close socket and TCP RST

close socket and TCP RST

[Celelibi]

Hello,

I'm not sure about the mailing list I should ask this. I would try
linux-net if it was still alive.

I spent a few days searching about a weird bug I had.
It is actually similar to this one:
http://marc.info/?l=linux-net&m=127651583824851&w=2 .

Summary for lazy people ^^: calling close(2) on a socket with a
non-empty receive kernel-buffer cause the connection to be ReSeT and
the send buffer discarded and not sent.


I have a server program (code included below)  that wait for a client.
When a client connect, the server:
- sent "Greet\n"
- read one char
- send "Hello "
- send "World\n"
- send "Quit\n"
- close the socket

And it happens that sometimes (quite often actually) that the client
does not recieve anything after "Hello " and the connection is just
closed.
Actually, after I dumped the TCP trafic with wireshark, I saw that
"World\n" and "Quit\n" were NOT sent, despite the send(2) succeded.
The server just send a packet with the RST flag to interrupt the
connection.
Even more strange: This only occurs when the client send more data to
the server than what was expected.
i.e.: the server read one byte, the client send one, the last messages
arrives just fine. The client send two bytes, the last messages never
arrives!

From what I intuited and understood from the Linux kernel code:
http://lxr.linux.no/linux+v3.3.1/net/ipv4/tcp.c#L1893
When the sever call close(2) on the socket file descriptor, the
connection is "reset" if the receive buffer was not empty. And in that
case, the output buffer is never sent to the client (whenever
SO_LINGER is set or not).


The workaround I found is to call shutdown(2) before calling close.
When the outgoing direction of the socket is shutdown the buffer is
flushed and sent, and it initiate a gentle connection ending. (While
the close still send a RST because of the non-empty input buffer.)

Therefore I have two questions:
1) Is this a standard behavior? Doesn't the RFC state that every
pending data is sent when the connection is closed?
2) Shouldn't that behavior be documented somewhere? I didn't found any
information about that anywhere. I looked at the man close(2),
shutdown(2), socket(7), tcp(7).

From this I deduce that shutdown must be called everytime we want to
close a socket. But this is not taught anywhere. :p



Here is the code of the server for those who want to try it. And since
it seems time related I also provide a client that exhibit the bug (at
least on my machine).

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netdb.h>



#define STRING_GREET "Greet\n"
#define STRING_HELLO "Hello "
#define STRING_WORLD "World"
#define STRING_QUIT  "Quit\n"




int create_socket(void) {
	struct addrinfo hints, *res, *rp;
	int sockfd = -1;
	int err;


	memset(&hints, 0, sizeof(hints));
	hints.ai_family = AF_UNSPEC;
	hints.ai_socktype = SOCK_STREAM;
	hints.ai_protocol = 0;
	hints.ai_flags = AI_V4MAPPED | AI_PASSIVE;

	err = getaddrinfo("127.0.0.1", "1337", &hints, &res);
	if (err) {
		fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(err));
		exit(EXIT_FAILURE);
	}

	for (rp = res; rp; rp = rp->ai_next) {
		int optval;

		sockfd = socket(rp->ai_family, rp->ai_socktype, rp->ai_protocol);
		if (sockfd == -1) {
			perror("socket");
			continue;
		}

		optval = 1;
		err = setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &optval, sizeof(optval));
		if (err == -1) {
			perror("setsockopt");
			close(sockfd);
			sockfd = -1;
			continue;
		}

		err = bind(sockfd, rp->ai_addr, rp->ai_addrlen);
		if (err == -1) {
			perror("bind");
			close(sockfd);
			sockfd = -1;
			continue;
		}

		break;
	}

	freeaddrinfo(res);

	if (sockfd == -1) {
		fprintf(stderr, "can't bind at all\n");
		exit(EXIT_FAILURE);
	}

	err = listen(sockfd, 5);
	if (err == -1) {
		perror("listen");
		close(sockfd);
		exit(EXIT_FAILURE);
	}

	return sockfd;
}



void handle_client(int sockfd, int s) {
	char c;
	int err;

	err = send(s, STRING_GREET, strlen(STRING_GREET), MSG_NOSIGNAL);
	if (err == -1) {
		perror("send");
		close(s);
		close(sockfd);
		exit(EXIT_FAILURE);
	}
	fprintf(stderr, "send(STRING_GREET) = %d\n", err);

	err = recv(s, &c, sizeof(c), 0);
	if (err == -1) {
		perror("recv");
		close(s);
		close(sockfd);
		exit(EXIT_FAILURE);
	}

	err = send(s, STRING_HELLO, strlen(STRING_HELLO), MSG_NOSIGNAL);
	if (err == -1) {
		perror("send");
		close(s);
		close(sockfd);
		exit(EXIT_FAILURE);
	}
	fprintf(stderr, "send(STRING_HELLO) = %d\n", err);

	err = send(s, STRING_WORLD, strlen(STRING_WORLD), MSG_NOSIGNAL);
	if (err == -1) {
		perror("send");
		close(s);
		close(sockfd);
		exit(EXIT_FAILURE);
	}
	fprintf(stderr, "send(STRING_WORLD) = %d\n", err);

	err = send(s, "\n", 1, 0);
	if (err == -1) {
		perror("send");
		close(s);
		close(sockfd);
		exit(EXIT_FAILURE);
	}
	fprintf(stderr, "send(\\n) = %d\n", err);

	err = send(s, STRING_QUIT, strlen(STRING_QUIT), MSG_NOSIGNAL);
	if (err == -1) {
		perror("send");
		close(s);
		close(sockfd);
		exit(EXIT_FAILURE);
	}
	fprintf(stderr, "send(STRING_QUIT) = %d\n", err);

	/*err = shutdown(s, SHUT_RDWR);
	if (err == -1) {
		perror("shutdown");
		close(sockfd);
		exit(EXIT_FAILURE);
	}*/

	err = close(s);
	if (err == -1) {
		perror("close");
		close(sockfd);
		exit(EXIT_FAILURE);
	}
}



int main(void) {
	struct linger lin;
	int sockfd;
	int err;

	sockfd = create_socket();

	lin.l_onoff = 1;
	lin.l_linger = 1;
	err = setsockopt(sockfd, SOL_SOCKET, SO_LINGER, &lin, sizeof(lin));
	if (err == -1) {
		perror("setsockopt(SO_LINGER)");
		close(sockfd);
		exit(EXIT_FAILURE);
	}


	while (1) {
		int s;

		s = accept(sockfd, NULL, NULL);
		if (s == -1) {
			perror("accept");
			close(sockfd);
			exit(EXIT_FAILURE);
		}

		handle_client(sockfd, s);
	}

	err = close(sockfd);
	if (err == -1) {
		perror("close");
		exit(EXIT_FAILURE);
	}

	return EXIT_SUCCESS;
}








The client:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netdb.h>


#define MIN(a, b) (((a) < (b)) ? (a) : (b))


size_t readline(int fd, unsigned char *out, size_t out_size) {
	static unsigned char buff[8 * 1024];
	static size_t data_buff = 0;
	size_t retval = 0;
	ssize_t nr;

	while (1) {
		unsigned char *nl;


		/* Is there already a \n in the buffer? */
		nl = memchr(buff, '\n', data_buff);
		if (nl) {
			size_t nc = MIN(out_size, nl - buff + 1UL);
			memcpy(out, buff, nc);
			retval += nc;
			memmove(buff, buff + nc, data_buff - nc);
			data_buff -= nc;
			break;
		} else {
			/* No \n found */
			if (data_buff >= out_size) {
				/* No space left in the out buffer */
				memcpy(out, buff, out_size);
				retval += out_size;
				memmove(buff, buff + out_size, data_buff - out_size);
				data_buff -= out_size;
				break;
			} else {
				/* No \n and some space left in the out buffer.
				 * copy _ALL_ the buffer! */
				memcpy(out, buff, data_buff);
				retval += data_buff;
				out += data_buff;
				out_size -= data_buff;
				data_buff = 0;
			}
		}


		nr = recv(fd, buff, sizeof(buff), 0);

		/* No matter the errors, we have data! */
		if (nr == -1 && retval != 0)
			break;

		if (nr == -1) {
			perror("recv");
			close(fd);
			exit(EXIT_FAILURE);
		}

		if (nr == 0)
			break;

		data_buff = nr;
	}

	return retval;
}



int main(void) {
	struct addrinfo hints, *res, *rp;
	int sockfd = -1;
	int err;
	unsigned char buff[8 * 1024];
	size_t line_size = 0;


	memset(&hints, 0, sizeof(hints));
	hints.ai_family = AF_INET;
	hints.ai_socktype = SOCK_STREAM;
	hints.ai_protocol = 0;
	hints.ai_flags = AI_V4MAPPED;

	err = getaddrinfo("127.0.0.1", "1337", &hints, &res);
	if (err) {
		fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(err));
		exit(EXIT_FAILURE);
	}

	for (rp = res; rp; rp = rp->ai_next) {
		sockfd = socket(rp->ai_family, rp->ai_socktype, rp->ai_protocol);
		if (sockfd == -1) {
			perror("socket");
			continue;
		}

		err = connect(sockfd, rp->ai_addr, rp->ai_addrlen);
		if (err == -1) {
			perror("connect");
			close(sockfd);
			sockfd = -1;
			continue;
		}

		break;
	}

	freeaddrinfo(res);

	if (sockfd == -1) {
		fprintf(stderr, "can't connect at all\n");
		exit(EXIT_FAILURE);
	}

	line_size = readline(sockfd, buff, sizeof(buff));
	write(STDOUT_FILENO, buff, line_size);

	err = send(sockfd, "XX", 2, 0);
	if (err == -1) {
		perror("send");
		close(sockfd);
		exit(EXIT_FAILURE);
	}

	line_size = readline(sockfd, buff, sizeof(buff));
	write(STDOUT_FILENO, buff, line_size);

	line_size = readline(sockfd, buff, sizeof(buff));
	write(STDOUT_FILENO, buff, line_size);

	err = close(sockfd);
	if (err == -1) {
		perror("close");
		exit(EXIT_FAILURE);
	}

	return EXIT_SUCCESS;
}


Thanks for reading that whole long message.

Celelibi

Re: close socket and TCP RST

[Bogdan Cristea]

On Tuesday 10 April 2012 01:46:45 you wrote:
> Hello,
> 
> I'm not sure about the mailing list I should ask this. I would try
> linux-net if it was still alive.
> 
> I spent a few days searching about a weird bug I had.
> It is actually similar to this one:
> http://marc.info/?l=linux-net&m=127651583824851&w=2 .
> 
> Summary for lazy people ^^: calling close(2) on a socket with a
> non-empty receive kernel-buffer cause the connection to be ReSeT and
> the send buffer discarded and not sent.
> 
> 
> I have a server program (code included below)  that wait for a client.
> When a client connect, the server:
> - sent "Greet\n"
> - read one char
> - send "Hello "
> - send "World\n"
> - send "Quit\n"
> - close the socket
> 
> And it happens that sometimes (quite often actually) that the client
> does not recieve anything after "Hello " and the connection is just
> closed.
> Actually, after I dumped the TCP trafic with wireshark, I saw that
> "World\n" and "Quit\n" were NOT sent, despite the send(2) succeded.
> The server just send a packet with the RST flag to interrupt the
> connection.
> Even more strange: This only occurs when the client send more data to
> the server than what was expected.
> i.e.: the server read one byte, the client send one, the last messages
> arrives just fine. The client send two bytes, the last messages never
> arrives!
> 
> From what I intuited and understood from the Linux kernel code:
> http://lxr.linux.no/linux+v3.3.1/net/ipv4/tcp.c#L1893
> When the sever call close(2) on the socket file descriptor, the
> connection is "reset" if the receive buffer was not empty. And in that
> case, the output buffer is never sent to the client (whenever
> SO_LINGER is set or not).
> 
> 
> The workaround I found is to call shutdown(2) before calling close.
> When the outgoing direction of the socket is shutdown the buffer is
> flushed and sent, and it initiate a gentle connection ending. (While
> the close still send a RST because of the non-empty input buffer.)
> 
> Therefore I have two questions:
> 1) Is this a standard behavior? Doesn't the RFC state that every
> pending data is sent when the connection is closed?
> 2) Shouldn't that behavior be documented somewhere? I didn't found any
> information about that anywhere. I looked at the man close(2),
> shutdown(2), socket(7), tcp(7).
> 
> From this I deduce that shutdown must be called everytime we want to
> close a socket. But this is not taught anywhere. :p
> 
> 
> 
> Here is the code of the server for those who want to try it. And since
> it seems time related I also provide a client that exhibit the bug (at
> least on my machine).
> 
> #include <stdio.h>
> #include <stdlib.h>
> #include <unistd.h>
> #include <string.h>
> #include <sys/types.h>
> #include <sys/socket.h>
> #include <netdb.h>
> 
> 
> 
> #define STRING_GREET "Greet\n"
> #define STRING_HELLO "Hello "
> #define STRING_WORLD "World"
> #define STRING_QUIT  "Quit\n"
> 
> 
> 
> 
> int create_socket(void) {
> 	struct addrinfo hints, *res, *rp;
> 	int sockfd = -1;
> 	int err;
> 
> 
> 	memset(&hints, 0, sizeof(hints));
> 	hints.ai_family = AF_UNSPEC;
> 	hints.ai_socktype = SOCK_STREAM;
> 	hints.ai_protocol = 0;
> 	hints.ai_flags = AI_V4MAPPED | AI_PASSIVE;
> 
> 	err = getaddrinfo("127.0.0.1", "1337", &hints, &res);
> 	if (err) {
> 		fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(err));
> 		exit(EXIT_FAILURE);
> 	}
> 
> 	for (rp = res; rp; rp = rp->ai_next) {
> 		int optval;
> 
> 		sockfd = socket(rp->ai_family, rp->ai_socktype, rp->ai_protocol);
> 		if (sockfd == -1) {
> 			perror("socket");
> 			continue;
> 		}
> 
> 		optval = 1;
> 		err = setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &optval,
> sizeof(optval)); if (err == -1) {
> 			perror("setsockopt");
> 			close(sockfd);
> 			sockfd = -1;
> 			continue;
> 		}
> 
> 		err = bind(sockfd, rp->ai_addr, rp->ai_addrlen);
> 		if (err == -1) {
> 			perror("bind");
> 			close(sockfd);
> 			sockfd = -1;
> 			continue;
> 		}
> 
> 		break;
> 	}
> 
> 	freeaddrinfo(res);
> 
> 	if (sockfd == -1) {
> 		fprintf(stderr, "can't bind at all\n");
> 		exit(EXIT_FAILURE);
> 	}
> 
> 	err = listen(sockfd, 5);
> 	if (err == -1) {
> 		perror("listen");
> 		close(sockfd);
> 		exit(EXIT_FAILURE);
> 	}
> 
> 	return sockfd;
> }
> 
> 
> 
> void handle_client(int sockfd, int s) {
> 	char c;
> 	int err;
> 
> 	err = send(s, STRING_GREET, strlen(STRING_GREET), MSG_NOSIGNAL);
> 	if (err == -1) {
> 		perror("send");
> 		close(s);
> 		close(sockfd);
> 		exit(EXIT_FAILURE);
> 	}
> 	fprintf(stderr, "send(STRING_GREET) = %d\n", err);
> 
> 	err = recv(s, &c, sizeof(c), 0);
> 	if (err == -1) {
> 		perror("recv");
> 		close(s);
> 		close(sockfd);
> 		exit(EXIT_FAILURE);
> 	}
> 
> 	err = send(s, STRING_HELLO, strlen(STRING_HELLO), MSG_NOSIGNAL);
> 	if (err == -1) {
> 		perror("send");
> 		close(s);
> 		close(sockfd);
> 		exit(EXIT_FAILURE);
> 	}
> 	fprintf(stderr, "send(STRING_HELLO) = %d\n", err);
> 
> 	err = send(s, STRING_WORLD, strlen(STRING_WORLD), MSG_NOSIGNAL);
> 	if (err == -1) {
> 		perror("send");
> 		close(s);
> 		close(sockfd);
> 		exit(EXIT_FAILURE);
> 	}
> 	fprintf(stderr, "send(STRING_WORLD) = %d\n", err);
> 
> 	err = send(s, "\n", 1, 0);
> 	if (err == -1) {
> 		perror("send");
> 		close(s);
> 		close(sockfd);
> 		exit(EXIT_FAILURE);
> 	}
> 	fprintf(stderr, "send(\\n) = %d\n", err);
> 
> 	err = send(s, STRING_QUIT, strlen(STRING_QUIT), MSG_NOSIGNAL);
> 	if (err == -1) {
> 		perror("send");
> 		close(s);
> 		close(sockfd);
> 		exit(EXIT_FAILURE);
> 	}
> 	fprintf(stderr, "send(STRING_QUIT) = %d\n", err);
> 
> 	/*err = shutdown(s, SHUT_RDWR);
> 	if (err == -1) {
> 		perror("shutdown");
> 		close(sockfd);
> 		exit(EXIT_FAILURE);
> 	}*/
> 
> 	err = close(s);
> 	if (err == -1) {
> 		perror("close");
> 		close(sockfd);
> 		exit(EXIT_FAILURE);
> 	}
> }
> 
> 
> 
> int main(void) {
> 	struct linger lin;
> 	int sockfd;
> 	int err;
> 
> 	sockfd = create_socket();
> 
> 	lin.l_onoff = 1;
> 	lin.l_linger = 1;
> 	err = setsockopt(sockfd, SOL_SOCKET, SO_LINGER, &lin, sizeof(lin));
> 	if (err == -1) {
> 		perror("setsockopt(SO_LINGER)");
> 		close(sockfd);
> 		exit(EXIT_FAILURE);
> 	}
> 
> 
> 	while (1) {
> 		int s;
> 
> 		s = accept(sockfd, NULL, NULL);
> 		if (s == -1) {
> 			perror("accept");
> 			close(sockfd);
> 			exit(EXIT_FAILURE);
> 		}
> 
> 		handle_client(sockfd, s);
> 	}
> 
> 	err = close(sockfd);
> 	if (err == -1) {
> 		perror("close");
> 		exit(EXIT_FAILURE);
> 	}
> 
> 	return EXIT_SUCCESS;
> }
> 
> 
> 
> 
> 
> 
> 
> 
> The client:
> 
> #include <stdio.h>
> #include <stdlib.h>
> #include <unistd.h>
> #include <string.h>
> #include <sys/types.h>
> #include <sys/socket.h>
> #include <netdb.h>
> 
> 
> #define MIN(a, b) (((a) < (b)) ? (a) : (b))
> 
> 
> size_t readline(int fd, unsigned char *out, size_t out_size) {
> 	static unsigned char buff[8 * 1024];
> 	static size_t data_buff = 0;
> 	size_t retval = 0;
> 	ssize_t nr;
> 
> 	while (1) {
> 		unsigned char *nl;
> 
> 
> 		/* Is there already a \n in the buffer? */
> 		nl = memchr(buff, '\n', data_buff);
> 		if (nl) {
> 			size_t nc = MIN(out_size, nl - buff + 1UL);
> 			memcpy(out, buff, nc);
> 			retval += nc;
> 			memmove(buff, buff + nc, data_buff - nc);
> 			data_buff -= nc;
> 			break;
> 		} else {
> 			/* No \n found */
> 			if (data_buff >= out_size) {
> 				/* No space left in the out buffer */
> 				memcpy(out, buff, out_size);
> 				retval += out_size;
> 				memmove(buff, buff + out_size, data_buff - out_size);
> 				data_buff -= out_size;
> 				break;
> 			} else {
> 				/* No \n and some space left in the out buffer.
> 				 * copy _ALL_ the buffer! */
> 				memcpy(out, buff, data_buff);
> 				retval += data_buff;
> 				out += data_buff;
> 				out_size -= data_buff;
> 				data_buff = 0;
> 			}
> 		}
> 
> 
> 		nr = recv(fd, buff, sizeof(buff), 0);
> 
> 		/* No matter the errors, we have data! */
> 		if (nr == -1 && retval != 0)
> 			break;
> 
> 		if (nr == -1) {
> 			perror("recv");
> 			close(fd);
> 			exit(EXIT_FAILURE);
> 		}
> 
> 		if (nr == 0)
> 			break;
> 
> 		data_buff = nr;
> 	}
> 
> 	return retval;
> }
> 
> 
> 
> int main(void) {
> 	struct addrinfo hints, *res, *rp;
> 	int sockfd = -1;
> 	int err;
> 	unsigned char buff[8 * 1024];
> 	size_t line_size = 0;
> 
> 
> 	memset(&hints, 0, sizeof(hints));
> 	hints.ai_family = AF_INET;
> 	hints.ai_socktype = SOCK_STREAM;
> 	hints.ai_protocol = 0;
> 	hints.ai_flags = AI_V4MAPPED;
> 
> 	err = getaddrinfo("127.0.0.1", "1337", &hints, &res);
> 	if (err) {
> 		fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(err));
> 		exit(EXIT_FAILURE);
> 	}
> 
> 	for (rp = res; rp; rp = rp->ai_next) {
> 		sockfd = socket(rp->ai_family, rp->ai_socktype, rp->ai_protocol);
> 		if (sockfd == -1) {
> 			perror("socket");
> 			continue;
> 		}
> 
> 		err = connect(sockfd, rp->ai_addr, rp->ai_addrlen);
> 		if (err == -1) {
> 			perror("connect");
> 			close(sockfd);
> 			sockfd = -1;
> 			continue;
> 		}
> 
> 		break;
> 	}
> 
> 	freeaddrinfo(res);
> 
> 	if (sockfd == -1) {
> 		fprintf(stderr, "can't connect at all\n");
> 		exit(EXIT_FAILURE);
> 	}
> 
> 	line_size = readline(sockfd, buff, sizeof(buff));
> 	write(STDOUT_FILENO, buff, line_size);
> 
> 	err = send(sockfd, "XX", 2, 0);
> 	if (err == -1) {
> 		perror("send");
> 		close(sockfd);
> 		exit(EXIT_FAILURE);
> 	}
> 
> 	line_size = readline(sockfd, buff, sizeof(buff));
> 	write(STDOUT_FILENO, buff, line_size);
> 
> 	line_size = readline(sockfd, buff, sizeof(buff));
> 	write(STDOUT_FILENO, buff, line_size);
> 
> 	err = close(sockfd);
> 	if (err == -1) {
> 		perror("close");
> 		exit(EXIT_FAILURE);
> 	}
> 
> 	return EXIT_SUCCESS;
> }
> 
> 
> Thanks for reading that whole long message.
> 
> Celelibi
> --
> To unsubscribe from this list: send the line "unsubscribe
> linux-c-programming" in the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hi

It seems indeed a little strange that with SO_LINGER option you cannot send 
everything. Anyway, at the first look it seems that you have several 
programming errors:
- socket ops for the server should be set before starting to listen on the 
socket
-in the main function you never exit from the while loop, so why bother to 
close the socket after this loop ?

Have a closer look at your code, there might be a bug in your program.

regards
-- 
Bogdan

Re: close socket and TCP RST

[Celelibi]

2012/4/10, Bogdan Cristea <cristeab@gmail.com>:
> On Tuesday 10 April 2012 01:46:45 you wrote:
>> Hello,
>>
>> I'm not sure about the mailing list I should ask this. I would try
>> linux-net if it was still alive.
>>
>> I spent a few days searching about a weird bug I had.
>> It is actually similar to this one:
>> http://marc.info/?l=linux-net&m=127651583824851&w=2 .
>>
>> Summary for lazy people ^^: calling close(2) on a socket with a
>> non-empty receive kernel-buffer cause the connection to be ReSeT and
>> the send buffer discarded and not sent.
>>
>>
>> I have a server program (code included below)  that wait for a client.
>> When a client connect, the server:
>> - sent "Greet\n"
>> - read one char
>> - send "Hello "
>> - send "World\n"
>> - send "Quit\n"
>> - close the socket
>>
>> And it happens that sometimes (quite often actually) that the client
>> does not recieve anything after "Hello " and the connection is just
>> closed.
>> Actually, after I dumped the TCP trafic with wireshark, I saw that
>> "World\n" and "Quit\n" were NOT sent, despite the send(2) succeded.
>> The server just send a packet with the RST flag to interrupt the
>> connection.
>> Even more strange: This only occurs when the client send more data to
>> the server than what was expected.
>> i.e.: the server read one byte, the client send one, the last messages
>> arrives just fine. The client send two bytes, the last messages never
>> arrives!
>>
>> From what I intuited and understood from the Linux kernel code:
>> http://lxr.linux.no/linux+v3.3.1/net/ipv4/tcp.c#L1893
>> When the sever call close(2) on the socket file descriptor, the
>> connection is "reset" if the receive buffer was not empty. And in that
>> case, the output buffer is never sent to the client (whenever
>> SO_LINGER is set or not).
>>
>>
>> The workaround I found is to call shutdown(2) before calling close.
>> When the outgoing direction of the socket is shutdown the buffer is
>> flushed and sent, and it initiate a gentle connection ending. (While
>> the close still send a RST because of the non-empty input buffer.)
>>
>> Therefore I have two questions:
>> 1) Is this a standard behavior? Doesn't the RFC state that every
>> pending data is sent when the connection is closed?
>> 2) Shouldn't that behavior be documented somewhere? I didn't found any
>> information about that anywhere. I looked at the man close(2),
>> shutdown(2), socket(7), tcp(7).
>>
>> From this I deduce that shutdown must be called everytime we want to
>> close a socket. But this is not taught anywhere. :p
>>
>>
>>
>> Here is the code of the server for those who want to try it. And since
>> it seems time related I also provide a client that exhibit the bug (at
>> least on my machine).
>>
>> #include <stdio.h>
>> #include <stdlib.h>
>> #include <unistd.h>
>> #include <string.h>
>> #include <sys/types.h>
>> #include <sys/socket.h>
>> #include <netdb.h>
>>
>>
>>
>> #define STRING_GREET "Greet\n"
>> #define STRING_HELLO "Hello "
>> #define STRING_WORLD "World"
>> #define STRING_QUIT  "Quit\n"
>>
>>
>>
>>
>> int create_socket(void) {
>> 	struct addrinfo hints, *res, *rp;
>> 	int sockfd = -1;
>> 	int err;
>>
>>
>> 	memset(&hints, 0, sizeof(hints));
>> 	hints.ai_family = AF_UNSPEC;
>> 	hints.ai_socktype = SOCK_STREAM;
>> 	hints.ai_protocol = 0;
>> 	hints.ai_flags = AI_V4MAPPED | AI_PASSIVE;
>>
>> 	err = getaddrinfo("127.0.0.1", "1337", &hints, &res);
>> 	if (err) {
>> 		fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(err));
>> 		exit(EXIT_FAILURE);
>> 	}
>>
>> 	for (rp = res; rp; rp = rp->ai_next) {
>> 		int optval;
>>
>> 		sockfd = socket(rp->ai_family, rp->ai_socktype, rp->ai_protocol);
>> 		if (sockfd == -1) {
>> 			perror("socket");
>> 			continue;
>> 		}
>>
>> 		optval = 1;
>> 		err = setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &optval,
>> sizeof(optval)); if (err == -1) {
>> 			perror("setsockopt");
>> 			close(sockfd);
>> 			sockfd = -1;
>> 			continue;
>> 		}
>>
>> 		err = bind(sockfd, rp->ai_addr, rp->ai_addrlen);
>> 		if (err == -1) {
>> 			perror("bind");
>> 			close(sockfd);
>> 			sockfd = -1;
>> 			continue;
>> 		}
>>
>> 		break;
>> 	}
>>
>> 	freeaddrinfo(res);
>>
>> 	if (sockfd == -1) {
>> 		fprintf(stderr, "can't bind at all\n");
>> 		exit(EXIT_FAILURE);
>> 	}
>>
>> 	err = listen(sockfd, 5);
>> 	if (err == -1) {
>> 		perror("listen");
>> 		close(sockfd);
>> 		exit(EXIT_FAILURE);
>> 	}
>>
>> 	return sockfd;
>> }
>>
>>
>>
>> void handle_client(int sockfd, int s) {
>> 	char c;
>> 	int err;
>>
>> 	err = send(s, STRING_GREET, strlen(STRING_GREET), MSG_NOSIGNAL);
>> 	if (err == -1) {
>> 		perror("send");
>> 		close(s);
>> 		close(sockfd);
>> 		exit(EXIT_FAILURE);
>> 	}
>> 	fprintf(stderr, "send(STRING_GREET) = %d\n", err);
>>
>> 	err = recv(s, &c, sizeof(c), 0);
>> 	if (err == -1) {
>> 		perror("recv");
>> 		close(s);
>> 		close(sockfd);
>> 		exit(EXIT_FAILURE);
>> 	}
>>
>> 	err = send(s, STRING_HELLO, strlen(STRING_HELLO), MSG_NOSIGNAL);
>> 	if (err == -1) {
>> 		perror("send");
>> 		close(s);
>> 		close(sockfd);
>> 		exit(EXIT_FAILURE);
>> 	}
>> 	fprintf(stderr, "send(STRING_HELLO) = %d\n", err);
>>
>> 	err = send(s, STRING_WORLD, strlen(STRING_WORLD), MSG_NOSIGNAL);
>> 	if (err == -1) {
>> 		perror("send");
>> 		close(s);
>> 		close(sockfd);
>> 		exit(EXIT_FAILURE);
>> 	}
>> 	fprintf(stderr, "send(STRING_WORLD) = %d\n", err);
>>
>> 	err = send(s, "\n", 1, 0);
>> 	if (err == -1) {
>> 		perror("send");
>> 		close(s);
>> 		close(sockfd);
>> 		exit(EXIT_FAILURE);
>> 	}
>> 	fprintf(stderr, "send(\\n) = %d\n", err);
>>
>> 	err = send(s, STRING_QUIT, strlen(STRING_QUIT), MSG_NOSIGNAL);
>> 	if (err == -1) {
>> 		perror("send");
>> 		close(s);
>> 		close(sockfd);
>> 		exit(EXIT_FAILURE);
>> 	}
>> 	fprintf(stderr, "send(STRING_QUIT) = %d\n", err);
>>
>> 	/*err = shutdown(s, SHUT_RDWR);
>> 	if (err == -1) {
>> 		perror("shutdown");
>> 		close(sockfd);
>> 		exit(EXIT_FAILURE);
>> 	}*/
>>
>> 	err = close(s);
>> 	if (err == -1) {
>> 		perror("close");
>> 		close(sockfd);
>> 		exit(EXIT_FAILURE);
>> 	}
>> }
>>
>>
>>
>> int main(void) {
>> 	struct linger lin;
>> 	int sockfd;
>> 	int err;
>>
>> 	sockfd = create_socket();
>>
>> 	lin.l_onoff = 1;
>> 	lin.l_linger = 1;
>> 	err = setsockopt(sockfd, SOL_SOCKET, SO_LINGER, &lin, sizeof(lin));
>> 	if (err == -1) {
>> 		perror("setsockopt(SO_LINGER)");
>> 		close(sockfd);
>> 		exit(EXIT_FAILURE);
>> 	}
>>
>>
>> 	while (1) {
>> 		int s;
>>
>> 		s = accept(sockfd, NULL, NULL);
>> 		if (s == -1) {
>> 			perror("accept");
>> 			close(sockfd);
>> 			exit(EXIT_FAILURE);
>> 		}
>>
>> 		handle_client(sockfd, s);
>> 	}
>>
>> 	err = close(sockfd);
>> 	if (err == -1) {
>> 		perror("close");
>> 		exit(EXIT_FAILURE);
>> 	}
>>
>> 	return EXIT_SUCCESS;
>> }
>>
>>
>>
>>
>>
>>
>>
>>
>> The client:
>>
>> #include <stdio.h>
>> #include <stdlib.h>
>> #include <unistd.h>
>> #include <string.h>
>> #include <sys/types.h>
>> #include <sys/socket.h>
>> #include <netdb.h>
>>
>>
>> #define MIN(a, b) (((a) < (b)) ? (a) : (b))
>>
>>
>> size_t readline(int fd, unsigned char *out, size_t out_size) {
>> 	static unsigned char buff[8 * 1024];
>> 	static size_t data_buff = 0;
>> 	size_t retval = 0;
>> 	ssize_t nr;
>>
>> 	while (1) {
>> 		unsigned char *nl;
>>
>>
>> 		/* Is there already a \n in the buffer? */
>> 		nl = memchr(buff, '\n', data_buff);
>> 		if (nl) {
>> 			size_t nc = MIN(out_size, nl - buff + 1UL);
>> 			memcpy(out, buff, nc);
>> 			retval += nc;
>> 			memmove(buff, buff + nc, data_buff - nc);
>> 			data_buff -= nc;
>> 			break;
>> 		} else {
>> 			/* No \n found */
>> 			if (data_buff >= out_size) {
>> 				/* No space left in the out buffer */
>> 				memcpy(out, buff, out_size);
>> 				retval += out_size;
>> 				memmove(buff, buff + out_size, data_buff - out_size);
>> 				data_buff -= out_size;
>> 				break;
>> 			} else {
>> 				/* No \n and some space left in the out buffer.
>> 				 * copy _ALL_ the buffer! */
>> 				memcpy(out, buff, data_buff);
>> 				retval += data_buff;
>> 				out += data_buff;
>> 				out_size -= data_buff;
>> 				data_buff = 0;
>> 			}
>> 		}
>>
>>
>> 		nr = recv(fd, buff, sizeof(buff), 0);
>>
>> 		/* No matter the errors, we have data! */
>> 		if (nr == -1 && retval != 0)
>> 			break;
>>
>> 		if (nr == -1) {
>> 			perror("recv");
>> 			close(fd);
>> 			exit(EXIT_FAILURE);
>> 		}
>>
>> 		if (nr == 0)
>> 			break;
>>
>> 		data_buff = nr;
>> 	}
>>
>> 	return retval;
>> }
>>
>>
>>
>> int main(void) {
>> 	struct addrinfo hints, *res, *rp;
>> 	int sockfd = -1;
>> 	int err;
>> 	unsigned char buff[8 * 1024];
>> 	size_t line_size = 0;
>>
>>
>> 	memset(&hints, 0, sizeof(hints));
>> 	hints.ai_family = AF_INET;
>> 	hints.ai_socktype = SOCK_STREAM;
>> 	hints.ai_protocol = 0;
>> 	hints.ai_flags = AI_V4MAPPED;
>>
>> 	err = getaddrinfo("127.0.0.1", "1337", &hints, &res);
>> 	if (err) {
>> 		fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(err));
>> 		exit(EXIT_FAILURE);
>> 	}
>>
>> 	for (rp = res; rp; rp = rp->ai_next) {
>> 		sockfd = socket(rp->ai_family, rp->ai_socktype, rp->ai_protocol);
>> 		if (sockfd == -1) {
>> 			perror("socket");
>> 			continue;
>> 		}
>>
>> 		err = connect(sockfd, rp->ai_addr, rp->ai_addrlen);
>> 		if (err == -1) {
>> 			perror("connect");
>> 			close(sockfd);
>> 			sockfd = -1;
>> 			continue;
>> 		}
>>
>> 		break;
>> 	}
>>
>> 	freeaddrinfo(res);
>>
>> 	if (sockfd == -1) {
>> 		fprintf(stderr, "can't connect at all\n");
>> 		exit(EXIT_FAILURE);
>> 	}
>>
>> 	line_size = readline(sockfd, buff, sizeof(buff));
>> 	write(STDOUT_FILENO, buff, line_size);
>>
>> 	err = send(sockfd, "XX", 2, 0);
>> 	if (err == -1) {
>> 		perror("send");
>> 		close(sockfd);
>> 		exit(EXIT_FAILURE);
>> 	}
>>
>> 	line_size = readline(sockfd, buff, sizeof(buff));
>> 	write(STDOUT_FILENO, buff, line_size);
>>
>> 	line_size = readline(sockfd, buff, sizeof(buff));
>> 	write(STDOUT_FILENO, buff, line_size);
>>
>> 	err = close(sockfd);
>> 	if (err == -1) {
>> 		perror("close");
>> 		exit(EXIT_FAILURE);
>> 	}
>>
>> 	return EXIT_SUCCESS;
>> }
>>
>>
>> Thanks for reading that whole long message.
>>
>> Celelibi
>> --
>> To unsubscribe from this list: send the line "unsubscribe
>> linux-c-programming" in the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> Hi
>
> It seems indeed a little strange that with SO_LINGER option you cannot send
> everything. Anyway, at the first look it seems that you have several
> programming errors:
> - socket ops for the server should be set before starting to listen on the
> socket
> -in the main function you never exit from the while loop, so why bother to
> close the socket after this loop ?
>
> Have a closer look at your code, there might be a bug in your program.
>
> regards
> --
> Bogdan
>


Hello, thanks for reading my long message.
By the way, I'm not sure I successfully subscribed to the mailing list
since your mail hadn't a List-Id header, and I do not have a majordomo
signature added.

My bad for the setsockopt. But wherever I put it, I see no change. And
I notice that everywhere I read some information about SO_LINGER, it
says that it is a way to LOSE data, and that by default the TCP will
try its best to deliver the data. I tried to move the setsockopt to
just after the socket() call. Changed nothing.

The close after the infinite loop is here because at first the server
exited after one client only. That's just a toy program to show the
case.

And as most part of my student had this bug last year, I guess this is
not a programming error.
I think the bug is most likely in:
- My understanding of the RFC + a lack of documentation
- Linux kernel
- Something I didn't think about.

But don't you agree this is a strange behavior that when send and
receive buffer are non-empty when I call close, the send buffer is not
actually sent?


Celelibi

Re: close socket and TCP RST

[Glynn Clements]

Celelibi wrote:

> Summary for lazy people ^^: calling close(2) on a socket with a
> non-empty receive kernel-buffer cause the connection to be ReSeT and
> the send buffer discarded and not sent.

Yes; this is all as it should be.

> 1) Is this a standard behavior?

Yes.

> Doesn't the RFC state that every pending data is sent when the
> connection is closed?

The RFCs describe the TCP protocol, not the sockets API.

> 2) Shouldn't that behavior be documented somewhere? I didn't found any
> information about that anywhere. I looked at the man close(2),
> shutdown(2), socket(7), tcp(7).
>
> >From this I deduce that shutdown must be called everytime we want to
> close a socket. But this is not taught anywhere. :p

In many cases, shutdown() is not necessary. Normally, one side knows
whether the other side will send more data. E.g. for (non-pipelined)
HTTP, the client sends a request, the server sends a response, then
closes the connection. At that point, the client sees EOF then
close()s the socket (or it could just close the socket once the amount
of data specified by the Content-Length header has been received).

With a request-response protocol, either the requestor sends a "quit"
command resulting in the responder closing the connection, or the
requestor will just close the connection instead of issuing a request. 
In the latter case, it will either perform a half-close or just wait
until any outstanding response has been received and perform a
full-close.

If you close the receive side of the connection while the other end is
still sending, the kernel needs to inform the sender that data was
discarded (analogous to EPIPE for a pipe). It does so by sending a
RST. A FIN merely indicates that it has ceased sending data; a RST
asserts that the connection no longer exists.

Once it has sent a RST, it cannot send any additional data. Doing so
would just result in the receiver discarding the data and sending a
RST, so there's no point.

If you want the other end to see EOF while your end still receives
data, use shutdown(fd, SHUT_WR) to perform a half-close. This sends a
FIN and effectively makes the descriptor read-only.

The classic example of a half-close is for the rsh protocol, where
each side transmits independently and the format of the data is
unknown to either the client or the server. If the user types Ctrl-D
(or whatever the EOF character is), the rsh client receives EOF which
needs to be passed to the server, which is done using a half-close. 
The server then closes the the descriptor used to write to the pty
master, which causes the shell to read EOF from the slave. Once all
processes writing to the slave have terminated, rshd reads EOF from
the master, closes the socket, server sends FIN to the client, which
the rsh client sees as EOF, at which point it terminates.

-- 
Glynn Clements <glynn@gclements.plus.com>

Re: close socket and TCP RST

[Bogdan Cristea]

On Wednesday 11 April 2012 14:33:37 you wrote:
> In many cases, shutdown() is not necessary. Normally, one side knows
> whether the other side will send more data. E.g. for (non-pipelined)
> HTTP, the client sends a request, the server sends a response, then
> closes the connection.

It is exactly what he does, but the question is how to close the connection so 
that the client receives the last message. 
He is using for this:

 err = setsockopt(sockfd, SOL_SOCKET, SO_LINGER, &lin, sizeof(lin));

but for some reason it does not work as it should.
-- 
Bogdan 

Re: close socket and TCP RST

[Glynn Clements]

Bogdan Cristea wrote:

> > In many cases, shutdown() is not necessary. Normally, one side knows
> > whether the other side will send more data. E.g. for (non-pipelined)
> > HTTP, the client sends a request, the server sends a response, then
> > closes the connection.
> 
> It is exactly what he does, but the question is how to close the connection so 
> that the client receives the last message. 
> He is using for this:
> 
>  err = setsockopt(sockfd, SOL_SOCKET, SO_LINGER, &lin, sizeof(lin));
> 
> but for some reason it does not work as it should.

IIUC, he close()s a socket which has unread data. Whether the data had
already been received when close() was called or whether it arrived
afterwards doesn't matter.

Normal behaviour is simply not to use close() or shutdown(SHUT_RD) if
you expect to receive more data, e.g. wait until you have seen EOF
(i.e. read(), recv() etc return a zero count) before closing the read
side of the socket.

Current Linux behaviour is that receiving data on a close()ed socket
sends a RST. No data can be sent after a RST.

SO_LINGER doesn't affect this; it just affects whether close() or
shutdown(SHUT_WR) wait until the data has been sent (i.e. for the FIN
to be ACK'd).

AFAIK, there is no way to make close() or shutdown(SHUT_RD) silently
discard subsequent inbound data.

The exact behaviour of SHUT_RD isn't specified by any standard. The
RFCs don't deal with the sockets API, and POSIX just says "Disables
further receive operations". For Unix-domain sockets, a writer will
receive SIGPIPE/EPIPE; Linux' current behaviour for TCP sockets is at
least consistent with that.

Alternatives include:

1. ACK and discard the data. But then there would be no way for the
sender to identify that it's writing to a closed socket.

2. Do nothing. The receive buffer will fill, the window will close,
and the sender will block until someone kills it (it won't time out
because probes will still be met with an ACK of the last byte which
fitted into the buffer).

If the OP wants to send outstanding data, but doesn't want to wait for
EOF from the sender, the solution is to use SO_LINGER with a long
timeout and shutdown(SHUT_WR). The shutdown() won't return until the
FIN (and everything before it) has been ACK'd. At that point, he can
just close() the socket; presumably it won't matter if subsequent data
results in a RST (if it does matter, there is no alternative to
reading until EOF).

-- 
Glynn Clements <glynn@gclements.plus.com>

Re: close socket and TCP RST

[Celelibi]

2012/4/11, Glynn Clements <glynn@gclements.plus.com>:
>
> Celelibi wrote:
>
>> Summary for lazy people ^^: calling close(2) on a socket with a
>> non-empty receive kernel-buffer cause the connection to be ReSeT and
>> the send buffer discarded and not sent.
>
> Yes; this is all as it should be.
>
>> 1) Is this a standard behavior?
>
> Yes.
>
>> Doesn't the RFC state that every pending data is sent when the
>> connection is closed?
>
> The RFCs describe the TCP protocol, not the sockets API.
>

I agree with that. But I didn't find any information about the
berkeley socket API standard.
Then, I assume that the close() syscall close the file descriptor and
close the socket (both ways) if it was the last file descriptor for
that socket.

However, when I look at the TCP's RFC 793 (a.k.a STD7):
http://www.rfc-editor.org/rfc/std/std7.txt
Section 3.5 "Closing a Connection" says:
"A TCP will reliably deliver all buffers SENT before the connection was CLOSED"

And RFC 1122 ("host requirement"): http://www.rfc-editor.org/rfc/rfc1122.txt
Section 4.2.2.13 "Closing a Connection" says:
"If such a host issues a CLOSE call while received data is still
pending in TCP, or if new data is received after CLOSE is called, its
TCP SHOULD send a RST to show that data was lost."

IMO both are not incompatible and should be done. But I may have
missed something somewhere.


>> 2) Shouldn't that behavior be documented somewhere? I didn't found any
>> information about that anywhere. I looked at the man close(2),
>> shutdown(2), socket(7), tcp(7).
>>
>> >From this I deduce that shutdown must be called everytime we want to
>> close a socket. But this is not taught anywhere. :p
>
> In many cases, shutdown() is not necessary. Normally, one side knows
> whether the other side will send more data. E.g. for (non-pipelined)
> HTTP, the client sends a request, the server sends a response, then
> closes the connection. At that point, the client sees EOF then
> close()s the socket (or it could just close the socket once the amount
> of data specified by the Content-Length header has been received).

My point is that if the client send one byte more than needed in its
request (like ending the HTTP request with 3 "\n" instead of 2), the
client will never receive the last bytes of the server's response. But
this makes more sens with the a request-response protocol as I will
explain just below.

>
> With a request-response protocol, either the requestor sends a "quit"
> command resulting in the responder closing the connection, or the
> requestor will just close the connection instead of issuing a request.
> In the latter case, it will either perform a half-close or just wait
> until any outstanding response has been received and perform a
> full-close.

Well, this is close to the original protocol. It was a kind of SQL
server. Every command is terminated by a semi-colon while every "\n"
are treated as spaces.

The client send "QUIT;\n", the server read 'Q', 'U', 'I', 'T', ';',
process "QUIT", send to the client Something that say "Ok, I quit" and
close() the connection.
However, a "\n" is still in the receive buffer of the server! That
makes the server RST the connection ASAP and discard the messages that
were not sent yet... Namely the "Ok, I quit" message which was delayed
because of Nagle's algorithm or whatever.
Annoying isn't it?

I totally agree with the RST. If the client send "QUIT;\nSELECT 42;\n"
a RST have to be sent.
I just disagree with the fact that the last messages that were sent
before the close() are not sent at all despite the success of the
send() calls.

> Once it has sent a RST, it cannot send any additional data. Doing so
> would just result in the receiver discarding the data and sending a
> RST, so there's no point.

That's right. I want to send the data *before* the RST is sent since I
called send() *before* close().

>
> If you want the other end to see EOF while your end still receives
> data, use shutdown(fd, SHUT_WR) to perform a half-close. This sends a
> FIN and effectively makes the descriptor read-only.

I don't care about the remaining data, and don't want to read it.


Celelibi

Re: close socket and TCP RST

[Celelibi]

2012/4/12, Glynn Clements <glynn@gclements.plus.com>:
> IIUC, he close()s a socket which has unread data. Whether the data had
> already been received when close() was called or whether it arrived
> afterwards doesn't matter.

Right.

>
> Normal behaviour is simply not to use close() or shutdown(SHUT_RD) if
> you expect to receive more data, e.g. wait until you have seen EOF
> (i.e. read(), recv() etc return a zero count) before closing the read
> side of the socket.

I don't care about these data. I don't want to read them. If there are
too much data after the client sent a QUIT command, it have to receive
a RST.

> AFAIK, there is no way to make close() or shutdown(SHUT_RD) silently
> discard subsequent inbound data.

And that's ok because that's not what I want. :)

> Alternatives include:
>
> 1. ACK and discard the data. But then there would be no way for the
> sender to identify that it's writing to a closed socket.

Right.

>
> 2. Do nothing. The receive buffer will fill, the window will close,
> and the sender will block until someone kills it (it won't time out
> because probes will still be met with an ACK of the last byte which
> fitted into the buffer).
>
> If the OP wants to send outstanding data, but doesn't want to wait for
> EOF from the sender, the solution is to use SO_LINGER with a long
> timeout and shutdown(SHUT_WR). The shutdown() won't return until the
> FIN (and everything before it) has been ACK'd. At that point, he can
> just close() the socket; presumably it won't matter if subsequent data
> results in a RST (if it does matter, there is no alternative to
> reading until EOF).
>

Actually, shutdown(SHUT_RDWR) do the trick since shutdown(SHUT_WR)
forces the unsent data to be actually sent with a FIN flag.
And since shutdown(SHUT_RDWR) send the outstanding data, why don't
close() do the same? I don't get it.

And of course, calling close() after shutdown() send a RST because of
the unread data.


Celelibi

Re: close socket and TCP RST

[Glynn Clements]

Celelibi wrote:

> However, when I look at the TCP's RFC 793 (a.k.a STD7):
> http://www.rfc-editor.org/rfc/std/std7.txt
> Section 3.5 "Closing a Connection" says:
> "A TCP will reliably deliver all buffers SENT before the connection was CLOSED"
> 
> And RFC 1122 ("host requirement"): http://www.rfc-editor.org/rfc/rfc1122.txt
> Section 4.2.2.13 "Closing a Connection" says:
> "If such a host issues a CLOSE call while received data is still
> pending in TCP, or if new data is received after CLOSE is called, its
> TCP SHOULD send a RST to show that data was lost."
> 
> IMO both are not incompatible and should be done. But I may have
> missed something somewhere.

First, the RFC's "close" doesn't necessarily equate to the Unix
close() call.

The host can't send outstanding data after a RST has been sent (if it
did, the receiver would just reply with an RST rather than an ACK, it
the sender definitely shouldn't send data after an RST has been
received).

Sending outstanding data then sending the RST afterwards is
potentially problematic as the receiver may not be able or willing to
read the data until its own data has been ACK'd, which would result in
deadlock.

> > With a request-response protocol, either the requestor sends a "quit"
> > command resulting in the responder closing the connection, or the
> > requestor will just close the connection instead of issuing a request.
> > In the latter case, it will either perform a half-close or just wait
> > until any outstanding response has been received and perform a
> > full-close.
> 
> Well, this is close to the original protocol. It was a kind of SQL
> server. Every command is terminated by a semi-colon while every "\n"
> are treated as spaces.
> 
> The client send "QUIT;\n", the server read 'Q', 'U', 'I', 'T', ';',
> process "QUIT", send to the client Something that say "Ok, I quit" and
> close() the connection.
> However, a "\n" is still in the receive buffer of the server! That
> makes the server RST the connection ASAP and discard the messages that
> were not sent yet... Namely the "Ok, I quit" message which was delayed
> because of Nagle's algorithm or whatever.
> Annoying isn't it?

If the server sends a response to the quit command, the client should
read the response then close() the socket. Provided that the server's
response is well-formed, there shouldn't be a problem.

Ultimately, if neither side can be relied upon to strictly adhere to
the protocol, then at least one side must perform a half-close, and
each side must wait for EOF before closing the read side of the
connnection.

Alternatively, using SHUT_WR with SO_LINGER allows the sender to
ensure that any data is sent before closing the read side, so even if
the connection does end in a RST, sent data won't be lost.

However, this may be a moot point, as a connection terminated by RST
may result in close() indicating an error. For some protocols, that
will cause any "transaction" to be aborted; e.g. for POP3, any
messages marked for deletion by the DELE command should only be
deleted if the connection is terminated normally.

-- 
Glynn Clements <glynn@gclements.plus.com>

Re: close socket and TCP RST

[Celelibi]

2012/4/13, Glynn Clements <glynn@gclements.plus.com>:
> Sending outstanding data then sending the RST afterwards is
> potentially problematic as the receiver may not be able or willing to
> read the data until its own data has been ACK'd, which would result in
> deadlock.

Could you tell me more about that?
I can't see a case which would be problematic.
Moreover shutdown(SHUT_WR) can actually send the outstanding data.

>> The client send "QUIT;\n", the server read 'Q', 'U', 'I', 'T', ';',
>> process "QUIT", send to the client Something that say "Ok, I quit" and
>> close() the connection.
>> However, a "\n" is still in the receive buffer of the server! That
>> makes the server RST the connection ASAP and discard the messages that
>> were not sent yet... Namely the "Ok, I quit" message which was delayed
>> because of Nagle's algorithm or whatever.
>> Annoying isn't it?
>
> If the server sends a response to the quit command, the client should
> read the response then close() the socket. Provided that the server's
> response is well-formed, there shouldn't be a problem.

The client is ok. As soon as it receive "Ok, I quit" from the server,
it close the connection. Although the server should also work with
basic clients like netcat or telnet, which won't close the connection
by themselves. Then the server just has to send its last message and
close the connection. That's the protocol we expect:
The server read the command "QUIT;", send its last message saying it's
going to close the connection, and close the connection. The client is
quite passive in the termination process, it read the response to the
"QUIT;" and wait for the connection to be closed (its read should
return -1).

The problem is on the server side. Because of the unread "\n" (after
the "QUIT;"), "Ok, I quit" is never sent.


> Alternatively, using SHUT_WR with SO_LINGER allows the sender to
> ensure that any data is sent before closing the read side, so even if
> the connection does end in a RST, sent data won't be lost.
>
> However, this may be a moot point, as a connection terminated by RST
> may result in close() indicating an error. For some protocols, that
> will cause any "transaction" to be aborted; e.g. for POP3, any
> messages marked for deletion by the DELE command should only be
> deleted if the connection is terminated normally.

That's not a problem since that was a student project. (I'm a teaching
assistant and wrote a test script for their server.) But that's still
worth noticing that the way a connection terminate can be meaningful.

BTW, now I understand better what happen and why it happen, don't you
think this behavior should be documented somewhere? Should I report a
bug for the man 7 tcp?


Celelibi

Re: close socket and TCP RST

[Glynn Clements]

Celelibi wrote:

> > Sending outstanding data then sending the RST afterwards is
> > potentially problematic as the receiver may not be able or willing to
> > read the data until its own data has been ACK'd, which would result in
> > deadlock.
> 
> Could you tell me more about that?
> I can't see a case which would be problematic.
> Moreover shutdown(SHUT_WR) can actually send the outstanding data.

Assume that the remote end is single threaded. At any given time, it's
either reading or writing. If it's writing, it won't be reading data,
and it won't read any data until it has finished writing. If the
reader doesn't consume the data (in the sense of ACK'ing it), the
writer's send buffer will fill, and the writer will block waiting for
the write to complete. While it's blocked, it isn't reading any data
sent in the other direction.

IOW: sending a RST doesn't require the cooperation of the remote end,
sending data does.

> > If the server sends a response to the quit command, the client should
> > read the response then close() the socket. Provided that the server's
> > response is well-formed, there shouldn't be a problem.
> 
> The client is ok. As soon as it receive "Ok, I quit" from the server,
> it close the connection. Although the server should also work with
> basic clients like netcat or telnet, which won't close the connection
> by themselves. Then the server just has to send its last message and
> close the connection. That's the protocol we expect:
> The server read the command "QUIT;", send its last message saying it's
> going to close the connection, and close the connection. The client is
> quite passive in the termination process, it read the response to the
> "QUIT;" and wait for the connection to be closed (its read should
> return -1).
> 
> The problem is on the server side. Because of the unread "\n" (after
> the "QUIT;"), "Ok, I quit" is never sent.

If you want to get by without either side performing a half-close, one
side must adhere strictly to the protocol, the other side must perform
the initial close.

If neither side can be relied upon to follow protocol exactly, you
must use a half-close (as mentioned previously, rsh works like this as
there isn't any "protocol" to follow; it's just free-form text in each
direction).

> BTW, now I understand better what happen and why it happen, don't you
> think this behavior should be documented somewhere? Should I report a
> bug for the man 7 tcp?

I wouldn't say it's a "bug" in the documentation. A manual page isn't
really the place for a high-level overview of the TCP protocol. And
the behaviour in question isn't related to a specific system call. 
close() isn't specific to sockets, shutdown() isn't specific to TCP
sockets, and the same issue arises if the socket is closed implicitly
as a result of process termination.

What it really boils down to is that "receiving" data on a TCP socket
which has been closed for reading will result in a RST; that much is
in the RFCs, I believe. The practical consequence of that fact at the
application layer is more appropriate in a tutorial or article.

-- 
Glynn Clements <glynn@gclements.plus.com>

Re: close socket and TCP RST

[Celelibi]

2012/4/14, Glynn Clements <glynn@gclements.plus.com>:
>
> Celelibi wrote:
>
>> > Sending outstanding data then sending the RST afterwards is
>> > potentially problematic as the receiver may not be able or willing to
>> > read the data until its own data has been ACK'd, which would result in
>> > deadlock.
>>
>> Could you tell me more about that?
>> I can't see a case which would be problematic.
>> Moreover shutdown(SHUT_WR) can actually send the outstanding data.
>
> Assume that the remote end is single threaded. At any given time, it's
> either reading or writing. If it's writing, it won't be reading data,
> and it won't read any data until it has finished writing. If the
> reader doesn't consume the data (in the sense of ACK'ing it), the
> writer's send buffer will fill, and the writer will block waiting for
> the write to complete. While it's blocked, it isn't reading any data
> sent in the other direction.
>
> IOW: sending a RST doesn't require the cooperation of the remote end,
> sending data does.
>

I can't see why it is important that the remote end is single threaded or not.
The data are ACK'd as soon as the kernel get them. There's nothing to
do at the application level.

Your explanation sounds a bit confusing to me. Let's see if I understood.

The remote kernel won't try to send the data over the network if the
window size has reached 0, but it can still receive data as long as
its own receive window has not reached 0.
That means, if the remote application is brusting with send(), it will
fill the server's receive window (since the server do not want to read
any data after a "QUIT;"), then it will fill it's own outgoing buffer
and then the send()/write() will become blocking.
And if, unfortunately, the first command from the send() burst
generated a lot of data send by the server to the client, the client's
receive window may be 0 too and server's kernel ougoing buffer may be
full too. And here is the deadlock.

But actually this kind of deadlock car occur everytime (and is quite
common when dealing with pipes).

But, well, I admit that a RST means here "protocol error" and should
imply a whole session rollback if possible.
And trying to send more data before the RST when the current remote
window size is 0 would either delay the RST (which would not be RFC
compliant) or "break the window".
But this is quite a corner case.
It may send together with the RST as much data as the window permit
and do not expect any ACK (as the connection has been reset).


> If you want to get by without either side performing a half-close, one
> side must adhere strictly to the protocol, the other side must perform
> the initial close.
>
> If neither side can be relied upon to follow protocol exactly, you
> must use a half-close (as mentioned previously, rsh works like this as
> there isn't any "protocol" to follow; it's just free-form text in each
> direction).
>

Ok, I understand that my protocol wasn't strict enough on the exact
amount of data to be sent. I mean, the existence of that trailing \n
should be specified.
Ok, let's say that the protocol say that the "QUIT;" can be followed
by one "\n". Even in that case, I'm forced to use a half-close
connection?


>> BTW, now I understand better what happen and why it happen, don't you
>> think this behavior should be documented somewhere? Should I report a
>> bug for the man 7 tcp?
>
> I wouldn't say it's a "bug" in the documentation. A manual page isn't
> really the place for a high-level overview of the TCP protocol. And
> the behaviour in question isn't related to a specific system call.
> close() isn't specific to sockets, shutdown() isn't specific to TCP
> sockets, and the same issue arises if the socket is closed implicitly
> as a result of process termination.

Actually, the section 7 contains (among other things) some high level
overviews of some parts of the system. The section 2 is meant for
system call documentation. The "man 7 socket" and "man 7 tcp" already
contains a lot of interesting stuff.

>
> What it really boils down to is that "receiving" data on a TCP socket
> which has been closed for reading will result in a RST; that much is
> in the RFCs, I believe. The practical consequence of that fact at the
> application layer is more appropriate in a tutorial or article.

The fact that outstanding data is discarded too is quite surprising
(even if it can makes sens after all). And still looks to me more like
an implementation choice than a real consequence of a RFC.


Celelibi

Re: close socket and TCP RST

[Glynn Clements]

Celelibi wrote:

> I can't see why it is important that the remote end is single
> threaded or not.

If it's single-threaded, it won't read data until it has finished
writing.

> The data are ACK'd as soon as the kernel get them.

Only if there's space in the receive buffer. Otherwise, the kernel
will ACK the last byte which was stored and report a receive window of
zero. The sender will not attempt to send more data until the receive
window opens.

> The remote kernel won't try to send the data over the network if the
> window size has reached 0, but it can still receive data as long as
> its own receive window has not reached 0.
> That means, if the remote application is brusting with send(), it will
> fill the server's receive window (since the server do not want to read
> any data after a "QUIT;"), then it will fill it's own outgoing buffer
> and then the send()/write() will become blocking.
> And if, unfortunately, the first command from the send() burst
> generated a lot of data send by the server to the client, the client's
> receive window may be 0 too and server's kernel ougoing buffer may be
> full too. And here is the deadlock.

Yes.

> But actually this kind of deadlock car occur everytime (and is quite
> common when dealing with pipes).

In this case, the problem is that the application may have terminated,
leaving the kernel to clean up the mess. If the kernel is going to
send a RST anyhow, it's debatable whether there's any benefit in going
to significant effort to send outstanding data which the receiver may
just ignore.

> But, well, I admit that a RST means here "protocol error" and should
> imply a whole session rollback if possible.
> And trying to send more data before the RST when the current remote
> window size is 0 would either delay the RST (which would not be RFC
> compliant) or "break the window".
> But this is quite a corner case.
> It may send together with the RST as much data as the window permit
> and do not expect any ACK (as the connection has been reset).

That would require a separate path, and is essentially pointless. If
it's not going to wait for an ACK (and retransmit if necessary),
there's not much point sending the data in the first place. If a
single dropped packet means that the data is permanently lost, it may
as well just discard it and save itself some trouble.

> > If you want to get by without either side performing a half-close, one
> > side must adhere strictly to the protocol, the other side must perform
> > the initial close.
> >
> > If neither side can be relied upon to follow protocol exactly, you
> > must use a half-close (as mentioned previously, rsh works like this as
> > there isn't any "protocol" to follow; it's just free-form text in each
> > direction).
> 
> Ok, I understand that my protocol wasn't strict enough on the exact
> amount of data to be sent. I mean, the existence of that trailing \n
> should be specified.
> Ok, let's say that the protocol say that the "QUIT;" can be followed
> by one "\n". Even in that case, I'm forced to use a half-close
> connection?

If the newline is optional and the receiver performs the initial
close, then at least one side needs to perform a half-close. Either:

1. The client sends "QUIT;\n...", the server sends a response and
performs a half-close, the client reads the response and EOF and calls
close(), the server reads the EOF and calls close(). Or:

2. The client sends "QUIT;\n...", the server sends the response, the
client performs a half-close, the server reads the EOF and calls
close(), the client reads the EOF and calls close().

If the client can be relied upon to send an exact termination
sequence, then the server can call close() as soon as it has read that
sequence, and the client can call close() once it has read EOF from
the server. EOF is always a reliable terminator; nothing else will be
received after that, so calling close() won't cause a RST.

> > What it really boils down to is that "receiving" data on a TCP socket
> > which has been closed for reading will result in a RST; that much is
> > in the RFCs, I believe. The practical consequence of that fact at the
> > application layer is more appropriate in a tutorial or article.
> 
> The fact that outstanding data is discarded too is quite surprising
> (even if it can makes sens after all).

Not really. RST indicates abnormal termination. Once it has been sent,
the kernel can (almost) completely forget that the connection ever
existed. Not only will unsent data be discarded, but so will
unacknowledged sent data. Any subsequent packets will be met with
another RST as they don't correspond to a "known" connection.

> And still looks to me more like an implementation choice than a real
> consequence of a RFC.

The very existence of SHUT_RD is an implementation choice, not just
its behaviour. The TCP protocol itself necessitates a mechanism to
terminate transmission, and each direction can be terminated
independently.

While it would be possible to implement an API where close() peformed
a half-close (SHUT_WR) with no "offical" mechanism for the SHUT_RD
case, the kernel still has to deal with an application which exit()s
or crashes.

-- 
Glynn Clements <glynn@gclements.plus.com>