disappearing listen()ed SO_REUSEPORT sockets across fork() when using epoll

* disappearing listen()ed SO_REUSEPORT sockets across fork() when using epoll
@ 2013-11-22 17:53 Shawn Landden
  2013-11-25 18:05 ` Jason Baron
  0 siblings, 1 reply; 5+ messages in thread
From: Shawn Landden @ 2013-11-22 17:53 UTC (permalink / raw)
  To: linux-fsdevel, linux-kernel, Tom Herbert

[-- Attachment #1: Type: text/plain, Size: 962 bytes --]

Hello, when running the attached program on 3.12 child processes
are missing a socket fd opened, set with SO_REUSEPORT, listen()ed to,
and added to epoll_ctl().

This is the output I get when pointing "wget http://localhost:5555/"
at the attached program:

main PID 31591
PID 31634 started
PID 31634 accept()ed connection
PID 31635 started
PID 31636 started
PID 31635 accept() failed: Bad file descriptor
PID 31636 accept() failed: Bad file descriptor
PID 31634 accept()ed connection
PID 31634 accept()ed connection
PID 31634 accept()ed connection
PID 31634 accept()ed connection

While I would expect something like:

main PID 31591
PID 31634 started
PID 31634 accept()ed connection
PID 31635 started
PID 31636 started
PID 31635 accept()ed connection
PID 31636 accept()ed connection

-more new processes, but inversely proportional to number of listening processes
-accept() always returns successfully

-- 

---
Shawn Landden
+1 360 389 3001 (SMS preferred)

[-- Attachment #2: epoll_test.c --]
[-- Type: text/x-csrc, Size: 2538 bytes --]

#include <stdio.h>
#include <sys/types.h> 
#include <sys/socket.h>
#include <netinet/in.h>
#include <stdlib.h>
#include <string.h>
#include <sys/epoll.h>

int main( int argc, char *argv[]) {
        int sockfd, epollfd, acceptfd, portno;
        struct epoll_event event = {EPOLLIN, NULL}, gotevent;
        char buffer[256];
        struct sockaddr_in serv_addr;
        int  n;
        pid_t pid;

        printf("main PID %d\n", getpid());

        memset((char *) &serv_addr, 0, sizeof(serv_addr));
        portno = 5555;
        serv_addr.sin_family = AF_INET;
        serv_addr.sin_addr.s_addr = INADDR_ANY;
        serv_addr.sin_port = htons(portno);

        if ((sockfd = socket(AF_INET, SOCK_STREAM, 0)) < 0) 
		return printf("socket() failed: %m\n");

        int optval = 1;
        if ((setsockopt(sockfd, SOL_SOCKET, SO_REUSEPORT, &optval, sizeof(optval))) < 0)
                return printf("setsockopt() failed: %m");

        if (bind(sockfd, (struct sockaddr *) &serv_addr,
                          sizeof(serv_addr)) < 0)
		return printf("bind() failed: %m\n");

        if (listen(sockfd, SOMAXCONN) < 0)
		return printf("listen() failed: %m\n");

	if ((epollfd = epoll_create1(0)) < 0)
	        return printf("epoll_create1() failed: %m\n");

        if (epoll_ctl(epollfd,  EPOLL_CTL_ADD, sockfd, &event) < 0)
                return printf("epoll_ctl() failed: %m\n");

        while (1) {
                if (epoll_wait(epollfd, &gotevent, 1, -1) != 1)
                        return printf("epoll_wait() failed: %m\n");

                pid = fork();
                if (pid == 0) {
                        printf("PID %d started\n", getpid());
                        while(1) {
                                struct sockaddr_in cli_addr;
                                socklen_t cli_size = sizeof cli_addr;

                                if ((acceptfd = accept(sockfd, (struct sockaddr *)&cli_addr, 
                                                        &cli_size)) < 0) {
	                                printf("PID %d accept() failed: %m\n", getpid());
	                                sleep(60);
	                                return 1;
                                }

                                printf("PID %d accept()ed connection\n", getpid());

                                if (close(acceptfd) < 0)
                                        return printf("close() failed: %m\n");
                        }
                }
                close(sockfd);
        }
}

^ permalink raw reply	[flat|nested] 5+ messages in thread