* POLLPRI/poll() behavior change since 2.6.31
@ 2011-01-06 15:50 Leonardo Chiquitto
2011-01-06 16:55 ` Eric Dumazet
2011-01-06 17:44 ` [PATCH] net: add POLLPRI to sock_def_readable() Eric Dumazet
0 siblings, 2 replies; 7+ messages in thread
From: Leonardo Chiquitto @ 2011-01-06 15:50 UTC (permalink / raw)
To: netdev; +Cc: Eric Dumazet, David S. Miller
Hello,
Since 2.6.31, poll() no longer returns when waiting exclusively for a
POLLPRI event. If we wait for POLLPRI | POLLIN, though, it will
correctly return POLLPRI as a received event.
The reproducer (code below) will print the following when running on
2.6.30:
$ ./pollpri-oob
main: starting
main: setup_pipe ok (sfd[0] = 5, sfd[1] = 4)
parent: child <pid 3790> started
child: polling...
parent: sending the message
parent: waiting for child to exit
child: poll(): some data <1,2> has arrived!
child: done
parent: done
... and will block when running on 2.6.37-rc7:
$ ./pollpri-oob
main: starting
main: setup_pipe ok (sfd[0] = 5, sfd[1] = 4)
parent: child <pid 14148> started
child: polling...
parent: sending the message
parent: waiting for child to exit
[hangs here]
I've bisected this behavior change to the following commit:
commit 4938d7e0233a455f04507bac81d0886c71529537
Author: Eric Dumazet <dada1@cosmosbay.com>
Date: Tue Jun 16 15:33:36 2009 -0700
poll: avoid extra wakeups in select/poll
After introduction of keyed wakeups Davide Libenzi did on epoll, we are
able to avoid spurious wakeups in poll()/select() code too.
For example, typical use of poll()/select() is to wait for incoming
network frames on many sockets. But TX completion for UDP/TCP frames call
sock_wfree() which in turn schedules thread.
When scheduled, thread does a full scan of all polled fds and can sleep
again, because nothing is really available. If number of fds is large,
this cause significant load.
This patch makes select()/poll() aware of keyed wakeups and useless
wakeups are avoided. This reduces number of context switches by about 50%
on some setups, and work performed by sofirq handlers.
I don't know if the new behavior is correct, but we've got one report of
an application that broke due to the change.
Thanks,
Leonardo
#define _BSD_SOURCE
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/ioctl.h>
#include <sys/wait.h>
#include <sys/stat.h>
#include <sys/un.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <errno.h>
#include <fcntl.h>
#include <poll.h>
int setup_pipe(int f[2])
{
int ret, server, client, client_ofl;
struct sockaddr_in own_sa;
struct sockaddr a_sa;
socklen_t a_len;
/* server side */
if ((server = socket(PF_INET, SOCK_STREAM, 0)) < 0)
return -1;
own_sa.sin_family = AF_INET;
own_sa.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
own_sa.sin_port = htons(10789);
if (bind(server, (struct sockaddr *)&own_sa, sizeof(own_sa)) != 0)
return -1;
if (listen(server, 1) < 0)
return -1;
/* client side */
if ((client = socket(PF_INET, SOCK_STREAM, 0)) < 0)
return -1;
if ((client_ofl = fcntl(client, F_GETFL)) < 0)
return -1;
if (fcntl(client, F_SETFL, client_ofl | O_NONBLOCK) < 0)
return -1;
ret = connect (client, (struct sockaddr *) &own_sa, sizeof(own_sa));
if (ret != 0 && errno != EINPROGRESS)
return -1;
if ((f[0] = accept(server, &a_sa, &a_len)) < 0)
return -1;
f[1] = client;
return 0;
}
int child_proc(int fd)
{
struct pollfd fds;
int ret;
fds.fd = fd;
fds.events = POLLPRI;
printf("child: polling...\n");
ret = poll(&fds, 1, -1);
if (ret > 0)
printf("child: poll(): some data <%d,%d> has arrived!\n",
ret, fds.revents);
printf("child: done\n");
return 0;
}
int main(int argc, char *argv[])
{
int sfd[2] = { -1, -1 };
pid_t child_pid;
printf("main: starting\n");
if (setup_pipe(sfd) == -1) {
fprintf(stderr, "main: error in setup_pipe()\n");
return -1;
}
printf("main: setup_pipe ok (sfd[0] = %d, sfd[1] = %d)\n",
sfd[0], sfd[1]);
switch (child_pid = fork()) {
case -1:
fprintf(stderr, "main: fork() error\n");
exit(1);
case 0:
return child_proc(sfd[0]);
default:
printf("parent: child <pid %d> started\n", child_pid);
sleep(1);
printf("parent: sending the message\n");
send(sfd[1], "a", 1, MSG_OOB);
printf("parent: waiting for child to exit\n");
waitpid(child_pid, NULL, 0);
printf("parent: done\n");
}
return 0;
}
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: POLLPRI/poll() behavior change since 2.6.31
2011-01-06 15:50 POLLPRI/poll() behavior change since 2.6.31 Leonardo Chiquitto
@ 2011-01-06 16:55 ` Eric Dumazet
2011-01-06 18:10 ` Leonardo Chiquitto
2011-01-06 22:40 ` Davide Libenzi
2011-01-06 17:44 ` [PATCH] net: add POLLPRI to sock_def_readable() Eric Dumazet
1 sibling, 2 replies; 7+ messages in thread
From: Eric Dumazet @ 2011-01-06 16:55 UTC (permalink / raw)
To: Leonardo Chiquitto; +Cc: netdev, David S. Miller
Le jeudi 06 janvier 2011 à 13:50 -0200, Leonardo Chiquitto a écrit :
> Hello,
>
> Since 2.6.31, poll() no longer returns when waiting exclusively for a
> POLLPRI event. If we wait for POLLPRI | POLLIN, though, it will
> correctly return POLLPRI as a received event.
>
> The reproducer (code below) will print the following when running on
> 2.6.30:
>
> $ ./pollpri-oob
> main: starting
> main: setup_pipe ok (sfd[0] = 5, sfd[1] = 4)
> parent: child <pid 3790> started
> child: polling...
> parent: sending the message
> parent: waiting for child to exit
> child: poll(): some data <1,2> has arrived!
> child: done
> parent: done
>
> ... and will block when running on 2.6.37-rc7:
>
> $ ./pollpri-oob
> main: starting
> main: setup_pipe ok (sfd[0] = 5, sfd[1] = 4)
> parent: child <pid 14148> started
> child: polling...
> parent: sending the message
> parent: waiting for child to exit
> [hangs here]
>
> I've bisected this behavior change to the following commit:
>
> commit 4938d7e0233a455f04507bac81d0886c71529537
> Author: Eric Dumazet <dada1@cosmosbay.com>
> Date: Tue Jun 16 15:33:36 2009 -0700
>
> poll: avoid extra wakeups in select/poll
>
> After introduction of keyed wakeups Davide Libenzi did on epoll, we are
> able to avoid spurious wakeups in poll()/select() code too.
>
> For example, typical use of poll()/select() is to wait for incoming
> network frames on many sockets. But TX completion for UDP/TCP frames call
> sock_wfree() which in turn schedules thread.
>
> When scheduled, thread does a full scan of all polled fds and can sleep
> again, because nothing is really available. If number of fds is large,
> this cause significant load.
>
> This patch makes select()/poll() aware of keyed wakeups and useless
> wakeups are avoided. This reduces number of context switches by about 50%
> on some setups, and work performed by sofirq handlers.
>
>
> I don't know if the new behavior is correct, but we've got one report of
> an application that broke due to the change.
Hi Leonardo
Hmm, this is because sock_def_readable() uses :
wake_up_interruptible_sync_poll(&wq->wait, POLLIN | POLLRDNORM |
POLLRDBAND);
So POLLPRI bit is not signaled.
I would just add POLLPRI flag in sock_def_readable()
(Alternatively, define a tcp_def_readable() function to pass POLLPRI
only if TCP_URG is set, but is it worth the pain for a seldom used
feature ?)
David, do you have an opinion on this ?
Thanks
diff --git a/net/core/sock.c b/net/core/sock.c
index e5af8d5..7fd3541 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -1907,7 +1907,7 @@ static void sock_def_readable(struct sock *sk, int len)
rcu_read_lock();
wq = rcu_dereference(sk->sk_wq);
if (wq_has_sleeper(wq))
- wake_up_interruptible_sync_poll(&wq->wait, POLLIN |
+ wake_up_interruptible_sync_poll(&wq->wait, POLLIN | POLLPRI |
POLLRDNORM | POLLRDBAND);
sk_wake_async(sk, SOCK_WAKE_WAITD, POLL_IN);
rcu_read_unlock();
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH] net: add POLLPRI to sock_def_readable()
2011-01-06 15:50 POLLPRI/poll() behavior change since 2.6.31 Leonardo Chiquitto
2011-01-06 16:55 ` Eric Dumazet
@ 2011-01-06 17:44 ` Eric Dumazet
2011-01-06 18:55 ` David Miller
1 sibling, 1 reply; 7+ messages in thread
From: Eric Dumazet @ 2011-01-06 17:44 UTC (permalink / raw)
To: Leonardo Chiquitto, David Miller; +Cc: netdev
Leonardo Chiquitto found poll() could block forever on tcp sockets and
Urgent data was received, if the event flag only contains POLLPRI.
He did a bisection and found commit 4938d7e0233 (poll: avoid extra
wakeups in select/poll) was the source of the problem.
Problem is TCP sockets use standard sock_def_readable() function for
their sk_data_ready() handler, and sock_def_readable() doesnt signal
POLLPRI.
Only TCP is affected by the problem. Adding POLLPRI to the list of flags
might trigger unnecessary schedules, but URGENT handling is such a
seldom used feature this seems a good compromise.
Thanks a lot to Leonardo for providing the bisection result and a test
program as well.
Reference : http://www.spinics.net/lists/netdev/msg151793.html
Reported-and-bisected-by: Leonardo Chiquitto <leonardo.lists@gmail.com>
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Tested-by: Eric Dumazet <eric.dumazet@gmail.com>
---
net/core/sock.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/net/core/sock.c b/net/core/sock.c
index e5af8d5..7fd3541 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -1907,7 +1907,7 @@ static void sock_def_readable(struct sock *sk, int len)
rcu_read_lock();
wq = rcu_dereference(sk->sk_wq);
if (wq_has_sleeper(wq))
- wake_up_interruptible_sync_poll(&wq->wait, POLLIN |
+ wake_up_interruptible_sync_poll(&wq->wait, POLLIN | POLLPRI |
POLLRDNORM | POLLRDBAND);
sk_wake_async(sk, SOCK_WAKE_WAITD, POLL_IN);
rcu_read_unlock();
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: POLLPRI/poll() behavior change since 2.6.31
2011-01-06 16:55 ` Eric Dumazet
@ 2011-01-06 18:10 ` Leonardo Chiquitto
2011-01-06 22:40 ` Davide Libenzi
1 sibling, 0 replies; 7+ messages in thread
From: Leonardo Chiquitto @ 2011-01-06 18:10 UTC (permalink / raw)
To: Eric Dumazet; +Cc: netdev, David S. Miller
On Thu, Jan 6, 2011 at 2:55 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> Le jeudi 06 janvier 2011 à 13:50 -0200, Leonardo Chiquitto a écrit :
>> Hello,
>>
>> Since 2.6.31, poll() no longer returns when waiting exclusively for a
>> POLLPRI event. If we wait for POLLPRI | POLLIN, though, it will
>> correctly return POLLPRI as a received event.
>>
>> The reproducer (code below) will print the following when running on
>> 2.6.30:
>>
>> $ ./pollpri-oob
>> main: starting
>> main: setup_pipe ok (sfd[0] = 5, sfd[1] = 4)
>> parent: child <pid 3790> started
>> child: polling...
>> parent: sending the message
>> parent: waiting for child to exit
>> child: poll(): some data <1,2> has arrived!
>> child: done
>> parent: done
>>
>> ... and will block when running on 2.6.37-rc7:
>>
>> $ ./pollpri-oob
>> main: starting
>> main: setup_pipe ok (sfd[0] = 5, sfd[1] = 4)
>> parent: child <pid 14148> started
>> child: polling...
>> parent: sending the message
>> parent: waiting for child to exit
>> [hangs here]
>>
>> I've bisected this behavior change to the following commit:
>>
>> commit 4938d7e0233a455f04507bac81d0886c71529537
>> Author: Eric Dumazet <dada1@cosmosbay.com>
>> Date: Tue Jun 16 15:33:36 2009 -0700
>>
>> poll: avoid extra wakeups in select/poll
>>
>> After introduction of keyed wakeups Davide Libenzi did on epoll, we are
>> able to avoid spurious wakeups in poll()/select() code too.
>>
>> For example, typical use of poll()/select() is to wait for incoming
>> network frames on many sockets. But TX completion for UDP/TCP frames call
>> sock_wfree() which in turn schedules thread.
>>
>> When scheduled, thread does a full scan of all polled fds and can sleep
>> again, because nothing is really available. If number of fds is large,
>> this cause significant load.
>>
>> This patch makes select()/poll() aware of keyed wakeups and useless
>> wakeups are avoided. This reduces number of context switches by about 50%
>> on some setups, and work performed by sofirq handlers.
>>
>>
>> I don't know if the new behavior is correct, but we've got one report of
>> an application that broke due to the change.
>
> Hi Leonardo
>
> Hmm, this is because sock_def_readable() uses :
>
> wake_up_interruptible_sync_poll(&wq->wait, POLLIN | POLLRDNORM |
> POLLRDBAND);
>
> So POLLPRI bit is not signaled.
>
> I would just add POLLPRI flag in sock_def_readable()
>
> (Alternatively, define a tcp_def_readable() function to pass POLLPRI
> only if TCP_URG is set, but is it worth the pain for a seldom used
> feature ?)
>
> David, do you have an opinion on this ?
>
> Thanks
>
> diff --git a/net/core/sock.c b/net/core/sock.c
> index e5af8d5..7fd3541 100644
> --- a/net/core/sock.c
> +++ b/net/core/sock.c
> @@ -1907,7 +1907,7 @@ static void sock_def_readable(struct sock *sk, int len)
> rcu_read_lock();
> wq = rcu_dereference(sk->sk_wq);
> if (wq_has_sleeper(wq))
> - wake_up_interruptible_sync_poll(&wq->wait, POLLIN |
> + wake_up_interruptible_sync_poll(&wq->wait, POLLIN | POLLPRI |
> POLLRDNORM | POLLRDBAND);
> sk_wake_async(sk, SOCK_WAKE_WAITD, POLL_IN);
> rcu_read_unlock();
Eric,
Thanks for the quick reply. I tested your patch and confirm that it resolves the
problem.
Regards,
Leonardo
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] net: add POLLPRI to sock_def_readable()
2011-01-06 17:44 ` [PATCH] net: add POLLPRI to sock_def_readable() Eric Dumazet
@ 2011-01-06 18:55 ` David Miller
0 siblings, 0 replies; 7+ messages in thread
From: David Miller @ 2011-01-06 18:55 UTC (permalink / raw)
To: eric.dumazet; +Cc: leonardo.lists, netdev
From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Thu, 06 Jan 2011 18:44:57 +0100
> Leonardo Chiquitto found poll() could block forever on tcp sockets and
> Urgent data was received, if the event flag only contains POLLPRI.
>
> He did a bisection and found commit 4938d7e0233 (poll: avoid extra
> wakeups in select/poll) was the source of the problem.
>
> Problem is TCP sockets use standard sock_def_readable() function for
> their sk_data_ready() handler, and sock_def_readable() doesnt signal
> POLLPRI.
>
> Only TCP is affected by the problem. Adding POLLPRI to the list of flags
> might trigger unnecessary schedules, but URGENT handling is such a
> seldom used feature this seems a good compromise.
>
> Thanks a lot to Leonardo for providing the bisection result and a test
> program as well.
>
> Reference : http://www.spinics.net/lists/netdev/msg151793.html
>
> Reported-and-bisected-by: Leonardo Chiquitto <leonardo.lists@gmail.com>
> Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
> Tested-by: Eric Dumazet <eric.dumazet@gmail.com>
Applied, thanks everyone.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: POLLPRI/poll() behavior change since 2.6.31
2011-01-06 16:55 ` Eric Dumazet
2011-01-06 18:10 ` Leonardo Chiquitto
@ 2011-01-06 22:40 ` Davide Libenzi
2011-01-07 13:31 ` Eric Dumazet
1 sibling, 1 reply; 7+ messages in thread
From: Davide Libenzi @ 2011-01-06 22:40 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Leonardo Chiquitto, netdev, David S. Miller
On Thu, 6 Jan 2011, Eric Dumazet wrote:
> Hmm, this is because sock_def_readable() uses :
>
> wake_up_interruptible_sync_poll(&wq->wait, POLLIN | POLLRDNORM |
> POLLRDBAND);
>
> So POLLPRI bit is not signaled.
>
> I would just add POLLPRI flag in sock_def_readable()
>
> (Alternatively, define a tcp_def_readable() function to pass POLLPRI
> only if TCP_URG is set, but is it worth the pain for a seldom used
> feature ?)
It would be kinda cleaner though, /me thinks.
- Davide
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: POLLPRI/poll() behavior change since 2.6.31
2011-01-06 22:40 ` Davide Libenzi
@ 2011-01-07 13:31 ` Eric Dumazet
0 siblings, 0 replies; 7+ messages in thread
From: Eric Dumazet @ 2011-01-07 13:31 UTC (permalink / raw)
To: Davide Libenzi; +Cc: Leonardo Chiquitto, netdev, David S. Miller
Le jeudi 06 janvier 2011 à 14:40 -0800, Davide Libenzi a écrit :
> On Thu, 6 Jan 2011, Eric Dumazet wrote:
>
> > Hmm, this is because sock_def_readable() uses :
> >
> > wake_up_interruptible_sync_poll(&wq->wait, POLLIN | POLLRDNORM |
> > POLLRDBAND);
> >
> > So POLLPRI bit is not signaled.
> >
> > I would just add POLLPRI flag in sock_def_readable()
> >
> > (Alternatively, define a tcp_def_readable() function to pass POLLPRI
> > only if TCP_URG is set, but is it worth the pain for a seldom used
> > feature ?)
>
> It would be kinda cleaner though, /me thinks.
>
Yep, we'll do this in net-next-2.6 for 2.6.39 :)
Thanks !
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2011-01-07 13:31 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-01-06 15:50 POLLPRI/poll() behavior change since 2.6.31 Leonardo Chiquitto
2011-01-06 16:55 ` Eric Dumazet
2011-01-06 18:10 ` Leonardo Chiquitto
2011-01-06 22:40 ` Davide Libenzi
2011-01-07 13:31 ` Eric Dumazet
2011-01-06 17:44 ` [PATCH] net: add POLLPRI to sock_def_readable() Eric Dumazet
2011-01-06 18:55 ` David Miller
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.