Re: PROBLEM: epoll_wait() does not return events when running in multiple threads

[Sergey Nikitin <nikitins@oktetlabs.ru>]

[-- Attachment #1: Type: text/plain, Size: 4099 bytes --]


On 10.09.2020 14:54, Al Viro wrote:
> On Thu, Sep 10, 2020 at 12:48:34PM +0300, Sergey Nikitin wrote:
>> Hi!
>>
>> epoll does not report an event to all the threads running epoll_wait() on
>> the same epoll descriptor.
>> The behavior appeared in recent kernel versions starting with 5.6 probably.
>>
>> How to reproduce:
>> - create a pair of sockets
>> - create epoll instance
>> - register the socket on the epoll instance, listen for EPOLLIN events
>> - start 2 threads running epoll_wait()
>> - send some data to the socket
>> - see that epoll_wait() within one of the threads reported an event, unlike
>> another.
> Could you reproduce it on mainline kernel and try to bisect it?

I rechecked the f4d51dffc6c0 Linux 5.9-rc4. The issue is still reproducible.

Bisect result:
339ddb53d373baee6e7946aec17c739c4924d6d9 is the first bad commit
commit 339ddb53d373baee6e7946aec17c739c4924d6d9
Author: Heiher <r@hev.cc>
Date:   Wed Dec 4 16:52:15 2019 -0800

     fs/epoll: remove unnecessary wakeups of nested epoll

     Take the case where we have:

             t0
              | (ew)
             e0
              | (et)
             e1
              | (lt)
             s0

     t0: thread 0
     e0: epoll fd 0
     e1: epoll fd 1
     s0: socket fd 0
     ew: epoll_wait
     et: edge-trigger
     lt: level-trigger

     We remove unnecessary wakeups to prevent the nested epoll that 
working in edge-
     triggered mode to waking up continuously.

     Test code:
      #include <unistd.h>
      #include <sys/epoll.h>
      #include <sys/socket.h>

      int main(int argc, char *argv[])
      {
             int sfd[2];
             int efd[2];
             struct epoll_event e;

             if (socketpair(AF_UNIX, SOCK_STREAM, 0, sfd) < 0)
                     goto out;

             efd[0] = epoll_create(1);
             if (efd[0] < 0)
                     goto out;

             efd[1] = epoll_create(1);
             if (efd[1] < 0)
                     goto out;

             e.events = EPOLLIN;
             if (epoll_ctl(efd[1], EPOLL_CTL_ADD, sfd[0], &e) < 0)
                     goto out;

             e.events = EPOLLIN | EPOLLET;
             if (epoll_ctl(efd[0], EPOLL_CTL_ADD, efd[1], &e) < 0)
                     goto out;

             if (write(sfd[1], "w", 1) != 1)
                     goto out;

             if (epoll_wait(efd[0], &e, 1, 0) != 1)
                     goto out;

             if (epoll_wait(efd[0], &e, 1, 0) != 0)
                     goto out;

             close(efd[0]);
             close(efd[1]);
             close(sfd[0]);
             close(sfd[1]);

             return 0;

      out:
             return -1;
      }

     More tests:
      https://github.com/heiher/epoll-wakeup

     Link: http://lkml.kernel.org/r/20191009060516.3577-1-r@hev.cc
     Signed-off-by: hev <r@hev.cc>
     Reviewed-by: Roman Penyaev <rpenyaev@suse.de>
     Cc: Al Viro <viro@ZenIV.linux.org.uk>
     Cc: Davide Libenzi <davidel@xmailserver.org>
     Cc: Davidlohr Bueso <dave@stgolabs.net>
     Cc: Dominik Brodowski <linux@dominikbrodowski.net>
     Cc: Eric Wong <e@80x24.org>
     Cc: Jason Baron <jbaron@akamai.com>
     Cc: Sridhar Samudrala <sridhar.samudrala@intel.com>
     Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
     Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

  fs/eventpoll.c | 16 ----------------
  1 file changed, 16 deletions(-)


Attaching a C reproducer which I was using to bisect.

-- 
Best regards,
Sergey Nikitin


[-- Attachment #2: reproducer.tar --]
[-- Type: application/x-tar, Size: 10240 bytes --]