From: Alan Burlison <Alan.Burlison@oracle.com>
To: Casper.Dik@oracle.com, Al Viro <viro@ZenIV.linux.org.uk>
Cc: David Miller <davem@davemloft.net>,
eric.dumazet@gmail.com, stephen@networkplumber.org,
netdev@vger.kernel.org, dholland-tech@netbsd.org
Subject: Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)
Date: Tue, 27 Oct 2015 10:52:46 +0000 [thread overview]
Message-ID: <562F577E.6000901@oracle.com> (raw)
In-Reply-To: <201510270908.t9R9873a001683@room101.nl.oracle.com>
On 27/10/2015 09:08, Casper.Dik@oracle.com wrote:
> Generally I wouldn't see that as a problem, but in the case of a socket
> blocking on accept indefinitely, I do see it as a problem especially as
> the thread actually wants to stop listening.
>
> But in general, this is basically a problem with the application: the file
> descriptor space is shared between threads and having one thread sniping
> at open files, you do have a problem and whatever the kernel does in that
> case perhaps doesn't matter all that much: the application needs to be
> fixed anyway.
The scenario in Hadoop is that the FD is being used by a thread that's
waiting in accept and another thread wants to shut it down, e.g. because
the application is terminating and needs to stop all threads cleanly. I
agree the use of shutdown()+close() on Linux or dup2() on Solaris is
pretty much an application-level hack - the concern in both cases is
that the file descriptor that's being used in the accept() might be
recycled by another thread. However that just begs the question of why
the FD isn't properly encapsulated by the application in a singleton
object, with the required shut down semantics provided by having a
mechanism to invalidate the singleton and its contained FD.
There are other mechanisms that could be used to do a clean shutdown
that don't require the OS to provide workarounds for arguably broken
application behaviour, for example by setting a 'shutdown' flag in the
object and then doing a dummy connect() to the accepting FD to kick it
off the accept() and thereby getting it to re-check the 'shutdown' flag
and not re-enter the accept().
If the object encapsulating a FD is invalidated and that prevents the FD
being used any more because the only access is via that object, then it
simply doesn't matter if the FD is reused elsewhere, there can be no
race so a complicated, platform-dependent dance isn't needed.
Unfortunately Hadoop isn't the only thing that pulls the shutdown()
trick, so I don't think there's a simple fix for this, as discussed
earlier in the thread. Having said that, if close() on Linux also did an
implicit shutdown() it would mean that well-written applications that
handled the scoping, sharing and reuse of FDs properly could just call
close() and have it work the same way across *NIX platforms.
--
Alan Burlison
--
next prev parent reply other threads:[~2015-10-27 10:52 UTC|newest]
Thread overview: 138+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-10-19 16:59 Fw: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3) Stephen Hemminger
2015-10-19 23:33 ` Eric Dumazet
2015-10-20 1:12 ` Alan Burlison
2015-10-20 1:45 ` Eric Dumazet
2015-10-20 9:59 ` Alan Burlison
2015-10-20 11:24 ` David Miller
2015-10-20 11:39 ` Alan Burlison
2015-10-20 13:19 ` Fw: " Eric Dumazet
2015-10-20 13:45 ` Alan Burlison
2015-10-20 15:30 ` Eric Dumazet
2015-10-20 18:31 ` Alan Burlison
2015-10-20 18:42 ` Eric Dumazet
2015-10-21 10:25 ` David Laight
2015-10-21 10:49 ` Alan Burlison
2015-10-21 11:28 ` Eric Dumazet
2015-10-21 13:03 ` Alan Burlison
2015-10-21 13:29 ` Eric Dumazet
2015-10-21 3:49 ` Al Viro
2015-10-21 14:38 ` Alan Burlison
2015-10-21 15:30 ` David Miller
2015-10-21 16:04 ` Casper.Dik
2015-10-21 21:18 ` Eric Dumazet
2015-10-21 21:28 ` Al Viro
2015-10-21 16:32 ` Fw: " Eric Dumazet
2015-10-21 18:51 ` Al Viro
2015-10-21 20:33 ` Casper.Dik
2015-10-22 4:21 ` Al Viro
2015-10-22 10:55 ` Alan Burlison
2015-10-22 18:16 ` Al Viro
2015-10-22 20:15 ` Alan Burlison
2015-11-02 10:03 ` David Laight
2015-11-02 10:29 ` Al Viro
2015-10-21 22:28 ` Alan Burlison
2015-10-22 1:29 ` David Miller
2015-10-22 4:17 ` Alan Burlison
2015-10-22 4:44 ` Al Viro
2015-10-22 6:03 ` Al Viro
2015-10-22 6:34 ` Casper.Dik
2015-10-22 17:21 ` Al Viro
2015-10-22 18:24 ` Casper.Dik
2015-10-22 19:07 ` Al Viro
2015-10-22 19:51 ` Casper.Dik
2015-10-22 21:57 ` Al Viro
2015-10-23 9:52 ` Casper.Dik
2015-10-23 13:02 ` Eric Dumazet
2015-10-23 13:20 ` Casper.Dik
2015-10-23 13:48 ` Eric Dumazet
2015-10-23 14:13 ` Eric Dumazet
2015-10-23 13:35 ` Alan Burlison
2015-10-23 14:21 ` Eric Dumazet
2015-10-23 15:46 ` Alan Burlison
2015-10-23 16:00 ` Eric Dumazet
2015-10-23 16:07 ` Alan Burlison
2015-10-23 16:19 ` Eric Dumazet
2015-10-23 16:40 ` Alan Burlison
2015-10-23 17:47 ` Eric Dumazet
2015-10-23 17:59 ` [PATCH net-next] af_unix: do not report POLLOUT on listeners Eric Dumazet
2015-10-25 13:45 ` David Miller
2015-10-24 2:30 ` [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3) Al Viro
2015-10-27 9:08 ` Casper.Dik
2015-10-27 10:52 ` Alan Burlison [this message]
2015-10-27 12:01 ` Eric Dumazet
2015-10-27 12:27 ` Alan Burlison
2015-10-27 12:44 ` Eric Dumazet
2015-10-27 13:42 ` David Miller
2015-10-27 13:37 ` Alan Burlison
2015-10-27 13:59 ` David Miller
2015-10-27 14:13 ` Alan Burlison
2015-10-27 14:39 ` David Miller
2015-10-27 14:39 ` Alan Burlison
2015-10-27 15:04 ` David Miller
2015-10-27 15:53 ` Alan Burlison
2015-10-27 23:17 ` Al Viro
2015-10-28 0:13 ` Eric Dumazet
2015-10-28 12:35 ` Al Viro
2015-10-28 13:24 ` Eric Dumazet
2015-10-28 14:47 ` Eric Dumazet
2015-10-28 21:13 ` Al Viro
2015-10-28 21:44 ` Eric Dumazet
2015-10-28 22:33 ` Al Viro
2015-10-28 23:08 ` Eric Dumazet
2015-10-29 0:15 ` Al Viro
2015-10-29 3:29 ` Eric Dumazet
2015-10-29 4:16 ` Al Viro
2015-10-29 12:35 ` Eric Dumazet
2015-10-29 13:48 ` Eric Dumazet
2015-10-30 17:18 ` Linus Torvalds
2015-10-30 21:02 ` Al Viro
2015-10-30 21:23 ` Linus Torvalds
2015-10-30 21:50 ` Linus Torvalds
2015-10-30 22:33 ` Al Viro
2015-10-30 23:52 ` Linus Torvalds
2015-10-31 0:09 ` Al Viro
2015-10-31 15:59 ` Eric Dumazet
2015-10-31 19:34 ` Al Viro
2015-10-31 19:54 ` Linus Torvalds
2015-10-31 20:29 ` Al Viro
2015-11-02 0:24 ` Al Viro
2015-11-02 0:59 ` Linus Torvalds
2015-11-02 2:14 ` Eric Dumazet
2015-11-02 6:22 ` Al Viro
2015-10-31 20:45 ` Eric Dumazet
2015-10-31 21:23 ` Linus Torvalds
2015-10-31 21:51 ` Al Viro
2015-10-31 22:34 ` Eric Dumazet
2015-10-31 1:07 ` Eric Dumazet
2015-10-28 16:04 ` Alan Burlison
2015-10-29 14:58 ` David Holland
2015-10-29 15:18 ` Alan Burlison
2015-10-29 16:01 ` David Holland
2015-10-29 16:15 ` Alan Burlison
2015-10-29 17:07 ` Al Viro
2015-10-29 17:12 ` Alan Burlison
2015-10-30 1:54 ` David Miller
2015-10-30 1:55 ` David Miller
2015-10-30 5:44 ` David Holland
2015-10-30 17:43 ` David Laight
2015-10-30 21:09 ` Al Viro
2015-11-04 15:54 ` David Laight
2015-11-04 16:27 ` Al Viro
2015-11-06 15:07 ` David Laight
2015-11-06 19:31 ` Al Viro
2015-10-22 6:51 ` Casper.Dik
2015-10-22 11:18 ` Alan Burlison
2015-10-22 11:15 ` Alan Burlison
2015-10-22 6:15 ` Casper.Dik
2015-10-22 11:30 ` Eric Dumazet
2015-10-22 11:58 ` Alan Burlison
2015-10-22 12:10 ` Eric Dumazet
2015-10-22 13:12 ` David Miller
2015-10-22 13:14 ` Alan Burlison
2015-10-22 17:05 ` Al Viro
2015-10-22 17:39 ` Alan Burlison
2015-10-22 18:56 ` Al Viro
2015-10-22 19:50 ` Casper.Dik
2015-10-23 17:09 ` Al Viro
2015-10-23 18:30 ` Fw: " David Holland
2015-10-23 19:51 ` Al Viro
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=562F577E.6000901@oracle.com \
--to=alan.burlison@oracle.com \
--cc=Casper.Dik@oracle.com \
--cc=davem@davemloft.net \
--cc=dholland-tech@netbsd.org \
--cc=eric.dumazet@gmail.com \
--cc=netdev@vger.kernel.org \
--cc=stephen@networkplumber.org \
--cc=viro@ZenIV.linux.org.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.