From: Nikolay Borisov <kernel@kyup.com>
To: Miklos Szeredi <mszeredi@redhat.com>,
Hannes Frederic Sowa <hannes@stressinduktion.org>
Cc: "Linux-Kernel@Vger. Kernel. Org" <linux-kernel@vger.kernel.org>,
netdev@vger.kernel.org
Subject: Re: kernel BUG at net/unix/garbage.c:149!"
Date: Tue, 30 Aug 2016 12:31:04 +0300 [thread overview]
Message-ID: <57C55258.8070509@kyup.com> (raw)
In-Reply-To: <CAOssrKcfncAYsQWkfLGFgoOxAQJVT2hYVWdBA6Cw7hhO8RJ_wQ@mail.gmail.com>
On 08/30/2016 12:18 PM, Miklos Szeredi wrote:
> On Tue, Aug 30, 2016 at 12:37 AM, Miklos Szeredi <mszeredi@redhat.com> wrote:
>> On Sat, Aug 27, 2016 at 11:55 AM, Miklos Szeredi <mszeredi@redhat.com> wrote:
>
>> crash> list -H gc_inflight_list unix_sock.link -s unix_sock.inflight |
>> grep counter | cut -d= -f2 | awk '{s+=$1} END {print s}'
>> 130
>> crash> p unix_tot_inflight
>> unix_tot_inflight = $2 = 135
>>
>> We've lost track of a total of five inflight sockets, so it's not a
>> one-off thing. Really weird... Now off to sleep, maybe I'll dream of
>> the solution.
>
> Okay, found one bug: gc assumes that in-flight sockets that don't have
> an external ref can't gain one while unix_gc_lock is held. That is
> true because unix_notinflight() will be called before detaching fds,
> which takes unix_gc_lock. Only MSG_PEEK was somehow overlooked. That
> one also clones the fds, also keeping them in the skb. But through
> MSG_PEEK an external reference can definitely be gained without ever
> touching unix_gc_lock.
>
> Not sure whether the reported bug can be explained by this. Can you
> confirm the MSG_PEEK was used in the setup?
>
> Does someone want to write a stress test for SCM_RIGHTS + MSG_PEEK?
>
> Anyway, attaching a fix that works by acquiring unix_gc_lock in case
> of MSG_PEEK also. It is trivially correct, but I haven't tested it.
I have no way of being 100% sure but looking through nginx's source code
it seems they do utilize MSG_PEEK on several occasions. This issue has
been apparently very hard to reproduce since I have 100s of servers
running a lot of NGINX processes and this has been triggered only once.
On a different note - if I inspect a live node without this patch should
the discrepancy between the gc_inflight_list and the unix_tot_inflight
be present VS with this patch applied?
>
> Thanks,
> Miklos
>
next prev parent reply other threads:[~2016-08-30 9:31 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-08-24 14:24 kernel BUG at net/unix/garbage.c:149!" Nikolay Borisov
2016-08-24 21:40 ` Hannes Frederic Sowa
2016-08-24 23:30 ` Nikolay Borisov
2016-08-26 20:24 ` Hannes Frederic Sowa
2016-08-27 9:55 ` Miklos Szeredi
2016-08-29 22:37 ` Miklos Szeredi
2016-08-30 9:18 ` Miklos Szeredi
2016-08-30 9:31 ` Nikolay Borisov [this message]
2016-08-30 9:39 ` Miklos Szeredi
2016-09-01 9:13 ` Hannes Frederic Sowa
2016-09-27 14:16 ` Nikolay Borisov
2016-09-27 14:43 ` Hannes Frederic Sowa
2016-09-28 2:05 ` David Miller
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=57C55258.8070509@kyup.com \
--to=kernel@kyup.com \
--cc=hannes@stressinduktion.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mszeredi@redhat.com \
--cc=netdev@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).