From: Dmitry Vyukov <dvyukov@google.com>
To: manfred <manfred@colorfullife.com>
Cc: syzbot+1145ec2e23165570c3ac@syzkaller.appspotmail.com,
Andrew Morton <akpm@linux-foundation.org>,
David Howells <dhowells@redhat.com>,
"Eric W. Biederman" <ebiederm@xmission.com>,
ktsanaktsidis@zendesk.com, LKML <linux-kernel@vger.kernel.org>,
Michal Hocko <mhocko@suse.com>,
Mike Rapoport <rppt@linux.vnet.ibm.com>,
Stephen Rothwell <sfr@canb.auug.org.au>,
syzkaller-bugs <syzkaller-bugs@googlegroups.com>,
Matthew Wilcox <willy@infradead.org>,
Davidlohr Bueso <dave@stgolabs.net>
Subject: Re: general protection fault in put_pid
Date: Sun, 23 Dec 2018 11:30:31 +0100 [thread overview]
Message-ID: <CACT4Y+ZitcDg=OA8NbJJTWQyHHzmiZ7rFekFkgxmdOOnSKTfXg@mail.gmail.com> (raw)
In-Reply-To: <CACT4Y+Zynq-URqgJNyCZyTPM80GU8DndtKv=BFDU9_U=f0KceA@mail.gmail.com>
On Sun, Dec 23, 2018 at 10:57 AM Dmitry Vyukov <dvyukov@google.com> wrote:
>
> On Sun, Dec 23, 2018 at 8:37 AM Dmitry Vyukov <dvyukov@google.com> wrote:
> >
> > On Sat, Dec 22, 2018 at 8:07 PM Manfred Spraul <manfred@colorfullife.com> wrote:
> > >
> > > Hi Dmitry,
> > >
> > > On 12/20/18 4:36 PM, Dmitry Vyukov wrote:
> > > > On Wed, Dec 19, 2018 at 10:04 AM Manfred Spraul
> > > > <manfred@colorfullife.com> wrote:
> > > >> Hello Dmitry,
> > > >>
> > > >> On 12/12/18 11:55 AM, Dmitry Vyukov wrote:
> > > >>> On Tue, Dec 11, 2018 at 9:23 PM syzbot
> > > >>> <syzbot+1145ec2e23165570c3ac@syzkaller.appspotmail.com> wrote:
> > > >>>> Hello,
> > > >>>>
> > > >>>> syzbot found the following crash on:
> > > >>>>
> > > >>>> HEAD commit: f5d582777bcb Merge branch 'for-linus' of git://git.kernel...
> > > >>>> git tree: upstream
> > > >>>> console output: https://syzkaller.appspot.com/x/log.txt?x=135bc547400000
> > > >>>> kernel config: https://syzkaller.appspot.com/x/.config?x=c8970c89a0efbb23
> > > >>>> dashboard link: https://syzkaller.appspot.com/bug?extid=1145ec2e23165570c3ac
> > > >>>> compiler: gcc (GCC) 8.0.1 20180413 (experimental)
> > > >>>> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=16803afb400000
> > > >>> +Manfred, this looks similar to the other few crashes related to
> > > >>> semget$private(0x0, 0x4000, 0x3f) that you looked at.
> > > >> I found one unexpected (incorrect?) locking, see the attached patch.
> > > >>
> > > >> But I doubt that this is the root cause of the crashes.
> > > >
> > > > But why? These one-off sporadic crashes reported by syzbot looks
> > > > exactly like a subtle race and your patch touches sem_exit_ns involved
> > > > in all reports.
> > > > So if you don't spot anything else, I would say close these 3 reports
> > > > with this patch (I see you already included Reported-by tags which is
> > > > great!) and then wait for syzbot reaction. Since we got 3 of them, if
> > > > it's still not fixed I would expect that syzbot will be able to
> > > > retrigger this later again.
> > >
> > > As I wrote, unless semop() is used, sma->use_global_lock is always 9 and
> > > nothing can happen.
> > >
> > > Every single-operation semop() reduces use_global_lock by one, i.e a
> > > single semop call as done here cannot trigger the bug:
> > >
> > > https://syzkaller.appspot.com/text?tag=ReproSyz&x=16803afb400000
> >
> > It contains "repeat":true,"procs":6, which means that it run 6
> > processes running this test in infinite loop. The last mark about
> > number of tests executed was:
> > 2018/12/11 18:38:02 executed programs: 2955
> >
> > > But, one more finding:
> > >
> > > https://syzkaller.appspot.com/bug?extid=1145ec2e23165570c3ac
> > >
> > > https://syzkaller.appspot.com/text?tag=CrashLog&x=109ecf6e400000
> > >
> > > The log file contain 1080 lines like these:
> > >
> > > > semget$private(..., 0x4003, ...)
> > > >
> > > > semget$private(..., 0x4006, ...)
> > > >
> > > > semget$private(..., 0x4007, ...)
> > >
> > > It ends up as kmalloc(128*0x400x), i.e. slightly more than 2 MB, an
> > > allocation in the 4 MB kmalloc buffer:
> > >
> > > > [ 1201.210245] kmalloc-4194304 4698112KB 4698112KB
> > > >
> > > i.e.: 1147 4 MB kmalloc blocks --> are we leaking nearly 100% of the
> > > semaphore arrays??
> >
> > /\/\/\/\/\/\
> >
> > Ha, this is definitely not healthy.
>
> I can reproduce this infinite memory consumption with the C program:
> https://gist.githubusercontent.com/dvyukov/03ec54b3429ade16fa07bf8b2379aff3/raw/ae4f654e279810de2505e8fa41b73dc1d77778e6/gistfile1.txt
>
> But this is working as intended, right? It just creates infinite
> number of large semaphore sets, which reasonably consumes infinite
> amount of memory.
> Except that it also violates the memcg bound and a process can have
> effectively unlimited amount of such "drum memory" in semaphores.
>
>
>
>
> > > This one looks similar:
> > >
> > > https://syzkaller.appspot.com/bug?extid=c92d3646e35bc5d1a909
> > >
> > > except that the array sizes are mixed, and thus there are kmalloc-1M and
> > > kmalloc-2M as well.
> > >
> > > (and I did not count the number of semget calls)
> > >
> > >
> > > The test apps use unshare(CLONE_NEWNS) and unshare(CLONE_NEWIPC), correct?
> > >
> > > I.e. no CLONE_NEWUSER.
> > >
> > > https://github.com/google/syzkaller/blob/master/executor/common_linux.h#L1523
> >
> > CLONE_NEWUSER is used on some instances as well:
> > https://github.com/google/syzkaller/blob/master/executor/common_linux.h#L1765
> > This crash happened on 2 different instances and 1 of them uses
> > CLONE_NEWUSER and another does not.
> > If it's important because of CAP_ADMIN in IPC namespace, then all
> > instances should have it (instances that don't use NEWUSER are just
> > root).
My naive attempts to re-reproduce this failed so far.
But I noticed that _all_ logs for these 3 crashes:
https://syzkaller.appspot.com/bug?extid=c92d3646e35bc5d1a909
https://syzkaller.appspot.com/bug?extid=1145ec2e23165570c3ac
https://syzkaller.appspot.com/bug?extid=9d8b6fa6ee7636f350c1
involve low memory conditions. My gut feeling says this is not a
coincidence. This is also probably the reason why all reproducers
create large sem sets. There must be some bad interaction between low
memory condition and semaphores/ipc namespaces.
next prev parent reply other threads:[~2018-12-23 10:30 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-12-11 20:23 general protection fault in put_pid syzbot
2018-12-12 10:55 ` Dmitry Vyukov
2018-12-19 9:04 ` Manfred Spraul
2018-12-20 15:36 ` Dmitry Vyukov
2018-12-22 19:07 ` Manfred Spraul
2018-12-23 7:37 ` Dmitry Vyukov
2018-12-23 9:57 ` Dmitry Vyukov
2018-12-23 10:30 ` Dmitry Vyukov [this message]
2018-12-23 10:42 ` Dmitry Vyukov
2018-12-23 12:32 ` Manfred Spraul
2018-12-25 9:35 ` Dmitry Vyukov
2018-12-26 9:03 ` Dmitry Vyukov
2018-12-30 9:31 ` Dmitry Vyukov
2018-12-31 6:35 ` Dmitry Vyukov
2018-12-25 9:41 ` Dmitry Vyukov
2018-12-23 12:25 ` Manfred Spraul
2019-01-03 22:18 ` Shakeel Butt
2019-01-07 18:04 ` Manfred Spraul
2019-01-07 18:22 ` Shakeel Butt
2019-03-27 20:10 ` syzbot
2019-03-27 20:27 ` Matthew Wilcox
2019-03-27 22:51 ` David Sterba
2019-11-07 13:42 ` syzbot
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CACT4Y+ZitcDg=OA8NbJJTWQyHHzmiZ7rFekFkgxmdOOnSKTfXg@mail.gmail.com' \
--to=dvyukov@google.com \
--cc=akpm@linux-foundation.org \
--cc=dave@stgolabs.net \
--cc=dhowells@redhat.com \
--cc=ebiederm@xmission.com \
--cc=ktsanaktsidis@zendesk.com \
--cc=linux-kernel@vger.kernel.org \
--cc=manfred@colorfullife.com \
--cc=mhocko@suse.com \
--cc=rppt@linux.vnet.ibm.com \
--cc=sfr@canb.auug.org.au \
--cc=syzbot+1145ec2e23165570c3ac@syzkaller.appspotmail.com \
--cc=syzkaller-bugs@googlegroups.com \
--cc=willy@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).