Re: WARNING in kmalloc_slab (3)

Re: WARNING in kmalloc_slab (3)

[Eric Biggers]

+Cc linux-block

On Sun, Dec 03, 2017 at 06:25:01AM -0800, syzbot wrote:
> WARNING: CPU: 0 PID: 3081 at mm/slab_common.c:971 kmalloc_slab+0x5d/0x70
> mm/slab_common.c:971
> Kernel panic - not syncing: panic_on_warn set ...
> 
[...]
>  __do_kmalloc mm/slab.c:3706 [inline]
>  __kmalloc+0x25/0x760 mm/slab.c:3720
>  kmalloc include/linux/slab.h:504 [inline]
>  relay_create_buf kernel/relay.c:172 [inline]
>  relay_open_buf.part.10+0xc8/0x9b0 kernel/relay.c:449
>  relay_open_buf kernel/relay.c:446 [inline]
>  relay_open+0x57a/0xa40 kernel/relay.c:596
>  do_blk_trace_setup+0x4a4/0xcd0 kernel/trace/blktrace.c:544
>  __blk_trace_setup+0xb6/0x140 kernel/trace/blktrace.c:589
>  blk_trace_ioctl+0x1d5/0x2a0 kernel/trace/blktrace.c:728
>  blkdev_ioctl+0x1845/0x1e00 block/ioctl.c:587
>  block_ioctl+0xea/0x130 fs/block_dev.c:1860
>  vfs_ioctl fs/ioctl.c:46 [inline]
>  do_vfs_ioctl+0x1b1/0x1530 fs/ioctl.c:686
>  SYSC_ioctl fs/ioctl.c:701 [inline]
>  SyS_ioctl+0x8f/0xc0 fs/ioctl.c:692
>  entry_SYSCALL_64_fastpath+0x1f/0x96

Looks like BLKTRACESETUP doesn't limit the '.buf_nr' parameter, allowing anyone
who can open a block device to cause an extremely large kmalloc.  Here's a
simplified reproducer:

#include <fcntl.h>
#include <linux/blktrace_api.h>
#include <linux/fs.h>
#include <sys/ioctl.h>

int main()
{
        int fd;
        struct blk_user_trace_setup setup = {
                .buf_size = 100,
                .buf_nr = 1000000,
        };

        fd = open("/dev/loop0", O_RDWR);
        ioctl(fd, BLKTRACESETUP, &setup);
}

Re: WARNING in kmalloc_slab (3)

[Dan Carpenter]

On Sun, Dec 03, 2017 at 12:16:08PM -0800, Eric Biggers wrote:
> Looks like BLKTRACESETUP doesn't limit the '.buf_nr' parameter, allowing anyone
> who can open a block device to cause an extremely large kmalloc.  Here's a
> simplified reproducer:
> 

There are lots of places which allow people to allocate as much as they
want.  With Syzcaller, you might want to just hard code a __GFP_NOWARN
in to disable it.

regards,
dan carpenter

Re: WARNING in kmalloc_slab (3)

[Dmitry Vyukov]

On Mon, Dec 4, 2017 at 9:14 AM, Dan Carpenter <dan.carpenter@oracle.com> wrote:
> On Sun, Dec 03, 2017 at 12:16:08PM -0800, Eric Biggers wrote:
>> Looks like BLKTRACESETUP doesn't limit the '.buf_nr' parameter, allowing anyone
>> who can open a block device to cause an extremely large kmalloc.  Here's a
>> simplified reproducer:
>>
>
> There are lots of places which allow people to allocate as much as they
> want.  With Syzcaller, you might want to just hard code a __GFP_NOWARN
> in to disable it.

Hi,

Hard code it where?

User-controllable allocation are supposed to use __GFP_NOWARN.

Re: WARNING in kmalloc_slab (3)

[Dan Carpenter]

On Mon, Dec 04, 2017 at 09:18:05AM +0100, Dmitry Vyukov wrote:
> On Mon, Dec 4, 2017 at 9:14 AM, Dan Carpenter <dan.carpenter@oracle.com> wrote:
> > On Sun, Dec 03, 2017 at 12:16:08PM -0800, Eric Biggers wrote:
> >> Looks like BLKTRACESETUP doesn't limit the '.buf_nr' parameter, allowing anyone
> >> who can open a block device to cause an extremely large kmalloc.  Here's a
> >> simplified reproducer:
> >>
> >
> > There are lots of places which allow people to allocate as much as they
> > want.  With Syzcaller, you might want to just hard code a __GFP_NOWARN
> > in to disable it.
> 
> Hi,
> 
> Hard code it where?

My idea was to just make warn_alloc() a no-op.

> 
> User-controllable allocation are supposed to use __GFP_NOWARN.

No that's not right.  What we don't want is unprivileged users to use
all the memory and we don't want unprivileged users to spam
/var/log/messages.  But you have to have slightly elevated permissions
to open block devices right?  The warning is helpful.  Admins should
"don't do that" if they don't want the warning.

The kernel really isn't designed to work with Oops on Warn.  I try to
tell people simple thinks like not printing a warning when
copy_from_user() fails because I don't want /var/log/messages to get
spammed.  But there are lots and lots of places which generate warnings.

regards,
dan carpenter

Re: WARNING in kmalloc_slab (3)

[Dmitry Vyukov]

On Mon, Dec 4, 2017 at 10:26 AM, Dan Carpenter <dan.carpenter@oracle.com> wrote:
> On Mon, Dec 04, 2017 at 09:18:05AM +0100, Dmitry Vyukov wrote:
>> On Mon, Dec 4, 2017 at 9:14 AM, Dan Carpenter <dan.carpenter@oracle.com> wrote:
>> > On Sun, Dec 03, 2017 at 12:16:08PM -0800, Eric Biggers wrote:
>> >> Looks like BLKTRACESETUP doesn't limit the '.buf_nr' parameter, allowing anyone
>> >> who can open a block device to cause an extremely large kmalloc.  Here's a
>> >> simplified reproducer:
>> >>
>> >
>> > There are lots of places which allow people to allocate as much as they
>> > want.  With Syzcaller, you might want to just hard code a __GFP_NOWARN
>> > in to disable it.
>>
>> Hi,
>>
>> Hard code it where?
>
> My idea was to just make warn_alloc() a no-op.

Yes, but how?
We specifically don't have any private patches, etc. That would cause
a bunch of much more serious problems. The system tracks HEAD of
multiple upstream repositories.
Starting testing non-upstream branches with all bad consequences,
especially for something that has an official solution and that
solution is very simple (adding __GFP_NOWARN), looks like a wrong
direction.


>> User-controllable allocation are supposed to use __GFP_NOWARN.
>
> No that's not right.  What we don't want is unprivileged users to use
> all the memory and we don't want unprivileged users to spam
> /var/log/messages.  But you have to have slightly elevated permissions
> to open block devices right?  The warning is helpful.  Admins should
> "don't do that" if they don't want the warning.
>
> The kernel really isn't designed to work with Oops on Warn.  I try to
> tell people simple thinks like not printing a warning when
> copy_from_user() fails because I don't want /var/log/messages to get
> spammed.  But there are lots and lots of places which generate warnings.


Yes, but we also want kernel to be testable. And preferably in mostly
automated way to not hire an army of monkeys to sort out all crash
reports (we currently hit around 14 crashes per minute).

I don't question that notifying user about incorrect arguments is
useful (though, kernel generally don't do for every "return -EINVAL").
But that doesn't need to be WARNING. pr_err can do. And if we are
talking about end user, pr_err can actually provide an much better
error message (for a non-kernel developer "WARNING: CPU: 0 PID: 3081
at mm/slab_common.c:971 kmalloc_slab+0x5d/0x70 mm/slab_common.c:971"
is like wat?).

Re: WARNING in kmalloc_slab (3)

[Eric Biggers]

On Mon, Dec 04, 2017 at 12:26:32PM +0300, Dan Carpenter wrote:
> On Mon, Dec 04, 2017 at 09:18:05AM +0100, Dmitry Vyukov wrote:
> > On Mon, Dec 4, 2017 at 9:14 AM, Dan Carpenter <dan.carpenter@oracle.com> wrote:
> > > On Sun, Dec 03, 2017 at 12:16:08PM -0800, Eric Biggers wrote:
> > >> Looks like BLKTRACESETUP doesn't limit the '.buf_nr' parameter, allowing anyone
> > >> who can open a block device to cause an extremely large kmalloc.  Here's a
> > >> simplified reproducer:
> > >>
> > >
> > > There are lots of places which allow people to allocate as much as they
> > > want.  With Syzcaller, you might want to just hard code a __GFP_NOWARN
> > > in to disable it.
> > 
> > Hi,
> > 
> > Hard code it where?
> 
> My idea was to just make warn_alloc() a no-op.
> 
> > 
> > User-controllable allocation are supposed to use __GFP_NOWARN.
> 
> No that's not right.  What we don't want is unprivileged users to use
> all the memory and we don't want unprivileged users to spam
> /var/log/messages.  But you have to have slightly elevated permissions
> to open block devices right?  The warning is helpful.  Admins should
> "don't do that" if they don't want the warning.

WARN_ON() should only be used for kernel bugs.  printk can be a different story.
If it's a "userspace shouldn't do this" kind of thing, then if there is any
message at all it should be a rate-limited printk that actually explains what
the problem is, not a random WARN_ON() that can only be interpreted by kernel
developers.

And yes, the fact that anyone with read access to any block device, even e.g. a
loop device, can cause the kernel to do an unbounded kmalloc *is* a bug.  It
needs to have a reasonable limit.  It is not a problem on all systems, but on
some systems "the admin" might give users read access to some block devices.

Eric

Re: WARNING in kmalloc_slab (3)

[Dmitry Vyukov]

On Tue, Dec 12, 2017 at 10:22 PM, Eric Biggers <ebiggers3@gmail.com> wrote:
> On Mon, Dec 04, 2017 at 12:26:32PM +0300, Dan Carpenter wrote:
>> On Mon, Dec 04, 2017 at 09:18:05AM +0100, Dmitry Vyukov wrote:
>> > On Mon, Dec 4, 2017 at 9:14 AM, Dan Carpenter <dan.carpenter@oracle.com> wrote:
>> > > On Sun, Dec 03, 2017 at 12:16:08PM -0800, Eric Biggers wrote:
>> > >> Looks like BLKTRACESETUP doesn't limit the '.buf_nr' parameter, allowing anyone
>> > >> who can open a block device to cause an extremely large kmalloc.  Here's a
>> > >> simplified reproducer:
>> > >>
>> > >
>> > > There are lots of places which allow people to allocate as much as they
>> > > want.  With Syzcaller, you might want to just hard code a __GFP_NOWARN
>> > > in to disable it.
>> >
>> > Hi,
>> >
>> > Hard code it where?
>>
>> My idea was to just make warn_alloc() a no-op.
>>
>> >
>> > User-controllable allocation are supposed to use __GFP_NOWARN.
>>
>> No that's not right.  What we don't want is unprivileged users to use
>> all the memory and we don't want unprivileged users to spam
>> /var/log/messages.  But you have to have slightly elevated permissions
>> to open block devices right?  The warning is helpful.  Admins should
>> "don't do that" if they don't want the warning.
>
> WARN_ON() should only be used for kernel bugs.  printk can be a different story.
> If it's a "userspace shouldn't do this" kind of thing, then if there is any
> message at all it should be a rate-limited printk that actually explains what
> the problem is, not a random WARN_ON() that can only be interpreted by kernel
> developers.
>
> And yes, the fact that anyone with read access to any block device, even e.g. a
> loop device, can cause the kernel to do an unbounded kmalloc *is* a bug.  It
> needs to have a reasonable limit.  It is not a problem on all systems, but on
> some systems "the admin" might give users read access to some block devices.



#syz fix: kernel/relay.c: limit kmalloc size to KMALLOC_MAX_SIZE