Re: [syzbot] general protection fault in __device_attach

Re: [syzbot] general protection fault in __device_attach

[Dmitry Vyukov]

On Mon, 6 Jun 2022 at 14:39, Dan Carpenter <dan.carpenter@oracle.com> wrote:
>
> On Sat, Jun 04, 2022 at 10:32:46AM +0200, 'Dmitry Vyukov' via syzkaller-bugs wrote:
> > On Fri, 3 Jun 2022 at 18:12, Greg KH <gregkh@linuxfoundation.org> wrote:
> > >
> > > But again, is this a "real and able to be triggered from userspace"
> > > problem, or just fault-injection-induced?
> >
> > Then this is something to fix in the fault injection subsystem.
> > Testing systems shouldn't be reporting false positives.
> > What allocations cannot fail in real life? Is it <=page_size?
> >
>
> Apparently in 2014, anything less than *EIGHT?!!* pages succeeded!
>
> https://lwn.net/Articles/627419/
>
> I have been on the look out since that article and never seen anyone
> mention it changing.  I think we should ignore that and say that
> anything over PAGE_SIZE can fail.  Possibly we could go smaller than
> PAGE_SIZE...

+linux-mm for GFP expertise re what allocations cannot possibly fail
and should be excluded from fault injection.

Interesting, thanks for the link.

PAGE_SIZE looks like a good start. Once we have the predicate in
place, we can refine it later when/if we have more inputs.

But I wonder about GFP flags. They definitely have some impact on allocations.
If GFP_ACCOUNT is set, all allocations can fail, right?
If GFP_DMA/DMA32 is set, allocations can fail, right? What about other zones?
If GFP_NORETRY is set, allocations can fail?
What about GFP_NOMEMALLOC and GFP_ATOMIC?
What about GFP_IO/GFP_FS/GFP_DIRECT_RECLAIM/GFP_KSWAPD_RECLAIM? At
least some of these need to be set for allocations to not fail? Which
ones?
Any other flags are required to be set/unset for allocations to not fail?

FTR here is quick link to flags list:
https://elixir.bootlin.com/linux/v5.19-rc1/source/include/linux/gfp.h#L32

Re: [syzbot] general protection fault in __device_attach

[Matthew Wilcox]

On Tue, Jun 07, 2022 at 09:15:09AM +0200, Dmitry Vyukov wrote:
> On Mon, 6 Jun 2022 at 14:39, Dan Carpenter <dan.carpenter@oracle.com> wrote:
> >
> > On Sat, Jun 04, 2022 at 10:32:46AM +0200, 'Dmitry Vyukov' via syzkaller-bugs wrote:
> > > On Fri, 3 Jun 2022 at 18:12, Greg KH <gregkh@linuxfoundation.org> wrote:
> > > >
> > > > But again, is this a "real and able to be triggered from userspace"
> > > > problem, or just fault-injection-induced?
> > >
> > > Then this is something to fix in the fault injection subsystem.
> > > Testing systems shouldn't be reporting false positives.
> > > What allocations cannot fail in real life? Is it <=page_size?
> > >
> >
> > Apparently in 2014, anything less than *EIGHT?!!* pages succeeded!
> >
> > https://lwn.net/Articles/627419/
> >
> > I have been on the look out since that article and never seen anyone
> > mention it changing.  I think we should ignore that and say that
> > anything over PAGE_SIZE can fail.  Possibly we could go smaller than
> > PAGE_SIZE...
> 
> +linux-mm for GFP expertise re what allocations cannot possibly fail
> and should be excluded from fault injection.
> 
> Interesting, thanks for the link.
> 
> PAGE_SIZE looks like a good start. Once we have the predicate in
> place, we can refine it later when/if we have more inputs.
> 
> But I wonder about GFP flags. They definitely have some impact on allocations.
> If GFP_ACCOUNT is set, all allocations can fail, right?
> If GFP_DMA/DMA32 is set, allocations can fail, right? What about other zones?
> If GFP_NORETRY is set, allocations can fail?
> What about GFP_NOMEMALLOC and GFP_ATOMIC?
> What about GFP_IO/GFP_FS/GFP_DIRECT_RECLAIM/GFP_KSWAPD_RECLAIM? At
> least some of these need to be set for allocations to not fail? Which
> ones?
> Any other flags are required to be set/unset for allocations to not fail?

I'm not the expert on page allocation, but ...

I don't think GFP_ACCOUNT makes allocations fail.  It might make reclaim
happen from within that cgroup, and it might cause an OOM kill for
something in that cgroup.  But I don't think it makes a (low order)
allocation more likely to fail.

There's usually less memory avilable in DMA/DMA32 zones, but we have
so few allocations from those zones, I question the utility of focusing
testing on those allocations.

GFP_ATOMIC allows access to emergency pools, so I would say _less_ likely
to fail.  KSWAPD_RECLAIM has no effect on whether _this_ allocation
succeeds or fails; it kicks kswapd to do reclaim, rather than doing
reclaim directly.  DIRECT_RECLAIM definitely makes allocations more likely
to succeed.  GFP_FS allows (direct) reclaim to happen from filesystems.
GFP_IO allows IO to start (ie writeback can start) in order to clean
dirty memory.

Anyway, I hope somebody who knows the page allocator better than I do
can say smarter things than this.  Even better if they can put it into
Documentation/ somewhere ;-)

https://www.kernel.org/doc/html/latest/core-api/memory-allocation.html
exists but isn't quite enough to answer this question.

Re: [syzbot] general protection fault in __device_attach

[Dmitry Vyukov]

On Wed, 8 Jun 2022 at 05:25, Matthew Wilcox <willy@infradead.org> wrote:
>
> On Tue, Jun 07, 2022 at 09:15:09AM +0200, Dmitry Vyukov wrote:
> > On Mon, 6 Jun 2022 at 14:39, Dan Carpenter <dan.carpenter@oracle.com> wrote:
> > >
> > > On Sat, Jun 04, 2022 at 10:32:46AM +0200, 'Dmitry Vyukov' via syzkaller-bugs wrote:
> > > > On Fri, 3 Jun 2022 at 18:12, Greg KH <gregkh@linuxfoundation.org> wrote:
> > > > >
> > > > > But again, is this a "real and able to be triggered from userspace"
> > > > > problem, or just fault-injection-induced?
> > > >
> > > > Then this is something to fix in the fault injection subsystem.
> > > > Testing systems shouldn't be reporting false positives.
> > > > What allocations cannot fail in real life? Is it <=page_size?
> > > >
> > >
> > > Apparently in 2014, anything less than *EIGHT?!!* pages succeeded!
> > >
> > > https://lwn.net/Articles/627419/
> > >
> > > I have been on the look out since that article and never seen anyone
> > > mention it changing.  I think we should ignore that and say that
> > > anything over PAGE_SIZE can fail.  Possibly we could go smaller than
> > > PAGE_SIZE...
> >
> > +linux-mm for GFP expertise re what allocations cannot possibly fail
> > and should be excluded from fault injection.
> >
> > Interesting, thanks for the link.
> >
> > PAGE_SIZE looks like a good start. Once we have the predicate in
> > place, we can refine it later when/if we have more inputs.
> >
> > But I wonder about GFP flags. They definitely have some impact on allocations.
> > If GFP_ACCOUNT is set, all allocations can fail, right?
> > If GFP_DMA/DMA32 is set, allocations can fail, right? What about other zones?
> > If GFP_NORETRY is set, allocations can fail?
> > What about GFP_NOMEMALLOC and GFP_ATOMIC?
> > What about GFP_IO/GFP_FS/GFP_DIRECT_RECLAIM/GFP_KSWAPD_RECLAIM? At
> > least some of these need to be set for allocations to not fail? Which
> > ones?
> > Any other flags are required to be set/unset for allocations to not fail?
>
> I'm not the expert on page allocation, but ...
>
> I don't think GFP_ACCOUNT makes allocations fail.  It might make reclaim
> happen from within that cgroup, and it might cause an OOM kill for
> something in that cgroup.  But I don't think it makes a (low order)
> allocation more likely to fail.

Interesting.
I was thinking of some malicious specifically crafted configurations
with very low limit and particular pattern of allocations. Also what
if there is just 1 process (current)? Is it possible to kill and
reclaim the current process when a thread is stuck in the middle of
the kernel on a kmalloc?
Also I see e.g.:
        Tasks with the OOM protection (oom_score_adj set to -1000)
        are treated as an exception and are never killed.

I am not an expert on this either, but I think it may be hard to fight
with a specifically crafted attack.


> There's usually less memory avilable in DMA/DMA32 zones, but we have
> so few allocations from those zones, I question the utility of focusing
> testing on those allocations.
>
> GFP_ATOMIC allows access to emergency pools, so I would say _less_ likely
> to fail.  KSWAPD_RECLAIM has no effect on whether _this_ allocation
> succeeds or fails; it kicks kswapd to do reclaim, rather than doing
> reclaim directly.  DIRECT_RECLAIM definitely makes allocations more likely
> to succeed.  GFP_FS allows (direct) reclaim to happen from filesystems.
> GFP_IO allows IO to start (ie writeback can start) in order to clean
> dirty memory.
>
> Anyway, I hope somebody who knows the page allocator better than I do
> can say smarter things than this.  Even better if they can put it into
> Documentation/ somewhere ;-)

Even better to put this into code as a predicate function that fault
injection will use. It will also serve as precise up-to-date
documentation.

> https://www.kernel.org/doc/html/latest/core-api/memory-allocation.html
> exists but isn't quite enough to answer this question.

Re: [syzbot] general protection fault in __device_attach

[Dmitry Vyukov]

On Wed, 8 Jun 2022 at 10:20, Dmitry Vyukov <dvyukov@google.com> wrote:
> > On Tue, Jun 07, 2022 at 09:15:09AM +0200, Dmitry Vyukov wrote:
> > > On Mon, 6 Jun 2022 at 14:39, Dan Carpenter <dan.carpenter@oracle.com> wrote:
> > > >
> > > > On Sat, Jun 04, 2022 at 10:32:46AM +0200, 'Dmitry Vyukov' via syzkaller-bugs wrote:
> > > > > On Fri, 3 Jun 2022 at 18:12, Greg KH <gregkh@linuxfoundation.org> wrote:
> > > > > >
> > > > > > But again, is this a "real and able to be triggered from userspace"
> > > > > > problem, or just fault-injection-induced?
> > > > >
> > > > > Then this is something to fix in the fault injection subsystem.
> > > > > Testing systems shouldn't be reporting false positives.
> > > > > What allocations cannot fail in real life? Is it <=page_size?
> > > > >
> > > >
> > > > Apparently in 2014, anything less than *EIGHT?!!* pages succeeded!
> > > >
> > > > https://lwn.net/Articles/627419/
> > > >
> > > > I have been on the look out since that article and never seen anyone
> > > > mention it changing.  I think we should ignore that and say that
> > > > anything over PAGE_SIZE can fail.  Possibly we could go smaller than
> > > > PAGE_SIZE...
> > >
> > > +linux-mm for GFP expertise re what allocations cannot possibly fail
> > > and should be excluded from fault injection.
> > >
> > > Interesting, thanks for the link.
> > >
> > > PAGE_SIZE looks like a good start. Once we have the predicate in
> > > place, we can refine it later when/if we have more inputs.
> > >
> > > But I wonder about GFP flags. They definitely have some impact on allocations.
> > > If GFP_ACCOUNT is set, all allocations can fail, right?
> > > If GFP_DMA/DMA32 is set, allocations can fail, right? What about other zones?
> > > If GFP_NORETRY is set, allocations can fail?
> > > What about GFP_NOMEMALLOC and GFP_ATOMIC?
> > > What about GFP_IO/GFP_FS/GFP_DIRECT_RECLAIM/GFP_KSWAPD_RECLAIM? At
> > > least some of these need to be set for allocations to not fail? Which
> > > ones?
> > > Any other flags are required to be set/unset for allocations to not fail?
> >
> > I'm not the expert on page allocation, but ...
> >
> > I don't think GFP_ACCOUNT makes allocations fail.  It might make reclaim
> > happen from within that cgroup, and it might cause an OOM kill for
> > something in that cgroup.  But I don't think it makes a (low order)
> > allocation more likely to fail.
>
> Interesting.
> I was thinking of some malicious specifically crafted configurations
> with very low limit and particular pattern of allocations. Also what
> if there is just 1 process (current)? Is it possible to kill and
> reclaim the current process when a thread is stuck in the middle of
> the kernel on a kmalloc?
> Also I see e.g.:
>         Tasks with the OOM protection (oom_score_adj set to -1000)
>         are treated as an exception and are never killed.
>
> I am not an expert on this either, but I think it may be hard to fight
> with a specifically crafted attack.
>
>
> > There's usually less memory avilable in DMA/DMA32 zones, but we have
> > so few allocations from those zones, I question the utility of focusing
> > testing on those allocations.
> >
> > GFP_ATOMIC allows access to emergency pools, so I would say _less_ likely
> > to fail.  KSWAPD_RECLAIM has no effect on whether _this_ allocation
> > succeeds or fails; it kicks kswapd to do reclaim, rather than doing
> > reclaim directly.  DIRECT_RECLAIM definitely makes allocations more likely
> > to succeed.  GFP_FS allows (direct) reclaim to happen from filesystems.
> > GFP_IO allows IO to start (ie writeback can start) in order to clean
> > dirty memory.
> >
> > Anyway, I hope somebody who knows the page allocator better than I do
> > can say smarter things than this.  Even better if they can put it into
> > Documentation/ somewhere ;-)
>
> Even better to put this into code as a predicate function that fault
> injection will use. It will also serve as precise up-to-date
> documentation.

Also at the end of kmalloc as:
WARN_ON(!ret && !cant_fail(size, gfp));
!

> > https://www.kernel.org/doc/html/latest/core-api/memory-allocation.html
> > exists but isn't quite enough to answer this question.