[next] arm64: allmodconfig: kernel BUG at include/linux/page-flags.h:509

[next] arm64: allmodconfig: kernel BUG at include/linux/page-flags.h:509

[Anders Roxell]

Hi,

I'm building and running an arm64 allmodconfig kernel on next.
Reacently I've seen this error below, for full log see [1].

[   56.000810][    T1] ------------[ cut here ]------------
[   56.003178][    T1] kernel BUG at include/linux/page-flags.h:509!
[   56.006291][    T1] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
[   56.008921][    T1] Modules linked in:
[   56.010600][    T1] CPU: 0 PID: 1 Comm: swapper/0 Tainted: G
        T 5.17.0-rc6-next-20220304 #1
ff4358f1e97b9d4d3a7966961f455fb8cb5c735d
[   56.015962][    T1] Hardware name: linux,dummy-virt (DT)
[   56.018189][    T1] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT
-SSBS BTYPE=--)
[   56.021355][    T1] pc : set_page_reservations+0xf8/0x3b8
[   56.023655][    T1] lr : set_page_reservations+0xec/0x3b8
[   56.025976][    T1] sp : ffff80000eb37c20
[   56.027699][    T1] x29: ffff80000eb37c20 x28: ffff0000079d8050
x27: 0000000000000001
[   56.031030][    T1] x26: ffff80000a2c6140 x25: ffff80000e240380
x24: ffff80000b780f70
[   56.034381][    T1] x23: fffffc0000000000 x22: 000000000028b100
x21: fffffc000028b100
[   56.037728][    T1] x20: ffff80000e23ffe0 x19: ffff80000e23ffe0
x18: 0000000000000000
[   56.041039][    T1] x17: 0000000000000000 x16: 0000000000000000
x15: 0000000000000000
[   56.044323][    T1] x14: 0000000000000000 x13: fffffffffffe34f0
x12: fffffffffffe3498
[   56.047644][    T1] x11: 0000000000000000 x10: 0000000000000000 x9
: 0000000000000000
[   56.050941][    T1] x8 : 0000000000000000 x7 : 0000000000000000 x6
: 0000000000000000
[   56.054258][    T1] x5 : 0000000000000000 x4 : 0000000000000000 x3
: 0000000000000000
[   56.057729][    T1] x2 : 0000000000000000 x1 : 0000000000000000 x0
: 0000000000000001
[   56.061075][    T1] Call trace:
[   56.062436][    T1]  set_page_reservations+0xf8/0x3b8
[   56.064709][    T1]  trace_events_user_init+0x1bc/0x3f8
[   56.067110][    T1]  do_one_initcall+0x2ac/0x6c0
[   56.069267][    T1]  do_initcalls+0x17c/0x244
[   56.071269][    T1]  kernel_init_freeable+0x2a0/0x344
[   56.073587][    T1]  kernel_init+0x34/0x180
[   56.075614][    T1]  ret_from_fork+0x10/0x20
[   56.077655][    T1] Code: 97937293 f9401a80 91000400 f9001a80 (d4210000)
[   56.080828][    T1] ---[ end trace 0000000000000000 ]---
[   56.083250][    T1] Kernel panic - not syncing: Oops - BUG: Fatal exception
[   56.086279][    T1] ---[ end Kernel panic - not syncing: Oops -
BUG: Fatal exception ]---

Reported-by: Anders Roxell <anders.roxell@linaro.org>

Any idea what happens?


Cheers,
Anders
[1] http://ix.io/3Rkj

Re: [next] arm64: allmodconfig: kernel BUG at include/linux/page-flags.h:509

[Steven Rostedt]

On Fri, 11 Mar 2022 10:43:32 +0100
Anders Roxell <anders.roxell@linaro.org> wrote:

> Hi,
> 
> I'm building and running an arm64 allmodconfig kernel on next.
> Reacently I've seen this error below, for full log see [1].
> 
> [   56.000810][    T1] ------------[ cut here ]------------
> [   56.003178][    T1] kernel BUG at include/linux/page-flags.h:509!
> [   56.006291][    T1] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
> [   56.008921][    T1] Modules linked in:
> [   56.010600][    T1] CPU: 0 PID: 1 Comm: swapper/0 Tainted: G
>         T 5.17.0-rc6-next-20220304 #1
> ff4358f1e97b9d4d3a7966961f455fb8cb5c735d
> [   56.015962][    T1] Hardware name: linux,dummy-virt (DT)
> [   56.018189][    T1] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT
> -SSBS BTYPE=--)
> [   56.021355][    T1] pc : set_page_reservations+0xf8/0x3b8
> [   56.023655][    T1] lr : set_page_reservations+0xec/0x3b8
> [   56.025976][    T1] sp : ffff80000eb37c20
> [   56.027699][    T1] x29: ffff80000eb37c20 x28: ffff0000079d8050
> x27: 0000000000000001
> [   56.031030][    T1] x26: ffff80000a2c6140 x25: ffff80000e240380
> x24: ffff80000b780f70
> [   56.034381][    T1] x23: fffffc0000000000 x22: 000000000028b100
> x21: fffffc000028b100
> [   56.037728][    T1] x20: ffff80000e23ffe0 x19: ffff80000e23ffe0
> x18: 0000000000000000
> [   56.041039][    T1] x17: 0000000000000000 x16: 0000000000000000
> x15: 0000000000000000
> [   56.044323][    T1] x14: 0000000000000000 x13: fffffffffffe34f0
> x12: fffffffffffe3498
> [   56.047644][    T1] x11: 0000000000000000 x10: 0000000000000000 x9
> : 0000000000000000
> [   56.050941][    T1] x8 : 0000000000000000 x7 : 0000000000000000 x6
> : 0000000000000000
> [   56.054258][    T1] x5 : 0000000000000000 x4 : 0000000000000000 x3
> : 0000000000000000
> [   56.057729][    T1] x2 : 0000000000000000 x1 : 0000000000000000 x0
> : 0000000000000001
> [   56.061075][    T1] Call trace:
> [   56.062436][    T1]  set_page_reservations+0xf8/0x3b8
> [   56.064709][    T1]  trace_events_user_init+0x1bc/0x3f8


Hmm,

static void set_page_reservations(bool set)
{
	int page;

	for (page = 0; page < MAX_PAGES; ++page) {
		void *addr = register_page_data + (PAGE_SIZE * page);

		if (set)
			SetPageReserved(virt_to_page(addr));
		else
			ClearPageReserved(virt_to_page(addr));
	}
}

static int __init trace_events_user_init(void)
{
	int ret;

	/* Zero all bits beside 0 (which is reserved for failures) */
	bitmap_zero(page_bitmap, MAX_EVENTS);
	set_bit(0, page_bitmap);

	register_page_data = kzalloc(MAX_EVENTS, GFP_KERNEL);

	if (!register_page_data)
		return -ENOMEM;

	set_page_reservations(true);

Beau,

Why are you reserving the page? Can't you just allocate a page and use that?

	page = alloc_pages(GFP_KERNEL | __GFP_ZERO, 0);
	register_page_data = page_address(page);

What was the purpose of setting it reserved?

-- Steve

> [   56.067110][    T1]  do_one_initcall+0x2ac/0x6c0
> [   56.069267][    T1]  do_initcalls+0x17c/0x244
> [   56.071269][    T1]  kernel_init_freeable+0x2a0/0x344
> [   56.073587][    T1]  kernel_init+0x34/0x180
> [   56.075614][    T1]  ret_from_fork+0x10/0x20
> [   56.077655][    T1] Code: 97937293 f9401a80 91000400 f9001a80
> (d4210000) [   56.080828][    T1] ---[ end trace 0000000000000000 ]---
> [   56.083250][    T1] Kernel panic - not syncing: Oops - BUG: Fatal
> exception [   56.086279][    T1] ---[ end Kernel panic - not syncing:
> Oops - BUG: Fatal exception ]---
> 
> Reported-by: Anders Roxell <anders.roxell@linaro.org>
> 
> Any idea what happens?
> 
> 
> Cheers,
> Anders
> [1] http://ix.io/3Rkj

Re: [next] arm64: allmodconfig: kernel BUG at include/linux/page-flags.h:509

[Beau Belgrave]

On Fri, Mar 11, 2022 at 11:24:26AM -0500, Steven Rostedt wrote:
> On Fri, 11 Mar 2022 10:43:32 +0100
> Anders Roxell <anders.roxell@linaro.org> wrote:
> 
> > Hi,
> > 
> > I'm building and running an arm64 allmodconfig kernel on next.
> > Reacently I've seen this error below, for full log see [1].
> > 
> > [   56.000810][    T1] ------------[ cut here ]------------
> > [   56.003178][    T1] kernel BUG at include/linux/page-flags.h:509!
> > [   56.006291][    T1] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
> > [   56.008921][    T1] Modules linked in:
> > [   56.010600][    T1] CPU: 0 PID: 1 Comm: swapper/0 Tainted: G
> >         T 5.17.0-rc6-next-20220304 #1
> > ff4358f1e97b9d4d3a7966961f455fb8cb5c735d
> > [   56.015962][    T1] Hardware name: linux,dummy-virt (DT)
> > [   56.018189][    T1] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT
> > -SSBS BTYPE=--)
> > [   56.021355][    T1] pc : set_page_reservations+0xf8/0x3b8
> > [   56.023655][    T1] lr : set_page_reservations+0xec/0x3b8
> > [   56.025976][    T1] sp : ffff80000eb37c20
> > [   56.027699][    T1] x29: ffff80000eb37c20 x28: ffff0000079d8050
> > x27: 0000000000000001
> > [   56.031030][    T1] x26: ffff80000a2c6140 x25: ffff80000e240380
> > x24: ffff80000b780f70
> > [   56.034381][    T1] x23: fffffc0000000000 x22: 000000000028b100
> > x21: fffffc000028b100
> > [   56.037728][    T1] x20: ffff80000e23ffe0 x19: ffff80000e23ffe0
> > x18: 0000000000000000
> > [   56.041039][    T1] x17: 0000000000000000 x16: 0000000000000000
> > x15: 0000000000000000
> > [   56.044323][    T1] x14: 0000000000000000 x13: fffffffffffe34f0
> > x12: fffffffffffe3498
> > [   56.047644][    T1] x11: 0000000000000000 x10: 0000000000000000 x9
> > : 0000000000000000
> > [   56.050941][    T1] x8 : 0000000000000000 x7 : 0000000000000000 x6
> > : 0000000000000000
> > [   56.054258][    T1] x5 : 0000000000000000 x4 : 0000000000000000 x3
> > : 0000000000000000
> > [   56.057729][    T1] x2 : 0000000000000000 x1 : 0000000000000000 x0
> > : 0000000000000001
> > [   56.061075][    T1] Call trace:
> > [   56.062436][    T1]  set_page_reservations+0xf8/0x3b8
> > [   56.064709][    T1]  trace_events_user_init+0x1bc/0x3f8
> 
> 
> Hmm,
> 
> static void set_page_reservations(bool set)
> {
> 	int page;
> 
> 	for (page = 0; page < MAX_PAGES; ++page) {
> 		void *addr = register_page_data + (PAGE_SIZE * page);
> 
> 		if (set)
> 			SetPageReserved(virt_to_page(addr));
> 		else
> 			ClearPageReserved(virt_to_page(addr));
> 	}
> }
> 
> static int __init trace_events_user_init(void)
> {
> 	int ret;
> 
> 	/* Zero all bits beside 0 (which is reserved for failures) */
> 	bitmap_zero(page_bitmap, MAX_EVENTS);
> 	set_bit(0, page_bitmap);
> 
> 	register_page_data = kzalloc(MAX_EVENTS, GFP_KERNEL);
> 
> 	if (!register_page_data)
> 		return -ENOMEM;
> 
> 	set_page_reservations(true);
> 
> Beau,
> 
> Why are you reserving the page? Can't you just allocate a page and use that?
> 
> 	page = alloc_pages(GFP_KERNEL | __GFP_ZERO, 0);
> 	register_page_data = page_address(page);
> 
> What was the purpose of setting it reserved?
> 

Maybe I am mistaken, but I remember reading in the case of a kernel
allocated page shared with user space they must be reserved. It was
stated that the PTE may not know the PFN belongs to user or kernel.

If this is not the case, I don't see why we couldn't allocate zero'd
pages as you describe. We just need to make sure we don't crash user
processes touching the page if it does get paged out (and ideally not
slow down their execution).

> -- Steve
> 
> > [   56.067110][    T1]  do_one_initcall+0x2ac/0x6c0
> > [   56.069267][    T1]  do_initcalls+0x17c/0x244
> > [   56.071269][    T1]  kernel_init_freeable+0x2a0/0x344
> > [   56.073587][    T1]  kernel_init+0x34/0x180
> > [   56.075614][    T1]  ret_from_fork+0x10/0x20
> > [   56.077655][    T1] Code: 97937293 f9401a80 91000400 f9001a80
> > (d4210000) [   56.080828][    T1] ---[ end trace 0000000000000000 ]---
> > [   56.083250][    T1] Kernel panic - not syncing: Oops - BUG: Fatal
> > exception [   56.086279][    T1] ---[ end Kernel panic - not syncing:
> > Oops - BUG: Fatal exception ]---
> > 
> > Reported-by: Anders Roxell <anders.roxell@linaro.org>
> > 
> > Any idea what happens?
> > 
> > 
> > Cheers,
> > Anders
> > [1] http://ix.io/3Rkj

Thanks,
-Beau

Re: [next] arm64: allmodconfig: kernel BUG at include/linux/page-flags.h:509

[Steven Rostedt]

On Fri, 11 Mar 2022 09:11:23 -0800
Beau Belgrave <beaub@linux.microsoft.com> wrote:

> Maybe I am mistaken, but I remember reading in the case of a kernel
> allocated page shared with user space they must be reserved. It was
> stated that the PTE may not know the PFN belongs to user or kernel.
> 
> If this is not the case, I don't see why we couldn't allocate zero'd
> pages as you describe. We just need to make sure we don't crash user
> processes touching the page if it does get paged out (and ideally not
> slow down their execution).

Anything allocated by the kernel via alloc_page() will never be paged out.
As the kernel may use it in any location (including NMIs). The ftrace ring
buffer allocates its pages this way.

Now if you were mapping some file system page, then that would be a
different story.

-- Steve

Re: [next] arm64: allmodconfig: kernel BUG at include/linux/page-flags.h:509

[Beau Belgrave]

On Fri, Mar 11, 2022 at 01:04:53PM -0500, Steven Rostedt wrote:
> On Fri, 11 Mar 2022 09:11:23 -0800
> Beau Belgrave <beaub@linux.microsoft.com> wrote:
> 
> > Maybe I am mistaken, but I remember reading in the case of a kernel
> > allocated page shared with user space they must be reserved. It was
> > stated that the PTE may not know the PFN belongs to user or kernel.
> > 
> > If this is not the case, I don't see why we couldn't allocate zero'd
> > pages as you describe. We just need to make sure we don't crash user
> > processes touching the page if it does get paged out (and ideally not
> > slow down their execution).
> 
> Anything allocated by the kernel via alloc_page() will never be paged out.
> As the kernel may use it in any location (including NMIs). The ftrace ring
> buffer allocates its pages this way.
> 

Is that true as well for the user process that has this mapped? Will the
user process virtual memory that maps to this ever page it out? I
believe that was the concern. My understanding is by reserving in
kernel, even though it won't page out on that side, marks the user side
entry to not let it page out.

The other thing is that this patch applied to 5.10.X on ARM64 does not
appear to hit this. Is it some weird interaction with something else or
was 5.10.X just getting lucky?

Thanks,
-Beau

Re: [next] arm64: allmodconfig: kernel BUG at include/linux/page-flags.h:509

[Steven Rostedt]

On Fri, 11 Mar 2022 10:13:28 -0800
Beau Belgrave <beaub@linux.microsoft.com> wrote:

> Is that true as well for the user process that has this mapped? Will the
> user process virtual memory that maps to this ever page it out? I
> believe that was the concern. My understanding is by reserving in
> kernel, even though it won't page out on that side, marks the user side
> entry to not let it page out.

My memory of the purpose of SetPageReserve() is a bit fuzzy, and there's
not much use of it in the kernel. Just a hand full. I'll try to investigate
it some more.

> 
> The other thing is that this patch applied to 5.10.X on ARM64 does not
> appear to hit this. Is it some weird interaction with something else or
> was 5.10.X just getting lucky?

Perhaps it's because you allocated the page with kzalloc and not just
getting a page directly?

Again, my knowledge in this area is lacking. I'm looking more into it.

-- Steve

Re: [next] arm64: allmodconfig: kernel BUG at include/linux/page-flags.h:509

[Beau Belgrave]

On Fri, Mar 11, 2022 at 06:15:00PM -0500, Steven Rostedt wrote:
> On Fri, 11 Mar 2022 10:13:28 -0800
> Beau Belgrave <beaub@linux.microsoft.com> wrote:
> 
> > Is that true as well for the user process that has this mapped? Will the
> > user process virtual memory that maps to this ever page it out? I
> > believe that was the concern. My understanding is by reserving in
> > kernel, even though it won't page out on that side, marks the user side
> > entry to not let it page out.
> 
> My memory of the purpose of SetPageReserve() is a bit fuzzy, and there's
> not much use of it in the kernel. Just a hand full. I'll try to investigate
> it some more.
> 

OK, you know more than I, so hopefully we can land on if we really need
it or not. For now I've left it in.

> > 
> > The other thing is that this patch applied to 5.10.X on ARM64 does not
> > appear to hit this. Is it some weird interaction with something else or
> > was 5.10.X just getting lucky?
> 
> Perhaps it's because you allocated the page with kzalloc and not just
> getting a page directly?
> 

Yeah, I think so too. I was able to repro locally and validate that using
alloc_pages directly fixes this by setting DEBUG_VM_PGFLAGS=y.

I've posted a patch for this:
https://lore.kernel.org/linux-trace-devel/20220311223028.1865-1-beaub@linux.microsoft.com/

> Again, my knowledge in this area is lacking. I'm looking more into it.
> 
> -- Steve

Thanks,
-Beau

Re: [next] arm64: allmodconfig: kernel BUG at include/linux/page-flags.h:509

[Steven Rostedt]

On Fri, 11 Mar 2022 15:28:09 -0800
Beau Belgrave <beaub@linux.microsoft.com> wrote:

> OK, you know more than I, so hopefully we can land on if we really need
> it or not. For now I've left it in.
> 
> > > 
> > > The other thing is that this patch applied to 5.10.X on ARM64 does not
> > > appear to hit this. Is it some weird interaction with something else or
> > > was 5.10.X just getting lucky?  
> > 
> > Perhaps it's because you allocated the page with kzalloc and not just
> > getting a page directly?

I sent a version that removes it, but I think we can add it back.

> >   
> 
> Yeah, I think so too. I was able to repro locally and validate that using
> alloc_pages directly fixes this by setting DEBUG_VM_PGFLAGS=y.
> 
> I've posted a patch for this:
> https://lore.kernel.org/linux-trace-devel/20220311223028.1865-1-beaub@linux.microsoft.com/

Just saw it (I screwed up my procmail settings getting rid of some more
spam, and ended up sending all my email to a ":" folder :-p)

If you want, I can send another version of my patch that doesn't remove the
reserve. But I like my patch as it sets the MAX_EVENTS based on a order and
not the other way around.

-- Steve

Re: [next] arm64: allmodconfig: kernel BUG at include/linux/page-flags.h:509

[Beau Belgrave]

On Fri, Mar 11, 2022 at 06:36:07PM -0500, Steven Rostedt wrote:
> On Fri, 11 Mar 2022 15:28:09 -0800
> Beau Belgrave <beaub@linux.microsoft.com> wrote:
> 
> > OK, you know more than I, so hopefully we can land on if we really need
> > it or not. For now I've left it in.
> > 
> > > > 
> > > > The other thing is that this patch applied to 5.10.X on ARM64 does not
> > > > appear to hit this. Is it some weird interaction with something else or
> > > > was 5.10.X just getting lucky?  
> > > 
> > > Perhaps it's because you allocated the page with kzalloc and not just
> > > getting a page directly?
> 
> I sent a version that removes it, but I think we can add it back.
> 
> > >   
> > 
> > Yeah, I think so too. I was able to repro locally and validate that using
> > alloc_pages directly fixes this by setting DEBUG_VM_PGFLAGS=y.
> > 
> > I've posted a patch for this:
> > https://lore.kernel.org/linux-trace-devel/20220311223028.1865-1-beaub@linux.microsoft.com/
> 
> Just saw it (I screwed up my procmail settings getting rid of some more
> spam, and ended up sending all my email to a ":" folder :-p)
> 
> If you want, I can send another version of my patch that doesn't remove the
> reserve. But I like my patch as it sets the MAX_EVENTS based on a order and
> not the other way around.

I never pass up free work (or drinks), I would prefer to keep the reserve
in there. Worst case it's redundant, best case it protects us from a
crash in a user process down the road.

Thanks,
-Beau