linux-next: boot failure after merge of the akpm tree

linux-next: boot failure after merge of the akpm tree

[Stephen Rothwell]

[-- Attachment #1: Type: text/plain, Size: 3393 bytes --]

Hi all,

After merging the akpm tree, today's linux-next boot test (powerpc
pseries_le_defconfig) failed like this:

[    0.025828][    T0] pid_max: default: 32768 minimum: 301
[    0.028445][    T0] Mount-cache hash table entries: 8192 (order: 0, 65536 bytes, linear)
[    0.028636][    T0] Mountpoint-cache hash table entries: 8192 (order: 0, 65536 bytes, linear)
[    0.055344][    T0] BUG: Unable to handle kernel data access on write at 0x2000000b7
[    0.056649][    T0] Faulting instruction address: 0xc00000000057435c
[    0.057458][    T0] Oops: Kernel access of bad area, sig: 11 [#1]
[    0.057734][    T0] LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries
[    0.058250][    T0] Modules linked in:
[    0.058816][    T0] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.16.0-rc3 #2
[    0.059410][    T0] NIP:  c00000000057435c LR: c000000000577f54 CTR: 0000000000000003
[    0.059608][    T0] REGS: c000000002787a80 TRAP: 0380   Not tainted  (5.16.0-rc3)
[    0.059999][    T0] MSR:  8000000002009033 <SF,VEC,EE,ME,IR,DR,RI,LE>  CR: 44000288  XER: 20000000
[    0.060654][    T0] CFAR: c000000000574464 IRQMASK: 0 
[    0.060654][    T0] GPR00: c000000000577f54 c000000002787d20 c000000002788500 c00000000252d5e8 
[    0.060654][    T0] GPR04: c000000000e42500 c00000000252d5e8 0000000000000000 c0000000073a02c9 
[    0.060654][    T0] GPR08: c000000000e42380 c000000000f89d48 c00000000274f700 ffffffffffffffff 
[    0.060654][    T0] GPR12: 0000000000000000 c000000002970000 0000000000000000 0000000000000000 
[    0.060654][    T0] GPR16: 0000000002bf00d0 000000000135bfd8 0000000000000000 0000000000000000 
[    0.060654][    T0] GPR20: c00000000252d6e8 c00000000252d740 0000000000000006 0000000000000000 
[    0.060654][    T0] GPR24: 0000000000000000 0000000000000002 c000000002662eb8 c00000000252d5e8 
[    0.060654][    T0] GPR28: 00000002000000b7 c000000001004108 0000000000000000 c00000000252d5c0 
[    0.062938][    T0] NIP [c00000000057435c] count_subheaders.part.0+0x9c/0x2b0
[    0.064395][    T0] LR [c000000000577f54] __register_sysctl_paths+0x64/0x320
[    0.065201][    T0] Call Trace:
[    0.065521][    T0] [c000000002787dc0] [c000000000577f54] __register_sysctl_paths+0x64/0x320
[    0.065960][    T0] [c000000002787e40] [c0000000005782a4] __register_sysctl_base+0x34/0x50
[    0.066570][    T0] [c000000002787e60] [c0000000020259bc] sysctl_init_bases+0x34/0x74
[    0.066960][    T0] [c000000002787e90] [c000000002041744] proc_sys_init+0x54/0x68
[    0.067306][    T0] [c000000002787eb0] [c0000000020410dc] proc_root_init+0xb8/0xdc
[    0.067763][    T0] [c000000002787ed0] [c0000000020044fc] start_kernel+0x838/0x8cc
[    0.068171][    T0] [c000000002787f90] [c00000000000d39c] start_here_common+0x1c/0x600
[    0.068782][    T0] Instruction dump:
[    0.069295][    T0] f821ff61 eab40018 2c350000 418201c8 e9350000 2c290000 4182020c 3b000000 
[    0.069694][    T0] 3a600000 eb950018 2c3c0000 41820198 <e93c0000> 2c290000 418201dc 3b200000 
[    0.071747][    T0] ---[ end trace df5d37c8b563422a ]---
[    0.071967][    T0] 
[    1.073058][    T0] Kernel panic - not syncing: Attempted to kill the idle task!

Bisected to commit

  f00ff38c1dfa ("sysctl: add and use base directory declarer and registration helper")

I have reverted that complete 6 patch series for today.

-- 
Cheers,
Stephen Rothwell

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: linux-next: boot failure after merge of the akpm tree

[Luis Chamberlain]

On Tue, Nov 30, 2021 at 06:46:22PM +1100, Stephen Rothwell wrote:
> Hi all,
> 
> After merging the akpm tree, today's linux-next boot test (powerpc
> pseries_le_defconfig) failed like this:
> 
> [    0.025828][    T0] pid_max: default: 32768 minimum: 301
> [    0.028445][    T0] Mount-cache hash table entries: 8192 (order: 0, 65536 bytes, linear)
> [    0.028636][    T0] Mountpoint-cache hash table entries: 8192 (order: 0, 65536 bytes, linear)
> [    0.055344][    T0] BUG: Unable to handle kernel data access on write at 0x2000000b7
> [    0.056649][    T0] Faulting instruction address: 0xc00000000057435c
> [    0.057458][    T0] Oops: Kernel access of bad area, sig: 11 [#1]
> [    0.057734][    T0] LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries
> [    0.058250][    T0] Modules linked in:
> [    0.058816][    T0] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.16.0-rc3 #2
> [    0.059410][    T0] NIP:  c00000000057435c LR: c000000000577f54 CTR: 0000000000000003
> [    0.059608][    T0] REGS: c000000002787a80 TRAP: 0380   Not tainted  (5.16.0-rc3)
> [    0.059999][    T0] MSR:  8000000002009033 <SF,VEC,EE,ME,IR,DR,RI,LE>  CR: 44000288  XER: 20000000
> [    0.060654][    T0] CFAR: c000000000574464 IRQMASK: 0 
> [    0.060654][    T0] GPR00: c000000000577f54 c000000002787d20 c000000002788500 c00000000252d5e8 
> [    0.060654][    T0] GPR04: c000000000e42500 c00000000252d5e8 0000000000000000 c0000000073a02c9 
> [    0.060654][    T0] GPR08: c000000000e42380 c000000000f89d48 c00000000274f700 ffffffffffffffff 
> [    0.060654][    T0] GPR12: 0000000000000000 c000000002970000 0000000000000000 0000000000000000 
> [    0.060654][    T0] GPR16: 0000000002bf00d0 000000000135bfd8 0000000000000000 0000000000000000 
> [    0.060654][    T0] GPR20: c00000000252d6e8 c00000000252d740 0000000000000006 0000000000000000 
> [    0.060654][    T0] GPR24: 0000000000000000 0000000000000002 c000000002662eb8 c00000000252d5e8 
> [    0.060654][    T0] GPR28: 00000002000000b7 c000000001004108 0000000000000000 c00000000252d5c0 
> [    0.062938][    T0] NIP [c00000000057435c] count_subheaders.part.0+0x9c/0x2b0
> [    0.064395][    T0] LR [c000000000577f54] __register_sysctl_paths+0x64/0x320
> [    0.065201][    T0] Call Trace:
> [    0.065521][    T0] [c000000002787dc0] [c000000000577f54] __register_sysctl_paths+0x64/0x320
> [    0.065960][    T0] [c000000002787e40] [c0000000005782a4] __register_sysctl_base+0x34/0x50
> [    0.066570][    T0] [c000000002787e60] [c0000000020259bc] sysctl_init_bases+0x34/0x74
> [    0.066960][    T0] [c000000002787e90] [c000000002041744] proc_sys_init+0x54/0x68
> [    0.067306][    T0] [c000000002787eb0] [c0000000020410dc] proc_root_init+0xb8/0xdc
> [    0.067763][    T0] [c000000002787ed0] [c0000000020044fc] start_kernel+0x838/0x8cc
> [    0.068171][    T0] [c000000002787f90] [c00000000000d39c] start_here_common+0x1c/0x600
> [    0.068782][    T0] Instruction dump:
> [    0.069295][    T0] f821ff61 eab40018 2c350000 418201c8 e9350000 2c290000 4182020c 3b000000 
> [    0.069694][    T0] 3a600000 eb950018 2c3c0000 41820198 <e93c0000> 2c290000 418201dc 3b200000 
> [    0.071747][    T0] ---[ end trace df5d37c8b563422a ]---
> [    0.071967][    T0] 
> [    1.073058][    T0] Kernel panic - not syncing: Attempted to kill the idle task!
> 
> Bisected to commit
> 
>   f00ff38c1dfa ("sysctl: add and use base directory declarer and registration helper")
> 
> I have reverted that complete 6 patch series for today.

I'll be trying to reproduce and fix, thanks for the report!

  Luis

Re: linux-next: boot failure after merge of the akpm tree

[Luis Chamberlain]

On Tue, Nov 30, 2021 at 08:21:52AM -0800, Luis Chamberlain wrote:
> On Tue, Nov 30, 2021 at 06:46:22PM +1100, Stephen Rothwell wrote:
> > Bisected to commit
> > 
> >   f00ff38c1dfa ("sysctl: add and use base directory declarer and registration helper")
> > 
> > I have reverted that complete 6 patch series for today.
> 
> I'll be trying to reproduce and fix, thanks for the report!

OK please try this patch, I can boot with this just fine on ppc64le:

diff --git a/include/linux/sysctl.h b/include/linux/sysctl.h
index 70acd2a100fd..180adf7da785 100644
--- a/include/linux/sysctl.h
+++ b/include/linux/sysctl.h
@@ -201,6 +201,7 @@ static struct ctl_table _name##_base_table[] = {			\
 		.mode		= 0555,					\
 		.child		= _table,				\
 	},								\
+	{ },								\
 }
 
 extern int __register_sysctl_base(struct ctl_table *base_table);

Re: linux-next: boot failure after merge of the akpm tree

[Luis Chamberlain]

On Wed, Dec 01, 2021 at 11:13:39AM -0800, Luis Chamberlain wrote:
> On Tue, Nov 30, 2021 at 08:21:52AM -0800, Luis Chamberlain wrote:
> > On Tue, Nov 30, 2021 at 06:46:22PM +1100, Stephen Rothwell wrote:
> > > Bisected to commit
> > > 
> > >   f00ff38c1dfa ("sysctl: add and use base directory declarer and registration helper")
> > > 
> > > I have reverted that complete 6 patch series for today.
> > 
> > I'll be trying to reproduce and fix, thanks for the report!
> 
> OK please try this patch, I can boot with this just fine on ppc64le:

And to be clear I reproduced the issue and this patch fixes it for me
on ppc64le.

  Luis

Re: linux-next: boot failure after merge of the akpm tree

[Stephen Rothwell]

[-- Attachment #1: Type: text/plain, Size: 1130 bytes --]

Hi Luis,

On Wed, 1 Dec 2021 11:13:39 -0800 Luis Chamberlain <mcgrof@kernel.org> wrote:
>
> On Tue, Nov 30, 2021 at 08:21:52AM -0800, Luis Chamberlain wrote:
> > On Tue, Nov 30, 2021 at 06:46:22PM +1100, Stephen Rothwell wrote:  
> > > Bisected to commit
> > > 
> > >   f00ff38c1dfa ("sysctl: add and use base directory declarer and registration helper")
> > > 
> > > I have reverted that complete 6 patch series for today.  
> > 
> > I'll be trying to reproduce and fix, thanks for the report!  
> 
> OK please try this patch, I can boot with this just fine on ppc64le:
> 
> diff --git a/include/linux/sysctl.h b/include/linux/sysctl.h
> index 70acd2a100fd..180adf7da785 100644
> --- a/include/linux/sysctl.h
> +++ b/include/linux/sysctl.h
> @@ -201,6 +201,7 @@ static struct ctl_table _name##_base_table[] = {			\
>  		.mode		= 0555,					\
>  		.child		= _table,				\
>  	},								\
> +	{ },								\
>  }
>  
>  extern int __register_sysctl_base(struct ctl_table *base_table);

That fixes it for me.  I will use that in linux-next today instead of
the reverts.

-- 
Cheers,
Stephen Rothwell

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: linux-next: boot failure after merge of the akpm tree

[Stephen Rothwell]

[-- Attachment #1: Type: text/plain, Size: 752 bytes --]

Hi all,

On Fri, 31 May 2019 12:27:58 +1000 Nicholas Piggin <npiggin@gmail.com> wrote:
>
> > I have reverted
> > 
> >   c353e2997976 ("mm/vmalloc: hugepage vmalloc mappings")
> >   a826492f28d9 ("mm: move ioremap page table mapping function to mm/")
> > 
> > (and my fix up) for today and things seem to work (if only because the
> > BUG() has been removed :-)).  
> 
> Good to know, maybe I didn't test powerpc without later enabling 
> patches...
> 
> The series also has a compile bug on ARM I have to work out, so
> yeah drop those for now, I'll post a v2. The large system map patches
> that I posted in that series can stay I think.

OK, I have removed them from the akpm-current tree today.

-- 
Cheers,
Stephen Rothwell

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: linux-next: boot failure after merge of the akpm tree

[Nicholas Piggin]

Stephen Rothwell's on May 30, 2019 4:17 pm:
> Hi all,
> 
> My qemu boot (PowerPC le guest on PowerPC le host, with and without kvm,
> using a kernel built with powerpc_pseries_le_defconfig) oopses during boot
> like this:
> 
> -----------------------------------------------------------------------------
> numa: Node 0 CPUs: 0
> Using standard scheduler topology
> devtmpfs: initialized
> clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
> futex hash table entries: 256 (order: -1, 32768 bytes)
> ------------[ cut here ]------------
> kernel BUG at mm/vmalloc.c:472!
> Oops: Exception in kernel mode, sig: 5 [#1]
> LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries
> Modules linked in:
> CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.2.0-rc2 #2
> NIP:  c000000000369b18 LR: c000000000369c74 CTR: c000000000176e30
> REGS: c00000007e6636e0 TRAP: 0700   Not tainted  (5.2.0-rc2)
> MSR:  8000000002029033 <SF,VEC,EE,ME,IR,DR,RI,LE>  CR: 24024882  XER: 20000000
> CFAR: c000000000369c78 IRQMASK: 0 
> GPR00: c000000000369c74 c00000007e663970 c00000000119c100 0000000000000001 
> GPR04: 000000007ec20000 00000001f4fe19cb 00000001f5398c84 c000000001380000 
> GPR08: 0000000000000000 0000000000000001 0000000000000001 00000000000002b2 
> GPR12: 0000000000004000 c000000001380000 c000000000010fc0 0000000000000001 
> GPR16: 0000000000010000 800000000000018e c000000000df9988 0000000000000000 
> GPR20: 0000000000010000 0000000000002dc2 0000000000000dc0 0000000000000022 
> GPR24: c00000007e2204c0 0000000000000dc2 0000000000010000 c00a000000000000 
> GPR28: c008000000000000 0000000000010000 ffffffffffffffff 0000000000000dc0 
> NIP [c000000000369b18] __vmalloc_node_range+0x1f8/0x410
> LR [c000000000369c74] __vmalloc_node_range+0x354/0x410
> Call Trace:
> [c00000007e663970] [c000000000369c74] __vmalloc_node_range+0x354/0x410 (unreliable)
> [c00000007e663a70] [c000000000369d80] __vmalloc+0x50/0x60
> [c00000007e663ae0] [c000000000299a98] bpf_prog_alloc_no_stats+0x58/0x120
> [c00000007e663b20] [c000000000299b90] bpf_prog_alloc+0x30/0xe0
> [c00000007e663b60] [c000000000a49dd8] bpf_prog_create+0x68/0x100
> [c00000007e663ba0] [c000000000f4f2a8] ptp_classifier_init+0x4c/0x80
> [c00000007e663be0] [c000000000f4b9e8] sock_init+0xe0/0x100
> [c00000007e663c10] [c000000000010b60] do_one_initcall+0x60/0x2c0
> [c00000007e663ce0] [c000000000ee45b0] kernel_init_freeable+0x37c/0x478
> [c00000007e663db0] [c000000000010fe4] kernel_init+0x2c/0x148
> [c00000007e663e20] [c00000000000c0cc] ret_from_kernel_thread+0x5c/0x70
> Instruction dump:
> 60000000 2c230000 418200dc e9580020 79e91f24 7c6a492a 40920170 8138002c 
> 394f0001 794f0020 7f895040 419dffbc <0fe00000> 60000000 3f400001 4bfffedc 
> ---[ end trace 49ed8f97d467e164 ]---
> 
> Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000005
> -----------------------------------------------------------------------------
> 
> The BUG is:
> 
> 	BUG_ON(page_shift != PAGE_SIZE);
> 
> in the !CONFIG_HAVE_ARCH_HUGE_VMAP version of vmap_hpages_range().
> 
> I am guessing this is something to do with the vmalloc changes in Andrew's
> patches (or it could be the fixup I did to Nick's patch).
> 
> I have reverted
> 
>   c353e2997976 ("mm/vmalloc: hugepage vmalloc mappings")
>   a826492f28d9 ("mm: move ioremap page table mapping function to mm/")
> 
> (and my fix up) for today and things seem to work (if only because the
> BUG() has been removed :-)).

Good to know, maybe I didn't test powerpc without later enabling 
patches...

The series also has a compile bug on ARM I have to work out, so
yeah drop those for now, I'll post a v2. The large system map patches
that I posted in that series can stay I think.

Thanks,
Nick

linux-next: boot failure after merge of the akpm tree

[Stephen Rothwell]

[-- Attachment #1: Type: text/plain, Size: 3382 bytes --]

Hi all,

My qemu boot (PowerPC le guest on PowerPC le host, with and without kvm,
using a kernel built with powerpc_pseries_le_defconfig) oopses during boot
like this:

-----------------------------------------------------------------------------
numa: Node 0 CPUs: 0
Using standard scheduler topology
devtmpfs: initialized
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
futex hash table entries: 256 (order: -1, 32768 bytes)
------------[ cut here ]------------
kernel BUG at mm/vmalloc.c:472!
Oops: Exception in kernel mode, sig: 5 [#1]
LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries
Modules linked in:
CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.2.0-rc2 #2
NIP:  c000000000369b18 LR: c000000000369c74 CTR: c000000000176e30
REGS: c00000007e6636e0 TRAP: 0700   Not tainted  (5.2.0-rc2)
MSR:  8000000002029033 <SF,VEC,EE,ME,IR,DR,RI,LE>  CR: 24024882  XER: 20000000
CFAR: c000000000369c78 IRQMASK: 0 
GPR00: c000000000369c74 c00000007e663970 c00000000119c100 0000000000000001 
GPR04: 000000007ec20000 00000001f4fe19cb 00000001f5398c84 c000000001380000 
GPR08: 0000000000000000 0000000000000001 0000000000000001 00000000000002b2 
GPR12: 0000000000004000 c000000001380000 c000000000010fc0 0000000000000001 
GPR16: 0000000000010000 800000000000018e c000000000df9988 0000000000000000 
GPR20: 0000000000010000 0000000000002dc2 0000000000000dc0 0000000000000022 
GPR24: c00000007e2204c0 0000000000000dc2 0000000000010000 c00a000000000000 
GPR28: c008000000000000 0000000000010000 ffffffffffffffff 0000000000000dc0 
NIP [c000000000369b18] __vmalloc_node_range+0x1f8/0x410
LR [c000000000369c74] __vmalloc_node_range+0x354/0x410
Call Trace:
[c00000007e663970] [c000000000369c74] __vmalloc_node_range+0x354/0x410 (unreliable)
[c00000007e663a70] [c000000000369d80] __vmalloc+0x50/0x60
[c00000007e663ae0] [c000000000299a98] bpf_prog_alloc_no_stats+0x58/0x120
[c00000007e663b20] [c000000000299b90] bpf_prog_alloc+0x30/0xe0
[c00000007e663b60] [c000000000a49dd8] bpf_prog_create+0x68/0x100
[c00000007e663ba0] [c000000000f4f2a8] ptp_classifier_init+0x4c/0x80
[c00000007e663be0] [c000000000f4b9e8] sock_init+0xe0/0x100
[c00000007e663c10] [c000000000010b60] do_one_initcall+0x60/0x2c0
[c00000007e663ce0] [c000000000ee45b0] kernel_init_freeable+0x37c/0x478
[c00000007e663db0] [c000000000010fe4] kernel_init+0x2c/0x148
[c00000007e663e20] [c00000000000c0cc] ret_from_kernel_thread+0x5c/0x70
Instruction dump:
60000000 2c230000 418200dc e9580020 79e91f24 7c6a492a 40920170 8138002c 
394f0001 794f0020 7f895040 419dffbc <0fe00000> 60000000 3f400001 4bfffedc 
---[ end trace 49ed8f97d467e164 ]---

Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000005
-----------------------------------------------------------------------------

The BUG is:

	BUG_ON(page_shift != PAGE_SIZE);

in the !CONFIG_HAVE_ARCH_HUGE_VMAP version of vmap_hpages_range().

I am guessing this is something to do with the vmalloc changes in Andrew's
patches (or it could be the fixup I did to Nick's patch).

I have reverted

  c353e2997976 ("mm/vmalloc: hugepage vmalloc mappings")
  a826492f28d9 ("mm: move ioremap page table mapping function to mm/")

(and my fix up) for today and things seem to work (if only because the
BUG() has been removed :-)).

-- 
Cheers,
Stephen Rothwell

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]