rp2470 boot crash in post-5.8 git

rp2470 boot crash in post-5.8 git

[Meelis Roos]

While 5.8.0 worked fine on my HP RP2470 (Gentoo with 10.2.0 kgcc), yesterdays dddcbc139e96 and todays a1d21081a60d crash on boot. I have started bisecting it.

Last output lines from 5.8.0-13161-gdddcbc139e96:

[    1.150985] 1. Crescendo 750 W2 at 0xfffffffffffa0000 [160] { 0, 0x0, 0x5e3, 0x00004 }
[    1.243570] 2. Memory at 0xfffffffffed08000 [8] { 1, 0x0, 0x09b, 0x00009 }
[    1.243792] 3. Astro BC Runway Port at 0xfffffffffed00000 [0] { 12, 0x0, 0x582, 0x0000b }
[    1.382893] 4. Elroy PCI Bridge at 0xfffffffffed30000 [0/0] { 13, 0x0, 0x782, 0x0000a }
[    1.522243] 5. Elroy PCI Bridge at 0xfffffffffed34000 [0/2] { 13, 0x0, 0x782, 0x0000a }
[    1.660767] 6. Elroy PCI Bridge at 0xfffffffffed38000 [0/4] { 13, 0x0, 0x782, 0x0000a }
[    1.799247] 7. Elroy PCI Bridge at 0xfffffffffed3c000 [0/6] { 13, 0x0, 0x782, 0x0000a }
[    1.799500]
********** VIRTUAL FRONT PANEL **********
System Boot detected
*****************************************
LEDs:  RUN      ATTENTION     FAULT     REMOTE     POWER
        ON       FLASH         OFF       ON         ON
LED State: There was a system interruption that did not take the system down.
Check Chassis and Console Logs for error messages.

processor                 system initialization      1C00

*****************************************

************ EARLY BOOT VFP *************
End of early boot detected
*****************************************
h support not available.
[    2.639001] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages

********** VIRTUAL FRONT PANEL **********
System Boot detected
*****************************************
LEDs:  RUN      ATTENTION     FAULT     REMOTE     POWER
        ON       FLASH         FLASH     ON         ON
LED State: System Running.  Unexpected Reboot.  Non-critical Error Detected.
Check Chassis and Console Logs for error messages.

processor                 system panic               1B00

---------------------------------------------------------------------------------------------------
Last output lines from 5.8.0-13249-ga1d21081a60d:

[    1.151696] 1. Crescendo 750 W2 at 0xfffffffffffa0000 [160] { 0, 0x0, 0x5e3, 0x00004 }
[    1.242819] 2. Memory at 0xfffffffffed08000 [8] { 1, 0x0, 0x09b, 0x00009 }
[    1.243041] 3. Astro BC Runway Port at 0xfffffffffed00000 [0] { 12, 0x0, 0x582, 0x0000b }
[    1.383414] 4. Elroy PCI Bridge at 0xfffffffffed30000 [0/0] { 13, 0x0, 0x782, 0x0000a }
[    1.523072] 5. Elroy PCI Bridge at 0xfffffffffed34000 [0/2] { 13, 0x0, 0x782, 0x0000a }
[    1.662053] 6. Elroy PCI Bridge at 0xfffffffffed38000 [0/4] { 13, 0x0, 0x782, 0x0000a }
[    1.800405] 7. Elroy PCI Bridge at 0xfffffffffed3c000 [0/6] { 13, 0x0, 0x782, 0x0000a }
[    1.800658]
********** VIRTUAL FRONT PANEL **********
System Boot detected
*****************************************
LEDs:  RUN      ATTENTION     FAULT     REMOTE     POWER
        ON       FLASH         OFF       ON         ON
LED State: There was a system interruption that did not take the system down.
Check Chassis and Console Logs for error messages.

processor                 system initialization      1C00

*****************************************

************ EARLY BOOT VFP *************
End of early boot detected
*****************************************

********** VIRTUAL FRONT PANEL **********
System Boot detected
*****************************************
LEDs:  RUN      ATTENTION     FAULT     REMOTE     POWER
        ON       FLASH         FLASH     ON         ON
LED State: System Running.  Unexpected Reboot.  Non-critical Error Detected.
Check Chassis and Console Logs for error messages.

processor                 system panic               1B00

*****************************************

************ EARLY BOOT VFP *************
End of early boot detected
*****************************************

********** VIRTUAL FRONT PANEL **********
System Boot detected
*****************************************
LEDs:  RUN      ATTENTION     FAULT     REMOTE     POWER
        ON       FLASH         FLASH     ON         ON
LED State: System Running.  Unexpected Reboot.  Non-critical Error Detected.
Check Chassis and Console Logs for error messages.

processor                 system panic               1B00


-- 
Meelis Roos <mroos@linux.ee>

(bisected) parisc boot crash in post-5.8 git - pgalloc changes

[Meelis Roos]

> While 5.8.0 worked fine on my HP RP2470 (Gentoo with 10.2.0 kgcc), yesterdays dddcbc139e96 and todays a1d21081a60d crash on boot. I have started bisecting it.

And finished now. I included all CC-s and SOB-s from the commit to this mail - sorry if too broad.

1355c31eeb7ea61a7f2f2937d17cd4e343a6b5af is the first bad commit
commit 1355c31eeb7ea61a7f2f2937d17cd4e343a6b5af
Author: Mike Rapoport <rppt@linux.ibm.com>
Date:   Thu Aug 6 23:22:39 2020 -0700

     asm-generic: pgalloc: provide generic pmd_alloc_one() and pmd_free_one()
     
     For most architectures that support >2 levels of page tables,
     pmd_alloc_one() is a wrapper for __get_free_pages(), sometimes with
     __GFP_ZERO and sometimes followed by memset(0) instead.
     
     More elaborate versions on arm64 and x86 account memory for the user page
     tables and call to pgtable_pmd_page_ctor() as the part of PMD page
     initialization.
     
     Move the arm64 version to include/asm-generic/pgalloc.h and use the
     generic version on several architectures.
     
     The pgtable_pmd_page_ctor() is a NOP when ARCH_ENABLE_SPLIT_PMD_PTLOCK is
     not enabled, so there is no functional change for most architectures
     except of the addition of __GFP_ACCOUNT for allocation of user page
     tables.
     
     The pmd_free() is a wrapper for free_page() in all the cases, so no
     functional change here.
     
     Signed-off-by: Mike Rapoport <rppt@linux.ibm.com>
     Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
     Reviewed-by: Pekka Enberg <penberg@kernel.org>
     Cc: Matthew Wilcox <willy@infradead.org>
     Cc: Abdul Haleem <abdhalee@linux.vnet.ibm.com>
     Cc: Andy Lutomirski <luto@kernel.org>
     Cc: Arnd Bergmann <arnd@arndb.de>
     Cc: Christophe Leroy <christophe.leroy@csgroup.eu>
     Cc: Joerg Roedel <joro@8bytes.org>
     Cc: Joerg Roedel <jroedel@suse.de>
     Cc: Max Filippov <jcmvbkbc@gmail.com>
     Cc: Peter Zijlstra (Intel) <peterz@infradead.org>
     Cc: Satheesh Rajendran <sathnaga@linux.vnet.ibm.com>
     Cc: Stafford Horne <shorne@gmail.com>
     Cc: Stephen Rothwell <sfr@canb.auug.org.au>
     Cc: Steven Rostedt <rostedt@goodmis.org>
     Cc: Geert Uytterhoeven <geert@linux-m68k.org>
     Link: http://lkml.kernel.org/r/20200627143453.31835-5-rppt@kernel.org
     Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

  arch/alpha/include/asm/pgalloc.h     | 15 +------------
  arch/arm/include/asm/pgalloc.h       | 11 ---------
  arch/arm64/include/asm/pgalloc.h     | 27 +---------------------
  arch/ia64/include/asm/pgalloc.h      | 10 ---------
  arch/mips/include/asm/pgalloc.h      |  8 ++-----
  arch/parisc/include/asm/pgalloc.h    | 11 ++-------
  arch/riscv/include/asm/pgalloc.h     | 13 +----------
  arch/sh/include/asm/pgalloc.h        |  3 +++
  arch/um/include/asm/pgalloc.h        |  8 +------
  arch/um/include/asm/pgtable-3level.h |  3 ---
  arch/um/kernel/mem.c                 | 12 ----------
  arch/x86/include/asm/pgalloc.h       | 26 +---------------------
  include/asm-generic/pgalloc.h        | 43 ++++++++++++++++++++++++++++++++++++
  13 files changed, 55 insertions(+), 135 deletions(-)

> Last output lines from 5.8.0-13161-gdddcbc139e96:
> 
> [    1.150985] 1. Crescendo 750 W2 at 0xfffffffffffa0000 [160] { 0, 0x0, 0x5e3, 0x00004 }
> [    1.243570] 2. Memory at 0xfffffffffed08000 [8] { 1, 0x0, 0x09b, 0x00009 }
> [    1.243792] 3. Astro BC Runway Port at 0xfffffffffed00000 [0] { 12, 0x0, 0x582, 0x0000b }
> [    1.382893] 4. Elroy PCI Bridge at 0xfffffffffed30000 [0/0] { 13, 0x0, 0x782, 0x0000a }
> [    1.522243] 5. Elroy PCI Bridge at 0xfffffffffed34000 [0/2] { 13, 0x0, 0x782, 0x0000a }
> [    1.660767] 6. Elroy PCI Bridge at 0xfffffffffed38000 [0/4] { 13, 0x0, 0x782, 0x0000a }
> [    1.799247] 7. Elroy PCI Bridge at 0xfffffffffed3c000 [0/6] { 13, 0x0, 0x782, 0x0000a }
> [    1.799500]
> ********** VIRTUAL FRONT PANEL **********
> System Boot detected
> *****************************************
> LEDs:  RUN      ATTENTION     FAULT     REMOTE     POWER
>         ON       FLASH         OFF       ON         ON
> LED State: There was a system interruption that did not take the system down.
> Check Chassis and Console Logs for error messages.
> 
> processor                 system initialization      1C00
> 
> *****************************************
> 
> ************ EARLY BOOT VFP *************
> End of early boot detected
> *****************************************
> h support not available.
> [    2.639001] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
> 
> ********** VIRTUAL FRONT PANEL **********
> System Boot detected
> *****************************************
> LEDs:  RUN      ATTENTION     FAULT     REMOTE     POWER
>         ON       FLASH         FLASH     ON         ON
> LED State: System Running.  Unexpected Reboot.  Non-critical Error Detected.
> Check Chassis and Console Logs for error messages.
> 
> processor                 system panic               1B00

-- 
Meelis Roos <mroos@linux.ee>

Re: (bisected) parisc boot crash in post-5.8 git - pgalloc changes

[Matthew Wilcox]

On Sat, Aug 15, 2020 at 09:35:05PM +0300, Meelis Roos wrote:
> > While 5.8.0 worked fine on my HP RP2470 (Gentoo with 10.2.0 kgcc), yesterdays dddcbc139e96 and todays a1d21081a60d crash on boot. I have started bisecting it.
> 
> And finished now. I included all CC-s and SOB-s from the commit to this mail - sorry if too broad.
> 
> 1355c31eeb7ea61a7f2f2937d17cd4e343a6b5af is the first bad commit

yes, I see the problem.

arch/parisc/include/asm/pgtable.h:#define PMD_ORDER     1 /* Number of pages per pmd */

-static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long address)
-{
-       pmd_t *pmd = (pmd_t *)__get_free_pages(GFP_KERNEL, PMD_ORDER);

+       page = alloc_pages(gfp, 0);

So I think the quick fix is to restore the parisc pmd_alloc_one and define
__HAVE_ARCH_PMD_ALLOC_ONE.  I wasn't following this series too closely,
so I shan't propose a patch, but leave it up to Mike.  I'd probably miss
some subtlety.

Re: (bisected) parisc boot crash in post-5.8 git - pgalloc changes

[Mike Rapoport]

Hi,

On Sat, Aug 15, 2020 at 07:42:00PM +0100, Matthew Wilcox wrote:
> On Sat, Aug 15, 2020 at 09:35:05PM +0300, Meelis Roos wrote:
> > > While 5.8.0 worked fine on my HP RP2470 (Gentoo with 10.2.0 kgcc), yesterdays dddcbc139e96 and todays a1d21081a60d crash on boot. I have started bisecting it.
> > 
> > And finished now. I included all CC-s and SOB-s from the commit to this mail - sorry if too broad.
> > 
> > 1355c31eeb7ea61a7f2f2937d17cd4e343a6b5af is the first bad commit
> 
> yes, I see the problem.
> 
> arch/parisc/include/asm/pgtable.h:#define PMD_ORDER     1 /* Number of pages per pmd */
> 
> -static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long address)
> -{
> -       pmd_t *pmd = (pmd_t *)__get_free_pages(GFP_KERNEL, PMD_ORDER);
> 
> +       page = alloc_pages(gfp, 0);
> 
> So I think the quick fix is to restore the parisc pmd_alloc_one and define
> __HAVE_ARCH_PMD_ALLOC_ONE.  I wasn't following this series too closely,
> so I shan't propose a patch, but leave it up to Mike.  I'd probably miss
> some subtlety.

You are right, I've missed that parisc has PMD_ORDER != 0 and the
simplest fix would be to revert changes to parisc pmd_alloc_one().

Meelis, can you please try the below patch:

diff --git a/arch/parisc/include/asm/pgalloc.h b/arch/parisc/include/asm/pgalloc.h
index cc7ecc2ef55d..a6482b2ce0ea 100644
--- a/arch/parisc/include/asm/pgalloc.h
+++ b/arch/parisc/include/asm/pgalloc.h
@@ -10,6 +10,7 @@
 
 #include <asm/cache.h>
 
+#define __HAVE_ARCH_PMD_ALLOC_ONE
 #define __HAVE_ARCH_PMD_FREE
 #define __HAVE_ARCH_PGD_FREE
 #include <asm-generic/pgalloc.h>
@@ -67,6 +68,11 @@ static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd)
 			(__u32)(__pa((unsigned long)pmd) >> PxD_VALUE_SHIFT)));
 }
 
+static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long address)
+{
+	return (pmd_t *)__get_free_pages(GFP_PGTABLE_KERNEL, PMD_ORDER);
+}
+
 static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd)
 {
 	if (pmd_flag(*pmd) & PxD_FLAG_ATTACHED) {

-- 
Sincerely yours,
Mike.

Re: (bisected) parisc boot crash in post-5.8 git - pgalloc changes

[Meelis Roos]

16.08.20 10:34 Mike Rapoport wrote:
> Hi,
> 
[...]
> You are right, I've missed that parisc has PMD_ORDER != 0 and the
> simplest fix would be to revert changes to parisc pmd_alloc_one().
> 
> Meelis, can you please try the below patch:

It works on top of current git 5.8.0-13249-ga1d21081a60d - thank you!

Tested-by: Meelis Roos <mroos@linux.ee>