[U-Boot] 83xx fails to boot with moderately sized kernels

[U-Boot] 83xx fails to boot with moderately sized kernels

[Ira W. Snyder]

Hello all,

I am using vanilla U-Boot on an MPC8349EMDS board with 256MB of RAM. I have
been tracking down a kernel hang very early during boot, and have traced it
back to a U-Boot bug. The bug has to do with the way CONFIG_SYS_BOOTMAPSZ
is set.

Please read through this thread for more lots more information:
http://lists.ozlabs.org/pipermail/linuxppc-dev/2010-September/085524.html

On most (all?) 83xx boards, CONFIG_SYS_BOOTMAPSZ is set to 8MB. The comment
says:

/*
 * For booting Linux, the board info and command line data
 * have to be in the first 8 MB of memory, since this is
 * the maximum mapped by the Linux kernel during initialization.
 */
#define CONFIG_SYS_BOOTMAPSZ	(8 << 20)	/* Initial Memory map for Linux*/

I am booting with a FIT image containing a kernel, FDT, and ramdisk.
U-Boot's bootm command makes sure that the FDT is relocated such that it is
located at an address lower than CONFIG_SYS_BOOTMAPSZ. The ramdisk is
placed at the end of memory. The relevant U-Boot messages:

   Booting using the fdt blob at 0x226a278
   Uncompressing Kernel Image ... OK
   Loading Ramdisk to 0fe9f000, end 0ff75699 ... OK
   Loading Device Tree to 007f8000, end 007ff78f ... OK

You'll notice the ramdisk was loaded@the end of RAM (near 256MB), while
the FDT was loaded near the end of CONFIG_SYS_BOOTMAPSZ (near 8MB).

Newer Linux kernels are getting pretty large. In fact, even a fairly
minimal kernel is large enough such that the uncompressed kernel code + BSS
is >8MB. When this happens, Linux overwrites the FDT and the boot hangs
silently.

U-Boot cannot detect this, since it does not load ELF images. It does not
know the BSS size.

Below is the size of my two kernels, toggling only the CONFIG_PROVE_LOCKING
feature on or off. As stated above, the rest of the kernel is relatively
minimal.

With CONFIG_PROVE_LOCKING:
c0369000 A __bss_start
c08a0c48 A __bss_stop

0xc08a0c48 - 0xc0369000 == 5471304 bytes BSS

Kernel size (uncompressed):
3573068 vmlinux.bin

Total (code + bss): 3573068 + 5471304 == 9044372 (8.625MB)

Without CONFIG_PROVE_LOCKING:
c0363000 A __bss_start
c0786c08 A __bss_stop

0xc0786c08 - 0xc0363000 == 4340744 bytes BSS

Kernel size (uncompressed):
3548492 vmlinux.bin

Total (code + bss): 3548492 + 4340744 == 7889236 (7.523MB)

So, just adding CONFIG_PROVE_LOCKING=y made a 1MB difference, most of it in
the bss section.

Now, as the U-Boot output shows (with CONFIG_SYS_BOOTMAPSZ == 8MB):

   Booting using the fdt blob at 0x226a278
   Uncompressing Kernel Image ... OK
   Loading Ramdisk to 0fe9f000, end 0ff75699 ... OK
   Loading Device Tree to 007f8000, end 007ff78f ... OK

Now, since the Linux kernel is loaded to address 0x0, this means that after
it is running and has initialized it's BSS, the end of the kernel will be
at address: 0x786154. The working kernel is just barely squeaking by with
about 450KB of space between the kernel and FDT.

Note that on PowerPC, Linux initializes it's BSS very, very early. The BSS
is initialized before the FDT is parsed.

Nothing that Timur or I can find indicate that CONFIG_SYS_BOOTMAPSZ is
actually limited to 8MB, as the comment states. Timur and Kumar chatted,
and realized that they had seen this bug before, on 85xx. They raised the
value of CONFIG_SYS_BOOTMAPSZ to 16MB without issue.

Reading through the Linux code, I cannot find any reason why this should be
limited to 8MB, or even 16MB. Should it be limited to CONFIG_MAX_MEM_MAPPED?

I'm worried that others either have hit this bug, or will hit it soon. By
building lots of drivers into the kernel, I'm sure that even the 16MB
boundary could be overrun.

Does anyone know the true maximum value for CONFIG_SYS_BOOTMAPSZ on Linux
(if one even exists)?

Thanks,
Ira

[U-Boot] 83xx fails to boot with moderately sized kernels

[Wolfgang Denk]

Dear "Ira W. Snyder",

In message <20100909225241.GI3496@ovro.caltech.edu> you wrote:
> 
> On most (all?) 83xx boards, CONFIG_SYS_BOOTMAPSZ is set to 8MB. The comment
> says:
...
> Does anyone know the true maximum value for CONFIG_SYS_BOOTMAPSZ on Linux
> (if one even exists)?

The CONFIG_SYS_BOOTMAPSZ thing is as old as U-Boot and PPCBoot
exists, i. e. well over a decade. IIRC there was such a limitation on
the then current 2.2.13 Linux kernels, at least on MPC8xx and PPC40x
systems, which is where all started from.


I am pretty sure that as long as nobody ran into any problems, nobody
looked into that code, so it was copied from architecture to
architecture without much thinking, if any.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
"I can call spirits from the vasty deep."
"Why so can I, or so can any man; but will they come when you do call
for them?"          - Shakespeare, 1 King Henry IV, Act III, Scene I.

[U-Boot] 83xx fails to boot with moderately sized kernels

[Stefan Roese]

Hi Ira & Wolfgang,

On Friday 10 September 2010 13:18:55 Wolfgang Denk wrote:
> > Does anyone know the true maximum value for CONFIG_SYS_BOOTMAPSZ on Linux
> > (if one even exists)?
> 
> The CONFIG_SYS_BOOTMAPSZ thing is as old as U-Boot and PPCBoot
> exists, i. e. well over a decade. IIRC there was such a limitation on
> the then current 2.2.13 Linux kernels, at least on MPC8xx and PPC40x
> systems, which is where all started from.
> 
> 
> I am pretty sure that as long as nobody ran into any problems, nobody
> looked into that code, so it was copied from architecture to
> architecture without much thinking, if any.

I looked at it a bit over a year ago and commited this change for the AMCC/APM 
eval boards:

commit 6942efc2be1b90054fa4afa5cda7023469fe08b9
Author: Stefan Roese <sr@denx.de>
Date:   Tue Jul 28 10:50:32 2009 +0200

    ppc4xx: amcc: Set CONFIG_SYS_BOOTMAPSZ to 16MB for big kernels
    
    This patch changes CONFIG_SYS_BOOTMAPSZ from 8MB to 16MB which is the
    initial TLB on 40x PPC's in the Linux kernel. With this change even bigger
    Linux kernels (> 8MB) can be booted.
    
    This patch also sets CONFIG_SYS_BOOTM_LEN to 16MB (default 8MB) to enable
    decompression of bigger images.
    
    Signed-off-by: Stefan Roese <sr@denx.de>


So we have this 16MiB initial TLB restriction at least for PPC405 (not 
PPC440). I'm pretty sure that 83xx has no such tight restrictions.

Cheers,
Stefan

--
DENX Software Engineering GmbH,      MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich,  Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-0 Fax: (+49)-8142-66989-80 Email: office at denx.de

[U-Boot] 83xx fails to boot with moderately sized kernels

[Ira W. Snyder]

On Fri, Sep 10, 2010 at 01:29:48PM +0200, Stefan Roese wrote:
> Hi Ira & Wolfgang,
> 
> On Friday 10 September 2010 13:18:55 Wolfgang Denk wrote:
> > > Does anyone know the true maximum value for CONFIG_SYS_BOOTMAPSZ on Linux
> > > (if one even exists)?
> > 
> > The CONFIG_SYS_BOOTMAPSZ thing is as old as U-Boot and PPCBoot
> > exists, i. e. well over a decade. IIRC there was such a limitation on
> > the then current 2.2.13 Linux kernels, at least on MPC8xx and PPC40x
> > systems, which is where all started from.
> > 
> > 
> > I am pretty sure that as long as nobody ran into any problems, nobody
> > looked into that code, so it was copied from architecture to
> > architecture without much thinking, if any.
> 
> I looked at it a bit over a year ago and commited this change for the AMCC/APM 
> eval boards:
> 
> commit 6942efc2be1b90054fa4afa5cda7023469fe08b9
> Author: Stefan Roese <sr@denx.de>
> Date:   Tue Jul 28 10:50:32 2009 +0200
> 
>     ppc4xx: amcc: Set CONFIG_SYS_BOOTMAPSZ to 16MB for big kernels
>     
>     This patch changes CONFIG_SYS_BOOTMAPSZ from 8MB to 16MB which is the
>     initial TLB on 40x PPC's in the Linux kernel. With this change even bigger
>     Linux kernels (> 8MB) can be booted.
>     
>     This patch also sets CONFIG_SYS_BOOTM_LEN to 16MB (default 8MB) to enable
>     decompression of bigger images.
>     
>     Signed-off-by: Stefan Roese <sr@denx.de>
> 
> 
> So we have this 16MiB initial TLB restriction at least for PPC405 (not 
> PPC440). I'm pretty sure that 83xx has no such tight restrictions.
> 

I've just gone through both the 40x code (as a source for comparison)
and the 6xx code (generic 32-bit powerpc: 83xx, 85xx, others?).

arch/powerpc/kernel/setup_32.c's machine_init() function is the first
thing to access the device tree. The full MMU setup has not been done at
this point; the initial MMU configuration is used at this point. The
initial MMU configuration is done in arch/powerpc/kernel/head_32.S's
initial_bats() function. On 6xx, it creates a 256MB mapping:

/*
 * On 601, we use 3 BATs to map up to 24M of RAM at _PAGE_OFFSET
 * (we keep one for debugging) and on others, we use one 256M BAT.
 */

Inside U-Boot, common/image.c's boot_relocate_fdt() function uses
lmb_alloc_base() to allocate memory to hold the FDT for Linux. That
shouldn't return an invalid memory location. (It doesn't return a
pointer to unpopulated memory on a board.)

Based on that, I think it should be fine to increase
CONFIG_SYS_BOOTMAPSZ to 256MB on all 32-bit 6xx processors. This
includes 83xx and 85xx. Is 86xx included too (IIRC, it has 64bit modes)?

A boot test on my MPC8349EMDS confirms that it works:

   Booting using the fdt blob at 0x2242d6c
   Uncompressing Kernel Image ... OK
   Loading Ramdisk to 0fe9f000, end 0ff75699 ... OK
   Loading Device Tree to 0fe97000, end 0fe9e84f ... OK
[    0.000000] Using MPC834x MDS machine description
[    0.000000] Linux version 2.6.31.12-00018-g306aebe [output trimmed]

Would you prefer a patch only for the MPC8349EMDS, or should I try and
convert the other boards too? How should I know which boards are safe?
Grep for CONFIG_E300?

Thanks,
Ira

[U-Boot] 83xx fails to boot with moderately sized kernels

[Kim Phillips]

On Fri, 10 Sep 2010 11:10:23 -0700
"Ira W. Snyder" <iws@ovro.caltech.edu> wrote:

> Would you prefer a patch only for the MPC8349EMDS, or should I try and
> convert the other boards too? How should I know which boards are safe?
> Grep for CONFIG_E300?

CONFIG_MPC83xx, since E300 gets hits for some mpc51xx parts.  Not to
mention it clarifies which git tree you're targeting, so as to not
confuse custodians.

Thanks,

Kim

[U-Boot] 83xx fails to boot with moderately sized kernels

[Wolfgang Denk]

Dear "Ira W. Snyder",

In message <20100910181022.GA18510@ovro.caltech.edu> you wrote:
>
> Would you prefer a patch only for the MPC8349EMDS, or should I try and
> convert the other boards too? How should I know which boards are safe?
> Grep for CONFIG_E300?

I think we should try and update all boards where this applies. I'm
not sure, but my wild guess is that this means all processors except
40x and 8xx (probably 5xx as well, but AFAICT there is no Linux
available for these anyway, so it does not matter).

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Those who do not  understand  Unix  are  condemned  to  reinvent  it,
poorly.              - Henry Spencer, University of Toronto Unix hack

[U-Boot] 83xx fails to boot with moderately sized kernels

[Wolfgang Denk]

Dear Kim Phillips,

In message <20100910132606.cb2debcd.kim.phillips@freescale.com> you wrote:
>
> > Would you prefer a patch only for the MPC8349EMDS, or should I try and
> > convert the other boards too? How should I know which boards are safe?
> > Grep for CONFIG_E300?
> 
> CONFIG_MPC83xx, since E300 gets hits for some mpc51xx parts.  Not to
> mention it clarifies which git tree you're targeting, so as to not
> confuse custodians.

Do you think mpc51xx behaves differently?

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Mistakes are often the stepping stones to utter failure.

[U-Boot] 83xx fails to boot with moderately sized kernels

[Kim Phillips]

On Fri, 10 Sep 2010 20:53:56 +0200
Wolfgang Denk <wd@denx.de> wrote:

> Dear Kim Phillips,
> 
> In message <20100910132606.cb2debcd.kim.phillips@freescale.com> you wrote:
> >
> > > Would you prefer a patch only for the MPC8349EMDS, or should I try and
> > > convert the other boards too? How should I know which boards are safe?
> > > Grep for CONFIG_E300?
> > 
> > CONFIG_MPC83xx, since E300 gets hits for some mpc51xx parts.  Not to
> > mention it clarifies which git tree you're targeting, so as to not
> > confuse custodians.
> 
> Do you think mpc51xx behaves differently?

I don't know, I'm just trying to encourage some sort of organization to
the breadth of application.

But based on your other email, this looks like it'll be a
Wolfgang-direct patch.

Kim

[U-Boot] 83xx fails to boot with moderately sized kernels

[Joakim Tjernlund]

>
> Dear "Ira W. Snyder",
>
> In message <20100910181022.GA18510@ovro.caltech.edu> you wrote:
> >
> > Would you prefer a patch only for the MPC8349EMDS, or should I try and
> > convert the other boards too? How should I know which boards are safe?
> > Grep for CONFIG_E300?
>
> I think we should try and update all boards where this applies. I'm
> not sure, but my wild guess is that this means all processors except
> 40x and 8xx (probably 5xx as well, but AFAICT there is no Linux
> available for these anyway, so it does not matter).

I think 8xx can be fixed quite easily to do 16 MB if needed. Probably
won't be needed though.

 Jocke

[U-Boot] 83xx fails to boot with moderately sized kernels

[Scott Wood]

On Sat, 11 Sep 2010 09:19:34 +0200
Joakim Tjernlund <joakim.tjernlund@transmode.se> wrote:

> >
> > Dear "Ira W. Snyder",
> >
> > In message <20100910181022.GA18510@ovro.caltech.edu> you wrote:
> > >
> > > Would you prefer a patch only for the MPC8349EMDS, or should I try and
> > > convert the other boards too? How should I know which boards are safe?
> > > Grep for CONFIG_E300?
> >
> > I think we should try and update all boards where this applies. I'm
> > not sure, but my wild guess is that this means all processors except
> > 40x and 8xx (probably 5xx as well, but AFAICT there is no Linux
> > available for these anyway, so it does not matter).
> 
> I think 8xx can be fixed quite easily to do 16 MB if needed. Probably
> won't be needed though.

Actually, I got a complaint a while ago (I think it was Ben H) about
problems with large kernels with debugging options on 8xx.

-Scott

[U-Boot] 83xx fails to boot with moderately sized kernels

[Joakim Tjernlund]

Scott Wood <scottwood@freescale.com> wrote on 2010/09/13 21:21:33:
> On Sat, 11 Sep 2010 09:19:34 +0200
> Joakim Tjernlund <joakim.tjernlund@transmode.se> wrote:
>
> > >
> > > Dear "Ira W. Snyder",
> > >
> > > In message <20100910181022.GA18510@ovro.caltech.edu> you wrote:
> > > >
> > > > Would you prefer a patch only for the MPC8349EMDS, or should I try and
> > > > convert the other boards too? How should I know which boards are safe?
> > > > Grep for CONFIG_E300?
> > >
> > > I think we should try and update all boards where this applies. I'm
> > > not sure, but my wild guess is that this means all processors except
> > > 40x and 8xx (probably 5xx as well, but AFAICT there is no Linux
> > > available for these anyway, so it does not matter).
> >
> > I think 8xx can be fixed quite easily to do 16 MB if needed. Probably
> > won't be needed though.
>
> Actually, I got a complaint a while ago (I think it was Ben H) about
> problems with large kernels with debugging options on 8xx.

I see. I think the fix is basically enable pinned TLBs. Then a much
bigger area will be mapped. Did they try that?

 Jocke

[U-Boot] 83xx fails to boot with moderately sized kernels

[Scott Wood]

On Mon, 13 Sep 2010 23:52:00 +0200
Joakim Tjernlund <joakim.tjernlund@transmode.se> wrote:

> Scott Wood <scottwood@freescale.com> wrote on 2010/09/13 21:21:33:
> > On Sat, 11 Sep 2010 09:19:34 +0200
> > Joakim Tjernlund <joakim.tjernlund@transmode.se> wrote:
> >
> > > >
> > > > Dear "Ira W. Snyder",
> > > >
> > > > In message <20100910181022.GA18510@ovro.caltech.edu> you wrote:
> > > > >
> > > > > Would you prefer a patch only for the MPC8349EMDS, or should I try and
> > > > > convert the other boards too? How should I know which boards are safe?
> > > > > Grep for CONFIG_E300?
> > > >
> > > > I think we should try and update all boards where this applies. I'm
> > > > not sure, but my wild guess is that this means all processors except
> > > > 40x and 8xx (probably 5xx as well, but AFAICT there is no Linux
> > > > available for these anyway, so it does not matter).
> > >
> > > I think 8xx can be fixed quite easily to do 16 MB if needed. Probably
> > > won't be needed though.
> >
> > Actually, I got a complaint a while ago (I think it was Ben H) about
> > problems with large kernels with debugging options on 8xx.
> 
> I see. I think the fix is basically enable pinned TLBs. Then a much
> bigger area will be mapped. Did they try that?

Don't know if the kernel in question fit in 16 MiB or not...  it
wasn't a request for assistance, just a "hey, I ran into this,
you might want to look at mapping more memory by default" note.

-Scott