Re: Where to put a large bootloader-supplied device tree on ARM ?

From: Albert ARIBAUD <albert.u.boot@aribaud.net>
To: Rob Herring <robherring2@gmail.com>
Cc: Mitch Bradley <wmb@firmworks.com>, U-Boot <u-boot@lists.denx.de>,
	"devicetree-discuss@lists.ozlabs.org"
	<devicetree-discuss@lists.ozlabs.org>,
	"linux-arm-kernel@lists.infradead.org"
	<linux-arm-kernel@lists.infradead.org>
Subject: Re: Where to put a large bootloader-supplied device tree on ARM ?
Date: Thu, 12 Jul 2012 23:38:01 +0200	[thread overview]
Message-ID: <20120712233801.0411daa7@lilith> (raw)
In-Reply-To: <4FFF34C9.8030000@gmail.com>

Hi Rob,

On Thu, 12 Jul 2012 15:34:17 -0500, Rob Herring <robherring2@gmail.com> wrote:
> [adding u-boot list]
> 
> On 07/12/2012 01:52 AM, Mitch Bradley wrote:
> > On 7/8/2012 6:30 PM, Nicolas Pitre wrote:
> >> On Fri, 6 Jul 2012, Mitch Bradley wrote:
> >>
> >>> On 7/6/2012 3:23 PM, David VomLehn (dvomlehn) wrote:
> >>>> The kernel *must* go where it is linked, but the FDT contains only relative
> >>>> references and is thus free to go anywhere. The same is true of ramdisks,
> >>>> which
> >>>> are usually placed after the kernel.
> >>
> >> The kernel must go where it is linked *only* if you are using the 
> >> 'Image' output.  When using 'zImage' you can put the kernel anywhere in 
> >> memory, or in the first 128MB of RAM if CONFIG_AUTO_ZRELADDR is used.
> >>
> >>> Right, but the kernel image is compressed, so after decompression it expands
> >>> into the area just after it.  Also, the .bss segment is in that vicinity.
> >>
> >> To be exact, the compressed kernel moves itself out of the region where 
> >> the decompressed kernel will end up before doing the decompression, but 
> >> only if necessary.  So it is a good idea to load zImage away from the 
> >> decompressed kernel area to avoid this extra move and save some fraction 
> >> of a second on boot time.
> >>
> >>> There's some code in arch/arm/boot/compressed/head.S to relocate
> >>> device tree blobs, but it requires CONFIG_ARM_APPENDED_DTB which
> >>> is not recommended - arch/arm/Kconfig recommends using the
> >>> documented boot protocol istead .
> >>
> >> This is in case a DTB is appended to zImage.  When the DTB is detected, 
> >> the moving of zImage out of the decompressed area must take care of 
> >> moving the DTB as well.
> >>
> >>> Documentation/arm/Booting says
> >>> to put the dtb "in a region of memory where the kernel decompressor
> >>> will not overwrite it", further recommending the first 16KiB.
> >>>
> >>> As noted, the first 16KiB loses if the dtb is too large.  And
> >>> "where the kernel decompressor will not overwrite it" says what
> >>> won't work, not what will.  It appears that the decompressor works
> >>> out its addresses dynamically, so there's no hard prescription even
> >>> for what to avoid.
> >>
> >> A good rule of thumb is to take the size of the decompressed kernel and 
> >> multiply this by 3.  Rounding up is also fine.  So for example if your 
> >> arch/arm/boot/Image is 5MB, then putting anciliary data such as a 
> >> ramdisk or a large DTB from 16MB into RAM or above should be fine.
> >>
> >>> For now, I'm putting the initrd at the end of memory and the dtb
> >>> below that.  That seems to work, but I'm unsure whether or not
> >>> I'm just "getting lucky".
> >>
> >> That's also perfectly fine.
> > 
> > 
> > Alas, that worked for machines with 512 MiB of main memory, but failed
> > on 1 GiB machines.  My guess is that, when the initrd and dtb are near
> > the top of a 1 GiB memory, the virtual address gets too near the top of
> > the kernel's 1 GiB of virtual space (which starts at 0xc0000000),
> > perhaps colliding with the VMALLOC space.
> > 
> > Putting them just below the 128 MiB boundary seems to work.
> 
> Interesting. I think this is also a problem on u-boot just waiting to
> happen. u-boot locates itself at the end of RAM and likes to copy the
> fdt and initrd to just below that. Any machine with 1G+ is going to hit
> this. I avoided it because I limited u-boot to 512MB on highbank.

If I'm not mistaken, yes U-Boot loads itself as high as it can, and I don't
know about the FDT, but no, U-Boot does not "like" to load initrd "just
below that": it loads initrd where the boot commands tell it to, and the
boot commands are written by board developers. Nothing in U-Boot forces
initrd to be loaded as high as possible.

That leaves the question of the FDT, though -- I'm not familiar enough
with it (yet) to tell if it is always located just under U-Boot or if
its placement is controllable by board commands.

Amicalement,
-- 
Albert.