imx8mm memory env in U-Boot

imx8mm memory env in U-Boot

[Tim Harvey]

Greetings,

I'm trying to understand what the best memory usage is in U-Boot for
IMX8M boards for generic distro configs such as: loadaddr,
kernel_addr_r, fdt_addr_r, ramdisk_addr, scriptaddr.

My understanding is that the following is a good rule of thumb:
loadaddr = DDR start + 32MB (as FIT images may load kernel at DDR
start; but this only allows for a 32MB kernel)
kernel_addr_r = $loadaddr
fdt_addr_r = $kernel_addr_r + 128MB (allows you up to 128MB for your
kernel; handy if you want a kernel with internal ramdisk)
ramdisk_addr = fdt_addr_r + 512KB (512KB should be plenty for a dt)
scriptaddr = $loadaddr

Looking at the various imx8mm boards upstream they are kind of all
over the place but do follow some patterns likely due to some of us
just going with what prior boards used.

While I'm at it I've encountered a couple other questions:
- why on IMX8MM is CONFIG_LOADADDR is 0x40480000 when DDR starts at
0x40000000. Why the 4608KB offset?
- what is CONFIG_SYS_INIT_RAM_SIZE? Most boards are setting this to
2MB but a couple (cl-iot-gate/phycore) set it to 512KB
- what are people using for the load address for the kernel within FIT
images? I expect start of DDR is appropriate (0x40000000) however for
whatever reason I've been using 0x40200000. This plays into the env as
you can't overlap where you loaded the FIT image with where you told
the FIT image to relocate the kernel to.

Best regards,

Tim

Re: imx8mm memory env in U-Boot

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 4346 bytes --]

On Thu, Aug 26, 2021 at 09:39:20AM -0700, Tim Harvey wrote:

> Greetings,
> 
> I'm trying to understand what the best memory usage is in U-Boot for
> IMX8M boards for generic distro configs such as: loadaddr,
> kernel_addr_r, fdt_addr_r, ramdisk_addr, scriptaddr.
> 
> My understanding is that the following is a good rule of thumb:
> loadaddr = DDR start + 32MB (as FIT images may load kernel at DDR
> start; but this only allows for a 32MB kernel)
> kernel_addr_r = $loadaddr
> fdt_addr_r = $kernel_addr_r + 128MB (allows you up to 128MB for your
> kernel; handy if you want a kernel with internal ramdisk)
> ramdisk_addr = fdt_addr_r + 512KB (512KB should be plenty for a dt)
> scriptaddr = $loadaddr

Missing from the list here is bootm_size, so that we make sure
everything that does need relocation is relocated within a specific size
range.  Where much of this comes from is (or should be) the huge comment
in ti_armv7_common.h that's based off of the Linux kernel arm "booting"
document (now converted to rST):
/*
 * We setup defaults based on constraints from the Linux kernel, which should
 * also be safe elsewhere.  We have the default load at 32MB into DDR (for
 * the kernel), FDT above 128MB (the maximum location for the end of the
 * kernel), and the ramdisk 512KB above that (allowing for hopefully never
 * seen large trees).  We say all of this must be within the first 256MB
 * as that will normally be within the kernel lowmem and thus visible via
 * bootm_size and we only run on platforms with 256MB or more of memory.
 *
 * As a temporary storage for DTBO blobs (which should be applied into DTB
 * blob), we use the location 15.5 MB above the ramdisk. If someone wants to
 * use ramdisk bigger than 15.5 MB, then DTBO can be loaded and applied to DTB
 * blob before loading the ramdisk, as DTBO location is only used as a temporary
 * storage, and can be re-used after 'fdt apply' command is done.
 */

At this point, re-reading and referencing both:
https://www.kernel.org/doc/Documentation/arm64/booting.rst
https://www.kernel.org/doc/Documentation/arm/Booting
would be good, and note that there's not currently a similar document
for RISC-V, they often follow the same guidelines.  And I know you're
talking about imx8 specifically right now but due to the copy/paste
nature of these kinds of values, I like to err on the side of maximal
safety.  Which means that we should bump the DTB size to 2MB, per arm64.

It also doesn't cover kernel_comp_addr_r / kernel_comp_size for
automatic decompression of Image files, but should.

Note that I believe (but would have to think on and re-read a bunch of
stuff to be sure), it's not that saying the kernel address is at 32MB
from the start of memory limits us to 32MB, but that it makes life
easier all around.

> Looking at the various imx8mm boards upstream they are kind of all
> over the place but do follow some patterns likely due to some of us
> just going with what prior boards used.
> 
> While I'm at it I've encountered a couple other questions:
> - why on IMX8MM is CONFIG_LOADADDR is 0x40480000 when DDR starts at
> 0x40000000. Why the 4608KB offset?
> - what is CONFIG_SYS_INIT_RAM_SIZE? Most boards are setting this to
> 2MB but a couple (cl-iot-gate/phycore) set it to 512KB

I feel like it's pretty likely CONFIG_SYS_INIT_RAM_SIZE has been
copy-pasted around as part of setting CONFIG_SYS_INIT_SP_OFFSET which is
unused.  A lot of unused (outside of m68k / PowerPC generally) options
in that area.

> - what are people using for the load address for the kernel within FIT
> images? I expect start of DDR is appropriate (0x40000000) however for
> whatever reason I've been using 0x40200000. This plays into the env as
> you can't overlap where you loaded the FIT image with where you told
> the FIT image to relocate the kernel to.

Getting some documentation under doc/ about both the environment
variables and optimal FIT layout would be good.  Since we're talking
about arm64 here (but this is true for RISC-V too, same header).
Reading over booti_setup(), the entry point we set is (so long as 2MB
aligned) where we relocate to.  So the 0x40200000 would be base + 2MB,
and there were points in history on arm64 where it had to be at that
offset, I believe.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: imx8mm memory env in U-Boot

[Tim Harvey]

On Thu, Aug 26, 2021 at 12:41 PM Tom Rini <trini@konsulko.com> wrote:
>
> On Thu, Aug 26, 2021 at 09:39:20AM -0700, Tim Harvey wrote:
>
> > Greetings,
> >
> > I'm trying to understand what the best memory usage is in U-Boot for
> > IMX8M boards for generic distro configs such as: loadaddr,
> > kernel_addr_r, fdt_addr_r, ramdisk_addr, scriptaddr.
> >
> > My understanding is that the following is a good rule of thumb:
> > loadaddr = DDR start + 32MB (as FIT images may load kernel at DDR
> > start; but this only allows for a 32MB kernel)
> > kernel_addr_r = $loadaddr
> > fdt_addr_r = $kernel_addr_r + 128MB (allows you up to 128MB for your
> > kernel; handy if you want a kernel with internal ramdisk)
> > ramdisk_addr = fdt_addr_r + 512KB (512KB should be plenty for a dt)
> > scriptaddr = $loadaddr
>

Hi Tom,

Thanks for the reply.

> Missing from the list here is bootm_size, so that we make sure
> everything that does need relocation is relocated within a specific size
> range.

I still don't quite understand bootm_size, you say it sets the limit
to where things are relocated to. Shouldn't this just be the size of
dram then? A few IMX8MM boards set this but most do not.

> Where much of this comes from is (or should be) the huge comment
> in ti_armv7_common.h that's based off of the Linux kernel arm "booting"
> document (now converted to rST):
> /*
>  * We setup defaults based on constraints from the Linux kernel, which should
>  * also be safe elsewhere.  We have the default load at 32MB into DDR (for
>  * the kernel), FDT above 128MB (the maximum location for the end of the
>  * kernel), and the ramdisk 512KB above that (allowing for hopefully never
>  * seen large trees).  We say all of this must be within the first 256MB
>  * as that will normally be within the kernel lowmem and thus visible via
>  * bootm_size and we only run on platforms with 256MB or more of memory.
>  *
>  * As a temporary storage for DTBO blobs (which should be applied into DTB
>  * blob), we use the location 15.5 MB above the ramdisk. If someone wants to
>  * use ramdisk bigger than 15.5 MB, then DTBO can be loaded and applied to DTB
>  * blob before loading the ramdisk, as DTBO location is only used as a temporary
>  * storage, and can be re-used after 'fdt apply' command is done.
>  */

Right, that's where I got my recommendations and I don't understand
the reasoning behind the default loadaddr be 32MB into DDR.

>
> At this point, re-reading and referencing both:
> https://www.kernel.org/doc/Documentation/arm64/booting.rst
> https://www.kernel.org/doc/Documentation/arm/Booting
> would be good, and note that there's not currently a similar document
> for RISC-V, they often follow the same guidelines.  And I know you're
> talking about imx8 specifically right now but due to the copy/paste
> nature of these kinds of values, I like to err on the side of maximal
> safety.  Which means that we should bump the DTB size to 2MB, per arm64.

ok, good to know 2MB should be allotted for dtb.

>
> It also doesn't cover kernel_comp_addr_r / kernel_comp_size for
> automatic decompression of Image files, but should.
>

interesting... I didn't even realize booti supported compressed images!

I see now commit 414c34ed55: ("image: Add compressed Image parsing
support in booti."). I'm not clear why the uncompressed kernel needs
to be moved to kernel_addr_r after decompression... why can't it
simply be decompressed directly to kernel_addr_r?

I would think kernel_comp_size would typically be set to filesize as
currently that is set by tftp as well as fs load commands.

> Note that I believe (but would have to think on and re-read a bunch of
> stuff to be sure), it's not that saying the kernel address is at 32MB
> from the start of memory limits us to 32MB, but that it makes life
> easier all around.
>
> > Looking at the various imx8mm boards upstream they are kind of all
> > over the place but do follow some patterns likely due to some of us
> > just going with what prior boards used.
> >
> > While I'm at it I've encountered a couple other questions:
> > - why on IMX8MM is CONFIG_LOADADDR is 0x40480000 when DDR starts at
> > 0x40000000. Why the 4608KB offset?

any idea why IMX8MM boards are using DDR+4608KB for loadaddr vs just
DDR? I am hoping some of the IMX8MM board maintainers I've cc'd here
can answer that.

> > - what is CONFIG_SYS_INIT_RAM_SIZE? Most boards are setting this to
> > 2MB but a couple (cl-iot-gate/phycore) set it to 512KB
>
> I feel like it's pretty likely CONFIG_SYS_INIT_RAM_SIZE has been
> copy-pasted around as part of setting CONFIG_SYS_INIT_SP_OFFSET which is
> unused.  A lot of unused (outside of m68k / PowerPC generally) options
> in that area.
>
> > - what are people using for the load address for the kernel within FIT
> > images? I expect start of DDR is appropriate (0x40000000) however for
> > whatever reason I've been using 0x40200000. This plays into the env as
> > you can't overlap where you loaded the FIT image with where you told
> > the FIT image to relocate the kernel to.
>
> Getting some documentation under doc/ about both the environment
> variables and optimal FIT layout would be good.  Since we're talking
> about arm64 here (but this is true for RISC-V too, same header).
> Reading over booti_setup(), the entry point we set is (so long as 2MB
> aligned) where we relocate to.  So the 0x40200000 would be base + 2MB,
> and there were points in history on arm64 where it had to be at that
> offset, I believe.

I did verify that using 0x40000000 for loadaddr in a FIT image booted
a modern kernel just fine so yes I guess the reasoning must have been
historical.

Best regards,

Tim

Re: imx8mm memory env in U-Boot

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 7360 bytes --]

On Fri, Aug 27, 2021 at 02:32:00PM -0700, Tim Harvey wrote:
> On Thu, Aug 26, 2021 at 12:41 PM Tom Rini <trini@konsulko.com> wrote:
> >
> > On Thu, Aug 26, 2021 at 09:39:20AM -0700, Tim Harvey wrote:
> >
> > > Greetings,
> > >
> > > I'm trying to understand what the best memory usage is in U-Boot for
> > > IMX8M boards for generic distro configs such as: loadaddr,
> > > kernel_addr_r, fdt_addr_r, ramdisk_addr, scriptaddr.
> > >
> > > My understanding is that the following is a good rule of thumb:
> > > loadaddr = DDR start + 32MB (as FIT images may load kernel at DDR
> > > start; but this only allows for a 32MB kernel)
> > > kernel_addr_r = $loadaddr
> > > fdt_addr_r = $kernel_addr_r + 128MB (allows you up to 128MB for your
> > > kernel; handy if you want a kernel with internal ramdisk)
> > > ramdisk_addr = fdt_addr_r + 512KB (512KB should be plenty for a dt)
> > > scriptaddr = $loadaddr
> >
> 
> Hi Tom,
> 
> Thanks for the reply.
> 
> > Missing from the list here is bootm_size, so that we make sure
> > everything that does need relocation is relocated within a specific size
> > range.
> 
> I still don't quite understand bootm_size, you say it sets the limit
> to where things are relocated to. Shouldn't this just be the size of
> dram then? A few IMX8MM boards set this but most do not.

Both ARM32 and ARM64 define limits on where in memory anything can
reside and still be seen / used by the kernel early on.  There's
certainly a lot of platforms where the whole of DRAM fits in that
window, being super safe for copy/paste is a big concern of mine over
all.  When I wrote out that comment for the TI platforms, I remember
seeing just how much arbitrary offsets had been reused over and over and
on many platforms without understanding why.

> > Where much of this comes from is (or should be) the huge comment
> > in ti_armv7_common.h that's based off of the Linux kernel arm "booting"
> > document (now converted to rST):
> > /*
> >  * We setup defaults based on constraints from the Linux kernel, which should
> >  * also be safe elsewhere.  We have the default load at 32MB into DDR (for
> >  * the kernel), FDT above 128MB (the maximum location for the end of the
> >  * kernel), and the ramdisk 512KB above that (allowing for hopefully never
> >  * seen large trees).  We say all of this must be within the first 256MB
> >  * as that will normally be within the kernel lowmem and thus visible via
> >  * bootm_size and we only run on platforms with 256MB or more of memory.
> >  *
> >  * As a temporary storage for DTBO blobs (which should be applied into DTB
> >  * blob), we use the location 15.5 MB above the ramdisk. If someone wants to
> >  * use ramdisk bigger than 15.5 MB, then DTBO can be loaded and applied to DTB
> >  * blob before loading the ramdisk, as DTBO location is only used as a temporary
> >  * storage, and can be re-used after 'fdt apply' command is done.
> >  */
> 
> Right, that's where I got my recommendations and I don't understand
> the reasoning behind the default loadaddr be 32MB into DDR.

I _think_, but would want to re-re-read
https://people.kernel.org/linusw/how-the-arm32-linux-kernel-decompresses
to make sure I'm remembering it right, if we put the zImage at 32MB
offset, that means the decompressor won't have to move things another
time, and saves boot time.  Not a concern for ARM64 because it's our
responsibility to do the decompression.

> > At this point, re-reading and referencing both:
> > https://www.kernel.org/doc/Documentation/arm64/booting.rst
> > https://www.kernel.org/doc/Documentation/arm/Booting
> > would be good, and note that there's not currently a similar document
> > for RISC-V, they often follow the same guidelines.  And I know you're
> > talking about imx8 specifically right now but due to the copy/paste
> > nature of these kinds of values, I like to err on the side of maximal
> > safety.  Which means that we should bump the DTB size to 2MB, per arm64.
> 
> ok, good to know 2MB should be allotted for dtb.
> 
> >
> > It also doesn't cover kernel_comp_addr_r / kernel_comp_size for
> > automatic decompression of Image files, but should.
> 
> interesting... I didn't even realize booti supported compressed images!
> 
> I see now commit 414c34ed55: ("image: Add compressed Image parsing
> support in booti."). I'm not clear why the uncompressed kernel needs
> to be moved to kernel_addr_r after decompression... why can't it
> simply be decompressed directly to kernel_addr_r?
> 
> I would think kernel_comp_size would typically be set to filesize as
> currently that is set by tftp as well as fs load commands.

Off-hand, I'm not sure it couldn't be further optimized.  I suspect it's
about avoiding overlaps, but there could be some checks made for optimal
aligned values and avoid that.

> > Note that I believe (but would have to think on and re-read a bunch of
> > stuff to be sure), it's not that saying the kernel address is at 32MB
> > from the start of memory limits us to 32MB, but that it makes life
> > easier all around.
> >
> > > Looking at the various imx8mm boards upstream they are kind of all
> > > over the place but do follow some patterns likely due to some of us
> > > just going with what prior boards used.
> > >
> > > While I'm at it I've encountered a couple other questions:
> > > - why on IMX8MM is CONFIG_LOADADDR is 0x40480000 when DDR starts at
> > > 0x40000000. Why the 4608KB offset?
> 
> any idea why IMX8MM boards are using DDR+4608KB for loadaddr vs just
> DDR? I am hoping some of the IMX8MM board maintainers I've cc'd here
> can answer that.

I too would be interested in knowing what's going on there.

> > > - what is CONFIG_SYS_INIT_RAM_SIZE? Most boards are setting this to
> > > 2MB but a couple (cl-iot-gate/phycore) set it to 512KB
> >
> > I feel like it's pretty likely CONFIG_SYS_INIT_RAM_SIZE has been
> > copy-pasted around as part of setting CONFIG_SYS_INIT_SP_OFFSET which is
> > unused.  A lot of unused (outside of m68k / PowerPC generally) options
> > in that area.
> >
> > > - what are people using for the load address for the kernel within FIT
> > > images? I expect start of DDR is appropriate (0x40000000) however for
> > > whatever reason I've been using 0x40200000. This plays into the env as
> > > you can't overlap where you loaded the FIT image with where you told
> > > the FIT image to relocate the kernel to.
> >
> > Getting some documentation under doc/ about both the environment
> > variables and optimal FIT layout would be good.  Since we're talking
> > about arm64 here (but this is true for RISC-V too, same header).
> > Reading over booti_setup(), the entry point we set is (so long as 2MB
> > aligned) where we relocate to.  So the 0x40200000 would be base + 2MB,
> > and there were points in history on arm64 where it had to be at that
> > offset, I believe.
> 
> I did verify that using 0x40000000 for loadaddr in a FIT image booted
> a modern kernel just fine so yes I guess the reasoning must have been
> historical.

Good to know.  Were there any other relocations shown?  One part of
these values was also to hope to avoid any avoidable relocations of the
kernel (and if possible, initrd, but that can be trickier).

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]