Broken support for 4GB DDR on 32-bit platforms

Broken support for 4GB DDR on 32-bit platforms

[Pali Rohár]

Hello! I tried to enable support for 2GB+ of DDR memory (with 4GB DDR3)
on powerpc P2020 board in 32-bit addressing mode and U-Boot crashed
during startup.

I figured out that issue is not powerpc specific, but rather generic to
all 32-bit platforms. U-Boot stores memory size into phys_size_t type
(gd->ram_size) and last mapped memory address increased by one byte into
phys_addr_t type (gd->ram_top).

Despite size 4GB fits into 32-bit addressing mode, it does not fit into
above two variables, and it overflows to zero. U-Boot then see zero RAM
size and crashes.

I tried to workaround this issue by changing both phys_size_t and
phys_addr_t types to 64-bit. But it did not helped because U-Boot on
many places cast gd->ram_size or gd->ram_top to ulong type, which is
32-bit on 32-bit platforms.

Next I changed ulong parameters of board_get_usable_ram_top() function
to u64.

This still was not enough because config value CONFIG_MAX_MEM_MAPPED is
ignored on one important place -- in function get_effective_memsize().
This config value takes effect only when also CONFIG_VERY_BIG_RAM is
set.

Finally With this change I was able to start U-Boot with more than 2GB
of DDR memory inserted in SODIMM slot on P2020.

How to fix issues with gd->ram_size and gd->ram_top? That +1 byte is
really stupid limitation. Changing phys_size_t and phys_addr_t types
unconditionally to 64-bit? Or something else?

And what is the purpose of CONFIG_VERY_BIG_RAM config option? Why is
CONFIG_MAX_MEM_MAPPED check skipped in get_effective_memsize() function,
but is not skipped on many more places?

Re: Broken support for 4GB DDR on 32-bit platforms

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 2616 bytes --]

On Sat, May 14, 2022 at 01:00:06AM +0200, Pali Rohár wrote:

> Hello! I tried to enable support for 2GB+ of DDR memory (with 4GB DDR3)
> on powerpc P2020 board in 32-bit addressing mode and U-Boot crashed
> during startup.
> 
> I figured out that issue is not powerpc specific, but rather generic to
> all 32-bit platforms. U-Boot stores memory size into phys_size_t type
> (gd->ram_size) and last mapped memory address increased by one byte into
> phys_addr_t type (gd->ram_top).
> 
> Despite size 4GB fits into 32-bit addressing mode, it does not fit into
> above two variables, and it overflows to zero. U-Boot then see zero RAM
> size and crashes.
> 
> I tried to workaround this issue by changing both phys_size_t and
> phys_addr_t types to 64-bit. But it did not helped because U-Boot on
> many places cast gd->ram_size or gd->ram_top to ulong type, which is
> 32-bit on 32-bit platforms.
> 
> Next I changed ulong parameters of board_get_usable_ram_top() function
> to u64.
> 
> This still was not enough because config value CONFIG_MAX_MEM_MAPPED is
> ignored on one important place -- in function get_effective_memsize().
> This config value takes effect only when also CONFIG_VERY_BIG_RAM is
> set.
> 
> Finally With this change I was able to start U-Boot with more than 2GB
> of DDR memory inserted in SODIMM slot on P2020.
> 
> How to fix issues with gd->ram_size and gd->ram_top? That +1 byte is
> really stupid limitation. Changing phys_size_t and phys_addr_t types
> unconditionally to 64-bit? Or something else?
> 
> And what is the purpose of CONFIG_VERY_BIG_RAM config option? Why is
> CONFIG_MAX_MEM_MAPPED check skipped in get_effective_memsize() function,
> but is not skipped on many more places?

So, there's two parts to this, as I recall it all.  First, even on 64bit
platforms we contain ourselves to 32bit address space and even then
something within the "old" 2GB window.  We then set a CONFIG option to
not mess with the memory node in DT which has the real value.  Second,
for 32bit platforms which can support 4GB memory, or more, some further
games need to be played, typically I believe around initializing the
memory controller (I'm more confident of that for dra7xx_evm, which I
don't have the big memory version of, just a small memory one) so that
Linux can do whatever needs doing to enable "36bit" typically address
support.  Looking at the other P*36BIT* configs might give you some more
clues about what to do on your platform, or at least who might still be
able to explain and test things on the PowerPC side.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: Broken support for 4GB DDR on 32-bit platforms

[Pali Rohár]

On Monday 16 May 2022 08:31:43 Tom Rini wrote:
> On Sat, May 14, 2022 at 01:00:06AM +0200, Pali Rohár wrote:
> 
> > Hello! I tried to enable support for 2GB+ of DDR memory (with 4GB DDR3)
> > on powerpc P2020 board in 32-bit addressing mode and U-Boot crashed
> > during startup.
> > 
> > I figured out that issue is not powerpc specific, but rather generic to
> > all 32-bit platforms. U-Boot stores memory size into phys_size_t type
> > (gd->ram_size) and last mapped memory address increased by one byte into
> > phys_addr_t type (gd->ram_top).
> > 
> > Despite size 4GB fits into 32-bit addressing mode, it does not fit into
> > above two variables, and it overflows to zero. U-Boot then see zero RAM
> > size and crashes.
> > 
> > I tried to workaround this issue by changing both phys_size_t and
> > phys_addr_t types to 64-bit. But it did not helped because U-Boot on
> > many places cast gd->ram_size or gd->ram_top to ulong type, which is
> > 32-bit on 32-bit platforms.
> > 
> > Next I changed ulong parameters of board_get_usable_ram_top() function
> > to u64.
> > 
> > This still was not enough because config value CONFIG_MAX_MEM_MAPPED is
> > ignored on one important place -- in function get_effective_memsize().
> > This config value takes effect only when also CONFIG_VERY_BIG_RAM is
> > set.
> > 
> > Finally With this change I was able to start U-Boot with more than 2GB
> > of DDR memory inserted in SODIMM slot on P2020.
> > 
> > How to fix issues with gd->ram_size and gd->ram_top? That +1 byte is
> > really stupid limitation. Changing phys_size_t and phys_addr_t types
> > unconditionally to 64-bit? Or something else?
> > 
> > And what is the purpose of CONFIG_VERY_BIG_RAM config option? Why is
> > CONFIG_MAX_MEM_MAPPED check skipped in get_effective_memsize() function,
> > but is not skipped on many more places?
> 
> So, there's two parts to this, as I recall it all.  First, even on 64bit
> platforms we contain ourselves to 32bit address space and even then
> something within the "old" 2GB window.  We then set a CONFIG option to
> not mess with the memory node in DT which has the real value.  Second,
> for 32bit platforms which can support 4GB memory, or more, some further
> games need to be played, typically I believe around initializing the
> memory controller (I'm more confident of that for dra7xx_evm, which I
> don't have the big memory version of, just a small memory one) so that
> Linux can do whatever needs doing to enable "36bit" typically address
> support.  Looking at the other P*36BIT* configs might give you some more
> clues about what to do on your platform, or at least who might still be
> able to explain and test things on the PowerPC side.

I know about 36-bit addressing on e500v2 but I'm not going to enable it
due to performance reasons (see Freescale AN4064 [1]). So I want to
stick with 32-bit addressing for 2GB+ memory usage (around 3GB; it is
4GB minus memory used by peripherals; which is still more than 2GB).

And due to 32-bit type for phys_size_t, phys_addr_t and casting these
types to ulong is an issue. Plus issue with CONFIG_VERY_BIG_RAM and
CONFIG_MAX_MEM_MAPPED as I already wrote.

[1] - https://www.nxp.com/docs/en/application-note/AN4064.pdf

Re: Broken support for 4GB DDR on 32-bit platforms

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 3729 bytes --]

On Mon, May 16, 2022 at 11:56:51PM +0200, Pali Rohár wrote:
> On Monday 16 May 2022 08:31:43 Tom Rini wrote:
> > On Sat, May 14, 2022 at 01:00:06AM +0200, Pali Rohár wrote:
> > 
> > > Hello! I tried to enable support for 2GB+ of DDR memory (with 4GB DDR3)
> > > on powerpc P2020 board in 32-bit addressing mode and U-Boot crashed
> > > during startup.
> > > 
> > > I figured out that issue is not powerpc specific, but rather generic to
> > > all 32-bit platforms. U-Boot stores memory size into phys_size_t type
> > > (gd->ram_size) and last mapped memory address increased by one byte into
> > > phys_addr_t type (gd->ram_top).
> > > 
> > > Despite size 4GB fits into 32-bit addressing mode, it does not fit into
> > > above two variables, and it overflows to zero. U-Boot then see zero RAM
> > > size and crashes.
> > > 
> > > I tried to workaround this issue by changing both phys_size_t and
> > > phys_addr_t types to 64-bit. But it did not helped because U-Boot on
> > > many places cast gd->ram_size or gd->ram_top to ulong type, which is
> > > 32-bit on 32-bit platforms.
> > > 
> > > Next I changed ulong parameters of board_get_usable_ram_top() function
> > > to u64.
> > > 
> > > This still was not enough because config value CONFIG_MAX_MEM_MAPPED is
> > > ignored on one important place -- in function get_effective_memsize().
> > > This config value takes effect only when also CONFIG_VERY_BIG_RAM is
> > > set.
> > > 
> > > Finally With this change I was able to start U-Boot with more than 2GB
> > > of DDR memory inserted in SODIMM slot on P2020.
> > > 
> > > How to fix issues with gd->ram_size and gd->ram_top? That +1 byte is
> > > really stupid limitation. Changing phys_size_t and phys_addr_t types
> > > unconditionally to 64-bit? Or something else?
> > > 
> > > And what is the purpose of CONFIG_VERY_BIG_RAM config option? Why is
> > > CONFIG_MAX_MEM_MAPPED check skipped in get_effective_memsize() function,
> > > but is not skipped on many more places?
> > 
> > So, there's two parts to this, as I recall it all.  First, even on 64bit
> > platforms we contain ourselves to 32bit address space and even then
> > something within the "old" 2GB window.  We then set a CONFIG option to
> > not mess with the memory node in DT which has the real value.  Second,
> > for 32bit platforms which can support 4GB memory, or more, some further
> > games need to be played, typically I believe around initializing the
> > memory controller (I'm more confident of that for dra7xx_evm, which I
> > don't have the big memory version of, just a small memory one) so that
> > Linux can do whatever needs doing to enable "36bit" typically address
> > support.  Looking at the other P*36BIT* configs might give you some more
> > clues about what to do on your platform, or at least who might still be
> > able to explain and test things on the PowerPC side.
> 
> I know about 36-bit addressing on e500v2 but I'm not going to enable it
> due to performance reasons (see Freescale AN4064 [1]). So I want to
> stick with 32-bit addressing for 2GB+ memory usage (around 3GB; it is
> 4GB minus memory used by peripherals; which is still more than 2GB).
> 
> And due to 32-bit type for phys_size_t, phys_addr_t and casting these
> types to ulong is an issue. Plus issue with CONFIG_VERY_BIG_RAM and
> CONFIG_MAX_MEM_MAPPED as I already wrote.

I'm not seeing the problem, sorry.  You run U-Boot in the normal 2GB
area and tell Linux there's 3GB available.  I kinda recall now that yes,
you can do a 3:1 split in Linux instead of 2:2 split, but in that case
you still do 2:2 in U-Boot.  As you have seen configuring for anything
else doesn't work.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: Broken support for 4GB DDR on 32-bit platforms

[Pali Rohár]

On Tuesday 17 May 2022 11:52:14 Tom Rini wrote:
> On Mon, May 16, 2022 at 11:56:51PM +0200, Pali Rohár wrote:
> > On Monday 16 May 2022 08:31:43 Tom Rini wrote:
> > > On Sat, May 14, 2022 at 01:00:06AM +0200, Pali Rohár wrote:
> > > 
> > > > Hello! I tried to enable support for 2GB+ of DDR memory (with 4GB DDR3)
> > > > on powerpc P2020 board in 32-bit addressing mode and U-Boot crashed
> > > > during startup.
> > > > 
> > > > I figured out that issue is not powerpc specific, but rather generic to
> > > > all 32-bit platforms. U-Boot stores memory size into phys_size_t type
> > > > (gd->ram_size) and last mapped memory address increased by one byte into
> > > > phys_addr_t type (gd->ram_top).
> > > > 
> > > > Despite size 4GB fits into 32-bit addressing mode, it does not fit into
> > > > above two variables, and it overflows to zero. U-Boot then see zero RAM
> > > > size and crashes.
> > > > 
> > > > I tried to workaround this issue by changing both phys_size_t and
> > > > phys_addr_t types to 64-bit. But it did not helped because U-Boot on
> > > > many places cast gd->ram_size or gd->ram_top to ulong type, which is
> > > > 32-bit on 32-bit platforms.
> > > > 
> > > > Next I changed ulong parameters of board_get_usable_ram_top() function
> > > > to u64.
> > > > 
> > > > This still was not enough because config value CONFIG_MAX_MEM_MAPPED is
> > > > ignored on one important place -- in function get_effective_memsize().
> > > > This config value takes effect only when also CONFIG_VERY_BIG_RAM is
> > > > set.
> > > > 
> > > > Finally With this change I was able to start U-Boot with more than 2GB
> > > > of DDR memory inserted in SODIMM slot on P2020.
> > > > 
> > > > How to fix issues with gd->ram_size and gd->ram_top? That +1 byte is
> > > > really stupid limitation. Changing phys_size_t and phys_addr_t types
> > > > unconditionally to 64-bit? Or something else?
> > > > 
> > > > And what is the purpose of CONFIG_VERY_BIG_RAM config option? Why is
> > > > CONFIG_MAX_MEM_MAPPED check skipped in get_effective_memsize() function,
> > > > but is not skipped on many more places?
> > > 
> > > So, there's two parts to this, as I recall it all.  First, even on 64bit
> > > platforms we contain ourselves to 32bit address space and even then
> > > something within the "old" 2GB window.  We then set a CONFIG option to
> > > not mess with the memory node in DT which has the real value.  Second,
> > > for 32bit platforms which can support 4GB memory, or more, some further
> > > games need to be played, typically I believe around initializing the
> > > memory controller (I'm more confident of that for dra7xx_evm, which I
> > > don't have the big memory version of, just a small memory one) so that
> > > Linux can do whatever needs doing to enable "36bit" typically address
> > > support.  Looking at the other P*36BIT* configs might give you some more
> > > clues about what to do on your platform, or at least who might still be
> > > able to explain and test things on the PowerPC side.
> > 
> > I know about 36-bit addressing on e500v2 but I'm not going to enable it
> > due to performance reasons (see Freescale AN4064 [1]). So I want to
> > stick with 32-bit addressing for 2GB+ memory usage (around 3GB; it is
> > 4GB minus memory used by peripherals; which is still more than 2GB).
> > 
> > And due to 32-bit type for phys_size_t, phys_addr_t and casting these
> > types to ulong is an issue. Plus issue with CONFIG_VERY_BIG_RAM and
> > CONFIG_MAX_MEM_MAPPED as I already wrote.
> 
> I'm not seeing the problem, sorry.  You run U-Boot in the normal 2GB

Ok, I will try to explain it again.

I want to run U-Boot in normal 2GB area. U-Boot normally store detected
RAM size into the gd->ram_size structure. And here is the issue.

32-bit unsigned C type cannot represent maximal 32-bit addressable size.
U-Boot tries to store 4GB value = 0x100000000 into gd->ram_size which
overflows to 0x00000000. And U-Boot then crashes because it expects that
can store some data at address "gd->ram_size - few_kb" as it expects
that RAM is in the area [0, gd->ram_size).

It is more clear what is the problem?

If I theoretically find 3.9GB RAM module (but such probably does not
exist) then U-Boot should work in normal 2GB area as number 3.9GB can be
stored into 32-bit unsigned type.

> area and tell Linux there's 3GB available.  I kinda recall now that yes,
> you can do a 3:1 split in Linux instead of 2:2 split, but in that case
> you still do 2:2 in U-Boot.  As you have seen configuring for anything
> else doesn't work.
> 
> -- 
> Tom

Re: Broken support for 4GB DDR on 32-bit platforms

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 4841 bytes --]

On Tue, May 17, 2022 at 06:00:16PM +0200, Pali Rohár wrote:
> On Tuesday 17 May 2022 11:52:14 Tom Rini wrote:
> > On Mon, May 16, 2022 at 11:56:51PM +0200, Pali Rohár wrote:
> > > On Monday 16 May 2022 08:31:43 Tom Rini wrote:
> > > > On Sat, May 14, 2022 at 01:00:06AM +0200, Pali Rohár wrote:
> > > > 
> > > > > Hello! I tried to enable support for 2GB+ of DDR memory (with 4GB DDR3)
> > > > > on powerpc P2020 board in 32-bit addressing mode and U-Boot crashed
> > > > > during startup.
> > > > > 
> > > > > I figured out that issue is not powerpc specific, but rather generic to
> > > > > all 32-bit platforms. U-Boot stores memory size into phys_size_t type
> > > > > (gd->ram_size) and last mapped memory address increased by one byte into
> > > > > phys_addr_t type (gd->ram_top).
> > > > > 
> > > > > Despite size 4GB fits into 32-bit addressing mode, it does not fit into
> > > > > above two variables, and it overflows to zero. U-Boot then see zero RAM
> > > > > size and crashes.
> > > > > 
> > > > > I tried to workaround this issue by changing both phys_size_t and
> > > > > phys_addr_t types to 64-bit. But it did not helped because U-Boot on
> > > > > many places cast gd->ram_size or gd->ram_top to ulong type, which is
> > > > > 32-bit on 32-bit platforms.
> > > > > 
> > > > > Next I changed ulong parameters of board_get_usable_ram_top() function
> > > > > to u64.
> > > > > 
> > > > > This still was not enough because config value CONFIG_MAX_MEM_MAPPED is
> > > > > ignored on one important place -- in function get_effective_memsize().
> > > > > This config value takes effect only when also CONFIG_VERY_BIG_RAM is
> > > > > set.
> > > > > 
> > > > > Finally With this change I was able to start U-Boot with more than 2GB
> > > > > of DDR memory inserted in SODIMM slot on P2020.
> > > > > 
> > > > > How to fix issues with gd->ram_size and gd->ram_top? That +1 byte is
> > > > > really stupid limitation. Changing phys_size_t and phys_addr_t types
> > > > > unconditionally to 64-bit? Or something else?
> > > > > 
> > > > > And what is the purpose of CONFIG_VERY_BIG_RAM config option? Why is
> > > > > CONFIG_MAX_MEM_MAPPED check skipped in get_effective_memsize() function,
> > > > > but is not skipped on many more places?
> > > > 
> > > > So, there's two parts to this, as I recall it all.  First, even on 64bit
> > > > platforms we contain ourselves to 32bit address space and even then
> > > > something within the "old" 2GB window.  We then set a CONFIG option to
> > > > not mess with the memory node in DT which has the real value.  Second,
> > > > for 32bit platforms which can support 4GB memory, or more, some further
> > > > games need to be played, typically I believe around initializing the
> > > > memory controller (I'm more confident of that for dra7xx_evm, which I
> > > > don't have the big memory version of, just a small memory one) so that
> > > > Linux can do whatever needs doing to enable "36bit" typically address
> > > > support.  Looking at the other P*36BIT* configs might give you some more
> > > > clues about what to do on your platform, or at least who might still be
> > > > able to explain and test things on the PowerPC side.
> > > 
> > > I know about 36-bit addressing on e500v2 but I'm not going to enable it
> > > due to performance reasons (see Freescale AN4064 [1]). So I want to
> > > stick with 32-bit addressing for 2GB+ memory usage (around 3GB; it is
> > > 4GB minus memory used by peripherals; which is still more than 2GB).
> > > 
> > > And due to 32-bit type for phys_size_t, phys_addr_t and casting these
> > > types to ulong is an issue. Plus issue with CONFIG_VERY_BIG_RAM and
> > > CONFIG_MAX_MEM_MAPPED as I already wrote.
> > 
> > I'm not seeing the problem, sorry.  You run U-Boot in the normal 2GB
> 
> Ok, I will try to explain it again.
> 
> I want to run U-Boot in normal 2GB area. U-Boot normally store detected
> RAM size into the gd->ram_size structure. And here is the issue.
> 
> 32-bit unsigned C type cannot represent maximal 32-bit addressable size.
> U-Boot tries to store 4GB value = 0x100000000 into gd->ram_size which
> overflows to 0x00000000. And U-Boot then crashes because it expects that
> can store some data at address "gd->ram_size - few_kb" as it expects
> that RAM is in the area [0, gd->ram_size).
> 
> It is more clear what is the problem?
> 
> If I theoretically find 3.9GB RAM module (but such probably does not
> exist) then U-Boot should work in normal 2GB area as number 3.9GB can be
> stored into 32-bit unsigned type.

Yes, you need to disable CONFIG_ARCH_FIXUP_FDT_MEMORY so that instead of
the memory size U-Boot calculates, you update the FDT in your board code
with the size you want, or you have it passed in correctly.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: Broken support for 4GB DDR on 32-bit platforms

[Pali Rohár]

On Tuesday 17 May 2022 12:38:43 Tom Rini wrote:
> On Tue, May 17, 2022 at 06:00:16PM +0200, Pali Rohár wrote:
> > On Tuesday 17 May 2022 11:52:14 Tom Rini wrote:
> > > On Mon, May 16, 2022 at 11:56:51PM +0200, Pali Rohár wrote:
> > > > On Monday 16 May 2022 08:31:43 Tom Rini wrote:
> > > > > On Sat, May 14, 2022 at 01:00:06AM +0200, Pali Rohár wrote:
> > > > > 
> > > > > > Hello! I tried to enable support for 2GB+ of DDR memory (with 4GB DDR3)
> > > > > > on powerpc P2020 board in 32-bit addressing mode and U-Boot crashed
> > > > > > during startup.
> > > > > > 
> > > > > > I figured out that issue is not powerpc specific, but rather generic to
> > > > > > all 32-bit platforms. U-Boot stores memory size into phys_size_t type
> > > > > > (gd->ram_size) and last mapped memory address increased by one byte into
> > > > > > phys_addr_t type (gd->ram_top).
> > > > > > 
> > > > > > Despite size 4GB fits into 32-bit addressing mode, it does not fit into
> > > > > > above two variables, and it overflows to zero. U-Boot then see zero RAM
> > > > > > size and crashes.
> > > > > > 
> > > > > > I tried to workaround this issue by changing both phys_size_t and
> > > > > > phys_addr_t types to 64-bit. But it did not helped because U-Boot on
> > > > > > many places cast gd->ram_size or gd->ram_top to ulong type, which is
> > > > > > 32-bit on 32-bit platforms.
> > > > > > 
> > > > > > Next I changed ulong parameters of board_get_usable_ram_top() function
> > > > > > to u64.
> > > > > > 
> > > > > > This still was not enough because config value CONFIG_MAX_MEM_MAPPED is
> > > > > > ignored on one important place -- in function get_effective_memsize().
> > > > > > This config value takes effect only when also CONFIG_VERY_BIG_RAM is
> > > > > > set.
> > > > > > 
> > > > > > Finally With this change I was able to start U-Boot with more than 2GB
> > > > > > of DDR memory inserted in SODIMM slot on P2020.
> > > > > > 
> > > > > > How to fix issues with gd->ram_size and gd->ram_top? That +1 byte is
> > > > > > really stupid limitation. Changing phys_size_t and phys_addr_t types
> > > > > > unconditionally to 64-bit? Or something else?
> > > > > > 
> > > > > > And what is the purpose of CONFIG_VERY_BIG_RAM config option? Why is
> > > > > > CONFIG_MAX_MEM_MAPPED check skipped in get_effective_memsize() function,
> > > > > > but is not skipped on many more places?
> > > > > 
> > > > > So, there's two parts to this, as I recall it all.  First, even on 64bit
> > > > > platforms we contain ourselves to 32bit address space and even then
> > > > > something within the "old" 2GB window.  We then set a CONFIG option to
> > > > > not mess with the memory node in DT which has the real value.  Second,
> > > > > for 32bit platforms which can support 4GB memory, or more, some further
> > > > > games need to be played, typically I believe around initializing the
> > > > > memory controller (I'm more confident of that for dra7xx_evm, which I
> > > > > don't have the big memory version of, just a small memory one) so that
> > > > > Linux can do whatever needs doing to enable "36bit" typically address
> > > > > support.  Looking at the other P*36BIT* configs might give you some more
> > > > > clues about what to do on your platform, or at least who might still be
> > > > > able to explain and test things on the PowerPC side.
> > > > 
> > > > I know about 36-bit addressing on e500v2 but I'm not going to enable it
> > > > due to performance reasons (see Freescale AN4064 [1]). So I want to
> > > > stick with 32-bit addressing for 2GB+ memory usage (around 3GB; it is
> > > > 4GB minus memory used by peripherals; which is still more than 2GB).
> > > > 
> > > > And due to 32-bit type for phys_size_t, phys_addr_t and casting these
> > > > types to ulong is an issue. Plus issue with CONFIG_VERY_BIG_RAM and
> > > > CONFIG_MAX_MEM_MAPPED as I already wrote.
> > > 
> > > I'm not seeing the problem, sorry.  You run U-Boot in the normal 2GB
> > 
> > Ok, I will try to explain it again.
> > 
> > I want to run U-Boot in normal 2GB area. U-Boot normally store detected
> > RAM size into the gd->ram_size structure. And here is the issue.
> > 
> > 32-bit unsigned C type cannot represent maximal 32-bit addressable size.
> > U-Boot tries to store 4GB value = 0x100000000 into gd->ram_size which
> > overflows to 0x00000000. And U-Boot then crashes because it expects that
> > can store some data at address "gd->ram_size - few_kb" as it expects
> > that RAM is in the area [0, gd->ram_size).
> > 
> > It is more clear what is the problem?
> > 
> > If I theoretically find 3.9GB RAM module (but such probably does not
> > exist) then U-Boot should work in normal 2GB area as number 3.9GB can be
> > stored into 32-bit unsigned type.
> 
> Yes, you need to disable CONFIG_ARCH_FIXUP_FDT_MEMORY

CONFIG_ARCH_FIXUP_FDT_MEMORY affects booting OS.

The issue is that **u-boot** is crashing during its initialization.

I'm not even able to go into the phase of booting OS.

So CONFIG_ARCH_FIXUP_FDT_MEMORY does absolutely nothing here.

> so that instead of
> the memory size U-Boot calculates, you update the FDT in your board code
> with the size you want, or you have it passed in correctly.
> 
> -- 
> Tom

Re: Broken support for 4GB DDR on 32-bit platforms

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 5342 bytes --]

On Wed, May 18, 2022 at 12:58:38PM +0200, Pali Rohár wrote:
> On Tuesday 17 May 2022 12:38:43 Tom Rini wrote:
> > On Tue, May 17, 2022 at 06:00:16PM +0200, Pali Rohár wrote:
> > > On Tuesday 17 May 2022 11:52:14 Tom Rini wrote:
> > > > On Mon, May 16, 2022 at 11:56:51PM +0200, Pali Rohár wrote:
> > > > > On Monday 16 May 2022 08:31:43 Tom Rini wrote:
> > > > > > On Sat, May 14, 2022 at 01:00:06AM +0200, Pali Rohár wrote:
> > > > > > 
> > > > > > > Hello! I tried to enable support for 2GB+ of DDR memory (with 4GB DDR3)
> > > > > > > on powerpc P2020 board in 32-bit addressing mode and U-Boot crashed
> > > > > > > during startup.
> > > > > > > 
> > > > > > > I figured out that issue is not powerpc specific, but rather generic to
> > > > > > > all 32-bit platforms. U-Boot stores memory size into phys_size_t type
> > > > > > > (gd->ram_size) and last mapped memory address increased by one byte into
> > > > > > > phys_addr_t type (gd->ram_top).
> > > > > > > 
> > > > > > > Despite size 4GB fits into 32-bit addressing mode, it does not fit into
> > > > > > > above two variables, and it overflows to zero. U-Boot then see zero RAM
> > > > > > > size and crashes.
> > > > > > > 
> > > > > > > I tried to workaround this issue by changing both phys_size_t and
> > > > > > > phys_addr_t types to 64-bit. But it did not helped because U-Boot on
> > > > > > > many places cast gd->ram_size or gd->ram_top to ulong type, which is
> > > > > > > 32-bit on 32-bit platforms.
> > > > > > > 
> > > > > > > Next I changed ulong parameters of board_get_usable_ram_top() function
> > > > > > > to u64.
> > > > > > > 
> > > > > > > This still was not enough because config value CONFIG_MAX_MEM_MAPPED is
> > > > > > > ignored on one important place -- in function get_effective_memsize().
> > > > > > > This config value takes effect only when also CONFIG_VERY_BIG_RAM is
> > > > > > > set.
> > > > > > > 
> > > > > > > Finally With this change I was able to start U-Boot with more than 2GB
> > > > > > > of DDR memory inserted in SODIMM slot on P2020.
> > > > > > > 
> > > > > > > How to fix issues with gd->ram_size and gd->ram_top? That +1 byte is
> > > > > > > really stupid limitation. Changing phys_size_t and phys_addr_t types
> > > > > > > unconditionally to 64-bit? Or something else?
> > > > > > > 
> > > > > > > And what is the purpose of CONFIG_VERY_BIG_RAM config option? Why is
> > > > > > > CONFIG_MAX_MEM_MAPPED check skipped in get_effective_memsize() function,
> > > > > > > but is not skipped on many more places?
> > > > > > 
> > > > > > So, there's two parts to this, as I recall it all.  First, even on 64bit
> > > > > > platforms we contain ourselves to 32bit address space and even then
> > > > > > something within the "old" 2GB window.  We then set a CONFIG option to
> > > > > > not mess with the memory node in DT which has the real value.  Second,
> > > > > > for 32bit platforms which can support 4GB memory, or more, some further
> > > > > > games need to be played, typically I believe around initializing the
> > > > > > memory controller (I'm more confident of that for dra7xx_evm, which I
> > > > > > don't have the big memory version of, just a small memory one) so that
> > > > > > Linux can do whatever needs doing to enable "36bit" typically address
> > > > > > support.  Looking at the other P*36BIT* configs might give you some more
> > > > > > clues about what to do on your platform, or at least who might still be
> > > > > > able to explain and test things on the PowerPC side.
> > > > > 
> > > > > I know about 36-bit addressing on e500v2 but I'm not going to enable it
> > > > > due to performance reasons (see Freescale AN4064 [1]). So I want to
> > > > > stick with 32-bit addressing for 2GB+ memory usage (around 3GB; it is
> > > > > 4GB minus memory used by peripherals; which is still more than 2GB).
> > > > > 
> > > > > And due to 32-bit type for phys_size_t, phys_addr_t and casting these
> > > > > types to ulong is an issue. Plus issue with CONFIG_VERY_BIG_RAM and
> > > > > CONFIG_MAX_MEM_MAPPED as I already wrote.
> > > > 
> > > > I'm not seeing the problem, sorry.  You run U-Boot in the normal 2GB
> > > 
> > > Ok, I will try to explain it again.
> > > 
> > > I want to run U-Boot in normal 2GB area. U-Boot normally store detected
> > > RAM size into the gd->ram_size structure. And here is the issue.
> > > 
> > > 32-bit unsigned C type cannot represent maximal 32-bit addressable size.
> > > U-Boot tries to store 4GB value = 0x100000000 into gd->ram_size which
> > > overflows to 0x00000000. And U-Boot then crashes because it expects that
> > > can store some data at address "gd->ram_size - few_kb" as it expects
> > > that RAM is in the area [0, gd->ram_size).
> > > 
> > > It is more clear what is the problem?
> > > 
> > > If I theoretically find 3.9GB RAM module (but such probably does not
> > > exist) then U-Boot should work in normal 2GB area as number 3.9GB can be
> > > stored into 32-bit unsigned type.
> > 
> > Yes, you need to disable CONFIG_ARCH_FIXUP_FDT_MEMORY
> 
> CONFIG_ARCH_FIXUP_FDT_MEMORY affects booting OS.
> 
> The issue is that **u-boot** is crashing during its initialization.

When configuring the controller for 4GB, but limiting U-Boot to 2GB?

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: Broken support for 4GB DDR on 32-bit platforms

[Pali Rohár]

On Wednesday 18 May 2022 08:16:55 Tom Rini wrote:
> On Wed, May 18, 2022 at 12:58:38PM +0200, Pali Rohár wrote:
> > On Tuesday 17 May 2022 12:38:43 Tom Rini wrote:
> > > On Tue, May 17, 2022 at 06:00:16PM +0200, Pali Rohár wrote:
> > > > On Tuesday 17 May 2022 11:52:14 Tom Rini wrote:
> > > > > On Mon, May 16, 2022 at 11:56:51PM +0200, Pali Rohár wrote:
> > > > > > On Monday 16 May 2022 08:31:43 Tom Rini wrote:
> > > > > > > On Sat, May 14, 2022 at 01:00:06AM +0200, Pali Rohár wrote:
> > > > > > > 
> > > > > > > > Hello! I tried to enable support for 2GB+ of DDR memory (with 4GB DDR3)
> > > > > > > > on powerpc P2020 board in 32-bit addressing mode and U-Boot crashed
> > > > > > > > during startup.
> > > > > > > > 
> > > > > > > > I figured out that issue is not powerpc specific, but rather generic to
> > > > > > > > all 32-bit platforms. U-Boot stores memory size into phys_size_t type
> > > > > > > > (gd->ram_size) and last mapped memory address increased by one byte into
> > > > > > > > phys_addr_t type (gd->ram_top).
> > > > > > > > 
> > > > > > > > Despite size 4GB fits into 32-bit addressing mode, it does not fit into
> > > > > > > > above two variables, and it overflows to zero. U-Boot then see zero RAM
> > > > > > > > size and crashes.
> > > > > > > > 
> > > > > > > > I tried to workaround this issue by changing both phys_size_t and
> > > > > > > > phys_addr_t types to 64-bit. But it did not helped because U-Boot on
> > > > > > > > many places cast gd->ram_size or gd->ram_top to ulong type, which is
> > > > > > > > 32-bit on 32-bit platforms.
> > > > > > > > 
> > > > > > > > Next I changed ulong parameters of board_get_usable_ram_top() function
> > > > > > > > to u64.
> > > > > > > > 
> > > > > > > > This still was not enough because config value CONFIG_MAX_MEM_MAPPED is
> > > > > > > > ignored on one important place -- in function get_effective_memsize().
> > > > > > > > This config value takes effect only when also CONFIG_VERY_BIG_RAM is
> > > > > > > > set.
> > > > > > > > 
> > > > > > > > Finally With this change I was able to start U-Boot with more than 2GB
> > > > > > > > of DDR memory inserted in SODIMM slot on P2020.
> > > > > > > > 
> > > > > > > > How to fix issues with gd->ram_size and gd->ram_top? That +1 byte is
> > > > > > > > really stupid limitation. Changing phys_size_t and phys_addr_t types
> > > > > > > > unconditionally to 64-bit? Or something else?
> > > > > > > > 
> > > > > > > > And what is the purpose of CONFIG_VERY_BIG_RAM config option? Why is
> > > > > > > > CONFIG_MAX_MEM_MAPPED check skipped in get_effective_memsize() function,
> > > > > > > > but is not skipped on many more places?
> > > > > > > 
> > > > > > > So, there's two parts to this, as I recall it all.  First, even on 64bit
> > > > > > > platforms we contain ourselves to 32bit address space and even then
> > > > > > > something within the "old" 2GB window.  We then set a CONFIG option to
> > > > > > > not mess with the memory node in DT which has the real value.  Second,
> > > > > > > for 32bit platforms which can support 4GB memory, or more, some further
> > > > > > > games need to be played, typically I believe around initializing the
> > > > > > > memory controller (I'm more confident of that for dra7xx_evm, which I
> > > > > > > don't have the big memory version of, just a small memory one) so that
> > > > > > > Linux can do whatever needs doing to enable "36bit" typically address
> > > > > > > support.  Looking at the other P*36BIT* configs might give you some more
> > > > > > > clues about what to do on your platform, or at least who might still be
> > > > > > > able to explain and test things on the PowerPC side.
> > > > > > 
> > > > > > I know about 36-bit addressing on e500v2 but I'm not going to enable it
> > > > > > due to performance reasons (see Freescale AN4064 [1]). So I want to
> > > > > > stick with 32-bit addressing for 2GB+ memory usage (around 3GB; it is
> > > > > > 4GB minus memory used by peripherals; which is still more than 2GB).
> > > > > > 
> > > > > > And due to 32-bit type for phys_size_t, phys_addr_t and casting these
> > > > > > types to ulong is an issue. Plus issue with CONFIG_VERY_BIG_RAM and
> > > > > > CONFIG_MAX_MEM_MAPPED as I already wrote.
> > > > > 
> > > > > I'm not seeing the problem, sorry.  You run U-Boot in the normal 2GB
> > > > 
> > > > Ok, I will try to explain it again.
> > > > 
> > > > I want to run U-Boot in normal 2GB area. U-Boot normally store detected
> > > > RAM size into the gd->ram_size structure. And here is the issue.
> > > > 
> > > > 32-bit unsigned C type cannot represent maximal 32-bit addressable size.
> > > > U-Boot tries to store 4GB value = 0x100000000 into gd->ram_size which
> > > > overflows to 0x00000000. And U-Boot then crashes because it expects that
> > > > can store some data at address "gd->ram_size - few_kb" as it expects
> > > > that RAM is in the area [0, gd->ram_size).
> > > > 
> > > > It is more clear what is the problem?
> > > > 
> > > > If I theoretically find 3.9GB RAM module (but such probably does not
> > > > exist) then U-Boot should work in normal 2GB area as number 3.9GB can be
> > > > stored into 32-bit unsigned type.
> > > 
> > > Yes, you need to disable CONFIG_ARCH_FIXUP_FDT_MEMORY
> > 
> > CONFIG_ARCH_FIXUP_FDT_MEMORY affects booting OS.
> > 
> > The issue is that **u-boot** is crashing during its initialization.
> 
> When configuring the controller for 4GB, but limiting U-Boot to 2GB?

Yes!

Re: Broken support for 4GB DDR on 32-bit platforms

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 6011 bytes --]

On Wed, May 18, 2022 at 02:17:39PM +0200, Pali Rohár wrote:
> On Wednesday 18 May 2022 08:16:55 Tom Rini wrote:
> > On Wed, May 18, 2022 at 12:58:38PM +0200, Pali Rohár wrote:
> > > On Tuesday 17 May 2022 12:38:43 Tom Rini wrote:
> > > > On Tue, May 17, 2022 at 06:00:16PM +0200, Pali Rohár wrote:
> > > > > On Tuesday 17 May 2022 11:52:14 Tom Rini wrote:
> > > > > > On Mon, May 16, 2022 at 11:56:51PM +0200, Pali Rohár wrote:
> > > > > > > On Monday 16 May 2022 08:31:43 Tom Rini wrote:
> > > > > > > > On Sat, May 14, 2022 at 01:00:06AM +0200, Pali Rohár wrote:
> > > > > > > > 
> > > > > > > > > Hello! I tried to enable support for 2GB+ of DDR memory (with 4GB DDR3)
> > > > > > > > > on powerpc P2020 board in 32-bit addressing mode and U-Boot crashed
> > > > > > > > > during startup.
> > > > > > > > > 
> > > > > > > > > I figured out that issue is not powerpc specific, but rather generic to
> > > > > > > > > all 32-bit platforms. U-Boot stores memory size into phys_size_t type
> > > > > > > > > (gd->ram_size) and last mapped memory address increased by one byte into
> > > > > > > > > phys_addr_t type (gd->ram_top).
> > > > > > > > > 
> > > > > > > > > Despite size 4GB fits into 32-bit addressing mode, it does not fit into
> > > > > > > > > above two variables, and it overflows to zero. U-Boot then see zero RAM
> > > > > > > > > size and crashes.
> > > > > > > > > 
> > > > > > > > > I tried to workaround this issue by changing both phys_size_t and
> > > > > > > > > phys_addr_t types to 64-bit. But it did not helped because U-Boot on
> > > > > > > > > many places cast gd->ram_size or gd->ram_top to ulong type, which is
> > > > > > > > > 32-bit on 32-bit platforms.
> > > > > > > > > 
> > > > > > > > > Next I changed ulong parameters of board_get_usable_ram_top() function
> > > > > > > > > to u64.
> > > > > > > > > 
> > > > > > > > > This still was not enough because config value CONFIG_MAX_MEM_MAPPED is
> > > > > > > > > ignored on one important place -- in function get_effective_memsize().
> > > > > > > > > This config value takes effect only when also CONFIG_VERY_BIG_RAM is
> > > > > > > > > set.
> > > > > > > > > 
> > > > > > > > > Finally With this change I was able to start U-Boot with more than 2GB
> > > > > > > > > of DDR memory inserted in SODIMM slot on P2020.
> > > > > > > > > 
> > > > > > > > > How to fix issues with gd->ram_size and gd->ram_top? That +1 byte is
> > > > > > > > > really stupid limitation. Changing phys_size_t and phys_addr_t types
> > > > > > > > > unconditionally to 64-bit? Or something else?
> > > > > > > > > 
> > > > > > > > > And what is the purpose of CONFIG_VERY_BIG_RAM config option? Why is
> > > > > > > > > CONFIG_MAX_MEM_MAPPED check skipped in get_effective_memsize() function,
> > > > > > > > > but is not skipped on many more places?
> > > > > > > > 
> > > > > > > > So, there's two parts to this, as I recall it all.  First, even on 64bit
> > > > > > > > platforms we contain ourselves to 32bit address space and even then
> > > > > > > > something within the "old" 2GB window.  We then set a CONFIG option to
> > > > > > > > not mess with the memory node in DT which has the real value.  Second,
> > > > > > > > for 32bit platforms which can support 4GB memory, or more, some further
> > > > > > > > games need to be played, typically I believe around initializing the
> > > > > > > > memory controller (I'm more confident of that for dra7xx_evm, which I
> > > > > > > > don't have the big memory version of, just a small memory one) so that
> > > > > > > > Linux can do whatever needs doing to enable "36bit" typically address
> > > > > > > > support.  Looking at the other P*36BIT* configs might give you some more
> > > > > > > > clues about what to do on your platform, or at least who might still be
> > > > > > > > able to explain and test things on the PowerPC side.
> > > > > > > 
> > > > > > > I know about 36-bit addressing on e500v2 but I'm not going to enable it
> > > > > > > due to performance reasons (see Freescale AN4064 [1]). So I want to
> > > > > > > stick with 32-bit addressing for 2GB+ memory usage (around 3GB; it is
> > > > > > > 4GB minus memory used by peripherals; which is still more than 2GB).
> > > > > > > 
> > > > > > > And due to 32-bit type for phys_size_t, phys_addr_t and casting these
> > > > > > > types to ulong is an issue. Plus issue with CONFIG_VERY_BIG_RAM and
> > > > > > > CONFIG_MAX_MEM_MAPPED as I already wrote.
> > > > > > 
> > > > > > I'm not seeing the problem, sorry.  You run U-Boot in the normal 2GB
> > > > > 
> > > > > Ok, I will try to explain it again.
> > > > > 
> > > > > I want to run U-Boot in normal 2GB area. U-Boot normally store detected
> > > > > RAM size into the gd->ram_size structure. And here is the issue.
> > > > > 
> > > > > 32-bit unsigned C type cannot represent maximal 32-bit addressable size.
> > > > > U-Boot tries to store 4GB value = 0x100000000 into gd->ram_size which
> > > > > overflows to 0x00000000. And U-Boot then crashes because it expects that
> > > > > can store some data at address "gd->ram_size - few_kb" as it expects
> > > > > that RAM is in the area [0, gd->ram_size).
> > > > > 
> > > > > It is more clear what is the problem?
> > > > > 
> > > > > If I theoretically find 3.9GB RAM module (but such probably does not
> > > > > exist) then U-Boot should work in normal 2GB area as number 3.9GB can be
> > > > > stored into 32-bit unsigned type.
> > > > 
> > > > Yes, you need to disable CONFIG_ARCH_FIXUP_FDT_MEMORY
> > > 
> > > CONFIG_ARCH_FIXUP_FDT_MEMORY affects booting OS.
> > > 
> > > The issue is that **u-boot** is crashing during its initialization.
> > 
> > When configuring the controller for 4GB, but limiting U-Boot to 2GB?
> 
> Yes!

Interesting.  Maybe you just have to do the 36BIT games on that SoC, to
support what you're trying to do.  If anyone at NXP knows something
perhaps they'll chime in.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: Broken support for 4GB DDR on 32-bit platforms

[Pali Rohár]

On Wednesday 18 May 2022 08:19:36 Tom Rini wrote:
> On Wed, May 18, 2022 at 02:17:39PM +0200, Pali Rohár wrote:
> > On Wednesday 18 May 2022 08:16:55 Tom Rini wrote:
> > > On Wed, May 18, 2022 at 12:58:38PM +0200, Pali Rohár wrote:
> > > > On Tuesday 17 May 2022 12:38:43 Tom Rini wrote:
> > > > > On Tue, May 17, 2022 at 06:00:16PM +0200, Pali Rohár wrote:
> > > > > > On Tuesday 17 May 2022 11:52:14 Tom Rini wrote:
> > > > > > > On Mon, May 16, 2022 at 11:56:51PM +0200, Pali Rohár wrote:
> > > > > > > > On Monday 16 May 2022 08:31:43 Tom Rini wrote:
> > > > > > > > > On Sat, May 14, 2022 at 01:00:06AM +0200, Pali Rohár wrote:
> > > > > > > > > 
> > > > > > > > > > Hello! I tried to enable support for 2GB+ of DDR memory (with 4GB DDR3)
> > > > > > > > > > on powerpc P2020 board in 32-bit addressing mode and U-Boot crashed
> > > > > > > > > > during startup.
> > > > > > > > > > 
> > > > > > > > > > I figured out that issue is not powerpc specific, but rather generic to
> > > > > > > > > > all 32-bit platforms. U-Boot stores memory size into phys_size_t type
> > > > > > > > > > (gd->ram_size) and last mapped memory address increased by one byte into
> > > > > > > > > > phys_addr_t type (gd->ram_top).
> > > > > > > > > > 
> > > > > > > > > > Despite size 4GB fits into 32-bit addressing mode, it does not fit into
> > > > > > > > > > above two variables, and it overflows to zero. U-Boot then see zero RAM
> > > > > > > > > > size and crashes.
> > > > > > > > > > 
> > > > > > > > > > I tried to workaround this issue by changing both phys_size_t and
> > > > > > > > > > phys_addr_t types to 64-bit. But it did not helped because U-Boot on
> > > > > > > > > > many places cast gd->ram_size or gd->ram_top to ulong type, which is
> > > > > > > > > > 32-bit on 32-bit platforms.
> > > > > > > > > > 
> > > > > > > > > > Next I changed ulong parameters of board_get_usable_ram_top() function
> > > > > > > > > > to u64.
> > > > > > > > > > 
> > > > > > > > > > This still was not enough because config value CONFIG_MAX_MEM_MAPPED is
> > > > > > > > > > ignored on one important place -- in function get_effective_memsize().
> > > > > > > > > > This config value takes effect only when also CONFIG_VERY_BIG_RAM is
> > > > > > > > > > set.
> > > > > > > > > > 
> > > > > > > > > > Finally With this change I was able to start U-Boot with more than 2GB
> > > > > > > > > > of DDR memory inserted in SODIMM slot on P2020.
> > > > > > > > > > 
> > > > > > > > > > How to fix issues with gd->ram_size and gd->ram_top? That +1 byte is
> > > > > > > > > > really stupid limitation. Changing phys_size_t and phys_addr_t types
> > > > > > > > > > unconditionally to 64-bit? Or something else?
> > > > > > > > > > 
> > > > > > > > > > And what is the purpose of CONFIG_VERY_BIG_RAM config option? Why is
> > > > > > > > > > CONFIG_MAX_MEM_MAPPED check skipped in get_effective_memsize() function,
> > > > > > > > > > but is not skipped on many more places?
> > > > > > > > > 
> > > > > > > > > So, there's two parts to this, as I recall it all.  First, even on 64bit
> > > > > > > > > platforms we contain ourselves to 32bit address space and even then
> > > > > > > > > something within the "old" 2GB window.  We then set a CONFIG option to
> > > > > > > > > not mess with the memory node in DT which has the real value.  Second,
> > > > > > > > > for 32bit platforms which can support 4GB memory, or more, some further
> > > > > > > > > games need to be played, typically I believe around initializing the
> > > > > > > > > memory controller (I'm more confident of that for dra7xx_evm, which I
> > > > > > > > > don't have the big memory version of, just a small memory one) so that
> > > > > > > > > Linux can do whatever needs doing to enable "36bit" typically address
> > > > > > > > > support.  Looking at the other P*36BIT* configs might give you some more
> > > > > > > > > clues about what to do on your platform, or at least who might still be
> > > > > > > > > able to explain and test things on the PowerPC side.
> > > > > > > > 
> > > > > > > > I know about 36-bit addressing on e500v2 but I'm not going to enable it
> > > > > > > > due to performance reasons (see Freescale AN4064 [1]). So I want to
> > > > > > > > stick with 32-bit addressing for 2GB+ memory usage (around 3GB; it is
> > > > > > > > 4GB minus memory used by peripherals; which is still more than 2GB).
> > > > > > > > 
> > > > > > > > And due to 32-bit type for phys_size_t, phys_addr_t and casting these
> > > > > > > > types to ulong is an issue. Plus issue with CONFIG_VERY_BIG_RAM and
> > > > > > > > CONFIG_MAX_MEM_MAPPED as I already wrote.
> > > > > > > 
> > > > > > > I'm not seeing the problem, sorry.  You run U-Boot in the normal 2GB
> > > > > > 
> > > > > > Ok, I will try to explain it again.
> > > > > > 
> > > > > > I want to run U-Boot in normal 2GB area. U-Boot normally store detected
> > > > > > RAM size into the gd->ram_size structure. And here is the issue.
> > > > > > 
> > > > > > 32-bit unsigned C type cannot represent maximal 32-bit addressable size.
> > > > > > U-Boot tries to store 4GB value = 0x100000000 into gd->ram_size which
> > > > > > overflows to 0x00000000. And U-Boot then crashes because it expects that
> > > > > > can store some data at address "gd->ram_size - few_kb" as it expects
> > > > > > that RAM is in the area [0, gd->ram_size).
> > > > > > 
> > > > > > It is more clear what is the problem?
> > > > > > 
> > > > > > If I theoretically find 3.9GB RAM module (but such probably does not
> > > > > > exist) then U-Boot should work in normal 2GB area as number 3.9GB can be
> > > > > > stored into 32-bit unsigned type.
> > > > > 
> > > > > Yes, you need to disable CONFIG_ARCH_FIXUP_FDT_MEMORY
> > > > 
> > > > CONFIG_ARCH_FIXUP_FDT_MEMORY affects booting OS.
> > > > 
> > > > The issue is that **u-boot** is crashing during its initialization.
> > > 
> > > When configuring the controller for 4GB, but limiting U-Boot to 2GB?
> > 
> > Yes!
> 
> Interesting.  Maybe you just have to do the 36BIT games on that SoC, to
> support what you're trying to do.  If anyone at NXP knows something
> perhaps they'll chime in.

Why? It is issue in U-Boot and affects all 32-bit platforms not only powerpc/NXP.

Re: Broken support for 4GB DDR on 32-bit platforms

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 7109 bytes --]

On Wed, May 18, 2022 at 03:19:19PM +0200, Pali Rohár wrote:
> On Wednesday 18 May 2022 08:19:36 Tom Rini wrote:
> > On Wed, May 18, 2022 at 02:17:39PM +0200, Pali Rohár wrote:
> > > On Wednesday 18 May 2022 08:16:55 Tom Rini wrote:
> > > > On Wed, May 18, 2022 at 12:58:38PM +0200, Pali Rohár wrote:
> > > > > On Tuesday 17 May 2022 12:38:43 Tom Rini wrote:
> > > > > > On Tue, May 17, 2022 at 06:00:16PM +0200, Pali Rohár wrote:
> > > > > > > On Tuesday 17 May 2022 11:52:14 Tom Rini wrote:
> > > > > > > > On Mon, May 16, 2022 at 11:56:51PM +0200, Pali Rohár wrote:
> > > > > > > > > On Monday 16 May 2022 08:31:43 Tom Rini wrote:
> > > > > > > > > > On Sat, May 14, 2022 at 01:00:06AM +0200, Pali Rohár wrote:
> > > > > > > > > > 
> > > > > > > > > > > Hello! I tried to enable support for 2GB+ of DDR memory (with 4GB DDR3)
> > > > > > > > > > > on powerpc P2020 board in 32-bit addressing mode and U-Boot crashed
> > > > > > > > > > > during startup.
> > > > > > > > > > > 
> > > > > > > > > > > I figured out that issue is not powerpc specific, but rather generic to
> > > > > > > > > > > all 32-bit platforms. U-Boot stores memory size into phys_size_t type
> > > > > > > > > > > (gd->ram_size) and last mapped memory address increased by one byte into
> > > > > > > > > > > phys_addr_t type (gd->ram_top).
> > > > > > > > > > > 
> > > > > > > > > > > Despite size 4GB fits into 32-bit addressing mode, it does not fit into
> > > > > > > > > > > above two variables, and it overflows to zero. U-Boot then see zero RAM
> > > > > > > > > > > size and crashes.
> > > > > > > > > > > 
> > > > > > > > > > > I tried to workaround this issue by changing both phys_size_t and
> > > > > > > > > > > phys_addr_t types to 64-bit. But it did not helped because U-Boot on
> > > > > > > > > > > many places cast gd->ram_size or gd->ram_top to ulong type, which is
> > > > > > > > > > > 32-bit on 32-bit platforms.
> > > > > > > > > > > 
> > > > > > > > > > > Next I changed ulong parameters of board_get_usable_ram_top() function
> > > > > > > > > > > to u64.
> > > > > > > > > > > 
> > > > > > > > > > > This still was not enough because config value CONFIG_MAX_MEM_MAPPED is
> > > > > > > > > > > ignored on one important place -- in function get_effective_memsize().
> > > > > > > > > > > This config value takes effect only when also CONFIG_VERY_BIG_RAM is
> > > > > > > > > > > set.
> > > > > > > > > > > 
> > > > > > > > > > > Finally With this change I was able to start U-Boot with more than 2GB
> > > > > > > > > > > of DDR memory inserted in SODIMM slot on P2020.
> > > > > > > > > > > 
> > > > > > > > > > > How to fix issues with gd->ram_size and gd->ram_top? That +1 byte is
> > > > > > > > > > > really stupid limitation. Changing phys_size_t and phys_addr_t types
> > > > > > > > > > > unconditionally to 64-bit? Or something else?
> > > > > > > > > > > 
> > > > > > > > > > > And what is the purpose of CONFIG_VERY_BIG_RAM config option? Why is
> > > > > > > > > > > CONFIG_MAX_MEM_MAPPED check skipped in get_effective_memsize() function,
> > > > > > > > > > > but is not skipped on many more places?
> > > > > > > > > > 
> > > > > > > > > > So, there's two parts to this, as I recall it all.  First, even on 64bit
> > > > > > > > > > platforms we contain ourselves to 32bit address space and even then
> > > > > > > > > > something within the "old" 2GB window.  We then set a CONFIG option to
> > > > > > > > > > not mess with the memory node in DT which has the real value.  Second,
> > > > > > > > > > for 32bit platforms which can support 4GB memory, or more, some further
> > > > > > > > > > games need to be played, typically I believe around initializing the
> > > > > > > > > > memory controller (I'm more confident of that for dra7xx_evm, which I
> > > > > > > > > > don't have the big memory version of, just a small memory one) so that
> > > > > > > > > > Linux can do whatever needs doing to enable "36bit" typically address
> > > > > > > > > > support.  Looking at the other P*36BIT* configs might give you some more
> > > > > > > > > > clues about what to do on your platform, or at least who might still be
> > > > > > > > > > able to explain and test things on the PowerPC side.
> > > > > > > > > 
> > > > > > > > > I know about 36-bit addressing on e500v2 but I'm not going to enable it
> > > > > > > > > due to performance reasons (see Freescale AN4064 [1]). So I want to
> > > > > > > > > stick with 32-bit addressing for 2GB+ memory usage (around 3GB; it is
> > > > > > > > > 4GB minus memory used by peripherals; which is still more than 2GB).
> > > > > > > > > 
> > > > > > > > > And due to 32-bit type for phys_size_t, phys_addr_t and casting these
> > > > > > > > > types to ulong is an issue. Plus issue with CONFIG_VERY_BIG_RAM and
> > > > > > > > > CONFIG_MAX_MEM_MAPPED as I already wrote.
> > > > > > > > 
> > > > > > > > I'm not seeing the problem, sorry.  You run U-Boot in the normal 2GB
> > > > > > > 
> > > > > > > Ok, I will try to explain it again.
> > > > > > > 
> > > > > > > I want to run U-Boot in normal 2GB area. U-Boot normally store detected
> > > > > > > RAM size into the gd->ram_size structure. And here is the issue.
> > > > > > > 
> > > > > > > 32-bit unsigned C type cannot represent maximal 32-bit addressable size.
> > > > > > > U-Boot tries to store 4GB value = 0x100000000 into gd->ram_size which
> > > > > > > overflows to 0x00000000. And U-Boot then crashes because it expects that
> > > > > > > can store some data at address "gd->ram_size - few_kb" as it expects
> > > > > > > that RAM is in the area [0, gd->ram_size).
> > > > > > > 
> > > > > > > It is more clear what is the problem?
> > > > > > > 
> > > > > > > If I theoretically find 3.9GB RAM module (but such probably does not
> > > > > > > exist) then U-Boot should work in normal 2GB area as number 3.9GB can be
> > > > > > > stored into 32-bit unsigned type.
> > > > > > 
> > > > > > Yes, you need to disable CONFIG_ARCH_FIXUP_FDT_MEMORY
> > > > > 
> > > > > CONFIG_ARCH_FIXUP_FDT_MEMORY affects booting OS.
> > > > > 
> > > > > The issue is that **u-boot** is crashing during its initialization.
> > > > 
> > > > When configuring the controller for 4GB, but limiting U-Boot to 2GB?
> > > 
> > > Yes!
> > 
> > Interesting.  Maybe you just have to do the 36BIT games on that SoC, to
> > support what you're trying to do.  If anyone at NXP knows something
> > perhaps they'll chime in.
> 
> Why? It is issue in U-Boot and affects all 32-bit platforms not only powerpc/NXP.

I disagree.  It's working as intended with other 32bit platforms, unless
it's stopped working on dra7xx_evm with >2GB memory (which I don't have,
unfortunately).  If you can't configure the memory controller to know
that 4GB exists, but limit yourself to running within 2GB, that's a
SoC-specific problem.  If you want to make the big changes so that we
can have a larger than 2GB memory space for U-Boot, period, that's a big
general change with challenges as you've already seen.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: Broken support for 4GB DDR on 32-bit platforms

[Pali Rohár]

On Wednesday 18 May 2022 11:18:18 Tom Rini wrote:
> On Wed, May 18, 2022 at 03:19:19PM +0200, Pali Rohár wrote:
> > On Wednesday 18 May 2022 08:19:36 Tom Rini wrote:
> > > On Wed, May 18, 2022 at 02:17:39PM +0200, Pali Rohár wrote:
> > > > On Wednesday 18 May 2022 08:16:55 Tom Rini wrote:
> > > > > On Wed, May 18, 2022 at 12:58:38PM +0200, Pali Rohár wrote:
> > > > > > On Tuesday 17 May 2022 12:38:43 Tom Rini wrote:
> > > > > > > On Tue, May 17, 2022 at 06:00:16PM +0200, Pali Rohár wrote:
> > > > > > > > On Tuesday 17 May 2022 11:52:14 Tom Rini wrote:
> > > > > > > > > On Mon, May 16, 2022 at 11:56:51PM +0200, Pali Rohár wrote:
> > > > > > > > > > On Monday 16 May 2022 08:31:43 Tom Rini wrote:
> > > > > > > > > > > On Sat, May 14, 2022 at 01:00:06AM +0200, Pali Rohár wrote:
> > > > > > > > > > > 
> > > > > > > > > > > > Hello! I tried to enable support for 2GB+ of DDR memory (with 4GB DDR3)
> > > > > > > > > > > > on powerpc P2020 board in 32-bit addressing mode and U-Boot crashed
> > > > > > > > > > > > during startup.
> > > > > > > > > > > > 
> > > > > > > > > > > > I figured out that issue is not powerpc specific, but rather generic to
> > > > > > > > > > > > all 32-bit platforms. U-Boot stores memory size into phys_size_t type
> > > > > > > > > > > > (gd->ram_size) and last mapped memory address increased by one byte into
> > > > > > > > > > > > phys_addr_t type (gd->ram_top).
> > > > > > > > > > > > 
> > > > > > > > > > > > Despite size 4GB fits into 32-bit addressing mode, it does not fit into
> > > > > > > > > > > > above two variables, and it overflows to zero. U-Boot then see zero RAM
> > > > > > > > > > > > size and crashes.
> > > > > > > > > > > > 
> > > > > > > > > > > > I tried to workaround this issue by changing both phys_size_t and
> > > > > > > > > > > > phys_addr_t types to 64-bit. But it did not helped because U-Boot on
> > > > > > > > > > > > many places cast gd->ram_size or gd->ram_top to ulong type, which is
> > > > > > > > > > > > 32-bit on 32-bit platforms.
> > > > > > > > > > > > 
> > > > > > > > > > > > Next I changed ulong parameters of board_get_usable_ram_top() function
> > > > > > > > > > > > to u64.
> > > > > > > > > > > > 
> > > > > > > > > > > > This still was not enough because config value CONFIG_MAX_MEM_MAPPED is
> > > > > > > > > > > > ignored on one important place -- in function get_effective_memsize().
> > > > > > > > > > > > This config value takes effect only when also CONFIG_VERY_BIG_RAM is
> > > > > > > > > > > > set.
> > > > > > > > > > > > 
> > > > > > > > > > > > Finally With this change I was able to start U-Boot with more than 2GB
> > > > > > > > > > > > of DDR memory inserted in SODIMM slot on P2020.
> > > > > > > > > > > > 
> > > > > > > > > > > > How to fix issues with gd->ram_size and gd->ram_top? That +1 byte is
> > > > > > > > > > > > really stupid limitation. Changing phys_size_t and phys_addr_t types
> > > > > > > > > > > > unconditionally to 64-bit? Or something else?
> > > > > > > > > > > > 
> > > > > > > > > > > > And what is the purpose of CONFIG_VERY_BIG_RAM config option? Why is
> > > > > > > > > > > > CONFIG_MAX_MEM_MAPPED check skipped in get_effective_memsize() function,
> > > > > > > > > > > > but is not skipped on many more places?
> > > > > > > > > > > 
> > > > > > > > > > > So, there's two parts to this, as I recall it all.  First, even on 64bit
> > > > > > > > > > > platforms we contain ourselves to 32bit address space and even then
> > > > > > > > > > > something within the "old" 2GB window.  We then set a CONFIG option to
> > > > > > > > > > > not mess with the memory node in DT which has the real value.  Second,
> > > > > > > > > > > for 32bit platforms which can support 4GB memory, or more, some further
> > > > > > > > > > > games need to be played, typically I believe around initializing the
> > > > > > > > > > > memory controller (I'm more confident of that for dra7xx_evm, which I
> > > > > > > > > > > don't have the big memory version of, just a small memory one) so that
> > > > > > > > > > > Linux can do whatever needs doing to enable "36bit" typically address
> > > > > > > > > > > support.  Looking at the other P*36BIT* configs might give you some more
> > > > > > > > > > > clues about what to do on your platform, or at least who might still be
> > > > > > > > > > > able to explain and test things on the PowerPC side.
> > > > > > > > > > 
> > > > > > > > > > I know about 36-bit addressing on e500v2 but I'm not going to enable it
> > > > > > > > > > due to performance reasons (see Freescale AN4064 [1]). So I want to
> > > > > > > > > > stick with 32-bit addressing for 2GB+ memory usage (around 3GB; it is
> > > > > > > > > > 4GB minus memory used by peripherals; which is still more than 2GB).
> > > > > > > > > > 
> > > > > > > > > > And due to 32-bit type for phys_size_t, phys_addr_t and casting these
> > > > > > > > > > types to ulong is an issue. Plus issue with CONFIG_VERY_BIG_RAM and
> > > > > > > > > > CONFIG_MAX_MEM_MAPPED as I already wrote.
> > > > > > > > > 
> > > > > > > > > I'm not seeing the problem, sorry.  You run U-Boot in the normal 2GB
> > > > > > > > 
> > > > > > > > Ok, I will try to explain it again.
> > > > > > > > 
> > > > > > > > I want to run U-Boot in normal 2GB area. U-Boot normally store detected
> > > > > > > > RAM size into the gd->ram_size structure. And here is the issue.
> > > > > > > > 
> > > > > > > > 32-bit unsigned C type cannot represent maximal 32-bit addressable size.
> > > > > > > > U-Boot tries to store 4GB value = 0x100000000 into gd->ram_size which
> > > > > > > > overflows to 0x00000000. And U-Boot then crashes because it expects that
> > > > > > > > can store some data at address "gd->ram_size - few_kb" as it expects
> > > > > > > > that RAM is in the area [0, gd->ram_size).
> > > > > > > > 
> > > > > > > > It is more clear what is the problem?
> > > > > > > > 
> > > > > > > > If I theoretically find 3.9GB RAM module (but such probably does not
> > > > > > > > exist) then U-Boot should work in normal 2GB area as number 3.9GB can be
> > > > > > > > stored into 32-bit unsigned type.
> > > > > > > 
> > > > > > > Yes, you need to disable CONFIG_ARCH_FIXUP_FDT_MEMORY
> > > > > > 
> > > > > > CONFIG_ARCH_FIXUP_FDT_MEMORY affects booting OS.
> > > > > > 
> > > > > > The issue is that **u-boot** is crashing during its initialization.
> > > > > 
> > > > > When configuring the controller for 4GB, but limiting U-Boot to 2GB?
> > > > 
> > > > Yes!
> > > 
> > > Interesting.  Maybe you just have to do the 36BIT games on that SoC, to
> > > support what you're trying to do.  If anyone at NXP knows something
> > > perhaps they'll chime in.
> > 
> > Why? It is issue in U-Boot and affects all 32-bit platforms not only powerpc/NXP.
> 
> I disagree.  It's working as intended with other 32bit platforms, unless

It really cannot work. See my previous emails with exact description. It
is mathematical limitation that you cannot store 4GB size of RAM into
U-Boot's gd structure. Maximal number which can be stored into ulong
type on 32-bit platforms is 2^32-1 which is less than 4GB.

So if U-Boot detects 4GB DDR module it crashes prior it tries to "limit"
usage just to e.g. 2GB.

> it's stopped working on dra7xx_evm with >2GB memory (which I don't have,
> unfortunately).  If you can't configure the memory controller to know
> that 4GB exists, but limit yourself to running within 2GB, that's a
> SoC-specific problem.

Have you really read what I wrote? It is not SoC-specific problem. Full
size of RAM (not mapped memory by U-Boot which may be less than full
size of RAM) is stored in gd->ram_size variable. And this variable is
cast to ulong type which cannot store 4GB. This is common U-Boot code,
not something SoC specific.

Plus CONFIG_MAX_MEM_MAPPED is ignored in some cases which I also
described in previous email. And code for ignoring is again **not** SoC
specific, but generic in common U-Boot code.

> If you want to make the big changes so that we
> can have a larger than 2GB memory space for U-Boot, period, that's a big
> general change with challenges as you've already seen.
> 
> -- 
> Tom

Re: Broken support for 4GB DDR on 32-bit platforms

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 9676 bytes --]

On Wed, May 18, 2022 at 05:27:09PM +0200, Pali Rohár wrote:
> On Wednesday 18 May 2022 11:18:18 Tom Rini wrote:
> > On Wed, May 18, 2022 at 03:19:19PM +0200, Pali Rohár wrote:
> > > On Wednesday 18 May 2022 08:19:36 Tom Rini wrote:
> > > > On Wed, May 18, 2022 at 02:17:39PM +0200, Pali Rohár wrote:
> > > > > On Wednesday 18 May 2022 08:16:55 Tom Rini wrote:
> > > > > > On Wed, May 18, 2022 at 12:58:38PM +0200, Pali Rohár wrote:
> > > > > > > On Tuesday 17 May 2022 12:38:43 Tom Rini wrote:
> > > > > > > > On Tue, May 17, 2022 at 06:00:16PM +0200, Pali Rohár wrote:
> > > > > > > > > On Tuesday 17 May 2022 11:52:14 Tom Rini wrote:
> > > > > > > > > > On Mon, May 16, 2022 at 11:56:51PM +0200, Pali Rohár wrote:
> > > > > > > > > > > On Monday 16 May 2022 08:31:43 Tom Rini wrote:
> > > > > > > > > > > > On Sat, May 14, 2022 at 01:00:06AM +0200, Pali Rohár wrote:
> > > > > > > > > > > > 
> > > > > > > > > > > > > Hello! I tried to enable support for 2GB+ of DDR memory (with 4GB DDR3)
> > > > > > > > > > > > > on powerpc P2020 board in 32-bit addressing mode and U-Boot crashed
> > > > > > > > > > > > > during startup.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > I figured out that issue is not powerpc specific, but rather generic to
> > > > > > > > > > > > > all 32-bit platforms. U-Boot stores memory size into phys_size_t type
> > > > > > > > > > > > > (gd->ram_size) and last mapped memory address increased by one byte into
> > > > > > > > > > > > > phys_addr_t type (gd->ram_top).
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Despite size 4GB fits into 32-bit addressing mode, it does not fit into
> > > > > > > > > > > > > above two variables, and it overflows to zero. U-Boot then see zero RAM
> > > > > > > > > > > > > size and crashes.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > I tried to workaround this issue by changing both phys_size_t and
> > > > > > > > > > > > > phys_addr_t types to 64-bit. But it did not helped because U-Boot on
> > > > > > > > > > > > > many places cast gd->ram_size or gd->ram_top to ulong type, which is
> > > > > > > > > > > > > 32-bit on 32-bit platforms.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Next I changed ulong parameters of board_get_usable_ram_top() function
> > > > > > > > > > > > > to u64.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > This still was not enough because config value CONFIG_MAX_MEM_MAPPED is
> > > > > > > > > > > > > ignored on one important place -- in function get_effective_memsize().
> > > > > > > > > > > > > This config value takes effect only when also CONFIG_VERY_BIG_RAM is
> > > > > > > > > > > > > set.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Finally With this change I was able to start U-Boot with more than 2GB
> > > > > > > > > > > > > of DDR memory inserted in SODIMM slot on P2020.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > How to fix issues with gd->ram_size and gd->ram_top? That +1 byte is
> > > > > > > > > > > > > really stupid limitation. Changing phys_size_t and phys_addr_t types
> > > > > > > > > > > > > unconditionally to 64-bit? Or something else?
> > > > > > > > > > > > > 
> > > > > > > > > > > > > And what is the purpose of CONFIG_VERY_BIG_RAM config option? Why is
> > > > > > > > > > > > > CONFIG_MAX_MEM_MAPPED check skipped in get_effective_memsize() function,
> > > > > > > > > > > > > but is not skipped on many more places?
> > > > > > > > > > > > 
> > > > > > > > > > > > So, there's two parts to this, as I recall it all.  First, even on 64bit
> > > > > > > > > > > > platforms we contain ourselves to 32bit address space and even then
> > > > > > > > > > > > something within the "old" 2GB window.  We then set a CONFIG option to
> > > > > > > > > > > > not mess with the memory node in DT which has the real value.  Second,
> > > > > > > > > > > > for 32bit platforms which can support 4GB memory, or more, some further
> > > > > > > > > > > > games need to be played, typically I believe around initializing the
> > > > > > > > > > > > memory controller (I'm more confident of that for dra7xx_evm, which I
> > > > > > > > > > > > don't have the big memory version of, just a small memory one) so that
> > > > > > > > > > > > Linux can do whatever needs doing to enable "36bit" typically address
> > > > > > > > > > > > support.  Looking at the other P*36BIT* configs might give you some more
> > > > > > > > > > > > clues about what to do on your platform, or at least who might still be
> > > > > > > > > > > > able to explain and test things on the PowerPC side.
> > > > > > > > > > > 
> > > > > > > > > > > I know about 36-bit addressing on e500v2 but I'm not going to enable it
> > > > > > > > > > > due to performance reasons (see Freescale AN4064 [1]). So I want to
> > > > > > > > > > > stick with 32-bit addressing for 2GB+ memory usage (around 3GB; it is
> > > > > > > > > > > 4GB minus memory used by peripherals; which is still more than 2GB).
> > > > > > > > > > > 
> > > > > > > > > > > And due to 32-bit type for phys_size_t, phys_addr_t and casting these
> > > > > > > > > > > types to ulong is an issue. Plus issue with CONFIG_VERY_BIG_RAM and
> > > > > > > > > > > CONFIG_MAX_MEM_MAPPED as I already wrote.
> > > > > > > > > > 
> > > > > > > > > > I'm not seeing the problem, sorry.  You run U-Boot in the normal 2GB
> > > > > > > > > 
> > > > > > > > > Ok, I will try to explain it again.
> > > > > > > > > 
> > > > > > > > > I want to run U-Boot in normal 2GB area. U-Boot normally store detected
> > > > > > > > > RAM size into the gd->ram_size structure. And here is the issue.
> > > > > > > > > 
> > > > > > > > > 32-bit unsigned C type cannot represent maximal 32-bit addressable size.
> > > > > > > > > U-Boot tries to store 4GB value = 0x100000000 into gd->ram_size which
> > > > > > > > > overflows to 0x00000000. And U-Boot then crashes because it expects that
> > > > > > > > > can store some data at address "gd->ram_size - few_kb" as it expects
> > > > > > > > > that RAM is in the area [0, gd->ram_size).
> > > > > > > > > 
> > > > > > > > > It is more clear what is the problem?
> > > > > > > > > 
> > > > > > > > > If I theoretically find 3.9GB RAM module (but such probably does not
> > > > > > > > > exist) then U-Boot should work in normal 2GB area as number 3.9GB can be
> > > > > > > > > stored into 32-bit unsigned type.
> > > > > > > > 
> > > > > > > > Yes, you need to disable CONFIG_ARCH_FIXUP_FDT_MEMORY
> > > > > > > 
> > > > > > > CONFIG_ARCH_FIXUP_FDT_MEMORY affects booting OS.
> > > > > > > 
> > > > > > > The issue is that **u-boot** is crashing during its initialization.
> > > > > > 
> > > > > > When configuring the controller for 4GB, but limiting U-Boot to 2GB?
> > > > > 
> > > > > Yes!
> > > > 
> > > > Interesting.  Maybe you just have to do the 36BIT games on that SoC, to
> > > > support what you're trying to do.  If anyone at NXP knows something
> > > > perhaps they'll chime in.
> > > 
> > > Why? It is issue in U-Boot and affects all 32-bit platforms not only powerpc/NXP.
> > 
> > I disagree.  It's working as intended with other 32bit platforms, unless
> 
> It really cannot work. See my previous emails with exact description. It
> is mathematical limitation that you cannot store 4GB size of RAM into
> U-Boot's gd structure. Maximal number which can be stored into ulong
> type on 32-bit platforms is 2^32-1 which is less than 4GB.
> 
> So if U-Boot detects 4GB DDR module it crashes prior it tries to "limit"
> usage just to e.g. 2GB.

Yes, this is a semi-intentional limitation of our design history.  We're
still in the very old days of a 2GB:2GB address space split.

> > it's stopped working on dra7xx_evm with >2GB memory (which I don't have,
> > unfortunately).  If you can't configure the memory controller to know
> > that 4GB exists, but limit yourself to running within 2GB, that's a
> > SoC-specific problem.
> 
> Have you really read what I wrote? It is not SoC-specific problem. Full
> size of RAM (not mapped memory by U-Boot which may be less than full
> size of RAM) is stored in gd->ram_size variable. And this variable is
> cast to ulong type which cannot store 4GB. This is common U-Boot code,
> not something SoC specific.
> 
> Plus CONFIG_MAX_MEM_MAPPED is ignored in some cases which I also
> described in previous email. And code for ignoring is again **not** SoC
> specific, but generic in common U-Boot code.

I guess no, I'm not seeing everything you've wrote, and you're not
seeing everything I wrote either.  If you have a 32bit platform and more
than 2GB of memory, you don't tell U-Boot that you have more than 2GB of
memory because it's not going to work right.  You lie, one way or
another, and have U-Boot work inside a 2GB window, and do whatever you
need so that the OS will see the correct amount and do the right thing.
All of the options you've noted I believe are related to the kludges
that have been required thus far to either enable 36bit support on some
PowerPC platforms (which you've explained you don't want to do) or the
36bit support for some ARM platforms (of which I'm asking around to see
if someone can re-test on current mainline).  I don't _think_ we have an
option today that would mirror the 1GB:3GB (peripheral:memory) split
that was an option at some point for Linux, as how to start handling
bigmem systems.

What I keep reading what you want to do is to support 4GB of memory
without doing some kludges.  I have no idea how much work would be
required to enable that, at this point.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: Broken support for 4GB DDR on 32-bit platforms

[Pali Rohár]

On Wednesday 18 May 2022 11:35:18 Tom Rini wrote:
> On Wed, May 18, 2022 at 05:27:09PM +0200, Pali Rohár wrote:
> > On Wednesday 18 May 2022 11:18:18 Tom Rini wrote:
> > > On Wed, May 18, 2022 at 03:19:19PM +0200, Pali Rohár wrote:
> > > > On Wednesday 18 May 2022 08:19:36 Tom Rini wrote:
> > > > > On Wed, May 18, 2022 at 02:17:39PM +0200, Pali Rohár wrote:
> > > > > > On Wednesday 18 May 2022 08:16:55 Tom Rini wrote:
> > > > > > > On Wed, May 18, 2022 at 12:58:38PM +0200, Pali Rohár wrote:
> > > > > > > > On Tuesday 17 May 2022 12:38:43 Tom Rini wrote:
> > > > > > > > > On Tue, May 17, 2022 at 06:00:16PM +0200, Pali Rohár wrote:
> > > > > > > > > > On Tuesday 17 May 2022 11:52:14 Tom Rini wrote:
> > > > > > > > > > > On Mon, May 16, 2022 at 11:56:51PM +0200, Pali Rohár wrote:
> > > > > > > > > > > > On Monday 16 May 2022 08:31:43 Tom Rini wrote:
> > > > > > > > > > > > > On Sat, May 14, 2022 at 01:00:06AM +0200, Pali Rohár wrote:
> > > > > > > > > > > > > 
> > > > > > > > > > > > > > Hello! I tried to enable support for 2GB+ of DDR memory (with 4GB DDR3)
> > > > > > > > > > > > > > on powerpc P2020 board in 32-bit addressing mode and U-Boot crashed
> > > > > > > > > > > > > > during startup.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > I figured out that issue is not powerpc specific, but rather generic to
> > > > > > > > > > > > > > all 32-bit platforms. U-Boot stores memory size into phys_size_t type
> > > > > > > > > > > > > > (gd->ram_size) and last mapped memory address increased by one byte into
> > > > > > > > > > > > > > phys_addr_t type (gd->ram_top).
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > Despite size 4GB fits into 32-bit addressing mode, it does not fit into
> > > > > > > > > > > > > > above two variables, and it overflows to zero. U-Boot then see zero RAM
> > > > > > > > > > > > > > size and crashes.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > I tried to workaround this issue by changing both phys_size_t and
> > > > > > > > > > > > > > phys_addr_t types to 64-bit. But it did not helped because U-Boot on
> > > > > > > > > > > > > > many places cast gd->ram_size or gd->ram_top to ulong type, which is
> > > > > > > > > > > > > > 32-bit on 32-bit platforms.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > Next I changed ulong parameters of board_get_usable_ram_top() function
> > > > > > > > > > > > > > to u64.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > This still was not enough because config value CONFIG_MAX_MEM_MAPPED is
> > > > > > > > > > > > > > ignored on one important place -- in function get_effective_memsize().
> > > > > > > > > > > > > > This config value takes effect only when also CONFIG_VERY_BIG_RAM is
> > > > > > > > > > > > > > set.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > Finally With this change I was able to start U-Boot with more than 2GB
> > > > > > > > > > > > > > of DDR memory inserted in SODIMM slot on P2020.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > How to fix issues with gd->ram_size and gd->ram_top? That +1 byte is
> > > > > > > > > > > > > > really stupid limitation. Changing phys_size_t and phys_addr_t types
> > > > > > > > > > > > > > unconditionally to 64-bit? Or something else?
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > And what is the purpose of CONFIG_VERY_BIG_RAM config option? Why is
> > > > > > > > > > > > > > CONFIG_MAX_MEM_MAPPED check skipped in get_effective_memsize() function,
> > > > > > > > > > > > > > but is not skipped on many more places?
> > > > > > > > > > > > > 
> > > > > > > > > > > > > So, there's two parts to this, as I recall it all.  First, even on 64bit
> > > > > > > > > > > > > platforms we contain ourselves to 32bit address space and even then
> > > > > > > > > > > > > something within the "old" 2GB window.  We then set a CONFIG option to
> > > > > > > > > > > > > not mess with the memory node in DT which has the real value.  Second,
> > > > > > > > > > > > > for 32bit platforms which can support 4GB memory, or more, some further
> > > > > > > > > > > > > games need to be played, typically I believe around initializing the
> > > > > > > > > > > > > memory controller (I'm more confident of that for dra7xx_evm, which I
> > > > > > > > > > > > > don't have the big memory version of, just a small memory one) so that
> > > > > > > > > > > > > Linux can do whatever needs doing to enable "36bit" typically address
> > > > > > > > > > > > > support.  Looking at the other P*36BIT* configs might give you some more
> > > > > > > > > > > > > clues about what to do on your platform, or at least who might still be
> > > > > > > > > > > > > able to explain and test things on the PowerPC side.
> > > > > > > > > > > > 
> > > > > > > > > > > > I know about 36-bit addressing on e500v2 but I'm not going to enable it
> > > > > > > > > > > > due to performance reasons (see Freescale AN4064 [1]). So I want to
> > > > > > > > > > > > stick with 32-bit addressing for 2GB+ memory usage (around 3GB; it is
> > > > > > > > > > > > 4GB minus memory used by peripherals; which is still more than 2GB).
> > > > > > > > > > > > 
> > > > > > > > > > > > And due to 32-bit type for phys_size_t, phys_addr_t and casting these
> > > > > > > > > > > > types to ulong is an issue. Plus issue with CONFIG_VERY_BIG_RAM and
> > > > > > > > > > > > CONFIG_MAX_MEM_MAPPED as I already wrote.
> > > > > > > > > > > 
> > > > > > > > > > > I'm not seeing the problem, sorry.  You run U-Boot in the normal 2GB
> > > > > > > > > > 
> > > > > > > > > > Ok, I will try to explain it again.
> > > > > > > > > > 
> > > > > > > > > > I want to run U-Boot in normal 2GB area. U-Boot normally store detected
> > > > > > > > > > RAM size into the gd->ram_size structure. And here is the issue.
> > > > > > > > > > 
> > > > > > > > > > 32-bit unsigned C type cannot represent maximal 32-bit addressable size.
> > > > > > > > > > U-Boot tries to store 4GB value = 0x100000000 into gd->ram_size which
> > > > > > > > > > overflows to 0x00000000. And U-Boot then crashes because it expects that
> > > > > > > > > > can store some data at address "gd->ram_size - few_kb" as it expects
> > > > > > > > > > that RAM is in the area [0, gd->ram_size).
> > > > > > > > > > 
> > > > > > > > > > It is more clear what is the problem?
> > > > > > > > > > 
> > > > > > > > > > If I theoretically find 3.9GB RAM module (but such probably does not
> > > > > > > > > > exist) then U-Boot should work in normal 2GB area as number 3.9GB can be
> > > > > > > > > > stored into 32-bit unsigned type.
> > > > > > > > > 
> > > > > > > > > Yes, you need to disable CONFIG_ARCH_FIXUP_FDT_MEMORY
> > > > > > > > 
> > > > > > > > CONFIG_ARCH_FIXUP_FDT_MEMORY affects booting OS.
> > > > > > > > 
> > > > > > > > The issue is that **u-boot** is crashing during its initialization.
> > > > > > > 
> > > > > > > When configuring the controller for 4GB, but limiting U-Boot to 2GB?
> > > > > > 
> > > > > > Yes!
> > > > > 
> > > > > Interesting.  Maybe you just have to do the 36BIT games on that SoC, to
> > > > > support what you're trying to do.  If anyone at NXP knows something
> > > > > perhaps they'll chime in.
> > > > 
> > > > Why? It is issue in U-Boot and affects all 32-bit platforms not only powerpc/NXP.
> > > 
> > > I disagree.  It's working as intended with other 32bit platforms, unless
> > 
> > It really cannot work. See my previous emails with exact description. It
> > is mathematical limitation that you cannot store 4GB size of RAM into
> > U-Boot's gd structure. Maximal number which can be stored into ulong
> > type on 32-bit platforms is 2^32-1 which is less than 4GB.
> > 
> > So if U-Boot detects 4GB DDR module it crashes prior it tries to "limit"
> > usage just to e.g. 2GB.
> 
> Yes, this is a semi-intentional limitation of our design history.  We're
> still in the very old days of a 2GB:2GB address space split.

This is just default value in option CONFIG_MAX_MEM_MAPPED or in
function get_effective_memsize().

> > > it's stopped working on dra7xx_evm with >2GB memory (which I don't have,
> > > unfortunately).  If you can't configure the memory controller to know
> > > that 4GB exists, but limit yourself to running within 2GB, that's a
> > > SoC-specific problem.
> > 
> > Have you really read what I wrote? It is not SoC-specific problem. Full
> > size of RAM (not mapped memory by U-Boot which may be less than full
> > size of RAM) is stored in gd->ram_size variable. And this variable is
> > cast to ulong type which cannot store 4GB. This is common U-Boot code,
> > not something SoC specific.
> > 
> > Plus CONFIG_MAX_MEM_MAPPED is ignored in some cases which I also
> > described in previous email. And code for ignoring is again **not** SoC
> > specific, but generic in common U-Boot code.
> 
> I guess no, I'm not seeing everything you've wrote, and you're not
> seeing everything I wrote either.  If you have a 32bit platform and more
> than 2GB of memory, you don't tell U-Boot that you have more than 2GB of
> memory because it's not going to work right.

And this is what I'm writing since beginning. It does not work because
of issues in U-Boot code with ulong type which I described in first
email.

> You lie, one way or
> another, and have U-Boot work inside a 2GB window, and do whatever you
> need so that the OS will see the correct amount and do the right thing.
> All of the options you've noted I believe are related to the kludges
> that have been required thus far to either enable 36bit support on some
> PowerPC platforms (which you've explained you don't want to do) or the
> 36bit support for some ARM platforms (of which I'm asking around to see
> if someone can re-test on current mainline).  I don't _think_ we have an
> option today that would mirror the 1GB:3GB (peripheral:memory) split
> that was an option at some point for Linux, as how to start handling
> bigmem systems.

This option is there and I'm writing about it in every email. Option
CONFIG_MAX_MEM_MAPPED. Also there is function get_effective_memsize()
with default implementation which use CONFIG_MAX_MEM_MAPPED, but not
always. And this function defines that split.

> What I keep reading what you want to do is to support 4GB of memory
> without doing some kludges.  I have no idea how much work would be
> required to enable that, at this point.
> 
> -- 
> Tom

Please look again at my first email in this thread. I have already wrote
what is wrong there.

The issue is really only in ulong type and in config option
CONFIG_VERY_BIG_RAM which I have not understand how and why is used and
why default get_effective_memsize() ignores CONFIG_MAX_MEM_MAPPED.