[U-Boot] Very slow mtest on i.MX6UL over dual-die DDR3 (two chip selects)

[U-Boot] Very slow mtest on i.MX6UL over dual-die DDR3 (two chip selects)

[Palacios, Hector]

Greetings,

I'm adding support to a 1GiB DDR3 chip that internally has two dies. I have configured the i.MX6UL memory controller for using two chip selects and assigned 512MiB to each. Although the RAM seems to work fine I'm experiencing extremely slow performance compared to a similar 1GiB chip with only one die (one chip select).

Testing one mtest loop over the first 496MiB takes:
 - 5s on the single die chip
 - 1m 16s on the dual die chip

I have run different memory benchmark tools in Linux like https://github.com/bingmann/pmbw.git and both memories perform similarly, but for some reason I see this weird behavior in U-Boot (v2015.04) with 'mtest'. 
The CONFIG_SYS_MEMTEST_START and CONFIG_SYS_MEMTEST_END are defined within secure limits and 'mtest' is not failing... it's just very slow.

Am I missing something obvious?
Thanks
--
Héctor Palacios

[U-Boot] Very slow mtest on i.MX6UL over dual-die DDR3 (two chip selects)

[Fabio Estevam]

Hi Hector,

On Tue, May 16, 2017 at 1:15 PM, Palacios, Hector
<Hector.Palacios@digi.com> wrote:
> Greetings,
>
> I'm adding support to a 1GiB DDR3 chip that internally has two dies. I have configured the i.MX6UL memory controller for using two chip selects and assigned 512MiB to each. Although the RAM seems to work fine I'm experiencing extremely slow performance compared to a similar 1GiB chip with only one die (one chip select).
>
> Testing one mtest loop over the first 496MiB takes:
>  - 5s on the single die chip
>  - 1m 16s on the dual die chip
>
> I have run different memory benchmark tools in Linux like https://github.com/bingmann/pmbw.git and both memories perform similarly, but for some reason I see this weird behavior in U-Boot (v2015.04) with 'mtest'.
> The CONFIG_SYS_MEMTEST_START and CONFIG_SYS_MEMTEST_END are defined within secure limits and 'mtest' is not failing... it's just very slow.
>
> Am I missing something obvious?

It is hard to tell without seeing your code.

What about adding your mx6ul board support upstream?

[U-Boot] Very slow mtest on i.MX6UL over dual-die DDR3 (two chip selects)

[Palacios, Hector]

Hi Fabio,

On Wed, 17 May 2017 00:34, Fabio Estevam wrote:
> Hi Hector,
> 
> On Tue, May 16, 2017 at 1:15 PM, Palacios, Hector
> <Hector.Palacios@digi.com> wrote:
>> Greetings,
>> 
>> I'm adding support to a 1GiB DDR3 chip that internally has two dies. I
> have configured the i.MX6UL memory controller for using two chip selects
> and assigned 512MiB to each. Although the RAM seems to work fine I'm
> experiencing extremely slow performance compared to a similar 1GiB chip
> with only one die (one chip select).
>> 
>> Testing one mtest loop over the first 496MiB takes:
>>  - 5s on the single die chip
>>  - 1m 16s on the dual die chip
>> I have run different memory benchmark tools in Linux like
> https://github.com/bingmann/pmbw.git and both memories perform similarly,
> but for some reason I see this weird behavior in U-Boot (v2015.04) with
> 'mtest'.
>> The CONFIG_SYS_MEMTEST_START and CONFIG_SYS_MEMTEST_END are defined within secure limits and
>> 'mtest' is not failing... it's just very slow.
>> 
>> Am I missing something obvious?
> 
> It is hard to tell without seeing your code.
> 
> What about adding your mx6ul board support upstream?

The code is on Github [1] (well, not the dual-die DDR3 yet) but there isn't much to see for this issue other than:

#define CONFIG_CMD_MEMTEST
#define CONFIG_SYS_MEMTEST_START	0x80000000
#define CONFIG_SYS_MEMTEST_END	0x9fffffff	/* 512 MiB */

For the dual-die DDR3, I apply the following change to the board .cfg file:

diff --git a/board/digi/ccimx6ul/imximage1GB.cfg b/board/digi/ccimx6ul/imximage1GB.cfg
index f72888bc2bfe..952458d45970 100644
--- a/board/digi/ccimx6ul/imximage1GB.cfg
+++ b/board/digi/ccimx6ul/imximage1GB.cfg
@@ -115,8 +115,8 @@ DATA 4 0x021B0018 0x00201740
 DATA 4 0x021B001C 0x00008000
 DATA 4 0x021B002C 0x000026D2
 DATA 4 0x021B0030 0x008F1023
-DATA 4 0x021B0040 0x0000005F
-DATA 4 0x021B0000 0x85180000
+DATA 4 0x021B0040 0x0000004F
+DATA 4 0x021B0000 0xC5180000
 
 /* Mode registers writes for CS0 */
 DATA 4 0x021B001C 0x02088032
@@ -125,6 +125,13 @@ DATA 4 0x021B001C 0x00048031
 DATA 4 0x021B001C 0x15208030
 DATA 4 0x021B001C 0x04008040
 
+/* Mode registers writes for CS1 */
+DATA 4 0x021B001C 0x0208803A
+DATA 4 0x021B001C 0x0000803B
+DATA 4 0x021B001C 0x00048039
+DATA 4 0x021B001C 0x15208038
+DATA 4 0x021B001C 0x04008048
+
 /* Final DDR setup */
 DATA 4 0x021B0020 0x00000800
 DATA 4 0x021B0818 0x00000227

So basically it's enabling the CS1 of the MMDC, and reducing the CS0_END to half the memory per chip select.

	[1] https://github.com/digi-embedded/u-boot


Thanks,
--
Héctor Palacios

[U-Boot] Very slow mtest on i.MX6UL over dual-die DDR3 (two chip selects)

[Fabio Estevam]

Hi Hector,

On Wed, May 17, 2017 at 5:50 AM, Palacios, Hector
<Hector.Palacios@digi.com> wrote:

> The code is on Github [1] (well, not the dual-die DDR3 yet) but there isn't much to see for this issue other than:

I looked at your code and I see:

int dram_init(void)
{
gd->ram_size = ((ulong)CONFIG_DDR_MB * SZ_1M);

return 0;
}

which may be worth investigating.

Take a look in this same function at
board/freescale/mx53loco/mx53loco.c and also in
include/configs/mx53loco.h (hint: we pass #define CONFIG_NR_DRAM_BANKS
2).

[U-Boot] Very slow mtest on i.MX6UL over dual-die DDR3 (two chip selects)

[Hector Palacios]

Hi Fabio,

On 05/17/2017 11:08 PM, Fabio Estevam wrote:
> Hi Hector,
> 
> On Wed, May 17, 2017 at 5:50 AM, Palacios, Hector
> <Hector.Palacios@digi.com> wrote:
> 
>> The code is on Github [1] (well, not the dual-die DDR3 yet) but there isn't much to see for this issue other than:
> 
> I looked at your code and I see:
> 
> int dram_init(void)
> {
> gd->ram_size = ((ulong)CONFIG_DDR_MB * SZ_1M);
> 
> return 0;
> }
> 
> which may be worth investigating.
> 
> Take a look in this same function at
> board/freescale/mx53loco/mx53loco.c and also in
> include/configs/mx53loco.h (hint: we pass #define CONFIG_NR_DRAM_BANKS
> 2).

In case I wasn't clear, I don't have any memory mapping problems. I have one single
DDR chip, which internally uses two dies and two chip selects. Setting
CONFIG_NR_DRAM_BANKS to 1 (and full size) or 2 (and half size) in U-Boot doesn't make
any difference in terms of performance.

Additional investigation showed the following:

NXP DDR stress test takes the same time to complete (successfully) in both variants
(single-die with one chip select and 1GB density per CS, and dual-die with two chip
select and 512MB per CS)

The slow memory access is global in U-Boot, not limited to 'mtest' command. A memory
copy command for 256M (time cp.l 90000000 80000000 4000000) takes:
- 1.304s on the single-die DDR3
- 15.866 seconds on the dual-die DDR3

Note that both data/instuction cache are ON on both devices, and that I'm only
exercising the lower memory (only CS0 on the dual-die chip) to avoid potential issues
or delays between changing from CS0 to CS1.

I also verified that configuring the MMDC0 for using only one CS on the dual-die DDR3
chip (and only half the size), does not help. This DDR is still performing slowly in
U-Boot, but I can't find the reason why.

Thanks for your help, anyway.
--
Hector Palacios

[U-Boot] Very slow mtest on i.MX6UL over dual-die DDR3 (two chip selects)

[Fabio Estevam]

Hi Hector,

On Thu, May 18, 2017 at 11:43 AM, Hector Palacios
<hector.palacios@digi.com> wrote:

> In case I wasn't clear, I don't have any memory mapping problems. I have one single
> DDR chip, which internally uses two dies and two chip selects. Setting
> CONFIG_NR_DRAM_BANKS to 1 (and full size) or 2 (and half size) in U-Boot doesn't make
> any difference in terms of performance.

Ok, thanks for the clarification.

> Additional investigation showed the following:
>
> NXP DDR stress test takes the same time to complete (successfully) in both variants
> (single-die with one chip select and 1GB density per CS, and dual-die with two chip
> select and 512MB per CS)
>
> The slow memory access is global in U-Boot, not limited to 'mtest' command. A memory
> copy command for 256M (time cp.l 90000000 80000000 4000000) takes:
> - 1.304s on the single-die DDR3
> - 15.866 seconds on the dual-die DDR3
>
> Note that both data/instuction cache are ON on both devices, and that I'm only
> exercising the lower memory (only CS0 on the dual-die chip) to avoid potential issues
> or delays between changing from CS0 to CS1.
>
> I also verified that configuring the MMDC0 for using only one CS on the dual-die DDR3
> chip (and only half the size), does not help. This DDR is still performing slowly in
> U-Boot, but I can't find the reason why.

I got very slow performance with MX6UL when U-Boot is loaded via
serial download mode.

This gets fixed by setting the SMP bit:
https://patchwork.ozlabs.org/patch/747074/

Looks like your issue is not related though.

Another experiment: could you try the same tests running on U-Boot 2017.05?

[U-Boot] Very slow mtest on i.MX6UL over dual-die DDR3 (two chip selects)

[Palacios, Hector]

Hi Fabio,

On Thu, 18 May 2017 17:52, Fabio Estevam wrote:
> I got very slow performance with MX6UL when U-Boot is loaded via
> serial download mode.

That's it, actually!
I was booting the target via USB, booting from the NAND shows normal performance.


> This gets fixed by setting the SMP bit:
> https://patchwork.ozlabs.org/patch/747074/

I don't have that patch. I guess a similar code exists somewhere that is not entered when booting from USB.

Many thanks for your help!
--
Héctor Palacios

[U-Boot] Very slow mtest on i.MX6UL over dual-die DDR3 (two chip selects)

[Fabio Estevam]

Hi Hector,

On Thu, May 18, 2017 at 2:06 PM, Palacios, Hector
<Hector.Palacios@digi.com> wrote:
> Hi Fabio,
>
> On Thu, 18 May 2017 17:52, Fabio Estevam wrote:
>> I got very slow performance with MX6UL when U-Boot is loaded via
>> serial download mode.
>
> That's it, actually!
> I was booting the target via USB, booting from the NAND shows normal performance.

Excellent, now we are making progress :-)

>> This gets fixed by setting the SMP bit:
>> https://patchwork.ozlabs.org/patch/747074/
>
> I don't have that patch. I guess a similar code exists somewhere that is not entered when booting from USB.

This patch has not been applied in U-Boot mainline yet. You can apply
it locally on your code for the time being.

When the board boots from SD/NAND the SMP bit is set by the ROM code
and then we do not have this problem.

When booting from serial download mode the SMP bit is not set the
system gets very slow because caches are disabled.

Regards,

Fabio Estevam