[U-Boot] A10 bring up and DRAM configuration procedure

[U-Boot] A10 bring up and DRAM configuration procedure

[Piotr Król]

Hi all,
I'm trying to power on custom A10 hardware based on Cubieboard1. DDR3
modules used are MT41J64M16 [1] - at least FBGA Code D9MNQ match this
hardware.

I'm using mainline U-Boot built using Buildroot with Cubieboard defconfig.
Unfortunately U-Boot hangs "randomly" at various locations. Farthest I can get
is:

## Booting kernel from Legacy Image at 48000000 ...
   Image Name:   Linux-3.4.103
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    4358824 Bytes = 4.2 MiB
   Load Address: 40008000
   Entry Point:  40008000
   Verifying Checksum ... 

This hangs in crc32_no_comp [2]. Although this is not 100% reproducible. Full
bootlog [5].

I made tests using various DRAM parameters configurations and those take
booting process farthest are with CONFIG_DRAM_CLK=240:

# below gives ~80% chance to get to Verifying Checksum
CONFIG_DRAM_CLK=240
CONFIG_DRAM_TIMINGS_DDR3_1066F_1333H=y

# below gives ~85% chance to get to Verifying Checksum
CONFIG_DRAM_CLK=240
CONFIG_DRAM_TIMINGS_VENDOR_MAGIC=y

Other configuration like CONFIG_DRAM_CLK=360 give worst results. For
240MHz I was even able get to "Starting kernel ...", but this is
sporadic. I disabled in U-Boot as much I could, my defconfig is here
[3].

From what I was able to found this is result of incorrect DRAM configuration
[4]. Anyone can confirm that this is correct diagnosis ? If not then what test
I can perform to gain better confidence ?

Other question is how to proceed from this point ? I have datasheet for DDR3,
but number of possible parameters to tweak seems to be overwhelming. Also as
sunxi wiki claim timing from datasheet have to be converted to number of cycles
- I'm not sure how to approach that. Is there any guide where to start ? sunxi
wiki describe calibration, when it is possible to run system, but what can be
done in situation when U-Boot cannot reach that state ?

If anyone more experienced can advise what can be done at this point I would
appreciate.

[1] http://www.micron.com/~/media/Documents/Products/Data%20Sheet/DRAM/DDR3/1Gb_DDR3_SDRAM.pdf
[2] http://git.denx.de/?p=u-boot.git;a=blob;f=lib/crc32.c;h=97592124867abb815d576899f8789545c3aef1aa;hb=HEAD#l200
[3] https://gist.github.com/pietrushnic/ea41a4ae38b7af8a6fa456113ce19af8
[4] http://www.denx.de/wiki/view/DULG/UBootCrashAfterRelocation
[5] https://gist.github.com/pietrushnic/d6a3d6c0b4a20a3226dfb67c4d129142

-- 
Best Regards,
Piotr Kr?l
Embedded Systems Consultant
http://3mdeb.com | @3mdeb_com

[U-Boot] [linux-sunxi] A10 bring up and DRAM configuration procedure

[Siarhei Siamashka]

Hi Piotr,

On Sun, 3 Apr 2016 16:26:44 +0200
Piotr Kr?l <piotr.krol@3mdeb.com> wrote:

> Hi all,
> I'm trying to power on custom A10 hardware based on Cubieboard1. DDR3
> modules used are MT41J64M16 [1] - at least FBGA Code D9MNQ match this
> hardware.
> 
> I'm using mainline U-Boot built using Buildroot with Cubieboard defconfig.
> Unfortunately U-Boot hangs "randomly" at various locations. Farthest I can get
> is:
> 
> ## Booting kernel from Legacy Image at 48000000 ...
>    Image Name:   Linux-3.4.103
>    Image Type:   ARM Linux Kernel Image (uncompressed)
>    Data Size:    4358824 Bytes = 4.2 MiB
>    Load Address: 40008000
>    Entry Point:  40008000
>    Verifying Checksum ... 
> 
> This hangs in crc32_no_comp [2]. Although this is not 100% reproducible. Full
> bootlog [5].
> 
> I made tests using various DRAM parameters configurations and those take
> booting process farthest are with CONFIG_DRAM_CLK=240:
> 
> # below gives ~80% chance to get to Verifying Checksum
> CONFIG_DRAM_CLK=240
> CONFIG_DRAM_TIMINGS_DDR3_1066F_1333H=y
> 
> # below gives ~85% chance to get to Verifying Checksum
> CONFIG_DRAM_CLK=240
> CONFIG_DRAM_TIMINGS_VENDOR_MAGIC=y
> 
> Other configuration like CONFIG_DRAM_CLK=360 give worst results. For
> 240MHz I was even able get to "Starting kernel ...", but this is
> sporadic. I disabled in U-Boot as much I could, my defconfig is here
> [3].
> 
> From what I was able to found this is result of incorrect DRAM configuration
> [4]. Anyone can confirm that this is correct diagnosis ? If not then what test
> I can perform to gain better confidence ?

If you get somewhat better results depending on the DRAM clock
frequency settings, then I would say that it pretty much points to
the DRAM related problems. So you are apparently on the right track.

> Other question is how to proceed from this point ? I have datasheet for DDR3,
> but number of possible parameters to tweak seems to be overwhelming. Also as
> sunxi wiki claim timing from datasheet have to be converted to number of cycles
> - I'm not sure how to approach that. Is there any guide where to start ?

Well, the standard JEDEC timings are already tabulated in 

    http://git.denx.de/?p=u-boot.git;a=blob;f=board/sunxi/dram_timings_sun4i.h

Some DRAM chips may have better timings than mandated by the standard,
but this is more like performance tuning. Either way, these timings are
a "digital" thing, they specify how many cycles to wait between sending
different commands to the DRAM chip. And they are rather unlikely to be
related to your reliability problems.

It is much more likely that you have "analog" problems and the signal
gets distorted too badly on the way from the DRAM controller in the
SoC to the DRAM chips. From what I heard, the DDR3 tracks routing is a
major PITA, one needs to get both the tracks length and the tracks
impedance matched within certain tolerance levels. As you are dealing
with a custom hardware design, then maybe the DDR3 routing is a little
bit screwed up?

The DRAM controller configuration registers allow to configure the
delays and drive/termination impedance to correct the PCB imperfection
to some extent. In practice, all the "professionally" designed Allwinner
devices work with the same generic Allwinner blessed DRAM settings,
because the PCB routing is apparently usually done and tested to work
fine with these default settings. Also reducing the DRAM clock speed
usually improves reliability. It was believed that any A10/A13/A20
device should work stable if we clock the DRAM down to 360MHz.

On the hardware side, you can maybe check the ZQ and RZQ resistors
(one connected to the SoC and also one extra resistor for each DRAM
chip). They should at least exist and be preferably 240 ohm. Some board
manufacturers tune the nominal value of these resistors, apparently
as a way to fine tune the reliability. That's the configuration knob
that is easily available to board manufacturers (they don't dare to
touch the DRAM settings in U-boot and probably don't know how to do
this).

One more thing is the VDD-DLL voltage. It is served from DCDC3 (AXP209
PMIC) and is usually set to 1.250V by default. Increasing this voltage
usually improves the DRAM reliability and allows it to be clocked
higher. You can try to increase this voltage to 1.300V to check
if this helps. Don't go beyond 1.325V on Allwinner A10 though.

> sunxi wiki describe calibration, when it is possible to run system, but what
> can be done in situation when U-Boot cannot reach that state ?

The whole idea of this calibration process is to find good ZQ and TPR3
settings. The settings are assumed to be good if the board is able to
pass the lima-memtester reliability test at a higher DRAM clock speed
without failing. We exploit the fact that lima-memster is by far more
sensitive to DRAM misconfiguration than the usual software. So the
borderline unreliable DRAM settings may not exhibit any visible bad
effects (the system is able to boot, the usual linux software works
fine too), but easily detected by lima-memtester. Because the system
is sufficiently usable at such borderline unstable DRAM settings, we
can run automation scripts to do a bruteforce search, which is probing
different TPR3 values without human supervision. That's what is
explained at:

    https://linux-sunxi.org/A10_DRAM_Controller_Calibration

But in your case, you can try different ZQ and TPR3 settings manually
in the U-Boot defconfig and instead of running lima-memtester (which
first needs a bootable userland) just monitor how far the system is
able to boot as an alternative selection criteria for picking better
configuration values. If you manage to boot the system up to a
semi-working userland, then you can switch to using lima-memtester
for the final tuning and validation.

> If anyone more experienced can advise what can be done at this point I would
> appreciate.

Maybe ask the hardware people if they have done the DDR3 routing right?
Also if it is just a single board, then it might be not misconfigured,
but defective. Trying another sample could be a good idea.

> [1] http://www.micron.com/~/media/Documents/Products/Data%20Sheet/DRAM/DDR3/1Gb_DDR3_SDRAM.pdf
> [2] http://git.denx.de/?p=u-boot.git;a=blob;f=lib/crc32.c;h=97592124867abb815d576899f8789545c3aef1aa;hb=HEAD#l200
> [3] https://gist.github.com/pietrushnic/ea41a4ae38b7af8a6fa456113ce19af8
> [4] http://www.denx.de/wiki/view/DULG/UBootCrashAfterRelocation
> [5] https://gist.github.com/pietrushnic/d6a3d6c0b4a20a3226dfb67c4d129142

-- 
Best regards,
Siarhei Siamashka

[U-Boot] [linux-sunxi] A10 bring up and DRAM configuration procedure

[Piotr Król]

On Sun, Apr 03, 2016 at 07:11:21PM +0300, Siarhei Siamashka wrote:
> Hi Piotr,

Hi Siarhei,
thanks for comprehensive reply.

> > Other question is how to proceed from this point ? I have datasheet for DDR3,
> > but number of possible parameters to tweak seems to be overwhelming. Also as
> > sunxi wiki claim timing from datasheet have to be converted to number of cycles
> > - I'm not sure how to approach that. Is there any guide where to start ?
> 
> Well, the standard JEDEC timings are already tabulated in 
> 
>     http://git.denx.de/?p=u-boot.git;a=blob;f=board/sunxi/dram_timings_sun4i.h
> 
> Some DRAM chips may have better timings than mandated by the standard,
> but this is more like performance tuning. Either way, these timings are
> a "digital" thing, they specify how many cycles to wait between sending
> different commands to the DRAM chip. And they are rather unlikely to be
> related to your reliability problems.
> 
> It is much more likely that you have "analog" problems and the signal
> gets distorted too badly on the way from the DRAM controller in the
> SoC to the DRAM chips. From what I heard, the DDR3 tracks routing is a
> major PITA, one needs to get both the tracks length and the tracks
> impedance matched within certain tolerance levels. As you are dealing
> with a custom hardware design, then maybe the DDR3 routing is a little
> bit screwed up?

You're right. I think this is the reason, because of that first thing I
tried when I saw corrupted serial output was decreasing DRAM_CLK.

> 
> The DRAM controller configuration registers allow to configure the
> delays and drive/termination impedance to correct the PCB imperfection
> to some extent. In practice, all the "professionally" designed Allwinner
> devices work with the same generic Allwinner blessed DRAM settings,
> because the PCB routing is apparently usually done and tested to work
> fine with these default settings. Also reducing the DRAM clock speed
> usually improves reliability. It was believed that any A10/A13/A20
> device should work stable if we clock the DRAM down to 360MHz.
> 
> On the hardware side, you can maybe check the ZQ and RZQ resistors
> (one connected to the SoC and also one extra resistor for each DRAM
> chip). They should at least exist and be preferably 240 ohm. Some board
> manufacturers tune the nominal value of these resistors, apparently
> as a way to fine tune the reliability. That's the configuration knob
> that is easily available to board manufacturers (they don't dare to
> touch the DRAM settings in U-boot and probably don't know how to do
> this).

I have schematics and confirmed that there is 240 ohm (%1 tolerance)
resistor although I cannot be sure about its quality. So there are
resistors on ZQ from DRAM die an SZQ (AA7) from SoC side. If I will get
to wall I will probably check if those have good enough quality and
maybe replace it. Is there any guide what can be used or I should just
try slightly higher/lower and see if situation change ?

> 
> One more thing is the VDD-DLL voltage. It is served from DCDC3 (AXP209
> PMIC) and is usually set to 1.250V by default. Increasing this voltage
> usually improves the DRAM reliability and allows it to be clocked
> higher. You can try to increase this voltage to 1.300V to check
> if this helps. Don't go beyond 1.325V on Allwinner A10 though.
> 
> > sunxi wiki describe calibration, when it is possible to run system, but what
> > can be done in situation when U-Boot cannot reach that state ?
> 
> The whole idea of this calibration process is to find good ZQ and TPR3
> settings. The settings are assumed to be good if the board is able to
> pass the lima-memtester reliability test at a higher DRAM clock speed
> without failing. We exploit the fact that lima-memster is by far more
> sensitive to DRAM misconfiguration than the usual software. So the
> borderline unreliable DRAM settings may not exhibit any visible bad
> effects (the system is able to boot, the usual linux software works
> fine too), but easily detected by lima-memtester. Because the system
> is sufficiently usable at such borderline unstable DRAM settings, we
> can run automation scripts to do a bruteforce search, which is probing
> different TPR3 values without human supervision. That's what is
> explained at:
> 
>     https://linux-sunxi.org/A10_DRAM_Controller_Calibration
> 
> But in your case, you can try different ZQ and TPR3 settings manually
> in the U-Boot defconfig and instead of running lima-memtester (which
> first needs a bootable userland) just monitor how far the system is
> able to boot as an alternative selection criteria for picking better
> configuration values. If you manage to boot the system up to a
> semi-working userland, then you can switch to using lima-memtester
> for the final tuning and validation.

I applied above suggestions those were very helpful. In addition, for
further reference, to tweak ZQ for A10 configuration requires enabling
ODT. So I applied those options:

CONFIG_AXP_DCDC3_VOLT=1300
CONFIG_DRAM_ZQ=123
CONFIG_DRAM_ODT_EN=y

And now I sporadically can start kernel. Unfortunately sunxi-3.4 crash
on:

mm/page_alloc.c:2317 __alloc_pages_nodemask+0x374/0x7b8()

and there are also couple BUGs:

BUG: Bad page state in process swapper  pfn:4ebb7

I wonder if tweaking ZQ can help with that. I will try and let you know.
If you have any suggestions related to above

> 
> > If anyone more experienced can advise what can be done at this point I would
> > appreciate.
> 
> Maybe ask the hardware people if they have done the DDR3 routing right?
> Also if it is just a single board, then it might be not misconfigured,
> but defective. Trying another sample could be a good idea.

It would be hard to verify hardware people answer. I think all points to
wrong DRAM wiring. Hardware people obviously didn't have previous
experience with Allwinner (ie. IDDET for USB OTG was not connected and
other minor problems). Are there any guides for DRAM wiring for
Allwinner ?

I have 2 boards and tried both although I need more statistics to gain
confidence in results.

> 
> > [1] http://www.micron.com/~/media/Documents/Products/Data%20Sheet/DRAM/DDR3/1Gb_DDR3_SDRAM.pdf
> > [2] http://git.denx.de/?p=u-boot.git;a=blob;f=lib/crc32.c;h=97592124867abb815d576899f8789545c3aef1aa;hb=HEAD#l200
> > [3] https://gist.github.com/pietrushnic/ea41a4ae38b7af8a6fa456113ce19af8
> > [4] http://www.denx.de/wiki/view/DULG/UBootCrashAfterRelocation
> > [5] https://gist.github.com/pietrushnic/d6a3d6c0b4a20a3226dfb67c4d129142
> 
> -- 
> Best regards,
> Siarhei Siamashka

-- 
Best Regards,
Piotr Kr?l
Embedded Systems Consultant
http://3mdeb.com | @3mdeb_com