[U-Boot] mx23-olinuxino: occasional memory errors

[U-Boot] mx23-olinuxino: occasional memory errors

[Kiril Zyapkov]

Hi all,

I was told I should report this.

The tests are performed on two imx233-olinuxino boards: maxi and mini.
I've removed R17 (120 ohms between CLK and CLKN) from the mini board
as advised in #olimex and the Olimex forum board, but kept it on the
maxi board.

I am testing U-Boot 2013-04 with a patchset for mx23, build recipe is
at [1] with a 3.7.2 kernel [2]. I ran `memtester 24M 100` 3 times on
each board, the memtester program is built using [3] on a computer
running ArchlinuxARM with armv5 architecture.

The maxi board shows no signs of errors -- somewhat strange, given the
popular opinion that R17 introduces instability and should be removed.

The mini board, despite having R17 removed, shows occasional memory
errors, here are some:

------------------------------------------------------------------------
Loop 3/100:
  Stuck Address       : testing   6FAILURE: possible bad address line
at offset 0x000392a0.
Skipping to next test...
...
Loop 15/100:
  Stuck Address       : testing   8FAILURE: possible bad address line
at offset 0x010321b0.
Skipping to next test...
...
FAILURE: 0x7e0ba0a0 != 0x7e4ba0a0 at offset 0x0082f8e0.
  Compare OR          : FAILURE: 0x4e032000 != 0x4e432000 at offset 0x0088bf50.
...
  8-bit Writes        : /FAILURE: 0xfe0f01b0 != 0xfe4f01b0 at offset 0x007de9a4.
------------------------------------------------------------------------

As advised by Marek, I've repeated the test with disabled pullup on
MX23_PAD_EMI_CLKN__EMI_CLKN, changing the line in
board/olimex/mx23_olinuxino/spl_boot.c to:
       MX23_PAD_EMI_CLKN__EMI_CLKN | (MXS_PAD_3V3 | MXS_PAD_12MA |
MXS_PAD_NOPULL),
, but this doesn't fix the errors on the mini board.

I've prepared a rootfs tarball which can be used to replicate the
tests. Please see [4].

Thanks,

-- Kiril

[1] https://github.com/kzyapkov/PKGBUILDs/blob/master/alarm/uboot-olinuxino/PKGBUILD
[2] https://github.com/kzyapkov/PKGBUILDs/blob/master/core/linux-olinuxino/PKGBUILD
[3] https://aur.archlinux.org/packages/memtester/
[4] http://promptly.undo.it/alarm-memtester/

[U-Boot] mx23-olinuxino: occasional memory errors

[Otavio Salvador]

On Wed, May 15, 2013 at 6:49 AM, Kiril Zyapkov <kiril.zyapkov@gmail.com> wrote:
...
> The mini board, despite having R17 removed, shows occasional memory
> errors, here are some:
...

What about including R17 and giving it a go?

--
Otavio Salvador                             O.S. Systems
E-mail: otavio at ossystems.com.br  http://www.ossystems.com.br
Mobile: +55 53 9981-7854              http://projetos.ossystems.com.br

[U-Boot] mx23-olinuxino: occasional memory errors

[Kiril Zyapkov]

On 15 May 2013 18:24, Otavio Salvador <otavio@ossystems.com.br> wrote:
> On Wed, May 15, 2013 at 6:49 AM, Kiril Zyapkov <kiril.zyapkov@gmail.com> wrote:
>> The mini board, despite having R17 removed, shows occasional memory
>> errors, here are some:
> What about including R17 and giving it a go?

I lost it and don't have another one. I could take the one from the
maxi board and put it on the mini, but I'd like to know if others see
stuff like this too.

[U-Boot] mx23-olinuxino: occasional memory errors

[Fabio Estevam]

On Wed, May 15, 2013 at 1:17 PM, Kiril Zyapkov <kiril.zyapkov@gmail.com> wrote:
> On 15 May 2013 18:24, Otavio Salvador <otavio@ossystems.com.br> wrote:
>> On Wed, May 15, 2013 at 6:49 AM, Kiril Zyapkov <kiril.zyapkov@gmail.com> wrote:
>>> The mini board, despite having R17 removed, shows occasional memory
>>> errors, here are some:
>> What about including R17 and giving it a go?
>
> I lost it and don't have another one. I could take the one from the
> maxi board and put it on the mini, but I'd like to know if others see
> stuff like this too.

Besides putting R17 in the olinuxino-mini board, it would also be
interesting to know the result of memtester when the kernel is loaded
via bootlets on this board.

[U-Boot] mx23-olinuxino: occasional memory errors

[Tsvetan]

Hi Kiril
something in uboot/kernel 3.x is broken with the memory setup
we also never succeed to build stabile image from 3.x
the only image which work rock solid is this one with kernel 2.6.x / elftosb
someone who is more familiar with kernel/uboot should take look at the 
memory/clock/plls initializations in elftosb and set them in exactly same 
manner in uboot/kernel 3.x
we used kernel 3.x image with Maxi on Embedded World to demonstrate how 
imx233 interface A13-LCD4.3" and MAXI was freezing at random time after 
playing video for 5-10 minutes on the LCD.
Same board works and plays video 24/7 with the old kernel image and elftosb

Tsvetan
----- Original Message ----- 
From: "Kiril Zyapkov" <kiril.zyapkov@gmail.com>
To: <u-boot@lists.denx.de>; "Support OLIMEX LTD" <support@olimex.com>; 
<contact@archlinuxarm.org>
Cc: "Robert Nelson" <robertcnelson@gmail.com>
Sent: Wednesday, May 15, 2013 12:49 PM
Subject: mx23-olinuxino: occasional memory errors


> Hi all,
>
> I was told I should report this.
>
> The tests are performed on two imx233-olinuxino boards: maxi and mini.
> I've removed R17 (120 ohms between CLK and CLKN) from the mini board
> as advised in #olimex and the Olimex forum board, but kept it on the
> maxi board.
>
> I am testing U-Boot 2013-04 with a patchset for mx23, build recipe is
> at [1] with a 3.7.2 kernel [2]. I ran `memtester 24M 100` 3 times on
> each board, the memtester program is built using [3] on a computer
> running ArchlinuxARM with armv5 architecture.
>
> The maxi board shows no signs of errors -- somewhat strange, given the
> popular opinion that R17 introduces instability and should be removed.
>
> The mini board, despite having R17 removed, shows occasional memory
> errors, here are some:
>
> ------------------------------------------------------------------------
> Loop 3/100:
>  Stuck Address       : testing   6FAILURE: possible bad address line
> at offset 0x000392a0.
> Skipping to next test...
> ...
> Loop 15/100:
>  Stuck Address       : testing   8FAILURE: possible bad address line
> at offset 0x010321b0.
> Skipping to next test...
> ...
> FAILURE: 0x7e0ba0a0 != 0x7e4ba0a0 at offset 0x0082f8e0.
>  Compare OR          : FAILURE: 0x4e032000 != 0x4e432000 at offset 
> 0x0088bf50.
> ...
>  8-bit Writes        : /FAILURE: 0xfe0f01b0 != 0xfe4f01b0 at offset 
> 0x007de9a4.
> ------------------------------------------------------------------------
>
> As advised by Marek, I've repeated the test with disabled pullup on
> MX23_PAD_EMI_CLKN__EMI_CLKN, changing the line in
> board/olimex/mx23_olinuxino/spl_boot.c to:
>       MX23_PAD_EMI_CLKN__EMI_CLKN | (MXS_PAD_3V3 | MXS_PAD_12MA |
> MXS_PAD_NOPULL),
> , but this doesn't fix the errors on the mini board.
>
> I've prepared a rootfs tarball which can be used to replicate the
> tests. Please see [4].
>
> Thanks,
>
> -- Kiril
>
> [1] 
> https://github.com/kzyapkov/PKGBUILDs/blob/master/alarm/uboot-olinuxino/PKGBUILD
> [2] 
> https://github.com/kzyapkov/PKGBUILDs/blob/master/core/linux-olinuxino/PKGBUILD
> [3] https://aur.archlinux.org/packages/memtester/
> [4] http://promptly.undo.it/alarm-memtester/ 

[U-Boot] mx23-olinuxino: occasional memory errors

[Tsvetan]

Kiril,

R17 adds instability to the SDRAM so you do not have to add it, that MAXI 
works better is because it uses 4 layer design and even with R17 could work 
without glitch, but if you remove R17 from MAXI it will work even better :)

Tsvetan
----- Original Message ----- 
From: "Kiril Zyapkov" <kiril.zyapkov@gmail.com>
To: "Otavio Salvador" <otavio@ossystems.com.br>
Cc: "U-Boot Mailing List" <u-boot@lists.denx.de>; "Support OLIMEX LTD" 
<support@olimex.com>; <contact@archlinuxarm.org>
Sent: Wednesday, May 15, 2013 7:17 PM
Subject: Re: [U-Boot] mx23-olinuxino: occasional memory errors


> On 15 May 2013 18:24, Otavio Salvador <otavio@ossystems.com.br> wrote:
>> On Wed, May 15, 2013 at 6:49 AM, Kiril Zyapkov <kiril.zyapkov@gmail.com> 
>> wrote:
>>> The mini board, despite having R17 removed, shows occasional memory
>>> errors, here are some:
>> What about including R17 and giving it a go?
>
> I lost it and don't have another one. I could take the one from the
> maxi board and put it on the mini, but I'd like to know if others see
> stuff like this too. 

[U-Boot] mx23-olinuxino: occasional memory errors

[Otavio Salvador]

On Thu, May 16, 2013 at 1:53 AM, Tsvetan Usunov, OLIMEX Ltd
<usunov@olimex.com> wrote:
> Kiril,
>
> R17 adds instability to the SDRAM so you do not have to add it, that MAXI
> works better is because it uses 4 layer design and even with R17 could work
> without glitch, but if you remove R17 from MAXI it will work even better :)

So new board revisions are coming without R17?

--
Otavio Salvador                             O.S. Systems
E-mail: otavio at ossystems.com.br  http://www.ossystems.com.br
Mobile: +55 53 9981-7854              http://projetos.ossystems.com.br

[U-Boot] mx23-olinuxino: occasional memory errors

[Otavio Salvador]

On Thu, May 16, 2013 at 1:34 AM, Tsvetan Usunov, OLIMEX Ltd
<usunov@olimex.com> wrote:
> Hi Kiril
> something in uboot/kernel 3.x is broken with the memory setup
> we also never succeed to build stabile image from 3.x
> the only image which work rock solid is this one with kernel 2.6.x / elftosb
> someone who is more familiar with kernel/uboot should take look at the
> memory/clock/plls initializations in elftosb and set them in exactly same
> manner in uboot/kernel 3.x
> we used kernel 3.x image with Maxi on Embedded World to demonstrate how
> imx233 interface A13-LCD4.3" and MAXI was freezing at random time after
> playing video for 5-10 minutes on the LCD.
> Same board works and plays video 24/7 with the old kernel image and elftosb

Fabio did a big work on this and made it work for memtester >24h.
These patches are in 2013.07 and
https://github.com/Freescale/u-boot-imx/commits/patches-2013.04

Can you test this one and check if it is stable?

--
Otavio Salvador                             O.S. Systems
E-mail: otavio at ossystems.com.br  http://www.ossystems.com.br
Mobile: +55 53 9981-7854              http://projetos.ossystems.com.br

[U-Boot] mx23-olinuxino: occasional memory errors

[Fabio Estevam]

On Thu, May 16, 2013 at 8:02 AM, Otavio Salvador
<otavio@ossystems.com.br> wrote:
> On Thu, May 16, 2013 at 1:34 AM, Tsvetan Usunov, OLIMEX Ltd
> <usunov@olimex.com> wrote:
>> Hi Kiril
>> something in uboot/kernel 3.x is broken with the memory setup
>> we also never succeed to build stabile image from 3.x
>> the only image which work rock solid is this one with kernel 2.6.x / elftosb
>> someone who is more familiar with kernel/uboot should take look at the
>> memory/clock/plls initializations in elftosb and set them in exactly same
>> manner in uboot/kernel 3.x
>> we used kernel 3.x image with Maxi on Embedded World to demonstrate how
>> imx233 interface A13-LCD4.3" and MAXI was freezing at random time after
>> playing video for 5-10 minutes on the LCD.
>> Same board works and plays video 24/7 with the old kernel image and elftosb
>
> Fabio did a big work on this and made it work for memtester >24h.
> These patches are in 2013.07 and
> https://github.com/Freescale/u-boot-imx/commits/patches-2013.04
>
> Can you test this one and check if it is stable?

Tsvetan, please try this branch and let us know how it goes. In fact
U-boot 2013.04 had issues with DDR setup and these issues were fixed
recently.

The problem with the mini variant is a different one. Does mini run
stable using old kernel and bootlets?

As the DDR layout is different we probably need to tweak
DRAM_HW_CTRL19 register for this board.

Regards,

Fabio Estevam

[U-Boot] mx23-olinuxino: occasional memory errors

[Tsvetan]

> The problem with the mini variant is a different one. Does mini run
> stable using old kernel and bootlets?

all imx233 boards use same image and kernel/bootlets and we have no issues 
with any of them with the old images

Tsvetan 

[U-Boot] mx23-olinuxino: occasional memory errors

[Fabio Estevam]

On Thu, May 16, 2013 at 9:36 AM, Tsvetan Usunov, OLIMEX Ltd
<usunov@olimex.com> wrote:
>> The problem with the mini variant is a different one. Does mini run
>> stable using old kernel and bootlets?
>
>
> all imx233 boards use same image and kernel/bootlets and we have no issues
> with any of them with the old images

Ok, so we have to isolate the issue then. I would suggest you to try:

- Kernel 3.9 + u-boot with mx23 ddr patches

- Kernel 3.9 + bootlets

This way you could see if the instability is coming from U-boot or
from the kernel.

[U-Boot] mx23-olinuxino: occasional memory errors

[Tsvetan]

I will ask our people to check!
Tsvetan
----- Original Message ----- 
From: "Fabio Estevam" <festevam@gmail.com>
To: "Tsvetan Usunov, OLIMEX Ltd" <usunov@olimex.com>
Cc: "Otavio Salvador" <otavio@ossystems.com.br>; "U-Boot Mailing List" 
<u-boot@lists.denx.de>; <contact@archlinuxarm.org>
Sent: Thursday, May 16, 2013 3:51 PM
Subject: Re: [U-Boot] mx23-olinuxino: occasional memory errors


> On Thu, May 16, 2013 at 9:36 AM, Tsvetan Usunov, OLIMEX Ltd
> <usunov@olimex.com> wrote:
>>> The problem with the mini variant is a different one. Does mini run
>>> stable using old kernel and bootlets?
>>
>>
>> all imx233 boards use same image and kernel/bootlets and we have no 
>> issues
>> with any of them with the old images
>
> Ok, so we have to isolate the issue then. I would suggest you to try:
>
> - Kernel 3.9 + u-boot with mx23 ddr patches
>
> - Kernel 3.9 + bootlets
>
> This way you could see if the instability is coming from U-boot or
> from the kernel.