OMAP baseline test results for v3.7-rc4

OMAP baseline test results for v3.7-rc4

[Paul Walmsley]

Here are some basic OMAP test results for Linux v3.7-rc4.
Logs and other details at:

    http://www.pwsan.com/omap/testlogs/test_v3.7-rc4/20121104142910/


Passing tests
-------------

Boot to userspace: 2420n800, 3517evm, 3530es3beagle, 3730beaglexm, 37xxevm,
                   4430es2panda, 5912osk, am335xbone

PM ret/off, suspend + dynamic idle: (none)


Failing tests: fixed by posted patches
--------------------------------------

Boot tests:

* 2430sdp: vfp_reload_hw oops during MMC initialization
  - Kernel attempts to save FP registers that don't exist; fix posted:
    - http://www.spinics.net/lists/arm-kernel/msg200646.html
    - added to rmk's patch system as 7566/1

* AM335x Beaglebone: omap2plus_defconfig kernels don't boot
  - Due to GPMC missing support for DT
  - Temporary workaround at 
http://www.spinics.net/lists/arm-kernel/msg200787.html
  - May be fixed now, pending retest:
    - http://marc.info/?l=linux-omap&m=135082257727502&w=2
  - Not yet part of the automated test suite
  * May be due to an old U-boot with FDT support problems?

PM tests:

* 3530es3beagle, 37xxevm, 3730beaglexm: I2C fails during resume from suspend
  - Causes MMC to become unusable since regulators are not reenabled
  - Caused by RT throttling
  - Fixed by http://www.spinics.net/lists/arm-kernel/msg202224.html
  - Patch in rmk's patch system as 7565/1

* 3530es3beagle: hangs during off-mode dynamic idle test
  - Appears to be caused by commit 6c31b2150ff96755d24e0ab6d6fea08a7bf5c44c:
    - http://marc.info/?l=linux-omap&m=135075364705188&w=2
  - Fixed by http://www.spinics.net/lists/arm-kernel/msg202116.html
 
Other:

* 2420N800: powers down 30 seconds after boot
  - Presumably due to missing CBUS patches for watchdog control
  - http://lkml.org/lkml/2012/9/3/265

* 4430es2panda: omap_hwmod: mcpdm: cannot be enabled for reset (3)
  - clock source is from an external I2C-controlled source
  - must skip reset until the switchover to hwmod late init
  - http://www.spinics.net/lists/arm-kernel/msg178138.html


Failing tests: needing investigation
------------------------------------

Boot tests:

* CM-T3517: L3 in-band error with IPSS during boot
  - Cause unknown but see http://marc.info/?l=linux-omap&m=134833869730129&w=2
  - Longstanding issue; does not occur on the 3517EVM

* 3517EVM & CM-T3517: boot hangs with NFS root
  - Likely some Kconfig, board file, and PM issues with EMAC

* CM-T3517: boot hangs with MMC boot
  - Due to missing MMC setup in board file

* 4460pandaes: boot fails early
  - Appears to be due to X-loader problems here
  - Need to note the X-loader version so we know it's broken

* 3530ES3 Beagle: I2C timeouts during userspace init
  - Intermittent, appears on 5 out of 6 boots here
  - Aaro Koskinen observes this also on N900
  - Appears to be caused by commit 3db11feffc1ad2ab9dea27789e6b5b3032827adc
    - http://marc.info/?l=linux-arm-kernel&m=135071372426971&w=2
    - http://marc.info/?l=linux-omap&m=135197877112220&w=2

PM tests:

* 3730 Beagle XM: does not serial wake from off-idle suspend when console
  UART doesn't clock gate ("debug ignore_loglevel")
  - Not shown in the current test logs; cause unknown
  - Pending re-test

Other:

* 4430es2panda: omap_hwmod: l3_instr: _wait_target_disable failed
  - Unknown cause; could be due to the lack of hierarchical enable/disable
    in hwmod code
  - Jon Hunter reports this does not appear with the same X-loader/bootloader
    on his 4430ES2.3 Panda, so could be ES-level dependent

vmlinux object size
(delta in bytes from test_v3.7-rc3 (8f0d8163b50e01f398b14bcd4dc039ac5ab18d64)):
   text     data      bss    total  kernel
   +750        0        0     +750  am33xx_only
   +208        0        0     +208  n800_multi_omap2xxx
   +208        0        0     +208  n800_only_a
  +2645        0        0    +2645  omap1_defconfig
  +2641        0        0    +2641  omap1_defconfig_1510innovator_only
  +2629        0        0    +2629  omap1_defconfig_5912osk_only
   +738        0        0     +738  omap2plus_defconfig
   +746        0        0     +746  omap2plus_defconfig_2430sdp_only
   +742        0        0     +742  omap2plus_defconfig_cpupm
   +742        0        0     +742  omap2plus_defconfig_no_pm
   +742        0        0     +742  omap2plus_defconfig_omap2_4_only
   +750        0        0     +750  omap2plus_defconfig_omap3_4_only
  +1105        0        0    +1105  rmk_omap3430_ldp_oldconfig
  +1527       +8        0    +1535  rmk_omap4430_sdp_oldconfig

An interesting 2.5KiB jump in OMAP1 kernel size.



- Paul

Re: OMAP baseline test results for v3.7-rc4

[Jon Hunter]

On 11/04/2012 08:46 PM, Paul Walmsley wrote:
> 
> Here are some basic OMAP test results for Linux v3.7-rc4.
> Logs and other details at:
> 
>     http://www.pwsan.com/omap/testlogs/test_v3.7-rc4/20121104142910/
> 
> 
> Passing tests
> -------------
> 
> Boot to userspace: 2420n800, 3517evm, 3530es3beagle, 3730beaglexm, 37xxevm,
>                    4430es2panda, 5912osk, am335xbone
> 
> PM ret/off, suspend + dynamic idle: (none)
> 
> 
> Failing tests: fixed by posted patches
> --------------------------------------
> 
> Boot tests:
> 
> * 2430sdp: vfp_reload_hw oops during MMC initialization
>   - Kernel attempts to save FP registers that don't exist; fix posted:
>     - http://www.spinics.net/lists/arm-kernel/msg200646.html
>     - added to rmk's patch system as 7566/1
> 
> * AM335x Beaglebone: omap2plus_defconfig kernels don't boot
>   - Due to GPMC missing support for DT
>   - Temporary workaround at 
> http://www.spinics.net/lists/arm-kernel/msg200787.html
>   - May be fixed now, pending retest:
>     - http://marc.info/?l=linux-omap&m=135082257727502&w=2
>   - Not yet part of the automated test suite
>   * May be due to an old U-boot with FDT support problems?
> 
> PM tests:
> 
> * 3530es3beagle, 37xxevm, 3730beaglexm: I2C fails during resume from suspend
>   - Causes MMC to become unusable since regulators are not reenabled
>   - Caused by RT throttling
>   - Fixed by http://www.spinics.net/lists/arm-kernel/msg202224.html
>   - Patch in rmk's patch system as 7565/1
> 
> * 3530es3beagle: hangs during off-mode dynamic idle test
>   - Appears to be caused by commit 6c31b2150ff96755d24e0ab6d6fea08a7bf5c44c:
>     - http://marc.info/?l=linux-omap&m=135075364705188&w=2
>   - Fixed by http://www.spinics.net/lists/arm-kernel/msg202116.html
>  
> Other:
> 
> * 2420N800: powers down 30 seconds after boot
>   - Presumably due to missing CBUS patches for watchdog control
>   - http://lkml.org/lkml/2012/9/3/265
> 
> * 4430es2panda: omap_hwmod: mcpdm: cannot be enabled for reset (3)
>   - clock source is from an external I2C-controlled source
>   - must skip reset until the switchover to hwmod late init
>   - http://www.spinics.net/lists/arm-kernel/msg178138.html
> 
> 
> Failing tests: needing investigation
> ------------------------------------
> 
> Boot tests:
> 
> * CM-T3517: L3 in-band error with IPSS during boot
>   - Cause unknown but see http://marc.info/?l=linux-omap&m=134833869730129&w=2
>   - Longstanding issue; does not occur on the 3517EVM
> 
> * 3517EVM & CM-T3517: boot hangs with NFS root
>   - Likely some Kconfig, board file, and PM issues with EMAC
> 
> * CM-T3517: boot hangs with MMC boot
>   - Due to missing MMC setup in board file
> 
> * 4460pandaes: boot fails early
>   - Appears to be due to X-loader problems here
>   - Need to note the X-loader version so we know it's broken

Are you still planning to investigate this further or migrate to a new
bootloader?

With 4460 and 3.7-rc4, I am seeing that resume from suspend is not
working (not sure when this broke though). I am just curious if anyone
else sees this. My u-boot version is v2012.10.

Cheers
Jon

Re: OMAP baseline test results for v3.7-rc4

[Paul Walmsley]

On Wed, 7 Nov 2012, Jon Hunter wrote:

> On 11/04/2012 08:46 PM, Paul Walmsley wrote:
>
> > * 4460pandaes: boot fails early
> >   - Appears to be due to X-loader problems here
> >   - Need to note the X-loader version so we know it's broken
> 
> Are you still planning to investigate this further or migrate to a new
> bootloader?

Yes.  The problem with the current boot setup is that the new 
bootloader is breaking as well! :-(

This can be seen by comparing:

http://www.pwsan.com/omap/testlogs/test_v3.7-rc1/20121017205513/boot/4460pandaes/4460pandaes_log.txt

(the old bootloader)

with

http://www.pwsan.com/omap/testlogs/test_v3.7-rc4/20121104142910/boot/4460pandaes/4460pandaes_log.txt

(the new bootloader)

Dealing with this will require hands-on access, and unfortunately, right 
now I am traveling and away from my testbed.  Next week, will take a 
closer look.

> With 4460 and 3.7-rc4, I am seeing that resume from suspend is not
> working (not sure when this broke though). I am just curious if anyone
> else sees this. My u-boot version is v2012.10.

If you have the chance, it would be useful to know if any of the v3.7-rcs 
worked.  Otherwise, will look at this next week.


- Paul

Re: OMAP baseline test results for v3.7-rc4

[Jon Hunter]

On 11/07/2012 11:32 AM, Paul Walmsley wrote:
> On Wed, 7 Nov 2012, Jon Hunter wrote:
> 
>> On 11/04/2012 08:46 PM, Paul Walmsley wrote:
>>
>>> * 4460pandaes: boot fails early
>>>   - Appears to be due to X-loader problems here
>>>   - Need to note the X-loader version so we know it's broken
>>
>> Are you still planning to investigate this further or migrate to a new
>> bootloader?
> 
> Yes.  The problem with the current boot setup is that the new 
> bootloader is breaking as well! :-(

Ugh!

> This can be seen by comparing:
> 
> http://www.pwsan.com/omap/testlogs/test_v3.7-rc1/20121017205513/boot/4460pandaes/4460pandaes_log.txt
> 
> (the old bootloader)
> 
> with
> 
> http://www.pwsan.com/omap/testlogs/test_v3.7-rc4/20121104142910/boot/4460pandaes/4460pandaes_log.txt
> 
> (the new bootloader)

If you get chance, it would be good to test u-boot v2012.10 release, as
I am using this and it is working fine. I see the "new" bootloader you
are using is v2012.07 (however, I though this was working too). Weird ...

> Dealing with this will require hands-on access, and unfortunately, right 
> now I am traveling and away from my testbed.  Next week, will take a 
> closer look.
> 
>> With 4460 and 3.7-rc4, I am seeing that resume from suspend is not
>> working (not sure when this broke though). I am just curious if anyone
>> else sees this. My u-boot version is v2012.10.
> 
> If you have the chance, it would be useful to know if any of the v3.7-rcs 
> worked.  Otherwise, will look at this next week.

Ok, I will see if I can figure out where this broke.

Cheers
Jon

Re: OMAP baseline test results for v3.7-rc4

[Jon Hunter]

On 11/07/2012 11:32 AM, Paul Walmsley wrote:
> On Wed, 7 Nov 2012, Jon Hunter wrote:
> 
>> On 11/04/2012 08:46 PM, Paul Walmsley wrote:
>>
>>> * 4460pandaes: boot fails early
>>>   - Appears to be due to X-loader problems here
>>>   - Need to note the X-loader version so we know it's broken
>>
>> Are you still planning to investigate this further or migrate to a new
>> bootloader?
> 
> Yes.  The problem with the current boot setup is that the new 
> bootloader is breaking as well! :-(
> 
> This can be seen by comparing:
> 
> http://www.pwsan.com/omap/testlogs/test_v3.7-rc1/20121017205513/boot/4460pandaes/4460pandaes_log.txt
> 
> (the old bootloader)
> 
> with
> 
> http://www.pwsan.com/omap/testlogs/test_v3.7-rc4/20121104142910/boot/4460pandaes/4460pandaes_log.txt
> 
> (the new bootloader)
> 
> Dealing with this will require hands-on access, and unfortunately, right 
> now I am traveling and away from my testbed.  Next week, will take a 
> closer look.
> 
>> With 4460 and 3.7-rc4, I am seeing that resume from suspend is not
>> working (not sure when this broke though). I am just curious if anyone
>> else sees this. My u-boot version is v2012.10.
> 
> If you have the chance, it would be useful to know if any of the v3.7-rcs 
> worked.  Otherwise, will look at this next week.

Kevin pointed me to his pm fixes for 3.8 as it is working there and I
have confirmed suspend is fixed on v3.7-rc4 with the rom code fix [1]
Kevin has queued for v3.8.

Cheers
Jon

[1]
http://git.kernel.org/?p=linux/kernel/git/khilman/linux-omap-pm.git;a=commit;h=ff999b8a0983ee15668394ed49e38d3568fc6859

Re: OMAP baseline test results for v3.7-rc4

[Kevin Hilman]

+Felipe, Venkat

Paul Walmsley <paul@pwsan.com> writes:

[...]

> Failing tests: needing investigation
> ------------------------------------
>

Here's another one for v3.7-rc4.

On all OMAP3 platforms I tested (3430/n900, 3530/Beagle, 3530/Overo,
3730/OveroSTORM, 3730/Beagle-xM), a system suspend (retention or off)
fails to wake up after a dynamic idle off-mode test (test script
below[1]).

I found that disabling MMC in the config made the tests pass again, so
then bisected it down to the commit below removing SYSCONFIG
accesses[2].  I haven't had the time to find out exactly the cause, but
clearly the code that was removed was doing something that hwmod is not
taking care of automatically.

Venkat, Felipe, could you please investigate this and hopefully get a
fix out for v3.7-rc?

Thanks,

Kevin


[1] Test for system suspend (retention) after off-idle

#!/bin/sh

echo 1 > /debug/pm_debug/enable_off_mode

echo 3000 > /sys/devices/platform/omap_uart.0/power/autosuspend_delay_ms
echo 3000 > /sys/devices/platform/omap_uart.1/power/autosuspend_delay_ms
echo 3000 > /sys/devices/platform/omap_uart.2/power/autosuspend_delay_ms
if [ -e /sys/devices/platform/omap_uart.3/power/autosuspend_delay_ms ]; then 
   echo 3000 > /sys/devices/platform/omap_uart.3/power/autosuspend_delay_ms
fi

sleep 5

echo -1 > /sys/devices/platform/omap_uart.0/power/autosuspend_delay_ms
echo -1 > /sys/devices/platform/omap_uart.1/power/autosuspend_delay_ms
echo -1 > /sys/devices/platform/omap_uart.2/power/autosuspend_delay_ms
if [ -e /sys/devices/platform/omap_uart.3/power/autosuspend_delay_ms ]; then 
   echo -1 > /sys/devices/platform/omap_uart.3/power/autosuspend_delay_ms
fi

echo 0 > /debug/pm_debug/enable_off_mode

rtcwake -m mem -s 1


[2]
commit 6c31b2150ff96755d24e0ab6d6fea08a7bf5c44c
Author: Venkatraman S <svenkatr@ti.com>
Date:   Wed Aug 8 14:26:52 2012 +0530

    mmc: omap_hsmmc: remove access to SYSCONFIG register
    
    SYSCONFIG register of HSMMC IP is managed by the omap hwmod
    abstraction layer. Resetting the IP and configuring the correct
    SYSCONFIG mode is centrally managed by hwmod.
    
    Remove code which manipulates IP reset and SYSCONFIG directly in
    the driver.
    
    Signed-off-by: Venkatraman S <svenkatr@ti.com>
    Acked-by: Felipe Balbi <balbi@ti.com>
    Signed-off-by: Chris Ball <cjb@laptop.org>

Re: OMAP baseline test results for v3.7-rc4

[Paul Walmsley]

On Fri, 9 Nov 2012, Kevin Hilman wrote:

> I found that disabling MMC in the config made the tests pass again, so
> then bisected it down to the commit below removing SYSCONFIG
> accesses[2].  I haven't had the time to find out exactly the cause, but
> clearly the code that was removed was doing something that hwmod is not
> taking care of automatically.
> 
> Venkat, Felipe, could you please investigate this and hopefully get a
> fix out for v3.7-rc?

This one might fix it -- it fixes another PM bug in the v3.7-rcs involving 
the same commit:

http://git.kernel.org/?p=linux/kernel/git/pjw/omap-pending.git;a=commit;h=613ad0e98c3596cd2524172fae2a795c3fc57e4a


- Paul

Re: OMAP baseline test results for v3.7-rc4

[Kevin Hilman]

Paul Walmsley <paul@pwsan.com> writes:

> On Fri, 9 Nov 2012, Kevin Hilman wrote:
>
>> I found that disabling MMC in the config made the tests pass again, so
>> then bisected it down to the commit below removing SYSCONFIG
>> accesses[2].  I haven't had the time to find out exactly the cause, but
>> clearly the code that was removed was doing something that hwmod is not
>> taking care of automatically.
>> 
>> Venkat, Felipe, could you please investigate this and hopefully get a
>> fix out for v3.7-rc?
>
> This one might fix it -- it fixes another PM bug in the v3.7-rcs involving 
> the same commit:
>
> http://git.kernel.org/?p=linux/kernel/git/pjw/omap-pending.git;a=commit;h=613ad0e98c3596cd2524172fae2a795c3fc57e4a
>

Indeed, that fixes it.  Somehow I thought I was already including that
fix, but I wasn't.  

Thanks,

Kevin