bisected regression: arm: omap3: voltage: fix channel configuration

bisected regression: arm: omap3: voltage: fix channel configuration

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1017 bytes --]


My GTA04 (mobile phone using OMAP3 and TWL4030 PMIC) has difficulty rebooting
with 3.5.
The first boot (after power on) works fine.  But when I reboot it hangs just
before

  Switched to new clocking rate (Crystal/Core/MPU): 26.0/332/800 MHz

is printed.  If I remove "mpurate=800" from the kernel command line the
problem goes away.

If I boot into 3.5, which works, and then reboot into an older working kernel
it freezes too.  So the 3.5 kernel leaves the device in a state which causes
any kernel to hang.

I spent a while bisecting and narrowed it down to 

commit f9d29f1617eb1b2f1fd41622bd1a0fc51658d2f0
Author: Tero Kristo <t-kristo@ti.com>
Date:   Mon Feb 20 12:26:06 2012 +0200

    arm: omap3: voltage: fix channel configuration


If I revert this patch then the problem goes away.  So that is the fix I'll
go with for now, but if anyone wants me to try something else to help find
out what the real problem is and to find a proper fix, I'm happy to help.

Thanks,
NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: bisected regression: arm: omap3: voltage: fix channel configuration

[Tero Kristo]

Hi Neil,

On Mon, 2012-07-23 at 21:06 +1000, NeilBrown wrote:
> My GTA04 (mobile phone using OMAP3 and TWL4030 PMIC) has difficulty rebooting
> with 3.5.
> The first boot (after power on) works fine.  But when I reboot it hangs just
> before
> 
>   Switched to new clocking rate (Crystal/Core/MPU): 26.0/332/800 MHz
> 
> is printed.  If I remove "mpurate=800" from the kernel command line the
> problem goes away.
> 
> If I boot into 3.5, which works, and then reboot into an older working kernel
> it freezes too.  So the 3.5 kernel leaves the device in a state which causes
> any kernel to hang.

TWL chip retains its settings over warm boot which most likely explains
this.

> 
> I spent a while bisecting and narrowed it down to 
> 
> commit f9d29f1617eb1b2f1fd41622bd1a0fc51658d2f0
> Author: Tero Kristo <t-kristo@ti.com>
> Date:   Mon Feb 20 12:26:06 2012 +0200
> 
>     arm: omap3: voltage: fix channel configuration
> 
> 
> If I revert this patch then the problem goes away.  So that is the fix I'll
> go with for now, but if anyone wants me to try something else to help find
> out what the real problem is and to find a proper fix, I'm happy to help.

Do you have cpufreq enabled in your build? If yes, then most likely the
crash occurs because cpufreq scales the used OPP to minimum after boot
and the warm reset doesn't switch the voltages properly and crashes as
it attempts to raise CPU frequency.

Also, if cpufreq is enabled, mpurate command line option doesn't really
do much... it will just switch the frequency to the desired target
during beginning of boot until cpufreq kicks in and switches the OPP to
its own understanding of desired cpu rate. I think the handling of
mpurate command line option should probably be ignored in cpufreq
builds, or moved within the cpufreq driver.

I tried to craft a quick boot time fix for this problem, but it seems
the voltage layer is not properly initialized yet when the mpurate
setting kicks in and it is not very straightforward to scale voltage
during this time without writing a serious hack.

If you don't have cpufreq enabled, then the vdd routing on your board
might be interesting (I don't think this is the case as the kernel works
okay after 1st boot.)

-Tero

Re: bisected regression: arm: omap3: voltage: fix channel configuration

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 3402 bytes --]

On Mon, 23 Jul 2012 17:24:54 +0300 Tero Kristo <t-kristo@ti.com> wrote:

> Hi Neil,
> 
> On Mon, 2012-07-23 at 21:06 +1000, NeilBrown wrote:
> > My GTA04 (mobile phone using OMAP3 and TWL4030 PMIC) has difficulty rebooting
> > with 3.5.
> > The first boot (after power on) works fine.  But when I reboot it hangs just
> > before
> > 
> >   Switched to new clocking rate (Crystal/Core/MPU): 26.0/332/800 MHz
> > 
> > is printed.  If I remove "mpurate=800" from the kernel command line the
> > problem goes away.
> > 
> > If I boot into 3.5, which works, and then reboot into an older working kernel
> > it freezes too.  So the 3.5 kernel leaves the device in a state which causes
> > any kernel to hang.
> 
> TWL chip retains its settings over warm boot which most likely explains
> this.
> 
> > 
> > I spent a while bisecting and narrowed it down to 
> > 
> > commit f9d29f1617eb1b2f1fd41622bd1a0fc51658d2f0
> > Author: Tero Kristo <t-kristo@ti.com>
> > Date:   Mon Feb 20 12:26:06 2012 +0200
> > 
> >     arm: omap3: voltage: fix channel configuration
> > 
> > 
> > If I revert this patch then the problem goes away.  So that is the fix I'll
> > go with for now, but if anyone wants me to try something else to help find
> > out what the real problem is and to find a proper fix, I'm happy to help.
> 
> Do you have cpufreq enabled in your build? If yes, then most likely the
> crash occurs because cpufreq scales the used OPP to minimum after boot
> and the warm reset doesn't switch the voltages properly and crashes as
> it attempts to raise CPU frequency.

Yes, I have cpufreq enabled.
CONFIG_ARCH_HAS_CPUFREQ=y
CONFIG_ARM_OMAP2PLUS_CPUFREQ=y

Interestingly cpufreq thinks the range is 300MHz to 600MHz:

/sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies:300000 600000 

where 600 is that the freq is at boot:

# dmesg | grep Cryst
[    0.000000] Clocking rate (Crystal/Core/MPU): 26.0/332/600 MHz
[    0.056365] Switched to new clocking rate (Crystal/Core/MPU): 26.0/332/800 MHz

Does that mean the mpurate=800 isn't doing anything at all?

> 
> Also, if cpufreq is enabled, mpurate command line option doesn't really
> do much... it will just switch the frequency to the desired target
> during beginning of boot until cpufreq kicks in and switches the OPP to
> its own understanding of desired cpu rate. I think the handling of
> mpurate command line option should probably be ignored in cpufreq
> builds, or moved within the cpufreq driver.

So it looks like you are suggesting that I simply remove the "mpurate=800"
as it isn't doing anything useful anyway.  Is that right?

Might there be some way to get it to scale higher than 600MHz?
The first message from U-boot says:

OMAP3630/3730-GP ES2.1, CPU-OPP2, L3-165MHz, Max CPU Clock 1 Ghz

and the board manufacturer thinks it should be capable of 800MHz.

Thanks,
NeilBrown


> 
> I tried to craft a quick boot time fix for this problem, but it seems
> the voltage layer is not properly initialized yet when the mpurate
> setting kicks in and it is not very straightforward to scale voltage
> during this time without writing a serious hack.
> 
> If you don't have cpufreq enabled, then the vdd routing on your board
> might be interesting (I don't think this is the case as the kernel works
> okay after 1st boot.)
> 
> -Tero
> 


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: bisected regression: arm: omap3: voltage: fix channel configuration

[Tero Kristo]

On Tue, 2012-07-24 at 11:30 +1000, NeilBrown wrote:
> On Mon, 23 Jul 2012 17:24:54 +0300 Tero Kristo <t-kristo@ti.com> wrote:
> 
> > Hi Neil,
> > 
> > On Mon, 2012-07-23 at 21:06 +1000, NeilBrown wrote:
> > > My GTA04 (mobile phone using OMAP3 and TWL4030 PMIC) has difficulty rebooting
> > > with 3.5.
> > > The first boot (after power on) works fine.  But when I reboot it hangs just
> > > before
> > > 
> > >   Switched to new clocking rate (Crystal/Core/MPU): 26.0/332/800 MHz
> > > 
> > > is printed.  If I remove "mpurate=800" from the kernel command line the
> > > problem goes away.
> > > 
> > > If I boot into 3.5, which works, and then reboot into an older working kernel
> > > it freezes too.  So the 3.5 kernel leaves the device in a state which causes
> > > any kernel to hang.
> > 
> > TWL chip retains its settings over warm boot which most likely explains
> > this.
> > 
> > > 
> > > I spent a while bisecting and narrowed it down to 
> > > 
> > > commit f9d29f1617eb1b2f1fd41622bd1a0fc51658d2f0
> > > Author: Tero Kristo <t-kristo@ti.com>
> > > Date:   Mon Feb 20 12:26:06 2012 +0200
> > > 
> > >     arm: omap3: voltage: fix channel configuration
> > > 
> > > 
> > > If I revert this patch then the problem goes away.  So that is the fix I'll
> > > go with for now, but if anyone wants me to try something else to help find
> > > out what the real problem is and to find a proper fix, I'm happy to help.
> > 
> > Do you have cpufreq enabled in your build? If yes, then most likely the
> > crash occurs because cpufreq scales the used OPP to minimum after boot
> > and the warm reset doesn't switch the voltages properly and crashes as
> > it attempts to raise CPU frequency.
> 
> Yes, I have cpufreq enabled.
> CONFIG_ARCH_HAS_CPUFREQ=y
> CONFIG_ARM_OMAP2PLUS_CPUFREQ=y
> 
> Interestingly cpufreq thinks the range is 300MHz to 600MHz:
> 
> /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies:300000 600000 

Yeah, this is the default setup for omap36xx. You can see current cpu
frequency by checking the contents of scaling_cur_freq, it is most
likely 300MHz.

> 
> where 600 is that the freq is at boot:
> 
> # dmesg | grep Cryst
> [    0.000000] Clocking rate (Crystal/Core/MPU): 26.0/332/600 MHz
> [    0.056365] Switched to new clocking rate (Crystal/Core/MPU): 26.0/332/800 MHz
> 
> Does that mean the mpurate=800 isn't doing anything at all?

It switches it temporarily to 800MHz, maybe for a few seconds or so.

> 
> > 
> > Also, if cpufreq is enabled, mpurate command line option doesn't really
> > do much... it will just switch the frequency to the desired target
> > during beginning of boot until cpufreq kicks in and switches the OPP to
> > its own understanding of desired cpu rate. I think the handling of
> > mpurate command line option should probably be ignored in cpufreq
> > builds, or moved within the cpufreq driver.
> 
> So it looks like you are suggesting that I simply remove the "mpurate=800"
> as it isn't doing anything useful anyway.  Is that right?
> 
> Might there be some way to get it to scale higher than 600MHz?
> The first message from U-boot says:
> 
> OMAP3630/3730-GP ES2.1, CPU-OPP2, L3-165MHz, Max CPU Clock 1 Ghz
> 
> and the board manufacturer thinks it should be capable of 800MHz.

You need to enable 800MHz OPP similarly to what is done in 
beagle_opp_init() in board-omap3beagle.c. I am not sure what your board
is detected as, depends on your boot loader (check /proc/cpuinfo.)

-Tero

> 
> Thanks,
> NeilBrown
> 
> 
> > 
> > I tried to craft a quick boot time fix for this problem, but it seems
> > the voltage layer is not properly initialized yet when the mpurate
> > setting kicks in and it is not very straightforward to scale voltage
> > during this time without writing a serious hack.
> > 
> > If you don't have cpufreq enabled, then the vdd routing on your board
> > might be interesting (I don't think this is the case as the kernel works
> > okay after 1st boot.)
> > 
> > -Tero
> > 
> 

Re: bisected regression: arm: omap3: voltage: fix channel configuration

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1873 bytes --]

On Tue, 24 Jul 2012 09:28:24 +0300 Tero Kristo <t-kristo@ti.com> wrote:

> On Tue, 2012-07-24 at 11:30 +1000, NeilBrown wrote:

> > Might there be some way to get it to scale higher than 600MHz?
> > The first message from U-boot says:
> > 
> > OMAP3630/3730-GP ES2.1, CPU-OPP2, L3-165MHz, Max CPU Clock 1 Ghz
> > 
> > and the board manufacturer thinks it should be capable of 800MHz.
> 
> You need to enable 800MHz OPP similarly to what is done in 
> beagle_opp_init() in board-omap3beagle.c. I am not sure what your board
> is detected as, depends on your boot loader (check /proc/cpuinfo.)

Thanks!
Yes, my board identifies as cpu_is_omap3630() so that code looks relevant.
I copied the opp_init across to my board file, it runs without complaining
but it now gets to

[ ok ] Loading cpufreq kernel modules...done (none).
[....] CPUFreq Utilities: Setting ondemand CPUFreq governor...CPU0...


and hangs.  magic SYSRQ doesn't respond.

Any hints?

Thanks,
NeilBrown


#
# CPU Power Management
#

#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=y
CONFIG_CPU_FREQ_STAT=y
CONFIG_CPU_FREQ_STAT_DETAILS=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set
CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=y
CONFIG_CPU_FREQ_GOV_USERSPACE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y

#
# ARM CPU frequency scaling drivers
#
CONFIG_ARM_OMAP2PLUS_CPUFREQ=y
# CONFIG_ARM_EXYNOS4210_CPUFREQ is not set
# CONFIG_ARM_EXYNOS4X12_CPUFREQ is not set
# CONFIG_ARM_EXYNOS5250_CPUFREQ is not set
CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
CONFIG_CPU_IDLE_GOV_MENU=y

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: bisected regression: arm: omap3: voltage: fix channel configuration

[Tero Kristo]

On Tue, 2012-07-24 at 17:34 +1000, NeilBrown wrote:
> On Tue, 24 Jul 2012 09:28:24 +0300 Tero Kristo <t-kristo@ti.com> wrote:
> 
> > On Tue, 2012-07-24 at 11:30 +1000, NeilBrown wrote:
> 
> > > Might there be some way to get it to scale higher than 600MHz?
> > > The first message from U-boot says:
> > > 
> > > OMAP3630/3730-GP ES2.1, CPU-OPP2, L3-165MHz, Max CPU Clock 1 Ghz
> > > 
> > > and the board manufacturer thinks it should be capable of 800MHz.
> > 
> > You need to enable 800MHz OPP similarly to what is done in 
> > beagle_opp_init() in board-omap3beagle.c. I am not sure what your board
> > is detected as, depends on your boot loader (check /proc/cpuinfo.)
> 
> Thanks!
> Yes, my board identifies as cpu_is_omap3630() so that code looks relevant.
> I copied the opp_init across to my board file, it runs without complaining
> but it now gets to
> 
> [ ok ] Loading cpufreq kernel modules...done (none).
> [....] CPUFreq Utilities: Setting ondemand CPUFreq governor...CPU0...
> 
> 
> and hangs.  magic SYSRQ doesn't respond.
> 
> Any hints?

You only enabled opp for 800MHz right? I am just wondering if the
voltage levels for mpu / core opps are correct for your board... You may
want to check the opp tables against the information provided by your
board manufacturer. The trace about cpufreq enabling ondemand governor
indicates it is switching to frequency throttling mode at that point, so
it most likely tries to raise MPU frequency to 800MHz (CPU load is
pretty high during boot.) You can try adding some trace to cpufreq
driver itself to see what is happening.

Another thing to verify would be to check the voltage rails on your
board schematics to see how the supply voltages are delivered.

-Tero

> 
> Thanks,
> NeilBrown
> 
> 
> #
> # CPU Power Management
> #
> 
> #
> # CPU Frequency scaling
> #
> CONFIG_CPU_FREQ=y
> CONFIG_CPU_FREQ_TABLE=y
> CONFIG_CPU_FREQ_STAT=y
> CONFIG_CPU_FREQ_STAT_DETAILS=y
> # CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set
> CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE=y
> # CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set
> # CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND is not set
> # CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set
> CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
> CONFIG_CPU_FREQ_GOV_POWERSAVE=y
> CONFIG_CPU_FREQ_GOV_USERSPACE=y
> CONFIG_CPU_FREQ_GOV_ONDEMAND=y
> CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y
> 
> #
> # ARM CPU frequency scaling drivers
> #
> CONFIG_ARM_OMAP2PLUS_CPUFREQ=y
> # CONFIG_ARM_EXYNOS4210_CPUFREQ is not set
> # CONFIG_ARM_EXYNOS4X12_CPUFREQ is not set
> # CONFIG_ARM_EXYNOS5250_CPUFREQ is not set
> CONFIG_CPU_IDLE=y
> CONFIG_CPU_IDLE_GOV_LADDER=y
> CONFIG_CPU_IDLE_GOV_MENU=y

Re: bisected regression: arm: omap3: voltage: fix channel configuration

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2598 bytes --]

On Tue, 24 Jul 2012 11:51:16 +0300 Tero Kristo <t-kristo@ti.com> wrote:

> On Tue, 2012-07-24 at 17:34 +1000, NeilBrown wrote:
> > On Tue, 24 Jul 2012 09:28:24 +0300 Tero Kristo <t-kristo@ti.com> wrote:
> > 
> > > On Tue, 2012-07-24 at 11:30 +1000, NeilBrown wrote:
> > 
> > > > Might there be some way to get it to scale higher than 600MHz?
> > > > The first message from U-boot says:
> > > > 
> > > > OMAP3630/3730-GP ES2.1, CPU-OPP2, L3-165MHz, Max CPU Clock 1 Ghz
> > > > 
> > > > and the board manufacturer thinks it should be capable of 800MHz.
> > > 
> > > You need to enable 800MHz OPP similarly to what is done in 
> > > beagle_opp_init() in board-omap3beagle.c. I am not sure what your board
> > > is detected as, depends on your boot loader (check /proc/cpuinfo.)
> > 
> > Thanks!
> > Yes, my board identifies as cpu_is_omap3630() so that code looks relevant.
> > I copied the opp_init across to my board file, it runs without complaining
> > but it now gets to
> > 
> > [ ok ] Loading cpufreq kernel modules...done (none).
> > [....] CPUFreq Utilities: Setting ondemand CPUFreq governor...CPU0...
> > 
> > 
> > and hangs.  magic SYSRQ doesn't respond.
> > 
> > Any hints?
> 
> You only enabled opp for 800MHz right? I am just wondering if the
> voltage levels for mpu / core opps are correct for your board... You may
> want to check the opp tables against the information provided by your
> board manufacturer. The trace about cpufreq enabling ondemand governor
> indicates it is switching to frequency throttling mode at that point, so
> it most likely tries to raise MPU frequency to 800MHz (CPU load is
> pretty high during boot.) You can try adding some trace to cpufreq
> driver itself to see what is happening.

Yes, just 800MHz.  I just copied the code from board-omap3beagle.c, so
it is actually 800MHz for MPU and 660MHz for IVA.

I tried running with that and with your patch reverted and it works
perfectly.  I even get 33% better speed on CPU-intensive workloads, so it
really is running at 800MHz.

> 
> Another thing to verify would be to check the voltage rails on your
> board schematics to see how the supply voltages are delivered.

I have a basic understanding of the schematics, but I don't really know what
to look for when it comes to power routing.
If you happen to have time and interest, the schematics are in the manual
that can be downloaded from 

    http://projects.goldelico.com/p/gta04-main/downloads/

I've got the GTA04A3, so GTA04A3-1-complete.pdf is the one you want.

Thanks,
NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: bisected regression: arm: omap3: voltage: fix channel configuration

[Tero Kristo]

On Wed, 2012-07-25 at 11:47 +1000, NeilBrown wrote:
> On Tue, 24 Jul 2012 11:51:16 +0300 Tero Kristo <t-kristo@ti.com> wrote:
> 
> > On Tue, 2012-07-24 at 17:34 +1000, NeilBrown wrote:
> > > On Tue, 24 Jul 2012 09:28:24 +0300 Tero Kristo <t-kristo@ti.com> wrote:
> > > 
> > > > On Tue, 2012-07-24 at 11:30 +1000, NeilBrown wrote:
> > > 
> > > > > Might there be some way to get it to scale higher than 600MHz?
> > > > > The first message from U-boot says:
> > > > > 
> > > > > OMAP3630/3730-GP ES2.1, CPU-OPP2, L3-165MHz, Max CPU Clock 1 Ghz
> > > > > 
> > > > > and the board manufacturer thinks it should be capable of 800MHz.
> > > > 
> > > > You need to enable 800MHz OPP similarly to what is done in 
> > > > beagle_opp_init() in board-omap3beagle.c. I am not sure what your board
> > > > is detected as, depends on your boot loader (check /proc/cpuinfo.)
> > > 
> > > Thanks!
> > > Yes, my board identifies as cpu_is_omap3630() so that code looks relevant.
> > > I copied the opp_init across to my board file, it runs without complaining
> > > but it now gets to
> > > 
> > > [ ok ] Loading cpufreq kernel modules...done (none).
> > > [....] CPUFreq Utilities: Setting ondemand CPUFreq governor...CPU0...
> > > 
> > > 
> > > and hangs.  magic SYSRQ doesn't respond.
> > > 
> > > Any hints?
> > 
> > You only enabled opp for 800MHz right? I am just wondering if the
> > voltage levels for mpu / core opps are correct for your board... You may
> > want to check the opp tables against the information provided by your
> > board manufacturer. The trace about cpufreq enabling ondemand governor
> > indicates it is switching to frequency throttling mode at that point, so
> > it most likely tries to raise MPU frequency to 800MHz (CPU load is
> > pretty high during boot.) You can try adding some trace to cpufreq
> > driver itself to see what is happening.
> 
> Yes, just 800MHz.  I just copied the code from board-omap3beagle.c, so
> it is actually 800MHz for MPU and 660MHz for IVA.
> 
> I tried running with that and with your patch reverted and it works
> perfectly.  I even get 33% better speed on CPU-intensive workloads, so it
> really is running at 800MHz.

Okay, so it just points to voltage control somehow. Are you using IVA
btw? I think current kernel doesn't work too well with the combination
of IVA + cpufreq, as I don't think we have devfreq support in place for
IVA, and the voltages used by these two is shared. If cpufreq decides
that it wants to go to a low OPP, it will also drop the voltages for
both and this might cause an issue.

> 
> > 
> > Another thing to verify would be to check the voltage rails on your
> > board schematics to see how the supply voltages are delivered.
> 
> I have a basic understanding of the schematics, but I don't really know what
> to look for when it comes to power routing.
> If you happen to have time and interest, the schematics are in the manual
> that can be downloaded from 
> 
>     http://projects.goldelico.com/p/gta04-main/downloads/
> 
> I've got the GTA04A3, so GTA04A3-1-complete.pdf is the one you want.

It seems the power routing should be okay, so don't really know whats
the problem here. There seems to be test pads for VDD1 and VDD2 voltages
in the device, so it would be good if you have possibility to measure
these in a working / non-working case and see what kind of voltage
levels you see there. Just be careful not to connect anything strange on
these test points as they are going directly to omap power supplies.

The situations to measure would be:

- working case with old kernel after boot (idle)
- working case with new kernel after boot (idle)
- hanged case with new kernel

You should see voltages between 1v...1.4v for vdd1 (mpu), and
0.9v...1.2v for vdd2 (core). You can also measure the voltage levels
with different cpufreq OPPs to see what it does (set the
scaling_min_freq for the cpufreq under sysfs, or use userspace governor
and just force a specific OPP.) If you are uncomfortable measuring the
voltages, just try the different OPPs with SW and see if you can figure
out which one is causing problems... or whether it is just a single
transition that kills it.

-Tero