All of lore.kernel.org
 help / color / mirror / Atom feed
From: Guenter Roeck <linux@roeck-us.net>
To: Tony Lindgren <tony@atomide.com>,
	"Rafael J. Wysocki" <rafael@kernel.org>
Cc: Viresh Kumar <viresh.kumar@linaro.org>,
	"linux-pm@vger.kernel.org" <linux-pm@vger.kernel.org>,
	Peter Zijlstra <peterz@infradead.org>,
	"Rafael J. Wysocki" <rafael.j.wysocki@intel.com>,
	Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
	linux-next@vger.kernel.org,
	"linux-arm-kernel@lists.infradead.org" 
	<linux-arm-kernel@lists.infradead.org>
Subject: Re: Crashes in arm qemu emulations due to 'cpufreq: governor: Replace timers with utilization ...'
Date: Mon, 15 Feb 2016 11:40:21 -0800	[thread overview]
Message-ID: <56C229A5.8020901@roeck-us.net> (raw)
In-Reply-To: <20160215190116.GX19432@atomide.com>

On 02/15/2016 11:01 AM, Tony Lindgren wrote:
> * Rafael J. Wysocki <rafael@kernel.org> [160215 10:44]:
>> On Mon, Feb 15, 2016 at 6:05 PM, Guenter Roeck <linux@roeck-us.net> wrote:
>>> Rafael,
>>
>> Hi,
>>
>> Thanks for the report!
>>
>>> I see crashes in various arm qemu tests due to 'cpufreq: governor: Replace
>>> timers with utilization update callbacks' with next-20160215. An example
>>> crash log and bisect results are attached below.
>>>
>>> Please let me know if there is anything I can do to help tracking down
>>> the problem.
>>
>> It looks like we've uncovered some nastiness in the arch ARM code (see below).
>>
>> [cut]
>>
>>> [    1.340000] Unable to handle kernel NULL pointer dereference at virtual address 00000000
>>> [    1.340000] pgd = c0204000
>>> [    1.340000] [00000000] *pgd=00000000
>>> [    1.340000] Internal error: Oops: 80000005 [#1] SMP ARM
>>> [    1.340000] Modules linked in:
>>> [    1.340000] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.5.0-rc4-next-20160215 #1
>>> [    1.340000] Hardware name: Generic OMAP3-GP (Flattened Device Tree)
>>> [    1.340000] task: cb060000 ti: cb05a000 task.ti: cb05a000
>>> [    1.340000] PC is at 0x0
>>> [    1.340000] LR is at arch_send_call_function_single_ipi+0x34/0x38
>>
>> Since this is ARM, arch_send_call_function_single_ipi() looks like this:
>>
>> void arch_send_call_function_single_ipi(int cpu)
>> {
>>           smp_cross_call(cpumask_of(cpu), IPI_CALL_FUNC_SINGLE);
>> }
>>
>> so I'm not sure how the NULL pointer deref is possible even.
>>
>> The only thing coming to mind would be that cpumask_of(cpu) triggers
>> this, but I'm not sure how exactly that can happen.
>>
>> I need help from somebody who knows how this low-level stuff works on ARM.
>
> That's not even an SMP machine? I suspect a bunch of out of the
> 65 boot failures here are related to this:
>
> https://kernelci.org/boot/all/job/next/kernel/next-20160215/
>
> The SMP ones seem to fail with some regulator issues?
>

There is another problem, introduced with 6a0712f6f199e ("PM / OPP: Add
dev_pm_opp_set_rate()"). The kernelci boot log for next-20160212:omap3-overo-tobi
and others experience that problem.

Essentially, the code now assumes that a CPU clock always has a voltage
regulator attached to it, which is not correct. I sent out a patch to fix
that problem a minute ago.

Guenter

WARNING: multiple messages have this Message-ID (diff)
From: linux@roeck-us.net (Guenter Roeck)
To: linux-arm-kernel@lists.infradead.org
Subject: Crashes in arm qemu emulations due to 'cpufreq: governor: Replace timers with utilization ...'
Date: Mon, 15 Feb 2016 11:40:21 -0800	[thread overview]
Message-ID: <56C229A5.8020901@roeck-us.net> (raw)
In-Reply-To: <20160215190116.GX19432@atomide.com>

On 02/15/2016 11:01 AM, Tony Lindgren wrote:
> * Rafael J. Wysocki <rafael@kernel.org> [160215 10:44]:
>> On Mon, Feb 15, 2016 at 6:05 PM, Guenter Roeck <linux@roeck-us.net> wrote:
>>> Rafael,
>>
>> Hi,
>>
>> Thanks for the report!
>>
>>> I see crashes in various arm qemu tests due to 'cpufreq: governor: Replace
>>> timers with utilization update callbacks' with next-20160215. An example
>>> crash log and bisect results are attached below.
>>>
>>> Please let me know if there is anything I can do to help tracking down
>>> the problem.
>>
>> It looks like we've uncovered some nastiness in the arch ARM code (see below).
>>
>> [cut]
>>
>>> [    1.340000] Unable to handle kernel NULL pointer dereference at virtual address 00000000
>>> [    1.340000] pgd = c0204000
>>> [    1.340000] [00000000] *pgd=00000000
>>> [    1.340000] Internal error: Oops: 80000005 [#1] SMP ARM
>>> [    1.340000] Modules linked in:
>>> [    1.340000] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.5.0-rc4-next-20160215 #1
>>> [    1.340000] Hardware name: Generic OMAP3-GP (Flattened Device Tree)
>>> [    1.340000] task: cb060000 ti: cb05a000 task.ti: cb05a000
>>> [    1.340000] PC is at 0x0
>>> [    1.340000] LR is at arch_send_call_function_single_ipi+0x34/0x38
>>
>> Since this is ARM, arch_send_call_function_single_ipi() looks like this:
>>
>> void arch_send_call_function_single_ipi(int cpu)
>> {
>>           smp_cross_call(cpumask_of(cpu), IPI_CALL_FUNC_SINGLE);
>> }
>>
>> so I'm not sure how the NULL pointer deref is possible even.
>>
>> The only thing coming to mind would be that cpumask_of(cpu) triggers
>> this, but I'm not sure how exactly that can happen.
>>
>> I need help from somebody who knows how this low-level stuff works on ARM.
>
> That's not even an SMP machine? I suspect a bunch of out of the
> 65 boot failures here are related to this:
>
> https://kernelci.org/boot/all/job/next/kernel/next-20160215/
>
> The SMP ones seem to fail with some regulator issues?
>

There is another problem, introduced with 6a0712f6f199e ("PM / OPP: Add
dev_pm_opp_set_rate()"). The kernelci boot log for next-20160212:omap3-overo-tobi
and others experience that problem.

Essentially, the code now assumes that a CPU clock always has a voltage
regulator attached to it, which is not correct. I sent out a patch to fix
that problem a minute ago.

Guenter

  reply	other threads:[~2016-02-15 19:40 UTC|newest]

Thread overview: 81+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2016-02-15 17:05 Crashes in arm qemu emulations due to 'cpufreq: governor: Replace timers with utilization ...' Guenter Roeck
2016-02-15 17:05 ` Guenter Roeck
2016-02-15 18:41 ` Rafael J. Wysocki
2016-02-15 18:41   ` Rafael J. Wysocki
2016-02-15 18:41   ` Rafael J. Wysocki
2016-02-15 18:49   ` Rafael J. Wysocki
2016-02-15 18:49     ` Rafael J. Wysocki
2016-02-15 18:49     ` Rafael J. Wysocki
2016-02-15 18:49   ` Marc Zyngier
2016-02-15 18:49     ` Marc Zyngier
2016-02-15 18:49     ` Marc Zyngier
2016-02-15 18:54     ` Rafael J. Wysocki
2016-02-15 18:54       ` Rafael J. Wysocki
2016-02-15 18:54       ` Rafael J. Wysocki
2016-02-15 19:03       ` Marc Zyngier
2016-02-15 19:03         ` Marc Zyngier
2016-02-15 19:03         ` Marc Zyngier
2016-02-15 19:12         ` Rafael J. Wysocki
2016-02-15 19:12           ` Rafael J. Wysocki
2016-02-15 19:12           ` Rafael J. Wysocki
2016-02-15 19:28           ` Rafael J. Wysocki
2016-02-15 19:28             ` Rafael J. Wysocki
2016-02-15 19:28             ` Rafael J. Wysocki
2016-02-15 19:42             ` Tony Lindgren
2016-02-15 19:42               ` Tony Lindgren
2016-02-15 19:42               ` Tony Lindgren
2016-02-15 19:46               ` Guenter Roeck
2016-02-15 19:46                 ` Guenter Roeck
2016-02-15 19:46                 ` Guenter Roeck
2016-02-15 19:57                 ` Tony Lindgren
2016-02-15 19:57                   ` Tony Lindgren
2016-02-15 19:57                   ` Tony Lindgren
2016-02-15 19:23         ` Russell King - ARM Linux
2016-02-15 19:23           ` Russell King - ARM Linux
2016-02-15 19:23           ` Russell King - ARM Linux
2016-02-15 20:41           ` Rafael J. Wysocki
2016-02-15 20:41             ` Rafael J. Wysocki
2016-02-15 20:41             ` Rafael J. Wysocki
2016-02-15 19:07       ` Russell King - ARM Linux
2016-02-15 19:07         ` Russell King - ARM Linux
2016-02-15 19:07         ` Russell King - ARM Linux
2016-02-15 19:01   ` Tony Lindgren
2016-02-15 19:01     ` Tony Lindgren
2016-02-15 19:01     ` Tony Lindgren
2016-02-15 19:40     ` Guenter Roeck [this message]
2016-02-15 19:40       ` Guenter Roeck
2016-02-15 19:40       ` Guenter Roeck
2016-02-15 19:58       ` Tony Lindgren
2016-02-15 19:58         ` Tony Lindgren
2016-02-15 19:58         ` Tony Lindgren
2016-02-15 20:09         ` Guenter Roeck
2016-02-15 20:09           ` Guenter Roeck
2016-02-15 20:09           ` Guenter Roeck
2016-02-15 20:38           ` Rafael J. Wysocki
2016-02-15 20:38             ` Rafael J. Wysocki
2016-02-15 20:38             ` Rafael J. Wysocki
2016-02-15 20:37         ` Rafael J. Wysocki
2016-02-15 20:37           ` Rafael J. Wysocki
2016-02-15 20:37           ` Rafael J. Wysocki
2016-02-15 21:36           ` Tony Lindgren
2016-02-15 21:36             ` Tony Lindgren
2016-02-15 21:36             ` Tony Lindgren
2016-02-16  1:38             ` Guenter Roeck
2016-02-16  1:38               ` Guenter Roeck
2016-02-16  1:38               ` Guenter Roeck
2016-02-15 19:02   ` Russell King - ARM Linux
2016-02-15 19:02     ` Russell King - ARM Linux
2016-02-15 19:02     ` Russell King - ARM Linux
2016-02-16  1:13   ` Viresh Kumar
2016-02-16  1:13     ` Viresh Kumar
2016-02-16  1:13     ` Viresh Kumar
2016-02-16  1:27     ` Rafael J. Wysocki
2016-02-16  1:27       ` Rafael J. Wysocki
2016-02-16  1:27       ` Rafael J. Wysocki
2016-02-16  1:36       ` Viresh Kumar
2016-02-16  1:36         ` Viresh Kumar
2016-02-16  1:36         ` Viresh Kumar
2016-02-15 22:29 ` Peter Maydell
2016-02-15 22:29   ` Peter Maydell
2016-02-15 23:19   ` Guenter Roeck
2016-02-15 23:19     ` Guenter Roeck

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=56C229A5.8020901@roeck-us.net \
    --to=linux@roeck-us.net \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-next@vger.kernel.org \
    --cc=linux-pm@vger.kernel.org \
    --cc=peterz@infradead.org \
    --cc=rafael.j.wysocki@intel.com \
    --cc=rafael@kernel.org \
    --cc=tony@atomide.com \
    --cc=viresh.kumar@linaro.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.