[PATCH 1/2] x86/apic: shutdown local APIC before I/O APIC during crash

[PATCH 1/2] x86/apic: shutdown local APIC before I/O APIC during crash

[Wei Jiangang]

commit <522e66464467> disables I/O APIC before shutdown of
the local APIC for both reboot and crash path.
and commit <2885432aaf15> declares that 'it still makes sense to
quiet IO APIC before disabling Local APIC'.

However, the former introduced a bug for crashdown.
If specify 'notsc' for capture-kernel, and then trigger crashdown.
The capture-kernel will be blocked at calibrate_delay_converge().

/* wait for "start of" clock tick */
ticks = jiffies;
while (ticks == jiffies)
    ; /* nothing */

serial console log as following,
............
[    0.000000] Linux version 4.7.0-rc2+ (root@localhost.localdomain)
(gcc version 4.8.2 20140120 (Red Hat 4.8.2-16) (GCC) ) #2 SMP Wed Jun
156
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-4.7.0-rc2+
root=/dev/mapper/centos-root ro rd.lvm.lv=centos/swap
vconsole.font=latarcyrheb-sun16 rd.lvm.lv=centos/root crashkernel=256M
vconsole.keymap=us console=tty0 console=ttyS0,115200n8 LANG=en_US.UTF-8
irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off numa=off
panic=10 rootflags=nofail acpi_no_memhotplug notsc
............
[    0.000000] tsc: Kernel compiled with CONFIG_X86_TSC, cannot disable
TSC completely
............
[    0.000000] clocksource: hpet: mask: 0xffffffff max_cycles:
0xffffffff, max_idle_ns: 133484882848 ns
[    0.000000] tsc: Fast TSC calibration using PIT
[    0.000000] tsc: Detected 3192.714 MHz processor
[    0.000000] Calibrating delay loop...

The bug remains and unsolved for a long time, since 2013.
I find the arch-criminal by bisect.

The commit <522e66464467> used to fix erratum AVR31 for "Intel Atom
Processor C2000 Product Family Specification Update".
You can find the doc at http://www.intel.com/content/dam/www/public/us
/en/documents/specification-updates/atom-c2000-family-spec-update.pdf.

IMO,
It doesn't make sense that change the order of disabling between
I/O APIC and local APIC just for a certain model C2000.
And I couldn't find any related descriptions for Intel 64 and IA-32 Arch.

so, I want to revert the crash part of commit <522e66464467>.

Signed-off-by: Wei Jiangang <weijg.fnst@cn.fujitsu.com>
---
 arch/x86/kernel/crash.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
index 9ef978d69c22..9aa0235c1f7d 100644
--- a/arch/x86/kernel/crash.c
+++ b/arch/x86/kernel/crash.c
@@ -179,12 +179,13 @@ void native_machine_crash_shutdown(struct pt_regs *regs)
 	 */
 	cpu_emergency_stop_pt();
 
+	lapic_shutdown();
 #ifdef CONFIG_X86_IO_APIC
 	/* Prevent crash_kexec() from deadlocking on ioapic_lock. */
 	ioapic_zap_locks();
 	disable_IO_APIC();
 #endif
-	lapic_shutdown();
+
 #ifdef CONFIG_HPET_TIMER
 	hpet_disable();
 #endif
-- 
1.9.3

[PATCH 2/2] time/tick-schede: fix typos

[Wei Jiangang]

When investigating kdump's failure with 'notsc' and
jiffies not incrementing, I found several spelling mistakes

Signed-off-by: Wei Jiangang <weijg.fnst@cn.fujitsu.com>
---
 kernel/time/tick-sched.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 536ada80f6dd..6d83e9c4a302 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -61,7 +61,7 @@ static void tick_do_update_jiffies64(ktime_t now)
 	if (delta.tv64 < tick_period.tv64)
 		return;
 
-	/* Reevalute with jiffies_lock held */
+	/* Reevaluate with jiffies_lock held */
 	write_seqlock(&jiffies_lock);
 
 	delta = ktime_sub(now, last_jiffies_update);
@@ -117,7 +117,7 @@ static void tick_sched_do_timer(ktime_t now)
 	/*
 	 * Check if the do_timer duty was dropped. We don't care about
 	 * concurrency: This happens only when the cpu in charge went
-	 * into a long sleep. If two cpus happen to assign themself to
+	 * into a long sleep. If two cpus happen to assign themselves to
 	 * this duty, then the jiffies update is still serialized by
 	 * jiffies_lock.
 	 */
@@ -571,7 +571,7 @@ static ktime_t tick_nohz_start_idle(struct tick_sched *ts)
  * @last_update_time: variable to store update time in. Do not update
  * counters if NULL.
  *
- * Return the cummulative idle time (since boot) for a given
+ * Return the cumulative idle time (since boot) for a given
  * CPU, in microseconds.
  *
  * This time is measured via accounting rather than sampling,
@@ -612,7 +612,7 @@ EXPORT_SYMBOL_GPL(get_cpu_idle_time_us);
  * @last_update_time: variable to store update time in. Do not update
  * counters if NULL.
  *
- * Return the cummulative iowait time (since boot) for a given
+ * Return the cumulative iowait time (since boot) for a given
  * CPU, in microseconds.
  *
  * This time is measured via accounting rather than sampling,
@@ -733,7 +733,7 @@ static ktime_t tick_nohz_stop_sched_tick(struct tick_sched *ts,
 	 * do_timer() never invoked. Keep track of the fact that it
 	 * was the one which had the do_timer() duty last. If this cpu
 	 * is the one which had the do_timer() duty last, we limit the
-	 * sleep time to the timekeeping max_deferement value.
+	 * sleep time to the timekeeping max_deferment value.
 	 * Otherwise we can sleep as long as we want.
 	 */
 	delta = timekeeping_max_deferment();
-- 
1.9.3

Re: [PATCH 1/2] x86/apic: shutdown local APIC before I/O APIC during crash

[Ingo Molnar]

* Wei Jiangang <weijg.fnst@cn.fujitsu.com> wrote:

> commit <522e66464467> disables I/O APIC before shutdown of
> the local APIC for both reboot and crash path.
> and commit <2885432aaf15> declares that 'it still makes sense to
> quiet IO APIC before disabling Local APIC'.

That's not how we refer to commits in changelogs.

> However, the former introduced a bug for crashdown.

What is 'crashdown'? It's not referred to in the kernel source even once.

> If specify 'notsc' for capture-kernel, and then trigger crashdown.
> The capture-kernel will be blocked at calibrate_delay_converge().

This is a more readable way of saying the same:

  If we specify the 'notsc' boot parameter for the dump-capture kernel,
  and then trigger a crash-down, then the dump-capture kernel will hang
  in calibrate_delay_converge():

(Assuming the changelog first explains what a 'crash-down' is.)

> /* wait for "start of" clock tick */
> ticks = jiffies;
> while (ticks == jiffies)
>     ; /* nothing */

Plase align quoted code to the right with at least a single tab.

> serial console log as following,

  serial log of the hang is as follows:

> ............
> [    0.000000] Linux version 4.7.0-rc2+ (root@localhost.localdomain)
> (gcc version 4.8.2 20140120 (Red Hat 4.8.2-16) (GCC) ) #2 SMP Wed Jun
> 156
> [    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-4.7.0-rc2+
> root=/dev/mapper/centos-root ro rd.lvm.lv=centos/swap
> vconsole.font=latarcyrheb-sun16 rd.lvm.lv=centos/root crashkernel=256M
> vconsole.keymap=us console=tty0 console=ttyS0,115200n8 LANG=en_US.UTF-8
> irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off numa=off
> panic=10 rootflags=nofail acpi_no_memhotplug notsc
> ............
> [    0.000000] tsc: Kernel compiled with CONFIG_X86_TSC, cannot disable
> TSC completely
> ............
> [    0.000000] clocksource: hpet: mask: 0xffffffff max_cycles:
> 0xffffffff, max_idle_ns: 133484882848 ns
> [    0.000000] tsc: Fast TSC calibration using PIT
> [    0.000000] tsc: Detected 3192.714 MHz processor
> [    0.000000] Calibrating delay loop...

Just quote the last few lines and skip the useless timestamp column. Also, please 
right-align this too.

> The bug remains and unsolved for a long time, since 2013.
> I find the arch-criminal by bisect.

What is an arch-criminal? Did you want to say:

  The bug has been introduced in 2013. I found the buggy commit via bisection.

?

> The commit <522e66464467> used to fix erratum AVR31 for "Intel Atom
> Processor C2000 Product Family Specification Update".
> You can find the doc at http://www.intel.com/content/dam/www/public/us
> /en/documents/specification-updates/atom-c2000-family-spec-update.pdf.
>
> IMO,
> It doesn't make sense that change the order of disabling between
> I/O APIC and local APIC just for a certain model C2000.
> And I couldn't find any related descriptions for Intel 64 and IA-32 Arch.
> 
> so, I want to revert the crash part of commit <522e66464467>.

So why does the crashdump kernel hang in calibrate_delay_converge()?

To me it appears this is a weakness in the crashdump kernel: it is unable to boot 
if we crash the original host system in a particular hardware state, right?

By reverting this change we'll just paper over the bug and re-introduce the bug 
that can result in certain CPUs hanging if the IO-APIC sends an APIC message if 
the lapic is disabled prematurely.

Thanks,

	Ingo

[tip:timers/core] timers/nohz: Fix several typos

[tip-bot for Wei Jiangang]

Commit-ID:  6168f8ed01dc46a277908938294f1132d723f58d
Gitweb:     http://git.kernel.org/tip/6168f8ed01dc46a277908938294f1132d723f58d
Author:     Wei Jiangang <weijg.fnst@cn.fujitsu.com>
AuthorDate: Wed, 29 Jun 2016 12:51:50 +0800
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Fri, 1 Jul 2016 12:39:22 +0200

timers/nohz: Fix several typos

Signed-off-by: Wei Jiangang <weijg.fnst@cn.fujitsu.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: fenghua.yu@intel.com
Link: http://lkml.kernel.org/r/1467175910-2966-2-git-send-email-weijg.fnst@cn.fujitsu.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 kernel/time/tick-sched.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 536ada8..6d83e9c 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -61,7 +61,7 @@ static void tick_do_update_jiffies64(ktime_t now)
 	if (delta.tv64 < tick_period.tv64)
 		return;
 
-	/* Reevalute with jiffies_lock held */
+	/* Reevaluate with jiffies_lock held */
 	write_seqlock(&jiffies_lock);
 
 	delta = ktime_sub(now, last_jiffies_update);
@@ -117,7 +117,7 @@ static void tick_sched_do_timer(ktime_t now)
 	/*
 	 * Check if the do_timer duty was dropped. We don't care about
 	 * concurrency: This happens only when the cpu in charge went
-	 * into a long sleep. If two cpus happen to assign themself to
+	 * into a long sleep. If two cpus happen to assign themselves to
 	 * this duty, then the jiffies update is still serialized by
 	 * jiffies_lock.
 	 */
@@ -571,7 +571,7 @@ static ktime_t tick_nohz_start_idle(struct tick_sched *ts)
  * @last_update_time: variable to store update time in. Do not update
  * counters if NULL.
  *
- * Return the cummulative idle time (since boot) for a given
+ * Return the cumulative idle time (since boot) for a given
  * CPU, in microseconds.
  *
  * This time is measured via accounting rather than sampling,
@@ -612,7 +612,7 @@ EXPORT_SYMBOL_GPL(get_cpu_idle_time_us);
  * @last_update_time: variable to store update time in. Do not update
  * counters if NULL.
  *
- * Return the cummulative iowait time (since boot) for a given
+ * Return the cumulative iowait time (since boot) for a given
  * CPU, in microseconds.
  *
  * This time is measured via accounting rather than sampling,
@@ -733,7 +733,7 @@ static ktime_t tick_nohz_stop_sched_tick(struct tick_sched *ts,
 	 * do_timer() never invoked. Keep track of the fact that it
 	 * was the one which had the do_timer() duty last. If this cpu
 	 * is the one which had the do_timer() duty last, we limit the
-	 * sleep time to the timekeeping max_deferement value.
+	 * sleep time to the timekeeping max_deferment value.
 	 * Otherwise we can sleep as long as we want.
 	 */
 	delta = timekeeping_max_deferment();

Re: [PATCH 1/2] x86/apic: shutdown local APIC before I/O APIC during crash

[Wei, Jiangang]

Hi, Ingo

Thanks for your comments firstly.

On Fri, 2016-07-01 at 12:36 +0200, Ingo Molnar wrote:
> * Wei Jiangang <weijg.fnst@cn.fujitsu.com> wrote:
> 
> > commit <522e66464467> disables I/O APIC before shutdown of
> > the local APIC for both reboot and crash path.
> > and commit <2885432aaf15> declares that 'it still makes sense to
> > quiet IO APIC before disabling Local APIC'.
> 
> That's not how we refer to commits in changelogs.
> 
OK, I will fix it and pay attention to it in the following patch.

> > However, the former introduced a bug for crashdown.
> 
> What is 'crashdown'? It's not referred to in the kernel source even once.

well, I mean ...
 
If we trigger kernel panic with the following commands, the capture
kernel should boot normally and captures the dump image. 

#echo 1 > /proc/sys/kernel/sysrq
#echo c > /proc/sysrq-trigger

But due to commit 522e66464467 changes the APIC shutdown sequence in
native_machine_crash_shutdown(), the capture kernel doesn't boot
normally and
hang in calibrate_delay_converge(), waiting for the jiffies to be
updated.

BTW, without commit 522e66464467, the capture kernel works well.

> 
> > If specify 'notsc' for capture-kernel, and then trigger crashdown.
> > The capture-kernel will be blocked at calibrate_delay_converge().
> 
> This is a more readable way of saying the same:
> 
>   If we specify the 'notsc' boot parameter for the dump-capture kernel,
>   and then trigger a crash-down, then the dump-capture kernel will hang
>   in calibrate_delay_converge():
> 
> (Assuming the changelog first explains what a 'crash-down' is.)
> 
> > /* wait for "start of" clock tick */
> > ticks = jiffies;
> > while (ticks == jiffies)
> >     ; /* nothing */
> 
> Plase align quoted code to the right with at least a single tab.
> 
OK
> > serial console log as following,
> 
>   serial log of the hang is as follows:
> 
> > ............
> > [    0.000000] Linux version 4.7.0-rc2+ (root@localhost.localdomain)
> > (gcc version 4.8.2 20140120 (Red Hat 4.8.2-16) (GCC) ) #2 SMP Wed Jun
> > 156
> > [    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-4.7.0-rc2+
> > root=/dev/mapper/centos-root ro rd.lvm.lv=centos/swap
> > vconsole.font=latarcyrheb-sun16 rd.lvm.lv=centos/root crashkernel=256M
> > vconsole.keymap=us console=tty0 console=ttyS0,115200n8 LANG=en_US.UTF-8
> > irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off numa=off
> > panic=10 rootflags=nofail acpi_no_memhotplug notsc
> > ............
> > [    0.000000] tsc: Kernel compiled with CONFIG_X86_TSC, cannot disable
> > TSC completely
> > ............
> > [    0.000000] clocksource: hpet: mask: 0xffffffff max_cycles:
> > 0xffffffff, max_idle_ns: 133484882848 ns
> > [    0.000000] tsc: Fast TSC calibration using PIT
> > [    0.000000] tsc: Detected 3192.714 MHz processor
> > [    0.000000] Calibrating delay loop...
> 
> Just quote the last few lines and skip the useless timestamp column. Also, please 
> right-align this too.
OK
> 
> > The bug remains and unsolved for a long time, since 2013.
> > I find the arch-criminal by bisect.
> 
> What is an arch-criminal? Did you want to say:
> 
>   The bug has been introduced in 2013. I found the buggy commit via bisection.
> 
> ?
Yes, That's what i want to say.

> 
> > The commit <522e66464467> used to fix erratum AVR31 for "Intel Atom
> > Processor C2000 Product Family Specification Update".
> > You can find the doc at http://www.intel.com/content/dam/www/public/us
> > /en/documents/specification-updates/atom-c2000-family-spec-update.pdf.
> >
> > IMO,
> > It doesn't make sense that change the order of disabling between
> > I/O APIC and local APIC just for a certain model C2000.
> > And I couldn't find any related descriptions for Intel 64 and IA-32 Arch.
> > 
> > so, I want to revert the crash part of commit <522e66464467>.
> 
> So why does the crashdump kernel hang in calibrate_delay_converge()?

The jiffies value doesn't increase, which causes the capture kernel hang
in calibrate_delay_converge().

It seems that there's a relationship with the shutdown(disable) order
between IO APIC and local APIC.   I'm not sure of this point ....

One thing for sure by debugging is that do_timer() is not called while
capture kernel boots up. I suspect the timer interrupts (irq0) is not
passed to cpu by APIC.

> 
> To me it appears this is a weakness in the crashdump kernel: it is unable to boot 
> if we crash the original host system in a particular hardware state, right?

Maybe you're right ...
I specify 'notsc' only for capture-kernel, not the original host
system(first kernel).

And I suspect the APIC shutdown sequence in first kernel maybe bring
some bad influence on capture kernel.
I need to do more investigation.
Do you have any advice?
Thanks in advance.

Wei
> By reverting this change we'll just paper over the bug and re-introduce the bug 
> that can result in certain CPUs hanging if the IO-APIC sends an APIC message if 
> the lapic is disabled prematurely.
> Thanks,
> 
> 	Ingo
> 
>