From mboxrd@z Thu Jan  1 00:00:00 1970
From: Amit Shah <amit.shah@redhat.com>
Subject: Re: x86: kvmclock: abstract save/restore sched_clock_state
Date: Thu, 9 Feb 2012 17:57:42 +0530
Message-ID: <20120209122742.GA25505@amit.redhat.com>
References: <20120207210542.GC20618@amt.cnet>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Cc: kvm@vger.kernel.org, tglx@linutronix.de, mingo@redhat.com,
	hpa@zytor.com, x86@kernel.org, johnstul@us.ibm.com,
	riel@redhat.com, avi@redhat.com, imammedo@redhat.com
To: Marcelo Tosatti <mtosatti@redhat.com>
Return-path: <kvm-owner@vger.kernel.org>
Received: from mx1.redhat.com ([209.132.183.28]:19163 "EHLO mx1.redhat.com"
	rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
	id S1753372Ab2BIM2C (ORCPT <rfc822;kvm@vger.kernel.org>);
	Thu, 9 Feb 2012 07:28:02 -0500
Content-Disposition: inline
In-Reply-To: <20120207210542.GC20618@amt.cnet>
Sender: kvm-owner@vger.kernel.org
List-ID: <kvm.vger.kernel.org>

On (Tue) 07 Feb 2012 [19:05:42], Marcelo Tosatti wrote:
> 
> Upon resume from hibernation, CPU 0's hvclock area contains the old
> values for system_time and tsc_timestamp. It is necessary for the
> hypervisor to update these values with uptodate ones before the CPU uses
> them.
> 
> Abstract TSC's save/restore sched_clock_state functions and use
> restore_state to write to KVM_SYSTEM_TIME MSR, forcing an update.
> 
> Fixes suspend-to-disk with kvmclock.

There are stalls after resume, see trace below.

./x86_64-softmmu/qemu-system-x86_64 -kernel ~/src/linux/arch/x86/boot/bzImage  -append 'root=/dev/vda1 console=tty0 console=ttyS0 no_console_suspend' -drive file=/guests/f14-suspend.qcow2,if=none,id=dr0 -device virtio-blk-pci,drive=dr0 -net nic,model=virtio -net user  -serial stdio  -enable-kvm -m 512  -cpu host -smp 4

<The following is on resume:>

Disabling non-boot CPUs ...
CPU 1 is now offline
CPU 2 is now offline
CPU 3 is now offline
lockdep: fixing up alternatives.
kvm-clock: cpu 0, msr 0:1f1d19c1, primary cpu clock, resume
PM: Restoring platform NVS memory
Enabling non-boot CPUs ...
lockdep: fixing up alternatives.
Booting Node 0 Processor 1 APIC 0x1
Calibrating delay loop (skipped) already calibrated this CPU
INFO: rcu_sched detected stalls on CPUs/tasks: { 1} (detected by 0, t=314192 jiffies)
Pid: 662, comm: bash Not tainted 3.3.0-rc2+ #293
Call Trace:
 <IRQ>  [<ffffffff8108b6c9>] __rcu_pending+0x268/0x3c5
 [<ffffffff8108bef5>] rcu_check_callbacks+0xab/0x108
 [<ffffffff81031208>] update_process_times+0x3f/0x75
 [<ffffffff810649b5>] tick_sched_timer+0x6d/0x8c
 [<ffffffff81042e68>] __run_hrtimer+0xc9/0x15c
 [<ffffffff81064948>] ? tick_nohz_handler+0xd5/0xd5
 [<ffffffff8104359b>] hrtimer_interrupt+0xea/0x1b4
 [<ffffffff81316192>] smp_apic_timer_interrupt+0x76/0x89
 [<ffffffff81315833>] apic_timer_interrupt+0x73/0x80
 <EOI>  [<ffffffff8130b9ec>] ? arch_local_irq_restore+0x6/0xd
 [<ffffffff8130840c>] native_cpu_up+0x108/0x127
 [<ffffffff81308f2b>] _cpu_up+0x92/0xfc
 [<ffffffff812facf7>] enable_nonboot_cpus+0x4d/0xb2
 [<ffffffff81059836>] hibernation_snapshot+0x1f3/0x2a0
 [<ffffffff81044406>] ? cleanup_srcu_struct+0x52/0x58
 [<ffffffff81059e6d>] hibernate+0x97/0x196
 [<ffffffff81057eb0>] state_store+0x5c/0x106
 [<ffffffff811af49b>] kobj_attr_store+0x17/0x19
 [<ffffffff811368fa>] sysfs_write_file+0x10e/0x14a
 [<ffffffff810eb5f6>] vfs_write+0xab/0xd2
 [<ffffffff810ec955>] ? fget_light+0x3a/0xa1
 [<ffffffff810eb7ad>] sys_write+0x4d/0x74
 [<ffffffff81314ce9>] system_call_fastpath+0x16/0x1b
INFO: rcu_preempt detected stalls on CPUs/tasks: { 1} (detected by 0, t=314193 jiffies)
Pid: 662, comm: bash Not tainted 3.3.0-rc2+ #293
Call Trace:
 <IRQ>  [<ffffffff8108b6c9>] __rcu_pending+0x268/0x3c5
 [<ffffffff8108bf33>] rcu_check_callbacks+0xe9/0x108
 [<ffffffff81031208>] update_process_times+0x3f/0x75
 [<ffffffff810649b5>] tick_sched_timer+0x6d/0x8c
 [<ffffffff81042e68>] __run_hrtimer+0xc9/0x15c
 [<ffffffff81064948>] ? tick_nohz_handler+0xd5/0xd5
 [<ffffffff8104359b>] hrtimer_interrupt+0xea/0x1b4
 [<ffffffff81316192>] smp_apic_timer_interrupt+0x76/0x89
 [<ffffffff81315833>] apic_timer_interrupt+0x73/0x80
 <EOI>  [<ffffffff8130b9ec>] ? arch_local_irq_restore+0x6/0xd
 [<ffffffff8130840c>] native_cpu_up+0x108/0x127
 [<ffffffff81308f2b>] _cpu_up+0x92/0xfc
 [<ffffffff812facf7>] enable_nonboot_cpus+0x4d/0xb2
 [<ffffffff81059836>] hibernation_snapshot+0x1f3/0x2a0
 [<ffffffff81044406>] ? cleanup_srcu_struct+0x52/0x58
 [<ffffffff81059e6d>] hibernate+0x97/0x196
 [<ffffffff81057eb0>] state_store+0x5c/0x106
 [<ffffffff811af49b>] kobj_attr_store+0x17/0x19
 [<ffffffff811368fa>] sysfs_write_file+0x10e/0x14a
 [<ffffffff810eb5f6>] vfs_write+0xab/0xd2
 [<ffffffff810ec955>] ? fget_light+0x3a/0xa1
 [<ffffffff810eb7ad>] sys_write+0x4d/0x74
 [<ffffffff81314ce9>] system_call_fastpath+0x16/0x1b
INFO: rcu_sched detected stalls on CPUs/tasks: { 1} (detected by 0, t=1713579840232 jiffies)
Pid: 662, comm: bash Not tainted 3.3.0-rc2+ #293
Call Trace:
 <IRQ>  [<ffffffff8108b6c9>] __rcu_pending+0x268/0x3c5
 [<ffffffff8108bef5>] rcu_check_callbacks+0xab/0x108
 [<ffffffff81031208>] update_process_times+0x3f/0x75
 [<ffffffff810649b5>] tick_sched_timer+0x6d/0x8c
 [<ffffffff81042e68>] __run_hrtimer+0xc9/0x15c
 [<ffffffff81064948>] ? tick_nohz_handler+0xd5/0xd5
 [<ffffffff8104359b>] hrtimer_interrupt+0xea/0x1b4
 [<ffffffff81316192>] smp_apic_timer_interrupt+0x76/0x89
 [<ffffffff81315833>] apic_timer_interrupt+0x73/0x80
 <EOI>  [<ffffffff8130b9ec>] ? arch_local_irq_restore+0x6/0xd
 [<ffffffff8130840c>] native_cpu_up+0x108/0x127
 [<ffffffff81308f2b>] _cpu_up+0x92/0xfc
 [<ffffffff812facf7>] enable_nonboot_cpus+0x4d/0xb2
 [<ffffffff81059836>] hibernation_snapshot+0x1f3/0x2a0
 [<ffffffff81044406>] ? cleanup_srcu_struct+0x52/0x58
 [<ffffffff81059e6d>] hibernate+0x97/0x196
 [<ffffffff81057eb0>] state_store+0x5c/0x106
 [<ffffffff811af49b>] kobj_attr_store+0x17/0x19
 [<ffffffff811368fa>] sysfs_write_file+0x10e/0x14a
 [<ffffffff810eb5f6>] vfs_write+0xab/0xd2
 [<ffffffff810ec955>] ? fget_light+0x3a/0xa1
 [<ffffffff810eb7ad>] sys_write+0x4d/0x74
 [<ffffffff81314ce9>] system_call_fastpath+0x16/0x1b
INFO: rcu_preempt detected stalls on CPUs/tasks: { 1} (detected by 0, t=1713579840233 jiffies)
Pid: 662, comm: bash Not tainted 3.3.0-rc2+ #293
Call Trace:
 <IRQ>  [<ffffffff8108b6c9>] __rcu_pending+0x268/0x3c5
 [<ffffffff8108bf33>] rcu_check_callbacks+0xe9/0x108
 [<ffffffff81031208>] update_process_times+0x3f/0x75
 [<ffffffff810649b5>] tick_sched_timer+0x6d/0x8c
 [<ffffffff81042e68>] __run_hrtimer+0xc9/0x15c
 [<ffffffff81064948>] ? tick_nohz_handler+0xd5/0xd5
 [<ffffffff8104359b>] hrtimer_interrupt+0xea/0x1b4
 [<ffffffff81316192>] smp_apic_timer_interrupt+0x76/0x89
 [<ffffffff81315833>] apic_timer_interrupt+0x73/0x80
 <EOI>  [<ffffffff8130b9ec>] ? arch_local_irq_restore+0x6/0xd
 [<ffffffff8130840c>] native_cpu_up+0x108/0x127
 [<ffffffff81308f2b>] _cpu_up+0x92/0xfc
 [<ffffffff812facf7>] enable_nonboot_cpus+0x4d/0xb2
 [<ffffffff81059836>] hibernation_snapshot+0x1f3/0x2a0
 [<ffffffff81044406>] ? cleanup_srcu_struct+0x52/0x58
 [<ffffffff81059e6d>] hibernate+0x97/0x196
 [<ffffffff81057eb0>] state_store+0x5c/0x106
 [<ffffffff811af49b>] kobj_attr_store+0x17/0x19
 [<ffffffff811368fa>] sysfs_write_file+0x10e/0x14a
 [<ffffffff810eb5f6>] vfs_write+0xab/0xd2
 [<ffffffff810ec955>] ? fget_light+0x3a/0xa1
 [<ffffffff810eb7ad>] sys_write+0x4d/0x74
 [<ffffffff81314ce9>] system_call_fastpath+0x16/0x1b


		Amit