From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67BC5C4332B for ; Wed, 3 Mar 2021 17:11:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3658864EDF for ; Wed, 3 Mar 2021 17:11:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1385107AbhCCQ6Q (ORCPT ); Wed, 3 Mar 2021 11:58:16 -0500 Received: from szxga05-in.huawei.com ([45.249.212.191]:13049 "EHLO szxga05-in.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234738AbhCCM3t (ORCPT ); Wed, 3 Mar 2021 07:29:49 -0500 Received: from DGGEMS410-HUB.china.huawei.com (unknown [172.30.72.58]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4DrCqw2qlCzMYFX; Wed, 3 Mar 2021 20:25:00 +0800 (CST) Received: from localhost (10.174.150.118) by DGGEMS410-HUB.china.huawei.com (10.3.19.210) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Mar 2021 20:26:59 +0800 From: To: CC: , , , Zhuang Yanying Subject: [PATCH] KVM: x86: fix cpu hang due to tsc adjustment when kvmclock in use Date: Wed, 3 Mar 2021 20:26:57 +0800 Message-ID: <20210303122657.23400-1-ann.zhuangyanying@huawei.com> X-Mailer: git-send-email 2.21.0.windows.1 MIME-Version: 1.0 Content-Transfer-Encoding: 7BIT Content-Type: text/plain; charset=US-ASCII X-Originating-IP: [10.174.150.118] X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Zhuang Yanying If the TSC frequency of the VM is not equal to the host, hot-plugging vCPU will cause the VM to be hang. The time of hang depends on the current TSC value of the VM. System time calculation of kvmclock is based on (tsc_timestamp, system_time), and adjusted by delta ( = rdtsc_ordered() - src->tsc_timestamp).The tsc of the hotplugged cpu is initialized to 0, which will trigger check_tsc_sync_target() to adjust the tsc of the hotplugged cpu according to another online cpu, that is, rdtsc_ordered() will change abruptly to a large value. Then system time based on kvmclock is modified at the same time. So after modifying the tsc offset, update vcpu->hv_clock immediately. Signed-off-by: Zhuang Yanying --- Host: Intel(R) Xeon(R) Gold 6161 CPU @ 2.20GHz linux-5.11 qemu-5.1 Guest: Centos8.1 (4.18.0-147.el8.x86_64) After Hotplug cpu, vm hang for 290s: [ 283.224026] CPU3 has been hot-added [ 283.226118] smpboot: Booting Node 0 Processor 3 APIC 0x3 [ 283.226964] kvm-clock: cpu 3, msr 9e5e010c1, secondary cpu clock [ 283.247200] TSC ADJUST compensate: CPU3 observed 867529151959 warp. Adjust: 867529151959 [ 572.445543] KVM setup async PF for cpu 3 [ 572.446412] kvm-stealtime: cpu 3, msr a16ce5040 [ 572.448108] Will online and init hotplugged CPU: 3 Feb 27 18:47:28 localhost kernel: CPU3 has been hot-added Feb 27 18:47:28 localhost kernel: smpboot: Booting Node 0 Processor 3 APIC 0x3 Feb 27 18:47:28 localhost kernel: kvm-clock: cpu 3, msr 9e5e010c1, secondary cpu clock Feb 27 18:47:28 localhost kernel: TSC ADJUST compensate: CPU3 observed 867529151959 warp. Adjust: 867529151959 Feb 27 18:47:28 localhost kernel: KVM setup async PF for cpu 3 Feb 27 18:47:28 localhost kernel: kvm-stealtime: cpu 3, msr a16ce5040 Feb 27 18:47:28 localhost kernel: Will online and init hotplugged CPU: 3 Feb 27 18:47:28 localhost systemd[1]: Started /usr/lib/udev/kdump-udev-throttler. [ 572.495181] clocksource: timekeeping watchdog on CPU2: Marking clocksource 'tsc' as unstable because the skew is too large: [ 572.495181] clocksource: 'kvm-clock' wd_now: 86ab1286a2 wd_last: 4344b44d09 mask: ffffffffffffffff [ 572.495181] clocksource: 'tsc' cs_now: ca313c563b cs_last: c9d88b54d2 mask: ffffffffffffffff [ 572.495181] tsc: Marking TSC unstable due to clocksource watchdog [ 572.495181] clocksource: Switched to clocksource kvm-clock Feb 27 18:47:28 localhost kernel: clocksource: timekeeping watchdog on CPU2: Marking clocksource 'tsc' as unstable because the skew Feb 27 18:47:28 localhost kernel: clocksource: 'kvm-clock' wd_now: 86ab1286a2 wd_last: 4344b44d09 mask: ffffff Feb 27 18:47:28 localhost kernel: clocksource: 'tsc' cs_now: ca313c563b cs_last: c9d88b54d2 mask: ffffffffffff Feb 27 18:47:28 localhost kernel: tsc: Marking TSC unstable due to clocksource watchdog Feb 27 18:47:28 localhost kernel: clocksource: Switched to clocksource kvm-clock Feb 27 18:47:28 localhost systemd[1]: Started Getty on tty2. Feb 27 18:47:29 localhost kdump-udev-throttler[3530]: kexec: unloaded kdump kernel Feb 27 18:47:29 localhost kdump-udev-throttler[3530]: Stopping kdump: [OK] Feb 27 18:47:29 localhost kdump-udev-throttler[3530]: kexec: loaded kdump kernel Feb 27 18:47:29 localhost kdump-udev-throttler[3530]: Starting kdump: [OK] --- arch/x86/kvm/x86.c | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 3712bb5245eb..429206d65989 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -3078,6 +3078,7 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) if (!msr_info->host_initiated) { s64 adj = data - vcpu->arch.ia32_tsc_adjust_msr; adjust_tsc_offset_guest(vcpu, adj); + kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu); } vcpu->arch.ia32_tsc_adjust_msr = data; } -- 2.23.0