From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99200C12002 for ; Fri, 16 Jul 2021 21:26:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 79D54613F5 for ; Fri, 16 Jul 2021 21:26:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236182AbhGPV3j (ORCPT ); Fri, 16 Jul 2021 17:29:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52726 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235844AbhGPV3h (ORCPT ); Fri, 16 Jul 2021 17:29:37 -0400 Received: from mail-il1-x149.google.com (mail-il1-x149.google.com [IPv6:2607:f8b0:4864:20::149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7B105C06175F for ; Fri, 16 Jul 2021 14:26:42 -0700 (PDT) Received: by mail-il1-x149.google.com with SMTP id i11-20020a056e02004bb029020269661e11so6138195ilr.13 for ; Fri, 16 Jul 2021 14:26:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=MJZQeREwgEv7K7M0v6j7xlL8wKPecrHyLbDAmoCt7vg=; b=lkAEWDv031NoddTxAys9lfBfrA6KmwD/4l+4g8DRaSQeMr2b6P7DqkuwKdvx0HMo6G tdsNTWAJWw2cGxWbVbAhCkWjzJgJLYdVkXEU+fi3KNaaz5g6Q+H6BKLzxxsLedbLVKJK Gbt/ojltMPlVEroSq0KfNzGz3FxSlP1OdauulQjJpaZMGjiYgXA7/nyg4xkMLm9JRd/1 g+173zUemf15PWK1Ui7ESvZMmG6zDxvfaNQ88BJs3b3mAZQCZ7c0WAjJ1L4g5B7oW/E3 JN2EJE4r8B4O7FhHutlDfqxhmvgYFEL3qAN7+T7t4sopFcYK8AQhmnOnlfuH99y2lTZx +QAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=MJZQeREwgEv7K7M0v6j7xlL8wKPecrHyLbDAmoCt7vg=; b=IgLidSfitiAmfFoODVKwcTeQWBC6xNypmAR/CxyP/uI7yWAY3hQB+bopDd0PSYZVnC +SoOF1OApd+mfrV85JpKGDj45HN617lvMpBAhp96o/kZ6DRDHI0RzSVaQzkBPEgOjlUQ gbDXQnp705EdAJmRvtJ61SSpIsEJdSNK1deoQBJvNeswpHgxkgI0hJfu54FJ3uEz6xs4 WhI73DJ1LTPNXhLmJYOoSJtfW7B9YcK2bMcH2XOZeSKZfa0EJBvC1UaB+JW1ejRtJr7Y C8p87sGjw4HvwbLJIiTjqpF4nntCMQ/DClHQxaLWp0RRzBDxn+R6C7pfFj+Za9jKxp0J q9fg== X-Gm-Message-State: AOAM531Vq+FqBe2Xlpctkuw+SSBIX2QXjA2RB47KZ/pVvLeNs0MzIcjp 1EIK+8AZeUPvFMaKIVPHSsRFUvQPmiSF1N/Y7lN1k0Sl4Owo+eecuO4MjJZM0oNvdcEcyZ7Z3rC iS4k62v7errLudWs9vj3botei8ABN49aaDf8B5H3ZSyaJEhkXAmKy9ek2CA== X-Google-Smtp-Source: ABdhPJwoxs4yEclO5kH0ab60ofNKeHKBJYurkbO5xn0o+KYDpWbntfW2OmI3RJILTEEV5yzbIG6R2eD3LJA= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a05:6602:2f09:: with SMTP id q9mr2363577iow.196.1626470801820; Fri, 16 Jul 2021 14:26:41 -0700 (PDT) Date: Fri, 16 Jul 2021 21:26:19 +0000 In-Reply-To: <20210716212629.2232756-1-oupton@google.com> Message-Id: <20210716212629.2232756-3-oupton@google.com> Mime-Version: 1.0 References: <20210716212629.2232756-1-oupton@google.com> X-Mailer: git-send-email 2.32.0.402.g57bb445576-goog Subject: [PATCH v2 02/12] KVM: x86: Refactor tsc synchronization code From: Oliver Upton To: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu Cc: Paolo Bonzini , Sean Christopherson , Marc Zyngier , Peter Shier , Jim Mattson , David Matlack , Ricardo Koller , Jing Zhang , Raghavendra Rao Anata , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Oliver Upton Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Refactor kvm_synchronize_tsc to make a new function that allows callers to specify TSC parameters (offset, value, nanoseconds, etc.) explicitly for the sake of participating in TSC synchronization. This changes the locking semantics around TSC writes. Writes to the TSC will now take the pvclock gtod lock while holding the tsc write lock, whereas before these locks were disjoint. Reviewed-by: David Matlack Signed-off-by: Oliver Upton --- Documentation/virt/kvm/locking.rst | 11 +++ arch/x86/kvm/x86.c | 106 +++++++++++++++++------------ 2 files changed, 74 insertions(+), 43 deletions(-) diff --git a/Documentation/virt/kvm/locking.rst b/Documentation/virt/kvm/locking.rst index 35eca377543d..ac62e1c76694 100644 --- a/Documentation/virt/kvm/locking.rst +++ b/Documentation/virt/kvm/locking.rst @@ -30,6 +30,9 @@ On x86: holding kvm->arch.mmu_lock (typically with ``read_lock``, otherwise there's no need to take kvm->arch.tdp_mmu_pages_lock at all). +- kvm->arch.tsc_write_lock is taken outside + kvm->arch.pvclock_gtod_sync_lock + Everything else is a leaf: no other lock is taken inside the critical sections. @@ -216,6 +219,14 @@ time it will be set using the Dirty tracking mechanism described above. :Comment: 'raw' because hardware enabling/disabling must be atomic /wrt migration. +:Name: kvm_arch::pvclock_gtod_sync_lock +:Type: raw_spinlock_t +:Arch: x86 +:Protects: kvm_arch::{cur_tsc_generation,cur_tsc_nsec,cur_tsc_write, + cur_tsc_offset,nr_vcpus_matched_tsc} +:Comment: 'raw' because updating the kvm master clock must not be + preempted. + :Name: kvm_arch::tsc_write_lock :Type: raw_spinlock :Arch: x86 diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 4e803632cdca..e1b7c8b67428 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -2441,13 +2441,73 @@ static inline bool kvm_check_tsc_unstable(void) return check_tsc_unstable(); } +/* + * Infers attempts to synchronize the guest's tsc from host writes. Sets the + * offset for the vcpu and tracks the TSC matching generation that the vcpu + * participates in. + * + * Must hold kvm->arch.tsc_write_lock to call this function. + */ +static void __kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 offset, u64 tsc, + u64 ns, bool matched) +{ + struct kvm *kvm = vcpu->kvm; + bool already_matched; + unsigned long flags; + + lockdep_assert_held(&kvm->arch.tsc_write_lock); + + already_matched = + (vcpu->arch.this_tsc_generation == kvm->arch.cur_tsc_generation); + + /* + * We track the most recent recorded KHZ, write and time to + * allow the matching interval to be extended at each write. + */ + kvm->arch.last_tsc_nsec = ns; + kvm->arch.last_tsc_write = tsc; + kvm->arch.last_tsc_khz = vcpu->arch.virtual_tsc_khz; + + vcpu->arch.last_guest_tsc = tsc; + + /* Keep track of which generation this VCPU has synchronized to */ + vcpu->arch.this_tsc_generation = kvm->arch.cur_tsc_generation; + vcpu->arch.this_tsc_nsec = kvm->arch.cur_tsc_nsec; + vcpu->arch.this_tsc_write = kvm->arch.cur_tsc_write; + + kvm_vcpu_write_tsc_offset(vcpu, offset); + + spin_lock_irqsave(&kvm->arch.pvclock_gtod_sync_lock, flags); + if (!matched) { + /* + * We split periods of matched TSC writes into generations. + * For each generation, we track the original measured + * nanosecond time, offset, and write, so if TSCs are in + * sync, we can match exact offset, and if not, we can match + * exact software computation in compute_guest_tsc() + * + * These values are tracked in kvm->arch.cur_xxx variables. + */ + kvm->arch.nr_vcpus_matched_tsc = 0; + kvm->arch.cur_tsc_generation++; + kvm->arch.cur_tsc_nsec = ns; + kvm->arch.cur_tsc_write = tsc; + kvm->arch.cur_tsc_offset = offset; + matched = false; + } else if (!already_matched) { + kvm->arch.nr_vcpus_matched_tsc++; + } + + kvm_track_tsc_matching(vcpu); + spin_unlock_irqrestore(&kvm->arch.pvclock_gtod_sync_lock, flags); +} + static void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 data) { struct kvm *kvm = vcpu->kvm; u64 offset, ns, elapsed; unsigned long flags; - bool matched; - bool already_matched; + bool matched = false; bool synchronizing = false; raw_spin_lock_irqsave(&kvm->arch.tsc_write_lock, flags); @@ -2493,51 +2553,11 @@ static void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 data) offset = kvm_compute_l1_tsc_offset(vcpu, data); } matched = true; - already_matched = (vcpu->arch.this_tsc_generation == kvm->arch.cur_tsc_generation); - } else { - /* - * We split periods of matched TSC writes into generations. - * For each generation, we track the original measured - * nanosecond time, offset, and write, so if TSCs are in - * sync, we can match exact offset, and if not, we can match - * exact software computation in compute_guest_tsc() - * - * These values are tracked in kvm->arch.cur_xxx variables. - */ - kvm->arch.cur_tsc_generation++; - kvm->arch.cur_tsc_nsec = ns; - kvm->arch.cur_tsc_write = data; - kvm->arch.cur_tsc_offset = offset; - matched = false; } - /* - * We also track th most recent recorded KHZ, write and time to - * allow the matching interval to be extended at each write. - */ - kvm->arch.last_tsc_nsec = ns; - kvm->arch.last_tsc_write = data; - kvm->arch.last_tsc_khz = vcpu->arch.virtual_tsc_khz; - - vcpu->arch.last_guest_tsc = data; + __kvm_synchronize_tsc(vcpu, offset, data, ns, matched); - /* Keep track of which generation this VCPU has synchronized to */ - vcpu->arch.this_tsc_generation = kvm->arch.cur_tsc_generation; - vcpu->arch.this_tsc_nsec = kvm->arch.cur_tsc_nsec; - vcpu->arch.this_tsc_write = kvm->arch.cur_tsc_write; - - kvm_vcpu_write_tsc_offset(vcpu, offset); raw_spin_unlock_irqrestore(&kvm->arch.tsc_write_lock, flags); - - spin_lock_irqsave(&kvm->arch.pvclock_gtod_sync_lock, flags); - if (!matched) { - kvm->arch.nr_vcpus_matched_tsc = 0; - } else if (!already_matched) { - kvm->arch.nr_vcpus_matched_tsc++; - } - - kvm_track_tsc_matching(vcpu); - spin_unlock_irqrestore(&kvm->arch.pvclock_gtod_sync_lock, flags); } static inline void adjust_tsc_offset_guest(struct kvm_vcpu *vcpu, -- 2.32.0.402.g57bb445576-goog From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B65B8C12002 for ; Fri, 16 Jul 2021 21:26:48 +0000 (UTC) Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by mail.kernel.org (Postfix) with ESMTP id 59464613F6 for ; Fri, 16 Jul 2021 21:26:48 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 59464613F6 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvmarm-bounces@lists.cs.columbia.edu Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 009344B0DD; Fri, 16 Jul 2021 17:26:47 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@google.com Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bapkNxYJPq5I; Fri, 16 Jul 2021 17:26:46 -0400 (EDT) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 959A04B0D9; Fri, 16 Jul 2021 17:26:46 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 80AB04B0CB for ; Fri, 16 Jul 2021 17:26:43 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SGqPGQs-97y7 for ; Fri, 16 Jul 2021 17:26:42 -0400 (EDT) Received: from mail-io1-f74.google.com (mail-io1-f74.google.com [209.85.166.74]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id 5FC664A1AF for ; Fri, 16 Jul 2021 17:26:42 -0400 (EDT) Received: by mail-io1-f74.google.com with SMTP id p7-20020a5d8d070000b02904c0978ed194so6891693ioj.10 for ; Fri, 16 Jul 2021 14:26:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=MJZQeREwgEv7K7M0v6j7xlL8wKPecrHyLbDAmoCt7vg=; b=lkAEWDv031NoddTxAys9lfBfrA6KmwD/4l+4g8DRaSQeMr2b6P7DqkuwKdvx0HMo6G tdsNTWAJWw2cGxWbVbAhCkWjzJgJLYdVkXEU+fi3KNaaz5g6Q+H6BKLzxxsLedbLVKJK Gbt/ojltMPlVEroSq0KfNzGz3FxSlP1OdauulQjJpaZMGjiYgXA7/nyg4xkMLm9JRd/1 g+173zUemf15PWK1Ui7ESvZMmG6zDxvfaNQ88BJs3b3mAZQCZ7c0WAjJ1L4g5B7oW/E3 JN2EJE4r8B4O7FhHutlDfqxhmvgYFEL3qAN7+T7t4sopFcYK8AQhmnOnlfuH99y2lTZx +QAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=MJZQeREwgEv7K7M0v6j7xlL8wKPecrHyLbDAmoCt7vg=; b=giBPvr/FxKi9STMa/bON6GbXd961rXs592sQvc3MNanXFbMCJbslU1EICXQHoUZXDz 1Z8jaGOZBVYfoKFbWDh92qYLYi9aeqRCP3fdJL5s3EJOHZyQD9m16IGJxfL0Z5Qh9Hyk JLMfNf5kSEwsqC2+lvGmQ7O3W+KzV5oFLUrJpVp7ywSOStgsVp8XECqctuOt0fy5DRw1 dQy5isKyMBuerMSGoaBxhUtUTamfbb7pyXJH6dUJTdHGVIN5z9oneRunWWFq8DO4gvPJ BCqoEf6LOO+YT5sfVTrsuXSZK1AAP58Y7WMTUxZU4IoEyW/hE8EWyuiJ3IEQ8Emdxzgu FDEQ== X-Gm-Message-State: AOAM532GmpmyAWnZ9jK5LY9/+ZA2stDrrMgJXKz8SyFJq48knuPBKd2+ X4eQX1PqZBjKrG6cc9PJAsUbEOK5s5U= X-Google-Smtp-Source: ABdhPJwoxs4yEclO5kH0ab60ofNKeHKBJYurkbO5xn0o+KYDpWbntfW2OmI3RJILTEEV5yzbIG6R2eD3LJA= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a05:6602:2f09:: with SMTP id q9mr2363577iow.196.1626470801820; Fri, 16 Jul 2021 14:26:41 -0700 (PDT) Date: Fri, 16 Jul 2021 21:26:19 +0000 In-Reply-To: <20210716212629.2232756-1-oupton@google.com> Message-Id: <20210716212629.2232756-3-oupton@google.com> Mime-Version: 1.0 References: <20210716212629.2232756-1-oupton@google.com> X-Mailer: git-send-email 2.32.0.402.g57bb445576-goog Subject: [PATCH v2 02/12] KVM: x86: Refactor tsc synchronization code From: Oliver Upton To: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu Cc: Marc Zyngier , Raghavendra Rao Anata , Peter Shier , Sean Christopherson , David Matlack , Paolo Bonzini , linux-arm-kernel@lists.infradead.org, Jim Mattson X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu Refactor kvm_synchronize_tsc to make a new function that allows callers to specify TSC parameters (offset, value, nanoseconds, etc.) explicitly for the sake of participating in TSC synchronization. This changes the locking semantics around TSC writes. Writes to the TSC will now take the pvclock gtod lock while holding the tsc write lock, whereas before these locks were disjoint. Reviewed-by: David Matlack Signed-off-by: Oliver Upton --- Documentation/virt/kvm/locking.rst | 11 +++ arch/x86/kvm/x86.c | 106 +++++++++++++++++------------ 2 files changed, 74 insertions(+), 43 deletions(-) diff --git a/Documentation/virt/kvm/locking.rst b/Documentation/virt/kvm/locking.rst index 35eca377543d..ac62e1c76694 100644 --- a/Documentation/virt/kvm/locking.rst +++ b/Documentation/virt/kvm/locking.rst @@ -30,6 +30,9 @@ On x86: holding kvm->arch.mmu_lock (typically with ``read_lock``, otherwise there's no need to take kvm->arch.tdp_mmu_pages_lock at all). +- kvm->arch.tsc_write_lock is taken outside + kvm->arch.pvclock_gtod_sync_lock + Everything else is a leaf: no other lock is taken inside the critical sections. @@ -216,6 +219,14 @@ time it will be set using the Dirty tracking mechanism described above. :Comment: 'raw' because hardware enabling/disabling must be atomic /wrt migration. +:Name: kvm_arch::pvclock_gtod_sync_lock +:Type: raw_spinlock_t +:Arch: x86 +:Protects: kvm_arch::{cur_tsc_generation,cur_tsc_nsec,cur_tsc_write, + cur_tsc_offset,nr_vcpus_matched_tsc} +:Comment: 'raw' because updating the kvm master clock must not be + preempted. + :Name: kvm_arch::tsc_write_lock :Type: raw_spinlock :Arch: x86 diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 4e803632cdca..e1b7c8b67428 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -2441,13 +2441,73 @@ static inline bool kvm_check_tsc_unstable(void) return check_tsc_unstable(); } +/* + * Infers attempts to synchronize the guest's tsc from host writes. Sets the + * offset for the vcpu and tracks the TSC matching generation that the vcpu + * participates in. + * + * Must hold kvm->arch.tsc_write_lock to call this function. + */ +static void __kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 offset, u64 tsc, + u64 ns, bool matched) +{ + struct kvm *kvm = vcpu->kvm; + bool already_matched; + unsigned long flags; + + lockdep_assert_held(&kvm->arch.tsc_write_lock); + + already_matched = + (vcpu->arch.this_tsc_generation == kvm->arch.cur_tsc_generation); + + /* + * We track the most recent recorded KHZ, write and time to + * allow the matching interval to be extended at each write. + */ + kvm->arch.last_tsc_nsec = ns; + kvm->arch.last_tsc_write = tsc; + kvm->arch.last_tsc_khz = vcpu->arch.virtual_tsc_khz; + + vcpu->arch.last_guest_tsc = tsc; + + /* Keep track of which generation this VCPU has synchronized to */ + vcpu->arch.this_tsc_generation = kvm->arch.cur_tsc_generation; + vcpu->arch.this_tsc_nsec = kvm->arch.cur_tsc_nsec; + vcpu->arch.this_tsc_write = kvm->arch.cur_tsc_write; + + kvm_vcpu_write_tsc_offset(vcpu, offset); + + spin_lock_irqsave(&kvm->arch.pvclock_gtod_sync_lock, flags); + if (!matched) { + /* + * We split periods of matched TSC writes into generations. + * For each generation, we track the original measured + * nanosecond time, offset, and write, so if TSCs are in + * sync, we can match exact offset, and if not, we can match + * exact software computation in compute_guest_tsc() + * + * These values are tracked in kvm->arch.cur_xxx variables. + */ + kvm->arch.nr_vcpus_matched_tsc = 0; + kvm->arch.cur_tsc_generation++; + kvm->arch.cur_tsc_nsec = ns; + kvm->arch.cur_tsc_write = tsc; + kvm->arch.cur_tsc_offset = offset; + matched = false; + } else if (!already_matched) { + kvm->arch.nr_vcpus_matched_tsc++; + } + + kvm_track_tsc_matching(vcpu); + spin_unlock_irqrestore(&kvm->arch.pvclock_gtod_sync_lock, flags); +} + static void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 data) { struct kvm *kvm = vcpu->kvm; u64 offset, ns, elapsed; unsigned long flags; - bool matched; - bool already_matched; + bool matched = false; bool synchronizing = false; raw_spin_lock_irqsave(&kvm->arch.tsc_write_lock, flags); @@ -2493,51 +2553,11 @@ static void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 data) offset = kvm_compute_l1_tsc_offset(vcpu, data); } matched = true; - already_matched = (vcpu->arch.this_tsc_generation == kvm->arch.cur_tsc_generation); - } else { - /* - * We split periods of matched TSC writes into generations. - * For each generation, we track the original measured - * nanosecond time, offset, and write, so if TSCs are in - * sync, we can match exact offset, and if not, we can match - * exact software computation in compute_guest_tsc() - * - * These values are tracked in kvm->arch.cur_xxx variables. - */ - kvm->arch.cur_tsc_generation++; - kvm->arch.cur_tsc_nsec = ns; - kvm->arch.cur_tsc_write = data; - kvm->arch.cur_tsc_offset = offset; - matched = false; } - /* - * We also track th most recent recorded KHZ, write and time to - * allow the matching interval to be extended at each write. - */ - kvm->arch.last_tsc_nsec = ns; - kvm->arch.last_tsc_write = data; - kvm->arch.last_tsc_khz = vcpu->arch.virtual_tsc_khz; - - vcpu->arch.last_guest_tsc = data; + __kvm_synchronize_tsc(vcpu, offset, data, ns, matched); - /* Keep track of which generation this VCPU has synchronized to */ - vcpu->arch.this_tsc_generation = kvm->arch.cur_tsc_generation; - vcpu->arch.this_tsc_nsec = kvm->arch.cur_tsc_nsec; - vcpu->arch.this_tsc_write = kvm->arch.cur_tsc_write; - - kvm_vcpu_write_tsc_offset(vcpu, offset); raw_spin_unlock_irqrestore(&kvm->arch.tsc_write_lock, flags); - - spin_lock_irqsave(&kvm->arch.pvclock_gtod_sync_lock, flags); - if (!matched) { - kvm->arch.nr_vcpus_matched_tsc = 0; - } else if (!already_matched) { - kvm->arch.nr_vcpus_matched_tsc++; - } - - kvm_track_tsc_matching(vcpu); - spin_unlock_irqrestore(&kvm->arch.pvclock_gtod_sync_lock, flags); } static inline void adjust_tsc_offset_guest(struct kvm_vcpu *vcpu, -- 2.32.0.402.g57bb445576-goog _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD8D5C12002 for ; Fri, 16 Jul 2021 21:39:42 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8EA0961158 for ; Fri, 16 Jul 2021 21:39:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8EA0961158 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=V6SiHmsjqMzjOoFIqb80VhukUtrZU3IgpoDmmRp/hIg=; b=KkPheppZHxP+Yrmlo45kESdr34 ta5EqpO4wO4PkPXvG5SqCsMpcG+wjJed9l8jAzRCAyFE1CP3eZWsclZY/G07GFnHRi0Le6bBQA/ng 3ft4PyXe7tFFavMhMcY3aTtJ3oXMN89xwNiZV9BdVVc3/4G/CZdqqBfYnQ3dbW1lL/53PBE7kiRPc 7wbsquQ7eov+QwQXMSoIoU06G1iOF5Yslej3JlAa/PODhbZGZfvddqRcAVNgHZ25t93nTaRp8QpbI KCDFhbLMn9X+MIkYbDTAHOaYfo3XK+l47pHZJMYiQoYPdy3I/EmxNVZ3V/qpSyU+4gyPW3vkHNzgx kvuM4sfw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1m4VWB-005MsK-9b; Fri, 16 Jul 2021 21:37:11 +0000 Received: from mail-il1-x14a.google.com ([2607:f8b0:4864:20::14a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1m4VM3-005IYE-Eu for linux-arm-kernel@lists.infradead.org; Fri, 16 Jul 2021 21:26:45 +0000 Received: by mail-il1-x14a.google.com with SMTP id e16-20020a056e0204b0b029020c886c9370so6142320ils.10 for ; Fri, 16 Jul 2021 14:26:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=MJZQeREwgEv7K7M0v6j7xlL8wKPecrHyLbDAmoCt7vg=; b=lkAEWDv031NoddTxAys9lfBfrA6KmwD/4l+4g8DRaSQeMr2b6P7DqkuwKdvx0HMo6G tdsNTWAJWw2cGxWbVbAhCkWjzJgJLYdVkXEU+fi3KNaaz5g6Q+H6BKLzxxsLedbLVKJK Gbt/ojltMPlVEroSq0KfNzGz3FxSlP1OdauulQjJpaZMGjiYgXA7/nyg4xkMLm9JRd/1 g+173zUemf15PWK1Ui7ESvZMmG6zDxvfaNQ88BJs3b3mAZQCZ7c0WAjJ1L4g5B7oW/E3 JN2EJE4r8B4O7FhHutlDfqxhmvgYFEL3qAN7+T7t4sopFcYK8AQhmnOnlfuH99y2lTZx +QAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=MJZQeREwgEv7K7M0v6j7xlL8wKPecrHyLbDAmoCt7vg=; b=iPDumhJeVkIvlb53EPoiw+keo0yQM1t4xUmSTPi1imJAp8jD+HcAXDucTg7BbLJkI7 HrQJ6pI7sGUoW8GHgdf/tQFtrW7Qn+Z039XKFUUT4ybMnQdD9273ytDISaW3waTCvDrk TJErf2HXZxAZQt/dJH1f1oPt4OCk/Ntj6swSb5mXMBGDNCP4UjMPuj39YsgblB86Qm2h Dbfx/ZGRfnVCiBDUz7+7aIvpK5LmnXNYjXSDaWOfHDvaxyAPBGQOQzjaiKJXqT8UTahj tffRDcqeKKkZgYbWQrhXBAaizLwrzB+FvEyhw6LgBlqLhyFvnmVzLAXb/PHcroZ9eMdl k7Xg== X-Gm-Message-State: AOAM532DKjVvWHeSO9bWbBATU5G6HTdtgpxr+H94miYz5cyjIdOnNNwp EI5w19PcjJEx/c/uBoZeO4cWPh5X+WM= X-Google-Smtp-Source: ABdhPJwoxs4yEclO5kH0ab60ofNKeHKBJYurkbO5xn0o+KYDpWbntfW2OmI3RJILTEEV5yzbIG6R2eD3LJA= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a05:6602:2f09:: with SMTP id q9mr2363577iow.196.1626470801820; Fri, 16 Jul 2021 14:26:41 -0700 (PDT) Date: Fri, 16 Jul 2021 21:26:19 +0000 In-Reply-To: <20210716212629.2232756-1-oupton@google.com> Message-Id: <20210716212629.2232756-3-oupton@google.com> Mime-Version: 1.0 References: <20210716212629.2232756-1-oupton@google.com> X-Mailer: git-send-email 2.32.0.402.g57bb445576-goog Subject: [PATCH v2 02/12] KVM: x86: Refactor tsc synchronization code From: Oliver Upton To: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu Cc: Paolo Bonzini , Sean Christopherson , Marc Zyngier , Peter Shier , Jim Mattson , David Matlack , Ricardo Koller , Jing Zhang , Raghavendra Rao Anata , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Oliver Upton X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210716_142643_587275_E68EB6FD X-CRM114-Status: GOOD ( 17.40 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Refactor kvm_synchronize_tsc to make a new function that allows callers to specify TSC parameters (offset, value, nanoseconds, etc.) explicitly for the sake of participating in TSC synchronization. This changes the locking semantics around TSC writes. Writes to the TSC will now take the pvclock gtod lock while holding the tsc write lock, whereas before these locks were disjoint. Reviewed-by: David Matlack Signed-off-by: Oliver Upton --- Documentation/virt/kvm/locking.rst | 11 +++ arch/x86/kvm/x86.c | 106 +++++++++++++++++------------ 2 files changed, 74 insertions(+), 43 deletions(-) diff --git a/Documentation/virt/kvm/locking.rst b/Documentation/virt/kvm/locking.rst index 35eca377543d..ac62e1c76694 100644 --- a/Documentation/virt/kvm/locking.rst +++ b/Documentation/virt/kvm/locking.rst @@ -30,6 +30,9 @@ On x86: holding kvm->arch.mmu_lock (typically with ``read_lock``, otherwise there's no need to take kvm->arch.tdp_mmu_pages_lock at all). +- kvm->arch.tsc_write_lock is taken outside + kvm->arch.pvclock_gtod_sync_lock + Everything else is a leaf: no other lock is taken inside the critical sections. @@ -216,6 +219,14 @@ time it will be set using the Dirty tracking mechanism described above. :Comment: 'raw' because hardware enabling/disabling must be atomic /wrt migration. +:Name: kvm_arch::pvclock_gtod_sync_lock +:Type: raw_spinlock_t +:Arch: x86 +:Protects: kvm_arch::{cur_tsc_generation,cur_tsc_nsec,cur_tsc_write, + cur_tsc_offset,nr_vcpus_matched_tsc} +:Comment: 'raw' because updating the kvm master clock must not be + preempted. + :Name: kvm_arch::tsc_write_lock :Type: raw_spinlock :Arch: x86 diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 4e803632cdca..e1b7c8b67428 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -2441,13 +2441,73 @@ static inline bool kvm_check_tsc_unstable(void) return check_tsc_unstable(); } +/* + * Infers attempts to synchronize the guest's tsc from host writes. Sets the + * offset for the vcpu and tracks the TSC matching generation that the vcpu + * participates in. + * + * Must hold kvm->arch.tsc_write_lock to call this function. + */ +static void __kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 offset, u64 tsc, + u64 ns, bool matched) +{ + struct kvm *kvm = vcpu->kvm; + bool already_matched; + unsigned long flags; + + lockdep_assert_held(&kvm->arch.tsc_write_lock); + + already_matched = + (vcpu->arch.this_tsc_generation == kvm->arch.cur_tsc_generation); + + /* + * We track the most recent recorded KHZ, write and time to + * allow the matching interval to be extended at each write. + */ + kvm->arch.last_tsc_nsec = ns; + kvm->arch.last_tsc_write = tsc; + kvm->arch.last_tsc_khz = vcpu->arch.virtual_tsc_khz; + + vcpu->arch.last_guest_tsc = tsc; + + /* Keep track of which generation this VCPU has synchronized to */ + vcpu->arch.this_tsc_generation = kvm->arch.cur_tsc_generation; + vcpu->arch.this_tsc_nsec = kvm->arch.cur_tsc_nsec; + vcpu->arch.this_tsc_write = kvm->arch.cur_tsc_write; + + kvm_vcpu_write_tsc_offset(vcpu, offset); + + spin_lock_irqsave(&kvm->arch.pvclock_gtod_sync_lock, flags); + if (!matched) { + /* + * We split periods of matched TSC writes into generations. + * For each generation, we track the original measured + * nanosecond time, offset, and write, so if TSCs are in + * sync, we can match exact offset, and if not, we can match + * exact software computation in compute_guest_tsc() + * + * These values are tracked in kvm->arch.cur_xxx variables. + */ + kvm->arch.nr_vcpus_matched_tsc = 0; + kvm->arch.cur_tsc_generation++; + kvm->arch.cur_tsc_nsec = ns; + kvm->arch.cur_tsc_write = tsc; + kvm->arch.cur_tsc_offset = offset; + matched = false; + } else if (!already_matched) { + kvm->arch.nr_vcpus_matched_tsc++; + } + + kvm_track_tsc_matching(vcpu); + spin_unlock_irqrestore(&kvm->arch.pvclock_gtod_sync_lock, flags); +} + static void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 data) { struct kvm *kvm = vcpu->kvm; u64 offset, ns, elapsed; unsigned long flags; - bool matched; - bool already_matched; + bool matched = false; bool synchronizing = false; raw_spin_lock_irqsave(&kvm->arch.tsc_write_lock, flags); @@ -2493,51 +2553,11 @@ static void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 data) offset = kvm_compute_l1_tsc_offset(vcpu, data); } matched = true; - already_matched = (vcpu->arch.this_tsc_generation == kvm->arch.cur_tsc_generation); - } else { - /* - * We split periods of matched TSC writes into generations. - * For each generation, we track the original measured - * nanosecond time, offset, and write, so if TSCs are in - * sync, we can match exact offset, and if not, we can match - * exact software computation in compute_guest_tsc() - * - * These values are tracked in kvm->arch.cur_xxx variables. - */ - kvm->arch.cur_tsc_generation++; - kvm->arch.cur_tsc_nsec = ns; - kvm->arch.cur_tsc_write = data; - kvm->arch.cur_tsc_offset = offset; - matched = false; } - /* - * We also track th most recent recorded KHZ, write and time to - * allow the matching interval to be extended at each write. - */ - kvm->arch.last_tsc_nsec = ns; - kvm->arch.last_tsc_write = data; - kvm->arch.last_tsc_khz = vcpu->arch.virtual_tsc_khz; - - vcpu->arch.last_guest_tsc = data; + __kvm_synchronize_tsc(vcpu, offset, data, ns, matched); - /* Keep track of which generation this VCPU has synchronized to */ - vcpu->arch.this_tsc_generation = kvm->arch.cur_tsc_generation; - vcpu->arch.this_tsc_nsec = kvm->arch.cur_tsc_nsec; - vcpu->arch.this_tsc_write = kvm->arch.cur_tsc_write; - - kvm_vcpu_write_tsc_offset(vcpu, offset); raw_spin_unlock_irqrestore(&kvm->arch.tsc_write_lock, flags); - - spin_lock_irqsave(&kvm->arch.pvclock_gtod_sync_lock, flags); - if (!matched) { - kvm->arch.nr_vcpus_matched_tsc = 0; - } else if (!already_matched) { - kvm->arch.nr_vcpus_matched_tsc++; - } - - kvm_track_tsc_matching(vcpu); - spin_unlock_irqrestore(&kvm->arch.pvclock_gtod_sync_lock, flags); } static inline void adjust_tsc_offset_guest(struct kvm_vcpu *vcpu, -- 2.32.0.402.g57bb445576-goog _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel