From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2888FC433F5 for ; Thu, 16 Sep 2021 18:15:46 +0000 (UTC) Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by mail.kernel.org (Postfix) with ESMTP id CF07E6120D for ; Thu, 16 Sep 2021 18:15:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org CF07E6120D Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.cs.columbia.edu Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 820F64B195; Thu, 16 Sep 2021 14:15:45 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@google.com Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tgS55cKh3E6d; Thu, 16 Sep 2021 14:15:44 -0400 (EDT) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 6863E4B0B8; Thu, 16 Sep 2021 14:15:44 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 0872C4B092 for ; Thu, 16 Sep 2021 14:15:43 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DFr4pIli69Ax for ; Thu, 16 Sep 2021 14:15:41 -0400 (EDT) Received: from mail-qv1-f73.google.com (mail-qv1-f73.google.com [209.85.219.73]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id D48444B090 for ; Thu, 16 Sep 2021 14:15:41 -0400 (EDT) Received: by mail-qv1-f73.google.com with SMTP id f14-20020a056214076e00b003773e95910bso63229341qvz.4 for ; Thu, 16 Sep 2021 11:15:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:message-id:mime-version:subject:from:to:cc; bh=Pe7x0KqNWpd27U8B0VgFmhUlzq7CaBZRDC7FTGm6BYo=; b=c/zDfdFNoGgGDawStlpHk9tXybqTTvO3qklooPCSh2RZyn1jO926yAJN3EkzROcnWV +7Jv8frVlp+AjiKuCpYLQYVNhudtlSi3aG5XE0qVftcRBxmfkVHYUFK2g3I/ufLuYsD5 yXWxHrl/35QQy9F3h5xRX4vnkMen65DAtcmDUlpWA2jh2qYxeZTWx7cuJq7rLgB37IjF FR1hGU1oD+z2TrGqXS1Hb1S8jn7kM7JHUntQZubO19eBArz72mowND7VvAXe/MPZogEM wCS3fOC64SO9FdoDCKIccrMyRTFwuxu7FOoW75kCgb27aM9H/lW5xEtfwu2ZTfFFx8KR /M9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=Pe7x0KqNWpd27U8B0VgFmhUlzq7CaBZRDC7FTGm6BYo=; b=p+60dl+XJdprYNaSnBmCLMjhIlspOvpTAZ7GY9xgYTow0jaKyyHekSx3y2wkJuobgA q15glCl+aAJt6K0OVQyL29dwpt5WyO1vaRg3GBbdWgb6FOnt7G7EmhBYX30GtkoPhzvv 7to8cAPCZghlCWnhrcbJV9/JmfUL365Owj5KMMW5S+qH0876LzncBJMcaEyRi1KKzmYa XnSHVn8YkKdN0XKP1+he0VHUXF/mAzgNTQCbIP6igf3lx3jWS81JZwyHP0kbpE0RhTH5 g1BMAtmnlhu2HE1xMqeWjvXSyoklNcaFf2ngZmiT7XAbZbYaRCYiDhGVDPCBj3DtxdTw xK2w== X-Gm-Message-State: AOAM530jx56Ppr23ZKTk1KMRwvGHmfkV6hnQojpQz5/twbaUHdH6wofG tXSoQGUlxgRX5MtjqIzac7VbmWzX9k4= X-Google-Smtp-Source: ABdhPJyvIbX+FueQ86OOu11wGIo24Yj/vbCnWhhfcLYDpB8XPsQIlfG3Y+cSd9yAQkcLoDzHF5ajFaxNvtk= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a0c:9146:: with SMTP id q64mr6704127qvq.38.1631816141556; Thu, 16 Sep 2021 11:15:41 -0700 (PDT) Date: Thu, 16 Sep 2021 18:15:31 +0000 Message-Id: <20210916181538.968978-1-oupton@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.33.0.464.g1972c5931b-goog Subject: [PATCH v8 0/7] KVM: x86: Add idempotent controls for migrating system counter state From: Oliver Upton To: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu Cc: Catalin Marinas , Will Deacon , Marc Zyngier , Peter Shier , Sean Christopherson , David Matlack , Paolo Bonzini , linux-arm-kernel@lists.infradead.org, Jim Mattson X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu KVM's current means of saving/restoring system counters is plagued with temporal issues. On x86, we migrate the guest's system counter by-value through the respective guest's IA32_TSC value. Restoring system counters by-value is brittle as the state is not idempotent: the host system counter is still oscillating between the attempted save and restore. Furthermore, VMMs may wish to transparently live migrate guest VMs, meaning that they include the elapsed time due to live migration blackout in the guest system counter view. The VMM thread could be preempted for any number of reasons (scheduler, L0 hypervisor under nested) between the time that it calculates the desired guest counter value and when KVM actually sets this counter state. Despite the value-based interface that we present to userspace, KVM actually has idempotent guest controls by way of the TSC offset. We can avoid all of the issues associated with a value-based interface by abstracting these offset controls in a new device attribute. This series introduces new vCPU device attributes to provide userspace access to the vCPU's system counter offset. Patches 1-2 are Paolo's refactorings around locking and the KVM_{GET,SET}_CLOCK ioctls. Patch 3 cures a race where use_master_clock is read outside of the pvclock lock in the KVM_GET_CLOCK ioctl. Patch 4 adopts Paolo's suggestion, augmenting the KVM_{GET,SET}_CLOCK ioctls to provide userspace with a (host_tsc, realtime) instant. This is essential for a VMM to perform precise migration of the guest's system counters. Patch 5 does away with the pvclock spin lock in favor of a sequence lock based on the tsc_write_lock. The original patch is from Paolo, I touched it up a bit to fix a deadlock and some unused variables that caused -Werror to scream. Patch 6 extracts the TSC synchronization tracking code in a way that it can be used for both offset-based and value-based TSC synchronization schemes. Finally, patch 7 implements a vCPU device attribute which allows VMMs to get at the TSC offset of a vCPU. This series was tested with the new KVM selftests for the KVM clock and system counter offset controls on Haswell hardware. Kernel was built with CONFIG_LOCKDEP given the new locking changes/lockdep assertions here. Note that these tests are mailed as a separate series due to the dependencies in both x86 and arm64. Applies cleanly to 5.15-rc1 v8: http://lore.kernel.org/r/20210816001130.3059564-1-oupton@google.com v7 -> v8: - Rebased to 5.15-rc1 - Picked up Paolo's version of the series, which includes locking changes - Make KVM advertise KVM_CAP_VCPU_ATTRIBUTES Oliver Upton (4): KVM: x86: Fix potential race in KVM_GET_CLOCK KVM: x86: Report host tsc and realtime values in KVM_GET_CLOCK KVM: x86: Refactor tsc synchronization code KVM: x86: Expose TSC offset controls to userspace Paolo Bonzini (3): kvm: x86: abstract locking around pvclock_update_vm_gtod_copy KVM: x86: extract KVM_GET_CLOCK/KVM_SET_CLOCK to separate functions kvm: x86: protect masterclock with a seqcount Documentation/virt/kvm/api.rst | 42 ++- Documentation/virt/kvm/devices/vcpu.rst | 57 +++ arch/x86/include/asm/kvm_host.h | 12 +- arch/x86/include/uapi/asm/kvm.h | 4 + arch/x86/kvm/x86.c | 458 ++++++++++++++++-------- include/uapi/linux/kvm.h | 7 +- 6 files changed, 419 insertions(+), 161 deletions(-) -- 2.33.0.309.g3052b89438-goog _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE999C433EF for ; Thu, 16 Sep 2021 18:20:40 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AB709611C4 for ; Thu, 16 Sep 2021 18:20:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org AB709611C4 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Mime-Version: Message-Id:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: References:List-Owner; bh=tOHy9mqsrWm3GQAS9cL/hEBBYK7hMBcGyw8wjUXyc2Y=; b=aQi 6rS7oA09YwXxw/rDZC7uqBhhC0rDWdsg1y9byR+BVfU9ZUK7ElGJ8M1b1oPWkHq64xohHwZdqk0lB UAkQJ93QbeRYwKWRs0l3HA/78Vc/cQH794X1wzsqTMYKz2FjUd8AX3FZJ7QQnKr1BNKNRAmvaM4Me dlFnTGHHOwKOjyexeH70x6d8ah6+NaqFbrw98+BqL9nUHIIJ/cJzuWlRGiponNCBmN8pDzvtxtCTX C2Ecj4tBMlA3FsHZc+orhkcHTdT5xufCgBau/J6pEAY64e6tAJY6dfgk2A48RP0Drcz7NLCJfQY/4 mfGmZLcf6C4OYQuD7g3Ubf2z+EFwKmQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mQvyJ-00C3Qm-MC; Thu, 16 Sep 2021 18:18:57 +0000 Received: from mail-qk1-x74a.google.com ([2607:f8b0:4864:20::74a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mQvvC-00C1wD-Vq for linux-arm-kernel@lists.infradead.org; Thu, 16 Sep 2021 18:15:44 +0000 Received: by mail-qk1-x74a.google.com with SMTP id j27-20020a05620a0a5b00b0042874883070so44656053qka.19 for ; Thu, 16 Sep 2021 11:15:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:message-id:mime-version:subject:from:to:cc; bh=Pe7x0KqNWpd27U8B0VgFmhUlzq7CaBZRDC7FTGm6BYo=; b=c/zDfdFNoGgGDawStlpHk9tXybqTTvO3qklooPCSh2RZyn1jO926yAJN3EkzROcnWV +7Jv8frVlp+AjiKuCpYLQYVNhudtlSi3aG5XE0qVftcRBxmfkVHYUFK2g3I/ufLuYsD5 yXWxHrl/35QQy9F3h5xRX4vnkMen65DAtcmDUlpWA2jh2qYxeZTWx7cuJq7rLgB37IjF FR1hGU1oD+z2TrGqXS1Hb1S8jn7kM7JHUntQZubO19eBArz72mowND7VvAXe/MPZogEM wCS3fOC64SO9FdoDCKIccrMyRTFwuxu7FOoW75kCgb27aM9H/lW5xEtfwu2ZTfFFx8KR /M9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=Pe7x0KqNWpd27U8B0VgFmhUlzq7CaBZRDC7FTGm6BYo=; b=4iifFdAUWPn8shamOLihyoI/ixbrS6qGADqkH0QYEfLrxvKW1XMZPKZOmDj9tQlDos M0rMGH0VUnhc10j/Zfi3QGO2d0h0qJtfWnvK1c6y6ZQJIKJL18SuegCOecXm2vEjSg6G /x4zSJJ+SoffO2U037zA0O+P3tf029SNJaKqKqRVoqbYgjPWOmEWb6kku6wslKV5+166 df7Yr/G5XYGaWgXsN+l6dOXgKDQiuqAzvUug9ExqzwGyMme7jTMh47PNShkABM0na4sb h55dPTWz5GuTdiUvtDv4bl4SY9imBIXCTloK65bbHNGGefwBi+P/k8JXTdMNufV4UbCr XpHQ== X-Gm-Message-State: AOAM532Wyc7QY7jnOHLxExBtMJjVkhPKpP/Ts30V5t9Rv++bYdwIpvZ1 rzXIKrKXD5z8wiX7NKHIZ8ARpHjeb44= X-Google-Smtp-Source: ABdhPJyvIbX+FueQ86OOu11wGIo24Yj/vbCnWhhfcLYDpB8XPsQIlfG3Y+cSd9yAQkcLoDzHF5ajFaxNvtk= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a0c:9146:: with SMTP id q64mr6704127qvq.38.1631816141556; Thu, 16 Sep 2021 11:15:41 -0700 (PDT) Date: Thu, 16 Sep 2021 18:15:31 +0000 Message-Id: <20210916181538.968978-1-oupton@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.33.0.464.g1972c5931b-goog Subject: [PATCH v8 0/7] KVM: x86: Add idempotent controls for migrating system counter state From: Oliver Upton To: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu Cc: Paolo Bonzini , Sean Christopherson , Marc Zyngier , Peter Shier , Jim Mattson , David Matlack , Ricardo Koller , Jing Zhang , Raghavendra Rao Anata , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Andrew Jones , Will Deacon , Catalin Marinas , Oliver Upton X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210916_111543_080148_AD1B52CA X-CRM114-Status: GOOD ( 18.00 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org KVM's current means of saving/restoring system counters is plagued with temporal issues. On x86, we migrate the guest's system counter by-value through the respective guest's IA32_TSC value. Restoring system counters by-value is brittle as the state is not idempotent: the host system counter is still oscillating between the attempted save and restore. Furthermore, VMMs may wish to transparently live migrate guest VMs, meaning that they include the elapsed time due to live migration blackout in the guest system counter view. The VMM thread could be preempted for any number of reasons (scheduler, L0 hypervisor under nested) between the time that it calculates the desired guest counter value and when KVM actually sets this counter state. Despite the value-based interface that we present to userspace, KVM actually has idempotent guest controls by way of the TSC offset. We can avoid all of the issues associated with a value-based interface by abstracting these offset controls in a new device attribute. This series introduces new vCPU device attributes to provide userspace access to the vCPU's system counter offset. Patches 1-2 are Paolo's refactorings around locking and the KVM_{GET,SET}_CLOCK ioctls. Patch 3 cures a race where use_master_clock is read outside of the pvclock lock in the KVM_GET_CLOCK ioctl. Patch 4 adopts Paolo's suggestion, augmenting the KVM_{GET,SET}_CLOCK ioctls to provide userspace with a (host_tsc, realtime) instant. This is essential for a VMM to perform precise migration of the guest's system counters. Patch 5 does away with the pvclock spin lock in favor of a sequence lock based on the tsc_write_lock. The original patch is from Paolo, I touched it up a bit to fix a deadlock and some unused variables that caused -Werror to scream. Patch 6 extracts the TSC synchronization tracking code in a way that it can be used for both offset-based and value-based TSC synchronization schemes. Finally, patch 7 implements a vCPU device attribute which allows VMMs to get at the TSC offset of a vCPU. This series was tested with the new KVM selftests for the KVM clock and system counter offset controls on Haswell hardware. Kernel was built with CONFIG_LOCKDEP given the new locking changes/lockdep assertions here. Note that these tests are mailed as a separate series due to the dependencies in both x86 and arm64. Applies cleanly to 5.15-rc1 v8: http://lore.kernel.org/r/20210816001130.3059564-1-oupton@google.com v7 -> v8: - Rebased to 5.15-rc1 - Picked up Paolo's version of the series, which includes locking changes - Make KVM advertise KVM_CAP_VCPU_ATTRIBUTES Oliver Upton (4): KVM: x86: Fix potential race in KVM_GET_CLOCK KVM: x86: Report host tsc and realtime values in KVM_GET_CLOCK KVM: x86: Refactor tsc synchronization code KVM: x86: Expose TSC offset controls to userspace Paolo Bonzini (3): kvm: x86: abstract locking around pvclock_update_vm_gtod_copy KVM: x86: extract KVM_GET_CLOCK/KVM_SET_CLOCK to separate functions kvm: x86: protect masterclock with a seqcount Documentation/virt/kvm/api.rst | 42 ++- Documentation/virt/kvm/devices/vcpu.rst | 57 +++ arch/x86/include/asm/kvm_host.h | 12 +- arch/x86/include/uapi/asm/kvm.h | 4 + arch/x86/kvm/x86.c | 458 ++++++++++++++++-------- include/uapi/linux/kvm.h | 7 +- 6 files changed, 419 insertions(+), 161 deletions(-) -- 2.33.0.309.g3052b89438-goog _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-21.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT, USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C66C0C433F5 for ; Thu, 16 Sep 2021 18:56:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B09266058D for ; Thu, 16 Sep 2021 18:56:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345072AbhIPS5x (ORCPT ); Thu, 16 Sep 2021 14:57:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52080 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245557AbhIPS53 (ORCPT ); Thu, 16 Sep 2021 14:57:29 -0400 Received: from mail-qv1-xf49.google.com (mail-qv1-xf49.google.com [IPv6:2607:f8b0:4864:20::f49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7E447C04A154 for ; Thu, 16 Sep 2021 11:15:42 -0700 (PDT) Received: by mail-qv1-xf49.google.com with SMTP id e8-20020a0cf348000000b0037a350958f2so63297015qvm.7 for ; Thu, 16 Sep 2021 11:15:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:message-id:mime-version:subject:from:to:cc; bh=Pe7x0KqNWpd27U8B0VgFmhUlzq7CaBZRDC7FTGm6BYo=; b=c/zDfdFNoGgGDawStlpHk9tXybqTTvO3qklooPCSh2RZyn1jO926yAJN3EkzROcnWV +7Jv8frVlp+AjiKuCpYLQYVNhudtlSi3aG5XE0qVftcRBxmfkVHYUFK2g3I/ufLuYsD5 yXWxHrl/35QQy9F3h5xRX4vnkMen65DAtcmDUlpWA2jh2qYxeZTWx7cuJq7rLgB37IjF FR1hGU1oD+z2TrGqXS1Hb1S8jn7kM7JHUntQZubO19eBArz72mowND7VvAXe/MPZogEM wCS3fOC64SO9FdoDCKIccrMyRTFwuxu7FOoW75kCgb27aM9H/lW5xEtfwu2ZTfFFx8KR /M9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=Pe7x0KqNWpd27U8B0VgFmhUlzq7CaBZRDC7FTGm6BYo=; b=ud4Sm5W+LUrcoBjrVG6hAFsJef1qVJGgN6oC4TWM0KomG4x5w88yyP5Kig6u2kFIOq sy6n8WkecWBiDLlNP3XKNjEzmXGF8BagGGriUGdOvlV6i+A3EeTQaLFDIBU98NZmVt3y YusDG4nkevrSHV3G3sKXWA8C91KDt+2BLXL/QjGzaqiWuJNRp//r6ovLgM+w2PQdI/+u 44uWfrhwlTRWZjvOY037Lw5vRSSwfQhSRRoAubq4qxLS7OxvmzhC9Isk5Suz6EwWFoHv K4GvFff13xrJnubmq6JvCxFxHCkrqJgJCqqiZkY+STNeWSsIM947+EGRbXGoFFgFnfCa J4qw== X-Gm-Message-State: AOAM533o5OGBHNOZY4HRFnzLO/ZGOG9axEmzQB/NpuRSYbEge7GTw/Zh oJPZ00+cUYzb7lYAtzhW6I7Y0H2QQ5WQNovNA4O+Igf7wCYPLoP+XeeNTFE0t4qbKGVWTWX7Xzd dZNn5QPzPQ1U91s5xpFJZdvnMZ/kOCtux4XQfgLc5JKLY9MJgAy+0O+21Bg== X-Google-Smtp-Source: ABdhPJyvIbX+FueQ86OOu11wGIo24Yj/vbCnWhhfcLYDpB8XPsQIlfG3Y+cSd9yAQkcLoDzHF5ajFaxNvtk= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a0c:9146:: with SMTP id q64mr6704127qvq.38.1631816141556; Thu, 16 Sep 2021 11:15:41 -0700 (PDT) Date: Thu, 16 Sep 2021 18:15:31 +0000 Message-Id: <20210916181538.968978-1-oupton@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.33.0.464.g1972c5931b-goog Subject: [PATCH v8 0/7] KVM: x86: Add idempotent controls for migrating system counter state From: Oliver Upton To: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu Cc: Paolo Bonzini , Sean Christopherson , Marc Zyngier , Peter Shier , Jim Mattson , David Matlack , Ricardo Koller , Jing Zhang , Raghavendra Rao Anata , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Andrew Jones , Will Deacon , Catalin Marinas , Oliver Upton Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org KVM's current means of saving/restoring system counters is plagued with temporal issues. On x86, we migrate the guest's system counter by-value through the respective guest's IA32_TSC value. Restoring system counters by-value is brittle as the state is not idempotent: the host system counter is still oscillating between the attempted save and restore. Furthermore, VMMs may wish to transparently live migrate guest VMs, meaning that they include the elapsed time due to live migration blackout in the guest system counter view. The VMM thread could be preempted for any number of reasons (scheduler, L0 hypervisor under nested) between the time that it calculates the desired guest counter value and when KVM actually sets this counter state. Despite the value-based interface that we present to userspace, KVM actually has idempotent guest controls by way of the TSC offset. We can avoid all of the issues associated with a value-based interface by abstracting these offset controls in a new device attribute. This series introduces new vCPU device attributes to provide userspace access to the vCPU's system counter offset. Patches 1-2 are Paolo's refactorings around locking and the KVM_{GET,SET}_CLOCK ioctls. Patch 3 cures a race where use_master_clock is read outside of the pvclock lock in the KVM_GET_CLOCK ioctl. Patch 4 adopts Paolo's suggestion, augmenting the KVM_{GET,SET}_CLOCK ioctls to provide userspace with a (host_tsc, realtime) instant. This is essential for a VMM to perform precise migration of the guest's system counters. Patch 5 does away with the pvclock spin lock in favor of a sequence lock based on the tsc_write_lock. The original patch is from Paolo, I touched it up a bit to fix a deadlock and some unused variables that caused -Werror to scream. Patch 6 extracts the TSC synchronization tracking code in a way that it can be used for both offset-based and value-based TSC synchronization schemes. Finally, patch 7 implements a vCPU device attribute which allows VMMs to get at the TSC offset of a vCPU. This series was tested with the new KVM selftests for the KVM clock and system counter offset controls on Haswell hardware. Kernel was built with CONFIG_LOCKDEP given the new locking changes/lockdep assertions here. Note that these tests are mailed as a separate series due to the dependencies in both x86 and arm64. Applies cleanly to 5.15-rc1 v8: http://lore.kernel.org/r/20210816001130.3059564-1-oupton@google.com v7 -> v8: - Rebased to 5.15-rc1 - Picked up Paolo's version of the series, which includes locking changes - Make KVM advertise KVM_CAP_VCPU_ATTRIBUTES Oliver Upton (4): KVM: x86: Fix potential race in KVM_GET_CLOCK KVM: x86: Report host tsc and realtime values in KVM_GET_CLOCK KVM: x86: Refactor tsc synchronization code KVM: x86: Expose TSC offset controls to userspace Paolo Bonzini (3): kvm: x86: abstract locking around pvclock_update_vm_gtod_copy KVM: x86: extract KVM_GET_CLOCK/KVM_SET_CLOCK to separate functions kvm: x86: protect masterclock with a seqcount Documentation/virt/kvm/api.rst | 42 ++- Documentation/virt/kvm/devices/vcpu.rst | 57 +++ arch/x86/include/asm/kvm_host.h | 12 +- arch/x86/include/uapi/asm/kvm.h | 4 + arch/x86/kvm/x86.c | 458 ++++++++++++++++-------- include/uapi/linux/kvm.h | 7 +- 6 files changed, 419 insertions(+), 161 deletions(-) -- 2.33.0.309.g3052b89438-goog