From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-21.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2C6BC4338F for ; Mon, 16 Aug 2021 00:11:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7960E613A1 for ; Mon, 16 Aug 2021 00:11:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231697AbhHPAMH (ORCPT ); Sun, 15 Aug 2021 20:12:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38980 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231379AbhHPAMF (ORCPT ); Sun, 15 Aug 2021 20:12:05 -0400 Received: from mail-io1-xd49.google.com (mail-io1-xd49.google.com [IPv6:2607:f8b0:4864:20::d49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F1BE9C061764 for ; Sun, 15 Aug 2021 17:11:34 -0700 (PDT) Received: by mail-io1-xd49.google.com with SMTP id s22-20020a5e98160000b02905afde383110so4774605ioj.8 for ; Sun, 15 Aug 2021 17:11:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:message-id:mime-version:subject:from:to:cc; bh=iJYsu7fpasA4KCIA5cuNFhT9/QgsjAss3yu7wk60u8Q=; b=aQj/I1mY6r1r03IW3BWlNheUhL3RDYxx/4BIrimzwkzt2gfhIN0A/tEFP0+K8V90Cb zKOXBSuylCsQDzsL6PKOCbhHzPGaGBvEjREeSHMXlsWBrwkycNJXsX+SXhJltmySqdbK 0cP3Dur3EwL6skxpgnFzlPK4SKLVQkZ3dYumZnisLLglIrZdTmu6K+DSv77KoVVtKS9+ XU3+hWyiPbJPXOwf+NesHBYEuz9EfXIqrStikdtcpAorkuSS1tERCKbQqUPs7FYKEZY3 O1roXtb6SaQoT+f3WJVvzZ1103URmkOQIs2VyJAu7ES9jAZAYzXa2e1wv7mCIVAjrSo3 hyjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=iJYsu7fpasA4KCIA5cuNFhT9/QgsjAss3yu7wk60u8Q=; b=chzaRG2lJW/RMG7LY/y8nYWZ1oqLiVTwGFUZ/1nb4YbBq+Cx1SBsRKTgZ3596w961a ktJgUHbxYdPy1DM0zkJnYk1uBjfivV5qOLTxpidsL9tZ4AnLkUZubZj+h4ZtLYZMncVD 8QwrLPQ5et5G84Nk5Z30ra9KmsoDDHSoi76kFib/rMqhzgQ+UBWGLSlyigCS9bj0ZSEL 3tSHU3AQ4tEImb8Nz2sYgTVFhF1952m84LA4KGWrvX246uOFn13as3jirxrJvXGA3EU7 TcmqdHSRq5hVWQiW9fm0PQBR/TLcgsABcxhrKBC5dqI8nqikUciRAodyez4LxSFL9vkN ysYg== X-Gm-Message-State: AOAM5319EEX9VpxRs+SWsbY7oWSECMsC2mzC2J3z+FQqOD59+3rdupzs O6fEy62zRgk13bGkpHtI94I8v730e8C/2v6gLu0KEuLIN3PXmkvnJqyoEHwUfBb66ADBqjkT46r rWhYUeffuR/oIx/qpL0FbgG7BqVAY7T5qlrrUYymaSnC+p7bASWGraXrQ6g== X-Google-Smtp-Source: ABdhPJyyEq1JPYrQ+owXHLwmCeY/SoZMfZe5R92+IYuMRt/23mdj7nVwa7/HPh0wkyoJLBOG25OQdvvomhs= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a5e:8915:: with SMTP id k21mr4192645ioj.137.1629072694172; Sun, 15 Aug 2021 17:11:34 -0700 (PDT) Date: Mon, 16 Aug 2021 00:11:24 +0000 Message-Id: <20210816001130.3059564-1-oupton@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.33.0.rc1.237.g0d66db33f3-goog Subject: [PATCH v7 0/6] KVM: x86: Add idempotent controls for migrating system counter state From: Oliver Upton To: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu Cc: Paolo Bonzini , Sean Christopherson , Marc Zyngier , Peter Shier , Jim Mattson , David Matlack , Ricardo Koller , Jing Zhang , Raghavendra Rao Anata , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Andrew Jones , Will Deacon , Catalin Marinas , Oliver Upton Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org KVM's current means of saving/restoring system counters is plagued with temporal issues. On x86, we migrate the guest's system counter by-value through the respective guest's IA32_TSC value. Restoring system counters by-value is brittle as the state is not idempotent: the host system counter is still oscillating between the attempted save and restore. Furthermore, VMMs may wish to transparently live migrate guest VMs, meaning that they include the elapsed time due to live migration blackout in the guest system counter view. The VMM thread could be preempted for any number of reasons (scheduler, L0 hypervisor under nested) between the time that it calculates the desired guest counter value and when KVM actually sets this counter state. Despite the value-based interface that we present to userspace, KVM actually has idempotent guest controls by way of the TSC offset. We can avoid all of the issues associated with a value-based interface by abstracting these offset controls in a new device attribute. This series introduces new vCPU device attributes to provide userspace access to the vCPU's system counter offset. Patch 1 addresses a possible race in KVM_GET_CLOCK where use_master_clock is read outside of the pvclock_gtod_sync_lock. Patch 2 is a cleanup, moving the implementation of KVM_{GET,SET}_CLOCK into helper methods. Patch 3 adopts Paolo's suggestion, augmenting the KVM_{GET,SET}_CLOCK ioctls to provide userspace with a (host_tsc, realtime) instant. This is essential for a VMM to perform precise migration of the guest's system counters. Patches 4-5 are some preparatory changes for exposing the TSC offset to userspace. Patch 6 provides a vCPU attribute to provide userspace access to the TSC offset. This series was tested with the new KVM selftests for the KVM clock and system counter offset controls on Haswell hardware. Note that these tests are mailed as a separate series due to the dependencies in both x86 and arm64. Applies cleanly to kvm/queue. Parent commit: a3e0b8bd99ab ("KVM: MMU: change tracepoints arguments to kvm_page_fault") v6: https://lore.kernel.org/r/20210804085819.846610-1-oupton@google.com v6 -> v7: - Separated x86, arm64, and selftests into different series - Rebased on top of kvm/queue Oliver Upton (6): KVM: x86: Fix potential race in KVM_GET_CLOCK KVM: x86: Create helper methods for KVM_{GET,SET}_CLOCK ioctls KVM: x86: Report host tsc and realtime values in KVM_GET_CLOCK KVM: x86: Take the pvclock sync lock behind the tsc_write_lock KVM: x86: Refactor tsc synchronization code KVM: x86: Expose TSC offset controls to userspace Documentation/virt/kvm/api.rst | 42 ++- Documentation/virt/kvm/devices/vcpu.rst | 57 ++++ Documentation/virt/kvm/locking.rst | 11 + arch/x86/include/asm/kvm_host.h | 4 + arch/x86/include/uapi/asm/kvm.h | 4 + arch/x86/kvm/x86.c | 362 +++++++++++++++++------- include/uapi/linux/kvm.h | 7 +- 7 files changed, 378 insertions(+), 109 deletions(-) -- 2.33.0.rc1.237.g0d66db33f3-goog From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4102AC4338F for ; Mon, 16 Aug 2021 00:11:42 +0000 (UTC) Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by mail.kernel.org (Postfix) with ESMTP id AC6E1613A1 for ; Mon, 16 Aug 2021 00:11:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org AC6E1613A1 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.cs.columbia.edu Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 1D7684B0BC; Sun, 15 Aug 2021 20:11:41 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@google.com Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hnFaSHFuqzLL; Sun, 15 Aug 2021 20:11:37 -0400 (EDT) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 325724B0D9; Sun, 15 Aug 2021 20:11:37 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 09C474B0CE for ; Sun, 15 Aug 2021 20:11:36 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2fOO6GCZjDqs for ; Sun, 15 Aug 2021 20:11:34 -0400 (EDT) Received: from mail-io1-f74.google.com (mail-io1-f74.google.com [209.85.166.74]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id C89D64B0AC for ; Sun, 15 Aug 2021 20:11:34 -0400 (EDT) Received: by mail-io1-f74.google.com with SMTP id d23-20020a056602281700b005b5b34670c7so1335395ioe.12 for ; Sun, 15 Aug 2021 17:11:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:message-id:mime-version:subject:from:to:cc; bh=iJYsu7fpasA4KCIA5cuNFhT9/QgsjAss3yu7wk60u8Q=; b=aQj/I1mY6r1r03IW3BWlNheUhL3RDYxx/4BIrimzwkzt2gfhIN0A/tEFP0+K8V90Cb zKOXBSuylCsQDzsL6PKOCbhHzPGaGBvEjREeSHMXlsWBrwkycNJXsX+SXhJltmySqdbK 0cP3Dur3EwL6skxpgnFzlPK4SKLVQkZ3dYumZnisLLglIrZdTmu6K+DSv77KoVVtKS9+ XU3+hWyiPbJPXOwf+NesHBYEuz9EfXIqrStikdtcpAorkuSS1tERCKbQqUPs7FYKEZY3 O1roXtb6SaQoT+f3WJVvzZ1103URmkOQIs2VyJAu7ES9jAZAYzXa2e1wv7mCIVAjrSo3 hyjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=iJYsu7fpasA4KCIA5cuNFhT9/QgsjAss3yu7wk60u8Q=; b=NvGR17r8qFlQDugyq/Y0qEcDB58NG/hY40fuKBcCKIjvzFePdi4YdGAGXRdHV5HyIu FWuUg1n9IotOk5jpf5ECVZ6OCFos4TgRvk2fjO7tbEvGuDpddavF1fdPPrUchxGJr5YT x+yJ3BAvpXvb4vUVIfOwH50XkqdkVMUWTBH5WQhaj2VSWW0XnAIYs5yE+RfdfsIpPIma uO+eZRM72FFEfi9lcAseHjsB8lfpnAmzGRmvdp+CvDNeOoDjvjfk4BmFMRn3PSAR1fiU axjE5pb6+XXUUyO50KCqJKCa+Lh0tT4PRuBUQxg8euMH0rr+QoRgBqcpIYNaiHvzcogz o03Q== X-Gm-Message-State: AOAM531QnYOtBxNVgEyY2LaqdpNMfx3hP0pjhF+9k0+7CT92h9oNVr8b GwZebDyB19AbH5qjapTMt8e2cVHZItE= X-Google-Smtp-Source: ABdhPJyyEq1JPYrQ+owXHLwmCeY/SoZMfZe5R92+IYuMRt/23mdj7nVwa7/HPh0wkyoJLBOG25OQdvvomhs= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a5e:8915:: with SMTP id k21mr4192645ioj.137.1629072694172; Sun, 15 Aug 2021 17:11:34 -0700 (PDT) Date: Mon, 16 Aug 2021 00:11:24 +0000 Message-Id: <20210816001130.3059564-1-oupton@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.33.0.rc1.237.g0d66db33f3-goog Subject: [PATCH v7 0/6] KVM: x86: Add idempotent controls for migrating system counter state From: Oliver Upton To: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu Cc: Catalin Marinas , Will Deacon , Marc Zyngier , Raghavendra Rao Anata , Peter Shier , Sean Christopherson , David Matlack , Paolo Bonzini , linux-arm-kernel@lists.infradead.org, Jim Mattson X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu KVM's current means of saving/restoring system counters is plagued with temporal issues. On x86, we migrate the guest's system counter by-value through the respective guest's IA32_TSC value. Restoring system counters by-value is brittle as the state is not idempotent: the host system counter is still oscillating between the attempted save and restore. Furthermore, VMMs may wish to transparently live migrate guest VMs, meaning that they include the elapsed time due to live migration blackout in the guest system counter view. The VMM thread could be preempted for any number of reasons (scheduler, L0 hypervisor under nested) between the time that it calculates the desired guest counter value and when KVM actually sets this counter state. Despite the value-based interface that we present to userspace, KVM actually has idempotent guest controls by way of the TSC offset. We can avoid all of the issues associated with a value-based interface by abstracting these offset controls in a new device attribute. This series introduces new vCPU device attributes to provide userspace access to the vCPU's system counter offset. Patch 1 addresses a possible race in KVM_GET_CLOCK where use_master_clock is read outside of the pvclock_gtod_sync_lock. Patch 2 is a cleanup, moving the implementation of KVM_{GET,SET}_CLOCK into helper methods. Patch 3 adopts Paolo's suggestion, augmenting the KVM_{GET,SET}_CLOCK ioctls to provide userspace with a (host_tsc, realtime) instant. This is essential for a VMM to perform precise migration of the guest's system counters. Patches 4-5 are some preparatory changes for exposing the TSC offset to userspace. Patch 6 provides a vCPU attribute to provide userspace access to the TSC offset. This series was tested with the new KVM selftests for the KVM clock and system counter offset controls on Haswell hardware. Note that these tests are mailed as a separate series due to the dependencies in both x86 and arm64. Applies cleanly to kvm/queue. Parent commit: a3e0b8bd99ab ("KVM: MMU: change tracepoints arguments to kvm_page_fault") v6: https://lore.kernel.org/r/20210804085819.846610-1-oupton@google.com v6 -> v7: - Separated x86, arm64, and selftests into different series - Rebased on top of kvm/queue Oliver Upton (6): KVM: x86: Fix potential race in KVM_GET_CLOCK KVM: x86: Create helper methods for KVM_{GET,SET}_CLOCK ioctls KVM: x86: Report host tsc and realtime values in KVM_GET_CLOCK KVM: x86: Take the pvclock sync lock behind the tsc_write_lock KVM: x86: Refactor tsc synchronization code KVM: x86: Expose TSC offset controls to userspace Documentation/virt/kvm/api.rst | 42 ++- Documentation/virt/kvm/devices/vcpu.rst | 57 ++++ Documentation/virt/kvm/locking.rst | 11 + arch/x86/include/asm/kvm_host.h | 4 + arch/x86/include/uapi/asm/kvm.h | 4 + arch/x86/kvm/x86.c | 362 +++++++++++++++++------- include/uapi/linux/kvm.h | 7 +- 7 files changed, 378 insertions(+), 109 deletions(-) -- 2.33.0.rc1.237.g0d66db33f3-goog _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13DFBC432BE for ; Mon, 16 Aug 2021 00:15:57 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D34C661221 for ; Mon, 16 Aug 2021 00:15:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org D34C661221 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Mime-Version: Message-Id:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: References:List-Owner; bh=/8euf7PpExtZ2ACevaHaXmJFlw/l7wO/IA4rNj63PYs=; b=0PN qE9/Xn7evoI5u/y0/ZfpWz01L7EUEP1wprHrRkQhH6I+l+iYPMrefdM2RemyY/mnsQgHHFfb6/oky 5bc8cefKpl1bbv6LPUB3Hrpk9ExKtFkGdiZyOi+kfaQ9AGAptdyv+En/T2ozIhmuVgZ42fHApTGlg oLI6sWq0fTZp2nWYEeLyW1OdhMrN8X1+tKQqv1MY+NRVsx9Jo5NLkYmxyU1qoJbvTgnoXdwnscOcA zw6vBTOO7FUcxBNgENLkw5ytladf1ubH/UtPU4L5C6wOYV4LIvORNFDb8bTT0EOIjZQS1iUWZFucW nQWS2i1Qh7uzuRij9QLTXt+6AUTuDGg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mFQE7-00Fl7S-RJ; Mon, 16 Aug 2021 00:11:40 +0000 Received: from mail-il1-x149.google.com ([2607:f8b0:4864:20::149]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mFQE4-00Fl6F-7Y for linux-arm-kernel@lists.infradead.org; Mon, 16 Aug 2021 00:11:37 +0000 Received: by mail-il1-x149.google.com with SMTP id c4-20020a056e020cc4b02902242bd90889so8683093ilj.20 for ; Sun, 15 Aug 2021 17:11:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:message-id:mime-version:subject:from:to:cc; bh=iJYsu7fpasA4KCIA5cuNFhT9/QgsjAss3yu7wk60u8Q=; b=aQj/I1mY6r1r03IW3BWlNheUhL3RDYxx/4BIrimzwkzt2gfhIN0A/tEFP0+K8V90Cb zKOXBSuylCsQDzsL6PKOCbhHzPGaGBvEjREeSHMXlsWBrwkycNJXsX+SXhJltmySqdbK 0cP3Dur3EwL6skxpgnFzlPK4SKLVQkZ3dYumZnisLLglIrZdTmu6K+DSv77KoVVtKS9+ XU3+hWyiPbJPXOwf+NesHBYEuz9EfXIqrStikdtcpAorkuSS1tERCKbQqUPs7FYKEZY3 O1roXtb6SaQoT+f3WJVvzZ1103URmkOQIs2VyJAu7ES9jAZAYzXa2e1wv7mCIVAjrSo3 hyjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=iJYsu7fpasA4KCIA5cuNFhT9/QgsjAss3yu7wk60u8Q=; b=mgDJ5UQ8JwnZgbFV4EbkNOyP1Ub2jZj58sdfzysDcs8hMlDf4wp52comBGWzcfRFR/ 2mYeTbmYvEoF6732x6GXwk9lkfZPBs40AUL8ZqaOvnY4u6kBWHngAfw5FXHmG4UJ5cI1 E5LzlAYRqr9toaGTnuSvnarrNg7CtlPG0fzLKVS3ugLYy8h5QXA+5+7bZU/JF0MhNpXT tXOKjnmJ76JUU/gyBov7417ISmPgA/k+ilG3Mm4fQr1hIfhzy3j4F/Bwyc9WvQH+5tpM 3eQjwgGkv+sjCvPOJJ6/eDayoVUc0WN7hb+vZzq+wHIEhmwFY6wLg9VqMxBnelSU0ane J0HQ== X-Gm-Message-State: AOAM5337xqBUPGLa11esx+RzVQwZbz0VrW2jTou9UeWFJ3EvwkYLAeFr 6vSapB6IAn4puJJgRhYvBJhKIjq8aPk= X-Google-Smtp-Source: ABdhPJyyEq1JPYrQ+owXHLwmCeY/SoZMfZe5R92+IYuMRt/23mdj7nVwa7/HPh0wkyoJLBOG25OQdvvomhs= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a5e:8915:: with SMTP id k21mr4192645ioj.137.1629072694172; Sun, 15 Aug 2021 17:11:34 -0700 (PDT) Date: Mon, 16 Aug 2021 00:11:24 +0000 Message-Id: <20210816001130.3059564-1-oupton@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.33.0.rc1.237.g0d66db33f3-goog Subject: [PATCH v7 0/6] KVM: x86: Add idempotent controls for migrating system counter state From: Oliver Upton To: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu Cc: Paolo Bonzini , Sean Christopherson , Marc Zyngier , Peter Shier , Jim Mattson , David Matlack , Ricardo Koller , Jing Zhang , Raghavendra Rao Anata , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Andrew Jones , Will Deacon , Catalin Marinas , Oliver Upton X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210815_171136_335712_681B7572 X-CRM114-Status: GOOD ( 15.42 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org KVM's current means of saving/restoring system counters is plagued with temporal issues. On x86, we migrate the guest's system counter by-value through the respective guest's IA32_TSC value. Restoring system counters by-value is brittle as the state is not idempotent: the host system counter is still oscillating between the attempted save and restore. Furthermore, VMMs may wish to transparently live migrate guest VMs, meaning that they include the elapsed time due to live migration blackout in the guest system counter view. The VMM thread could be preempted for any number of reasons (scheduler, L0 hypervisor under nested) between the time that it calculates the desired guest counter value and when KVM actually sets this counter state. Despite the value-based interface that we present to userspace, KVM actually has idempotent guest controls by way of the TSC offset. We can avoid all of the issues associated with a value-based interface by abstracting these offset controls in a new device attribute. This series introduces new vCPU device attributes to provide userspace access to the vCPU's system counter offset. Patch 1 addresses a possible race in KVM_GET_CLOCK where use_master_clock is read outside of the pvclock_gtod_sync_lock. Patch 2 is a cleanup, moving the implementation of KVM_{GET,SET}_CLOCK into helper methods. Patch 3 adopts Paolo's suggestion, augmenting the KVM_{GET,SET}_CLOCK ioctls to provide userspace with a (host_tsc, realtime) instant. This is essential for a VMM to perform precise migration of the guest's system counters. Patches 4-5 are some preparatory changes for exposing the TSC offset to userspace. Patch 6 provides a vCPU attribute to provide userspace access to the TSC offset. This series was tested with the new KVM selftests for the KVM clock and system counter offset controls on Haswell hardware. Note that these tests are mailed as a separate series due to the dependencies in both x86 and arm64. Applies cleanly to kvm/queue. Parent commit: a3e0b8bd99ab ("KVM: MMU: change tracepoints arguments to kvm_page_fault") v6: https://lore.kernel.org/r/20210804085819.846610-1-oupton@google.com v6 -> v7: - Separated x86, arm64, and selftests into different series - Rebased on top of kvm/queue Oliver Upton (6): KVM: x86: Fix potential race in KVM_GET_CLOCK KVM: x86: Create helper methods for KVM_{GET,SET}_CLOCK ioctls KVM: x86: Report host tsc and realtime values in KVM_GET_CLOCK KVM: x86: Take the pvclock sync lock behind the tsc_write_lock KVM: x86: Refactor tsc synchronization code KVM: x86: Expose TSC offset controls to userspace Documentation/virt/kvm/api.rst | 42 ++- Documentation/virt/kvm/devices/vcpu.rst | 57 ++++ Documentation/virt/kvm/locking.rst | 11 + arch/x86/include/asm/kvm_host.h | 4 + arch/x86/include/uapi/asm/kvm.h | 4 + arch/x86/kvm/x86.c | 362 +++++++++++++++++------- include/uapi/linux/kvm.h | 7 +- 7 files changed, 378 insertions(+), 109 deletions(-) -- 2.33.0.rc1.237.g0d66db33f3-goog _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel