From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13F72C4338F for ; Wed, 11 Aug 2021 18:56:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EA0076108C for ; Wed, 11 Aug 2021 18:56:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230147AbhHKS47 (ORCPT ); Wed, 11 Aug 2021 14:56:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35470 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229655AbhHKS47 (ORCPT ); Wed, 11 Aug 2021 14:56:59 -0400 Received: from mail-lf1-x12c.google.com (mail-lf1-x12c.google.com [IPv6:2a00:1450:4864:20::12c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2E6AFC061765 for ; Wed, 11 Aug 2021 11:56:35 -0700 (PDT) Received: by mail-lf1-x12c.google.com with SMTP id z2so7951525lft.1 for ; Wed, 11 Aug 2021 11:56:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ScGXpGBVKuemsdnjrMZtpirVRiYqp0nxEwpdjzvAcQc=; b=BKzD/phz2V62L+BYy2l6IT1uRjXbNsfEnzJ1FP0i4QNPcJzu7wq5do8D8vcLmOFMdd 7KpQTTNtf52WOneMY8EYnz8BGgQdqj2Hn1/PyugV0cMwUsu28juY0d7gBj+7y3ap/Yr/ wbBDtL9syqcrir563sm91vau1YSpqnaTb42w7lyS7D4tUUzvIWRWTiL6mDX8BbNyH+6/ 6bi6aiP5Zmp3R99aKSL7YAvBriWg+BAJBHE+BFjlz2Hbb+1bY+8a49+1+3ekTgERfjHp OF7r5f7z69rrUu95q0CIjZuxKjsDyvAJjD6vCEwiKG1XrUAe0CFLx0GFn3FX5zjSO+Sv UhHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ScGXpGBVKuemsdnjrMZtpirVRiYqp0nxEwpdjzvAcQc=; b=cax6XEr/wVhzFmY0PeMwmiE+ChhlkC8k2w1qpuMt/nVBVXunV5pIO9L/f79DiZZnvV vl+CfhfsnZ2ylXNrkRSUCwRsn4+WoFN365YuSWA3Nz+V/UTg/loetQrny+kCAZHTgG4G CFCFDM8HiVVMZSN7hAYabYf/ak0nPCO99VOLOd5q4+2ZhyEBOPJLAWLD3R/WBU+iAkzR n+tbEwCZI7K7cXyVByEUhiWEhtzFONIIr3/J/Jk2y7Un4cFW/ZmhWJRC5vbZF452oFWB 3yRJ7vek/Y3wHCghltOOkbs1TJLiv/KHw6689vaEZLMjB6Q2Me3pUYNuIbha2nz69l4O UQsA== X-Gm-Message-State: AOAM530HiZx3z8ortNWm0EVsOKlQRhEiFPDZyzs9yqf4ogm/Y3mxqFdS 4Y1wTFqbbfC19IknYqmK/rZ9az7+tlS8NohYts++ng== X-Google-Smtp-Source: ABdhPJwGtbhMqHwJKvW91HNOMAe2ToUWMYoIqEwOcawUfgK0gps7jUCiF6LzH6yuRTJPb08sopXFSG2mGtm6DyX1DtU= X-Received: by 2002:ac2:5fc7:: with SMTP id q7mr183843lfg.524.1628708193196; Wed, 11 Aug 2021 11:56:33 -0700 (PDT) MIME-Version: 1.0 References: <20210804085819.846610-1-oupton@google.com> <927240ff-a4f4-fcc6-ae1b-92cefeda9e59@redhat.com> In-Reply-To: <927240ff-a4f4-fcc6-ae1b-92cefeda9e59@redhat.com> From: Oliver Upton Date: Wed, 11 Aug 2021 11:56:22 -0700 Message-ID: Subject: Re: [PATCH v6 00/21] KVM: Add idempotent controls for migrating system counter state To: Paolo Bonzini Cc: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, Sean Christopherson , Marc Zyngier , Peter Shier , Jim Mattson , David Matlack , Ricardo Koller , Jing Zhang , Raghavendra Rao Anata , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Andrew Jones , Will Deacon , Catalin Marinas Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org On Wed, Aug 11, 2021 at 6:05 AM Paolo Bonzini wrote: > > On 04/08/21 10:57, Oliver Upton wrote: > > KVM's current means of saving/restoring system counters is plagued with > > temporal issues. At least on ARM64 and x86, we migrate the guest's > > system counter by-value through the respective guest system register > > values (cntvct_el0, ia32_tsc). Restoring system counters by-value is > > brittle as the state is not idempotent: the host system counter is still > > oscillating between the attempted save and restore. Furthermore, VMMs > > may wish to transparently live migrate guest VMs, meaning that they > > include the elapsed time due to live migration blackout in the guest > > system counter view. The VMM thread could be preempted for any number of > > reasons (scheduler, L0 hypervisor under nested) between the time that > > it calculates the desired guest counter value and when KVM actually sets > > this counter state. > > > > Despite the value-based interface that we present to userspace, KVM > > actually has idempotent guest controls by way of system counter offsets. > > We can avoid all of the issues associated with a value-based interface > > by abstracting these offset controls in new ioctls. This series > > introduces new vCPU device attributes to provide userspace access to the > > vCPU's system counter offset. > > > > Patch 1 addresses a possible race in KVM_GET_CLOCK where > > use_master_clock is read outside of the pvclock_gtod_sync_lock. > > > > Patch 2 adopts Paolo's suggestion, augmenting the KVM_{GET,SET}_CLOCK > > ioctls to provide userspace with a (host_tsc, realtime) instant. This is > > essential for a VMM to perform precise migration of the guest's system > > counters. > > > > Patches 3-4 are some preparatory changes for exposing the TSC offset to > > userspace. Patch 5 provides a vCPU attribute to provide userspace access > > to the TSC offset. > > > > Patches 6-7 implement a test for the new additions to > > KVM_{GET,SET}_CLOCK. > > > > Patch 8 fixes some assertions in the kvm device attribute helpers. > > > > Patches 9-10 implement at test for the tsc offset attribute introduced in > > patch 5. > > The x86 parts look good, except that patch 3 is a bit redundant with my > idea of altogether getting rid of the pvclock_gtod_sync_lock. That said > I agree that patches 1 and 2 (and extracting kvm_vm_ioctl_get_clock and > kvm_vm_ioctl_set_clock) should be done before whatever locking changes > have to be done. Following up on patch 3. > Time is ticking for 5.15 due to my vacation, I'll see if I have some > time to look at it further next week. > > I agree that arm64 can be done separately from x86. Marc, just a disclaimer: I'm going to separate these two series, although there will still exist dependencies in the selftests changes. Otherwise, kernel changes are disjoint. -- Thanks, Oliver From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.5 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3661CC4338F for ; Wed, 11 Aug 2021 18:56:42 +0000 (UTC) Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by mail.kernel.org (Postfix) with ESMTP id 9E5E860560 for ; Wed, 11 Aug 2021 18:56:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 9E5E860560 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.cs.columbia.edu Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 0275949DE7; Wed, 11 Aug 2021 14:56:41 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@google.com Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 22Kz1iidj31F; Wed, 11 Aug 2021 14:56:37 -0400 (EDT) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 0DA5C4A32E; Wed, 11 Aug 2021 14:56:37 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 0E4AC4A2E5 for ; Wed, 11 Aug 2021 14:56:36 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8AK4J-9RnKgi for ; Wed, 11 Aug 2021 14:56:35 -0400 (EDT) Received: from mail-lf1-f46.google.com (mail-lf1-f46.google.com [209.85.167.46]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id CB4964A1AF for ; Wed, 11 Aug 2021 14:56:34 -0400 (EDT) Received: by mail-lf1-f46.google.com with SMTP id c24so7834805lfi.11 for ; Wed, 11 Aug 2021 11:56:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ScGXpGBVKuemsdnjrMZtpirVRiYqp0nxEwpdjzvAcQc=; b=BKzD/phz2V62L+BYy2l6IT1uRjXbNsfEnzJ1FP0i4QNPcJzu7wq5do8D8vcLmOFMdd 7KpQTTNtf52WOneMY8EYnz8BGgQdqj2Hn1/PyugV0cMwUsu28juY0d7gBj+7y3ap/Yr/ wbBDtL9syqcrir563sm91vau1YSpqnaTb42w7lyS7D4tUUzvIWRWTiL6mDX8BbNyH+6/ 6bi6aiP5Zmp3R99aKSL7YAvBriWg+BAJBHE+BFjlz2Hbb+1bY+8a49+1+3ekTgERfjHp OF7r5f7z69rrUu95q0CIjZuxKjsDyvAJjD6vCEwiKG1XrUAe0CFLx0GFn3FX5zjSO+Sv UhHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ScGXpGBVKuemsdnjrMZtpirVRiYqp0nxEwpdjzvAcQc=; b=cOosXKg9viNwFlUR8SggpganHLisz6vmYlPfQQUj2tQEnMaia8PE4X29ImDXI+tyMm GJKRMdi3/+Ago1J40hiDgBCNUbasO/h+/HjcEx96uzh8iSSYr0TSUb9d8rHFTsy7FpTt BHkDovoOw7KUqxtDMfmZkPWTaIlnmHPIDlVQUaaYoKTh++ObTGG9DNR/kUMa2OM2fO5V MRD8L29Y08XTfAW/oKo7w+9Nu3XNAlafMCoIQ6YbIBmK6FIeLokFIvke+RE5pS3iJn6W LAS/6Iq2b4IMZtR3i89ZpxChutjGUDltUWjfy8ZgfkLr/bSQHeAfyREXyq6+Zi2ngQFR 9Xbw== X-Gm-Message-State: AOAM533xUK2Ui2eLPhc0yghthsNVujd+krRK8Ayf+m5YViiMgrGFYwOa CebN1GZs/InrkDL0jmLk0T4OKlUMM7zwqtJPAQXb8A== X-Google-Smtp-Source: ABdhPJwGtbhMqHwJKvW91HNOMAe2ToUWMYoIqEwOcawUfgK0gps7jUCiF6LzH6yuRTJPb08sopXFSG2mGtm6DyX1DtU= X-Received: by 2002:ac2:5fc7:: with SMTP id q7mr183843lfg.524.1628708193196; Wed, 11 Aug 2021 11:56:33 -0700 (PDT) MIME-Version: 1.0 References: <20210804085819.846610-1-oupton@google.com> <927240ff-a4f4-fcc6-ae1b-92cefeda9e59@redhat.com> In-Reply-To: <927240ff-a4f4-fcc6-ae1b-92cefeda9e59@redhat.com> From: Oliver Upton Date: Wed, 11 Aug 2021 11:56:22 -0700 Message-ID: Subject: Re: [PATCH v6 00/21] KVM: Add idempotent controls for migrating system counter state To: Paolo Bonzini Cc: Catalin Marinas , kvm@vger.kernel.org, Will Deacon , Sean Christopherson , Raghavendra Rao Anata , Peter Shier , Marc Zyngier , David Matlack , kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, Jim Mattson X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu On Wed, Aug 11, 2021 at 6:05 AM Paolo Bonzini wrote: > > On 04/08/21 10:57, Oliver Upton wrote: > > KVM's current means of saving/restoring system counters is plagued with > > temporal issues. At least on ARM64 and x86, we migrate the guest's > > system counter by-value through the respective guest system register > > values (cntvct_el0, ia32_tsc). Restoring system counters by-value is > > brittle as the state is not idempotent: the host system counter is still > > oscillating between the attempted save and restore. Furthermore, VMMs > > may wish to transparently live migrate guest VMs, meaning that they > > include the elapsed time due to live migration blackout in the guest > > system counter view. The VMM thread could be preempted for any number of > > reasons (scheduler, L0 hypervisor under nested) between the time that > > it calculates the desired guest counter value and when KVM actually sets > > this counter state. > > > > Despite the value-based interface that we present to userspace, KVM > > actually has idempotent guest controls by way of system counter offsets. > > We can avoid all of the issues associated with a value-based interface > > by abstracting these offset controls in new ioctls. This series > > introduces new vCPU device attributes to provide userspace access to the > > vCPU's system counter offset. > > > > Patch 1 addresses a possible race in KVM_GET_CLOCK where > > use_master_clock is read outside of the pvclock_gtod_sync_lock. > > > > Patch 2 adopts Paolo's suggestion, augmenting the KVM_{GET,SET}_CLOCK > > ioctls to provide userspace with a (host_tsc, realtime) instant. This is > > essential for a VMM to perform precise migration of the guest's system > > counters. > > > > Patches 3-4 are some preparatory changes for exposing the TSC offset to > > userspace. Patch 5 provides a vCPU attribute to provide userspace access > > to the TSC offset. > > > > Patches 6-7 implement a test for the new additions to > > KVM_{GET,SET}_CLOCK. > > > > Patch 8 fixes some assertions in the kvm device attribute helpers. > > > > Patches 9-10 implement at test for the tsc offset attribute introduced in > > patch 5. > > The x86 parts look good, except that patch 3 is a bit redundant with my > idea of altogether getting rid of the pvclock_gtod_sync_lock. That said > I agree that patches 1 and 2 (and extracting kvm_vm_ioctl_get_clock and > kvm_vm_ioctl_set_clock) should be done before whatever locking changes > have to be done. Following up on patch 3. > Time is ticking for 5.15 due to my vacation, I'll see if I have some > time to look at it further next week. > > I agree that arm64 can be done separately from x86. Marc, just a disclaimer: I'm going to separate these two series, although there will still exist dependencies in the selftests changes. Otherwise, kernel changes are disjoint. -- Thanks, Oliver _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4368FC4338F for ; Wed, 11 Aug 2021 18:58:54 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 001416105A for ; Wed, 11 Aug 2021 18:58:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 001416105A Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=BM3T6QIxeYJ0JbVdY38H/YXoNSa0D/7/0KnNyxSyId8=; b=413eA9xOUIbWQd sssQhHxSW3Dy951Shcy/PGcfwH6+pgCTl8mJpuT7WSnsXo9gsrPtpSPbTpFRZuBPKvlYZnW0B0wTa UvKcnnpog/wrc7KAXehYbxPPs/VG0MckuzjHrIkqnb5TActEFELmQaZGbBGU4OePntBwf+b/+WoMP mwK1hkq7dLrcAS0ObN70KbWdsnHq5s2JczAFHI3f0p0Duimo5lNgxAoyFZvfok/vL3jYkuEbALe58 2HnhWMQ8Onwxaf3lAA1VJwILjpvSfgi8/W01Fwi3WMdd2j08htnWiFqayMxaU+uBGT3+QJbdnJu46 FSh8Dd4vqvXSooiKbKpA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mDtP8-007w8j-2K; Wed, 11 Aug 2021 18:56:42 +0000 Received: from mail-lf1-x12a.google.com ([2a00:1450:4864:20::12a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mDtP1-007w7T-H5 for linux-arm-kernel@lists.infradead.org; Wed, 11 Aug 2021 18:56:40 +0000 Received: by mail-lf1-x12a.google.com with SMTP id g30so7920330lfv.4 for ; Wed, 11 Aug 2021 11:56:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ScGXpGBVKuemsdnjrMZtpirVRiYqp0nxEwpdjzvAcQc=; b=BKzD/phz2V62L+BYy2l6IT1uRjXbNsfEnzJ1FP0i4QNPcJzu7wq5do8D8vcLmOFMdd 7KpQTTNtf52WOneMY8EYnz8BGgQdqj2Hn1/PyugV0cMwUsu28juY0d7gBj+7y3ap/Yr/ wbBDtL9syqcrir563sm91vau1YSpqnaTb42w7lyS7D4tUUzvIWRWTiL6mDX8BbNyH+6/ 6bi6aiP5Zmp3R99aKSL7YAvBriWg+BAJBHE+BFjlz2Hbb+1bY+8a49+1+3ekTgERfjHp OF7r5f7z69rrUu95q0CIjZuxKjsDyvAJjD6vCEwiKG1XrUAe0CFLx0GFn3FX5zjSO+Sv UhHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ScGXpGBVKuemsdnjrMZtpirVRiYqp0nxEwpdjzvAcQc=; b=LVBXxmVruO+aDg6lIHLgkFUMqPuDTWUbGlV50o7hGjnCrTq1nm6en/83czDI2Xx8PD R+eViWSNwnhkp0COSgml5im76hMwY4dIZyrEnVKsbGdL2n1zj3Mokt3FXSsyHS3QqVyg jJ1q7/GKDrehDbRZ6V7YnruSDe3lcC/m/Yz227jt5acR/lWNE7s5PS95iU61Ht30Ps2l enDzyF5ku4pKiFEfhkrp5G7t49HywOL3lY/YOv9lO9QwEW/2E0KuUBv1IpDl4avXXeOZ pQhY/24tYUh+Wr5KLaG1htJGhFentI40z31pKAG6K542IZNV6c6zREfcRG5yLYtvDoWr mi/A== X-Gm-Message-State: AOAM531r/dWTTdZg3nm2VAS0QZcxR8SRh/uf4wRA3sqgXQbBcw9mPHfK imMr8U/dKGkCGc20AaOECE0+gnO6ax6oEuVFO6z6Ew== X-Google-Smtp-Source: ABdhPJwGtbhMqHwJKvW91HNOMAe2ToUWMYoIqEwOcawUfgK0gps7jUCiF6LzH6yuRTJPb08sopXFSG2mGtm6DyX1DtU= X-Received: by 2002:ac2:5fc7:: with SMTP id q7mr183843lfg.524.1628708193196; Wed, 11 Aug 2021 11:56:33 -0700 (PDT) MIME-Version: 1.0 References: <20210804085819.846610-1-oupton@google.com> <927240ff-a4f4-fcc6-ae1b-92cefeda9e59@redhat.com> In-Reply-To: <927240ff-a4f4-fcc6-ae1b-92cefeda9e59@redhat.com> From: Oliver Upton Date: Wed, 11 Aug 2021 11:56:22 -0700 Message-ID: Subject: Re: [PATCH v6 00/21] KVM: Add idempotent controls for migrating system counter state To: Paolo Bonzini Cc: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, Sean Christopherson , Marc Zyngier , Peter Shier , Jim Mattson , David Matlack , Ricardo Koller , Jing Zhang , Raghavendra Rao Anata , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Andrew Jones , Will Deacon , Catalin Marinas X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210811_115635_624921_E5A0FAF6 X-CRM114-Status: GOOD ( 28.33 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, Aug 11, 2021 at 6:05 AM Paolo Bonzini wrote: > > On 04/08/21 10:57, Oliver Upton wrote: > > KVM's current means of saving/restoring system counters is plagued with > > temporal issues. At least on ARM64 and x86, we migrate the guest's > > system counter by-value through the respective guest system register > > values (cntvct_el0, ia32_tsc). Restoring system counters by-value is > > brittle as the state is not idempotent: the host system counter is still > > oscillating between the attempted save and restore. Furthermore, VMMs > > may wish to transparently live migrate guest VMs, meaning that they > > include the elapsed time due to live migration blackout in the guest > > system counter view. The VMM thread could be preempted for any number of > > reasons (scheduler, L0 hypervisor under nested) between the time that > > it calculates the desired guest counter value and when KVM actually sets > > this counter state. > > > > Despite the value-based interface that we present to userspace, KVM > > actually has idempotent guest controls by way of system counter offsets. > > We can avoid all of the issues associated with a value-based interface > > by abstracting these offset controls in new ioctls. This series > > introduces new vCPU device attributes to provide userspace access to the > > vCPU's system counter offset. > > > > Patch 1 addresses a possible race in KVM_GET_CLOCK where > > use_master_clock is read outside of the pvclock_gtod_sync_lock. > > > > Patch 2 adopts Paolo's suggestion, augmenting the KVM_{GET,SET}_CLOCK > > ioctls to provide userspace with a (host_tsc, realtime) instant. This is > > essential for a VMM to perform precise migration of the guest's system > > counters. > > > > Patches 3-4 are some preparatory changes for exposing the TSC offset to > > userspace. Patch 5 provides a vCPU attribute to provide userspace access > > to the TSC offset. > > > > Patches 6-7 implement a test for the new additions to > > KVM_{GET,SET}_CLOCK. > > > > Patch 8 fixes some assertions in the kvm device attribute helpers. > > > > Patches 9-10 implement at test for the tsc offset attribute introduced in > > patch 5. > > The x86 parts look good, except that patch 3 is a bit redundant with my > idea of altogether getting rid of the pvclock_gtod_sync_lock. That said > I agree that patches 1 and 2 (and extracting kvm_vm_ioctl_get_clock and > kvm_vm_ioctl_set_clock) should be done before whatever locking changes > have to be done. Following up on patch 3. > Time is ticking for 5.15 due to my vacation, I'll see if I have some > time to look at it further next week. > > I agree that arm64 can be done separately from x86. Marc, just a disclaimer: I'm going to separate these two series, although there will still exist dependencies in the selftests changes. Otherwise, kernel changes are disjoint. -- Thanks, Oliver _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel