From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0CD2DC433F5 for ; Fri, 1 Oct 2021 15:12:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EA2E461242 for ; Fri, 1 Oct 2021 15:12:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1354806AbhJAPOK (ORCPT ); Fri, 1 Oct 2021 11:14:10 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:44452 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231890AbhJAPOJ (ORCPT ); Fri, 1 Oct 2021 11:14:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1633101144; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LW6fp9rG/lvp5l8prGyc0xe8qMlUDZGZChEO4If5ox0=; b=cuh1A0UdTnY4C7/r5YpXA/HDZdcZRHAH0A4gSo61eS6Q6bvNilXlPMVfGSUCyuwmKis9UW d9Zjldz9t86jJba30rS0Y4gkuAlD+rwF25Vi+mJFOhOOQ36ncRL+sI2aA4IN494+ZZ04yL F8X4X6KLO+EiK9dnDq4ILEMbegAALHc= Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-500-CqBgkAwoO12iTrT9UGR6uw-1; Fri, 01 Oct 2021 11:12:23 -0400 X-MC-Unique: CqBgkAwoO12iTrT9UGR6uw-1 Received: by mail-ed1-f69.google.com with SMTP id ec14-20020a0564020d4e00b003cf5630c190so10769058edb.3 for ; Fri, 01 Oct 2021 08:12:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent:subject :content-language:to:cc:references:from:in-reply-to :content-transfer-encoding; bh=LW6fp9rG/lvp5l8prGyc0xe8qMlUDZGZChEO4If5ox0=; b=IEi7p2KDidZ4+PRytJowgmd5dcJ1hUX14SvaFl3/6XD7ldWdS2UxdQqICPLEH4xOZp 8Nqs/1AXGE4i8eCpVkofb72wS+auYCnNxQ1xA4Fo+WTJM7LyiqBBjTkgZuXmPgOk+G1n UGtPzLPdBf6VLgUQ6zyEU+0nii/zCWzuXKmLSHBkc6jJiFMeRucEdrEPVvLqhm32ZR+Z sX4Lf9KAGZDMzyEGzzTjoZU4y0FAPGrz++WUWDFKc4uhVh8NQfNjxJThIktu8qGgiS1k bq3MLdXLWPQTK74E9OJBsiET7cAVc0oJvwF1nlRBxuTz672Ne/oWJn6qsz1cJDn6eGrN rQgA== X-Gm-Message-State: AOAM530L918AEOwQGzc0RBZJHRBFbHRfDputxh2cyFJVI/5Jwfo1zPr7 I9g+WphC3KjdPGryj1IlL1st6yRE+02eLNkkJG7mBtjMoVPli1b5lMN8bXJCz5K5xL1IMvJ02x1 nqybSDEZQ46uJ X-Received: by 2002:a17:907:3e03:: with SMTP id hp3mr6732921ejc.183.1633101142618; Fri, 01 Oct 2021 08:12:22 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxSleQpwCGZXW71w2qzpx6TzvbZz4Gx6iqhZH1Md89m/5OKOBFh+S0Y4ZfLrRfA1AXKxqbKQQ== X-Received: by 2002:a17:907:3e03:: with SMTP id hp3mr6732898ejc.183.1633101142335; Fri, 01 Oct 2021 08:12:22 -0700 (PDT) Received: from [192.168.10.118] ([93.56.162.200]) by smtp.gmail.com with ESMTPSA id ee13sm1227751edb.14.2021.10.01.08.12.20 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 01 Oct 2021 08:12:21 -0700 (PDT) Message-ID: <7901cb84-052d-92b6-1e6a-028396c2c691@redhat.com> Date: Fri, 1 Oct 2021 17:12:20 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.1.0 Subject: Re: [PATCH v8 7/7] KVM: x86: Expose TSC offset controls to userspace Content-Language: en-US To: Marcelo Tosatti Cc: Oliver Upton , kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, Sean Christopherson , Marc Zyngier , Peter Shier , Jim Mattson , David Matlack , Ricardo Koller , Jing Zhang , Raghavendra Rao Anata , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Andrew Jones , Will Deacon , Catalin Marinas References: <20210916181538.968978-1-oupton@google.com> <20210916181538.968978-8-oupton@google.com> <20210930191416.GA19068@fuller.cnet> <48151d08-ee29-2b98-b6e1-f3c8a1ff26bc@redhat.com> <20211001103200.GA39746@fuller.cnet> From: Paolo Bonzini In-Reply-To: <20211001103200.GA39746@fuller.cnet> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org On 01/10/21 12:32, Marcelo Tosatti wrote: >> +1. Invoke the KVM_GET_CLOCK ioctl to record the host TSC (t_0), + >> kvmclock nanoseconds (k_0), and realtime nanoseconds (r_0). + [...] >> +4. Invoke the KVM_SET_CLOCK ioctl, providing the kvmclock >> nanoseconds + (k_0) and realtime nanoseconds (r_0) in their >> respective fields. + Ensure that the KVM_CLOCK_REALTIME flag is >> set in the provided + structure. KVM will advance the VM's >> kvmclock to account for elapsed + time since recording the clock >> values. > > You can't advance both kvmclock (kvmclock_offset variable) and the > TSCs, which would be double counting. > > So you have to either add the elapsed realtime (1) between > KVM_GET_CLOCK to kvmclock (which this patch is doing), or to the > TSCs. If you do both, there is double counting. Am i missing > something? Probably one of these two (but it's worth pointing out both of them): 1) the attribute that's introduced here *replaces* KVM_SET_MSR(MSR_IA32_TSC), so the TSC is not added. 2) the adjustment formula later in the algorithm does not care about how much time passed between step 1 and step 4. It just takes two well known (TSC, kvmclock) pairs, and uses them to ensure the guest TSC is the same on the destination as if the guest was still running on the source. It is irrelevant that one of them is before migration and one is after, all it matters is that one is on the source and one is on the destination. Perhaps we can add to step 6 something like: > +6. Adjust the guest TSC offsets for every vCPU to account for (1) > time + elapsed since recording state and (2) difference in TSCs > between the + source and destination machine: + + new_off_n = t_0 > + off_n + (k_1 - k_0) * freq - t_1 + "off + t - k * freq" is the guest TSC value corresponding to a time of 0 in kvmclock. The above formula ensures that it is the same on the destination as it was on the source. Also, the names are a bit hard to follow. Perhaps t_0 tsc_src t_1 tsc_dest k_0 guest_src k_1 guest_dest r_0 host_src off_n ofs_src[i] new_off_n ofs_dest[i] Paolo From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 747EEC4332F for ; Fri, 1 Oct 2021 15:12:28 +0000 (UTC) Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by mail.kernel.org (Postfix) with ESMTP id F336861242 for ; Fri, 1 Oct 2021 15:12:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org F336861242 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.cs.columbia.edu Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 6E3E44B099; Fri, 1 Oct 2021 11:12:27 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@redhat.com Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IiBeMBxmf2LO; Fri, 1 Oct 2021 11:12:26 -0400 (EDT) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 5CE644B0A3; Fri, 1 Oct 2021 11:12:26 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id F0C874B099 for ; Fri, 1 Oct 2021 11:12:25 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5StGxPNejc9T for ; Fri, 1 Oct 2021 11:12:25 -0400 (EDT) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 19C1640573 for ; Fri, 1 Oct 2021 11:12:25 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1633101144; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LW6fp9rG/lvp5l8prGyc0xe8qMlUDZGZChEO4If5ox0=; b=cuh1A0UdTnY4C7/r5YpXA/HDZdcZRHAH0A4gSo61eS6Q6bvNilXlPMVfGSUCyuwmKis9UW d9Zjldz9t86jJba30rS0Y4gkuAlD+rwF25Vi+mJFOhOOQ36ncRL+sI2aA4IN494+ZZ04yL F8X4X6KLO+EiK9dnDq4ILEMbegAALHc= Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-200-ShlJEXMqPXqkrBVjuSQPdg-1; Fri, 01 Oct 2021 11:12:23 -0400 X-MC-Unique: ShlJEXMqPXqkrBVjuSQPdg-1 Received: by mail-ed1-f69.google.com with SMTP id w6-20020a50d786000000b003dabc563406so6319027edi.17 for ; Fri, 01 Oct 2021 08:12:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent:subject :content-language:to:cc:references:from:in-reply-to :content-transfer-encoding; bh=LW6fp9rG/lvp5l8prGyc0xe8qMlUDZGZChEO4If5ox0=; b=400u0qVPl+qKqupn5P4xM3/Lpu4BGxAApw7Nil0c2T5r8wQBB4Ksh3eANJGsp9gTS7 OD/ihlbq9PVU7EXYZIYRn56p+mjUsoKfAq2OaUrYRUBhsW1S93ThhV0ucWS4RgbX/hqJ Yowffq/DShh9xt69G1S4Ab8jFZuZvTvafxg0bhr/IBvHchO05edx0D8Q1hxtxShwOK1j p0dEfv/O85LkJ1/MLfxmn8KlWiIVCZ5Xd3LFM2vDC+h1KaYMWjUDa2wHee8K4rD6InHi LjFqZowpgLHZsaRnod+B29MPHsCUeSmlS7uVAX9E65ySl+odVSvMiIwoNNUmAIYueHsO fZSA== X-Gm-Message-State: AOAM531Vn4O1XnWnCmjFWpHaWzJpVpHXIlEBNclKSoxyVOgvLyXTJWRc hMLMxZOLTcgoepUUP/9AAFkrsfUndF4aqNEcAMm1R0J+QKPdVS9QtqaWGXHzp8+bIfi+ElWgPq0 4ge5qiUW56D+esIChnWwCxELl X-Received: by 2002:a17:907:3e03:: with SMTP id hp3mr6732923ejc.183.1633101142621; Fri, 01 Oct 2021 08:12:22 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxSleQpwCGZXW71w2qzpx6TzvbZz4Gx6iqhZH1Md89m/5OKOBFh+S0Y4ZfLrRfA1AXKxqbKQQ== X-Received: by 2002:a17:907:3e03:: with SMTP id hp3mr6732898ejc.183.1633101142335; Fri, 01 Oct 2021 08:12:22 -0700 (PDT) Received: from [192.168.10.118] ([93.56.162.200]) by smtp.gmail.com with ESMTPSA id ee13sm1227751edb.14.2021.10.01.08.12.20 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 01 Oct 2021 08:12:21 -0700 (PDT) Message-ID: <7901cb84-052d-92b6-1e6a-028396c2c691@redhat.com> Date: Fri, 1 Oct 2021 17:12:20 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.1.0 Subject: Re: [PATCH v8 7/7] KVM: x86: Expose TSC offset controls to userspace To: Marcelo Tosatti References: <20210916181538.968978-1-oupton@google.com> <20210916181538.968978-8-oupton@google.com> <20210930191416.GA19068@fuller.cnet> <48151d08-ee29-2b98-b6e1-f3c8a1ff26bc@redhat.com> <20211001103200.GA39746@fuller.cnet> From: Paolo Bonzini In-Reply-To: <20211001103200.GA39746@fuller.cnet> Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=pbonzini@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Cc: Catalin Marinas , kvm@vger.kernel.org, Peter Shier , Marc Zyngier , David Matlack , Will Deacon , kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, Jim Mattson X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu On 01/10/21 12:32, Marcelo Tosatti wrote: >> +1. Invoke the KVM_GET_CLOCK ioctl to record the host TSC (t_0), + >> kvmclock nanoseconds (k_0), and realtime nanoseconds (r_0). + [...] >> +4. Invoke the KVM_SET_CLOCK ioctl, providing the kvmclock >> nanoseconds + (k_0) and realtime nanoseconds (r_0) in their >> respective fields. + Ensure that the KVM_CLOCK_REALTIME flag is >> set in the provided + structure. KVM will advance the VM's >> kvmclock to account for elapsed + time since recording the clock >> values. > > You can't advance both kvmclock (kvmclock_offset variable) and the > TSCs, which would be double counting. > > So you have to either add the elapsed realtime (1) between > KVM_GET_CLOCK to kvmclock (which this patch is doing), or to the > TSCs. If you do both, there is double counting. Am i missing > something? Probably one of these two (but it's worth pointing out both of them): 1) the attribute that's introduced here *replaces* KVM_SET_MSR(MSR_IA32_TSC), so the TSC is not added. 2) the adjustment formula later in the algorithm does not care about how much time passed between step 1 and step 4. It just takes two well known (TSC, kvmclock) pairs, and uses them to ensure the guest TSC is the same on the destination as if the guest was still running on the source. It is irrelevant that one of them is before migration and one is after, all it matters is that one is on the source and one is on the destination. Perhaps we can add to step 6 something like: > +6. Adjust the guest TSC offsets for every vCPU to account for (1) > time + elapsed since recording state and (2) difference in TSCs > between the + source and destination machine: + + new_off_n = t_0 > + off_n + (k_1 - k_0) * freq - t_1 + "off + t - k * freq" is the guest TSC value corresponding to a time of 0 in kvmclock. The above formula ensures that it is the same on the destination as it was on the source. Also, the names are a bit hard to follow. Perhaps t_0 tsc_src t_1 tsc_dest k_0 guest_src k_1 guest_dest r_0 host_src off_n ofs_src[i] new_off_n ofs_dest[i] Paolo _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6EE3C433F5 for ; Fri, 1 Oct 2021 15:14:41 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7470161ABA for ; Fri, 1 Oct 2021 15:14:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 7470161ABA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Content-Type: Content-Transfer-Encoding:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:From:References:Cc:To:Subject: MIME-Version:Date:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ZqWunw002cEROUhCT3+IuC7NDZKDV0qteOOr9ndwxvY=; b=NLo1xtBIxp1yZG D/Tr12MLdIIWDKOvWTqcz6+cY1+AHw0cr++xrVoI7P19+3EdkdocmzVzKcVGHaUNPxLbLdUGuMKpC POfp6hmWLQoIWc6sQvH63pYRXLePJcOq/e14GuLfLsdDefPFhCkBkBri8S3ucz84rIiHKutBXPeyC DeaLmSZGgKsVTm83nni8H3DsF9W3bqzAG+L6Rn/MPO1was/UIViBR1mbiT1uVsH6+cTdn5MnjqHU1 1jCT8AVFz+nuKZstbjnM1GG+z5qxRMtPNIiQCHBOud/hhbnfWeD06aBkd6vyZdRb4A713BdLGFZqX o6WUer5LawjGxVPOCPEA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mWKDx-000gV0-2O; Fri, 01 Oct 2021 15:13:21 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mWKDt-000gUE-8E for linux-arm-kernel@lists.infradead.org; Fri, 01 Oct 2021 15:13:19 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1633101194; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LW6fp9rG/lvp5l8prGyc0xe8qMlUDZGZChEO4If5ox0=; b=Al/Db8XhhPODEAiBaLPPbsUI5HxzcbJi6pso+o/R6EujgOzsedjyRGy9dw22ZvwxPKO1L3 UvZfs1FCQpwRSGBklty8T/tQfH2lvPO5xlZB8QAWr22/qKlDl2rXA0wuNDzuYgDoHQCl9v 4wMhEl9c3d8guw4ks95ssDLaY41eops= Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-202-bD-I8sRNOpaoSNSlon-Lhg-1; Fri, 01 Oct 2021 11:12:50 -0400 X-MC-Unique: bD-I8sRNOpaoSNSlon-Lhg-1 Received: by mail-ed1-f72.google.com with SMTP id x7-20020a50ba87000000b003dabd8354f1so6030161ede.7 for ; Fri, 01 Oct 2021 08:12:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent:subject :content-language:to:cc:references:from:in-reply-to :content-transfer-encoding; bh=LW6fp9rG/lvp5l8prGyc0xe8qMlUDZGZChEO4If5ox0=; b=gcFGcrnARSOBSr+UMBv5384gc6iWBIe779ofb9u6znMZGQg/fTSdYsok/jddjV+m13 KRRTJ9EnRWTVmCVb2h2LO1uUwy0CEw0Zazo6MMagHqc2gpIK5mF9wswpyMz9snomxz0w Mfk1ouv0JPafy3xx0iD6bOV8VprzY2DCB7aB5NiPtd8uwbBJqJJWsDTXjXUfbH36Vrap KQO3yXxc4I1VXBcb7/UnfROtzpFmRcqXqxlRozf2U3GJ9xSEmgieI2U0CzZ5sVYpNDNy OYb2tu3pF9tvv3a8XYDHSv6boLFqqkzraxBagparOGjK5co1Th7WYL4otmzyQjeNCe2S gFeA== X-Gm-Message-State: AOAM533tHm5pkXx/LPlZ45zOBtoM8tZXmt6DIktEi7KhHUIY3g9iVEc0 h/ALyGoRa1lUkPP31BEL1OPRk1lY4xiAsLhdDio6IN3C2z3FJgxRKoLnr4rGZ893kdRd9h/Va0y 0szCOn/4Tgr8ejwB7wp5PTIXUigb1lXXq8mk= X-Received: by 2002:a17:907:3e03:: with SMTP id hp3mr6732922ejc.183.1633101142626; Fri, 01 Oct 2021 08:12:22 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxSleQpwCGZXW71w2qzpx6TzvbZz4Gx6iqhZH1Md89m/5OKOBFh+S0Y4ZfLrRfA1AXKxqbKQQ== X-Received: by 2002:a17:907:3e03:: with SMTP id hp3mr6732898ejc.183.1633101142335; Fri, 01 Oct 2021 08:12:22 -0700 (PDT) Received: from [192.168.10.118] ([93.56.162.200]) by smtp.gmail.com with ESMTPSA id ee13sm1227751edb.14.2021.10.01.08.12.20 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 01 Oct 2021 08:12:21 -0700 (PDT) Message-ID: <7901cb84-052d-92b6-1e6a-028396c2c691@redhat.com> Date: Fri, 1 Oct 2021 17:12:20 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.1.0 Subject: Re: [PATCH v8 7/7] KVM: x86: Expose TSC offset controls to userspace To: Marcelo Tosatti Cc: Oliver Upton , kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, Sean Christopherson , Marc Zyngier , Peter Shier , Jim Mattson , David Matlack , Ricardo Koller , Jing Zhang , Raghavendra Rao Anata , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Andrew Jones , Will Deacon , Catalin Marinas References: <20210916181538.968978-1-oupton@google.com> <20210916181538.968978-8-oupton@google.com> <20210930191416.GA19068@fuller.cnet> <48151d08-ee29-2b98-b6e1-f3c8a1ff26bc@redhat.com> <20211001103200.GA39746@fuller.cnet> From: Paolo Bonzini In-Reply-To: <20211001103200.GA39746@fuller.cnet> Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=pbonzini@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211001_081317_911207_AAB89CD0 X-CRM114-Status: GOOD ( 20.40 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 01/10/21 12:32, Marcelo Tosatti wrote: >> +1. Invoke the KVM_GET_CLOCK ioctl to record the host TSC (t_0), + >> kvmclock nanoseconds (k_0), and realtime nanoseconds (r_0). + [...] >> +4. Invoke the KVM_SET_CLOCK ioctl, providing the kvmclock >> nanoseconds + (k_0) and realtime nanoseconds (r_0) in their >> respective fields. + Ensure that the KVM_CLOCK_REALTIME flag is >> set in the provided + structure. KVM will advance the VM's >> kvmclock to account for elapsed + time since recording the clock >> values. > > You can't advance both kvmclock (kvmclock_offset variable) and the > TSCs, which would be double counting. > > So you have to either add the elapsed realtime (1) between > KVM_GET_CLOCK to kvmclock (which this patch is doing), or to the > TSCs. If you do both, there is double counting. Am i missing > something? Probably one of these two (but it's worth pointing out both of them): 1) the attribute that's introduced here *replaces* KVM_SET_MSR(MSR_IA32_TSC), so the TSC is not added. 2) the adjustment formula later in the algorithm does not care about how much time passed between step 1 and step 4. It just takes two well known (TSC, kvmclock) pairs, and uses them to ensure the guest TSC is the same on the destination as if the guest was still running on the source. It is irrelevant that one of them is before migration and one is after, all it matters is that one is on the source and one is on the destination. Perhaps we can add to step 6 something like: > +6. Adjust the guest TSC offsets for every vCPU to account for (1) > time + elapsed since recording state and (2) difference in TSCs > between the + source and destination machine: + + new_off_n = t_0 > + off_n + (k_1 - k_0) * freq - t_1 + "off + t - k * freq" is the guest TSC value corresponding to a time of 0 in kvmclock. The above formula ensures that it is the same on the destination as it was on the source. Also, the names are a bit hard to follow. Perhaps t_0 tsc_src t_1 tsc_dest k_0 guest_src k_1 guest_dest r_0 host_src off_n ofs_src[i] new_off_n ofs_dest[i] Paolo _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel