From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6A07C4338F for ; Wed, 11 Aug 2021 13:05:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B65ED60FA0 for ; Wed, 11 Aug 2021 13:05:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231175AbhHKNF7 (ORCPT ); Wed, 11 Aug 2021 09:05:59 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:55749 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229766AbhHKNF6 (ORCPT ); Wed, 11 Aug 2021 09:05:58 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1628687134; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ejky1L45fjpIqglpN+AycHkOBEyLVHC4mSrGMJh6Vhk=; b=R6jKYdjEjs3FUXNbrLa7BVwvfC0aKyH5ntWKOYxgjEy2XtHfngeIrGXk00jEWOkQgoA2XM sp53/RbdLPmcEWK2HHejBKDqZrPZGPjPbsxkrUIsCM0HLDZ/Zx1Lt23OUzHyosdO17VnyO y1wGTslXwTcbU77i5oTQDEL26kFRgjY= Received: from mail-ej1-f70.google.com (mail-ej1-f70.google.com [209.85.218.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-251-T3F45O6rPa-GSlGyFV3krA-1; Wed, 11 Aug 2021 09:05:33 -0400 X-MC-Unique: T3F45O6rPa-GSlGyFV3krA-1 Received: by mail-ej1-f70.google.com with SMTP id e1-20020a170906c001b02905b53c2f6542so663256ejz.7 for ; Wed, 11 Aug 2021 06:05:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=ejky1L45fjpIqglpN+AycHkOBEyLVHC4mSrGMJh6Vhk=; b=fcbsc3OP5Q7fN/bnFtFQjTQNBhxwoKMzZbH78ZdJ7BCNJytWzU4z36bhElHN5orcJf A/vcuMEUHIjXl8qp53G9dXyo9nkhPD4ExUxUdUwqhIE7FSamE0875uWHB5rpzDb7e0Us 7jQ2XerUrHCvDl40ECJ+IHI6TX2P7jGib83qmZw6MuTjXNdbjwyt9IRmrPVkP9M+O5kd n1a3zRh62XoTSUBYapCZByUwf2f5HLSdKxho3aOEQPRFAl5hNG+GG2HIwCulkezX6P5W 14BOCVRGUf9qmIpNQrKAhp3bOYOs9+dT4XCxH7h5r+lgEgTXF61Kf4zY3bb60FPBwXJ6 amNA== X-Gm-Message-State: AOAM530L2TAI01jwSPnR2ZGUqkYbkPkAJQKlFGl+WzOAEf8z2mYfJFts lVDvFHEb4zmj8qsPxYL5p87O7HFVxeET4pI76XG6yrI69SKRx1kdzSImvWZsSCZm1GvN7rNN9o6 yyFqmtqJoa+fr X-Received: by 2002:a17:907:3d93:: with SMTP id he19mr3569807ejc.179.1628687132019; Wed, 11 Aug 2021 06:05:32 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwv0o2qcd2qyszkVd6eUDUWXqakuUjyDYC+wCYZeaGx1y3ijN1A/umGBfrGXBRNz7hWK5walg== X-Received: by 2002:a17:907:3d93:: with SMTP id he19mr3569774ejc.179.1628687131737; Wed, 11 Aug 2021 06:05:31 -0700 (PDT) Received: from ?IPv6:2001:b07:6468:f312:63a7:c72e:ea0e:6045? ([2001:b07:6468:f312:63a7:c72e:ea0e:6045]) by smtp.gmail.com with ESMTPSA id y1sm2657015ejf.2.2021.08.11.06.05.30 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 11 Aug 2021 06:05:31 -0700 (PDT) Subject: Re: [PATCH v6 00/21] KVM: Add idempotent controls for migrating system counter state To: Oliver Upton , kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu Cc: Sean Christopherson , Marc Zyngier , Peter Shier , Jim Mattson , David Matlack , Ricardo Koller , Jing Zhang , Raghavendra Rao Anata , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Andrew Jones , Will Deacon , Catalin Marinas References: <20210804085819.846610-1-oupton@google.com> From: Paolo Bonzini Message-ID: <927240ff-a4f4-fcc6-ae1b-92cefeda9e59@redhat.com> Date: Wed, 11 Aug 2021 15:05:29 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <20210804085819.846610-1-oupton@google.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org On 04/08/21 10:57, Oliver Upton wrote: > KVM's current means of saving/restoring system counters is plagued with > temporal issues. At least on ARM64 and x86, we migrate the guest's > system counter by-value through the respective guest system register > values (cntvct_el0, ia32_tsc). Restoring system counters by-value is > brittle as the state is not idempotent: the host system counter is still > oscillating between the attempted save and restore. Furthermore, VMMs > may wish to transparently live migrate guest VMs, meaning that they > include the elapsed time due to live migration blackout in the guest > system counter view. The VMM thread could be preempted for any number of > reasons (scheduler, L0 hypervisor under nested) between the time that > it calculates the desired guest counter value and when KVM actually sets > this counter state. > > Despite the value-based interface that we present to userspace, KVM > actually has idempotent guest controls by way of system counter offsets. > We can avoid all of the issues associated with a value-based interface > by abstracting these offset controls in new ioctls. This series > introduces new vCPU device attributes to provide userspace access to the > vCPU's system counter offset. > > Patch 1 addresses a possible race in KVM_GET_CLOCK where > use_master_clock is read outside of the pvclock_gtod_sync_lock. > > Patch 2 adopts Paolo's suggestion, augmenting the KVM_{GET,SET}_CLOCK > ioctls to provide userspace with a (host_tsc, realtime) instant. This is > essential for a VMM to perform precise migration of the guest's system > counters. > > Patches 3-4 are some preparatory changes for exposing the TSC offset to > userspace. Patch 5 provides a vCPU attribute to provide userspace access > to the TSC offset. > > Patches 6-7 implement a test for the new additions to > KVM_{GET,SET}_CLOCK. > > Patch 8 fixes some assertions in the kvm device attribute helpers. > > Patches 9-10 implement at test for the tsc offset attribute introduced in > patch 5. The x86 parts look good, except that patch 3 is a bit redundant with my idea of altogether getting rid of the pvclock_gtod_sync_lock. That said I agree that patches 1 and 2 (and extracting kvm_vm_ioctl_get_clock and kvm_vm_ioctl_set_clock) should be done before whatever locking changes have to be done. Time is ticking for 5.15 due to my vacation, I'll see if I have some time to look at it further next week. I agree that arm64 can be done separately from x86. Paolo From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.1 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B3BFC4338F for ; Wed, 11 Aug 2021 13:05:43 +0000 (UTC) Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by mail.kernel.org (Postfix) with ESMTP id B5D12600CD for ; Wed, 11 Aug 2021 13:05:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org B5D12600CD Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.cs.columbia.edu Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 26ED240874; Wed, 11 Aug 2021 09:05:42 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@redhat.com Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id B9XwedxU3+SK; Wed, 11 Aug 2021 09:05:38 -0400 (EDT) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 3C46D40878; Wed, 11 Aug 2021 09:05:38 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 73E1540878 for ; Wed, 11 Aug 2021 09:05:36 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zgg-0Co5l841 for ; Wed, 11 Aug 2021 09:05:34 -0400 (EDT) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by mm01.cs.columbia.edu (Postfix) with ESMTP id A363C40874 for ; Wed, 11 Aug 2021 09:05:34 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1628687134; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ejky1L45fjpIqglpN+AycHkOBEyLVHC4mSrGMJh6Vhk=; b=R6jKYdjEjs3FUXNbrLa7BVwvfC0aKyH5ntWKOYxgjEy2XtHfngeIrGXk00jEWOkQgoA2XM sp53/RbdLPmcEWK2HHejBKDqZrPZGPjPbsxkrUIsCM0HLDZ/Zx1Lt23OUzHyosdO17VnyO y1wGTslXwTcbU77i5oTQDEL26kFRgjY= Received: from mail-ej1-f71.google.com (mail-ej1-f71.google.com [209.85.218.71]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-499-cpatxCwZORKyRUjI6rW8_A-1; Wed, 11 Aug 2021 09:05:33 -0400 X-MC-Unique: cpatxCwZORKyRUjI6rW8_A-1 Received: by mail-ej1-f71.google.com with SMTP id h17-20020a1709070b11b02905b5ced62193so664442ejl.1 for ; Wed, 11 Aug 2021 06:05:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=ejky1L45fjpIqglpN+AycHkOBEyLVHC4mSrGMJh6Vhk=; b=FQugXtUXDEjv/aJjN2qAZe/gyX5A7/wfsxHFtEN6LaJ57UiwWGrGWTfzDKAaoWSozC Ode8o6+rGtCKDYdZVU5ljLfP1mMeELmDXUE2kB6EPHQSX+EKRDPvUzm+Y1Z3vWgI0cH/ Iv9bkB70h34ZHN59I9glyWJ+1k14qCB8ekEu6hpc1ca5zKRzndHsIJifumokvN66Ga7D ONQ58eh+TIDpHb9eynoQZtmNfeA03qfvq0yOOMkKTcqgU6ChjhJP/tLEuC5A1BxAC3+o HMdZPViLjC1Vs1p74qse95ddG92kA1olDxVPnBVy0K1zVD3UtK66fL2i+1JrH+IXNLTB Bo2Q== X-Gm-Message-State: AOAM533pHnLpM0gcpTnapXIedZlOH6RX80KWUgRfvoJ1b8GfCCvHnQC/ VJwI9XgoTHgD0TR3RJuXc2MnDtE9bdnD3wQfmgDCEqd8zqL8XZodUcDg1BUheSJ+EyojDFpXMuF o3up7eS9NFX5o55EopPP2vpac X-Received: by 2002:a17:907:3d93:: with SMTP id he19mr3569805ejc.179.1628687132018; Wed, 11 Aug 2021 06:05:32 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwv0o2qcd2qyszkVd6eUDUWXqakuUjyDYC+wCYZeaGx1y3ijN1A/umGBfrGXBRNz7hWK5walg== X-Received: by 2002:a17:907:3d93:: with SMTP id he19mr3569774ejc.179.1628687131737; Wed, 11 Aug 2021 06:05:31 -0700 (PDT) Received: from ?IPv6:2001:b07:6468:f312:63a7:c72e:ea0e:6045? ([2001:b07:6468:f312:63a7:c72e:ea0e:6045]) by smtp.gmail.com with ESMTPSA id y1sm2657015ejf.2.2021.08.11.06.05.30 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 11 Aug 2021 06:05:31 -0700 (PDT) Subject: Re: [PATCH v6 00/21] KVM: Add idempotent controls for migrating system counter state To: Oliver Upton , kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu References: <20210804085819.846610-1-oupton@google.com> From: Paolo Bonzini Message-ID: <927240ff-a4f4-fcc6-ae1b-92cefeda9e59@redhat.com> Date: Wed, 11 Aug 2021 15:05:29 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <20210804085819.846610-1-oupton@google.com> Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=pbonzini@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Cc: Catalin Marinas , Will Deacon , Sean Christopherson , Raghavendra Rao Anata , Peter Shier , Marc Zyngier , David Matlack , linux-arm-kernel@lists.infradead.org, Jim Mattson X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu On 04/08/21 10:57, Oliver Upton wrote: > KVM's current means of saving/restoring system counters is plagued with > temporal issues. At least on ARM64 and x86, we migrate the guest's > system counter by-value through the respective guest system register > values (cntvct_el0, ia32_tsc). Restoring system counters by-value is > brittle as the state is not idempotent: the host system counter is still > oscillating between the attempted save and restore. Furthermore, VMMs > may wish to transparently live migrate guest VMs, meaning that they > include the elapsed time due to live migration blackout in the guest > system counter view. The VMM thread could be preempted for any number of > reasons (scheduler, L0 hypervisor under nested) between the time that > it calculates the desired guest counter value and when KVM actually sets > this counter state. > > Despite the value-based interface that we present to userspace, KVM > actually has idempotent guest controls by way of system counter offsets. > We can avoid all of the issues associated with a value-based interface > by abstracting these offset controls in new ioctls. This series > introduces new vCPU device attributes to provide userspace access to the > vCPU's system counter offset. > > Patch 1 addresses a possible race in KVM_GET_CLOCK where > use_master_clock is read outside of the pvclock_gtod_sync_lock. > > Patch 2 adopts Paolo's suggestion, augmenting the KVM_{GET,SET}_CLOCK > ioctls to provide userspace with a (host_tsc, realtime) instant. This is > essential for a VMM to perform precise migration of the guest's system > counters. > > Patches 3-4 are some preparatory changes for exposing the TSC offset to > userspace. Patch 5 provides a vCPU attribute to provide userspace access > to the TSC offset. > > Patches 6-7 implement a test for the new additions to > KVM_{GET,SET}_CLOCK. > > Patch 8 fixes some assertions in the kvm device attribute helpers. > > Patches 9-10 implement at test for the tsc offset attribute introduced in > patch 5. The x86 parts look good, except that patch 3 is a bit redundant with my idea of altogether getting rid of the pvclock_gtod_sync_lock. That said I agree that patches 1 and 2 (and extracting kvm_vm_ioctl_get_clock and kvm_vm_ioctl_set_clock) should be done before whatever locking changes have to be done. Time is ticking for 5.15 due to my vacation, I'll see if I have some time to look at it further next week. I agree that arm64 can be done separately from x86. Paolo _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07CAEC4338F for ; Wed, 11 Aug 2021 13:07:58 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B9A4860E09 for ; Wed, 11 Aug 2021 13:07:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org B9A4860E09 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Content-Type: Content-Transfer-Encoding:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date:Message-ID:From: References:Cc:To:Subject:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Ah0YfWs72yV4zugK98Fus4bw+glg89gEExTC7rg63Fw=; b=AIqMDGCnxLWR4gh26kD2imaIzZ ME24pgMc5h+R8kJDQCs1x8NRAEwuXcYGQo4onNyjEwvoGIPEd7Ga+8qJOKAX6Sh6mKCNRD24sdwV/ U6hRihAxCG96LlIHAzSZAjfz5mSL+hZO1iWGhmsTacC1gfvJW4el3wAIMcPAAuxPCcZ5jY088lUgq C9wojpPIg7pK7OiCzT5UVR2EpLzhTNi5Jful8RiM2baAS0gPX84nT5tg/Yalu8iSXuuZhzM+jjkrB KLqICUBKfSBNKol6O3iEPugC9IMr2gk4BsX9e4kQBsmwmGXSP+lZfQaAsddAfVZGXyoRRSRxYzE8C q+9at4sg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mDnvQ-007GvZ-LU; Wed, 11 Aug 2021 13:05:40 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mDnvM-007Guf-1L for linux-arm-kernel@lists.infradead.org; Wed, 11 Aug 2021 13:05:37 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1628687134; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ejky1L45fjpIqglpN+AycHkOBEyLVHC4mSrGMJh6Vhk=; b=R6jKYdjEjs3FUXNbrLa7BVwvfC0aKyH5ntWKOYxgjEy2XtHfngeIrGXk00jEWOkQgoA2XM sp53/RbdLPmcEWK2HHejBKDqZrPZGPjPbsxkrUIsCM0HLDZ/Zx1Lt23OUzHyosdO17VnyO y1wGTslXwTcbU77i5oTQDEL26kFRgjY= Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-48-Fv0ZmQCYOWi6RII07l5Pwg-1; Wed, 11 Aug 2021 09:05:33 -0400 X-MC-Unique: Fv0ZmQCYOWi6RII07l5Pwg-1 Received: by mail-ed1-f70.google.com with SMTP id s8-20020a0564025208b02903bd8539e1caso1146083edd.22 for ; Wed, 11 Aug 2021 06:05:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=ejky1L45fjpIqglpN+AycHkOBEyLVHC4mSrGMJh6Vhk=; b=S0U+5V3LJKi0durgtM1fVbwbi7eXDH04HjBhX0NufrpKdP01SAG76Fsa+P98yr7O9c ZxJbStEV5CizmbIRyh18vFzNLS0dFNAeHtQYPlJtUVi7ZuDsCYnh9HIYZiI5C0prRr7H BdVqF2iTRJAspF8ukXH7V26ykH4q+eS6zPC0vidSYx8xu+gWOgmOp6iHMGTnPPyhHNd0 WSqzq6nco86ZAoXxyGOJ6XrJHuePFpA1/X0Wmmd6aGGbb7RWHWqwoajAWtZSd2m+X7D/ BmKcsswJWZwZ4iIMSa/KvLzQF+/B5t/hoZ5U8bG8w7WQoc7q9BtgjH9mBLpfvZ5AarWN CnPA== X-Gm-Message-State: AOAM530z68EURLaHtsHoartrOdzoRR38mSWmwTqnpr6YZEhXo/xHi3YW qi7+J/bFOhMcG5PGSjPMenfZGh59wnO5VucAN2xP2bnmUzx2PokbIop5IfQBnZknQrg2Zlla9Ti iWQWd4mIaXlhjwScJAfRXFFy6hbtc8mnWgQ8= X-Received: by 2002:a17:907:3d93:: with SMTP id he19mr3569815ejc.179.1628687132026; Wed, 11 Aug 2021 06:05:32 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwv0o2qcd2qyszkVd6eUDUWXqakuUjyDYC+wCYZeaGx1y3ijN1A/umGBfrGXBRNz7hWK5walg== X-Received: by 2002:a17:907:3d93:: with SMTP id he19mr3569774ejc.179.1628687131737; Wed, 11 Aug 2021 06:05:31 -0700 (PDT) Received: from ?IPv6:2001:b07:6468:f312:63a7:c72e:ea0e:6045? ([2001:b07:6468:f312:63a7:c72e:ea0e:6045]) by smtp.gmail.com with ESMTPSA id y1sm2657015ejf.2.2021.08.11.06.05.30 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 11 Aug 2021 06:05:31 -0700 (PDT) Subject: Re: [PATCH v6 00/21] KVM: Add idempotent controls for migrating system counter state To: Oliver Upton , kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu Cc: Sean Christopherson , Marc Zyngier , Peter Shier , Jim Mattson , David Matlack , Ricardo Koller , Jing Zhang , Raghavendra Rao Anata , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Andrew Jones , Will Deacon , Catalin Marinas References: <20210804085819.846610-1-oupton@google.com> From: Paolo Bonzini Message-ID: <927240ff-a4f4-fcc6-ae1b-92cefeda9e59@redhat.com> Date: Wed, 11 Aug 2021 15:05:29 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <20210804085819.846610-1-oupton@google.com> Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=pbonzini@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210811_060536_214601_424EAB4B X-CRM114-Status: GOOD ( 23.44 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 04/08/21 10:57, Oliver Upton wrote: > KVM's current means of saving/restoring system counters is plagued with > temporal issues. At least on ARM64 and x86, we migrate the guest's > system counter by-value through the respective guest system register > values (cntvct_el0, ia32_tsc). Restoring system counters by-value is > brittle as the state is not idempotent: the host system counter is still > oscillating between the attempted save and restore. Furthermore, VMMs > may wish to transparently live migrate guest VMs, meaning that they > include the elapsed time due to live migration blackout in the guest > system counter view. The VMM thread could be preempted for any number of > reasons (scheduler, L0 hypervisor under nested) between the time that > it calculates the desired guest counter value and when KVM actually sets > this counter state. > > Despite the value-based interface that we present to userspace, KVM > actually has idempotent guest controls by way of system counter offsets. > We can avoid all of the issues associated with a value-based interface > by abstracting these offset controls in new ioctls. This series > introduces new vCPU device attributes to provide userspace access to the > vCPU's system counter offset. > > Patch 1 addresses a possible race in KVM_GET_CLOCK where > use_master_clock is read outside of the pvclock_gtod_sync_lock. > > Patch 2 adopts Paolo's suggestion, augmenting the KVM_{GET,SET}_CLOCK > ioctls to provide userspace with a (host_tsc, realtime) instant. This is > essential for a VMM to perform precise migration of the guest's system > counters. > > Patches 3-4 are some preparatory changes for exposing the TSC offset to > userspace. Patch 5 provides a vCPU attribute to provide userspace access > to the TSC offset. > > Patches 6-7 implement a test for the new additions to > KVM_{GET,SET}_CLOCK. > > Patch 8 fixes some assertions in the kvm device attribute helpers. > > Patches 9-10 implement at test for the tsc offset attribute introduced in > patch 5. The x86 parts look good, except that patch 3 is a bit redundant with my idea of altogether getting rid of the pvclock_gtod_sync_lock. That said I agree that patches 1 and 2 (and extracting kvm_vm_ioctl_get_clock and kvm_vm_ioctl_set_clock) should be done before whatever locking changes have to be done. Time is ticking for 5.15 due to my vacation, I'll see if I have some time to look at it further next week. I agree that arm64 can be done separately from x86. Paolo _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel