From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 185CDC4338F for ; Tue, 10 Aug 2021 00:04:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E41BD60FE6 for ; Tue, 10 Aug 2021 00:04:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233281AbhHJAFO (ORCPT ); Mon, 9 Aug 2021 20:05:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35602 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230317AbhHJAFN (ORCPT ); Mon, 9 Aug 2021 20:05:13 -0400 Received: from mail-lf1-x129.google.com (mail-lf1-x129.google.com [IPv6:2a00:1450:4864:20::129]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 165D7C0613D3 for ; Mon, 9 Aug 2021 17:04:52 -0700 (PDT) Received: by mail-lf1-x129.google.com with SMTP id b6so21281603lff.10 for ; Mon, 09 Aug 2021 17:04:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=tfgnvia9bhtFcWQg+6QIhSvfTwR0SecrlC0psAaPUJM=; b=YcoZnYHnJR0j6OfE2Q111pTfgMM2XYw5tZ1HXZlY9djjH7FBTqXN0tWuHbPIhhiaD3 ylfUTDihwoeTQNTMsJDgQoq5TyD03kKOGO6bpgAFJWIzx3oz0Lp9C/WT7aW+m9lGISJh FdR7GEIAEoAvyNl1eUhRv1Vf4efcHMB9PIG2AOr4fZoAMgnF/5nGnzrGyPbMAqZuTGa5 jFqZOAX9MzpNLR/SHw/pefvTaDhtxdvbcqRRdy4JsCGjTqoqMWt8Zm6lgxyrsaLZGFp5 D9iPwPvsnEv8JtJDFFDe/vkPXGWWVyqSF06C7RVHdaUqQu5PRNkRb29ms5ECPa8E4V6f 0EYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=tfgnvia9bhtFcWQg+6QIhSvfTwR0SecrlC0psAaPUJM=; b=FEEx8Ddpl6gj95GClnz4mOljQGpALDlCLqbSGRcEUXNzlVe042EZvf1u9b9bzTYtHH n6VbQ9Qn96IXawVSP0JCxAgt74cI/vb9LxAoSaf72/9DqQeMqJTqFbwRxgnrB6fDNT+y j0Jx4SdgTSKaaOUqNvCYAldTAwbOLMDulQE+XW4y0377ol/hk2Erzxp2sas2FtBiKPsH rDF2EFr7o/nrIQTYM6y3dukYvVInYfE19Et8YodRFm+19L7RaL6jLsjIUJZoTfJHSLCM y3JtbCwhjloD6r0iSCZkriiNekb1AfDmJQUsE+feH97fG6XjOIh/iFv4lyDgC0OhDd/Z fkxA== X-Gm-Message-State: AOAM5315yL7w5VJ0/Nu8PbSNBr2I1p5FKk5OvvngkvYCRpTNwKTEq0VC UfJQrYJ7zqJ8NOz0M5FGlzwxMp+c0e29pw/DcSvkH/Ss7rw= X-Google-Smtp-Source: ABdhPJxsIzr/4rGMJmdeT8OQd21bYuQu+LPC3VNK01JmxBPugVic9SDMpmYmwTnX1Zf01tRLm21WNr2mxSwUfxcra3g= X-Received: by 2002:ac2:429a:: with SMTP id m26mr19513903lfh.80.1628553889715; Mon, 09 Aug 2021 17:04:49 -0700 (PDT) MIME-Version: 1.0 References: <20210804085819.846610-1-oupton@google.com> In-Reply-To: From: Oliver Upton Date: Mon, 9 Aug 2021 17:04:38 -0700 Message-ID: Subject: Re: [PATCH v6 00/21] KVM: Add idempotent controls for migrating system counter state To: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu Cc: Paolo Bonzini , Sean Christopherson , Marc Zyngier , Peter Shier , Jim Mattson , David Matlack , Ricardo Koller , Jing Zhang , Raghavendra Rao Anata , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Andrew Jones , Will Deacon , Catalin Marinas Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org On Wed, Aug 4, 2021 at 3:03 PM Oliver Upton wrote: > > On Wed, Aug 4, 2021 at 4:05 AM Oliver Upton wrote: > > > > On Wed, Aug 4, 2021 at 1:58 AM Oliver Upton wrote: > > > > > > KVM's current means of saving/restoring system counters is plagued with > > > temporal issues. At least on ARM64 and x86, we migrate the guest's > > > system counter by-value through the respective guest system register > > > values (cntvct_el0, ia32_tsc). Restoring system counters by-value is > > > brittle as the state is not idempotent: the host system counter is still > > > oscillating between the attempted save and restore. Furthermore, VMMs > > > may wish to transparently live migrate guest VMs, meaning that they > > > include the elapsed time due to live migration blackout in the guest > > > system counter view. The VMM thread could be preempted for any number of > > > reasons (scheduler, L0 hypervisor under nested) between the time that > > > it calculates the desired guest counter value and when KVM actually sets > > > this counter state. > > > > > > Despite the value-based interface that we present to userspace, KVM > > > actually has idempotent guest controls by way of system counter offsets. > > > We can avoid all of the issues associated with a value-based interface > > > by abstracting these offset controls in new ioctls. This series > > > introduces new vCPU device attributes to provide userspace access to the > > > vCPU's system counter offset. > > > > > > Patch 1 addresses a possible race in KVM_GET_CLOCK where > > > use_master_clock is read outside of the pvclock_gtod_sync_lock. > > > > > > Patch 2 adopts Paolo's suggestion, augmenting the KVM_{GET,SET}_CLOCK > > > ioctls to provide userspace with a (host_tsc, realtime) instant. This is > > > essential for a VMM to perform precise migration of the guest's system > > > counters. > > > > > > Patches 3-4 are some preparatory changes for exposing the TSC offset to > > > userspace. Patch 5 provides a vCPU attribute to provide userspace access > > > to the TSC offset. > > > > > > Patches 6-7 implement a test for the new additions to > > > KVM_{GET,SET}_CLOCK. > > > > > > Patch 8 fixes some assertions in the kvm device attribute helpers. > > > > > > Patches 9-10 implement at test for the tsc offset attribute introduced in > > > patch 5. Paolo, Is there anything else you're waiting to see for the x86 portion of this series after addressing Sean's comments? There's some work remaining on the arm64 side, though I believe the two architectures are now disjoint for this series. -- Thanks, Oliver > > > Patches 11-12 lay the groundwork for patch 13, which exposes CNTVOFF_EL2 > > > through the ONE_REG interface. > > > > > > Patches 14-15 add test cases for userspace manipulation of the virtual > > > counter-timer. > > > > > > Patches 16-17 add a vCPU attribute to adjust the host-guest offset of an > > > ARM vCPU, but only implements support for ECV hosts. Patches 18-19 add > > > support for non-ECV hosts by emulating physical counter offsetting. > > > > > > Patch 20 adds test cases for adjusting the host-guest offset, and > > > finally patch 21 adds a test to measure the emulation overhead of > > > CNTPCT_EL2. > > > > > > This series was tested on both an Ampere Mt. Jade and Haswell systems. > > > Unfortunately, the ECV portions of this series are untested, as there is > > > no ECV-capable hardware and the ARM fast models only partially implement > > > ECV. > > > > Small correction: I was only using the foundation model. Apparently > > the AEM FVP provides full ECV support. > > Ok. I've now tested this series on the FVP Base RevC fast model@v8.6 + > ECV=2. Passes on VHE, fails on nVHE. > > I'll respin this series with the fix for nVHE+ECV soon. > > -- > Thanks, > Oliver > > > > > > > > > Physical counter benchmark > > > -------------------------- > > > > > > The following data was collected by running 10000 iterations of the > > > benchmark test from Patch 21 on an Ampere Mt. Jade reference server, A 2S > > > machine with 2 80-core Ampere Altra SoCs. Measurements were collected > > > for both VHE and nVHE operation using the `kvm-arm.mode=` command-line > > > parameter. > > > > > > nVHE > > > ---- > > > > > > +--------------------+--------+---------+ > > > | Metric | Native | Trapped | > > > +--------------------+--------+---------+ > > > | Average | 54ns | 148ns | > > > | Standard Deviation | 124ns | 122ns | > > > | 95th Percentile | 258ns | 348ns | > > > +--------------------+--------+---------+ > > > > > > VHE > > > --- > > > > > > +--------------------+--------+---------+ > > > | Metric | Native | Trapped | > > > +--------------------+--------+---------+ > > > | Average | 53ns | 152ns | > > > | Standard Deviation | 92ns | 94ns | > > > | 95th Percentile | 204ns | 307ns | > > > +--------------------+--------+---------+ > > > > > > This series applies cleanly to kvm/queue at the following commit: > > > > > > 6cd974485e25 ("KVM: selftests: Add a test of an unbacked nested PI descriptor") > > > > > > v1 -> v2: > > > - Reimplemented as vCPU device attributes instead of a distinct ioctl. > > > - Added the (realtime, host_tsc) instant support to KVM_{GET,SET}_CLOCK > > > - Changed the arm64 implementation to broadcast counter > > > offset values to all vCPUs in a guest. This upholds the > > > architectural expectations of a consistent counter-timer across CPUs. > > > - Fixed a bug with traps in VHE mode. We now configure traps on every > > > transition into a guest to handle differing VMs (trapped, emulated). > > > > > > v2 -> v3: > > > - Added documentation for additions to KVM_{GET,SET}_CLOCK > > > - Added documentation for all new vCPU attributes > > > - Added documentation for suggested algorithm to migrate a guest's > > > TSC(s) > > > - Bug fixes throughout series > > > - Rename KVM_CLOCK_REAL_TIME -> KVM_CLOCK_REALTIME > > > > > > v3 -> v4: > > > - Added patch to address incorrect device helper assertions (Drew) > > > - Carried Drew's r-b tags where appropriate > > > - x86 selftest cleanup > > > - Removed stale kvm_timer_init_vhe() function > > > - Removed unnecessary GUEST_DONE() from selftests > > > > > > v4 -> v5: > > > - Fix typo in TSC migration algorithm > > > - Carry more of Drew's r-b tags > > > - clean up run loop logic in counter emulation benchmark (missed from > > > Drew's comments on v3) > > > > > > v5 -> v6: > > > - Add fix for race in KVM_GET_CLOCK (Sean) > > > - Fix 32-bit build issues in series + use of uninitialized host tsc > > > value (Sean) > > > - General style cleanups > > > - Rework ARM virtual counter offsetting to match guest behavior. Use > > > the ONE_REG interface instead of a VM attribute (Marc) > > > - Maintain a single host-guest counter offset, which applies to both > > > physical and virtual counters > > > - Dropped some of Drew's r-b tags due to nontrivial patch changes > > > (sorry for the churn!) > > > > > > v1: https://lore.kernel.org/kvm/20210608214742.1897483-1-oupton@google.com/ > > > v2: https://lore.kernel.org/r/20210716212629.2232756-1-oupton@google.com > > > v3: https://lore.kernel.org/r/20210719184949.1385910-1-oupton@google.com > > > v4: https://lore.kernel.org/r/20210729001012.70394-1-oupton@google.com > > > v5: https://lore.kernel.org/r/20210729173300.181775-1-oupton@google.com > > > > > > Oliver Upton (21): > > > KVM: x86: Fix potential race in KVM_GET_CLOCK > > > KVM: x86: Report host tsc and realtime values in KVM_GET_CLOCK > > > KVM: x86: Take the pvclock sync lock behind the tsc_write_lock > > > KVM: x86: Refactor tsc synchronization code > > > KVM: x86: Expose TSC offset controls to userspace > > > tools: arch: x86: pull in pvclock headers > > > selftests: KVM: Add test for KVM_{GET,SET}_CLOCK > > > selftests: KVM: Fix kvm device helper ioctl assertions > > > selftests: KVM: Add helpers for vCPU device attributes > > > selftests: KVM: Introduce system counter offset test > > > KVM: arm64: Refactor update_vtimer_cntvoff() > > > KVM: arm64: Separate guest/host counter offset values > > > KVM: arm64: Allow userspace to configure a vCPU's virtual offset > > > selftests: KVM: Add helper to check for register presence > > > selftests: KVM: Add support for aarch64 to system_counter_offset_test > > > arm64: cpufeature: Enumerate support for Enhanced Counter > > > Virtualization > > > KVM: arm64: Allow userspace to configure a guest's counter-timer > > > offset > > > KVM: arm64: Configure timer traps in vcpu_load() for VHE > > > KVM: arm64: Emulate physical counter offsetting on non-ECV systems > > > selftests: KVM: Test physical counter offsetting > > > selftests: KVM: Add counter emulation benchmark > > > > > > Documentation/virt/kvm/api.rst | 52 ++- > > > Documentation/virt/kvm/devices/vcpu.rst | 85 ++++ > > > Documentation/virt/kvm/locking.rst | 11 + > > > arch/arm64/include/asm/kvm_asm.h | 2 + > > > arch/arm64/include/asm/sysreg.h | 5 + > > > arch/arm64/include/uapi/asm/kvm.h | 2 + > > > arch/arm64/kernel/cpufeature.c | 10 + > > > arch/arm64/kvm/arch_timer.c | 224 ++++++++++- > > > arch/arm64/kvm/arm.c | 4 +- > > > arch/arm64/kvm/guest.c | 6 +- > > > arch/arm64/kvm/hyp/include/hyp/switch.h | 29 ++ > > > arch/arm64/kvm/hyp/nvhe/hyp-main.c | 6 + > > > arch/arm64/kvm/hyp/nvhe/timer-sr.c | 16 +- > > > arch/arm64/kvm/hyp/vhe/timer-sr.c | 5 + > > > arch/arm64/tools/cpucaps | 1 + > > > arch/x86/include/asm/kvm_host.h | 4 + > > > arch/x86/include/uapi/asm/kvm.h | 4 + > > > arch/x86/kvm/x86.c | 364 +++++++++++++----- > > > include/clocksource/arm_arch_timer.h | 1 + > > > include/kvm/arm_arch_timer.h | 6 +- > > > include/uapi/linux/kvm.h | 7 +- > > > tools/arch/x86/include/asm/pvclock-abi.h | 48 +++ > > > tools/arch/x86/include/asm/pvclock.h | 103 +++++ > > > tools/testing/selftests/kvm/.gitignore | 3 + > > > tools/testing/selftests/kvm/Makefile | 4 + > > > .../kvm/aarch64/counter_emulation_benchmark.c | 207 ++++++++++ > > > .../selftests/kvm/include/aarch64/processor.h | 24 ++ > > > .../testing/selftests/kvm/include/kvm_util.h | 13 + > > > tools/testing/selftests/kvm/lib/kvm_util.c | 63 ++- > > > .../kvm/system_counter_offset_test.c | 211 ++++++++++ > > > .../selftests/kvm/x86_64/kvm_clock_test.c | 204 ++++++++++ > > > 31 files changed, 1581 insertions(+), 143 deletions(-) > > > create mode 100644 tools/arch/x86/include/asm/pvclock-abi.h > > > create mode 100644 tools/arch/x86/include/asm/pvclock.h > > > create mode 100644 tools/testing/selftests/kvm/aarch64/counter_emulation_benchmark.c > > > create mode 100644 tools/testing/selftests/kvm/system_counter_offset_test.c > > > create mode 100644 tools/testing/selftests/kvm/x86_64/kvm_clock_test.c > > > > > > -- > > > 2.32.0.605.g8dce9f2422-goog > > > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.6 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41D11C4338F for ; Tue, 10 Aug 2021 00:05:02 +0000 (UTC) Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by mail.kernel.org (Postfix) with ESMTP id 80F9960FE6 for ; Tue, 10 Aug 2021 00:05:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 80F9960FE6 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.cs.columbia.edu Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id EB48E404F8; Mon, 9 Aug 2021 20:05:00 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@google.com Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VGL64jX5H-Zq; Mon, 9 Aug 2021 20:04:57 -0400 (EDT) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 038EE4B0D2; Mon, 9 Aug 2021 20:04:57 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id DD50E4B0C7 for ; Mon, 9 Aug 2021 20:04:55 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kA564ZfMlhfg for ; Mon, 9 Aug 2021 20:04:51 -0400 (EDT) Received: from mail-lf1-f51.google.com (mail-lf1-f51.google.com [209.85.167.51]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id 6D354404F8 for ; Mon, 9 Aug 2021 20:04:51 -0400 (EDT) Received: by mail-lf1-f51.google.com with SMTP id z20so476539lfd.2 for ; Mon, 09 Aug 2021 17:04:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=tfgnvia9bhtFcWQg+6QIhSvfTwR0SecrlC0psAaPUJM=; b=YcoZnYHnJR0j6OfE2Q111pTfgMM2XYw5tZ1HXZlY9djjH7FBTqXN0tWuHbPIhhiaD3 ylfUTDihwoeTQNTMsJDgQoq5TyD03kKOGO6bpgAFJWIzx3oz0Lp9C/WT7aW+m9lGISJh FdR7GEIAEoAvyNl1eUhRv1Vf4efcHMB9PIG2AOr4fZoAMgnF/5nGnzrGyPbMAqZuTGa5 jFqZOAX9MzpNLR/SHw/pefvTaDhtxdvbcqRRdy4JsCGjTqoqMWt8Zm6lgxyrsaLZGFp5 D9iPwPvsnEv8JtJDFFDe/vkPXGWWVyqSF06C7RVHdaUqQu5PRNkRb29ms5ECPa8E4V6f 0EYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=tfgnvia9bhtFcWQg+6QIhSvfTwR0SecrlC0psAaPUJM=; b=UhGu65tqYonxnLkj4yItDnWDCT1sePNgeBY9iPMm+E3rAScTud8ioL3XcsjkeTEFRl yTp4InNWWZ3RDQBsTcjlCAKu0m27MahoK6HPyOpzimqhUEr21UojZAHMuN5Hz87d4yg0 0pmI42fEzU7yrdoOJJZCa1VKWR9WWYDDxxK3pUOn9nOuOWe7T6m1bCIUbrqG9TCSiZz9 60GBiLXigUtJ0re4R8pGFr93n/XJq7yR2NvTzHUKqacx1lSEY4MHEiFVVdOQh7C9SkQv 4O+KdUEK+RPC7A5ZBd+oKf89fprqjNWjq71RZpIFSyg89peHJY/4MAj7s1vmvvY1ugIt R1iA== X-Gm-Message-State: AOAM533wdR4WcgtH3habwGASN0DLk2xciuHFsnnP8A5P5p+/LB0gwtWF 3+HgnAxyvyiq7z+wR+0ByXvlTI4ng8WWfkOMQ3hn4A== X-Google-Smtp-Source: ABdhPJxsIzr/4rGMJmdeT8OQd21bYuQu+LPC3VNK01JmxBPugVic9SDMpmYmwTnX1Zf01tRLm21WNr2mxSwUfxcra3g= X-Received: by 2002:ac2:429a:: with SMTP id m26mr19513903lfh.80.1628553889715; Mon, 09 Aug 2021 17:04:49 -0700 (PDT) MIME-Version: 1.0 References: <20210804085819.846610-1-oupton@google.com> In-Reply-To: From: Oliver Upton Date: Mon, 9 Aug 2021 17:04:38 -0700 Message-ID: Subject: Re: [PATCH v6 00/21] KVM: Add idempotent controls for migrating system counter state To: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu Cc: Catalin Marinas , Will Deacon , Marc Zyngier , Raghavendra Rao Anata , Peter Shier , Sean Christopherson , David Matlack , Paolo Bonzini , linux-arm-kernel@lists.infradead.org, Jim Mattson X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu On Wed, Aug 4, 2021 at 3:03 PM Oliver Upton wrote: > > On Wed, Aug 4, 2021 at 4:05 AM Oliver Upton wrote: > > > > On Wed, Aug 4, 2021 at 1:58 AM Oliver Upton wrote: > > > > > > KVM's current means of saving/restoring system counters is plagued with > > > temporal issues. At least on ARM64 and x86, we migrate the guest's > > > system counter by-value through the respective guest system register > > > values (cntvct_el0, ia32_tsc). Restoring system counters by-value is > > > brittle as the state is not idempotent: the host system counter is still > > > oscillating between the attempted save and restore. Furthermore, VMMs > > > may wish to transparently live migrate guest VMs, meaning that they > > > include the elapsed time due to live migration blackout in the guest > > > system counter view. The VMM thread could be preempted for any number of > > > reasons (scheduler, L0 hypervisor under nested) between the time that > > > it calculates the desired guest counter value and when KVM actually sets > > > this counter state. > > > > > > Despite the value-based interface that we present to userspace, KVM > > > actually has idempotent guest controls by way of system counter offsets. > > > We can avoid all of the issues associated with a value-based interface > > > by abstracting these offset controls in new ioctls. This series > > > introduces new vCPU device attributes to provide userspace access to the > > > vCPU's system counter offset. > > > > > > Patch 1 addresses a possible race in KVM_GET_CLOCK where > > > use_master_clock is read outside of the pvclock_gtod_sync_lock. > > > > > > Patch 2 adopts Paolo's suggestion, augmenting the KVM_{GET,SET}_CLOCK > > > ioctls to provide userspace with a (host_tsc, realtime) instant. This is > > > essential for a VMM to perform precise migration of the guest's system > > > counters. > > > > > > Patches 3-4 are some preparatory changes for exposing the TSC offset to > > > userspace. Patch 5 provides a vCPU attribute to provide userspace access > > > to the TSC offset. > > > > > > Patches 6-7 implement a test for the new additions to > > > KVM_{GET,SET}_CLOCK. > > > > > > Patch 8 fixes some assertions in the kvm device attribute helpers. > > > > > > Patches 9-10 implement at test for the tsc offset attribute introduced in > > > patch 5. Paolo, Is there anything else you're waiting to see for the x86 portion of this series after addressing Sean's comments? There's some work remaining on the arm64 side, though I believe the two architectures are now disjoint for this series. -- Thanks, Oliver > > > Patches 11-12 lay the groundwork for patch 13, which exposes CNTVOFF_EL2 > > > through the ONE_REG interface. > > > > > > Patches 14-15 add test cases for userspace manipulation of the virtual > > > counter-timer. > > > > > > Patches 16-17 add a vCPU attribute to adjust the host-guest offset of an > > > ARM vCPU, but only implements support for ECV hosts. Patches 18-19 add > > > support for non-ECV hosts by emulating physical counter offsetting. > > > > > > Patch 20 adds test cases for adjusting the host-guest offset, and > > > finally patch 21 adds a test to measure the emulation overhead of > > > CNTPCT_EL2. > > > > > > This series was tested on both an Ampere Mt. Jade and Haswell systems. > > > Unfortunately, the ECV portions of this series are untested, as there is > > > no ECV-capable hardware and the ARM fast models only partially implement > > > ECV. > > > > Small correction: I was only using the foundation model. Apparently > > the AEM FVP provides full ECV support. > > Ok. I've now tested this series on the FVP Base RevC fast model@v8.6 + > ECV=2. Passes on VHE, fails on nVHE. > > I'll respin this series with the fix for nVHE+ECV soon. > > -- > Thanks, > Oliver > > > > > > > > > Physical counter benchmark > > > -------------------------- > > > > > > The following data was collected by running 10000 iterations of the > > > benchmark test from Patch 21 on an Ampere Mt. Jade reference server, A 2S > > > machine with 2 80-core Ampere Altra SoCs. Measurements were collected > > > for both VHE and nVHE operation using the `kvm-arm.mode=` command-line > > > parameter. > > > > > > nVHE > > > ---- > > > > > > +--------------------+--------+---------+ > > > | Metric | Native | Trapped | > > > +--------------------+--------+---------+ > > > | Average | 54ns | 148ns | > > > | Standard Deviation | 124ns | 122ns | > > > | 95th Percentile | 258ns | 348ns | > > > +--------------------+--------+---------+ > > > > > > VHE > > > --- > > > > > > +--------------------+--------+---------+ > > > | Metric | Native | Trapped | > > > +--------------------+--------+---------+ > > > | Average | 53ns | 152ns | > > > | Standard Deviation | 92ns | 94ns | > > > | 95th Percentile | 204ns | 307ns | > > > +--------------------+--------+---------+ > > > > > > This series applies cleanly to kvm/queue at the following commit: > > > > > > 6cd974485e25 ("KVM: selftests: Add a test of an unbacked nested PI descriptor") > > > > > > v1 -> v2: > > > - Reimplemented as vCPU device attributes instead of a distinct ioctl. > > > - Added the (realtime, host_tsc) instant support to KVM_{GET,SET}_CLOCK > > > - Changed the arm64 implementation to broadcast counter > > > offset values to all vCPUs in a guest. This upholds the > > > architectural expectations of a consistent counter-timer across CPUs. > > > - Fixed a bug with traps in VHE mode. We now configure traps on every > > > transition into a guest to handle differing VMs (trapped, emulated). > > > > > > v2 -> v3: > > > - Added documentation for additions to KVM_{GET,SET}_CLOCK > > > - Added documentation for all new vCPU attributes > > > - Added documentation for suggested algorithm to migrate a guest's > > > TSC(s) > > > - Bug fixes throughout series > > > - Rename KVM_CLOCK_REAL_TIME -> KVM_CLOCK_REALTIME > > > > > > v3 -> v4: > > > - Added patch to address incorrect device helper assertions (Drew) > > > - Carried Drew's r-b tags where appropriate > > > - x86 selftest cleanup > > > - Removed stale kvm_timer_init_vhe() function > > > - Removed unnecessary GUEST_DONE() from selftests > > > > > > v4 -> v5: > > > - Fix typo in TSC migration algorithm > > > - Carry more of Drew's r-b tags > > > - clean up run loop logic in counter emulation benchmark (missed from > > > Drew's comments on v3) > > > > > > v5 -> v6: > > > - Add fix for race in KVM_GET_CLOCK (Sean) > > > - Fix 32-bit build issues in series + use of uninitialized host tsc > > > value (Sean) > > > - General style cleanups > > > - Rework ARM virtual counter offsetting to match guest behavior. Use > > > the ONE_REG interface instead of a VM attribute (Marc) > > > - Maintain a single host-guest counter offset, which applies to both > > > physical and virtual counters > > > - Dropped some of Drew's r-b tags due to nontrivial patch changes > > > (sorry for the churn!) > > > > > > v1: https://lore.kernel.org/kvm/20210608214742.1897483-1-oupton@google.com/ > > > v2: https://lore.kernel.org/r/20210716212629.2232756-1-oupton@google.com > > > v3: https://lore.kernel.org/r/20210719184949.1385910-1-oupton@google.com > > > v4: https://lore.kernel.org/r/20210729001012.70394-1-oupton@google.com > > > v5: https://lore.kernel.org/r/20210729173300.181775-1-oupton@google.com > > > > > > Oliver Upton (21): > > > KVM: x86: Fix potential race in KVM_GET_CLOCK > > > KVM: x86: Report host tsc and realtime values in KVM_GET_CLOCK > > > KVM: x86: Take the pvclock sync lock behind the tsc_write_lock > > > KVM: x86: Refactor tsc synchronization code > > > KVM: x86: Expose TSC offset controls to userspace > > > tools: arch: x86: pull in pvclock headers > > > selftests: KVM: Add test for KVM_{GET,SET}_CLOCK > > > selftests: KVM: Fix kvm device helper ioctl assertions > > > selftests: KVM: Add helpers for vCPU device attributes > > > selftests: KVM: Introduce system counter offset test > > > KVM: arm64: Refactor update_vtimer_cntvoff() > > > KVM: arm64: Separate guest/host counter offset values > > > KVM: arm64: Allow userspace to configure a vCPU's virtual offset > > > selftests: KVM: Add helper to check for register presence > > > selftests: KVM: Add support for aarch64 to system_counter_offset_test > > > arm64: cpufeature: Enumerate support for Enhanced Counter > > > Virtualization > > > KVM: arm64: Allow userspace to configure a guest's counter-timer > > > offset > > > KVM: arm64: Configure timer traps in vcpu_load() for VHE > > > KVM: arm64: Emulate physical counter offsetting on non-ECV systems > > > selftests: KVM: Test physical counter offsetting > > > selftests: KVM: Add counter emulation benchmark > > > > > > Documentation/virt/kvm/api.rst | 52 ++- > > > Documentation/virt/kvm/devices/vcpu.rst | 85 ++++ > > > Documentation/virt/kvm/locking.rst | 11 + > > > arch/arm64/include/asm/kvm_asm.h | 2 + > > > arch/arm64/include/asm/sysreg.h | 5 + > > > arch/arm64/include/uapi/asm/kvm.h | 2 + > > > arch/arm64/kernel/cpufeature.c | 10 + > > > arch/arm64/kvm/arch_timer.c | 224 ++++++++++- > > > arch/arm64/kvm/arm.c | 4 +- > > > arch/arm64/kvm/guest.c | 6 +- > > > arch/arm64/kvm/hyp/include/hyp/switch.h | 29 ++ > > > arch/arm64/kvm/hyp/nvhe/hyp-main.c | 6 + > > > arch/arm64/kvm/hyp/nvhe/timer-sr.c | 16 +- > > > arch/arm64/kvm/hyp/vhe/timer-sr.c | 5 + > > > arch/arm64/tools/cpucaps | 1 + > > > arch/x86/include/asm/kvm_host.h | 4 + > > > arch/x86/include/uapi/asm/kvm.h | 4 + > > > arch/x86/kvm/x86.c | 364 +++++++++++++----- > > > include/clocksource/arm_arch_timer.h | 1 + > > > include/kvm/arm_arch_timer.h | 6 +- > > > include/uapi/linux/kvm.h | 7 +- > > > tools/arch/x86/include/asm/pvclock-abi.h | 48 +++ > > > tools/arch/x86/include/asm/pvclock.h | 103 +++++ > > > tools/testing/selftests/kvm/.gitignore | 3 + > > > tools/testing/selftests/kvm/Makefile | 4 + > > > .../kvm/aarch64/counter_emulation_benchmark.c | 207 ++++++++++ > > > .../selftests/kvm/include/aarch64/processor.h | 24 ++ > > > .../testing/selftests/kvm/include/kvm_util.h | 13 + > > > tools/testing/selftests/kvm/lib/kvm_util.c | 63 ++- > > > .../kvm/system_counter_offset_test.c | 211 ++++++++++ > > > .../selftests/kvm/x86_64/kvm_clock_test.c | 204 ++++++++++ > > > 31 files changed, 1581 insertions(+), 143 deletions(-) > > > create mode 100644 tools/arch/x86/include/asm/pvclock-abi.h > > > create mode 100644 tools/arch/x86/include/asm/pvclock.h > > > create mode 100644 tools/testing/selftests/kvm/aarch64/counter_emulation_benchmark.c > > > create mode 100644 tools/testing/selftests/kvm/system_counter_offset_test.c > > > create mode 100644 tools/testing/selftests/kvm/x86_64/kvm_clock_test.c > > > > > > -- > > > 2.32.0.605.g8dce9f2422-goog > > > _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18461C4338F for ; Tue, 10 Aug 2021 00:08:47 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C3A2B60FE6 for ; Tue, 10 Aug 2021 00:08:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org C3A2B60FE6 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=N8cONwnAD3vjfqthLsGkCC6X15qPkPhTkuKtUwPhDUk=; b=lZqIPQlJeIHfho n2fUuqNFG9gJ32bwNAq2aDv4DLMzckHIdXFOne/iZ6P82v4BRZEM68JpgnwMvURL0gfg1B4y9EViQ h36V/Gk/yBgIAyqwjSXPadWBmp+BV7Dn8ubOi33vTN27Dv2GYRB7GdqnGIfoX7cSDLCREEL6oUKDn zZQ+LnYAUOzx/XEkNUQFCHPVafamIxGBmO1P0e3izpw3AZx9l5UjJcsaYFlcoD7x0K5YsWvK94NXb sY2whHgJQ1cPgsVvZc4L2luSg4191YoUzI8s5RyC+Hkil0MoE/dWrZIBDGRhZDlNgR1UqTbyERTcW 1xI4gaFBrd/F28KSBgZg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mDFGn-0029Q4-HS; Tue, 10 Aug 2021 00:05:25 +0000 Received: from mail-lf1-x12f.google.com ([2a00:1450:4864:20::12f]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mDFGK-0029PG-OX for linux-arm-kernel@lists.infradead.org; Tue, 10 Aug 2021 00:04:59 +0000 Received: by mail-lf1-x12f.google.com with SMTP id b6so21281587lff.10 for ; Mon, 09 Aug 2021 17:04:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=tfgnvia9bhtFcWQg+6QIhSvfTwR0SecrlC0psAaPUJM=; b=YcoZnYHnJR0j6OfE2Q111pTfgMM2XYw5tZ1HXZlY9djjH7FBTqXN0tWuHbPIhhiaD3 ylfUTDihwoeTQNTMsJDgQoq5TyD03kKOGO6bpgAFJWIzx3oz0Lp9C/WT7aW+m9lGISJh FdR7GEIAEoAvyNl1eUhRv1Vf4efcHMB9PIG2AOr4fZoAMgnF/5nGnzrGyPbMAqZuTGa5 jFqZOAX9MzpNLR/SHw/pefvTaDhtxdvbcqRRdy4JsCGjTqoqMWt8Zm6lgxyrsaLZGFp5 D9iPwPvsnEv8JtJDFFDe/vkPXGWWVyqSF06C7RVHdaUqQu5PRNkRb29ms5ECPa8E4V6f 0EYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=tfgnvia9bhtFcWQg+6QIhSvfTwR0SecrlC0psAaPUJM=; b=OO9P7d/nLyK+bxUXZsCQ1JM8CLgJmkCdUe48JuX8eZrJyi/Jdm9DC4hkDe6xjTSsbF eicj1ku+svw0KdWX9MnMpCswPUyv6qKcofXSYV3U0IRNLWn9uIhtLMcw3W6oihLj2Ndz nAQ6jF1w2tyGEIVWCf7GW60OvBK2G6feuKSbPyortPCqlVrtD2YOZERcWUoaHX1z7nqk mFxxVPtTEeS+TFUlwTSuVZw5C66/zxDBTgq7DmJCqob8oi2GsOBuFjQaxcOfRhWe7dL0 KHt24Gw2hGz2i3N1xaYKw+WK3nJ/CAN3K6b+I7PQjZ3hGfKyt3NNLhSIn81SdhJBDEjv nKqQ== X-Gm-Message-State: AOAM532M3t8qsnL67KD/4z0E5OhSNXLNzYp1IVxthB1u+0Eo7LDZdUsk S12sAWn5K2tUumjJzYKsWjUCKaIC+lOiKn8psDFSqg== X-Google-Smtp-Source: ABdhPJxsIzr/4rGMJmdeT8OQd21bYuQu+LPC3VNK01JmxBPugVic9SDMpmYmwTnX1Zf01tRLm21WNr2mxSwUfxcra3g= X-Received: by 2002:ac2:429a:: with SMTP id m26mr19513903lfh.80.1628553889715; Mon, 09 Aug 2021 17:04:49 -0700 (PDT) MIME-Version: 1.0 References: <20210804085819.846610-1-oupton@google.com> In-Reply-To: From: Oliver Upton Date: Mon, 9 Aug 2021 17:04:38 -0700 Message-ID: Subject: Re: [PATCH v6 00/21] KVM: Add idempotent controls for migrating system counter state To: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu Cc: Paolo Bonzini , Sean Christopherson , Marc Zyngier , Peter Shier , Jim Mattson , David Matlack , Ricardo Koller , Jing Zhang , Raghavendra Rao Anata , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Andrew Jones , Will Deacon , Catalin Marinas X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210809_170456_852083_83B40D4A X-CRM114-Status: GOOD ( 50.73 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, Aug 4, 2021 at 3:03 PM Oliver Upton wrote: > > On Wed, Aug 4, 2021 at 4:05 AM Oliver Upton wrote: > > > > On Wed, Aug 4, 2021 at 1:58 AM Oliver Upton wrote: > > > > > > KVM's current means of saving/restoring system counters is plagued with > > > temporal issues. At least on ARM64 and x86, we migrate the guest's > > > system counter by-value through the respective guest system register > > > values (cntvct_el0, ia32_tsc). Restoring system counters by-value is > > > brittle as the state is not idempotent: the host system counter is still > > > oscillating between the attempted save and restore. Furthermore, VMMs > > > may wish to transparently live migrate guest VMs, meaning that they > > > include the elapsed time due to live migration blackout in the guest > > > system counter view. The VMM thread could be preempted for any number of > > > reasons (scheduler, L0 hypervisor under nested) between the time that > > > it calculates the desired guest counter value and when KVM actually sets > > > this counter state. > > > > > > Despite the value-based interface that we present to userspace, KVM > > > actually has idempotent guest controls by way of system counter offsets. > > > We can avoid all of the issues associated with a value-based interface > > > by abstracting these offset controls in new ioctls. This series > > > introduces new vCPU device attributes to provide userspace access to the > > > vCPU's system counter offset. > > > > > > Patch 1 addresses a possible race in KVM_GET_CLOCK where > > > use_master_clock is read outside of the pvclock_gtod_sync_lock. > > > > > > Patch 2 adopts Paolo's suggestion, augmenting the KVM_{GET,SET}_CLOCK > > > ioctls to provide userspace with a (host_tsc, realtime) instant. This is > > > essential for a VMM to perform precise migration of the guest's system > > > counters. > > > > > > Patches 3-4 are some preparatory changes for exposing the TSC offset to > > > userspace. Patch 5 provides a vCPU attribute to provide userspace access > > > to the TSC offset. > > > > > > Patches 6-7 implement a test for the new additions to > > > KVM_{GET,SET}_CLOCK. > > > > > > Patch 8 fixes some assertions in the kvm device attribute helpers. > > > > > > Patches 9-10 implement at test for the tsc offset attribute introduced in > > > patch 5. Paolo, Is there anything else you're waiting to see for the x86 portion of this series after addressing Sean's comments? There's some work remaining on the arm64 side, though I believe the two architectures are now disjoint for this series. -- Thanks, Oliver > > > Patches 11-12 lay the groundwork for patch 13, which exposes CNTVOFF_EL2 > > > through the ONE_REG interface. > > > > > > Patches 14-15 add test cases for userspace manipulation of the virtual > > > counter-timer. > > > > > > Patches 16-17 add a vCPU attribute to adjust the host-guest offset of an > > > ARM vCPU, but only implements support for ECV hosts. Patches 18-19 add > > > support for non-ECV hosts by emulating physical counter offsetting. > > > > > > Patch 20 adds test cases for adjusting the host-guest offset, and > > > finally patch 21 adds a test to measure the emulation overhead of > > > CNTPCT_EL2. > > > > > > This series was tested on both an Ampere Mt. Jade and Haswell systems. > > > Unfortunately, the ECV portions of this series are untested, as there is > > > no ECV-capable hardware and the ARM fast models only partially implement > > > ECV. > > > > Small correction: I was only using the foundation model. Apparently > > the AEM FVP provides full ECV support. > > Ok. I've now tested this series on the FVP Base RevC fast model@v8.6 + > ECV=2. Passes on VHE, fails on nVHE. > > I'll respin this series with the fix for nVHE+ECV soon. > > -- > Thanks, > Oliver > > > > > > > > > Physical counter benchmark > > > -------------------------- > > > > > > The following data was collected by running 10000 iterations of the > > > benchmark test from Patch 21 on an Ampere Mt. Jade reference server, A 2S > > > machine with 2 80-core Ampere Altra SoCs. Measurements were collected > > > for both VHE and nVHE operation using the `kvm-arm.mode=` command-line > > > parameter. > > > > > > nVHE > > > ---- > > > > > > +--------------------+--------+---------+ > > > | Metric | Native | Trapped | > > > +--------------------+--------+---------+ > > > | Average | 54ns | 148ns | > > > | Standard Deviation | 124ns | 122ns | > > > | 95th Percentile | 258ns | 348ns | > > > +--------------------+--------+---------+ > > > > > > VHE > > > --- > > > > > > +--------------------+--------+---------+ > > > | Metric | Native | Trapped | > > > +--------------------+--------+---------+ > > > | Average | 53ns | 152ns | > > > | Standard Deviation | 92ns | 94ns | > > > | 95th Percentile | 204ns | 307ns | > > > +--------------------+--------+---------+ > > > > > > This series applies cleanly to kvm/queue at the following commit: > > > > > > 6cd974485e25 ("KVM: selftests: Add a test of an unbacked nested PI descriptor") > > > > > > v1 -> v2: > > > - Reimplemented as vCPU device attributes instead of a distinct ioctl. > > > - Added the (realtime, host_tsc) instant support to KVM_{GET,SET}_CLOCK > > > - Changed the arm64 implementation to broadcast counter > > > offset values to all vCPUs in a guest. This upholds the > > > architectural expectations of a consistent counter-timer across CPUs. > > > - Fixed a bug with traps in VHE mode. We now configure traps on every > > > transition into a guest to handle differing VMs (trapped, emulated). > > > > > > v2 -> v3: > > > - Added documentation for additions to KVM_{GET,SET}_CLOCK > > > - Added documentation for all new vCPU attributes > > > - Added documentation for suggested algorithm to migrate a guest's > > > TSC(s) > > > - Bug fixes throughout series > > > - Rename KVM_CLOCK_REAL_TIME -> KVM_CLOCK_REALTIME > > > > > > v3 -> v4: > > > - Added patch to address incorrect device helper assertions (Drew) > > > - Carried Drew's r-b tags where appropriate > > > - x86 selftest cleanup > > > - Removed stale kvm_timer_init_vhe() function > > > - Removed unnecessary GUEST_DONE() from selftests > > > > > > v4 -> v5: > > > - Fix typo in TSC migration algorithm > > > - Carry more of Drew's r-b tags > > > - clean up run loop logic in counter emulation benchmark (missed from > > > Drew's comments on v3) > > > > > > v5 -> v6: > > > - Add fix for race in KVM_GET_CLOCK (Sean) > > > - Fix 32-bit build issues in series + use of uninitialized host tsc > > > value (Sean) > > > - General style cleanups > > > - Rework ARM virtual counter offsetting to match guest behavior. Use > > > the ONE_REG interface instead of a VM attribute (Marc) > > > - Maintain a single host-guest counter offset, which applies to both > > > physical and virtual counters > > > - Dropped some of Drew's r-b tags due to nontrivial patch changes > > > (sorry for the churn!) > > > > > > v1: https://lore.kernel.org/kvm/20210608214742.1897483-1-oupton@google.com/ > > > v2: https://lore.kernel.org/r/20210716212629.2232756-1-oupton@google.com > > > v3: https://lore.kernel.org/r/20210719184949.1385910-1-oupton@google.com > > > v4: https://lore.kernel.org/r/20210729001012.70394-1-oupton@google.com > > > v5: https://lore.kernel.org/r/20210729173300.181775-1-oupton@google.com > > > > > > Oliver Upton (21): > > > KVM: x86: Fix potential race in KVM_GET_CLOCK > > > KVM: x86: Report host tsc and realtime values in KVM_GET_CLOCK > > > KVM: x86: Take the pvclock sync lock behind the tsc_write_lock > > > KVM: x86: Refactor tsc synchronization code > > > KVM: x86: Expose TSC offset controls to userspace > > > tools: arch: x86: pull in pvclock headers > > > selftests: KVM: Add test for KVM_{GET,SET}_CLOCK > > > selftests: KVM: Fix kvm device helper ioctl assertions > > > selftests: KVM: Add helpers for vCPU device attributes > > > selftests: KVM: Introduce system counter offset test > > > KVM: arm64: Refactor update_vtimer_cntvoff() > > > KVM: arm64: Separate guest/host counter offset values > > > KVM: arm64: Allow userspace to configure a vCPU's virtual offset > > > selftests: KVM: Add helper to check for register presence > > > selftests: KVM: Add support for aarch64 to system_counter_offset_test > > > arm64: cpufeature: Enumerate support for Enhanced Counter > > > Virtualization > > > KVM: arm64: Allow userspace to configure a guest's counter-timer > > > offset > > > KVM: arm64: Configure timer traps in vcpu_load() for VHE > > > KVM: arm64: Emulate physical counter offsetting on non-ECV systems > > > selftests: KVM: Test physical counter offsetting > > > selftests: KVM: Add counter emulation benchmark > > > > > > Documentation/virt/kvm/api.rst | 52 ++- > > > Documentation/virt/kvm/devices/vcpu.rst | 85 ++++ > > > Documentation/virt/kvm/locking.rst | 11 + > > > arch/arm64/include/asm/kvm_asm.h | 2 + > > > arch/arm64/include/asm/sysreg.h | 5 + > > > arch/arm64/include/uapi/asm/kvm.h | 2 + > > > arch/arm64/kernel/cpufeature.c | 10 + > > > arch/arm64/kvm/arch_timer.c | 224 ++++++++++- > > > arch/arm64/kvm/arm.c | 4 +- > > > arch/arm64/kvm/guest.c | 6 +- > > > arch/arm64/kvm/hyp/include/hyp/switch.h | 29 ++ > > > arch/arm64/kvm/hyp/nvhe/hyp-main.c | 6 + > > > arch/arm64/kvm/hyp/nvhe/timer-sr.c | 16 +- > > > arch/arm64/kvm/hyp/vhe/timer-sr.c | 5 + > > > arch/arm64/tools/cpucaps | 1 + > > > arch/x86/include/asm/kvm_host.h | 4 + > > > arch/x86/include/uapi/asm/kvm.h | 4 + > > > arch/x86/kvm/x86.c | 364 +++++++++++++----- > > > include/clocksource/arm_arch_timer.h | 1 + > > > include/kvm/arm_arch_timer.h | 6 +- > > > include/uapi/linux/kvm.h | 7 +- > > > tools/arch/x86/include/asm/pvclock-abi.h | 48 +++ > > > tools/arch/x86/include/asm/pvclock.h | 103 +++++ > > > tools/testing/selftests/kvm/.gitignore | 3 + > > > tools/testing/selftests/kvm/Makefile | 4 + > > > .../kvm/aarch64/counter_emulation_benchmark.c | 207 ++++++++++ > > > .../selftests/kvm/include/aarch64/processor.h | 24 ++ > > > .../testing/selftests/kvm/include/kvm_util.h | 13 + > > > tools/testing/selftests/kvm/lib/kvm_util.c | 63 ++- > > > .../kvm/system_counter_offset_test.c | 211 ++++++++++ > > > .../selftests/kvm/x86_64/kvm_clock_test.c | 204 ++++++++++ > > > 31 files changed, 1581 insertions(+), 143 deletions(-) > > > create mode 100644 tools/arch/x86/include/asm/pvclock-abi.h > > > create mode 100644 tools/arch/x86/include/asm/pvclock.h > > > create mode 100644 tools/testing/selftests/kvm/aarch64/counter_emulation_benchmark.c > > > create mode 100644 tools/testing/selftests/kvm/system_counter_offset_test.c > > > create mode 100644 tools/testing/selftests/kvm/x86_64/kvm_clock_test.c > > > > > > -- > > > 2.32.0.605.g8dce9f2422-goog > > > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel