From: Oliver Upton <oupton@google.com>
To: Paolo Bonzini <pbonzini@redhat.com>
Cc: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu,
Sean Christopherson <seanjc@google.com>,
Marc Zyngier <maz@kernel.org>, Peter Shier <pshier@google.com>,
Jim Mattson <jmattson@google.com>,
David Matlack <dmatlack@google.com>,
Ricardo Koller <ricarkol@google.com>,
Jing Zhang <jingzhangos@google.com>,
Raghavendra Rao Anata <rananta@google.com>,
James Morse <james.morse@arm.com>,
Alexandru Elisei <Alexandru.Elisei@arm.com>,
Suzuki K Poulose <suzuki.poulose@arm.com>,
linux-arm-kernel@lists.infradead.org,
Andrew Jones <drjones@redhat.com>, Will Deacon <will@kernel.org>,
Catalin Marinas <catalin.marinas@arm.com>
Subject: Re: [PATCH v6 00/21] KVM: Add idempotent controls for migrating system counter state
Date: Wed, 11 Aug 2021 11:56:22 -0700 [thread overview]
Message-ID: <CAOQ_Qsjby+z_kU49_s0PDKo5c3V-UD+FRg-2PcPycNq-p2gPVg@mail.gmail.com> (raw)
In-Reply-To: <927240ff-a4f4-fcc6-ae1b-92cefeda9e59@redhat.com>
On Wed, Aug 11, 2021 at 6:05 AM Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> On 04/08/21 10:57, Oliver Upton wrote:
> > KVM's current means of saving/restoring system counters is plagued with
> > temporal issues. At least on ARM64 and x86, we migrate the guest's
> > system counter by-value through the respective guest system register
> > values (cntvct_el0, ia32_tsc). Restoring system counters by-value is
> > brittle as the state is not idempotent: the host system counter is still
> > oscillating between the attempted save and restore. Furthermore, VMMs
> > may wish to transparently live migrate guest VMs, meaning that they
> > include the elapsed time due to live migration blackout in the guest
> > system counter view. The VMM thread could be preempted for any number of
> > reasons (scheduler, L0 hypervisor under nested) between the time that
> > it calculates the desired guest counter value and when KVM actually sets
> > this counter state.
> >
> > Despite the value-based interface that we present to userspace, KVM
> > actually has idempotent guest controls by way of system counter offsets.
> > We can avoid all of the issues associated with a value-based interface
> > by abstracting these offset controls in new ioctls. This series
> > introduces new vCPU device attributes to provide userspace access to the
> > vCPU's system counter offset.
> >
> > Patch 1 addresses a possible race in KVM_GET_CLOCK where
> > use_master_clock is read outside of the pvclock_gtod_sync_lock.
> >
> > Patch 2 adopts Paolo's suggestion, augmenting the KVM_{GET,SET}_CLOCK
> > ioctls to provide userspace with a (host_tsc, realtime) instant. This is
> > essential for a VMM to perform precise migration of the guest's system
> > counters.
> >
> > Patches 3-4 are some preparatory changes for exposing the TSC offset to
> > userspace. Patch 5 provides a vCPU attribute to provide userspace access
> > to the TSC offset.
> >
> > Patches 6-7 implement a test for the new additions to
> > KVM_{GET,SET}_CLOCK.
> >
> > Patch 8 fixes some assertions in the kvm device attribute helpers.
> >
> > Patches 9-10 implement at test for the tsc offset attribute introduced in
> > patch 5.
>
> The x86 parts look good, except that patch 3 is a bit redundant with my
> idea of altogether getting rid of the pvclock_gtod_sync_lock. That said
> I agree that patches 1 and 2 (and extracting kvm_vm_ioctl_get_clock and
> kvm_vm_ioctl_set_clock) should be done before whatever locking changes
> have to be done.
Following up on patch 3.
> Time is ticking for 5.15 due to my vacation, I'll see if I have some
> time to look at it further next week.
>
> I agree that arm64 can be done separately from x86.
Marc, just a disclaimer:
I'm going to separate these two series, although there will still
exist dependencies in the selftests changes. Otherwise, kernel changes
are disjoint.
--
Thanks,
Oliver
WARNING: multiple messages have this Message-ID (diff)
From: Oliver Upton <oupton@google.com>
To: Paolo Bonzini <pbonzini@redhat.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>,
kvm@vger.kernel.org, Will Deacon <will@kernel.org>,
Sean Christopherson <seanjc@google.com>,
Raghavendra Rao Anata <rananta@google.com>,
Peter Shier <pshier@google.com>, Marc Zyngier <maz@kernel.org>,
David Matlack <dmatlack@google.com>,
kvmarm@lists.cs.columbia.edu,
linux-arm-kernel@lists.infradead.org,
Jim Mattson <jmattson@google.com>
Subject: Re: [PATCH v6 00/21] KVM: Add idempotent controls for migrating system counter state
Date: Wed, 11 Aug 2021 11:56:22 -0700 [thread overview]
Message-ID: <CAOQ_Qsjby+z_kU49_s0PDKo5c3V-UD+FRg-2PcPycNq-p2gPVg@mail.gmail.com> (raw)
In-Reply-To: <927240ff-a4f4-fcc6-ae1b-92cefeda9e59@redhat.com>
On Wed, Aug 11, 2021 at 6:05 AM Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> On 04/08/21 10:57, Oliver Upton wrote:
> > KVM's current means of saving/restoring system counters is plagued with
> > temporal issues. At least on ARM64 and x86, we migrate the guest's
> > system counter by-value through the respective guest system register
> > values (cntvct_el0, ia32_tsc). Restoring system counters by-value is
> > brittle as the state is not idempotent: the host system counter is still
> > oscillating between the attempted save and restore. Furthermore, VMMs
> > may wish to transparently live migrate guest VMs, meaning that they
> > include the elapsed time due to live migration blackout in the guest
> > system counter view. The VMM thread could be preempted for any number of
> > reasons (scheduler, L0 hypervisor under nested) between the time that
> > it calculates the desired guest counter value and when KVM actually sets
> > this counter state.
> >
> > Despite the value-based interface that we present to userspace, KVM
> > actually has idempotent guest controls by way of system counter offsets.
> > We can avoid all of the issues associated with a value-based interface
> > by abstracting these offset controls in new ioctls. This series
> > introduces new vCPU device attributes to provide userspace access to the
> > vCPU's system counter offset.
> >
> > Patch 1 addresses a possible race in KVM_GET_CLOCK where
> > use_master_clock is read outside of the pvclock_gtod_sync_lock.
> >
> > Patch 2 adopts Paolo's suggestion, augmenting the KVM_{GET,SET}_CLOCK
> > ioctls to provide userspace with a (host_tsc, realtime) instant. This is
> > essential for a VMM to perform precise migration of the guest's system
> > counters.
> >
> > Patches 3-4 are some preparatory changes for exposing the TSC offset to
> > userspace. Patch 5 provides a vCPU attribute to provide userspace access
> > to the TSC offset.
> >
> > Patches 6-7 implement a test for the new additions to
> > KVM_{GET,SET}_CLOCK.
> >
> > Patch 8 fixes some assertions in the kvm device attribute helpers.
> >
> > Patches 9-10 implement at test for the tsc offset attribute introduced in
> > patch 5.
>
> The x86 parts look good, except that patch 3 is a bit redundant with my
> idea of altogether getting rid of the pvclock_gtod_sync_lock. That said
> I agree that patches 1 and 2 (and extracting kvm_vm_ioctl_get_clock and
> kvm_vm_ioctl_set_clock) should be done before whatever locking changes
> have to be done.
Following up on patch 3.
> Time is ticking for 5.15 due to my vacation, I'll see if I have some
> time to look at it further next week.
>
> I agree that arm64 can be done separately from x86.
Marc, just a disclaimer:
I'm going to separate these two series, although there will still
exist dependencies in the selftests changes. Otherwise, kernel changes
are disjoint.
--
Thanks,
Oliver
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
WARNING: multiple messages have this Message-ID (diff)
From: Oliver Upton <oupton@google.com>
To: Paolo Bonzini <pbonzini@redhat.com>
Cc: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu,
Sean Christopherson <seanjc@google.com>,
Marc Zyngier <maz@kernel.org>, Peter Shier <pshier@google.com>,
Jim Mattson <jmattson@google.com>,
David Matlack <dmatlack@google.com>,
Ricardo Koller <ricarkol@google.com>,
Jing Zhang <jingzhangos@google.com>,
Raghavendra Rao Anata <rananta@google.com>,
James Morse <james.morse@arm.com>,
Alexandru Elisei <Alexandru.Elisei@arm.com>,
Suzuki K Poulose <suzuki.poulose@arm.com>,
linux-arm-kernel@lists.infradead.org,
Andrew Jones <drjones@redhat.com>, Will Deacon <will@kernel.org>,
Catalin Marinas <catalin.marinas@arm.com>
Subject: Re: [PATCH v6 00/21] KVM: Add idempotent controls for migrating system counter state
Date: Wed, 11 Aug 2021 11:56:22 -0700 [thread overview]
Message-ID: <CAOQ_Qsjby+z_kU49_s0PDKo5c3V-UD+FRg-2PcPycNq-p2gPVg@mail.gmail.com> (raw)
In-Reply-To: <927240ff-a4f4-fcc6-ae1b-92cefeda9e59@redhat.com>
On Wed, Aug 11, 2021 at 6:05 AM Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> On 04/08/21 10:57, Oliver Upton wrote:
> > KVM's current means of saving/restoring system counters is plagued with
> > temporal issues. At least on ARM64 and x86, we migrate the guest's
> > system counter by-value through the respective guest system register
> > values (cntvct_el0, ia32_tsc). Restoring system counters by-value is
> > brittle as the state is not idempotent: the host system counter is still
> > oscillating between the attempted save and restore. Furthermore, VMMs
> > may wish to transparently live migrate guest VMs, meaning that they
> > include the elapsed time due to live migration blackout in the guest
> > system counter view. The VMM thread could be preempted for any number of
> > reasons (scheduler, L0 hypervisor under nested) between the time that
> > it calculates the desired guest counter value and when KVM actually sets
> > this counter state.
> >
> > Despite the value-based interface that we present to userspace, KVM
> > actually has idempotent guest controls by way of system counter offsets.
> > We can avoid all of the issues associated with a value-based interface
> > by abstracting these offset controls in new ioctls. This series
> > introduces new vCPU device attributes to provide userspace access to the
> > vCPU's system counter offset.
> >
> > Patch 1 addresses a possible race in KVM_GET_CLOCK where
> > use_master_clock is read outside of the pvclock_gtod_sync_lock.
> >
> > Patch 2 adopts Paolo's suggestion, augmenting the KVM_{GET,SET}_CLOCK
> > ioctls to provide userspace with a (host_tsc, realtime) instant. This is
> > essential for a VMM to perform precise migration of the guest's system
> > counters.
> >
> > Patches 3-4 are some preparatory changes for exposing the TSC offset to
> > userspace. Patch 5 provides a vCPU attribute to provide userspace access
> > to the TSC offset.
> >
> > Patches 6-7 implement a test for the new additions to
> > KVM_{GET,SET}_CLOCK.
> >
> > Patch 8 fixes some assertions in the kvm device attribute helpers.
> >
> > Patches 9-10 implement at test for the tsc offset attribute introduced in
> > patch 5.
>
> The x86 parts look good, except that patch 3 is a bit redundant with my
> idea of altogether getting rid of the pvclock_gtod_sync_lock. That said
> I agree that patches 1 and 2 (and extracting kvm_vm_ioctl_get_clock and
> kvm_vm_ioctl_set_clock) should be done before whatever locking changes
> have to be done.
Following up on patch 3.
> Time is ticking for 5.15 due to my vacation, I'll see if I have some
> time to look at it further next week.
>
> I agree that arm64 can be done separately from x86.
Marc, just a disclaimer:
I'm going to separate these two series, although there will still
exist dependencies in the selftests changes. Otherwise, kernel changes
are disjoint.
--
Thanks,
Oliver
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2021-08-11 18:56 UTC|newest]
Thread overview: 153+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-08-04 8:57 [PATCH v6 00/21] KVM: Add idempotent controls for migrating system counter state Oliver Upton
2021-08-04 8:57 ` Oliver Upton
2021-08-04 8:57 ` Oliver Upton
2021-08-04 8:57 ` [PATCH v6 01/21] KVM: x86: Fix potential race in KVM_GET_CLOCK Oliver Upton
2021-08-04 8:57 ` Oliver Upton
2021-08-04 8:57 ` Oliver Upton
2021-08-11 12:23 ` Paolo Bonzini
2021-08-11 12:23 ` Paolo Bonzini
2021-08-11 12:23 ` Paolo Bonzini
2021-08-13 10:39 ` Oliver Upton
2021-08-13 10:39 ` Oliver Upton
2021-08-13 10:39 ` Oliver Upton
2021-08-13 10:44 ` Paolo Bonzini
2021-08-13 10:44 ` Paolo Bonzini
2021-08-13 10:44 ` Paolo Bonzini
2021-08-13 17:46 ` Oliver Upton
2021-08-13 17:46 ` Oliver Upton
2021-08-13 17:46 ` Oliver Upton
2021-08-04 8:58 ` [PATCH v6 02/21] KVM: x86: Report host tsc and realtime values " Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 8:58 ` [PATCH v6 03/21] KVM: x86: Take the pvclock sync lock behind the tsc_write_lock Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 8:58 ` [PATCH v6 04/21] KVM: x86: Refactor tsc synchronization code Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 8:58 ` [PATCH v6 05/21] KVM: x86: Expose TSC offset controls to userspace Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 8:58 ` [PATCH v6 06/21] tools: arch: x86: pull in pvclock headers Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 8:58 ` [PATCH v6 07/21] selftests: KVM: Add test for KVM_{GET,SET}_CLOCK Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 8:58 ` [PATCH v6 08/21] selftests: KVM: Fix kvm device helper ioctl assertions Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 8:58 ` [PATCH v6 09/21] selftests: KVM: Add helpers for vCPU device attributes Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 8:58 ` [PATCH v6 10/21] selftests: KVM: Introduce system counter offset test Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 8:58 ` [PATCH v6 11/21] KVM: arm64: Refactor update_vtimer_cntvoff() Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 9:23 ` Andrew Jones
2021-08-04 9:23 ` Andrew Jones
2021-08-04 9:23 ` Andrew Jones
2021-08-04 8:58 ` [PATCH v6 12/21] KVM: arm64: Separate guest/host counter offset values Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 10:19 ` Andrew Jones
2021-08-04 10:19 ` Andrew Jones
2021-08-04 10:19 ` Andrew Jones
2021-08-04 8:58 ` [PATCH v6 13/21] KVM: arm64: Allow userspace to configure a vCPU's virtual offset Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 10:20 ` Andrew Jones
2021-08-04 10:20 ` Andrew Jones
2021-08-04 10:20 ` Andrew Jones
2021-08-10 9:35 ` Marc Zyngier
2021-08-10 9:35 ` Marc Zyngier
2021-08-10 9:35 ` Marc Zyngier
2021-08-10 9:44 ` Oliver Upton
2021-08-10 9:44 ` Oliver Upton
2021-08-10 9:44 ` Oliver Upton
2021-08-11 15:22 ` Marc Zyngier
2021-08-11 15:22 ` Marc Zyngier
2021-08-11 15:22 ` Marc Zyngier
2021-08-04 8:58 ` [PATCH v6 14/21] selftests: KVM: Add helper to check for register presence Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 9:14 ` Andrew Jones
2021-08-04 9:14 ` Andrew Jones
2021-08-04 9:14 ` Andrew Jones
2021-08-04 8:58 ` [PATCH v6 15/21] selftests: KVM: Add support for aarch64 to system_counter_offset_test Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 8:58 ` [PATCH v6 16/21] arm64: cpufeature: Enumerate support for Enhanced Counter Virtualization Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-10 9:38 ` Marc Zyngier
2021-08-10 9:38 ` Marc Zyngier
2021-08-10 9:38 ` Marc Zyngier
2021-08-04 8:58 ` [PATCH v6 17/21] KVM: arm64: Allow userspace to configure a guest's counter-timer offset Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 10:17 ` Andrew Jones
2021-08-04 10:17 ` Andrew Jones
2021-08-04 10:17 ` Andrew Jones
2021-08-04 10:22 ` Oliver Upton
2021-08-04 10:22 ` Oliver Upton
2021-08-04 10:22 ` Oliver Upton
2021-08-10 10:56 ` Marc Zyngier
2021-08-10 10:56 ` Marc Zyngier
2021-08-10 10:56 ` Marc Zyngier
2021-08-10 17:55 ` Oliver Upton
2021-08-10 17:55 ` Oliver Upton
2021-08-10 17:55 ` Oliver Upton
2021-08-11 9:01 ` Marc Zyngier
2021-08-11 9:01 ` Marc Zyngier
2021-08-11 9:01 ` Marc Zyngier
2021-08-04 8:58 ` [PATCH v6 18/21] KVM: arm64: Configure timer traps in vcpu_load() for VHE Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 10:25 ` Andrew Jones
2021-08-04 10:25 ` Andrew Jones
2021-08-04 10:25 ` Andrew Jones
2021-08-04 8:58 ` [PATCH v6 19/21] KVM: arm64: Emulate physical counter offsetting on non-ECV systems Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 11:05 ` Andrew Jones
2021-08-04 11:05 ` Andrew Jones
2021-08-04 11:05 ` Andrew Jones
2021-08-05 6:27 ` Oliver Upton
2021-08-05 6:27 ` Oliver Upton
2021-08-05 6:27 ` Oliver Upton
2021-08-10 11:27 ` Marc Zyngier
2021-08-10 11:27 ` Marc Zyngier
2021-08-10 11:27 ` Marc Zyngier
2021-08-04 8:58 ` [PATCH v6 20/21] selftests: KVM: Test physical counter offsetting Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 11:03 ` Andrew Jones
2021-08-04 11:03 ` Andrew Jones
2021-08-04 11:03 ` Andrew Jones
2021-08-04 8:58 ` [PATCH v6 21/21] selftests: KVM: Add counter emulation benchmark Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 8:58 ` Oliver Upton
2021-08-04 11:05 ` [PATCH v6 00/21] KVM: Add idempotent controls for migrating system counter state Oliver Upton
2021-08-04 11:05 ` Oliver Upton
2021-08-04 11:05 ` Oliver Upton
2021-08-04 22:03 ` Oliver Upton
2021-08-04 22:03 ` Oliver Upton
2021-08-04 22:03 ` Oliver Upton
2021-08-10 0:04 ` Oliver Upton
2021-08-10 0:04 ` Oliver Upton
2021-08-10 0:04 ` Oliver Upton
2021-08-10 12:30 ` Marc Zyngier
2021-08-10 12:30 ` Marc Zyngier
2021-08-10 12:30 ` Marc Zyngier
2021-08-11 13:05 ` Paolo Bonzini
2021-08-11 13:05 ` Paolo Bonzini
2021-08-11 13:05 ` Paolo Bonzini
2021-08-11 18:56 ` Oliver Upton [this message]
2021-08-11 18:56 ` Oliver Upton
2021-08-11 18:56 ` Oliver Upton
2021-08-11 19:01 ` Marc Zyngier
2021-08-11 19:01 ` Marc Zyngier
2021-08-11 19:01 ` Marc Zyngier
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAOQ_Qsjby+z_kU49_s0PDKo5c3V-UD+FRg-2PcPycNq-p2gPVg@mail.gmail.com \
--to=oupton@google.com \
--cc=Alexandru.Elisei@arm.com \
--cc=catalin.marinas@arm.com \
--cc=dmatlack@google.com \
--cc=drjones@redhat.com \
--cc=james.morse@arm.com \
--cc=jingzhangos@google.com \
--cc=jmattson@google.com \
--cc=kvm@vger.kernel.org \
--cc=kvmarm@lists.cs.columbia.edu \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=maz@kernel.org \
--cc=pbonzini@redhat.com \
--cc=pshier@google.com \
--cc=rananta@google.com \
--cc=ricarkol@google.com \
--cc=seanjc@google.com \
--cc=suzuki.poulose@arm.com \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.