From: Roman Bolshakov <r.bolshakov@yadro.com>
To: Paolo Bonzini <pbonzini@redhat.com>
Cc: Eduardo Habkost <ehabkost@redhat.com>,
qemu-devel@nongnu.org, Cameron Esfahani <dirty@apple.com>,
Claudio Fontana <cfontana@suse.de>,
Richard Henderson <rth@twiddle.net>
Subject: Re: [PATCH 4/8] i386: hvf: Implement CPU kick
Date: Thu, 25 Jun 2020 18:57:12 +0300 [thread overview]
Message-ID: <20200625155712.GI25104@SPB-NB-133.local> (raw)
In-Reply-To: <9d63db4e-25a7-9c77-6f05-e5f808b8b33c@redhat.com>
On Thu, Jun 25, 2020 at 12:28:26PM +0200, Paolo Bonzini wrote:
> On 25/06/20 00:58, Roman Bolshakov wrote:
> > HVF doesn't have a CPU kick and without it it's not possible to perform
> > an action on CPU thread until a VMEXIT happens. The kick is also needed
> > for timely interrupt delivery.
> >
> > Existing implementation of CPU kick sends SIG_IPI (aka SIGUSR1) to vCPU
> > thread, but it's different from what hv_vcpu_interrupt does. The latter
> > one results in invocation of mp_cpus_kick() in XNU kernel [1].
> >
> > While at it, correct type of hvf_fd to the type of hv_vcpuid_t to avoid
> > compilation warnings.
> >
> > 1. https://opensource.apple.com/source/xnu/xnu-6153.81.5/osfmk/i386/mp.c
> >
> > Cc: Cameron Esfahani <dirty@apple.com>
> > Signed-off-by: Roman Bolshakov <r.bolshakov@yadro.com>
> > ---
> > cpus.c | 13 +++++++++----
> > include/hw/core/cpu.h | 2 +-
> > include/sysemu/hvf.h | 1 +
> > target/i386/hvf/hvf.c | 11 +++++++++++
> > 4 files changed, 22 insertions(+), 5 deletions(-)
> >
> > diff --git a/cpus.c b/cpus.c
> > index 26709677d3..36f38ce5c8 100644
> > --- a/cpus.c
> > +++ b/cpus.c
> > @@ -1783,10 +1783,15 @@ static void qemu_cpu_kick_thread(CPUState *cpu)
> > return;
> > }
> > cpu->thread_kicked = true;
> > - err = pthread_kill(cpu->thread->thread, SIG_IPI);
> > - if (err && err != ESRCH) {
> > - fprintf(stderr, "qemu:%s: %s", __func__, strerror(err));
> > - exit(1);
> > +
> > + if (hvf_enabled()) {
> > + hvf_vcpu_kick(cpu);
> > + } else {
> > + err = pthread_kill(cpu->thread->thread, SIG_IPI);
> > + if (err && err != ESRCH) {
> > + fprintf(stderr, "qemu:%s: %s", __func__, strerror(err));
> > + exit(1);
> > + }
> > }
> > #else /* _WIN32 */
> > if (!qemu_cpu_is_self(cpu)) {
> > diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
> > index b3f4b79318..288a2bd57e 100644
> > --- a/include/hw/core/cpu.h
> > +++ b/include/hw/core/cpu.h
> > @@ -438,7 +438,7 @@ struct CPUState {
> >
> > struct hax_vcpu_state *hax_vcpu;
> >
> > - int hvf_fd;
> > + unsigned hvf_fd;
> >
> > /* track IOMMUs whose translations we've cached in the TCG TLB */
> > GArray *iommu_notifiers;
> > diff --git a/include/sysemu/hvf.h b/include/sysemu/hvf.h
> > index 1d40a8ec01..aaa00cbf05 100644
> > --- a/include/sysemu/hvf.h
> > +++ b/include/sysemu/hvf.h
> > @@ -25,6 +25,7 @@ extern bool hvf_allowed;
> >
> > int hvf_init_vcpu(CPUState *);
> > int hvf_vcpu_exec(CPUState *);
> > +void hvf_vcpu_kick(CPUState *);
> > void hvf_cpu_synchronize_state(CPUState *);
> > void hvf_cpu_synchronize_post_reset(CPUState *);
> > void hvf_cpu_synchronize_post_init(CPUState *);
> > diff --git a/target/i386/hvf/hvf.c b/target/i386/hvf/hvf.c
> > index efe9802962..4d254a477a 100644
> > --- a/target/i386/hvf/hvf.c
> > +++ b/target/i386/hvf/hvf.c
> > @@ -966,6 +966,17 @@ int hvf_vcpu_exec(CPUState *cpu)
> > return ret;
> > }
> >
> > +void hvf_vcpu_kick(CPUState *cpu)
> > +{
> > + hv_return_t err;
> > +
> > + err = hv_vcpu_interrupt(&cpu->hvf_fd, 1);
> > + if (err) {
> > + fprintf(stderr, "qemu:%s error %#x\n", __func__, err);
> > + exit(1);
> > + }
> > +}
> > +
> > bool hvf_allowed;
> >
> > static int hvf_accel_init(MachineState *ms)
> >
>
> The documentation isn't clear on whether hv_vcpu_interrupt is able to
> interrupt a *subsequent* hv_vcpu_run, similar to WHPX
> WHvCancelRunVirtualProcessor (is it possible to decompile
> hv_vcpu_interrupt and see what it does?).
hv_vcpu_interrupt sends a KICK IPI using mp_cpus_kick() only if the
destination vCPU thread is running as far as I undrestand the
mp_cpus_kick():
void
mp_cpus_kick(cpumask_t cpus)
{
cpu_t cpu;
boolean_t intrs_enabled = FALSE;
intrs_enabled = ml_set_interrupts_enabled(FALSE);
mp_safe_spin_lock(&x86_topo_lock);
for (cpu = 0; cpu < (cpu_t) real_ncpus; cpu++) {
if ((cpu == (cpu_t) cpu_number())
|| ((cpu_to_cpumask(cpu) & cpus) == 0)
|| !cpu_is_running(cpu)) {
continue;
}
lapic_send_ipi(cpu, LAPIC_VECTOR(KICK));
}
simple_unlock(&x86_topo_lock);
ml_set_interrupts_enabled(intrs_enabled);
}
So, the kick is not delivered to self and in case if destination cpu is
not running. I think it can't interrupt subsequent hv_vcpu_run.
> If not, you can reduce a bit the race window by setting a variable in
> cpu, like
>
> atomic_set(&cpu->deadline, 0);
> hv_vcpu_interrupt(...)
>
> and in the vCPU thread
>
> hv_vcpu_run_until(..., atomic_read(&cpu->deadline));
> atomic_set(&cpu->deadline, HV_DEADLINE_FOREVER);
>
Sure, could you please explain who'll be racing? There's a race if a
kick was sent after VMEXIT, right? So essentially we need a way to
"requeue" a kick that was received outside of hv_vcpu_run to avoid loss
of it?
hv_vcpu_run_until is only available on macOS 10.15+ and we can't use yet
because of three release support rule.
(https://developer.apple.com/documentation/hypervisor/3181548-hv_vcpu_run_until?language=objc)
BTW, I'm totally okay to send v2 if kicks are lost and/or the patch
needs improvements. (and I can address EFER to VMCS Entry Controls
synchronization as well)
Paolo, do you know any particular test in kvm-unit-tests that can
exhibit the issue?
Thanks,
Roman
next prev parent reply other threads:[~2020-06-25 16:10 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-06-24 22:58 [PATCH 0/8] Improve synchronization between QEMU and HVF Roman Bolshakov
2020-06-24 22:58 ` [PATCH 1/8] i386: hvf: Set env->eip in macvm_set_rip() Roman Bolshakov
2020-06-24 22:58 ` [PATCH 2/8] i386: hvf: Move synchronize functions to sysemu Roman Bolshakov
2020-06-25 7:09 ` Claudio Fontana
2020-06-24 22:58 ` [PATCH 3/8] i386: hvf: Add hvf_cpu_synchronize_pre_loadvm() Roman Bolshakov
2020-06-24 22:58 ` [PATCH 4/8] i386: hvf: Implement CPU kick Roman Bolshakov
2020-06-25 7:07 ` Claudio Fontana
2020-06-25 10:51 ` Roman Bolshakov
2020-06-25 10:28 ` Paolo Bonzini
2020-06-25 15:57 ` Roman Bolshakov [this message]
2020-06-25 18:34 ` Paolo Bonzini
2020-06-29 11:31 ` Roman Bolshakov
2020-06-29 13:03 ` Paolo Bonzini
2020-06-29 13:29 ` Roman Bolshakov
2020-06-29 13:35 ` Paolo Bonzini
2020-06-29 14:04 ` Roman Bolshakov
2020-06-29 14:18 ` Paolo Bonzini
2020-06-30 10:12 ` Roman Bolshakov
2020-06-30 10:43 ` Paolo Bonzini
2020-06-24 22:58 ` [PATCH 5/8] i386: hvf: Don't duplicate register reset Roman Bolshakov
2020-06-24 22:58 ` [PATCH 6/8] i386: hvf: Drop hvf_reset_vcpu() Roman Bolshakov
2020-06-25 10:31 ` Paolo Bonzini
2020-06-25 12:36 ` Roman Bolshakov
2020-06-25 13:30 ` Paolo Bonzini
2020-06-25 15:02 ` Roman Bolshakov
2020-06-25 18:26 ` Paolo Bonzini
2020-06-29 12:58 ` Roman Bolshakov
2020-06-24 22:58 ` [PATCH 7/8] i386: hvf: Clean up synchronize functions Roman Bolshakov
2020-06-24 22:58 ` [PATCH 8/8] MAINTAINERS: Add Cameron as HVF co-maintainer Roman Bolshakov
2020-06-25 11:08 ` [PATCH 0/8] Improve synchronization between QEMU and HVF Paolo Bonzini
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200625155712.GI25104@SPB-NB-133.local \
--to=r.bolshakov@yadro.com \
--cc=cfontana@suse.de \
--cc=dirty@apple.com \
--cc=ehabkost@redhat.com \
--cc=pbonzini@redhat.com \
--cc=qemu-devel@nongnu.org \
--cc=rth@twiddle.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).