Re: [PATCH] KVM: VMX cannot yet handle migration-after-preemption, so disable it

Re: [PATCH] KVM: VMX cannot yet handle migration-after-preemption, so disable it

[Gregory Haskins]

On Wed, 2007-07-25 at 06:20 +0300, Avi Kivity wrote:

> 
> I'd rather wait for the real fix, especially as this is an experimental
> branch.  Can you provide details of the issue?
> 

Sure.  Keep in mind I am working with the RT kernel, so things might be
tighter and/or different than the vanilla kernel w.r.t. what I am about
to say:

But in a nutshell, the current code calls vcpu_clear() during a sched-in
operation.  If the vcpu is simply uninitialized, no problem.  We clear
it inline in the handler.  However, if the vcpu is migrating we have an
issue.  A synchronous smp_call_function_single() is invoked to
vcpu_clear the remote pCPU, but you must have interrupts enabled to call
this function to avoid a deadlock in the IPI handshake.  sched-in()
handlers are (and should be) called with interrupts and preemption
disabled. 

One solution is the one in which I provided already: Hobble migration
while the vcpu is locked.  Not very elegant, but it works.

Another solution (of which I am very close to getting working now) still
allows for a high degree of preemption _and_ migration.  It accomplishes
this by essentially deferring the migration to a workqueue and taking
itself off the run-queue all from inside the sched-in handler.  This is
all very tricky since the sched-X handlers are more or less like IRQ
context, but I think I have a suitable solution.  Stay tuned.

Regards,
-Greg


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

Re: [PATCH] KVM: VMX cannot yet handle migration-after-preemption, so disable it

[Avi Kivity]

Gregory Haskins wrote:
> On Wed, 2007-07-25 at 06:20 +0300, Avi Kivity wrote:
>
>   
>> I'd rather wait for the real fix, especially as this is an experimental
>> branch.  Can you provide details of the issue?
>>
>>     
>
> Sure.  Keep in mind I am working with the RT kernel, so things might be
> tighter and/or different than the vanilla kernel w.r.t. what I am about
> to say:
>
> But in a nutshell, the current code calls vcpu_clear() during a sched-in
> operation.  If the vcpu is simply uninitialized, no problem.  We clear
> it inline in the handler.  However, if the vcpu is migrating we have an
> issue.  A synchronous smp_call_function_single() is invoked to
> vcpu_clear the remote pCPU, but you must have interrupts enabled to call
> this function to avoid a deadlock in the IPI handshake.  sched-in()
> handlers are (and should be) called with interrupts and preemption
> disabled. 
>
>   

Preemption disabled yes, but interrupts? IIRC I picked the place where
sched_in is called to have interrupts enabled, but maybe -rt changes
that.  Seems like finding a better position for that hook would solve
the problem.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

Re: [PATCH] KVM: VMX cannot yet handle migration-after-preemption, so disable it

[Gregory Haskins]

On Wed, 2007-07-25 at 07:13 +0300, Avi Kivity wrote:

> 
> Preemption disabled yes, but interrupts? 

How do you think I found this? ;)

>From smp_call_function_single in -rt:

WARN_ON(irqs_disabled());



> IIRC I picked the place where
> sched_in is called to have interrupts enabled, but maybe -rt changes
> that.

Perhaps...I haven't investigated what vanilla does.


>   Seems like finding a better position for that hook would solve
> the problem.

I looked and there didn't appear to be a better place.  At the point
interrupts are restored we were well beyond the context_switch code and
it got plain ugly.  Maybe I should look again.

It may be moot, as I believe I am past the problem of getting the
VMCLEAR operation done on the owning pCPU.  Now I am wrestling with that
issue that I mentioned on IRC:  Problems actually using a migrated
context once we get scheduled back in.  Stay tuned.

-Greg



-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

Re: [PATCH] KVM: VMX cannot yet handle migration-after-preemption, so disable it

[Avi Kivity]

Gregory Haskins wrote:
> This patch applies to the preempt-hooks branch to enable KVM to function
> properly on VMX.  With preemption enabled, the KVM context can get scheduled
> out at arbitrary points.  The preempt-hooks allow us to correctly restore
> the host state while being preemptible, but preemption causes problems on the
> schedule-in side if the vcpu-task migrates to a new pCPU.
>
> Ideally we can handle the migration scenario better (patch for this in
> progress).  However, this patch provides correctness in the interim at the
> expense of a small amount of flexibility. E.g. we can only migrate when the
> vcpu_lock is not held.
>
>   

I'd rather wait for the real fix, especially as this is an experimental
branch.  Can you provide details of the issue?

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

[PATCH] KVM: VMX cannot yet handle migration-after-preemption, so disable it

[Gregory Haskins]

This patch applies to the preempt-hooks branch to enable KVM to function
properly on VMX.  With preemption enabled, the KVM context can get scheduled
out at arbitrary points.  The preempt-hooks allow us to correctly restore
the host state while being preemptible, but preemption causes problems on the
schedule-in side if the vcpu-task migrates to a new pCPU.

Ideally we can handle the migration scenario better (patch for this in
progress).  However, this patch provides correctness in the interim at the
expense of a small amount of flexibility. E.g. we can only migrate when the
vcpu_lock is not held.

Signed-off-by: Gregory Haskins <ghaskins-Et1tbQHTxzrQT0dZR+AlfA@public.gmane.org>
---

 drivers/kvm/kvm.h      |    1 +
 drivers/kvm/kvm_main.c |    3 +++
 2 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/drivers/kvm/kvm.h b/drivers/kvm/kvm.h
index 03f374e..d18537c 100644
--- a/drivers/kvm/kvm.h
+++ b/drivers/kvm/kvm.h
@@ -318,6 +318,7 @@ struct kvm_vcpu {
 		struct vcpu_svm *svm;
 	};
 	struct mutex mutex;
+	cpumask_t cpumask;
 	int   cpu;
 	int   launched;
 	u64 host_tsc;
diff --git a/drivers/kvm/kvm_main.c b/drivers/kvm/kvm_main.c
index ef67572..4a19f6f 100644
--- a/drivers/kvm/kvm_main.c
+++ b/drivers/kvm/kvm_main.c
@@ -245,6 +245,8 @@ static void vcpu_load(struct kvm_vcpu *vcpu)
 
 	mutex_lock(&vcpu->mutex);
 	cpu = get_cpu();
+	vcpu->cpumask = current->cpus_allowed;
+	set_cpus_allowed(current, cpumask_of_cpu(cpu));
 	preempt_hook_register(&vcpu->preempt_hook);
 	kvm_arch_ops->vcpu_load(vcpu, cpu);
 	put_cpu();
@@ -255,6 +257,7 @@ static void vcpu_put(struct kvm_vcpu *vcpu)
 	preempt_disable();
 	kvm_arch_ops->vcpu_put(vcpu);
 	preempt_hook_unregister(&vcpu->preempt_hook);
+	set_cpus_allowed(current, vcpu->cpumask);
 	preempt_enable();
 	mutex_unlock(&vcpu->mutex);
 }


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/