* KVM: x86: use smp_send_reschedule in kvm_vcpu_kick @ 2009-03-03 0:14 Marcelo Tosatti 2009-03-03 7:53 ` Zhang, Xiantao 2009-03-09 10:17 ` KVM: " Avi Kivity 0 siblings, 2 replies; 17+ messages in thread From: Marcelo Tosatti @ 2009-03-03 0:14 UTC (permalink / raw) To: kvm; +Cc: Gleb Natapov, Ingo Molnar, Peter Zijlstra KVM uses a function call IPI to cause the exit of a guest running on a physical cpu. For virtual interrupt notification there is no need to wait on IPI receival, or to execute any function. This is exactly what the reschedule IPI does, without the overhead of function IPI. So use it instead of smp_call_function_single in kvm_vcpu_kick. Also change the "guest_mode" variable to a bit in vcpu->requests, and use that to collapse multiple IPI's that would be issued between the first one and zeroing of guest mode. This allows kvm_vcpu_kick to called from interrupt context. diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index b556b6a..c68168c 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -3133,6 +3133,9 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run) local_irq_disable(); + clear_bit(KVM_REQ_KICK, &vcpu->requests); + smp_mb__after_clear_bit(); + if (vcpu->requests || need_resched() || signal_pending(current)) { local_irq_enable(); preempt_enable(); @@ -3140,13 +3143,6 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run) goto out; } - vcpu->guest_mode = 1; - /* - * Make sure that guest_mode assignment won't happen after - * testing the pending IRQ vector bitmap. - */ - smp_wmb(); - if (vcpu->arch.exception.pending) __queue_exception(vcpu); else if (irqchip_in_kernel(vcpu->kvm)) @@ -3188,7 +3184,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run) set_debugreg(vcpu->arch.host_dr6, 6); set_debugreg(vcpu->arch.host_dr7, 7); - vcpu->guest_mode = 0; + set_bit(KVM_REQ_KICK, &vcpu->requests); local_irq_enable(); ++vcpu->stat.exits; @@ -4429,28 +4425,19 @@ int kvm_arch_vcpu_runnable(struct kvm_vcpu *vcpu) || vcpu->arch.nmi_pending; } -static void vcpu_kick_intr(void *info) -{ -#ifdef DEBUG - struct kvm_vcpu *vcpu = (struct kvm_vcpu *)info; - printk(KERN_DEBUG "vcpu_kick_intr %p \n", vcpu); -#endif -} - void kvm_vcpu_kick(struct kvm_vcpu *vcpu) { - int ipi_pcpu = vcpu->cpu; - int cpu = get_cpu(); + int me = get_cpu(); + int cpu = vcpu->cpu; if (waitqueue_active(&vcpu->wq)) { wake_up_interruptible(&vcpu->wq); ++vcpu->stat.halt_wakeup; } - /* - * We may be called synchronously with irqs disabled in guest mode, - * So need not to call smp_call_function_single() in that case. - */ - if (vcpu->guest_mode && vcpu->cpu != cpu) - smp_call_function_single(ipi_pcpu, vcpu_kick_intr, vcpu, 0); + + if (cpu != me && (unsigned)cpu < nr_cpu_ids && cpu_online(cpu)) + if (!test_and_set_bit(KVM_REQ_KICK, &vcpu->requests)) + smp_send_reschedule(cpu); + put_cpu(); } diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 3832243..ef7c239 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -38,6 +38,7 @@ #define KVM_REQ_UNHALT 6 #define KVM_REQ_MMU_SYNC 7 #define KVM_REQ_KVMCLOCK_UPDATE 8 +#define KVM_REQ_KICK 9 #define KVM_USERSPACE_IRQ_SOURCE_ID 0 @@ -72,7 +73,6 @@ struct kvm_vcpu { struct mutex mutex; int cpu; struct kvm_run *run; - int guest_mode; unsigned long requests; unsigned long guest_debug; int fpu_active; ^ permalink raw reply related [flat|nested] 17+ messages in thread
* RE: x86: use smp_send_reschedule in kvm_vcpu_kick 2009-03-03 0:14 KVM: x86: use smp_send_reschedule in kvm_vcpu_kick Marcelo Tosatti @ 2009-03-03 7:53 ` Zhang, Xiantao 2009-03-03 13:25 ` Marcelo Tosatti 2009-03-09 10:17 ` KVM: " Avi Kivity 1 sibling, 1 reply; 17+ messages in thread From: Zhang, Xiantao @ 2009-03-03 7:53 UTC (permalink / raw) To: Marcelo Tosatti, kvm; +Cc: Gleb Natapov, Ingo Molnar, Peter Zijlstra Marcelo Tosatti wrote: > @@ -72,7 +73,6 @@ struct kvm_vcpu { > struct mutex mutex; > int cpu; > struct kvm_run *run; > - int guest_mode; > unsigned long requests; > unsigned long guest_debug; > int fpu_active; The deletion will break other archs which depend on the filed. Xiantao ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: x86: use smp_send_reschedule in kvm_vcpu_kick 2009-03-03 7:53 ` Zhang, Xiantao @ 2009-03-03 13:25 ` Marcelo Tosatti 2009-03-03 13:25 ` Gleb Natapov 0 siblings, 1 reply; 17+ messages in thread From: Marcelo Tosatti @ 2009-03-03 13:25 UTC (permalink / raw) To: Zhang, Xiantao; +Cc: kvm, Gleb Natapov, Ingo Molnar, Peter Zijlstra On Tue, Mar 03, 2009 at 03:53:50PM +0800, Zhang, Xiantao wrote: > Marcelo Tosatti wrote: > > @@ -72,7 +73,6 @@ struct kvm_vcpu { > > struct mutex mutex; > > int cpu; > > struct kvm_run *run; > > - int guest_mode; > > unsigned long requests; > > unsigned long guest_debug; > > int fpu_active; > > The deletion will break other archs which depend on the filed. Indeed. Will keep it, thanks. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: x86: use smp_send_reschedule in kvm_vcpu_kick 2009-03-03 13:25 ` Marcelo Tosatti @ 2009-03-03 13:25 ` Gleb Natapov 2009-03-03 13:56 ` Marcelo Tosatti 0 siblings, 1 reply; 17+ messages in thread From: Gleb Natapov @ 2009-03-03 13:25 UTC (permalink / raw) To: Marcelo Tosatti; +Cc: Zhang, Xiantao, kvm, Ingo Molnar, Peter Zijlstra On Tue, Mar 03, 2009 at 10:25:20AM -0300, Marcelo Tosatti wrote: > On Tue, Mar 03, 2009 at 03:53:50PM +0800, Zhang, Xiantao wrote: > > Marcelo Tosatti wrote: > > > @@ -72,7 +73,6 @@ struct kvm_vcpu { > > > struct mutex mutex; > > > int cpu; > > > struct kvm_run *run; > > > - int guest_mode; > > > unsigned long requests; > > > unsigned long guest_debug; > > > int fpu_active; > > > > The deletion will break other archs which depend on the filed. > > Indeed. Will keep it, thanks. The only other arch is ia64 and its kvm_vcpu_kick() should be fixed the same way. No? -- Gleb. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: x86: use smp_send_reschedule in kvm_vcpu_kick 2009-03-03 13:25 ` Gleb Natapov @ 2009-03-03 13:56 ` Marcelo Tosatti 0 siblings, 0 replies; 17+ messages in thread From: Marcelo Tosatti @ 2009-03-03 13:56 UTC (permalink / raw) To: Gleb Natapov; +Cc: Zhang, Xiantao, kvm, Ingo Molnar, Peter Zijlstra On Tue, Mar 03, 2009 at 03:25:57PM +0200, Gleb Natapov wrote: > On Tue, Mar 03, 2009 at 10:25:20AM -0300, Marcelo Tosatti wrote: > > On Tue, Mar 03, 2009 at 03:53:50PM +0800, Zhang, Xiantao wrote: > > > Marcelo Tosatti wrote: > > > > @@ -72,7 +73,6 @@ struct kvm_vcpu { > > > > struct mutex mutex; > > > > int cpu; > > > > struct kvm_run *run; > > > > - int guest_mode; > > > > unsigned long requests; > > > > unsigned long guest_debug; > > > > int fpu_active; > > > > > > The deletion will break other archs which depend on the filed. > > > > Indeed. Will keep it, thanks. > The only other arch is ia64 and its kvm_vcpu_kick() should be fixed > the same way. No? Probably yes. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: KVM: x86: use smp_send_reschedule in kvm_vcpu_kick 2009-03-03 0:14 KVM: x86: use smp_send_reschedule in kvm_vcpu_kick Marcelo Tosatti 2009-03-03 7:53 ` Zhang, Xiantao @ 2009-03-09 10:17 ` Avi Kivity 2009-03-09 10:58 ` Ingo Molnar 1 sibling, 1 reply; 17+ messages in thread From: Avi Kivity @ 2009-03-09 10:17 UTC (permalink / raw) To: Marcelo Tosatti; +Cc: kvm, Gleb Natapov, Ingo Molnar, Peter Zijlstra Marcelo Tosatti wrote: > KVM uses a function call IPI to cause the exit of a guest running on a > physical cpu. For virtual interrupt notification there is no need to > wait on IPI receival, or to execute any function. > > This is exactly what the reschedule IPI does, without the overhead > of function IPI. So use it instead of smp_call_function_single in > kvm_vcpu_kick. > > Also change the "guest_mode" variable to a bit in vcpu->requests, and > use that to collapse multiple IPI's that would be issued between the > first one and zeroing of guest mode. > > This allows kvm_vcpu_kick to called from interrupt context. > Looks good. The only worry I have is that we depend on smp_reschedule_interrupt() being a no-op. I guess that's a reasonable assumption though. -- error compiling committee.c: too many arguments to function ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: KVM: x86: use smp_send_reschedule in kvm_vcpu_kick 2009-03-09 10:17 ` KVM: " Avi Kivity @ 2009-03-09 10:58 ` Ingo Molnar 2009-03-09 11:09 ` Peter Zijlstra 0 siblings, 1 reply; 17+ messages in thread From: Ingo Molnar @ 2009-03-09 10:58 UTC (permalink / raw) To: Avi Kivity; +Cc: Marcelo Tosatti, kvm, Gleb Natapov, Peter Zijlstra * Avi Kivity <avi@redhat.com> wrote: > Marcelo Tosatti wrote: >> KVM uses a function call IPI to cause the exit of a guest running on a >> physical cpu. For virtual interrupt notification there is no need to >> wait on IPI receival, or to execute any function. >> >> This is exactly what the reschedule IPI does, without the overhead >> of function IPI. So use it instead of smp_call_function_single in >> kvm_vcpu_kick. >> >> Also change the "guest_mode" variable to a bit in vcpu->requests, and >> use that to collapse multiple IPI's that would be issued between the >> first one and zeroing of guest mode. >> >> This allows kvm_vcpu_kick to called from interrupt context. >> > > Looks good. The only worry I have is that we depend on > smp_reschedule_interrupt() being a no-op. I guess that's a > reasonable assumption though. It's a reasonable current assumption - but it might change in the future - so please also put it into the changelog that KVM will revert it or fix it differently if the scheduler grows some functionality there. Ingo ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: KVM: x86: use smp_send_reschedule in kvm_vcpu_kick 2009-03-09 10:58 ` Ingo Molnar @ 2009-03-09 11:09 ` Peter Zijlstra 2009-03-09 23:23 ` Marcelo Tosatti 0 siblings, 1 reply; 17+ messages in thread From: Peter Zijlstra @ 2009-03-09 11:09 UTC (permalink / raw) To: Ingo Molnar Cc: Avi Kivity, Marcelo Tosatti, kvm, Gleb Natapov, Peter Zijlstra On Mon, 2009-03-09 at 11:58 +0100, Ingo Molnar wrote: > * Avi Kivity <avi@redhat.com> wrote: > > > Marcelo Tosatti wrote: > >> KVM uses a function call IPI to cause the exit of a guest running on a > >> physical cpu. For virtual interrupt notification there is no need to > >> wait on IPI receival, or to execute any function. > >> > >> This is exactly what the reschedule IPI does, without the overhead > >> of function IPI. So use it instead of smp_call_function_single in > >> kvm_vcpu_kick. > >> > >> Also change the "guest_mode" variable to a bit in vcpu->requests, and > >> use that to collapse multiple IPI's that would be issued between the > >> first one and zeroing of guest mode. > >> > >> This allows kvm_vcpu_kick to called from interrupt context. > >> > > > > Looks good. The only worry I have is that we depend on > > smp_reschedule_interrupt() being a no-op. I guess that's a > > reasonable assumption though. > > It's a reasonable current assumption - but it might change in > the future - so please also put it into the changelog that KVM > will revert it or fix it differently if the scheduler grows some > functionality there. Alternatively, do something like the below, then anybody poking at that code will know to prod the KVM folks when they change anything. [ I suspect other kvm arches will need to a similar comment ] --- arch/x86/kernel/smp.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/arch/x86/kernel/smp.c b/arch/x86/kernel/smp.c index 13f33ea..3b2e55e 100644 --- a/arch/x86/kernel/smp.c +++ b/arch/x86/kernel/smp.c @@ -172,6 +172,9 @@ void smp_reschedule_interrupt(struct pt_regs *regs) { ack_APIC_irq(); inc_irq_stat(irq_resched_count); + /* + * KVM uses this interrupt to force a cpu out of guest mode + */ } void smp_call_function_interrupt(struct pt_regs *regs) ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: KVM: x86: use smp_send_reschedule in kvm_vcpu_kick 2009-03-09 11:09 ` Peter Zijlstra @ 2009-03-09 23:23 ` Marcelo Tosatti 2009-03-10 8:01 ` Avi Kivity 0 siblings, 1 reply; 17+ messages in thread From: Marcelo Tosatti @ 2009-03-09 23:23 UTC (permalink / raw) To: Avi Kivity, Zhang, Xiantao; +Cc: Ingo Molnar, kvm, Gleb Natapov, Peter Zijlstra On Mon, Mar 09, 2009 at 12:09:06PM +0100, Peter Zijlstra wrote: > On Mon, 2009-03-09 at 11:58 +0100, Ingo Molnar wrote: > > * Avi Kivity <avi@redhat.com> wrote: > > > > > Looks good. The only worry I have is that we depend on > > > smp_reschedule_interrupt() being a no-op. I guess that's a > > > reasonable assumption though. > > > > It's a reasonable current assumption - but it might change in > > the future - so please also put it into the changelog that KVM > > will revert it or fix it differently if the scheduler grows some > > functionality there. > > Alternatively, do something like the below, then anybody poking at that > code will know to prod the KVM folks when they change anything. > > [ I suspect other kvm arches will need to a similar comment ] OK, reworked patch: - change ia64 in addition to x86 - add comment on smp send reschedule handlers about KVM's usage Untested on IA64. KVM: use smp_send_reschedule in kvm_vcpu_kick KVM uses a function call IPI to cause the exit of a guest running on a physical cpu. For virtual interrupt notification there is no need to wait on IPI receival, or to execute any function. This is exactly what the reschedule IPI does, without the overhead of function IPI. So use it instead of smp_call_function_single in kvm_vcpu_kick. Also change the "guest_mode" variable to a bit in vcpu->requests, and use that to collapse multiple IPI's that would be issued between the first one and zeroing of guest mode. This allows kvm_vcpu_kick to called with interrupts disabled. Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com> diff --git a/arch/ia64/kernel/irq_ia64.c b/arch/ia64/kernel/irq_ia64.c index 28d3d48..0e8242d 100644 --- a/arch/ia64/kernel/irq_ia64.c +++ b/arch/ia64/kernel/irq_ia64.c @@ -607,6 +607,9 @@ static struct irqaction ipi_irqaction = { .name = "IPI" }; +/* + * KVM uses this interrupt to force a cpu out of guest mode + */ static struct irqaction resched_irqaction = { .handler = dummy_handler, .flags = IRQF_DISABLED, diff --git a/arch/ia64/kvm/kvm-ia64.c b/arch/ia64/kvm/kvm-ia64.c index e1984cb..080dcdb 100644 --- a/arch/ia64/kvm/kvm-ia64.c +++ b/arch/ia64/kvm/kvm-ia64.c @@ -633,7 +633,7 @@ again: goto out; } - vcpu->guest_mode = 1; + clear_bit(KVM_REQ_KICK, &vcpu->requests); kvm_guest_enter(); down_read(&vcpu->kvm->slots_lock); r = vti_vcpu_run(vcpu, kvm_run); @@ -645,7 +645,7 @@ again: } vcpu->arch.launched = 1; - vcpu->guest_mode = 0; + set_bit(KVM_REQ_KICK, &vcpu->requests); local_irq_enable(); /* @@ -1802,24 +1802,17 @@ void kvm_arch_hardware_unsetup(void) { } -static void vcpu_kick_intr(void *info) -{ -#ifdef DEBUG - struct kvm_vcpu *vcpu = (struct kvm_vcpu *)info; - printk(KERN_DEBUG"vcpu_kick_intr %p \n", vcpu); -#endif -} - void kvm_vcpu_kick(struct kvm_vcpu *vcpu) { - int ipi_pcpu = vcpu->cpu; - int cpu = get_cpu(); + int me = get_cpu(); + int cpu = vcpu->cpu; if (waitqueue_active(&vcpu->wq)) wake_up_interruptible(&vcpu->wq); - if (vcpu->guest_mode && cpu != ipi_pcpu) - smp_call_function_single(ipi_pcpu, vcpu_kick_intr, vcpu, 0); + if (cpu != me && (unsigned) cpu < nr_cpu_ids && cpu_online(cpu)) + if (!test_and_set_bit(KVM_REQ_KICK, &vcpu->requests)) + smp_send_reschedule(cpu); put_cpu(); } diff --git a/arch/x86/kernel/smp.c b/arch/x86/kernel/smp.c index e6faa33..ca4fa7f 100644 --- a/arch/x86/kernel/smp.c +++ b/arch/x86/kernel/smp.c @@ -173,6 +173,9 @@ void smp_reschedule_interrupt(struct pt_regs *regs) { ack_APIC_irq(); inc_irq_stat(irq_resched_count); + /* + * KVM uses this interrupt to force a cpu out of guest mode + */ } void smp_call_function_interrupt(struct pt_regs *regs) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index b556b6a..c68168c 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -3133,6 +3133,9 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run) local_irq_disable(); + clear_bit(KVM_REQ_KICK, &vcpu->requests); + smp_mb__after_clear_bit(); + if (vcpu->requests || need_resched() || signal_pending(current)) { local_irq_enable(); preempt_enable(); @@ -3140,13 +3143,6 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run) goto out; } - vcpu->guest_mode = 1; - /* - * Make sure that guest_mode assignment won't happen after - * testing the pending IRQ vector bitmap. - */ - smp_wmb(); - if (vcpu->arch.exception.pending) __queue_exception(vcpu); else if (irqchip_in_kernel(vcpu->kvm)) @@ -3188,7 +3184,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run) set_debugreg(vcpu->arch.host_dr6, 6); set_debugreg(vcpu->arch.host_dr7, 7); - vcpu->guest_mode = 0; + set_bit(KVM_REQ_KICK, &vcpu->requests); local_irq_enable(); ++vcpu->stat.exits; @@ -4429,28 +4425,19 @@ int kvm_arch_vcpu_runnable(struct kvm_vcpu *vcpu) || vcpu->arch.nmi_pending; } -static void vcpu_kick_intr(void *info) -{ -#ifdef DEBUG - struct kvm_vcpu *vcpu = (struct kvm_vcpu *)info; - printk(KERN_DEBUG "vcpu_kick_intr %p \n", vcpu); -#endif -} - void kvm_vcpu_kick(struct kvm_vcpu *vcpu) { - int ipi_pcpu = vcpu->cpu; - int cpu = get_cpu(); + int me = get_cpu(); + int cpu = vcpu->cpu; if (waitqueue_active(&vcpu->wq)) { wake_up_interruptible(&vcpu->wq); ++vcpu->stat.halt_wakeup; } - /* - * We may be called synchronously with irqs disabled in guest mode, - * So need not to call smp_call_function_single() in that case. - */ - if (vcpu->guest_mode && vcpu->cpu != cpu) - smp_call_function_single(ipi_pcpu, vcpu_kick_intr, vcpu, 0); + + if (cpu != me && (unsigned)cpu < nr_cpu_ids && cpu_online(cpu)) + if (!test_and_set_bit(KVM_REQ_KICK, &vcpu->requests)) + smp_send_reschedule(cpu); + put_cpu(); } diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 3b91ec9..4d7da7d 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -38,6 +38,7 @@ #define KVM_REQ_UNHALT 6 #define KVM_REQ_MMU_SYNC 7 #define KVM_REQ_KVMCLOCK_UPDATE 8 +#define KVM_REQ_KICK 9 #define KVM_USERSPACE_IRQ_SOURCE_ID 0 @@ -72,7 +73,6 @@ struct kvm_vcpu { struct mutex mutex; int cpu; struct kvm_run *run; - int guest_mode; unsigned long requests; unsigned long guest_debug; int fpu_active; ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: KVM: x86: use smp_send_reschedule in kvm_vcpu_kick 2009-03-09 23:23 ` Marcelo Tosatti @ 2009-03-10 8:01 ` Avi Kivity 2009-03-12 2:31 ` Zhang, Xiantao 0 siblings, 1 reply; 17+ messages in thread From: Avi Kivity @ 2009-03-10 8:01 UTC (permalink / raw) To: Marcelo Tosatti Cc: Zhang, Xiantao, Ingo Molnar, kvm, Gleb Natapov, Peter Zijlstra Marcelo Tosatti wrote: > OK, reworked patch: > - change ia64 in addition to x86 > - add comment on smp send reschedule handlers about KVM's usage > > Untested on IA64. > > KVM: use smp_send_reschedule in kvm_vcpu_kick > > KVM uses a function call IPI to cause the exit of a guest running on a > physical cpu. For virtual interrupt notification there is no need to > wait on IPI receival, or to execute any function. > > This is exactly what the reschedule IPI does, without the overhead > of function IPI. So use it instead of smp_call_function_single in > kvm_vcpu_kick. > > Also change the "guest_mode" variable to a bit in vcpu->requests, and > use that to collapse multiple IPI's that would be issued between the > first one and zeroing of guest mode. > > This allows kvm_vcpu_kick to called with interrupts disabled. > Looks good. Will wait for Xiantao's test-n-ack before applying. -- error compiling committee.c: too many arguments to function ^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: x86: use smp_send_reschedule in kvm_vcpu_kick 2009-03-10 8:01 ` Avi Kivity @ 2009-03-12 2:31 ` Zhang, Xiantao 2009-03-12 7:06 ` Avi Kivity 2009-03-12 11:04 ` Gleb Natapov 0 siblings, 2 replies; 17+ messages in thread From: Zhang, Xiantao @ 2009-03-12 2:31 UTC (permalink / raw) To: Avi Kivity, Marcelo Tosatti Cc: Ingo Molnar, kvm, Gleb Natapov, Peter Zijlstra Avi Kivity wrote: > Marcelo Tosatti wrote: >> OK, reworked patch: >> - change ia64 in addition to x86 >> - add comment on smp send reschedule handlers about KVM's usage >> >> Untested on IA64. >> >> KVM: use smp_send_reschedule in kvm_vcpu_kick >> >> KVM uses a function call IPI to cause the exit of a guest running on >> a physical cpu. For virtual interrupt notification there is no need >> to wait on IPI receival, or to execute any function. >> >> This is exactly what the reschedule IPI does, without the overhead >> of function IPI. So use it instead of smp_call_function_single in >> kvm_vcpu_kick. >> >> Also change the "guest_mode" variable to a bit in vcpu->requests, and >> use that to collapse multiple IPI's that would be issued between the >> first one and zeroing of guest mode. >> >> This allows kvm_vcpu_kick to called with interrupts disabled. >> > > Looks good. Will wait for Xiantao's test-n-ack before applying. kvm-ia64 is broken due to recent check-ins about irq-bits, and we are trying to fix it. For this patch, ia64 has to export the symbol smp_send_reschedule before applying the patch. Xiantao ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: x86: use smp_send_reschedule in kvm_vcpu_kick 2009-03-12 2:31 ` Zhang, Xiantao @ 2009-03-12 7:06 ` Avi Kivity 2009-03-12 11:04 ` Gleb Natapov 1 sibling, 0 replies; 17+ messages in thread From: Avi Kivity @ 2009-03-12 7:06 UTC (permalink / raw) To: Zhang, Xiantao Cc: Marcelo Tosatti, Ingo Molnar, kvm, Gleb Natapov, Peter Zijlstra Zhang, Xiantao wrote: >> >> Looks good. Will wait for Xiantao's test-n-ack before applying. >> > > kvm-ia64 is broken due to recent check-ins about irq-bits, and we are trying to fix it. Sorry about that, see Gleb's patch as well. > For this patch, ia64 has to export the symbol smp_send_reschedule before applying the patch. Please send a patch for that once you've tested it. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: x86: use smp_send_reschedule in kvm_vcpu_kick 2009-03-12 2:31 ` Zhang, Xiantao 2009-03-12 7:06 ` Avi Kivity @ 2009-03-12 11:04 ` Gleb Natapov 2009-03-13 2:15 ` Zhang, Xiantao 1 sibling, 1 reply; 17+ messages in thread From: Gleb Natapov @ 2009-03-12 11:04 UTC (permalink / raw) To: Zhang, Xiantao Cc: Avi Kivity, Marcelo Tosatti, Ingo Molnar, kvm, Peter Zijlstra On Thu, Mar 12, 2009 at 10:31:47AM +0800, Zhang, Xiantao wrote: > Avi Kivity wrote: > > Marcelo Tosatti wrote: > >> OK, reworked patch: > >> - change ia64 in addition to x86 > >> - add comment on smp send reschedule handlers about KVM's usage > >> > >> Untested on IA64. > >> > >> KVM: use smp_send_reschedule in kvm_vcpu_kick > >> > >> KVM uses a function call IPI to cause the exit of a guest running on > >> a physical cpu. For virtual interrupt notification there is no need > >> to wait on IPI receival, or to execute any function. > >> > >> This is exactly what the reschedule IPI does, without the overhead > >> of function IPI. So use it instead of smp_call_function_single in > >> kvm_vcpu_kick. > >> > >> Also change the "guest_mode" variable to a bit in vcpu->requests, and > >> use that to collapse multiple IPI's that would be issued between the > >> first one and zeroing of guest mode. > >> > >> This allows kvm_vcpu_kick to called with interrupts disabled. > >> > > > > Looks good. Will wait for Xiantao's test-n-ack before applying. > > kvm-ia64 is broken due to recent check-ins about irq-bits, and we are trying to fix it. For this patch, ia64 has to export the symbol smp_send_reschedule before applying the patch. Can you try this patch please: http://patchwork.kernel.org/patch/11103/ -- Gleb. ^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: x86: use smp_send_reschedule in kvm_vcpu_kick 2009-03-12 11:04 ` Gleb Natapov @ 2009-03-13 2:15 ` Zhang, Xiantao 2009-03-19 7:38 ` Gleb Natapov 0 siblings, 1 reply; 17+ messages in thread From: Zhang, Xiantao @ 2009-03-13 2:15 UTC (permalink / raw) To: Gleb Natapov Cc: Avi Kivity, Marcelo Tosatti, Ingo Molnar, kvm, Peter Zijlstra We also hacked the source like the patch. But the issue is not caused by it. We are still trying to figure the reason out. Thanks! Xiantao -----Original Message----- From: Gleb Natapov [mailto:gleb@redhat.com] Sent: Thursday, March 12, 2009 7:04 PM To: Zhang, Xiantao Cc: Avi Kivity; Marcelo Tosatti; Ingo Molnar; kvm@vger.kernel.org; Peter Zijlstra Subject: Re: x86: use smp_send_reschedule in kvm_vcpu_kick On Thu, Mar 12, 2009 at 10:31:47AM +0800, Zhang, Xiantao wrote: > Avi Kivity wrote: > > Marcelo Tosatti wrote: > >> OK, reworked patch: > >> - change ia64 in addition to x86 > >> - add comment on smp send reschedule handlers about KVM's usage > >> > >> Untested on IA64. > >> > >> KVM: use smp_send_reschedule in kvm_vcpu_kick > >> > >> KVM uses a function call IPI to cause the exit of a guest running on > >> a physical cpu. For virtual interrupt notification there is no need > >> to wait on IPI receival, or to execute any function. > >> > >> This is exactly what the reschedule IPI does, without the overhead > >> of function IPI. So use it instead of smp_call_function_single in > >> kvm_vcpu_kick. > >> > >> Also change the "guest_mode" variable to a bit in vcpu->requests, and > >> use that to collapse multiple IPI's that would be issued between the > >> first one and zeroing of guest mode. > >> > >> This allows kvm_vcpu_kick to called with interrupts disabled. > >> > > > > Looks good. Will wait for Xiantao's test-n-ack before applying. > > kvm-ia64 is broken due to recent check-ins about irq-bits, and we are trying to fix it. For this patch, ia64 has to export the symbol smp_send_reschedule before applying the patch. Can you try this patch please: http://patchwork.kernel.org/patch/11103/ -- Gleb. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: x86: use smp_send_reschedule in kvm_vcpu_kick 2009-03-13 2:15 ` Zhang, Xiantao @ 2009-03-19 7:38 ` Gleb Natapov 2009-03-19 7:56 ` Zhang, Xiantao 0 siblings, 1 reply; 17+ messages in thread From: Gleb Natapov @ 2009-03-19 7:38 UTC (permalink / raw) To: Zhang, Xiantao Cc: Avi Kivity, Marcelo Tosatti, Ingo Molnar, kvm, Peter Zijlstra On Fri, Mar 13, 2009 at 10:15:22AM +0800, Zhang, Xiantao wrote: > We also hacked the source like the patch. But the issue is not caused by it. We are still trying to figure the reason out. Thanks! > Xiantao > With the patch below I am able to compile kvm-userspace on IA64 and run linux guest. Network doesn't work since there is some kind of problem with PCI interrupts. They are not delivered to a guest. Signed-off-by: Gleb Natapov <gleb@redhat.com> diff --git a/qemu/hw/ipf.c b/qemu/hw/ipf.c index e2b53ee..bbe789a 100644 --- a/qemu/hw/ipf.c +++ b/qemu/hw/ipf.c @@ -700,7 +700,7 @@ void ioapic_set_irq(void *opaque, int irq_num, int level) ioapic_irq_count[vector] -= 1; if (kvm_enabled()) { - if (kvm_set_irq(vector, ioapic_irq_count[vector] == 0)) + if (kvm_set_irq(vector, ioapic_irq_count[vector] == 0, NULL)) return; } } @@ -709,3 +709,17 @@ int ipf_map_irq(PCIDevice *pci_dev, int irq_num) { return ioapic_map_irq(pci_dev->devfn, irq_num); } + +static int apic_irq_delivered; +void apic_reset_irq_delivered(void) +{ + apic_irq_delivered = 0; +} +int apic_get_irq_delivered(void) +{ + return apic_irq_delivered; +} +void apic_set_irq_delivered(void) +{ + apic_irq_delivered = 1; +} diff --git a/qemu/target-ia64/cpu.h b/qemu/target-ia64/cpu.h index 9bad6f6..5960bb9 100644 --- a/qemu/target-ia64/cpu.h +++ b/qemu/target-ia64/cpu.h @@ -29,7 +29,7 @@ #define TARGET_LONG_BITS 64 -#define TARGET_PAGE_BITS 16 +#define TARGET_PAGE_BITS 14 #define ELF_MACHINE EM_IA_64 @@ -40,13 +40,14 @@ #include "cpu-defs.h" #include "softfloat.h" +#define CPUState struct CPUIA64State + typedef struct CPUIA64State { CPU_COMMON; uint32_t hflags; int mp_state; } CPUIA64State; -#define CPUState CPUIA64State #define cpu_gen_code cpu_ia64_gen_code #define cpu_init cpu_ia64_init #define cpu_signal_handler cpu_ia64_signal_handler -- Gleb. ^ permalink raw reply related [flat|nested] 17+ messages in thread
* RE: x86: use smp_send_reschedule in kvm_vcpu_kick 2009-03-19 7:38 ` Gleb Natapov @ 2009-03-19 7:56 ` Zhang, Xiantao 2009-03-19 8:11 ` Gleb Natapov 0 siblings, 1 reply; 17+ messages in thread From: Zhang, Xiantao @ 2009-03-19 7:56 UTC (permalink / raw) To: Gleb Natapov Cc: Avi Kivity, Marcelo Tosatti, Ingo Molnar, kvm, Peter Zijlstra Gleb Natapov wrote: > On Fri, Mar 13, 2009 at 10:15:22AM +0800, Zhang, Xiantao wrote: >> We also hacked the source like the patch. But the issue is not >> caused by it. We are still trying to figure the reason out. Thanks! >> Xiantao >> > > With the patch below I am able to compile kvm-userspace on IA64 and > run linux guest. >Network doesn't work since there is some kind of > problem with PCI interrupts. They are not delivered to a guest. Network should work before kvm-82, and not sure latest irq-bits changes has some side-impact for ipf side. You can reference ioapic_map_irq in ipf.c, and the mapping should match the PRT of bios. > @@ -29,7 +29,7 @@ > > #define TARGET_LONG_BITS 64 > > -#define TARGET_PAGE_BITS 16 > +#define TARGET_PAGE_BITS 14 If the host kernel's page size is 64k ,bits should be 16, and 16k with 14. We may reference kernel's page size here. > #define ELF_MACHINE EM_IA_64 > > @@ -40,13 +40,14 @@ > #include "cpu-defs.h" > > #include "softfloat.h" > +#define CPUState struct CPUIA64State > + > typedef struct CPUIA64State { > CPU_COMMON; > uint32_t hflags; > int mp_state; > } CPUIA64State; > > -#define CPUState CPUIA64State > #define cpu_gen_code cpu_ia64_gen_code > #define cpu_init cpu_ia64_init > #define cpu_signal_handler cpu_ia64_signal_handler ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: x86: use smp_send_reschedule in kvm_vcpu_kick 2009-03-19 7:56 ` Zhang, Xiantao @ 2009-03-19 8:11 ` Gleb Natapov 0 siblings, 0 replies; 17+ messages in thread From: Gleb Natapov @ 2009-03-19 8:11 UTC (permalink / raw) To: Zhang, Xiantao Cc: Avi Kivity, Marcelo Tosatti, Ingo Molnar, kvm, Peter Zijlstra On Thu, Mar 19, 2009 at 03:56:07PM +0800, Zhang, Xiantao wrote: > Gleb Natapov wrote: > > On Fri, Mar 13, 2009 at 10:15:22AM +0800, Zhang, Xiantao wrote: > >> We also hacked the source like the patch. But the issue is not > >> caused by it. We are still trying to figure the reason out. Thanks! > >> Xiantao > >> > > > > With the patch below I am able to compile kvm-userspace on IA64 and > > run linux guest. > >Network doesn't work since there is some kind of > > problem with PCI interrupts. They are not delivered to a guest. > > Network should work before kvm-82, and not sure latest irq-bits changes has some side-impact for ipf side. > You can reference ioapic_map_irq in ipf.c, and the mapping should match the PRT of bios. > Can you send me your Flash.fd? > > @@ -29,7 +29,7 @@ > > > > #define TARGET_LONG_BITS 64 > > > > -#define TARGET_PAGE_BITS 16 > > +#define TARGET_PAGE_BITS 14 > > If the host kernel's page size is 64k ,bits should be 16, and 16k with 14. We may reference kernel's page size here. It should be determined by configure script. I am not sure PAGE_SIZE define is exported to userspace. -- Gleb. ^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2009-03-19 8:15 UTC | newest] Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2009-03-03 0:14 KVM: x86: use smp_send_reschedule in kvm_vcpu_kick Marcelo Tosatti 2009-03-03 7:53 ` Zhang, Xiantao 2009-03-03 13:25 ` Marcelo Tosatti 2009-03-03 13:25 ` Gleb Natapov 2009-03-03 13:56 ` Marcelo Tosatti 2009-03-09 10:17 ` KVM: " Avi Kivity 2009-03-09 10:58 ` Ingo Molnar 2009-03-09 11:09 ` Peter Zijlstra 2009-03-09 23:23 ` Marcelo Tosatti 2009-03-10 8:01 ` Avi Kivity 2009-03-12 2:31 ` Zhang, Xiantao 2009-03-12 7:06 ` Avi Kivity 2009-03-12 11:04 ` Gleb Natapov 2009-03-13 2:15 ` Zhang, Xiantao 2009-03-19 7:38 ` Gleb Natapov 2009-03-19 7:56 ` Zhang, Xiantao 2009-03-19 8:11 ` Gleb Natapov
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.