* [PATCH RT 0/5] Linux 3.12.40-rt56-rc1
@ 2015-05-14 14:01 Steven Rostedt
2015-05-14 14:01 ` [PATCH RT 1/5] kernel/irq_work: fix no_hz deadlock Steven Rostedt
` (5 more replies)
0 siblings, 6 replies; 9+ messages in thread
From: Steven Rostedt @ 2015-05-14 14:01 UTC (permalink / raw)
To: linux-kernel, linux-rt-users
Cc: Thomas Gleixner, Carsten Emde, Sebastian Andrzej Siewior,
John Kacur, Paul Gortmaker
Dear RT Folks,
This is the RT stable review cycle of patch 3.12.40-rt56-rc1.
Please scream at me if I messed something up. Please test the patches too.
The -rc release will be uploaded to kernel.org and will be deleted when
the final release is out. This is just a review release (or release candidate).
The pre-releases will not be pushed to the git repository, only the
final release is.
If all goes well, this patch will be converted to the next main release
on 5/18/2015.
Enjoy,
-- Steve
To build 3.12.40-rt56-rc1 directly, the following patches should be applied:
http://www.kernel.org/pub/linux/kernel/v3.x/linux-3.12.tar.xz
http://www.kernel.org/pub/linux/kernel/v3.x/patch-3.12.40.xz
http://www.kernel.org/pub/linux/kernel/projects/rt/3.12/patch-3.12.40-rt56-rc1.patch.xz
You can also build from 3.12.40-rt55 by applying the incremental patch:
http://www.kernel.org/pub/linux/kernel/projects/rt/3.12/incr/patch-3.12.40-rt55-rt56-rc1.patch.xz
Changes from 3.12.40-rt55:
---
Marcelo Tosatti (2):
KVM: lapic: mark LAPIC timer handler as irqsafe
KVM: use simple waitqueue for vcpu->wq
Mike Galbraith (1):
hotplug: Use set_cpus_allowed_ptr() in sync_unplug_thread()
Sebastian Andrzej Siewior (1):
kernel/irq_work: fix no_hz deadlock
Steven Rostedt (Red Hat) (1):
Linux 3.12.40-rt56-rc1
----
arch/arm/kvm/arm.c | 4 ++--
arch/arm/kvm/psci.c | 4 ++--
arch/powerpc/include/asm/kvm_host.h | 4 ++--
arch/powerpc/kernel/time.c | 2 +-
arch/powerpc/kvm/book3s_hv.c | 20 ++++++++--------
arch/s390/include/asm/kvm_host.h | 2 +-
arch/sparc/kernel/pcr.c | 2 --
arch/x86/kernel/irq_work.c | 2 --
arch/x86/kvm/lapic.c | 48 ++++++++++++++++++++++++++++++++-----
include/linux/kvm_host.h | 4 ++--
kernel/cpu.c | 2 +-
kernel/irq_work.c | 5 +---
kernel/time/tick-sched.c | 5 ++++
kernel/timer.c | 2 +-
localversion-rt | 2 +-
virt/kvm/async_pf.c | 4 ++--
virt/kvm/kvm_main.c | 16 ++++++-------
17 files changed, 81 insertions(+), 47 deletions(-)
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH RT 1/5] kernel/irq_work: fix no_hz deadlock
2015-05-14 14:01 [PATCH RT 0/5] Linux 3.12.40-rt56-rc1 Steven Rostedt
@ 2015-05-14 14:01 ` Steven Rostedt
2015-05-14 14:01 ` [PATCH RT 2/5] KVM: lapic: mark LAPIC timer handler as irqsafe Steven Rostedt
` (4 subsequent siblings)
5 siblings, 0 replies; 9+ messages in thread
From: Steven Rostedt @ 2015-05-14 14:01 UTC (permalink / raw)
To: linux-kernel, linux-rt-users
Cc: Thomas Gleixner, Carsten Emde, Sebastian Andrzej Siewior,
John Kacur, Paul Gortmaker, stable-rt
[-- Attachment #1: 0001-kernel-irq_work-fix-no_hz-deadlock.patch --]
[-- Type: text/plain, Size: 4261 bytes --]
3.12.40-rt56-rc1 stable review patch.
If anyone has any objections, please let me know.
------------------
From: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Invoking NO_HZ's irq_work callback from timer irq is not working very
well if the callback decides to invoke hrtimer_cancel():
|hrtimer_try_to_cancel+0x55/0x5f
|hrtimer_cancel+0x16/0x28
|tick_nohz_restart+0x17/0x72
|__tick_nohz_full_check+0x8e/0x93
|nohz_full_kick_work_func+0xe/0x10
|irq_work_run_list+0x39/0x57
|irq_work_tick+0x60/0x67
|update_process_times+0x57/0x67
|tick_sched_handle+0x4a/0x59
|tick_sched_timer+0x3b/0x64
|__run_hrtimer+0x7a/0x149
|hrtimer_interrupt+0x1cc/0x2c5
and here we deadlock while waiting for the lock which we are holding.
To fix this I'm doing the same thing that upstream is doing: is the
irq_work dedicated IRQ and use it only for what is marked as "hirq"
which should only be the FULL_NO_HZ related work.
Cc: stable-rt@vger.kernel.org
Reported-by: Carsten Emde <C.Emde@osadl.org>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
[ Added check for in_irq() before calling irq_work_run() ]
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
arch/powerpc/kernel/time.c | 2 +-
arch/sparc/kernel/pcr.c | 2 --
arch/x86/kernel/irq_work.c | 2 --
kernel/irq_work.c | 5 +----
kernel/time/tick-sched.c | 5 +++++
kernel/timer.c | 2 +-
6 files changed, 8 insertions(+), 10 deletions(-)
diff --git a/arch/powerpc/kernel/time.c b/arch/powerpc/kernel/time.c
index 03cbe6f231dc..945505e6ae9e 100644
--- a/arch/powerpc/kernel/time.c
+++ b/arch/powerpc/kernel/time.c
@@ -423,7 +423,7 @@ unsigned long profile_pc(struct pt_regs *regs)
EXPORT_SYMBOL(profile_pc);
#endif
-#if defined(CONFIG_IRQ_WORK) && !defined(CONFIG_PREEMPT_RT_FULL)
+#if defined(CONFIG_IRQ_WORK)
/*
* 64-bit uses a byte in the PACA, 32-bit uses a per-cpu variable...
diff --git a/arch/sparc/kernel/pcr.c b/arch/sparc/kernel/pcr.c
index 927d9c5e50f5..7e967c8018c8 100644
--- a/arch/sparc/kernel/pcr.c
+++ b/arch/sparc/kernel/pcr.c
@@ -43,12 +43,10 @@ void __irq_entry deferred_pcr_work_irq(int irq, struct pt_regs *regs)
set_irq_regs(old_regs);
}
-#ifndef CONFIG_PREEMPT_RT_FULL
void arch_irq_work_raise(void)
{
set_softint(1 << PIL_DEFERRED_PCR_WORK);
}
-#endif
const struct pcr_ops *pcr_ops;
EXPORT_SYMBOL_GPL(pcr_ops);
diff --git a/arch/x86/kernel/irq_work.c b/arch/x86/kernel/irq_work.c
index 3d21f7bd7b42..1de84e3ab4e0 100644
--- a/arch/x86/kernel/irq_work.c
+++ b/arch/x86/kernel/irq_work.c
@@ -38,7 +38,6 @@ __visible void smp_trace_irq_work_interrupt(struct pt_regs *regs)
exiting_irq();
}
-#ifndef CONFIG_PREEMPT_RT_FULL
void arch_irq_work_raise(void)
{
#ifdef CONFIG_X86_LOCAL_APIC
@@ -49,4 +48,3 @@ void arch_irq_work_raise(void)
apic_wait_icr_idle();
#endif
}
-#endif
diff --git a/kernel/irq_work.c b/kernel/irq_work.c
index 35d21f93bbe8..5f7d93d89c7f 100644
--- a/kernel/irq_work.c
+++ b/kernel/irq_work.c
@@ -16,6 +16,7 @@
#include <linux/tick.h>
#include <linux/cpu.h>
#include <linux/notifier.h>
+#include <linux/interrupt.h>
#include <asm/processor.h>
@@ -51,11 +52,7 @@ static bool irq_work_claim(struct irq_work *work)
return true;
}
-#ifdef CONFIG_PREEMPT_RT_FULL
-void arch_irq_work_raise(void)
-#else
void __weak arch_irq_work_raise(void)
-#endif
{
/*
* Lame architectures will get the timer tick callback
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 7e5c94b245e8..2cfd4c740042 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -175,6 +175,11 @@ static bool can_stop_full_tick(void)
return false;
}
+ if (!arch_irq_work_has_interrupt()) {
+ trace_tick_stop(0, "missing irq work interrupt\n");
+ return false;
+ }
+
/* sched_clock_tick() needs us? */
#ifdef CONFIG_HAVE_UNSTABLE_SCHED_CLOCK
/*
diff --git a/kernel/timer.c b/kernel/timer.c
index 2d9ba1b9f698..286ca044f767 100644
--- a/kernel/timer.c
+++ b/kernel/timer.c
@@ -1450,7 +1450,7 @@ void update_process_times(int user_tick)
scheduler_tick();
run_local_timers();
rcu_check_callbacks(cpu, user_tick);
-#if defined(CONFIG_IRQ_WORK)
+#if defined(CONFIG_IRQ_WORK) && !defined(CONFIG_PREEMPT_RT_FULL)
if (in_irq())
irq_work_run();
#endif
--
2.1.4
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH RT 2/5] KVM: lapic: mark LAPIC timer handler as irqsafe
2015-05-14 14:01 [PATCH RT 0/5] Linux 3.12.40-rt56-rc1 Steven Rostedt
2015-05-14 14:01 ` [PATCH RT 1/5] kernel/irq_work: fix no_hz deadlock Steven Rostedt
@ 2015-05-14 14:01 ` Steven Rostedt
2015-05-14 14:01 ` [PATCH RT 3/5] KVM: use simple waitqueue for vcpu->wq Steven Rostedt
` (3 subsequent siblings)
5 siblings, 0 replies; 9+ messages in thread
From: Steven Rostedt @ 2015-05-14 14:01 UTC (permalink / raw)
To: linux-kernel, linux-rt-users
Cc: Thomas Gleixner, Carsten Emde, Sebastian Andrzej Siewior,
John Kacur, Paul Gortmaker, stable-rt, Marcelo Tosatti
[-- Attachment #1: 0002-KVM-lapic-mark-LAPIC-timer-handler-as-irqsafe.patch --]
[-- Type: text/plain, Size: 3262 bytes --]
3.12.40-rt56-rc1 stable review patch.
If anyone has any objections, please let me know.
------------------
From: Marcelo Tosatti <mtosatti@redhat.com>
Since lapic timer handler only wakes up a simple waitqueue,
it can be executed from hardirq context.
Also handle the case where hrtimer_start_expires fails due to -ETIME,
by injecting the interrupt to the guest immediately.
Reduces average cyclictest latency by 3us.
Cc: stable-rt@vger.kernel.org
Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
arch/x86/kvm/lapic.c | 42 +++++++++++++++++++++++++++++++++++++++---
1 file changed, 39 insertions(+), 3 deletions(-)
diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
index 92bbb397f59d..0c4ebde0452f 100644
--- a/arch/x86/kvm/lapic.c
+++ b/arch/x86/kvm/lapic.c
@@ -1010,8 +1010,38 @@ static void update_divide_count(struct kvm_lapic *apic)
apic->divide_count);
}
+
+static enum hrtimer_restart apic_timer_fn(struct hrtimer *data);
+
+static void apic_timer_expired(struct hrtimer *data)
+{
+ int ret, i = 0;
+ enum hrtimer_restart r;
+ struct kvm_timer *ktimer = container_of(data, struct kvm_timer, timer);
+
+ r = apic_timer_fn(data);
+
+ if (r == HRTIMER_RESTART) {
+ do {
+ ret = hrtimer_start_expires(data, HRTIMER_MODE_ABS);
+ if (ret == -ETIME)
+ hrtimer_add_expires_ns(&ktimer->timer,
+ ktimer->period);
+ i++;
+ } while (ret == -ETIME && i < 10);
+
+ if (ret == -ETIME) {
+ printk_once(KERN_ERR "%s: failed to reprogram timer\n",
+ __func__);
+ WARN_ON_ONCE(1);
+ }
+ }
+}
+
+
static void start_apic_timer(struct kvm_lapic *apic)
{
+ int ret;
ktime_t now;
atomic_set(&apic->lapic_timer.pending, 0);
@@ -1041,9 +1071,11 @@ static void start_apic_timer(struct kvm_lapic *apic)
}
}
- hrtimer_start(&apic->lapic_timer.timer,
+ ret = hrtimer_start(&apic->lapic_timer.timer,
ktime_add_ns(now, apic->lapic_timer.period),
HRTIMER_MODE_ABS);
+ if (ret == -ETIME)
+ apic_timer_expired(&apic->lapic_timer.timer);
apic_debug("%s: bus cycle is %" PRId64 "ns, now 0x%016"
PRIx64 ", "
@@ -1073,8 +1105,10 @@ static void start_apic_timer(struct kvm_lapic *apic)
ns = (tscdeadline - guest_tsc) * 1000000ULL;
do_div(ns, this_tsc_khz);
}
- hrtimer_start(&apic->lapic_timer.timer,
+ ret = hrtimer_start(&apic->lapic_timer.timer,
ktime_add_ns(now, ns), HRTIMER_MODE_ABS);
+ if (ret == -ETIME)
+ apic_timer_expired(&apic->lapic_timer.timer);
local_irq_restore(flags);
}
@@ -1560,6 +1594,7 @@ int kvm_create_lapic(struct kvm_vcpu *vcpu)
hrtimer_init(&apic->lapic_timer.timer, CLOCK_MONOTONIC,
HRTIMER_MODE_ABS);
apic->lapic_timer.timer.function = apic_timer_fn;
+ apic->lapic_timer.timer.irqsafe = 1;
/*
* APIC is created enabled. This will prevent kvm_lapic_set_base from
@@ -1673,7 +1708,8 @@ void __kvm_migrate_apic_timer(struct kvm_vcpu *vcpu)
timer = &vcpu->arch.apic->lapic_timer.timer;
if (hrtimer_cancel(timer))
- hrtimer_start_expires(timer, HRTIMER_MODE_ABS);
+ if (hrtimer_start_expires(timer, HRTIMER_MODE_ABS) == -ETIME)
+ apic_timer_expired(timer);
}
/*
--
2.1.4
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH RT 3/5] KVM: use simple waitqueue for vcpu->wq
2015-05-14 14:01 [PATCH RT 0/5] Linux 3.12.40-rt56-rc1 Steven Rostedt
2015-05-14 14:01 ` [PATCH RT 1/5] kernel/irq_work: fix no_hz deadlock Steven Rostedt
2015-05-14 14:01 ` [PATCH RT 2/5] KVM: lapic: mark LAPIC timer handler as irqsafe Steven Rostedt
@ 2015-05-14 14:01 ` Steven Rostedt
2015-05-14 14:01 ` [PATCH RT 4/5] hotplug: Use set_cpus_allowed_ptr() in sync_unplug_thread() Steven Rostedt
` (2 subsequent siblings)
5 siblings, 0 replies; 9+ messages in thread
From: Steven Rostedt @ 2015-05-14 14:01 UTC (permalink / raw)
To: linux-kernel, linux-rt-users
Cc: Thomas Gleixner, Carsten Emde, Sebastian Andrzej Siewior,
John Kacur, Paul Gortmaker, stable-rt, Marcelo Tosatti
[-- Attachment #1: 0003-KVM-use-simple-waitqueue-for-vcpu-wq.patch --]
[-- Type: text/plain, Size: 9570 bytes --]
3.12.40-rt56-rc1 stable review patch.
If anyone has any objections, please let me know.
------------------
From: Marcelo Tosatti <mtosatti@redhat.com>
The problem:
On -RT, an emulated LAPIC timer instances has the following path:
1) hard interrupt
2) ksoftirqd is scheduled
3) ksoftirqd wakes up vcpu thread
4) vcpu thread is scheduled
This extra context switch introduces unnecessary latency in the
LAPIC path for a KVM guest.
The solution:
Allow waking up vcpu thread from hardirq context,
thus avoiding the need for ksoftirqd to be scheduled.
Normal waitqueues make use of spinlocks, which on -RT
are sleepable locks. Therefore, waking up a waitqueue
waiter involves locking a sleeping lock, which
is not allowed from hard interrupt context.
cyclictest command line:
This patch reduces the average latency in my tests from 14us to 11us.
Cc: stable-rt@vger.kernel.org
Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
arch/arm/kvm/arm.c | 4 ++--
arch/arm/kvm/psci.c | 4 ++--
arch/powerpc/include/asm/kvm_host.h | 4 ++--
arch/powerpc/kvm/book3s_hv.c | 20 ++++++++++----------
arch/s390/include/asm/kvm_host.h | 2 +-
arch/x86/kvm/lapic.c | 6 +++---
include/linux/kvm_host.h | 4 ++--
virt/kvm/async_pf.c | 4 ++--
virt/kvm/kvm_main.c | 16 ++++++++--------
9 files changed, 32 insertions(+), 32 deletions(-)
diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
index 9c697db2787e..5409d4d7c208 100644
--- a/arch/arm/kvm/arm.c
+++ b/arch/arm/kvm/arm.c
@@ -491,9 +491,9 @@ static int kvm_vcpu_first_run_init(struct kvm_vcpu *vcpu)
static void vcpu_pause(struct kvm_vcpu *vcpu)
{
- wait_queue_head_t *wq = kvm_arch_vcpu_wq(vcpu);
+ struct swait_head *wq = kvm_arch_vcpu_wq(vcpu);
- wait_event_interruptible(*wq, !vcpu->arch.pause);
+ swait_event_interruptible(*wq, !vcpu->arch.pause);
}
static int kvm_vcpu_initialized(struct kvm_vcpu *vcpu)
diff --git a/arch/arm/kvm/psci.c b/arch/arm/kvm/psci.c
index 86a693a02ba3..49aed024ad78 100644
--- a/arch/arm/kvm/psci.c
+++ b/arch/arm/kvm/psci.c
@@ -35,7 +35,7 @@ static unsigned long kvm_psci_vcpu_on(struct kvm_vcpu *source_vcpu)
{
struct kvm *kvm = source_vcpu->kvm;
struct kvm_vcpu *vcpu;
- wait_queue_head_t *wq;
+ struct swait_head *wq;
unsigned long cpu_id;
phys_addr_t target_pc;
@@ -66,7 +66,7 @@ static unsigned long kvm_psci_vcpu_on(struct kvm_vcpu *source_vcpu)
vcpu->arch.pause = false;
smp_mb(); /* Make sure the above is visible */
- wake_up_interruptible(wq);
+ swait_wake_interruptible(wq);
return KVM_PSCI_RET_SUCCESS;
}
diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h
index 33283532e9d8..5fa50fc31ab3 100644
--- a/arch/powerpc/include/asm/kvm_host.h
+++ b/arch/powerpc/include/asm/kvm_host.h
@@ -290,7 +290,7 @@ struct kvmppc_vcore {
u8 in_guest;
struct list_head runnable_threads;
spinlock_t lock;
- wait_queue_head_t wq;
+ struct swait_head wq;
u64 stolen_tb;
u64 preempt_tb;
struct kvm_vcpu *runner;
@@ -562,7 +562,7 @@ struct kvm_vcpu_arch {
u8 prodded;
u32 last_inst;
- wait_queue_head_t *wqp;
+ struct swait_head *wqp;
struct kvmppc_vcore *vcore;
int ret;
int trap;
diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
index 211974a386d6..a4e71ea12cf7 100644
--- a/arch/powerpc/kvm/book3s_hv.c
+++ b/arch/powerpc/kvm/book3s_hv.c
@@ -70,11 +70,11 @@ void kvmppc_fast_vcpu_kick(struct kvm_vcpu *vcpu)
{
int me;
int cpu = vcpu->cpu;
- wait_queue_head_t *wqp;
+ struct swait_head *wqp;
wqp = kvm_arch_vcpu_wq(vcpu);
- if (waitqueue_active(wqp)) {
- wake_up_interruptible(wqp);
+ if (swaitqueue_active(wqp)) {
+ swait_wake_interruptible(wqp);
++vcpu->stat.halt_wakeup;
}
@@ -534,8 +534,8 @@ int kvmppc_pseries_do_hcall(struct kvm_vcpu *vcpu)
tvcpu->arch.prodded = 1;
smp_mb();
if (vcpu->arch.ceded) {
- if (waitqueue_active(&vcpu->wq)) {
- wake_up_interruptible(&vcpu->wq);
+ if (swaitqueue_active(&vcpu->wq)) {
+ swait_wake_interruptible(&vcpu->wq);
vcpu->stat.halt_wakeup++;
}
}
@@ -941,7 +941,7 @@ struct kvm_vcpu *kvmppc_core_vcpu_create(struct kvm *kvm, unsigned int id)
if (vcore) {
INIT_LIST_HEAD(&vcore->runnable_threads);
spin_lock_init(&vcore->lock);
- init_waitqueue_head(&vcore->wq);
+ init_swait_head(&vcore->wq);
vcore->preempt_tb = TB_NIL;
}
kvm->arch.vcores[core] = vcore;
@@ -1312,13 +1312,13 @@ static void kvmppc_wait_for_exec(struct kvm_vcpu *vcpu, int wait_state)
*/
static void kvmppc_vcore_blocked(struct kvmppc_vcore *vc)
{
- DEFINE_WAIT(wait);
+ DEFINE_SWAITER(wait);
- prepare_to_wait(&vc->wq, &wait, TASK_INTERRUPTIBLE);
+ swait_prepare(&vc->wq, &wait, TASK_INTERRUPTIBLE);
vc->vcore_state = VCORE_SLEEPING;
spin_unlock(&vc->lock);
schedule();
- finish_wait(&vc->wq, &wait);
+ swait_finish(&vc->wq, &wait);
spin_lock(&vc->lock);
vc->vcore_state = VCORE_INACTIVE;
}
@@ -1360,7 +1360,7 @@ static int kvmppc_run_vcpu(struct kvm_run *kvm_run, struct kvm_vcpu *vcpu)
kvmppc_create_dtl_entry(vcpu, vc);
kvmppc_start_thread(vcpu);
} else if (vc->vcore_state == VCORE_SLEEPING) {
- wake_up(&vc->wq);
+ swait_wake(&vc->wq);
}
}
diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index e87ecaa2c569..295aab26f605 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -228,7 +228,7 @@ struct kvm_s390_local_interrupt {
atomic_t active;
struct kvm_s390_float_interrupt *float_int;
int timer_due; /* event indicator for waitqueue below */
- wait_queue_head_t *wq;
+ struct swait_head *wq;
atomic_t *cpuflags;
unsigned int action_bits;
};
diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
index 0c4ebde0452f..9644aac4334e 100644
--- a/arch/x86/kvm/lapic.c
+++ b/arch/x86/kvm/lapic.c
@@ -1546,7 +1546,7 @@ static enum hrtimer_restart apic_timer_fn(struct hrtimer *data)
struct kvm_timer *ktimer = container_of(data, struct kvm_timer, timer);
struct kvm_lapic *apic = container_of(ktimer, struct kvm_lapic, lapic_timer);
struct kvm_vcpu *vcpu = apic->vcpu;
- wait_queue_head_t *q = &vcpu->wq;
+ struct swait_head *q = &vcpu->wq;
/*
* There is a race window between reading and incrementing, but we do
@@ -1560,8 +1560,8 @@ static enum hrtimer_restart apic_timer_fn(struct hrtimer *data)
kvm_make_request(KVM_REQ_PENDING_TIMER, vcpu);
}
- if (waitqueue_active(q))
- wake_up_interruptible(q);
+ if (swaitqueue_active(q))
+ swait_wake_interruptible(q);
if (lapic_is_periodic(apic)) {
hrtimer_add_expires_ns(&ktimer->timer, ktimer->period);
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index e47c7e2f4d04..7bb6c5ac2ad7 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -234,7 +234,7 @@ struct kvm_vcpu {
int fpu_active;
int guest_fpu_loaded, guest_xcr0_loaded;
- wait_queue_head_t wq;
+ struct swait_head wq;
struct pid *pid;
int sigset_active;
sigset_t sigset;
@@ -669,7 +669,7 @@ static inline void kvm_arch_free_vm(struct kvm *kvm)
}
#endif
-static inline wait_queue_head_t *kvm_arch_vcpu_wq(struct kvm_vcpu *vcpu)
+static inline struct swait_head *kvm_arch_vcpu_wq(struct kvm_vcpu *vcpu)
{
#ifdef __KVM_HAVE_ARCH_WQP
return vcpu->arch.wqp;
diff --git a/virt/kvm/async_pf.c b/virt/kvm/async_pf.c
index 8a39dda7a325..c8b49a1f9278 100644
--- a/virt/kvm/async_pf.c
+++ b/virt/kvm/async_pf.c
@@ -85,8 +85,8 @@ static void async_pf_execute(struct work_struct *work)
trace_kvm_async_pf_completed(addr, page, gva);
- if (waitqueue_active(&vcpu->wq))
- wake_up_interruptible(&vcpu->wq);
+ if (swaitqueue_active(&vcpu->wq))
+ swait_wake_interruptible(&vcpu->wq);
mmdrop(mm);
kvm_put_kvm(vcpu->kvm);
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index b9bf29490b12..efcd02d2a5c6 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -213,7 +213,7 @@ int kvm_vcpu_init(struct kvm_vcpu *vcpu, struct kvm *kvm, unsigned id)
vcpu->kvm = kvm;
vcpu->vcpu_id = id;
vcpu->pid = NULL;
- init_waitqueue_head(&vcpu->wq);
+ init_swait_head(&vcpu->wq);
kvm_async_pf_vcpu_init(vcpu);
page = alloc_page(GFP_KERNEL | __GFP_ZERO);
@@ -1672,10 +1672,10 @@ void mark_page_dirty(struct kvm *kvm, gfn_t gfn)
*/
void kvm_vcpu_block(struct kvm_vcpu *vcpu)
{
- DEFINE_WAIT(wait);
+ DEFINE_SWAITER(wait);
for (;;) {
- prepare_to_wait(&vcpu->wq, &wait, TASK_INTERRUPTIBLE);
+ swait_prepare(&vcpu->wq, &wait, TASK_INTERRUPTIBLE);
if (kvm_arch_vcpu_runnable(vcpu)) {
kvm_make_request(KVM_REQ_UNHALT, vcpu);
@@ -1689,7 +1689,7 @@ void kvm_vcpu_block(struct kvm_vcpu *vcpu)
schedule();
}
- finish_wait(&vcpu->wq, &wait);
+ swait_finish(&vcpu->wq, &wait);
}
#ifndef CONFIG_S390
@@ -1700,11 +1700,11 @@ void kvm_vcpu_kick(struct kvm_vcpu *vcpu)
{
int me;
int cpu = vcpu->cpu;
- wait_queue_head_t *wqp;
+ struct swait_head *wqp;
wqp = kvm_arch_vcpu_wq(vcpu);
- if (waitqueue_active(wqp)) {
- wake_up_interruptible(wqp);
+ if (swaitqueue_active(wqp)) {
+ swait_wake_interruptible(wqp);
++vcpu->stat.halt_wakeup;
}
@@ -1816,7 +1816,7 @@ void kvm_vcpu_on_spin(struct kvm_vcpu *me)
continue;
if (vcpu == me)
continue;
- if (waitqueue_active(&vcpu->wq))
+ if (swaitqueue_active(&vcpu->wq))
continue;
if (!kvm_vcpu_eligible_for_directed_yield(vcpu))
continue;
--
2.1.4
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH RT 4/5] hotplug: Use set_cpus_allowed_ptr() in sync_unplug_thread()
2015-05-14 14:01 [PATCH RT 0/5] Linux 3.12.40-rt56-rc1 Steven Rostedt
` (2 preceding siblings ...)
2015-05-14 14:01 ` [PATCH RT 3/5] KVM: use simple waitqueue for vcpu->wq Steven Rostedt
@ 2015-05-14 14:01 ` Steven Rostedt
2015-05-14 14:01 ` [PATCH RT 5/5] Linux 3.12.40-rt56-rc1 Steven Rostedt
2015-05-20 2:59 ` [PATCH RT] rt, nohz_full: fix nohz_full for PREEMPT_RT_FULL Steven Rostedt
5 siblings, 0 replies; 9+ messages in thread
From: Steven Rostedt @ 2015-05-14 14:01 UTC (permalink / raw)
To: linux-kernel, linux-rt-users
Cc: Thomas Gleixner, Carsten Emde, Sebastian Andrzej Siewior,
John Kacur, Paul Gortmaker, Mike Galbraith, stable-rt
[-- Attachment #1: 0004-hotplug-Use-set_cpus_allowed_ptr-in-sync_unplug_thre.patch --]
[-- Type: text/plain, Size: 2131 bytes --]
3.12.40-rt56-rc1 stable review patch.
If anyone has any objections, please let me know.
------------------
From: Mike Galbraith <umgwanakikbuti@gmail.com>
do_set_cpus_allowed() is not safe vs ->sched_class change.
crash> bt
PID: 11676 TASK: ffff88026f979da0 CPU: 22 COMMAND: "sync_unplug/22"
#0 [ffff880274d25bc8] machine_kexec at ffffffff8103b41c
#1 [ffff880274d25c18] crash_kexec at ffffffff810d881a
#2 [ffff880274d25cd8] oops_end at ffffffff81525818
#3 [ffff880274d25cf8] do_invalid_op at ffffffff81003096
#4 [ffff880274d25d90] invalid_op at ffffffff8152d3de
[exception RIP: set_cpus_allowed_rt+18]
RIP: ffffffff8109e012 RSP: ffff880274d25e48 RFLAGS: 00010202
RAX: ffffffff8109e000 RBX: ffff88026f979da0 RCX: ffff8802770cb6e8
RDX: 0000000000000000 RSI: ffffffff81add700 RDI: ffff88026f979da0
RBP: ffff880274d25e78 R8: ffffffff816112e0 R9: 0000000000000001
R10: 0000000000000001 R11: 0000000000011940 R12: ffff88026f979da0
R13: ffff8802770cb6d0 R14: ffff880274d25fd8 R15: 0000000000000000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#5 [ffff880274d25e60] do_set_cpus_allowed at ffffffff8108e65f
#6 [ffff880274d25e80] sync_unplug_thread at ffffffff81058c08
#7 [ffff880274d25ed8] kthread at ffffffff8107cad6
#8 [ffff880274d25f50] ret_from_fork at ffffffff8152bbbc
crash> task_struct ffff88026f979da0 | grep class
sched_class = 0xffffffff816111e0 <fair_sched_class+64>,
Signed-off-by: Mike Galbraith <umgwanakikbuti@gmail.com>
Cc: stable-rt@vger.kernel.org
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
kernel/cpu.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/cpu.c b/kernel/cpu.c
index 6d2cab15a585..3566e74fae83 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -242,7 +242,7 @@ static int sync_unplug_thread(void *data)
* we don't want any more work on this CPU.
*/
current->flags &= ~PF_NO_SETAFFINITY;
- do_set_cpus_allowed(current, cpu_present_mask);
+ set_cpus_allowed_ptr(current, cpu_present_mask);
migrate_me();
return 0;
}
--
2.1.4
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH RT 5/5] Linux 3.12.40-rt56-rc1
2015-05-14 14:01 [PATCH RT 0/5] Linux 3.12.40-rt56-rc1 Steven Rostedt
` (3 preceding siblings ...)
2015-05-14 14:01 ` [PATCH RT 4/5] hotplug: Use set_cpus_allowed_ptr() in sync_unplug_thread() Steven Rostedt
@ 2015-05-14 14:01 ` Steven Rostedt
2015-05-20 2:59 ` [PATCH RT] rt, nohz_full: fix nohz_full for PREEMPT_RT_FULL Steven Rostedt
5 siblings, 0 replies; 9+ messages in thread
From: Steven Rostedt @ 2015-05-14 14:01 UTC (permalink / raw)
To: linux-kernel, linux-rt-users
Cc: Thomas Gleixner, Carsten Emde, Sebastian Andrzej Siewior,
John Kacur, Paul Gortmaker
[-- Attachment #1: 0005-Linux-3.12.40-rt56-rc1.patch --]
[-- Type: text/plain, Size: 414 bytes --]
3.12.40-rt56-rc1 stable review patch.
If anyone has any objections, please let me know.
------------------
From: "Steven Rostedt (Red Hat)" <rostedt@goodmis.org>
---
localversion-rt | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/localversion-rt b/localversion-rt
index 51b05e9abe6f..a6adf38a57d4 100644
--- a/localversion-rt
+++ b/localversion-rt
@@ -1 +1 @@
--rt55
+-rt56-rc1
--
2.1.4
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH RT] rt, nohz_full: fix nohz_full for PREEMPT_RT_FULL
2015-05-14 14:01 [PATCH RT 0/5] Linux 3.12.40-rt56-rc1 Steven Rostedt
` (4 preceding siblings ...)
2015-05-14 14:01 ` [PATCH RT 5/5] Linux 3.12.40-rt56-rc1 Steven Rostedt
@ 2015-05-20 2:59 ` Steven Rostedt
5 siblings, 0 replies; 9+ messages in thread
From: Steven Rostedt @ 2015-05-20 2:59 UTC (permalink / raw)
To: linux-kernel, linux-rt-users
Cc: Thomas Gleixner, Carsten Emde, Sebastian Andrzej Siewior,
John Kacur, Paul Gortmaker, Mike Galbraith, Muli Baron
I'm adding this to the next 3.12-rt release.
-- Steve
>From 239f9f7716c6a2b466fcbb605fa0102de483c116 Mon Sep 17 00:00:00 2001
From: Mike Galbraith <umgwanakikbuti@gmail.com>
Date: Sat, 11 Apr 2015 15:15:59 +0200
Subject: [PATCH] rt, nohz_full: fix nohz_full for PREEMPT_RT_FULL
A task being ticked and trying to shut the tick down will fail due
to having just awakened ksoftirqd, subtract it from nr_running.
Signed-off-by: Mike Galbraith <umgwanakikbuti@gmail.com>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
kernel/sched/core.c | 29 +++++++++++++++++++++++------
kernel/time/tick-sched.c | 5 +++++
2 files changed, 28 insertions(+), 6 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index af7418b7d212..76873807655a 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -696,17 +696,34 @@ static inline bool got_nohz_idle_kick(void)
#endif /* CONFIG_NO_HZ_COMMON */
#ifdef CONFIG_NO_HZ_FULL
+
+static int ksoftirqd_running(void)
+{
+ struct task_struct *softirqd;
+
+ if (!IS_ENABLED(CONFIG_PREEMPT_RT_FULL))
+ return 0;
+ softirqd = this_cpu_ksoftirqd();
+ if (softirqd && softirqd->on_rq)
+ return 1;
+ return 0;
+}
+
bool sched_can_stop_tick(void)
{
- struct rq *rq;
+ struct rq *rq;
- rq = this_rq();
+ rq = this_rq();
- /* Make sure rq->nr_running update is visible after the IPI */
- smp_rmb();
+ /* Make sure rq->nr_running update is visible after the IPI */
+ smp_rmb();
- /* More than one running task need preemption */
- if (rq->nr_running > 1)
+ /*
+ * More than one running task need preemption
+ *
+ * NOTE, RT: if ksoftirqd is awake, subtract it.
+ */
+ if (rq->nr_running - ksoftirqd_running() > 1)
return false;
return true;
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 2cfd4c740042..3238cfbfbeeb 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -221,7 +221,12 @@ void __tick_nohz_full_check(void)
static void nohz_full_kick_work_func(struct irq_work *work)
{
+ unsigned long flags;
+
+ /* ksoftirqd processes sirqs with interrupts enabled */
+ local_irq_save(flags);
__tick_nohz_full_check();
+ local_irq_restore(flags);
}
static DEFINE_PER_CPU(struct irq_work, nohz_full_kick_work) = {
--
2.1.4
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH RT 3/5] KVM: use simple waitqueue for vcpu->wq
2015-05-14 14:02 [PATCH RT 0/5] Linux 3.10.75-rt81-rc1 Steven Rostedt
@ 2015-05-14 14:02 ` Steven Rostedt
0 siblings, 0 replies; 9+ messages in thread
From: Steven Rostedt @ 2015-05-14 14:02 UTC (permalink / raw)
To: linux-kernel, linux-rt-users
Cc: Thomas Gleixner, Carsten Emde, Sebastian Andrzej Siewior,
John Kacur, Paul Gortmaker, stable-rt, Marcelo Tosatti
[-- Attachment #1: 0003-KVM-use-simple-waitqueue-for-vcpu-wq.patch --]
[-- Type: text/plain, Size: 9568 bytes --]
3.10.75-rt81-rc1 stable review patch.
If anyone has any objections, please let me know.
------------------
From: Marcelo Tosatti <mtosatti@redhat.com>
The problem:
On -RT, an emulated LAPIC timer instances has the following path:
1) hard interrupt
2) ksoftirqd is scheduled
3) ksoftirqd wakes up vcpu thread
4) vcpu thread is scheduled
This extra context switch introduces unnecessary latency in the
LAPIC path for a KVM guest.
The solution:
Allow waking up vcpu thread from hardirq context,
thus avoiding the need for ksoftirqd to be scheduled.
Normal waitqueues make use of spinlocks, which on -RT
are sleepable locks. Therefore, waking up a waitqueue
waiter involves locking a sleeping lock, which
is not allowed from hard interrupt context.
cyclictest command line:
This patch reduces the average latency in my tests from 14us to 11us.
Cc: stable-rt@vger.kernel.org
Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
arch/arm/kvm/arm.c | 4 ++--
arch/arm/kvm/psci.c | 4 ++--
arch/powerpc/include/asm/kvm_host.h | 4 ++--
arch/powerpc/kvm/book3s_hv.c | 20 ++++++++++----------
arch/s390/include/asm/kvm_host.h | 2 +-
arch/x86/kvm/lapic.c | 6 +++---
include/linux/kvm_host.h | 4 ++--
virt/kvm/async_pf.c | 4 ++--
virt/kvm/kvm_main.c | 16 ++++++++--------
9 files changed, 32 insertions(+), 32 deletions(-)
diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
index ef1703b9587b..6837ef09934c 100644
--- a/arch/arm/kvm/arm.c
+++ b/arch/arm/kvm/arm.c
@@ -487,9 +487,9 @@ static int kvm_vcpu_first_run_init(struct kvm_vcpu *vcpu)
static void vcpu_pause(struct kvm_vcpu *vcpu)
{
- wait_queue_head_t *wq = kvm_arch_vcpu_wq(vcpu);
+ struct swait_head *wq = kvm_arch_vcpu_wq(vcpu);
- wait_event_interruptible(*wq, !vcpu->arch.pause);
+ swait_event_interruptible(*wq, !vcpu->arch.pause);
}
static int kvm_vcpu_initialized(struct kvm_vcpu *vcpu)
diff --git a/arch/arm/kvm/psci.c b/arch/arm/kvm/psci.c
index 7ee5bb7a3667..6da25764094f 100644
--- a/arch/arm/kvm/psci.c
+++ b/arch/arm/kvm/psci.c
@@ -35,7 +35,7 @@ static unsigned long kvm_psci_vcpu_on(struct kvm_vcpu *source_vcpu)
{
struct kvm *kvm = source_vcpu->kvm;
struct kvm_vcpu *vcpu;
- wait_queue_head_t *wq;
+ struct swait_head *wq;
unsigned long cpu_id;
phys_addr_t target_pc;
@@ -66,7 +66,7 @@ static unsigned long kvm_psci_vcpu_on(struct kvm_vcpu *source_vcpu)
vcpu->arch.pause = false;
smp_mb(); /* Make sure the above is visible */
- wake_up_interruptible(wq);
+ swait_wake_interruptible(wq);
return KVM_PSCI_RET_SUCCESS;
}
diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h
index af326cde7cb6..286e65ad2fa1 100644
--- a/arch/powerpc/include/asm/kvm_host.h
+++ b/arch/powerpc/include/asm/kvm_host.h
@@ -294,7 +294,7 @@ struct kvmppc_vcore {
u8 in_guest;
struct list_head runnable_threads;
spinlock_t lock;
- wait_queue_head_t wq;
+ struct swait_head wq;
u64 stolen_tb;
u64 preempt_tb;
struct kvm_vcpu *runner;
@@ -566,7 +566,7 @@ struct kvm_vcpu_arch {
u8 prodded;
u32 last_inst;
- wait_queue_head_t *wqp;
+ struct swait_head *wqp;
struct kvmppc_vcore *vcore;
int ret;
int trap;
diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
index 102ad8a255f3..47193e4ae46f 100644
--- a/arch/powerpc/kvm/book3s_hv.c
+++ b/arch/powerpc/kvm/book3s_hv.c
@@ -70,11 +70,11 @@ void kvmppc_fast_vcpu_kick(struct kvm_vcpu *vcpu)
{
int me;
int cpu = vcpu->cpu;
- wait_queue_head_t *wqp;
+ struct swait_head *wqp;
wqp = kvm_arch_vcpu_wq(vcpu);
- if (waitqueue_active(wqp)) {
- wake_up_interruptible(wqp);
+ if (swaitqueue_active(wqp)) {
+ swait_wake_interruptible(wqp);
++vcpu->stat.halt_wakeup;
}
@@ -534,8 +534,8 @@ int kvmppc_pseries_do_hcall(struct kvm_vcpu *vcpu)
tvcpu->arch.prodded = 1;
smp_mb();
if (vcpu->arch.ceded) {
- if (waitqueue_active(&vcpu->wq)) {
- wake_up_interruptible(&vcpu->wq);
+ if (swaitqueue_active(&vcpu->wq)) {
+ swait_wake_interruptible(&vcpu->wq);
vcpu->stat.halt_wakeup++;
}
}
@@ -942,7 +942,7 @@ struct kvm_vcpu *kvmppc_core_vcpu_create(struct kvm *kvm, unsigned int id)
if (vcore) {
INIT_LIST_HEAD(&vcore->runnable_threads);
spin_lock_init(&vcore->lock);
- init_waitqueue_head(&vcore->wq);
+ init_swait_head(&vcore->wq);
vcore->preempt_tb = TB_NIL;
}
kvm->arch.vcores[core] = vcore;
@@ -1315,13 +1315,13 @@ static void kvmppc_wait_for_exec(struct kvm_vcpu *vcpu, int wait_state)
*/
static void kvmppc_vcore_blocked(struct kvmppc_vcore *vc)
{
- DEFINE_WAIT(wait);
+ DEFINE_SWAITER(wait);
- prepare_to_wait(&vc->wq, &wait, TASK_INTERRUPTIBLE);
+ swait_prepare(&vc->wq, &wait, TASK_INTERRUPTIBLE);
vc->vcore_state = VCORE_SLEEPING;
spin_unlock(&vc->lock);
schedule();
- finish_wait(&vc->wq, &wait);
+ swait_finish(&vc->wq, &wait);
spin_lock(&vc->lock);
vc->vcore_state = VCORE_INACTIVE;
}
@@ -1363,7 +1363,7 @@ static int kvmppc_run_vcpu(struct kvm_run *kvm_run, struct kvm_vcpu *vcpu)
kvmppc_create_dtl_entry(vcpu, vc);
kvmppc_start_thread(vcpu);
} else if (vc->vcore_state == VCORE_SLEEPING) {
- wake_up(&vc->wq);
+ swait_wake(&vc->wq);
}
}
diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index 16bd5d169cdb..1a492e2f1075 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -219,7 +219,7 @@ struct kvm_s390_local_interrupt {
atomic_t active;
struct kvm_s390_float_interrupt *float_int;
int timer_due; /* event indicator for waitqueue below */
- wait_queue_head_t wq;
+ struct swait_head wq;
atomic_t *cpuflags;
unsigned int action_bits;
};
diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
index 4500d312f8d9..dce9273a3361 100644
--- a/arch/x86/kvm/lapic.c
+++ b/arch/x86/kvm/lapic.c
@@ -1581,7 +1581,7 @@ static enum hrtimer_restart apic_timer_fn(struct hrtimer *data)
struct kvm_timer *ktimer = container_of(data, struct kvm_timer, timer);
struct kvm_lapic *apic = container_of(ktimer, struct kvm_lapic, lapic_timer);
struct kvm_vcpu *vcpu = apic->vcpu;
- wait_queue_head_t *q = &vcpu->wq;
+ struct swait_head *q = &vcpu->wq;
/*
* There is a race window between reading and incrementing, but we do
@@ -1595,8 +1595,8 @@ static enum hrtimer_restart apic_timer_fn(struct hrtimer *data)
kvm_make_request(KVM_REQ_PENDING_TIMER, vcpu);
}
- if (waitqueue_active(q))
- wake_up_interruptible(q);
+ if (swaitqueue_active(q))
+ swait_wake_interruptible(q);
if (lapic_is_periodic(apic)) {
hrtimer_add_expires_ns(&ktimer->timer, ktimer->period);
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 8db53cfaccdb..be9bcd06c571 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -220,7 +220,7 @@ struct kvm_vcpu {
int fpu_active;
int guest_fpu_loaded, guest_xcr0_loaded;
- wait_queue_head_t wq;
+ struct swait_head wq;
struct pid *pid;
int sigset_active;
sigset_t sigset;
@@ -655,7 +655,7 @@ static inline void kvm_arch_free_vm(struct kvm *kvm)
}
#endif
-static inline wait_queue_head_t *kvm_arch_vcpu_wq(struct kvm_vcpu *vcpu)
+static inline struct swait_head *kvm_arch_vcpu_wq(struct kvm_vcpu *vcpu)
{
#ifdef __KVM_HAVE_ARCH_WQP
return vcpu->arch.wqp;
diff --git a/virt/kvm/async_pf.c b/virt/kvm/async_pf.c
index ea475cd03511..59a9f1eada67 100644
--- a/virt/kvm/async_pf.c
+++ b/virt/kvm/async_pf.c
@@ -85,8 +85,8 @@ static void async_pf_execute(struct work_struct *work)
trace_kvm_async_pf_completed(addr, page, gva);
- if (waitqueue_active(&vcpu->wq))
- wake_up_interruptible(&vcpu->wq);
+ if (swaitqueue_active(&vcpu->wq))
+ swait_wake_interruptible(&vcpu->wq);
mmdrop(mm);
kvm_put_kvm(vcpu->kvm);
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index a17f190be58e..71a1f09cd94b 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -233,7 +233,7 @@ int kvm_vcpu_init(struct kvm_vcpu *vcpu, struct kvm *kvm, unsigned id)
vcpu->kvm = kvm;
vcpu->vcpu_id = id;
vcpu->pid = NULL;
- init_waitqueue_head(&vcpu->wq);
+ init_swait_head(&vcpu->wq);
kvm_async_pf_vcpu_init(vcpu);
page = alloc_page(GFP_KERNEL | __GFP_ZERO);
@@ -1672,10 +1672,10 @@ void mark_page_dirty(struct kvm *kvm, gfn_t gfn)
*/
void kvm_vcpu_block(struct kvm_vcpu *vcpu)
{
- DEFINE_WAIT(wait);
+ DEFINE_SWAITER(wait);
for (;;) {
- prepare_to_wait(&vcpu->wq, &wait, TASK_INTERRUPTIBLE);
+ swait_prepare(&vcpu->wq, &wait, TASK_INTERRUPTIBLE);
if (kvm_arch_vcpu_runnable(vcpu)) {
kvm_make_request(KVM_REQ_UNHALT, vcpu);
@@ -1689,7 +1689,7 @@ void kvm_vcpu_block(struct kvm_vcpu *vcpu)
schedule();
}
- finish_wait(&vcpu->wq, &wait);
+ swait_finish(&vcpu->wq, &wait);
}
#ifndef CONFIG_S390
@@ -1700,11 +1700,11 @@ void kvm_vcpu_kick(struct kvm_vcpu *vcpu)
{
int me;
int cpu = vcpu->cpu;
- wait_queue_head_t *wqp;
+ struct swait_head *wqp;
wqp = kvm_arch_vcpu_wq(vcpu);
- if (waitqueue_active(wqp)) {
- wake_up_interruptible(wqp);
+ if (swaitqueue_active(wqp)) {
+ swait_wake_interruptible(wqp);
++vcpu->stat.halt_wakeup;
}
@@ -1816,7 +1816,7 @@ void kvm_vcpu_on_spin(struct kvm_vcpu *me)
continue;
if (vcpu == me)
continue;
- if (waitqueue_active(&vcpu->wq))
+ if (swaitqueue_active(&vcpu->wq))
continue;
if (!kvm_vcpu_eligible_for_directed_yield(vcpu))
continue;
--
2.1.4
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH RT 3/5] KVM: use simple waitqueue for vcpu->wq
2015-05-14 13:59 [PATCH RT 0/5] Linux 3.14.39-rt38-rc1 Steven Rostedt
@ 2015-05-14 13:59 ` Steven Rostedt
0 siblings, 0 replies; 9+ messages in thread
From: Steven Rostedt @ 2015-05-14 13:59 UTC (permalink / raw)
To: linux-kernel, linux-rt-users
Cc: Thomas Gleixner, Carsten Emde, Sebastian Andrzej Siewior,
John Kacur, Paul Gortmaker, stable-rt, Marcelo Tosatti
[-- Attachment #1: 0003-KVM-use-simple-waitqueue-for-vcpu-wq.patch --]
[-- Type: text/plain, Size: 9664 bytes --]
3.14.39-rt38-rc1 stable review patch.
If anyone has any objections, please let me know.
------------------
From: Marcelo Tosatti <mtosatti@redhat.com>
The problem:
On -RT, an emulated LAPIC timer instances has the following path:
1) hard interrupt
2) ksoftirqd is scheduled
3) ksoftirqd wakes up vcpu thread
4) vcpu thread is scheduled
This extra context switch introduces unnecessary latency in the
LAPIC path for a KVM guest.
The solution:
Allow waking up vcpu thread from hardirq context,
thus avoiding the need for ksoftirqd to be scheduled.
Normal waitqueues make use of spinlocks, which on -RT
are sleepable locks. Therefore, waking up a waitqueue
waiter involves locking a sleeping lock, which
is not allowed from hard interrupt context.
cyclictest command line:
This patch reduces the average latency in my tests from 14us to 11us.
Cc: stable-rt@vger.kernel.org
Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
arch/arm/kvm/arm.c | 4 ++--
arch/arm/kvm/psci.c | 4 ++--
arch/powerpc/include/asm/kvm_host.h | 4 ++--
arch/powerpc/kvm/book3s_hv.c | 20 ++++++++++----------
arch/s390/include/asm/kvm_host.h | 2 +-
arch/x86/kvm/lapic.c | 6 +++---
include/linux/kvm_host.h | 4 ++--
virt/kvm/async_pf.c | 4 ++--
virt/kvm/kvm_main.c | 16 ++++++++--------
9 files changed, 32 insertions(+), 32 deletions(-)
diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
index bd18bb8b2770..9100f8a609a2 100644
--- a/arch/arm/kvm/arm.c
+++ b/arch/arm/kvm/arm.c
@@ -495,9 +495,9 @@ static int kvm_vcpu_first_run_init(struct kvm_vcpu *vcpu)
static void vcpu_pause(struct kvm_vcpu *vcpu)
{
- wait_queue_head_t *wq = kvm_arch_vcpu_wq(vcpu);
+ struct swait_head *wq = kvm_arch_vcpu_wq(vcpu);
- wait_event_interruptible(*wq, !vcpu->arch.pause);
+ swait_event_interruptible(*wq, !vcpu->arch.pause);
}
static int kvm_vcpu_initialized(struct kvm_vcpu *vcpu)
diff --git a/arch/arm/kvm/psci.c b/arch/arm/kvm/psci.c
index 448f60e8d23c..19c6c9c2118c 100644
--- a/arch/arm/kvm/psci.c
+++ b/arch/arm/kvm/psci.c
@@ -36,7 +36,7 @@ static unsigned long kvm_psci_vcpu_on(struct kvm_vcpu *source_vcpu)
{
struct kvm *kvm = source_vcpu->kvm;
struct kvm_vcpu *vcpu = NULL, *tmp;
- wait_queue_head_t *wq;
+ struct swait_head *wq;
unsigned long cpu_id;
unsigned long mpidr;
phys_addr_t target_pc;
@@ -80,7 +80,7 @@ static unsigned long kvm_psci_vcpu_on(struct kvm_vcpu *source_vcpu)
smp_mb(); /* Make sure the above is visible */
wq = kvm_arch_vcpu_wq(vcpu);
- wake_up_interruptible(wq);
+ swait_wake_interruptible(wq);
return KVM_PSCI_RET_SUCCESS;
}
diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h
index 1eaea2dea174..dfaf2c2d8e7f 100644
--- a/arch/powerpc/include/asm/kvm_host.h
+++ b/arch/powerpc/include/asm/kvm_host.h
@@ -295,7 +295,7 @@ struct kvmppc_vcore {
u8 in_guest;
struct list_head runnable_threads;
spinlock_t lock;
- wait_queue_head_t wq;
+ struct swait_head wq;
u64 stolen_tb;
u64 preempt_tb;
struct kvm_vcpu *runner;
@@ -612,7 +612,7 @@ struct kvm_vcpu_arch {
u8 prodded;
u32 last_inst;
- wait_queue_head_t *wqp;
+ struct swait_head *wqp;
struct kvmppc_vcore *vcore;
int ret;
int trap;
diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
index 7a25d9218a05..0019281510da 100644
--- a/arch/powerpc/kvm/book3s_hv.c
+++ b/arch/powerpc/kvm/book3s_hv.c
@@ -74,11 +74,11 @@ static void kvmppc_fast_vcpu_kick_hv(struct kvm_vcpu *vcpu)
{
int me;
int cpu = vcpu->cpu;
- wait_queue_head_t *wqp;
+ struct swait_head *wqp;
wqp = kvm_arch_vcpu_wq(vcpu);
- if (waitqueue_active(wqp)) {
- wake_up_interruptible(wqp);
+ if (swaitqueue_active(wqp)) {
+ swait_wake_interruptible(wqp);
++vcpu->stat.halt_wakeup;
}
@@ -583,8 +583,8 @@ int kvmppc_pseries_do_hcall(struct kvm_vcpu *vcpu)
tvcpu->arch.prodded = 1;
smp_mb();
if (vcpu->arch.ceded) {
- if (waitqueue_active(&vcpu->wq)) {
- wake_up_interruptible(&vcpu->wq);
+ if (swaitqueue_active(&vcpu->wq)) {
+ swait_wake_interruptible(&vcpu->wq);
vcpu->stat.halt_wakeup++;
}
}
@@ -1199,7 +1199,7 @@ static struct kvm_vcpu *kvmppc_core_vcpu_create_hv(struct kvm *kvm,
if (vcore) {
INIT_LIST_HEAD(&vcore->runnable_threads);
spin_lock_init(&vcore->lock);
- init_waitqueue_head(&vcore->wq);
+ init_swait_head(&vcore->wq);
vcore->preempt_tb = TB_NIL;
vcore->lpcr = kvm->arch.lpcr;
vcore->first_vcpuid = core * threads_per_core;
@@ -1566,13 +1566,13 @@ static void kvmppc_wait_for_exec(struct kvm_vcpu *vcpu, int wait_state)
*/
static void kvmppc_vcore_blocked(struct kvmppc_vcore *vc)
{
- DEFINE_WAIT(wait);
+ DEFINE_SWAITER(wait);
- prepare_to_wait(&vc->wq, &wait, TASK_INTERRUPTIBLE);
+ swait_prepare(&vc->wq, &wait, TASK_INTERRUPTIBLE);
vc->vcore_state = VCORE_SLEEPING;
spin_unlock(&vc->lock);
schedule();
- finish_wait(&vc->wq, &wait);
+ swait_finish(&vc->wq, &wait);
spin_lock(&vc->lock);
vc->vcore_state = VCORE_INACTIVE;
}
@@ -1613,7 +1613,7 @@ static int kvmppc_run_vcpu(struct kvm_run *kvm_run, struct kvm_vcpu *vcpu)
kvmppc_create_dtl_entry(vcpu, vc);
kvmppc_start_thread(vcpu);
} else if (vc->vcore_state == VCORE_SLEEPING) {
- wake_up(&vc->wq);
+ swait_wake(&vc->wq);
}
}
diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index eef3dd3fd9a9..65402d300a26 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -233,7 +233,7 @@ struct kvm_s390_local_interrupt {
atomic_t active;
struct kvm_s390_float_interrupt *float_int;
int timer_due; /* event indicator for waitqueue below */
- wait_queue_head_t *wq;
+ struct swait_head *wq;
atomic_t *cpuflags;
unsigned int action_bits;
};
diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
index 6326cce914ab..8b25d010d5e4 100644
--- a/arch/x86/kvm/lapic.c
+++ b/arch/x86/kvm/lapic.c
@@ -1567,7 +1567,7 @@ static enum hrtimer_restart apic_timer_fn(struct hrtimer *data)
struct kvm_timer *ktimer = container_of(data, struct kvm_timer, timer);
struct kvm_lapic *apic = container_of(ktimer, struct kvm_lapic, lapic_timer);
struct kvm_vcpu *vcpu = apic->vcpu;
- wait_queue_head_t *q = &vcpu->wq;
+ struct swait_head *q = &vcpu->wq;
/*
* There is a race window between reading and incrementing, but we do
@@ -1581,8 +1581,8 @@ static enum hrtimer_restart apic_timer_fn(struct hrtimer *data)
kvm_make_request(KVM_REQ_PENDING_TIMER, vcpu);
}
- if (waitqueue_active(q))
- wake_up_interruptible(q);
+ if (swaitqueue_active(q))
+ swait_wake_interruptible(q);
if (lapic_is_periodic(apic)) {
hrtimer_add_expires_ns(&ktimer->timer, ktimer->period);
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index b8e9a43e501a..aebf20c9bfab 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -231,7 +231,7 @@ struct kvm_vcpu {
int fpu_active;
int guest_fpu_loaded, guest_xcr0_loaded;
- wait_queue_head_t wq;
+ struct swait_head wq;
struct pid *pid;
int sigset_active;
sigset_t sigset;
@@ -677,7 +677,7 @@ static inline bool kvm_arch_has_noncoherent_dma(struct kvm *kvm)
}
#endif
-static inline wait_queue_head_t *kvm_arch_vcpu_wq(struct kvm_vcpu *vcpu)
+static inline struct swait_head *kvm_arch_vcpu_wq(struct kvm_vcpu *vcpu)
{
#ifdef __KVM_HAVE_ARCH_WQP
return vcpu->arch.wqp;
diff --git a/virt/kvm/async_pf.c b/virt/kvm/async_pf.c
index f2c80d5451c3..10634baf3b08 100644
--- a/virt/kvm/async_pf.c
+++ b/virt/kvm/async_pf.c
@@ -82,8 +82,8 @@ static void async_pf_execute(struct work_struct *work)
trace_kvm_async_pf_completed(addr, gva);
- if (waitqueue_active(&vcpu->wq))
- wake_up_interruptible(&vcpu->wq);
+ if (swaitqueue_active(&vcpu->wq))
+ swait_wake_interruptible(&vcpu->wq);
mmput(mm);
kvm_put_kvm(vcpu->kvm);
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 66112533b1e9..38a0d7a5ce74 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -219,7 +219,7 @@ int kvm_vcpu_init(struct kvm_vcpu *vcpu, struct kvm *kvm, unsigned id)
vcpu->kvm = kvm;
vcpu->vcpu_id = id;
vcpu->pid = NULL;
- init_waitqueue_head(&vcpu->wq);
+ init_swait_head(&vcpu->wq);
kvm_async_pf_vcpu_init(vcpu);
page = alloc_page(GFP_KERNEL | __GFP_ZERO);
@@ -1675,10 +1675,10 @@ EXPORT_SYMBOL_GPL(mark_page_dirty);
*/
void kvm_vcpu_block(struct kvm_vcpu *vcpu)
{
- DEFINE_WAIT(wait);
+ DEFINE_SWAITER(wait);
for (;;) {
- prepare_to_wait(&vcpu->wq, &wait, TASK_INTERRUPTIBLE);
+ swait_prepare(&vcpu->wq, &wait, TASK_INTERRUPTIBLE);
if (kvm_arch_vcpu_runnable(vcpu)) {
kvm_make_request(KVM_REQ_UNHALT, vcpu);
@@ -1692,7 +1692,7 @@ void kvm_vcpu_block(struct kvm_vcpu *vcpu)
schedule();
}
- finish_wait(&vcpu->wq, &wait);
+ swait_finish(&vcpu->wq, &wait);
}
EXPORT_SYMBOL_GPL(kvm_vcpu_block);
@@ -1704,11 +1704,11 @@ void kvm_vcpu_kick(struct kvm_vcpu *vcpu)
{
int me;
int cpu = vcpu->cpu;
- wait_queue_head_t *wqp;
+ struct swait_head *wqp;
wqp = kvm_arch_vcpu_wq(vcpu);
- if (waitqueue_active(wqp)) {
- wake_up_interruptible(wqp);
+ if (swaitqueue_active(wqp)) {
+ swait_wake_interruptible(wqp);
++vcpu->stat.halt_wakeup;
}
@@ -1814,7 +1814,7 @@ void kvm_vcpu_on_spin(struct kvm_vcpu *me)
continue;
if (vcpu == me)
continue;
- if (waitqueue_active(&vcpu->wq))
+ if (swaitqueue_active(&vcpu->wq))
continue;
if (!kvm_vcpu_eligible_for_directed_yield(vcpu))
continue;
--
2.1.4
^ permalink raw reply related [flat|nested] 9+ messages in thread
end of thread, other threads:[~2015-05-20 2:58 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-05-14 14:01 [PATCH RT 0/5] Linux 3.12.40-rt56-rc1 Steven Rostedt
2015-05-14 14:01 ` [PATCH RT 1/5] kernel/irq_work: fix no_hz deadlock Steven Rostedt
2015-05-14 14:01 ` [PATCH RT 2/5] KVM: lapic: mark LAPIC timer handler as irqsafe Steven Rostedt
2015-05-14 14:01 ` [PATCH RT 3/5] KVM: use simple waitqueue for vcpu->wq Steven Rostedt
2015-05-14 14:01 ` [PATCH RT 4/5] hotplug: Use set_cpus_allowed_ptr() in sync_unplug_thread() Steven Rostedt
2015-05-14 14:01 ` [PATCH RT 5/5] Linux 3.12.40-rt56-rc1 Steven Rostedt
2015-05-20 2:59 ` [PATCH RT] rt, nohz_full: fix nohz_full for PREEMPT_RT_FULL Steven Rostedt
-- strict thread matches above, loose matches on Subject: below --
2015-05-14 14:02 [PATCH RT 0/5] Linux 3.10.75-rt81-rc1 Steven Rostedt
2015-05-14 14:02 ` [PATCH RT 3/5] KVM: use simple waitqueue for vcpu->wq Steven Rostedt
2015-05-14 13:59 [PATCH RT 0/5] Linux 3.14.39-rt38-rc1 Steven Rostedt
2015-05-14 13:59 ` [PATCH RT 3/5] KVM: use simple waitqueue for vcpu->wq Steven Rostedt
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.