* [PATCH 1/5] rcuwait: Fix stale wake call name in comment
2020-04-22 4:07 [PATCH v3 -tip 0/5] kvm: Use rcuwait for vcpu blocking Davidlohr Bueso
@ 2020-04-22 4:07 ` Davidlohr Bueso
2020-04-22 4:07 ` [PATCH 2/5] rcuwait: Let rcuwait_wake_up() return whether or not a task was awoken Davidlohr Bueso
` (4 subsequent siblings)
5 siblings, 0 replies; 16+ messages in thread
From: Davidlohr Bueso @ 2020-04-22 4:07 UTC (permalink / raw)
To: tglx, pbonzini
Cc: bigeasy, peterz, rostedt, torvalds, will, joel, linux-kernel,
kvm, dave, Davidlohr Bueso
The 'trywake' name was renamed to simply 'wake', update the comment.
Signed-off-by: Davidlohr Bueso <dbueso@suse.de>
---
kernel/exit.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/exit.c b/kernel/exit.c
index 389a88cb3081..9f9015f3f6b0 100644
--- a/kernel/exit.c
+++ b/kernel/exit.c
@@ -236,7 +236,7 @@ void rcuwait_wake_up(struct rcuwait *w)
/*
* Order condition vs @task, such that everything prior to the load
* of @task is visible. This is the condition as to why the user called
- * rcuwait_trywake() in the first place. Pairs with set_current_state()
+ * rcuwait_wake() in the first place. Pairs with set_current_state()
* barrier (A) in rcuwait_wait_event().
*
* WAIT WAKE
--
2.16.4
^ permalink raw reply related [flat|nested] 16+ messages in thread* [PATCH 2/5] rcuwait: Let rcuwait_wake_up() return whether or not a task was awoken
2020-04-22 4:07 [PATCH v3 -tip 0/5] kvm: Use rcuwait for vcpu blocking Davidlohr Bueso
2020-04-22 4:07 ` [PATCH 1/5] rcuwait: Fix stale wake call name in comment Davidlohr Bueso
@ 2020-04-22 4:07 ` Davidlohr Bueso
2020-04-22 4:07 ` [PATCH 3/5] rcuwait: Introduce prepare_to and finish_rcuwait Davidlohr Bueso
` (3 subsequent siblings)
5 siblings, 0 replies; 16+ messages in thread
From: Davidlohr Bueso @ 2020-04-22 4:07 UTC (permalink / raw)
To: tglx, pbonzini
Cc: bigeasy, peterz, rostedt, torvalds, will, joel, linux-kernel,
kvm, dave, Davidlohr Bueso
Propagating the return value of wake_up_process() back to the caller
can come in handy for future users, such as for statistics or
accounting purposes.
Signed-off-by: Davidlohr Bueso <dbueso@suse.de>
---
include/linux/rcuwait.h | 2 +-
kernel/exit.c | 7 +++++--
2 files changed, 6 insertions(+), 3 deletions(-)
diff --git a/include/linux/rcuwait.h b/include/linux/rcuwait.h
index 2ffe1ee6d482..6ebb23258a27 100644
--- a/include/linux/rcuwait.h
+++ b/include/linux/rcuwait.h
@@ -25,7 +25,7 @@ static inline void rcuwait_init(struct rcuwait *w)
w->task = NULL;
}
-extern void rcuwait_wake_up(struct rcuwait *w);
+extern int rcuwait_wake_up(struct rcuwait *w);
/*
* The caller is responsible for locking around rcuwait_wait_event(),
diff --git a/kernel/exit.c b/kernel/exit.c
index 9f9015f3f6b0..f3beb637acf7 100644
--- a/kernel/exit.c
+++ b/kernel/exit.c
@@ -227,8 +227,9 @@ void release_task(struct task_struct *p)
goto repeat;
}
-void rcuwait_wake_up(struct rcuwait *w)
+int rcuwait_wake_up(struct rcuwait *w)
{
+ int ret = 0;
struct task_struct *task;
rcu_read_lock();
@@ -248,8 +249,10 @@ void rcuwait_wake_up(struct rcuwait *w)
task = rcu_dereference(w->task);
if (task)
- wake_up_process(task);
+ ret = wake_up_process(task);
rcu_read_unlock();
+
+ return ret;
}
EXPORT_SYMBOL_GPL(rcuwait_wake_up);
--
2.16.4
^ permalink raw reply related [flat|nested] 16+ messages in thread* [PATCH 3/5] rcuwait: Introduce prepare_to and finish_rcuwait
2020-04-22 4:07 [PATCH v3 -tip 0/5] kvm: Use rcuwait for vcpu blocking Davidlohr Bueso
2020-04-22 4:07 ` [PATCH 1/5] rcuwait: Fix stale wake call name in comment Davidlohr Bueso
2020-04-22 4:07 ` [PATCH 2/5] rcuwait: Let rcuwait_wake_up() return whether or not a task was awoken Davidlohr Bueso
@ 2020-04-22 4:07 ` Davidlohr Bueso
2020-04-23 9:23 ` Peter Zijlstra
2020-04-22 4:07 ` [PATCH 4/5] kvm: Replace vcpu->swait with rcuwait Davidlohr Bueso
` (2 subsequent siblings)
5 siblings, 1 reply; 16+ messages in thread
From: Davidlohr Bueso @ 2020-04-22 4:07 UTC (permalink / raw)
To: tglx, pbonzini
Cc: bigeasy, peterz, rostedt, torvalds, will, joel, linux-kernel,
kvm, dave, Davidlohr Bueso
This allows further flexibility for some callers to implement
ad-hoc versions of the generic rcuwait_wait_event(). For example,
kvm will need this to maintain tracing semantics. The naming
is of course similar to what waitqueue apis offer.
Signed-off-by: Davidlohr Bueso <dbueso@suse.de>
---
include/linux/rcuwait.h | 21 ++++++++++++++++-----
1 file changed, 16 insertions(+), 5 deletions(-)
diff --git a/include/linux/rcuwait.h b/include/linux/rcuwait.h
index 6ebb23258a27..0c6a3d0d25ab 100644
--- a/include/linux/rcuwait.h
+++ b/include/linux/rcuwait.h
@@ -29,12 +29,25 @@ extern int rcuwait_wake_up(struct rcuwait *w);
/*
* The caller is responsible for locking around rcuwait_wait_event(),
- * such that writes to @task are properly serialized.
+ * and prepare_to_rcuwait() such that writes to @task are properly
+ * serialized.
*/
+
+static inline void prepare_to_rcuwait(struct rcuwait *w)
+{
+ rcu_assign_pointer(w->task, current);
+}
+
+static inline void finish_rcuwait(struct rcuwait *w)
+{
+ WRITE_ONCE(w->task, NULL);
+ __set_current_state(TASK_RUNNING);
+}
+
#define rcuwait_wait_event(w, condition, state) \
({ \
int __ret = 0; \
- rcu_assign_pointer((w)->task, current); \
+ prepare_to_rcuwait(w); \
for (;;) { \
/* \
* Implicit barrier (A) pairs with (B) in \
@@ -51,9 +64,7 @@ extern int rcuwait_wake_up(struct rcuwait *w);
\
schedule(); \
} \
- \
- WRITE_ONCE((w)->task, NULL); \
- __set_current_state(TASK_RUNNING); \
+ finish_rcuwait(w); \
__ret; \
})
--
2.16.4
^ permalink raw reply related [flat|nested] 16+ messages in thread* Re: [PATCH 3/5] rcuwait: Introduce prepare_to and finish_rcuwait
2020-04-22 4:07 ` [PATCH 3/5] rcuwait: Introduce prepare_to and finish_rcuwait Davidlohr Bueso
@ 2020-04-23 9:23 ` Peter Zijlstra
0 siblings, 0 replies; 16+ messages in thread
From: Peter Zijlstra @ 2020-04-23 9:23 UTC (permalink / raw)
To: Davidlohr Bueso
Cc: tglx, pbonzini, bigeasy, rostedt, torvalds, will, joel,
linux-kernel, kvm, Davidlohr Bueso
On Tue, Apr 21, 2020 at 09:07:37PM -0700, Davidlohr Bueso wrote:
> +static inline void prepare_to_rcuwait(struct rcuwait *w)
> +{
> + rcu_assign_pointer(w->task, current);
> +}
> +
> +static inline void finish_rcuwait(struct rcuwait *w)
> +{
> + WRITE_ONCE(w->task, NULL);
I think that wants to be:
rcu_assign_pointer(w->task, NULL);
There is a special case in rcu_assign_pointer() that looses the barrier,
but it will keep the __rcu sparse people happy. That is w->task is
__rcu, and WRITE_ONCE ignores that etc.. blah.
The alternative is using RCU_INIT_POINTER() I suppose.
> + __set_current_state(TASK_RUNNING);
> +}
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH 4/5] kvm: Replace vcpu->swait with rcuwait
2020-04-22 4:07 [PATCH v3 -tip 0/5] kvm: Use rcuwait for vcpu blocking Davidlohr Bueso
` (2 preceding siblings ...)
2020-04-22 4:07 ` [PATCH 3/5] rcuwait: Introduce prepare_to and finish_rcuwait Davidlohr Bueso
@ 2020-04-22 4:07 ` Davidlohr Bueso
2020-04-22 8:32 ` Paolo Bonzini
` (2 more replies)
2020-04-22 4:07 ` [PATCH 5/5] sched/swait: Reword some of the main description Davidlohr Bueso
2020-04-22 11:33 ` [PATCH v3 -tip 0/5] kvm: Use rcuwait for vcpu blocking Peter Zijlstra
5 siblings, 3 replies; 16+ messages in thread
From: Davidlohr Bueso @ 2020-04-22 4:07 UTC (permalink / raw)
To: tglx, pbonzini
Cc: bigeasy, peterz, rostedt, torvalds, will, joel, linux-kernel,
kvm, dave, Paul Mackerras, kvmarm, linux-mips, Davidlohr Bueso
The use of any sort of waitqueue (simple or regular) for
wait/waking vcpus has always been an overkill and semantically
wrong. Because this is per-vcpu (which is blocked) there is
only ever a single waiting vcpu, thus no need for any sort of
queue.
As such, make use of the rcuwait primitive, with the following
considerations:
- rcuwait already provides the proper barriers that serialize
concurrent waiter and waker.
- Task wakeup is done in rcu read critical region, with a
stable task pointer.
- Because there is no concurrency among waiters, we need
not worry about rcuwait_wait_event() calls corrupting
the wait->task. As a consequence, this saves the locking
done in swait when modifying the queue. This also applies
to per-vcore wait for powerpc kvm-hv.
The x86 tscdeadline_latency test mentioned in 8577370fb0cb
("KVM: Use simple waitqueue for vcpu->wq") shows that, on avg,
latency is reduced by around 15-20% with this change.
Cc: Paul Mackerras <paulus@ozlabs.org>
Cc: kvmarm@lists.cs.columbia.edu
Cc: linux-mips@vger.kernel.org
Signed-off-by: Davidlohr Bueso <dbueso@suse.de>
---
arch/mips/kvm/mips.c | 6 ++----
arch/powerpc/include/asm/kvm_book3s.h | 2 +-
arch/powerpc/include/asm/kvm_host.h | 2 +-
arch/powerpc/kvm/book3s_hv.c | 22 ++++++++--------------
arch/powerpc/kvm/powerpc.c | 2 +-
arch/x86/kvm/lapic.c | 2 +-
include/linux/kvm_host.h | 10 +++++-----
virt/kvm/arm/arch_timer.c | 2 +-
virt/kvm/arm/arm.c | 9 +++++----
virt/kvm/async_pf.c | 3 +--
virt/kvm/kvm_main.c | 19 +++++++++----------
11 files changed, 35 insertions(+), 44 deletions(-)
diff --git a/arch/mips/kvm/mips.c b/arch/mips/kvm/mips.c
index 8f05dd0a0f4e..fad6acce46e4 100644
--- a/arch/mips/kvm/mips.c
+++ b/arch/mips/kvm/mips.c
@@ -284,8 +284,7 @@ static enum hrtimer_restart kvm_mips_comparecount_wakeup(struct hrtimer *timer)
kvm_mips_callbacks->queue_timer_int(vcpu);
vcpu->arch.wait = 0;
- if (swq_has_sleeper(&vcpu->wq))
- swake_up_one(&vcpu->wq);
+ rcuwait_wake_up(&vcpu->wait);
return kvm_mips_count_timeout(vcpu);
}
@@ -511,8 +510,7 @@ int kvm_vcpu_ioctl_interrupt(struct kvm_vcpu *vcpu,
dvcpu->arch.wait = 0;
- if (swq_has_sleeper(&dvcpu->wq))
- swake_up_one(&dvcpu->wq);
+ rcuwait_wake_up(&dvcpu->wait);
return 0;
}
diff --git a/arch/powerpc/include/asm/kvm_book3s.h b/arch/powerpc/include/asm/kvm_book3s.h
index 506e4df2d730..6e5d85ba588d 100644
--- a/arch/powerpc/include/asm/kvm_book3s.h
+++ b/arch/powerpc/include/asm/kvm_book3s.h
@@ -78,7 +78,7 @@ struct kvmppc_vcore {
struct kvm_vcpu *runnable_threads[MAX_SMT_THREADS];
struct list_head preempt_list;
spinlock_t lock;
- struct swait_queue_head wq;
+ struct rcuwait wait;
spinlock_t stoltb_lock; /* protects stolen_tb and preempt_tb */
u64 stolen_tb;
u64 preempt_tb;
diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h
index 1dc63101ffe1..337047ba4a56 100644
--- a/arch/powerpc/include/asm/kvm_host.h
+++ b/arch/powerpc/include/asm/kvm_host.h
@@ -751,7 +751,7 @@ struct kvm_vcpu_arch {
u8 irq_pending; /* Used by XIVE to signal pending guest irqs */
u32 last_inst;
- struct swait_queue_head *wqp;
+ struct rcuwait *waitp;
struct kvmppc_vcore *vcore;
int ret;
int trap;
diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
index 93493f0cbfe8..b8d42f523ca7 100644
--- a/arch/powerpc/kvm/book3s_hv.c
+++ b/arch/powerpc/kvm/book3s_hv.c
@@ -230,13 +230,11 @@ static bool kvmppc_ipi_thread(int cpu)
static void kvmppc_fast_vcpu_kick_hv(struct kvm_vcpu *vcpu)
{
int cpu;
- struct swait_queue_head *wqp;
+ struct rcuwait *wait;
- wqp = kvm_arch_vcpu_wq(vcpu);
- if (swq_has_sleeper(wqp)) {
- swake_up_one(wqp);
+ wait = kvm_arch_vcpu_get_wait(vcpu);
+ if (rcuwait_wake_up(wait))
++vcpu->stat.halt_wakeup;
- }
cpu = READ_ONCE(vcpu->arch.thread_cpu);
if (cpu >= 0 && kvmppc_ipi_thread(cpu))
@@ -2125,7 +2123,7 @@ static struct kvmppc_vcore *kvmppc_vcore_create(struct kvm *kvm, int id)
spin_lock_init(&vcore->lock);
spin_lock_init(&vcore->stoltb_lock);
- init_swait_queue_head(&vcore->wq);
+ rcuwait_init(&vcore->wait);
vcore->preempt_tb = TB_NIL;
vcore->lpcr = kvm->arch.lpcr;
vcore->first_vcpuid = id;
@@ -3784,7 +3782,6 @@ static void kvmppc_vcore_blocked(struct kvmppc_vcore *vc)
ktime_t cur, start_poll, start_wait;
int do_sleep = 1;
u64 block_ns;
- DECLARE_SWAITQUEUE(wait);
/* Poll for pending exceptions and ceded state */
cur = start_poll = ktime_get();
@@ -3812,10 +3809,7 @@ static void kvmppc_vcore_blocked(struct kvmppc_vcore *vc)
}
}
- prepare_to_swait_exclusive(&vc->wq, &wait, TASK_INTERRUPTIBLE);
-
if (kvmppc_vcore_check_block(vc)) {
- finish_swait(&vc->wq, &wait);
do_sleep = 0;
/* If we polled, count this as a successful poll */
if (vc->halt_poll_ns)
@@ -3828,8 +3822,8 @@ static void kvmppc_vcore_blocked(struct kvmppc_vcore *vc)
vc->vcore_state = VCORE_SLEEPING;
trace_kvmppc_vcore_blocked(vc, 0);
spin_unlock(&vc->lock);
- schedule();
- finish_swait(&vc->wq, &wait);
+ rcuwait_wait_event(&vc->wait,
+ kvmppc_vcore_check_block(vc), TASK_INTERRUPTIBLE);
spin_lock(&vc->lock);
vc->vcore_state = VCORE_INACTIVE;
trace_kvmppc_vcore_blocked(vc, 1);
@@ -3940,7 +3934,7 @@ static int kvmppc_run_vcpu(struct kvm_run *kvm_run, struct kvm_vcpu *vcpu)
kvmppc_start_thread(vcpu, vc);
trace_kvm_guest_enter(vcpu);
} else if (vc->vcore_state == VCORE_SLEEPING) {
- swake_up_one(&vc->wq);
+ rcuwait_wake_up(&vc->wait);
}
}
@@ -4279,7 +4273,7 @@ static int kvmppc_vcpu_run_hv(struct kvm_run *run, struct kvm_vcpu *vcpu)
}
user_vrsave = mfspr(SPRN_VRSAVE);
- vcpu->arch.wqp = &vcpu->arch.vcore->wq;
+ vcpu->arch.waitp = &vcpu->arch.vcore->wait;
vcpu->arch.pgdir = kvm->mm->pgd;
vcpu->arch.state = KVMPPC_VCPU_BUSY_IN_HOST;
diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c
index e15166b0a16d..4a074b587520 100644
--- a/arch/powerpc/kvm/powerpc.c
+++ b/arch/powerpc/kvm/powerpc.c
@@ -751,7 +751,7 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
if (err)
goto out_vcpu_uninit;
- vcpu->arch.wqp = &vcpu->wq;
+ vcpu->arch.waitp = &vcpu->wait;
kvmppc_create_vcpu_debugfs(vcpu, vcpu->vcpu_id);
return 0;
diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
index 9af25c97612a..fb2f56ba171d 100644
--- a/arch/x86/kvm/lapic.c
+++ b/arch/x86/kvm/lapic.c
@@ -1833,7 +1833,7 @@ void kvm_lapic_expired_hv_timer(struct kvm_vcpu *vcpu)
/* If the preempt notifier has already run, it also called apic_timer_expired */
if (!apic->lapic_timer.hv_timer_in_use)
goto out;
- WARN_ON(swait_active(&vcpu->wq));
+ WARN_ON(rcu_dereference(vcpu->wait.task));
cancel_hv_timer(apic);
apic_timer_expired(apic);
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 6d58beb65454..fc34021546bd 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -23,7 +23,7 @@
#include <linux/irqflags.h>
#include <linux/context_tracking.h>
#include <linux/irqbypass.h>
-#include <linux/swait.h>
+#include <linux/rcuwait.h>
#include <linux/refcount.h>
#include <linux/nospec.h>
#include <asm/signal.h>
@@ -277,7 +277,7 @@ struct kvm_vcpu {
struct mutex mutex;
struct kvm_run *run;
- struct swait_queue_head wq;
+ struct rcuwait wait;
struct pid __rcu *pid;
int sigset_active;
sigset_t sigset;
@@ -956,12 +956,12 @@ static inline bool kvm_arch_has_assigned_device(struct kvm *kvm)
}
#endif
-static inline struct swait_queue_head *kvm_arch_vcpu_wq(struct kvm_vcpu *vcpu)
+static inline struct rcuwait *kvm_arch_vcpu_get_wait(struct kvm_vcpu *vcpu)
{
#ifdef __KVM_HAVE_ARCH_WQP
- return vcpu->arch.wqp;
+ return vcpu->arch.waitp;
#else
- return &vcpu->wq;
+ return &vcpu->wait;
#endif
}
diff --git a/virt/kvm/arm/arch_timer.c b/virt/kvm/arm/arch_timer.c
index 93bd59b46848..b2805105bbe5 100644
--- a/virt/kvm/arm/arch_timer.c
+++ b/virt/kvm/arm/arch_timer.c
@@ -593,7 +593,7 @@ void kvm_timer_vcpu_put(struct kvm_vcpu *vcpu)
if (map.emul_ptimer)
soft_timer_cancel(&map.emul_ptimer->hrtimer);
- if (swait_active(kvm_arch_vcpu_wq(vcpu)))
+ if (rcu_dereference(kvm_arch_vpu_get_wait(vcpu)) != NULL)
kvm_timer_blocking(vcpu);
/*
diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c
index 48d0ec44ad77..f94a10bb1251 100644
--- a/virt/kvm/arm/arm.c
+++ b/virt/kvm/arm/arm.c
@@ -579,16 +579,17 @@ void kvm_arm_resume_guest(struct kvm *kvm)
kvm_for_each_vcpu(i, vcpu, kvm) {
vcpu->arch.pause = false;
- swake_up_one(kvm_arch_vcpu_wq(vcpu));
+ rcuwait_wake_up(kvm_arch_vcpu_get_wait(vcpu));
}
}
static void vcpu_req_sleep(struct kvm_vcpu *vcpu)
{
- struct swait_queue_head *wq = kvm_arch_vcpu_wq(vcpu);
+ struct rcuwait *wait = kvm_arch_vcpu_get_wait(vcpu);
- swait_event_interruptible_exclusive(*wq, ((!vcpu->arch.power_off) &&
- (!vcpu->arch.pause)));
+ rcuwait_wait_event(*wait,
+ (!vcpu->arch.power_off) &&(!vcpu->arch.pause),
+ TASK_INTERRUPTIBLE);
if (vcpu->arch.power_off || vcpu->arch.pause) {
/* Awaken to handle a signal, request we sleep again later. */
diff --git a/virt/kvm/async_pf.c b/virt/kvm/async_pf.c
index 15e5b037f92d..10b533f641a6 100644
--- a/virt/kvm/async_pf.c
+++ b/virt/kvm/async_pf.c
@@ -80,8 +80,7 @@ static void async_pf_execute(struct work_struct *work)
trace_kvm_async_pf_completed(addr, cr2_or_gpa);
- if (swq_has_sleeper(&vcpu->wq))
- swake_up_one(&vcpu->wq);
+ rcuwait_wake_up(&vcpu->wait);
mmput(mm);
kvm_put_kvm(vcpu->kvm);
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 74bdb7bf3295..835fb109badf 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -341,7 +341,7 @@ static void kvm_vcpu_init(struct kvm_vcpu *vcpu, struct kvm *kvm, unsigned id)
vcpu->kvm = kvm;
vcpu->vcpu_id = id;
vcpu->pid = NULL;
- init_swait_queue_head(&vcpu->wq);
+ rcuwait_init(&vcpu->wait);
kvm_async_pf_vcpu_init(vcpu);
vcpu->pre_pcpu = -1;
@@ -2671,7 +2671,6 @@ static int kvm_vcpu_check_block(struct kvm_vcpu *vcpu)
void kvm_vcpu_block(struct kvm_vcpu *vcpu)
{
ktime_t start, cur;
- DECLARE_SWAITQUEUE(wait);
bool waited = false;
u64 block_ns;
@@ -2697,8 +2696,9 @@ void kvm_vcpu_block(struct kvm_vcpu *vcpu)
} while (single_task_running() && ktime_before(cur, stop));
}
+ prepare_to_rcuwait(&vcpu->wait);
for (;;) {
- prepare_to_swait_exclusive(&vcpu->wq, &wait, TASK_INTERRUPTIBLE);
+ set_current_state(TASK_INTERRUPTIBLE);
if (kvm_vcpu_check_block(vcpu) < 0)
break;
@@ -2706,8 +2706,7 @@ void kvm_vcpu_block(struct kvm_vcpu *vcpu)
waited = true;
schedule();
}
-
- finish_swait(&vcpu->wq, &wait);
+ finish_rcuwait(&vcpu->wait);
cur = ktime_get();
out:
kvm_arch_vcpu_unblocking(vcpu);
@@ -2738,11 +2737,10 @@ EXPORT_SYMBOL_GPL(kvm_vcpu_block);
bool kvm_vcpu_wake_up(struct kvm_vcpu *vcpu)
{
- struct swait_queue_head *wqp;
+ struct rcuwait *wait;
- wqp = kvm_arch_vcpu_wq(vcpu);
- if (swq_has_sleeper(wqp)) {
- swake_up_one(wqp);
+ wait = kvm_arch_vcpu_get_wait(vcpu);
+ if (rcuwait_wake_up(wait)) {
WRITE_ONCE(vcpu->ready, true);
++vcpu->stat.halt_wakeup;
return true;
@@ -2884,7 +2882,8 @@ void kvm_vcpu_on_spin(struct kvm_vcpu *me, bool yield_to_kernel_mode)
continue;
if (vcpu == me)
continue;
- if (swait_active(&vcpu->wq) && !vcpu_dy_runnable(vcpu))
+ if (rcu_dereference(vcpu->wait.task) &&
+ !vcpu_dy_runnable(vcpu))
continue;
if (READ_ONCE(vcpu->preempted) && yield_to_kernel_mode &&
!kvm_arch_vcpu_in_kernel(vcpu))
--
2.16.4
^ permalink raw reply related [flat|nested] 16+ messages in thread* Re: [PATCH 4/5] kvm: Replace vcpu->swait with rcuwait
2020-04-22 4:07 ` [PATCH 4/5] kvm: Replace vcpu->swait with rcuwait Davidlohr Bueso
@ 2020-04-22 8:32 ` Paolo Bonzini
2020-04-22 10:14 ` Marc Zyngier
2020-04-23 8:41 ` Marc Zyngier
2 siblings, 0 replies; 16+ messages in thread
From: Paolo Bonzini @ 2020-04-22 8:32 UTC (permalink / raw)
To: Davidlohr Bueso, tglx
Cc: bigeasy, peterz, rostedt, torvalds, will, joel, linux-kernel,
kvm, Paul Mackerras, kvmarm, linux-mips, Davidlohr Bueso
On 22/04/20 06:07, Davidlohr Bueso wrote:
> The use of any sort of waitqueue (simple or regular) for
> wait/waking vcpus has always been an overkill and semantically
> wrong. Because this is per-vcpu (which is blocked) there is
> only ever a single waiting vcpu, thus no need for any sort of
> queue.
>
> As such, make use of the rcuwait primitive, with the following
> considerations:
>
> - rcuwait already provides the proper barriers that serialize
> concurrent waiter and waker.
>
> - Task wakeup is done in rcu read critical region, with a
> stable task pointer.
>
> - Because there is no concurrency among waiters, we need
> not worry about rcuwait_wait_event() calls corrupting
> the wait->task. As a consequence, this saves the locking
> done in swait when modifying the queue. This also applies
> to per-vcore wait for powerpc kvm-hv.
>
> The x86 tscdeadline_latency test mentioned in 8577370fb0cb
> ("KVM: Use simple waitqueue for vcpu->wq") shows that, on avg,
> latency is reduced by around 15-20% with this change.
>
> Cc: Paul Mackerras <paulus@ozlabs.org>
> Cc: kvmarm@lists.cs.columbia.edu
> Cc: linux-mips@vger.kernel.org
> Signed-off-by: Davidlohr Bueso <dbueso@suse.de>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
> arch/mips/kvm/mips.c | 6 ++----
> arch/powerpc/include/asm/kvm_book3s.h | 2 +-
> arch/powerpc/include/asm/kvm_host.h | 2 +-
> arch/powerpc/kvm/book3s_hv.c | 22 ++++++++--------------
> arch/powerpc/kvm/powerpc.c | 2 +-
> arch/x86/kvm/lapic.c | 2 +-
> include/linux/kvm_host.h | 10 +++++-----
> virt/kvm/arm/arch_timer.c | 2 +-
> virt/kvm/arm/arm.c | 9 +++++----
> virt/kvm/async_pf.c | 3 +--
> virt/kvm/kvm_main.c | 19 +++++++++----------
> 11 files changed, 35 insertions(+), 44 deletions(-)
>
> diff --git a/arch/mips/kvm/mips.c b/arch/mips/kvm/mips.c
> index 8f05dd0a0f4e..fad6acce46e4 100644
> --- a/arch/mips/kvm/mips.c
> +++ b/arch/mips/kvm/mips.c
> @@ -284,8 +284,7 @@ static enum hrtimer_restart kvm_mips_comparecount_wakeup(struct hrtimer *timer)
> kvm_mips_callbacks->queue_timer_int(vcpu);
>
> vcpu->arch.wait = 0;
> - if (swq_has_sleeper(&vcpu->wq))
> - swake_up_one(&vcpu->wq);
> + rcuwait_wake_up(&vcpu->wait);
>
> return kvm_mips_count_timeout(vcpu);
> }
> @@ -511,8 +510,7 @@ int kvm_vcpu_ioctl_interrupt(struct kvm_vcpu *vcpu,
>
> dvcpu->arch.wait = 0;
>
> - if (swq_has_sleeper(&dvcpu->wq))
> - swake_up_one(&dvcpu->wq);
> + rcuwait_wake_up(&dvcpu->wait);
>
> return 0;
> }
> diff --git a/arch/powerpc/include/asm/kvm_book3s.h b/arch/powerpc/include/asm/kvm_book3s.h
> index 506e4df2d730..6e5d85ba588d 100644
> --- a/arch/powerpc/include/asm/kvm_book3s.h
> +++ b/arch/powerpc/include/asm/kvm_book3s.h
> @@ -78,7 +78,7 @@ struct kvmppc_vcore {
> struct kvm_vcpu *runnable_threads[MAX_SMT_THREADS];
> struct list_head preempt_list;
> spinlock_t lock;
> - struct swait_queue_head wq;
> + struct rcuwait wait;
> spinlock_t stoltb_lock; /* protects stolen_tb and preempt_tb */
> u64 stolen_tb;
> u64 preempt_tb;
> diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h
> index 1dc63101ffe1..337047ba4a56 100644
> --- a/arch/powerpc/include/asm/kvm_host.h
> +++ b/arch/powerpc/include/asm/kvm_host.h
> @@ -751,7 +751,7 @@ struct kvm_vcpu_arch {
> u8 irq_pending; /* Used by XIVE to signal pending guest irqs */
> u32 last_inst;
>
> - struct swait_queue_head *wqp;
> + struct rcuwait *waitp;
> struct kvmppc_vcore *vcore;
> int ret;
> int trap;
> diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
> index 93493f0cbfe8..b8d42f523ca7 100644
> --- a/arch/powerpc/kvm/book3s_hv.c
> +++ b/arch/powerpc/kvm/book3s_hv.c
> @@ -230,13 +230,11 @@ static bool kvmppc_ipi_thread(int cpu)
> static void kvmppc_fast_vcpu_kick_hv(struct kvm_vcpu *vcpu)
> {
> int cpu;
> - struct swait_queue_head *wqp;
> + struct rcuwait *wait;
>
> - wqp = kvm_arch_vcpu_wq(vcpu);
> - if (swq_has_sleeper(wqp)) {
> - swake_up_one(wqp);
> + wait = kvm_arch_vcpu_get_wait(vcpu);
> + if (rcuwait_wake_up(wait))
> ++vcpu->stat.halt_wakeup;
> - }
>
> cpu = READ_ONCE(vcpu->arch.thread_cpu);
> if (cpu >= 0 && kvmppc_ipi_thread(cpu))
> @@ -2125,7 +2123,7 @@ static struct kvmppc_vcore *kvmppc_vcore_create(struct kvm *kvm, int id)
>
> spin_lock_init(&vcore->lock);
> spin_lock_init(&vcore->stoltb_lock);
> - init_swait_queue_head(&vcore->wq);
> + rcuwait_init(&vcore->wait);
> vcore->preempt_tb = TB_NIL;
> vcore->lpcr = kvm->arch.lpcr;
> vcore->first_vcpuid = id;
> @@ -3784,7 +3782,6 @@ static void kvmppc_vcore_blocked(struct kvmppc_vcore *vc)
> ktime_t cur, start_poll, start_wait;
> int do_sleep = 1;
> u64 block_ns;
> - DECLARE_SWAITQUEUE(wait);
>
> /* Poll for pending exceptions and ceded state */
> cur = start_poll = ktime_get();
> @@ -3812,10 +3809,7 @@ static void kvmppc_vcore_blocked(struct kvmppc_vcore *vc)
> }
> }
>
> - prepare_to_swait_exclusive(&vc->wq, &wait, TASK_INTERRUPTIBLE);
> -
> if (kvmppc_vcore_check_block(vc)) {
> - finish_swait(&vc->wq, &wait);
> do_sleep = 0;
> /* If we polled, count this as a successful poll */
> if (vc->halt_poll_ns)
> @@ -3828,8 +3822,8 @@ static void kvmppc_vcore_blocked(struct kvmppc_vcore *vc)
> vc->vcore_state = VCORE_SLEEPING;
> trace_kvmppc_vcore_blocked(vc, 0);
> spin_unlock(&vc->lock);
> - schedule();
> - finish_swait(&vc->wq, &wait);
> + rcuwait_wait_event(&vc->wait,
> + kvmppc_vcore_check_block(vc), TASK_INTERRUPTIBLE);
> spin_lock(&vc->lock);
> vc->vcore_state = VCORE_INACTIVE;
> trace_kvmppc_vcore_blocked(vc, 1);
> @@ -3940,7 +3934,7 @@ static int kvmppc_run_vcpu(struct kvm_run *kvm_run, struct kvm_vcpu *vcpu)
> kvmppc_start_thread(vcpu, vc);
> trace_kvm_guest_enter(vcpu);
> } else if (vc->vcore_state == VCORE_SLEEPING) {
> - swake_up_one(&vc->wq);
> + rcuwait_wake_up(&vc->wait);
> }
>
> }
> @@ -4279,7 +4273,7 @@ static int kvmppc_vcpu_run_hv(struct kvm_run *run, struct kvm_vcpu *vcpu)
> }
> user_vrsave = mfspr(SPRN_VRSAVE);
>
> - vcpu->arch.wqp = &vcpu->arch.vcore->wq;
> + vcpu->arch.waitp = &vcpu->arch.vcore->wait;
> vcpu->arch.pgdir = kvm->mm->pgd;
> vcpu->arch.state = KVMPPC_VCPU_BUSY_IN_HOST;
>
> diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c
> index e15166b0a16d..4a074b587520 100644
> --- a/arch/powerpc/kvm/powerpc.c
> +++ b/arch/powerpc/kvm/powerpc.c
> @@ -751,7 +751,7 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
> if (err)
> goto out_vcpu_uninit;
>
> - vcpu->arch.wqp = &vcpu->wq;
> + vcpu->arch.waitp = &vcpu->wait;
> kvmppc_create_vcpu_debugfs(vcpu, vcpu->vcpu_id);
> return 0;
>
> diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
> index 9af25c97612a..fb2f56ba171d 100644
> --- a/arch/x86/kvm/lapic.c
> +++ b/arch/x86/kvm/lapic.c
> @@ -1833,7 +1833,7 @@ void kvm_lapic_expired_hv_timer(struct kvm_vcpu *vcpu)
> /* If the preempt notifier has already run, it also called apic_timer_expired */
> if (!apic->lapic_timer.hv_timer_in_use)
> goto out;
> - WARN_ON(swait_active(&vcpu->wq));
> + WARN_ON(rcu_dereference(vcpu->wait.task));
> cancel_hv_timer(apic);
> apic_timer_expired(apic);
>
> diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
> index 6d58beb65454..fc34021546bd 100644
> --- a/include/linux/kvm_host.h
> +++ b/include/linux/kvm_host.h
> @@ -23,7 +23,7 @@
> #include <linux/irqflags.h>
> #include <linux/context_tracking.h>
> #include <linux/irqbypass.h>
> -#include <linux/swait.h>
> +#include <linux/rcuwait.h>
> #include <linux/refcount.h>
> #include <linux/nospec.h>
> #include <asm/signal.h>
> @@ -277,7 +277,7 @@ struct kvm_vcpu {
> struct mutex mutex;
> struct kvm_run *run;
>
> - struct swait_queue_head wq;
> + struct rcuwait wait;
> struct pid __rcu *pid;
> int sigset_active;
> sigset_t sigset;
> @@ -956,12 +956,12 @@ static inline bool kvm_arch_has_assigned_device(struct kvm *kvm)
> }
> #endif
>
> -static inline struct swait_queue_head *kvm_arch_vcpu_wq(struct kvm_vcpu *vcpu)
> +static inline struct rcuwait *kvm_arch_vcpu_get_wait(struct kvm_vcpu *vcpu)
> {
> #ifdef __KVM_HAVE_ARCH_WQP
> - return vcpu->arch.wqp;
> + return vcpu->arch.waitp;
> #else
> - return &vcpu->wq;
> + return &vcpu->wait;
> #endif
> }
>
> diff --git a/virt/kvm/arm/arch_timer.c b/virt/kvm/arm/arch_timer.c
> index 93bd59b46848..b2805105bbe5 100644
> --- a/virt/kvm/arm/arch_timer.c
> +++ b/virt/kvm/arm/arch_timer.c
> @@ -593,7 +593,7 @@ void kvm_timer_vcpu_put(struct kvm_vcpu *vcpu)
> if (map.emul_ptimer)
> soft_timer_cancel(&map.emul_ptimer->hrtimer);
>
> - if (swait_active(kvm_arch_vcpu_wq(vcpu)))
> + if (rcu_dereference(kvm_arch_vpu_get_wait(vcpu)) != NULL)
> kvm_timer_blocking(vcpu);
>
> /*
> diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c
> index 48d0ec44ad77..f94a10bb1251 100644
> --- a/virt/kvm/arm/arm.c
> +++ b/virt/kvm/arm/arm.c
> @@ -579,16 +579,17 @@ void kvm_arm_resume_guest(struct kvm *kvm)
>
> kvm_for_each_vcpu(i, vcpu, kvm) {
> vcpu->arch.pause = false;
> - swake_up_one(kvm_arch_vcpu_wq(vcpu));
> + rcuwait_wake_up(kvm_arch_vcpu_get_wait(vcpu));
> }
> }
>
> static void vcpu_req_sleep(struct kvm_vcpu *vcpu)
> {
> - struct swait_queue_head *wq = kvm_arch_vcpu_wq(vcpu);
> + struct rcuwait *wait = kvm_arch_vcpu_get_wait(vcpu);
>
> - swait_event_interruptible_exclusive(*wq, ((!vcpu->arch.power_off) &&
> - (!vcpu->arch.pause)));
> + rcuwait_wait_event(*wait,
> + (!vcpu->arch.power_off) &&(!vcpu->arch.pause),
> + TASK_INTERRUPTIBLE);
>
> if (vcpu->arch.power_off || vcpu->arch.pause) {
> /* Awaken to handle a signal, request we sleep again later. */
> diff --git a/virt/kvm/async_pf.c b/virt/kvm/async_pf.c
> index 15e5b037f92d..10b533f641a6 100644
> --- a/virt/kvm/async_pf.c
> +++ b/virt/kvm/async_pf.c
> @@ -80,8 +80,7 @@ static void async_pf_execute(struct work_struct *work)
>
> trace_kvm_async_pf_completed(addr, cr2_or_gpa);
>
> - if (swq_has_sleeper(&vcpu->wq))
> - swake_up_one(&vcpu->wq);
> + rcuwait_wake_up(&vcpu->wait);
>
> mmput(mm);
> kvm_put_kvm(vcpu->kvm);
> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> index 74bdb7bf3295..835fb109badf 100644
> --- a/virt/kvm/kvm_main.c
> +++ b/virt/kvm/kvm_main.c
> @@ -341,7 +341,7 @@ static void kvm_vcpu_init(struct kvm_vcpu *vcpu, struct kvm *kvm, unsigned id)
> vcpu->kvm = kvm;
> vcpu->vcpu_id = id;
> vcpu->pid = NULL;
> - init_swait_queue_head(&vcpu->wq);
> + rcuwait_init(&vcpu->wait);
> kvm_async_pf_vcpu_init(vcpu);
>
> vcpu->pre_pcpu = -1;
> @@ -2671,7 +2671,6 @@ static int kvm_vcpu_check_block(struct kvm_vcpu *vcpu)
> void kvm_vcpu_block(struct kvm_vcpu *vcpu)
> {
> ktime_t start, cur;
> - DECLARE_SWAITQUEUE(wait);
> bool waited = false;
> u64 block_ns;
>
> @@ -2697,8 +2696,9 @@ void kvm_vcpu_block(struct kvm_vcpu *vcpu)
> } while (single_task_running() && ktime_before(cur, stop));
> }
>
> + prepare_to_rcuwait(&vcpu->wait);
> for (;;) {
> - prepare_to_swait_exclusive(&vcpu->wq, &wait, TASK_INTERRUPTIBLE);
> + set_current_state(TASK_INTERRUPTIBLE);
>
> if (kvm_vcpu_check_block(vcpu) < 0)
> break;
> @@ -2706,8 +2706,7 @@ void kvm_vcpu_block(struct kvm_vcpu *vcpu)
> waited = true;
> schedule();
> }
> -
> - finish_swait(&vcpu->wq, &wait);
> + finish_rcuwait(&vcpu->wait);
> cur = ktime_get();
> out:
> kvm_arch_vcpu_unblocking(vcpu);
> @@ -2738,11 +2737,10 @@ EXPORT_SYMBOL_GPL(kvm_vcpu_block);
>
> bool kvm_vcpu_wake_up(struct kvm_vcpu *vcpu)
> {
> - struct swait_queue_head *wqp;
> + struct rcuwait *wait;
>
> - wqp = kvm_arch_vcpu_wq(vcpu);
> - if (swq_has_sleeper(wqp)) {
> - swake_up_one(wqp);
> + wait = kvm_arch_vcpu_get_wait(vcpu);
> + if (rcuwait_wake_up(wait)) {
> WRITE_ONCE(vcpu->ready, true);
> ++vcpu->stat.halt_wakeup;
> return true;
> @@ -2884,7 +2882,8 @@ void kvm_vcpu_on_spin(struct kvm_vcpu *me, bool yield_to_kernel_mode)
> continue;
> if (vcpu == me)
> continue;
> - if (swait_active(&vcpu->wq) && !vcpu_dy_runnable(vcpu))
> + if (rcu_dereference(vcpu->wait.task) &&
> + !vcpu_dy_runnable(vcpu))
> continue;
> if (READ_ONCE(vcpu->preempted) && yield_to_kernel_mode &&
> !kvm_arch_vcpu_in_kernel(vcpu))
>
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: [PATCH 4/5] kvm: Replace vcpu->swait with rcuwait
2020-04-22 4:07 ` [PATCH 4/5] kvm: Replace vcpu->swait with rcuwait Davidlohr Bueso
2020-04-22 8:32 ` Paolo Bonzini
@ 2020-04-22 10:14 ` Marc Zyngier
2020-04-23 8:41 ` Marc Zyngier
2 siblings, 0 replies; 16+ messages in thread
From: Marc Zyngier @ 2020-04-22 10:14 UTC (permalink / raw)
To: Davidlohr Bueso
Cc: tglx, pbonzini, kvm, Davidlohr Bueso, peterz, torvalds, bigeasy,
linux-kernel, rostedt, linux-mips, Paul Mackerras, joel, will,
kvmarm
On 2020-04-22 05:07, Davidlohr Bueso wrote:
> The use of any sort of waitqueue (simple or regular) for
> wait/waking vcpus has always been an overkill and semantically
> wrong. Because this is per-vcpu (which is blocked) there is
> only ever a single waiting vcpu, thus no need for any sort of
> queue.
>
> As such, make use of the rcuwait primitive, with the following
> considerations:
>
> - rcuwait already provides the proper barriers that serialize
> concurrent waiter and waker.
>
> - Task wakeup is done in rcu read critical region, with a
> stable task pointer.
>
> - Because there is no concurrency among waiters, we need
> not worry about rcuwait_wait_event() calls corrupting
> the wait->task. As a consequence, this saves the locking
> done in swait when modifying the queue. This also applies
> to per-vcore wait for powerpc kvm-hv.
>
> The x86 tscdeadline_latency test mentioned in 8577370fb0cb
> ("KVM: Use simple waitqueue for vcpu->wq") shows that, on avg,
> latency is reduced by around 15-20% with this change.
>
> Cc: Paul Mackerras <paulus@ozlabs.org>
> Cc: kvmarm@lists.cs.columbia.edu
> Cc: linux-mips@vger.kernel.org
> Signed-off-by: Davidlohr Bueso <dbueso@suse.de>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Thanks,
M.
--
Who you jivin' with that Cosmik Debris?
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: [PATCH 4/5] kvm: Replace vcpu->swait with rcuwait
2020-04-22 4:07 ` [PATCH 4/5] kvm: Replace vcpu->swait with rcuwait Davidlohr Bueso
2020-04-22 8:32 ` Paolo Bonzini
2020-04-22 10:14 ` Marc Zyngier
@ 2020-04-23 8:41 ` Marc Zyngier
2020-04-23 8:57 ` Paolo Bonzini
2 siblings, 1 reply; 16+ messages in thread
From: Marc Zyngier @ 2020-04-23 8:41 UTC (permalink / raw)
To: Davidlohr Bueso
Cc: tglx, pbonzini, kvm, Davidlohr Bueso, peterz, torvalds, bigeasy,
linux-kernel, rostedt, linux-mips, Paul Mackerras, joel, will,
kvmarm
On Tue, 21 Apr 2020 21:07:38 -0700
Davidlohr Bueso <dave@stgolabs.net> wrote:
> The use of any sort of waitqueue (simple or regular) for
> wait/waking vcpus has always been an overkill and semantically
> wrong. Because this is per-vcpu (which is blocked) there is
> only ever a single waiting vcpu, thus no need for any sort of
> queue.
>
> As such, make use of the rcuwait primitive, with the following
> considerations:
>
> - rcuwait already provides the proper barriers that serialize
> concurrent waiter and waker.
>
> - Task wakeup is done in rcu read critical region, with a
> stable task pointer.
>
> - Because there is no concurrency among waiters, we need
> not worry about rcuwait_wait_event() calls corrupting
> the wait->task. As a consequence, this saves the locking
> done in swait when modifying the queue. This also applies
> to per-vcore wait for powerpc kvm-hv.
>
> The x86 tscdeadline_latency test mentioned in 8577370fb0cb
> ("KVM: Use simple waitqueue for vcpu->wq") shows that, on avg,
> latency is reduced by around 15-20% with this change.
>
> Cc: Paul Mackerras <paulus@ozlabs.org>
> Cc: kvmarm@lists.cs.columbia.edu
> Cc: linux-mips@vger.kernel.org
> Signed-off-by: Davidlohr Bueso <dbueso@suse.de>
> ---
> arch/mips/kvm/mips.c | 6 ++----
> arch/powerpc/include/asm/kvm_book3s.h | 2 +-
> arch/powerpc/include/asm/kvm_host.h | 2 +-
> arch/powerpc/kvm/book3s_hv.c | 22 ++++++++--------------
> arch/powerpc/kvm/powerpc.c | 2 +-
> arch/x86/kvm/lapic.c | 2 +-
> include/linux/kvm_host.h | 10 +++++-----
> virt/kvm/arm/arch_timer.c | 2 +-
> virt/kvm/arm/arm.c | 9 +++++----
> virt/kvm/async_pf.c | 3 +--
> virt/kvm/kvm_main.c | 19 +++++++++----------
> 11 files changed, 35 insertions(+), 44 deletions(-)
[...]
I should have tested it *before* acking it, really.
> diff --git a/virt/kvm/arm/arch_timer.c b/virt/kvm/arm/arch_timer.c
> index 93bd59b46848..b2805105bbe5 100644
> --- a/virt/kvm/arm/arch_timer.c
> +++ b/virt/kvm/arm/arch_timer.c
> @@ -593,7 +593,7 @@ void kvm_timer_vcpu_put(struct kvm_vcpu *vcpu)
> if (map.emul_ptimer)
> soft_timer_cancel(&map.emul_ptimer->hrtimer);
>
> - if (swait_active(kvm_arch_vcpu_wq(vcpu)))
> + if (rcu_dereference(kvm_arch_vpu_get_wait(vcpu)) != NULL)
This doesn't compile (wrong function name, and rcu_dereference takes a
variable). But whatever it would do if we fixed it looks dodgy. it isn't
the rcuwait structure that you want to dereference, but rcuwait->task
(we are checking whether we are called because we are blocking or being
preempted).
> kvm_timer_blocking(vcpu);
>
> /*
> diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c
> index 48d0ec44ad77..f94a10bb1251 100644
> --- a/virt/kvm/arm/arm.c
> +++ b/virt/kvm/arm/arm.c
> @@ -579,16 +579,17 @@ void kvm_arm_resume_guest(struct kvm *kvm)
>
> kvm_for_each_vcpu(i, vcpu, kvm) {
> vcpu->arch.pause = false;
> - swake_up_one(kvm_arch_vcpu_wq(vcpu));
> + rcuwait_wake_up(kvm_arch_vcpu_get_wait(vcpu));
> }
> }
>
> static void vcpu_req_sleep(struct kvm_vcpu *vcpu)
> {
> - struct swait_queue_head *wq = kvm_arch_vcpu_wq(vcpu);
> + struct rcuwait *wait = kvm_arch_vcpu_get_wait(vcpu);
>
> - swait_event_interruptible_exclusive(*wq, ((!vcpu->arch.power_off) &&
> - (!vcpu->arch.pause)));
> + rcuwait_wait_event(*wait,
> + (!vcpu->arch.power_off) &&(!vcpu->arch.pause),
> + TASK_INTERRUPTIBLE);
As noticed by the kbuild robot, this doesn't compile either.
I fixed it as follow, and it survived a very basic test run in a model
(more testing later).
Thanks,
M.
diff --git a/virt/kvm/arm/arch_timer.c b/virt/kvm/arm/arch_timer.c
index b2805105bbe56..2dbd14dcae9fb 100644
--- a/virt/kvm/arm/arch_timer.c
+++ b/virt/kvm/arm/arch_timer.c
@@ -569,6 +569,7 @@ bool kvm_timer_should_notify_user(struct kvm_vcpu *vcpu)
void kvm_timer_vcpu_put(struct kvm_vcpu *vcpu)
{
+ struct rcuwait *wait = kvm_arch_vcpu_get_wait(vcpu);
struct arch_timer_cpu *timer = vcpu_timer(vcpu);
struct timer_map map;
@@ -593,7 +594,7 @@ void kvm_timer_vcpu_put(struct kvm_vcpu *vcpu)
if (map.emul_ptimer)
soft_timer_cancel(&map.emul_ptimer->hrtimer);
- if (rcu_dereference(kvm_arch_vpu_get_wait(vcpu)) != NULL)
+ if (rcu_dereference(wait->task))
kvm_timer_blocking(vcpu);
/*
diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c
index f94a10bb1251b..479f36d02418d 100644
--- a/virt/kvm/arm/arm.c
+++ b/virt/kvm/arm/arm.c
@@ -587,7 +587,7 @@ static void vcpu_req_sleep(struct kvm_vcpu *vcpu)
{
struct rcuwait *wait = kvm_arch_vcpu_get_wait(vcpu);
- rcuwait_wait_event(*wait,
+ rcuwait_wait_event(wait,
(!vcpu->arch.power_off) &&(!vcpu->arch.pause),
TASK_INTERRUPTIBLE);
--
Jazz is not dead. It just smells funny...
^ permalink raw reply related [flat|nested] 16+ messages in thread* Re: [PATCH 4/5] kvm: Replace vcpu->swait with rcuwait
2020-04-23 8:41 ` Marc Zyngier
@ 2020-04-23 8:57 ` Paolo Bonzini
2020-04-23 9:19 ` Peter Zijlstra
0 siblings, 1 reply; 16+ messages in thread
From: Paolo Bonzini @ 2020-04-23 8:57 UTC (permalink / raw)
To: Marc Zyngier, Davidlohr Bueso
Cc: tglx, kvm, Davidlohr Bueso, peterz, torvalds, bigeasy,
linux-kernel, rostedt, linux-mips, Paul Mackerras, joel, will,
kvmarm
On 23/04/20 10:41, Marc Zyngier wrote:
>>
>> - if (swait_active(kvm_arch_vcpu_wq(vcpu)))
>> + if (rcu_dereference(kvm_arch_vpu_get_wait(vcpu)) != NULL)
> This doesn't compile (wrong function name, and rcu_dereference takes a
> variable). But whatever it would do if we fixed it looks dodgy. it isn't
> the rcuwait structure that you want to dereference, but rcuwait->task
> (we are checking whether we are called because we are blocking or being
> preempted).
>
Yes, I agree. Replacing swait with rcuwait is all good, but please make
the API look the same first. Just like you added prepare_to_rcuwait and
finish_rcuwait, let's add rcuwait_active as well.
Actually let's do it like this:
1) Davidlohr, please post only patches 1-3 to "equalize" the swait and
rcuwait APIs.
2) Peter, please prepare a topic branch for those, or provide Acked-by
3) let's get everything else through the KVM tree.
Thanks,
Paolo
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 4/5] kvm: Replace vcpu->swait with rcuwait
2020-04-23 8:57 ` Paolo Bonzini
@ 2020-04-23 9:19 ` Peter Zijlstra
2020-04-23 9:26 ` Paolo Bonzini
0 siblings, 1 reply; 16+ messages in thread
From: Peter Zijlstra @ 2020-04-23 9:19 UTC (permalink / raw)
To: Paolo Bonzini
Cc: Marc Zyngier, Davidlohr Bueso, tglx, kvm, Davidlohr Bueso,
torvalds, bigeasy, linux-kernel, rostedt, linux-mips,
Paul Mackerras, joel, will, kvmarm
On Thu, Apr 23, 2020 at 10:57:57AM +0200, Paolo Bonzini wrote:
> On 23/04/20 10:41, Marc Zyngier wrote:
> >>
> >> - if (swait_active(kvm_arch_vcpu_wq(vcpu)))
> >> + if (rcu_dereference(kvm_arch_vpu_get_wait(vcpu)) != NULL)
> > This doesn't compile (wrong function name, and rcu_dereference takes a
> > variable). But whatever it would do if we fixed it looks dodgy. it isn't
> > the rcuwait structure that you want to dereference, but rcuwait->task
> > (we are checking whether we are called because we are blocking or being
> > preempted).
> >
>
> Yes, I agree. Replacing swait with rcuwait is all good, but please make
> the API look the same first. Just like you added prepare_to_rcuwait and
> finish_rcuwait, let's add rcuwait_active as well.
>
> Actually let's do it like this:
>
> 1) Davidlohr, please post only patches 1-3 to "equalize" the swait and
> rcuwait APIs.
>
> 2) Peter, please prepare a topic branch for those, or provide Acked-by
I don't think I have anything that conflicts with this, so sure, take
the whole thing through KVM.
For 1-3 (and I'll send a small niggle for 3 right after this):
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
I'll keep 5 as it is unrelated.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 4/5] kvm: Replace vcpu->swait with rcuwait
2020-04-23 9:19 ` Peter Zijlstra
@ 2020-04-23 9:26 ` Paolo Bonzini
0 siblings, 0 replies; 16+ messages in thread
From: Paolo Bonzini @ 2020-04-23 9:26 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Marc Zyngier, Davidlohr Bueso, tglx, kvm, Davidlohr Bueso,
torvalds, bigeasy, linux-kernel, rostedt, linux-mips,
Paul Mackerras, joel, will, kvmarm
On 23/04/20 11:19, Peter Zijlstra wrote:
>>
>> 1) Davidlohr, please post only patches 1-3 to "equalize" the swait and
>> rcuwait APIs.
>>
>> 2) Peter, please prepare a topic branch for those, or provide Acked-by
> I don't think I have anything that conflicts with this, so sure, take
> the whole thing through KVM.
>
> For 1-3 (and I'll send a small niggle for 3 right after this):
>
> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Great thanks. I assume you have no issue with rcuwait_active either.
Paolo
> I'll keep 5 as it is unrelated.
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH 5/5] sched/swait: Reword some of the main description
2020-04-22 4:07 [PATCH v3 -tip 0/5] kvm: Use rcuwait for vcpu blocking Davidlohr Bueso
` (3 preceding siblings ...)
2020-04-22 4:07 ` [PATCH 4/5] kvm: Replace vcpu->swait with rcuwait Davidlohr Bueso
@ 2020-04-22 4:07 ` Davidlohr Bueso
2020-05-01 18:22 ` [tip: sched/core] " tip-bot2 for Davidlohr Bueso
2020-04-22 11:33 ` [PATCH v3 -tip 0/5] kvm: Use rcuwait for vcpu blocking Peter Zijlstra
5 siblings, 1 reply; 16+ messages in thread
From: Davidlohr Bueso @ 2020-04-22 4:07 UTC (permalink / raw)
To: tglx, pbonzini
Cc: bigeasy, peterz, rostedt, torvalds, will, joel, linux-kernel,
kvm, dave, Davidlohr Bueso
With both the increased use of swait and kvm no longer using
it, we can reword some of the comments. While removing Linus'
valid rant, I've also cared to explicitly mention that swait
is very different than regular wait. In addition it is
mentioned against using swait in favor of the regular flavor.
Signed-off-by: Davidlohr Bueso <dbueso@suse.de>
---
include/linux/swait.h | 23 +++++------------------
1 file changed, 5 insertions(+), 18 deletions(-)
diff --git a/include/linux/swait.h b/include/linux/swait.h
index 73e06e9986d4..6a8c22b8c2a5 100644
--- a/include/linux/swait.h
+++ b/include/linux/swait.h
@@ -9,23 +9,10 @@
#include <asm/current.h>
/*
- * BROKEN wait-queues.
- *
- * These "simple" wait-queues are broken garbage, and should never be
- * used. The comments below claim that they are "similar" to regular
- * wait-queues, but the semantics are actually completely different, and
- * every single user we have ever had has been buggy (or pointless).
- *
- * A "swake_up_one()" only wakes up _one_ waiter, which is not at all what
- * "wake_up()" does, and has led to problems. In other cases, it has
- * been fine, because there's only ever one waiter (kvm), but in that
- * case gthe whole "simple" wait-queue is just pointless to begin with,
- * since there is no "queue". Use "wake_up_process()" with a direct
- * pointer instead.
- *
- * While these are very similar to regular wait queues (wait.h) the most
- * important difference is that the simple waitqueue allows for deterministic
- * behaviour -- IOW it has strictly bounded IRQ and lock hold times.
+ * Simple waitqueues are semantically very different to regular wait queues
+ * (wait.h). The most important difference is that the simple waitqueue allows
+ * for deterministic behaviour -- IOW it has strictly bounded IRQ and lock hold
+ * times.
*
* Mainly, this is accomplished by two things. Firstly not allowing swake_up_all
* from IRQ disabled, and dropping the lock upon every wakeup, giving a higher
@@ -39,7 +26,7 @@
* sleeper state.
*
* - the !exclusive mode; because that leads to O(n) wakeups, everything is
- * exclusive.
+ * exclusive. As such swake_up_one will only ever awake _one_ waiter.
*
* - custom wake callback functions; because you cannot give any guarantees
* about random code. This also allows swait to be used in RT, such that
--
2.16.4
^ permalink raw reply related [flat|nested] 16+ messages in thread* [tip: sched/core] sched/swait: Reword some of the main description
2020-04-22 4:07 ` [PATCH 5/5] sched/swait: Reword some of the main description Davidlohr Bueso
@ 2020-05-01 18:22 ` tip-bot2 for Davidlohr Bueso
0 siblings, 0 replies; 16+ messages in thread
From: tip-bot2 for Davidlohr Bueso @ 2020-05-01 18:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Davidlohr Bueso, Peter Zijlstra (Intel), x86, LKML
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 12ac6782a40ad7636b6ef45680741825b64ab221
Gitweb: https://git.kernel.org/tip/12ac6782a40ad7636b6ef45680741825b64ab221
Author: Davidlohr Bueso <dave@stgolabs.net>
AuthorDate: Tue, 21 Apr 2020 21:07:39 -07:00
Committer: Peter Zijlstra <peterz@infradead.org>
CommitterDate: Thu, 30 Apr 2020 20:14:41 +02:00
sched/swait: Reword some of the main description
With both the increased use of swait and kvm no longer using
it, we can reword some of the comments. While removing Linus'
valid rant, I've also cared to explicitly mention that swait
is very different than regular wait. In addition it is
mentioned against using swait in favor of the regular flavor.
Signed-off-by: Davidlohr Bueso <dbueso@suse.de>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://lkml.kernel.org/r/20200422040739.18601-6-dave@stgolabs.net
---
include/linux/swait.h | 23 +++++------------------
1 file changed, 5 insertions(+), 18 deletions(-)
diff --git a/include/linux/swait.h b/include/linux/swait.h
index 73e06e9..6a8c22b 100644
--- a/include/linux/swait.h
+++ b/include/linux/swait.h
@@ -9,23 +9,10 @@
#include <asm/current.h>
/*
- * BROKEN wait-queues.
- *
- * These "simple" wait-queues are broken garbage, and should never be
- * used. The comments below claim that they are "similar" to regular
- * wait-queues, but the semantics are actually completely different, and
- * every single user we have ever had has been buggy (or pointless).
- *
- * A "swake_up_one()" only wakes up _one_ waiter, which is not at all what
- * "wake_up()" does, and has led to problems. In other cases, it has
- * been fine, because there's only ever one waiter (kvm), but in that
- * case gthe whole "simple" wait-queue is just pointless to begin with,
- * since there is no "queue". Use "wake_up_process()" with a direct
- * pointer instead.
- *
- * While these are very similar to regular wait queues (wait.h) the most
- * important difference is that the simple waitqueue allows for deterministic
- * behaviour -- IOW it has strictly bounded IRQ and lock hold times.
+ * Simple waitqueues are semantically very different to regular wait queues
+ * (wait.h). The most important difference is that the simple waitqueue allows
+ * for deterministic behaviour -- IOW it has strictly bounded IRQ and lock hold
+ * times.
*
* Mainly, this is accomplished by two things. Firstly not allowing swake_up_all
* from IRQ disabled, and dropping the lock upon every wakeup, giving a higher
@@ -39,7 +26,7 @@
* sleeper state.
*
* - the !exclusive mode; because that leads to O(n) wakeups, everything is
- * exclusive.
+ * exclusive. As such swake_up_one will only ever awake _one_ waiter.
*
* - custom wake callback functions; because you cannot give any guarantees
* about random code. This also allows swait to be used in RT, such that
^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH v3 -tip 0/5] kvm: Use rcuwait for vcpu blocking
2020-04-22 4:07 [PATCH v3 -tip 0/5] kvm: Use rcuwait for vcpu blocking Davidlohr Bueso
` (4 preceding siblings ...)
2020-04-22 4:07 ` [PATCH 5/5] sched/swait: Reword some of the main description Davidlohr Bueso
@ 2020-04-22 11:33 ` Peter Zijlstra
5 siblings, 0 replies; 16+ messages in thread
From: Peter Zijlstra @ 2020-04-22 11:33 UTC (permalink / raw)
To: Davidlohr Bueso
Cc: tglx, pbonzini, bigeasy, rostedt, torvalds, will, joel,
linux-kernel, kvm
On Tue, Apr 21, 2020 at 09:07:34PM -0700, Davidlohr Bueso wrote:
> Davidlohr Bueso (5):
> rcuwait: Fix stale wake call name in comment
> rcuwait: Let rcuwait_wake_up() return whether or not a task was awoken
> rcuwait: Introduce prepare_to and finish_rcuwait
> kvm: Replace vcpu->swait with rcuwait
> sched/swait: Reword some of the main description
Thanks Dave!
^ permalink raw reply [flat|nested] 16+ messages in thread