* [PATCH v4] powerpc: kvm: fix rare but potential deadlock scene
@ 2013-11-15 8:35 Liu Ping Fan
2013-11-15 8:35 ` [PATCH v2] powerpc: kvm: optimize "sc 1" as fast return Liu Ping Fan
2013-11-16 6:55 ` [PATCH v4] powerpc: kvm: fix rare but potential deadlock scene Paul Mackerras
0 siblings, 2 replies; 7+ messages in thread
From: Liu Ping Fan @ 2013-11-15 8:35 UTC (permalink / raw)
To: linuxppc-dev, kvm-ppc; +Cc: Paul Mackerras, Alexander Graf
Since kvmppc_hv_find_lock_hpte() is called from both virtmode and
realmode, so it can trigger the deadlock.
Suppose the following scene:
Two physical cpuM, cpuN, two VM instances A, B, each VM has a group of
vcpus.
If on cpuM, vcpu_A_1 holds bitlock X (HPTE_V_HVLOCK), then is switched
out, and on cpuN, vcpu_A_2 try to lock X in realmode, then cpuN will be
caught in realmode for a long time.
What makes things even worse if the following happens,
On cpuM, bitlockX is hold, on cpuN, Y is hold.
vcpu_B_2 try to lock Y on cpuM in realmode
vcpu_A_2 try to lock X on cpuN in realmode
Oops! deadlock happens
Signed-off-by: Liu Ping Fan <pingfank@linux.vnet.ibm.com>
---
v4: remove the over-engineered part and keep it simple, also add some notes.
---
arch/powerpc/kvm/book3s_64_mmu_hv.c | 6 +++++-
arch/powerpc/kvm/book3s_hv_rm_mmu.c | 4 ++++
2 files changed, 9 insertions(+), 1 deletion(-)
diff --git a/arch/powerpc/kvm/book3s_64_mmu_hv.c b/arch/powerpc/kvm/book3s_64_mmu_hv.c
index 842f081..abf81fe 100644
--- a/arch/powerpc/kvm/book3s_64_mmu_hv.c
+++ b/arch/powerpc/kvm/book3s_64_mmu_hv.c
@@ -473,11 +473,14 @@ static int kvmppc_mmu_book3s_64_hv_xlate(struct kvm_vcpu *vcpu, gva_t eaddr,
slb_v = vcpu->kvm->arch.vrma_slb_v;
}
+ preempt_disable();
/* Find the HPTE in the hash table */
index = kvmppc_hv_find_lock_hpte(kvm, eaddr, slb_v,
HPTE_V_VALID | HPTE_V_ABSENT);
- if (index < 0)
+ if (index < 0) {
+ preempt_enable();
return -ENOENT;
+ }
hptep = (unsigned long *)(kvm->arch.hpt_virt + (index << 4));
v = hptep[0] & ~HPTE_V_HVLOCK;
gr = kvm->arch.revmap[index].guest_rpte;
@@ -485,6 +488,7 @@ static int kvmppc_mmu_book3s_64_hv_xlate(struct kvm_vcpu *vcpu, gva_t eaddr,
/* Unlock the HPTE */
asm volatile("lwsync" : : : "memory");
hptep[0] = v;
+ preempt_enable();
gpte->eaddr = eaddr;
gpte->vpage = ((v & HPTE_V_AVPN) << 4) | ((eaddr >> 12) & 0xfff);
diff --git a/arch/powerpc/kvm/book3s_hv_rm_mmu.c b/arch/powerpc/kvm/book3s_hv_rm_mmu.c
index 9c51544..ea17b30 100644
--- a/arch/powerpc/kvm/book3s_hv_rm_mmu.c
+++ b/arch/powerpc/kvm/book3s_hv_rm_mmu.c
@@ -749,6 +749,10 @@ static int slb_base_page_shift[4] = {
20, /* 1M, unsupported */
};
+/* When called from virtmode, this func should be protected by
+ * preempt_disable(), otherwise, the holding of HPTE_V_HVLOCK
+ * can trigger deadlock issue.
+ */
long kvmppc_hv_find_lock_hpte(struct kvm *kvm, gva_t eaddr, unsigned long slb_v,
unsigned long valid)
{
--
1.8.1.4
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH v2] powerpc: kvm: optimize "sc 1" as fast return
2013-11-15 8:35 [PATCH v4] powerpc: kvm: fix rare but potential deadlock scene Liu Ping Fan
@ 2013-11-15 8:35 ` Liu Ping Fan
2013-11-16 7:00 ` Paul Mackerras
2013-11-16 6:55 ` [PATCH v4] powerpc: kvm: fix rare but potential deadlock scene Paul Mackerras
1 sibling, 1 reply; 7+ messages in thread
From: Liu Ping Fan @ 2013-11-15 8:35 UTC (permalink / raw)
To: linuxppc-dev, kvm-ppc; +Cc: Paul Mackerras, Alexander Graf
In some scene, e.g openstack CI, PR guest can trigger "sc 1" frequently,
this patch optimizes the path by directly delivering BOOK3S_INTERRUPT_SYSCALL
to HV guest, so powernv can return to HV guest without heavy exit, i.e,
no need to swap TLB, HTAB,.. etc
Signed-off-by: Liu Ping Fan <pingfank@linux.vnet.ibm.com>
---
arch/powerpc/kvm/book3s_hv.c | 6 ------
arch/powerpc/kvm/book3s_hv_rmhandlers.S | 11 ++++++++++-
2 files changed, 10 insertions(+), 7 deletions(-)
diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
index 62a2b5a..73dc852 100644
--- a/arch/powerpc/kvm/book3s_hv.c
+++ b/arch/powerpc/kvm/book3s_hv.c
@@ -628,12 +628,6 @@ static int kvmppc_handle_exit(struct kvm_run *run, struct kvm_vcpu *vcpu,
/* hcall - punt to userspace */
int i;
- if (vcpu->arch.shregs.msr & MSR_PR) {
- /* sc 1 from userspace - reflect to guest syscall */
- kvmppc_book3s_queue_irqprio(vcpu, BOOK3S_INTERRUPT_SYSCALL);
- r = RESUME_GUEST;
- break;
- }
run->papr_hcall.nr = kvmppc_get_gpr(vcpu, 3);
for (i = 0; i < 9; ++i)
run->papr_hcall.args[i] = kvmppc_get_gpr(vcpu, 4 + i);
diff --git a/arch/powerpc/kvm/book3s_hv_rmhandlers.S b/arch/powerpc/kvm/book3s_hv_rmhandlers.S
index c71103b..a463f08 100644
--- a/arch/powerpc/kvm/book3s_hv_rmhandlers.S
+++ b/arch/powerpc/kvm/book3s_hv_rmhandlers.S
@@ -1388,7 +1388,8 @@ kvmppc_hisi:
hcall_try_real_mode:
ld r3,VCPU_GPR(R3)(r9)
andi. r0,r11,MSR_PR
- bne guest_exit_cont
+ /* sc 1 from userspace - reflect to guest syscall */
+ bne sc_1_fast_return
clrrdi r3,r3,2
cmpldi r3,hcall_real_table_end - hcall_real_table
bge guest_exit_cont
@@ -1409,6 +1410,14 @@ hcall_try_real_mode:
ld r11,VCPU_MSR(r4)
b fast_guest_return
+sc_1_fast_return:
+ mtspr SPRN_SRR0,r10
+ mtspr SPRN_SRR1,r11
+ li r10, BOOK3S_INTERRUPT_SYSCALL
+ li r11, (MSR_ME << 1) | 1 /* synthesize MSR_SF | MSR_ME */
+ rotldi r11, r11, 63
+ b fast_guest_return
+
/* We've attempted a real mode hcall, but it's punted it back
* to userspace. We need to restore some clobbered volatiles
* before resuming the pass-it-to-qemu path */
--
1.8.1.4
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH v4] powerpc: kvm: fix rare but potential deadlock scene
2013-11-15 8:35 [PATCH v4] powerpc: kvm: fix rare but potential deadlock scene Liu Ping Fan
2013-11-15 8:35 ` [PATCH v2] powerpc: kvm: optimize "sc 1" as fast return Liu Ping Fan
@ 2013-11-16 6:55 ` Paul Mackerras
2013-11-18 21:32 ` Alexander Graf
1 sibling, 1 reply; 7+ messages in thread
From: Paul Mackerras @ 2013-11-16 6:55 UTC (permalink / raw)
To: Liu Ping Fan; +Cc: linuxppc-dev, Alexander Graf, kvm-ppc
On Fri, Nov 15, 2013 at 04:35:00PM +0800, Liu Ping Fan wrote:
> Since kvmppc_hv_find_lock_hpte() is called from both virtmode and
> realmode, so it can trigger the deadlock.
>
> Suppose the following scene:
>
> Two physical cpuM, cpuN, two VM instances A, B, each VM has a group of
> vcpus.
>
> If on cpuM, vcpu_A_1 holds bitlock X (HPTE_V_HVLOCK), then is switched
> out, and on cpuN, vcpu_A_2 try to lock X in realmode, then cpuN will be
> caught in realmode for a long time.
>
> What makes things even worse if the following happens,
> On cpuM, bitlockX is hold, on cpuN, Y is hold.
> vcpu_B_2 try to lock Y on cpuM in realmode
> vcpu_A_2 try to lock X on cpuN in realmode
>
> Oops! deadlock happens
>
> Signed-off-by: Liu Ping Fan <pingfank@linux.vnet.ibm.com>
Reviewed-by: Paul Mackerras <paulus@samba.org>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v2] powerpc: kvm: optimize "sc 1" as fast return
2013-11-15 8:35 ` [PATCH v2] powerpc: kvm: optimize "sc 1" as fast return Liu Ping Fan
@ 2013-11-16 7:00 ` Paul Mackerras
2013-11-18 1:06 ` liu ping fan
0 siblings, 1 reply; 7+ messages in thread
From: Paul Mackerras @ 2013-11-16 7:00 UTC (permalink / raw)
To: Liu Ping Fan; +Cc: linuxppc-dev, Alexander Graf, kvm-ppc
On Fri, Nov 15, 2013 at 04:35:01PM +0800, Liu Ping Fan wrote:
>
> +sc_1_fast_return:
> + mtspr SPRN_SRR0,r10
> + mtspr SPRN_SRR1,r11
> + li r10, BOOK3S_INTERRUPT_SYSCALL
> + li r11, (MSR_ME << 1) | 1 /* synthesize MSR_SF | MSR_ME */
> + rotldi r11, r11, 63
You need a "mr r4, r9" instruction here, because fast_guest_return
needs the vcpu pointer in r4. Apart from that this looks fine.
> + b fast_guest_return
Paul.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v2] powerpc: kvm: optimize "sc 1" as fast return
2013-11-16 7:00 ` Paul Mackerras
@ 2013-11-18 1:06 ` liu ping fan
0 siblings, 0 replies; 7+ messages in thread
From: liu ping fan @ 2013-11-18 1:06 UTC (permalink / raw)
To: Paul Mackerras; +Cc: linuxppc-dev, Alexander Graf, kvm-ppc
On Sat, Nov 16, 2013 at 3:00 PM, Paul Mackerras <paulus@samba.org> wrote:
> On Fri, Nov 15, 2013 at 04:35:01PM +0800, Liu Ping Fan wrote:
>>
>> +sc_1_fast_return:
>> + mtspr SPRN_SRR0,r10
>> + mtspr SPRN_SRR1,r11
>> + li r10, BOOK3S_INTERRUPT_SYSCALL
>> + li r11, (MSR_ME << 1) | 1 /* synthesize MSR_SF | MSR_ME */
>> + rotldi r11, r11, 63
>
> You need a "mr r4, r9" instruction here, because fast_guest_return
> needs the vcpu pointer in r4. Apart from that this looks fine.
>
Will fix it.
Thanks and regards,
Pingfan
>> + b fast_guest_return
>
> Paul.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v4] powerpc: kvm: fix rare but potential deadlock scene
2013-11-16 6:55 ` [PATCH v4] powerpc: kvm: fix rare but potential deadlock scene Paul Mackerras
@ 2013-11-18 21:32 ` Alexander Graf
2013-11-18 21:43 ` Alexander Graf
0 siblings, 1 reply; 7+ messages in thread
From: Alexander Graf @ 2013-11-18 21:32 UTC (permalink / raw)
To: Paul Mackerras; +Cc: linuxppc-dev, Liu Ping Fan, kvm-ppc
On 16.11.2013, at 01:55, Paul Mackerras <paulus@samba.org> wrote:
> On Fri, Nov 15, 2013 at 04:35:00PM +0800, Liu Ping Fan wrote:
>> Since kvmppc_hv_find_lock_hpte() is called from both virtmode and
>> realmode, so it can trigger the deadlock.
>>
>> Suppose the following scene:
>>
>> Two physical cpuM, cpuN, two VM instances A, B, each VM has a group of
>> vcpus.
>>
>> If on cpuM, vcpu_A_1 holds bitlock X (HPTE_V_HVLOCK), then is switched
>> out, and on cpuN, vcpu_A_2 try to lock X in realmode, then cpuN will be
>> caught in realmode for a long time.
>>
>> What makes things even worse if the following happens,
>> On cpuM, bitlockX is hold, on cpuN, Y is hold.
>> vcpu_B_2 try to lock Y on cpuM in realmode
>> vcpu_A_2 try to lock X on cpuN in realmode
>>
>> Oops! deadlock happens
>>
>> Signed-off-by: Liu Ping Fan <pingfank@linux.vnet.ibm.com>
>
> Reviewed-by: Paul Mackerras <paulus@samba.org>
Thanks, applied to kvm-ppc-queue.
Alex
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v4] powerpc: kvm: fix rare but potential deadlock scene
2013-11-18 21:32 ` Alexander Graf
@ 2013-11-18 21:43 ` Alexander Graf
0 siblings, 0 replies; 7+ messages in thread
From: Alexander Graf @ 2013-11-18 21:43 UTC (permalink / raw)
To: Paul Mackerras
Cc: linuxppc-dev, Liu Ping Fan, kvm-ppc, kvm@vger.kernel.org mailing list
On 18.11.2013, at 16:32, Alexander Graf <agraf@suse.de> wrote:
>=20
> On 16.11.2013, at 01:55, Paul Mackerras <paulus@samba.org> wrote:
>=20
>> On Fri, Nov 15, 2013 at 04:35:00PM +0800, Liu Ping Fan wrote:
>>> Since kvmppc_hv_find_lock_hpte() is called from both virtmode and
>>> realmode, so it can trigger the deadlock.
>>>=20
>>> Suppose the following scene:
>>>=20
>>> Two physical cpuM, cpuN, two VM instances A, B, each VM has a group =
of
>>> vcpus.
>>>=20
>>> If on cpuM, vcpu_A_1 holds bitlock X (HPTE_V_HVLOCK), then is =
switched
>>> out, and on cpuN, vcpu_A_2 try to lock X in realmode, then cpuN will =
be
>>> caught in realmode for a long time.
>>>=20
>>> What makes things even worse if the following happens,
>>> On cpuM, bitlockX is hold, on cpuN, Y is hold.
>>> vcpu_B_2 try to lock Y on cpuM in realmode
>>> vcpu_A_2 try to lock X on cpuN in realmode
>>>=20
>>> Oops! deadlock happens
>>>=20
>>> Signed-off-by: Liu Ping Fan <pingfank@linux.vnet.ibm.com>
>>=20
>> Reviewed-by: Paul Mackerras <paulus@samba.org>
>=20
> Thanks, applied to kvm-ppc-queue.
Actually, I've changed my mind and moved the patch to the for-3.13 =
branch instead. Please make sure to CC kvm@vger on all patches you =
submit though.
Alex
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2013-11-18 21:43 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-11-15 8:35 [PATCH v4] powerpc: kvm: fix rare but potential deadlock scene Liu Ping Fan
2013-11-15 8:35 ` [PATCH v2] powerpc: kvm: optimize "sc 1" as fast return Liu Ping Fan
2013-11-16 7:00 ` Paul Mackerras
2013-11-18 1:06 ` liu ping fan
2013-11-16 6:55 ` [PATCH v4] powerpc: kvm: fix rare but potential deadlock scene Paul Mackerras
2013-11-18 21:32 ` Alexander Graf
2013-11-18 21:43 ` Alexander Graf
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).