* Re: device-assignment deadlock
@ 2009-05-12 9:05 Yang, Sheng
2009-05-12 9:32 ` [PATCH 1/1] KVM: Fix potentially recursively get kvm lock Sheng Yang
0 siblings, 1 reply; 16+ messages in thread
From: Yang, Sheng @ 2009-05-12 9:05 UTC (permalink / raw)
To: Alex Williamson; +Cc: kvm
[I was kicked off from the mailing list by periodic unknown reason last
Friday... Sorry]
> Hi Sheng,
>
> I think I'm running into the following deadlock in the kvm kernel module
> when trying to use device assignment:
>
> CPU A CPU B
> kvm_vm_ioctl_deassign_dev_irq()
> mutex_lock(&kvm->lock); worker_thread()
> -> kvm_deassign_irq() ->
>kvm_assigned_dev_interrupt_work_handler()
> -> deassign_host_irq() mutex_lock(&kvm->lock);
> -> cancel_work_sync() [blocked]
> I wonder if we need finer granularity locking to avoid this.
> Suggestions? Thanks,
This part again...
I think simply move kvm_deassign_irq() out of critical region is OK, and I
also add the lock which seems missing in deassign_guest_irq(). Would post a
patch soon.
--
regards
Yang, Sheng
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH 1/1] KVM: Fix potentially recursively get kvm lock
2009-05-12 9:05 device-assignment deadlock Yang, Sheng
@ 2009-05-12 9:32 ` Sheng Yang
2009-05-12 11:55 ` Marcelo Tosatti
0 siblings, 1 reply; 16+ messages in thread
From: Sheng Yang @ 2009-05-12 9:32 UTC (permalink / raw)
To: Avi Kivity; +Cc: Marcelo Tosatti, Alex Williamson, kvm, Sheng Yang
kvm_vm_ioctl_deassign_dev_irq() would potentially recursively get kvm->lock,
because it called kvm_deassigned_irq() which implicit hold kvm->lock by calling
deassign_host_irq().
Fix it by move kvm_deassign_irq() out of critial region. And add the missing
lock for deassign_guest_irq().
Reported-by: Alex Williamson <alex.williamson@hp.com>
Signed-off-by: Sheng Yang <sheng@linux.intel.com>
---
virt/kvm/kvm_main.c | 14 +++++++-------
1 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 4d00942..3c69655 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -215,6 +215,8 @@ static void kvm_assigned_dev_ack_irq(struct kvm_irq_ack_notifier *kian)
static void deassign_guest_irq(struct kvm *kvm,
struct kvm_assigned_dev_kernel *assigned_dev)
{
+ mutex_lock(&kvm->lock);
+
kvm_unregister_irq_ack_notifier(&assigned_dev->ack_notifier);
assigned_dev->ack_notifier.gsi = -1;
@@ -222,6 +224,8 @@ static void deassign_guest_irq(struct kvm *kvm,
kvm_free_irq_source_id(kvm, assigned_dev->irq_source_id);
assigned_dev->irq_source_id = -1;
assigned_dev->irq_requested_type &= ~(KVM_DEV_IRQ_GUEST_MASK);
+
+ mutex_unlock(&kvm->lock);
}
/* The function implicit hold kvm->lock mutex due to cancel_work_sync() */
@@ -558,20 +562,16 @@ static int kvm_vm_ioctl_deassign_dev_irq(struct kvm *kvm,
struct kvm_assigned_irq
*assigned_irq)
{
- int r = -ENODEV;
struct kvm_assigned_dev_kernel *match;
mutex_lock(&kvm->lock);
-
match = kvm_find_assigned_dev(&kvm->arch.assigned_dev_head,
assigned_irq->assigned_dev_id);
+ mutex_unlock(&kvm->lock);
if (!match)
- goto out;
+ return -ENODEV;
- r = kvm_deassign_irq(kvm, match, assigned_irq->flags);
-out:
- mutex_unlock(&kvm->lock);
- return r;
+ return kvm_deassign_irq(kvm, match, assigned_irq->flags);
}
static int kvm_vm_ioctl_assign_device(struct kvm *kvm,
--
1.5.4.5
^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH 1/1] KVM: Fix potentially recursively get kvm lock
2009-05-12 9:32 ` [PATCH 1/1] KVM: Fix potentially recursively get kvm lock Sheng Yang
@ 2009-05-12 11:55 ` Marcelo Tosatti
2009-05-12 14:13 ` Yang, Sheng
0 siblings, 1 reply; 16+ messages in thread
From: Marcelo Tosatti @ 2009-05-12 11:55 UTC (permalink / raw)
To: Sheng Yang; +Cc: Avi Kivity, Alex Williamson, kvm
On Tue, May 12, 2009 at 05:32:09PM +0800, Sheng Yang wrote:
> kvm_vm_ioctl_deassign_dev_irq() would potentially recursively get kvm->lock,
> because it called kvm_deassigned_irq() which implicit hold kvm->lock by calling
> deassign_host_irq().
>
> Fix it by move kvm_deassign_irq() out of critial region. And add the missing
> lock for deassign_guest_irq().
>
> Reported-by: Alex Williamson <alex.williamson@hp.com>
> Signed-off-by: Sheng Yang <sheng@linux.intel.com>
> ---
> virt/kvm/kvm_main.c | 14 +++++++-------
> 1 files changed, 7 insertions(+), 7 deletions(-)
>
> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> index 4d00942..3c69655 100644
> --- a/virt/kvm/kvm_main.c
> +++ b/virt/kvm/kvm_main.c
> @@ -215,6 +215,8 @@ static void kvm_assigned_dev_ack_irq(struct kvm_irq_ack_notifier *kian)
> static void deassign_guest_irq(struct kvm *kvm,
> struct kvm_assigned_dev_kernel *assigned_dev)
> {
> + mutex_lock(&kvm->lock);
> +
> kvm_unregister_irq_ack_notifier(&assigned_dev->ack_notifier);
> assigned_dev->ack_notifier.gsi = -1;
>
> @@ -222,6 +224,8 @@ static void deassign_guest_irq(struct kvm *kvm,
> kvm_free_irq_source_id(kvm, assigned_dev->irq_source_id);
> assigned_dev->irq_source_id = -1;
> assigned_dev->irq_requested_type &= ~(KVM_DEV_IRQ_GUEST_MASK);
> +
> + mutex_unlock(&kvm->lock);
> }
>
> /* The function implicit hold kvm->lock mutex due to cancel_work_sync() */
> @@ -558,20 +562,16 @@ static int kvm_vm_ioctl_deassign_dev_irq(struct kvm *kvm,
> struct kvm_assigned_irq
> *assigned_irq)
> {
> - int r = -ENODEV;
> struct kvm_assigned_dev_kernel *match;
>
> mutex_lock(&kvm->lock);
> -
> match = kvm_find_assigned_dev(&kvm->arch.assigned_dev_head,
> assigned_irq->assigned_dev_id);
> + mutex_unlock(&kvm->lock);
assigned_dev list is protected by kvm->lock. So you could have another
ioctl adding to it at the same time you're searching.
Could either have a separate kvm->assigned_devs_lock, to protect
kvm->arch.assigned_dev_head (users are ioctls that manipulate it), or
change the IRQ injection to use a separate spinlock, kill the workqueue
and call kvm_set_irq from the assigned device interrupt handler.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 1/1] KVM: Fix potentially recursively get kvm lock
2009-05-12 11:55 ` Marcelo Tosatti
@ 2009-05-12 14:13 ` Yang, Sheng
2009-05-12 14:30 ` Marcelo Tosatti
0 siblings, 1 reply; 16+ messages in thread
From: Yang, Sheng @ 2009-05-12 14:13 UTC (permalink / raw)
To: Marcelo Tosatti; +Cc: Avi Kivity, Alex Williamson, kvm
On Tuesday 12 May 2009 19:55:24 Marcelo Tosatti wrote:
> On Tue, May 12, 2009 at 05:32:09PM +0800, Sheng Yang wrote:
> > kvm_vm_ioctl_deassign_dev_irq() would potentially recursively get
> > kvm->lock, because it called kvm_deassigned_irq() which implicit hold
> > kvm->lock by calling deassign_host_irq().
> >
> > Fix it by move kvm_deassign_irq() out of critial region. And add the
> > missing lock for deassign_guest_irq().
> >
> > Reported-by: Alex Williamson <alex.williamson@hp.com>
> > Signed-off-by: Sheng Yang <sheng@linux.intel.com>
> > ---
> > virt/kvm/kvm_main.c | 14 +++++++-------
> > 1 files changed, 7 insertions(+), 7 deletions(-)
> >
> > diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> > index 4d00942..3c69655 100644
> > --- a/virt/kvm/kvm_main.c
> > +++ b/virt/kvm/kvm_main.c
> > @@ -215,6 +215,8 @@ static void kvm_assigned_dev_ack_irq(struct
> > kvm_irq_ack_notifier *kian) static void deassign_guest_irq(struct kvm
> > *kvm,
> > struct kvm_assigned_dev_kernel *assigned_dev)
> > {
> > + mutex_lock(&kvm->lock);
> > +
> > kvm_unregister_irq_ack_notifier(&assigned_dev->ack_notifier);
> > assigned_dev->ack_notifier.gsi = -1;
> >
> > @@ -222,6 +224,8 @@ static void deassign_guest_irq(struct kvm *kvm,
> > kvm_free_irq_source_id(kvm, assigned_dev->irq_source_id);
> > assigned_dev->irq_source_id = -1;
> > assigned_dev->irq_requested_type &= ~(KVM_DEV_IRQ_GUEST_MASK);
> > +
> > + mutex_unlock(&kvm->lock);
> > }
> >
> > /* The function implicit hold kvm->lock mutex due to cancel_work_sync()
> > */ @@ -558,20 +562,16 @@ static int kvm_vm_ioctl_deassign_dev_irq(struct
> > kvm *kvm, struct kvm_assigned_irq
> > *assigned_irq)
> > {
> > - int r = -ENODEV;
> > struct kvm_assigned_dev_kernel *match;
> >
> > mutex_lock(&kvm->lock);
> > -
> > match = kvm_find_assigned_dev(&kvm->arch.assigned_dev_head,
> > assigned_irq->assigned_dev_id);
> > + mutex_unlock(&kvm->lock);
>
> assigned_dev list is protected by kvm->lock. So you could have another
> ioctl adding to it at the same time you're searching.
Oh, yes... My fault...
> Could either have a separate kvm->assigned_devs_lock, to protect
> kvm->arch.assigned_dev_head (users are ioctls that manipulate it), or
> change the IRQ injection to use a separate spinlock, kill the workqueue
> and call kvm_set_irq from the assigned device interrupt handler.
Peferred the latter, though needs more work. But the only reason for put a
workqueue here is because kvm->lock is a mutex? I can't believe... If so, I
think we had made a big mistake - we have to fix all kinds of racy problem
caused by this, but finally find it's unnecessary...
Maybe another reason is kvm_kick_vcpu(), but have already fixed by you.
Continue to check the code...
--
regards
Yang, Sheng
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 1/1] KVM: Fix potentially recursively get kvm lock
2009-05-12 14:13 ` Yang, Sheng
@ 2009-05-12 14:30 ` Marcelo Tosatti
2009-05-12 15:59 ` Marcelo Tosatti
` (2 more replies)
0 siblings, 3 replies; 16+ messages in thread
From: Marcelo Tosatti @ 2009-05-12 14:30 UTC (permalink / raw)
To: Yang, Sheng; +Cc: Avi Kivity, Alex Williamson, kvm
On Tue, May 12, 2009 at 10:13:36PM +0800, Yang, Sheng wrote:
> > > + mutex_unlock(&kvm->lock);
> >
> > assigned_dev list is protected by kvm->lock. So you could have another
> > ioctl adding to it at the same time you're searching.
>
> Oh, yes... My fault...
>
> > Could either have a separate kvm->assigned_devs_lock, to protect
> > kvm->arch.assigned_dev_head (users are ioctls that manipulate it), or
> > change the IRQ injection to use a separate spinlock, kill the workqueue
> > and call kvm_set_irq from the assigned device interrupt handler.
>
> Peferred the latter, though needs more work. But the only reason for put a
> workqueue here is because kvm->lock is a mutex? I can't believe... If so, I
> think we had made a big mistake - we have to fix all kinds of racy problem
> caused by this, but finally find it's unnecessary...
One issue is that kvm_set_irq can take too long while interrupts are
blocked, and you'd have to disable interrupts in other contexes that
inject interrupts (say qemu->ioctl(SET_INTERRUPT)->...->), so all i can
see is a tradeoff.
<guess mode on>
But the interrupt injection path seems to be pretty short and efficient
to happen in host interrupt context.
<guess mode off>
Avi, Gleb?
> Maybe another reason is kvm_kick_vcpu(), but have already fixed by you.
Note you tested the spinlock_irq patch with GigE and there was no
significant performance regression right?
>
> Continue to check the code...
>
> --
> regards
> Yang, Sheng
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 1/1] KVM: Fix potentially recursively get kvm lock
2009-05-12 14:30 ` Marcelo Tosatti
@ 2009-05-12 15:59 ` Marcelo Tosatti
2009-05-12 19:44 ` Marcelo Tosatti
2009-05-13 11:43 ` Gleb Natapov
2 siblings, 0 replies; 16+ messages in thread
From: Marcelo Tosatti @ 2009-05-12 15:59 UTC (permalink / raw)
To: Yang, Sheng; +Cc: Avi Kivity, Alex Williamson, kvm
On Tue, May 12, 2009 at 11:30:21AM -0300, Marcelo Tosatti wrote:
> On Tue, May 12, 2009 at 10:13:36PM +0800, Yang, Sheng wrote:
> > > > + mutex_unlock(&kvm->lock);
> > >
> > > assigned_dev list is protected by kvm->lock. So you could have another
> > > ioctl adding to it at the same time you're searching.
> >
> > Oh, yes... My fault...
> >
> > > Could either have a separate kvm->assigned_devs_lock, to protect
> > > kvm->arch.assigned_dev_head (users are ioctls that manipulate it), or
> > > change the IRQ injection to use a separate spinlock, kill the workqueue
> > > and call kvm_set_irq from the assigned device interrupt handler.
> >
> > Peferred the latter, though needs more work. But the only reason for put a
> > workqueue here is because kvm->lock is a mutex? I can't believe... If so, I
> > think we had made a big mistake - we have to fix all kinds of racy problem
> > caused by this, but finally find it's unnecessary...
>
> One issue is that kvm_set_irq can take too long while interrupts are
> blocked, and you'd have to disable interrupts in other contexes that
> inject interrupts (say qemu->ioctl(SET_INTERRUPT)->...->), so all i can
> see is a tradeoff.
Or multiple kvm_set_irq calls for MSI.
>
> <guess mode on>
>
> But the interrupt injection path seems to be pretty short and efficient
> to happen in host interrupt context.
>
> <guess mode off>
>
> Avi, Gleb?
>
> > Maybe another reason is kvm_kick_vcpu(), but have already fixed by you.
>
> Note you tested the spinlock_irq patch with GigE and there was no
> significant performance regression right?
>
> >
> > Continue to check the code...
> >
> > --
> > regards
> > Yang, Sheng
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 1/1] KVM: Fix potentially recursively get kvm lock
2009-05-12 14:30 ` Marcelo Tosatti
2009-05-12 15:59 ` Marcelo Tosatti
@ 2009-05-12 19:44 ` Marcelo Tosatti
2009-05-12 21:36 ` Alex Williamson
2009-05-13 11:43 ` Gleb Natapov
2 siblings, 1 reply; 16+ messages in thread
From: Marcelo Tosatti @ 2009-05-12 19:44 UTC (permalink / raw)
To: Yang, Sheng; +Cc: Avi Kivity, Alex Williamson, kvm
On Tue, May 12, 2009 at 11:30:21AM -0300, Marcelo Tosatti wrote:
> On Tue, May 12, 2009 at 10:13:36PM +0800, Yang, Sheng wrote:
> > > > + mutex_unlock(&kvm->lock);
> > >
> > > assigned_dev list is protected by kvm->lock. So you could have another
> > > ioctl adding to it at the same time you're searching.
> >
> > Oh, yes... My fault...
> >
> > > Could either have a separate kvm->assigned_devs_lock, to protect
> > > kvm->arch.assigned_dev_head (users are ioctls that manipulate it), or
> > > change the IRQ injection to use a separate spinlock, kill the workqueue
> > > and call kvm_set_irq from the assigned device interrupt handler.
> >
> > Peferred the latter, though needs more work. But the only reason for put a
> > workqueue here is because kvm->lock is a mutex? I can't believe... If so, I
> > think we had made a big mistake - we have to fix all kinds of racy problem
> > caused by this, but finally find it's unnecessary...
>
> One issue is that kvm_set_irq can take too long while interrupts are
> blocked, and you'd have to disable interrupts in other contexes that
> inject interrupts (say qemu->ioctl(SET_INTERRUPT)->...->), so all i can
> see is a tradeoff.
>
> <guess mode on>
>
> But the interrupt injection path seems to be pretty short and efficient
> to happen in host interrupt context.
>
> <guess mode off>
>
> Avi, Gleb?
>
> > Maybe another reason is kvm_kick_vcpu(), but have already fixed by you.
>
> Note you tested the spinlock_irq patch with GigE and there was no
> significant performance regression right?
>
> >
> > Continue to check the code...
OK, it might take some time for bigger changes to happen. I've changed
your patch to drop the lock only around cancel_work_sync. Can deadlock
if someone else tries to mess with the assigned device at the same time,
but the VM won't go away under it because of the vmfd reference.
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 4d00942..ba067db 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -250,7 +250,15 @@ static void deassign_host_irq(struct kvm *kvm,
disable_irq_nosync(assigned_dev->
host_msix_entries[i].vector);
+ /*
+ * FIXME: kvm_assigned_dev_interrupt_work_handler can deadlock
+ * with cancel_work_sync, since it requires kvm->lock for irq
+ * injection. This is a hack, the irq code must use
+ * a separate lock.
+ */
+ mutex_unlock(&kvm->lock);
cancel_work_sync(&assigned_dev->interrupt_work);
+ mutex_lock(&kvm->lock);
for (i = 0; i < assigned_dev->entries_nr; i++)
free_irq(assigned_dev->host_msix_entries[i].vector,
^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH 1/1] KVM: Fix potentially recursively get kvm lock
2009-05-12 19:44 ` Marcelo Tosatti
@ 2009-05-12 21:36 ` Alex Williamson
2009-05-12 22:09 ` Marcelo Tosatti
0 siblings, 1 reply; 16+ messages in thread
From: Alex Williamson @ 2009-05-12 21:36 UTC (permalink / raw)
To: Marcelo Tosatti; +Cc: Yang, Sheng, Avi Kivity, kvm
On Tue, 2009-05-12 at 16:44 -0300, Marcelo Tosatti wrote:
> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> index 4d00942..ba067db 100644
> --- a/virt/kvm/kvm_main.c
> +++ b/virt/kvm/kvm_main.c
> @@ -250,7 +250,15 @@ static void deassign_host_irq(struct kvm *kvm,
> disable_irq_nosync(assigned_dev->
> host_msix_entries[i].vector);
>
> + /*
> + * FIXME: kvm_assigned_dev_interrupt_work_handler can deadlock
> + * with cancel_work_sync, since it requires kvm->lock for irq
> + * injection. This is a hack, the irq code must use
> + * a separate lock.
> + */
> + mutex_unlock(&kvm->lock);
> cancel_work_sync(&assigned_dev->interrupt_work);
> + mutex_lock(&kvm->lock);
Seems to work, I assume you've got a similar unlock/lock for the
MSI/INTx block. Thanks,
Alex
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 1/1] KVM: Fix potentially recursively get kvm lock
2009-05-12 21:36 ` Alex Williamson
@ 2009-05-12 22:09 ` Marcelo Tosatti
2009-05-12 22:17 ` Alex Williamson
2009-05-13 2:07 ` Yang, Sheng
0 siblings, 2 replies; 16+ messages in thread
From: Marcelo Tosatti @ 2009-05-12 22:09 UTC (permalink / raw)
To: Alex Williamson; +Cc: Yang, Sheng, Avi Kivity, kvm
[-- Attachment #1: Type: text/plain, Size: 2672 bytes --]
On Tue, May 12, 2009 at 03:36:27PM -0600, Alex Williamson wrote:
> On Tue, 2009-05-12 at 16:44 -0300, Marcelo Tosatti wrote:
> > diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> > index 4d00942..ba067db 100644
> > --- a/virt/kvm/kvm_main.c
> > +++ b/virt/kvm/kvm_main.c
> > @@ -250,7 +250,15 @@ static void deassign_host_irq(struct kvm *kvm,
> > disable_irq_nosync(assigned_dev->
> > host_msix_entries[i].vector);
> >
> > + /*
> > + * FIXME: kvm_assigned_dev_interrupt_work_handler can deadlock
> > + * with cancel_work_sync, since it requires kvm->lock for irq
> > + * injection. This is a hack, the irq code must use
> > + * a separate lock.
> > + */
> > + mutex_unlock(&kvm->lock);
> > cancel_work_sync(&assigned_dev->interrupt_work);
> > + mutex_lock(&kvm->lock);
>
> Seems to work, I assume you've got a similar unlock/lock for the
> MSI/INTx block. Thanks,
KVM: workaround workqueue / deassign_host_irq deadlock
I think I'm running into the following deadlock in the kvm kernel module
when trying to use device assignment:
CPU A CPU B
kvm_vm_ioctl_deassign_dev_irq()
mutex_lock(&kvm->lock); worker_thread()
-> kvm_deassign_irq() ->
kvm_assigned_dev_interrupt_work_handler()
-> deassign_host_irq() mutex_lock(&kvm->lock);
-> cancel_work_sync() [blocked]
Workaround the issue by dropping kvm->lock for cancel_work_sync().
Reported-by: Alex Williamson <alex.williamson@hp.com>
From: Sheng Yang <sheng.yang@intel.com>
Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 4d00942..d4af719 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -250,7 +250,15 @@ static void deassign_host_irq(struct kvm *kvm,
disable_irq_nosync(assigned_dev->
host_msix_entries[i].vector);
+ /*
+ * FIXME: kvm_assigned_dev_interrupt_work_handler can deadlock
+ * with cancel_work_sync, since it requires kvm->lock for irq
+ * injection. This is a hack, the irq code must use
+ * a separate lock. Same below for MSI.
+ */
+ mutex_unlock(&kvm->lock);
cancel_work_sync(&assigned_dev->interrupt_work);
+ mutex_lock(&kvm->lock);
for (i = 0; i < assigned_dev->entries_nr; i++)
free_irq(assigned_dev->host_msix_entries[i].vector,
@@ -263,7 +271,9 @@ static void deassign_host_irq(struct kvm *kvm,
} else {
/* Deal with MSI and INTx */
disable_irq_nosync(assigned_dev->host_irq);
+ mutex_unlock(&kvm->lock);
cancel_work_sync(&assigned_dev->interrupt_work);
+ mutex_lock(&kvm->lock);
free_irq(assigned_dev->host_irq, (void *)assigned_dev);
[-- Attachment #2: assigned-dev-cancel-work-deadlock.patch --]
[-- Type: text/plain, Size: 1070 bytes --]
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 4d00942..d4af719 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -250,7 +250,15 @@ static void deassign_host_irq(struct kvm *kvm,
disable_irq_nosync(assigned_dev->
host_msix_entries[i].vector);
+ /*
+ * FIXME: kvm_assigned_dev_interrupt_work_handler can deadlock
+ * with cancel_work_sync, since it requires kvm->lock for irq
+ * injection. This is a hack, the irq code must use
+ * a separate lock. Same below for MSI.
+ */
+ mutex_unlock(&kvm->lock);
cancel_work_sync(&assigned_dev->interrupt_work);
+ mutex_lock(&kvm->lock);
for (i = 0; i < assigned_dev->entries_nr; i++)
free_irq(assigned_dev->host_msix_entries[i].vector,
@@ -263,7 +271,9 @@ static void deassign_host_irq(struct kvm *kvm,
} else {
/* Deal with MSI and INTx */
disable_irq_nosync(assigned_dev->host_irq);
+ mutex_unlock(&kvm->lock);
cancel_work_sync(&assigned_dev->interrupt_work);
+ mutex_lock(&kvm->lock);
free_irq(assigned_dev->host_irq, (void *)assigned_dev);
^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH 1/1] KVM: Fix potentially recursively get kvm lock
2009-05-12 22:09 ` Marcelo Tosatti
@ 2009-05-12 22:17 ` Alex Williamson
2009-05-22 15:06 ` Chris Wright
2009-05-13 2:07 ` Yang, Sheng
1 sibling, 1 reply; 16+ messages in thread
From: Alex Williamson @ 2009-05-12 22:17 UTC (permalink / raw)
To: Marcelo Tosatti; +Cc: Yang, Sheng, Avi Kivity, kvm
On Tue, 2009-05-12 at 19:09 -0300, Marcelo Tosatti wrote:
> KVM: workaround workqueue / deassign_host_irq deadlock
>
> I think I'm running into the following deadlock in the kvm kernel module
> when trying to use device assignment:
>
> CPU A CPU B
> kvm_vm_ioctl_deassign_dev_irq()
> mutex_lock(&kvm->lock); worker_thread()
> -> kvm_deassign_irq() ->
> kvm_assigned_dev_interrupt_work_handler()
> -> deassign_host_irq() mutex_lock(&kvm->lock);
> -> cancel_work_sync() [blocked]
>
> Workaround the issue by dropping kvm->lock for cancel_work_sync().
>
> Reported-by: Alex Williamson <alex.williamson@hp.com>
> From: Sheng Yang <sheng.yang@intel.com>
> Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
Perfect, thanks.
Acked-by: Alex Williamson <alex.williamson@hp.com>
> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> index 4d00942..d4af719 100644
> --- a/virt/kvm/kvm_main.c
> +++ b/virt/kvm/kvm_main.c
> @@ -250,7 +250,15 @@ static void deassign_host_irq(struct kvm *kvm,
> disable_irq_nosync(assigned_dev->
> host_msix_entries[i].vector);
>
> + /*
> + * FIXME: kvm_assigned_dev_interrupt_work_handler can deadlock
> + * with cancel_work_sync, since it requires kvm->lock for irq
> + * injection. This is a hack, the irq code must use
> + * a separate lock. Same below for MSI.
> + */
> + mutex_unlock(&kvm->lock);
> cancel_work_sync(&assigned_dev->interrupt_work);
> + mutex_lock(&kvm->lock);
>
> for (i = 0; i < assigned_dev->entries_nr; i++)
> free_irq(assigned_dev->host_msix_entries[i].vector,
> @@ -263,7 +271,9 @@ static void deassign_host_irq(struct kvm *kvm,
> } else {
> /* Deal with MSI and INTx */
> disable_irq_nosync(assigned_dev->host_irq);
> + mutex_unlock(&kvm->lock);
> cancel_work_sync(&assigned_dev->interrupt_work);
> + mutex_lock(&kvm->lock);
>
> free_irq(assigned_dev->host_irq, (void *)assigned_dev);
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 1/1] KVM: Fix potentially recursively get kvm lock
2009-05-12 22:09 ` Marcelo Tosatti
2009-05-12 22:17 ` Alex Williamson
@ 2009-05-13 2:07 ` Yang, Sheng
2009-05-13 13:14 ` Marcelo Tosatti
1 sibling, 1 reply; 16+ messages in thread
From: Yang, Sheng @ 2009-05-13 2:07 UTC (permalink / raw)
To: Marcelo Tosatti; +Cc: Alex Williamson, Avi Kivity, kvm
On Wednesday 13 May 2009 06:09:08 Marcelo Tosatti wrote:
> On Tue, May 12, 2009 at 03:36:27PM -0600, Alex Williamson wrote:
> > On Tue, 2009-05-12 at 16:44 -0300, Marcelo Tosatti wrote:
> > > diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> > > index 4d00942..ba067db 100644
> > > --- a/virt/kvm/kvm_main.c
> > > +++ b/virt/kvm/kvm_main.c
> > > @@ -250,7 +250,15 @@ static void deassign_host_irq(struct kvm *kvm,
> > > disable_irq_nosync(assigned_dev->
> > > host_msix_entries[i].vector);
> > >
> > > + /*
> > > + * FIXME: kvm_assigned_dev_interrupt_work_handler can deadlock
> > > + * with cancel_work_sync, since it requires kvm->lock for irq
> > > + * injection. This is a hack, the irq code must use
> > > + * a separate lock.
> > > + */
> > > + mutex_unlock(&kvm->lock);
> > > cancel_work_sync(&assigned_dev->interrupt_work);
> > > + mutex_lock(&kvm->lock);
> >
> > Seems to work, I assume you've got a similar unlock/lock for the
> > MSI/INTx block. Thanks,
>
> KVM: workaround workqueue / deassign_host_irq deadlock
>
> I think I'm running into the following deadlock in the kvm kernel module
> when trying to use device assignment:
>
> CPU A CPU B
> kvm_vm_ioctl_deassign_dev_irq()
> mutex_lock(&kvm->lock); worker_thread()
> -> kvm_deassign_irq() ->
> kvm_assigned_dev_interrupt_work_handler()
> -> deassign_host_irq() mutex_lock(&kvm->lock);
> -> cancel_work_sync() [blocked]
>
> Workaround the issue by dropping kvm->lock for cancel_work_sync().
>
> Reported-by: Alex Williamson <alex.williamson@hp.com>
> From: Sheng Yang <sheng.yang@intel.com>
> Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
Another calling path(kvm_free_all_assigned_devices()) don't hold kvm->lock...
Seems it need the lock for travel assigned dev list?
--
regards
Yang, Sheng
> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> index 4d00942..d4af719 100644
> --- a/virt/kvm/kvm_main.c
> +++ b/virt/kvm/kvm_main.c
> @@ -250,7 +250,15 @@ static void deassign_host_irq(struct kvm *kvm,
> disable_irq_nosync(assigned_dev->
> host_msix_entries[i].vector);
>
> + /*
> + * FIXME: kvm_assigned_dev_interrupt_work_handler can deadlock
> + * with cancel_work_sync, since it requires kvm->lock for irq
> + * injection. This is a hack, the irq code must use
> + * a separate lock. Same below for MSI.
> + */
> + mutex_unlock(&kvm->lock);
> cancel_work_sync(&assigned_dev->interrupt_work);
> + mutex_lock(&kvm->lock);
>
> for (i = 0; i < assigned_dev->entries_nr; i++)
> free_irq(assigned_dev->host_msix_entries[i].vector,
> @@ -263,7 +271,9 @@ static void deassign_host_irq(struct kvm *kvm,
> } else {
> /* Deal with MSI and INTx */
> disable_irq_nosync(assigned_dev->host_irq);
> + mutex_unlock(&kvm->lock);
> cancel_work_sync(&assigned_dev->interrupt_work);
> + mutex_lock(&kvm->lock);
>
> free_irq(assigned_dev->host_irq, (void *)assigned_dev);
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 1/1] KVM: Fix potentially recursively get kvm lock
2009-05-12 14:30 ` Marcelo Tosatti
2009-05-12 15:59 ` Marcelo Tosatti
2009-05-12 19:44 ` Marcelo Tosatti
@ 2009-05-13 11:43 ` Gleb Natapov
2 siblings, 0 replies; 16+ messages in thread
From: Gleb Natapov @ 2009-05-13 11:43 UTC (permalink / raw)
To: Marcelo Tosatti; +Cc: Yang, Sheng, Avi Kivity, Alex Williamson, kvm
On Tue, May 12, 2009 at 11:30:21AM -0300, Marcelo Tosatti wrote:
> On Tue, May 12, 2009 at 10:13:36PM +0800, Yang, Sheng wrote:
> > > > + mutex_unlock(&kvm->lock);
> > >
> > > assigned_dev list is protected by kvm->lock. So you could have another
> > > ioctl adding to it at the same time you're searching.
> >
> > Oh, yes... My fault...
> >
> > > Could either have a separate kvm->assigned_devs_lock, to protect
> > > kvm->arch.assigned_dev_head (users are ioctls that manipulate it), or
> > > change the IRQ injection to use a separate spinlock, kill the workqueue
> > > and call kvm_set_irq from the assigned device interrupt handler.
> >
> > Peferred the latter, though needs more work. But the only reason for put a
> > workqueue here is because kvm->lock is a mutex? I can't believe... If so, I
> > think we had made a big mistake - we have to fix all kinds of racy problem
> > caused by this, but finally find it's unnecessary...
>
> One issue is that kvm_set_irq can take too long while interrupts are
> blocked, and you'd have to disable interrupts in other contexes that
> inject interrupts (say qemu->ioctl(SET_INTERRUPT)->...->), so all i can
> see is a tradeoff.
>
> <guess mode on>
>
> But the interrupt injection path seems to be pretty short and efficient
> to happen in host interrupt context.
>
> <guess mode off>
>
> Avi, Gleb?
>
Interrupt injection path also use IRQ routing data structures so access
to them should be protected by the same lock. And of cause in kernel
device (apic/ioapic/pic) mmio is done holding this lock so interrupt
injection cannot happen in parallel with device reconfiguration. May
be we want more parallelism here.
> > Maybe another reason is kvm_kick_vcpu(), but have already fixed by you.
>
> Note you tested the spinlock_irq patch with GigE and there was no
> significant performance regression right?
>
> >
> > Continue to check the code...
> >
> > --
> > regards
> > Yang, Sheng
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Gleb.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 1/1] KVM: Fix potentially recursively get kvm lock
2009-05-13 2:07 ` Yang, Sheng
@ 2009-05-13 13:14 ` Marcelo Tosatti
0 siblings, 0 replies; 16+ messages in thread
From: Marcelo Tosatti @ 2009-05-13 13:14 UTC (permalink / raw)
To: Yang, Sheng; +Cc: Alex Williamson, Avi Kivity, kvm
On Wed, May 13, 2009 at 10:07:54AM +0800, Yang, Sheng wrote:
> > KVM: workaround workqueue / deassign_host_irq deadlock
> >
> > I think I'm running into the following deadlock in the kvm kernel module
> > when trying to use device assignment:
> >
> > CPU A CPU B
> > kvm_vm_ioctl_deassign_dev_irq()
> > mutex_lock(&kvm->lock); worker_thread()
> > -> kvm_deassign_irq() ->
> > kvm_assigned_dev_interrupt_work_handler()
> > -> deassign_host_irq() mutex_lock(&kvm->lock);
> > -> cancel_work_sync() [blocked]
> >
> > Workaround the issue by dropping kvm->lock for cancel_work_sync().
> >
> > Reported-by: Alex Williamson <alex.williamson@hp.com>
> > From: Sheng Yang <sheng.yang@intel.com>
> > Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
>
> Another calling path(kvm_free_all_assigned_devices()) don't hold kvm->lock...
> Seems it need the lock for travel assigned dev list?
Sheng,
The task executing the deassign irq ioctl has a reference to the vm
instance. This solution is just temporary though until the locks can be
split and then dropping kvm->lock around cancel_work_sync will not be
necessary anymore.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 1/1] KVM: Fix potentially recursively get kvm lock
2009-05-12 22:17 ` Alex Williamson
@ 2009-05-22 15:06 ` Chris Wright
2009-05-22 15:34 ` Alex Williamson
0 siblings, 1 reply; 16+ messages in thread
From: Chris Wright @ 2009-05-22 15:06 UTC (permalink / raw)
To: Alex Williamson; +Cc: Marcelo Tosatti, Yang, Sheng, Avi Kivity, kvm
* Alex Williamson (alex.williamson@hp.com) wrote:
> On Tue, 2009-05-12 at 19:09 -0300, Marcelo Tosatti wrote:
> > KVM: workaround workqueue / deassign_host_irq deadlock
> >
> > I think I'm running into the following deadlock in the kvm kernel module
> > when trying to use device assignment:
> >
> > CPU A CPU B
> > kvm_vm_ioctl_deassign_dev_irq()
> > mutex_lock(&kvm->lock); worker_thread()
> > -> kvm_deassign_irq() ->
> > kvm_assigned_dev_interrupt_work_handler()
> > -> deassign_host_irq() mutex_lock(&kvm->lock);
> > -> cancel_work_sync() [blocked]
> >
> > Workaround the issue by dropping kvm->lock for cancel_work_sync().
> >
> > Reported-by: Alex Williamson <alex.williamson@hp.com>
> > From: Sheng Yang <sheng.yang@intel.com>
> > Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
>
> Perfect, thanks.
>
> Acked-by: Alex Williamson <alex.williamson@hp.com>
Is this still pending?
thanks,
-chris
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 1/1] KVM: Fix potentially recursively get kvm lock
2009-05-22 15:06 ` Chris Wright
@ 2009-05-22 15:34 ` Alex Williamson
2009-05-22 15:36 ` Chris Wright
0 siblings, 1 reply; 16+ messages in thread
From: Alex Williamson @ 2009-05-22 15:34 UTC (permalink / raw)
To: Chris Wright; +Cc: Marcelo Tosatti, Yang, Sheng, Avi Kivity, kvm
On Fri, 2009-05-22 at 08:06 -0700, Chris Wright wrote:
> * Alex Williamson (alex.williamson@hp.com) wrote:
> > On Tue, 2009-05-12 at 19:09 -0300, Marcelo Tosatti wrote:
> > > KVM: workaround workqueue / deassign_host_irq deadlock
> > >
> > > I think I'm running into the following deadlock in the kvm kernel module
> > > when trying to use device assignment:
> > >
> > > CPU A CPU B
> > > kvm_vm_ioctl_deassign_dev_irq()
> > > mutex_lock(&kvm->lock); worker_thread()
> > > -> kvm_deassign_irq() ->
> > > kvm_assigned_dev_interrupt_work_handler()
> > > -> deassign_host_irq() mutex_lock(&kvm->lock);
> > > -> cancel_work_sync() [blocked]
> > >
> > > Workaround the issue by dropping kvm->lock for cancel_work_sync().
>
> Is this still pending?
I haven't seen this particular workaround make it into a tree, however
Marcelo has been working on a set of patches to properly fix this. Most
recent version was sent on 5/20.
Alex
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 1/1] KVM: Fix potentially recursively get kvm lock
2009-05-22 15:34 ` Alex Williamson
@ 2009-05-22 15:36 ` Chris Wright
0 siblings, 0 replies; 16+ messages in thread
From: Chris Wright @ 2009-05-22 15:36 UTC (permalink / raw)
To: Alex Williamson
Cc: Chris Wright, Marcelo Tosatti, Yang, Sheng, Avi Kivity, kvm
* Alex Williamson (alex.williamson@hp.com) wrote:
> On Fri, 2009-05-22 at 08:06 -0700, Chris Wright wrote:
> > * Alex Williamson (alex.williamson@hp.com) wrote:
> > > On Tue, 2009-05-12 at 19:09 -0300, Marcelo Tosatti wrote:
> > > > KVM: workaround workqueue / deassign_host_irq deadlock
> > > >
> > > > I think I'm running into the following deadlock in the kvm kernel module
> > > > when trying to use device assignment:
> > > >
> > > > CPU A CPU B
> > > > kvm_vm_ioctl_deassign_dev_irq()
> > > > mutex_lock(&kvm->lock); worker_thread()
> > > > -> kvm_deassign_irq() ->
> > > > kvm_assigned_dev_interrupt_work_handler()
> > > > -> deassign_host_irq() mutex_lock(&kvm->lock);
> > > > -> cancel_work_sync() [blocked]
> > > >
> > > > Workaround the issue by dropping kvm->lock for cancel_work_sync().
> >
> > Is this still pending?
>
> I haven't seen this particular workaround make it into a tree, however
> Marcelo has been working on a set of patches to properly fix this. Most
> recent version was sent on 5/20.
Great, thanks.
-chris
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2009-05-22 15:36 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2009-05-12 9:05 device-assignment deadlock Yang, Sheng
2009-05-12 9:32 ` [PATCH 1/1] KVM: Fix potentially recursively get kvm lock Sheng Yang
2009-05-12 11:55 ` Marcelo Tosatti
2009-05-12 14:13 ` Yang, Sheng
2009-05-12 14:30 ` Marcelo Tosatti
2009-05-12 15:59 ` Marcelo Tosatti
2009-05-12 19:44 ` Marcelo Tosatti
2009-05-12 21:36 ` Alex Williamson
2009-05-12 22:09 ` Marcelo Tosatti
2009-05-12 22:17 ` Alex Williamson
2009-05-22 15:06 ` Chris Wright
2009-05-22 15:34 ` Alex Williamson
2009-05-22 15:36 ` Chris Wright
2009-05-13 2:07 ` Yang, Sheng
2009-05-13 13:14 ` Marcelo Tosatti
2009-05-13 11:43 ` Gleb Natapov
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.