[PATCH v2 vfio 0/3] pds/vfio: Fixes for locking bugs

[PATCH v2 vfio 0/3] pds/vfio: Fixes for locking bugs

[Brett Creeley]

This series contains fixes for locking bugs in the recently introduced
pds-vfio-pci driver. There was an initial bug reported by Dan Carpenter
at:

https://lore.kernel.org/kvm/1f9bc27b-3de9-4891-9687-ba2820c1b390@moroto.mountain/

However, more locking bugs were found when looking into the previously
mentioned issue. So, all fixes are included in this series.

v2:
https://lore.kernel.org/kvm/20230914191540.54946-1-brett.creeley@amd.com/
- Trim the OOPs in the patch commit messages
- Rework patch 3/3 to only unlock the spinlock once
- Destroy the state_mutex in the driver specific vfio_device_ops.release
  callback

Brett Creeley (3):
  pds/vfio: Fix spinlock bad magic BUG
  pds/vfio: Fix mutex lock->magic != lock warning
  pds/vfio: Fix possible sleep while in atomic context

 drivers/vfio/pci/pds/vfio_dev.c | 27 ++++++++++++++++++++-------
 1 file changed, 20 insertions(+), 7 deletions(-)

-- 
2.17.1

[PATCH v2 vfio 1/3] pds/vfio: Fix spinlock bad magic BUG

[Brett Creeley]

The following BUG was found when running on a kernel with
CONFIG_DEBUG_SPINLOCK=y set:

BUG: spinlock bad magic on CPU#2, bash/2481
 lock: 0xffff8d6052a88f50, .magic: 00000000, .owner: <none>/-1, .owner_cpu: 0
Call Trace:
 <TASK>
 dump_stack_lvl+0x36/0x50
 do_raw_spin_lock+0x79/0xc0
 pds_vfio_reset+0x1d/0x60 [pds_vfio_pci]
 pci_reset_function+0x4b/0x70
 reset_store+0x5b/0xa0
 kernfs_fop_write_iter+0x137/0x1d0
 vfs_write+0x2de/0x410
 ksys_write+0x5d/0xd0
 do_syscall_64+0x3b/0x90
 entry_SYSCALL_64_after_hwframe+0x6e/0xd8

As shown, the .magic: 00000000, does not match the expected value. This
is because spin_lock_init() is never called for the reset_lock. Fix
this by calling spin_lock_init(&pds_vfio->reset_lock) when initializing
the device.

Signed-off-by: Brett Creeley <brett.creeley@amd.com>
Reviewed-by: Shannon Nelson <shannon.nelson@amd.com>
---
 drivers/vfio/pci/pds/vfio_dev.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/vfio/pci/pds/vfio_dev.c b/drivers/vfio/pci/pds/vfio_dev.c
index 649b18ee394b..c351f588fa13 100644
--- a/drivers/vfio/pci/pds/vfio_dev.c
+++ b/drivers/vfio/pci/pds/vfio_dev.c
@@ -155,6 +155,8 @@ static int pds_vfio_init_device(struct vfio_device *vdev)
 
 	pds_vfio->vf_id = vf_id;
 
+	spin_lock_init(&pds_vfio->reset_lock);
+
 	vdev->migration_flags = VFIO_MIGRATION_STOP_COPY | VFIO_MIGRATION_P2P;
 	vdev->mig_ops = &pds_vfio_lm_ops;
 	vdev->log_ops = &pds_vfio_log_ops;
-- 
2.17.1

[PATCH v2 vfio 2/3] pds/vfio: Fix mutex lock->magic != lock warning

[Brett Creeley]

The following BUG was found when running on a kernel with
CONFIG_DEBUG_MUTEXES=y set:

DEBUG_LOCKS_WARN_ON(lock->magic != lock)
RIP: 0010:mutex_trylock+0x10d/0x120
Call Trace:
 <TASK>
 ? __warn+0x85/0x140
 ? mutex_trylock+0x10d/0x120
 ? report_bug+0xfc/0x1e0
 ? handle_bug+0x3f/0x70
 ? exc_invalid_op+0x17/0x70
 ? asm_exc_invalid_op+0x1a/0x20
 ? mutex_trylock+0x10d/0x120
 ? mutex_trylock+0x10d/0x120
 pds_vfio_reset+0x3a/0x60 [pds_vfio_pci]
 pci_reset_function+0x4b/0x70
 reset_store+0x5b/0xa0
 kernfs_fop_write_iter+0x137/0x1d0
 vfs_write+0x2de/0x410
 ksys_write+0x5d/0xd0
 do_syscall_64+0x3b/0x90
 entry_SYSCALL_64_after_hwframe+0x6e/0xd8

As shown, lock->magic != lock. This is because
mutex_init(&pds_vfio->state_mutex) is called in the VFIO open path. So,
if a reset is initiated before the VFIO device is opened the mutex will
have never been initialized. Fix this by calling
mutex_init(&pds_vfio->state_mutex) in the VFIO init path.

Also, don't destroy the mutex on close because the device may
be re-opened, which would cause mutex to be uninitialized. Fix this by
implementing a driver specific vfio_device_ops.release callback that
destroys the mutex before calling vfio_pci_core_release_dev().

Signed-off-by: Brett Creeley <brett.creeley@amd.com>
Reviewed-by: Shannon Nelson <shannon.nelson@amd.com>
---
 drivers/vfio/pci/pds/vfio_dev.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/drivers/vfio/pci/pds/vfio_dev.c b/drivers/vfio/pci/pds/vfio_dev.c
index c351f588fa13..306b1c25f016 100644
--- a/drivers/vfio/pci/pds/vfio_dev.c
+++ b/drivers/vfio/pci/pds/vfio_dev.c
@@ -155,6 +155,7 @@ static int pds_vfio_init_device(struct vfio_device *vdev)
 
 	pds_vfio->vf_id = vf_id;
 
+	mutex_init(&pds_vfio->state_mutex);
 	spin_lock_init(&pds_vfio->reset_lock);
 
 	vdev->migration_flags = VFIO_MIGRATION_STOP_COPY | VFIO_MIGRATION_P2P;
@@ -170,6 +171,16 @@ static int pds_vfio_init_device(struct vfio_device *vdev)
 	return 0;
 }
 
+static void pds_vfio_release_device(struct vfio_device *vdev)
+{
+	struct pds_vfio_pci_device *pds_vfio =
+		container_of(vdev, struct pds_vfio_pci_device,
+			     vfio_coredev.vdev);
+
+	mutex_destroy(&pds_vfio->state_mutex);
+	vfio_pci_core_release_dev(vdev);
+}
+
 static int pds_vfio_open_device(struct vfio_device *vdev)
 {
 	struct pds_vfio_pci_device *pds_vfio =
@@ -181,7 +192,6 @@ static int pds_vfio_open_device(struct vfio_device *vdev)
 	if (err)
 		return err;
 
-	mutex_init(&pds_vfio->state_mutex);
 	pds_vfio->state = VFIO_DEVICE_STATE_RUNNING;
 	pds_vfio->deferred_reset_state = VFIO_DEVICE_STATE_RUNNING;
 
@@ -201,14 +211,13 @@ static void pds_vfio_close_device(struct vfio_device *vdev)
 	pds_vfio_put_save_file(pds_vfio);
 	pds_vfio_dirty_disable(pds_vfio, true);
 	mutex_unlock(&pds_vfio->state_mutex);
-	mutex_destroy(&pds_vfio->state_mutex);
 	vfio_pci_core_close_device(vdev);
 }
 
 static const struct vfio_device_ops pds_vfio_ops = {
 	.name = "pds-vfio",
 	.init = pds_vfio_init_device,
-	.release = vfio_pci_core_release_dev,
+	.release = pds_vfio_release_device,
 	.open_device = pds_vfio_open_device,
 	.close_device = pds_vfio_close_device,
 	.ioctl = vfio_pci_core_ioctl,
-- 
2.17.1

[PATCH v2 vfio 3/3] pds/vfio: Fix possible sleep while in atomic context

[Brett Creeley]

The driver could possibly sleep while in atomic context resulting
in the following call trace while CONFIG_DEBUG_ATOMIC_SLEEP=y is
set:

BUG: sleeping function called from invalid context at kernel/locking/mutex.c:283
in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 2817, name: bash
preempt_count: 1, expected: 0
RCU nest depth: 0, expected: 0
Call Trace:
 <TASK>
 dump_stack_lvl+0x36/0x50
 __might_resched+0x123/0x170
 mutex_lock+0x1e/0x50
 pds_vfio_put_lm_file+0x1e/0xa0 [pds_vfio_pci]
 pds_vfio_put_save_file+0x19/0x30 [pds_vfio_pci]
 pds_vfio_state_mutex_unlock+0x2e/0x80 [pds_vfio_pci]
 pci_reset_function+0x4b/0x70
 reset_store+0x5b/0xa0
 kernfs_fop_write_iter+0x137/0x1d0
 vfs_write+0x2de/0x410
 ksys_write+0x5d/0xd0
 do_syscall_64+0x3b/0x90
 entry_SYSCALL_64_after_hwframe+0x6e/0xd8

This can happen if pds_vfio_put_restore_file() and/or
pds_vfio_put_save_file() grab the mutex_lock(&lm_file->lock)
while the spin_lock(&pds_vfio->reset_lock) is held, which can
happen during while calling pds_vfio_state_mutex_unlock().

Fix this by using a local state variable to represent the current
vfio device state and update all the next state values. Then
release the spin_unlock(&pds_vfio->reset_lock) before checking if
the current vfio device state is VFIO_DEVICE_STATE_ERROR. Then
calling pds_vfio_put_restore_file() and pds_vfio_put_save_file() are
no longer being done while holding the reset_lock.

The only possible concerns are other threads that may call
pds_vfio_put_restore_file(), pds_vfio_put_save_file(), and/or
pds_vfio_dirty_disable(). However, those paths are already protected
by the state mutex_lock(), which is held in this context.

Reported-by: Dan Carpenter <dan.carpenter@linaro.org>
Closes: https://lore.kernel.org/kvm/1f9bc27b-3de9-4891-9687-ba2820c1b390@moroto.mountain/
Signed-off-by: Brett Creeley <brett.creeley@amd.com>
Reviewed-by: Shannon Nelson <shannon.nelson@amd.com>
---
 drivers/vfio/pci/pds/vfio_dev.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/drivers/vfio/pci/pds/vfio_dev.c b/drivers/vfio/pci/pds/vfio_dev.c
index 306b1c25f016..cf7f639ba0ec 100644
--- a/drivers/vfio/pci/pds/vfio_dev.c
+++ b/drivers/vfio/pci/pds/vfio_dev.c
@@ -31,15 +31,17 @@ void pds_vfio_state_mutex_unlock(struct pds_vfio_pci_device *pds_vfio)
 again:
 	spin_lock(&pds_vfio->reset_lock);
 	if (pds_vfio->deferred_reset) {
+		enum vfio_device_mig_state current_state = pds_vfio->state;
+
 		pds_vfio->deferred_reset = false;
-		if (pds_vfio->state == VFIO_DEVICE_STATE_ERROR) {
+		pds_vfio->state = pds_vfio->deferred_reset_state;
+		pds_vfio->deferred_reset_state = VFIO_DEVICE_STATE_RUNNING;
+		spin_unlock(&pds_vfio->reset_lock);
+		if (current_state == VFIO_DEVICE_STATE_ERROR) {
 			pds_vfio_put_restore_file(pds_vfio);
 			pds_vfio_put_save_file(pds_vfio);
 			pds_vfio_dirty_disable(pds_vfio, false);
 		}
-		pds_vfio->state = pds_vfio->deferred_reset_state;
-		pds_vfio->deferred_reset_state = VFIO_DEVICE_STATE_RUNNING;
-		spin_unlock(&pds_vfio->reset_lock);
 		goto again;
 	}
 	mutex_unlock(&pds_vfio->state_mutex);
-- 
2.17.1

RE: [PATCH v2 vfio 1/3] pds/vfio: Fix spinlock bad magic BUG

[Shameerali Kolothum Thodi]

> -----Original Message-----
> From: Brett Creeley [mailto:brett.creeley@amd.com]
> Sent: 12 October 2023 00:01
> To: jgg@ziepe.ca; yishaih@nvidia.com; Shameerali Kolothum Thodi
> <shameerali.kolothum.thodi@huawei.com>; kevin.tian@intel.com;
> alex.williamson@redhat.com; dan.carpenter@linaro.org
> Cc: kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> brett.creeley@amd.com; shannon.nelson@amd.com
> Subject: [PATCH v2 vfio 1/3] pds/vfio: Fix spinlock bad magic BUG
> 
> The following BUG was found when running on a kernel with
> CONFIG_DEBUG_SPINLOCK=y set:
> 
> BUG: spinlock bad magic on CPU#2, bash/2481
>  lock: 0xffff8d6052a88f50, .magic: 00000000, .owner:
> <none>/-1, .owner_cpu: 0
> Call Trace:
>  <TASK>
>  dump_stack_lvl+0x36/0x50
>  do_raw_spin_lock+0x79/0xc0
>  pds_vfio_reset+0x1d/0x60 [pds_vfio_pci]
>  pci_reset_function+0x4b/0x70
>  reset_store+0x5b/0xa0
>  kernfs_fop_write_iter+0x137/0x1d0
>  vfs_write+0x2de/0x410
>  ksys_write+0x5d/0xd0
>  do_syscall_64+0x3b/0x90
>  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
> 
> As shown, the .magic: 00000000, does not match the expected value. This
> is because spin_lock_init() is never called for the reset_lock. Fix
> this by calling spin_lock_init(&pds_vfio->reset_lock) when initializing
> the device.
> 
> Signed-off-by: Brett Creeley <brett.creeley@amd.com>
> Reviewed-by: Shannon Nelson <shannon.nelson@amd.com>

Reviewed-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>

I think this needs to be fixed in HiSilicon driver as well. I will send out a 
patch.

Thanks,
Shameer

> ---
>  drivers/vfio/pci/pds/vfio_dev.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/drivers/vfio/pci/pds/vfio_dev.c b/drivers/vfio/pci/pds/vfio_dev.c
> index 649b18ee394b..c351f588fa13 100644
> --- a/drivers/vfio/pci/pds/vfio_dev.c
> +++ b/drivers/vfio/pci/pds/vfio_dev.c
> @@ -155,6 +155,8 @@ static int pds_vfio_init_device(struct vfio_device
> *vdev)
> 
>  	pds_vfio->vf_id = vf_id;
> 
> +	spin_lock_init(&pds_vfio->reset_lock);
> +
>  	vdev->migration_flags = VFIO_MIGRATION_STOP_COPY |
> VFIO_MIGRATION_P2P;
>  	vdev->mig_ops = &pds_vfio_lm_ops;
>  	vdev->log_ops = &pds_vfio_log_ops;
> --
> 2.17.1

RE: [PATCH v2 vfio 2/3] pds/vfio: Fix mutex lock->magic != lock warning

[Shameerali Kolothum Thodi]

> -----Original Message-----
> From: Brett Creeley [mailto:brett.creeley@amd.com]
> Sent: 12 October 2023 00:01
> To: jgg@ziepe.ca; yishaih@nvidia.com; Shameerali Kolothum Thodi
> <shameerali.kolothum.thodi@huawei.com>; kevin.tian@intel.com;
> alex.williamson@redhat.com; dan.carpenter@linaro.org
> Cc: kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> brett.creeley@amd.com; shannon.nelson@amd.com
> Subject: [PATCH v2 vfio 2/3] pds/vfio: Fix mutex lock->magic != lock warning
> 
> The following BUG was found when running on a kernel with
> CONFIG_DEBUG_MUTEXES=y set:
> 
> DEBUG_LOCKS_WARN_ON(lock->magic != lock)
> RIP: 0010:mutex_trylock+0x10d/0x120
> Call Trace:
>  <TASK>
>  ? __warn+0x85/0x140
>  ? mutex_trylock+0x10d/0x120
>  ? report_bug+0xfc/0x1e0
>  ? handle_bug+0x3f/0x70
>  ? exc_invalid_op+0x17/0x70
>  ? asm_exc_invalid_op+0x1a/0x20
>  ? mutex_trylock+0x10d/0x120
>  ? mutex_trylock+0x10d/0x120
>  pds_vfio_reset+0x3a/0x60 [pds_vfio_pci]
>  pci_reset_function+0x4b/0x70
>  reset_store+0x5b/0xa0
>  kernfs_fop_write_iter+0x137/0x1d0
>  vfs_write+0x2de/0x410
>  ksys_write+0x5d/0xd0
>  do_syscall_64+0x3b/0x90
>  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
> 
> As shown, lock->magic != lock. This is because
> mutex_init(&pds_vfio->state_mutex) is called in the VFIO open path. So,
> if a reset is initiated before the VFIO device is opened the mutex will
> have never been initialized. Fix this by calling
> mutex_init(&pds_vfio->state_mutex) in the VFIO init path.
> 
> Also, don't destroy the mutex on close because the device may
> be re-opened, which would cause mutex to be uninitialized. Fix this by
> implementing a driver specific vfio_device_ops.release callback that
> destroys the mutex before calling vfio_pci_core_release_dev().
> 
> Signed-off-by: Brett Creeley <brett.creeley@amd.com>
> Reviewed-by: Shannon Nelson <shannon.nelson@amd.com>

Reviewed-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>

Looks like mutex destruction logic needs to be added to HiSilicon driver as well.
From a quick look mlx5 also doesn't destroy the state_mutex.

Thanks,
Shameer
 
> ---
>  drivers/vfio/pci/pds/vfio_dev.c | 15 ++++++++++++---
>  1 file changed, 12 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/vfio/pci/pds/vfio_dev.c b/drivers/vfio/pci/pds/vfio_dev.c
> index c351f588fa13..306b1c25f016 100644
> --- a/drivers/vfio/pci/pds/vfio_dev.c
> +++ b/drivers/vfio/pci/pds/vfio_dev.c
> @@ -155,6 +155,7 @@ static int pds_vfio_init_device(struct vfio_device
> *vdev)
> 
>  	pds_vfio->vf_id = vf_id;
> 
> +	mutex_init(&pds_vfio->state_mutex);
>  	spin_lock_init(&pds_vfio->reset_lock);
> 
>  	vdev->migration_flags = VFIO_MIGRATION_STOP_COPY |
> VFIO_MIGRATION_P2P;
> @@ -170,6 +171,16 @@ static int pds_vfio_init_device(struct vfio_device
> *vdev)
>  	return 0;
>  }
> 
> +static void pds_vfio_release_device(struct vfio_device *vdev)
> +{
> +	struct pds_vfio_pci_device *pds_vfio =
> +		container_of(vdev, struct pds_vfio_pci_device,
> +			     vfio_coredev.vdev);
> +
> +	mutex_destroy(&pds_vfio->state_mutex);
> +	vfio_pci_core_release_dev(vdev);
> +}
> +
>  static int pds_vfio_open_device(struct vfio_device *vdev)
>  {
>  	struct pds_vfio_pci_device *pds_vfio =
> @@ -181,7 +192,6 @@ static int pds_vfio_open_device(struct vfio_device
> *vdev)
>  	if (err)
>  		return err;
> 
> -	mutex_init(&pds_vfio->state_mutex);
>  	pds_vfio->state = VFIO_DEVICE_STATE_RUNNING;
>  	pds_vfio->deferred_reset_state = VFIO_DEVICE_STATE_RUNNING;
> 
> @@ -201,14 +211,13 @@ static void pds_vfio_close_device(struct
> vfio_device *vdev)
>  	pds_vfio_put_save_file(pds_vfio);
>  	pds_vfio_dirty_disable(pds_vfio, true);
>  	mutex_unlock(&pds_vfio->state_mutex);
> -	mutex_destroy(&pds_vfio->state_mutex);
>  	vfio_pci_core_close_device(vdev);
>  }
> 
>  static const struct vfio_device_ops pds_vfio_ops = {
>  	.name = "pds-vfio",
>  	.init = pds_vfio_init_device,
> -	.release = vfio_pci_core_release_dev,
> +	.release = pds_vfio_release_device,
>  	.open_device = pds_vfio_open_device,
>  	.close_device = pds_vfio_close_device,
>  	.ioctl = vfio_pci_core_ioctl,
> --
> 2.17.1

Re: [PATCH v2 vfio 2/3] pds/vfio: Fix mutex lock->magic != lock warning

[Jason Gunthorpe]

On Thu, Oct 12, 2023 at 09:00:09AM +0000, Shameerali Kolothum Thodi wrote:

> > Also, don't destroy the mutex on close because the device may
> > be re-opened, which would cause mutex to be uninitialized. Fix this by
> > implementing a driver specific vfio_device_ops.release callback that
> > destroys the mutex before calling vfio_pci_core_release_dev().
> > 
> > Signed-off-by: Brett Creeley <brett.creeley@amd.com>
> > Reviewed-by: Shannon Nelson <shannon.nelson@amd.com>
> 
> Reviewed-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
> 
> Looks like mutex destruction logic needs to be added to HiSilicon driver as well.
> From a quick look mlx5 also doesn't destroy the state_mutex.

FWIW, mutex_destroy is a debugging feature, it is nice if you can do it, but
not a bug if it is missing..

Jason

Re: [PATCH v2 vfio 0/3] pds/vfio: Fixes for locking bugs

[Alex Williamson]

On Wed, 11 Oct 2023 16:01:12 -0700
Brett Creeley <brett.creeley@amd.com> wrote:

> This series contains fixes for locking bugs in the recently introduced
> pds-vfio-pci driver. There was an initial bug reported by Dan Carpenter
> at:
> 
> https://lore.kernel.org/kvm/1f9bc27b-3de9-4891-9687-ba2820c1b390@moroto.mountain/
> 
> However, more locking bugs were found when looking into the previously
> mentioned issue. So, all fixes are included in this series.
> 
> v2:
> https://lore.kernel.org/kvm/20230914191540.54946-1-brett.creeley@amd.com/
> - Trim the OOPs in the patch commit messages
> - Rework patch 3/3 to only unlock the spinlock once

I thought we determined the spinlock, and thus the atomic context, was
an arbitrary choice.  I would have figured we simply convert it to a
mutex.  Why didn't we take that route?  Thanks,

Alex

> - Destroy the state_mutex in the driver specific vfio_device_ops.release
>   callback
> 
> Brett Creeley (3):
>   pds/vfio: Fix spinlock bad magic BUG
>   pds/vfio: Fix mutex lock->magic != lock warning
>   pds/vfio: Fix possible sleep while in atomic context
> 
>  drivers/vfio/pci/pds/vfio_dev.c | 27 ++++++++++++++++++++-------
>  1 file changed, 20 insertions(+), 7 deletions(-)
> 

Re: [PATCH v2 vfio 0/3] pds/vfio: Fixes for locking bugs

[Brett Creeley]

On 10/13/2023 2:09 PM, Alex Williamson wrote:
> Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
> 
> 
> On Wed, 11 Oct 2023 16:01:12 -0700
> Brett Creeley <brett.creeley@amd.com> wrote:
> 
>> This series contains fixes for locking bugs in the recently introduced
>> pds-vfio-pci driver. There was an initial bug reported by Dan Carpenter
>> at:
>>
>> https://lore.kernel.org/kvm/1f9bc27b-3de9-4891-9687-ba2820c1b390@moroto.mountain/
>>
>> However, more locking bugs were found when looking into the previously
>> mentioned issue. So, all fixes are included in this series.
>>
>> v2:
>> https://lore.kernel.org/kvm/20230914191540.54946-1-brett.creeley@amd.com/
>> - Trim the OOPs in the patch commit messages
>> - Rework patch 3/3 to only unlock the spinlock once
> 
> I thought we determined the spinlock, and thus the atomic context, was
> an arbitrary choice.  I would have figured we simply convert it to a
> mutex.  Why didn't we take that route?  Thanks,
> 
> Alex

Hmm. I guess it wasn't completely clear that was the expected solution 
even after Jason's response. However, I can resubmit a v3 with that change.

Thanks,

Brett
> 
>> - Destroy the state_mutex in the driver specific vfio_device_ops.release
>>    callback
>>
>> Brett Creeley (3):
>>    pds/vfio: Fix spinlock bad magic BUG
>>    pds/vfio: Fix mutex lock->magic != lock warning
>>    pds/vfio: Fix possible sleep while in atomic context
>>
>>   drivers/vfio/pci/pds/vfio_dev.c | 27 ++++++++++++++++++++-------
>>   1 file changed, 20 insertions(+), 7 deletions(-)
>>
>