[PATCHv3 0/3] vfio/mdev: Improve vfio/mdev core module

[PATCHv3 0/3] vfio/mdev: Improve vfio/mdev core module

[Parav Pandit]

As we would like to use mdev subsystem for wider use case as
discussed in [1], [2] apart from an offline discussion.
This use case is also discussed with wider forum in [4] in track
'Lightweight NIC HW functions for container offload use cases'.

This series is prep-work and improves vfio/mdev module in following ways.

Patch-1 Improves the mdev create/remove sequence to match Linux
bus, device model
Patch-2 Avoid recreating remove file on stale device to eliminate
call trace
Patch-3 Fix race conditions of create/remove with parent removal.
This is improved version than using srcu as srcu can take seconds
to minutes.

This series is tested using
(a) mtty with VM using vfio_mdev driver for positive tests and device
removal while device in use by VM using vfio_mdev driver.

(b) mlx5 core driver using RFC patches [3] and internal patches.
Internal patches are large and cannot be combined with this prep-work
patches. It will posted once prep-work completes.

[1] https://www.spinics.net/lists/netdev/msg556978.html
[2] https://lkml.org/lkml/2019/3/7/696
[3] https://lkml.org/lkml/2019/3/8/819
[4] https://netdevconf.org/0x13/session.html?workshop-hardware-offload

---
Changelog:
---
v2->v3:
 - Addressed comment from Cornelia
 - Corrected several errors in commit log, updated commit log
 - Dropped already merged 7 patches
v1->v2:
 - Addressed comments from Alex
 - Rebased
 - Inserted the device checking loop in Patch-6 as original code
 - Added patch 7 to 10
 - Added fixes for race condition in create/remove with parent removal
   Patch-10 uses simplified refcount and completion, instead of srcu
   which might take seconds to minutes on busy system.
 - Added fix for device create/remove sequence to match
   Linux device, bus model
v0->v1:
 - Dropped device placement on bus sequence patch for this series
 - Addressed below comments from Alex, Kirti, Maxim.
 - Added Review-by tag for already reviewed patches.
 - Dropped incorrect patch of put_device().
 - Corrected Fixes commit tag for sysfs remove sequence fix
 - Split last 8th patch to smaller refactor and fixes patch
 - Following coding style commenting format
 - Fixed accidental delete of mutex_lock in mdev_unregister_device
 - Renamed remove helped to mdev_device_remove_common().
 - Rebased for uuid/guid change

Parav Pandit (3):
  vfio/mdev: Improve the create/remove sequence
  vfio/mdev: Avoid creating sysfs remove file on stale device removal
  vfio/mdev: Synchronize device create/remove with parent removal

 drivers/vfio/mdev/mdev_core.c    | 150 ++++++++++++++-----------------
 drivers/vfio/mdev/mdev_private.h |   8 +-
 drivers/vfio/mdev/mdev_sysfs.c   |   6 +-
 3 files changed, 73 insertions(+), 91 deletions(-)

-- 
2.19.2

[PATCHv3 1/3] vfio/mdev: Improve the create/remove sequence

[Parav Pandit]

This patch addresses below two issues and prepares the code to address
3rd issue listed below.

1. mdev device is placed on the mdev bus before it is created in the
vendor driver. Once a device is placed on the mdev bus without creating
its supporting underlying vendor device, mdev driver's probe() gets triggered.
However there isn't a stable mdev available to work on.

   create_store()
     mdev_create_device()
       device_register()
          ...
         vfio_mdev_probe()
        [...]
        parent->ops->create()
          vfio_ap_mdev_create()
            mdev_set_drvdata(mdev, matrix_mdev);
            /* Valid pointer set above */

Due to this way of initialization, mdev driver who wants to use the mdev,
doesn't have a valid mdev to work on.

2. Current creation sequence is,
   parent->ops_create()
   groups_register()

Remove sequence is,
   parent->ops->remove()
   groups_unregister()

However, remove sequence should be exact mirror of creation sequence.
Once this is achieved, all users of the mdev will be terminated first
before removing underlying vendor device.
(Follow standard linux driver model).
At that point vendor's remove() ops shouldn't fail because taking the
device off the bus should terminate any usage.

3. When remove operation fails, mdev sysfs removal attempts to add the
file back on already removed device. Following call trace [1] is observed.

[1] call trace:
kernel: WARNING: CPU: 2 PID: 9348 at fs/sysfs/file.c:327 sysfs_create_file_ns+0x7f/0x90
kernel: CPU: 2 PID: 9348 Comm: bash Kdump: loaded Not tainted 5.1.0-rc6-vdevbus+ #6
kernel: Hardware name: Supermicro SYS-6028U-TR4+/X10DRU-i+, BIOS 2.0b 08/09/2016
kernel: RIP: 0010:sysfs_create_file_ns+0x7f/0x90
kernel: Call Trace:
kernel: remove_store+0xdc/0x100 [mdev]
kernel: kernfs_fop_write+0x113/0x1a0
kernel: vfs_write+0xad/0x1b0
kernel: ksys_write+0x5a/0xe0
kernel: do_syscall_64+0x5a/0x210
kernel: entry_SYSCALL_64_after_hwframe+0x49/0xbe

Therefore, mdev core is improved in following ways.

1. Split the device registration/deregistration sequence so that some
things can be done between initialization of the device and
hooking it up to the bus respectively after deregistering it
from the bus but before giving up our final reference.
In particular, this means invoking the ->create and ->remove
callbacks in those new windows. This gives the vendor driver an
initialized mdev device to work with during creation.
At the same time, a bus driver who wish to bind to mdev driver also
gets initialized mdev device.

This follows standard Linux kernel bus and device model.

2. During remove flow, first remove the device from the bus. This
ensures that any bus specific devices are removed.
Once device is taken off the mdev bus, invoke remove() of mdev
from the vendor driver.

3. The driver core device model provides way to register and auto
unregister the device sysfs attribute groups at dev->groups.
Make use of dev->groups to let core create the groups and eliminate
code to avoid explicit groups creation and removal.

To ensure, that new sequence is solid, a below stack dump of a
process is taken who attempts to remove the device while device is in
use by vfio driver and user application.
This stack dump validates that vfio driver guards against such device
removal when device is in use.

 cat /proc/21962/stack
[<0>] vfio_del_group_dev+0x216/0x3c0 [vfio]
[<0>] mdev_remove+0x21/0x40 [mdev]
[<0>] device_release_driver_internal+0xe8/0x1b0
[<0>] bus_remove_device+0xf9/0x170
[<0>] device_del+0x168/0x350
[<0>] mdev_device_remove_common+0x1d/0x50 [mdev]
[<0>] mdev_device_remove+0x8c/0xd0 [mdev]
[<0>] remove_store+0x71/0x90 [mdev]
[<0>] kernfs_fop_write+0x113/0x1a0
[<0>] vfs_write+0xad/0x1b0
[<0>] ksys_write+0x5a/0xe0
[<0>] do_syscall_64+0x5a/0x210
[<0>] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[<0>] 0xffffffffffffffff

This prepares the code to eliminate calling device_create_file() in
subsquent patch.

Signed-off-by: Parav Pandit <parav@mellanox.com>
---
 drivers/vfio/mdev/mdev_core.c    | 94 +++++++++-----------------------
 drivers/vfio/mdev/mdev_private.h |  2 +-
 drivers/vfio/mdev/mdev_sysfs.c   |  2 +-
 3 files changed, 27 insertions(+), 71 deletions(-)

diff --git a/drivers/vfio/mdev/mdev_core.c b/drivers/vfio/mdev/mdev_core.c
index 3cc1a05fde1c..0bef0cae1d4b 100644
--- a/drivers/vfio/mdev/mdev_core.c
+++ b/drivers/vfio/mdev/mdev_core.c
@@ -102,55 +102,10 @@ static void mdev_put_parent(struct mdev_parent *parent)
 		kref_put(&parent->ref, mdev_release_parent);
 }
 
-static int mdev_device_create_ops(struct kobject *kobj,
-				  struct mdev_device *mdev)
-{
-	struct mdev_parent *parent = mdev->parent;
-	int ret;
-
-	ret = parent->ops->create(kobj, mdev);
-	if (ret)
-		return ret;
-
-	ret = sysfs_create_groups(&mdev->dev.kobj,
-				  parent->ops->mdev_attr_groups);
-	if (ret)
-		parent->ops->remove(mdev);
-
-	return ret;
-}
-
-/*
- * mdev_device_remove_ops gets called from sysfs's 'remove' and when parent
- * device is being unregistered from mdev device framework.
- * - 'force_remove' is set to 'false' when called from sysfs's 'remove' which
- *   indicates that if the mdev device is active, used by VMM or userspace
- *   application, vendor driver could return error then don't remove the device.
- * - 'force_remove' is set to 'true' when called from mdev_unregister_device()
- *   which indicate that parent device is being removed from mdev device
- *   framework so remove mdev device forcefully.
- */
-static int mdev_device_remove_ops(struct mdev_device *mdev, bool force_remove)
-{
-	struct mdev_parent *parent = mdev->parent;
-	int ret;
-
-	/*
-	 * Vendor driver can return error if VMM or userspace application is
-	 * using this mdev device.
-	 */
-	ret = parent->ops->remove(mdev);
-	if (ret && !force_remove)
-		return ret;
-
-	sysfs_remove_groups(&mdev->dev.kobj, parent->ops->mdev_attr_groups);
-	return 0;
-}
-
 static int mdev_device_remove_cb(struct device *dev, void *data)
 {
 	if (dev_is_mdev(dev))
-		mdev_device_remove(dev, true);
+		mdev_device_remove(dev);
 
 	return 0;
 }
@@ -310,41 +265,43 @@ int mdev_device_create(struct kobject *kobj,
 
 	mdev->parent = parent;
 
+	device_initialize(&mdev->dev);
 	mdev->dev.parent  = dev;
 	mdev->dev.bus     = &mdev_bus_type;
 	mdev->dev.release = mdev_device_release;
 	dev_set_name(&mdev->dev, "%pUl", uuid);
+	mdev->dev.groups = parent->ops->mdev_attr_groups;
+	mdev->type_kobj = kobj;
 
-	ret = device_register(&mdev->dev);
-	if (ret) {
-		put_device(&mdev->dev);
-		goto mdev_fail;
-	}
+	ret = parent->ops->create(kobj, mdev);
+	if (ret)
+		goto ops_create_fail;
 
-	ret = mdev_device_create_ops(kobj, mdev);
+	ret = device_add(&mdev->dev);
 	if (ret)
-		goto create_fail;
+		goto add_fail;
 
 	ret = mdev_create_sysfs_files(&mdev->dev, type);
-	if (ret) {
-		mdev_device_remove_ops(mdev, true);
-		goto create_fail;
-	}
+	if (ret)
+		goto sysfs_fail;
 
-	mdev->type_kobj = kobj;
 	mdev->active = true;
 	dev_dbg(&mdev->dev, "MDEV: created\n");
 
 	return 0;
 
-create_fail:
-	device_unregister(&mdev->dev);
+sysfs_fail:
+	device_del(&mdev->dev);
+add_fail:
+	parent->ops->remove(mdev);
+ops_create_fail:
+	put_device(&mdev->dev);
 mdev_fail:
 	mdev_put_parent(parent);
 	return ret;
 }
 
-int mdev_device_remove(struct device *dev, bool force_remove)
+int mdev_device_remove(struct device *dev)
 {
 	struct mdev_device *mdev, *tmp;
 	struct mdev_parent *parent;
@@ -373,16 +330,15 @@ int mdev_device_remove(struct device *dev, bool force_remove)
 	mutex_unlock(&mdev_list_lock);
 
 	type = to_mdev_type(mdev->type_kobj);
+	mdev_remove_sysfs_files(dev, type);
+	device_del(&mdev->dev);
 	parent = mdev->parent;
+	ret = parent->ops->remove(mdev);
+	if (ret)
+		dev_err(&mdev->dev, "Remove failed: err=%d\n", ret);
 
-	ret = mdev_device_remove_ops(mdev, force_remove);
-	if (ret) {
-		mdev->active = true;
-		return ret;
-	}
-
-	mdev_remove_sysfs_files(dev, type);
-	device_unregister(dev);
+	/* Balances with device_initialize() */
+	put_device(&mdev->dev);
 	mdev_put_parent(parent);
 
 	return 0;
diff --git a/drivers/vfio/mdev/mdev_private.h b/drivers/vfio/mdev/mdev_private.h
index 36cbbdb754de..924ed2274941 100644
--- a/drivers/vfio/mdev/mdev_private.h
+++ b/drivers/vfio/mdev/mdev_private.h
@@ -60,6 +60,6 @@ void mdev_remove_sysfs_files(struct device *dev, struct mdev_type *type);
 
 int  mdev_device_create(struct kobject *kobj,
 			struct device *dev, const guid_t *uuid);
-int  mdev_device_remove(struct device *dev, bool force_remove);
+int  mdev_device_remove(struct device *dev);
 
 #endif /* MDEV_PRIVATE_H */
diff --git a/drivers/vfio/mdev/mdev_sysfs.c b/drivers/vfio/mdev/mdev_sysfs.c
index cbf94b8165ea..9f774b91d275 100644
--- a/drivers/vfio/mdev/mdev_sysfs.c
+++ b/drivers/vfio/mdev/mdev_sysfs.c
@@ -236,7 +236,7 @@ static ssize_t remove_store(struct device *dev, struct device_attribute *attr,
 	if (val && device_remove_file_self(dev, attr)) {
 		int ret;
 
-		ret = mdev_device_remove(dev, false);
+		ret = mdev_device_remove(dev);
 		if (ret) {
 			device_create_file(dev, attr);
 			return ret;
-- 
2.19.2

[PATCHv3 2/3] vfio/mdev: Avoid creating sysfs remove file on stale device removal

[Parav Pandit]

If device is removal is initiated by two threads as below, mdev core
attempts to create a syfs remove file on stale device.
During this flow, below [1] call trace is observed.

     cpu-0                                    cpu-1
     -----                                    -----
  mdev_unregister_device()
    device_for_each_child
       mdev_device_remove_cb
          mdev_device_remove
                                       user_syscall
                                         remove_store()
                                           mdev_device_remove()
                                        [..]
   unregister device();
                                       /* not found in list or
                                        * active=false.
                                        */
                                          sysfs_create_file()
                                          ..Call trace

Now that mdev core follows correct device removal system of the linux
bus model, remove shouldn't fail in normal cases. If it fails, there is
no point of creating a stale file or checking for specific error status.

kernel: WARNING: CPU: 2 PID: 9348 at fs/sysfs/file.c:327
sysfs_create_file_ns+0x7f/0x90
kernel: CPU: 2 PID: 9348 Comm: bash Kdump: loaded Not tainted
5.1.0-rc6-vdevbus+ #6
kernel: Hardware name: Supermicro SYS-6028U-TR4+/X10DRU-i+, BIOS 2.0b
08/09/2016
kernel: RIP: 0010:sysfs_create_file_ns+0x7f/0x90
kernel: Call Trace:
kernel: remove_store+0xdc/0x100 [mdev]
kernel: kernfs_fop_write+0x113/0x1a0
kernel: vfs_write+0xad/0x1b0
kernel: ksys_write+0x5a/0xe0
kernel: do_syscall_64+0x5a/0x210
kernel: entry_SYSCALL_64_after_hwframe+0x49/0xbe

Signed-off-by: Parav Pandit <parav@mellanox.com>
---
 drivers/vfio/mdev/mdev_sysfs.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/drivers/vfio/mdev/mdev_sysfs.c b/drivers/vfio/mdev/mdev_sysfs.c
index 9f774b91d275..ffa3dcebf201 100644
--- a/drivers/vfio/mdev/mdev_sysfs.c
+++ b/drivers/vfio/mdev/mdev_sysfs.c
@@ -237,10 +237,8 @@ static ssize_t remove_store(struct device *dev, struct device_attribute *attr,
 		int ret;
 
 		ret = mdev_device_remove(dev);
-		if (ret) {
-			device_create_file(dev, attr);
+		if (ret)
 			return ret;
-		}
 	}
 
 	return count;
-- 
2.19.2

[PATCHv3 3/3] vfio/mdev: Synchronize device create/remove with parent removal

[Parav Pandit]

In following sequences, child devices created while removing mdev parent
device can be left out, or it may lead to race of removing half
initialized child mdev devices.

issue-1:
--------
       cpu-0                         cpu-1
       -----                         -----
                                  mdev_unregister_device()
                                    device_for_each_child()
                                      mdev_device_remove_cb()
                                        mdev_device_remove()
create_store()
  mdev_device_create()                   [...]
    device_add()
                                  parent_remove_sysfs_files()

/* BUG: device added by cpu-0
 * whose parent is getting removed
 * and it won't process this mdev.
 */

issue-2:
--------
Below crash is observed when user initiated remove is in progress
and mdev_unregister_driver() completes parent unregistration.

       cpu-0                         cpu-1
       -----                         -----
remove_store()
   mdev_device_remove()
   active = false;
                                  mdev_unregister_device()
                                  parent device removed.
   [...]
   parents->ops->remove()
 /*
  * BUG: Accessing invalid parent.
  */

This is similar race like create() racing with mdev_unregister_device().

BUG: unable to handle kernel paging request at ffffffffc0585668
PGD e8f618067 P4D e8f618067 PUD e8f61a067 PMD 85adca067 PTE 0
Oops: 0000 [#1] SMP PTI
CPU: 41 PID: 37403 Comm: bash Kdump: loaded Not tainted 5.1.0-rc6-vdevbus+ #6
Hardware name: Supermicro SYS-6028U-TR4+/X10DRU-i+, BIOS 2.0b 08/09/2016
RIP: 0010:mdev_device_remove+0xfa/0x140 [mdev]
Call Trace:
 remove_store+0x71/0x90 [mdev]
 kernfs_fop_write+0x113/0x1a0
 vfs_write+0xad/0x1b0
 ksys_write+0x5a/0xe0
 do_syscall_64+0x5a/0x210
 entry_SYSCALL_64_after_hwframe+0x49/0xbe

Therefore, mdev core is improved as below to overcome above issues.

Wait for any ongoing mdev create() and remove() to finish before
unregistering parent device using refcount and completion.
This continues to allow multiple create and remove to progress in
parallel for different mdev devices as most common case.
At the same time guard parent removal while parent is being access by
create() and remove callbacks.

Code is simplified from kref to use refcount as unregister_device() has
to wait anyway for all create/remove to finish.

While removing mdev devices during parent unregistration, there isn't
need to acquire refcount of parent device, hence code is restructured
using mdev_device_remove_common() to avoid it.

Fixes: 7b96953bc640 ("vfio: Mediated device Core driver")
Signed-off-by: Parav Pandit <parav@mellanox.com>
---
 drivers/vfio/mdev/mdev_core.c    | 86 ++++++++++++++++++++------------
 drivers/vfio/mdev/mdev_private.h |  6 ++-
 2 files changed, 60 insertions(+), 32 deletions(-)

diff --git a/drivers/vfio/mdev/mdev_core.c b/drivers/vfio/mdev/mdev_core.c
index 0bef0cae1d4b..ca33246c1dc3 100644
--- a/drivers/vfio/mdev/mdev_core.c
+++ b/drivers/vfio/mdev/mdev_core.c
@@ -78,34 +78,41 @@ static struct mdev_parent *__find_parent_device(struct device *dev)
 	return NULL;
 }
 
-static void mdev_release_parent(struct kref *kref)
+static bool mdev_try_get_parent(struct mdev_parent *parent)
 {
-	struct mdev_parent *parent = container_of(kref, struct mdev_parent,
-						  ref);
-	struct device *dev = parent->dev;
-
-	kfree(parent);
-	put_device(dev);
+	if (parent)
+		return refcount_inc_not_zero(&parent->refcount);
+	return false;
 }
 
-static struct mdev_parent *mdev_get_parent(struct mdev_parent *parent)
+static void mdev_put_parent(struct mdev_parent *parent)
 {
-	if (parent)
-		kref_get(&parent->ref);
-
-	return parent;
+	if (parent && refcount_dec_and_test(&parent->refcount))
+		complete(&parent->unreg_completion);
 }
 
-static void mdev_put_parent(struct mdev_parent *parent)
+static void mdev_device_remove_common(struct mdev_device *mdev)
 {
-	if (parent)
-		kref_put(&parent->ref, mdev_release_parent);
+	struct mdev_parent *parent;
+	struct mdev_type *type;
+	int ret;
+
+	type = to_mdev_type(mdev->type_kobj);
+	mdev_remove_sysfs_files(&mdev->dev, type);
+	device_del(&mdev->dev);
+	parent = mdev->parent;
+	ret = parent->ops->remove(mdev);
+	if (ret)
+		dev_err(&mdev->dev, "Remove failed: err=%d\n", ret);
+
+	/* Balances with device_initialize() */
+	put_device(&mdev->dev);
 }
 
 static int mdev_device_remove_cb(struct device *dev, void *data)
 {
 	if (dev_is_mdev(dev))
-		mdev_device_remove(dev);
+		mdev_device_remove_common(to_mdev_device(dev));
 
 	return 0;
 }
@@ -147,7 +154,8 @@ int mdev_register_device(struct device *dev, const struct mdev_parent_ops *ops)
 		goto add_dev_err;
 	}
 
-	kref_init(&parent->ref);
+	refcount_set(&parent->refcount, 1);
+	init_completion(&parent->unreg_completion);
 
 	parent->dev = dev;
 	parent->ops = ops;
@@ -206,14 +214,27 @@ void mdev_unregister_device(struct device *dev)
 	dev_info(dev, "MDEV: Unregistering\n");
 
 	list_del(&parent->next);
+	mutex_unlock(&parent_list_lock);
+
+	/* Release the initial reference so that new create cannot start */
+	mdev_put_parent(parent);
+
+	/*
+	 * Wait for all the create and remove references to drop.
+	 */
+	wait_for_completion(&parent->unreg_completion);
+
+	/*
+	 * New references cannot be taken and all users are done
+	 * using the parent. So it is safe to unregister parent.
+	 */
 	class_compat_remove_link(mdev_bus_compat_class, dev, NULL);
 
 	device_for_each_child(dev, NULL, mdev_device_remove_cb);
 
 	parent_remove_sysfs_files(parent);
-
-	mutex_unlock(&parent_list_lock);
-	mdev_put_parent(parent);
+	kfree(parent);
+	put_device(dev);
 }
 EXPORT_SYMBOL(mdev_unregister_device);
 
@@ -237,10 +258,11 @@ int mdev_device_create(struct kobject *kobj,
 	struct mdev_parent *parent;
 	struct mdev_type *type = to_mdev_type(kobj);
 
-	parent = mdev_get_parent(type->parent);
-	if (!parent)
+	if (!mdev_try_get_parent(type->parent))
 		return -EINVAL;
 
+	parent = type->parent;
+
 	mutex_lock(&mdev_list_lock);
 
 	/* Check for duplicate */
@@ -287,6 +309,7 @@ int mdev_device_create(struct kobject *kobj,
 
 	mdev->active = true;
 	dev_dbg(&mdev->dev, "MDEV: created\n");
+	mdev_put_parent(parent);
 
 	return 0;
 
@@ -306,7 +329,6 @@ int mdev_device_remove(struct device *dev)
 	struct mdev_device *mdev, *tmp;
 	struct mdev_parent *parent;
 	struct mdev_type *type;
-	int ret;
 
 	mdev = to_mdev_device(dev);
 
@@ -330,15 +352,17 @@ int mdev_device_remove(struct device *dev)
 	mutex_unlock(&mdev_list_lock);
 
 	type = to_mdev_type(mdev->type_kobj);
-	mdev_remove_sysfs_files(dev, type);
-	device_del(&mdev->dev);
-	parent = mdev->parent;
-	ret = parent->ops->remove(mdev);
-	if (ret)
-		dev_err(&mdev->dev, "Remove failed: err=%d\n", ret);
+	if (!mdev_try_get_parent(type->parent)) {
+		/*
+		 * Parent unregistration have started.
+		 * No need to remove here.
+		 */
+		mutex_unlock(&mdev_list_lock);
+		return -ENODEV;
+	}
 
-	/* Balances with device_initialize() */
-	put_device(&mdev->dev);
+	parent = mdev->parent;
+	mdev_device_remove_common(mdev);
 	mdev_put_parent(parent);
 
 	return 0;
diff --git a/drivers/vfio/mdev/mdev_private.h b/drivers/vfio/mdev/mdev_private.h
index 924ed2274941..55ebab0af7b0 100644
--- a/drivers/vfio/mdev/mdev_private.h
+++ b/drivers/vfio/mdev/mdev_private.h
@@ -19,7 +19,11 @@ void mdev_bus_unregister(void);
 struct mdev_parent {
 	struct device *dev;
 	const struct mdev_parent_ops *ops;
-	struct kref ref;
+	/* Protects unregistration to wait until create/remove
+	 * are completed.
+	 */
+	refcount_t refcount;
+	struct completion unreg_completion;
 	struct list_head next;
 	struct kset *mdev_types_kset;
 	struct list_head type_list;
-- 
2.19.2

Re: [PATCHv3 3/3] vfio/mdev: Synchronize device create/remove with parent removal

[Cornelia Huck]

On Thu, 16 May 2019 18:30:34 -0500
Parav Pandit <parav@mellanox.com> wrote:

> In following sequences, child devices created while removing mdev parent
> device can be left out, or it may lead to race of removing half
> initialized child mdev devices.
> 
> issue-1:
> --------
>        cpu-0                         cpu-1
>        -----                         -----
>                                   mdev_unregister_device()
>                                     device_for_each_child()
>                                       mdev_device_remove_cb()
>                                         mdev_device_remove()
> create_store()
>   mdev_device_create()                   [...]
>     device_add()
>                                   parent_remove_sysfs_files()
> 
> /* BUG: device added by cpu-0
>  * whose parent is getting removed
>  * and it won't process this mdev.
>  */
> 
> issue-2:
> --------
> Below crash is observed when user initiated remove is in progress
> and mdev_unregister_driver() completes parent unregistration.
> 
>        cpu-0                         cpu-1
>        -----                         -----
> remove_store()
>    mdev_device_remove()
>    active = false;
>                                   mdev_unregister_device()
>                                   parent device removed.
>    [...]
>    parents->ops->remove()
>  /*
>   * BUG: Accessing invalid parent.
>   */
> 
> This is similar race like create() racing with mdev_unregister_device().
> 
> BUG: unable to handle kernel paging request at ffffffffc0585668
> PGD e8f618067 P4D e8f618067 PUD e8f61a067 PMD 85adca067 PTE 0
> Oops: 0000 [#1] SMP PTI
> CPU: 41 PID: 37403 Comm: bash Kdump: loaded Not tainted 5.1.0-rc6-vdevbus+ #6
> Hardware name: Supermicro SYS-6028U-TR4+/X10DRU-i+, BIOS 2.0b 08/09/2016
> RIP: 0010:mdev_device_remove+0xfa/0x140 [mdev]
> Call Trace:
>  remove_store+0x71/0x90 [mdev]
>  kernfs_fop_write+0x113/0x1a0
>  vfs_write+0xad/0x1b0
>  ksys_write+0x5a/0xe0
>  do_syscall_64+0x5a/0x210
>  entry_SYSCALL_64_after_hwframe+0x49/0xbe
> 
> Therefore, mdev core is improved as below to overcome above issues.
> 
> Wait for any ongoing mdev create() and remove() to finish before
> unregistering parent device using refcount and completion.
> This continues to allow multiple create and remove to progress in
> parallel for different mdev devices as most common case.
> At the same time guard parent removal while parent is being access by
> create() and remove callbacks.
> 
> Code is simplified from kref to use refcount as unregister_device() has
> to wait anyway for all create/remove to finish.
> 
> While removing mdev devices during parent unregistration, there isn't
> need to acquire refcount of parent device, hence code is restructured
> using mdev_device_remove_common() to avoid it.
> 
> Fixes: 7b96953bc640 ("vfio: Mediated device Core driver")
> Signed-off-by: Parav Pandit <parav@mellanox.com>
> ---
>  drivers/vfio/mdev/mdev_core.c    | 86 ++++++++++++++++++++------------
>  drivers/vfio/mdev/mdev_private.h |  6 ++-
>  2 files changed, 60 insertions(+), 32 deletions(-)

I'm still not quite happy with this patch. I think most of my dislike
comes from how you are using a member called 'refcount' vs. what I
believe a refcount actually is. See below.

> 
> diff --git a/drivers/vfio/mdev/mdev_core.c b/drivers/vfio/mdev/mdev_core.c
> index 0bef0cae1d4b..ca33246c1dc3 100644
> --- a/drivers/vfio/mdev/mdev_core.c
> +++ b/drivers/vfio/mdev/mdev_core.c
> @@ -78,34 +78,41 @@ static struct mdev_parent *__find_parent_device(struct device *dev)
>  	return NULL;
>  }
>  
> -static void mdev_release_parent(struct kref *kref)
> +static bool mdev_try_get_parent(struct mdev_parent *parent)
>  {
> -	struct mdev_parent *parent = container_of(kref, struct mdev_parent,
> -						  ref);
> -	struct device *dev = parent->dev;
> -
> -	kfree(parent);
> -	put_device(dev);
> +	if (parent)
> +		return refcount_inc_not_zero(&parent->refcount);
> +	return false;
>  }
>  
> -static struct mdev_parent *mdev_get_parent(struct mdev_parent *parent)
> +static void mdev_put_parent(struct mdev_parent *parent)
>  {
> -	if (parent)
> -		kref_get(&parent->ref);
> -
> -	return parent;
> +	if (parent && refcount_dec_and_test(&parent->refcount))
> +		complete(&parent->unreg_completion);
>  }

So far, this is "obtain a reference if the reference is not 0 (implying
the object is not ready to use) and notify waiters when the last
reference is dropped". This still looks idiomatic enough.

>  
> -static void mdev_put_parent(struct mdev_parent *parent)
> +static void mdev_device_remove_common(struct mdev_device *mdev)
>  {
> -	if (parent)
> -		kref_put(&parent->ref, mdev_release_parent);
> +	struct mdev_parent *parent;
> +	struct mdev_type *type;
> +	int ret;
> +
> +	type = to_mdev_type(mdev->type_kobj);
> +	mdev_remove_sysfs_files(&mdev->dev, type);
> +	device_del(&mdev->dev);
> +	parent = mdev->parent;
> +	ret = parent->ops->remove(mdev);
> +	if (ret)
> +		dev_err(&mdev->dev, "Remove failed: err=%d\n", ret);
> +
> +	/* Balances with device_initialize() */
> +	put_device(&mdev->dev);
>  }
>  
>  static int mdev_device_remove_cb(struct device *dev, void *data)
>  {
>  	if (dev_is_mdev(dev))
> -		mdev_device_remove(dev);
> +		mdev_device_remove_common(to_mdev_device(dev));
>  
>  	return 0;
>  }
> @@ -147,7 +154,8 @@ int mdev_register_device(struct device *dev, const struct mdev_parent_ops *ops)
>  		goto add_dev_err;
>  	}
>  
> -	kref_init(&parent->ref);
> +	refcount_set(&parent->refcount, 1);

Initializing to 1 when creating is also fine.

> +	init_completion(&parent->unreg_completion);
>  
>  	parent->dev = dev;
>  	parent->ops = ops;
> @@ -206,14 +214,27 @@ void mdev_unregister_device(struct device *dev)
>  	dev_info(dev, "MDEV: Unregistering\n");
>  
>  	list_del(&parent->next);
> +	mutex_unlock(&parent_list_lock);
> +
> +	/* Release the initial reference so that new create cannot start */
> +	mdev_put_parent(parent);

The comment is confusing: We do drop one reference, but this does not
imply we're going to 0 (which would be the one thing that would block
creating new devices).

> +
> +	/*
> +	 * Wait for all the create and remove references to drop.
> +	 */
> +	wait_for_completion(&parent->unreg_completion);

It only reaches 0 after this wait.

> +
> +	/*
> +	 * New references cannot be taken and all users are done
> +	 * using the parent. So it is safe to unregister parent.
> +	 */
>  	class_compat_remove_link(mdev_bus_compat_class, dev, NULL);
>  
>  	device_for_each_child(dev, NULL, mdev_device_remove_cb);
>  
>  	parent_remove_sysfs_files(parent);
> -
> -	mutex_unlock(&parent_list_lock);
> -	mdev_put_parent(parent);
> +	kfree(parent);
> +	put_device(dev);
>  }
>  EXPORT_SYMBOL(mdev_unregister_device);
>  
> @@ -237,10 +258,11 @@ int mdev_device_create(struct kobject *kobj,
>  	struct mdev_parent *parent;
>  	struct mdev_type *type = to_mdev_type(kobj);
>  
> -	parent = mdev_get_parent(type->parent);
> -	if (!parent)
> +	if (!mdev_try_get_parent(type->parent))

If other calls are still running, the refcount won't be 0, and this
will succeed, even if we really want to get rid of the device.

>  		return -EINVAL;
>  
> +	parent = type->parent;
> +
>  	mutex_lock(&mdev_list_lock);
>  
>  	/* Check for duplicate */
> @@ -287,6 +309,7 @@ int mdev_device_create(struct kobject *kobj,
>  
>  	mdev->active = true;
>  	dev_dbg(&mdev->dev, "MDEV: created\n");
> +	mdev_put_parent(parent);
>  
>  	return 0;
>  
> @@ -306,7 +329,6 @@ int mdev_device_remove(struct device *dev)
>  	struct mdev_device *mdev, *tmp;
>  	struct mdev_parent *parent;
>  	struct mdev_type *type;
> -	int ret;
>  
>  	mdev = to_mdev_device(dev);
>  
> @@ -330,15 +352,17 @@ int mdev_device_remove(struct device *dev)
>  	mutex_unlock(&mdev_list_lock);
>  
>  	type = to_mdev_type(mdev->type_kobj);
> -	mdev_remove_sysfs_files(dev, type);
> -	device_del(&mdev->dev);
> -	parent = mdev->parent;
> -	ret = parent->ops->remove(mdev);
> -	if (ret)
> -		dev_err(&mdev->dev, "Remove failed: err=%d\n", ret);
> +	if (!mdev_try_get_parent(type->parent)) {

Same here: Is there really a guarantee that the refcount is 0 when the
parent is going away?

> +		/*
> +		 * Parent unregistration have started.
> +		 * No need to remove here.
> +		 */
> +		mutex_unlock(&mdev_list_lock);

Btw., you already unlocked above.

> +		return -ENODEV;
> +	}
>  
> -	/* Balances with device_initialize() */
> -	put_device(&mdev->dev);
> +	parent = mdev->parent;
> +	mdev_device_remove_common(mdev);
>  	mdev_put_parent(parent);
>  
>  	return 0;
> diff --git a/drivers/vfio/mdev/mdev_private.h b/drivers/vfio/mdev/mdev_private.h
> index 924ed2274941..55ebab0af7b0 100644
> --- a/drivers/vfio/mdev/mdev_private.h
> +++ b/drivers/vfio/mdev/mdev_private.h
> @@ -19,7 +19,11 @@ void mdev_bus_unregister(void);
>  struct mdev_parent {
>  	struct device *dev;
>  	const struct mdev_parent_ops *ops;
> -	struct kref ref;
> +	/* Protects unregistration to wait until create/remove
> +	 * are completed.
> +	 */
> +	refcount_t refcount;
> +	struct completion unreg_completion;
>  	struct list_head next;
>  	struct kset *mdev_types_kset;
>  	struct list_head type_list;

I think what's really needed is to split up the different needs and not
overload the 'refcount' concept.

- If we need to make sure that a reference to the parent is held so
  that the parent may not go away while still in use, we should
  continue to use the kref (in the idiomatic way it is used before this
  patch.)
- We need to protect against creation of new devices if the parent is
  going away. Maybe set a going_away marker in the parent structure for
  that so that creation bails out immediately? What happens if the
  creation has already started when parent removal kicks in, though?
  Do we need some child list locking and an indication whether a child
  is in progress of being registered/unregistered?
- We also need to protect against removal of devices while unregister
  is in progress (same mechanism as above?) The second issue you
  describe above should be fixed then if the children keep a reference
  of the parent.

RE: [PATCHv3 3/3] vfio/mdev: Synchronize device create/remove with parent removal

[Parav Pandit]

Hi Cornelia,

> -----Original Message-----
> From: Cornelia Huck <cohuck@redhat.com>
> Sent: Friday, May 17, 2019 6:22 AM
> To: Parav Pandit <parav@mellanox.com>
> Cc: kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> kwankhede@nvidia.com; alex.williamson@redhat.com; cjia@nvidia.com
> Subject: Re: [PATCHv3 3/3] vfio/mdev: Synchronize device create/remove
> with parent removal
> 
> On Thu, 16 May 2019 18:30:34 -0500
> Parav Pandit <parav@mellanox.com> wrote:
> 
> > In following sequences, child devices created while removing mdev
> > parent device can be left out, or it may lead to race of removing half
> > initialized child mdev devices.
> >
> > issue-1:
> > --------
> >        cpu-0                         cpu-1
> >        -----                         -----
> >                                   mdev_unregister_device()
> >                                     device_for_each_child()
> >                                       mdev_device_remove_cb()
> >                                         mdev_device_remove()
> > create_store()
> >   mdev_device_create()                   [...]
> >     device_add()
> >                                   parent_remove_sysfs_files()
> >
> > /* BUG: device added by cpu-0
> >  * whose parent is getting removed
> >  * and it won't process this mdev.
> >  */
> >
> > issue-2:
> > --------
> > Below crash is observed when user initiated remove is in progress and
> > mdev_unregister_driver() completes parent unregistration.
> >
> >        cpu-0                         cpu-1
> >        -----                         -----
> > remove_store()
> >    mdev_device_remove()
> >    active = false;
> >                                   mdev_unregister_device()
> >                                   parent device removed.
> >    [...]
> >    parents->ops->remove()
> >  /*
> >   * BUG: Accessing invalid parent.
> >   */
> >
> > This is similar race like create() racing with mdev_unregister_device().
> >
> > BUG: unable to handle kernel paging request at ffffffffc0585668 PGD
> > e8f618067 P4D e8f618067 PUD e8f61a067 PMD 85adca067 PTE 0
> > Oops: 0000 [#1] SMP PTI
> > CPU: 41 PID: 37403 Comm: bash Kdump: loaded Not tainted
> > 5.1.0-rc6-vdevbus+ #6 Hardware name: Supermicro
> > SYS-6028U-TR4+/X10DRU-i+, BIOS 2.0b 08/09/2016
> > RIP: 0010:mdev_device_remove+0xfa/0x140 [mdev] Call Trace:
> >  remove_store+0x71/0x90 [mdev]
> >  kernfs_fop_write+0x113/0x1a0
> >  vfs_write+0xad/0x1b0
> >  ksys_write+0x5a/0xe0
> >  do_syscall_64+0x5a/0x210
> >  entry_SYSCALL_64_after_hwframe+0x49/0xbe
> >
> > Therefore, mdev core is improved as below to overcome above issues.
> >
> > Wait for any ongoing mdev create() and remove() to finish before
> > unregistering parent device using refcount and completion.
> > This continues to allow multiple create and remove to progress in
> > parallel for different mdev devices as most common case.
> > At the same time guard parent removal while parent is being access by
> > create() and remove callbacks.
> >
> > Code is simplified from kref to use refcount as unregister_device()
> > has to wait anyway for all create/remove to finish.
> >
> > While removing mdev devices during parent unregistration, there isn't
> > need to acquire refcount of parent device, hence code is restructured
> > using mdev_device_remove_common() to avoid it.
> >
> > Fixes: 7b96953bc640 ("vfio: Mediated device Core driver")
> > Signed-off-by: Parav Pandit <parav@mellanox.com>
> > ---
> >  drivers/vfio/mdev/mdev_core.c    | 86 ++++++++++++++++++++------------
> >  drivers/vfio/mdev/mdev_private.h |  6 ++-
> >  2 files changed, 60 insertions(+), 32 deletions(-)
> 
> I'm still not quite happy with this patch. I think most of my dislike comes
> from how you are using a member called 'refcount' vs. what I believe a
> refcount actually is. See below.
>
Comments below.
 
> >
> > diff --git a/drivers/vfio/mdev/mdev_core.c
> > b/drivers/vfio/mdev/mdev_core.c index 0bef0cae1d4b..ca33246c1dc3
> > 100644
> > --- a/drivers/vfio/mdev/mdev_core.c
> > +++ b/drivers/vfio/mdev/mdev_core.c
> > @@ -78,34 +78,41 @@ static struct mdev_parent
> *__find_parent_device(struct device *dev)
> >  	return NULL;
> >  }
> >
> > -static void mdev_release_parent(struct kref *kref)
> > +static bool mdev_try_get_parent(struct mdev_parent *parent)
> >  {
> > -	struct mdev_parent *parent = container_of(kref, struct
> mdev_parent,
> > -						  ref);
> > -	struct device *dev = parent->dev;
> > -
> > -	kfree(parent);
> > -	put_device(dev);
> > +	if (parent)
> > +		return refcount_inc_not_zero(&parent->refcount);
> > +	return false;
> >  }
> >
> > -static struct mdev_parent *mdev_get_parent(struct mdev_parent
> > *parent)
> > +static void mdev_put_parent(struct mdev_parent *parent)
> >  {
> > -	if (parent)
> > -		kref_get(&parent->ref);
> > -
> > -	return parent;
> > +	if (parent && refcount_dec_and_test(&parent->refcount))
> > +		complete(&parent->unreg_completion);
> >  }
> 
> So far, this is "obtain a reference if the reference is not 0 (implying the object
> is not ready to use) and notify waiters when the last reference is dropped".
> This still looks idiomatic enough.
> 
> >
> > -static void mdev_put_parent(struct mdev_parent *parent)
> > +static void mdev_device_remove_common(struct mdev_device *mdev)
> >  {
> > -	if (parent)
> > -		kref_put(&parent->ref, mdev_release_parent);
> > +	struct mdev_parent *parent;
> > +	struct mdev_type *type;
> > +	int ret;
> > +
> > +	type = to_mdev_type(mdev->type_kobj);
> > +	mdev_remove_sysfs_files(&mdev->dev, type);
> > +	device_del(&mdev->dev);
> > +	parent = mdev->parent;
> > +	ret = parent->ops->remove(mdev);
> > +	if (ret)
> > +		dev_err(&mdev->dev, "Remove failed: err=%d\n", ret);
> > +
> > +	/* Balances with device_initialize() */
> > +	put_device(&mdev->dev);
> >  }
> >
> >  static int mdev_device_remove_cb(struct device *dev, void *data)  {
> >  	if (dev_is_mdev(dev))
> > -		mdev_device_remove(dev);
> > +		mdev_device_remove_common(to_mdev_device(dev));
> >
> >  	return 0;
> >  }
> > @@ -147,7 +154,8 @@ int mdev_register_device(struct device *dev, const
> struct mdev_parent_ops *ops)
> >  		goto add_dev_err;
> >  	}
> >
> > -	kref_init(&parent->ref);
> > +	refcount_set(&parent->refcount, 1);
> 
> Initializing to 1 when creating is also fine.
> 
> > +	init_completion(&parent->unreg_completion);
> >
> >  	parent->dev = dev;
> >  	parent->ops = ops;
> > @@ -206,14 +214,27 @@ void mdev_unregister_device(struct device *dev)
> >  	dev_info(dev, "MDEV: Unregistering\n");
> >
> >  	list_del(&parent->next);
> > +	mutex_unlock(&parent_list_lock);
> > +
> > +	/* Release the initial reference so that new create cannot start */
> > +	mdev_put_parent(parent);
> 
> The comment is confusing: We do drop one reference, but this does not
> imply we're going to 0 (which would be the one thing that would block
> creating new devices).
> 
Ok. How about below comment.
/* Balance with initial reference init */

> > +
> > +	/*
> > +	 * Wait for all the create and remove references to drop.
> > +	 */
> > +	wait_for_completion(&parent->unreg_completion);
> 
> It only reaches 0 after this wait.
>
Yes.
 
> > +
> > +	/*
> > +	 * New references cannot be taken and all users are done
> > +	 * using the parent. So it is safe to unregister parent.
> > +	 */
> >  	class_compat_remove_link(mdev_bus_compat_class, dev, NULL);
> >
> >  	device_for_each_child(dev, NULL, mdev_device_remove_cb);
> >
> >  	parent_remove_sysfs_files(parent);
> > -
> > -	mutex_unlock(&parent_list_lock);
> > -	mdev_put_parent(parent);
> > +	kfree(parent);
> > +	put_device(dev);
> >  }
> >  EXPORT_SYMBOL(mdev_unregister_device);
> >
> > @@ -237,10 +258,11 @@ int mdev_device_create(struct kobject *kobj,
> >  	struct mdev_parent *parent;
> >  	struct mdev_type *type = to_mdev_type(kobj);
> >
> > -	parent = mdev_get_parent(type->parent);
> > -	if (!parent)
> > +	if (!mdev_try_get_parent(type->parent))
> 
> If other calls are still running, the refcount won't be 0, and this will succeed,
> even if we really want to get rid of the device.
> 
Sure, if other calls are running, refcount won't be 0. Process creating them will eventually complete, and refcount will drop to zero.
And new processes won't be able to start any more.
So there is no differentiation between 'already in creation stage' and 'about to start' processes.

> >  		return -EINVAL;
> >
> > +	parent = type->parent;
> > +
> >  	mutex_lock(&mdev_list_lock);
> >
> >  	/* Check for duplicate */
> > @@ -287,6 +309,7 @@ int mdev_device_create(struct kobject *kobj,
> >
> >  	mdev->active = true;
> >  	dev_dbg(&mdev->dev, "MDEV: created\n");
> > +	mdev_put_parent(parent);
> >
> >  	return 0;
> >
> > @@ -306,7 +329,6 @@ int mdev_device_remove(struct device *dev)
> >  	struct mdev_device *mdev, *tmp;
> >  	struct mdev_parent *parent;
> >  	struct mdev_type *type;
> > -	int ret;
> >
> >  	mdev = to_mdev_device(dev);
> >
> > @@ -330,15 +352,17 @@ int mdev_device_remove(struct device *dev)
> >  	mutex_unlock(&mdev_list_lock);
> >
> >  	type = to_mdev_type(mdev->type_kobj);
> > -	mdev_remove_sysfs_files(dev, type);
> > -	device_del(&mdev->dev);
> > -	parent = mdev->parent;
> > -	ret = parent->ops->remove(mdev);
> > -	if (ret)
> > -		dev_err(&mdev->dev, "Remove failed: err=%d\n", ret);
> > +	if (!mdev_try_get_parent(type->parent)) {
> 
> Same here: Is there really a guarantee that the refcount is 0 when the parent
> is going away?
A WARN_ON after wait_for_completion or in freeing the parent is good to catch bugs.

> 
> > +		/*
> > +		 * Parent unregistration have started.
> > +		 * No need to remove here.
> > +		 */
> > +		mutex_unlock(&mdev_list_lock);
> 
> Btw., you already unlocked above.
>
You are right. This unlock is wrong. I will revise the patch.
 
> > +		return -ENODEV;
> > +	}
> >
> > -	/* Balances with device_initialize() */
> > -	put_device(&mdev->dev);
> > +	parent = mdev->parent;
> > +	mdev_device_remove_common(mdev);
> >  	mdev_put_parent(parent);
> >
> >  	return 0;
> > diff --git a/drivers/vfio/mdev/mdev_private.h
> > b/drivers/vfio/mdev/mdev_private.h
> > index 924ed2274941..55ebab0af7b0 100644
> > --- a/drivers/vfio/mdev/mdev_private.h
> > +++ b/drivers/vfio/mdev/mdev_private.h
> > @@ -19,7 +19,11 @@ void mdev_bus_unregister(void);  struct
> mdev_parent
> > {
> >  	struct device *dev;
> >  	const struct mdev_parent_ops *ops;
> > -	struct kref ref;
> > +	/* Protects unregistration to wait until create/remove
> > +	 * are completed.
> > +	 */
> > +	refcount_t refcount;
> > +	struct completion unreg_completion;
> >  	struct list_head next;
> >  	struct kset *mdev_types_kset;
> >  	struct list_head type_list;
> 
> I think what's really needed is to split up the different needs and not
> overload the 'refcount' concept.
> 
Refcount tells that how many active references are present for this parent device.
Those active reference could be create/remove processes and mdev core itself.

So when parent unregisters, mdev module publishes that it is going away through this refcount.
Hence new users cannot start.

> - If we need to make sure that a reference to the parent is held so
>   that the parent may not go away while still in use, we should
>   continue to use the kref (in the idiomatic way it is used before this
>   patch.)
> - We need to protect against creation of new devices if the parent is
>   going away. Maybe set a going_away marker in the parent structure for
>   that so that creation bails out immediately? 
Such marker will help to not start new processes.
So an additional marker can be added to improve mdev_try_get_parent().
But I couldn't justify why differentiating those two users on time scale is desired.
One reason could be that user continuously tries to create mdev and parent never gets a chance, to unregister?
I guess, parent will run out mdev devices before this can happen.

Additionally a stop marker is needed (counter) to tell that all users are done accessing it.
Both purposes are served using a refcount scheme.

> What happens if the
>   creation has already started when parent removal kicks in, though?
That particular creation will succeed but newer cannot start, because mdev_put_parent() is done.

>   Do we need some child list locking and an indication whether a child
>   is in progress of being registered/unregistered?
> - We also need to protect against removal of devices while unregister
>   is in progress (same mechanism as above?) The second issue you
>   describe above should be fixed then if the children keep a reference
>   of the parent.
Parent unregistration publishes that its going away first, so no new device removal from user can start.
Already on going removal by users anyway complete first.

Once all remove() users are done, parent is getting unregistered.

Re: [PATCHv3 3/3] vfio/mdev: Synchronize device create/remove with parent removal

[Cornelia Huck]

On Fri, 17 May 2019 14:18:26 +0000
Parav Pandit <parav@mellanox.com> wrote:

> > > @@ -206,14 +214,27 @@ void mdev_unregister_device(struct device *dev)
> > >  	dev_info(dev, "MDEV: Unregistering\n");
> > >
> > >  	list_del(&parent->next);
> > > +	mutex_unlock(&parent_list_lock);
> > > +
> > > +	/* Release the initial reference so that new create cannot start */
> > > +	mdev_put_parent(parent);  
> > 
> > The comment is confusing: We do drop one reference, but this does not
> > imply we're going to 0 (which would be the one thing that would block
> > creating new devices).
> >   
> Ok. How about below comment.
> /* Balance with initial reference init */

Well, 'release the initial reference' is fine; it's just the second
part that is confusing.

One thing that continues to irk me (and I'm sorry if I sound like a
broken record) is that you give up the initial reference and then
continue to use parent. For the more usual semantics of a reference
count, that would be a bug (as the structure would be freed if the
reference count dropped to zero), even though it is not a bug here.

> 
> > > +
> > > +	/*
> > > +	 * Wait for all the create and remove references to drop.
> > > +	 */
> > > +	wait_for_completion(&parent->unreg_completion);  
> > 
> > It only reaches 0 after this wait.
> >  
> Yes.
>  
> > > +
> > > +	/*
> > > +	 * New references cannot be taken and all users are done
> > > +	 * using the parent. So it is safe to unregister parent.
> > > +	 */
> > >  	class_compat_remove_link(mdev_bus_compat_class, dev, NULL);
> > >
> > >  	device_for_each_child(dev, NULL, mdev_device_remove_cb);
> > >
> > >  	parent_remove_sysfs_files(parent);
> > > -
> > > -	mutex_unlock(&parent_list_lock);
> > > -	mdev_put_parent(parent);
> > > +	kfree(parent);
> > > +	put_device(dev);
> > >  }
> > >  EXPORT_SYMBOL(mdev_unregister_device);
> > >
> > > @@ -237,10 +258,11 @@ int mdev_device_create(struct kobject *kobj,
> > >  	struct mdev_parent *parent;
> > >  	struct mdev_type *type = to_mdev_type(kobj);
> > >
> > > -	parent = mdev_get_parent(type->parent);
> > > -	if (!parent)
> > > +	if (!mdev_try_get_parent(type->parent))  
> > 
> > If other calls are still running, the refcount won't be 0, and this will succeed,
> > even if we really want to get rid of the device.
> >   
> Sure, if other calls are running, refcount won't be 0. Process creating them will eventually complete, and refcount will drop to zero.
> And new processes won't be able to start any more.
> So there is no differentiation between 'already in creation stage' and 'about to start' processes.

Does it really make sense to allow creation to start if the parent is
going away?

> 
> > >  		return -EINVAL;
> > >
> > > +	parent = type->parent;
> > > +
> > >  	mutex_lock(&mdev_list_lock);
> > >
> > >  	/* Check for duplicate */
> > > @@ -287,6 +309,7 @@ int mdev_device_create(struct kobject *kobj,
> > >
> > >  	mdev->active = true;
> > >  	dev_dbg(&mdev->dev, "MDEV: created\n");
> > > +	mdev_put_parent(parent);
> > >
> > >  	return 0;
> > >
> > > @@ -306,7 +329,6 @@ int mdev_device_remove(struct device *dev)
> > >  	struct mdev_device *mdev, *tmp;
> > >  	struct mdev_parent *parent;
> > >  	struct mdev_type *type;
> > > -	int ret;
> > >
> > >  	mdev = to_mdev_device(dev);
> > >
> > > @@ -330,15 +352,17 @@ int mdev_device_remove(struct device *dev)
> > >  	mutex_unlock(&mdev_list_lock);
> > >
> > >  	type = to_mdev_type(mdev->type_kobj);
> > > -	mdev_remove_sysfs_files(dev, type);
> > > -	device_del(&mdev->dev);
> > > -	parent = mdev->parent;
> > > -	ret = parent->ops->remove(mdev);
> > > -	if (ret)
> > > -		dev_err(&mdev->dev, "Remove failed: err=%d\n", ret);
> > > +	if (!mdev_try_get_parent(type->parent)) {  
> > 
> > Same here: Is there really a guarantee that the refcount is 0 when the parent
> > is going away?  
> A WARN_ON after wait_for_completion or in freeing the parent is good to catch bugs.

I'd rather prefer to avoid having to add WARN_ONs :)

This looks like it is supposed to be an early exit. However, if some
other thread does any create or remove operation at the same time,
we'll still do the remove, and we still might have have a race window
(and this is getting really hard to follow in the code).

> 
> >   
> > > +		/*
> > > +		 * Parent unregistration have started.
> > > +		 * No need to remove here.
> > > +		 */
> > > +		mutex_unlock(&mdev_list_lock);  
> > 
> > Btw., you already unlocked above.
> >  
> You are right. This unlock is wrong. I will revise the patch.
>  
> > > +		return -ENODEV;
> > > +	}
> > >
> > > -	/* Balances with device_initialize() */
> > > -	put_device(&mdev->dev);
> > > +	parent = mdev->parent;
> > > +	mdev_device_remove_common(mdev);
> > >  	mdev_put_parent(parent);
> > >
> > >  	return 0;
> > > diff --git a/drivers/vfio/mdev/mdev_private.h
> > > b/drivers/vfio/mdev/mdev_private.h
> > > index 924ed2274941..55ebab0af7b0 100644
> > > --- a/drivers/vfio/mdev/mdev_private.h
> > > +++ b/drivers/vfio/mdev/mdev_private.h
> > > @@ -19,7 +19,11 @@ void mdev_bus_unregister(void);  struct  
> > mdev_parent  
> > > {
> > >  	struct device *dev;
> > >  	const struct mdev_parent_ops *ops;
> > > -	struct kref ref;
> > > +	/* Protects unregistration to wait until create/remove
> > > +	 * are completed.
> > > +	 */
> > > +	refcount_t refcount;
> > > +	struct completion unreg_completion;
> > >  	struct list_head next;
> > >  	struct kset *mdev_types_kset;
> > >  	struct list_head type_list;  
> > 
> > I think what's really needed is to split up the different needs and not
> > overload the 'refcount' concept.
> >   
> Refcount tells that how many active references are present for this parent device.
> Those active reference could be create/remove processes and mdev core itself.
> 
> So when parent unregisters, mdev module publishes that it is going away through this refcount.
> Hence new users cannot start.

But it does not actually do that -- if there are other create/remove
operations running, userspace can still trigger a new create/remove. If
it triggers enough create/remove processes, it can keep the parent
around (even though that really is a pathological case.)

> 
> > - If we need to make sure that a reference to the parent is held so
> >   that the parent may not go away while still in use, we should
> >   continue to use the kref (in the idiomatic way it is used before this
> >   patch.)
> > - We need to protect against creation of new devices if the parent is
> >   going away. Maybe set a going_away marker in the parent structure for
> >   that so that creation bails out immediately?   
> Such marker will help to not start new processes.
> So an additional marker can be added to improve mdev_try_get_parent().
> But I couldn't justify why differentiating those two users on time scale is desired.
> One reason could be that user continuously tries to create mdev and parent never gets a chance, to unregister?
> I guess, parent will run out mdev devices before this can happen.

They can also run remove tasks in parallel (see above).

> 
> Additionally a stop marker is needed (counter) to tell that all users are done accessing it.
> Both purposes are served using a refcount scheme.

Why not stop new create/remove tasks on remove, and do the final
cleanup asynchronously? I think a refcount is fine to track accesses,
but not to block new tasks.

> 
> > What happens if the
> >   creation has already started when parent removal kicks in, though?  
> That particular creation will succeed but newer cannot start, because mdev_put_parent() is done.
> 
> >   Do we need some child list locking and an indication whether a child
> >   is in progress of being registered/unregistered?
> > - We also need to protect against removal of devices while unregister
> >   is in progress (same mechanism as above?) The second issue you
> >   describe above should be fixed then if the children keep a reference
> >   of the parent.  
> Parent unregistration publishes that its going away first, so no new device removal from user can start.

I don't think that this actually works as intended (see above).

> Already on going removal by users anyway complete first.
> 
> Once all remove() users are done, parent is getting unregistered.

RE: [PATCHv3 3/3] vfio/mdev: Synchronize device create/remove with parent removal

[Parav Pandit]

> -----Original Message-----
> From: Cornelia Huck <cohuck@redhat.com>
> Sent: Monday, May 20, 2019 6:29 AM
> To: Parav Pandit <parav@mellanox.com>
> Cc: kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> kwankhede@nvidia.com; alex.williamson@redhat.com; cjia@nvidia.com
> Subject: Re: [PATCHv3 3/3] vfio/mdev: Synchronize device create/remove
> with parent removal
> 
> On Fri, 17 May 2019 14:18:26 +0000
> Parav Pandit <parav@mellanox.com> wrote:
> 
> > > > @@ -206,14 +214,27 @@ void mdev_unregister_device(struct device
> *dev)
> > > >  	dev_info(dev, "MDEV: Unregistering\n");
> > > >
> > > >  	list_del(&parent->next);
> > > > +	mutex_unlock(&parent_list_lock);
> > > > +
> > > > +	/* Release the initial reference so that new create cannot start */
> > > > +	mdev_put_parent(parent);
> > >
> > > The comment is confusing: We do drop one reference, but this does
> > > not imply we're going to 0 (which would be the one thing that would
> > > block creating new devices).
> > >
> > Ok. How about below comment.
> > /* Balance with initial reference init */
> 
> Well, 'release the initial reference' is fine; it's just the second part that is
> confusing.
> 
> One thing that continues to irk me (and I'm sorry if I sound like a broken
> record) is that you give up the initial reference and then continue to use
> parent. For the more usual semantics of a reference count, that would be a
> bug (as the structure would be freed if the reference count dropped to zero),
> even though it is not a bug here.
> 
Well, refcount cannot drop to zero if user is using it.
But I understand that mdev_device caches it the parent in it, and hence it uses it.
However, mdev_device child devices are terminated first when parent goes away, ensuring that no more parent user is active.
So as you mentioned, its not a bug here.

> >
> > > > +
> > > > +	/*
> > > > +	 * Wait for all the create and remove references to drop.
> > > > +	 */
> > > > +	wait_for_completion(&parent->unreg_completion);
> > >
> > > It only reaches 0 after this wait.
> > >
> > Yes.
> >
> > > > +
> > > > +	/*
> > > > +	 * New references cannot be taken and all users are done
> > > > +	 * using the parent. So it is safe to unregister parent.
> > > > +	 */
> > > >  	class_compat_remove_link(mdev_bus_compat_class, dev, NULL);
> > > >
> > > >  	device_for_each_child(dev, NULL, mdev_device_remove_cb);
> > > >
> > > >  	parent_remove_sysfs_files(parent);
> > > > -
> > > > -	mutex_unlock(&parent_list_lock);
> > > > -	mdev_put_parent(parent);
> > > > +	kfree(parent);
> > > > +	put_device(dev);
> > > >  }
> > > >  EXPORT_SYMBOL(mdev_unregister_device);
> > > >
> > > > @@ -237,10 +258,11 @@ int mdev_device_create(struct kobject *kobj,
> > > >  	struct mdev_parent *parent;
> > > >  	struct mdev_type *type = to_mdev_type(kobj);
> > > >
> > > > -	parent = mdev_get_parent(type->parent);
> > > > -	if (!parent)
> > > > +	if (!mdev_try_get_parent(type->parent))
> > >
> > > If other calls are still running, the refcount won't be 0, and this
> > > will succeed, even if we really want to get rid of the device.
> > >
> > Sure, if other calls are running, refcount won't be 0. Process creating them
> will eventually complete, and refcount will drop to zero.
> > And new processes won't be able to start any more.
> > So there is no differentiation between 'already in creation stage' and
> 'about to start' processes.
> 
> Does it really make sense to allow creation to start if the parent is going
> away?
> 
Its really a small time window, on how we draw the line.
But it has important note that if user continues to keep creating, removing, parent is blocked on removal.

> >
> > > >  		return -EINVAL;
> > > >
> > > > +	parent = type->parent;
> > > > +
> > > >  	mutex_lock(&mdev_list_lock);
> > > >
> > > >  	/* Check for duplicate */
> > > > @@ -287,6 +309,7 @@ int mdev_device_create(struct kobject *kobj,
> > > >
> > > >  	mdev->active = true;
> > > >  	dev_dbg(&mdev->dev, "MDEV: created\n");
> > > > +	mdev_put_parent(parent);
> > > >
> > > >  	return 0;
> > > >
> > > > @@ -306,7 +329,6 @@ int mdev_device_remove(struct device *dev)
> > > >  	struct mdev_device *mdev, *tmp;
> > > >  	struct mdev_parent *parent;
> > > >  	struct mdev_type *type;
> > > > -	int ret;
> > > >
> > > >  	mdev = to_mdev_device(dev);
> > > >
> > > > @@ -330,15 +352,17 @@ int mdev_device_remove(struct device *dev)
> > > >  	mutex_unlock(&mdev_list_lock);
> > > >
> > > >  	type = to_mdev_type(mdev->type_kobj);
> > > > -	mdev_remove_sysfs_files(dev, type);
> > > > -	device_del(&mdev->dev);
> > > > -	parent = mdev->parent;
> > > > -	ret = parent->ops->remove(mdev);
> > > > -	if (ret)
> > > > -		dev_err(&mdev->dev, "Remove failed: err=%d\n", ret);
> > > > +	if (!mdev_try_get_parent(type->parent)) {
> > >
> > > Same here: Is there really a guarantee that the refcount is 0 when
> > > the parent is going away?
> > A WARN_ON after wait_for_completion or in freeing the parent is good to
> catch bugs.
> 
> I'd rather prefer to avoid having to add WARN_ONs :)
> 
> This looks like it is supposed to be an early exit. 
remove() is doing early exit if it doesn't get reference to its parent.
mdev_device_remove_common().

> However, if some
> other thread does any create or remove operation at the same time,
> we'll still do the remove, and we still might have have a race window
> (and this is getting really hard to follow in the code).
> 
Which part?
We have only 4 functions to follow, register_device(), unregister_device(), create() and remove().

If you meant, two removes racing with each other?
If so, that is currently guarded using not_so_well_defined active flag.
I will cleanup that later once this series is done.

> >
> > >
> > > > +		/*
> > > > +		 * Parent unregistration have started.
> > > > +		 * No need to remove here.
> > > > +		 */
> > > > +		mutex_unlock(&mdev_list_lock);
> > >
> > > Btw., you already unlocked above.
> > >
> > You are right. This unlock is wrong. I will revise the patch.
> >
> > > > +		return -ENODEV;
> > > > +	}
> > > >
> > > > -	/* Balances with device_initialize() */
> > > > -	put_device(&mdev->dev);
> > > > +	parent = mdev->parent;
> > > > +	mdev_device_remove_common(mdev);
> > > >  	mdev_put_parent(parent);
> > > >
> > > >  	return 0;
> > > > diff --git a/drivers/vfio/mdev/mdev_private.h
> > > > b/drivers/vfio/mdev/mdev_private.h
> > > > index 924ed2274941..55ebab0af7b0 100644
> > > > --- a/drivers/vfio/mdev/mdev_private.h
> > > > +++ b/drivers/vfio/mdev/mdev_private.h
> > > > @@ -19,7 +19,11 @@ void mdev_bus_unregister(void);  struct
> > > mdev_parent
> > > > {
> > > >  	struct device *dev;
> > > >  	const struct mdev_parent_ops *ops;
> > > > -	struct kref ref;
> > > > +	/* Protects unregistration to wait until create/remove
> > > > +	 * are completed.
> > > > +	 */
> > > > +	refcount_t refcount;
> > > > +	struct completion unreg_completion;
> > > >  	struct list_head next;
> > > >  	struct kset *mdev_types_kset;
> > > >  	struct list_head type_list;
> > >
> > > I think what's really needed is to split up the different needs and not
> > > overload the 'refcount' concept.
> > >
> > Refcount tells that how many active references are present for this parent
> device.
> > Those active reference could be create/remove processes and mdev core
> itself.
> >
> > So when parent unregisters, mdev module publishes that it is going away
> through this refcount.
> > Hence new users cannot start.
> 
> But it does not actually do that -- if there are other create/remove
> operations running, userspace can still trigger a new create/remove. If
> it triggers enough create/remove processes, it can keep the parent
> around (even though that really is a pathological case.)
> 
Yes. I agree that is still possible. And an extra flag can guard it.
I see it as try_get_parent() can be improved as incremental to implement and honor that flag.
Do you want to roll that flag in same patch in v4?

> >
> > > - If we need to make sure that a reference to the parent is held so
> > >   that the parent may not go away while still in use, we should
> > >   continue to use the kref (in the idiomatic way it is used before this
> > >   patch.)
> > > - We need to protect against creation of new devices if the parent is
> > >   going away. Maybe set a going_away marker in the parent structure for
> > >   that so that creation bails out immediately?
> > Such marker will help to not start new processes.
> > So an additional marker can be added to improve mdev_try_get_parent().
> > But I couldn't justify why differentiating those two users on time scale is
> desired.
> > One reason could be that user continuously tries to create mdev and
> parent never gets a chance, to unregister?
> > I guess, parent will run out mdev devices before this can happen.
> 
> They can also run remove tasks in parallel (see above).
> 
Yes, remove() is guarded using active flag.

> >
> > Additionally a stop marker is needed (counter) to tell that all users are
> done accessing it.
> > Both purposes are served using a refcount scheme.
> 
> Why not stop new create/remove tasks on remove, and do the final
> cleanup asynchronously? I think a refcount is fine to track accesses,
> but not to block new tasks.
> 
So a new flag will guard new create/remove tasks by enhancing try_get_parent().
I just didn't see it as critical fix, but it's doable. See above.

Async is certainly not a good idea.
mdev_release_parent() in current code doesn't nothing other than freeing memory and parent reference.
It take away the parent from the list early on, which is also wrong, because it was added to the list at the end.
Unregister() sequence should be mirror image.
Parent device files has to be removed before unregister_device() finishes, because they were added in register_device().
Otherwise, parent device_del() might be done, but files are still created under it.

If we want to keep the memory around of parent, until kref drops, than we need two refcounts.
One ensure that create and remove are done using it, other one that ensures that child are done using it.
I fail to justify adding complexity of two counters, because such two_counter_desire hints that somehow child devices may be still active even after remove() calls are finished.
And that should not be the case. Unless I miss such case.

> >
> > > What happens if the
> > >   creation has already started when parent removal kicks in, though?
> > That particular creation will succeed but newer cannot start, because
> mdev_put_parent() is done.
> >
> > >   Do we need some child list locking and an indication whether a child
> > >   is in progress of being registered/unregistered?
> > > - We also need to protect against removal of devices while unregister
> > >   is in progress (same mechanism as above?) The second issue you
> > >   describe above should be fixed then if the children keep a reference
> > >   of the parent.
> > Parent unregistration publishes that its going away first, so no new device
> removal from user can start.
> 
> I don't think that this actually works as intended (see above).
> 
It does work in most cases. Only if user space is creating hundreds of processes for creating mdevs, before they actually run out of creating new one.
But as we talked a flag will guard it.

So if refcount is ok, I can enhance it for flag.

> > Already on going removal by users anyway complete first.
> >
> > Once all remove() users are done, parent is getting unregistered.

RE: [PATCHv3 1/3] vfio/mdev: Improve the create/remove sequence

[Parav Pandit]

Hi Alex, Cornelia,


> -----Original Message-----
> From: linux-kernel-owner@vger.kernel.org <linux-kernel-
> owner@vger.kernel.org> On Behalf Of Parav Pandit
> Sent: Thursday, May 16, 2019 6:31 PM
> To: kvm@vger.kernel.org; linux-kernel@vger.kernel.org; cohuck@redhat.com;
> kwankhede@nvidia.com; alex.williamson@redhat.com
> Cc: cjia@nvidia.com; Parav Pandit <parav@mellanox.com>
> Subject: [PATCHv3 1/3] vfio/mdev: Improve the create/remove sequence
> 
> This patch addresses below two issues and prepares the code to address 3rd
> issue listed below.
> 
> 1. mdev device is placed on the mdev bus before it is created in the vendor
> driver. Once a device is placed on the mdev bus without creating its
> supporting underlying vendor device, mdev driver's probe() gets triggered.
> However there isn't a stable mdev available to work on.
> 
>    create_store()
>      mdev_create_device()
>        device_register()
>           ...
>          vfio_mdev_probe()
>         [...]
>         parent->ops->create()
>           vfio_ap_mdev_create()
>             mdev_set_drvdata(mdev, matrix_mdev);
>             /* Valid pointer set above */
> 
> Due to this way of initialization, mdev driver who wants to use the mdev,
> doesn't have a valid mdev to work on.
> 
> 2. Current creation sequence is,
>    parent->ops_create()
>    groups_register()
> 
> Remove sequence is,
>    parent->ops->remove()
>    groups_unregister()
> 
> However, remove sequence should be exact mirror of creation sequence.
> Once this is achieved, all users of the mdev will be terminated first before
> removing underlying vendor device.
> (Follow standard linux driver model).
> At that point vendor's remove() ops shouldn't fail because taking the device
> off the bus should terminate any usage.
> 
> 3. When remove operation fails, mdev sysfs removal attempts to add the file
> back on already removed device. Following call trace [1] is observed.
> 
> [1] call trace:
> kernel: WARNING: CPU: 2 PID: 9348 at fs/sysfs/file.c:327
> sysfs_create_file_ns+0x7f/0x90
> kernel: CPU: 2 PID: 9348 Comm: bash Kdump: loaded Not tainted 5.1.0-rc6-
> vdevbus+ #6
> kernel: Hardware name: Supermicro SYS-6028U-TR4+/X10DRU-i+, BIOS 2.0b
> 08/09/2016
> kernel: RIP: 0010:sysfs_create_file_ns+0x7f/0x90
> kernel: Call Trace:
> kernel: remove_store+0xdc/0x100 [mdev]
> kernel: kernfs_fop_write+0x113/0x1a0
> kernel: vfs_write+0xad/0x1b0
> kernel: ksys_write+0x5a/0xe0
> kernel: do_syscall_64+0x5a/0x210
> kernel: entry_SYSCALL_64_after_hwframe+0x49/0xbe
> 
> Therefore, mdev core is improved in following ways.
> 
> 1. Split the device registration/deregistration sequence so that some things
> can be done between initialization of the device and hooking it up to the
> bus respectively after deregistering it from the bus but before giving up our
> final reference.
> In particular, this means invoking the ->create and ->remove callbacks in
> those new windows. This gives the vendor driver an initialized mdev device
> to work with during creation.
> At the same time, a bus driver who wish to bind to mdev driver also gets
> initialized mdev device.
> 
> This follows standard Linux kernel bus and device model.
> 
> 2. During remove flow, first remove the device from the bus. This ensures
> that any bus specific devices are removed.
> Once device is taken off the mdev bus, invoke remove() of mdev from the
> vendor driver.
> 
> 3. The driver core device model provides way to register and auto unregister
> the device sysfs attribute groups at dev->groups.
> Make use of dev->groups to let core create the groups and eliminate code to
> avoid explicit groups creation and removal.
> 
> To ensure, that new sequence is solid, a below stack dump of a process is
> taken who attempts to remove the device while device is in use by vfio
> driver and user application.
> This stack dump validates that vfio driver guards against such device removal
> when device is in use.
> 
>  cat /proc/21962/stack
> [<0>] vfio_del_group_dev+0x216/0x3c0 [vfio] [<0>] mdev_remove+0x21/0x40
> [mdev] [<0>] device_release_driver_internal+0xe8/0x1b0
> [<0>] bus_remove_device+0xf9/0x170
> [<0>] device_del+0x168/0x350
> [<0>] mdev_device_remove_common+0x1d/0x50 [mdev] [<0>]
> mdev_device_remove+0x8c/0xd0 [mdev] [<0>] remove_store+0x71/0x90
> [mdev] [<0>] kernfs_fop_write+0x113/0x1a0 [<0>] vfs_write+0xad/0x1b0
> [<0>] ksys_write+0x5a/0xe0 [<0>] do_syscall_64+0x5a/0x210 [<0>]
> entry_SYSCALL_64_after_hwframe+0x49/0xbe
> [<0>] 0xffffffffffffffff
> 
> This prepares the code to eliminate calling device_create_file() in subsquent
> patch.
> 
> Signed-off-by: Parav Pandit <parav@mellanox.com>
> ---
>  drivers/vfio/mdev/mdev_core.c    | 94 +++++++++-----------------------
>  drivers/vfio/mdev/mdev_private.h |  2 +-
>  drivers/vfio/mdev/mdev_sysfs.c   |  2 +-
>  3 files changed, 27 insertions(+), 71 deletions(-)
> 
> diff --git a/drivers/vfio/mdev/mdev_core.c b/drivers/vfio/mdev/mdev_core.c
> index 3cc1a05fde1c..0bef0cae1d4b 100644
> --- a/drivers/vfio/mdev/mdev_core.c
> +++ b/drivers/vfio/mdev/mdev_core.c
> @@ -102,55 +102,10 @@ static void mdev_put_parent(struct mdev_parent
> *parent)
>  		kref_put(&parent->ref, mdev_release_parent);  }
> 
> -static int mdev_device_create_ops(struct kobject *kobj,
> -				  struct mdev_device *mdev)
> -{
> -	struct mdev_parent *parent = mdev->parent;
> -	int ret;
> -
> -	ret = parent->ops->create(kobj, mdev);
> -	if (ret)
> -		return ret;
> -
> -	ret = sysfs_create_groups(&mdev->dev.kobj,
> -				  parent->ops->mdev_attr_groups);
> -	if (ret)
> -		parent->ops->remove(mdev);
> -
> -	return ret;
> -}
> -
> -/*
> - * mdev_device_remove_ops gets called from sysfs's 'remove' and when
> parent
> - * device is being unregistered from mdev device framework.
> - * - 'force_remove' is set to 'false' when called from sysfs's 'remove' which
> - *   indicates that if the mdev device is active, used by VMM or userspace
> - *   application, vendor driver could return error then don't remove the
> device.
> - * - 'force_remove' is set to 'true' when called from
> mdev_unregister_device()
> - *   which indicate that parent device is being removed from mdev device
> - *   framework so remove mdev device forcefully.
> - */
> -static int mdev_device_remove_ops(struct mdev_device *mdev, bool
> force_remove) -{
> -	struct mdev_parent *parent = mdev->parent;
> -	int ret;
> -
> -	/*
> -	 * Vendor driver can return error if VMM or userspace application is
> -	 * using this mdev device.
> -	 */
> -	ret = parent->ops->remove(mdev);
> -	if (ret && !force_remove)
> -		return ret;
> -
> -	sysfs_remove_groups(&mdev->dev.kobj, parent->ops-
> >mdev_attr_groups);
> -	return 0;
> -}
> -
>  static int mdev_device_remove_cb(struct device *dev, void *data)  {
>  	if (dev_is_mdev(dev))
> -		mdev_device_remove(dev, true);
> +		mdev_device_remove(dev);
> 
>  	return 0;
>  }
> @@ -310,41 +265,43 @@ int mdev_device_create(struct kobject *kobj,
> 
>  	mdev->parent = parent;
> 
> +	device_initialize(&mdev->dev);
>  	mdev->dev.parent  = dev;
>  	mdev->dev.bus     = &mdev_bus_type;
>  	mdev->dev.release = mdev_device_release;
>  	dev_set_name(&mdev->dev, "%pUl", uuid);
> +	mdev->dev.groups = parent->ops->mdev_attr_groups;
> +	mdev->type_kobj = kobj;
> 
> -	ret = device_register(&mdev->dev);
> -	if (ret) {
> -		put_device(&mdev->dev);
> -		goto mdev_fail;
> -	}
> +	ret = parent->ops->create(kobj, mdev);
> +	if (ret)
> +		goto ops_create_fail;
> 
> -	ret = mdev_device_create_ops(kobj, mdev);
> +	ret = device_add(&mdev->dev);
>  	if (ret)
> -		goto create_fail;
> +		goto add_fail;
> 
>  	ret = mdev_create_sysfs_files(&mdev->dev, type);
> -	if (ret) {
> -		mdev_device_remove_ops(mdev, true);
> -		goto create_fail;
> -	}
> +	if (ret)
> +		goto sysfs_fail;
> 
> -	mdev->type_kobj = kobj;
>  	mdev->active = true;
>  	dev_dbg(&mdev->dev, "MDEV: created\n");
> 
>  	return 0;
> 
> -create_fail:
> -	device_unregister(&mdev->dev);
> +sysfs_fail:
> +	device_del(&mdev->dev);
> +add_fail:
> +	parent->ops->remove(mdev);
> +ops_create_fail:
> +	put_device(&mdev->dev);
>  mdev_fail:
>  	mdev_put_parent(parent);
>  	return ret;
>  }
> 
> -int mdev_device_remove(struct device *dev, bool force_remove)
> +int mdev_device_remove(struct device *dev)
>  {
>  	struct mdev_device *mdev, *tmp;
>  	struct mdev_parent *parent;
> @@ -373,16 +330,15 @@ int mdev_device_remove(struct device *dev, bool
> force_remove)
>  	mutex_unlock(&mdev_list_lock);
> 
>  	type = to_mdev_type(mdev->type_kobj);
> +	mdev_remove_sysfs_files(dev, type);
> +	device_del(&mdev->dev);
>  	parent = mdev->parent;
> +	ret = parent->ops->remove(mdev);
> +	if (ret)
> +		dev_err(&mdev->dev, "Remove failed: err=%d\n", ret);
> 
> -	ret = mdev_device_remove_ops(mdev, force_remove);
> -	if (ret) {
> -		mdev->active = true;
> -		return ret;
> -	}
> -
> -	mdev_remove_sysfs_files(dev, type);
> -	device_unregister(dev);
> +	/* Balances with device_initialize() */
> +	put_device(&mdev->dev);
>  	mdev_put_parent(parent);
> 
>  	return 0;
> diff --git a/drivers/vfio/mdev/mdev_private.h
> b/drivers/vfio/mdev/mdev_private.h
> index 36cbbdb754de..924ed2274941 100644
> --- a/drivers/vfio/mdev/mdev_private.h
> +++ b/drivers/vfio/mdev/mdev_private.h
> @@ -60,6 +60,6 @@ void mdev_remove_sysfs_files(struct device *dev, struct
> mdev_type *type);
> 
>  int  mdev_device_create(struct kobject *kobj,
>  			struct device *dev, const guid_t *uuid); -int
> mdev_device_remove(struct device *dev, bool force_remove);
> +int  mdev_device_remove(struct device *dev);
> 
>  #endif /* MDEV_PRIVATE_H */
> diff --git a/drivers/vfio/mdev/mdev_sysfs.c b/drivers/vfio/mdev/mdev_sysfs.c
> index cbf94b8165ea..9f774b91d275 100644
> --- a/drivers/vfio/mdev/mdev_sysfs.c
> +++ b/drivers/vfio/mdev/mdev_sysfs.c
> @@ -236,7 +236,7 @@ static ssize_t remove_store(struct device *dev, struct
> device_attribute *attr,
>  	if (val && device_remove_file_self(dev, attr)) {
>  		int ret;
> 
> -		ret = mdev_device_remove(dev, false);
> +		ret = mdev_device_remove(dev);
>  		if (ret) {
>  			device_create_file(dev, attr);
>  			return ret;
> --
> 2.19.2

Seems 3rd patch will take few more days to settle down with new flag and its review.
Given that fix of 3rd patch is fixing a different race condition, if patch 1 and 2 look ok, shall we move forward with those 2 fixes it in 5.2-rc?
Fixes 1,2 prepare mdev to be usable for non vfio use case for the series we are working on.

Re: [PATCHv3 3/3] vfio/mdev: Synchronize device create/remove with parent removal

[Alex Williamson]

On Mon, 20 May 2019 19:15:15 +0000
Parav Pandit <parav@mellanox.com> wrote:

> > -----Original Message-----
> > From: Cornelia Huck <cohuck@redhat.com>
> > Sent: Monday, May 20, 2019 6:29 AM
> > To: Parav Pandit <parav@mellanox.com>
> > Cc: kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> > kwankhede@nvidia.com; alex.williamson@redhat.com; cjia@nvidia.com
> > Subject: Re: [PATCHv3 3/3] vfio/mdev: Synchronize device create/remove
> > with parent removal
> > 
> > On Fri, 17 May 2019 14:18:26 +0000
> > Parav Pandit <parav@mellanox.com> wrote:
> >   
> > > > > @@ -206,14 +214,27 @@ void mdev_unregister_device(struct device  
> > *dev)  
> > > > >  	dev_info(dev, "MDEV: Unregistering\n");
> > > > >
> > > > >  	list_del(&parent->next);
> > > > > +	mutex_unlock(&parent_list_lock);
> > > > > +
> > > > > +	/* Release the initial reference so that new create cannot start */
> > > > > +	mdev_put_parent(parent);  
> > > >
> > > > The comment is confusing: We do drop one reference, but this does
> > > > not imply we're going to 0 (which would be the one thing that would
> > > > block creating new devices).
> > > >  
> > > Ok. How about below comment.
> > > /* Balance with initial reference init */  
> > 
> > Well, 'release the initial reference' is fine; it's just the second part that is
> > confusing.
> > 
> > One thing that continues to irk me (and I'm sorry if I sound like a broken
> > record) is that you give up the initial reference and then continue to use
> > parent. For the more usual semantics of a reference count, that would be a
> > bug (as the structure would be freed if the reference count dropped to zero),
> > even though it is not a bug here.
> >   
> Well, refcount cannot drop to zero if user is using it.
> But I understand that mdev_device caches it the parent in it, and hence it uses it.
> However, mdev_device child devices are terminated first when parent goes away, ensuring that no more parent user is active.
> So as you mentioned, its not a bug here.
> 
> > >  
> > > > > +
> > > > > +	/*
> > > > > +	 * Wait for all the create and remove references to drop.
> > > > > +	 */
> > > > > +	wait_for_completion(&parent->unreg_completion);  
> > > >
> > > > It only reaches 0 after this wait.
> > > >  
> > > Yes.
> > >  
> > > > > +
> > > > > +	/*
> > > > > +	 * New references cannot be taken and all users are done
> > > > > +	 * using the parent. So it is safe to unregister parent.
> > > > > +	 */
> > > > >  	class_compat_remove_link(mdev_bus_compat_class, dev, NULL);
> > > > >
> > > > >  	device_for_each_child(dev, NULL, mdev_device_remove_cb);
> > > > >
> > > > >  	parent_remove_sysfs_files(parent);
> > > > > -
> > > > > -	mutex_unlock(&parent_list_lock);
> > > > > -	mdev_put_parent(parent);
> > > > > +	kfree(parent);
> > > > > +	put_device(dev);
> > > > >  }
> > > > >  EXPORT_SYMBOL(mdev_unregister_device);
> > > > >
> > > > > @@ -237,10 +258,11 @@ int mdev_device_create(struct kobject *kobj,
> > > > >  	struct mdev_parent *parent;
> > > > >  	struct mdev_type *type = to_mdev_type(kobj);
> > > > >
> > > > > -	parent = mdev_get_parent(type->parent);
> > > > > -	if (!parent)
> > > > > +	if (!mdev_try_get_parent(type->parent))  
> > > >
> > > > If other calls are still running, the refcount won't be 0, and this
> > > > will succeed, even if we really want to get rid of the device.
> > > >  
> > > Sure, if other calls are running, refcount won't be 0. Process creating them  
> > will eventually complete, and refcount will drop to zero.  
> > > And new processes won't be able to start any more.
> > > So there is no differentiation between 'already in creation stage' and  
> > 'about to start' processes.
> > 
> > Does it really make sense to allow creation to start if the parent is going
> > away?
> >   
> Its really a small time window, on how we draw the line.
> But it has important note that if user continues to keep creating, removing, parent is blocked on removal.
> 
> > >  
> > > > >  		return -EINVAL;
> > > > >
> > > > > +	parent = type->parent;
> > > > > +
> > > > >  	mutex_lock(&mdev_list_lock);
> > > > >
> > > > >  	/* Check for duplicate */
> > > > > @@ -287,6 +309,7 @@ int mdev_device_create(struct kobject *kobj,
> > > > >
> > > > >  	mdev->active = true;
> > > > >  	dev_dbg(&mdev->dev, "MDEV: created\n");
> > > > > +	mdev_put_parent(parent);
> > > > >
> > > > >  	return 0;
> > > > >
> > > > > @@ -306,7 +329,6 @@ int mdev_device_remove(struct device *dev)
> > > > >  	struct mdev_device *mdev, *tmp;
> > > > >  	struct mdev_parent *parent;
> > > > >  	struct mdev_type *type;
> > > > > -	int ret;
> > > > >
> > > > >  	mdev = to_mdev_device(dev);
> > > > >
> > > > > @@ -330,15 +352,17 @@ int mdev_device_remove(struct device *dev)
> > > > >  	mutex_unlock(&mdev_list_lock);
> > > > >
> > > > >  	type = to_mdev_type(mdev->type_kobj);
> > > > > -	mdev_remove_sysfs_files(dev, type);
> > > > > -	device_del(&mdev->dev);
> > > > > -	parent = mdev->parent;
> > > > > -	ret = parent->ops->remove(mdev);
> > > > > -	if (ret)
> > > > > -		dev_err(&mdev->dev, "Remove failed: err=%d\n", ret);
> > > > > +	if (!mdev_try_get_parent(type->parent)) {  
> > > >
> > > > Same here: Is there really a guarantee that the refcount is 0 when
> > > > the parent is going away?  
> > > A WARN_ON after wait_for_completion or in freeing the parent is good to  
> > catch bugs.
> > 
> > I'd rather prefer to avoid having to add WARN_ONs :)
> > 
> > This looks like it is supposed to be an early exit.   
> remove() is doing early exit if it doesn't get reference to its parent.
> mdev_device_remove_common().
> 
> > However, if some
> > other thread does any create or remove operation at the same time,
> > we'll still do the remove, and we still might have have a race window
> > (and this is getting really hard to follow in the code).
> >   
> Which part?
> We have only 4 functions to follow, register_device(), unregister_device(), create() and remove().
> 
> If you meant, two removes racing with each other?
> If so, that is currently guarded using not_so_well_defined active flag.
> I will cleanup that later once this series is done.
> 
> > >  
> > > >  
> > > > > +		/*
> > > > > +		 * Parent unregistration have started.
> > > > > +		 * No need to remove here.
> > > > > +		 */
> > > > > +		mutex_unlock(&mdev_list_lock);  
> > > >
> > > > Btw., you already unlocked above.
> > > >  
> > > You are right. This unlock is wrong. I will revise the patch.
> > >  
> > > > > +		return -ENODEV;
> > > > > +	}
> > > > >
> > > > > -	/* Balances with device_initialize() */
> > > > > -	put_device(&mdev->dev);
> > > > > +	parent = mdev->parent;
> > > > > +	mdev_device_remove_common(mdev);
> > > > >  	mdev_put_parent(parent);
> > > > >
> > > > >  	return 0;
> > > > > diff --git a/drivers/vfio/mdev/mdev_private.h
> > > > > b/drivers/vfio/mdev/mdev_private.h
> > > > > index 924ed2274941..55ebab0af7b0 100644
> > > > > --- a/drivers/vfio/mdev/mdev_private.h
> > > > > +++ b/drivers/vfio/mdev/mdev_private.h
> > > > > @@ -19,7 +19,11 @@ void mdev_bus_unregister(void);  struct  
> > > > mdev_parent  
> > > > > {
> > > > >  	struct device *dev;
> > > > >  	const struct mdev_parent_ops *ops;
> > > > > -	struct kref ref;
> > > > > +	/* Protects unregistration to wait until create/remove
> > > > > +	 * are completed.
> > > > > +	 */
> > > > > +	refcount_t refcount;
> > > > > +	struct completion unreg_completion;
> > > > >  	struct list_head next;
> > > > >  	struct kset *mdev_types_kset;
> > > > >  	struct list_head type_list;  
> > > >
> > > > I think what's really needed is to split up the different needs and not
> > > > overload the 'refcount' concept.
> > > >  
> > > Refcount tells that how many active references are present for this parent  
> > device.  
> > > Those active reference could be create/remove processes and mdev core  
> > itself.  
> > >
> > > So when parent unregisters, mdev module publishes that it is going away  
> > through this refcount.  
> > > Hence new users cannot start.  
> > 
> > But it does not actually do that -- if there are other create/remove
> > operations running, userspace can still trigger a new create/remove. If
> > it triggers enough create/remove processes, it can keep the parent
> > around (even though that really is a pathological case.)
> >   
> Yes. I agree that is still possible. And an extra flag can guard it.
> I see it as try_get_parent() can be improved as incremental to implement and honor that flag.
> Do you want to roll that flag in same patch in v4?
> 
> > >  
> > > > - If we need to make sure that a reference to the parent is held so
> > > >   that the parent may not go away while still in use, we should
> > > >   continue to use the kref (in the idiomatic way it is used before this
> > > >   patch.)
> > > > - We need to protect against creation of new devices if the parent is
> > > >   going away. Maybe set a going_away marker in the parent structure for
> > > >   that so that creation bails out immediately?  
> > > Such marker will help to not start new processes.
> > > So an additional marker can be added to improve mdev_try_get_parent().
> > > But I couldn't justify why differentiating those two users on time scale is  
> > desired.  
> > > One reason could be that user continuously tries to create mdev and  
> > parent never gets a chance, to unregister?  
> > > I guess, parent will run out mdev devices before this can happen.  
> > 
> > They can also run remove tasks in parallel (see above).
> >   
> Yes, remove() is guarded using active flag.
> 
> > >
> > > Additionally a stop marker is needed (counter) to tell that all users are  
> > done accessing it.  
> > > Both purposes are served using a refcount scheme.  
> > 
> > Why not stop new create/remove tasks on remove, and do the final
> > cleanup asynchronously? I think a refcount is fine to track accesses,
> > but not to block new tasks.
> >   
> So a new flag will guard new create/remove tasks by enhancing try_get_parent().
> I just didn't see it as critical fix, but it's doable. See above.
> 
> Async is certainly not a good idea.
> mdev_release_parent() in current code doesn't nothing other than freeing memory and parent reference.
> It take away the parent from the list early on, which is also wrong, because it was added to the list at the end.
> Unregister() sequence should be mirror image.
> Parent device files has to be removed before unregister_device() finishes, because they were added in register_device().
> Otherwise, parent device_del() might be done, but files are still created under it.
> 
> If we want to keep the memory around of parent, until kref drops, than we need two refcounts.
> One ensure that create and remove are done using it, other one that ensures that child are done using it.
> I fail to justify adding complexity of two counters, because such two_counter_desire hints that somehow child devices may be still active even after remove() calls are finished.
> And that should not be the case. Unless I miss such case.
> 
> > >  
> > > > What happens if the
> > > >   creation has already started when parent removal kicks in, though?  
> > > That particular creation will succeed but newer cannot start, because  
> > mdev_put_parent() is done.  
> > >  
> > > >   Do we need some child list locking and an indication whether a child
> > > >   is in progress of being registered/unregistered?
> > > > - We also need to protect against removal of devices while unregister
> > > >   is in progress (same mechanism as above?) The second issue you
> > > >   describe above should be fixed then if the children keep a reference
> > > >   of the parent.  
> > > Parent unregistration publishes that its going away first, so no new device  
> > removal from user can start.
> > 
> > I don't think that this actually works as intended (see above).
> >   
> It does work in most cases. Only if user space is creating hundreds of processes for creating mdevs, before they actually run out of creating new one.
> But as we talked a flag will guard it.
> 
> So if refcount is ok, I can enhance it for flag.

I agree with Connie's dislike of the refcount, where it seems we're
really just using it as a read-writer lock.  So why not simply use a
rwsem?  The parent unregistration path would do a down_write() and all
the ancillary paths would do a down_read_trylock() as they should never
see read contention unless the parent is being removed.  As a bonus, we
don't need to invent our own fairness algorithm, nor do we need to
remove the krefs as they're at least useful to validate we haven't
missed anyone.  Thanks,

Alex

Re: [PATCHv3 1/3] vfio/mdev: Improve the create/remove sequence

[Cornelia Huck]

On Thu, 16 May 2019 18:30:32 -0500
Parav Pandit <parav@mellanox.com> wrote:

> This patch addresses below two issues and prepares the code to address
> 3rd issue listed below.
> 
> 1. mdev device is placed on the mdev bus before it is created in the
> vendor driver. Once a device is placed on the mdev bus without creating
> its supporting underlying vendor device, mdev driver's probe() gets triggered.
> However there isn't a stable mdev available to work on.
> 
>    create_store()
>      mdev_create_device()
>        device_register()
>           ...
>          vfio_mdev_probe()
>         [...]
>         parent->ops->create()
>           vfio_ap_mdev_create()
>             mdev_set_drvdata(mdev, matrix_mdev);
>             /* Valid pointer set above */
> 
> Due to this way of initialization, mdev driver who wants to use the mdev,
> doesn't have a valid mdev to work on.
> 
> 2. Current creation sequence is,
>    parent->ops_create()
>    groups_register()
> 
> Remove sequence is,
>    parent->ops->remove()
>    groups_unregister()
> 
> However, remove sequence should be exact mirror of creation sequence.
> Once this is achieved, all users of the mdev will be terminated first
> before removing underlying vendor device.
> (Follow standard linux driver model).
> At that point vendor's remove() ops shouldn't fail because taking the
> device off the bus should terminate any usage.
> 
> 3. When remove operation fails, mdev sysfs removal attempts to add the
> file back on already removed device. Following call trace [1] is observed.
> 
> [1] call trace:
> kernel: WARNING: CPU: 2 PID: 9348 at fs/sysfs/file.c:327 sysfs_create_file_ns+0x7f/0x90
> kernel: CPU: 2 PID: 9348 Comm: bash Kdump: loaded Not tainted 5.1.0-rc6-vdevbus+ #6
> kernel: Hardware name: Supermicro SYS-6028U-TR4+/X10DRU-i+, BIOS 2.0b 08/09/2016
> kernel: RIP: 0010:sysfs_create_file_ns+0x7f/0x90
> kernel: Call Trace:
> kernel: remove_store+0xdc/0x100 [mdev]
> kernel: kernfs_fop_write+0x113/0x1a0
> kernel: vfs_write+0xad/0x1b0
> kernel: ksys_write+0x5a/0xe0
> kernel: do_syscall_64+0x5a/0x210
> kernel: entry_SYSCALL_64_after_hwframe+0x49/0xbe
> 
> Therefore, mdev core is improved in following ways.
> 
> 1. Split the device registration/deregistration sequence so that some
> things can be done between initialization of the device and
> hooking it up to the bus respectively after deregistering it
> from the bus but before giving up our final reference.
> In particular, this means invoking the ->create and ->remove
> callbacks in those new windows. This gives the vendor driver an
> initialized mdev device to work with during creation.
> At the same time, a bus driver who wish to bind to mdev driver also

s/who wish/that wishes/

> gets initialized mdev device.
> 
> This follows standard Linux kernel bus and device model.
> 
> 2. During remove flow, first remove the device from the bus. This
> ensures that any bus specific devices are removed.
> Once device is taken off the mdev bus, invoke remove() of mdev
> from the vendor driver.
> 
> 3. The driver core device model provides way to register and auto
> unregister the device sysfs attribute groups at dev->groups.
> Make use of dev->groups to let core create the groups and eliminate
> code to avoid explicit groups creation and removal.
> 
> To ensure, that new sequence is solid, a below stack dump of a
> process is taken who attempts to remove the device while device is in
> use by vfio driver and user application.
> This stack dump validates that vfio driver guards against such device
> removal when device is in use.
> 
>  cat /proc/21962/stack
> [<0>] vfio_del_group_dev+0x216/0x3c0 [vfio]
> [<0>] mdev_remove+0x21/0x40 [mdev]
> [<0>] device_release_driver_internal+0xe8/0x1b0
> [<0>] bus_remove_device+0xf9/0x170
> [<0>] device_del+0x168/0x350
> [<0>] mdev_device_remove_common+0x1d/0x50 [mdev]
> [<0>] mdev_device_remove+0x8c/0xd0 [mdev]
> [<0>] remove_store+0x71/0x90 [mdev]
> [<0>] kernfs_fop_write+0x113/0x1a0
> [<0>] vfs_write+0xad/0x1b0
> [<0>] ksys_write+0x5a/0xe0
> [<0>] do_syscall_64+0x5a/0x210
> [<0>] entry_SYSCALL_64_after_hwframe+0x49/0xbe
> [<0>] 0xffffffffffffffff
> 
> This prepares the code to eliminate calling device_create_file() in
> subsquent patch.
> 
> Signed-off-by: Parav Pandit <parav@mellanox.com>
> ---
>  drivers/vfio/mdev/mdev_core.c    | 94 +++++++++-----------------------
>  drivers/vfio/mdev/mdev_private.h |  2 +-
>  drivers/vfio/mdev/mdev_sysfs.c   |  2 +-
>  3 files changed, 27 insertions(+), 71 deletions(-)

Personally, I'd do a more compact patch description, but there's
nothing really wrong with yours, either.

Patch also seems sane to me, although I'd probably have merged this and
the next patch. But no reason to quibble further.

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [PATCHv3 2/3] vfio/mdev: Avoid creating sysfs remove file on stale device removal

[Cornelia Huck]

On Thu, 16 May 2019 18:30:33 -0500
Parav Pandit <parav@mellanox.com> wrote:

> If device is removal is initiated by two threads as below, mdev core
> attempts to create a syfs remove file on stale device.
> During this flow, below [1] call trace is observed.
> 
>      cpu-0                                    cpu-1
>      -----                                    -----
>   mdev_unregister_device()
>     device_for_each_child
>        mdev_device_remove_cb
>           mdev_device_remove
>                                        user_syscall
>                                          remove_store()
>                                            mdev_device_remove()
>                                         [..]
>    unregister device();
>                                        /* not found in list or
>                                         * active=false.
>                                         */
>                                           sysfs_create_file()
>                                           ..Call trace
> 
> Now that mdev core follows correct device removal system of the linux
> bus model, remove shouldn't fail in normal cases. If it fails, there is
> no point of creating a stale file or checking for specific error status.
> 
> kernel: WARNING: CPU: 2 PID: 9348 at fs/sysfs/file.c:327
> sysfs_create_file_ns+0x7f/0x90
> kernel: CPU: 2 PID: 9348 Comm: bash Kdump: loaded Not tainted
> 5.1.0-rc6-vdevbus+ #6
> kernel: Hardware name: Supermicro SYS-6028U-TR4+/X10DRU-i+, BIOS 2.0b
> 08/09/2016
> kernel: RIP: 0010:sysfs_create_file_ns+0x7f/0x90
> kernel: Call Trace:
> kernel: remove_store+0xdc/0x100 [mdev]
> kernel: kernfs_fop_write+0x113/0x1a0
> kernel: vfs_write+0xad/0x1b0
> kernel: ksys_write+0x5a/0xe0
> kernel: do_syscall_64+0x5a/0x210
> kernel: entry_SYSCALL_64_after_hwframe+0x49/0xbe
> 
> Signed-off-by: Parav Pandit <parav@mellanox.com>
> ---
>  drivers/vfio/mdev/mdev_sysfs.c | 4 +---
>  1 file changed, 1 insertion(+), 3 deletions(-)
> 
> diff --git a/drivers/vfio/mdev/mdev_sysfs.c b/drivers/vfio/mdev/mdev_sysfs.c
> index 9f774b91d275..ffa3dcebf201 100644
> --- a/drivers/vfio/mdev/mdev_sysfs.c
> +++ b/drivers/vfio/mdev/mdev_sysfs.c
> @@ -237,10 +237,8 @@ static ssize_t remove_store(struct device *dev, struct device_attribute *attr,
>  		int ret;
>  
>  		ret = mdev_device_remove(dev);
> -		if (ret) {
> -			device_create_file(dev, attr);
> +		if (ret)
>  			return ret;
> -		}
>  	}
>  
>  	return count;

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [PATCHv3 1/3] vfio/mdev: Improve the create/remove sequence

[Cornelia Huck]

On Mon, 20 May 2019 19:54:59 +0000
Parav Pandit <parav@mellanox.com> wrote:

> Seems 3rd patch will take few more days to settle down with new flag and its review.
> Given that fix of 3rd patch is fixing a different race condition, if patch 1 and 2 look ok, shall we move forward with those 2 fixes it in 5.2-rc?
> Fixes 1,2 prepare mdev to be usable for non vfio use case for the series we are working on.

I think it's fine to merge the first two patches (I've given my R-b.)

The only issue I'm aware of is that the vfio-ap code might need some
fixes as well; but I'll leave that to the vfio-ap folks to sort out.

RE: [PATCHv3 3/3] vfio/mdev: Synchronize device create/remove with parent removal

[Parav Pandit]

Hi Alex,

I was on travel for last 3 days, hence the slow response.
Started working now. Please see inline response below.

> -----Original Message-----
> From: Alex Williamson <alex.williamson@redhat.com>
> Sent: Tuesday, May 21, 2019 3:42 AM
> To: Parav Pandit <parav@mellanox.com>
> Cc: Cornelia Huck <cohuck@redhat.com>; kvm@vger.kernel.org; linux-
> kernel@vger.kernel.org; kwankhede@nvidia.com; cjia@nvidia.com
> Subject: Re: [PATCHv3 3/3] vfio/mdev: Synchronize device create/remove with
> parent removal
> 
> On Mon, 20 May 2019 19:15:15 +0000
> Parav Pandit <parav@mellanox.com> wrote:
> 
> > > -----Original Message-----
> > > From: Cornelia Huck <cohuck@redhat.com>
> > > Sent: Monday, May 20, 2019 6:29 AM
> > > To: Parav Pandit <parav@mellanox.com>
> > > Cc: kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> > > kwankhede@nvidia.com; alex.williamson@redhat.com; cjia@nvidia.com
> > > Subject: Re: [PATCHv3 3/3] vfio/mdev: Synchronize device
> > > create/remove with parent removal
> > >
> > > On Fri, 17 May 2019 14:18:26 +0000
> > > Parav Pandit <parav@mellanox.com> wrote:
> > >
> > > > > > @@ -206,14 +214,27 @@ void mdev_unregister_device(struct
> > > > > > device
> > > *dev)
> > > > > >  	dev_info(dev, "MDEV: Unregistering\n");
> > > > > >
> > > > > >  	list_del(&parent->next);
> > > > > > +	mutex_unlock(&parent_list_lock);
> > > > > > +
> > > > > > +	/* Release the initial reference so that new create cannot start
> */
> > > > > > +	mdev_put_parent(parent);
> > > > >
> > > > > The comment is confusing: We do drop one reference, but this
> > > > > does not imply we're going to 0 (which would be the one thing
> > > > > that would block creating new devices).
> > > > >
> > > > Ok. How about below comment.
> > > > /* Balance with initial reference init */
> > >
> > > Well, 'release the initial reference' is fine; it's just the second
> > > part that is confusing.
> > >
> > > One thing that continues to irk me (and I'm sorry if I sound like a
> > > broken
> > > record) is that you give up the initial reference and then continue
> > > to use parent. For the more usual semantics of a reference count,
> > > that would be a bug (as the structure would be freed if the
> > > reference count dropped to zero), even though it is not a bug here.
> > >
> > Well, refcount cannot drop to zero if user is using it.
> > But I understand that mdev_device caches it the parent in it, and hence it
> uses it.
> > However, mdev_device child devices are terminated first when parent goes
> away, ensuring that no more parent user is active.
> > So as you mentioned, its not a bug here.
> >
> > > >
> > > > > > +
> > > > > > +	/*
> > > > > > +	 * Wait for all the create and remove references to drop.
> > > > > > +	 */
> > > > > > +	wait_for_completion(&parent->unreg_completion);
> > > > >
> > > > > It only reaches 0 after this wait.
> > > > >
> > > > Yes.
> > > >
> > > > > > +
> > > > > > +	/*
> > > > > > +	 * New references cannot be taken and all users are done
> > > > > > +	 * using the parent. So it is safe to unregister parent.
> > > > > > +	 */
> > > > > >  	class_compat_remove_link(mdev_bus_compat_class, dev,
> NULL);
> > > > > >
> > > > > >  	device_for_each_child(dev, NULL, mdev_device_remove_cb);
> > > > > >
> > > > > >  	parent_remove_sysfs_files(parent);
> > > > > > -
> > > > > > -	mutex_unlock(&parent_list_lock);
> > > > > > -	mdev_put_parent(parent);
> > > > > > +	kfree(parent);
> > > > > > +	put_device(dev);
> > > > > >  }
> > > > > >  EXPORT_SYMBOL(mdev_unregister_device);
> > > > > >
> > > > > > @@ -237,10 +258,11 @@ int mdev_device_create(struct kobject
> *kobj,
> > > > > >  	struct mdev_parent *parent;
> > > > > >  	struct mdev_type *type = to_mdev_type(kobj);
> > > > > >
> > > > > > -	parent = mdev_get_parent(type->parent);
> > > > > > -	if (!parent)
> > > > > > +	if (!mdev_try_get_parent(type->parent))
> > > > >
> > > > > If other calls are still running, the refcount won't be 0, and
> > > > > this will succeed, even if we really want to get rid of the device.
> > > > >
> > > > Sure, if other calls are running, refcount won't be 0. Process
> > > > creating them
> > > will eventually complete, and refcount will drop to zero.
> > > > And new processes won't be able to start any more.
> > > > So there is no differentiation between 'already in creation stage'
> > > > and
> > > 'about to start' processes.
> > >
> > > Does it really make sense to allow creation to start if the parent
> > > is going away?
> > >
> > Its really a small time window, on how we draw the line.
> > But it has important note that if user continues to keep creating, removing,
> parent is blocked on removal.
> >
> > > >
> > > > > >  		return -EINVAL;
> > > > > >
> > > > > > +	parent = type->parent;
> > > > > > +
> > > > > >  	mutex_lock(&mdev_list_lock);
> > > > > >
> > > > > >  	/* Check for duplicate */
> > > > > > @@ -287,6 +309,7 @@ int mdev_device_create(struct kobject
> > > > > > *kobj,
> > > > > >
> > > > > >  	mdev->active = true;
> > > > > >  	dev_dbg(&mdev->dev, "MDEV: created\n");
> > > > > > +	mdev_put_parent(parent);
> > > > > >
> > > > > >  	return 0;
> > > > > >
> > > > > > @@ -306,7 +329,6 @@ int mdev_device_remove(struct device *dev)
> > > > > >  	struct mdev_device *mdev, *tmp;
> > > > > >  	struct mdev_parent *parent;
> > > > > >  	struct mdev_type *type;
> > > > > > -	int ret;
> > > > > >
> > > > > >  	mdev = to_mdev_device(dev);
> > > > > >
> > > > > > @@ -330,15 +352,17 @@ int mdev_device_remove(struct device
> *dev)
> > > > > >  	mutex_unlock(&mdev_list_lock);
> > > > > >
> > > > > >  	type = to_mdev_type(mdev->type_kobj);
> > > > > > -	mdev_remove_sysfs_files(dev, type);
> > > > > > -	device_del(&mdev->dev);
> > > > > > -	parent = mdev->parent;
> > > > > > -	ret = parent->ops->remove(mdev);
> > > > > > -	if (ret)
> > > > > > -		dev_err(&mdev->dev, "Remove failed: err=%d\n", ret);
> > > > > > +	if (!mdev_try_get_parent(type->parent)) {
> > > > >
> > > > > Same here: Is there really a guarantee that the refcount is 0
> > > > > when the parent is going away?
> > > > A WARN_ON after wait_for_completion or in freeing the parent is
> > > > good to
> > > catch bugs.
> > >
> > > I'd rather prefer to avoid having to add WARN_ONs :)
> > >
> > > This looks like it is supposed to be an early exit.
> > remove() is doing early exit if it doesn't get reference to its parent.
> > mdev_device_remove_common().
> >
> > > However, if some
> > > other thread does any create or remove operation at the same time,
> > > we'll still do the remove, and we still might have have a race
> > > window (and this is getting really hard to follow in the code).
> > >
> > Which part?
> > We have only 4 functions to follow, register_device(), unregister_device(),
> create() and remove().
> >
> > If you meant, two removes racing with each other?
> > If so, that is currently guarded using not_so_well_defined active flag.
> > I will cleanup that later once this series is done.
> >
> > > >
> > > > >
> > > > > > +		/*
> > > > > > +		 * Parent unregistration have started.
> > > > > > +		 * No need to remove here.
> > > > > > +		 */
> > > > > > +		mutex_unlock(&mdev_list_lock);
> > > > >
> > > > > Btw., you already unlocked above.
> > > > >
> > > > You are right. This unlock is wrong. I will revise the patch.
> > > >
> > > > > > +		return -ENODEV;
> > > > > > +	}
> > > > > >
> > > > > > -	/* Balances with device_initialize() */
> > > > > > -	put_device(&mdev->dev);
> > > > > > +	parent = mdev->parent;
> > > > > > +	mdev_device_remove_common(mdev);
> > > > > >  	mdev_put_parent(parent);
> > > > > >
> > > > > >  	return 0;
> > > > > > diff --git a/drivers/vfio/mdev/mdev_private.h
> > > > > > b/drivers/vfio/mdev/mdev_private.h
> > > > > > index 924ed2274941..55ebab0af7b0 100644
> > > > > > --- a/drivers/vfio/mdev/mdev_private.h
> > > > > > +++ b/drivers/vfio/mdev/mdev_private.h
> > > > > > @@ -19,7 +19,11 @@ void mdev_bus_unregister(void);  struct
> > > > > mdev_parent
> > > > > > {
> > > > > >  	struct device *dev;
> > > > > >  	const struct mdev_parent_ops *ops;
> > > > > > -	struct kref ref;
> > > > > > +	/* Protects unregistration to wait until create/remove
> > > > > > +	 * are completed.
> > > > > > +	 */
> > > > > > +	refcount_t refcount;
> > > > > > +	struct completion unreg_completion;
> > > > > >  	struct list_head next;
> > > > > >  	struct kset *mdev_types_kset;
> > > > > >  	struct list_head type_list;
> > > > >
> > > > > I think what's really needed is to split up the different needs
> > > > > and not overload the 'refcount' concept.
> > > > >
> > > > Refcount tells that how many active references are present for
> > > > this parent
> > > device.
> > > > Those active reference could be create/remove processes and mdev
> > > > core
> > > itself.
> > > >
> > > > So when parent unregisters, mdev module publishes that it is going
> > > > away
> > > through this refcount.
> > > > Hence new users cannot start.
> > >
> > > But it does not actually do that -- if there are other create/remove
> > > operations running, userspace can still trigger a new create/remove.
> > > If it triggers enough create/remove processes, it can keep the
> > > parent around (even though that really is a pathological case.)
> > >
> > Yes. I agree that is still possible. And an extra flag can guard it.
> > I see it as try_get_parent() can be improved as incremental to implement and
> honor that flag.
> > Do you want to roll that flag in same patch in v4?
> >
> > > >
> > > > > - If we need to make sure that a reference to the parent is held so
> > > > >   that the parent may not go away while still in use, we should
> > > > >   continue to use the kref (in the idiomatic way it is used before this
> > > > >   patch.)
> > > > > - We need to protect against creation of new devices if the parent is
> > > > >   going away. Maybe set a going_away marker in the parent structure
> for
> > > > >   that so that creation bails out immediately?
> > > > Such marker will help to not start new processes.
> > > > So an additional marker can be added to improve mdev_try_get_parent().
> > > > But I couldn't justify why differentiating those two users on time
> > > > scale is
> > > desired.
> > > > One reason could be that user continuously tries to create mdev
> > > > and
> > > parent never gets a chance, to unregister?
> > > > I guess, parent will run out mdev devices before this can happen.
> > >
> > > They can also run remove tasks in parallel (see above).
> > >
> > Yes, remove() is guarded using active flag.
> >
> > > >
> > > > Additionally a stop marker is needed (counter) to tell that all
> > > > users are
> > > done accessing it.
> > > > Both purposes are served using a refcount scheme.
> > >
> > > Why not stop new create/remove tasks on remove, and do the final
> > > cleanup asynchronously? I think a refcount is fine to track
> > > accesses, but not to block new tasks.
> > >
> > So a new flag will guard new create/remove tasks by enhancing
> try_get_parent().
> > I just didn't see it as critical fix, but it's doable. See above.
> >
> > Async is certainly not a good idea.
> > mdev_release_parent() in current code doesn't nothing other than freeing
> memory and parent reference.
> > It take away the parent from the list early on, which is also wrong, because it
> was added to the list at the end.
> > Unregister() sequence should be mirror image.
> > Parent device files has to be removed before unregister_device() finishes,
> because they were added in register_device().
> > Otherwise, parent device_del() might be done, but files are still created
> under it.
> >
> > If we want to keep the memory around of parent, until kref drops, than we
> need two refcounts.
> > One ensure that create and remove are done using it, other one that ensures
> that child are done using it.
> > I fail to justify adding complexity of two counters, because such
> two_counter_desire hints that somehow child devices may be still active even
> after remove() calls are finished.
> > And that should not be the case. Unless I miss such case.
> >
> > > >
> > > > > What happens if the
> > > > >   creation has already started when parent removal kicks in, though?
> > > > That particular creation will succeed but newer cannot start,
> > > > because
> > > mdev_put_parent() is done.
> > > >
> > > > >   Do we need some child list locking and an indication whether a child
> > > > >   is in progress of being registered/unregistered?
> > > > > - We also need to protect against removal of devices while unregister
> > > > >   is in progress (same mechanism as above?) The second issue you
> > > > >   describe above should be fixed then if the children keep a reference
> > > > >   of the parent.
> > > > Parent unregistration publishes that its going away first, so no
> > > > new device
> > > removal from user can start.
> > >
> > > I don't think that this actually works as intended (see above).
> > >
> > It does work in most cases. Only if user space is creating hundreds of
> processes for creating mdevs, before they actually run out of creating new one.
> > But as we talked a flag will guard it.
> >
> > So if refcount is ok, I can enhance it for flag.
> 
> I agree with Connie's dislike of the refcount, where it seems we're really just
> using it as a read-writer lock.  So why not simply use a rwsem?  The parent
> unregistration path would do a down_write() and all the ancillary paths would
> do a down_read_trylock() as they should never see read contention unless the
> parent is being removed.  
Ok. sounds good. I will send v4 using rwsem without removing kref.

> As a bonus, we don't need to invent our own fairness
> algorithm, nor do we need to remove the krefs as they're at least useful to
> validate we haven't missed anyone.  Thanks,
Well if we really care for kref, put on parent kref should be done in mdev_device_release().
I do not see how device can be still using the parent after device_del() is done.
Anyways, kref cleanup is different patch in 5.3.

> 
> Alex

RE: [PATCHv3 1/3] vfio/mdev: Improve the create/remove sequence

[Parav Pandit]

Hi Alex, Cornelia,

> -----Original Message-----
> From: Cornelia Huck <cohuck@redhat.com>
> Sent: Wednesday, May 22, 2019 3:25 PM
> To: Parav Pandit <parav@mellanox.com>
> Cc: kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> kwankhede@nvidia.com; alex.williamson@redhat.com; cjia@nvidia.com
> Subject: Re: [PATCHv3 1/3] vfio/mdev: Improve the create/remove sequence
> 
> On Thu, 16 May 2019 18:30:32 -0500
> Parav Pandit <parav@mellanox.com> wrote:
> 
> > This patch addresses below two issues and prepares the code to address
> > 3rd issue listed below.
> >
> > 1. mdev device is placed on the mdev bus before it is created in the
> > vendor driver. Once a device is placed on the mdev bus without
> > creating its supporting underlying vendor device, mdev driver's probe()
> gets triggered.
> > However there isn't a stable mdev available to work on.
> >
> >    create_store()
> >      mdev_create_device()
> >        device_register()
> >           ...
> >          vfio_mdev_probe()
> >         [...]
> >         parent->ops->create()
> >           vfio_ap_mdev_create()
> >             mdev_set_drvdata(mdev, matrix_mdev);
> >             /* Valid pointer set above */
> >
> > Due to this way of initialization, mdev driver who wants to use the
> > mdev, doesn't have a valid mdev to work on.
> >
> > 2. Current creation sequence is,
> >    parent->ops_create()
> >    groups_register()
> >
> > Remove sequence is,
> >    parent->ops->remove()
> >    groups_unregister()
> >
> > However, remove sequence should be exact mirror of creation sequence.
> > Once this is achieved, all users of the mdev will be terminated first
> > before removing underlying vendor device.
> > (Follow standard linux driver model).
> > At that point vendor's remove() ops shouldn't fail because taking the
> > device off the bus should terminate any usage.
> >
> > 3. When remove operation fails, mdev sysfs removal attempts to add the
> > file back on already removed device. Following call trace [1] is observed.
> >
> > [1] call trace:
> > kernel: WARNING: CPU: 2 PID: 9348 at fs/sysfs/file.c:327
> > sysfs_create_file_ns+0x7f/0x90
> > kernel: CPU: 2 PID: 9348 Comm: bash Kdump: loaded Not tainted
> > 5.1.0-rc6-vdevbus+ #6
> > kernel: Hardware name: Supermicro SYS-6028U-TR4+/X10DRU-i+, BIOS 2.0b
> > 08/09/2016
> > kernel: RIP: 0010:sysfs_create_file_ns+0x7f/0x90
> > kernel: Call Trace:
> > kernel: remove_store+0xdc/0x100 [mdev]
> > kernel: kernfs_fop_write+0x113/0x1a0
> > kernel: vfs_write+0xad/0x1b0
> > kernel: ksys_write+0x5a/0xe0
> > kernel: do_syscall_64+0x5a/0x210
> > kernel: entry_SYSCALL_64_after_hwframe+0x49/0xbe
> >
> > Therefore, mdev core is improved in following ways.
> >
> > 1. Split the device registration/deregistration sequence so that some
> > things can be done between initialization of the device and hooking it
> > up to the bus respectively after deregistering it from the bus but
> > before giving up our final reference.
> > In particular, this means invoking the ->create and ->remove callbacks
> > in those new windows. This gives the vendor driver an initialized mdev
> > device to work with during creation.
> > At the same time, a bus driver who wish to bind to mdev driver also
> 
> s/who wish/that wishes/
> 
> > gets initialized mdev device.
> >
> > This follows standard Linux kernel bus and device model.
> >
> > 2. During remove flow, first remove the device from the bus. This
> > ensures that any bus specific devices are removed.
> > Once device is taken off the mdev bus, invoke remove() of mdev from
> > the vendor driver.
> >
> > 3. The driver core device model provides way to register and auto
> > unregister the device sysfs attribute groups at dev->groups.
> > Make use of dev->groups to let core create the groups and eliminate
> > code to avoid explicit groups creation and removal.
> >
> > To ensure, that new sequence is solid, a below stack dump of a process
> > is taken who attempts to remove the device while device is in use by
> > vfio driver and user application.
> > This stack dump validates that vfio driver guards against such device
> > removal when device is in use.
> >
> >  cat /proc/21962/stack
> > [<0>] vfio_del_group_dev+0x216/0x3c0 [vfio] [<0>]
> > mdev_remove+0x21/0x40 [mdev] [<0>]
> > device_release_driver_internal+0xe8/0x1b0
> > [<0>] bus_remove_device+0xf9/0x170
> > [<0>] device_del+0x168/0x350
> > [<0>] mdev_device_remove_common+0x1d/0x50 [mdev] [<0>]
> > mdev_device_remove+0x8c/0xd0 [mdev] [<0>] remove_store+0x71/0x90
> > [mdev] [<0>] kernfs_fop_write+0x113/0x1a0 [<0>] vfs_write+0xad/0x1b0
> > [<0>] ksys_write+0x5a/0xe0 [<0>] do_syscall_64+0x5a/0x210 [<0>]
> > entry_SYSCALL_64_after_hwframe+0x49/0xbe
> > [<0>] 0xffffffffffffffff
> >
> > This prepares the code to eliminate calling device_create_file() in
> > subsquent patch.
> >
> > Signed-off-by: Parav Pandit <parav@mellanox.com>
> > ---
> >  drivers/vfio/mdev/mdev_core.c    | 94 +++++++++-----------------------
> >  drivers/vfio/mdev/mdev_private.h |  2 +-
> >  drivers/vfio/mdev/mdev_sysfs.c   |  2 +-
> >  3 files changed, 27 insertions(+), 71 deletions(-)
> 
> Personally, I'd do a more compact patch description, but there's nothing
> really wrong with yours, either.
> 
> Patch also seems sane to me, although I'd probably have merged this and
> the next patch. But no reason to quibble further.
> 
> Reviewed-by: Cornelia Huck <cohuck@redhat.com>

I missed to add your RB signature in v4.
If you send all 3 or just fist 2 of them in 5.2-rc, please include Cornelia's RB tag.