* [PATCH v4 0/4] cxl/mem: Fix memdev device setup
@ 2021-04-01 14:33 Dan Williams
2021-04-01 14:33 ` [PATCH v4 1/4] cxl/mem: Use sysfs_emit() for attribute show routines Dan Williams
` (3 more replies)
0 siblings, 4 replies; 5+ messages in thread
From: Dan Williams @ 2021-04-01 14:33 UTC (permalink / raw)
To: linux-cxl
Cc: Ben Widawsky, Jason Gunthorpe, linux-kernel, vishal.l.verma,
ira.weiny, alison.schofield
Changes since v3 [1]:
- Drop cxl_memdev_activate(). An open-coded pointer assignment is
sufficient relative to cdev_device_add() publishing the device (Jason)
[1]: http://lore.kernel.org/r/161714738634.2168142.10860201861152789544.stgit@dwillia2-desk3.amr.corp.intel.com
---
A collection of fixes initially inspired by Jason's recognition of
dev_set_name() error handling mistakes on other driver review, but also
from a deeper discussion of idiomatic device operation shutdown flows.
The end result is easier to reason about and validate. Thank you, Jason.
The sysfs_emit() fixup and unpublishing of device power management files
are independent sanity cleanups.
---
Dan Williams (4):
cxl/mem: Use sysfs_emit() for attribute show routines
cxl/mem: Fix synchronization mechanism for device removal vs ioctl operations
cxl/mem: Do not rely on device_add() side effects for dev_set_name() failures
cxl/mem: Disable cxl device power management
drivers/cxl/mem.c | 141 +++++++++++++++++++++++++++++++----------------------
1 file changed, 82 insertions(+), 59 deletions(-)
base-commit: a38fd8748464831584a19438cbb3082b5a2dab15
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH v4 1/4] cxl/mem: Use sysfs_emit() for attribute show routines
2021-04-01 14:33 [PATCH v4 0/4] cxl/mem: Fix memdev device setup Dan Williams
@ 2021-04-01 14:33 ` Dan Williams
2021-04-01 14:33 ` [PATCH v4 2/4] cxl/mem: Fix synchronization mechanism for device removal vs ioctl operations Dan Williams
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: Dan Williams @ 2021-04-01 14:33 UTC (permalink / raw)
To: linux-cxl
Cc: Jason Gunthorpe, Ben Widawsky, Jason Gunthorpe, linux-kernel,
vishal.l.verma, ira.weiny, alison.schofield
While none the CXL sysfs attributes are threatening to overrun a
PAGE_SIZE of output, it is good form to use the recommended helpers.
Fixes: b39cb1052a5c ("cxl/mem: Register CXL memX devices")
Reported-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Ben Widawsky <ben.widawsky@intel.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
drivers/cxl/mem.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index ecfc9ccdba8d..30bf4f0f3c17 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -1066,7 +1066,7 @@ static ssize_t firmware_version_show(struct device *dev,
struct cxl_memdev *cxlmd = to_cxl_memdev(dev);
struct cxl_mem *cxlm = cxlmd->cxlm;
- return sprintf(buf, "%.16s\n", cxlm->firmware_version);
+ return sysfs_emit(buf, "%.16s\n", cxlm->firmware_version);
}
static DEVICE_ATTR_RO(firmware_version);
@@ -1076,7 +1076,7 @@ static ssize_t payload_max_show(struct device *dev,
struct cxl_memdev *cxlmd = to_cxl_memdev(dev);
struct cxl_mem *cxlm = cxlmd->cxlm;
- return sprintf(buf, "%zu\n", cxlm->payload_size);
+ return sysfs_emit(buf, "%zu\n", cxlm->payload_size);
}
static DEVICE_ATTR_RO(payload_max);
@@ -1087,7 +1087,7 @@ static ssize_t ram_size_show(struct device *dev, struct device_attribute *attr,
struct cxl_mem *cxlm = cxlmd->cxlm;
unsigned long long len = range_len(&cxlm->ram_range);
- return sprintf(buf, "%#llx\n", len);
+ return sysfs_emit(buf, "%#llx\n", len);
}
static struct device_attribute dev_attr_ram_size =
@@ -1100,7 +1100,7 @@ static ssize_t pmem_size_show(struct device *dev, struct device_attribute *attr,
struct cxl_mem *cxlm = cxlmd->cxlm;
unsigned long long len = range_len(&cxlm->pmem_range);
- return sprintf(buf, "%#llx\n", len);
+ return sysfs_emit(buf, "%#llx\n", len);
}
static struct device_attribute dev_attr_pmem_size =
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH v4 2/4] cxl/mem: Fix synchronization mechanism for device removal vs ioctl operations
2021-04-01 14:33 [PATCH v4 0/4] cxl/mem: Fix memdev device setup Dan Williams
2021-04-01 14:33 ` [PATCH v4 1/4] cxl/mem: Use sysfs_emit() for attribute show routines Dan Williams
@ 2021-04-01 14:33 ` Dan Williams
2021-04-01 14:33 ` [PATCH v4 3/4] cxl/mem: Do not rely on device_add() side effects for dev_set_name() failures Dan Williams
2021-04-01 14:33 ` [PATCH v4 4/4] cxl/mem: Disable cxl device power management Dan Williams
3 siblings, 0 replies; 5+ messages in thread
From: Dan Williams @ 2021-04-01 14:33 UTC (permalink / raw)
To: linux-cxl
Cc: Jason Gunthorpe, Jason Gunthorpe, linux-kernel, vishal.l.verma,
ira.weiny, alison.schofield
The percpu_ref to gate whether cxl_memdev_ioctl() is free to use the
driver context (@cxlm) to issue I/O is overkill, implemented incorrectly
(missing a device reference before accessing the percpu_ref), and the
complexities of shutting down a percpu_ref contributed to a bug in the
error unwind in cxl_mem_add_memdev() (missing put_device() to be fixed
separately).
Use an rwsem to explicitly synchronize the usage of cxlmd->cxlm, and add
the missing reference counting for cxlmd in cxl_memdev_open() and
cxl_memdev_release_file().
Fixes: b39cb1052a5c ("cxl/mem: Register CXL memX devices")
Reported-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
drivers/cxl/mem.c | 97 +++++++++++++++++++++++++++--------------------------
1 file changed, 50 insertions(+), 47 deletions(-)
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index 30bf4f0f3c17..c77d375fad95 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -96,21 +96,18 @@ struct mbox_cmd {
* @dev: driver core device object
* @cdev: char dev core object for ioctl operations
* @cxlm: pointer to the parent device driver data
- * @ops_active: active user of @cxlm in ops handlers
- * @ops_dead: completion when all @cxlm ops users have exited
* @id: id number of this memdev instance.
*/
struct cxl_memdev {
struct device dev;
struct cdev cdev;
struct cxl_mem *cxlm;
- struct percpu_ref ops_active;
- struct completion ops_dead;
int id;
};
static int cxl_mem_major;
static DEFINE_IDA(cxl_memdev_ida);
+static DECLARE_RWSEM(cxl_memdev_rwsem);
static struct dentry *cxl_debugfs;
static bool cxl_raw_allow_all;
@@ -776,26 +773,43 @@ static long __cxl_memdev_ioctl(struct cxl_memdev *cxlmd, unsigned int cmd,
static long cxl_memdev_ioctl(struct file *file, unsigned int cmd,
unsigned long arg)
{
- struct cxl_memdev *cxlmd;
- struct inode *inode;
- int rc = -ENOTTY;
+ struct cxl_memdev *cxlmd = file->private_data;
+ int rc = -ENXIO;
- inode = file_inode(file);
- cxlmd = container_of(inode->i_cdev, typeof(*cxlmd), cdev);
+ down_read(&cxl_memdev_rwsem);
+ if (cxlmd->cxlm)
+ rc = __cxl_memdev_ioctl(cxlmd, cmd, arg);
+ up_read(&cxl_memdev_rwsem);
- if (!percpu_ref_tryget_live(&cxlmd->ops_active))
- return -ENXIO;
+ return rc;
+}
- rc = __cxl_memdev_ioctl(cxlmd, cmd, arg);
+static int cxl_memdev_open(struct inode *inode, struct file *file)
+{
+ struct cxl_memdev *cxlmd =
+ container_of(inode->i_cdev, typeof(*cxlmd), cdev);
- percpu_ref_put(&cxlmd->ops_active);
+ get_device(&cxlmd->dev);
+ file->private_data = cxlmd;
- return rc;
+ return 0;
+}
+
+static int cxl_memdev_release_file(struct inode *inode, struct file *file)
+{
+ struct cxl_memdev *cxlmd =
+ container_of(inode->i_cdev, typeof(*cxlmd), cdev);
+
+ put_device(&cxlmd->dev);
+
+ return 0;
}
static const struct file_operations cxl_memdev_fops = {
.owner = THIS_MODULE,
.unlocked_ioctl = cxl_memdev_ioctl,
+ .open = cxl_memdev_open,
+ .release = cxl_memdev_release_file,
.compat_ioctl = compat_ptr_ioctl,
.llseek = noop_llseek,
};
@@ -1049,7 +1063,6 @@ static void cxl_memdev_release(struct device *dev)
{
struct cxl_memdev *cxlmd = to_cxl_memdev(dev);
- percpu_ref_exit(&cxlmd->ops_active);
ida_free(&cxl_memdev_ida, cxlmd->id);
kfree(cxlmd);
}
@@ -1150,24 +1163,21 @@ static const struct device_type cxl_memdev_type = {
.groups = cxl_memdev_attribute_groups,
};
-static void cxlmdev_unregister(void *_cxlmd)
+static void cxl_memdev_shutdown(struct cxl_memdev *cxlmd)
{
- struct cxl_memdev *cxlmd = _cxlmd;
- struct device *dev = &cxlmd->dev;
-
- percpu_ref_kill(&cxlmd->ops_active);
- cdev_device_del(&cxlmd->cdev, dev);
- wait_for_completion(&cxlmd->ops_dead);
+ down_write(&cxl_memdev_rwsem);
cxlmd->cxlm = NULL;
- put_device(dev);
+ up_write(&cxl_memdev_rwsem);
}
-static void cxlmdev_ops_active_release(struct percpu_ref *ref)
+static void cxl_memdev_unregister(void *_cxlmd)
{
- struct cxl_memdev *cxlmd =
- container_of(ref, typeof(*cxlmd), ops_active);
+ struct cxl_memdev *cxlmd = _cxlmd;
+ struct device *dev = &cxlmd->dev;
- complete(&cxlmd->ops_dead);
+ cdev_device_del(&cxlmd->cdev, dev);
+ cxl_memdev_shutdown(cxlmd);
+ put_device(dev);
}
static int cxl_mem_add_memdev(struct cxl_mem *cxlm)
@@ -1181,17 +1191,6 @@ static int cxl_mem_add_memdev(struct cxl_mem *cxlm)
cxlmd = kzalloc(sizeof(*cxlmd), GFP_KERNEL);
if (!cxlmd)
return -ENOMEM;
- init_completion(&cxlmd->ops_dead);
-
- /*
- * @cxlm is deallocated when the driver unbinds so operations
- * that are using it need to hold a live reference.
- */
- cxlmd->cxlm = cxlm;
- rc = percpu_ref_init(&cxlmd->ops_active, cxlmdev_ops_active_release, 0,
- GFP_KERNEL);
- if (rc)
- goto err_ref;
rc = ida_alloc_range(&cxl_memdev_ida, 0, CXL_MEM_MAX_DEVS, GFP_KERNEL);
if (rc < 0)
@@ -1209,23 +1208,27 @@ static int cxl_mem_add_memdev(struct cxl_mem *cxlm)
cdev = &cxlmd->cdev;
cdev_init(cdev, &cxl_memdev_fops);
+ /*
+ * Activate ioctl operations, no cxl_memdev_rwsem manipulation
+ * needed as this is ordered with cdev_add() publishing the device.
+ */
+ cxlmd->cxlm = cxlm;
+
rc = cdev_device_add(cdev, dev);
if (rc)
goto err_add;
- return devm_add_action_or_reset(dev->parent, cxlmdev_unregister, cxlmd);
+ return devm_add_action_or_reset(dev->parent, cxl_memdev_unregister,
+ cxlmd);
err_add:
- ida_free(&cxl_memdev_ida, cxlmd->id);
-err_id:
/*
- * Theoretically userspace could have already entered the fops,
- * so flush ops_active.
+ * The cdev was briefly live, shutdown any ioctl operations that
+ * saw that state.
*/
- percpu_ref_kill(&cxlmd->ops_active);
- wait_for_completion(&cxlmd->ops_dead);
- percpu_ref_exit(&cxlmd->ops_active);
-err_ref:
+ cxl_memdev_shutdown(cxlmd);
+ ida_free(&cxl_memdev_ida, cxlmd->id);
+err_id:
kfree(cxlmd);
return rc;
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH v4 3/4] cxl/mem: Do not rely on device_add() side effects for dev_set_name() failures
2021-04-01 14:33 [PATCH v4 0/4] cxl/mem: Fix memdev device setup Dan Williams
2021-04-01 14:33 ` [PATCH v4 1/4] cxl/mem: Use sysfs_emit() for attribute show routines Dan Williams
2021-04-01 14:33 ` [PATCH v4 2/4] cxl/mem: Fix synchronization mechanism for device removal vs ioctl operations Dan Williams
@ 2021-04-01 14:33 ` Dan Williams
2021-04-01 14:33 ` [PATCH v4 4/4] cxl/mem: Disable cxl device power management Dan Williams
3 siblings, 0 replies; 5+ messages in thread
From: Dan Williams @ 2021-04-01 14:33 UTC (permalink / raw)
To: linux-cxl
Cc: Jason Gunthorpe, Jason Gunthorpe, linux-kernel, vishal.l.verma,
ira.weiny, alison.schofield
While device_add() will happen to catch dev_set_name() failures it is a
broken pattern to follow given that the core may try to fall back to a
different name.
Add explicit checking for dev_set_name() failures to be cleaned up by
put_device(). Skip cdev_device_add() and proceed directly to
put_device() if the name set fails.
This type of bug is easier to see if 'alloc' is split from 'add'
operations that require put_device() on failure. So cxl_memdev_alloc()
is split out as a result.
Fixes: b39cb1052a5c ("cxl/mem: Register CXL memX devices")
Reported-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
drivers/cxl/mem.c | 39 +++++++++++++++++++++++++++++----------
1 file changed, 29 insertions(+), 10 deletions(-)
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index c77d375fad95..e59b373694d3 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -1180,7 +1180,7 @@ static void cxl_memdev_unregister(void *_cxlmd)
put_device(dev);
}
-static int cxl_mem_add_memdev(struct cxl_mem *cxlm)
+static struct cxl_memdev *cxl_memdev_alloc(struct cxl_mem *cxlm)
{
struct pci_dev *pdev = cxlm->pdev;
struct cxl_memdev *cxlmd;
@@ -1190,11 +1190,11 @@ static int cxl_mem_add_memdev(struct cxl_mem *cxlm)
cxlmd = kzalloc(sizeof(*cxlmd), GFP_KERNEL);
if (!cxlmd)
- return -ENOMEM;
+ return ERR_PTR(-ENOMEM);
rc = ida_alloc_range(&cxl_memdev_ida, 0, CXL_MEM_MAX_DEVS, GFP_KERNEL);
if (rc < 0)
- goto err_id;
+ goto err;
cxlmd->id = rc;
dev = &cxlmd->dev;
@@ -1203,10 +1203,31 @@ static int cxl_mem_add_memdev(struct cxl_mem *cxlm)
dev->bus = &cxl_bus_type;
dev->devt = MKDEV(cxl_mem_major, cxlmd->id);
dev->type = &cxl_memdev_type;
- dev_set_name(dev, "mem%d", cxlmd->id);
cdev = &cxlmd->cdev;
cdev_init(cdev, &cxl_memdev_fops);
+ return cxlmd;
+
+err:
+ kfree(cxlmd);
+ return ERR_PTR(rc);
+}
+
+static int cxl_mem_add_memdev(struct cxl_mem *cxlm)
+{
+ struct cxl_memdev *cxlmd;
+ struct device *dev;
+ struct cdev *cdev;
+ int rc;
+
+ cxlmd = cxl_memdev_alloc(cxlm);
+ if (IS_ERR(cxlmd))
+ return PTR_ERR(cxlmd);
+
+ dev = &cxlmd->dev;
+ rc = dev_set_name(dev, "mem%d", cxlmd->id);
+ if (rc)
+ goto err;
/*
* Activate ioctl operations, no cxl_memdev_rwsem manipulation
@@ -1214,23 +1235,21 @@ static int cxl_mem_add_memdev(struct cxl_mem *cxlm)
*/
cxlmd->cxlm = cxlm;
+ cdev = &cxlmd->cdev;
rc = cdev_device_add(cdev, dev);
if (rc)
- goto err_add;
+ goto err;
return devm_add_action_or_reset(dev->parent, cxl_memdev_unregister,
cxlmd);
-err_add:
+err:
/*
* The cdev was briefly live, shutdown any ioctl operations that
* saw that state.
*/
cxl_memdev_shutdown(cxlmd);
- ida_free(&cxl_memdev_ida, cxlmd->id);
-err_id:
- kfree(cxlmd);
-
+ put_device(dev);
return rc;
}
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH v4 4/4] cxl/mem: Disable cxl device power management
2021-04-01 14:33 [PATCH v4 0/4] cxl/mem: Fix memdev device setup Dan Williams
` (2 preceding siblings ...)
2021-04-01 14:33 ` [PATCH v4 3/4] cxl/mem: Do not rely on device_add() side effects for dev_set_name() failures Dan Williams
@ 2021-04-01 14:33 ` Dan Williams
3 siblings, 0 replies; 5+ messages in thread
From: Dan Williams @ 2021-04-01 14:33 UTC (permalink / raw)
To: linux-cxl
Cc: Ben Widawsky, linux-kernel, vishal.l.verma, ira.weiny, alison.schofield
There is no power management of cxl virtual devices, disable
device-power-management and runtime-power-management to prevent
userspace from growing expectations of those attributes appearing. They
can be added back in the future if needed.
Reviewed-by: Ben Widawsky <ben.widawsky@intel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
drivers/cxl/mem.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index e59b373694d3..e3003f49b329 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -1203,6 +1203,7 @@ static struct cxl_memdev *cxl_memdev_alloc(struct cxl_mem *cxlm)
dev->bus = &cxl_bus_type;
dev->devt = MKDEV(cxl_mem_major, cxlmd->id);
dev->type = &cxl_memdev_type;
+ device_set_pm_not_required(dev);
cdev = &cxlmd->cdev;
cdev_init(cdev, &cxl_memdev_fops);
^ permalink raw reply related [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-04-01 18:17 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-01 14:33 [PATCH v4 0/4] cxl/mem: Fix memdev device setup Dan Williams
2021-04-01 14:33 ` [PATCH v4 1/4] cxl/mem: Use sysfs_emit() for attribute show routines Dan Williams
2021-04-01 14:33 ` [PATCH v4 2/4] cxl/mem: Fix synchronization mechanism for device removal vs ioctl operations Dan Williams
2021-04-01 14:33 ` [PATCH v4 3/4] cxl/mem: Do not rely on device_add() side effects for dev_set_name() failures Dan Williams
2021-04-01 14:33 ` [PATCH v4 4/4] cxl/mem: Disable cxl device power management Dan Williams
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).