[PATCH v6 0/4] user space client interface driver

[PATCH v6 0/4] user space client interface driver

[Hemant Kumar]

V6:
- Moved uci.c to mhi directory.
- Updated Kconfig to add module information.
- Updated Makefile to rename uci object file name as mhi_uci
- Removed kref for open count

V5:
- Removed mhi_uci_drv structure.
- Used idr instead of creating global list of uci devices.
- Used kref instead of local ref counting for uci device and
  open count.
- Removed unlikely macro.

V4:
- Fix locking to protect proper struct members.
- Updated documentation describing uci client driver use cases.
- Fixed uci ref counting in mhi_uci_open for error case.
- Addressed style related review comments.

V3: Added documentation for MHI UCI driver.

V2: Added mutex lock to prevent multiple readers to access same
mhi buffer which can result into use after free.

Hemant Kumar (4):
  bus: mhi: core: Add helper API to return number of free TREs
  bus: mhi: core: Move MHI_MAX_MTU to external header file
  docs: Add documentation for userspace client interface
  bus: mhi: Add userspace client interface driver

 Documentation/mhi/index.rst     |   1 +
 Documentation/mhi/uci.rst       |  39 +++
 drivers/bus/mhi/Kconfig         |  13 +
 drivers/bus/mhi/Makefile        |   4 +
 drivers/bus/mhi/core/internal.h |   1 -
 drivers/bus/mhi/core/main.c     |  12 +
 drivers/bus/mhi/uci.c           | 657 ++++++++++++++++++++++++++++++++++++++++
 include/linux/mhi.h             |  12 +
 8 files changed, 738 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/mhi/uci.rst
 create mode 100644 drivers/bus/mhi/uci.c

-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

[PATCH v6 1/4] bus: mhi: core: Add helper API to return number of free TREs

[Hemant Kumar]

Introduce mhi_get_no_free_descriptors() API to return number
of TREs available to queue buffer. MHI clients can use this
API to know before hand if ring is full without calling queue
API.

Signed-off-by: Hemant Kumar <hemantk@codeaurora.org>
---
 drivers/bus/mhi/core/main.c | 12 ++++++++++++
 include/linux/mhi.h         |  9 +++++++++
 2 files changed, 21 insertions(+)

diff --git a/drivers/bus/mhi/core/main.c b/drivers/bus/mhi/core/main.c
index 2cff5dd..0599e7d 100644
--- a/drivers/bus/mhi/core/main.c
+++ b/drivers/bus/mhi/core/main.c
@@ -258,6 +258,18 @@ int mhi_destroy_device(struct device *dev, void *data)
 	return 0;
 }
 
+int mhi_get_no_free_descriptors(struct mhi_device *mhi_dev,
+				enum dma_data_direction dir)
+{
+	struct mhi_controller *mhi_cntrl = mhi_dev->mhi_cntrl;
+	struct mhi_chan *mhi_chan = (dir == DMA_TO_DEVICE) ?
+		mhi_dev->ul_chan : mhi_dev->dl_chan;
+	struct mhi_ring *tre_ring = &mhi_chan->tre_ring;
+
+	return get_nr_avail_ring_elements(mhi_cntrl, tre_ring);
+}
+EXPORT_SYMBOL_GPL(mhi_get_no_free_descriptors);
+
 void mhi_notify(struct mhi_device *mhi_dev, enum mhi_callback cb_reason)
 {
 	struct mhi_driver *mhi_drv;
diff --git a/include/linux/mhi.h b/include/linux/mhi.h
index a35d876..6565528 100644
--- a/include/linux/mhi.h
+++ b/include/linux/mhi.h
@@ -600,6 +600,15 @@ void mhi_set_mhi_state(struct mhi_controller *mhi_cntrl,
 void mhi_notify(struct mhi_device *mhi_dev, enum mhi_callback cb_reason);
 
 /**
+ * mhi_get_no_free_descriptors - Get transfer ring length
+ * Get # of TD available to queue buffers
+ * @mhi_dev: Device associated with the channels
+ * @dir: Direction of the channel
+ */
+int mhi_get_no_free_descriptors(struct mhi_device *mhi_dev,
+				enum dma_data_direction dir);
+
+/**
  * mhi_prepare_for_power_up - Do pre-initialization before power up.
  *                            This is optional, call this before power up if
  *                            the controller does not want bus framework to
-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

Re: [PATCH v6 1/4] bus: mhi: core: Add helper API to return number of free TREs

[Manivannan Sadhasivam]

On Wed, Sep 16, 2020 at 12:56:04PM -0700, Hemant Kumar wrote:
> Introduce mhi_get_no_free_descriptors() API to return number
> of TREs available to queue buffer. MHI clients can use this
> API to know before hand if ring is full without calling queue
> API.
> 
> Signed-off-by: Hemant Kumar <hemantk@codeaurora.org>
> ---
>  drivers/bus/mhi/core/main.c | 12 ++++++++++++
>  include/linux/mhi.h         |  9 +++++++++
>  2 files changed, 21 insertions(+)
> 
> diff --git a/drivers/bus/mhi/core/main.c b/drivers/bus/mhi/core/main.c
> index 2cff5dd..0599e7d 100644
> --- a/drivers/bus/mhi/core/main.c
> +++ b/drivers/bus/mhi/core/main.c
> @@ -258,6 +258,18 @@ int mhi_destroy_device(struct device *dev, void *data)
>  	return 0;
>  }
>  
> +int mhi_get_no_free_descriptors(struct mhi_device *mhi_dev,
> +				enum dma_data_direction dir)
> +{
> +	struct mhi_controller *mhi_cntrl = mhi_dev->mhi_cntrl;
> +	struct mhi_chan *mhi_chan = (dir == DMA_TO_DEVICE) ?
> +		mhi_dev->ul_chan : mhi_dev->dl_chan;
> +	struct mhi_ring *tre_ring = &mhi_chan->tre_ring;
> +
> +	return get_nr_avail_ring_elements(mhi_cntrl, tre_ring);

Hmm, so this is essentially a wrapper for get_nr_avail_ring_elements().
Why can't you call this API directly?

> +}
> +EXPORT_SYMBOL_GPL(mhi_get_no_free_descriptors);
> +
>  void mhi_notify(struct mhi_device *mhi_dev, enum mhi_callback cb_reason)
>  {
>  	struct mhi_driver *mhi_drv;
> diff --git a/include/linux/mhi.h b/include/linux/mhi.h
> index a35d876..6565528 100644
> --- a/include/linux/mhi.h
> +++ b/include/linux/mhi.h
> @@ -600,6 +600,15 @@ void mhi_set_mhi_state(struct mhi_controller *mhi_cntrl,
>  void mhi_notify(struct mhi_device *mhi_dev, enum mhi_callback cb_reason);
>  
>  /**
> + * mhi_get_no_free_descriptors - Get transfer ring length

mhi_get_nr_free_descriptors?

> + * Get # of TD available to queue buffers
> + * @mhi_dev: Device associated with the channels
> + * @dir: Direction of the channel
> + */
> +int mhi_get_no_free_descriptors(struct mhi_device *mhi_dev,
> +				enum dma_data_direction dir);

Align enum with start of "("

Thanks,
Mani

> +
> +/**
>   * mhi_prepare_for_power_up - Do pre-initialization before power up.
>   *                            This is optional, call this before power up if
>   *                            the controller does not want bus framework to
> -- 
> The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
> a Linux Foundation Collaborative Project
> 

[PATCH v6 2/4] bus: mhi: core: Move MHI_MAX_MTU to external header file

[Hemant Kumar]

Currently this macro is defined in internal MHI header as
a TRE length mask. Moving it to external header allows MHI
client drivers to set this upper bound for the transmit
buffer size.

Signed-off-by: Hemant Kumar <hemantk@codeaurora.org>
---
 drivers/bus/mhi/core/internal.h | 1 -
 include/linux/mhi.h             | 3 +++
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/bus/mhi/core/internal.h b/drivers/bus/mhi/core/internal.h
index 7989269..4abf0cf 100644
--- a/drivers/bus/mhi/core/internal.h
+++ b/drivers/bus/mhi/core/internal.h
@@ -453,7 +453,6 @@ enum mhi_pm_state {
 #define CMD_EL_PER_RING			128
 #define PRIMARY_CMD_RING		0
 #define MHI_DEV_WAKE_DB			127
-#define MHI_MAX_MTU			0xffff
 #define MHI_RANDOM_U32_NONZERO(bmsk)	(prandom_u32_max(bmsk) + 1)
 
 enum mhi_er_type {
diff --git a/include/linux/mhi.h b/include/linux/mhi.h
index 6565528..610f3b0 100644
--- a/include/linux/mhi.h
+++ b/include/linux/mhi.h
@@ -16,6 +16,9 @@
 #include <linux/wait.h>
 #include <linux/workqueue.h>
 
+/* MHI client drivers to set this upper bound for tx buffer */
+#define MHI_MAX_MTU 0xffff
+
 #define MHI_MAX_OEM_PK_HASH_SEGMENTS 16
 
 struct mhi_chan;
-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

Re: [PATCH v6 2/4] bus: mhi: core: Move MHI_MAX_MTU to external header file

[Manivannan Sadhasivam]

On Wed, Sep 16, 2020 at 12:56:05PM -0700, Hemant Kumar wrote:
> Currently this macro is defined in internal MHI header as
> a TRE length mask. Moving it to external header allows MHI
> client drivers to set this upper bound for the transmit
> buffer size.
> 
> Signed-off-by: Hemant Kumar <hemantk@codeaurora.org>

Reviewed-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>

Thanks,
Mani

> ---
>  drivers/bus/mhi/core/internal.h | 1 -
>  include/linux/mhi.h             | 3 +++
>  2 files changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/bus/mhi/core/internal.h b/drivers/bus/mhi/core/internal.h
> index 7989269..4abf0cf 100644
> --- a/drivers/bus/mhi/core/internal.h
> +++ b/drivers/bus/mhi/core/internal.h
> @@ -453,7 +453,6 @@ enum mhi_pm_state {
>  #define CMD_EL_PER_RING			128
>  #define PRIMARY_CMD_RING		0
>  #define MHI_DEV_WAKE_DB			127
> -#define MHI_MAX_MTU			0xffff
>  #define MHI_RANDOM_U32_NONZERO(bmsk)	(prandom_u32_max(bmsk) + 1)
>  
>  enum mhi_er_type {
> diff --git a/include/linux/mhi.h b/include/linux/mhi.h
> index 6565528..610f3b0 100644
> --- a/include/linux/mhi.h
> +++ b/include/linux/mhi.h
> @@ -16,6 +16,9 @@
>  #include <linux/wait.h>
>  #include <linux/workqueue.h>
>  
> +/* MHI client drivers to set this upper bound for tx buffer */
> +#define MHI_MAX_MTU 0xffff
> +
>  #define MHI_MAX_OEM_PK_HASH_SEGMENTS 16
>  
>  struct mhi_chan;
> -- 
> The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
> a Linux Foundation Collaborative Project
> 

[PATCH v6 3/4] docs: Add documentation for userspace client interface

[Hemant Kumar]

MHI userspace client driver is creating device file node
for user application to perform file operations. File
operations are handled by MHI core driver. Currently
Loopback MHI channel is supported by this driver.

Signed-off-by: Hemant Kumar <hemantk@codeaurora.org>
---
 Documentation/mhi/index.rst |  1 +
 Documentation/mhi/uci.rst   | 39 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 40 insertions(+)
 create mode 100644 Documentation/mhi/uci.rst

diff --git a/Documentation/mhi/index.rst b/Documentation/mhi/index.rst
index 1d8dec3..c75a371 100644
--- a/Documentation/mhi/index.rst
+++ b/Documentation/mhi/index.rst
@@ -9,6 +9,7 @@ MHI
 
    mhi
    topology
+   uci
 
 .. only::  subproject and html
 
diff --git a/Documentation/mhi/uci.rst b/Documentation/mhi/uci.rst
new file mode 100644
index 0000000..5d92939
--- /dev/null
+++ b/Documentation/mhi/uci.rst
@@ -0,0 +1,39 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=================================
+Userspace Client Interface (UCI)
+=================================
+
+UCI driver enables userspace clients to communicate to external MHI devices
+like modem and WLAN. It creates standard character device file nodes for user
+space clients to perform open, read, write, poll and close file operations.
+
+Device file node is created with format:-
+
+/dev/mhi_<controller_name>_<mhi_device_name>
+
+controller_name is the name of underlying bus used to transfer data.
+mhi_device_name is the name of the MHI channel being used by MHI client in
+userspace to send or receive data using MHI protocol.
+
+There is a separate character device file node created for each channel specified
+in mhi device id table. MHI channels are statically defined by MHI specification.
+Driver currently supports LOOPBACK channel 0 (Host to device) and 1 (Device to Host).
+
+LOOPBACK Channel
+----------------
+
+Userspace MHI client using LOOPBACK channel opens device file node. As part of
+open operation TREs to transfer ring of LOOPBACK channel 1 gets queued and channel
+doorbell is rung. When userspace MHI client performs write operation on device node,
+data buffer gets queued as a TRE to transfer ring of LOOPBACK channel 0. MHI Core
+driver rings the channel doorbell for MHI device to move data over underlying bus.
+When userspace MHI client driver performs read operation, same data gets looped back
+to MHI host using LOOPBACK channel 1. LOOPBACK channel is used to verify data path
+and data integrity between MHI Host and MHI device.
+
+Other Use Cases
+---------------
+
+Getting MHI device specific diagnostics information to userspace MHI diag client
+using DIAG channel 4 (Host to device) and 5 (Device to Host).
-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

[PATCH v6 4/4] bus: mhi: Add userspace client interface driver

[Hemant Kumar]

This MHI client driver allows userspace clients to transfer
raw data between MHI device and host using standard file operations.
Device file node is created with format

/dev/mhi_<controller_name>_<mhi_device_name>

Currently it supports LOOPBACK channel.

Signed-off-by: Hemant Kumar <hemantk@codeaurora.org>
---
 drivers/bus/mhi/Kconfig  |  13 +
 drivers/bus/mhi/Makefile |   4 +
 drivers/bus/mhi/uci.c    | 657 +++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 674 insertions(+)
 create mode 100644 drivers/bus/mhi/uci.c

diff --git a/drivers/bus/mhi/Kconfig b/drivers/bus/mhi/Kconfig
index 6a217ff..8aebe8b 100644
--- a/drivers/bus/mhi/Kconfig
+++ b/drivers/bus/mhi/Kconfig
@@ -20,3 +20,16 @@ config MHI_BUS_DEBUG
 	 Enable debugfs support for use with the MHI transport. Allows
 	 reading and/or modifying some values within the MHI controller
 	 for debug and test purposes.
+
+config MHI_UCI
+       tristate "MHI UCI"
+       depends on MHI_BUS
+       help
+	 MHI based userspace client interface driver is used for transferring
+	 raw data between host and device using standard file operations from
+	 userspace. Open, read, write, and close operations are supported
+	 by this driver. Please check mhi_uci_match_table for all supported
+	 channels that are exposed to userspace.
+
+	 To compile this driver as a module, choose M here: the module will be
+	 called mhi_uci.
diff --git a/drivers/bus/mhi/Makefile b/drivers/bus/mhi/Makefile
index 19e6443..80feefb 100644
--- a/drivers/bus/mhi/Makefile
+++ b/drivers/bus/mhi/Makefile
@@ -1,2 +1,6 @@
 # core layer
 obj-y += core/
+
+# MHI client
+mhi_uci-y := uci.o
+obj-$(CONFIG_MHI_UCI) += mhi_uci.o
diff --git a/drivers/bus/mhi/uci.c b/drivers/bus/mhi/uci.c
new file mode 100644
index 0000000..d6758f2
--- /dev/null
+++ b/drivers/bus/mhi/uci.c
@@ -0,0 +1,657 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright (c) 2018-2020, The Linux Foundation. All rights reserved.*/
+
+#include <linux/kernel.h>
+#include <linux/mhi.h>
+#include <linux/mod_devicetable.h>
+#include <linux/module.h>
+#include <linux/poll.h>
+
+#define DEVICE_NAME "mhi"
+#define MHI_UCI_DRIVER_NAME "mhi_uci"
+#define MAX_UCI_MINORS (128)
+
+static DEFINE_IDR(uci_idr);
+static DEFINE_MUTEX(uci_idr_mutex);
+static struct class *uci_dev_class;
+static int uci_dev_major;
+
+/**
+ * struct uci_chan - MHI channel for a uci device
+ * @wq: wait queue for reader/writer
+ * @lock: spin lock
+ * @pending: list of rx buffers userspace is waiting to read
+ * @cur_buf: current buffer userspace is reading
+ * @rx_size: size of the current rx buffer userspace is reading
+ */
+struct uci_chan {
+	wait_queue_head_t wq;
+
+	/* protects pending and cur_buf members in bh context */
+	spinlock_t lock;
+
+	struct list_head pending;
+	struct uci_buf *cur_buf;
+	size_t rx_size;
+};
+
+/**
+ * struct uci_buf - uci buffer
+ * @data: data buffer
+ * @len: length of data buffer
+ * @node: list node of the uci buffer
+ */
+struct uci_buf {
+	void *data;
+	size_t len;
+	struct list_head node;
+};
+
+/**
+ * struct uci_dev - MHI uci device
+ * @minor: uci device node minor number
+ * @mhi_dev: associated mhi device object
+ * @chan: MHI channel name
+ * @lock: mutex lock
+ * @ul_chan: uplink uci channel object
+ * @dl_chan: downlink uci channel object
+ * @mtu: max tx buffer length
+ * @actual_mtu: maximum size of incoming buffer
+ * @open: open called for device node
+ * @enabled: uci device probed
+ * @ref_count: uci_dev reference count
+ */
+struct uci_dev {
+	unsigned int minor;
+	struct mhi_device *mhi_dev;
+	const char *chan;
+
+	/* protects uci_dev struct members */
+	struct mutex lock;
+
+	struct uci_chan ul_chan;
+	struct uci_chan dl_chan;
+	size_t mtu;
+	size_t actual_mtu;
+	bool enabled;
+	struct kref ref_count;
+};
+
+static int mhi_queue_inbound(struct uci_dev *udev)
+{
+	struct mhi_device *mhi_dev = udev->mhi_dev;
+	struct device *dev = &mhi_dev->dev;
+	size_t mtu = udev->mtu;
+	size_t actual_mtu = udev->actual_mtu;
+	int nr_trbs, i, ret = -EIO;
+	void *buf;
+	struct uci_buf *uci_buf;
+
+	nr_trbs = mhi_get_no_free_descriptors(mhi_dev, DMA_FROM_DEVICE);
+
+	for (i = 0; i < nr_trbs; i++) {
+		buf = kmalloc(mtu, GFP_KERNEL);
+		if (!buf)
+			return -ENOMEM;
+
+		uci_buf = buf + actual_mtu;
+		uci_buf->data = buf;
+
+		dev_dbg(dev, "Allocated buf %d of %d size %ld\n", i, nr_trbs,
+			actual_mtu);
+
+		ret = mhi_queue_buf(mhi_dev, DMA_FROM_DEVICE, buf, actual_mtu,
+				    MHI_EOT);
+		if (ret) {
+			kfree(buf);
+			dev_err(dev, "Failed to queue buffer %d\n", i);
+			return ret;
+		}
+	}
+
+	return ret;
+}
+
+static void mhi_uci_dev_release(struct kref *ref)
+{
+	struct uci_dev *udev =
+		container_of(ref, struct uci_dev, ref_count);
+
+	mutex_destroy(&udev->lock);
+
+	dev_set_drvdata(&udev->mhi_dev->dev, NULL);
+
+	kfree(udev);
+}
+
+static int mhi_uci_release(struct inode *inode, struct file *file)
+{
+	struct uci_dev *udev = file->private_data;
+	struct uci_buf *itr, *tmp;
+	struct uci_chan *uchan;
+
+	if (kref_read(&udev->ref_count) > 2)
+		goto exit_uci_release;
+
+	if (udev->enabled)
+		mhi_unprepare_from_transfer(udev->mhi_dev);
+
+	/* clean inbound channel */
+	uchan = &udev->dl_chan;
+
+	spin_lock_bh(&uchan->lock);
+	list_for_each_entry_safe(itr, tmp, &uchan->pending, node) {
+		list_del(&itr->node);
+		kfree(itr->data);
+	}
+
+	if (uchan->cur_buf)
+		kfree(uchan->cur_buf->data);
+
+	uchan->cur_buf = NULL;
+	spin_unlock_bh(&uchan->lock);
+
+exit_uci_release:
+	kref_put(&udev->ref_count, mhi_uci_dev_release);
+
+	return 0;
+}
+
+static __poll_t mhi_uci_poll(struct file *file, poll_table *wait)
+{
+	struct uci_dev *udev = file->private_data;
+	struct mhi_device *mhi_dev = udev->mhi_dev;
+	struct device *dev = &mhi_dev->dev;
+	struct uci_chan *uchan;
+	__poll_t mask = 0;
+
+	poll_wait(file, &udev->dl_chan.wq, wait);
+	poll_wait(file, &udev->ul_chan.wq, wait);
+
+	if (!udev->enabled) {
+		mask = EPOLLERR;
+	} else {
+		uchan = &udev->dl_chan;
+		spin_lock_bh(&uchan->lock);
+		if (!list_empty(&uchan->pending) || uchan->cur_buf) {
+			dev_dbg(dev, "Client can read from node\n");
+			mask |= EPOLLIN | EPOLLRDNORM;
+		}
+		spin_unlock_bh(&uchan->lock);
+	}
+
+	if (!udev->enabled) {
+		mask |= EPOLLERR;
+	} else if (mhi_get_no_free_descriptors(mhi_dev, DMA_TO_DEVICE) > 0) {
+		dev_dbg(dev, "Client can write to node\n");
+		mask |= EPOLLOUT | EPOLLWRNORM;
+	}
+
+	dev_dbg(dev, "Client attempted to poll, returning mask 0x%x\n", mask);
+
+	return mask;
+}
+
+static ssize_t mhi_uci_write(struct file *file,
+			     const char __user *buf,
+			     size_t count,
+			     loff_t *offp)
+{
+	struct uci_dev *udev = file->private_data;
+	struct mhi_device *mhi_dev = udev->mhi_dev;
+	struct device *dev = &mhi_dev->dev;
+	struct uci_chan *uchan = &udev->ul_chan;
+	size_t bytes_xfered = 0;
+	int ret, nr_avail = 0;
+
+	if (!buf || !count)
+		return -EINVAL;
+
+	/* confirm channel is active */
+	mutex_lock(&udev->lock);
+	if (!udev->enabled) {
+		ret = -ENODEV;
+		goto err_mtx_unlock;
+	}
+
+	dev_dbg(dev, "%s: to xfer: %lu bytes\n", __func__, count);
+
+	while (count) {
+		size_t xfer_size;
+		void *kbuf;
+		enum mhi_flags flags;
+
+		mutex_unlock(&udev->lock);
+		/* wait for free descriptors */
+		ret = wait_event_interruptible(uchan->wq,
+					       (!udev->enabled) ||
+				(nr_avail = mhi_get_no_free_descriptors(mhi_dev,
+					       DMA_TO_DEVICE)) > 0);
+
+		mutex_lock(&udev->lock);
+		if (ret == -ERESTARTSYS) {
+			dev_dbg(dev, "Exit signal caught for node\n");
+			goto err_mtx_unlock;
+		}
+
+		if (!udev->enabled) {
+			ret = -ENODEV;
+			goto err_mtx_unlock;
+		}
+
+		xfer_size = min_t(size_t, count, udev->mtu);
+		kbuf = kmalloc(xfer_size, GFP_KERNEL);
+		if (!kbuf) {
+			ret = -ENOMEM;
+			goto err_mtx_unlock;
+		}
+
+		ret = copy_from_user(kbuf, buf, xfer_size);
+		if (ret) {
+			kfree(kbuf);
+			ret = -EFAULT;
+			goto err_mtx_unlock;
+		}
+
+		/* if ring is full after this force EOT */
+		if (nr_avail > 1 && (count - xfer_size))
+			flags = MHI_CHAIN;
+		else
+			flags = MHI_EOT;
+
+		if (udev->enabled)
+			ret = mhi_queue_buf(mhi_dev, DMA_TO_DEVICE, kbuf,
+					    xfer_size, flags);
+		else
+			ret = -ENODEV;
+
+		if (ret) {
+			kfree(kbuf);
+			goto err_mtx_unlock;
+		}
+
+		bytes_xfered += xfer_size;
+		count -= xfer_size;
+		buf += xfer_size;
+	}
+
+	mutex_unlock(&udev->lock);
+	dev_dbg(dev, "%s: bytes xferred: %lu\n", __func__, bytes_xfered);
+
+	return bytes_xfered;
+
+err_mtx_unlock:
+	mutex_unlock(&udev->lock);
+
+	return ret;
+}
+
+static ssize_t mhi_uci_read(struct file *file,
+			    char __user *buf,
+			    size_t count,
+			    loff_t *ppos)
+{
+	struct uci_dev *udev = file->private_data;
+	struct mhi_device *mhi_dev = udev->mhi_dev;
+	struct uci_chan *uchan = &udev->dl_chan;
+	struct device *dev = &mhi_dev->dev;
+	struct uci_buf *ubuf;
+	char *ptr;
+	size_t to_copy;
+	int ret = 0;
+
+	if (!buf)
+		return -EINVAL;
+
+	mutex_lock(&udev->lock);
+	/* confirm channel is active */
+	if (!udev->enabled) {
+		ret = -ENODEV;
+		goto err_mtx_unlock;
+	}
+
+	spin_lock_bh(&uchan->lock);
+	/* No data available to read, wait */
+	if (!uchan->cur_buf && list_empty(&uchan->pending)) {
+		dev_dbg(dev, "No data available to read waiting\n");
+
+		spin_unlock_bh(&uchan->lock);
+		mutex_unlock(&udev->lock);
+		ret = wait_event_interruptible(uchan->wq,
+					       (!udev->enabled ||
+					      !list_empty(&uchan->pending)));
+
+		mutex_lock(&udev->lock);
+		if (ret == -ERESTARTSYS) {
+			dev_dbg(dev, "Exit signal caught for node\n");
+			goto err_mtx_unlock;
+		}
+
+		if (!udev->enabled) {
+			ret = -ENODEV;
+			goto err_mtx_unlock;
+		}
+		spin_lock_bh(&uchan->lock);
+	}
+
+	/* new read, get the next descriptor from the list */
+	if (!uchan->cur_buf) {
+		ubuf = list_first_entry_or_null(&uchan->pending,
+						struct uci_buf, node);
+		if (!ubuf) {
+			ret = -EIO;
+			goto err_spin_unlock;
+		}
+
+		list_del(&ubuf->node);
+		uchan->cur_buf = ubuf;
+		uchan->rx_size = ubuf->len;
+		dev_dbg(dev, "Got pkt of size: %zu\n", uchan->rx_size);
+	}
+
+	ubuf = uchan->cur_buf;
+
+	/* Copy the buffer to user space */
+	to_copy = min_t(size_t, count, uchan->rx_size);
+	ptr = ubuf->data + (ubuf->len - uchan->rx_size);
+	spin_unlock_bh(&uchan->lock);
+
+	ret = copy_to_user(buf, ptr, to_copy);
+	if (ret) {
+		ret = -EFAULT;
+		goto err_mtx_unlock;
+	}
+
+	spin_lock_bh(&uchan->lock);
+
+	dev_dbg(dev, "Copied %lu of %lu bytes\n", to_copy, uchan->rx_size);
+	uchan->rx_size -= to_copy;
+
+	/* we finished with this buffer, queue it back to hardware */
+	if (!uchan->rx_size) {
+		uchan->cur_buf = NULL;
+
+		if (udev->enabled)
+			ret = mhi_queue_buf(mhi_dev, DMA_FROM_DEVICE,
+					    ubuf->data,
+					    udev->actual_mtu, MHI_EOT);
+		else
+			ret = -ENODEV;
+
+		if (ret) {
+			dev_err(dev, "Failed to recycle element: %d\n", ret);
+			kfree(ubuf->data);
+			goto err_spin_unlock;
+		}
+	}
+	spin_unlock_bh(&uchan->lock);
+	mutex_unlock(&udev->lock);
+
+	dev_dbg(dev, "%s: Returning %lu bytes\n", __func__, to_copy);
+
+	return to_copy;
+
+err_spin_unlock:
+	spin_unlock_bh(&uchan->lock);
+err_mtx_unlock:
+	mutex_unlock(&udev->lock);
+	return ret;
+}
+
+static int mhi_uci_open(struct inode *inode, struct file *filp)
+{
+	struct uci_dev *udev = NULL;
+	unsigned int minor = iminor(inode);
+	int ret = -EIO;
+	struct uci_buf *buf_itr, *tmp;
+	struct uci_chan *dl_chan;
+	struct mhi_device *mhi_dev;
+	struct device *dev;
+
+	mutex_lock(&uci_idr_mutex);
+	udev = idr_find(&uci_idr, minor);
+	mutex_unlock(&uci_idr_mutex);
+	if (!udev) {
+		pr_err("uci dev: minor %d not found\n", minor);
+		ret = -ENODEV;
+		goto error_no_dev;
+	}
+
+	kref_get(&udev->ref_count);
+
+	mhi_dev = udev->mhi_dev;
+	dev = &mhi_dev->dev;
+
+	mutex_lock(&udev->lock);
+	if (kref_read(&udev->ref_count) > 2) {
+		dev_dbg(dev, "Node already opened\n");
+		goto exit_uci_open;
+	}
+
+	if (!udev->enabled) {
+		dev_info(dev, "Node exists, but is not in active state!\n");
+		goto error_open_chan;
+	}
+
+	dev_dbg(dev, "Starting channel\n");
+	ret = mhi_prepare_for_transfer(udev->mhi_dev);
+	if (ret) {
+		dev_err(dev, "Error starting transfer channels\n");
+		goto error_open_chan;
+	}
+
+	ret = mhi_queue_inbound(udev);
+	if (ret)
+		goto error_rx_queue;
+
+exit_uci_open:
+	filp->private_data = udev;
+	mutex_unlock(&udev->lock);
+
+	return 0;
+
+error_rx_queue:
+	dl_chan = &udev->dl_chan;
+	mhi_unprepare_from_transfer(udev->mhi_dev);
+	list_for_each_entry_safe(buf_itr, tmp, &dl_chan->pending, node) {
+		list_del(&buf_itr->node);
+		kfree(buf_itr->data);
+	}
+error_open_chan:
+	mutex_unlock(&udev->lock);
+	kref_put(&udev->ref_count, mhi_uci_dev_release);
+error_no_dev:
+	return ret;
+}
+
+static const struct file_operations mhidev_fops = {
+	.owner = THIS_MODULE,
+	.open = mhi_uci_open,
+	.release = mhi_uci_release,
+	.read = mhi_uci_read,
+	.write = mhi_uci_write,
+	.poll = mhi_uci_poll,
+};
+
+static void mhi_ul_xfer_cb(struct mhi_device *mhi_dev,
+			   struct mhi_result *mhi_result)
+{
+	struct uci_dev *udev = dev_get_drvdata(&mhi_dev->dev);
+	struct uci_chan *uchan = &udev->ul_chan;
+	struct device *dev = &mhi_dev->dev;
+
+	dev_dbg(dev, "status: %d xfer_len: %zu\n",
+		mhi_result->transaction_status, mhi_result->bytes_xferd);
+
+	kfree(mhi_result->buf_addr);
+
+	if (!mhi_result->transaction_status)
+		wake_up(&uchan->wq);
+}
+
+static void mhi_dl_xfer_cb(struct mhi_device *mhi_dev,
+			   struct mhi_result *mhi_result)
+{
+	struct uci_dev *udev = dev_get_drvdata(&mhi_dev->dev);
+	struct uci_chan *uchan = &udev->dl_chan;
+	struct device *dev = &mhi_dev->dev;
+	struct uci_buf *buf;
+
+	dev_dbg(dev, "status: %d receive_len: %zu\n",
+		mhi_result->transaction_status, mhi_result->bytes_xferd);
+
+	if (mhi_result->transaction_status == -ENOTCONN) {
+		kfree(mhi_result->buf_addr);
+		return;
+	}
+
+	spin_lock_bh(&uchan->lock);
+	buf = mhi_result->buf_addr + udev->actual_mtu;
+	buf->data = mhi_result->buf_addr;
+	buf->len = mhi_result->bytes_xferd;
+	list_add_tail(&buf->node, &uchan->pending);
+	spin_unlock_bh(&uchan->lock);
+
+	wake_up(&uchan->wq);
+}
+
+static int mhi_uci_probe(struct mhi_device *mhi_dev,
+			 const struct mhi_device_id *id)
+{
+	struct uci_dev *udev;
+	struct mhi_controller *mhi_cntrl = mhi_dev->mhi_cntrl;
+	struct device *dev;
+	int index, dir;
+
+	udev = kzalloc(sizeof(*udev), GFP_KERNEL);
+	if (!udev)
+		return -ENOMEM;
+
+	kref_init(&udev->ref_count);
+	mutex_init(&udev->lock);
+	udev->mhi_dev = mhi_dev;
+
+	mutex_lock(&udev->lock);
+
+	mutex_lock(&uci_idr_mutex);
+	index = idr_alloc(&uci_idr, udev, 0, MAX_UCI_MINORS, GFP_KERNEL);
+	mutex_unlock(&uci_idr_mutex);
+	if (index < 0) {
+		mutex_unlock(&udev->lock);
+		kfree(udev);
+		return index;
+	}
+
+	udev->minor = index;
+
+	/* create device file node /dev/mhi_<cntrl_dev_name>_<mhi_dev_name> */
+	dev = device_create(uci_dev_class, &mhi_dev->dev,
+			    MKDEV(uci_dev_major, index), udev,
+			    DEVICE_NAME "_%s_%s",
+			    dev_name(mhi_cntrl->cntrl_dev), mhi_dev->name);
+	if (IS_ERR(dev)) {
+		mutex_lock(&uci_idr_mutex);
+		idr_remove(&uci_idr, udev->minor);
+		mutex_unlock(&uci_idr_mutex);
+		mutex_unlock(&udev->lock);
+		kfree(udev);
+		return PTR_ERR(dev);
+	}
+
+	for (dir = 0; dir < 2; dir++) {
+		struct uci_chan *uchan = (dir) ?
+			&udev->ul_chan : &udev->dl_chan;
+		spin_lock_init(&uchan->lock);
+		init_waitqueue_head(&uchan->wq);
+		INIT_LIST_HEAD(&uchan->pending);
+	}
+
+	udev->mtu = min_t(size_t, id->driver_data, MHI_MAX_MTU);
+	udev->actual_mtu = udev->mtu - sizeof(struct uci_buf);
+	dev_set_drvdata(&mhi_dev->dev, udev);
+	udev->enabled = true;
+
+	mutex_unlock(&udev->lock);
+
+	dev_info(&mhi_dev->dev, "probed uci dev: minor %d\n", index);
+
+	return 0;
+};
+
+static void mhi_uci_remove(struct mhi_device *mhi_dev)
+{
+	struct uci_dev *udev = dev_get_drvdata(&mhi_dev->dev);
+
+	mutex_lock(&udev->lock);
+
+	/* disable the node */
+	udev->enabled = false;
+
+	wake_up(&udev->dl_chan.wq);
+	wake_up(&udev->ul_chan.wq);
+
+	/* delete the node to prevent new opens */
+	device_destroy(uci_dev_class, MKDEV(uci_dev_major, udev->minor));
+
+	mutex_lock(&uci_idr_mutex);
+	idr_remove(&uci_idr, udev->minor);
+	mutex_unlock(&uci_idr_mutex);
+
+	mutex_unlock(&udev->lock);
+
+	kref_put(&udev->ref_count, mhi_uci_dev_release);
+}
+
+/* .driver_data stores max mtu */
+static const struct mhi_device_id mhi_uci_match_table[] = {
+	{ .chan = "LOOPBACK", .driver_data = 0x1000},
+	{},
+};
+MODULE_DEVICE_TABLE(mhi, mhi_uci_match_table);
+
+static struct mhi_driver mhi_uci_driver = {
+	.id_table = mhi_uci_match_table,
+	.remove = mhi_uci_remove,
+	.probe = mhi_uci_probe,
+	.ul_xfer_cb = mhi_ul_xfer_cb,
+	.dl_xfer_cb = mhi_dl_xfer_cb,
+	.driver = {
+		.name = MHI_UCI_DRIVER_NAME,
+	},
+};
+
+static int mhi_uci_init(void)
+{
+	int ret;
+
+	ret = register_chrdev(0, MHI_UCI_DRIVER_NAME, &mhidev_fops);
+	if (ret < 0)
+		return ret;
+
+	uci_dev_major = ret;
+	uci_dev_class = class_create(THIS_MODULE, MHI_UCI_DRIVER_NAME);
+	if (IS_ERR(uci_dev_class)) {
+		unregister_chrdev(uci_dev_major, MHI_UCI_DRIVER_NAME);
+		return -ENODEV;
+	}
+
+	ret = mhi_driver_register(&mhi_uci_driver);
+	if (ret) {
+		class_destroy(uci_dev_class);
+		unregister_chrdev(uci_dev_major, MHI_UCI_DRIVER_NAME);
+	}
+
+	return ret;
+}
+
+static void __exit mhi_uci_exit(void)
+{
+	mhi_driver_unregister(&mhi_uci_driver);
+	class_destroy(uci_dev_class);
+	unregister_chrdev(uci_dev_major, MHI_UCI_DRIVER_NAME);
+}
+
+module_init(mhi_uci_init);
+module_exit(mhi_uci_exit);
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("MHI UCI Driver");
-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

Re: [PATCH v6 4/4] bus: mhi: Add userspace client interface driver

[Randy Dunlap]

On 9/16/20 12:56 PM, Hemant Kumar wrote:
> diff --git a/drivers/bus/mhi/Kconfig b/drivers/bus/mhi/Kconfig
> index 6a217ff..8aebe8b 100644
> --- a/drivers/bus/mhi/Kconfig
> +++ b/drivers/bus/mhi/Kconfig
> @@ -20,3 +20,16 @@ config MHI_BUS_DEBUG
>  	 Enable debugfs support for use with the MHI transport. Allows
>  	 reading and/or modifying some values within the MHI controller
>  	 for debug and test purposes.

Hi,
Please indent Kconfig keywords with one tab only, and indent
help text with one tab + 2 spaces, as documented in
Documentation/process/coding-style.rst:

"""
For all of the Kconfig* configuration files throughout the source tree,
the indentation is somewhat different.  Lines under a ``config`` definition
are indented with one tab, while help text is indented an additional two
spaces.
"""

> +
> +config MHI_UCI
> +       tristate "MHI UCI"
> +       depends on MHI_BUS
> +       help
> +	 MHI based userspace client interface driver is used for transferring
> +	 raw data between host and device using standard file operations from
> +	 userspace. Open, read, write, and close operations are supported
> +	 by this driver. Please check mhi_uci_match_table for all supported
> +	 channels that are exposed to userspace.
> +
> +	 To compile this driver as a module, choose M here: the module will be
> +	 called mhi_uci.

thanks.
-- 
~Randy

Re: [PATCH v6 4/4] bus: mhi: Add userspace client interface driver

[Greg KH]

On Wed, Sep 16, 2020 at 12:56:07PM -0700, Hemant Kumar wrote:
> +/**
> + * struct uci_dev - MHI uci device
> + * @minor: uci device node minor number
> + * @mhi_dev: associated mhi device object
> + * @chan: MHI channel name
> + * @lock: mutex lock
> + * @ul_chan: uplink uci channel object
> + * @dl_chan: downlink uci channel object
> + * @mtu: max tx buffer length
> + * @actual_mtu: maximum size of incoming buffer
> + * @open: open called for device node
> + * @enabled: uci device probed
> + * @ref_count: uci_dev reference count
> + */
> +struct uci_dev {
> +	unsigned int minor;
> +	struct mhi_device *mhi_dev;
> +	const char *chan;
> +
> +	/* protects uci_dev struct members */
> +	struct mutex lock;
> +
> +	struct uci_chan ul_chan;
> +	struct uci_chan dl_chan;
> +	size_t mtu;
> +	size_t actual_mtu;
> +	bool enabled;
> +	struct kref ref_count;
> +};

I don't think you actually made the kernel documentation based on these
lines.

Or if you did, you ignored the warnings :(

Please test build your patches before sending them out...

thanks,

greg k-h

Re: [PATCH v6 4/4] bus: mhi: Add userspace client interface driver

[Hemant Kumar]

Hi Greg,

On 9/17/20 9:40 AM, Greg KH wrote:
> On Wed, Sep 16, 2020 at 12:56:07PM -0700, Hemant Kumar wrote:
>> +/**
>> + * struct uci_dev - MHI uci device
>> + * @minor: uci device node minor number
>> + * @mhi_dev: associated mhi device object
>> + * @chan: MHI channel name
>> + * @lock: mutex lock
>> + * @ul_chan: uplink uci channel object
>> + * @dl_chan: downlink uci channel object
>> + * @mtu: max tx buffer length
>> + * @actual_mtu: maximum size of incoming buffer
>> + * @open: open called for device node
>> + * @enabled: uci device probed
>> + * @ref_count: uci_dev reference count
>> + */
>> +struct uci_dev {
>> +	unsigned int minor;
>> +	struct mhi_device *mhi_dev;
>> +	const char *chan;
>> +
>> +	/* protects uci_dev struct members */
>> +	struct mutex lock;
>> +
>> +	struct uci_chan ul_chan;
>> +	struct uci_chan dl_chan;
>> +	size_t mtu;
>> +	size_t actual_mtu;
>> +	bool enabled;
>> +	struct kref ref_count;
>> +};
> 
> I don't think you actually made the kernel documentation based on these
> lines.
> 
> Or if you did, you ignored the warnings :(
> 
> Please test build your patches before sending them out...
Thanks for pointing out the kernel doc, it was a miss. i will fix it.
> 
> thanks,
> 
> greg k-h
> 

-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

Re: [PATCH v6 4/4] bus: mhi: Add userspace client interface driver

[Greg KH]

On Wed, Sep 16, 2020 at 12:56:07PM -0700, Hemant Kumar wrote:
> This MHI client driver allows userspace clients to transfer
> raw data between MHI device and host using standard file operations.
> Device file node is created with format
> 
> /dev/mhi_<controller_name>_<mhi_device_name>
> 
> Currently it supports LOOPBACK channel.
> 
> Signed-off-by: Hemant Kumar <hemantk@codeaurora.org>
> ---
>  drivers/bus/mhi/Kconfig  |  13 +
>  drivers/bus/mhi/Makefile |   4 +
>  drivers/bus/mhi/uci.c    | 657 +++++++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 674 insertions(+)
>  create mode 100644 drivers/bus/mhi/uci.c
> 
> diff --git a/drivers/bus/mhi/Kconfig b/drivers/bus/mhi/Kconfig
> index 6a217ff..8aebe8b 100644
> --- a/drivers/bus/mhi/Kconfig
> +++ b/drivers/bus/mhi/Kconfig
> @@ -20,3 +20,16 @@ config MHI_BUS_DEBUG
>  	 Enable debugfs support for use with the MHI transport. Allows
>  	 reading and/or modifying some values within the MHI controller
>  	 for debug and test purposes.
> +
> +config MHI_UCI
> +       tristate "MHI UCI"
> +       depends on MHI_BUS
> +       help
> +	 MHI based userspace client interface driver is used for transferring
> +	 raw data between host and device using standard file operations from
> +	 userspace. Open, read, write, and close operations are supported
> +	 by this driver. Please check mhi_uci_match_table for all supported
> +	 channels that are exposed to userspace.
> +
> +	 To compile this driver as a module, choose M here: the module will be
> +	 called mhi_uci.
> diff --git a/drivers/bus/mhi/Makefile b/drivers/bus/mhi/Makefile
> index 19e6443..80feefb 100644
> --- a/drivers/bus/mhi/Makefile
> +++ b/drivers/bus/mhi/Makefile
> @@ -1,2 +1,6 @@
>  # core layer
>  obj-y += core/
> +
> +# MHI client
> +mhi_uci-y := uci.o
> +obj-$(CONFIG_MHI_UCI) += mhi_uci.o
> diff --git a/drivers/bus/mhi/uci.c b/drivers/bus/mhi/uci.c
> new file mode 100644
> index 0000000..d6758f2
> --- /dev/null
> +++ b/drivers/bus/mhi/uci.c
> @@ -0,0 +1,657 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/* Copyright (c) 2018-2020, The Linux Foundation. All rights reserved.*/
> +
> +#include <linux/kernel.h>
> +#include <linux/mhi.h>
> +#include <linux/mod_devicetable.h>
> +#include <linux/module.h>
> +#include <linux/poll.h>
> +
> +#define DEVICE_NAME "mhi"
> +#define MHI_UCI_DRIVER_NAME "mhi_uci"
> +#define MAX_UCI_MINORS (128)
> +
> +static DEFINE_IDR(uci_idr);
> +static DEFINE_MUTEX(uci_idr_mutex);
> +static struct class *uci_dev_class;
> +static int uci_dev_major;
> +
> +/**
> + * struct uci_chan - MHI channel for a uci device
> + * @wq: wait queue for reader/writer
> + * @lock: spin lock
> + * @pending: list of rx buffers userspace is waiting to read
> + * @cur_buf: current buffer userspace is reading
> + * @rx_size: size of the current rx buffer userspace is reading
> + */
> +struct uci_chan {
> +	wait_queue_head_t wq;
> +
> +	/* protects pending and cur_buf members in bh context */
> +	spinlock_t lock;
> +
> +	struct list_head pending;
> +	struct uci_buf *cur_buf;
> +	size_t rx_size;
> +};
> +
> +/**
> + * struct uci_buf - uci buffer
> + * @data: data buffer
> + * @len: length of data buffer
> + * @node: list node of the uci buffer
> + */
> +struct uci_buf {
> +	void *data;
> +	size_t len;
> +	struct list_head node;
> +};
> +
> +/**
> + * struct uci_dev - MHI uci device
> + * @minor: uci device node minor number
> + * @mhi_dev: associated mhi device object
> + * @chan: MHI channel name
> + * @lock: mutex lock
> + * @ul_chan: uplink uci channel object
> + * @dl_chan: downlink uci channel object
> + * @mtu: max tx buffer length
> + * @actual_mtu: maximum size of incoming buffer
> + * @open: open called for device node
> + * @enabled: uci device probed
> + * @ref_count: uci_dev reference count
> + */
> +struct uci_dev {
> +	unsigned int minor;
> +	struct mhi_device *mhi_dev;
> +	const char *chan;
> +
> +	/* protects uci_dev struct members */
> +	struct mutex lock;
> +
> +	struct uci_chan ul_chan;
> +	struct uci_chan dl_chan;
> +	size_t mtu;
> +	size_t actual_mtu;
> +	bool enabled;
> +	struct kref ref_count;
> +};
> +
> +static int mhi_queue_inbound(struct uci_dev *udev)
> +{
> +	struct mhi_device *mhi_dev = udev->mhi_dev;
> +	struct device *dev = &mhi_dev->dev;
> +	size_t mtu = udev->mtu;
> +	size_t actual_mtu = udev->actual_mtu;
> +	int nr_trbs, i, ret = -EIO;
> +	void *buf;
> +	struct uci_buf *uci_buf;
> +
> +	nr_trbs = mhi_get_no_free_descriptors(mhi_dev, DMA_FROM_DEVICE);
> +
> +	for (i = 0; i < nr_trbs; i++) {
> +		buf = kmalloc(mtu, GFP_KERNEL);
> +		if (!buf)
> +			return -ENOMEM;
> +
> +		uci_buf = buf + actual_mtu;
> +		uci_buf->data = buf;
> +
> +		dev_dbg(dev, "Allocated buf %d of %d size %ld\n", i, nr_trbs,
> +			actual_mtu);
> +
> +		ret = mhi_queue_buf(mhi_dev, DMA_FROM_DEVICE, buf, actual_mtu,
> +				    MHI_EOT);
> +		if (ret) {
> +			kfree(buf);
> +			dev_err(dev, "Failed to queue buffer %d\n", i);
> +			return ret;
> +		}
> +	}
> +
> +	return ret;
> +}
> +
> +static void mhi_uci_dev_release(struct kref *ref)
> +{
> +	struct uci_dev *udev =
> +		container_of(ref, struct uci_dev, ref_count);
> +
> +	mutex_destroy(&udev->lock);
> +
> +	dev_set_drvdata(&udev->mhi_dev->dev, NULL);
> +
> +	kfree(udev);
> +}
> +
> +static int mhi_uci_release(struct inode *inode, struct file *file)
> +{
> +	struct uci_dev *udev = file->private_data;
> +	struct uci_buf *itr, *tmp;
> +	struct uci_chan *uchan;
> +
> +	if (kref_read(&udev->ref_count) > 2)
> +		goto exit_uci_release;
> +
> +	if (udev->enabled)
> +		mhi_unprepare_from_transfer(udev->mhi_dev);
> +
> +	/* clean inbound channel */
> +	uchan = &udev->dl_chan;
> +
> +	spin_lock_bh(&uchan->lock);
> +	list_for_each_entry_safe(itr, tmp, &uchan->pending, node) {
> +		list_del(&itr->node);
> +		kfree(itr->data);
> +	}
> +
> +	if (uchan->cur_buf)
> +		kfree(uchan->cur_buf->data);
> +
> +	uchan->cur_buf = NULL;
> +	spin_unlock_bh(&uchan->lock);
> +
> +exit_uci_release:
> +	kref_put(&udev->ref_count, mhi_uci_dev_release);
> +
> +	return 0;
> +}
> +
> +static __poll_t mhi_uci_poll(struct file *file, poll_table *wait)
> +{
> +	struct uci_dev *udev = file->private_data;
> +	struct mhi_device *mhi_dev = udev->mhi_dev;
> +	struct device *dev = &mhi_dev->dev;
> +	struct uci_chan *uchan;
> +	__poll_t mask = 0;
> +
> +	poll_wait(file, &udev->dl_chan.wq, wait);
> +	poll_wait(file, &udev->ul_chan.wq, wait);
> +
> +	if (!udev->enabled) {
> +		mask = EPOLLERR;
> +	} else {
> +		uchan = &udev->dl_chan;
> +		spin_lock_bh(&uchan->lock);
> +		if (!list_empty(&uchan->pending) || uchan->cur_buf) {
> +			dev_dbg(dev, "Client can read from node\n");
> +			mask |= EPOLLIN | EPOLLRDNORM;
> +		}
> +		spin_unlock_bh(&uchan->lock);
> +	}
> +
> +	if (!udev->enabled) {
> +		mask |= EPOLLERR;
> +	} else if (mhi_get_no_free_descriptors(mhi_dev, DMA_TO_DEVICE) > 0) {
> +		dev_dbg(dev, "Client can write to node\n");
> +		mask |= EPOLLOUT | EPOLLWRNORM;
> +	}
> +
> +	dev_dbg(dev, "Client attempted to poll, returning mask 0x%x\n", mask);
> +
> +	return mask;
> +}
> +
> +static ssize_t mhi_uci_write(struct file *file,
> +			     const char __user *buf,
> +			     size_t count,
> +			     loff_t *offp)
> +{
> +	struct uci_dev *udev = file->private_data;
> +	struct mhi_device *mhi_dev = udev->mhi_dev;
> +	struct device *dev = &mhi_dev->dev;
> +	struct uci_chan *uchan = &udev->ul_chan;
> +	size_t bytes_xfered = 0;
> +	int ret, nr_avail = 0;
> +
> +	if (!buf || !count)
> +		return -EINVAL;
> +
> +	/* confirm channel is active */
> +	mutex_lock(&udev->lock);
> +	if (!udev->enabled) {
> +		ret = -ENODEV;
> +		goto err_mtx_unlock;
> +	}
> +
> +	dev_dbg(dev, "%s: to xfer: %lu bytes\n", __func__, count);
> +
> +	while (count) {
> +		size_t xfer_size;
> +		void *kbuf;
> +		enum mhi_flags flags;
> +
> +		mutex_unlock(&udev->lock);
> +		/* wait for free descriptors */
> +		ret = wait_event_interruptible(uchan->wq,
> +					       (!udev->enabled) ||
> +				(nr_avail = mhi_get_no_free_descriptors(mhi_dev,
> +					       DMA_TO_DEVICE)) > 0);
> +
> +		mutex_lock(&udev->lock);
> +		if (ret == -ERESTARTSYS) {
> +			dev_dbg(dev, "Exit signal caught for node\n");
> +			goto err_mtx_unlock;
> +		}
> +
> +		if (!udev->enabled) {
> +			ret = -ENODEV;
> +			goto err_mtx_unlock;
> +		}
> +
> +		xfer_size = min_t(size_t, count, udev->mtu);
> +		kbuf = kmalloc(xfer_size, GFP_KERNEL);
> +		if (!kbuf) {
> +			ret = -ENOMEM;
> +			goto err_mtx_unlock;
> +		}
> +
> +		ret = copy_from_user(kbuf, buf, xfer_size);
> +		if (ret) {
> +			kfree(kbuf);
> +			ret = -EFAULT;
> +			goto err_mtx_unlock;
> +		}
> +
> +		/* if ring is full after this force EOT */
> +		if (nr_avail > 1 && (count - xfer_size))
> +			flags = MHI_CHAIN;
> +		else
> +			flags = MHI_EOT;
> +
> +		if (udev->enabled)
> +			ret = mhi_queue_buf(mhi_dev, DMA_TO_DEVICE, kbuf,
> +					    xfer_size, flags);
> +		else
> +			ret = -ENODEV;
> +
> +		if (ret) {
> +			kfree(kbuf);
> +			goto err_mtx_unlock;
> +		}
> +
> +		bytes_xfered += xfer_size;
> +		count -= xfer_size;
> +		buf += xfer_size;
> +	}
> +
> +	mutex_unlock(&udev->lock);
> +	dev_dbg(dev, "%s: bytes xferred: %lu\n", __func__, bytes_xfered);
> +
> +	return bytes_xfered;
> +
> +err_mtx_unlock:
> +	mutex_unlock(&udev->lock);
> +
> +	return ret;
> +}
> +
> +static ssize_t mhi_uci_read(struct file *file,
> +			    char __user *buf,
> +			    size_t count,
> +			    loff_t *ppos)
> +{
> +	struct uci_dev *udev = file->private_data;
> +	struct mhi_device *mhi_dev = udev->mhi_dev;
> +	struct uci_chan *uchan = &udev->dl_chan;
> +	struct device *dev = &mhi_dev->dev;
> +	struct uci_buf *ubuf;
> +	char *ptr;
> +	size_t to_copy;
> +	int ret = 0;
> +
> +	if (!buf)
> +		return -EINVAL;
> +
> +	mutex_lock(&udev->lock);
> +	/* confirm channel is active */
> +	if (!udev->enabled) {
> +		ret = -ENODEV;
> +		goto err_mtx_unlock;
> +	}
> +
> +	spin_lock_bh(&uchan->lock);
> +	/* No data available to read, wait */
> +	if (!uchan->cur_buf && list_empty(&uchan->pending)) {
> +		dev_dbg(dev, "No data available to read waiting\n");
> +
> +		spin_unlock_bh(&uchan->lock);
> +		mutex_unlock(&udev->lock);
> +		ret = wait_event_interruptible(uchan->wq,
> +					       (!udev->enabled ||
> +					      !list_empty(&uchan->pending)));
> +
> +		mutex_lock(&udev->lock);
> +		if (ret == -ERESTARTSYS) {
> +			dev_dbg(dev, "Exit signal caught for node\n");
> +			goto err_mtx_unlock;
> +		}
> +
> +		if (!udev->enabled) {
> +			ret = -ENODEV;
> +			goto err_mtx_unlock;
> +		}
> +		spin_lock_bh(&uchan->lock);
> +	}
> +
> +	/* new read, get the next descriptor from the list */
> +	if (!uchan->cur_buf) {
> +		ubuf = list_first_entry_or_null(&uchan->pending,
> +						struct uci_buf, node);
> +		if (!ubuf) {
> +			ret = -EIO;
> +			goto err_spin_unlock;
> +		}
> +
> +		list_del(&ubuf->node);
> +		uchan->cur_buf = ubuf;
> +		uchan->rx_size = ubuf->len;
> +		dev_dbg(dev, "Got pkt of size: %zu\n", uchan->rx_size);
> +	}
> +
> +	ubuf = uchan->cur_buf;
> +
> +	/* Copy the buffer to user space */
> +	to_copy = min_t(size_t, count, uchan->rx_size);
> +	ptr = ubuf->data + (ubuf->len - uchan->rx_size);
> +	spin_unlock_bh(&uchan->lock);
> +
> +	ret = copy_to_user(buf, ptr, to_copy);
> +	if (ret) {
> +		ret = -EFAULT;
> +		goto err_mtx_unlock;
> +	}
> +
> +	spin_lock_bh(&uchan->lock);
> +
> +	dev_dbg(dev, "Copied %lu of %lu bytes\n", to_copy, uchan->rx_size);
> +	uchan->rx_size -= to_copy;
> +
> +	/* we finished with this buffer, queue it back to hardware */
> +	if (!uchan->rx_size) {
> +		uchan->cur_buf = NULL;
> +
> +		if (udev->enabled)
> +			ret = mhi_queue_buf(mhi_dev, DMA_FROM_DEVICE,
> +					    ubuf->data,
> +					    udev->actual_mtu, MHI_EOT);
> +		else
> +			ret = -ENODEV;
> +
> +		if (ret) {
> +			dev_err(dev, "Failed to recycle element: %d\n", ret);
> +			kfree(ubuf->data);
> +			goto err_spin_unlock;
> +		}
> +	}
> +	spin_unlock_bh(&uchan->lock);
> +	mutex_unlock(&udev->lock);
> +
> +	dev_dbg(dev, "%s: Returning %lu bytes\n", __func__, to_copy);
> +
> +	return to_copy;
> +
> +err_spin_unlock:
> +	spin_unlock_bh(&uchan->lock);
> +err_mtx_unlock:
> +	mutex_unlock(&udev->lock);
> +	return ret;
> +}
> +
> +static int mhi_uci_open(struct inode *inode, struct file *filp)
> +{
> +	struct uci_dev *udev = NULL;
> +	unsigned int minor = iminor(inode);
> +	int ret = -EIO;
> +	struct uci_buf *buf_itr, *tmp;
> +	struct uci_chan *dl_chan;
> +	struct mhi_device *mhi_dev;
> +	struct device *dev;
> +
> +	mutex_lock(&uci_idr_mutex);
> +	udev = idr_find(&uci_idr, minor);
> +	mutex_unlock(&uci_idr_mutex);
> +	if (!udev) {
> +		pr_err("uci dev: minor %d not found\n", minor);

Don't spam the kernel log for things that users can do :(

> +		ret = -ENODEV;
> +		goto error_no_dev;
> +	}
> +
> +	kref_get(&udev->ref_count);

Why grab a reference?  What does that help with?

> +
> +	mhi_dev = udev->mhi_dev;
> +	dev = &mhi_dev->dev;
> +
> +	mutex_lock(&udev->lock);
> +	if (kref_read(&udev->ref_count) > 2) {
> +		dev_dbg(dev, "Node already opened\n");

Nope, this is NOT doing what you think it is doing.

I told you before, do not try to keep a device node from being opened
multiple times, as it will always fail (think about passing file handles
around between programs...)

If userspace wants to do this, it will do it.  If your driver can't
handle that, that's fine, userspace will learn not to do that.  But the
kernel can not prevent this from happening.

Also note that reading a kref value is a HUGE sign that the code is
incorrect, you should never care about the value of a reference.  Maybe
if it is 0, but that's a special case...

Anyway, given that you ignored my previous review comments here, I'm
loath to keep reviewing this patch series.  Please get others to review
it first before sending it back as I don't like being the only one doing
this type of work...

thanks,

greg k-h

Re: [PATCH v6 4/4] bus: mhi: Add userspace client interface driver

[Hemant Kumar]

Hi Greg,

On 9/17/20 9:44 AM, Greg KH wrote:
> On Wed, Sep 16, 2020 at 12:56:07PM -0700, Hemant Kumar wrote:
...
...
>> +
>> +static int mhi_uci_open(struct inode *inode, struct file *filp)
>> +{
>> +	struct uci_dev *udev = NULL;
>> +	unsigned int minor = iminor(inode);
>> +	int ret = -EIO;
>> +	struct uci_buf *buf_itr, *tmp;
>> +	struct uci_chan *dl_chan;
>> +	struct mhi_device *mhi_dev;
>> +	struct device *dev;
>> +
>> +	mutex_lock(&uci_idr_mutex);
>> +	udev = idr_find(&uci_idr, minor);
>> +	mutex_unlock(&uci_idr_mutex);
>> +	if (!udev) {
>> +		pr_err("uci dev: minor %d not found\n", minor);
> 
> Don't spam the kernel log for things that users can do :(
i will change it to a pr_debug, as it helps to debug why open() is failing.
> 
>> +		ret = -ENODEV;
>> +		goto error_no_dev;
>> +	}
>> +
>> +	kref_get(&udev->ref_count);
> 
> Why grab a reference?  What does that help with?
In case open() and driver remove() are racing, it helps to prevent use 
after free of udev in open().
> 
>> +
>> +	mhi_dev = udev->mhi_dev;
>> +	dev = &mhi_dev->dev;
>> +
>> +	mutex_lock(&udev->lock);
>> +	if (kref_read(&udev->ref_count) > 2) {
>> +		dev_dbg(dev, "Node already opened\n");
> 
> Nope, this is NOT doing what you think it is doing.
> 
> I told you before, do not try to keep a device node from being opened
> multiple times, as it will always fail (think about passing file handles
> around between programs...)
> 
> If userspace wants to do this, it will do it.  If your driver can't
> handle that, that's fine, userspace will learn not to do that.  But the
> kernel can not prevent this from happening.
This check is not returning error, instead just setting 
filp->private_data = udev; and return 0; It is skipping channel prepare
and queuing of inbound buffers which was done by first open().
> 
> Also note that reading a kref value is a HUGE sign that the code is
> incorrect, you should never care about the value of a reference.  Maybe
> if it is 0, but that's a special case...
In previous patch this was done using separate open reference count and
after removing that i was relying on udev ref count. MHI channel prepare
and buffer allocation for a give channel suppose to happen at open() and
only for first open() call.
> 
> Anyway, given that you ignored my previous review comments here, I'm
> loath to keep reviewing this patch series.  Please get others to review
> it first before sending it back as I don't like being the only one doing
> this type of work...
Thanks for reviewing my patch series Greg and help making it a better 
driver!
> 
> thanks,
> 
> greg k-h
> 

-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

Re: [PATCH v6 4/4] bus: mhi: Add userspace client interface driver

[Greg KH]

On Fri, Sep 18, 2020 at 11:14:08AM -0700, Hemant Kumar wrote:
> Hi Greg,
> 
> On 9/17/20 9:44 AM, Greg KH wrote:
> > On Wed, Sep 16, 2020 at 12:56:07PM -0700, Hemant Kumar wrote:
> ...
> ...
> > > +
> > > +static int mhi_uci_open(struct inode *inode, struct file *filp)
> > > +{
> > > +	struct uci_dev *udev = NULL;
> > > +	unsigned int minor = iminor(inode);
> > > +	int ret = -EIO;
> > > +	struct uci_buf *buf_itr, *tmp;
> > > +	struct uci_chan *dl_chan;
> > > +	struct mhi_device *mhi_dev;
> > > +	struct device *dev;
> > > +
> > > +	mutex_lock(&uci_idr_mutex);
> > > +	udev = idr_find(&uci_idr, minor);
> > > +	mutex_unlock(&uci_idr_mutex);
> > > +	if (!udev) {
> > > +		pr_err("uci dev: minor %d not found\n", minor);
> > 
> > Don't spam the kernel log for things that users can do :(
> i will change it to a pr_debug, as it helps to debug why open() is failing.
> > 
> > > +		ret = -ENODEV;
> > > +		goto error_no_dev;
> > > +	}
> > > +
> > > +	kref_get(&udev->ref_count);
> > 
> > Why grab a reference?  What does that help with?
> In case open() and driver remove() are racing, it helps to prevent use after
> free of udev in open().

Are you sure it prevents that?  Where is the lock that handles dropping
a reference count and incrementing it at the same time?

krefs are not "lock free" entirely, they need to have some type of other
control somewhere to prevent foolish things from happening :)

> > > +
> > > +	mhi_dev = udev->mhi_dev;
> > > +	dev = &mhi_dev->dev;
> > > +
> > > +	mutex_lock(&udev->lock);
> > > +	if (kref_read(&udev->ref_count) > 2) {
> > > +		dev_dbg(dev, "Node already opened\n");
> > 
> > Nope, this is NOT doing what you think it is doing.
> > 
> > I told you before, do not try to keep a device node from being opened
> > multiple times, as it will always fail (think about passing file handles
> > around between programs...)
> > 
> > If userspace wants to do this, it will do it.  If your driver can't
> > handle that, that's fine, userspace will learn not to do that.  But the
> > kernel can not prevent this from happening.
> This check is not returning error, instead just setting filp->private_data =
> udev; and return 0; It is skipping channel prepare
> and queuing of inbound buffers which was done by first open().

But don't do that by checking a kref value.  You should never care about
the value of it, that is not how you use it at all, and one reason I
hate that function is even present in the kernel...

thanks,

greg k-h

Re: [PATCH v6 4/4] bus: mhi: Add userspace client interface driver

[Jeffrey Hugo]

On 9/16/2020 1:56 PM, Hemant Kumar wrote:
> This MHI client driver allows userspace clients to transfer
> raw data between MHI device and host using standard file operations.
> Device file node is created with format
> 
> /dev/mhi_<controller_name>_<mhi_device_name>
> 
> Currently it supports LOOPBACK channel.
> 
> Signed-off-by: Hemant Kumar <hemantk@codeaurora.org>
> ---
>   drivers/bus/mhi/Kconfig  |  13 +
>   drivers/bus/mhi/Makefile |   4 +
>   drivers/bus/mhi/uci.c    | 657 +++++++++++++++++++++++++++++++++++++++++++++++
>   3 files changed, 674 insertions(+)
>   create mode 100644 drivers/bus/mhi/uci.c
> 
> diff --git a/drivers/bus/mhi/Kconfig b/drivers/bus/mhi/Kconfig
> index 6a217ff..8aebe8b 100644
> --- a/drivers/bus/mhi/Kconfig
> +++ b/drivers/bus/mhi/Kconfig
> @@ -20,3 +20,16 @@ config MHI_BUS_DEBUG
>   	 Enable debugfs support for use with the MHI transport. Allows
>   	 reading and/or modifying some values within the MHI controller
>   	 for debug and test purposes.
> +
> +config MHI_UCI
> +       tristate "MHI UCI"
> +       depends on MHI_BUS
> +       help
> +	 MHI based userspace client interface driver is used for transferring
> +	 raw data between host and device using standard file operations from
> +	 userspace. Open, read, write, and close operations are supported
> +	 by this driver. Please check mhi_uci_match_table for all supported
> +	 channels that are exposed to userspace.
> +
> +	 To compile this driver as a module, choose M here: the module will be
> +	 called mhi_uci.
> diff --git a/drivers/bus/mhi/Makefile b/drivers/bus/mhi/Makefile
> index 19e6443..80feefb 100644
> --- a/drivers/bus/mhi/Makefile
> +++ b/drivers/bus/mhi/Makefile
> @@ -1,2 +1,6 @@
>   # core layer
>   obj-y += core/
> +
> +# MHI client
> +mhi_uci-y := uci.o
> +obj-$(CONFIG_MHI_UCI) += mhi_uci.o
> diff --git a/drivers/bus/mhi/uci.c b/drivers/bus/mhi/uci.c
> new file mode 100644
> index 0000000..d6758f2
> --- /dev/null
> +++ b/drivers/bus/mhi/uci.c
> @@ -0,0 +1,657 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/* Copyright (c) 2018-2020, The Linux Foundation. All rights reserved.*/
> +
> +#include <linux/kernel.h>
> +#include <linux/mhi.h>
> +#include <linux/mod_devicetable.h>
> +#include <linux/module.h>
> +#include <linux/poll.h>
> +
> +#define DEVICE_NAME "mhi"
> +#define MHI_UCI_DRIVER_NAME "mhi_uci"
> +#define MAX_UCI_MINORS (128)
> +
> +static DEFINE_IDR(uci_idr);
> +static DEFINE_MUTEX(uci_idr_mutex);
> +static struct class *uci_dev_class;
> +static int uci_dev_major;
> +
> +/**
> + * struct uci_chan - MHI channel for a uci device
> + * @wq: wait queue for reader/writer
> + * @lock: spin lock
> + * @pending: list of rx buffers userspace is waiting to read
> + * @cur_buf: current buffer userspace is reading
> + * @rx_size: size of the current rx buffer userspace is reading
> + */
> +struct uci_chan {
> +	wait_queue_head_t wq;
> +
> +	/* protects pending and cur_buf members in bh context */
> +	spinlock_t lock;
> +
> +	struct list_head pending;
> +	struct uci_buf *cur_buf;
> +	size_t rx_size;
> +};
> +
> +/**
> + * struct uci_buf - uci buffer
> + * @data: data buffer
> + * @len: length of data buffer
> + * @node: list node of the uci buffer
> + */
> +struct uci_buf {
> +	void *data;
> +	size_t len;
> +	struct list_head node;
> +};
> +
> +/**
> + * struct uci_dev - MHI uci device
> + * @minor: uci device node minor number
> + * @mhi_dev: associated mhi device object
> + * @chan: MHI channel name
> + * @lock: mutex lock
> + * @ul_chan: uplink uci channel object
> + * @dl_chan: downlink uci channel object
> + * @mtu: max tx buffer length
> + * @actual_mtu: maximum size of incoming buffer
> + * @open: open called for device node
> + * @enabled: uci device probed
> + * @ref_count: uci_dev reference count
> + */
> +struct uci_dev {
> +	unsigned int minor;
> +	struct mhi_device *mhi_dev;
> +	const char *chan;
> +
> +	/* protects uci_dev struct members */
> +	struct mutex lock;
> +
> +	struct uci_chan ul_chan;
> +	struct uci_chan dl_chan;
> +	size_t mtu;
> +	size_t actual_mtu;
> +	bool enabled;
> +	struct kref ref_count;
> +};
> +
> +static int mhi_queue_inbound(struct uci_dev *udev)
> +{
> +	struct mhi_device *mhi_dev = udev->mhi_dev;
> +	struct device *dev = &mhi_dev->dev;
> +	size_t mtu = udev->mtu;
> +	size_t actual_mtu = udev->actual_mtu;
> +	int nr_trbs, i, ret = -EIO;
> +	void *buf;
> +	struct uci_buf *uci_buf;
> +
> +	nr_trbs = mhi_get_no_free_descriptors(mhi_dev, DMA_FROM_DEVICE);
> +
> +	for (i = 0; i < nr_trbs; i++) {
> +		buf = kmalloc(mtu, GFP_KERNEL);
> +		if (!buf)
> +			return -ENOMEM;
> +
> +		uci_buf = buf + actual_mtu;
> +		uci_buf->data = buf;
> +
> +		dev_dbg(dev, "Allocated buf %d of %d size %ld\n", i, nr_trbs,
> +			actual_mtu);
> +
> +		ret = mhi_queue_buf(mhi_dev, DMA_FROM_DEVICE, buf, actual_mtu,
> +				    MHI_EOT);
> +		if (ret) {
> +			kfree(buf);
> +			dev_err(dev, "Failed to queue buffer %d\n", i);
> +			return ret;
> +		}
> +	}
> +
> +	return ret;
> +}
> +
> +static void mhi_uci_dev_release(struct kref *ref)
> +{
> +	struct uci_dev *udev =
> +		container_of(ref, struct uci_dev, ref_count);
> +
> +	mutex_destroy(&udev->lock);
> +
> +	dev_set_drvdata(&udev->mhi_dev->dev, NULL);
> +
> +	kfree(udev);
> +}
> +
> +static int mhi_uci_release(struct inode *inode, struct file *file)
> +{
> +	struct uci_dev *udev = file->private_data;
> +	struct uci_buf *itr, *tmp;
> +	struct uci_chan *uchan;
> +
> +	if (kref_read(&udev->ref_count) > 2)
> +		goto exit_uci_release;
> +
> +	if (udev->enabled)
> +		mhi_unprepare_from_transfer(udev->mhi_dev);
> +
> +	/* clean inbound channel */
> +	uchan = &udev->dl_chan;
> +
> +	spin_lock_bh(&uchan->lock);
> +	list_for_each_entry_safe(itr, tmp, &uchan->pending, node) {
> +		list_del(&itr->node);
> +		kfree(itr->data);
> +	}
> +
> +	if (uchan->cur_buf)
> +		kfree(uchan->cur_buf->data);
> +
> +	uchan->cur_buf = NULL;
> +	spin_unlock_bh(&uchan->lock);
> +
> +exit_uci_release:
> +	kref_put(&udev->ref_count, mhi_uci_dev_release);
> +
> +	return 0;
> +}
> +
> +static __poll_t mhi_uci_poll(struct file *file, poll_table *wait)
> +{
> +	struct uci_dev *udev = file->private_data;
> +	struct mhi_device *mhi_dev = udev->mhi_dev;
> +	struct device *dev = &mhi_dev->dev;
> +	struct uci_chan *uchan;
> +	__poll_t mask = 0;
> +
> +	poll_wait(file, &udev->dl_chan.wq, wait);
> +	poll_wait(file, &udev->ul_chan.wq, wait);
> +
> +	if (!udev->enabled) {
> +		mask = EPOLLERR;
> +	} else {
> +		uchan = &udev->dl_chan;
> +		spin_lock_bh(&uchan->lock);
> +		if (!list_empty(&uchan->pending) || uchan->cur_buf) {
> +			dev_dbg(dev, "Client can read from node\n");
> +			mask |= EPOLLIN | EPOLLRDNORM;
> +		}
> +		spin_unlock_bh(&uchan->lock);
> +	}
> +
> +	if (!udev->enabled) {
> +		mask |= EPOLLERR;
> +	} else if (mhi_get_no_free_descriptors(mhi_dev, DMA_TO_DEVICE) > 0) {
> +		dev_dbg(dev, "Client can write to node\n");
> +		mask |= EPOLLOUT | EPOLLWRNORM;
> +	}
> +
> +	dev_dbg(dev, "Client attempted to poll, returning mask 0x%x\n", mask);
> +
> +	return mask;
> +}
> +
> +static ssize_t mhi_uci_write(struct file *file,
> +			     const char __user *buf,
> +			     size_t count,
> +			     loff_t *offp)
> +{
> +	struct uci_dev *udev = file->private_data;
> +	struct mhi_device *mhi_dev = udev->mhi_dev;
> +	struct device *dev = &mhi_dev->dev;
> +	struct uci_chan *uchan = &udev->ul_chan;
> +	size_t bytes_xfered = 0;
> +	int ret, nr_avail = 0;
> +
> +	if (!buf || !count)
> +		return -EINVAL;
> +
> +	/* confirm channel is active */
> +	mutex_lock(&udev->lock);
> +	if (!udev->enabled) {
> +		ret = -ENODEV;
> +		goto err_mtx_unlock;
> +	}
> +
> +	dev_dbg(dev, "%s: to xfer: %lu bytes\n", __func__, count);
> +
> +	while (count) {
> +		size_t xfer_size;
> +		void *kbuf;
> +		enum mhi_flags flags;
> +
> +		mutex_unlock(&udev->lock);
> +		/* wait for free descriptors */
> +		ret = wait_event_interruptible(uchan->wq,
> +					       (!udev->enabled) ||
> +				(nr_avail = mhi_get_no_free_descriptors(mhi_dev,
> +					       DMA_TO_DEVICE)) > 0);
> +
> +		mutex_lock(&udev->lock);

It feels like there is a race here.  What if we wait because there are 
no free descriptors, one becomes available, but someone else manages to 
grab the mutex before we do, and consume it?  mhi_queue_buf() will fail 
later on, and I don't think thats intended.

> +		if (ret == -ERESTARTSYS) {
> +			dev_dbg(dev, "Exit signal caught for node\n");
> +			goto err_mtx_unlock;
> +		}
> +
> +		if (!udev->enabled) {
> +			ret = -ENODEV;
> +			goto err_mtx_unlock;
> +		}
> +
> +		xfer_size = min_t(size_t, count, udev->mtu);
> +		kbuf = kmalloc(xfer_size, GFP_KERNEL);
> +		if (!kbuf) {
> +			ret = -ENOMEM;
> +			goto err_mtx_unlock;
> +		}
> +
> +		ret = copy_from_user(kbuf, buf, xfer_size);
> +		if (ret) {
> +			kfree(kbuf);
> +			ret = -EFAULT;
> +			goto err_mtx_unlock;
> +		}
> +
> +		/* if ring is full after this force EOT */
> +		if (nr_avail > 1 && (count - xfer_size))
> +			flags = MHI_CHAIN;
> +		else
> +			flags = MHI_EOT;
> +
> +		if (udev->enabled)
> +			ret = mhi_queue_buf(mhi_dev, DMA_TO_DEVICE, kbuf,
> +					    xfer_size, flags);
> +		else
> +			ret = -ENODEV;
> +
> +		if (ret) {
> +			kfree(kbuf);
> +			goto err_mtx_unlock;
> +		}
> +
> +		bytes_xfered += xfer_size;
> +		count -= xfer_size;
> +		buf += xfer_size;
> +	}
> +
> +	mutex_unlock(&udev->lock);
> +	dev_dbg(dev, "%s: bytes xferred: %lu\n", __func__, bytes_xfered);
> +
> +	return bytes_xfered;
> +
> +err_mtx_unlock:
> +	mutex_unlock(&udev->lock);
> +
> +	return ret;
> +}
> +
> +static ssize_t mhi_uci_read(struct file *file,
> +			    char __user *buf,
> +			    size_t count,
> +			    loff_t *ppos)
> +{
> +	struct uci_dev *udev = file->private_data;
> +	struct mhi_device *mhi_dev = udev->mhi_dev;
> +	struct uci_chan *uchan = &udev->dl_chan;
> +	struct device *dev = &mhi_dev->dev;
> +	struct uci_buf *ubuf;
> +	char *ptr;
> +	size_t to_copy;
> +	int ret = 0;
> +
> +	if (!buf)
> +		return -EINVAL;
> +
> +	mutex_lock(&udev->lock);
> +	/* confirm channel is active */
> +	if (!udev->enabled) {
> +		ret = -ENODEV;
> +		goto err_mtx_unlock;
> +	}
> +
> +	spin_lock_bh(&uchan->lock);
> +	/* No data available to read, wait */
> +	if (!uchan->cur_buf && list_empty(&uchan->pending)) {
> +		dev_dbg(dev, "No data available to read waiting\n");
> +
> +		spin_unlock_bh(&uchan->lock);
> +		mutex_unlock(&udev->lock);
> +		ret = wait_event_interruptible(uchan->wq,
> +					       (!udev->enabled ||
> +					      !list_empty(&uchan->pending)));
> +
> +		mutex_lock(&udev->lock);
> +		if (ret == -ERESTARTSYS) {
> +			dev_dbg(dev, "Exit signal caught for node\n");
> +			goto err_mtx_unlock;
> +		}
> +
> +		if (!udev->enabled) {
> +			ret = -ENODEV;
> +			goto err_mtx_unlock;
> +		}
> +		spin_lock_bh(&uchan->lock);
> +	}
> +
> +	/* new read, get the next descriptor from the list */
> +	if (!uchan->cur_buf) {
> +		ubuf = list_first_entry_or_null(&uchan->pending,
> +						struct uci_buf, node);
> +		if (!ubuf) {
> +			ret = -EIO;
> +			goto err_spin_unlock;
> +		}
> +
> +		list_del(&ubuf->node);
> +		uchan->cur_buf = ubuf;
> +		uchan->rx_size = ubuf->len;
> +		dev_dbg(dev, "Got pkt of size: %zu\n", uchan->rx_size);
> +	}
> +
> +	ubuf = uchan->cur_buf;
> +
> +	/* Copy the buffer to user space */
> +	to_copy = min_t(size_t, count, uchan->rx_size);
> +	ptr = ubuf->data + (ubuf->len - uchan->rx_size);
> +	spin_unlock_bh(&uchan->lock);
> +
> +	ret = copy_to_user(buf, ptr, to_copy);
> +	if (ret) {
> +		ret = -EFAULT;
> +		goto err_mtx_unlock;
> +	}
> +
> +	spin_lock_bh(&uchan->lock);
> +
> +	dev_dbg(dev, "Copied %lu of %lu bytes\n", to_copy, uchan->rx_size);
> +	uchan->rx_size -= to_copy;
> +
> +	/* we finished with this buffer, queue it back to hardware */
> +	if (!uchan->rx_size) {
> +		uchan->cur_buf = NULL;
> +
> +		if (udev->enabled)
> +			ret = mhi_queue_buf(mhi_dev, DMA_FROM_DEVICE,
> +					    ubuf->data,
> +					    udev->actual_mtu, MHI_EOT);
> +		else
> +			ret = -ENODEV;
> +
> +		if (ret) {
> +			dev_err(dev, "Failed to recycle element: %d\n", ret);
> +			kfree(ubuf->data);
> +			goto err_spin_unlock;
> +		}
> +	}
> +	spin_unlock_bh(&uchan->lock);
> +	mutex_unlock(&udev->lock);
> +
> +	dev_dbg(dev, "%s: Returning %lu bytes\n", __func__, to_copy);
> +
> +	return to_copy;
> +
> +err_spin_unlock:
> +	spin_unlock_bh(&uchan->lock);
> +err_mtx_unlock:
> +	mutex_unlock(&udev->lock);
> +	return ret;
> +}
> +
> +static int mhi_uci_open(struct inode *inode, struct file *filp)
> +{
> +	struct uci_dev *udev = NULL;
> +	unsigned int minor = iminor(inode);
> +	int ret = -EIO;
> +	struct uci_buf *buf_itr, *tmp;
> +	struct uci_chan *dl_chan;
> +	struct mhi_device *mhi_dev;
> +	struct device *dev;
> +
> +	mutex_lock(&uci_idr_mutex);
> +	udev = idr_find(&uci_idr, minor);
> +	mutex_unlock(&uci_idr_mutex);
> +	if (!udev) {
> +		pr_err("uci dev: minor %d not found\n", minor);
> +		ret = -ENODEV;
> +		goto error_no_dev;
> +	}
> +
> +	kref_get(&udev->ref_count);
> +
> +	mhi_dev = udev->mhi_dev;

mhi_dev never gets used.  You could use it down at the 
prepare_for_transfer, but you don't.  Might as well just remove this 
from the stack.

> +	dev = &mhi_dev->dev;
> +
> +	mutex_lock(&udev->lock);
> +	if (kref_read(&udev->ref_count) > 2) {
> +		dev_dbg(dev, "Node already opened\n");
> +		goto exit_uci_open;
> +	}
> +
> +	if (!udev->enabled) {
> +		dev_info(dev, "Node exists, but is not in active state!\n");
> +		goto error_open_chan;
> +	}
> +
> +	dev_dbg(dev, "Starting channel\n");
> +	ret = mhi_prepare_for_transfer(udev->mhi_dev);
> +	if (ret) {
> +		dev_err(dev, "Error starting transfer channels\n");
> +		goto error_open_chan;
> +	}
> +
> +	ret = mhi_queue_inbound(udev);
> +	if (ret)
> +		goto error_rx_queue;
> +
> +exit_uci_open:
> +	filp->private_data = udev;
> +	mutex_unlock(&udev->lock);
> +
> +	return 0;
> +
> +error_rx_queue:
> +	dl_chan = &udev->dl_chan;
> +	mhi_unprepare_from_transfer(udev->mhi_dev);
> +	list_for_each_entry_safe(buf_itr, tmp, &dl_chan->pending, node) {
> +		list_del(&buf_itr->node);
> +		kfree(buf_itr->data);
> +	}
> +error_open_chan:
> +	mutex_unlock(&udev->lock);
> +	kref_put(&udev->ref_count, mhi_uci_dev_release);
> +error_no_dev:
> +	return ret;
> +}
> +
> +static const struct file_operations mhidev_fops = {
> +	.owner = THIS_MODULE,
> +	.open = mhi_uci_open,
> +	.release = mhi_uci_release,
> +	.read = mhi_uci_read,
> +	.write = mhi_uci_write,
> +	.poll = mhi_uci_poll,
> +};
> +
> +static void mhi_ul_xfer_cb(struct mhi_device *mhi_dev,
> +			   struct mhi_result *mhi_result)
> +{
> +	struct uci_dev *udev = dev_get_drvdata(&mhi_dev->dev);
> +	struct uci_chan *uchan = &udev->ul_chan;
> +	struct device *dev = &mhi_dev->dev;
> +
> +	dev_dbg(dev, "status: %d xfer_len: %zu\n",
> +		mhi_result->transaction_status, mhi_result->bytes_xferd);
> +
> +	kfree(mhi_result->buf_addr);
> +
> +	if (!mhi_result->transaction_status)
> +		wake_up(&uchan->wq);
> +}
> +
> +static void mhi_dl_xfer_cb(struct mhi_device *mhi_dev,
> +			   struct mhi_result *mhi_result)
> +{
> +	struct uci_dev *udev = dev_get_drvdata(&mhi_dev->dev);
> +	struct uci_chan *uchan = &udev->dl_chan;
> +	struct device *dev = &mhi_dev->dev;
> +	struct uci_buf *buf;
> +
> +	dev_dbg(dev, "status: %d receive_len: %zu\n",
> +		mhi_result->transaction_status, mhi_result->bytes_xferd);
> +
> +	if (mhi_result->transaction_status == -ENOTCONN) {
> +		kfree(mhi_result->buf_addr);
> +		return;
> +	}
> +
> +	spin_lock_bh(&uchan->lock);
> +	buf = mhi_result->buf_addr + udev->actual_mtu;
> +	buf->data = mhi_result->buf_addr;
> +	buf->len = mhi_result->bytes_xferd;
> +	list_add_tail(&buf->node, &uchan->pending);
> +	spin_unlock_bh(&uchan->lock);
> +
> +	wake_up(&uchan->wq);
> +}
> +
> +static int mhi_uci_probe(struct mhi_device *mhi_dev,
> +			 const struct mhi_device_id *id)
> +{
> +	struct uci_dev *udev;
> +	struct mhi_controller *mhi_cntrl = mhi_dev->mhi_cntrl;
> +	struct device *dev;
> +	int index, dir;
> +
> +	udev = kzalloc(sizeof(*udev), GFP_KERNEL);

What if you defer creating the udev until the first open(), and tie the 
xfer prepare state to the ref count of the udev instance?

> +	if (!udev)
> +		return -ENOMEM;
> +
> +	kref_init(&udev->ref_count);
> +	mutex_init(&udev->lock);
> +	udev->mhi_dev = mhi_dev;
> +
> +	mutex_lock(&udev->lock);
> +
> +	mutex_lock(&uci_idr_mutex);
> +	index = idr_alloc(&uci_idr, udev, 0, MAX_UCI_MINORS, GFP_KERNEL);
> +	mutex_unlock(&uci_idr_mutex);
> +	if (index < 0) {
> +		mutex_unlock(&udev->lock);
> +		kfree(udev);
> +		return index;
> +	}
> +
> +	udev->minor = index;
> +
> +	/* create device file node /dev/mhi_<cntrl_dev_name>_<mhi_dev_name> */
> +	dev = device_create(uci_dev_class, &mhi_dev->dev,
> +			    MKDEV(uci_dev_major, index), udev,
> +			    DEVICE_NAME "_%s_%s",

DEVICE_NAME is only used here.  Doesn't seem like having a macro is 
providing much value.  I suggest just inlining the value.

> +			    dev_name(mhi_cntrl->cntrl_dev), mhi_dev->name);
> +	if (IS_ERR(dev)) {
> +		mutex_lock(&uci_idr_mutex);
> +		idr_remove(&uci_idr, udev->minor);
> +		mutex_unlock(&uci_idr_mutex);
> +		mutex_unlock(&udev->lock);
> +		kfree(udev);
> +		return PTR_ERR(dev);
> +	}
> +
> +	for (dir = 0; dir < 2; dir++) {
> +		struct uci_chan *uchan = (dir) ?
> +			&udev->ul_chan : &udev->dl_chan;
> +		spin_lock_init(&uchan->lock);
> +		init_waitqueue_head(&uchan->wq);
> +		INIT_LIST_HEAD(&uchan->pending);
> +	}
> +
> +	udev->mtu = min_t(size_t, id->driver_data, MHI_MAX_MTU);
> +	udev->actual_mtu = udev->mtu - sizeof(struct uci_buf);

"mtu" vs "actual_mtu" seems to be very confusing.  To start, why not 
"tx_mtu" and "rx_mtu"?  But going from there, why have different mtus 
for tx and rx?  Why not just increase the rx allocations by the required 
"header" (struct uci_buf) as needed to keep symetry?

> +	dev_set_drvdata(&mhi_dev->dev, udev);
> +	udev->enabled = true;
> +
> +	mutex_unlock(&udev->lock);
> +
> +	dev_info(&mhi_dev->dev, "probed uci dev: minor %d\n", index);
> +
> +	return 0;
> +};
> +
> +static void mhi_uci_remove(struct mhi_device *mhi_dev)
> +{
> +	struct uci_dev *udev = dev_get_drvdata(&mhi_dev->dev);
> +
> +	mutex_lock(&udev->lock);
> +
> +	/* disable the node */
> +	udev->enabled = false;
> +
> +	wake_up(&udev->dl_chan.wq);
> +	wake_up(&udev->ul_chan.wq);
> +
> +	/* delete the node to prevent new opens */
> +	device_destroy(uci_dev_class, MKDEV(uci_dev_major, udev->minor));
> +
> +	mutex_lock(&uci_idr_mutex);
> +	idr_remove(&uci_idr, udev->minor);
> +	mutex_unlock(&uci_idr_mutex);
> +
> +	mutex_unlock(&udev->lock);
> +
> +	kref_put(&udev->ref_count, mhi_uci_dev_release);
> +}
> +
> +/* .driver_data stores max mtu */
> +static const struct mhi_device_id mhi_uci_match_table[] = {
> +	{ .chan = "LOOPBACK", .driver_data = 0x1000},
> +	{},
> +};
> +MODULE_DEVICE_TABLE(mhi, mhi_uci_match_table);
> +
> +static struct mhi_driver mhi_uci_driver = {
> +	.id_table = mhi_uci_match_table,
> +	.remove = mhi_uci_remove,
> +	.probe = mhi_uci_probe,
> +	.ul_xfer_cb = mhi_ul_xfer_cb,
> +	.dl_xfer_cb = mhi_dl_xfer_cb,
> +	.driver = {
> +		.name = MHI_UCI_DRIVER_NAME,
> +	},
> +};
> +
> +static int mhi_uci_init(void)
> +{
> +	int ret;
> +
> +	ret = register_chrdev(0, MHI_UCI_DRIVER_NAME, &mhidev_fops);
> +	if (ret < 0)
> +		return ret;
> +
> +	uci_dev_major = ret;
> +	uci_dev_class = class_create(THIS_MODULE, MHI_UCI_DRIVER_NAME);
> +	if (IS_ERR(uci_dev_class)) {
> +		unregister_chrdev(uci_dev_major, MHI_UCI_DRIVER_NAME);
> +		return -ENODEV;
> +	}
> +
> +	ret = mhi_driver_register(&mhi_uci_driver);
> +	if (ret) {
> +		class_destroy(uci_dev_class);
> +		unregister_chrdev(uci_dev_major, MHI_UCI_DRIVER_NAME);
> +	}
> +
> +	return ret;
> +}
> +
> +static void __exit mhi_uci_exit(void)
> +{
> +	mhi_driver_unregister(&mhi_uci_driver);
> +	class_destroy(uci_dev_class);
> +	unregister_chrdev(uci_dev_major, MHI_UCI_DRIVER_NAME);

I think you need an idr_destroy() of uci_idr here.

> +}
> +
> +module_init(mhi_uci_init);
> +module_exit(mhi_uci_exit);
> +MODULE_LICENSE("GPL v2");
> +MODULE_DESCRIPTION("MHI UCI Driver");
>

-- 
Jeffrey Hugo
Qualcomm Technologies, Inc. is a member of the
Code Aurora Forum, a Linux Foundation Collaborative Project.

Re: [PATCH v6 4/4] bus: mhi: Add userspace client interface driver

[Hemant Kumar]

Hi Jeff,

On 9/18/20 1:08 PM, Jeffrey Hugo wrote:
> On 9/16/2020 1:56 PM, Hemant Kumar wrote:
>> This MHI client driver allows userspace clients to transfer
>> raw data between MHI device and host using standard file operations.
>> Device file node is created with format
>>
>> /dev/mhi_<controller_name>_<mhi_device_name>
>>
>> Currently it supports LOOPBACK channel.
>>
>> Signed-off-by: Hemant Kumar <hemantk@codeaurora.org>
...
...
>> +
>> +static ssize_t mhi_uci_write(struct file *file,
>> +                 const char __user *buf,
>> +                 size_t count,
>> +                 loff_t *offp)
>> +{
>> +    struct uci_dev *udev = file->private_data;
>> +    struct mhi_device *mhi_dev = udev->mhi_dev;
>> +    struct device *dev = &mhi_dev->dev;
>> +    struct uci_chan *uchan = &udev->ul_chan;
>> +    size_t bytes_xfered = 0;
>> +    int ret, nr_avail = 0;
>> +
>> +    if (!buf || !count)
>> +        return -EINVAL;
>> +
>> +    /* confirm channel is active */
>> +    mutex_lock(&udev->lock);
>> +    if (!udev->enabled) {
>> +        ret = -ENODEV;
>> +        goto err_mtx_unlock;
>> +    }
>> +
>> +    dev_dbg(dev, "%s: to xfer: %lu bytes\n", __func__, count);
>> +
>> +    while (count) {
>> +        size_t xfer_size;
>> +        void *kbuf;
>> +        enum mhi_flags flags;
>> +
>> +        mutex_unlock(&udev->lock);
>> +        /* wait for free descriptors */
>> +        ret = wait_event_interruptible(uchan->wq,
>> +                           (!udev->enabled) ||
>> +                (nr_avail = mhi_get_no_free_descriptors(mhi_dev,
>> +                           DMA_TO_DEVICE)) > 0);
>> +
>> +        mutex_lock(&udev->lock);
> 
> It feels like there is a race here.  What if we wait because there are 
> no free descriptors, one becomes available, but someone else manages to 
> grab the mutex before we do, and consume it?  mhi_queue_buf() will fail 
> later on, and I don't think thats intended.
I agree Jeff, one option i can think of having separate read and write 
lock so that writer can hold lock if TREs are not available but reader 
can continue.
> 
>> +        if (ret == -ERESTARTSYS) {
>> +            dev_dbg(dev, "Exit signal caught for node\n");
>> +            goto err_mtx_unlock;
>> +        }
>> +
>> +        if (!udev->enabled) {
>> +            ret = -ENODEV;
>> +            goto err_mtx_unlock;
>> +        }
>> +
>> +        xfer_size = min_t(size_t, count, udev->mtu);
>> +        kbuf = kmalloc(xfer_size, GFP_KERNEL);
>> +        if (!kbuf) {
>> +            ret = -ENOMEM;
>> +            goto err_mtx_unlock;
>> +        }
>> +
>> +        ret = copy_from_user(kbuf, buf, xfer_size);
>> +        if (ret) {
>> +            kfree(kbuf);
>> +            ret = -EFAULT;
>> +            goto err_mtx_unlock;
>> +        }
>> +
>> +        /* if ring is full after this force EOT */
>> +        if (nr_avail > 1 && (count - xfer_size))
>> +            flags = MHI_CHAIN;
>> +        else
>> +            flags = MHI_EOT;
>> +
>> +        if (udev->enabled)
>> +            ret = mhi_queue_buf(mhi_dev, DMA_TO_DEVICE, kbuf,
>> +                        xfer_size, flags);
>> +        else
>> +            ret = -ENODEV;
>> +
>> +        if (ret) {
>> +            kfree(kbuf);
>> +            goto err_mtx_unlock;
>> +        }
>> +
>> +        bytes_xfered += xfer_size;
>> +        count -= xfer_size;
>> +        buf += xfer_size;
>> +    }
>> +
>> +    mutex_unlock(&udev->lock);
>> +    dev_dbg(dev, "%s: bytes xferred: %lu\n", __func__, bytes_xfered);
>> +
>> +    return bytes_xfered;
>> +
>> +err_mtx_unlock:
>> +    mutex_unlock(&udev->lock);
>> +
>> +    return ret;
>> +}
>> +
>> +static ssize_t mhi_uci_read(struct file *file,
>> +                char __user *buf,
>> +                size_t count,
>> +                loff_t *ppos)
>> +{
>> +    struct uci_dev *udev = file->private_data;
>> +    struct mhi_device *mhi_dev = udev->mhi_dev;
>> +    struct uci_chan *uchan = &udev->dl_chan;
>> +    struct device *dev = &mhi_dev->dev;
>> +    struct uci_buf *ubuf;
>> +    char *ptr;
>> +    size_t to_copy;
>> +    int ret = 0;
>> +
>> +    if (!buf)
>> +        return -EINVAL;
>> +
>> +    mutex_lock(&udev->lock);
>> +    /* confirm channel is active */
>> +    if (!udev->enabled) {
>> +        ret = -ENODEV;
>> +        goto err_mtx_unlock;
>> +    }
>> +
>> +    spin_lock_bh(&uchan->lock);
>> +    /* No data available to read, wait */
>> +    if (!uchan->cur_buf && list_empty(&uchan->pending)) {
>> +        dev_dbg(dev, "No data available to read waiting\n");
>> +
>> +        spin_unlock_bh(&uchan->lock);
>> +        mutex_unlock(&udev->lock);
>> +        ret = wait_event_interruptible(uchan->wq,
>> +                           (!udev->enabled ||
>> +                          !list_empty(&uchan->pending)));
>> +
>> +        mutex_lock(&udev->lock);
>> +        if (ret == -ERESTARTSYS) {
>> +            dev_dbg(dev, "Exit signal caught for node\n");
>> +            goto err_mtx_unlock;
>> +        }
>> +
>> +        if (!udev->enabled) {
>> +            ret = -ENODEV;
>> +            goto err_mtx_unlock;
>> +        }
>> +        spin_lock_bh(&uchan->lock);
>> +    }
>> +
>> +    /* new read, get the next descriptor from the list */
>> +    if (!uchan->cur_buf) {
>> +        ubuf = list_first_entry_or_null(&uchan->pending,
>> +                        struct uci_buf, node);
>> +        if (!ubuf) {
>> +            ret = -EIO;
>> +            goto err_spin_unlock;
>> +        }
>> +
>> +        list_del(&ubuf->node);
>> +        uchan->cur_buf = ubuf;
>> +        uchan->rx_size = ubuf->len;
>> +        dev_dbg(dev, "Got pkt of size: %zu\n", uchan->rx_size);
>> +    }
>> +
>> +    ubuf = uchan->cur_buf;
>> +
>> +    /* Copy the buffer to user space */
>> +    to_copy = min_t(size_t, count, uchan->rx_size);
>> +    ptr = ubuf->data + (ubuf->len - uchan->rx_size);
>> +    spin_unlock_bh(&uchan->lock);
>> +
>> +    ret = copy_to_user(buf, ptr, to_copy);
>> +    if (ret) {
>> +        ret = -EFAULT;
>> +        goto err_mtx_unlock;
>> +    }
>> +
>> +    spin_lock_bh(&uchan->lock);
>> +
>> +    dev_dbg(dev, "Copied %lu of %lu bytes\n", to_copy, uchan->rx_size);
>> +    uchan->rx_size -= to_copy;
>> +
>> +    /* we finished with this buffer, queue it back to hardware */
>> +    if (!uchan->rx_size) {
>> +        uchan->cur_buf = NULL;
>> +
>> +        if (udev->enabled)
>> +            ret = mhi_queue_buf(mhi_dev, DMA_FROM_DEVICE,
>> +                        ubuf->data,
>> +                        udev->actual_mtu, MHI_EOT);
>> +        else
>> +            ret = -ENODEV;
>> +
>> +        if (ret) {
>> +            dev_err(dev, "Failed to recycle element: %d\n", ret);
>> +            kfree(ubuf->data);
>> +            goto err_spin_unlock;
>> +        }
>> +    }
>> +    spin_unlock_bh(&uchan->lock);
>> +    mutex_unlock(&udev->lock);
>> +
>> +    dev_dbg(dev, "%s: Returning %lu bytes\n", __func__, to_copy);
>> +
>> +    return to_copy;
>> +
>> +err_spin_unlock:
>> +    spin_unlock_bh(&uchan->lock);
>> +err_mtx_unlock:
>> +    mutex_unlock(&udev->lock);
>> +    return ret;
>> +}
>> +
>> +static int mhi_uci_open(struct inode *inode, struct file *filp)
>> +{
>> +    struct uci_dev *udev = NULL;
>> +    unsigned int minor = iminor(inode);
>> +    int ret = -EIO;
>> +    struct uci_buf *buf_itr, *tmp;
>> +    struct uci_chan *dl_chan;
>> +    struct mhi_device *mhi_dev;
>> +    struct device *dev;
>> +
>> +    mutex_lock(&uci_idr_mutex);
>> +    udev = idr_find(&uci_idr, minor);
>> +    mutex_unlock(&uci_idr_mutex);
>> +    if (!udev) {
>> +        pr_err("uci dev: minor %d not found\n", minor);
>> +        ret = -ENODEV;
>> +        goto error_no_dev;
>> +    }
>> +
>> +    kref_get(&udev->ref_count);
>> +
>> +    mhi_dev = udev->mhi_dev;
> 
> mhi_dev never gets used.  You could use it down at the 
> prepare_for_transfer, but you don't.  Might as well just remove this 
> from the stack.
this was done to pass dev pointer instead of udev->mhi_dev->dev.
> 
>> +    dev = &mhi_dev->dev;
>> +
>> +    mutex_lock(&udev->lock);
>> +    if (kref_read(&udev->ref_count) > 2) {
>> +        dev_dbg(dev, "Node already opened\n");
>> +        goto exit_uci_open;
>> +    }
>> +
>> +    if (!udev->enabled) {
>> +        dev_info(dev, "Node exists, but is not in active state!\n");
>> +        goto error_open_chan;
>> +    }
>> +
>> +    dev_dbg(dev, "Starting channel\n");
>> +    ret = mhi_prepare_for_transfer(udev->mhi_dev);
>> +    if (ret) {
>> +        dev_err(dev, "Error starting transfer channels\n");
>> +        goto error_open_chan;
>> +    }
>> +
>> +    ret = mhi_queue_inbound(udev);
>> +    if (ret)
>> +        goto error_rx_queue;
>> +
>> +exit_uci_open:
>> +    filp->private_data = udev;
>> +    mutex_unlock(&udev->lock);
>> +
>> +    return 0;
>> +
>> +error_rx_queue:
>> +    dl_chan = &udev->dl_chan;
>> +    mhi_unprepare_from_transfer(udev->mhi_dev);
>> +    list_for_each_entry_safe(buf_itr, tmp, &dl_chan->pending, node) {
>> +        list_del(&buf_itr->node);
>> +        kfree(buf_itr->data);
>> +    }
>> +error_open_chan:
>> +    mutex_unlock(&udev->lock);
>> +    kref_put(&udev->ref_count, mhi_uci_dev_release);
>> +error_no_dev:
>> +    return ret;
>> +}
>> +
>> +static const struct file_operations mhidev_fops = {
>> +    .owner = THIS_MODULE,
>> +    .open = mhi_uci_open,
>> +    .release = mhi_uci_release,
>> +    .read = mhi_uci_read,
>> +    .write = mhi_uci_write,
>> +    .poll = mhi_uci_poll,
>> +};
>> +
>> +static void mhi_ul_xfer_cb(struct mhi_device *mhi_dev,
>> +               struct mhi_result *mhi_result)
>> +{
>> +    struct uci_dev *udev = dev_get_drvdata(&mhi_dev->dev);
>> +    struct uci_chan *uchan = &udev->ul_chan;
>> +    struct device *dev = &mhi_dev->dev;
>> +
>> +    dev_dbg(dev, "status: %d xfer_len: %zu\n",
>> +        mhi_result->transaction_status, mhi_result->bytes_xferd);
>> +
>> +    kfree(mhi_result->buf_addr);
>> +
>> +    if (!mhi_result->transaction_status)
>> +        wake_up(&uchan->wq);
>> +}
>> +
>> +static void mhi_dl_xfer_cb(struct mhi_device *mhi_dev,
>> +               struct mhi_result *mhi_result)
>> +{
>> +    struct uci_dev *udev = dev_get_drvdata(&mhi_dev->dev);
>> +    struct uci_chan *uchan = &udev->dl_chan;
>> +    struct device *dev = &mhi_dev->dev;
>> +    struct uci_buf *buf;
>> +
>> +    dev_dbg(dev, "status: %d receive_len: %zu\n",
>> +        mhi_result->transaction_status, mhi_result->bytes_xferd);
>> +
>> +    if (mhi_result->transaction_status == -ENOTCONN) {
>> +        kfree(mhi_result->buf_addr);
>> +        return;
>> +    }
>> +
>> +    spin_lock_bh(&uchan->lock);
>> +    buf = mhi_result->buf_addr + udev->actual_mtu;
>> +    buf->data = mhi_result->buf_addr;
>> +    buf->len = mhi_result->bytes_xferd;
>> +    list_add_tail(&buf->node, &uchan->pending);
>> +    spin_unlock_bh(&uchan->lock);
>> +
>> +    wake_up(&uchan->wq);
>> +}
>> +
>> +static int mhi_uci_probe(struct mhi_device *mhi_dev,
>> +             const struct mhi_device_id *id)
>> +{
>> +    struct uci_dev *udev;
>> +    struct mhi_controller *mhi_cntrl = mhi_dev->mhi_cntrl;
>> +    struct device *dev;
>> +    int index, dir;
>> +
>> +    udev = kzalloc(sizeof(*udev), GFP_KERNEL);
> 
> What if you defer creating the udev until the first open(), and tie the 
> xfer prepare state to the ref count of the udev instance?
With current implementation udev and minor are mapped using idr_alloc.
i was thinking about another option of adding kref for uci_chann object
to take care of channel and buffer ref count.
> 
>> +    if (!udev)
>> +        return -ENOMEM;
>> +
>> +    kref_init(&udev->ref_count);
>> +    mutex_init(&udev->lock);
>> +    udev->mhi_dev = mhi_dev;
>> +
>> +    mutex_lock(&udev->lock);
>> +
>> +    mutex_lock(&uci_idr_mutex);
>> +    index = idr_alloc(&uci_idr, udev, 0, MAX_UCI_MINORS, GFP_KERNEL);
>> +    mutex_unlock(&uci_idr_mutex);
>> +    if (index < 0) {
>> +        mutex_unlock(&udev->lock);
>> +        kfree(udev);
>> +        return index;
>> +    }
>> +
>> +    udev->minor = index;
>> +
>> +    /* create device file node 
>> /dev/mhi_<cntrl_dev_name>_<mhi_dev_name> */
>> +    dev = device_create(uci_dev_class, &mhi_dev->dev,
>> +                MKDEV(uci_dev_major, index), udev,
>> +                DEVICE_NAME "_%s_%s",
> 
> DEVICE_NAME is only used here.  Doesn't seem like having a macro is 
> providing much value.  I suggest just inlining the value.
> 
>> +                dev_name(mhi_cntrl->cntrl_dev), mhi_dev->name);
>> +    if (IS_ERR(dev)) {
>> +        mutex_lock(&uci_idr_mutex);
>> +        idr_remove(&uci_idr, udev->minor);
>> +        mutex_unlock(&uci_idr_mutex);
>> +        mutex_unlock(&udev->lock);
>> +        kfree(udev);
>> +        return PTR_ERR(dev);
>> +    }
>> +
>> +    for (dir = 0; dir < 2; dir++) {
>> +        struct uci_chan *uchan = (dir) ?
>> +            &udev->ul_chan : &udev->dl_chan;
>> +        spin_lock_init(&uchan->lock);
>> +        init_waitqueue_head(&uchan->wq);
>> +        INIT_LIST_HEAD(&uchan->pending);
>> +    }
>> +
>> +    udev->mtu = min_t(size_t, id->driver_data, MHI_MAX_MTU);
>> +    udev->actual_mtu = udev->mtu - sizeof(struct uci_buf);
> 
> "mtu" vs "actual_mtu" seems to be very confusing.  To start, why not 
> "tx_mtu" and "rx_mtu"?  But going from there, why have different mtus 
> for tx and rx?  Why not just increase the rx allocations by the required 
> "header" (struct uci_buf) as needed to keep symetry?
in past we had issues with order of 3 allocation failures for bigger mtu 
size, so we decided to use memory within the mtu buffer. i can add 
tx_mtu and rx_mtu if that makes it clear.
> 
>> +    dev_set_drvdata(&mhi_dev->dev, udev);
>> +    udev->enabled = true;
>> +
>> +    mutex_unlock(&udev->lock);
>> +
>> +    dev_info(&mhi_dev->dev, "probed uci dev: minor %d\n", index);
>> +
>> +    return 0;
>> +};
>> +
>> +static void mhi_uci_remove(struct mhi_device *mhi_dev)
>> +{
>> +    struct uci_dev *udev = dev_get_drvdata(&mhi_dev->dev);
>> +
>> +    mutex_lock(&udev->lock);
>> +
>> +    /* disable the node */
>> +    udev->enabled = false;
>> +
>> +    wake_up(&udev->dl_chan.wq);
>> +    wake_up(&udev->ul_chan.wq);
>> +
>> +    /* delete the node to prevent new opens */
>> +    device_destroy(uci_dev_class, MKDEV(uci_dev_major, udev->minor));
>> +
>> +    mutex_lock(&uci_idr_mutex);
>> +    idr_remove(&uci_idr, udev->minor);
>> +    mutex_unlock(&uci_idr_mutex);
>> +
>> +    mutex_unlock(&udev->lock);
>> +
>> +    kref_put(&udev->ref_count, mhi_uci_dev_release);
>> +}
>> +
>> +/* .driver_data stores max mtu */
>> +static const struct mhi_device_id mhi_uci_match_table[] = {
>> +    { .chan = "LOOPBACK", .driver_data = 0x1000},
>> +    {},
>> +};
>> +MODULE_DEVICE_TABLE(mhi, mhi_uci_match_table);
>> +
>> +static struct mhi_driver mhi_uci_driver = {
>> +    .id_table = mhi_uci_match_table,
>> +    .remove = mhi_uci_remove,
>> +    .probe = mhi_uci_probe,
>> +    .ul_xfer_cb = mhi_ul_xfer_cb,
>> +    .dl_xfer_cb = mhi_dl_xfer_cb,
>> +    .driver = {
>> +        .name = MHI_UCI_DRIVER_NAME,
>> +    },
>> +};
>> +
>> +static int mhi_uci_init(void)
>> +{
>> +    int ret;
>> +
>> +    ret = register_chrdev(0, MHI_UCI_DRIVER_NAME, &mhidev_fops);
>> +    if (ret < 0)
>> +        return ret;
>> +
>> +    uci_dev_major = ret;
>> +    uci_dev_class = class_create(THIS_MODULE, MHI_UCI_DRIVER_NAME);
>> +    if (IS_ERR(uci_dev_class)) {
>> +        unregister_chrdev(uci_dev_major, MHI_UCI_DRIVER_NAME);
>> +        return -ENODEV;
>> +    }
>> +
>> +    ret = mhi_driver_register(&mhi_uci_driver);
>> +    if (ret) {
>> +        class_destroy(uci_dev_class);
>> +        unregister_chrdev(uci_dev_major, MHI_UCI_DRIVER_NAME);
>> +    }
>> +
>> +    return ret;
>> +}
>> +
>> +static void __exit mhi_uci_exit(void)
>> +{
>> +    mhi_driver_unregister(&mhi_uci_driver);
>> +    class_destroy(uci_dev_class);
>> +    unregister_chrdev(uci_dev_major, MHI_UCI_DRIVER_NAME);
> 
> I think you need an idr_destroy() of uci_idr here.
Done.
> 
>> +}
>> +
>> +module_init(mhi_uci_init);
>> +module_exit(mhi_uci_exit);
>> +MODULE_LICENSE("GPL v2");
>> +MODULE_DESCRIPTION("MHI UCI Driver");
>>
> 

-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

Re: [PATCH v6 4/4] bus: mhi: Add userspace client interface driver

[Loic Poulain]

Hi Hemant,

See comments inline, but globally, the locking and ref counting is
more complicated than it should be.

On Wed, 16 Sep 2020 at 21:57, Hemant Kumar <hemantk@codeaurora.org> wrote:
>
> This MHI client driver allows userspace clients to transfer
> raw data between MHI device and host using standard file operations.
> Device file node is created with format
>
> /dev/mhi_<controller_name>_<mhi_device_name>
>
> Currently it supports LOOPBACK channel.
>
> Signed-off-by: Hemant Kumar <hemantk@codeaurora.org>
> ---
>  drivers/bus/mhi/Kconfig  |  13 +
>  drivers/bus/mhi/Makefile |   4 +
>  drivers/bus/mhi/uci.c    | 657 +++++++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 674 insertions(+)
>  create mode 100644 drivers/bus/mhi/uci.c
>
> diff --git a/drivers/bus/mhi/Kconfig b/drivers/bus/mhi/Kconfig
> index 6a217ff..8aebe8b 100644
> --- a/drivers/bus/mhi/Kconfig
> +++ b/drivers/bus/mhi/Kconfig
> @@ -20,3 +20,16 @@ config MHI_BUS_DEBUG
>          Enable debugfs support for use with the MHI transport. Allows
>          reading and/or modifying some values within the MHI controller
>          for debug and test purposes.
> +
> +config MHI_UCI
> +       tristate "MHI UCI"
> +       depends on MHI_BUS
> +       help
> +        MHI based userspace client interface driver is used for transferring
> +        raw data between host and device using standard file operations from
> +        userspace. Open, read, write, and close operations are supported
> +        by this driver. Please check mhi_uci_match_table for all supported
> +        channels that are exposed to userspace.
> +
> +        To compile this driver as a module, choose M here: the module will be
> +        called mhi_uci.
> diff --git a/drivers/bus/mhi/Makefile b/drivers/bus/mhi/Makefile
> index 19e6443..80feefb 100644
> --- a/drivers/bus/mhi/Makefile
> +++ b/drivers/bus/mhi/Makefile
> @@ -1,2 +1,6 @@
>  # core layer
>  obj-y += core/
> +
> +# MHI client
> +mhi_uci-y := uci.o
> +obj-$(CONFIG_MHI_UCI) += mhi_uci.o
> diff --git a/drivers/bus/mhi/uci.c b/drivers/bus/mhi/uci.c
> new file mode 100644
> index 0000000..d6758f2
> --- /dev/null
> +++ b/drivers/bus/mhi/uci.c
> @@ -0,0 +1,657 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/* Copyright (c) 2018-2020, The Linux Foundation. All rights reserved.*/
> +
> +#include <linux/kernel.h>
> +#include <linux/mhi.h>
> +#include <linux/mod_devicetable.h>
> +#include <linux/module.h>
> +#include <linux/poll.h>
> +
> +#define DEVICE_NAME "mhi"
> +#define MHI_UCI_DRIVER_NAME "mhi_uci"
> +#define MAX_UCI_MINORS (128)
> +
> +static DEFINE_IDR(uci_idr);
> +static DEFINE_MUTEX(uci_idr_mutex);
> +static struct class *uci_dev_class;
> +static int uci_dev_major;
> +
> +/**
> + * struct uci_chan - MHI channel for a uci device
> + * @wq: wait queue for reader/writer
> + * @lock: spin lock
> + * @pending: list of rx buffers userspace is waiting to read
> + * @cur_buf: current buffer userspace is reading
> + * @rx_size: size of the current rx buffer userspace is reading
> + */
> +struct uci_chan {
> +       wait_queue_head_t wq;
> +
> +       /* protects pending and cur_buf members in bh context */
> +       spinlock_t lock;
> +
> +       struct list_head pending;
> +       struct uci_buf *cur_buf;
> +       size_t rx_size;
> +};
> +
> +/**
> + * struct uci_buf - uci buffer
> + * @data: data buffer
> + * @len: length of data buffer
> + * @node: list node of the uci buffer
> + */
> +struct uci_buf {
> +       void *data;
> +       size_t len;
> +       struct list_head node;
> +};
> +
> +/**
> + * struct uci_dev - MHI uci device
> + * @minor: uci device node minor number
> + * @mhi_dev: associated mhi device object
> + * @chan: MHI channel name
> + * @lock: mutex lock
> + * @ul_chan: uplink uci channel object
> + * @dl_chan: downlink uci channel object
> + * @mtu: max tx buffer length
> + * @actual_mtu: maximum size of incoming buffer
> + * @open: open called for device node
> + * @enabled: uci device probed
> + * @ref_count: uci_dev reference count
> + */
> +struct uci_dev {
> +       unsigned int minor;
> +       struct mhi_device *mhi_dev;
> +       const char *chan;
> +
> +       /* protects uci_dev struct members */
> +       struct mutex lock;
> +
> +       struct uci_chan ul_chan;
> +       struct uci_chan dl_chan;
> +       size_t mtu;
> +       size_t actual_mtu;
> +       bool enabled;
> +       struct kref ref_count;
> +};
> +
> +static int mhi_queue_inbound(struct uci_dev *udev)
> +{
> +       struct mhi_device *mhi_dev = udev->mhi_dev;
> +       struct device *dev = &mhi_dev->dev;
> +       size_t mtu = udev->mtu;
> +       size_t actual_mtu = udev->actual_mtu;
> +       int nr_trbs, i, ret = -EIO;
> +       void *buf;
> +       struct uci_buf *uci_buf;
> +
> +       nr_trbs = mhi_get_no_free_descriptors(mhi_dev, DMA_FROM_DEVICE);
> +
> +       for (i = 0; i < nr_trbs; i++) {
> +               buf = kmalloc(mtu, GFP_KERNEL);
> +               if (!buf)
> +                       return -ENOMEM;
> +
> +               uci_buf = buf + actual_mtu;
> +               uci_buf->data = buf;
> +
> +               dev_dbg(dev, "Allocated buf %d of %d size %ld\n", i, nr_trbs,
> +                       actual_mtu);
> +
> +               ret = mhi_queue_buf(mhi_dev, DMA_FROM_DEVICE, buf, actual_mtu,
> +                                   MHI_EOT);
> +               if (ret) {
> +                       kfree(buf);
> +                       dev_err(dev, "Failed to queue buffer %d\n", i);
> +                       return ret;
> +               }
> +       }
> +
> +       return ret;
> +}
> +
> +static void mhi_uci_dev_release(struct kref *ref)
> +{
> +       struct uci_dev *udev =
> +               container_of(ref, struct uci_dev, ref_count);
> +
> +       mutex_destroy(&udev->lock);
> +
> +       dev_set_drvdata(&udev->mhi_dev->dev, NULL);
> +
> +       kfree(udev);
> +}
> +
> +static int mhi_uci_release(struct inode *inode, struct file *file)
> +{
> +       struct uci_dev *udev = file->private_data;
> +       struct uci_buf *itr, *tmp;
> +       struct uci_chan *uchan;
> +
> +       if (kref_read(&udev->ref_count) > 2)
> +               goto exit_uci_release;
> +
> +       if (udev->enabled)
> +               mhi_unprepare_from_transfer(udev->mhi_dev);
> +
> +       /* clean inbound channel */
> +       uchan = &udev->dl_chan;
> +
> +       spin_lock_bh(&uchan->lock);
> +       list_for_each_entry_safe(itr, tmp, &uchan->pending, node) {
> +               list_del(&itr->node);
> +               kfree(itr->data);
> +       }
> +
> +       if (uchan->cur_buf)
> +               kfree(uchan->cur_buf->data);
> +
> +       uchan->cur_buf = NULL;
> +       spin_unlock_bh(&uchan->lock);
> +
> +exit_uci_release:
> +       kref_put(&udev->ref_count, mhi_uci_dev_release);
> +
> +       return 0;
> +}
> +
> +static __poll_t mhi_uci_poll(struct file *file, poll_table *wait)
> +{
> +       struct uci_dev *udev = file->private_data;
> +       struct mhi_device *mhi_dev = udev->mhi_dev;
> +       struct device *dev = &mhi_dev->dev;
> +       struct uci_chan *uchan;
> +       __poll_t mask = 0;
> +
> +       poll_wait(file, &udev->dl_chan.wq, wait);
> +       poll_wait(file, &udev->ul_chan.wq, wait);
> +
> +       if (!udev->enabled) {
> +               mask = EPOLLERR;
> +       } else {
> +               uchan = &udev->dl_chan;
> +               spin_lock_bh(&uchan->lock);
> +               if (!list_empty(&uchan->pending) || uchan->cur_buf) {
> +                       dev_dbg(dev, "Client can read from node\n");
> +                       mask |= EPOLLIN | EPOLLRDNORM;
> +               }
> +               spin_unlock_bh(&uchan->lock);
> +       }
> +
> +       if (!udev->enabled) {
> +               mask |= EPOLLERR;
> +       } else if (mhi_get_no_free_descriptors(mhi_dev, DMA_TO_DEVICE) > 0) {
> +               dev_dbg(dev, "Client can write to node\n");
> +               mask |= EPOLLOUT | EPOLLWRNORM;
> +       }
> +
> +       dev_dbg(dev, "Client attempted to poll, returning mask 0x%x\n", mask);
> +
> +       return mask;
> +}
> +
> +static ssize_t mhi_uci_write(struct file *file,
> +                            const char __user *buf,
> +                            size_t count,
> +                            loff_t *offp)
> +{
> +       struct uci_dev *udev = file->private_data;
> +       struct mhi_device *mhi_dev = udev->mhi_dev;
> +       struct device *dev = &mhi_dev->dev;
> +       struct uci_chan *uchan = &udev->ul_chan;
> +       size_t bytes_xfered = 0;
> +       int ret, nr_avail = 0;
> +
> +       if (!buf || !count)
> +               return -EINVAL;
> +
> +       /* confirm channel is active */
> +       mutex_lock(&udev->lock);
> +       if (!udev->enabled) {
> +               ret = -ENODEV;
> +               goto err_mtx_unlock;
> +       }
> +
> +       dev_dbg(dev, "%s: to xfer: %lu bytes\n", __func__, count);
> +
> +       while (count) {
> +               size_t xfer_size;
> +               void *kbuf;
> +               enum mhi_flags flags;
> +
> +               mutex_unlock(&udev->lock);
> +               /* wait for free descriptors */
> +               ret = wait_event_interruptible(uchan->wq,
> +                                              (!udev->enabled) ||
> +                               (nr_avail = mhi_get_no_free_descriptors(mhi_dev,
> +                                              DMA_TO_DEVICE)) > 0);
> +
> +               mutex_lock(&udev->lock);

All this locking unlocking is odd:
- why do you need locking for testing wait_event return code?
- why do you need this udev->enabled?
- The MHI core should be thread safe + mhi_queue_buf should simply
fail if removing is ongoing.

> +               if (ret == -ERESTARTSYS) {
> +                       dev_dbg(dev, "Exit signal caught for node\n");
> +                       goto err_mtx_unlock;
> +               }
> +
> +               if (!udev->enabled) {
> +                       ret = -ENODEV;
> +                       goto err_mtx_unlock;
> +               }
> +
> +               xfer_size = min_t(size_t, count, udev->mtu);
> +               kbuf = kmalloc(xfer_size, GFP_KERNEL);
> +               if (!kbuf) {
> +                       ret = -ENOMEM;
> +                       goto err_mtx_unlock;
> +               }
> +
> +               ret = copy_from_user(kbuf, buf, xfer_size);
> +               if (ret) {
> +                       kfree(kbuf);
> +                       ret = -EFAULT;
> +                       goto err_mtx_unlock;
> +               }
> +
> +               /* if ring is full after this force EOT */
> +               if (nr_avail > 1 && (count - xfer_size))
> +                       flags = MHI_CHAIN;
> +               else
> +                       flags = MHI_EOT;
> +
> +               if (udev->enabled)
> +                       ret = mhi_queue_buf(mhi_dev, DMA_TO_DEVICE, kbuf,
> +                                           xfer_size, flags);
> +               else
> +                       ret = -ENODEV;
> +
> +               if (ret) {
> +                       kfree(kbuf);
> +                       goto err_mtx_unlock;
> +               }
> +
> +               bytes_xfered += xfer_size;
> +               count -= xfer_size;
> +               buf += xfer_size;
> +       }
> +
> +       mutex_unlock(&udev->lock);
> +       dev_dbg(dev, "%s: bytes xferred: %lu\n", __func__, bytes_xfered);
> +
> +       return bytes_xfered;
> +
> +err_mtx_unlock:
> +       mutex_unlock(&udev->lock);
> +
> +       return ret;
> +}
> +
> +static ssize_t mhi_uci_read(struct file *file,
> +                           char __user *buf,
> +                           size_t count,
> +                           loff_t *ppos)
> +{
> +       struct uci_dev *udev = file->private_data;
> +       struct mhi_device *mhi_dev = udev->mhi_dev;
> +       struct uci_chan *uchan = &udev->dl_chan;
> +       struct device *dev = &mhi_dev->dev;
> +       struct uci_buf *ubuf;
> +       char *ptr;
> +       size_t to_copy;
> +       int ret = 0;
> +
> +       if (!buf)
> +               return -EINVAL;
> +
> +       mutex_lock(&udev->lock);
> +       /* confirm channel is active */
> +       if (!udev->enabled) {
> +               ret = -ENODEV;
> +               goto err_mtx_unlock;
> +       }
> +
> +       spin_lock_bh(&uchan->lock);
> +       /* No data available to read, wait */
> +       if (!uchan->cur_buf && list_empty(&uchan->pending)) {
> +               dev_dbg(dev, "No data available to read waiting\n");
> +
> +               spin_unlock_bh(&uchan->lock);
> +               mutex_unlock(&udev->lock);
> +               ret = wait_event_interruptible(uchan->wq,
> +                                              (!udev->enabled ||
> +                                             !list_empty(&uchan->pending)));
> +
> +               mutex_lock(&udev->lock);
> +               if (ret == -ERESTARTSYS) {
> +                       dev_dbg(dev, "Exit signal caught for node\n");
> +                       goto err_mtx_unlock;
> +               }
> +
> +               if (!udev->enabled) {
> +                       ret = -ENODEV;
> +                       goto err_mtx_unlock;
> +               }
> +               spin_lock_bh(&uchan->lock);
> +       }
> +
> +       /* new read, get the next descriptor from the list */
> +       if (!uchan->cur_buf) {
> +               ubuf = list_first_entry_or_null(&uchan->pending,
> +                                               struct uci_buf, node);
> +               if (!ubuf) {
> +                       ret = -EIO;
> +                       goto err_spin_unlock;
> +               }
> +
> +               list_del(&ubuf->node);
> +               uchan->cur_buf = ubuf;
> +               uchan->rx_size = ubuf->len;
> +               dev_dbg(dev, "Got pkt of size: %zu\n", uchan->rx_size);
> +       }
> +
> +       ubuf = uchan->cur_buf;
> +
> +       /* Copy the buffer to user space */
> +       to_copy = min_t(size_t, count, uchan->rx_size);
> +       ptr = ubuf->data + (ubuf->len - uchan->rx_size);
> +       spin_unlock_bh(&uchan->lock);
> +
> +       ret = copy_to_user(buf, ptr, to_copy);
> +       if (ret) {
> +               ret = -EFAULT;
> +               goto err_mtx_unlock;
> +       }
> +
> +       spin_lock_bh(&uchan->lock);
> +
> +       dev_dbg(dev, "Copied %lu of %lu bytes\n", to_copy, uchan->rx_size);
> +       uchan->rx_size -= to_copy;
> +
> +       /* we finished with this buffer, queue it back to hardware */
> +       if (!uchan->rx_size) {
> +               uchan->cur_buf = NULL;
> +
> +               if (udev->enabled)
> +                       ret = mhi_queue_buf(mhi_dev, DMA_FROM_DEVICE,
> +                                           ubuf->data,
> +                                           udev->actual_mtu, MHI_EOT);
> +               else
> +                       ret = -ENODEV;
> +
> +               if (ret) {
> +                       dev_err(dev, "Failed to recycle element: %d\n", ret);
> +                       kfree(ubuf->data);
> +                       goto err_spin_unlock;
> +               }
> +       }
> +       spin_unlock_bh(&uchan->lock);
> +       mutex_unlock(&udev->lock);
> +
> +       dev_dbg(dev, "%s: Returning %lu bytes\n", __func__, to_copy);
> +
> +       return to_copy;
> +
> +err_spin_unlock:
> +       spin_unlock_bh(&uchan->lock);
> +err_mtx_unlock:
> +       mutex_unlock(&udev->lock);
> +       return ret;
> +}
> +
> +static int mhi_uci_open(struct inode *inode, struct file *filp)
> +{
> +       struct uci_dev *udev = NULL;
> +       unsigned int minor = iminor(inode);
> +       int ret = -EIO;
> +       struct uci_buf *buf_itr, *tmp;
> +       struct uci_chan *dl_chan;
> +       struct mhi_device *mhi_dev;
> +       struct device *dev;
> +
> +       mutex_lock(&uci_idr_mutex);
> +       udev = idr_find(&uci_idr, minor);
> +       mutex_unlock(&uci_idr_mutex);
> +       if (!udev) {
> +               pr_err("uci dev: minor %d not found\n", minor);
> +               ret = -ENODEV;
> +               goto error_no_dev;
> +       }
> +
> +       kref_get(&udev->ref_count);
> +
> +       mhi_dev = udev->mhi_dev;
> +       dev = &mhi_dev->dev;
> +
> +       mutex_lock(&udev->lock);
> +       if (kref_read(&udev->ref_count) > 2) {
> +               dev_dbg(dev, "Node already opened\n");
> +               goto exit_uci_open;
> +       }
> +
> +       if (!udev->enabled) {
> +               dev_info(dev, "Node exists, but is not in active state!\n");
> +               goto error_open_chan;
> +       }
> +
> +       dev_dbg(dev, "Starting channel\n");
> +       ret = mhi_prepare_for_transfer(udev->mhi_dev);
> +       if (ret) {
> +               dev_err(dev, "Error starting transfer channels\n");
> +               goto error_open_chan;
> +       }
> +
> +       ret = mhi_queue_inbound(udev);
> +       if (ret)
> +               goto error_rx_queue;
> +
> +exit_uci_open:
> +       filp->private_data = udev;
> +       mutex_unlock(&udev->lock);
> +
> +       return 0;
> +
> +error_rx_queue:
> +       dl_chan = &udev->dl_chan;
> +       mhi_unprepare_from_transfer(udev->mhi_dev);
> +       list_for_each_entry_safe(buf_itr, tmp, &dl_chan->pending, node) {
> +               list_del(&buf_itr->node);
> +               kfree(buf_itr->data);
> +       }
> +error_open_chan:
> +       mutex_unlock(&udev->lock);
> +       kref_put(&udev->ref_count, mhi_uci_dev_release);
> +error_no_dev:
> +       return ret;
> +}
> +
> +static const struct file_operations mhidev_fops = {
> +       .owner = THIS_MODULE,
> +       .open = mhi_uci_open,
> +       .release = mhi_uci_release,
> +       .read = mhi_uci_read,
> +       .write = mhi_uci_write,
> +       .poll = mhi_uci_poll,
> +};
> +
> +static void mhi_ul_xfer_cb(struct mhi_device *mhi_dev,
> +                          struct mhi_result *mhi_result)
> +{
> +       struct uci_dev *udev = dev_get_drvdata(&mhi_dev->dev);
> +       struct uci_chan *uchan = &udev->ul_chan;
> +       struct device *dev = &mhi_dev->dev;
> +
> +       dev_dbg(dev, "status: %d xfer_len: %zu\n",
> +               mhi_result->transaction_status, mhi_result->bytes_xferd);
> +
> +       kfree(mhi_result->buf_addr);
> +
> +       if (!mhi_result->transaction_status)
> +               wake_up(&uchan->wq);
> +}
> +
> +static void mhi_dl_xfer_cb(struct mhi_device *mhi_dev,
> +                          struct mhi_result *mhi_result)
> +{
> +       struct uci_dev *udev = dev_get_drvdata(&mhi_dev->dev);
> +       struct uci_chan *uchan = &udev->dl_chan;
> +       struct device *dev = &mhi_dev->dev;
> +       struct uci_buf *buf;
> +
> +       dev_dbg(dev, "status: %d receive_len: %zu\n",
> +               mhi_result->transaction_status, mhi_result->bytes_xferd);
> +
> +       if (mhi_result->transaction_status == -ENOTCONN) {
> +               kfree(mhi_result->buf_addr);
> +               return;
> +       }
> +
> +       spin_lock_bh(&uchan->lock);
> +       buf = mhi_result->buf_addr + udev->actual_mtu;
> +       buf->data = mhi_result->buf_addr;
> +       buf->len = mhi_result->bytes_xferd;

You don't need to protect buf here, only uchan pending list, right? so
move lock here.

> +       list_add_tail(&buf->node, &uchan->pending);
> +       spin_unlock_bh(&uchan->lock);
> +
> +       wake_up(&uchan->wq);
> +}
> +
> +static int mhi_uci_probe(struct mhi_device *mhi_dev,
> +                        const struct mhi_device_id *id)
> +{
> +       struct uci_dev *udev;
> +       struct mhi_controller *mhi_cntrl = mhi_dev->mhi_cntrl;
> +       struct device *dev;
> +       int index, dir;
> +
> +       udev = kzalloc(sizeof(*udev), GFP_KERNEL);
> +       if (!udev)
> +               return -ENOMEM;
> +
> +       kref_init(&udev->ref_count);
> +       mutex_init(&udev->lock);
> +       udev->mhi_dev = mhi_dev;
> +
> +       mutex_lock(&udev->lock);

Why locking here? udev has just been created, it cannot be used concurrently.

> +
> +       mutex_lock(&uci_idr_mutex);
> +       index = idr_alloc(&uci_idr, udev, 0, MAX_UCI_MINORS, GFP_KERNEL);
> +       mutex_unlock(&uci_idr_mutex);
> +       if (index < 0) {
> +               mutex_unlock(&udev->lock);
> +               kfree(udev);
> +               return index;
> +       }
> +
> +       udev->minor = index;
> +
> +       /* create device file node /dev/mhi_<cntrl_dev_name>_<mhi_dev_name> */
> +       dev = device_create(uci_dev_class, &mhi_dev->dev,
> +                           MKDEV(uci_dev_major, index), udev,
> +                           DEVICE_NAME "_%s_%s",
> +                           dev_name(mhi_cntrl->cntrl_dev), mhi_dev->name);
> +       if (IS_ERR(dev)) {
> +               mutex_lock(&uci_idr_mutex);
> +               idr_remove(&uci_idr, udev->minor);
> +               mutex_unlock(&uci_idr_mutex);
> +               mutex_unlock(&udev->lock);
> +               kfree(udev);
> +               return PTR_ERR(dev);
> +       }

I would suggest adding the device at the very end of the probe, when
everything has been initialized.

Regards,
Loic



> +
> +       for (dir = 0; dir < 2; dir++) {
> +               struct uci_chan *uchan = (dir) ?
> +                       &udev->ul_chan : &udev->dl_chan;
> +               spin_lock_init(&uchan->lock);
> +               init_waitqueue_head(&uchan->wq);
> +               INIT_LIST_HEAD(&uchan->pending);
> +       }
> +
> +       udev->mtu = min_t(size_t, id->driver_data, MHI_MAX_MTU);
> +       udev->actual_mtu = udev->mtu - sizeof(struct uci_buf);
> +       dev_set_drvdata(&mhi_dev->dev, udev);
> +       udev->enabled = true;
> +
> +       mutex_unlock(&udev->lock);
> +
> +       dev_info(&mhi_dev->dev, "probed uci dev: minor %d\n", index);
> +
> +       return 0;
> +};
> +
> +static void mhi_uci_remove(struct mhi_device *mhi_dev)
> +{
> +       struct uci_dev *udev = dev_get_drvdata(&mhi_dev->dev);
> +
> +       mutex_lock(&udev->lock);
> +
> +       /* disable the node */
> +       udev->enabled = false;
> +
> +       wake_up(&udev->dl_chan.wq);
> +       wake_up(&udev->ul_chan.wq);
> +
> +       /* delete the node to prevent new opens */
> +       device_destroy(uci_dev_class, MKDEV(uci_dev_major, udev->minor));
> +
> +       mutex_lock(&uci_idr_mutex);
> +       idr_remove(&uci_idr, udev->minor);
> +       mutex_unlock(&uci_idr_mutex);
> +
> +       mutex_unlock(&udev->lock);
> +
> +       kref_put(&udev->ref_count, mhi_uci_dev_release);
> +}
> +
> +/* .driver_data stores max mtu */
> +static const struct mhi_device_id mhi_uci_match_table[] = {
> +       { .chan = "LOOPBACK", .driver_data = 0x1000},
> +       {},
> +};
> +MODULE_DEVICE_TABLE(mhi, mhi_uci_match_table);
> +
> +static struct mhi_driver mhi_uci_driver = {
> +       .id_table = mhi_uci_match_table,
> +       .remove = mhi_uci_remove,
> +       .probe = mhi_uci_probe,
> +       .ul_xfer_cb = mhi_ul_xfer_cb,
> +       .dl_xfer_cb = mhi_dl_xfer_cb,
> +       .driver = {
> +               .name = MHI_UCI_DRIVER_NAME,
> +       },
> +};
> +
> +static int mhi_uci_init(void)
> +{
> +       int ret;
> +
> +       ret = register_chrdev(0, MHI_UCI_DRIVER_NAME, &mhidev_fops);
> +       if (ret < 0)
> +               return ret;
> +
> +       uci_dev_major = ret;
> +       uci_dev_class = class_create(THIS_MODULE, MHI_UCI_DRIVER_NAME);
> +       if (IS_ERR(uci_dev_class)) {
> +               unregister_chrdev(uci_dev_major, MHI_UCI_DRIVER_NAME);
> +               return -ENODEV;
> +       }
> +
> +       ret = mhi_driver_register(&mhi_uci_driver);
> +       if (ret) {
> +               class_destroy(uci_dev_class);
> +               unregister_chrdev(uci_dev_major, MHI_UCI_DRIVER_NAME);
> +       }
> +
> +       return ret;
> +}
> +
> +static void __exit mhi_uci_exit(void)
> +{
> +       mhi_driver_unregister(&mhi_uci_driver);
> +       class_destroy(uci_dev_class);
> +       unregister_chrdev(uci_dev_major, MHI_UCI_DRIVER_NAME);
> +}
> +
> +module_init(mhi_uci_init);
> +module_exit(mhi_uci_exit);
> +MODULE_LICENSE("GPL v2");
> +MODULE_DESCRIPTION("MHI UCI Driver");
> --
> The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
> a Linux Foundation Collaborative Project
>

Re: [PATCH v6 4/4] bus: mhi: Add userspace client interface driver

[Hemant Kumar]

Hi Loic,

On 9/22/20 4:10 AM, Loic Poulain wrote:
> Hi Hemant,
> 
> See comments inline, but globally, the locking and ref counting is
> more complicated than it should be.
> 
> On Wed, 16 Sep 2020 at 21:57, Hemant Kumar <hemantk@codeaurora.org> wrote:
>>
>> This MHI client driver allows userspace clients to transfer
>> raw data between MHI device and host using standard file operations.
>> Device file node is created with format
>>
>> /dev/mhi_<controller_name>_<mhi_device_name>
>>
>> Currently it supports LOOPBACK channel.
>>
>> Signed-off-by: Hemant Kumar <hemantk@codeaurora.org>
>> ---
>>   drivers/bus/mhi/Kconfig  |  13 +
>>   drivers/bus/mhi/Makefile |   4 +
>>   drivers/bus/mhi/uci.c    | 657 +++++++++++++++++++++++++++++++++++++++++++++++
>>   3 files changed, 674 insertions(+)
>>   create mode 100644 drivers/bus/mhi/uci.c
>>
>> diff --git a/drivers/bus/mhi/Kconfig b/drivers/bus/mhi/Kconfig
>> index 6a217ff..8aebe8b 100644
>> --- a/drivers/bus/mhi/Kconfig
>> +++ b/drivers/bus/mhi/Kconfig
>> @@ -20,3 +20,16 @@ config MHI_BUS_DEBUG
>>           Enable debugfs support for use with the MHI transport. Allows
>>           reading and/or modifying some values within the MHI controller
>>           for debug and test purposes.
>> +
>> +config MHI_UCI
>> +       tristate "MHI UCI"
>> +       depends on MHI_BUS
>> +       help
>> +        MHI based userspace client interface driver is used for transferring
>> +        raw data between host and device using standard file operations from
>> +        userspace. Open, read, write, and close operations are supported
>> +        by this driver. Please check mhi_uci_match_table for all supported
>> +        channels that are exposed to userspace.
>> +
>> +        To compile this driver as a module, choose M here: the module will be
>> +        called mhi_uci.
>> diff --git a/drivers/bus/mhi/Makefile b/drivers/bus/mhi/Makefile
>> index 19e6443..80feefb 100644
>> --- a/drivers/bus/mhi/Makefile
>> +++ b/drivers/bus/mhi/Makefile
>> @@ -1,2 +1,6 @@
>>   # core layer
>>   obj-y += core/
>> +
>> +# MHI client
>> +mhi_uci-y := uci.o
>> +obj-$(CONFIG_MHI_UCI) += mhi_uci.o
>> diff --git a/drivers/bus/mhi/uci.c b/drivers/bus/mhi/uci.c
>> new file mode 100644
>> index 0000000..d6758f2
>> --- /dev/null
>> +++ b/drivers/bus/mhi/uci.c
>> @@ -0,0 +1,657 @@
>> +// SPDX-License-Identifier: GPL-2.0-only
>> +/* Copyright (c) 2018-2020, The Linux Foundation. All rights reserved.*/
>> +
>> +#include <linux/kernel.h>
>> +#include <linux/mhi.h>
>> +#include <linux/mod_devicetable.h>
>> +#include <linux/module.h>
>> +#include <linux/poll.h>
>> +
>> +#define DEVICE_NAME "mhi"
>> +#define MHI_UCI_DRIVER_NAME "mhi_uci"
>> +#define MAX_UCI_MINORS (128)
>> +
>> +static DEFINE_IDR(uci_idr);
>> +static DEFINE_MUTEX(uci_idr_mutex);
>> +static struct class *uci_dev_class;
>> +static int uci_dev_major;
>> +
>> +/**
>> + * struct uci_chan - MHI channel for a uci device
>> + * @wq: wait queue for reader/writer
>> + * @lock: spin lock
>> + * @pending: list of rx buffers userspace is waiting to read
>> + * @cur_buf: current buffer userspace is reading
>> + * @rx_size: size of the current rx buffer userspace is reading
>> + */
>> +struct uci_chan {
>> +       wait_queue_head_t wq;
>> +
>> +       /* protects pending and cur_buf members in bh context */
>> +       spinlock_t lock;
>> +
>> +       struct list_head pending;
>> +       struct uci_buf *cur_buf;
>> +       size_t rx_size;
>> +};
>> +
>> +/**
>> + * struct uci_buf - uci buffer
>> + * @data: data buffer
>> + * @len: length of data buffer
>> + * @node: list node of the uci buffer
>> + */
>> +struct uci_buf {
>> +       void *data;
>> +       size_t len;
>> +       struct list_head node;
>> +};
>> +
>> +/**
>> + * struct uci_dev - MHI uci device
>> + * @minor: uci device node minor number
>> + * @mhi_dev: associated mhi device object
>> + * @chan: MHI channel name
>> + * @lock: mutex lock
>> + * @ul_chan: uplink uci channel object
>> + * @dl_chan: downlink uci channel object
>> + * @mtu: max tx buffer length
>> + * @actual_mtu: maximum size of incoming buffer
>> + * @open: open called for device node
>> + * @enabled: uci device probed
>> + * @ref_count: uci_dev reference count
>> + */
>> +struct uci_dev {
>> +       unsigned int minor;
>> +       struct mhi_device *mhi_dev;
>> +       const char *chan;
>> +
>> +       /* protects uci_dev struct members */
>> +       struct mutex lock;
>> +
>> +       struct uci_chan ul_chan;
>> +       struct uci_chan dl_chan;
>> +       size_t mtu;
>> +       size_t actual_mtu;
>> +       bool enabled;
>> +       struct kref ref_count;
>> +};
>> +
>> +static int mhi_queue_inbound(struct uci_dev *udev)
>> +{
>> +       struct mhi_device *mhi_dev = udev->mhi_dev;
>> +       struct device *dev = &mhi_dev->dev;
>> +       size_t mtu = udev->mtu;
>> +       size_t actual_mtu = udev->actual_mtu;
>> +       int nr_trbs, i, ret = -EIO;
>> +       void *buf;
>> +       struct uci_buf *uci_buf;
>> +
>> +       nr_trbs = mhi_get_no_free_descriptors(mhi_dev, DMA_FROM_DEVICE);
>> +
>> +       for (i = 0; i < nr_trbs; i++) {
>> +               buf = kmalloc(mtu, GFP_KERNEL);
>> +               if (!buf)
>> +                       return -ENOMEM;
>> +
>> +               uci_buf = buf + actual_mtu;
>> +               uci_buf->data = buf;
>> +
>> +               dev_dbg(dev, "Allocated buf %d of %d size %ld\n", i, nr_trbs,
>> +                       actual_mtu);
>> +
>> +               ret = mhi_queue_buf(mhi_dev, DMA_FROM_DEVICE, buf, actual_mtu,
>> +                                   MHI_EOT);
>> +               if (ret) {
>> +                       kfree(buf);
>> +                       dev_err(dev, "Failed to queue buffer %d\n", i);
>> +                       return ret;
>> +               }
>> +       }
>> +
>> +       return ret;
>> +}
>> +
>> +static void mhi_uci_dev_release(struct kref *ref)
>> +{
>> +       struct uci_dev *udev =
>> +               container_of(ref, struct uci_dev, ref_count);
>> +
>> +       mutex_destroy(&udev->lock);
>> +
>> +       dev_set_drvdata(&udev->mhi_dev->dev, NULL);
>> +
>> +       kfree(udev);
>> +}
>> +
>> +static int mhi_uci_release(struct inode *inode, struct file *file)
>> +{
>> +       struct uci_dev *udev = file->private_data;
>> +       struct uci_buf *itr, *tmp;
>> +       struct uci_chan *uchan;
>> +
>> +       if (kref_read(&udev->ref_count) > 2)
>> +               goto exit_uci_release;
>> +
>> +       if (udev->enabled)
>> +               mhi_unprepare_from_transfer(udev->mhi_dev);
>> +
>> +       /* clean inbound channel */
>> +       uchan = &udev->dl_chan;
>> +
>> +       spin_lock_bh(&uchan->lock);
>> +       list_for_each_entry_safe(itr, tmp, &uchan->pending, node) {
>> +               list_del(&itr->node);
>> +               kfree(itr->data);
>> +       }
>> +
>> +       if (uchan->cur_buf)
>> +               kfree(uchan->cur_buf->data);
>> +
>> +       uchan->cur_buf = NULL;
>> +       spin_unlock_bh(&uchan->lock);
>> +
>> +exit_uci_release:
>> +       kref_put(&udev->ref_count, mhi_uci_dev_release);
>> +
>> +       return 0;
>> +}
>> +
>> +static __poll_t mhi_uci_poll(struct file *file, poll_table *wait)
>> +{
>> +       struct uci_dev *udev = file->private_data;
>> +       struct mhi_device *mhi_dev = udev->mhi_dev;
>> +       struct device *dev = &mhi_dev->dev;
>> +       struct uci_chan *uchan;
>> +       __poll_t mask = 0;
>> +
>> +       poll_wait(file, &udev->dl_chan.wq, wait);
>> +       poll_wait(file, &udev->ul_chan.wq, wait);
>> +
>> +       if (!udev->enabled) {
>> +               mask = EPOLLERR;
>> +       } else {
>> +               uchan = &udev->dl_chan;
>> +               spin_lock_bh(&uchan->lock);
>> +               if (!list_empty(&uchan->pending) || uchan->cur_buf) {
>> +                       dev_dbg(dev, "Client can read from node\n");
>> +                       mask |= EPOLLIN | EPOLLRDNORM;
>> +               }
>> +               spin_unlock_bh(&uchan->lock);
>> +       }
>> +
>> +       if (!udev->enabled) {
>> +               mask |= EPOLLERR;
>> +       } else if (mhi_get_no_free_descriptors(mhi_dev, DMA_TO_DEVICE) > 0) {
>> +               dev_dbg(dev, "Client can write to node\n");
>> +               mask |= EPOLLOUT | EPOLLWRNORM;
>> +       }
>> +
>> +       dev_dbg(dev, "Client attempted to poll, returning mask 0x%x\n", mask);
>> +
>> +       return mask;
>> +}
>> +
>> +static ssize_t mhi_uci_write(struct file *file,
>> +                            const char __user *buf,
>> +                            size_t count,
>> +                            loff_t *offp)
>> +{
>> +       struct uci_dev *udev = file->private_data;
>> +       struct mhi_device *mhi_dev = udev->mhi_dev;
>> +       struct device *dev = &mhi_dev->dev;
>> +       struct uci_chan *uchan = &udev->ul_chan;
>> +       size_t bytes_xfered = 0;
>> +       int ret, nr_avail = 0;
>> +
>> +       if (!buf || !count)
>> +               return -EINVAL;
>> +
>> +       /* confirm channel is active */
>> +       mutex_lock(&udev->lock);
>> +       if (!udev->enabled) {
>> +               ret = -ENODEV;
>> +               goto err_mtx_unlock;
>> +       }
>> +
>> +       dev_dbg(dev, "%s: to xfer: %lu bytes\n", __func__, count);
>> +
>> +       while (count) {
>> +               size_t xfer_size;
>> +               void *kbuf;
>> +               enum mhi_flags flags;
>> +
>> +               mutex_unlock(&udev->lock);
>> +               /* wait for free descriptors */
>> +               ret = wait_event_interruptible(uchan->wq,
>> +                                              (!udev->enabled) ||
>> +                               (nr_avail = mhi_get_no_free_descriptors(mhi_dev,
>> +                                              DMA_TO_DEVICE)) > 0);
>> +
>> +               mutex_lock(&udev->lock);
> 
> All this locking unlocking is odd:
> - why do you need locking for testing wait_event return code?
> - why do you need this udev->enabled?
udev->enabled used in wait_event_interruptible to unblock in remove. I 
am going to get rid of udev->enabled check that is added on top of this 
function and also before calling mhi_queue_buf below.
> - The MHI core should be thread safe + mhi_queue_buf should simply
> fail if removing is ongoing.
> 
>> +               if (ret == -ERESTARTSYS) {
>> +                       dev_dbg(dev, "Exit signal caught for node\n");
>> +                       goto err_mtx_unlock;
>> +               }
>> +
>> +               if (!udev->enabled) {
>> +                       ret = -ENODEV;
>> +                       goto err_mtx_unlock;
>> +               }
>> +
>> +               xfer_size = min_t(size_t, count, udev->mtu);
>> +               kbuf = kmalloc(xfer_size, GFP_KERNEL);
>> +               if (!kbuf) {
>> +                       ret = -ENOMEM;
>> +                       goto err_mtx_unlock;
>> +               }
>> +
>> +               ret = copy_from_user(kbuf, buf, xfer_size);
>> +               if (ret) {
>> +                       kfree(kbuf);
>> +                       ret = -EFAULT;
>> +                       goto err_mtx_unlock;
>> +               }
>> +
>> +               /* if ring is full after this force EOT */
>> +               if (nr_avail > 1 && (count - xfer_size))
>> +                       flags = MHI_CHAIN;
>> +               else
>> +                       flags = MHI_EOT;
>> +
>> +               if (udev->enabled)
>> +                       ret = mhi_queue_buf(mhi_dev, DMA_TO_DEVICE, kbuf,
>> +                                           xfer_size, flags);
>> +               else
>> +                       ret = -ENODEV;
>> +
>> +               if (ret) {
>> +                       kfree(kbuf);
>> +                       goto err_mtx_unlock;
>> +               }
>> +
>> +               bytes_xfered += xfer_size;
>> +               count -= xfer_size;
>> +               buf += xfer_size;
>> +       }
>> +
>> +       mutex_unlock(&udev->lock);
>> +       dev_dbg(dev, "%s: bytes xferred: %lu\n", __func__, bytes_xfered);
>> +
>> +       return bytes_xfered;
>> +
>> +err_mtx_unlock:
>> +       mutex_unlock(&udev->lock);
>> +
>> +       return ret;
>> +}
>> +
>> +static ssize_t mhi_uci_read(struct file *file,
>> +                           char __user *buf,
>> +                           size_t count,
>> +                           loff_t *ppos)
>> +{
>> +       struct uci_dev *udev = file->private_data;
>> +       struct mhi_device *mhi_dev = udev->mhi_dev;
>> +       struct uci_chan *uchan = &udev->dl_chan;
>> +       struct device *dev = &mhi_dev->dev;
>> +       struct uci_buf *ubuf;
>> +       char *ptr;
>> +       size_t to_copy;
>> +       int ret = 0;
>> +
>> +       if (!buf)
>> +               return -EINVAL;
>> +
>> +       mutex_lock(&udev->lock);
>> +       /* confirm channel is active */
>> +       if (!udev->enabled) {
>> +               ret = -ENODEV;
>> +               goto err_mtx_unlock;
>> +       }
>> +
>> +       spin_lock_bh(&uchan->lock);
>> +       /* No data available to read, wait */
>> +       if (!uchan->cur_buf && list_empty(&uchan->pending)) {
>> +               dev_dbg(dev, "No data available to read waiting\n");
>> +
>> +               spin_unlock_bh(&uchan->lock);
>> +               mutex_unlock(&udev->lock);
>> +               ret = wait_event_interruptible(uchan->wq,
>> +                                              (!udev->enabled ||
>> +                                             !list_empty(&uchan->pending)));
>> +
>> +               mutex_lock(&udev->lock);
>> +               if (ret == -ERESTARTSYS) {
>> +                       dev_dbg(dev, "Exit signal caught for node\n");
>> +                       goto err_mtx_unlock;
>> +               }
>> +
>> +               if (!udev->enabled) {
>> +                       ret = -ENODEV;
>> +                       goto err_mtx_unlock;
>> +               }
>> +               spin_lock_bh(&uchan->lock);
>> +       }
>> +
>> +       /* new read, get the next descriptor from the list */
>> +       if (!uchan->cur_buf) {
>> +               ubuf = list_first_entry_or_null(&uchan->pending,
>> +                                               struct uci_buf, node);
>> +               if (!ubuf) {
>> +                       ret = -EIO;
>> +                       goto err_spin_unlock;
>> +               }
>> +
>> +               list_del(&ubuf->node);
>> +               uchan->cur_buf = ubuf;
>> +               uchan->rx_size = ubuf->len;
>> +               dev_dbg(dev, "Got pkt of size: %zu\n", uchan->rx_size);
>> +       }
>> +
>> +       ubuf = uchan->cur_buf;
>> +
>> +       /* Copy the buffer to user space */
>> +       to_copy = min_t(size_t, count, uchan->rx_size);
>> +       ptr = ubuf->data + (ubuf->len - uchan->rx_size);
>> +       spin_unlock_bh(&uchan->lock);
>> +
>> +       ret = copy_to_user(buf, ptr, to_copy);
>> +       if (ret) {
>> +               ret = -EFAULT;
>> +               goto err_mtx_unlock;
>> +       }
>> +
>> +       spin_lock_bh(&uchan->lock);
>> +
>> +       dev_dbg(dev, "Copied %lu of %lu bytes\n", to_copy, uchan->rx_size);
>> +       uchan->rx_size -= to_copy;
>> +
>> +       /* we finished with this buffer, queue it back to hardware */
>> +       if (!uchan->rx_size) {
>> +               uchan->cur_buf = NULL;
>> +
>> +               if (udev->enabled)
>> +                       ret = mhi_queue_buf(mhi_dev, DMA_FROM_DEVICE,
>> +                                           ubuf->data,
>> +                                           udev->actual_mtu, MHI_EOT);
>> +               else
>> +                       ret = -ENODEV;
>> +
>> +               if (ret) {
>> +                       dev_err(dev, "Failed to recycle element: %d\n", ret);
>> +                       kfree(ubuf->data);
>> +                       goto err_spin_unlock;
>> +               }
>> +       }
>> +       spin_unlock_bh(&uchan->lock);
>> +       mutex_unlock(&udev->lock);
>> +
>> +       dev_dbg(dev, "%s: Returning %lu bytes\n", __func__, to_copy);
>> +
>> +       return to_copy;
>> +
>> +err_spin_unlock:
>> +       spin_unlock_bh(&uchan->lock);
>> +err_mtx_unlock:
>> +       mutex_unlock(&udev->lock);
>> +       return ret;
>> +}
>> +
>> +static int mhi_uci_open(struct inode *inode, struct file *filp)
>> +{
>> +       struct uci_dev *udev = NULL;
>> +       unsigned int minor = iminor(inode);
>> +       int ret = -EIO;
>> +       struct uci_buf *buf_itr, *tmp;
>> +       struct uci_chan *dl_chan;
>> +       struct mhi_device *mhi_dev;
>> +       struct device *dev;
>> +
>> +       mutex_lock(&uci_idr_mutex);
>> +       udev = idr_find(&uci_idr, minor);
>> +       mutex_unlock(&uci_idr_mutex);
>> +       if (!udev) {
>> +               pr_err("uci dev: minor %d not found\n", minor);
>> +               ret = -ENODEV;
>> +               goto error_no_dev;
>> +       }
>> +
>> +       kref_get(&udev->ref_count);
>> +
>> +       mhi_dev = udev->mhi_dev;
>> +       dev = &mhi_dev->dev;
>> +
>> +       mutex_lock(&udev->lock);
>> +       if (kref_read(&udev->ref_count) > 2) {
>> +               dev_dbg(dev, "Node already opened\n");
>> +               goto exit_uci_open;
>> +       }
>> +
>> +       if (!udev->enabled) {
>> +               dev_info(dev, "Node exists, but is not in active state!\n");
>> +               goto error_open_chan;
>> +       }
>> +
>> +       dev_dbg(dev, "Starting channel\n");
>> +       ret = mhi_prepare_for_transfer(udev->mhi_dev);
>> +       if (ret) {
>> +               dev_err(dev, "Error starting transfer channels\n");
>> +               goto error_open_chan;
>> +       }
>> +
>> +       ret = mhi_queue_inbound(udev);
>> +       if (ret)
>> +               goto error_rx_queue;
>> +
>> +exit_uci_open:
>> +       filp->private_data = udev;
>> +       mutex_unlock(&udev->lock);
>> +
>> +       return 0;
>> +
>> +error_rx_queue:
>> +       dl_chan = &udev->dl_chan;
>> +       mhi_unprepare_from_transfer(udev->mhi_dev);
>> +       list_for_each_entry_safe(buf_itr, tmp, &dl_chan->pending, node) {
>> +               list_del(&buf_itr->node);
>> +               kfree(buf_itr->data);
>> +       }
>> +error_open_chan:
>> +       mutex_unlock(&udev->lock);
>> +       kref_put(&udev->ref_count, mhi_uci_dev_release);
>> +error_no_dev:
>> +       return ret;
>> +}
>> +
>> +static const struct file_operations mhidev_fops = {
>> +       .owner = THIS_MODULE,
>> +       .open = mhi_uci_open,
>> +       .release = mhi_uci_release,
>> +       .read = mhi_uci_read,
>> +       .write = mhi_uci_write,
>> +       .poll = mhi_uci_poll,
>> +};
>> +
>> +static void mhi_ul_xfer_cb(struct mhi_device *mhi_dev,
>> +                          struct mhi_result *mhi_result)
>> +{
>> +       struct uci_dev *udev = dev_get_drvdata(&mhi_dev->dev);
>> +       struct uci_chan *uchan = &udev->ul_chan;
>> +       struct device *dev = &mhi_dev->dev;
>> +
>> +       dev_dbg(dev, "status: %d xfer_len: %zu\n",
>> +               mhi_result->transaction_status, mhi_result->bytes_xferd);
>> +
>> +       kfree(mhi_result->buf_addr);
>> +
>> +       if (!mhi_result->transaction_status)
>> +               wake_up(&uchan->wq);
>> +}
>> +
>> +static void mhi_dl_xfer_cb(struct mhi_device *mhi_dev,
>> +                          struct mhi_result *mhi_result)
>> +{
>> +       struct uci_dev *udev = dev_get_drvdata(&mhi_dev->dev);
>> +       struct uci_chan *uchan = &udev->dl_chan;
>> +       struct device *dev = &mhi_dev->dev;
>> +       struct uci_buf *buf;
>> +
>> +       dev_dbg(dev, "status: %d receive_len: %zu\n",
>> +               mhi_result->transaction_status, mhi_result->bytes_xferd);
>> +
>> +       if (mhi_result->transaction_status == -ENOTCONN) {
>> +               kfree(mhi_result->buf_addr);
>> +               return;
>> +       }
>> +
>> +       spin_lock_bh(&uchan->lock);
>> +       buf = mhi_result->buf_addr + udev->actual_mtu;
>> +       buf->data = mhi_result->buf_addr;
>> +       buf->len = mhi_result->bytes_xferd;
> 
> You don't need to protect buf here, only uchan pending list, right? so
> move lock here.
Done.
> 
>> +       list_add_tail(&buf->node, &uchan->pending);
>> +       spin_unlock_bh(&uchan->lock);
>> +
>> +       wake_up(&uchan->wq);
>> +}
>> +
>> +static int mhi_uci_probe(struct mhi_device *mhi_dev,
>> +                        const struct mhi_device_id *id)
>> +{
>> +       struct uci_dev *udev;
>> +       struct mhi_controller *mhi_cntrl = mhi_dev->mhi_cntrl;
>> +       struct device *dev;
>> +       int index, dir;
>> +
>> +       udev = kzalloc(sizeof(*udev), GFP_KERNEL);
>> +       if (!udev)
>> +               return -ENOMEM;
>> +
>> +       kref_init(&udev->ref_count);
>> +       mutex_init(&udev->lock);
>> +       udev->mhi_dev = mhi_dev;
>> +
>> +       mutex_lock(&udev->lock);
> 
> Why locking here? udev has just been created, it cannot be used concurrently.
Done, will remove it.
> 
>> +
>> +       mutex_lock(&uci_idr_mutex);
>> +       index = idr_alloc(&uci_idr, udev, 0, MAX_UCI_MINORS, GFP_KERNEL);
>> +       mutex_unlock(&uci_idr_mutex);
>> +       if (index < 0) {
>> +               mutex_unlock(&udev->lock);
>> +               kfree(udev);
>> +               return index;
>> +       }
>> +
>> +       udev->minor = index;
>> +
>> +       /* create device file node /dev/mhi_<cntrl_dev_name>_<mhi_dev_name> */
>> +       dev = device_create(uci_dev_class, &mhi_dev->dev,
>> +                           MKDEV(uci_dev_major, index), udev,
>> +                           DEVICE_NAME "_%s_%s",
>> +                           dev_name(mhi_cntrl->cntrl_dev), mhi_dev->name);
>> +       if (IS_ERR(dev)) {
>> +               mutex_lock(&uci_idr_mutex);
>> +               idr_remove(&uci_idr, udev->minor);
>> +               mutex_unlock(&uci_idr_mutex);
>> +               mutex_unlock(&udev->lock);
>> +               kfree(udev);
>> +               return PTR_ERR(dev);
>> +       }
> 
> I would suggest adding the device at the very end of the probe, when
> everything has been initialized.
Done, will do that.
> 
> Regards,
> Loic
> 
> 
> 
>> +
>> +       for (dir = 0; dir < 2; dir++) {
>> +               struct uci_chan *uchan = (dir) ?
>> +                       &udev->ul_chan : &udev->dl_chan;
>> +               spin_lock_init(&uchan->lock);
>> +               init_waitqueue_head(&uchan->wq);
>> +               INIT_LIST_HEAD(&uchan->pending);
>> +       }
>> +
>> +       udev->mtu = min_t(size_t, id->driver_data, MHI_MAX_MTU);
>> +       udev->actual_mtu = udev->mtu - sizeof(struct uci_buf);
>> +       dev_set_drvdata(&mhi_dev->dev, udev);
>> +       udev->enabled = true;
>> +
>> +       mutex_unlock(&udev->lock);
>> +
>> +       dev_info(&mhi_dev->dev, "probed uci dev: minor %d\n", index);
>> +
>> +       return 0;
>> +};
>> +
>> +static void mhi_uci_remove(struct mhi_device *mhi_dev)
>> +{
>> +       struct uci_dev *udev = dev_get_drvdata(&mhi_dev->dev);
>> +
>> +       mutex_lock(&udev->lock);
>> +
>> +       /* disable the node */
>> +       udev->enabled = false;
>> +
>> +       wake_up(&udev->dl_chan.wq);
>> +       wake_up(&udev->ul_chan.wq);
>> +
>> +       /* delete the node to prevent new opens */
>> +       device_destroy(uci_dev_class, MKDEV(uci_dev_major, udev->minor));
>> +
>> +       mutex_lock(&uci_idr_mutex);
>> +       idr_remove(&uci_idr, udev->minor);
>> +       mutex_unlock(&uci_idr_mutex);
>> +
>> +       mutex_unlock(&udev->lock);
>> +
>> +       kref_put(&udev->ref_count, mhi_uci_dev_release);
>> +}
>> +
>> +/* .driver_data stores max mtu */
>> +static const struct mhi_device_id mhi_uci_match_table[] = {
>> +       { .chan = "LOOPBACK", .driver_data = 0x1000},
>> +       {},
>> +};
>> +MODULE_DEVICE_TABLE(mhi, mhi_uci_match_table);
>> +
>> +static struct mhi_driver mhi_uci_driver = {
>> +       .id_table = mhi_uci_match_table,
>> +       .remove = mhi_uci_remove,
>> +       .probe = mhi_uci_probe,
>> +       .ul_xfer_cb = mhi_ul_xfer_cb,
>> +       .dl_xfer_cb = mhi_dl_xfer_cb,
>> +       .driver = {
>> +               .name = MHI_UCI_DRIVER_NAME,
>> +       },
>> +};
>> +
>> +static int mhi_uci_init(void)
>> +{
>> +       int ret;
>> +
>> +       ret = register_chrdev(0, MHI_UCI_DRIVER_NAME, &mhidev_fops);
>> +       if (ret < 0)
>> +               return ret;
>> +
>> +       uci_dev_major = ret;
>> +       uci_dev_class = class_create(THIS_MODULE, MHI_UCI_DRIVER_NAME);
>> +       if (IS_ERR(uci_dev_class)) {
>> +               unregister_chrdev(uci_dev_major, MHI_UCI_DRIVER_NAME);
>> +               return -ENODEV;
>> +       }
>> +
>> +       ret = mhi_driver_register(&mhi_uci_driver);
>> +       if (ret) {
>> +               class_destroy(uci_dev_class);
>> +               unregister_chrdev(uci_dev_major, MHI_UCI_DRIVER_NAME);
>> +       }
>> +
>> +       return ret;
>> +}
>> +
>> +static void __exit mhi_uci_exit(void)
>> +{
>> +       mhi_driver_unregister(&mhi_uci_driver);
>> +       class_destroy(uci_dev_class);
>> +       unregister_chrdev(uci_dev_major, MHI_UCI_DRIVER_NAME);
>> +}
>> +
>> +module_init(mhi_uci_init);
>> +module_exit(mhi_uci_exit);
>> +MODULE_LICENSE("GPL v2");
>> +MODULE_DESCRIPTION("MHI UCI Driver");
>> --
>> The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
>> a Linux Foundation Collaborative Project
>>
Thanks for reviewing my change Loic, based on concerns from Greg and 
Jeff, i am working on pushing another patch set. This patch set also 
address locking related concern that you mentioned. Basically intent for 
adding udev ref count is to take care of race between driver remove() 
and file open(). So i am keeping udev ref count in the next patch set 
but in order to take care of corner cases i need to guard udev ref 
counting with global mutex. udev needs to ref count the channel so that 
mhi channel is started when first open is called and channel is stopped 
only when the last release is called. So we need some kind of ref 
counting for channel usage as well. Will remove the current way of doing 
channel ref counting using udev ref count value.

Thanks,
Hemant
-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

Re: [PATCH v6 0/4] user space client interface driver

[Christoph Hellwig]

"user space client interface driver" is a very generic and there is
absolutel no explanation in this cover letter.  What is this supposed
to be?