[PATCH V4 0/2] gpio: Add virtio based driver

[PATCH V4 0/2] gpio: Add virtio based driver

[Viresh Kumar]

Hello,

This adds Virtio GPIO driver based on the proposed specification [1].

The specification for basic GPIO operations is already reviewed by Linus and
Arnd, while the IRQ stuff is still under discussion and not finalized.

I am sharing the code, so everyone gets more clarity on how it will work
eventually in Linux.

I have tested this patchset with Qemu guest with help of the libgpiod utility.
I have also tested basic handling of interrupts on the guest side. It works as
expected.

The host side virtio-backend isn't ready yet and my tests only tested the flow
control between guest and host, but didn't play with real GPIO pins.  That will
be done once I have a working backend in place (WIP).

V3->V4:
- Lots of changes, as the specification changed too much. Better forget
  everything we have done until now :)

--
Viresh

[1] https://lists.oasis-open.org/archives/virtio-dev/202107/msg00232.html

Viresh Kumar (2):
  gpio: Add virtio-gpio driver
  gpio: virtio: Add IRQ support

 MAINTAINERS                      |   7 +
 drivers/gpio/Kconfig             |  10 +
 drivers/gpio/Makefile            |   1 +
 drivers/gpio/gpio-virtio.c       | 648 +++++++++++++++++++++++++++++++
 include/uapi/linux/virtio_gpio.h |  72 ++++
 5 files changed, 738 insertions(+)
 create mode 100644 drivers/gpio/gpio-virtio.c
 create mode 100644 include/uapi/linux/virtio_gpio.h

-- 
2.31.1.272.g89b43f80a514

[PATCH V4 1/2] gpio: Add virtio-gpio driver

[Viresh Kumar]

This patch adds a new driver for Virtio based GPIO devices.

This allows a guest VM running Linux to access GPIO lines provided by
the host. It supports all basic operations, except interrupts for the
GPIO lines.

Based on the initial work posted by:
"Enrico Weigelt, metux IT consult" <lkml@metux.net>.

Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org>
---
Enrico, lemme know if you want me to add your co-developed by and
signed-off-by. Didn't want to add without checking with you first.
---
 MAINTAINERS                      |   7 +
 drivers/gpio/Kconfig             |   9 +
 drivers/gpio/Makefile            |   1 +
 drivers/gpio/gpio-virtio.c       | 375 +++++++++++++++++++++++++++++++
 include/uapi/linux/virtio_gpio.h |  47 ++++
 5 files changed, 439 insertions(+)
 create mode 100644 drivers/gpio/gpio-virtio.c
 create mode 100644 include/uapi/linux/virtio_gpio.h

diff --git a/MAINTAINERS b/MAINTAINERS
index a61f4f3b78a9..f632acd7d98c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -19647,6 +19647,13 @@ F:	Documentation/filesystems/virtiofs.rst
 F:	fs/fuse/virtio_fs.c
 F:	include/uapi/linux/virtio_fs.h
 
+VIRTIO GPIO DRIVER
+M:	Enrico Weigelt, metux IT consult <info@metux.net>
+M:	Viresh Kumar <vireshk@kernel.org>
+S:	Maintained
+F:	drivers/gpio/gpio-virtio.c
+F:	include/uapi/linux/virtio_gpio.h
+
 VIRTIO GPU DRIVER
 M:	David Airlie <airlied@linux.ie>
 M:	Gerd Hoffmann <kraxel@redhat.com>
diff --git a/drivers/gpio/Kconfig b/drivers/gpio/Kconfig
index fab571016adf..e5993d6864fb 100644
--- a/drivers/gpio/Kconfig
+++ b/drivers/gpio/Kconfig
@@ -1669,6 +1669,15 @@ config GPIO_MOCKUP
 	  tools/testing/selftests/gpio/gpio-mockup.sh. Reference the usage in
 	  it.
 
+config GPIO_VIRTIO
+	tristate "VirtIO GPIO support"
+	depends on VIRTIO
+	help
+	  Say Y here to enable guest support for virtio-based GPIO controllers.
+
+	  These virtual GPIOs can be routed to real GPIOs or attached to
+	  simulators on the host (like QEMU).
+
 endmenu
 
 endif
diff --git a/drivers/gpio/Makefile b/drivers/gpio/Makefile
index 32a32659866a..e0301cfedd8d 100644
--- a/drivers/gpio/Makefile
+++ b/drivers/gpio/Makefile
@@ -165,6 +165,7 @@ obj-$(CONFIG_GPIO_UCB1400)		+= gpio-ucb1400.o
 obj-$(CONFIG_GPIO_UNIPHIER)		+= gpio-uniphier.o
 obj-$(CONFIG_GPIO_VF610)		+= gpio-vf610.o
 obj-$(CONFIG_GPIO_VIPERBOARD)		+= gpio-viperboard.o
+obj-$(CONFIG_GPIO_VIRTIO)		+= gpio-virtio.o
 obj-$(CONFIG_GPIO_VISCONTI)		+= gpio-visconti.o
 obj-$(CONFIG_GPIO_VR41XX)		+= gpio-vr41xx.o
 obj-$(CONFIG_GPIO_VX855)		+= gpio-vx855.o
diff --git a/drivers/gpio/gpio-virtio.c b/drivers/gpio/gpio-virtio.c
new file mode 100644
index 000000000000..3ed4240fe670
--- /dev/null
+++ b/drivers/gpio/gpio-virtio.c
@@ -0,0 +1,375 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * GPIO driver for virtio-based virtual GPIO controllers
+ *
+ * Copyright (C) 2021 metux IT consult
+ * Enrico Weigelt, metux IT consult <info@metux.net>
+ *
+ * Copyright (C) 2021 Linaro.
+ * Viresh Kumar <viresh.kumar@linaro.org>
+ */
+
+#include <linux/completion.h>
+#include <linux/err.h>
+#include <linux/gpio/driver.h>
+#include <linux/io.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/virtio_config.h>
+#include <uapi/linux/virtio_gpio.h>
+#include <uapi/linux/virtio_ids.h>
+
+struct virtio_gpio_line {
+	struct mutex lock; /* Protects line operation */
+	struct completion completion;
+	struct virtio_gpio_request req;
+	struct virtio_gpio_response res;
+	unsigned int rxlen;
+};
+
+struct virtio_gpio {
+	struct virtio_device *vdev;
+	struct mutex lock; /* Protects virtqueue operation */
+	struct gpio_chip gc;
+	struct virtio_gpio_config config;
+	struct virtio_gpio_line *lines;
+	struct virtqueue *request_vq;
+};
+
+static int _virtio_gpio_req(struct virtio_gpio *vgpio, u16 type, u16 gpio,
+			    u8 txvalue, u8 *rxvalue, void *response, u32 rxlen)
+{
+	struct virtio_gpio_line *line = &vgpio->lines[gpio];
+	struct virtio_gpio_request *req = &line->req;
+	struct virtio_gpio_response *res = response;
+	struct scatterlist *sgs[2], req_sg, res_sg;
+	struct device *dev = &vgpio->vdev->dev;
+	int ret;
+
+	/*
+	 * Prevent multiple concurrent requests for the same line since we have
+	 * pre-allocated request/response buffers for each GPIO line. Moreover
+	 * Linux always accesses a GPIO line sequentially, so this locking shall
+	 * always go through without any delays.
+	 */
+	mutex_lock(&line->lock);
+
+	req->type = cpu_to_le16(type);
+	req->gpio = cpu_to_le16(gpio);
+	req->value = txvalue;
+
+	sg_init_one(&req_sg, req, sizeof(*req));
+	sg_init_one(&res_sg, res, rxlen);
+	sgs[0] = &req_sg;
+	sgs[1] = &res_sg;
+
+	line->rxlen = 0;
+	reinit_completion(&line->completion);
+
+	/*
+	 * Virtqueue callers need to ensure they don't call its APIs with other
+	 * virtqueue operations at the same time.
+	 */
+	mutex_lock(&vgpio->lock);
+	ret = virtqueue_add_sgs(vgpio->request_vq, sgs, 1, 1, line, GFP_KERNEL);
+	if (ret) {
+		dev_err(dev, "failed to add request to vq\n");
+		mutex_unlock(&vgpio->lock);
+		goto out;
+	}
+
+	virtqueue_kick(vgpio->request_vq);
+	mutex_unlock(&vgpio->lock);
+
+	if (!wait_for_completion_timeout(&line->completion, HZ)) {
+		dev_err(dev, "GPIO operation timed out\n");
+		ret = -ETIMEDOUT;
+		goto out;
+	}
+
+	if (unlikely(res->status != VIRTIO_GPIO_STATUS_OK)) {
+		dev_err(dev, "GPIO request failed: %d\n", gpio);
+		ret = -EINVAL;
+		goto out;
+	}
+
+	if (unlikely(line->rxlen != rxlen)) {
+		dev_err(dev, "GPIO operation returned incorrect len (%u : %u)\n",
+			rxlen, line->rxlen);
+		ret = -EINVAL;
+		goto out;
+	}
+
+	if (rxvalue)
+		*rxvalue = res->value;
+
+out:
+	mutex_unlock(&line->lock);
+	return ret;
+}
+
+static int virtio_gpio_req(struct virtio_gpio *vgpio, u16 type, u16 gpio,
+			   u8 txvalue, u8 *rxvalue)
+{
+	struct virtio_gpio_line *line = &vgpio->lines[gpio];
+	struct virtio_gpio_response *res = &line->res;
+
+	return _virtio_gpio_req(vgpio, type, gpio, txvalue, rxvalue, res,
+				sizeof(*res));
+}
+
+static void virtio_gpio_free(struct gpio_chip *gc, unsigned int gpio)
+{
+	struct virtio_gpio *vgpio = gpiochip_get_data(gc);
+
+	virtio_gpio_req(vgpio, VIRTIO_GPIO_MSG_SET_DIRECTION, gpio,
+			VIRTIO_GPIO_DIRECTION_NONE, NULL);
+}
+
+static int virtio_gpio_get_direction(struct gpio_chip *gc, unsigned int gpio)
+{
+	struct virtio_gpio *vgpio = gpiochip_get_data(gc);
+	u8 direction;
+	int ret;
+
+	ret = virtio_gpio_req(vgpio, VIRTIO_GPIO_MSG_GET_DIRECTION, gpio, 0,
+			      &direction);
+	if (ret)
+		return ret;
+
+	switch (direction) {
+	case VIRTIO_GPIO_DIRECTION_IN:
+		return GPIO_LINE_DIRECTION_IN;
+	case VIRTIO_GPIO_DIRECTION_OUT:
+		return GPIO_LINE_DIRECTION_OUT;
+	default:
+		return -EINVAL;
+	}
+}
+
+static int virtio_gpio_direction_input(struct gpio_chip *gc, unsigned int gpio)
+{
+	struct virtio_gpio *vgpio = gpiochip_get_data(gc);
+
+	return virtio_gpio_req(vgpio, VIRTIO_GPIO_MSG_SET_DIRECTION, gpio,
+			       VIRTIO_GPIO_DIRECTION_IN, NULL);
+}
+
+static int virtio_gpio_direction_output(struct gpio_chip *gc, unsigned int gpio,
+					int value)
+{
+	struct virtio_gpio *vgpio = gpiochip_get_data(gc);
+	int ret;
+
+	ret = virtio_gpio_req(vgpio, VIRTIO_GPIO_MSG_SET_VALUE, gpio, value, NULL);
+	if (ret)
+		return ret;
+
+	return virtio_gpio_req(vgpio, VIRTIO_GPIO_MSG_SET_DIRECTION, gpio,
+			       VIRTIO_GPIO_DIRECTION_OUT, NULL);
+}
+
+static int virtio_gpio_get(struct gpio_chip *gc, unsigned int gpio)
+{
+	struct virtio_gpio *vgpio = gpiochip_get_data(gc);
+	u8 value;
+	int ret;
+
+	ret = virtio_gpio_req(vgpio, VIRTIO_GPIO_MSG_GET_VALUE, gpio, 0, &value);
+	return ret ? ret : value;
+}
+
+static void virtio_gpio_set(struct gpio_chip *gc, unsigned int gpio, int value)
+{
+	struct virtio_gpio *vgpio = gpiochip_get_data(gc);
+
+	virtio_gpio_req(vgpio, VIRTIO_GPIO_MSG_SET_VALUE, gpio, value, NULL);
+}
+
+static void virtio_gpio_request_vq(struct virtqueue *vq)
+{
+	struct virtio_gpio_line *line;
+	unsigned int len;
+
+	do {
+		line = virtqueue_get_buf(vq, &len);
+		if (!line)
+			return;
+
+		line->rxlen = len;
+		complete(&line->completion);
+	} while (1);
+}
+
+static void virtio_gpio_free_vqs(struct virtio_device *vdev)
+{
+	vdev->config->reset(vdev);
+	vdev->config->del_vqs(vdev);
+}
+
+static int virtio_gpio_alloc_vqs(struct virtio_gpio *vgpio,
+				 struct virtio_device *vdev)
+{
+	const char * const names[] = { "requestq" };
+	vq_callback_t *cbs[] = {
+		virtio_gpio_request_vq,
+	};
+	struct virtqueue *vqs[1] = { NULL };
+	int ret;
+
+	ret = virtio_find_vqs(vdev, 1, vqs, cbs, names, NULL);
+	if (ret) {
+		dev_err(&vdev->dev, "failed to find vqs: %d\n", ret);
+		return ret;
+	}
+
+	if (!vqs[0]) {
+		dev_err(&vdev->dev, "failed to find requestq vq\n");
+		return -ENODEV;
+	}
+	vgpio->request_vq = vqs[0];
+
+	return 0;
+}
+
+static const char **virtio_gpio_get_names(struct virtio_gpio *vgpio)
+{
+	struct virtio_gpio_config *config = &vgpio->config;
+	struct virtio_gpio_response_get_names *res;
+	struct device *dev = &vgpio->vdev->dev;
+	u8 *gpio_names, *str;
+	const char **names;
+	int i, ret, len;
+
+	if (!config->gpio_names_size)
+		return NULL;
+
+	len = sizeof(*res) + config->gpio_names_size;
+	res = devm_kzalloc(dev, len, GFP_KERNEL);
+	if (!res)
+		return NULL;
+	gpio_names = res->value;
+
+	ret = _virtio_gpio_req(vgpio, VIRTIO_GPIO_MSG_GET_NAMES, 0, 0, NULL,
+			       res, len);
+	if (ret) {
+		dev_err(dev, "Failed to get GPIO names: %d\n", ret);
+		return NULL;
+	}
+
+	names = devm_kcalloc(dev, config->ngpio, sizeof(names), GFP_KERNEL);
+	if (!names)
+		return NULL;
+
+	/* NULL terminate the string instead of checking it */
+	gpio_names[config->gpio_names_size - 1] = '\0';
+
+	for (i = 0, str = gpio_names; i < config->ngpio; i++) {
+		names[i] = str;
+		str += strlen(str) + 1; /* zero-length strings are allowed */
+
+		if (str > gpio_names + config->gpio_names_size) {
+			dev_err(dev, "gpio_names block is too short (%d)\n", i);
+			return NULL;
+		}
+	}
+
+	return names;
+}
+
+static int virtio_gpio_probe(struct virtio_device *vdev)
+{
+	struct virtio_gpio_config *config;
+	struct device *dev = &vdev->dev;
+	struct virtio_gpio *vgpio;
+	int ret, i;
+
+	vgpio = devm_kzalloc(dev, sizeof(*vgpio), GFP_KERNEL);
+	if (!vgpio)
+		return -ENOMEM;
+
+	config = &vgpio->config;
+
+	/* Read configuration */
+	virtio_cread_bytes(vdev, 0, config, sizeof(*config));
+	config->gpio_names_size = le32_to_cpu(config->gpio_names_size);
+	config->ngpio = le16_to_cpu(config->ngpio);
+	if (!config->ngpio) {
+		dev_err(dev, "Number of GPIOs can't be zero\n");
+		return -EINVAL;
+	}
+
+	vgpio->lines = devm_kcalloc(dev, config->ngpio, sizeof(*vgpio->lines), GFP_KERNEL);
+	if (!vgpio->lines)
+		return -ENOMEM;
+
+	for (i = 0; i < config->ngpio; i++) {
+		mutex_init(&vgpio->lines[i].lock);
+		init_completion(&vgpio->lines[i].completion);
+	}
+
+	mutex_init(&vgpio->lock);
+	vdev->priv = vgpio;
+
+	vgpio->vdev			= vdev;
+	vgpio->gc.free			= virtio_gpio_free;
+	vgpio->gc.get_direction		= virtio_gpio_get_direction;
+	vgpio->gc.direction_input	= virtio_gpio_direction_input;
+	vgpio->gc.direction_output	= virtio_gpio_direction_output;
+	vgpio->gc.get			= virtio_gpio_get;
+	vgpio->gc.set			= virtio_gpio_set;
+	vgpio->gc.ngpio			= config->ngpio;
+	vgpio->gc.base			= -1; /* Allocate base dynamically */
+	vgpio->gc.label			= dev_name(dev);
+	vgpio->gc.parent		= dev;
+	vgpio->gc.owner			= THIS_MODULE;
+	vgpio->gc.can_sleep		= true;
+
+	ret = virtio_gpio_alloc_vqs(vgpio, vdev);
+	if (ret)
+		return ret;
+
+	/* Mark the device ready to perform operations from within probe() */
+	virtio_device_ready(vdev);
+
+	vgpio->gc.names = virtio_gpio_get_names(vgpio);
+
+	ret = gpiochip_add_data(&vgpio->gc, vgpio);
+	if (ret) {
+		virtio_gpio_free_vqs(vdev);
+		dev_err(dev, "Failed to add virtio-gpio controller\n");
+	}
+
+	return ret;
+}
+
+static void virtio_gpio_remove(struct virtio_device *vdev)
+{
+	struct virtio_gpio *vgpio = vdev->priv;
+
+	gpiochip_remove(&vgpio->gc);
+	virtio_gpio_free_vqs(vdev);
+}
+
+static const struct virtio_device_id id_table[] = {
+	{ VIRTIO_ID_GPIO, VIRTIO_DEV_ANY_ID },
+	{},
+};
+MODULE_DEVICE_TABLE(virtio, id_table);
+
+static struct virtio_driver virtio_gpio_driver = {
+	.id_table		= id_table,
+	.probe			= virtio_gpio_probe,
+	.remove			= virtio_gpio_remove,
+	.driver			= {
+		.name		= KBUILD_MODNAME,
+		.owner		= THIS_MODULE,
+	},
+};
+module_virtio_driver(virtio_gpio_driver);
+
+MODULE_AUTHOR("Enrico Weigelt, metux IT consult <info@metux.net>");
+MODULE_AUTHOR("Viresh Kumar <viresh.kumar@linaro.org>");
+MODULE_DESCRIPTION("VirtIO GPIO driver");
+MODULE_LICENSE("GPL");
diff --git a/include/uapi/linux/virtio_gpio.h b/include/uapi/linux/virtio_gpio.h
new file mode 100644
index 000000000000..844574acf095
--- /dev/null
+++ b/include/uapi/linux/virtio_gpio.h
@@ -0,0 +1,47 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+
+#ifndef _LINUX_VIRTIO_GPIO_H
+#define _LINUX_VIRTIO_GPIO_H
+
+#include <linux/types.h>
+
+/* Virtio GPIO request types */
+#define VIRTIO_GPIO_MSG_GET_NAMES		0x0001
+#define VIRTIO_GPIO_MSG_GET_DIRECTION		0x0002
+#define VIRTIO_GPIO_MSG_SET_DIRECTION		0x0003
+#define VIRTIO_GPIO_MSG_GET_VALUE		0x0004
+#define VIRTIO_GPIO_MSG_SET_VALUE		0x0005
+
+/* Possible values of the status field */
+#define VIRTIO_GPIO_STATUS_OK			0x0
+#define VIRTIO_GPIO_STATUS_ERR			0x1
+
+/* Direction types */
+#define VIRTIO_GPIO_DIRECTION_NONE		0x00
+#define VIRTIO_GPIO_DIRECTION_OUT		0x01
+#define VIRTIO_GPIO_DIRECTION_IN		0x02
+
+struct virtio_gpio_config {
+	__u16 ngpio;
+	__u8 padding[2];
+	__u32 gpio_names_size;
+} __packed;
+
+/* Virtio GPIO Request / Response */
+struct virtio_gpio_request {
+	__u16 type;
+	__u16 gpio;
+	__u32 value;
+};
+
+struct virtio_gpio_response {
+	__u8 status;
+	__u8 value;
+};
+
+struct virtio_gpio_response_get_names {
+	__u8 status;
+	__u8 value[];
+};
+
+#endif /* _LINUX_VIRTIO_GPIO_H */
-- 
2.31.1.272.g89b43f80a514

[PATCH V4 2/2] gpio: virtio: Add IRQ support

[Viresh Kumar]

This patch adds IRQ support for the virtio GPIO driver. Note that this
uses the irq_bus_lock/unlock() callbacks, since those operations over
virtio may sleep. Also the notifications for the eventq are processed
using a work item to allow sleep-able operations.

Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org>
---
 drivers/gpio/Kconfig             |   1 +
 drivers/gpio/gpio-virtio.c       | 281 ++++++++++++++++++++++++++++++-
 include/uapi/linux/virtio_gpio.h |  25 +++
 3 files changed, 303 insertions(+), 4 deletions(-)

diff --git a/drivers/gpio/Kconfig b/drivers/gpio/Kconfig
index e5993d6864fb..222f4ae98a35 100644
--- a/drivers/gpio/Kconfig
+++ b/drivers/gpio/Kconfig
@@ -1672,6 +1672,7 @@ config GPIO_MOCKUP
 config GPIO_VIRTIO
 	tristate "VirtIO GPIO support"
 	depends on VIRTIO
+	select GPIOLIB_IRQCHIP
 	help
 	  Say Y here to enable guest support for virtio-based GPIO controllers.
 
diff --git a/drivers/gpio/gpio-virtio.c b/drivers/gpio/gpio-virtio.c
index 3ed4240fe670..0be132d75396 100644
--- a/drivers/gpio/gpio-virtio.c
+++ b/drivers/gpio/gpio-virtio.c
@@ -17,6 +17,7 @@
 #include <linux/module.h>
 #include <linux/mutex.h>
 #include <linux/virtio_config.h>
+#include <linux/workqueue.h>
 #include <uapi/linux/virtio_gpio.h>
 #include <uapi/linux/virtio_ids.h>
 
@@ -28,6 +29,16 @@ struct virtio_gpio_line {
 	unsigned int rxlen;
 };
 
+struct vgpio_irq_line {
+	u8 type;
+	bool masked;
+	bool update_pending;
+	bool queued;
+
+	struct virtio_gpio_irq_request ireq;
+	struct virtio_gpio_irq_response ires;
+};
+
 struct virtio_gpio {
 	struct virtio_device *vdev;
 	struct mutex lock; /* Protects virtqueue operation */
@@ -35,6 +46,12 @@ struct virtio_gpio {
 	struct virtio_gpio_config config;
 	struct virtio_gpio_line *lines;
 	struct virtqueue *request_vq;
+
+	/* fields for irq support */
+	struct virtqueue *event_vq;
+	struct mutex irq_lock; /* Protects irq operation */
+	struct work_struct work;
+	struct vgpio_irq_line *irq_lines;
 };
 
 static int _virtio_gpio_req(struct virtio_gpio *vgpio, u16 type, u16 gpio,
@@ -187,6 +204,220 @@ static void virtio_gpio_set(struct gpio_chip *gc, unsigned int gpio, int value)
 	virtio_gpio_req(vgpio, VIRTIO_GPIO_MSG_SET_VALUE, gpio, value, NULL);
 }
 
+/* Interrupt handling */
+static void virtio_gpio_irq_prepare(struct virtio_gpio *vgpio, u16 gpio)
+{
+	struct vgpio_irq_line *irq_line = &vgpio->irq_lines[gpio];
+	struct virtio_gpio_irq_request *ireq = &irq_line->ireq;
+	struct virtio_gpio_irq_response *ires = &irq_line->ires;
+	struct scatterlist *sgs[2], req_sg, res_sg;
+	int ret;
+
+	ireq->gpio = cpu_to_le16(gpio);
+	sg_init_one(&req_sg, ireq, sizeof(*ireq));
+	sg_init_one(&res_sg, ires, sizeof(*ires));
+	sgs[0] = &req_sg;
+	sgs[1] = &res_sg;
+
+	ret = virtqueue_add_sgs(vgpio->event_vq, sgs, 1, 1, irq_line, GFP_KERNEL);
+	if (ret) {
+		dev_err(&vgpio->vdev->dev, "failed to add request to eventq\n");
+		return;
+	}
+
+	WARN_ON(irq_line->queued);
+
+	irq_line->queued = true;
+	virtqueue_kick(vgpio->event_vq);
+}
+
+static void virtio_gpio_irq_eoi(struct irq_data *d)
+{
+	/*
+	 * Queue buffers, by calling virtio_gpio_irq_prepare(), from
+	 * virtio_gpio_event_vq() itself, after taking into consideration the
+	 * masking status of the interrupt.
+	 */
+}
+
+static void virtio_gpio_irq_mask(struct irq_data *d)
+{
+	struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
+	struct virtio_gpio *vgpio = gpiochip_get_data(gc);
+	struct vgpio_irq_line *irq_line = &vgpio->irq_lines[d->hwirq];
+
+	irq_line->masked = true;
+	irq_line->update_pending = true;
+}
+
+static void virtio_gpio_irq_unmask(struct irq_data *d)
+{
+	struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
+	struct virtio_gpio *vgpio = gpiochip_get_data(gc);
+	struct vgpio_irq_line *irq_line = &vgpio->irq_lines[d->hwirq];
+
+	irq_line->masked = false;
+	irq_line->update_pending = true;
+}
+
+static int virtio_gpio_irq_set_type(struct irq_data *d, unsigned int type)
+{
+	struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
+	struct virtio_gpio *vgpio = gpiochip_get_data(gc);
+	struct vgpio_irq_line *irq_line = &vgpio->irq_lines[d->hwirq];
+
+	switch (type) {
+	case IRQ_TYPE_NONE:
+		type = VIRTIO_GPIO_IRQ_TYPE_NONE;
+		break;
+	case IRQ_TYPE_EDGE_RISING:
+		type = VIRTIO_GPIO_IRQ_TYPE_EDGE_RISING;
+		break;
+	case IRQ_TYPE_EDGE_FALLING:
+		type = VIRTIO_GPIO_IRQ_TYPE_EDGE_FALLING;
+		break;
+	case IRQ_TYPE_EDGE_BOTH:
+		type = VIRTIO_GPIO_IRQ_TYPE_EDGE_BOTH;
+		break;
+	case IRQ_TYPE_LEVEL_LOW:
+		type = VIRTIO_GPIO_IRQ_TYPE_LEVEL_LOW;
+		break;
+	case IRQ_TYPE_LEVEL_HIGH:
+		type = VIRTIO_GPIO_IRQ_TYPE_LEVEL_HIGH;
+		break;
+	default:
+		dev_err(&vgpio->vdev->dev, "unsupported irq type: %u\n", type);
+		return -EINVAL;
+	}
+
+	irq_line->type = type;
+	irq_line->update_pending = true;
+
+	return 0;
+}
+
+static void update_irq_type(struct virtio_gpio *vgpio, u16 gpio, u8 type)
+{
+	virtio_gpio_req(vgpio, VIRTIO_GPIO_MSG_IRQ_TYPE, gpio, type, NULL);
+}
+
+static void virtio_gpio_irq_bus_lock(struct irq_data *d)
+{
+	struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
+	struct virtio_gpio *vgpio = gpiochip_get_data(gc);
+
+	mutex_lock(&vgpio->irq_lock);
+}
+
+static void virtio_gpio_irq_bus_sync_unlock(struct irq_data *d)
+{
+	struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
+	struct virtio_gpio *vgpio = gpiochip_get_data(gc);
+	int gpio = d->hwirq;
+	struct vgpio_irq_line *irq_line = &vgpio->irq_lines[gpio];
+
+	if (unlikely(!irq_line->update_pending))
+		goto out;
+
+	if (irq_line->masked) {
+		update_irq_type(vgpio, gpio, VIRTIO_GPIO_IRQ_TYPE_NONE);
+	} else {
+		update_irq_type(vgpio, gpio, irq_line->type);
+		virtio_gpio_irq_prepare(vgpio, gpio);
+	}
+
+	irq_line->update_pending = false;
+
+out:
+	mutex_unlock(&vgpio->irq_lock);
+}
+
+static struct irq_chip vgpio_irq_chip = {
+	.name			= "virtio-gpio",
+	.irq_eoi		= virtio_gpio_irq_eoi,
+	.irq_mask		= virtio_gpio_irq_mask,
+	.irq_unmask		= virtio_gpio_irq_unmask,
+	.irq_set_type		= virtio_gpio_irq_set_type,
+
+	/* These are required to implement irqchip for slow busses */
+	.irq_bus_lock		= virtio_gpio_irq_bus_lock,
+	.irq_bus_sync_unlock	= virtio_gpio_irq_bus_sync_unlock,
+};
+
+static void vgpio_work_handler(struct work_struct *work)
+{
+	struct virtio_gpio *vgpio = container_of(work, struct virtio_gpio,
+						 work);
+	struct device *dev = &vgpio->vdev->dev;
+	struct vgpio_irq_line *irq_line;
+	int irq, gpio, ret;
+	unsigned int len;
+
+	mutex_lock(&vgpio->irq_lock);
+
+	while (true) {
+		irq_line = virtqueue_get_buf(vgpio->event_vq, &len);
+		if (!irq_line)
+			break;
+
+		if (len != sizeof(irq_line->ires)) {
+			dev_err(dev, "irq with incorrect length (%u : %lu)\n",
+				len, sizeof(irq_line->ires));
+			continue;
+		}
+
+		WARN_ON(!irq_line->queued);
+		irq_line->queued = false;
+
+		/* Buffer is returned after interrupt is masked */
+		if (irq_line->ires.status == VIRTIO_GPIO_IRQ_STATUS_INVALID)
+			continue;
+
+		if (WARN_ON(irq_line->ires.status != VIRTIO_GPIO_IRQ_STATUS_VALID))
+			continue;
+
+		/*
+		 * Find GPIO line number from the offset of irq_line within the
+		 * irq_lines block. We can also get GPIO number from
+		 * irq-request, but better not rely on a value returned by
+		 * remote.
+		 */
+		gpio = irq_line - vgpio->irq_lines;
+		WARN_ON(gpio >= vgpio->config.ngpio);
+
+		irq = irq_find_mapping(vgpio->gc.irq.domain, gpio);
+		WARN_ON(!irq);
+
+		local_irq_disable();
+		ret = generic_handle_irq(irq);
+		local_irq_enable();
+
+		if (ret)
+			dev_err(dev, "failed to handle interrupt: %d\n", ret);
+
+		/* The interrupt may have been disabled by now */
+		if (irq_line->update_pending && irq_line->masked)
+			update_irq_type(vgpio, gpio, VIRTIO_GPIO_IRQ_TYPE_NONE);
+		else
+			virtio_gpio_irq_prepare(vgpio, gpio);
+
+		irq_line->update_pending = false;
+	};
+
+	mutex_unlock(&vgpio->irq_lock);
+}
+
+static void virtio_gpio_event_vq(struct virtqueue *vq)
+{
+	struct virtio_gpio *vgpio = vq->vdev->priv;
+
+	/*
+	 * We can't initiate virtio-gpio operations from hard irq context, as
+	 * they need sleep-able context.
+	 */
+	schedule_work(&vgpio->work);
+}
+
 static void virtio_gpio_request_vq(struct virtqueue *vq)
 {
 	struct virtio_gpio_line *line;
@@ -211,14 +442,15 @@ static void virtio_gpio_free_vqs(struct virtio_device *vdev)
 static int virtio_gpio_alloc_vqs(struct virtio_gpio *vgpio,
 				 struct virtio_device *vdev)
 {
-	const char * const names[] = { "requestq" };
+	const char * const names[] = { "requestq", "eventq" };
 	vq_callback_t *cbs[] = {
 		virtio_gpio_request_vq,
+		virtio_gpio_event_vq,
 	};
-	struct virtqueue *vqs[1] = { NULL };
+	struct virtqueue *vqs[2] = { NULL, NULL };
 	int ret;
 
-	ret = virtio_find_vqs(vdev, 1, vqs, cbs, names, NULL);
+	ret = virtio_find_vqs(vdev, vgpio->irq_lines ? 2 : 1, vqs, cbs, names, NULL);
 	if (ret) {
 		dev_err(&vdev->dev, "failed to find vqs: %d\n", ret);
 		return ret;
@@ -226,11 +458,23 @@ static int virtio_gpio_alloc_vqs(struct virtio_gpio *vgpio,
 
 	if (!vqs[0]) {
 		dev_err(&vdev->dev, "failed to find requestq vq\n");
-		return -ENODEV;
+		goto out;
 	}
 	vgpio->request_vq = vqs[0];
 
+	if (vgpio->irq_lines && !vqs[1]) {
+		dev_err(&vdev->dev, "failed to find eventq vq\n");
+		goto out;
+	}
+	vgpio->event_vq = vqs[1];
+
 	return 0;
+
+out:
+	if (vqs[0] || vqs[1])
+		virtio_gpio_free_vqs(vdev);
+
+	return -ENODEV;
 }
 
 static const char **virtio_gpio_get_names(struct virtio_gpio *vgpio)
@@ -326,6 +570,29 @@ static int virtio_gpio_probe(struct virtio_device *vdev)
 	vgpio->gc.owner			= THIS_MODULE;
 	vgpio->gc.can_sleep		= true;
 
+	/* Interrupt support */
+	if (virtio_has_feature(vdev, VIRTIO_GPIO_F_IRQ)) {
+		vgpio->irq_lines = devm_kcalloc(dev, config->ngpio,
+						sizeof(*vgpio->irq_lines),
+						GFP_KERNEL);
+		if (!vgpio->irq_lines)
+			return -ENOMEM;
+
+		/* The event comes from the outside so no parent handler */
+		vgpio->gc.irq.parent_handler	= NULL;
+		vgpio->gc.irq.num_parents	= 0;
+		vgpio->gc.irq.parents		= NULL;
+		vgpio->gc.irq.default_type	= IRQ_TYPE_NONE;
+		vgpio->gc.irq.handler		= handle_fasteoi_irq;
+		vgpio->gc.irq.chip		= &vgpio_irq_chip;
+
+		for (i = 0; i < config->ngpio; i++)
+			vgpio->irq_lines[i].type = VIRTIO_GPIO_IRQ_TYPE_NONE;
+
+		mutex_init(&vgpio->irq_lock);
+		INIT_WORK(&vgpio->work, vgpio_work_handler);
+	}
+
 	ret = virtio_gpio_alloc_vqs(vgpio, vdev);
 	if (ret)
 		return ret;
@@ -358,7 +625,13 @@ static const struct virtio_device_id id_table[] = {
 };
 MODULE_DEVICE_TABLE(virtio, id_table);
 
+static const unsigned int features[] = {
+	VIRTIO_GPIO_F_IRQ,
+};
+
 static struct virtio_driver virtio_gpio_driver = {
+	.feature_table		= features,
+	.feature_table_size	= ARRAY_SIZE(features),
 	.id_table		= id_table,
 	.probe			= virtio_gpio_probe,
 	.remove			= virtio_gpio_remove,
diff --git a/include/uapi/linux/virtio_gpio.h b/include/uapi/linux/virtio_gpio.h
index 844574acf095..297ffdae1a5d 100644
--- a/include/uapi/linux/virtio_gpio.h
+++ b/include/uapi/linux/virtio_gpio.h
@@ -5,12 +5,16 @@
 
 #include <linux/types.h>
 
+/* Virtio GPIO Feature bits */
+#define VIRTIO_GPIO_F_IRQ			0
+
 /* Virtio GPIO request types */
 #define VIRTIO_GPIO_MSG_GET_NAMES		0x0001
 #define VIRTIO_GPIO_MSG_GET_DIRECTION		0x0002
 #define VIRTIO_GPIO_MSG_SET_DIRECTION		0x0003
 #define VIRTIO_GPIO_MSG_GET_VALUE		0x0004
 #define VIRTIO_GPIO_MSG_SET_VALUE		0x0005
+#define VIRTIO_GPIO_MSG_IRQ_TYPE		0x0006
 
 /* Possible values of the status field */
 #define VIRTIO_GPIO_STATUS_OK			0x0
@@ -21,6 +25,14 @@
 #define VIRTIO_GPIO_DIRECTION_OUT		0x01
 #define VIRTIO_GPIO_DIRECTION_IN		0x02
 
+/* Virtio GPIO IRQ types */
+#define VIRTIO_GPIO_IRQ_TYPE_NONE		0x00
+#define VIRTIO_GPIO_IRQ_TYPE_EDGE_RISING	0x01
+#define VIRTIO_GPIO_IRQ_TYPE_EDGE_FALLING	0x02
+#define VIRTIO_GPIO_IRQ_TYPE_EDGE_BOTH		0x03
+#define VIRTIO_GPIO_IRQ_TYPE_LEVEL_HIGH		0x04
+#define VIRTIO_GPIO_IRQ_TYPE_LEVEL_LOW		0x08
+
 struct virtio_gpio_config {
 	__u16 ngpio;
 	__u8 padding[2];
@@ -44,4 +56,17 @@ struct virtio_gpio_response_get_names {
 	__u8 value[];
 };
 
+/* Virtio GPIO IRQ Request / Response */
+struct virtio_gpio_irq_request {
+	__u16 gpio;
+};
+
+struct virtio_gpio_irq_response {
+	__u8 status;
+};
+
+/* Possible values of the interrupt status field */
+#define VIRTIO_GPIO_IRQ_STATUS_INVALID		0x0
+#define VIRTIO_GPIO_IRQ_STATUS_VALID		0x1
+
 #endif /* _LINUX_VIRTIO_GPIO_H */
-- 
2.31.1.272.g89b43f80a514

Re: [PATCH V4 2/2] gpio: virtio: Add IRQ support

[Arnd Bergmann]

On Tue, Aug 3, 2021 at 1:36 PM Viresh Kumar <viresh.kumar@linaro.org> wrote:
>
> This patch adds IRQ support for the virtio GPIO driver. Note that this
> uses the irq_bus_lock/unlock() callbacks, since those operations over
> virtio may sleep. Also the notifications for the eventq are processed
> using a work item to allow sleep-able operations.
>
> Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org>
> ---
>  drivers/gpio/Kconfig             |   1 +
>  drivers/gpio/gpio-virtio.c       | 281 ++++++++++++++++++++++++++++++-
>  include/uapi/linux/virtio_gpio.h |  25 +++
>  3 files changed, 303 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/gpio/Kconfig b/drivers/gpio/Kconfig
> index e5993d6864fb..222f4ae98a35 100644
> --- a/drivers/gpio/Kconfig
> +++ b/drivers/gpio/Kconfig
> @@ -1672,6 +1672,7 @@ config GPIO_MOCKUP
>  config GPIO_VIRTIO
>         tristate "VirtIO GPIO support"
>         depends on VIRTIO
> +       select GPIOLIB_IRQCHIP
>         help
>           Say Y here to enable guest support for virtio-based GPIO controllers.
>

> +struct vgpio_irq_line {
> +       u8 type;
> +       bool masked;
> +       bool update_pending;
> +       bool queued;
> +
> +       struct virtio_gpio_irq_request ireq;
> +       struct virtio_gpio_irq_response ires;
> +};

I think the last two members should be marked as __cacheline_aligned,
since they are transferred separately over DMA.

> +static void virtio_gpio_irq_eoi(struct irq_data *d)
> +{
> +       /*
> +        * Queue buffers, by calling virtio_gpio_irq_prepare(), from
> +        * virtio_gpio_event_vq() itself, after taking into consideration the
> +        * masking status of the interrupt.
> +        */
> +}

Shouldn't this just requeue the interrupt? There is no reason to
defer this, and I think we want the normal operation to not have
to involve any scheduling.

> +static void virtio_gpio_irq_unmask(struct irq_data *d)
> +{
> +       struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
> +       struct virtio_gpio *vgpio = gpiochip_get_data(gc);
> +       struct vgpio_irq_line *irq_line = &vgpio->irq_lines[d->hwirq];
> +
> +       irq_line->masked = false;
> +       irq_line->update_pending = true;
> +}

Same here. unmask is probably less important, but it's the
same operation: if you want interrupts to become active
again when they are not, just queue the request

> +static void virtio_gpio_irq_mask(struct irq_data *d)
> +{
> +       struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
> +       struct virtio_gpio *vgpio = gpiochip_get_data(gc);
> +       struct vgpio_irq_line *irq_line = &vgpio->irq_lines[d->hwirq];
> +
> +       irq_line->masked = true;
> +       irq_line->update_pending = true;
> +}

This is of course the tricky bit, I was hoping you had found a solution
based on what I wrote above for eio() and unmask().

> +static void vgpio_work_handler(struct work_struct *work)
> +{
> +       struct virtio_gpio *vgpio = container_of(work, struct virtio_gpio,
> +                                                work);
> +       struct device *dev = &vgpio->vdev->dev;
> +       struct vgpio_irq_line *irq_line;
> +       int irq, gpio, ret;
> +       unsigned int len;
> +
> +       mutex_lock(&vgpio->irq_lock);
> +
> +       while (true) {
> +               irq_line = virtqueue_get_buf(vgpio->event_vq, &len);
> +               if (!irq_line)
> +                       break;

Related to above, I think all the eventq handling should be moved into the
virtio_gpio_event_vq() function directly.

> +               /* The interrupt may have been disabled by now */
> +               if (irq_line->update_pending && irq_line->masked)
> +                       update_irq_type(vgpio, gpio, VIRTIO_GPIO_IRQ_TYPE_NONE);

This is a part I'm not sure about, and I suppose it's the same part that
Marc was also confused by.

As far as I can tell, the update_irq_type() message would lead to the
interrupt getting delivered when it was armed and is now getting disabled,
but I don't see why we would call update_irq_type() as a result of the
eventq notification.

      Arnd

Re: [PATCH V4 2/2] gpio: virtio: Add IRQ support

[Viresh Kumar]

On 03-08-21, 17:01, Arnd Bergmann wrote:
> On Tue, Aug 3, 2021 at 1:36 PM Viresh Kumar <viresh.kumar@linaro.org> wrote:
> > +struct vgpio_irq_line {
> > +       u8 type;
> > +       bool masked;
> > +       bool update_pending;
> > +       bool queued;
> > +
> > +       struct virtio_gpio_irq_request ireq;
> > +       struct virtio_gpio_irq_response ires;
> > +};
> 
> I think the last two members should be marked as __cacheline_aligned,
> since they are transferred separately over DMA.

Right.

> > +static void virtio_gpio_irq_eoi(struct irq_data *d)
> > +{
> > +       /*
> > +        * Queue buffers, by calling virtio_gpio_irq_prepare(), from
> > +        * virtio_gpio_event_vq() itself, after taking into consideration the
> > +        * masking status of the interrupt.
> > +        */
> > +}
> 
> Shouldn't this just requeue the interrupt? There is no reason to
> defer this, and I think we want the normal operation to not have
> to involve any scheduling.
> 
> > +static void virtio_gpio_irq_unmask(struct irq_data *d)
> > +{
> > +       struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
> > +       struct virtio_gpio *vgpio = gpiochip_get_data(gc);
> > +       struct vgpio_irq_line *irq_line = &vgpio->irq_lines[d->hwirq];
> > +
> > +       irq_line->masked = false;
> > +       irq_line->update_pending = true;
> > +}
> 
> Same here. unmask is probably less important, but it's the
> same operation: if you want interrupts to become active
> again when they are not, just queue the request

We can't because its a slow bus ? And unmask can be called from
irq-context. That's exactly why we had the irq_bus_lock/unlock
discussion earlier.

> > +static void virtio_gpio_irq_mask(struct irq_data *d)
> > +{
> > +       struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
> > +       struct virtio_gpio *vgpio = gpiochip_get_data(gc);
> > +       struct vgpio_irq_line *irq_line = &vgpio->irq_lines[d->hwirq];
> > +
> > +       irq_line->masked = true;
> > +       irq_line->update_pending = true;
> > +}
> 
> This is of course the tricky bit, I was hoping you had found a solution
> based on what I wrote above for eio() and unmask().
> 
> > +static void vgpio_work_handler(struct work_struct *work)
> > +{
> > +       struct virtio_gpio *vgpio = container_of(work, struct virtio_gpio,
> > +                                                work);
> > +       struct device *dev = &vgpio->vdev->dev;
> > +       struct vgpio_irq_line *irq_line;
> > +       int irq, gpio, ret;
> > +       unsigned int len;
> > +
> > +       mutex_lock(&vgpio->irq_lock);
> > +
> > +       while (true) {
> > +               irq_line = virtqueue_get_buf(vgpio->event_vq, &len);
> > +               if (!irq_line)
> > +                       break;
> 
> Related to above, I think all the eventq handling should be moved into the
> virtio_gpio_event_vq() function directly.

You mean without scheduling a work ?

> > +               /* The interrupt may have been disabled by now */
> > +               if (irq_line->update_pending && irq_line->masked)
> > +                       update_irq_type(vgpio, gpio, VIRTIO_GPIO_IRQ_TYPE_NONE);
> 
> This is a part I'm not sure about, and I suppose it's the same part that
> Marc was also confused by.
> 
> As far as I can tell, the update_irq_type() message would lead to the
> interrupt getting delivered when it was armed and is now getting disabled,
> but I don't see why we would call update_irq_type() as a result of the
> eventq notification.

Lemme try to explain answer to all other question together here.

The irq related functions get called in two scenarios:

- request_irq() or irq_set_irq_type(), enable/disable_irq(), etc:

  The call sequence here is like this:

  ->irq_bus_lock()

  ->spin-lock-irqsave
  ->irq_mask()/irq_unmask()/irq_set_type()..
  ->spin-unlock-irqsave

  ->irq_bus_unlock()


  So the mask/unmask/set-type routines can't issue virtio requests and
  we need to do that from irq_bus_unlock(). This shall answer your
  question about mask/unmask, right ? Or maybe I misunderstood them
  then ?


- Interrupt: i.e. buffer sent back by the host over virtio.

  virtio_gpio_event_vq() schedules a work item, which processes the
  items from the eventq virtqueue and eventually calls
  generic_handle_irq(). The irq-core can issue calls to
  ->irq_mask/unmask() here without a prior call to
  irq_bus_lock/unlock(), normally they will balance out by the end,
  but I am not sure if it is guaranteed. Moreover, interrupt should be
  re-enabled only after unmask() is called (for ONESHOT) and not at
  EOI, right ?

  I chose not to queue the buffers back from eoi() as it is possible
  that we won't want to queue them at all, as the interrupt needs to
  be disabled by the time generic_handle_irq() returns. And so I did
  everything from the end of vgpio_work_handler()'s loop, i.e. either
  disable the interrupts with VIRTIO_GPIO_IRQ_TYPE_NONE or enable the
  interrupt again by re-queuing the buffer.

Regarding irq handling using work-item, I had to move to that to take
care of locking for re-queuing the buffers for a GPIO line from
irq-handler and bus-unlock. Nothing else seemed to work, though I am
continuing to look into that to see if there is an alternative here.

-- 
viresh

Re: [PATCH V4 2/2] gpio: virtio: Add IRQ support

[Arnd Bergmann]

On Wed, Aug 4, 2021 at 9:05 AM Viresh Kumar <viresh.kumar@linaro.org> wrote:
> On 03-08-21, 17:01, Arnd Bergmann wrote:
> > >
> > > +static void virtio_gpio_irq_unmask(struct irq_data *d)
> > > +{
> > > +       struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
> > > +       struct virtio_gpio *vgpio = gpiochip_get_data(gc);
> > > +       struct vgpio_irq_line *irq_line = &vgpio->irq_lines[d->hwirq];
> > > +
> > > +       irq_line->masked = false;
> > > +       irq_line->update_pending = true;
> > > +}
> >
> > Same here. unmask is probably less important, but it's the
> > same operation: if you want interrupts to become active
> > again when they are not, just queue the request
>
> We can't because its a slow bus ? And unmask can be called from
> irq-context. That's exactly why we had the irq_bus_lock/unlock
> discussion earlier.

I thought only 'mask' is slow, since that has to wait for the completion,
but 'unmask' just involves sending the eventq request without having
to wait for it.

> > > +static void vgpio_work_handler(struct work_struct *work)
> > > +{
> > > +       struct virtio_gpio *vgpio = container_of(work, struct virtio_gpio,
> > > +                                                work);
> > > +       struct device *dev = &vgpio->vdev->dev;
> > > +       struct vgpio_irq_line *irq_line;
> > > +       int irq, gpio, ret;
> > > +       unsigned int len;
> > > +
> > > +       mutex_lock(&vgpio->irq_lock);
> > > +
> > > +       while (true) {
> > > +               irq_line = virtqueue_get_buf(vgpio->event_vq, &len);
> > > +               if (!irq_line)
> > > +                       break;
> >
> > Related to above, I think all the eventq handling should be moved into the
> > virtio_gpio_event_vq() function directly.
>
> You mean without scheduling a work ?

Yes.

> > > +               /* The interrupt may have been disabled by now */
> > > +               if (irq_line->update_pending && irq_line->masked)
> > > +                       update_irq_type(vgpio, gpio, VIRTIO_GPIO_IRQ_TYPE_NONE);
> >
> > This is a part I'm not sure about, and I suppose it's the same part that
> > Marc was also confused by.
> >
> > As far as I can tell, the update_irq_type() message would lead to the
> > interrupt getting delivered when it was armed and is now getting disabled,
> > but I don't see why we would call update_irq_type() as a result of the
> > eventq notification.
>
> Lemme try to explain answer to all other question together here.
>
> The irq related functions get called in two scenarios:
>
> - request_irq() or irq_set_irq_type(), enable/disable_irq(), etc:
>
>   The call sequence here is like this:
>
>   ->irq_bus_lock()
>
>   ->spin-lock-irqsave
>   ->irq_mask()/irq_unmask()/irq_set_type()..
>   ->spin-unlock-irqsave
>
>   ->irq_bus_unlock()
>
>
>   So the mask/unmask/set-type routines can't issue virtio requests and
>   we need to do that from irq_bus_unlock(). This shall answer your
>   question about mask/unmask, right ? Or maybe I misunderstood them
>   then ?

I don't think it is correct that you cannot issue virtio requests from
atomic context, only that you cannot wait for the reply.

For 'unmask', there is no waiting, since the reply is the actual IRQ
event. For the others, the sequence makes sense.

> - Interrupt: i.e. buffer sent back by the host over virtio.
>
>   virtio_gpio_event_vq() schedules a work item, which processes the
>   items from the eventq virtqueue and eventually calls
>   generic_handle_irq(). The irq-core can issue calls to
>   ->irq_mask/unmask() here without a prior call to
>   irq_bus_lock/unlock(), normally they will balance out by the end,
>   but I am not sure if it is guaranteed. Moreover, interrupt should be
>   re-enabled only after unmask() is called (for ONESHOT) and not at
>   EOI, right ?
>
>   I chose not to queue the buffers back from eoi() as it is possible
>   that we won't want to queue them at all, as the interrupt needs to
>   be disabled by the time generic_handle_irq() returns. And so I did
>   everything from the end of vgpio_work_handler()'s loop, i.e. either
>   disable the interrupts with VIRTIO_GPIO_IRQ_TYPE_NONE or enable the
>   interrupt again by re-queuing the buffer.
>
> Regarding irq handling using work-item, I had to move to that to take
> care of locking for re-queuing the buffers for a GPIO line from
> irq-handler and bus-unlock. Nothing else seemed to work, though I am
> continuing to look into that to see if there is an alternative here.

I don't think it makes sense to optimize for the rare case that the
irq handler disables the irq, when that makes the common case
(irq remains unmasked and enabled) much slower.

       Arnd

Re: [PATCH V4 2/2] gpio: virtio: Add IRQ support

[Viresh Kumar]

On 04-08-21, 10:27, Arnd Bergmann wrote:
> On Wed, Aug 4, 2021 at 9:05 AM Viresh Kumar <viresh.kumar@linaro.org> wrote:
> > On 03-08-21, 17:01, Arnd Bergmann wrote:
> > > >
> > > > +static void virtio_gpio_irq_unmask(struct irq_data *d)
> > > > +{
> > > > +       struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
> > > > +       struct virtio_gpio *vgpio = gpiochip_get_data(gc);
> > > > +       struct vgpio_irq_line *irq_line = &vgpio->irq_lines[d->hwirq];
> > > > +
> > > > +       irq_line->masked = false;
> > > > +       irq_line->update_pending = true;
> > > > +}
> > >
> > > Same here. unmask is probably less important, but it's the
> > > same operation: if you want interrupts to become active
> > > again when they are not, just queue the request
> >
> > We can't because its a slow bus ? And unmask can be called from
> > irq-context. That's exactly why we had the irq_bus_lock/unlock
> > discussion earlier.
> 
> I thought only 'mask' is slow, since that has to wait for the completion,
> but 'unmask' just involves sending the eventq request without having
> to wait for it.

Ahh, so you are suggesting of just sending the request without waiting
for the response. We can do it, but I am not sure if it is going to be
worth it considering the additional complexity in the driver.

Maybe lets come back to this later, if this is going to make anything
better, once the basic support for irqs look good here.

> I don't think it is correct that you cannot issue virtio requests from
> atomic context, only that you cannot wait for the reply.

Yes, you are right. I was having a mess because of all the races and
different ways things were getting called.

> For 'unmask', there is no waiting, since the reply is the actual IRQ
> event. For the others, the sequence makes sense.
 
> > - Interrupt: i.e. buffer sent back by the host over virtio.
> >
> >   virtio_gpio_event_vq() schedules a work item, which processes the
> >   items from the eventq virtqueue and eventually calls
> >   generic_handle_irq(). The irq-core can issue calls to
> >   ->irq_mask/unmask() here without a prior call to
> >   irq_bus_lock/unlock(), normally they will balance out by the end,
> >   but I am not sure if it is guaranteed. Moreover, interrupt should be
> >   re-enabled only after unmask() is called (for ONESHOT) and not at
> >   EOI, right ?
> >
> >   I chose not to queue the buffers back from eoi() as it is possible
> >   that we won't want to queue them at all, as the interrupt needs to
> >   be disabled by the time generic_handle_irq() returns. And so I did
> >   everything from the end of vgpio_work_handler()'s loop, i.e. either
> >   disable the interrupts with VIRTIO_GPIO_IRQ_TYPE_NONE or enable the
> >   interrupt again by re-queuing the buffer.
> >
> > Regarding irq handling using work-item, I had to move to that to take
> > care of locking for re-queuing the buffers for a GPIO line from
> > irq-handler and bus-unlock. Nothing else seemed to work, though I am
> > continuing to look into that to see if there is an alternative here.
> 
> I don't think it makes sense to optimize for the rare case that the
> irq handler disables the irq, when that makes the common case
> (irq remains unmasked and enabled) much slower.

I am not worried about that case only, i.e. where the
interrupt-handler disables the interrupt. It should work fine though,
even if it isn't optimized.

A) If the client has registered with irq with IRQF_ONESHOT, irq core
   will call ->irq_mask(), irq-handler(), ->irq_eoi(), and finally
   ->irq_unmask(). Looking at this basic sequence itself, irq_eoi()
   shouldn't enable the interrupt again by re-queuing the buffer,
   rather it should happen from unmask in this case.

B) If the client has registered a threaded-irq, it gets even more
   complex. The core does this in that case:

   call ->irq_mask(), ->irq_eoi(), and return control back to
   virtio_gpio_event_vq().

   The thread gets scheduled then and calls the irq-handler and then
   the irq-core calls irq-bus-lock(), ->irq_unmask(),
   irq-bus-unlock().

Looking at these, eoi shouldn't be queuing the buffer. Also, I wonder
if we should use handle_fasteoi_irq() here or something else.

-- 
viresh

Re: [PATCH V4 2/2] gpio: virtio: Add IRQ support

[Viresh Kumar]

On 03-08-21, 17:01, Arnd Bergmann wrote:
> As far as I can tell, the update_irq_type() message would lead to the
> interrupt getting delivered when it was armed and is now getting disabled,
> but I don't see why we would call update_irq_type() as a result of the
> eventq notification.

Based on discussion we had today (offline), I changed the design a bit
and used handle_level_irq() instead, as it provides consistent calls
to mask/unmask(), which simplified the whole thing a bit.

Also I have broken the rule from specs, maybe we should update spec
with that, where the specs said that the buffer must not be queued
before enabling the interrupt. I just queue the buffer unconditionally
now from unmask().

I am not sure but there may be some race around the "queued" flag and
I wonder if we can land in a scenario where the buffer is left
un-queued somehow, while an interrupt is enabled.

diff --git a/drivers/gpio/gpio-virtio.c b/drivers/gpio/gpio-virtio.c
index 199f8ace1e88..114ce2640944 100644
--- a/drivers/gpio/gpio-virtio.c
+++ b/drivers/gpio/gpio-virtio.c
@@ -28,6 +28,17 @@ struct virtio_gpio_line {
 	unsigned int rxlen;
 };
 
+struct vgpio_irq_line {
+	u8 type;
+	bool type_pending;
+	bool masked;
+	bool mask_pending;
+	bool queued;
+
+	struct virtio_gpio_irq_request ireq ____cacheline_aligned;
+	struct virtio_gpio_irq_response ires ____cacheline_aligned;
+};
+
 struct virtio_gpio {
 	struct virtio_device *vdev;
 	struct mutex lock; /* Protects virtqueue operation */
@@ -35,6 +46,11 @@ struct virtio_gpio {
 	struct virtio_gpio_config config;
 	struct virtio_gpio_line *lines;
 	struct virtqueue *request_vq;
+
+	/* irq support */
+	struct virtqueue *event_vq;
+	struct mutex irq_lock; /* Protects irq operation */
+	struct vgpio_irq_line *irq_lines;
 };
 
 static int _virtio_gpio_req(struct virtio_gpio *vgpio, u16 type, u16 gpio,
@@ -187,6 +203,186 @@ static void virtio_gpio_set(struct gpio_chip *gc, unsigned int gpio, int value)
 	virtio_gpio_req(vgpio, VIRTIO_GPIO_MSG_SET_VALUE, gpio, value, NULL);
 }
 
+/* Interrupt handling */
+static void virtio_gpio_irq_prepare(struct virtio_gpio *vgpio, u16 gpio)
+{
+	struct vgpio_irq_line *irq_line = &vgpio->irq_lines[gpio];
+	struct virtio_gpio_irq_request *ireq = &irq_line->ireq;
+	struct virtio_gpio_irq_response *ires = &irq_line->ires;
+	struct scatterlist *sgs[2], req_sg, res_sg;
+	int ret;
+
+	if (unlikely(irq_line->queued))
+		return;
+
+	ireq->gpio = cpu_to_le16(gpio);
+	sg_init_one(&req_sg, ireq, sizeof(*ireq));
+	sg_init_one(&res_sg, ires, sizeof(*ires));
+	sgs[0] = &req_sg;
+	sgs[1] = &res_sg;
+
+	ret = virtqueue_add_sgs(vgpio->event_vq, sgs, 1, 1, irq_line, GFP_ATOMIC);
+	if (ret) {
+		dev_err(&vgpio->vdev->dev, "failed to add request to eventq\n");
+		return;
+	}
+
+	irq_line->queued = true;
+	virtqueue_kick(vgpio->event_vq);
+}
+
+static void virtio_gpio_irq_mask(struct irq_data *d)
+{
+	struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
+	struct virtio_gpio *vgpio = gpiochip_get_data(gc);
+	struct vgpio_irq_line *irq_line = &vgpio->irq_lines[d->hwirq];
+
+	if (irq_line->masked)
+		return;
+
+	irq_line->masked = true;
+	irq_line->mask_pending = !irq_line->mask_pending;
+}
+
+static void virtio_gpio_irq_unmask(struct irq_data *d)
+{
+	struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
+	struct virtio_gpio *vgpio = gpiochip_get_data(gc);
+	struct vgpio_irq_line *irq_line = &vgpio->irq_lines[d->hwirq];
+
+	/* Queue the buffer unconditionally on unmask */
+	virtio_gpio_irq_prepare(vgpio, d->hwirq);
+
+	if (!irq_line->masked)
+		return;
+
+	irq_line->masked = false;
+	irq_line->mask_pending = !irq_line->mask_pending;
+}
+
+static int virtio_gpio_irq_set_type(struct irq_data *d, unsigned int type)
+{
+	struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
+	struct virtio_gpio *vgpio = gpiochip_get_data(gc);
+	struct vgpio_irq_line *irq_line = &vgpio->irq_lines[d->hwirq];
+
+	switch (type) {
+	case IRQ_TYPE_NONE:
+		type = VIRTIO_GPIO_IRQ_TYPE_NONE;
+		break;
+	case IRQ_TYPE_EDGE_RISING:
+		type = VIRTIO_GPIO_IRQ_TYPE_EDGE_RISING;
+		break;
+	case IRQ_TYPE_EDGE_FALLING:
+		type = VIRTIO_GPIO_IRQ_TYPE_EDGE_FALLING;
+		break;
+	case IRQ_TYPE_EDGE_BOTH:
+		type = VIRTIO_GPIO_IRQ_TYPE_EDGE_BOTH;
+		break;
+	case IRQ_TYPE_LEVEL_LOW:
+		type = VIRTIO_GPIO_IRQ_TYPE_LEVEL_LOW;
+		break;
+	case IRQ_TYPE_LEVEL_HIGH:
+		type = VIRTIO_GPIO_IRQ_TYPE_LEVEL_HIGH;
+		break;
+	default:
+		dev_err(&vgpio->vdev->dev, "unsupported irq type: %u\n", type);
+		return -EINVAL;
+	}
+
+	irq_line->type = type;
+	irq_line->type_pending = true;
+
+	return 0;
+}
+
+static void update_irq_type(struct virtio_gpio *vgpio, u16 gpio, u8 type)
+{
+	virtio_gpio_req(vgpio, VIRTIO_GPIO_MSG_IRQ_TYPE, gpio, type, NULL);
+}
+
+static void virtio_gpio_irq_bus_lock(struct irq_data *d)
+{
+	struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
+	struct virtio_gpio *vgpio = gpiochip_get_data(gc);
+
+	mutex_lock(&vgpio->irq_lock);
+}
+
+static void virtio_gpio_irq_bus_sync_unlock(struct irq_data *d)
+{
+	struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
+	struct virtio_gpio *vgpio = gpiochip_get_data(gc);
+	struct vgpio_irq_line *irq_line = &vgpio->irq_lines[d->hwirq];
+	u8 type = irq_line->masked ? VIRTIO_GPIO_IRQ_TYPE_NONE : irq_line->type;
+
+	if (irq_line->mask_pending || irq_line->type_pending) {
+		irq_line->mask_pending = false;
+		irq_line->type_pending = false;
+		update_irq_type(vgpio, d->hwirq, type);
+	}
+
+	mutex_unlock(&vgpio->irq_lock);
+}
+
+static struct irq_chip vgpio_irq_chip = {
+	.name			= "virtio-gpio",
+	.irq_mask		= virtio_gpio_irq_mask,
+	.irq_unmask		= virtio_gpio_irq_unmask,
+	.irq_set_type		= virtio_gpio_irq_set_type,
+
+	/* These are required to implement irqchip for slow busses */
+	.irq_bus_lock		= virtio_gpio_irq_bus_lock,
+	.irq_bus_sync_unlock	= virtio_gpio_irq_bus_sync_unlock,
+};
+
+static void virtio_gpio_event_vq(struct virtqueue *vq)
+{
+	struct virtio_gpio *vgpio = vq->vdev->priv;
+	struct device *dev = &vgpio->vdev->dev;
+	struct vgpio_irq_line *irq_line;
+	int irq, gpio, ret;
+	unsigned int len;
+
+	while (true) {
+		irq_line = virtqueue_get_buf(vgpio->event_vq, &len);
+		if (!irq_line)
+			break;
+
+		if (len != sizeof(irq_line->ires)) {
+			dev_err(dev, "irq with incorrect length (%u : %u)\n",
+				len, (unsigned)sizeof(irq_line->ires));
+			continue;
+		}
+
+		WARN_ON(!irq_line->queued);
+		irq_line->queued = false;
+
+		/* Buffer is returned after interrupt is masked */
+		if (irq_line->ires.status == VIRTIO_GPIO_IRQ_STATUS_INVALID)
+			continue;
+
+		if (WARN_ON(irq_line->ires.status != VIRTIO_GPIO_IRQ_STATUS_VALID))
+			continue;
+
+		/*
+		 * Find GPIO line number from the offset of irq_line within the
+		 * irq_lines block. We can also get GPIO number from
+		 * irq-request, but better not rely on a value returned by
+		 * remote.
+		 */
+		gpio = irq_line - vgpio->irq_lines;
+		WARN_ON(gpio >= vgpio->config.ngpio);
+
+		irq = irq_find_mapping(vgpio->gc.irq.domain, gpio);
+		WARN_ON(!irq);
+
+		ret = generic_handle_irq(irq);
+		if (ret)
+			dev_err(dev, "failed to handle interrupt: %d\n", ret);
+	};
+}
+
 static void virtio_gpio_request_vq(struct virtqueue *vq)
 {
 	struct virtio_gpio_line *line;
@@ -211,14 +407,15 @@ static void virtio_gpio_free_vqs(struct virtio_device *vdev)
 static int virtio_gpio_alloc_vqs(struct virtio_gpio *vgpio,
 				 struct virtio_device *vdev)
 {
-	const char * const names[] = { "requestq" };
+	const char * const names[] = { "requestq", "eventq" };
 	vq_callback_t *cbs[] = {
 		virtio_gpio_request_vq,
+		virtio_gpio_event_vq,
 	};
-	struct virtqueue *vqs[1] = { NULL };
+	struct virtqueue *vqs[2] = { NULL, NULL };
 	int ret;
 
-	ret = virtio_find_vqs(vdev, 1, vqs, cbs, names, NULL);
+	ret = virtio_find_vqs(vdev, vgpio->irq_lines ? 2 : 1, vqs, cbs, names, NULL);
 	if (ret) {
 		dev_err(&vdev->dev, "failed to find vqs: %d\n", ret);
 		return ret;
@@ -226,11 +423,23 @@ static int virtio_gpio_alloc_vqs(struct virtio_gpio *vgpio,
 
 	if (!vqs[0]) {
 		dev_err(&vdev->dev, "failed to find requestq vq\n");
-		return -ENODEV;
+		goto out;
 	}
 	vgpio->request_vq = vqs[0];
 
+	if (vgpio->irq_lines && !vqs[1]) {
+		dev_err(&vdev->dev, "failed to find eventq vq\n");
+		goto out;
+	}
+	vgpio->event_vq = vqs[1];
+
 	return 0;
+
+out:
+	if (vqs[0] || vqs[1])
+		virtio_gpio_free_vqs(vdev);
+
+	return -ENODEV;
 }
 
 static const char **virtio_gpio_get_names(struct virtio_gpio *vgpio)
@@ -326,6 +535,30 @@ static int virtio_gpio_probe(struct virtio_device *vdev)
 	vgpio->gc.owner			= THIS_MODULE;
 	vgpio->gc.can_sleep		= true;
 
+	/* Interrupt support */
+	if (virtio_has_feature(vdev, VIRTIO_GPIO_F_IRQ)) {
+		vgpio->irq_lines = devm_kcalloc(dev, config->ngpio,
+						sizeof(*vgpio->irq_lines),
+						GFP_KERNEL);
+		if (!vgpio->irq_lines)
+			return -ENOMEM;
+
+		/* The event comes from the outside so no parent handler */
+		vgpio->gc.irq.parent_handler	= NULL;
+		vgpio->gc.irq.num_parents	= 0;
+		vgpio->gc.irq.parents		= NULL;
+		vgpio->gc.irq.default_type	= IRQ_TYPE_NONE;
+		vgpio->gc.irq.handler		= handle_level_irq;
+		vgpio->gc.irq.chip		= &vgpio_irq_chip;
+
+		for (i = 0; i < config->ngpio; i++) {
+			vgpio->irq_lines[i].type = VIRTIO_GPIO_IRQ_TYPE_NONE;
+			vgpio->irq_lines[i].masked = true;
+		}
+
+		mutex_init(&vgpio->irq_lock);
+	}
+
 	ret = virtio_gpio_alloc_vqs(vgpio, vdev);
 	if (ret)
 		return ret;
@@ -358,7 +591,13 @@ static const struct virtio_device_id id_table[] = {
 };
 MODULE_DEVICE_TABLE(virtio, id_table);
 
+static const unsigned int features[] = {
+	VIRTIO_GPIO_F_IRQ,
+};
+
 static struct virtio_driver virtio_gpio_driver = {
+	.feature_table		= features,
+	.feature_table_size	= ARRAY_SIZE(features),
 	.id_table		= id_table,
 	.probe			= virtio_gpio_probe,
 	.remove			= virtio_gpio_remove,

Re: [Stratos-dev] [PATCH V4 2/2] gpio: virtio: Add IRQ support

[Arnd Bergmann]

On Thu, Aug 5, 2021 at 1:26 PM Viresh Kumar via Stratos-dev
<stratos-dev@op-lists.linaro.org> wrote:
>
> On 03-08-21, 17:01, Arnd Bergmann wrote:
> > As far as I can tell, the update_irq_type() message would lead to the
> > interrupt getting delivered when it was armed and is now getting disabled,
> > but I don't see why we would call update_irq_type() as a result of the
> > eventq notification.
>
> Based on discussion we had today (offline), I changed the design a bit
> and used handle_level_irq() instead, as it provides consistent calls
> to mask/unmask(), which simplified the whole thing a bit.

The new flow looks much nicer to me, without the workqueue, and
doing the requeue directly in the unmask() operation.

I don't quite understand the purpose of the type_pending and
mask_pending flags yet, can you explain what they actually
do?

Also, I have no idea about whether using the handle_level_irq()
function is actually correct here. I suppose if necessary, the driver
could provide its own irq.handler callback in place of that.

> Also I have broken the rule from specs, maybe we should update spec
> with that, where the specs said that the buffer must not be queued
> before enabling the interrupt. I just queue the buffer unconditionally
> now from unmask().
>
> I am not sure but there may be some race around the "queued" flag and
> I wonder if we can land in a scenario where the buffer is left
> un-queued somehow, while an interrupt is enabled.

Can that be integrated with the "masked" state now? It looks like
the two flags are always opposites now.

      Arnd

Re: [Stratos-dev] [PATCH V4 2/2] gpio: virtio: Add IRQ support

[Viresh Kumar]

On 05-08-21, 14:03, Arnd Bergmann wrote:
> On Thu, Aug 5, 2021 at 1:26 PM Viresh Kumar via Stratos-dev
> > Based on discussion we had today (offline), I changed the design a bit
> > and used handle_level_irq() instead, as it provides consistent calls
> > to mask/unmask(), which simplified the whole thing a bit.
> 
> The new flow looks much nicer to me, without the workqueue, and
> doing the requeue directly in the unmask() operation.
> 
> I don't quite understand the purpose of the type_pending and
> mask_pending flags yet, can you explain what they actually
> do?

They are required to make sure we don't send unnecessary
VIRTIO_GPIO_MSG_IRQ_TYPE events to the device, every time bus_unlock()
is called.

mask_pending tracks if the masked state has changed since the time
last bus_unlock() was called. So on an interrupt, both mask() and
unmask() will get called by the irq-core now and mask_pending will
change to true (in mask()} and then false (in unmask()). And
eventually in bus_unlock() we won't send an extra
VIRTIO_GPIO_MSG_IRQ_TYPE message.

> Also, I have no idea about whether using the handle_level_irq()
> function is actually correct here. I suppose if necessary, the driver
> could provide its own irq.handler callback in place of that.

After looking at internals of these, I felt handle_level_irq() suits
much better in our case as we need to queue the buffer only at
unmask(). With handle_fasteoi_irq(), we would be required to do the
same from multiple places, unmask(), eoi().

> > Also I have broken the rule from specs, maybe we should update spec
> > with that, where the specs said that the buffer must not be queued
> > before enabling the interrupt. I just queue the buffer unconditionally
> > now from unmask().
> >
> > I am not sure but there may be some race around the "queued" flag and
> > I wonder if we can land in a scenario where the buffer is left
> > un-queued somehow, while an interrupt is enabled.
> 
> Can that be integrated with the "masked" state now? It looks like
> the two flags are always opposites now.

Yeah, but then there can be races and keeping them separate is a
better thing IMO.

I was thinking of something on these lines, disable_irq() followed by
enable_irq():

CPU0                                                 CPU1

disable_irq()
 -> irq_bus_lock()
 -> irq_mask()
 -> irq_bus_sync_unlock()
   -> sends blocking VIRTIO_GPIO_MSG_IRQ_TYPE
      to disable interrupt
                                                     Backend (at host) disables irq and 
                                                     returns the unused buffer.

enable_irq()
 -> irq_bus_lock()
 -> irq_unmask()
   -> Tries to queues buffer again
      Finds it already queued and returns.
                                                     - virtio_gpio_event_vq() runs at guest
                                                     - Finds VIRTIO_GPIO_IRQ_STATUS_INVALID in status
                                                     - returns without doing anything
 -> irq_bus_sync_unlock()
   -> sends blocking VIRTIO_GPIO_MSG_IRQ_TYPE
      to enable interrupt



So the irq is still enabled and the buffer isn't queued. Yes, need
some locking here for sure, confirmed :)

-- 
viresh

Re: [Stratos-dev] [PATCH V4 2/2] gpio: virtio: Add IRQ support

[Arnd Bergmann]

On Thu, Aug 5, 2021 at 2:49 PM Viresh Kumar <viresh.kumar@linaro.org> wrote:
>
> On 05-08-21, 14:03, Arnd Bergmann wrote:
> > On Thu, Aug 5, 2021 at 1:26 PM Viresh Kumar via Stratos-dev
> > > Based on discussion we had today (offline), I changed the design a bit
> > > and used handle_level_irq() instead, as it provides consistent calls
> > > to mask/unmask(), which simplified the whole thing a bit.
> >
> > The new flow looks much nicer to me, without the workqueue, and
> > doing the requeue directly in the unmask() operation.
> >
> > I don't quite understand the purpose of the type_pending and
> > mask_pending flags yet, can you explain what they actually
> > do?
>
> They are required to make sure we don't send unnecessary
> VIRTIO_GPIO_MSG_IRQ_TYPE events to the device, every time bus_unlock()
> is called.
>
> mask_pending tracks if the masked state has changed since the time
> last bus_unlock() was called. So on an interrupt, both mask() and
> unmask() will get called by the irq-core now and mask_pending will
> change to true (in mask()} and then false (in unmask()). And
> eventually in bus_unlock() we won't send an extra
> VIRTIO_GPIO_MSG_IRQ_TYPE message.

I hope this can still be simplified by working out better which state
transitions are needed exactly. In particular, I would expect that we
can get away with not sending a VIRTIO_GPIO_MSG_IRQ_TYPE
for 'mask' state changes at all, but use that only for forcing 'enabled'
state changes.

One part that I think is missing though is remembering the case
when an eventq message came in after an interrupt got masked
when the message was already armed. In this case, the
virtio_gpio_event_vq() function would not call the irq handler,
but the subsequent "unmask" callback would need to arrange
having it called.

        Arnd

Re: [Stratos-dev] [PATCH V4 2/2] gpio: virtio: Add IRQ support

[Viresh Kumar]

On 05-08-21, 15:10, Arnd Bergmann wrote:
> I hope this can still be simplified by working out better which state
> transitions are needed exactly. In particular, I would expect that we
> can get away with not sending a VIRTIO_GPIO_MSG_IRQ_TYPE
> for 'mask' state changes at all, but use that only for forcing 'enabled'
> state changes.

Something like this ?

struct vgpio_irq_line {
	u8 type;
	bool masked;
	bool update_pending;
	bool queued;

	struct virtio_gpio_irq_request ireq ____cacheline_aligned;
	struct virtio_gpio_irq_response ires ____cacheline_aligned;
};

static void virtio_gpio_irq_disable(struct irq_data *d)
{
	struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
	struct virtio_gpio *vgpio = gpiochip_get_data(gc);
	struct vgpio_irq_line *irq_line = &vgpio->irq_lines[d->hwirq];

	irq_line->masked = true;
	irq_line->update_pending = true;
}

static void virtio_gpio_irq_enable(struct irq_data *d)
{
	struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
	struct virtio_gpio *vgpio = gpiochip_get_data(gc);
	struct vgpio_irq_line *irq_line = &vgpio->irq_lines[d->hwirq];

	irq_line->masked = false;
	irq_line->update_pending = true;

	/* Queue the buffer unconditionally on enable */
	virtio_gpio_irq_prepare(vgpio, d->hwirq);
}

static void virtio_gpio_irq_mask(struct irq_data *d)
{
	/* Nothing to do here */
}

static void virtio_gpio_irq_unmask(struct irq_data *d)
{
	struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
	struct virtio_gpio *vgpio = gpiochip_get_data(gc);

	/* Queue the buffer unconditionally on unmask */
	virtio_gpio_irq_prepare(vgpio, d->hwirq);
}

static int virtio_gpio_irq_set_type(struct irq_data *d, unsigned int type)
{
	struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
	struct virtio_gpio *vgpio = gpiochip_get_data(gc);
	struct vgpio_irq_line *irq_line = &vgpio->irq_lines[d->hwirq];

	switch (type) {
	case IRQ_TYPE_NONE:
		type = VIRTIO_GPIO_IRQ_TYPE_NONE;
		break;
	case IRQ_TYPE_EDGE_RISING:
		type = VIRTIO_GPIO_IRQ_TYPE_EDGE_RISING;
		break;
	case IRQ_TYPE_EDGE_FALLING:
		type = VIRTIO_GPIO_IRQ_TYPE_EDGE_FALLING;
		break;
	case IRQ_TYPE_EDGE_BOTH:
		type = VIRTIO_GPIO_IRQ_TYPE_EDGE_BOTH;
		break;
	case IRQ_TYPE_LEVEL_LOW:
		type = VIRTIO_GPIO_IRQ_TYPE_LEVEL_LOW;
		break;
	case IRQ_TYPE_LEVEL_HIGH:
		type = VIRTIO_GPIO_IRQ_TYPE_LEVEL_HIGH;
		break;
	default:
		dev_err(&vgpio->vdev->dev, "unsupported irq type: %u\n", type);
		return -EINVAL;
	}

	irq_line->type = type;
	irq_line->update_pending = true;

	return 0;
}

static void update_irq_type(struct virtio_gpio *vgpio, u16 gpio, u8 type)
{
	virtio_gpio_req(vgpio, VIRTIO_GPIO_MSG_IRQ_TYPE, gpio, type, NULL);
}

static void virtio_gpio_irq_bus_lock(struct irq_data *d)
{
	struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
	struct virtio_gpio *vgpio = gpiochip_get_data(gc);

	mutex_lock(&vgpio->irq_lock);
}

static void virtio_gpio_irq_bus_sync_unlock(struct irq_data *d)
{
	struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
	struct virtio_gpio *vgpio = gpiochip_get_data(gc);
	struct vgpio_irq_line *irq_line = &vgpio->irq_lines[d->hwirq];
	u8 type = irq_line->masked ? VIRTIO_GPIO_IRQ_TYPE_NONE : irq_line->type;

	if (irq_line->update_pending) {
		irq_line->update_pending = false;
		update_irq_type(vgpio, d->hwirq, type);
	}

	mutex_unlock(&vgpio->irq_lock);
}

static struct irq_chip vgpio_irq_chip = {
	.name			= "virtio-gpio",
	.irq_enable		= virtio_gpio_irq_enable,
	.irq_disable		= virtio_gpio_irq_disable,
	.irq_mask		= virtio_gpio_irq_mask,
	.irq_unmask		= virtio_gpio_irq_unmask,
	.irq_set_type		= virtio_gpio_irq_set_type,

	/* These are required to implement irqchip for slow busses */
	.irq_bus_lock		= virtio_gpio_irq_bus_lock,
	.irq_bus_sync_unlock	= virtio_gpio_irq_bus_sync_unlock,
};

> One part that I think is missing though is remembering the case
> when an eventq message came in after an interrupt got masked
> when the message was already armed. In this case, the
> virtio_gpio_event_vq() function would not call the irq handler,
> but the subsequent "unmask" callback would need to arrange
> having it called.

I will come back to this.

-- 
viresh

Re: [Stratos-dev] [PATCH V4 2/2] gpio: virtio: Add IRQ support

[Arnd Bergmann]

On Fri, Aug 6, 2021 at 9:44 AM Viresh Kumar via Stratos-dev
<stratos-dev@op-lists.linaro.org> wrote:
>
> On 05-08-21, 15:10, Arnd Bergmann wrote:
> > I hope this can still be simplified by working out better which state
> > transitions are needed exactly. In particular, I would expect that we
> > can get away with not sending a VIRTIO_GPIO_MSG_IRQ_TYPE
> > for 'mask' state changes at all, but use that only for forcing 'enabled'
> > state changes.
>
> Something like this ?

> static void virtio_gpio_irq_mask(struct irq_data *d)
> {
>         /* Nothing to do here */
> }

You'd have to do /something/ here I think, if only setting the flag
that we don't want to deliver the next interrupt.

> static void virtio_gpio_irq_unmask(struct irq_data *d)
> {
>         struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
>         struct virtio_gpio *vgpio = gpiochip_get_data(gc);
>
>         /* Queue the buffer unconditionally on unmask */
>         virtio_gpio_irq_prepare(vgpio, d->hwirq);
> }

And check the flag here to not requeue it if it's masked.

Now, there is already a flag in the irq descriptor, so rather than
having double accounting, the easy way may be to
just use irqd_irq_masked()/irq_state_set_masked(), or
have the irq core take care of this.

     Arnd

Re: [Stratos-dev] [PATCH V4 2/2] gpio: virtio: Add IRQ support

[Viresh Kumar]

On 06-08-21, 10:00, Arnd Bergmann wrote:
> On Fri, Aug 6, 2021 at 9:44 AM Viresh Kumar via Stratos-dev
> <stratos-dev@op-lists.linaro.org> wrote:
> >
> > On 05-08-21, 15:10, Arnd Bergmann wrote:
> > > I hope this can still be simplified by working out better which state
> > > transitions are needed exactly. In particular, I would expect that we
> > > can get away with not sending a VIRTIO_GPIO_MSG_IRQ_TYPE
> > > for 'mask' state changes at all, but use that only for forcing 'enabled'
> > > state changes.
> >
> > Something like this ?
> 
> > static void virtio_gpio_irq_mask(struct irq_data *d)
> > {
> >         /* Nothing to do here */
> > }
> 
> You'd have to do /something/ here I think, if only setting the flag
> that we don't want to deliver the next interrupt.
> 
> > static void virtio_gpio_irq_unmask(struct irq_data *d)
> > {
> >         struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
> >         struct virtio_gpio *vgpio = gpiochip_get_data(gc);
> >
> >         /* Queue the buffer unconditionally on unmask */
> >         virtio_gpio_irq_prepare(vgpio, d->hwirq);
> > }
> 
> And check the flag here to not requeue it if it's masked.

I am not sure I understand why this would be required. If the
interrupt is getting disabled, then unmask will not get called here
and so we won't requeue anything. Same will happen with threaded
handlers where the interrupt gets unmasked at a later point of time.

> Now, there is already a flag in the irq descriptor, so rather than
> having double accounting, the easy way may be to
> just use irqd_irq_masked()/irq_state_set_masked(), or
> have the irq core take care of this.

-- 
viresh

Re: [Stratos-dev] [PATCH V4 2/2] gpio: virtio: Add IRQ support

[Arnd Bergmann]

On Mon, Aug 9, 2021 at 9:30 AM Viresh Kumar <viresh.kumar@linaro.org> wrote:
>
> On 06-08-21, 10:00, Arnd Bergmann wrote:
> > On Fri, Aug 6, 2021 at 9:44 AM Viresh Kumar via Stratos-dev
> > <stratos-dev@op-lists.linaro.org> wrote:
> > >
> > > On 05-08-21, 15:10, Arnd Bergmann wrote:
> > > > I hope this can still be simplified by working out better which state
> > > > transitions are needed exactly. In particular, I would expect that we
> > > > can get away with not sending a VIRTIO_GPIO_MSG_IRQ_TYPE
> > > > for 'mask' state changes at all, but use that only for forcing 'enabled'
> > > > state changes.
> > >
> > > Something like this ?
> >
> > > static void virtio_gpio_irq_mask(struct irq_data *d)
> > > {
> > >         /* Nothing to do here */
> > > }
> >
> > You'd have to do /something/ here I think, if only setting the flag
> > that we don't want to deliver the next interrupt.
> >
> > > static void virtio_gpio_irq_unmask(struct irq_data *d)
> > > {
> > >         struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
> > >         struct virtio_gpio *vgpio = gpiochip_get_data(gc);
> > >
> > >         /* Queue the buffer unconditionally on unmask */
> > >         virtio_gpio_irq_prepare(vgpio, d->hwirq);
> > > }
> >
> > And check the flag here to not requeue it if it's masked.
>
> I am not sure I understand why this would be required. If the
> interrupt is getting disabled, then unmask will not get called here
> and so we won't requeue anything. Same will happen with threaded
> handlers where the interrupt gets unmasked at a later point of time.

Ah, right. There is already a flag that gets checked by the caller.

It does feel odd to have an empty 'irq_mask' callback though, so
maybe there is still something missing, just not what I thought.

It's probably the result of calling handle_level_irq(), which as you
said is closer to what we want, but is not exactly what we need for
this protocol.

        Arnd

Re: [Stratos-dev] [PATCH V4 2/2] gpio: virtio: Add IRQ support

[Viresh Kumar]

On 09-08-21, 09:55, Arnd Bergmann wrote:
> Ah, right. There is already a flag that gets checked by the caller.
> 
> It does feel odd to have an empty 'irq_mask' callback though, so
> maybe there is still something missing, just not what I thought.
> 
> It's probably the result of calling handle_level_irq(), which as you
> said is closer to what we want, but is not exactly what we need for
> this protocol.

Okay, I have tried to take care of locking as well now and used local
flags only to make sure I can depend on them to get the locking
working properly. Lets see what's broken in this now :)

-- 
viresh

-------------------------8<-------------------------

diff --git a/drivers/gpio/gpio-virtio.c b/drivers/gpio/gpio-virtio.c
index 199f8ace1e88..48dc469cfc64 100644
--- a/drivers/gpio/gpio-virtio.c
+++ b/drivers/gpio/gpio-virtio.c
@@ -16,6 +16,7 @@
 #include <linux/kernel.h>
 #include <linux/module.h>
 #include <linux/mutex.h>
+#include <linux/spinlock.h>
 #include <linux/virtio_config.h>
 #include <uapi/linux/virtio_gpio.h>
 #include <uapi/linux/virtio_ids.h>
@@ -28,6 +29,17 @@ struct virtio_gpio_line {
 	unsigned int rxlen;
 };
 
+struct vgpio_irq_line {
+	u8 type;
+	bool disabled;
+	bool update_pending;
+	bool masked;
+	bool queued;
+
+	struct virtio_gpio_irq_request ireq ____cacheline_aligned;
+	struct virtio_gpio_irq_response ires ____cacheline_aligned;
+};
+
 struct virtio_gpio {
 	struct virtio_device *vdev;
 	struct mutex lock; /* Protects virtqueue operation */
@@ -35,6 +47,12 @@ struct virtio_gpio {
 	struct virtio_gpio_config config;
 	struct virtio_gpio_line *lines;
 	struct virtqueue *request_vq;
+
+	/* irq support */
+	struct virtqueue *event_vq;
+	struct mutex irq_lock; /* Protects irq operation */
+	spinlock_t eventq_lock; /* Protects queuing of the buffer */
+	struct vgpio_irq_line *irq_lines;
 };
 
 static int _virtio_gpio_req(struct virtio_gpio *vgpio, u16 type, u16 gpio,
@@ -187,6 +205,240 @@ static void virtio_gpio_set(struct gpio_chip *gc, unsigned int gpio, int value)
 	virtio_gpio_req(vgpio, VIRTIO_GPIO_MSG_SET_VALUE, gpio, value, NULL);
 }
 
+/* Interrupt handling */
+static void virtio_gpio_irq_prepare(struct virtio_gpio *vgpio, u16 gpio)
+{
+	struct vgpio_irq_line *irq_line = &vgpio->irq_lines[gpio];
+	struct virtio_gpio_irq_request *ireq = &irq_line->ireq;
+	struct virtio_gpio_irq_response *ires = &irq_line->ires;
+	struct scatterlist *sgs[2], req_sg, res_sg;
+	int ret;
+
+	if (WARN_ON(irq_line->queued || irq_line->masked || irq_line->disabled))
+		return;
+
+	ireq->gpio = cpu_to_le16(gpio);
+	sg_init_one(&req_sg, ireq, sizeof(*ireq));
+	sg_init_one(&res_sg, ires, sizeof(*ires));
+	sgs[0] = &req_sg;
+	sgs[1] = &res_sg;
+
+	ret = virtqueue_add_sgs(vgpio->event_vq, sgs, 1, 1, irq_line, GFP_ATOMIC);
+	if (ret) {
+		dev_err(&vgpio->vdev->dev, "failed to add request to eventq\n");
+		return;
+	}
+
+	irq_line->queued = true;
+	virtqueue_kick(vgpio->event_vq);
+}
+
+static void virtio_gpio_irq_enable(struct irq_data *d)
+{
+	struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
+	struct virtio_gpio *vgpio = gpiochip_get_data(gc);
+	struct vgpio_irq_line *irq_line = &vgpio->irq_lines[d->hwirq];
+
+	spin_lock(&vgpio->eventq_lock);
+	irq_line->disabled = false;
+	irq_line->masked = false;
+
+	/* Queue the buffer unconditionally on enable */
+	virtio_gpio_irq_prepare(vgpio, d->hwirq);
+	spin_unlock(&vgpio->eventq_lock);
+
+	irq_line->update_pending = true;
+}
+
+static void virtio_gpio_irq_disable(struct irq_data *d)
+{
+	struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
+	struct virtio_gpio *vgpio = gpiochip_get_data(gc);
+	struct vgpio_irq_line *irq_line = &vgpio->irq_lines[d->hwirq];
+
+	spin_lock(&vgpio->eventq_lock);
+	irq_line->disabled = true;
+	irq_line->masked = true;
+	spin_unlock(&vgpio->eventq_lock);
+
+	irq_line->update_pending = true;
+}
+
+static void virtio_gpio_irq_mask(struct irq_data *d)
+{
+	struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
+	struct virtio_gpio *vgpio = gpiochip_get_data(gc);
+	struct vgpio_irq_line *irq_line = &vgpio->irq_lines[d->hwirq];
+
+	spin_lock(&vgpio->eventq_lock);
+	irq_line->masked = true;
+	spin_unlock(&vgpio->eventq_lock);
+}
+
+static void virtio_gpio_irq_unmask(struct irq_data *d)
+{
+	struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
+	struct virtio_gpio *vgpio = gpiochip_get_data(gc);
+	struct vgpio_irq_line *irq_line = &vgpio->irq_lines[d->hwirq];
+
+	spin_lock(&vgpio->eventq_lock);
+	irq_line->masked = false;
+
+	/* Queue the buffer unconditionally on unmask */
+	virtio_gpio_irq_prepare(vgpio, d->hwirq);
+	spin_unlock(&vgpio->eventq_lock);
+}
+
+static int virtio_gpio_irq_set_type(struct irq_data *d, unsigned int type)
+{
+	struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
+	struct virtio_gpio *vgpio = gpiochip_get_data(gc);
+	struct vgpio_irq_line *irq_line = &vgpio->irq_lines[d->hwirq];
+
+	switch (type) {
+	case IRQ_TYPE_NONE:
+		type = VIRTIO_GPIO_IRQ_TYPE_NONE;
+		break;
+	case IRQ_TYPE_EDGE_RISING:
+		type = VIRTIO_GPIO_IRQ_TYPE_EDGE_RISING;
+		break;
+	case IRQ_TYPE_EDGE_FALLING:
+		type = VIRTIO_GPIO_IRQ_TYPE_EDGE_FALLING;
+		break;
+	case IRQ_TYPE_EDGE_BOTH:
+		type = VIRTIO_GPIO_IRQ_TYPE_EDGE_BOTH;
+		break;
+	case IRQ_TYPE_LEVEL_LOW:
+		type = VIRTIO_GPIO_IRQ_TYPE_LEVEL_LOW;
+		break;
+	case IRQ_TYPE_LEVEL_HIGH:
+		type = VIRTIO_GPIO_IRQ_TYPE_LEVEL_HIGH;
+		break;
+	default:
+		dev_err(&vgpio->vdev->dev, "unsupported irq type: %u\n", type);
+		return -EINVAL;
+	}
+
+	irq_line->type = type;
+	irq_line->update_pending = true;
+
+	return 0;
+}
+
+static void update_irq_type(struct virtio_gpio *vgpio, u16 gpio, u8 type)
+{
+	virtio_gpio_req(vgpio, VIRTIO_GPIO_MSG_IRQ_TYPE, gpio, type, NULL);
+}
+
+static void virtio_gpio_irq_bus_lock(struct irq_data *d)
+{
+	struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
+	struct virtio_gpio *vgpio = gpiochip_get_data(gc);
+
+	mutex_lock(&vgpio->irq_lock);
+}
+
+static void virtio_gpio_irq_bus_sync_unlock(struct irq_data *d)
+{
+	struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
+	struct virtio_gpio *vgpio = gpiochip_get_data(gc);
+	struct vgpio_irq_line *irq_line = &vgpio->irq_lines[d->hwirq];
+	u8 type = irq_line->disabled ? VIRTIO_GPIO_IRQ_TYPE_NONE : irq_line->type;
+
+	if (irq_line->update_pending) {
+		irq_line->update_pending = false;
+		update_irq_type(vgpio, d->hwirq, type);
+	}
+
+	mutex_unlock(&vgpio->irq_lock);
+}
+
+static struct irq_chip vgpio_irq_chip = {
+	.name			= "virtio-gpio",
+	.irq_enable		= virtio_gpio_irq_enable,
+	.irq_disable		= virtio_gpio_irq_disable,
+	.irq_mask		= virtio_gpio_irq_mask,
+	.irq_unmask		= virtio_gpio_irq_unmask,
+	.irq_set_type		= virtio_gpio_irq_set_type,
+
+	/* These are required to implement irqchip for slow busses */
+	.irq_bus_lock		= virtio_gpio_irq_bus_lock,
+	.irq_bus_sync_unlock	= virtio_gpio_irq_bus_sync_unlock,
+};
+
+static bool skip_irq_event(struct virtio_gpio *vgpio, int gpio,
+			   struct vgpio_irq_line *irq_line)
+{
+	bool skip = false;
+
+	spin_lock(&vgpio->eventq_lock);
+	irq_line->queued = false;
+
+	/* Interrupt is disabled currently */
+	if (irq_line->masked || irq_line->disabled) {
+		skip = true;
+		goto unlock;
+	}
+
+	/*
+	 * Buffer is returned after the interrupt is masked. The interrupt is
+	 * already enabled again now, requeue the buffers.
+	 */
+	if (irq_line->ires.status == VIRTIO_GPIO_IRQ_STATUS_INVALID) {
+		virtio_gpio_irq_prepare(vgpio, gpio);
+		skip = true;
+		goto unlock;
+	}
+
+	if (WARN_ON(irq_line->ires.status != VIRTIO_GPIO_IRQ_STATUS_VALID))
+		skip = true;
+
+unlock:
+	spin_unlock(&vgpio->eventq_lock);
+
+	return skip;
+}
+
+static void virtio_gpio_event_vq(struct virtqueue *vq)
+{
+	struct virtio_gpio *vgpio = vq->vdev->priv;
+	struct device *dev = &vgpio->vdev->dev;
+	struct vgpio_irq_line *irq_line;
+	int irq, gpio, ret;
+	unsigned int len;
+
+	while (true) {
+		irq_line = virtqueue_get_buf(vgpio->event_vq, &len);
+		if (!irq_line)
+			break;
+
+		if (len != sizeof(irq_line->ires)) {
+			dev_err(dev, "irq with incorrect length (%u : %u)\n",
+				len, (unsigned)sizeof(irq_line->ires));
+			continue;
+		}
+
+		/*
+		 * Find GPIO line number from the offset of irq_line within the
+		 * irq_lines block. We can also get GPIO number from
+		 * irq-request, but better not rely on a value returned by
+		 * remote.
+		 */
+		gpio = irq_line - vgpio->irq_lines;
+		WARN_ON(gpio >= vgpio->config.ngpio);
+
+		if (skip_irq_event(vgpio, gpio, irq_line))
+			continue;
+
+		irq = irq_find_mapping(vgpio->gc.irq.domain, gpio);
+		WARN_ON(!irq);
+
+		ret = generic_handle_irq(irq);
+		if (ret)
+			dev_err(dev, "failed to handle interrupt: %d\n", ret);
+	};
+}
+
 static void virtio_gpio_request_vq(struct virtqueue *vq)
 {
 	struct virtio_gpio_line *line;
@@ -211,14 +463,15 @@ static void virtio_gpio_free_vqs(struct virtio_device *vdev)
 static int virtio_gpio_alloc_vqs(struct virtio_gpio *vgpio,
 				 struct virtio_device *vdev)
 {
-	const char * const names[] = { "requestq" };
+	const char * const names[] = { "requestq", "eventq" };
 	vq_callback_t *cbs[] = {
 		virtio_gpio_request_vq,
+		virtio_gpio_event_vq,
 	};
-	struct virtqueue *vqs[1] = { NULL };
+	struct virtqueue *vqs[2] = { NULL, NULL };
 	int ret;
 
-	ret = virtio_find_vqs(vdev, 1, vqs, cbs, names, NULL);
+	ret = virtio_find_vqs(vdev, vgpio->irq_lines ? 2 : 1, vqs, cbs, names, NULL);
 	if (ret) {
 		dev_err(&vdev->dev, "failed to find vqs: %d\n", ret);
 		return ret;
@@ -226,11 +479,23 @@ static int virtio_gpio_alloc_vqs(struct virtio_gpio *vgpio,
 
 	if (!vqs[0]) {
 		dev_err(&vdev->dev, "failed to find requestq vq\n");
-		return -ENODEV;
+		goto out;
 	}
 	vgpio->request_vq = vqs[0];
 
+	if (vgpio->irq_lines && !vqs[1]) {
+		dev_err(&vdev->dev, "failed to find eventq vq\n");
+		goto out;
+	}
+	vgpio->event_vq = vqs[1];
+
 	return 0;
+
+out:
+	if (vqs[0] || vqs[1])
+		virtio_gpio_free_vqs(vdev);
+
+	return -ENODEV;
 }
 
 static const char **virtio_gpio_get_names(struct virtio_gpio *vgpio)
@@ -326,6 +591,32 @@ static int virtio_gpio_probe(struct virtio_device *vdev)
 	vgpio->gc.owner			= THIS_MODULE;
 	vgpio->gc.can_sleep		= true;
 
+	/* Interrupt support */
+	if (virtio_has_feature(vdev, VIRTIO_GPIO_F_IRQ)) {
+		vgpio->irq_lines = devm_kcalloc(dev, config->ngpio,
+						sizeof(*vgpio->irq_lines),
+						GFP_KERNEL);
+		if (!vgpio->irq_lines)
+			return -ENOMEM;
+
+		/* The event comes from the outside so no parent handler */
+		vgpio->gc.irq.parent_handler	= NULL;
+		vgpio->gc.irq.num_parents	= 0;
+		vgpio->gc.irq.parents		= NULL;
+		vgpio->gc.irq.default_type	= IRQ_TYPE_NONE;
+		vgpio->gc.irq.handler		= handle_level_irq;
+		vgpio->gc.irq.chip		= &vgpio_irq_chip;
+
+		for (i = 0; i < config->ngpio; i++) {
+			vgpio->irq_lines[i].type = VIRTIO_GPIO_IRQ_TYPE_NONE;
+			vgpio->irq_lines[i].disabled = true;
+			vgpio->irq_lines[i].masked = true;
+		}
+
+		mutex_init(&vgpio->irq_lock);
+		spin_lock_init(&vgpio->eventq_lock);
+	}
+
 	ret = virtio_gpio_alloc_vqs(vgpio, vdev);
 	if (ret)
 		return ret;
@@ -358,7 +649,13 @@ static const struct virtio_device_id id_table[] = {
 };
 MODULE_DEVICE_TABLE(virtio, id_table);
 
+static const unsigned int features[] = {
+	VIRTIO_GPIO_F_IRQ,
+};
+
 static struct virtio_driver virtio_gpio_driver = {
+	.feature_table		= features,
+	.feature_table_size	= ARRAY_SIZE(features),
 	.id_table		= id_table,
 	.probe			= virtio_gpio_probe,
 	.remove			= virtio_gpio_remove,

Re: [Stratos-dev] [PATCH V4 2/2] gpio: virtio: Add IRQ support

[Arnd Bergmann]

On Mon, Aug 9, 2021 at 12:47 PM Viresh Kumar via Stratos-dev
<stratos-dev@op-lists.linaro.org> wrote:
>
> On 09-08-21, 09:55, Arnd Bergmann wrote:
> > Ah, right. There is already a flag that gets checked by the caller.
> >
> > It does feel odd to have an empty 'irq_mask' callback though, so
> > maybe there is still something missing, just not what I thought.
> >
> > It's probably the result of calling handle_level_irq(), which as you
> > said is closer to what we want, but is not exactly what we need for
> > this protocol.
>
> Okay, I have tried to take care of locking as well now and used local
> flags only to make sure I can depend on them to get the locking
> working properly. Lets see what's broken in this now :)

I don't see anything wrong with this version, but let's see what
Marc thinks. I expect that he can still poke some holes in it, or
at least find some simplifications.

I was slightly surprised at the relation between the disabled and
masked states, where 'disable' always implies 'mask' and
'enable' always implies 'unmask', but I don't actually know how
those two are actually defined in the irqchip code in Linux, so
I assume you did this correctly.

          Arnd

Re: [Stratos-dev] [PATCH V4 2/2] gpio: virtio: Add IRQ support

[Viresh Kumar]

On 09-08-21, 13:19, Arnd Bergmann wrote:
> I don't see anything wrong with this version,

Great.

> but let's see what
> Marc thinks. I expect that he can still poke some holes in it, or
> at least find some simplifications.

Right, though I may resend the patches properly first to make it less
confusing.

> I was slightly surprised at the relation between the disabled and
> masked states, where 'disable' always implies 'mask' and
> 'enable' always implies 'unmask', but I don't actually know how
> those two are actually defined in the irqchip code in Linux, so
> I assume you did this correctly.

I did have a look at the irq-core, but didn't go in real depth. I
rather saw how stuff happens at the driver's end.

- On setup-irq, the core only calls enable() and not unmask().
- On interrupt, the core calls mask(), followed by unmask() (which can
  be delayed for threaded irqs).
- On disable_irq(), the core only calls disable().
- On enable_irq(), the core only calls enable().

And so I ended up at this version :)

-- 
viresh