[PATCH 0/5] VFIO platform reset

[PATCH 0/5] VFIO platform reset

[Eric Auger]

In situations where the userspace driver is stopped abnormally and the
VFIO platform device is released, the assigned HW device currently is
left running. As a consequence the HW device might continue issuing IRQs
and performing DMA accesses.

On release, no physical IRQ handler is setup anymore. Also the DMA buffers
are unmapped leading to IOMMU aborts. So there is no serious consequence.

However when assigning that HW device again to another userspace driver,
this latter might face some unexpected IRQs and DMA accesses, which are
the result of the previous assignment.

In virtualization use-case, a VM newly granted with that HW device may be
impacted by the assignment of that device to a previous VM:
- IRQs may be injected very early when booting the new guest, even before
  the guest driver has initialized leading to possible driver state
  inconsistency.
- DMA accesses may hit the newly mapped VM address space at addresses that
  may jeopardize the integrity of the newly installed VM.

Obviously the criticity depends on the assigned HW devcie.

As opposed to PCI, there is no standard mechanism to reset the platform
device.

This series proposes to implement device specific reset functions in
separate vfio modules (in-kernel or external). The reset module only
registers/unregisters a reset function to the vfio platform driver using
a dedicated API, paired with the device compatibility string.

The platform driver then look for the reset function that matches the
compatibility string of the device. In case of match the device specific
reset can be applied on release and on VFIO_DEVICE_GET_INFO and
VFIO_DEVICE_RESET ioctls.

A first reset module is provided: the vfio-platform-calxedaxgmac
module. It implements a basic reset for the Calxeda xgmac.

Amba device reset is not tested.

Best Regards

Eric

The series can be found at
https://git.linaro.org/people/eric.auger/linux.git/shortlog/refs/heads/v4.1-rc2-reset

RFC -> PATCH v1:
- solution now based on a lookup list instead of specialized driver


Eric Auger (5):
  VFIO: platform: add reset_list and register/unregister functions
  VFIO: platform: add get_device callback
  VFIO: platform: add reset callback
  VFIO: platform: populate reset function according to compat
  VFIO: platform: VFIO platform Calxeda xgmac reset module

 drivers/vfio/platform/Kconfig                      |   2 +
 drivers/vfio/platform/Makefile                     |   2 +
 drivers/vfio/platform/reset/Kconfig                |   7 ++
 drivers/vfio/platform/reset/Makefile               |   5 +
 .../platform/reset/vfio_platform_calxedaxgmac.c    | 121 ++++++++++++++++++++
 drivers/vfio/platform/vfio_amba.c                  |   9 ++
 drivers/vfio/platform/vfio_platform.c              |  10 ++
 drivers/vfio/platform/vfio_platform_common.c       | 125 ++++++++++++++++++++-
 drivers/vfio/platform/vfio_platform_private.h      |  15 +++
 9 files changed, 294 insertions(+), 2 deletions(-)
 create mode 100644 drivers/vfio/platform/reset/Kconfig
 create mode 100644 drivers/vfio/platform/reset/Makefile
 create mode 100644 drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c

-- 
1.9.1

[PATCH 0/5] VFIO platform reset

[Eric Auger]

In situations where the userspace driver is stopped abnormally and the
VFIO platform device is released, the assigned HW device currently is
left running. As a consequence the HW device might continue issuing IRQs
and performing DMA accesses.

On release, no physical IRQ handler is setup anymore. Also the DMA buffers
are unmapped leading to IOMMU aborts. So there is no serious consequence.

However when assigning that HW device again to another userspace driver,
this latter might face some unexpected IRQs and DMA accesses, which are
the result of the previous assignment.

In virtualization use-case, a VM newly granted with that HW device may be
impacted by the assignment of that device to a previous VM:
- IRQs may be injected very early when booting the new guest, even before
  the guest driver has initialized leading to possible driver state
  inconsistency.
- DMA accesses may hit the newly mapped VM address space at addresses that
  may jeopardize the integrity of the newly installed VM.

Obviously the criticity depends on the assigned HW devcie.

As opposed to PCI, there is no standard mechanism to reset the platform
device.

This series proposes to implement device specific reset functions in
separate vfio modules (in-kernel or external). The reset module only
registers/unregisters a reset function to the vfio platform driver using
a dedicated API, paired with the device compatibility string.

The platform driver then look for the reset function that matches the
compatibility string of the device. In case of match the device specific
reset can be applied on release and on VFIO_DEVICE_GET_INFO and
VFIO_DEVICE_RESET ioctls.

A first reset module is provided: the vfio-platform-calxedaxgmac
module. It implements a basic reset for the Calxeda xgmac.

Amba device reset is not tested.

Best Regards

Eric

The series can be found at
https://git.linaro.org/people/eric.auger/linux.git/shortlog/refs/heads/v4.1-rc2-reset

RFC -> PATCH v1:
- solution now based on a lookup list instead of specialized driver


Eric Auger (5):
  VFIO: platform: add reset_list and register/unregister functions
  VFIO: platform: add get_device callback
  VFIO: platform: add reset callback
  VFIO: platform: populate reset function according to compat
  VFIO: platform: VFIO platform Calxeda xgmac reset module

 drivers/vfio/platform/Kconfig                      |   2 +
 drivers/vfio/platform/Makefile                     |   2 +
 drivers/vfio/platform/reset/Kconfig                |   7 ++
 drivers/vfio/platform/reset/Makefile               |   5 +
 .../platform/reset/vfio_platform_calxedaxgmac.c    | 121 ++++++++++++++++++++
 drivers/vfio/platform/vfio_amba.c                  |   9 ++
 drivers/vfio/platform/vfio_platform.c              |  10 ++
 drivers/vfio/platform/vfio_platform_common.c       | 125 ++++++++++++++++++++-
 drivers/vfio/platform/vfio_platform_private.h      |  15 +++
 9 files changed, 294 insertions(+), 2 deletions(-)
 create mode 100644 drivers/vfio/platform/reset/Kconfig
 create mode 100644 drivers/vfio/platform/reset/Makefile
 create mode 100644 drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c

-- 
1.9.1

[PATCH 1/5] VFIO: platform: add reset_list and register/unregister functions

[Eric Auger]

vfio_platform_common now stores a lists of available reset functions.
Two functions are exposed to register/unregister a reset function. A
reset function is paired with a compat string.

Signed-off-by: Eric Auger <eric.auger@linaro.org>
---
 drivers/vfio/platform/vfio_platform_common.c  | 63 +++++++++++++++++++++++++++
 drivers/vfio/platform/vfio_platform_private.h | 13 ++++++
 2 files changed, 76 insertions(+)

diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c
index abcff7a..edbf24c 100644
--- a/drivers/vfio/platform/vfio_platform_common.c
+++ b/drivers/vfio/platform/vfio_platform_common.c
@@ -23,6 +23,9 @@
 
 #include "vfio_platform_private.h"
 
+struct list_head reset_list;
+LIST_HEAD(reset_list);
+
 static DEFINE_MUTEX(driver_lock);
 
 static int vfio_platform_regions_init(struct vfio_platform_device *vdev)
@@ -511,6 +514,13 @@ EXPORT_SYMBOL_GPL(vfio_platform_probe_common);
 struct vfio_platform_device *vfio_platform_remove_common(struct device *dev)
 {
 	struct vfio_platform_device *vdev;
+	struct vfio_platform_reset_node *iter, *tmp;
+
+	list_for_each_entry_safe(iter, tmp, &reset_list, link) {
+		list_del(&iter->link);
+		kfree(iter->compat);
+		kfree(iter);
+	}
 
 	vdev = vfio_del_group_dev(dev);
 	if (vdev)
@@ -519,3 +529,56 @@ struct vfio_platform_device *vfio_platform_remove_common(struct device *dev)
 	return vdev;
 }
 EXPORT_SYMBOL_GPL(vfio_platform_remove_common);
+
+int vfio_platform_register_reset(char *compat, struct module *reset_owner,
+				 vfio_platform_reset_fn_t reset)
+{
+	struct vfio_platform_reset_node *node, *iter;
+	bool found = false;
+
+	list_for_each_entry(iter, &reset_list, link) {
+		if (!strcmp(iter->compat, compat)) {
+			found = true;
+			break;
+		}
+	}
+	if (found)
+		return -EINVAL;
+
+	node = kmalloc(sizeof(*node), GFP_KERNEL);
+	if (!node)
+		return -ENOMEM;
+
+	node->compat = kstrdup(compat, GFP_KERNEL);
+	if (!node->compat)
+		return -ENOMEM;
+
+	node->owner = reset_owner;
+	node->reset = reset;
+
+	list_add(&node->link, &reset_list);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(vfio_platform_register_reset);
+
+int vfio_platform_unregister_reset(char *compat)
+{
+	struct vfio_platform_reset_node *iter;
+	bool found = false;
+
+	list_for_each_entry(iter, &reset_list, link) {
+		if (!strcmp(iter->compat, compat)) {
+			found = true;
+			break;
+		}
+	}
+	if (!found)
+		return -EINVAL;
+
+	list_del(&iter->link);
+	kfree(iter->compat);
+	kfree(iter);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(vfio_platform_unregister_reset);
+
diff --git a/drivers/vfio/platform/vfio_platform_private.h b/drivers/vfio/platform/vfio_platform_private.h
index 5d31e04..da2d60b 100644
--- a/drivers/vfio/platform/vfio_platform_private.h
+++ b/drivers/vfio/platform/vfio_platform_private.h
@@ -69,6 +69,15 @@ struct vfio_platform_device {
 	int	(*get_irq)(struct vfio_platform_device *vdev, int i);
 };
 
+typedef int (*vfio_platform_reset_fn_t)(struct vfio_platform_device *vdev);
+
+struct vfio_platform_reset_node {
+	struct list_head link;
+	char *compat;
+	struct module *owner;
+	vfio_platform_reset_fn_t reset;
+};
+
 extern int vfio_platform_probe_common(struct vfio_platform_device *vdev,
 				      struct device *dev);
 extern struct vfio_platform_device *vfio_platform_remove_common
@@ -82,4 +91,8 @@ extern int vfio_platform_set_irqs_ioctl(struct vfio_platform_device *vdev,
 					unsigned start, unsigned count,
 					void *data);
 
+extern int vfio_platform_register_reset(char *compat, struct module *owner,
+					vfio_platform_reset_fn_t reset);
+extern int vfio_platform_unregister_reset(char *compat);
+
 #endif /* VFIO_PLATFORM_PRIVATE_H */
-- 
1.9.1

[PATCH 1/5] VFIO: platform: add reset_list and register/unregister functions

[Eric Auger]

vfio_platform_common now stores a lists of available reset functions.
Two functions are exposed to register/unregister a reset function. A
reset function is paired with a compat string.

Signed-off-by: Eric Auger <eric.auger@linaro.org>
---
 drivers/vfio/platform/vfio_platform_common.c  | 63 +++++++++++++++++++++++++++
 drivers/vfio/platform/vfio_platform_private.h | 13 ++++++
 2 files changed, 76 insertions(+)

diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c
index abcff7a..edbf24c 100644
--- a/drivers/vfio/platform/vfio_platform_common.c
+++ b/drivers/vfio/platform/vfio_platform_common.c
@@ -23,6 +23,9 @@
 
 #include "vfio_platform_private.h"
 
+struct list_head reset_list;
+LIST_HEAD(reset_list);
+
 static DEFINE_MUTEX(driver_lock);
 
 static int vfio_platform_regions_init(struct vfio_platform_device *vdev)
@@ -511,6 +514,13 @@ EXPORT_SYMBOL_GPL(vfio_platform_probe_common);
 struct vfio_platform_device *vfio_platform_remove_common(struct device *dev)
 {
 	struct vfio_platform_device *vdev;
+	struct vfio_platform_reset_node *iter, *tmp;
+
+	list_for_each_entry_safe(iter, tmp, &reset_list, link) {
+		list_del(&iter->link);
+		kfree(iter->compat);
+		kfree(iter);
+	}
 
 	vdev = vfio_del_group_dev(dev);
 	if (vdev)
@@ -519,3 +529,56 @@ struct vfio_platform_device *vfio_platform_remove_common(struct device *dev)
 	return vdev;
 }
 EXPORT_SYMBOL_GPL(vfio_platform_remove_common);
+
+int vfio_platform_register_reset(char *compat, struct module *reset_owner,
+				 vfio_platform_reset_fn_t reset)
+{
+	struct vfio_platform_reset_node *node, *iter;
+	bool found = false;
+
+	list_for_each_entry(iter, &reset_list, link) {
+		if (!strcmp(iter->compat, compat)) {
+			found = true;
+			break;
+		}
+	}
+	if (found)
+		return -EINVAL;
+
+	node = kmalloc(sizeof(*node), GFP_KERNEL);
+	if (!node)
+		return -ENOMEM;
+
+	node->compat = kstrdup(compat, GFP_KERNEL);
+	if (!node->compat)
+		return -ENOMEM;
+
+	node->owner = reset_owner;
+	node->reset = reset;
+
+	list_add(&node->link, &reset_list);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(vfio_platform_register_reset);
+
+int vfio_platform_unregister_reset(char *compat)
+{
+	struct vfio_platform_reset_node *iter;
+	bool found = false;
+
+	list_for_each_entry(iter, &reset_list, link) {
+		if (!strcmp(iter->compat, compat)) {
+			found = true;
+			break;
+		}
+	}
+	if (!found)
+		return -EINVAL;
+
+	list_del(&iter->link);
+	kfree(iter->compat);
+	kfree(iter);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(vfio_platform_unregister_reset);
+
diff --git a/drivers/vfio/platform/vfio_platform_private.h b/drivers/vfio/platform/vfio_platform_private.h
index 5d31e04..da2d60b 100644
--- a/drivers/vfio/platform/vfio_platform_private.h
+++ b/drivers/vfio/platform/vfio_platform_private.h
@@ -69,6 +69,15 @@ struct vfio_platform_device {
 	int	(*get_irq)(struct vfio_platform_device *vdev, int i);
 };
 
+typedef int (*vfio_platform_reset_fn_t)(struct vfio_platform_device *vdev);
+
+struct vfio_platform_reset_node {
+	struct list_head link;
+	char *compat;
+	struct module *owner;
+	vfio_platform_reset_fn_t reset;
+};
+
 extern int vfio_platform_probe_common(struct vfio_platform_device *vdev,
 				      struct device *dev);
 extern struct vfio_platform_device *vfio_platform_remove_common
@@ -82,4 +91,8 @@ extern int vfio_platform_set_irqs_ioctl(struct vfio_platform_device *vdev,
 					unsigned start, unsigned count,
 					void *data);
 
+extern int vfio_platform_register_reset(char *compat, struct module *owner,
+					vfio_platform_reset_fn_t reset);
+extern int vfio_platform_unregister_reset(char *compat);
+
 #endif /* VFIO_PLATFORM_PRIVATE_H */
-- 
1.9.1

[PATCH 2/5] VFIO: platform: add get_device callback

[Eric Auger]

It is needed to introduce a new callback enabling to retrieve the
struct device* from the vfio_platform_device. Implementation depends
on the underlying device, platform or amba. This will be used to retrieve
the compatibility string of the device.

Signed-off-by: Eric Auger <eric.auger@linaro.org>
---
 drivers/vfio/platform/vfio_amba.c             |  9 +++++++++
 drivers/vfio/platform/vfio_platform.c         | 10 ++++++++++
 drivers/vfio/platform/vfio_platform_private.h |  1 +
 3 files changed, 20 insertions(+)

diff --git a/drivers/vfio/platform/vfio_amba.c b/drivers/vfio/platform/vfio_amba.c
index ff0331f..fd68115 100644
--- a/drivers/vfio/platform/vfio_amba.c
+++ b/drivers/vfio/platform/vfio_amba.c
@@ -48,6 +48,14 @@ static int get_amba_irq(struct vfio_platform_device *vdev, int i)
 	return ret ? ret : -ENXIO;
 }
 
+static struct device *vfio_amba_get_device(struct vfio_platform_device *vdev)
+{
+	struct amba_device *pdev = (struct amba_device *) vdev->opaque;
+	struct device *dev = &pdev->dev;
+
+	return dev;
+}
+
 static int vfio_amba_probe(struct amba_device *adev, const struct amba_id *id)
 {
 	struct vfio_platform_device *vdev;
@@ -67,6 +75,7 @@ static int vfio_amba_probe(struct amba_device *adev, const struct amba_id *id)
 	vdev->flags = VFIO_DEVICE_FLAGS_AMBA;
 	vdev->get_resource = get_amba_resource;
 	vdev->get_irq = get_amba_irq;
+	vdev->get_device = vfio_amba_get_device;
 
 	ret = vfio_platform_probe_common(vdev, &adev->dev);
 	if (ret) {
diff --git a/drivers/vfio/platform/vfio_platform.c b/drivers/vfio/platform/vfio_platform.c
index cef645c..c025d76 100644
--- a/drivers/vfio/platform/vfio_platform.c
+++ b/drivers/vfio/platform/vfio_platform.c
@@ -51,6 +51,15 @@ static int get_platform_irq(struct vfio_platform_device *vdev, int i)
 	return platform_get_irq(pdev, i);
 }
 
+static struct device *vfio_platform_get_device(
+				struct vfio_platform_device *vdev)
+{
+	struct platform_device *pdev = (struct platform_device *) vdev->opaque;
+	struct device *dev = &pdev->dev;
+
+	return dev;
+}
+
 static int vfio_platform_probe(struct platform_device *pdev)
 {
 	struct vfio_platform_device *vdev;
@@ -65,6 +74,7 @@ static int vfio_platform_probe(struct platform_device *pdev)
 	vdev->flags = VFIO_DEVICE_FLAGS_PLATFORM;
 	vdev->get_resource = get_platform_resource;
 	vdev->get_irq = get_platform_irq;
+	vdev->get_device = vfio_platform_get_device;
 
 	ret = vfio_platform_probe_common(vdev, &pdev->dev);
 	if (ret)
diff --git a/drivers/vfio/platform/vfio_platform_private.h b/drivers/vfio/platform/vfio_platform_private.h
index da2d60b..68909a4 100644
--- a/drivers/vfio/platform/vfio_platform_private.h
+++ b/drivers/vfio/platform/vfio_platform_private.h
@@ -67,6 +67,7 @@ struct vfio_platform_device {
 	struct resource*
 		(*get_resource)(struct vfio_platform_device *vdev, int i);
 	int	(*get_irq)(struct vfio_platform_device *vdev, int i);
+	struct device *(*get_device)(struct vfio_platform_device *vdev);
 };
 
 typedef int (*vfio_platform_reset_fn_t)(struct vfio_platform_device *vdev);
-- 
1.9.1

[PATCH 2/5] VFIO: platform: add get_device callback

[Eric Auger]

It is needed to introduce a new callback enabling to retrieve the
struct device* from the vfio_platform_device. Implementation depends
on the underlying device, platform or amba. This will be used to retrieve
the compatibility string of the device.

Signed-off-by: Eric Auger <eric.auger@linaro.org>
---
 drivers/vfio/platform/vfio_amba.c             |  9 +++++++++
 drivers/vfio/platform/vfio_platform.c         | 10 ++++++++++
 drivers/vfio/platform/vfio_platform_private.h |  1 +
 3 files changed, 20 insertions(+)

diff --git a/drivers/vfio/platform/vfio_amba.c b/drivers/vfio/platform/vfio_amba.c
index ff0331f..fd68115 100644
--- a/drivers/vfio/platform/vfio_amba.c
+++ b/drivers/vfio/platform/vfio_amba.c
@@ -48,6 +48,14 @@ static int get_amba_irq(struct vfio_platform_device *vdev, int i)
 	return ret ? ret : -ENXIO;
 }
 
+static struct device *vfio_amba_get_device(struct vfio_platform_device *vdev)
+{
+	struct amba_device *pdev = (struct amba_device *) vdev->opaque;
+	struct device *dev = &pdev->dev;
+
+	return dev;
+}
+
 static int vfio_amba_probe(struct amba_device *adev, const struct amba_id *id)
 {
 	struct vfio_platform_device *vdev;
@@ -67,6 +75,7 @@ static int vfio_amba_probe(struct amba_device *adev, const struct amba_id *id)
 	vdev->flags = VFIO_DEVICE_FLAGS_AMBA;
 	vdev->get_resource = get_amba_resource;
 	vdev->get_irq = get_amba_irq;
+	vdev->get_device = vfio_amba_get_device;
 
 	ret = vfio_platform_probe_common(vdev, &adev->dev);
 	if (ret) {
diff --git a/drivers/vfio/platform/vfio_platform.c b/drivers/vfio/platform/vfio_platform.c
index cef645c..c025d76 100644
--- a/drivers/vfio/platform/vfio_platform.c
+++ b/drivers/vfio/platform/vfio_platform.c
@@ -51,6 +51,15 @@ static int get_platform_irq(struct vfio_platform_device *vdev, int i)
 	return platform_get_irq(pdev, i);
 }
 
+static struct device *vfio_platform_get_device(
+				struct vfio_platform_device *vdev)
+{
+	struct platform_device *pdev = (struct platform_device *) vdev->opaque;
+	struct device *dev = &pdev->dev;
+
+	return dev;
+}
+
 static int vfio_platform_probe(struct platform_device *pdev)
 {
 	struct vfio_platform_device *vdev;
@@ -65,6 +74,7 @@ static int vfio_platform_probe(struct platform_device *pdev)
 	vdev->flags = VFIO_DEVICE_FLAGS_PLATFORM;
 	vdev->get_resource = get_platform_resource;
 	vdev->get_irq = get_platform_irq;
+	vdev->get_device = vfio_platform_get_device;
 
 	ret = vfio_platform_probe_common(vdev, &pdev->dev);
 	if (ret)
diff --git a/drivers/vfio/platform/vfio_platform_private.h b/drivers/vfio/platform/vfio_platform_private.h
index da2d60b..68909a4 100644
--- a/drivers/vfio/platform/vfio_platform_private.h
+++ b/drivers/vfio/platform/vfio_platform_private.h
@@ -67,6 +67,7 @@ struct vfio_platform_device {
 	struct resource*
 		(*get_resource)(struct vfio_platform_device *vdev, int i);
 	int	(*get_irq)(struct vfio_platform_device *vdev, int i);
+	struct device *(*get_device)(struct vfio_platform_device *vdev);
 };
 
 typedef int (*vfio_platform_reset_fn_t)(struct vfio_platform_device *vdev);
-- 
1.9.1

[PATCH 3/5] VFIO: platform: add reset callback

[Eric Auger]

A new reset callback is introduced. If this callback is populated,
the reset is invoked on device release or upon userspace ioctl. The
modality is exposed on VFIO_DEVICE_GET_INFO.

Signed-off-by: Eric Auger <eric.auger@linaro.org>

---

v2 -> v3:
- change patch title and commit message. Use IS_ERR_OR_NULL to
  anticipate for getter returned value.

v1 -> v2:
- On VFIO_DEVICE_RESET returns -EINVAL in case the callback is not
  populated (instead of -ENOTTY)
---
 drivers/vfio/platform/vfio_platform_common.c  | 12 ++++++++++--
 drivers/vfio/platform/vfio_platform_private.h |  1 +
 2 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c
index edbf24c..0d10018 100644
--- a/drivers/vfio/platform/vfio_platform_common.c
+++ b/drivers/vfio/platform/vfio_platform_common.c
@@ -103,6 +103,8 @@ static void vfio_platform_release(void *device_data)
 	mutex_lock(&driver_lock);
 
 	if (!(--vdev->refcnt)) {
+		if (!IS_ERR_OR_NULL(vdev->reset))
+			vdev->reset(vdev);
 		vfio_platform_regions_cleanup(vdev);
 		vfio_platform_irq_cleanup(vdev);
 	}
@@ -162,6 +164,8 @@ static long vfio_platform_ioctl(void *device_data,
 		if (info.argsz < minsz)
 			return -EINVAL;
 
+		if (!IS_ERR_OR_NULL(vdev->reset))
+			vdev->flags |= VFIO_DEVICE_FLAGS_RESET;
 		info.flags = vdev->flags;
 		info.num_regions = vdev->num_regions;
 		info.num_irqs = vdev->num_irqs;
@@ -255,8 +259,12 @@ static long vfio_platform_ioctl(void *device_data,
 
 		return ret;
 
-	} else if (cmd == VFIO_DEVICE_RESET)
-		return -EINVAL;
+	} else if (cmd == VFIO_DEVICE_RESET) {
+		if (!IS_ERR_OR_NULL(vdev->reset))
+			return vdev->reset(vdev);
+		else
+			return -EINVAL;
+	}
 
 	return -ENOTTY;
 }
diff --git a/drivers/vfio/platform/vfio_platform_private.h b/drivers/vfio/platform/vfio_platform_private.h
index 68909a4..84ccf6d 100644
--- a/drivers/vfio/platform/vfio_platform_private.h
+++ b/drivers/vfio/platform/vfio_platform_private.h
@@ -68,6 +68,7 @@ struct vfio_platform_device {
 		(*get_resource)(struct vfio_platform_device *vdev, int i);
 	int	(*get_irq)(struct vfio_platform_device *vdev, int i);
 	struct device *(*get_device)(struct vfio_platform_device *vdev);
+	int	(*reset)(struct vfio_platform_device *vdev);
 };
 
 typedef int (*vfio_platform_reset_fn_t)(struct vfio_platform_device *vdev);
-- 
1.9.1

[PATCH 3/5] VFIO: platform: add reset callback

[Eric Auger]

A new reset callback is introduced. If this callback is populated,
the reset is invoked on device release or upon userspace ioctl. The
modality is exposed on VFIO_DEVICE_GET_INFO.

Signed-off-by: Eric Auger <eric.auger@linaro.org>

---

v2 -> v3:
- change patch title and commit message. Use IS_ERR_OR_NULL to
  anticipate for getter returned value.

v1 -> v2:
- On VFIO_DEVICE_RESET returns -EINVAL in case the callback is not
  populated (instead of -ENOTTY)
---
 drivers/vfio/platform/vfio_platform_common.c  | 12 ++++++++++--
 drivers/vfio/platform/vfio_platform_private.h |  1 +
 2 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c
index edbf24c..0d10018 100644
--- a/drivers/vfio/platform/vfio_platform_common.c
+++ b/drivers/vfio/platform/vfio_platform_common.c
@@ -103,6 +103,8 @@ static void vfio_platform_release(void *device_data)
 	mutex_lock(&driver_lock);
 
 	if (!(--vdev->refcnt)) {
+		if (!IS_ERR_OR_NULL(vdev->reset))
+			vdev->reset(vdev);
 		vfio_platform_regions_cleanup(vdev);
 		vfio_platform_irq_cleanup(vdev);
 	}
@@ -162,6 +164,8 @@ static long vfio_platform_ioctl(void *device_data,
 		if (info.argsz < minsz)
 			return -EINVAL;
 
+		if (!IS_ERR_OR_NULL(vdev->reset))
+			vdev->flags |= VFIO_DEVICE_FLAGS_RESET;
 		info.flags = vdev->flags;
 		info.num_regions = vdev->num_regions;
 		info.num_irqs = vdev->num_irqs;
@@ -255,8 +259,12 @@ static long vfio_platform_ioctl(void *device_data,
 
 		return ret;
 
-	} else if (cmd == VFIO_DEVICE_RESET)
-		return -EINVAL;
+	} else if (cmd == VFIO_DEVICE_RESET) {
+		if (!IS_ERR_OR_NULL(vdev->reset))
+			return vdev->reset(vdev);
+		else
+			return -EINVAL;
+	}
 
 	return -ENOTTY;
 }
diff --git a/drivers/vfio/platform/vfio_platform_private.h b/drivers/vfio/platform/vfio_platform_private.h
index 68909a4..84ccf6d 100644
--- a/drivers/vfio/platform/vfio_platform_private.h
+++ b/drivers/vfio/platform/vfio_platform_private.h
@@ -68,6 +68,7 @@ struct vfio_platform_device {
 		(*get_resource)(struct vfio_platform_device *vdev, int i);
 	int	(*get_irq)(struct vfio_platform_device *vdev, int i);
 	struct device *(*get_device)(struct vfio_platform_device *vdev);
+	int	(*reset)(struct vfio_platform_device *vdev);
 };
 
 typedef int (*vfio_platform_reset_fn_t)(struct vfio_platform_device *vdev);
-- 
1.9.1

[PATCH 3/5] VFIO: platform: add reset callback

[Eric Auger]

A new reset callback is introduced. If this callback is populated,
the reset is invoked on device release or upon userspace ioctl. The
modality is exposed on VFIO_DEVICE_GET_INFO.

Signed-off-by: Eric Auger <eric.auger@linaro.org>

---

v2 -> v3:
- change patch title and commit message. Use IS_ERR_OR_NULL to
  anticipate for getter returned value.

v1 -> v2:
- On VFIO_DEVICE_RESET returns -EINVAL in case the callback is not
  populated (instead of -ENOTTY)
---
 drivers/vfio/platform/vfio_platform_common.c  | 12 ++++++++++--
 drivers/vfio/platform/vfio_platform_private.h |  1 +
 2 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c
index edbf24c..0d10018 100644
--- a/drivers/vfio/platform/vfio_platform_common.c
+++ b/drivers/vfio/platform/vfio_platform_common.c
@@ -103,6 +103,8 @@ static void vfio_platform_release(void *device_data)
 	mutex_lock(&driver_lock);
 
 	if (!(--vdev->refcnt)) {
+		if (!IS_ERR_OR_NULL(vdev->reset))
+			vdev->reset(vdev);
 		vfio_platform_regions_cleanup(vdev);
 		vfio_platform_irq_cleanup(vdev);
 	}
@@ -162,6 +164,8 @@ static long vfio_platform_ioctl(void *device_data,
 		if (info.argsz < minsz)
 			return -EINVAL;
 
+		if (!IS_ERR_OR_NULL(vdev->reset))
+			vdev->flags |= VFIO_DEVICE_FLAGS_RESET;
 		info.flags = vdev->flags;
 		info.num_regions = vdev->num_regions;
 		info.num_irqs = vdev->num_irqs;
@@ -255,8 +259,12 @@ static long vfio_platform_ioctl(void *device_data,
 
 		return ret;
 
-	} else if (cmd == VFIO_DEVICE_RESET)
-		return -EINVAL;
+	} else if (cmd == VFIO_DEVICE_RESET) {
+		if (!IS_ERR_OR_NULL(vdev->reset))
+			return vdev->reset(vdev);
+		else
+			return -EINVAL;
+	}
 
 	return -ENOTTY;
 }
diff --git a/drivers/vfio/platform/vfio_platform_private.h b/drivers/vfio/platform/vfio_platform_private.h
index 68909a4..84ccf6d 100644
--- a/drivers/vfio/platform/vfio_platform_private.h
+++ b/drivers/vfio/platform/vfio_platform_private.h
@@ -68,6 +68,7 @@ struct vfio_platform_device {
 		(*get_resource)(struct vfio_platform_device *vdev, int i);
 	int	(*get_irq)(struct vfio_platform_device *vdev, int i);
 	struct device *(*get_device)(struct vfio_platform_device *vdev);
+	int	(*reset)(struct vfio_platform_device *vdev);
 };
 
 typedef int (*vfio_platform_reset_fn_t)(struct vfio_platform_device *vdev);
-- 
1.9.1

[PATCH 4/5] VFIO: platform: populate reset function according to compat

[Eric Auger]

Add the reset function lookup according to the device compat
string. This lookup is added at different places:
- on VFIO_DEVICE_GET_INFO
- on VFIO_DEVICE_RESET
- on device release

A reference to the module implementing the reset function is taken
on first reset function lookup and released on vfio platform device
release.

Signed-off-by: Eric Auger <eric.auger@linaro.org>
---
 drivers/vfio/platform/vfio_platform_common.c | 50 ++++++++++++++++++++++++++++
 1 file changed, 50 insertions(+)

diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c
index 0d10018..bd7e44c 100644
--- a/drivers/vfio/platform/vfio_platform_common.c
+++ b/drivers/vfio/platform/vfio_platform_common.c
@@ -28,6 +28,52 @@ LIST_HEAD(reset_list);
 
 static DEFINE_MUTEX(driver_lock);
 
+static vfio_platform_reset_fn_t vfio_platform_lookup_reset(const char *compat)
+{
+	struct vfio_platform_reset_node *iter;
+
+	list_for_each_entry(iter, &reset_list, link) {
+		if (!strcmp(iter->compat, compat) &&
+			try_module_get(iter->owner))
+			return iter->reset;
+	}
+
+	return ERR_PTR(-ENODEV);
+}
+
+static vfio_platform_reset_fn_t vfio_platform_get_reset(
+					struct vfio_platform_device *vdev)
+{
+	struct device *dev = vdev->get_device(vdev);
+	const char *compat_str_array[2];
+	vfio_platform_reset_fn_t reset;
+	int ret;
+
+	if (!IS_ERR_OR_NULL(vdev->reset))
+		return vdev->reset;
+
+	ret = device_property_read_string_array(dev, "compatible",
+						compat_str_array, 2);
+	if (!ret)
+		return ERR_PTR(ret);
+
+	reset = vfio_platform_lookup_reset(compat_str_array[0]);
+	return reset;
+}
+
+static void vfio_platform_put_reset(struct vfio_platform_device *vdev)
+{
+	struct vfio_platform_reset_node *iter;
+
+	list_for_each_entry(iter, &reset_list, link) {
+		if (iter->reset == vdev->reset) {
+			module_put(iter->owner);
+			vdev->reset = NULL;
+			return;
+		}
+	}
+}
+
 static int vfio_platform_regions_init(struct vfio_platform_device *vdev)
 {
 	int cnt = 0, i;
@@ -103,10 +149,12 @@ static void vfio_platform_release(void *device_data)
 	mutex_lock(&driver_lock);
 
 	if (!(--vdev->refcnt)) {
+		vdev->reset = vfio_platform_get_reset(vdev);
 		if (!IS_ERR_OR_NULL(vdev->reset))
 			vdev->reset(vdev);
 		vfio_platform_regions_cleanup(vdev);
 		vfio_platform_irq_cleanup(vdev);
+		vfio_platform_put_reset(vdev);
 	}
 
 	mutex_unlock(&driver_lock);
@@ -164,6 +212,7 @@ static long vfio_platform_ioctl(void *device_data,
 		if (info.argsz < minsz)
 			return -EINVAL;
 
+		vdev->reset = vfio_platform_get_reset(vdev);
 		if (!IS_ERR_OR_NULL(vdev->reset))
 			vdev->flags |= VFIO_DEVICE_FLAGS_RESET;
 		info.flags = vdev->flags;
@@ -260,6 +309,7 @@ static long vfio_platform_ioctl(void *device_data,
 		return ret;
 
 	} else if (cmd == VFIO_DEVICE_RESET) {
+		vdev->reset = vfio_platform_get_reset(vdev);
 		if (!IS_ERR_OR_NULL(vdev->reset))
 			return vdev->reset(vdev);
 		else
-- 
1.9.1

[PATCH 4/5] VFIO: platform: populate reset function according to compat

[Eric Auger]

Add the reset function lookup according to the device compat
string. This lookup is added at different places:
- on VFIO_DEVICE_GET_INFO
- on VFIO_DEVICE_RESET
- on device release

A reference to the module implementing the reset function is taken
on first reset function lookup and released on vfio platform device
release.

Signed-off-by: Eric Auger <eric.auger@linaro.org>
---
 drivers/vfio/platform/vfio_platform_common.c | 50 ++++++++++++++++++++++++++++
 1 file changed, 50 insertions(+)

diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c
index 0d10018..bd7e44c 100644
--- a/drivers/vfio/platform/vfio_platform_common.c
+++ b/drivers/vfio/platform/vfio_platform_common.c
@@ -28,6 +28,52 @@ LIST_HEAD(reset_list);
 
 static DEFINE_MUTEX(driver_lock);
 
+static vfio_platform_reset_fn_t vfio_platform_lookup_reset(const char *compat)
+{
+	struct vfio_platform_reset_node *iter;
+
+	list_for_each_entry(iter, &reset_list, link) {
+		if (!strcmp(iter->compat, compat) &&
+			try_module_get(iter->owner))
+			return iter->reset;
+	}
+
+	return ERR_PTR(-ENODEV);
+}
+
+static vfio_platform_reset_fn_t vfio_platform_get_reset(
+					struct vfio_platform_device *vdev)
+{
+	struct device *dev = vdev->get_device(vdev);
+	const char *compat_str_array[2];
+	vfio_platform_reset_fn_t reset;
+	int ret;
+
+	if (!IS_ERR_OR_NULL(vdev->reset))
+		return vdev->reset;
+
+	ret = device_property_read_string_array(dev, "compatible",
+						compat_str_array, 2);
+	if (!ret)
+		return ERR_PTR(ret);
+
+	reset = vfio_platform_lookup_reset(compat_str_array[0]);
+	return reset;
+}
+
+static void vfio_platform_put_reset(struct vfio_platform_device *vdev)
+{
+	struct vfio_platform_reset_node *iter;
+
+	list_for_each_entry(iter, &reset_list, link) {
+		if (iter->reset == vdev->reset) {
+			module_put(iter->owner);
+			vdev->reset = NULL;
+			return;
+		}
+	}
+}
+
 static int vfio_platform_regions_init(struct vfio_platform_device *vdev)
 {
 	int cnt = 0, i;
@@ -103,10 +149,12 @@ static void vfio_platform_release(void *device_data)
 	mutex_lock(&driver_lock);
 
 	if (!(--vdev->refcnt)) {
+		vdev->reset = vfio_platform_get_reset(vdev);
 		if (!IS_ERR_OR_NULL(vdev->reset))
 			vdev->reset(vdev);
 		vfio_platform_regions_cleanup(vdev);
 		vfio_platform_irq_cleanup(vdev);
+		vfio_platform_put_reset(vdev);
 	}
 
 	mutex_unlock(&driver_lock);
@@ -164,6 +212,7 @@ static long vfio_platform_ioctl(void *device_data,
 		if (info.argsz < minsz)
 			return -EINVAL;
 
+		vdev->reset = vfio_platform_get_reset(vdev);
 		if (!IS_ERR_OR_NULL(vdev->reset))
 			vdev->flags |= VFIO_DEVICE_FLAGS_RESET;
 		info.flags = vdev->flags;
@@ -260,6 +309,7 @@ static long vfio_platform_ioctl(void *device_data,
 		return ret;
 
 	} else if (cmd == VFIO_DEVICE_RESET) {
+		vdev->reset = vfio_platform_get_reset(vdev);
 		if (!IS_ERR_OR_NULL(vdev->reset))
 			return vdev->reset(vdev);
 		else
-- 
1.9.1

[PATCH 4/5] VFIO: platform: populate reset function according to compat

[Eric Auger]

Add the reset function lookup according to the device compat
string. This lookup is added at different places:
- on VFIO_DEVICE_GET_INFO
- on VFIO_DEVICE_RESET
- on device release

A reference to the module implementing the reset function is taken
on first reset function lookup and released on vfio platform device
release.

Signed-off-by: Eric Auger <eric.auger@linaro.org>
---
 drivers/vfio/platform/vfio_platform_common.c | 50 ++++++++++++++++++++++++++++
 1 file changed, 50 insertions(+)

diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c
index 0d10018..bd7e44c 100644
--- a/drivers/vfio/platform/vfio_platform_common.c
+++ b/drivers/vfio/platform/vfio_platform_common.c
@@ -28,6 +28,52 @@ LIST_HEAD(reset_list);
 
 static DEFINE_MUTEX(driver_lock);
 
+static vfio_platform_reset_fn_t vfio_platform_lookup_reset(const char *compat)
+{
+	struct vfio_platform_reset_node *iter;
+
+	list_for_each_entry(iter, &reset_list, link) {
+		if (!strcmp(iter->compat, compat) &&
+			try_module_get(iter->owner))
+			return iter->reset;
+	}
+
+	return ERR_PTR(-ENODEV);
+}
+
+static vfio_platform_reset_fn_t vfio_platform_get_reset(
+					struct vfio_platform_device *vdev)
+{
+	struct device *dev = vdev->get_device(vdev);
+	const char *compat_str_array[2];
+	vfio_platform_reset_fn_t reset;
+	int ret;
+
+	if (!IS_ERR_OR_NULL(vdev->reset))
+		return vdev->reset;
+
+	ret = device_property_read_string_array(dev, "compatible",
+						compat_str_array, 2);
+	if (!ret)
+		return ERR_PTR(ret);
+
+	reset = vfio_platform_lookup_reset(compat_str_array[0]);
+	return reset;
+}
+
+static void vfio_platform_put_reset(struct vfio_platform_device *vdev)
+{
+	struct vfio_platform_reset_node *iter;
+
+	list_for_each_entry(iter, &reset_list, link) {
+		if (iter->reset == vdev->reset) {
+			module_put(iter->owner);
+			vdev->reset = NULL;
+			return;
+		}
+	}
+}
+
 static int vfio_platform_regions_init(struct vfio_platform_device *vdev)
 {
 	int cnt = 0, i;
@@ -103,10 +149,12 @@ static void vfio_platform_release(void *device_data)
 	mutex_lock(&driver_lock);
 
 	if (!(--vdev->refcnt)) {
+		vdev->reset = vfio_platform_get_reset(vdev);
 		if (!IS_ERR_OR_NULL(vdev->reset))
 			vdev->reset(vdev);
 		vfio_platform_regions_cleanup(vdev);
 		vfio_platform_irq_cleanup(vdev);
+		vfio_platform_put_reset(vdev);
 	}
 
 	mutex_unlock(&driver_lock);
@@ -164,6 +212,7 @@ static long vfio_platform_ioctl(void *device_data,
 		if (info.argsz < minsz)
 			return -EINVAL;
 
+		vdev->reset = vfio_platform_get_reset(vdev);
 		if (!IS_ERR_OR_NULL(vdev->reset))
 			vdev->flags |= VFIO_DEVICE_FLAGS_RESET;
 		info.flags = vdev->flags;
@@ -260,6 +309,7 @@ static long vfio_platform_ioctl(void *device_data,
 		return ret;
 
 	} else if (cmd == VFIO_DEVICE_RESET) {
+		vdev->reset = vfio_platform_get_reset(vdev);
 		if (!IS_ERR_OR_NULL(vdev->reset))
 			return vdev->reset(vdev);
 		else
-- 
1.9.1

[PATCH 5/5] VFIO: platform: VFIO platform Calxeda xgmac reset module

[Eric Auger]

This patch introduces a module that registers and implements a basic
reset function for the Calxeda xgmac device. This latter basically disables
interrupts and stops DMA transfers.

The reset function code is inherited from the native calxeda xgmac driver.

Signed-off-by: Eric Auger <eric.auger@linaro.org>
---
 drivers/vfio/platform/Kconfig                      |   2 +
 drivers/vfio/platform/Makefile                     |   2 +
 drivers/vfio/platform/reset/Kconfig                |   7 ++
 drivers/vfio/platform/reset/Makefile               |   5 +
 .../platform/reset/vfio_platform_calxedaxgmac.c    | 121 +++++++++++++++++++++
 5 files changed, 137 insertions(+)
 create mode 100644 drivers/vfio/platform/reset/Kconfig
 create mode 100644 drivers/vfio/platform/reset/Makefile
 create mode 100644 drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c

diff --git a/drivers/vfio/platform/Kconfig b/drivers/vfio/platform/Kconfig
index 9a4403e..1df7477 100644
--- a/drivers/vfio/platform/Kconfig
+++ b/drivers/vfio/platform/Kconfig
@@ -18,3 +18,5 @@ config VFIO_AMBA
 	  framework.
 
 	  If you don't know what to do here, say N.
+
+source "drivers/vfio/platform/reset/Kconfig"
diff --git a/drivers/vfio/platform/Makefile b/drivers/vfio/platform/Makefile
index 81de144..9ce8afe 100644
--- a/drivers/vfio/platform/Makefile
+++ b/drivers/vfio/platform/Makefile
@@ -2,7 +2,9 @@
 vfio-platform-y := vfio_platform.o vfio_platform_common.o vfio_platform_irq.o
 
 obj-$(CONFIG_VFIO_PLATFORM) += vfio-platform.o
+obj-$(CONFIG_VFIO_PLATFORM) += reset/
 
 vfio-amba-y := vfio_amba.o
 
 obj-$(CONFIG_VFIO_AMBA) += vfio-amba.o
+obj-$(CONFIG_VFIO_AMBA) += reset/
diff --git a/drivers/vfio/platform/reset/Kconfig b/drivers/vfio/platform/reset/Kconfig
new file mode 100644
index 0000000..746b96b
--- /dev/null
+++ b/drivers/vfio/platform/reset/Kconfig
@@ -0,0 +1,7 @@
+config VFIO_PLATFORM_CALXEDAXGMAC_RESET
+	tristate "VFIO support for calxeda xgmac reset"
+	depends on VFIO_PLATFORM
+	help
+	  Enables the VFIO platform driver to handle reset for Calxeda xgmac
+
+	  If you don't know what to do here, say N.
diff --git a/drivers/vfio/platform/reset/Makefile b/drivers/vfio/platform/reset/Makefile
new file mode 100644
index 0000000..2a486af
--- /dev/null
+++ b/drivers/vfio/platform/reset/Makefile
@@ -0,0 +1,5 @@
+vfio-platform-calxedaxgmac-y := vfio_platform_calxedaxgmac.o
+
+ccflags-y += -Idrivers/vfio/platform
+
+obj-$(CONFIG_VFIO_PLATFORM_CALXEDAXGMAC_RESET) += vfio-platform-calxedaxgmac.o
diff --git a/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c b/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
new file mode 100644
index 0000000..3c6e428
--- /dev/null
+++ b/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
@@ -0,0 +1,121 @@
+/*
+ * VFIO platform driver specialized for Calxeda xgmac reset
+ * reset code is inherited from calxeda xgmac native driver
+ *
+ * Copyright 2010-2011 Calxeda, Inc.
+ * Copyright (c) 2015 Linaro Ltd.
+ *              www.linaro.org
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <linux/module.h>
+#include <linux/kernel.h>
+#include <linux/init.h>
+#include <linux/io.h>
+
+#include "vfio_platform_private.h"
+
+#define DRIVER_VERSION  "0.1"
+#define DRIVER_AUTHOR   "Eric Auger <eric.auger@linaro.org>"
+#define DRIVER_DESC     "Reset support for Calxeda xgmac vfio platform device"
+
+#define CALXEDAXGMAC_COMPAT "calxeda,hb-xgmac"
+
+/* XGMAC Register definitions */
+#define XGMAC_CONTROL           0x00000000      /* MAC Configuration */
+
+/* DMA Control and Status Registers */
+#define XGMAC_DMA_CONTROL       0x00000f18      /* Ctrl (Operational Mode) */
+#define XGMAC_DMA_INTR_ENA      0x00000f1c      /* Interrupt Enable */
+
+/* DMA Control registe defines */
+#define DMA_CONTROL_ST          0x00002000      /* Start/Stop Transmission */
+#define DMA_CONTROL_SR          0x00000002      /* Start/Stop Receive */
+
+/* Common MAC defines */
+#define MAC_ENABLE_TX           0x00000008      /* Transmitter Enable */
+#define MAC_ENABLE_RX           0x00000004      /* Receiver Enable */
+
+static inline void xgmac_mac_disable(void __iomem *ioaddr)
+{
+	u32 value = readl(ioaddr + XGMAC_DMA_CONTROL);
+
+	value &= ~(DMA_CONTROL_ST | DMA_CONTROL_SR);
+	writel(value, ioaddr + XGMAC_DMA_CONTROL);
+
+	value = readl(ioaddr + XGMAC_CONTROL);
+	value &= ~(MAC_ENABLE_TX | MAC_ENABLE_RX);
+	writel(value, ioaddr + XGMAC_CONTROL);
+}
+
+int vfio_platform_calxedaxgmac_reset(struct vfio_platform_device *vdev)
+{
+	struct vfio_platform_region reg = vdev->regions[0];
+
+	if (!reg.ioaddr) {
+		reg.ioaddr =
+			ioremap_nocache(reg.addr, reg.size);
+		if (!reg.ioaddr)
+			return -ENOMEM;
+	}
+
+	/* disable IRQ */
+	writel(0, reg.ioaddr + XGMAC_DMA_INTR_ENA);
+
+	/* Disable the MAC core */
+	xgmac_mac_disable(reg.ioaddr);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(vfio_platform_calxedaxgmac_reset);
+
+static int __init vfio_platform_calxedaxgmac_init(void)
+{
+	int (*register_reset)(char *, struct module *,
+				vfio_platform_reset_fn_t);
+	int ret;
+
+	register_reset = symbol_get(vfio_platform_register_reset);
+	if (!register_reset)
+		return -EINVAL;
+
+	ret = register_reset(CALXEDAXGMAC_COMPAT, THIS_MODULE,
+			     vfio_platform_calxedaxgmac_reset);
+
+	symbol_put(vfio_platform_register_reset);
+
+	return ret;
+}
+
+static void __exit vfio_platform_calxedaxgmac_exit(void)
+{
+	int (*unregister_reset)(char *);
+	int ret;
+
+	unregister_reset = symbol_get(vfio_platform_unregister_reset);
+	if (!unregister_reset)
+		return;
+
+	ret = unregister_reset(CALXEDAXGMAC_COMPAT);
+
+	symbol_put(vfio_platform_unregister_reset);
+}
+
+module_init(vfio_platform_calxedaxgmac_init);
+module_exit(vfio_platform_calxedaxgmac_exit);
+
+MODULE_VERSION(DRIVER_VERSION);
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR(DRIVER_AUTHOR);
+MODULE_DESCRIPTION(DRIVER_DESC);
-- 
1.9.1

[PATCH 5/5] VFIO: platform: VFIO platform Calxeda xgmac reset module

[Eric Auger]

This patch introduces a module that registers and implements a basic
reset function for the Calxeda xgmac device. This latter basically disables
interrupts and stops DMA transfers.

The reset function code is inherited from the native calxeda xgmac driver.

Signed-off-by: Eric Auger <eric.auger@linaro.org>
---
 drivers/vfio/platform/Kconfig                      |   2 +
 drivers/vfio/platform/Makefile                     |   2 +
 drivers/vfio/platform/reset/Kconfig                |   7 ++
 drivers/vfio/platform/reset/Makefile               |   5 +
 .../platform/reset/vfio_platform_calxedaxgmac.c    | 121 +++++++++++++++++++++
 5 files changed, 137 insertions(+)
 create mode 100644 drivers/vfio/platform/reset/Kconfig
 create mode 100644 drivers/vfio/platform/reset/Makefile
 create mode 100644 drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c

diff --git a/drivers/vfio/platform/Kconfig b/drivers/vfio/platform/Kconfig
index 9a4403e..1df7477 100644
--- a/drivers/vfio/platform/Kconfig
+++ b/drivers/vfio/platform/Kconfig
@@ -18,3 +18,5 @@ config VFIO_AMBA
 	  framework.
 
 	  If you don't know what to do here, say N.
+
+source "drivers/vfio/platform/reset/Kconfig"
diff --git a/drivers/vfio/platform/Makefile b/drivers/vfio/platform/Makefile
index 81de144..9ce8afe 100644
--- a/drivers/vfio/platform/Makefile
+++ b/drivers/vfio/platform/Makefile
@@ -2,7 +2,9 @@
 vfio-platform-y := vfio_platform.o vfio_platform_common.o vfio_platform_irq.o
 
 obj-$(CONFIG_VFIO_PLATFORM) += vfio-platform.o
+obj-$(CONFIG_VFIO_PLATFORM) += reset/
 
 vfio-amba-y := vfio_amba.o
 
 obj-$(CONFIG_VFIO_AMBA) += vfio-amba.o
+obj-$(CONFIG_VFIO_AMBA) += reset/
diff --git a/drivers/vfio/platform/reset/Kconfig b/drivers/vfio/platform/reset/Kconfig
new file mode 100644
index 0000000..746b96b
--- /dev/null
+++ b/drivers/vfio/platform/reset/Kconfig
@@ -0,0 +1,7 @@
+config VFIO_PLATFORM_CALXEDAXGMAC_RESET
+	tristate "VFIO support for calxeda xgmac reset"
+	depends on VFIO_PLATFORM
+	help
+	  Enables the VFIO platform driver to handle reset for Calxeda xgmac
+
+	  If you don't know what to do here, say N.
diff --git a/drivers/vfio/platform/reset/Makefile b/drivers/vfio/platform/reset/Makefile
new file mode 100644
index 0000000..2a486af
--- /dev/null
+++ b/drivers/vfio/platform/reset/Makefile
@@ -0,0 +1,5 @@
+vfio-platform-calxedaxgmac-y := vfio_platform_calxedaxgmac.o
+
+ccflags-y += -Idrivers/vfio/platform
+
+obj-$(CONFIG_VFIO_PLATFORM_CALXEDAXGMAC_RESET) += vfio-platform-calxedaxgmac.o
diff --git a/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c b/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
new file mode 100644
index 0000000..3c6e428
--- /dev/null
+++ b/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
@@ -0,0 +1,121 @@
+/*
+ * VFIO platform driver specialized for Calxeda xgmac reset
+ * reset code is inherited from calxeda xgmac native driver
+ *
+ * Copyright 2010-2011 Calxeda, Inc.
+ * Copyright (c) 2015 Linaro Ltd.
+ *              www.linaro.org
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <linux/module.h>
+#include <linux/kernel.h>
+#include <linux/init.h>
+#include <linux/io.h>
+
+#include "vfio_platform_private.h"
+
+#define DRIVER_VERSION  "0.1"
+#define DRIVER_AUTHOR   "Eric Auger <eric.auger@linaro.org>"
+#define DRIVER_DESC     "Reset support for Calxeda xgmac vfio platform device"
+
+#define CALXEDAXGMAC_COMPAT "calxeda,hb-xgmac"
+
+/* XGMAC Register definitions */
+#define XGMAC_CONTROL           0x00000000      /* MAC Configuration */
+
+/* DMA Control and Status Registers */
+#define XGMAC_DMA_CONTROL       0x00000f18      /* Ctrl (Operational Mode) */
+#define XGMAC_DMA_INTR_ENA      0x00000f1c      /* Interrupt Enable */
+
+/* DMA Control registe defines */
+#define DMA_CONTROL_ST          0x00002000      /* Start/Stop Transmission */
+#define DMA_CONTROL_SR          0x00000002      /* Start/Stop Receive */
+
+/* Common MAC defines */
+#define MAC_ENABLE_TX           0x00000008      /* Transmitter Enable */
+#define MAC_ENABLE_RX           0x00000004      /* Receiver Enable */
+
+static inline void xgmac_mac_disable(void __iomem *ioaddr)
+{
+	u32 value = readl(ioaddr + XGMAC_DMA_CONTROL);
+
+	value &= ~(DMA_CONTROL_ST | DMA_CONTROL_SR);
+	writel(value, ioaddr + XGMAC_DMA_CONTROL);
+
+	value = readl(ioaddr + XGMAC_CONTROL);
+	value &= ~(MAC_ENABLE_TX | MAC_ENABLE_RX);
+	writel(value, ioaddr + XGMAC_CONTROL);
+}
+
+int vfio_platform_calxedaxgmac_reset(struct vfio_platform_device *vdev)
+{
+	struct vfio_platform_region reg = vdev->regions[0];
+
+	if (!reg.ioaddr) {
+		reg.ioaddr =
+			ioremap_nocache(reg.addr, reg.size);
+		if (!reg.ioaddr)
+			return -ENOMEM;
+	}
+
+	/* disable IRQ */
+	writel(0, reg.ioaddr + XGMAC_DMA_INTR_ENA);
+
+	/* Disable the MAC core */
+	xgmac_mac_disable(reg.ioaddr);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(vfio_platform_calxedaxgmac_reset);
+
+static int __init vfio_platform_calxedaxgmac_init(void)
+{
+	int (*register_reset)(char *, struct module *,
+				vfio_platform_reset_fn_t);
+	int ret;
+
+	register_reset = symbol_get(vfio_platform_register_reset);
+	if (!register_reset)
+		return -EINVAL;
+
+	ret = register_reset(CALXEDAXGMAC_COMPAT, THIS_MODULE,
+			     vfio_platform_calxedaxgmac_reset);
+
+	symbol_put(vfio_platform_register_reset);
+
+	return ret;
+}
+
+static void __exit vfio_platform_calxedaxgmac_exit(void)
+{
+	int (*unregister_reset)(char *);
+	int ret;
+
+	unregister_reset = symbol_get(vfio_platform_unregister_reset);
+	if (!unregister_reset)
+		return;
+
+	ret = unregister_reset(CALXEDAXGMAC_COMPAT);
+
+	symbol_put(vfio_platform_unregister_reset);
+}
+
+module_init(vfio_platform_calxedaxgmac_init);
+module_exit(vfio_platform_calxedaxgmac_exit);
+
+MODULE_VERSION(DRIVER_VERSION);
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR(DRIVER_AUTHOR);
+MODULE_DESCRIPTION(DRIVER_DESC);
-- 
1.9.1

[PATCH 5/5] VFIO: platform: VFIO platform Calxeda xgmac reset module

[Eric Auger]

This patch introduces a module that registers and implements a basic
reset function for the Calxeda xgmac device. This latter basically disables
interrupts and stops DMA transfers.

The reset function code is inherited from the native calxeda xgmac driver.

Signed-off-by: Eric Auger <eric.auger@linaro.org>
---
 drivers/vfio/platform/Kconfig                      |   2 +
 drivers/vfio/platform/Makefile                     |   2 +
 drivers/vfio/platform/reset/Kconfig                |   7 ++
 drivers/vfio/platform/reset/Makefile               |   5 +
 .../platform/reset/vfio_platform_calxedaxgmac.c    | 121 +++++++++++++++++++++
 5 files changed, 137 insertions(+)
 create mode 100644 drivers/vfio/platform/reset/Kconfig
 create mode 100644 drivers/vfio/platform/reset/Makefile
 create mode 100644 drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c

diff --git a/drivers/vfio/platform/Kconfig b/drivers/vfio/platform/Kconfig
index 9a4403e..1df7477 100644
--- a/drivers/vfio/platform/Kconfig
+++ b/drivers/vfio/platform/Kconfig
@@ -18,3 +18,5 @@ config VFIO_AMBA
 	  framework.
 
 	  If you don't know what to do here, say N.
+
+source "drivers/vfio/platform/reset/Kconfig"
diff --git a/drivers/vfio/platform/Makefile b/drivers/vfio/platform/Makefile
index 81de144..9ce8afe 100644
--- a/drivers/vfio/platform/Makefile
+++ b/drivers/vfio/platform/Makefile
@@ -2,7 +2,9 @@
 vfio-platform-y := vfio_platform.o vfio_platform_common.o vfio_platform_irq.o
 
 obj-$(CONFIG_VFIO_PLATFORM) += vfio-platform.o
+obj-$(CONFIG_VFIO_PLATFORM) += reset/
 
 vfio-amba-y := vfio_amba.o
 
 obj-$(CONFIG_VFIO_AMBA) += vfio-amba.o
+obj-$(CONFIG_VFIO_AMBA) += reset/
diff --git a/drivers/vfio/platform/reset/Kconfig b/drivers/vfio/platform/reset/Kconfig
new file mode 100644
index 0000000..746b96b
--- /dev/null
+++ b/drivers/vfio/platform/reset/Kconfig
@@ -0,0 +1,7 @@
+config VFIO_PLATFORM_CALXEDAXGMAC_RESET
+	tristate "VFIO support for calxeda xgmac reset"
+	depends on VFIO_PLATFORM
+	help
+	  Enables the VFIO platform driver to handle reset for Calxeda xgmac
+
+	  If you don't know what to do here, say N.
diff --git a/drivers/vfio/platform/reset/Makefile b/drivers/vfio/platform/reset/Makefile
new file mode 100644
index 0000000..2a486af
--- /dev/null
+++ b/drivers/vfio/platform/reset/Makefile
@@ -0,0 +1,5 @@
+vfio-platform-calxedaxgmac-y := vfio_platform_calxedaxgmac.o
+
+ccflags-y += -Idrivers/vfio/platform
+
+obj-$(CONFIG_VFIO_PLATFORM_CALXEDAXGMAC_RESET) += vfio-platform-calxedaxgmac.o
diff --git a/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c b/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
new file mode 100644
index 0000000..3c6e428
--- /dev/null
+++ b/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
@@ -0,0 +1,121 @@
+/*
+ * VFIO platform driver specialized for Calxeda xgmac reset
+ * reset code is inherited from calxeda xgmac native driver
+ *
+ * Copyright 2010-2011 Calxeda, Inc.
+ * Copyright (c) 2015 Linaro Ltd.
+ *              www.linaro.org
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <linux/module.h>
+#include <linux/kernel.h>
+#include <linux/init.h>
+#include <linux/io.h>
+
+#include "vfio_platform_private.h"
+
+#define DRIVER_VERSION  "0.1"
+#define DRIVER_AUTHOR   "Eric Auger <eric.auger@linaro.org>"
+#define DRIVER_DESC     "Reset support for Calxeda xgmac vfio platform device"
+
+#define CALXEDAXGMAC_COMPAT "calxeda,hb-xgmac"
+
+/* XGMAC Register definitions */
+#define XGMAC_CONTROL           0x00000000      /* MAC Configuration */
+
+/* DMA Control and Status Registers */
+#define XGMAC_DMA_CONTROL       0x00000f18      /* Ctrl (Operational Mode) */
+#define XGMAC_DMA_INTR_ENA      0x00000f1c      /* Interrupt Enable */
+
+/* DMA Control registe defines */
+#define DMA_CONTROL_ST          0x00002000      /* Start/Stop Transmission */
+#define DMA_CONTROL_SR          0x00000002      /* Start/Stop Receive */
+
+/* Common MAC defines */
+#define MAC_ENABLE_TX           0x00000008      /* Transmitter Enable */
+#define MAC_ENABLE_RX           0x00000004      /* Receiver Enable */
+
+static inline void xgmac_mac_disable(void __iomem *ioaddr)
+{
+	u32 value = readl(ioaddr + XGMAC_DMA_CONTROL);
+
+	value &= ~(DMA_CONTROL_ST | DMA_CONTROL_SR);
+	writel(value, ioaddr + XGMAC_DMA_CONTROL);
+
+	value = readl(ioaddr + XGMAC_CONTROL);
+	value &= ~(MAC_ENABLE_TX | MAC_ENABLE_RX);
+	writel(value, ioaddr + XGMAC_CONTROL);
+}
+
+int vfio_platform_calxedaxgmac_reset(struct vfio_platform_device *vdev)
+{
+	struct vfio_platform_region reg = vdev->regions[0];
+
+	if (!reg.ioaddr) {
+		reg.ioaddr =
+			ioremap_nocache(reg.addr, reg.size);
+		if (!reg.ioaddr)
+			return -ENOMEM;
+	}
+
+	/* disable IRQ */
+	writel(0, reg.ioaddr + XGMAC_DMA_INTR_ENA);
+
+	/* Disable the MAC core */
+	xgmac_mac_disable(reg.ioaddr);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(vfio_platform_calxedaxgmac_reset);
+
+static int __init vfio_platform_calxedaxgmac_init(void)
+{
+	int (*register_reset)(char *, struct module *,
+				vfio_platform_reset_fn_t);
+	int ret;
+
+	register_reset = symbol_get(vfio_platform_register_reset);
+	if (!register_reset)
+		return -EINVAL;
+
+	ret = register_reset(CALXEDAXGMAC_COMPAT, THIS_MODULE,
+			     vfio_platform_calxedaxgmac_reset);
+
+	symbol_put(vfio_platform_register_reset);
+
+	return ret;
+}
+
+static void __exit vfio_platform_calxedaxgmac_exit(void)
+{
+	int (*unregister_reset)(char *);
+	int ret;
+
+	unregister_reset = symbol_get(vfio_platform_unregister_reset);
+	if (!unregister_reset)
+		return;
+
+	ret = unregister_reset(CALXEDAXGMAC_COMPAT);
+
+	symbol_put(vfio_platform_unregister_reset);
+}
+
+module_init(vfio_platform_calxedaxgmac_init);
+module_exit(vfio_platform_calxedaxgmac_exit);
+
+MODULE_VERSION(DRIVER_VERSION);
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR(DRIVER_AUTHOR);
+MODULE_DESCRIPTION(DRIVER_DESC);
-- 
1.9.1

Re: [PATCH 1/5] VFIO: platform: add reset_list and register/unregister functions

[Alex Williamson]

On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
> vfio_platform_common now stores a lists of available reset functions.
> Two functions are exposed to register/unregister a reset function. A
> reset function is paired with a compat string.
> 
> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> ---
>  drivers/vfio/platform/vfio_platform_common.c  | 63 +++++++++++++++++++++++++++
>  drivers/vfio/platform/vfio_platform_private.h | 13 ++++++
>  2 files changed, 76 insertions(+)
> 
> diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c
> index abcff7a..edbf24c 100644
> --- a/drivers/vfio/platform/vfio_platform_common.c
> +++ b/drivers/vfio/platform/vfio_platform_common.c
> @@ -23,6 +23,9 @@
>  
>  #include "vfio_platform_private.h"
>  
> +struct list_head reset_list;
> +LIST_HEAD(reset_list);
> +

Redundant?  Static?

>  static DEFINE_MUTEX(driver_lock);
>  
>  static int vfio_platform_regions_init(struct vfio_platform_device *vdev)
> @@ -511,6 +514,13 @@ EXPORT_SYMBOL_GPL(vfio_platform_probe_common);
>  struct vfio_platform_device *vfio_platform_remove_common(struct device *dev)
>  {
>  	struct vfio_platform_device *vdev;
> +	struct vfio_platform_reset_node *iter, *tmp;
> +
> +	list_for_each_entry_safe(iter, tmp, &reset_list, link) {
> +		list_del(&iter->link);
> +		kfree(iter->compat);
> +		kfree(iter);
> +	}


This doesn't make sense.  We allow reset functions to be registered and
unregistered, but we forget them all when any device is released?!

>  
>  	vdev = vfio_del_group_dev(dev);
>  	if (vdev)
> @@ -519,3 +529,56 @@ struct vfio_platform_device *vfio_platform_remove_common(struct device *dev)
>  	return vdev;
>  }
>  EXPORT_SYMBOL_GPL(vfio_platform_remove_common);
> +
> +int vfio_platform_register_reset(char *compat, struct module *reset_owner,
> +				 vfio_platform_reset_fn_t reset)
> +{
> +	struct vfio_platform_reset_node *node, *iter;
> +	bool found = false;
> +
> +	list_for_each_entry(iter, &reset_list, link) {
> +		if (!strcmp(iter->compat, compat)) {
> +			found = true;

Just return errno here

> +			break;
> +		}
> +	}
> +	if (found)
> +		return -EINVAL;
> +
> +	node = kmalloc(sizeof(*node), GFP_KERNEL);
> +	if (!node)
> +		return -ENOMEM;
> +
> +	node->compat = kstrdup(compat, GFP_KERNEL);
> +	if (!node->compat)

Leaking node

> +		return -ENOMEM;
> +
> +	node->owner = reset_owner;
> +	node->reset = reset;
> +
> +	list_add(&node->link, &reset_list);

Isn't this racy?  Don't we need some locks around the list?

> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(vfio_platform_register_reset);
> +
> +int vfio_platform_unregister_reset(char *compat)
> +{
> +	struct vfio_platform_reset_node *iter;
> +	bool found = false;
> +
> +	list_for_each_entry(iter, &reset_list, link) {
> +		if (!strcmp(iter->compat, compat)) {

Return errno here

> +			found = true;
> +			break;
> +		}
> +	}
> +	if (!found)
> +		return -EINVAL;
> +
> +	list_del(&iter->link);

Racy

> +	kfree(iter->compat);
> +	kfree(iter);
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(vfio_platform_unregister_reset);
> +
> diff --git a/drivers/vfio/platform/vfio_platform_private.h b/drivers/vfio/platform/vfio_platform_private.h
> index 5d31e04..da2d60b 100644
> --- a/drivers/vfio/platform/vfio_platform_private.h
> +++ b/drivers/vfio/platform/vfio_platform_private.h
> @@ -69,6 +69,15 @@ struct vfio_platform_device {
>  	int	(*get_irq)(struct vfio_platform_device *vdev, int i);
>  };
>  
> +typedef int (*vfio_platform_reset_fn_t)(struct vfio_platform_device *vdev);

Seems like this ought to be in a non-private header if we're exporting
the [un]register functions.
> +
> +struct vfio_platform_reset_node {
> +	struct list_head link;
> +	char *compat;
> +	struct module *owner;
> +	vfio_platform_reset_fn_t reset;
> +};
> +
>  extern int vfio_platform_probe_common(struct vfio_platform_device *vdev,
>  				      struct device *dev);
>  extern struct vfio_platform_device *vfio_platform_remove_common
> @@ -82,4 +91,8 @@ extern int vfio_platform_set_irqs_ioctl(struct vfio_platform_device *vdev,
>  					unsigned start, unsigned count,
>  					void *data);
>  
> +extern int vfio_platform_register_reset(char *compat, struct module *owner,
> +					vfio_platform_reset_fn_t reset);
> +extern int vfio_platform_unregister_reset(char *compat);
> +
>  #endif /* VFIO_PLATFORM_PRIVATE_H */

Re: [PATCH 1/5] VFIO: platform: add reset_list and register/unregister functions

[Alex Williamson]

On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
> vfio_platform_common now stores a lists of available reset functions.
> Two functions are exposed to register/unregister a reset function. A
> reset function is paired with a compat string.
> 
> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> ---
>  drivers/vfio/platform/vfio_platform_common.c  | 63 +++++++++++++++++++++++++++
>  drivers/vfio/platform/vfio_platform_private.h | 13 ++++++
>  2 files changed, 76 insertions(+)
> 
> diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c
> index abcff7a..edbf24c 100644
> --- a/drivers/vfio/platform/vfio_platform_common.c
> +++ b/drivers/vfio/platform/vfio_platform_common.c
> @@ -23,6 +23,9 @@
>  
>  #include "vfio_platform_private.h"
>  
> +struct list_head reset_list;
> +LIST_HEAD(reset_list);
> +

Redundant?  Static?

>  static DEFINE_MUTEX(driver_lock);
>  
>  static int vfio_platform_regions_init(struct vfio_platform_device *vdev)
> @@ -511,6 +514,13 @@ EXPORT_SYMBOL_GPL(vfio_platform_probe_common);
>  struct vfio_platform_device *vfio_platform_remove_common(struct device *dev)
>  {
>  	struct vfio_platform_device *vdev;
> +	struct vfio_platform_reset_node *iter, *tmp;
> +
> +	list_for_each_entry_safe(iter, tmp, &reset_list, link) {
> +		list_del(&iter->link);
> +		kfree(iter->compat);
> +		kfree(iter);
> +	}


This doesn't make sense.  We allow reset functions to be registered and
unregistered, but we forget them all when any device is released?!

>  
>  	vdev = vfio_del_group_dev(dev);
>  	if (vdev)
> @@ -519,3 +529,56 @@ struct vfio_platform_device *vfio_platform_remove_common(struct device *dev)
>  	return vdev;
>  }
>  EXPORT_SYMBOL_GPL(vfio_platform_remove_common);
> +
> +int vfio_platform_register_reset(char *compat, struct module *reset_owner,
> +				 vfio_platform_reset_fn_t reset)
> +{
> +	struct vfio_platform_reset_node *node, *iter;
> +	bool found = false;
> +
> +	list_for_each_entry(iter, &reset_list, link) {
> +		if (!strcmp(iter->compat, compat)) {
> +			found = true;

Just return errno here

> +			break;
> +		}
> +	}
> +	if (found)
> +		return -EINVAL;
> +
> +	node = kmalloc(sizeof(*node), GFP_KERNEL);
> +	if (!node)
> +		return -ENOMEM;
> +
> +	node->compat = kstrdup(compat, GFP_KERNEL);
> +	if (!node->compat)

Leaking node

> +		return -ENOMEM;
> +
> +	node->owner = reset_owner;
> +	node->reset = reset;
> +
> +	list_add(&node->link, &reset_list);

Isn't this racy?  Don't we need some locks around the list?

> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(vfio_platform_register_reset);
> +
> +int vfio_platform_unregister_reset(char *compat)
> +{
> +	struct vfio_platform_reset_node *iter;
> +	bool found = false;
> +
> +	list_for_each_entry(iter, &reset_list, link) {
> +		if (!strcmp(iter->compat, compat)) {

Return errno here

> +			found = true;
> +			break;
> +		}
> +	}
> +	if (!found)
> +		return -EINVAL;
> +
> +	list_del(&iter->link);

Racy

> +	kfree(iter->compat);
> +	kfree(iter);
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(vfio_platform_unregister_reset);
> +
> diff --git a/drivers/vfio/platform/vfio_platform_private.h b/drivers/vfio/platform/vfio_platform_private.h
> index 5d31e04..da2d60b 100644
> --- a/drivers/vfio/platform/vfio_platform_private.h
> +++ b/drivers/vfio/platform/vfio_platform_private.h
> @@ -69,6 +69,15 @@ struct vfio_platform_device {
>  	int	(*get_irq)(struct vfio_platform_device *vdev, int i);
>  };
>  
> +typedef int (*vfio_platform_reset_fn_t)(struct vfio_platform_device *vdev);

Seems like this ought to be in a non-private header if we're exporting
the [un]register functions.
> +
> +struct vfio_platform_reset_node {
> +	struct list_head link;
> +	char *compat;
> +	struct module *owner;
> +	vfio_platform_reset_fn_t reset;
> +};
> +
>  extern int vfio_platform_probe_common(struct vfio_platform_device *vdev,
>  				      struct device *dev);
>  extern struct vfio_platform_device *vfio_platform_remove_common
> @@ -82,4 +91,8 @@ extern int vfio_platform_set_irqs_ioctl(struct vfio_platform_device *vdev,
>  					unsigned start, unsigned count,
>  					void *data);
>  
> +extern int vfio_platform_register_reset(char *compat, struct module *owner,
> +					vfio_platform_reset_fn_t reset);
> +extern int vfio_platform_unregister_reset(char *compat);
> +
>  #endif /* VFIO_PLATFORM_PRIVATE_H */

[PATCH 1/5] VFIO: platform: add reset_list and register/unregister functions

[Alex Williamson]

On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
> vfio_platform_common now stores a lists of available reset functions.
> Two functions are exposed to register/unregister a reset function. A
> reset function is paired with a compat string.
> 
> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> ---
>  drivers/vfio/platform/vfio_platform_common.c  | 63 +++++++++++++++++++++++++++
>  drivers/vfio/platform/vfio_platform_private.h | 13 ++++++
>  2 files changed, 76 insertions(+)
> 
> diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c
> index abcff7a..edbf24c 100644
> --- a/drivers/vfio/platform/vfio_platform_common.c
> +++ b/drivers/vfio/platform/vfio_platform_common.c
> @@ -23,6 +23,9 @@
>  
>  #include "vfio_platform_private.h"
>  
> +struct list_head reset_list;
> +LIST_HEAD(reset_list);
> +

Redundant?  Static?

>  static DEFINE_MUTEX(driver_lock);
>  
>  static int vfio_platform_regions_init(struct vfio_platform_device *vdev)
> @@ -511,6 +514,13 @@ EXPORT_SYMBOL_GPL(vfio_platform_probe_common);
>  struct vfio_platform_device *vfio_platform_remove_common(struct device *dev)
>  {
>  	struct vfio_platform_device *vdev;
> +	struct vfio_platform_reset_node *iter, *tmp;
> +
> +	list_for_each_entry_safe(iter, tmp, &reset_list, link) {
> +		list_del(&iter->link);
> +		kfree(iter->compat);
> +		kfree(iter);
> +	}


This doesn't make sense.  We allow reset functions to be registered and
unregistered, but we forget them all when any device is released?!

>  
>  	vdev = vfio_del_group_dev(dev);
>  	if (vdev)
> @@ -519,3 +529,56 @@ struct vfio_platform_device *vfio_platform_remove_common(struct device *dev)
>  	return vdev;
>  }
>  EXPORT_SYMBOL_GPL(vfio_platform_remove_common);
> +
> +int vfio_platform_register_reset(char *compat, struct module *reset_owner,
> +				 vfio_platform_reset_fn_t reset)
> +{
> +	struct vfio_platform_reset_node *node, *iter;
> +	bool found = false;
> +
> +	list_for_each_entry(iter, &reset_list, link) {
> +		if (!strcmp(iter->compat, compat)) {
> +			found = true;

Just return errno here

> +			break;
> +		}
> +	}
> +	if (found)
> +		return -EINVAL;
> +
> +	node = kmalloc(sizeof(*node), GFP_KERNEL);
> +	if (!node)
> +		return -ENOMEM;
> +
> +	node->compat = kstrdup(compat, GFP_KERNEL);
> +	if (!node->compat)

Leaking node

> +		return -ENOMEM;
> +
> +	node->owner = reset_owner;
> +	node->reset = reset;
> +
> +	list_add(&node->link, &reset_list);

Isn't this racy?  Don't we need some locks around the list?

> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(vfio_platform_register_reset);
> +
> +int vfio_platform_unregister_reset(char *compat)
> +{
> +	struct vfio_platform_reset_node *iter;
> +	bool found = false;
> +
> +	list_for_each_entry(iter, &reset_list, link) {
> +		if (!strcmp(iter->compat, compat)) {

Return errno here

> +			found = true;
> +			break;
> +		}
> +	}
> +	if (!found)
> +		return -EINVAL;
> +
> +	list_del(&iter->link);

Racy

> +	kfree(iter->compat);
> +	kfree(iter);
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(vfio_platform_unregister_reset);
> +
> diff --git a/drivers/vfio/platform/vfio_platform_private.h b/drivers/vfio/platform/vfio_platform_private.h
> index 5d31e04..da2d60b 100644
> --- a/drivers/vfio/platform/vfio_platform_private.h
> +++ b/drivers/vfio/platform/vfio_platform_private.h
> @@ -69,6 +69,15 @@ struct vfio_platform_device {
>  	int	(*get_irq)(struct vfio_platform_device *vdev, int i);
>  };
>  
> +typedef int (*vfio_platform_reset_fn_t)(struct vfio_platform_device *vdev);

Seems like this ought to be in a non-private header if we're exporting
the [un]register functions.
> +
> +struct vfio_platform_reset_node {
> +	struct list_head link;
> +	char *compat;
> +	struct module *owner;
> +	vfio_platform_reset_fn_t reset;
> +};
> +
>  extern int vfio_platform_probe_common(struct vfio_platform_device *vdev,
>  				      struct device *dev);
>  extern struct vfio_platform_device *vfio_platform_remove_common
> @@ -82,4 +91,8 @@ extern int vfio_platform_set_irqs_ioctl(struct vfio_platform_device *vdev,
>  					unsigned start, unsigned count,
>  					void *data);
>  
> +extern int vfio_platform_register_reset(char *compat, struct module *owner,
> +					vfio_platform_reset_fn_t reset);
> +extern int vfio_platform_unregister_reset(char *compat);
> +
>  #endif /* VFIO_PLATFORM_PRIVATE_H */

Re: [PATCH 2/5] VFIO: platform: add get_device callback

[Alex Williamson]

On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
> It is needed to introduce a new callback enabling to retrieve the
> struct device* from the vfio_platform_device. Implementation depends
> on the underlying device, platform or amba. This will be used to retrieve
> the compatibility string of the device.
> 
> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> ---
>  drivers/vfio/platform/vfio_amba.c             |  9 +++++++++
>  drivers/vfio/platform/vfio_platform.c         | 10 ++++++++++
>  drivers/vfio/platform/vfio_platform_private.h |  1 +
>  3 files changed, 20 insertions(+)
> 
> diff --git a/drivers/vfio/platform/vfio_amba.c b/drivers/vfio/platform/vfio_amba.c
> index ff0331f..fd68115 100644
> --- a/drivers/vfio/platform/vfio_amba.c
> +++ b/drivers/vfio/platform/vfio_amba.c
> @@ -48,6 +48,14 @@ static int get_amba_irq(struct vfio_platform_device *vdev, int i)
>  	return ret ? ret : -ENXIO;
>  }
>  
> +static struct device *vfio_amba_get_device(struct vfio_platform_device *vdev)
> +{
> +	struct amba_device *pdev = (struct amba_device *) vdev->opaque;
> +	struct device *dev = &pdev->dev;
> +
> +	return dev;
> +}

*get* device implies a increment in the reference count.  There's
nothing like that happening here.  Do you just want a to_device wrapper?
Why doesn't struct vfio_platform_device just cache a pointer to the
struct device?

> +
>  static int vfio_amba_probe(struct amba_device *adev, const struct amba_id *id)
>  {
>  	struct vfio_platform_device *vdev;
> @@ -67,6 +75,7 @@ static int vfio_amba_probe(struct amba_device *adev, const struct amba_id *id)
>  	vdev->flags = VFIO_DEVICE_FLAGS_AMBA;
>  	vdev->get_resource = get_amba_resource;
>  	vdev->get_irq = get_amba_irq;
> +	vdev->get_device = vfio_amba_get_device;
>  
>  	ret = vfio_platform_probe_common(vdev, &adev->dev);
>  	if (ret) {
> diff --git a/drivers/vfio/platform/vfio_platform.c b/drivers/vfio/platform/vfio_platform.c
> index cef645c..c025d76 100644
> --- a/drivers/vfio/platform/vfio_platform.c
> +++ b/drivers/vfio/platform/vfio_platform.c
> @@ -51,6 +51,15 @@ static int get_platform_irq(struct vfio_platform_device *vdev, int i)
>  	return platform_get_irq(pdev, i);
>  }
>  
> +static struct device *vfio_platform_get_device(
> +				struct vfio_platform_device *vdev)
> +{
> +	struct platform_device *pdev = (struct platform_device *) vdev->opaque;
> +	struct device *dev = &pdev->dev;
> +
> +	return dev;
> +}
> +
>  static int vfio_platform_probe(struct platform_device *pdev)
>  {
>  	struct vfio_platform_device *vdev;
> @@ -65,6 +74,7 @@ static int vfio_platform_probe(struct platform_device *pdev)
>  	vdev->flags = VFIO_DEVICE_FLAGS_PLATFORM;
>  	vdev->get_resource = get_platform_resource;
>  	vdev->get_irq = get_platform_irq;
> +	vdev->get_device = vfio_platform_get_device;
>  
>  	ret = vfio_platform_probe_common(vdev, &pdev->dev);
>  	if (ret)
> diff --git a/drivers/vfio/platform/vfio_platform_private.h b/drivers/vfio/platform/vfio_platform_private.h
> index da2d60b..68909a4 100644
> --- a/drivers/vfio/platform/vfio_platform_private.h
> +++ b/drivers/vfio/platform/vfio_platform_private.h
> @@ -67,6 +67,7 @@ struct vfio_platform_device {
>  	struct resource*
>  		(*get_resource)(struct vfio_platform_device *vdev, int i);
>  	int	(*get_irq)(struct vfio_platform_device *vdev, int i);
> +	struct device *(*get_device)(struct vfio_platform_device *vdev);
>  };
>  
>  typedef int (*vfio_platform_reset_fn_t)(struct vfio_platform_device *vdev);

[PATCH 2/5] VFIO: platform: add get_device callback

[Alex Williamson]

On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
> It is needed to introduce a new callback enabling to retrieve the
> struct device* from the vfio_platform_device. Implementation depends
> on the underlying device, platform or amba. This will be used to retrieve
> the compatibility string of the device.
> 
> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> ---
>  drivers/vfio/platform/vfio_amba.c             |  9 +++++++++
>  drivers/vfio/platform/vfio_platform.c         | 10 ++++++++++
>  drivers/vfio/platform/vfio_platform_private.h |  1 +
>  3 files changed, 20 insertions(+)
> 
> diff --git a/drivers/vfio/platform/vfio_amba.c b/drivers/vfio/platform/vfio_amba.c
> index ff0331f..fd68115 100644
> --- a/drivers/vfio/platform/vfio_amba.c
> +++ b/drivers/vfio/platform/vfio_amba.c
> @@ -48,6 +48,14 @@ static int get_amba_irq(struct vfio_platform_device *vdev, int i)
>  	return ret ? ret : -ENXIO;
>  }
>  
> +static struct device *vfio_amba_get_device(struct vfio_platform_device *vdev)
> +{
> +	struct amba_device *pdev = (struct amba_device *) vdev->opaque;
> +	struct device *dev = &pdev->dev;
> +
> +	return dev;
> +}

*get* device implies a increment in the reference count.  There's
nothing like that happening here.  Do you just want a to_device wrapper?
Why doesn't struct vfio_platform_device just cache a pointer to the
struct device?

> +
>  static int vfio_amba_probe(struct amba_device *adev, const struct amba_id *id)
>  {
>  	struct vfio_platform_device *vdev;
> @@ -67,6 +75,7 @@ static int vfio_amba_probe(struct amba_device *adev, const struct amba_id *id)
>  	vdev->flags = VFIO_DEVICE_FLAGS_AMBA;
>  	vdev->get_resource = get_amba_resource;
>  	vdev->get_irq = get_amba_irq;
> +	vdev->get_device = vfio_amba_get_device;
>  
>  	ret = vfio_platform_probe_common(vdev, &adev->dev);
>  	if (ret) {
> diff --git a/drivers/vfio/platform/vfio_platform.c b/drivers/vfio/platform/vfio_platform.c
> index cef645c..c025d76 100644
> --- a/drivers/vfio/platform/vfio_platform.c
> +++ b/drivers/vfio/platform/vfio_platform.c
> @@ -51,6 +51,15 @@ static int get_platform_irq(struct vfio_platform_device *vdev, int i)
>  	return platform_get_irq(pdev, i);
>  }
>  
> +static struct device *vfio_platform_get_device(
> +				struct vfio_platform_device *vdev)
> +{
> +	struct platform_device *pdev = (struct platform_device *) vdev->opaque;
> +	struct device *dev = &pdev->dev;
> +
> +	return dev;
> +}
> +
>  static int vfio_platform_probe(struct platform_device *pdev)
>  {
>  	struct vfio_platform_device *vdev;
> @@ -65,6 +74,7 @@ static int vfio_platform_probe(struct platform_device *pdev)
>  	vdev->flags = VFIO_DEVICE_FLAGS_PLATFORM;
>  	vdev->get_resource = get_platform_resource;
>  	vdev->get_irq = get_platform_irq;
> +	vdev->get_device = vfio_platform_get_device;
>  
>  	ret = vfio_platform_probe_common(vdev, &pdev->dev);
>  	if (ret)
> diff --git a/drivers/vfio/platform/vfio_platform_private.h b/drivers/vfio/platform/vfio_platform_private.h
> index da2d60b..68909a4 100644
> --- a/drivers/vfio/platform/vfio_platform_private.h
> +++ b/drivers/vfio/platform/vfio_platform_private.h
> @@ -67,6 +67,7 @@ struct vfio_platform_device {
>  	struct resource*
>  		(*get_resource)(struct vfio_platform_device *vdev, int i);
>  	int	(*get_irq)(struct vfio_platform_device *vdev, int i);
> +	struct device *(*get_device)(struct vfio_platform_device *vdev);
>  };
>  
>  typedef int (*vfio_platform_reset_fn_t)(struct vfio_platform_device *vdev);

Re: [PATCH 3/5] VFIO: platform: add reset callback

[Alex Williamson]

On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
> A new reset callback is introduced. If this callback is populated,
> the reset is invoked on device release or upon userspace ioctl. The
> modality is exposed on VFIO_DEVICE_GET_INFO.
> 
> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> 
> ---
> 
> v2 -> v3:
> - change patch title and commit message. Use IS_ERR_OR_NULL to
>   anticipate for getter returned value.
> 
> v1 -> v2:
> - On VFIO_DEVICE_RESET returns -EINVAL in case the callback is not
>   populated (instead of -ENOTTY)
> ---
>  drivers/vfio/platform/vfio_platform_common.c  | 12 ++++++++++--
>  drivers/vfio/platform/vfio_platform_private.h |  1 +
>  2 files changed, 11 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c
> index edbf24c..0d10018 100644
> --- a/drivers/vfio/platform/vfio_platform_common.c
> +++ b/drivers/vfio/platform/vfio_platform_common.c
> @@ -103,6 +103,8 @@ static void vfio_platform_release(void *device_data)
>  	mutex_lock(&driver_lock);
>  
>  	if (!(--vdev->refcnt)) {
> +		if (!IS_ERR_OR_NULL(vdev->reset))
> +			vdev->reset(vdev);


It seems sloppy to have a function pointer that could be ERR_PTR.

Should we also be trying to reset on device open?

>  		vfio_platform_regions_cleanup(vdev);
>  		vfio_platform_irq_cleanup(vdev);
>  	}
> @@ -162,6 +164,8 @@ static long vfio_platform_ioctl(void *device_data,
>  		if (info.argsz < minsz)
>  			return -EINVAL;
>  
> +		if (!IS_ERR_OR_NULL(vdev->reset))
> +			vdev->flags |= VFIO_DEVICE_FLAGS_RESET;
>  		info.flags = vdev->flags;
>  		info.num_regions = vdev->num_regions;
>  		info.num_irqs = vdev->num_irqs;
> @@ -255,8 +259,12 @@ static long vfio_platform_ioctl(void *device_data,
>  
>  		return ret;
>  
> -	} else if (cmd == VFIO_DEVICE_RESET)
> -		return -EINVAL;
> +	} else if (cmd == VFIO_DEVICE_RESET) {
> +		if (!IS_ERR_OR_NULL(vdev->reset))
> +			return vdev->reset(vdev);
> +		else
> +			return -EINVAL;
> +	}
>  
>  	return -ENOTTY;
>  }
> diff --git a/drivers/vfio/platform/vfio_platform_private.h b/drivers/vfio/platform/vfio_platform_private.h
> index 68909a4..84ccf6d 100644
> --- a/drivers/vfio/platform/vfio_platform_private.h
> +++ b/drivers/vfio/platform/vfio_platform_private.h
> @@ -68,6 +68,7 @@ struct vfio_platform_device {
>  		(*get_resource)(struct vfio_platform_device *vdev, int i);
>  	int	(*get_irq)(struct vfio_platform_device *vdev, int i);
>  	struct device *(*get_device)(struct vfio_platform_device *vdev);
> +	int	(*reset)(struct vfio_platform_device *vdev);
>  };
>  
>  typedef int (*vfio_platform_reset_fn_t)(struct vfio_platform_device *vdev);

Re: [PATCH 3/5] VFIO: platform: add reset callback

[Alex Williamson]

On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
> A new reset callback is introduced. If this callback is populated,
> the reset is invoked on device release or upon userspace ioctl. The
> modality is exposed on VFIO_DEVICE_GET_INFO.
> 
> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> 
> ---
> 
> v2 -> v3:
> - change patch title and commit message. Use IS_ERR_OR_NULL to
>   anticipate for getter returned value.
> 
> v1 -> v2:
> - On VFIO_DEVICE_RESET returns -EINVAL in case the callback is not
>   populated (instead of -ENOTTY)
> ---
>  drivers/vfio/platform/vfio_platform_common.c  | 12 ++++++++++--
>  drivers/vfio/platform/vfio_platform_private.h |  1 +
>  2 files changed, 11 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c
> index edbf24c..0d10018 100644
> --- a/drivers/vfio/platform/vfio_platform_common.c
> +++ b/drivers/vfio/platform/vfio_platform_common.c
> @@ -103,6 +103,8 @@ static void vfio_platform_release(void *device_data)
>  	mutex_lock(&driver_lock);
>  
>  	if (!(--vdev->refcnt)) {
> +		if (!IS_ERR_OR_NULL(vdev->reset))
> +			vdev->reset(vdev);


It seems sloppy to have a function pointer that could be ERR_PTR.

Should we also be trying to reset on device open?

>  		vfio_platform_regions_cleanup(vdev);
>  		vfio_platform_irq_cleanup(vdev);
>  	}
> @@ -162,6 +164,8 @@ static long vfio_platform_ioctl(void *device_data,
>  		if (info.argsz < minsz)
>  			return -EINVAL;
>  
> +		if (!IS_ERR_OR_NULL(vdev->reset))
> +			vdev->flags |= VFIO_DEVICE_FLAGS_RESET;
>  		info.flags = vdev->flags;
>  		info.num_regions = vdev->num_regions;
>  		info.num_irqs = vdev->num_irqs;
> @@ -255,8 +259,12 @@ static long vfio_platform_ioctl(void *device_data,
>  
>  		return ret;
>  
> -	} else if (cmd == VFIO_DEVICE_RESET)
> -		return -EINVAL;
> +	} else if (cmd == VFIO_DEVICE_RESET) {
> +		if (!IS_ERR_OR_NULL(vdev->reset))
> +			return vdev->reset(vdev);
> +		else
> +			return -EINVAL;
> +	}
>  
>  	return -ENOTTY;
>  }
> diff --git a/drivers/vfio/platform/vfio_platform_private.h b/drivers/vfio/platform/vfio_platform_private.h
> index 68909a4..84ccf6d 100644
> --- a/drivers/vfio/platform/vfio_platform_private.h
> +++ b/drivers/vfio/platform/vfio_platform_private.h
> @@ -68,6 +68,7 @@ struct vfio_platform_device {
>  		(*get_resource)(struct vfio_platform_device *vdev, int i);
>  	int	(*get_irq)(struct vfio_platform_device *vdev, int i);
>  	struct device *(*get_device)(struct vfio_platform_device *vdev);
> +	int	(*reset)(struct vfio_platform_device *vdev);
>  };
>  
>  typedef int (*vfio_platform_reset_fn_t)(struct vfio_platform_device *vdev);

[PATCH 3/5] VFIO: platform: add reset callback

[Alex Williamson]

On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
> A new reset callback is introduced. If this callback is populated,
> the reset is invoked on device release or upon userspace ioctl. The
> modality is exposed on VFIO_DEVICE_GET_INFO.
> 
> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> 
> ---
> 
> v2 -> v3:
> - change patch title and commit message. Use IS_ERR_OR_NULL to
>   anticipate for getter returned value.
> 
> v1 -> v2:
> - On VFIO_DEVICE_RESET returns -EINVAL in case the callback is not
>   populated (instead of -ENOTTY)
> ---
>  drivers/vfio/platform/vfio_platform_common.c  | 12 ++++++++++--
>  drivers/vfio/platform/vfio_platform_private.h |  1 +
>  2 files changed, 11 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c
> index edbf24c..0d10018 100644
> --- a/drivers/vfio/platform/vfio_platform_common.c
> +++ b/drivers/vfio/platform/vfio_platform_common.c
> @@ -103,6 +103,8 @@ static void vfio_platform_release(void *device_data)
>  	mutex_lock(&driver_lock);
>  
>  	if (!(--vdev->refcnt)) {
> +		if (!IS_ERR_OR_NULL(vdev->reset))
> +			vdev->reset(vdev);


It seems sloppy to have a function pointer that could be ERR_PTR.

Should we also be trying to reset on device open?

>  		vfio_platform_regions_cleanup(vdev);
>  		vfio_platform_irq_cleanup(vdev);
>  	}
> @@ -162,6 +164,8 @@ static long vfio_platform_ioctl(void *device_data,
>  		if (info.argsz < minsz)
>  			return -EINVAL;
>  
> +		if (!IS_ERR_OR_NULL(vdev->reset))
> +			vdev->flags |= VFIO_DEVICE_FLAGS_RESET;
>  		info.flags = vdev->flags;
>  		info.num_regions = vdev->num_regions;
>  		info.num_irqs = vdev->num_irqs;
> @@ -255,8 +259,12 @@ static long vfio_platform_ioctl(void *device_data,
>  
>  		return ret;
>  
> -	} else if (cmd == VFIO_DEVICE_RESET)
> -		return -EINVAL;
> +	} else if (cmd == VFIO_DEVICE_RESET) {
> +		if (!IS_ERR_OR_NULL(vdev->reset))
> +			return vdev->reset(vdev);
> +		else
> +			return -EINVAL;
> +	}
>  
>  	return -ENOTTY;
>  }
> diff --git a/drivers/vfio/platform/vfio_platform_private.h b/drivers/vfio/platform/vfio_platform_private.h
> index 68909a4..84ccf6d 100644
> --- a/drivers/vfio/platform/vfio_platform_private.h
> +++ b/drivers/vfio/platform/vfio_platform_private.h
> @@ -68,6 +68,7 @@ struct vfio_platform_device {
>  		(*get_resource)(struct vfio_platform_device *vdev, int i);
>  	int	(*get_irq)(struct vfio_platform_device *vdev, int i);
>  	struct device *(*get_device)(struct vfio_platform_device *vdev);
> +	int	(*reset)(struct vfio_platform_device *vdev);
>  };
>  
>  typedef int (*vfio_platform_reset_fn_t)(struct vfio_platform_device *vdev);

Re: [PATCH 4/5] VFIO: platform: populate reset function according to compat

[Alex Williamson]

On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
> Add the reset function lookup according to the device compat
> string. This lookup is added at different places:
> - on VFIO_DEVICE_GET_INFO
> - on VFIO_DEVICE_RESET
> - on device release
> 
> A reference to the module implementing the reset function is taken
> on first reset function lookup and released on vfio platform device
> release.
> 
> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> ---
>  drivers/vfio/platform/vfio_platform_common.c | 50 ++++++++++++++++++++++++++++
>  1 file changed, 50 insertions(+)
> 
> diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c
> index 0d10018..bd7e44c 100644
> --- a/drivers/vfio/platform/vfio_platform_common.c
> +++ b/drivers/vfio/platform/vfio_platform_common.c
> @@ -28,6 +28,52 @@ LIST_HEAD(reset_list);
>  
>  static DEFINE_MUTEX(driver_lock);
>  
> +static vfio_platform_reset_fn_t vfio_platform_lookup_reset(const char *compat)
> +{
> +	struct vfio_platform_reset_node *iter;
> +
> +	list_for_each_entry(iter, &reset_list, link) {

Racy

> +		if (!strcmp(iter->compat, compat) &&
> +			try_module_get(iter->owner))
> +			return iter->reset;
> +	}
> +
> +	return ERR_PTR(-ENODEV);

return NULL imo

> +}
> +
> +static vfio_platform_reset_fn_t vfio_platform_get_reset(
> +					struct vfio_platform_device *vdev)
> +{
> +	struct device *dev = vdev->get_device(vdev);
> +	const char *compat_str_array[2];
> +	vfio_platform_reset_fn_t reset;
> +	int ret;
> +
> +	if (!IS_ERR_OR_NULL(vdev->reset))
> +		return vdev->reset;
> +
> +	ret = device_property_read_string_array(dev, "compatible",
> +						compat_str_array, 2);
> +	if (!ret)
> +		return ERR_PTR(ret);
> +
> +	reset = vfio_platform_lookup_reset(compat_str_array[0]);
> +	return reset;

Something got allocated into compat_str_array and gets leaked here.

> +}
> +
> +static void vfio_platform_put_reset(struct vfio_platform_device *vdev)
> +{
> +	struct vfio_platform_reset_node *iter;
> +
> +	list_for_each_entry(iter, &reset_list, link) {

Racy

> +		if (iter->reset == vdev->reset) {
> +			module_put(iter->owner);
> +			vdev->reset = NULL;
> +			return;
> +		}
> +	}
> +}
> +
>  static int vfio_platform_regions_init(struct vfio_platform_device *vdev)
>  {
>  	int cnt = 0, i;
> @@ -103,10 +149,12 @@ static void vfio_platform_release(void *device_data)
>  	mutex_lock(&driver_lock);
>  
>  	if (!(--vdev->refcnt)) {
> +		vdev->reset = vfio_platform_get_reset(vdev);
>  		if (!IS_ERR_OR_NULL(vdev->reset))
>  			vdev->reset(vdev);
>  		vfio_platform_regions_cleanup(vdev);
>  		vfio_platform_irq_cleanup(vdev);
> +		vfio_platform_put_reset(vdev);
>  	}
>  
>  	mutex_unlock(&driver_lock);
> @@ -164,6 +212,7 @@ static long vfio_platform_ioctl(void *device_data,
>  		if (info.argsz < minsz)
>  			return -EINVAL;
>  
> +		vdev->reset = vfio_platform_get_reset(vdev);
>  		if (!IS_ERR_OR_NULL(vdev->reset))
>  			vdev->flags |= VFIO_DEVICE_FLAGS_RESET;
>  		info.flags = vdev->flags;
> @@ -260,6 +309,7 @@ static long vfio_platform_ioctl(void *device_data,
>  		return ret;
>  
>  	} else if (cmd == VFIO_DEVICE_RESET) {
> +		vdev->reset = vfio_platform_get_reset(vdev);
>  		if (!IS_ERR_OR_NULL(vdev->reset))
>  			return vdev->reset(vdev);
>  		else

I count 3 gets and 1 put, isn't the module reference count increase
showing that?  This looks like it hasn't been tested.  Why would we do a
get every time we want to do a reset?  Do one get when the device is
registered or opened and one put when the device is unregistered or
closed.  We don't want erratic userspace behavior that the reset
property of a device can come and go.

Re: [PATCH 4/5] VFIO: platform: populate reset function according to compat

[Alex Williamson]

On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
> Add the reset function lookup according to the device compat
> string. This lookup is added at different places:
> - on VFIO_DEVICE_GET_INFO
> - on VFIO_DEVICE_RESET
> - on device release
> 
> A reference to the module implementing the reset function is taken
> on first reset function lookup and released on vfio platform device
> release.
> 
> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> ---
>  drivers/vfio/platform/vfio_platform_common.c | 50 ++++++++++++++++++++++++++++
>  1 file changed, 50 insertions(+)
> 
> diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c
> index 0d10018..bd7e44c 100644
> --- a/drivers/vfio/platform/vfio_platform_common.c
> +++ b/drivers/vfio/platform/vfio_platform_common.c
> @@ -28,6 +28,52 @@ LIST_HEAD(reset_list);
>  
>  static DEFINE_MUTEX(driver_lock);
>  
> +static vfio_platform_reset_fn_t vfio_platform_lookup_reset(const char *compat)
> +{
> +	struct vfio_platform_reset_node *iter;
> +
> +	list_for_each_entry(iter, &reset_list, link) {

Racy

> +		if (!strcmp(iter->compat, compat) &&
> +			try_module_get(iter->owner))
> +			return iter->reset;
> +	}
> +
> +	return ERR_PTR(-ENODEV);

return NULL imo

> +}
> +
> +static vfio_platform_reset_fn_t vfio_platform_get_reset(
> +					struct vfio_platform_device *vdev)
> +{
> +	struct device *dev = vdev->get_device(vdev);
> +	const char *compat_str_array[2];
> +	vfio_platform_reset_fn_t reset;
> +	int ret;
> +
> +	if (!IS_ERR_OR_NULL(vdev->reset))
> +		return vdev->reset;
> +
> +	ret = device_property_read_string_array(dev, "compatible",
> +						compat_str_array, 2);
> +	if (!ret)
> +		return ERR_PTR(ret);
> +
> +	reset = vfio_platform_lookup_reset(compat_str_array[0]);
> +	return reset;

Something got allocated into compat_str_array and gets leaked here.

> +}
> +
> +static void vfio_platform_put_reset(struct vfio_platform_device *vdev)
> +{
> +	struct vfio_platform_reset_node *iter;
> +
> +	list_for_each_entry(iter, &reset_list, link) {

Racy

> +		if (iter->reset == vdev->reset) {
> +			module_put(iter->owner);
> +			vdev->reset = NULL;
> +			return;
> +		}
> +	}
> +}
> +
>  static int vfio_platform_regions_init(struct vfio_platform_device *vdev)
>  {
>  	int cnt = 0, i;
> @@ -103,10 +149,12 @@ static void vfio_platform_release(void *device_data)
>  	mutex_lock(&driver_lock);
>  
>  	if (!(--vdev->refcnt)) {
> +		vdev->reset = vfio_platform_get_reset(vdev);
>  		if (!IS_ERR_OR_NULL(vdev->reset))
>  			vdev->reset(vdev);
>  		vfio_platform_regions_cleanup(vdev);
>  		vfio_platform_irq_cleanup(vdev);
> +		vfio_platform_put_reset(vdev);
>  	}
>  
>  	mutex_unlock(&driver_lock);
> @@ -164,6 +212,7 @@ static long vfio_platform_ioctl(void *device_data,
>  		if (info.argsz < minsz)
>  			return -EINVAL;
>  
> +		vdev->reset = vfio_platform_get_reset(vdev);
>  		if (!IS_ERR_OR_NULL(vdev->reset))
>  			vdev->flags |= VFIO_DEVICE_FLAGS_RESET;
>  		info.flags = vdev->flags;
> @@ -260,6 +309,7 @@ static long vfio_platform_ioctl(void *device_data,
>  		return ret;
>  
>  	} else if (cmd == VFIO_DEVICE_RESET) {
> +		vdev->reset = vfio_platform_get_reset(vdev);
>  		if (!IS_ERR_OR_NULL(vdev->reset))
>  			return vdev->reset(vdev);
>  		else

I count 3 gets and 1 put, isn't the module reference count increase
showing that?  This looks like it hasn't been tested.  Why would we do a
get every time we want to do a reset?  Do one get when the device is
registered or opened and one put when the device is unregistered or
closed.  We don't want erratic userspace behavior that the reset
property of a device can come and go.

[PATCH 4/5] VFIO: platform: populate reset function according to compat

[Alex Williamson]

On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
> Add the reset function lookup according to the device compat
> string. This lookup is added at different places:
> - on VFIO_DEVICE_GET_INFO
> - on VFIO_DEVICE_RESET
> - on device release
> 
> A reference to the module implementing the reset function is taken
> on first reset function lookup and released on vfio platform device
> release.
> 
> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> ---
>  drivers/vfio/platform/vfio_platform_common.c | 50 ++++++++++++++++++++++++++++
>  1 file changed, 50 insertions(+)
> 
> diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c
> index 0d10018..bd7e44c 100644
> --- a/drivers/vfio/platform/vfio_platform_common.c
> +++ b/drivers/vfio/platform/vfio_platform_common.c
> @@ -28,6 +28,52 @@ LIST_HEAD(reset_list);
>  
>  static DEFINE_MUTEX(driver_lock);
>  
> +static vfio_platform_reset_fn_t vfio_platform_lookup_reset(const char *compat)
> +{
> +	struct vfio_platform_reset_node *iter;
> +
> +	list_for_each_entry(iter, &reset_list, link) {

Racy

> +		if (!strcmp(iter->compat, compat) &&
> +			try_module_get(iter->owner))
> +			return iter->reset;
> +	}
> +
> +	return ERR_PTR(-ENODEV);

return NULL imo

> +}
> +
> +static vfio_platform_reset_fn_t vfio_platform_get_reset(
> +					struct vfio_platform_device *vdev)
> +{
> +	struct device *dev = vdev->get_device(vdev);
> +	const char *compat_str_array[2];
> +	vfio_platform_reset_fn_t reset;
> +	int ret;
> +
> +	if (!IS_ERR_OR_NULL(vdev->reset))
> +		return vdev->reset;
> +
> +	ret = device_property_read_string_array(dev, "compatible",
> +						compat_str_array, 2);
> +	if (!ret)
> +		return ERR_PTR(ret);
> +
> +	reset = vfio_platform_lookup_reset(compat_str_array[0]);
> +	return reset;

Something got allocated into compat_str_array and gets leaked here.

> +}
> +
> +static void vfio_platform_put_reset(struct vfio_platform_device *vdev)
> +{
> +	struct vfio_platform_reset_node *iter;
> +
> +	list_for_each_entry(iter, &reset_list, link) {

Racy

> +		if (iter->reset == vdev->reset) {
> +			module_put(iter->owner);
> +			vdev->reset = NULL;
> +			return;
> +		}
> +	}
> +}
> +
>  static int vfio_platform_regions_init(struct vfio_platform_device *vdev)
>  {
>  	int cnt = 0, i;
> @@ -103,10 +149,12 @@ static void vfio_platform_release(void *device_data)
>  	mutex_lock(&driver_lock);
>  
>  	if (!(--vdev->refcnt)) {
> +		vdev->reset = vfio_platform_get_reset(vdev);
>  		if (!IS_ERR_OR_NULL(vdev->reset))
>  			vdev->reset(vdev);
>  		vfio_platform_regions_cleanup(vdev);
>  		vfio_platform_irq_cleanup(vdev);
> +		vfio_platform_put_reset(vdev);
>  	}
>  
>  	mutex_unlock(&driver_lock);
> @@ -164,6 +212,7 @@ static long vfio_platform_ioctl(void *device_data,
>  		if (info.argsz < minsz)
>  			return -EINVAL;
>  
> +		vdev->reset = vfio_platform_get_reset(vdev);
>  		if (!IS_ERR_OR_NULL(vdev->reset))
>  			vdev->flags |= VFIO_DEVICE_FLAGS_RESET;
>  		info.flags = vdev->flags;
> @@ -260,6 +309,7 @@ static long vfio_platform_ioctl(void *device_data,
>  		return ret;
>  
>  	} else if (cmd == VFIO_DEVICE_RESET) {
> +		vdev->reset = vfio_platform_get_reset(vdev);
>  		if (!IS_ERR_OR_NULL(vdev->reset))
>  			return vdev->reset(vdev);
>  		else

I count 3 gets and 1 put, isn't the module reference count increase
showing that?  This looks like it hasn't been tested.  Why would we do a
get every time we want to do a reset?  Do one get when the device is
registered or opened and one put when the device is unregistered or
closed.  We don't want erratic userspace behavior that the reset
property of a device can come and go.

Re: [PATCH 5/5] VFIO: platform: VFIO platform Calxeda xgmac reset module

[Alex Williamson]

On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
> This patch introduces a module that registers and implements a basic
> reset function for the Calxeda xgmac device. This latter basically disables
> interrupts and stops DMA transfers.
> 
> The reset function code is inherited from the native calxeda xgmac driver.
> 
> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> ---
>  drivers/vfio/platform/Kconfig                      |   2 +
>  drivers/vfio/platform/Makefile                     |   2 +
>  drivers/vfio/platform/reset/Kconfig                |   7 ++
>  drivers/vfio/platform/reset/Makefile               |   5 +
>  .../platform/reset/vfio_platform_calxedaxgmac.c    | 121 +++++++++++++++++++++
>  5 files changed, 137 insertions(+)
>  create mode 100644 drivers/vfio/platform/reset/Kconfig
>  create mode 100644 drivers/vfio/platform/reset/Makefile
>  create mode 100644 drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
> 
> diff --git a/drivers/vfio/platform/Kconfig b/drivers/vfio/platform/Kconfig
> index 9a4403e..1df7477 100644
> --- a/drivers/vfio/platform/Kconfig
> +++ b/drivers/vfio/platform/Kconfig
> @@ -18,3 +18,5 @@ config VFIO_AMBA
>  	  framework.
>  
>  	  If you don't know what to do here, say N.
> +
> +source "drivers/vfio/platform/reset/Kconfig"
> diff --git a/drivers/vfio/platform/Makefile b/drivers/vfio/platform/Makefile
> index 81de144..9ce8afe 100644
> --- a/drivers/vfio/platform/Makefile
> +++ b/drivers/vfio/platform/Makefile
> @@ -2,7 +2,9 @@
>  vfio-platform-y := vfio_platform.o vfio_platform_common.o vfio_platform_irq.o
>  
>  obj-$(CONFIG_VFIO_PLATFORM) += vfio-platform.o
> +obj-$(CONFIG_VFIO_PLATFORM) += reset/
>  
>  vfio-amba-y := vfio_amba.o
>  
>  obj-$(CONFIG_VFIO_AMBA) += vfio-amba.o
> +obj-$(CONFIG_VFIO_AMBA) += reset/
> diff --git a/drivers/vfio/platform/reset/Kconfig b/drivers/vfio/platform/reset/Kconfig
> new file mode 100644
> index 0000000..746b96b
> --- /dev/null
> +++ b/drivers/vfio/platform/reset/Kconfig
> @@ -0,0 +1,7 @@
> +config VFIO_PLATFORM_CALXEDAXGMAC_RESET
> +	tristate "VFIO support for calxeda xgmac reset"
> +	depends on VFIO_PLATFORM
> +	help
> +	  Enables the VFIO platform driver to handle reset for Calxeda xgmac
> +
> +	  If you don't know what to do here, say N.
> diff --git a/drivers/vfio/platform/reset/Makefile b/drivers/vfio/platform/reset/Makefile
> new file mode 100644
> index 0000000..2a486af
> --- /dev/null
> +++ b/drivers/vfio/platform/reset/Makefile
> @@ -0,0 +1,5 @@
> +vfio-platform-calxedaxgmac-y := vfio_platform_calxedaxgmac.o
> +
> +ccflags-y += -Idrivers/vfio/platform
> +
> +obj-$(CONFIG_VFIO_PLATFORM_CALXEDAXGMAC_RESET) += vfio-platform-calxedaxgmac.o
> diff --git a/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c b/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
> new file mode 100644
> index 0000000..3c6e428
> --- /dev/null
> +++ b/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
> @@ -0,0 +1,121 @@
> +/*
> + * VFIO platform driver specialized for Calxeda xgmac reset
> + * reset code is inherited from calxeda xgmac native driver
> + *
> + * Copyright 2010-2011 Calxeda, Inc.
> + * Copyright (c) 2015 Linaro Ltd.
> + *              www.linaro.org
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms and conditions of the GNU General Public License,
> + * version 2, as published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope it will be useful, but WITHOUT
> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
> + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
> + * more details.
> + *
> + * You should have received a copy of the GNU General Public License along with
> + * this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <linux/module.h>
> +#include <linux/kernel.h>
> +#include <linux/init.h>
> +#include <linux/io.h>
> +
> +#include "vfio_platform_private.h"
> +
> +#define DRIVER_VERSION  "0.1"
> +#define DRIVER_AUTHOR   "Eric Auger <eric.auger@linaro.org>"
> +#define DRIVER_DESC     "Reset support for Calxeda xgmac vfio platform device"
> +
> +#define CALXEDAXGMAC_COMPAT "calxeda,hb-xgmac"
> +
> +/* XGMAC Register definitions */
> +#define XGMAC_CONTROL           0x00000000      /* MAC Configuration */
> +
> +/* DMA Control and Status Registers */
> +#define XGMAC_DMA_CONTROL       0x00000f18      /* Ctrl (Operational Mode) */
> +#define XGMAC_DMA_INTR_ENA      0x00000f1c      /* Interrupt Enable */
> +
> +/* DMA Control registe defines */
> +#define DMA_CONTROL_ST          0x00002000      /* Start/Stop Transmission */
> +#define DMA_CONTROL_SR          0x00000002      /* Start/Stop Receive */
> +
> +/* Common MAC defines */
> +#define MAC_ENABLE_TX           0x00000008      /* Transmitter Enable */
> +#define MAC_ENABLE_RX           0x00000004      /* Receiver Enable */
> +
> +static inline void xgmac_mac_disable(void __iomem *ioaddr)
> +{
> +	u32 value = readl(ioaddr + XGMAC_DMA_CONTROL);
> +
> +	value &= ~(DMA_CONTROL_ST | DMA_CONTROL_SR);
> +	writel(value, ioaddr + XGMAC_DMA_CONTROL);
> +
> +	value = readl(ioaddr + XGMAC_CONTROL);
> +	value &= ~(MAC_ENABLE_TX | MAC_ENABLE_RX);
> +	writel(value, ioaddr + XGMAC_CONTROL);
> +}
> +
> +int vfio_platform_calxedaxgmac_reset(struct vfio_platform_device *vdev)
> +{
> +	struct vfio_platform_region reg = vdev->regions[0];
> +
> +	if (!reg.ioaddr) {
> +		reg.ioaddr =
> +			ioremap_nocache(reg.addr, reg.size);
> +		if (!reg.ioaddr)
> +			return -ENOMEM;
> +	}
> +
> +	/* disable IRQ */
> +	writel(0, reg.ioaddr + XGMAC_DMA_INTR_ENA);
> +
> +	/* Disable the MAC core */
> +	xgmac_mac_disable(reg.ioaddr);
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(vfio_platform_calxedaxgmac_reset);
> +
> +static int __init vfio_platform_calxedaxgmac_init(void)
> +{
> +	int (*register_reset)(char *, struct module *,
> +				vfio_platform_reset_fn_t);
> +	int ret;
> +
> +	register_reset = symbol_get(vfio_platform_register_reset);
> +	if (!register_reset)
> +		return -EINVAL;
> +
> +	ret = register_reset(CALXEDAXGMAC_COMPAT, THIS_MODULE,
> +			     vfio_platform_calxedaxgmac_reset);
> +
> +	symbol_put(vfio_platform_register_reset);
> +
> +	return ret;
> +}
> +
> +static void __exit vfio_platform_calxedaxgmac_exit(void)
> +{
> +	int (*unregister_reset)(char *);
> +	int ret;
> +
> +	unregister_reset = symbol_get(vfio_platform_unregister_reset);

Shouldn't we have gotten the symbol during the module init so that this
cannot fail?

> +	if (!unregister_reset)
> +		return;
> +
> +	ret = unregister_reset(CALXEDAXGMAC_COMPAT);

unregister_reset() should just return void

> +
> +	symbol_put(vfio_platform_unregister_reset);
> +}
> +
> +module_init(vfio_platform_calxedaxgmac_init);
> +module_exit(vfio_platform_calxedaxgmac_exit);
> +
> +MODULE_VERSION(DRIVER_VERSION);
> +MODULE_LICENSE("GPL v2");
> +MODULE_AUTHOR(DRIVER_AUTHOR);
> +MODULE_DESCRIPTION(DRIVER_DESC);


So a user needs to manually load this module to get a working reset for
this device?  That's never going to happen.

Re: [PATCH 5/5] VFIO: platform: VFIO platform Calxeda xgmac reset module

[Alex Williamson]

On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
> This patch introduces a module that registers and implements a basic
> reset function for the Calxeda xgmac device. This latter basically disables
> interrupts and stops DMA transfers.
> 
> The reset function code is inherited from the native calxeda xgmac driver.
> 
> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> ---
>  drivers/vfio/platform/Kconfig                      |   2 +
>  drivers/vfio/platform/Makefile                     |   2 +
>  drivers/vfio/platform/reset/Kconfig                |   7 ++
>  drivers/vfio/platform/reset/Makefile               |   5 +
>  .../platform/reset/vfio_platform_calxedaxgmac.c    | 121 +++++++++++++++++++++
>  5 files changed, 137 insertions(+)
>  create mode 100644 drivers/vfio/platform/reset/Kconfig
>  create mode 100644 drivers/vfio/platform/reset/Makefile
>  create mode 100644 drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
> 
> diff --git a/drivers/vfio/platform/Kconfig b/drivers/vfio/platform/Kconfig
> index 9a4403e..1df7477 100644
> --- a/drivers/vfio/platform/Kconfig
> +++ b/drivers/vfio/platform/Kconfig
> @@ -18,3 +18,5 @@ config VFIO_AMBA
>  	  framework.
>  
>  	  If you don't know what to do here, say N.
> +
> +source "drivers/vfio/platform/reset/Kconfig"
> diff --git a/drivers/vfio/platform/Makefile b/drivers/vfio/platform/Makefile
> index 81de144..9ce8afe 100644
> --- a/drivers/vfio/platform/Makefile
> +++ b/drivers/vfio/platform/Makefile
> @@ -2,7 +2,9 @@
>  vfio-platform-y := vfio_platform.o vfio_platform_common.o vfio_platform_irq.o
>  
>  obj-$(CONFIG_VFIO_PLATFORM) += vfio-platform.o
> +obj-$(CONFIG_VFIO_PLATFORM) += reset/
>  
>  vfio-amba-y := vfio_amba.o
>  
>  obj-$(CONFIG_VFIO_AMBA) += vfio-amba.o
> +obj-$(CONFIG_VFIO_AMBA) += reset/
> diff --git a/drivers/vfio/platform/reset/Kconfig b/drivers/vfio/platform/reset/Kconfig
> new file mode 100644
> index 0000000..746b96b
> --- /dev/null
> +++ b/drivers/vfio/platform/reset/Kconfig
> @@ -0,0 +1,7 @@
> +config VFIO_PLATFORM_CALXEDAXGMAC_RESET
> +	tristate "VFIO support for calxeda xgmac reset"
> +	depends on VFIO_PLATFORM
> +	help
> +	  Enables the VFIO platform driver to handle reset for Calxeda xgmac
> +
> +	  If you don't know what to do here, say N.
> diff --git a/drivers/vfio/platform/reset/Makefile b/drivers/vfio/platform/reset/Makefile
> new file mode 100644
> index 0000000..2a486af
> --- /dev/null
> +++ b/drivers/vfio/platform/reset/Makefile
> @@ -0,0 +1,5 @@
> +vfio-platform-calxedaxgmac-y := vfio_platform_calxedaxgmac.o
> +
> +ccflags-y += -Idrivers/vfio/platform
> +
> +obj-$(CONFIG_VFIO_PLATFORM_CALXEDAXGMAC_RESET) += vfio-platform-calxedaxgmac.o
> diff --git a/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c b/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
> new file mode 100644
> index 0000000..3c6e428
> --- /dev/null
> +++ b/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
> @@ -0,0 +1,121 @@
> +/*
> + * VFIO platform driver specialized for Calxeda xgmac reset
> + * reset code is inherited from calxeda xgmac native driver
> + *
> + * Copyright 2010-2011 Calxeda, Inc.
> + * Copyright (c) 2015 Linaro Ltd.
> + *              www.linaro.org
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms and conditions of the GNU General Public License,
> + * version 2, as published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope it will be useful, but WITHOUT
> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
> + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
> + * more details.
> + *
> + * You should have received a copy of the GNU General Public License along with
> + * this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <linux/module.h>
> +#include <linux/kernel.h>
> +#include <linux/init.h>
> +#include <linux/io.h>
> +
> +#include "vfio_platform_private.h"
> +
> +#define DRIVER_VERSION  "0.1"
> +#define DRIVER_AUTHOR   "Eric Auger <eric.auger@linaro.org>"
> +#define DRIVER_DESC     "Reset support for Calxeda xgmac vfio platform device"
> +
> +#define CALXEDAXGMAC_COMPAT "calxeda,hb-xgmac"
> +
> +/* XGMAC Register definitions */
> +#define XGMAC_CONTROL           0x00000000      /* MAC Configuration */
> +
> +/* DMA Control and Status Registers */
> +#define XGMAC_DMA_CONTROL       0x00000f18      /* Ctrl (Operational Mode) */
> +#define XGMAC_DMA_INTR_ENA      0x00000f1c      /* Interrupt Enable */
> +
> +/* DMA Control registe defines */
> +#define DMA_CONTROL_ST          0x00002000      /* Start/Stop Transmission */
> +#define DMA_CONTROL_SR          0x00000002      /* Start/Stop Receive */
> +
> +/* Common MAC defines */
> +#define MAC_ENABLE_TX           0x00000008      /* Transmitter Enable */
> +#define MAC_ENABLE_RX           0x00000004      /* Receiver Enable */
> +
> +static inline void xgmac_mac_disable(void __iomem *ioaddr)
> +{
> +	u32 value = readl(ioaddr + XGMAC_DMA_CONTROL);
> +
> +	value &= ~(DMA_CONTROL_ST | DMA_CONTROL_SR);
> +	writel(value, ioaddr + XGMAC_DMA_CONTROL);
> +
> +	value = readl(ioaddr + XGMAC_CONTROL);
> +	value &= ~(MAC_ENABLE_TX | MAC_ENABLE_RX);
> +	writel(value, ioaddr + XGMAC_CONTROL);
> +}
> +
> +int vfio_platform_calxedaxgmac_reset(struct vfio_platform_device *vdev)
> +{
> +	struct vfio_platform_region reg = vdev->regions[0];
> +
> +	if (!reg.ioaddr) {
> +		reg.ioaddr =
> +			ioremap_nocache(reg.addr, reg.size);
> +		if (!reg.ioaddr)
> +			return -ENOMEM;
> +	}
> +
> +	/* disable IRQ */
> +	writel(0, reg.ioaddr + XGMAC_DMA_INTR_ENA);
> +
> +	/* Disable the MAC core */
> +	xgmac_mac_disable(reg.ioaddr);
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(vfio_platform_calxedaxgmac_reset);
> +
> +static int __init vfio_platform_calxedaxgmac_init(void)
> +{
> +	int (*register_reset)(char *, struct module *,
> +				vfio_platform_reset_fn_t);
> +	int ret;
> +
> +	register_reset = symbol_get(vfio_platform_register_reset);
> +	if (!register_reset)
> +		return -EINVAL;
> +
> +	ret = register_reset(CALXEDAXGMAC_COMPAT, THIS_MODULE,
> +			     vfio_platform_calxedaxgmac_reset);
> +
> +	symbol_put(vfio_platform_register_reset);
> +
> +	return ret;
> +}
> +
> +static void __exit vfio_platform_calxedaxgmac_exit(void)
> +{
> +	int (*unregister_reset)(char *);
> +	int ret;
> +
> +	unregister_reset = symbol_get(vfio_platform_unregister_reset);

Shouldn't we have gotten the symbol during the module init so that this
cannot fail?

> +	if (!unregister_reset)
> +		return;
> +
> +	ret = unregister_reset(CALXEDAXGMAC_COMPAT);

unregister_reset() should just return void

> +
> +	symbol_put(vfio_platform_unregister_reset);
> +}
> +
> +module_init(vfio_platform_calxedaxgmac_init);
> +module_exit(vfio_platform_calxedaxgmac_exit);
> +
> +MODULE_VERSION(DRIVER_VERSION);
> +MODULE_LICENSE("GPL v2");
> +MODULE_AUTHOR(DRIVER_AUTHOR);
> +MODULE_DESCRIPTION(DRIVER_DESC);


So a user needs to manually load this module to get a working reset for
this device?  That's never going to happen.

[PATCH 5/5] VFIO: platform: VFIO platform Calxeda xgmac reset module

[Alex Williamson]

On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
> This patch introduces a module that registers and implements a basic
> reset function for the Calxeda xgmac device. This latter basically disables
> interrupts and stops DMA transfers.
> 
> The reset function code is inherited from the native calxeda xgmac driver.
> 
> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> ---
>  drivers/vfio/platform/Kconfig                      |   2 +
>  drivers/vfio/platform/Makefile                     |   2 +
>  drivers/vfio/platform/reset/Kconfig                |   7 ++
>  drivers/vfio/platform/reset/Makefile               |   5 +
>  .../platform/reset/vfio_platform_calxedaxgmac.c    | 121 +++++++++++++++++++++
>  5 files changed, 137 insertions(+)
>  create mode 100644 drivers/vfio/platform/reset/Kconfig
>  create mode 100644 drivers/vfio/platform/reset/Makefile
>  create mode 100644 drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
> 
> diff --git a/drivers/vfio/platform/Kconfig b/drivers/vfio/platform/Kconfig
> index 9a4403e..1df7477 100644
> --- a/drivers/vfio/platform/Kconfig
> +++ b/drivers/vfio/platform/Kconfig
> @@ -18,3 +18,5 @@ config VFIO_AMBA
>  	  framework.
>  
>  	  If you don't know what to do here, say N.
> +
> +source "drivers/vfio/platform/reset/Kconfig"
> diff --git a/drivers/vfio/platform/Makefile b/drivers/vfio/platform/Makefile
> index 81de144..9ce8afe 100644
> --- a/drivers/vfio/platform/Makefile
> +++ b/drivers/vfio/platform/Makefile
> @@ -2,7 +2,9 @@
>  vfio-platform-y := vfio_platform.o vfio_platform_common.o vfio_platform_irq.o
>  
>  obj-$(CONFIG_VFIO_PLATFORM) += vfio-platform.o
> +obj-$(CONFIG_VFIO_PLATFORM) += reset/
>  
>  vfio-amba-y := vfio_amba.o
>  
>  obj-$(CONFIG_VFIO_AMBA) += vfio-amba.o
> +obj-$(CONFIG_VFIO_AMBA) += reset/
> diff --git a/drivers/vfio/platform/reset/Kconfig b/drivers/vfio/platform/reset/Kconfig
> new file mode 100644
> index 0000000..746b96b
> --- /dev/null
> +++ b/drivers/vfio/platform/reset/Kconfig
> @@ -0,0 +1,7 @@
> +config VFIO_PLATFORM_CALXEDAXGMAC_RESET
> +	tristate "VFIO support for calxeda xgmac reset"
> +	depends on VFIO_PLATFORM
> +	help
> +	  Enables the VFIO platform driver to handle reset for Calxeda xgmac
> +
> +	  If you don't know what to do here, say N.
> diff --git a/drivers/vfio/platform/reset/Makefile b/drivers/vfio/platform/reset/Makefile
> new file mode 100644
> index 0000000..2a486af
> --- /dev/null
> +++ b/drivers/vfio/platform/reset/Makefile
> @@ -0,0 +1,5 @@
> +vfio-platform-calxedaxgmac-y := vfio_platform_calxedaxgmac.o
> +
> +ccflags-y += -Idrivers/vfio/platform
> +
> +obj-$(CONFIG_VFIO_PLATFORM_CALXEDAXGMAC_RESET) += vfio-platform-calxedaxgmac.o
> diff --git a/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c b/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
> new file mode 100644
> index 0000000..3c6e428
> --- /dev/null
> +++ b/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
> @@ -0,0 +1,121 @@
> +/*
> + * VFIO platform driver specialized for Calxeda xgmac reset
> + * reset code is inherited from calxeda xgmac native driver
> + *
> + * Copyright 2010-2011 Calxeda, Inc.
> + * Copyright (c) 2015 Linaro Ltd.
> + *              www.linaro.org
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms and conditions of the GNU General Public License,
> + * version 2, as published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope it will be useful, but WITHOUT
> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
> + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
> + * more details.
> + *
> + * You should have received a copy of the GNU General Public License along with
> + * this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <linux/module.h>
> +#include <linux/kernel.h>
> +#include <linux/init.h>
> +#include <linux/io.h>
> +
> +#include "vfio_platform_private.h"
> +
> +#define DRIVER_VERSION  "0.1"
> +#define DRIVER_AUTHOR   "Eric Auger <eric.auger@linaro.org>"
> +#define DRIVER_DESC     "Reset support for Calxeda xgmac vfio platform device"
> +
> +#define CALXEDAXGMAC_COMPAT "calxeda,hb-xgmac"
> +
> +/* XGMAC Register definitions */
> +#define XGMAC_CONTROL           0x00000000      /* MAC Configuration */
> +
> +/* DMA Control and Status Registers */
> +#define XGMAC_DMA_CONTROL       0x00000f18      /* Ctrl (Operational Mode) */
> +#define XGMAC_DMA_INTR_ENA      0x00000f1c      /* Interrupt Enable */
> +
> +/* DMA Control registe defines */
> +#define DMA_CONTROL_ST          0x00002000      /* Start/Stop Transmission */
> +#define DMA_CONTROL_SR          0x00000002      /* Start/Stop Receive */
> +
> +/* Common MAC defines */
> +#define MAC_ENABLE_TX           0x00000008      /* Transmitter Enable */
> +#define MAC_ENABLE_RX           0x00000004      /* Receiver Enable */
> +
> +static inline void xgmac_mac_disable(void __iomem *ioaddr)
> +{
> +	u32 value = readl(ioaddr + XGMAC_DMA_CONTROL);
> +
> +	value &= ~(DMA_CONTROL_ST | DMA_CONTROL_SR);
> +	writel(value, ioaddr + XGMAC_DMA_CONTROL);
> +
> +	value = readl(ioaddr + XGMAC_CONTROL);
> +	value &= ~(MAC_ENABLE_TX | MAC_ENABLE_RX);
> +	writel(value, ioaddr + XGMAC_CONTROL);
> +}
> +
> +int vfio_platform_calxedaxgmac_reset(struct vfio_platform_device *vdev)
> +{
> +	struct vfio_platform_region reg = vdev->regions[0];
> +
> +	if (!reg.ioaddr) {
> +		reg.ioaddr =
> +			ioremap_nocache(reg.addr, reg.size);
> +		if (!reg.ioaddr)
> +			return -ENOMEM;
> +	}
> +
> +	/* disable IRQ */
> +	writel(0, reg.ioaddr + XGMAC_DMA_INTR_ENA);
> +
> +	/* Disable the MAC core */
> +	xgmac_mac_disable(reg.ioaddr);
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(vfio_platform_calxedaxgmac_reset);
> +
> +static int __init vfio_platform_calxedaxgmac_init(void)
> +{
> +	int (*register_reset)(char *, struct module *,
> +				vfio_platform_reset_fn_t);
> +	int ret;
> +
> +	register_reset = symbol_get(vfio_platform_register_reset);
> +	if (!register_reset)
> +		return -EINVAL;
> +
> +	ret = register_reset(CALXEDAXGMAC_COMPAT, THIS_MODULE,
> +			     vfio_platform_calxedaxgmac_reset);
> +
> +	symbol_put(vfio_platform_register_reset);
> +
> +	return ret;
> +}
> +
> +static void __exit vfio_platform_calxedaxgmac_exit(void)
> +{
> +	int (*unregister_reset)(char *);
> +	int ret;
> +
> +	unregister_reset = symbol_get(vfio_platform_unregister_reset);

Shouldn't we have gotten the symbol during the module init so that this
cannot fail?

> +	if (!unregister_reset)
> +		return;
> +
> +	ret = unregister_reset(CALXEDAXGMAC_COMPAT);

unregister_reset() should just return void

> +
> +	symbol_put(vfio_platform_unregister_reset);
> +}
> +
> +module_init(vfio_platform_calxedaxgmac_init);
> +module_exit(vfio_platform_calxedaxgmac_exit);
> +
> +MODULE_VERSION(DRIVER_VERSION);
> +MODULE_LICENSE("GPL v2");
> +MODULE_AUTHOR(DRIVER_AUTHOR);
> +MODULE_DESCRIPTION(DRIVER_DESC);


So a user needs to manually load this module to get a working reset for
this device?  That's never going to happen.

Re: [PATCH 1/5] VFIO: platform: add reset_list and register/unregister functions

[Eric Auger]

Hi Alex,
On 05/13/2015 08:32 PM, Alex Williamson wrote:
> On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
>> vfio_platform_common now stores a lists of available reset functions.
>> Two functions are exposed to register/unregister a reset function. A
>> reset function is paired with a compat string.
>>
>> Signed-off-by: Eric Auger <eric.auger@linaro.org>
>> ---
>>  drivers/vfio/platform/vfio_platform_common.c  | 63 +++++++++++++++++++++++++++
>>  drivers/vfio/platform/vfio_platform_private.h | 13 ++++++
>>  2 files changed, 76 insertions(+)
>>
>> diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c
>> index abcff7a..edbf24c 100644
>> --- a/drivers/vfio/platform/vfio_platform_common.c
>> +++ b/drivers/vfio/platform/vfio_platform_common.c
>> @@ -23,6 +23,9 @@
>>  
>>  #include "vfio_platform_private.h"
>>  
>> +struct list_head reset_list;
>> +LIST_HEAD(reset_list);
>> +
> 
> Redundant?  Static?
static, yes
> 
>>  static DEFINE_MUTEX(driver_lock);
>>  
>>  static int vfio_platform_regions_init(struct vfio_platform_device *vdev)
>> @@ -511,6 +514,13 @@ EXPORT_SYMBOL_GPL(vfio_platform_probe_common);
>>  struct vfio_platform_device *vfio_platform_remove_common(struct device *dev)
>>  {
>>  	struct vfio_platform_device *vdev;
>> +	struct vfio_platform_reset_node *iter, *tmp;
>> +
>> +	list_for_each_entry_safe(iter, tmp, &reset_list, link) {
>> +		list_del(&iter->link);
>> +		kfree(iter->compat);
>> +		kfree(iter);
>> +	}
> 
> 
> This doesn't make sense.  We allow reset functions to be registered and
> unregistered, but we forget them all when any device is released?!
I acknowledge indeed. Can I rely on the reset module exit and associated
unregister_reset or shall I take this action in the vfio driver itself,
core?
> 
>>  
>>  	vdev = vfio_del_group_dev(dev);
>>  	if (vdev)
>> @@ -519,3 +529,56 @@ struct vfio_platform_device *vfio_platform_remove_common(struct device *dev)
>>  	return vdev;
>>  }
>>  EXPORT_SYMBOL_GPL(vfio_platform_remove_common);
>> +
>> +int vfio_platform_register_reset(char *compat, struct module *reset_owner,
>> +				 vfio_platform_reset_fn_t reset)
>> +{
>> +	struct vfio_platform_reset_node *node, *iter;
>> +	bool found = false;
>> +
>> +	list_for_each_entry(iter, &reset_list, link) {
>> +		if (!strcmp(iter->compat, compat)) {
>> +			found = true;
> 
> Just return errno here
ok
> 
>> +			break;
>> +		}
>> +	}
>> +	if (found)
>> +		return -EINVAL;
>> +
>> +	node = kmalloc(sizeof(*node), GFP_KERNEL);
>> +	if (!node)
>> +		return -ENOMEM;
>> +
>> +	node->compat = kstrdup(compat, GFP_KERNEL);
>> +	if (!node->compat)
> 
> Leaking node
ok
> 
>> +		return -ENOMEM;
>> +
>> +	node->owner = reset_owner;
>> +	node->reset = reset;
>> +
>> +	list_add(&node->link, &reset_list);
> 
> Isn't this racy?  Don't we need some locks around the list?
I will add a lock to protect access to the list.
> 
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(vfio_platform_register_reset);
>> +
>> +int vfio_platform_unregister_reset(char *compat)
>> +{
>> +	struct vfio_platform_reset_node *iter;
>> +	bool found = false;
>> +
>> +	list_for_each_entry(iter, &reset_list, link) {
>> +		if (!strcmp(iter->compat, compat)) {
> 
> Return errno here
ok
> 
>> +			found = true;
>> +			break;
>> +		}
>> +	}
>> +	if (!found)
>> +		return -EINVAL;
>> +
>> +	list_del(&iter->link);
> 
> Racy
> 
>> +	kfree(iter->compat);
>> +	kfree(iter);
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(vfio_platform_unregister_reset);
>> +
>> diff --git a/drivers/vfio/platform/vfio_platform_private.h b/drivers/vfio/platform/vfio_platform_private.h
>> index 5d31e04..da2d60b 100644
>> --- a/drivers/vfio/platform/vfio_platform_private.h
>> +++ b/drivers/vfio/platform/vfio_platform_private.h
>> @@ -69,6 +69,15 @@ struct vfio_platform_device {
>>  	int	(*get_irq)(struct vfio_platform_device *vdev, int i);
>>  };
>>  
>> +typedef int (*vfio_platform_reset_fn_t)(struct vfio_platform_device *vdev);
> 
> Seems like this ought to be in a non-private header if we're exporting
> the [un]register functions.
I considered the vfio reset modules were internal to the vfio subsystem
but if you prefer I can expose that in vfio.h. I guess
register/unregister should become an external API then?

Thanks

Eric
>> +
>> +struct vfio_platform_reset_node {
>> +	struct list_head link;
>> +	char *compat;
>> +	struct module *owner;
>> +	vfio_platform_reset_fn_t reset;
>> +};
>> +
>>  extern int vfio_platform_probe_common(struct vfio_platform_device *vdev,
>>  				      struct device *dev);
>>  extern struct vfio_platform_device *vfio_platform_remove_common
>> @@ -82,4 +91,8 @@ extern int vfio_platform_set_irqs_ioctl(struct vfio_platform_device *vdev,
>>  					unsigned start, unsigned count,
>>  					void *data);
>>  
>> +extern int vfio_platform_register_reset(char *compat, struct module *owner,
>> +					vfio_platform_reset_fn_t reset);
>> +extern int vfio_platform_unregister_reset(char *compat);
>> +
>>  #endif /* VFIO_PLATFORM_PRIVATE_H */
> 
> 
> 

Re: [PATCH 1/5] VFIO: platform: add reset_list and register/unregister functions

[Eric Auger]

Hi Alex,
On 05/13/2015 08:32 PM, Alex Williamson wrote:
> On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
>> vfio_platform_common now stores a lists of available reset functions.
>> Two functions are exposed to register/unregister a reset function. A
>> reset function is paired with a compat string.
>>
>> Signed-off-by: Eric Auger <eric.auger@linaro.org>
>> ---
>>  drivers/vfio/platform/vfio_platform_common.c  | 63 +++++++++++++++++++++++++++
>>  drivers/vfio/platform/vfio_platform_private.h | 13 ++++++
>>  2 files changed, 76 insertions(+)
>>
>> diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c
>> index abcff7a..edbf24c 100644
>> --- a/drivers/vfio/platform/vfio_platform_common.c
>> +++ b/drivers/vfio/platform/vfio_platform_common.c
>> @@ -23,6 +23,9 @@
>>  
>>  #include "vfio_platform_private.h"
>>  
>> +struct list_head reset_list;
>> +LIST_HEAD(reset_list);
>> +
> 
> Redundant?  Static?
static, yes
> 
>>  static DEFINE_MUTEX(driver_lock);
>>  
>>  static int vfio_platform_regions_init(struct vfio_platform_device *vdev)
>> @@ -511,6 +514,13 @@ EXPORT_SYMBOL_GPL(vfio_platform_probe_common);
>>  struct vfio_platform_device *vfio_platform_remove_common(struct device *dev)
>>  {
>>  	struct vfio_platform_device *vdev;
>> +	struct vfio_platform_reset_node *iter, *tmp;
>> +
>> +	list_for_each_entry_safe(iter, tmp, &reset_list, link) {
>> +		list_del(&iter->link);
>> +		kfree(iter->compat);
>> +		kfree(iter);
>> +	}
> 
> 
> This doesn't make sense.  We allow reset functions to be registered and
> unregistered, but we forget them all when any device is released?!
I acknowledge indeed. Can I rely on the reset module exit and associated
unregister_reset or shall I take this action in the vfio driver itself,
core?
> 
>>  
>>  	vdev = vfio_del_group_dev(dev);
>>  	if (vdev)
>> @@ -519,3 +529,56 @@ struct vfio_platform_device *vfio_platform_remove_common(struct device *dev)
>>  	return vdev;
>>  }
>>  EXPORT_SYMBOL_GPL(vfio_platform_remove_common);
>> +
>> +int vfio_platform_register_reset(char *compat, struct module *reset_owner,
>> +				 vfio_platform_reset_fn_t reset)
>> +{
>> +	struct vfio_platform_reset_node *node, *iter;
>> +	bool found = false;
>> +
>> +	list_for_each_entry(iter, &reset_list, link) {
>> +		if (!strcmp(iter->compat, compat)) {
>> +			found = true;
> 
> Just return errno here
ok
> 
>> +			break;
>> +		}
>> +	}
>> +	if (found)
>> +		return -EINVAL;
>> +
>> +	node = kmalloc(sizeof(*node), GFP_KERNEL);
>> +	if (!node)
>> +		return -ENOMEM;
>> +
>> +	node->compat = kstrdup(compat, GFP_KERNEL);
>> +	if (!node->compat)
> 
> Leaking node
ok
> 
>> +		return -ENOMEM;
>> +
>> +	node->owner = reset_owner;
>> +	node->reset = reset;
>> +
>> +	list_add(&node->link, &reset_list);
> 
> Isn't this racy?  Don't we need some locks around the list?
I will add a lock to protect access to the list.
> 
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(vfio_platform_register_reset);
>> +
>> +int vfio_platform_unregister_reset(char *compat)
>> +{
>> +	struct vfio_platform_reset_node *iter;
>> +	bool found = false;
>> +
>> +	list_for_each_entry(iter, &reset_list, link) {
>> +		if (!strcmp(iter->compat, compat)) {
> 
> Return errno here
ok
> 
>> +			found = true;
>> +			break;
>> +		}
>> +	}
>> +	if (!found)
>> +		return -EINVAL;
>> +
>> +	list_del(&iter->link);
> 
> Racy
> 
>> +	kfree(iter->compat);
>> +	kfree(iter);
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(vfio_platform_unregister_reset);
>> +
>> diff --git a/drivers/vfio/platform/vfio_platform_private.h b/drivers/vfio/platform/vfio_platform_private.h
>> index 5d31e04..da2d60b 100644
>> --- a/drivers/vfio/platform/vfio_platform_private.h
>> +++ b/drivers/vfio/platform/vfio_platform_private.h
>> @@ -69,6 +69,15 @@ struct vfio_platform_device {
>>  	int	(*get_irq)(struct vfio_platform_device *vdev, int i);
>>  };
>>  
>> +typedef int (*vfio_platform_reset_fn_t)(struct vfio_platform_device *vdev);
> 
> Seems like this ought to be in a non-private header if we're exporting
> the [un]register functions.
I considered the vfio reset modules were internal to the vfio subsystem
but if you prefer I can expose that in vfio.h. I guess
register/unregister should become an external API then?

Thanks

Eric
>> +
>> +struct vfio_platform_reset_node {
>> +	struct list_head link;
>> +	char *compat;
>> +	struct module *owner;
>> +	vfio_platform_reset_fn_t reset;
>> +};
>> +
>>  extern int vfio_platform_probe_common(struct vfio_platform_device *vdev,
>>  				      struct device *dev);
>>  extern struct vfio_platform_device *vfio_platform_remove_common
>> @@ -82,4 +91,8 @@ extern int vfio_platform_set_irqs_ioctl(struct vfio_platform_device *vdev,
>>  					unsigned start, unsigned count,
>>  					void *data);
>>  
>> +extern int vfio_platform_register_reset(char *compat, struct module *owner,
>> +					vfio_platform_reset_fn_t reset);
>> +extern int vfio_platform_unregister_reset(char *compat);
>> +
>>  #endif /* VFIO_PLATFORM_PRIVATE_H */
> 
> 
> 

[PATCH 1/5] VFIO: platform: add reset_list and register/unregister functions

[Eric Auger]

Hi Alex,
On 05/13/2015 08:32 PM, Alex Williamson wrote:
> On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
>> vfio_platform_common now stores a lists of available reset functions.
>> Two functions are exposed to register/unregister a reset function. A
>> reset function is paired with a compat string.
>>
>> Signed-off-by: Eric Auger <eric.auger@linaro.org>
>> ---
>>  drivers/vfio/platform/vfio_platform_common.c  | 63 +++++++++++++++++++++++++++
>>  drivers/vfio/platform/vfio_platform_private.h | 13 ++++++
>>  2 files changed, 76 insertions(+)
>>
>> diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c
>> index abcff7a..edbf24c 100644
>> --- a/drivers/vfio/platform/vfio_platform_common.c
>> +++ b/drivers/vfio/platform/vfio_platform_common.c
>> @@ -23,6 +23,9 @@
>>  
>>  #include "vfio_platform_private.h"
>>  
>> +struct list_head reset_list;
>> +LIST_HEAD(reset_list);
>> +
> 
> Redundant?  Static?
static, yes
> 
>>  static DEFINE_MUTEX(driver_lock);
>>  
>>  static int vfio_platform_regions_init(struct vfio_platform_device *vdev)
>> @@ -511,6 +514,13 @@ EXPORT_SYMBOL_GPL(vfio_platform_probe_common);
>>  struct vfio_platform_device *vfio_platform_remove_common(struct device *dev)
>>  {
>>  	struct vfio_platform_device *vdev;
>> +	struct vfio_platform_reset_node *iter, *tmp;
>> +
>> +	list_for_each_entry_safe(iter, tmp, &reset_list, link) {
>> +		list_del(&iter->link);
>> +		kfree(iter->compat);
>> +		kfree(iter);
>> +	}
> 
> 
> This doesn't make sense.  We allow reset functions to be registered and
> unregistered, but we forget them all when any device is released?!
I acknowledge indeed. Can I rely on the reset module exit and associated
unregister_reset or shall I take this action in the vfio driver itself,
core?
> 
>>  
>>  	vdev = vfio_del_group_dev(dev);
>>  	if (vdev)
>> @@ -519,3 +529,56 @@ struct vfio_platform_device *vfio_platform_remove_common(struct device *dev)
>>  	return vdev;
>>  }
>>  EXPORT_SYMBOL_GPL(vfio_platform_remove_common);
>> +
>> +int vfio_platform_register_reset(char *compat, struct module *reset_owner,
>> +				 vfio_platform_reset_fn_t reset)
>> +{
>> +	struct vfio_platform_reset_node *node, *iter;
>> +	bool found = false;
>> +
>> +	list_for_each_entry(iter, &reset_list, link) {
>> +		if (!strcmp(iter->compat, compat)) {
>> +			found = true;
> 
> Just return errno here
ok
> 
>> +			break;
>> +		}
>> +	}
>> +	if (found)
>> +		return -EINVAL;
>> +
>> +	node = kmalloc(sizeof(*node), GFP_KERNEL);
>> +	if (!node)
>> +		return -ENOMEM;
>> +
>> +	node->compat = kstrdup(compat, GFP_KERNEL);
>> +	if (!node->compat)
> 
> Leaking node
ok
> 
>> +		return -ENOMEM;
>> +
>> +	node->owner = reset_owner;
>> +	node->reset = reset;
>> +
>> +	list_add(&node->link, &reset_list);
> 
> Isn't this racy?  Don't we need some locks around the list?
I will add a lock to protect access to the list.
> 
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(vfio_platform_register_reset);
>> +
>> +int vfio_platform_unregister_reset(char *compat)
>> +{
>> +	struct vfio_platform_reset_node *iter;
>> +	bool found = false;
>> +
>> +	list_for_each_entry(iter, &reset_list, link) {
>> +		if (!strcmp(iter->compat, compat)) {
> 
> Return errno here
ok
> 
>> +			found = true;
>> +			break;
>> +		}
>> +	}
>> +	if (!found)
>> +		return -EINVAL;
>> +
>> +	list_del(&iter->link);
> 
> Racy
> 
>> +	kfree(iter->compat);
>> +	kfree(iter);
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(vfio_platform_unregister_reset);
>> +
>> diff --git a/drivers/vfio/platform/vfio_platform_private.h b/drivers/vfio/platform/vfio_platform_private.h
>> index 5d31e04..da2d60b 100644
>> --- a/drivers/vfio/platform/vfio_platform_private.h
>> +++ b/drivers/vfio/platform/vfio_platform_private.h
>> @@ -69,6 +69,15 @@ struct vfio_platform_device {
>>  	int	(*get_irq)(struct vfio_platform_device *vdev, int i);
>>  };
>>  
>> +typedef int (*vfio_platform_reset_fn_t)(struct vfio_platform_device *vdev);
> 
> Seems like this ought to be in a non-private header if we're exporting
> the [un]register functions.
I considered the vfio reset modules were internal to the vfio subsystem
but if you prefer I can expose that in vfio.h. I guess
register/unregister should become an external API then?

Thanks

Eric
>> +
>> +struct vfio_platform_reset_node {
>> +	struct list_head link;
>> +	char *compat;
>> +	struct module *owner;
>> +	vfio_platform_reset_fn_t reset;
>> +};
>> +
>>  extern int vfio_platform_probe_common(struct vfio_platform_device *vdev,
>>  				      struct device *dev);
>>  extern struct vfio_platform_device *vfio_platform_remove_common
>> @@ -82,4 +91,8 @@ extern int vfio_platform_set_irqs_ioctl(struct vfio_platform_device *vdev,
>>  					unsigned start, unsigned count,
>>  					void *data);
>>  
>> +extern int vfio_platform_register_reset(char *compat, struct module *owner,
>> +					vfio_platform_reset_fn_t reset);
>> +extern int vfio_platform_unregister_reset(char *compat);
>> +
>>  #endif /* VFIO_PLATFORM_PRIVATE_H */
> 
> 
> 

Re: [PATCH 2/5] VFIO: platform: add get_device callback

[Eric Auger]

On 05/13/2015 08:32 PM, Alex Williamson wrote:
> On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
>> It is needed to introduce a new callback enabling to retrieve the
>> struct device* from the vfio_platform_device. Implementation depends
>> on the underlying device, platform or amba. This will be used to retrieve
>> the compatibility string of the device.
>>
>> Signed-off-by: Eric Auger <eric.auger@linaro.org>
>> ---
>>  drivers/vfio/platform/vfio_amba.c             |  9 +++++++++
>>  drivers/vfio/platform/vfio_platform.c         | 10 ++++++++++
>>  drivers/vfio/platform/vfio_platform_private.h |  1 +
>>  3 files changed, 20 insertions(+)
>>
>> diff --git a/drivers/vfio/platform/vfio_amba.c b/drivers/vfio/platform/vfio_amba.c
>> index ff0331f..fd68115 100644
>> --- a/drivers/vfio/platform/vfio_amba.c
>> +++ b/drivers/vfio/platform/vfio_amba.c
>> @@ -48,6 +48,14 @@ static int get_amba_irq(struct vfio_platform_device *vdev, int i)
>>  	return ret ? ret : -ENXIO;
>>  }
>>  
>> +static struct device *vfio_amba_get_device(struct vfio_platform_device *vdev)
>> +{
>> +	struct amba_device *pdev = (struct amba_device *) vdev->opaque;
>> +	struct device *dev = &pdev->dev;
>> +
>> +	return dev;
>> +}
> 
> *get* device implies a increment in the reference count.  There's
> nothing like that happening here.  Do you just want a to_device wrapper?
Yes you already told that in the past and I should have learnt ...
> Why doesn't struct vfio_platform_device just cache a pointer to the
> struct device?
simpler indeed if agreed.
> 
>> +
>>  static int vfio_amba_probe(struct amba_device *adev, const struct amba_id *id)
>>  {
>>  	struct vfio_platform_device *vdev;
>> @@ -67,6 +75,7 @@ static int vfio_amba_probe(struct amba_device *adev, const struct amba_id *id)
>>  	vdev->flags = VFIO_DEVICE_FLAGS_AMBA;
>>  	vdev->get_resource = get_amba_resource;
>>  	vdev->get_irq = get_amba_irq;
>> +	vdev->get_device = vfio_amba_get_device;
>>  
>>  	ret = vfio_platform_probe_common(vdev, &adev->dev);
>>  	if (ret) {
>> diff --git a/drivers/vfio/platform/vfio_platform.c b/drivers/vfio/platform/vfio_platform.c
>> index cef645c..c025d76 100644
>> --- a/drivers/vfio/platform/vfio_platform.c
>> +++ b/drivers/vfio/platform/vfio_platform.c
>> @@ -51,6 +51,15 @@ static int get_platform_irq(struct vfio_platform_device *vdev, int i)
>>  	return platform_get_irq(pdev, i);
>>  }
>>  
>> +static struct device *vfio_platform_get_device(
>> +				struct vfio_platform_device *vdev)
>> +{
>> +	struct platform_device *pdev = (struct platform_device *) vdev->opaque;
>> +	struct device *dev = &pdev->dev;
>> +
>> +	return dev;
>> +}
>> +
>>  static int vfio_platform_probe(struct platform_device *pdev)
>>  {
>>  	struct vfio_platform_device *vdev;
>> @@ -65,6 +74,7 @@ static int vfio_platform_probe(struct platform_device *pdev)
>>  	vdev->flags = VFIO_DEVICE_FLAGS_PLATFORM;
>>  	vdev->get_resource = get_platform_resource;
>>  	vdev->get_irq = get_platform_irq;
>> +	vdev->get_device = vfio_platform_get_device;
>>  
>>  	ret = vfio_platform_probe_common(vdev, &pdev->dev);
>>  	if (ret)
>> diff --git a/drivers/vfio/platform/vfio_platform_private.h b/drivers/vfio/platform/vfio_platform_private.h
>> index da2d60b..68909a4 100644
>> --- a/drivers/vfio/platform/vfio_platform_private.h
>> +++ b/drivers/vfio/platform/vfio_platform_private.h
>> @@ -67,6 +67,7 @@ struct vfio_platform_device {
>>  	struct resource*
>>  		(*get_resource)(struct vfio_platform_device *vdev, int i);
>>  	int	(*get_irq)(struct vfio_platform_device *vdev, int i);
>> +	struct device *(*get_device)(struct vfio_platform_device *vdev);
>>  };
>>  
>>  typedef int (*vfio_platform_reset_fn_t)(struct vfio_platform_device *vdev);
> 
> 
> 

Re: [PATCH 2/5] VFIO: platform: add get_device callback

[Eric Auger]

On 05/13/2015 08:32 PM, Alex Williamson wrote:
> On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
>> It is needed to introduce a new callback enabling to retrieve the
>> struct device* from the vfio_platform_device. Implementation depends
>> on the underlying device, platform or amba. This will be used to retrieve
>> the compatibility string of the device.
>>
>> Signed-off-by: Eric Auger <eric.auger@linaro.org>
>> ---
>>  drivers/vfio/platform/vfio_amba.c             |  9 +++++++++
>>  drivers/vfio/platform/vfio_platform.c         | 10 ++++++++++
>>  drivers/vfio/platform/vfio_platform_private.h |  1 +
>>  3 files changed, 20 insertions(+)
>>
>> diff --git a/drivers/vfio/platform/vfio_amba.c b/drivers/vfio/platform/vfio_amba.c
>> index ff0331f..fd68115 100644
>> --- a/drivers/vfio/platform/vfio_amba.c
>> +++ b/drivers/vfio/platform/vfio_amba.c
>> @@ -48,6 +48,14 @@ static int get_amba_irq(struct vfio_platform_device *vdev, int i)
>>  	return ret ? ret : -ENXIO;
>>  }
>>  
>> +static struct device *vfio_amba_get_device(struct vfio_platform_device *vdev)
>> +{
>> +	struct amba_device *pdev = (struct amba_device *) vdev->opaque;
>> +	struct device *dev = &pdev->dev;
>> +
>> +	return dev;
>> +}
> 
> *get* device implies a increment in the reference count.  There's
> nothing like that happening here.  Do you just want a to_device wrapper?
Yes you already told that in the past and I should have learnt ...
> Why doesn't struct vfio_platform_device just cache a pointer to the
> struct device?
simpler indeed if agreed.
> 
>> +
>>  static int vfio_amba_probe(struct amba_device *adev, const struct amba_id *id)
>>  {
>>  	struct vfio_platform_device *vdev;
>> @@ -67,6 +75,7 @@ static int vfio_amba_probe(struct amba_device *adev, const struct amba_id *id)
>>  	vdev->flags = VFIO_DEVICE_FLAGS_AMBA;
>>  	vdev->get_resource = get_amba_resource;
>>  	vdev->get_irq = get_amba_irq;
>> +	vdev->get_device = vfio_amba_get_device;
>>  
>>  	ret = vfio_platform_probe_common(vdev, &adev->dev);
>>  	if (ret) {
>> diff --git a/drivers/vfio/platform/vfio_platform.c b/drivers/vfio/platform/vfio_platform.c
>> index cef645c..c025d76 100644
>> --- a/drivers/vfio/platform/vfio_platform.c
>> +++ b/drivers/vfio/platform/vfio_platform.c
>> @@ -51,6 +51,15 @@ static int get_platform_irq(struct vfio_platform_device *vdev, int i)
>>  	return platform_get_irq(pdev, i);
>>  }
>>  
>> +static struct device *vfio_platform_get_device(
>> +				struct vfio_platform_device *vdev)
>> +{
>> +	struct platform_device *pdev = (struct platform_device *) vdev->opaque;
>> +	struct device *dev = &pdev->dev;
>> +
>> +	return dev;
>> +}
>> +
>>  static int vfio_platform_probe(struct platform_device *pdev)
>>  {
>>  	struct vfio_platform_device *vdev;
>> @@ -65,6 +74,7 @@ static int vfio_platform_probe(struct platform_device *pdev)
>>  	vdev->flags = VFIO_DEVICE_FLAGS_PLATFORM;
>>  	vdev->get_resource = get_platform_resource;
>>  	vdev->get_irq = get_platform_irq;
>> +	vdev->get_device = vfio_platform_get_device;
>>  
>>  	ret = vfio_platform_probe_common(vdev, &pdev->dev);
>>  	if (ret)
>> diff --git a/drivers/vfio/platform/vfio_platform_private.h b/drivers/vfio/platform/vfio_platform_private.h
>> index da2d60b..68909a4 100644
>> --- a/drivers/vfio/platform/vfio_platform_private.h
>> +++ b/drivers/vfio/platform/vfio_platform_private.h
>> @@ -67,6 +67,7 @@ struct vfio_platform_device {
>>  	struct resource*
>>  		(*get_resource)(struct vfio_platform_device *vdev, int i);
>>  	int	(*get_irq)(struct vfio_platform_device *vdev, int i);
>> +	struct device *(*get_device)(struct vfio_platform_device *vdev);
>>  };
>>  
>>  typedef int (*vfio_platform_reset_fn_t)(struct vfio_platform_device *vdev);
> 
> 
> 

[PATCH 2/5] VFIO: platform: add get_device callback

[Eric Auger]

On 05/13/2015 08:32 PM, Alex Williamson wrote:
> On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
>> It is needed to introduce a new callback enabling to retrieve the
>> struct device* from the vfio_platform_device. Implementation depends
>> on the underlying device, platform or amba. This will be used to retrieve
>> the compatibility string of the device.
>>
>> Signed-off-by: Eric Auger <eric.auger@linaro.org>
>> ---
>>  drivers/vfio/platform/vfio_amba.c             |  9 +++++++++
>>  drivers/vfio/platform/vfio_platform.c         | 10 ++++++++++
>>  drivers/vfio/platform/vfio_platform_private.h |  1 +
>>  3 files changed, 20 insertions(+)
>>
>> diff --git a/drivers/vfio/platform/vfio_amba.c b/drivers/vfio/platform/vfio_amba.c
>> index ff0331f..fd68115 100644
>> --- a/drivers/vfio/platform/vfio_amba.c
>> +++ b/drivers/vfio/platform/vfio_amba.c
>> @@ -48,6 +48,14 @@ static int get_amba_irq(struct vfio_platform_device *vdev, int i)
>>  	return ret ? ret : -ENXIO;
>>  }
>>  
>> +static struct device *vfio_amba_get_device(struct vfio_platform_device *vdev)
>> +{
>> +	struct amba_device *pdev = (struct amba_device *) vdev->opaque;
>> +	struct device *dev = &pdev->dev;
>> +
>> +	return dev;
>> +}
> 
> *get* device implies a increment in the reference count.  There's
> nothing like that happening here.  Do you just want a to_device wrapper?
Yes you already told that in the past and I should have learnt ...
> Why doesn't struct vfio_platform_device just cache a pointer to the
> struct device?
simpler indeed if agreed.
> 
>> +
>>  static int vfio_amba_probe(struct amba_device *adev, const struct amba_id *id)
>>  {
>>  	struct vfio_platform_device *vdev;
>> @@ -67,6 +75,7 @@ static int vfio_amba_probe(struct amba_device *adev, const struct amba_id *id)
>>  	vdev->flags = VFIO_DEVICE_FLAGS_AMBA;
>>  	vdev->get_resource = get_amba_resource;
>>  	vdev->get_irq = get_amba_irq;
>> +	vdev->get_device = vfio_amba_get_device;
>>  
>>  	ret = vfio_platform_probe_common(vdev, &adev->dev);
>>  	if (ret) {
>> diff --git a/drivers/vfio/platform/vfio_platform.c b/drivers/vfio/platform/vfio_platform.c
>> index cef645c..c025d76 100644
>> --- a/drivers/vfio/platform/vfio_platform.c
>> +++ b/drivers/vfio/platform/vfio_platform.c
>> @@ -51,6 +51,15 @@ static int get_platform_irq(struct vfio_platform_device *vdev, int i)
>>  	return platform_get_irq(pdev, i);
>>  }
>>  
>> +static struct device *vfio_platform_get_device(
>> +				struct vfio_platform_device *vdev)
>> +{
>> +	struct platform_device *pdev = (struct platform_device *) vdev->opaque;
>> +	struct device *dev = &pdev->dev;
>> +
>> +	return dev;
>> +}
>> +
>>  static int vfio_platform_probe(struct platform_device *pdev)
>>  {
>>  	struct vfio_platform_device *vdev;
>> @@ -65,6 +74,7 @@ static int vfio_platform_probe(struct platform_device *pdev)
>>  	vdev->flags = VFIO_DEVICE_FLAGS_PLATFORM;
>>  	vdev->get_resource = get_platform_resource;
>>  	vdev->get_irq = get_platform_irq;
>> +	vdev->get_device = vfio_platform_get_device;
>>  
>>  	ret = vfio_platform_probe_common(vdev, &pdev->dev);
>>  	if (ret)
>> diff --git a/drivers/vfio/platform/vfio_platform_private.h b/drivers/vfio/platform/vfio_platform_private.h
>> index da2d60b..68909a4 100644
>> --- a/drivers/vfio/platform/vfio_platform_private.h
>> +++ b/drivers/vfio/platform/vfio_platform_private.h
>> @@ -67,6 +67,7 @@ struct vfio_platform_device {
>>  	struct resource*
>>  		(*get_resource)(struct vfio_platform_device *vdev, int i);
>>  	int	(*get_irq)(struct vfio_platform_device *vdev, int i);
>> +	struct device *(*get_device)(struct vfio_platform_device *vdev);
>>  };
>>  
>>  typedef int (*vfio_platform_reset_fn_t)(struct vfio_platform_device *vdev);
> 
> 
> 

Re: [PATCH 3/5] VFIO: platform: add reset callback

[Eric Auger]

On 05/13/2015 08:32 PM, Alex Williamson wrote:
> On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
>> A new reset callback is introduced. If this callback is populated,
>> the reset is invoked on device release or upon userspace ioctl. The
>> modality is exposed on VFIO_DEVICE_GET_INFO.
>>
>> Signed-off-by: Eric Auger <eric.auger@linaro.org>
>>
>> ---
>>
>> v2 -> v3:
>> - change patch title and commit message. Use IS_ERR_OR_NULL to
>>   anticipate for getter returned value.
>>
>> v1 -> v2:
>> - On VFIO_DEVICE_RESET returns -EINVAL in case the callback is not
>>   populated (instead of -ENOTTY)
>> ---
>>  drivers/vfio/platform/vfio_platform_common.c  | 12 ++++++++++--
>>  drivers/vfio/platform/vfio_platform_private.h |  1 +
>>  2 files changed, 11 insertions(+), 2 deletions(-)
>>
>> diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c
>> index edbf24c..0d10018 100644
>> --- a/drivers/vfio/platform/vfio_platform_common.c
>> +++ b/drivers/vfio/platform/vfio_platform_common.c
>> @@ -103,6 +103,8 @@ static void vfio_platform_release(void *device_data)
>>  	mutex_lock(&driver_lock);
>>  
>>  	if (!(--vdev->refcnt)) {
>> +		if (!IS_ERR_OR_NULL(vdev->reset))
>> +			vdev->reset(vdev);
> 
> 
> It seems sloppy to have a function pointer that could be ERR_PTR.
The reset lookup function currently can return errors. I will change this.
> 
> Should we also be trying to reset on device open?
may be yes. I was influenced by qemu use case where reset is also
explicitly invoked through iotcl by the reset notifier. If we do so we
might have duplicated resets.

- Eric
> 
>>  		vfio_platform_regions_cleanup(vdev);
>>  		vfio_platform_irq_cleanup(vdev);
>>  	}
>> @@ -162,6 +164,8 @@ static long vfio_platform_ioctl(void *device_data,
>>  		if (info.argsz < minsz)
>>  			return -EINVAL;
>>  
>> +		if (!IS_ERR_OR_NULL(vdev->reset))
>> +			vdev->flags |= VFIO_DEVICE_FLAGS_RESET;
>>  		info.flags = vdev->flags;
>>  		info.num_regions = vdev->num_regions;
>>  		info.num_irqs = vdev->num_irqs;
>> @@ -255,8 +259,12 @@ static long vfio_platform_ioctl(void *device_data,
>>  
>>  		return ret;
>>  
>> -	} else if (cmd == VFIO_DEVICE_RESET)
>> -		return -EINVAL;
>> +	} else if (cmd == VFIO_DEVICE_RESET) {
>> +		if (!IS_ERR_OR_NULL(vdev->reset))
>> +			return vdev->reset(vdev);
>> +		else
>> +			return -EINVAL;
>> +	}
>>  
>>  	return -ENOTTY;
>>  }
>> diff --git a/drivers/vfio/platform/vfio_platform_private.h b/drivers/vfio/platform/vfio_platform_private.h
>> index 68909a4..84ccf6d 100644
>> --- a/drivers/vfio/platform/vfio_platform_private.h
>> +++ b/drivers/vfio/platform/vfio_platform_private.h
>> @@ -68,6 +68,7 @@ struct vfio_platform_device {
>>  		(*get_resource)(struct vfio_platform_device *vdev, int i);
>>  	int	(*get_irq)(struct vfio_platform_device *vdev, int i);
>>  	struct device *(*get_device)(struct vfio_platform_device *vdev);
>> +	int	(*reset)(struct vfio_platform_device *vdev);
>>  };
>>  
>>  typedef int (*vfio_platform_reset_fn_t)(struct vfio_platform_device *vdev);
> 
> 
> 

Re: [PATCH 3/5] VFIO: platform: add reset callback

[Eric Auger]

On 05/13/2015 08:32 PM, Alex Williamson wrote:
> On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
>> A new reset callback is introduced. If this callback is populated,
>> the reset is invoked on device release or upon userspace ioctl. The
>> modality is exposed on VFIO_DEVICE_GET_INFO.
>>
>> Signed-off-by: Eric Auger <eric.auger@linaro.org>
>>
>> ---
>>
>> v2 -> v3:
>> - change patch title and commit message. Use IS_ERR_OR_NULL to
>>   anticipate for getter returned value.
>>
>> v1 -> v2:
>> - On VFIO_DEVICE_RESET returns -EINVAL in case the callback is not
>>   populated (instead of -ENOTTY)
>> ---
>>  drivers/vfio/platform/vfio_platform_common.c  | 12 ++++++++++--
>>  drivers/vfio/platform/vfio_platform_private.h |  1 +
>>  2 files changed, 11 insertions(+), 2 deletions(-)
>>
>> diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c
>> index edbf24c..0d10018 100644
>> --- a/drivers/vfio/platform/vfio_platform_common.c
>> +++ b/drivers/vfio/platform/vfio_platform_common.c
>> @@ -103,6 +103,8 @@ static void vfio_platform_release(void *device_data)
>>  	mutex_lock(&driver_lock);
>>  
>>  	if (!(--vdev->refcnt)) {
>> +		if (!IS_ERR_OR_NULL(vdev->reset))
>> +			vdev->reset(vdev);
> 
> 
> It seems sloppy to have a function pointer that could be ERR_PTR.
The reset lookup function currently can return errors. I will change this.
> 
> Should we also be trying to reset on device open?
may be yes. I was influenced by qemu use case where reset is also
explicitly invoked through iotcl by the reset notifier. If we do so we
might have duplicated resets.

- Eric
> 
>>  		vfio_platform_regions_cleanup(vdev);
>>  		vfio_platform_irq_cleanup(vdev);
>>  	}
>> @@ -162,6 +164,8 @@ static long vfio_platform_ioctl(void *device_data,
>>  		if (info.argsz < minsz)
>>  			return -EINVAL;
>>  
>> +		if (!IS_ERR_OR_NULL(vdev->reset))
>> +			vdev->flags |= VFIO_DEVICE_FLAGS_RESET;
>>  		info.flags = vdev->flags;
>>  		info.num_regions = vdev->num_regions;
>>  		info.num_irqs = vdev->num_irqs;
>> @@ -255,8 +259,12 @@ static long vfio_platform_ioctl(void *device_data,
>>  
>>  		return ret;
>>  
>> -	} else if (cmd == VFIO_DEVICE_RESET)
>> -		return -EINVAL;
>> +	} else if (cmd == VFIO_DEVICE_RESET) {
>> +		if (!IS_ERR_OR_NULL(vdev->reset))
>> +			return vdev->reset(vdev);
>> +		else
>> +			return -EINVAL;
>> +	}
>>  
>>  	return -ENOTTY;
>>  }
>> diff --git a/drivers/vfio/platform/vfio_platform_private.h b/drivers/vfio/platform/vfio_platform_private.h
>> index 68909a4..84ccf6d 100644
>> --- a/drivers/vfio/platform/vfio_platform_private.h
>> +++ b/drivers/vfio/platform/vfio_platform_private.h
>> @@ -68,6 +68,7 @@ struct vfio_platform_device {
>>  		(*get_resource)(struct vfio_platform_device *vdev, int i);
>>  	int	(*get_irq)(struct vfio_platform_device *vdev, int i);
>>  	struct device *(*get_device)(struct vfio_platform_device *vdev);
>> +	int	(*reset)(struct vfio_platform_device *vdev);
>>  };
>>  
>>  typedef int (*vfio_platform_reset_fn_t)(struct vfio_platform_device *vdev);
> 
> 
> 

[PATCH 3/5] VFIO: platform: add reset callback

[Eric Auger]

On 05/13/2015 08:32 PM, Alex Williamson wrote:
> On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
>> A new reset callback is introduced. If this callback is populated,
>> the reset is invoked on device release or upon userspace ioctl. The
>> modality is exposed on VFIO_DEVICE_GET_INFO.
>>
>> Signed-off-by: Eric Auger <eric.auger@linaro.org>
>>
>> ---
>>
>> v2 -> v3:
>> - change patch title and commit message. Use IS_ERR_OR_NULL to
>>   anticipate for getter returned value.
>>
>> v1 -> v2:
>> - On VFIO_DEVICE_RESET returns -EINVAL in case the callback is not
>>   populated (instead of -ENOTTY)
>> ---
>>  drivers/vfio/platform/vfio_platform_common.c  | 12 ++++++++++--
>>  drivers/vfio/platform/vfio_platform_private.h |  1 +
>>  2 files changed, 11 insertions(+), 2 deletions(-)
>>
>> diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c
>> index edbf24c..0d10018 100644
>> --- a/drivers/vfio/platform/vfio_platform_common.c
>> +++ b/drivers/vfio/platform/vfio_platform_common.c
>> @@ -103,6 +103,8 @@ static void vfio_platform_release(void *device_data)
>>  	mutex_lock(&driver_lock);
>>  
>>  	if (!(--vdev->refcnt)) {
>> +		if (!IS_ERR_OR_NULL(vdev->reset))
>> +			vdev->reset(vdev);
> 
> 
> It seems sloppy to have a function pointer that could be ERR_PTR.
The reset lookup function currently can return errors. I will change this.
> 
> Should we also be trying to reset on device open?
may be yes. I was influenced by qemu use case where reset is also
explicitly invoked through iotcl by the reset notifier. If we do so we
might have duplicated resets.

- Eric
> 
>>  		vfio_platform_regions_cleanup(vdev);
>>  		vfio_platform_irq_cleanup(vdev);
>>  	}
>> @@ -162,6 +164,8 @@ static long vfio_platform_ioctl(void *device_data,
>>  		if (info.argsz < minsz)
>>  			return -EINVAL;
>>  
>> +		if (!IS_ERR_OR_NULL(vdev->reset))
>> +			vdev->flags |= VFIO_DEVICE_FLAGS_RESET;
>>  		info.flags = vdev->flags;
>>  		info.num_regions = vdev->num_regions;
>>  		info.num_irqs = vdev->num_irqs;
>> @@ -255,8 +259,12 @@ static long vfio_platform_ioctl(void *device_data,
>>  
>>  		return ret;
>>  
>> -	} else if (cmd == VFIO_DEVICE_RESET)
>> -		return -EINVAL;
>> +	} else if (cmd == VFIO_DEVICE_RESET) {
>> +		if (!IS_ERR_OR_NULL(vdev->reset))
>> +			return vdev->reset(vdev);
>> +		else
>> +			return -EINVAL;
>> +	}
>>  
>>  	return -ENOTTY;
>>  }
>> diff --git a/drivers/vfio/platform/vfio_platform_private.h b/drivers/vfio/platform/vfio_platform_private.h
>> index 68909a4..84ccf6d 100644
>> --- a/drivers/vfio/platform/vfio_platform_private.h
>> +++ b/drivers/vfio/platform/vfio_platform_private.h
>> @@ -68,6 +68,7 @@ struct vfio_platform_device {
>>  		(*get_resource)(struct vfio_platform_device *vdev, int i);
>>  	int	(*get_irq)(struct vfio_platform_device *vdev, int i);
>>  	struct device *(*get_device)(struct vfio_platform_device *vdev);
>> +	int	(*reset)(struct vfio_platform_device *vdev);
>>  };
>>  
>>  typedef int (*vfio_platform_reset_fn_t)(struct vfio_platform_device *vdev);
> 
> 
> 

Re: [PATCH 4/5] VFIO: platform: populate reset function according to compat

[Eric Auger]

On 05/13/2015 08:33 PM, Alex Williamson wrote:
> On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
>> Add the reset function lookup according to the device compat
>> string. This lookup is added at different places:
>> - on VFIO_DEVICE_GET_INFO
>> - on VFIO_DEVICE_RESET
>> - on device release
>>
>> A reference to the module implementing the reset function is taken
>> on first reset function lookup and released on vfio platform device
>> release.
>>
>> Signed-off-by: Eric Auger <eric.auger@linaro.org>
>> ---
>>  drivers/vfio/platform/vfio_platform_common.c | 50 ++++++++++++++++++++++++++++
>>  1 file changed, 50 insertions(+)
>>
>> diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c
>> index 0d10018..bd7e44c 100644
>> --- a/drivers/vfio/platform/vfio_platform_common.c
>> +++ b/drivers/vfio/platform/vfio_platform_common.c
>> @@ -28,6 +28,52 @@ LIST_HEAD(reset_list);
>>  
>>  static DEFINE_MUTEX(driver_lock);
>>  
>> +static vfio_platform_reset_fn_t vfio_platform_lookup_reset(const char *compat)
>> +{
>> +	struct vfio_platform_reset_node *iter;
>> +
>> +	list_for_each_entry(iter, &reset_list, link) {
> 
> Racy
ok
> 
>> +		if (!strcmp(iter->compat, compat) &&
>> +			try_module_get(iter->owner))
>> +			return iter->reset;
>> +	}
>> +
>> +	return ERR_PTR(-ENODEV);
> 
> return NULL imo
ok
> 
>> +}
>> +
>> +static vfio_platform_reset_fn_t vfio_platform_get_reset(
>> +					struct vfio_platform_device *vdev)
>> +{
>> +	struct device *dev = vdev->get_device(vdev);
>> +	const char *compat_str_array[2];
>> +	vfio_platform_reset_fn_t reset;
>> +	int ret;
>> +
>> +	if (!IS_ERR_OR_NULL(vdev->reset))
>> +		return vdev->reset;
>> +
>> +	ret = device_property_read_string_array(dev, "compatible",
>> +						compat_str_array, 2);
>> +	if (!ret)
>> +		return ERR_PTR(ret);
>> +
>> +	reset = vfio_platform_lookup_reset(compat_str_array[0]);
>> +	return reset;
> 
> Something got allocated into compat_str_array and gets leaked here.
is there any allocation? device_property_read_string_array does not
return -ENOMEM.

> 
>> +}
>> +
>> +static void vfio_platform_put_reset(struct vfio_platform_device *vdev)
>> +{
>> +	struct vfio_platform_reset_node *iter;
>> +
>> +	list_for_each_entry(iter, &reset_list, link) {
> 
> Racy
ok
> 
>> +		if (iter->reset == vdev->reset) {
>> +			module_put(iter->owner);
>> +			vdev->reset = NULL;
>> +			return;
>> +		}
>> +	}
>> +}
>> +
>>  static int vfio_platform_regions_init(struct vfio_platform_device *vdev)
>>  {
>>  	int cnt = 0, i;
>> @@ -103,10 +149,12 @@ static void vfio_platform_release(void *device_data)
>>  	mutex_lock(&driver_lock);
>>  
>>  	if (!(--vdev->refcnt)) {
>> +		vdev->reset = vfio_platform_get_reset(vdev);
>>  		if (!IS_ERR_OR_NULL(vdev->reset))
>>  			vdev->reset(vdev);
>>  		vfio_platform_regions_cleanup(vdev);
>>  		vfio_platform_irq_cleanup(vdev);
>> +		vfio_platform_put_reset(vdev);
>>  	}
>>  
>>  	mutex_unlock(&driver_lock);
>> @@ -164,6 +212,7 @@ static long vfio_platform_ioctl(void *device_data,
>>  		if (info.argsz < minsz)
>>  			return -EINVAL;
>>  
>> +		vdev->reset = vfio_platform_get_reset(vdev);
>>  		if (!IS_ERR_OR_NULL(vdev->reset))
>>  			vdev->flags |= VFIO_DEVICE_FLAGS_RESET;
>>  		info.flags = vdev->flags;
>> @@ -260,6 +309,7 @@ static long vfio_platform_ioctl(void *device_data,
>>  		return ret;
>>  
>>  	} else if (cmd == VFIO_DEVICE_RESET) {
>> +		vdev->reset = vfio_platform_get_reset(vdev);
>>  		if (!IS_ERR_OR_NULL(vdev->reset))
>>  			return vdev->reset(vdev);
>>  		else
> 
> I count 3 gets and 1 put, isn't the module reference count increase
> showing that?

vfio_platform_get_reset simply returns if the function pointer already
is populated so there is no systematic ref increment.

  This looks like it hasn't been tested.

It did testing with external and in-kernel modules through
  Why would we do a
> get every time we want to do a reset?

My doubt were about the order of probing between the
vfio-platform_driver and the vfio reset module? This question was the
rationale of this implementation choice. But again the actual ref count
increment is devised to be done once on the first entry point (iotcl or
internal release)

  Do one get when the device is
> registered or opened and one put when the device is unregistered or
> closed.  We don't want erratic userspace behavior that the reset
> property of a device can come and go.

> 

Re: [PATCH 4/5] VFIO: platform: populate reset function according to compat

[Eric Auger]

On 05/13/2015 08:33 PM, Alex Williamson wrote:
> On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
>> Add the reset function lookup according to the device compat
>> string. This lookup is added at different places:
>> - on VFIO_DEVICE_GET_INFO
>> - on VFIO_DEVICE_RESET
>> - on device release
>>
>> A reference to the module implementing the reset function is taken
>> on first reset function lookup and released on vfio platform device
>> release.
>>
>> Signed-off-by: Eric Auger <eric.auger@linaro.org>
>> ---
>>  drivers/vfio/platform/vfio_platform_common.c | 50 ++++++++++++++++++++++++++++
>>  1 file changed, 50 insertions(+)
>>
>> diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c
>> index 0d10018..bd7e44c 100644
>> --- a/drivers/vfio/platform/vfio_platform_common.c
>> +++ b/drivers/vfio/platform/vfio_platform_common.c
>> @@ -28,6 +28,52 @@ LIST_HEAD(reset_list);
>>  
>>  static DEFINE_MUTEX(driver_lock);
>>  
>> +static vfio_platform_reset_fn_t vfio_platform_lookup_reset(const char *compat)
>> +{
>> +	struct vfio_platform_reset_node *iter;
>> +
>> +	list_for_each_entry(iter, &reset_list, link) {
> 
> Racy
ok
> 
>> +		if (!strcmp(iter->compat, compat) &&
>> +			try_module_get(iter->owner))
>> +			return iter->reset;
>> +	}
>> +
>> +	return ERR_PTR(-ENODEV);
> 
> return NULL imo
ok
> 
>> +}
>> +
>> +static vfio_platform_reset_fn_t vfio_platform_get_reset(
>> +					struct vfio_platform_device *vdev)
>> +{
>> +	struct device *dev = vdev->get_device(vdev);
>> +	const char *compat_str_array[2];
>> +	vfio_platform_reset_fn_t reset;
>> +	int ret;
>> +
>> +	if (!IS_ERR_OR_NULL(vdev->reset))
>> +		return vdev->reset;
>> +
>> +	ret = device_property_read_string_array(dev, "compatible",
>> +						compat_str_array, 2);
>> +	if (!ret)
>> +		return ERR_PTR(ret);
>> +
>> +	reset = vfio_platform_lookup_reset(compat_str_array[0]);
>> +	return reset;
> 
> Something got allocated into compat_str_array and gets leaked here.
is there any allocation? device_property_read_string_array does not
return -ENOMEM.

> 
>> +}
>> +
>> +static void vfio_platform_put_reset(struct vfio_platform_device *vdev)
>> +{
>> +	struct vfio_platform_reset_node *iter;
>> +
>> +	list_for_each_entry(iter, &reset_list, link) {
> 
> Racy
ok
> 
>> +		if (iter->reset == vdev->reset) {
>> +			module_put(iter->owner);
>> +			vdev->reset = NULL;
>> +			return;
>> +		}
>> +	}
>> +}
>> +
>>  static int vfio_platform_regions_init(struct vfio_platform_device *vdev)
>>  {
>>  	int cnt = 0, i;
>> @@ -103,10 +149,12 @@ static void vfio_platform_release(void *device_data)
>>  	mutex_lock(&driver_lock);
>>  
>>  	if (!(--vdev->refcnt)) {
>> +		vdev->reset = vfio_platform_get_reset(vdev);
>>  		if (!IS_ERR_OR_NULL(vdev->reset))
>>  			vdev->reset(vdev);
>>  		vfio_platform_regions_cleanup(vdev);
>>  		vfio_platform_irq_cleanup(vdev);
>> +		vfio_platform_put_reset(vdev);
>>  	}
>>  
>>  	mutex_unlock(&driver_lock);
>> @@ -164,6 +212,7 @@ static long vfio_platform_ioctl(void *device_data,
>>  		if (info.argsz < minsz)
>>  			return -EINVAL;
>>  
>> +		vdev->reset = vfio_platform_get_reset(vdev);
>>  		if (!IS_ERR_OR_NULL(vdev->reset))
>>  			vdev->flags |= VFIO_DEVICE_FLAGS_RESET;
>>  		info.flags = vdev->flags;
>> @@ -260,6 +309,7 @@ static long vfio_platform_ioctl(void *device_data,
>>  		return ret;
>>  
>>  	} else if (cmd == VFIO_DEVICE_RESET) {
>> +		vdev->reset = vfio_platform_get_reset(vdev);
>>  		if (!IS_ERR_OR_NULL(vdev->reset))
>>  			return vdev->reset(vdev);
>>  		else
> 
> I count 3 gets and 1 put, isn't the module reference count increase
> showing that?

vfio_platform_get_reset simply returns if the function pointer already
is populated so there is no systematic ref increment.

  This looks like it hasn't been tested.

It did testing with external and in-kernel modules through
  Why would we do a
> get every time we want to do a reset?

My doubt were about the order of probing between the
vfio-platform_driver and the vfio reset module? This question was the
rationale of this implementation choice. But again the actual ref count
increment is devised to be done once on the first entry point (iotcl or
internal release)

  Do one get when the device is
> registered or opened and one put when the device is unregistered or
> closed.  We don't want erratic userspace behavior that the reset
> property of a device can come and go.

> 

[PATCH 4/5] VFIO: platform: populate reset function according to compat

[Eric Auger]

On 05/13/2015 08:33 PM, Alex Williamson wrote:
> On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
>> Add the reset function lookup according to the device compat
>> string. This lookup is added at different places:
>> - on VFIO_DEVICE_GET_INFO
>> - on VFIO_DEVICE_RESET
>> - on device release
>>
>> A reference to the module implementing the reset function is taken
>> on first reset function lookup and released on vfio platform device
>> release.
>>
>> Signed-off-by: Eric Auger <eric.auger@linaro.org>
>> ---
>>  drivers/vfio/platform/vfio_platform_common.c | 50 ++++++++++++++++++++++++++++
>>  1 file changed, 50 insertions(+)
>>
>> diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c
>> index 0d10018..bd7e44c 100644
>> --- a/drivers/vfio/platform/vfio_platform_common.c
>> +++ b/drivers/vfio/platform/vfio_platform_common.c
>> @@ -28,6 +28,52 @@ LIST_HEAD(reset_list);
>>  
>>  static DEFINE_MUTEX(driver_lock);
>>  
>> +static vfio_platform_reset_fn_t vfio_platform_lookup_reset(const char *compat)
>> +{
>> +	struct vfio_platform_reset_node *iter;
>> +
>> +	list_for_each_entry(iter, &reset_list, link) {
> 
> Racy
ok
> 
>> +		if (!strcmp(iter->compat, compat) &&
>> +			try_module_get(iter->owner))
>> +			return iter->reset;
>> +	}
>> +
>> +	return ERR_PTR(-ENODEV);
> 
> return NULL imo
ok
> 
>> +}
>> +
>> +static vfio_platform_reset_fn_t vfio_platform_get_reset(
>> +					struct vfio_platform_device *vdev)
>> +{
>> +	struct device *dev = vdev->get_device(vdev);
>> +	const char *compat_str_array[2];
>> +	vfio_platform_reset_fn_t reset;
>> +	int ret;
>> +
>> +	if (!IS_ERR_OR_NULL(vdev->reset))
>> +		return vdev->reset;
>> +
>> +	ret = device_property_read_string_array(dev, "compatible",
>> +						compat_str_array, 2);
>> +	if (!ret)
>> +		return ERR_PTR(ret);
>> +
>> +	reset = vfio_platform_lookup_reset(compat_str_array[0]);
>> +	return reset;
> 
> Something got allocated into compat_str_array and gets leaked here.
is there any allocation? device_property_read_string_array does not
return -ENOMEM.

> 
>> +}
>> +
>> +static void vfio_platform_put_reset(struct vfio_platform_device *vdev)
>> +{
>> +	struct vfio_platform_reset_node *iter;
>> +
>> +	list_for_each_entry(iter, &reset_list, link) {
> 
> Racy
ok
> 
>> +		if (iter->reset == vdev->reset) {
>> +			module_put(iter->owner);
>> +			vdev->reset = NULL;
>> +			return;
>> +		}
>> +	}
>> +}
>> +
>>  static int vfio_platform_regions_init(struct vfio_platform_device *vdev)
>>  {
>>  	int cnt = 0, i;
>> @@ -103,10 +149,12 @@ static void vfio_platform_release(void *device_data)
>>  	mutex_lock(&driver_lock);
>>  
>>  	if (!(--vdev->refcnt)) {
>> +		vdev->reset = vfio_platform_get_reset(vdev);
>>  		if (!IS_ERR_OR_NULL(vdev->reset))
>>  			vdev->reset(vdev);
>>  		vfio_platform_regions_cleanup(vdev);
>>  		vfio_platform_irq_cleanup(vdev);
>> +		vfio_platform_put_reset(vdev);
>>  	}
>>  
>>  	mutex_unlock(&driver_lock);
>> @@ -164,6 +212,7 @@ static long vfio_platform_ioctl(void *device_data,
>>  		if (info.argsz < minsz)
>>  			return -EINVAL;
>>  
>> +		vdev->reset = vfio_platform_get_reset(vdev);
>>  		if (!IS_ERR_OR_NULL(vdev->reset))
>>  			vdev->flags |= VFIO_DEVICE_FLAGS_RESET;
>>  		info.flags = vdev->flags;
>> @@ -260,6 +309,7 @@ static long vfio_platform_ioctl(void *device_data,
>>  		return ret;
>>  
>>  	} else if (cmd == VFIO_DEVICE_RESET) {
>> +		vdev->reset = vfio_platform_get_reset(vdev);
>>  		if (!IS_ERR_OR_NULL(vdev->reset))
>>  			return vdev->reset(vdev);
>>  		else
> 
> I count 3 gets and 1 put, isn't the module reference count increase
> showing that?

vfio_platform_get_reset simply returns if the function pointer already
is populated so there is no systematic ref increment.

  This looks like it hasn't been tested.

It did testing with external and in-kernel modules through
  Why would we do a
> get every time we want to do a reset?

My doubt were about the order of probing between the
vfio-platform_driver and the vfio reset module? This question was the
rationale of this implementation choice. But again the actual ref count
increment is devised to be done once on the first entry point (iotcl or
internal release)

  Do one get when the device is
> registered or opened and one put when the device is unregistered or
> closed.  We don't want erratic userspace behavior that the reset
> property of a device can come and go.

> 

Re: [PATCH 5/5] VFIO: platform: VFIO platform Calxeda xgmac reset module

[Eric Auger]

Alex,
On 05/13/2015 08:33 PM, Alex Williamson wrote:
> On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
>> This patch introduces a module that registers and implements a basic
>> reset function for the Calxeda xgmac device. This latter basically disables
>> interrupts and stops DMA transfers.
>>
>> The reset function code is inherited from the native calxeda xgmac driver.
>>
>> Signed-off-by: Eric Auger <eric.auger@linaro.org>
>> ---
>>  drivers/vfio/platform/Kconfig                      |   2 +
>>  drivers/vfio/platform/Makefile                     |   2 +
>>  drivers/vfio/platform/reset/Kconfig                |   7 ++
>>  drivers/vfio/platform/reset/Makefile               |   5 +
>>  .../platform/reset/vfio_platform_calxedaxgmac.c    | 121 +++++++++++++++++++++
>>  5 files changed, 137 insertions(+)
>>  create mode 100644 drivers/vfio/platform/reset/Kconfig
>>  create mode 100644 drivers/vfio/platform/reset/Makefile
>>  create mode 100644 drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
>>
>> diff --git a/drivers/vfio/platform/Kconfig b/drivers/vfio/platform/Kconfig
>> index 9a4403e..1df7477 100644
>> --- a/drivers/vfio/platform/Kconfig
>> +++ b/drivers/vfio/platform/Kconfig
>> @@ -18,3 +18,5 @@ config VFIO_AMBA
>>  	  framework.
>>  
>>  	  If you don't know what to do here, say N.
>> +
>> +source "drivers/vfio/platform/reset/Kconfig"
>> diff --git a/drivers/vfio/platform/Makefile b/drivers/vfio/platform/Makefile
>> index 81de144..9ce8afe 100644
>> --- a/drivers/vfio/platform/Makefile
>> +++ b/drivers/vfio/platform/Makefile
>> @@ -2,7 +2,9 @@
>>  vfio-platform-y := vfio_platform.o vfio_platform_common.o vfio_platform_irq.o
>>  
>>  obj-$(CONFIG_VFIO_PLATFORM) += vfio-platform.o
>> +obj-$(CONFIG_VFIO_PLATFORM) += reset/
>>  
>>  vfio-amba-y := vfio_amba.o
>>  
>>  obj-$(CONFIG_VFIO_AMBA) += vfio-amba.o
>> +obj-$(CONFIG_VFIO_AMBA) += reset/
>> diff --git a/drivers/vfio/platform/reset/Kconfig b/drivers/vfio/platform/reset/Kconfig
>> new file mode 100644
>> index 0000000..746b96b
>> --- /dev/null
>> +++ b/drivers/vfio/platform/reset/Kconfig
>> @@ -0,0 +1,7 @@
>> +config VFIO_PLATFORM_CALXEDAXGMAC_RESET
>> +	tristate "VFIO support for calxeda xgmac reset"
>> +	depends on VFIO_PLATFORM
>> +	help
>> +	  Enables the VFIO platform driver to handle reset for Calxeda xgmac
>> +
>> +	  If you don't know what to do here, say N.
>> diff --git a/drivers/vfio/platform/reset/Makefile b/drivers/vfio/platform/reset/Makefile
>> new file mode 100644
>> index 0000000..2a486af
>> --- /dev/null
>> +++ b/drivers/vfio/platform/reset/Makefile
>> @@ -0,0 +1,5 @@
>> +vfio-platform-calxedaxgmac-y := vfio_platform_calxedaxgmac.o
>> +
>> +ccflags-y += -Idrivers/vfio/platform
>> +
>> +obj-$(CONFIG_VFIO_PLATFORM_CALXEDAXGMAC_RESET) += vfio-platform-calxedaxgmac.o
>> diff --git a/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c b/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
>> new file mode 100644
>> index 0000000..3c6e428
>> --- /dev/null
>> +++ b/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
>> @@ -0,0 +1,121 @@
>> +/*
>> + * VFIO platform driver specialized for Calxeda xgmac reset
>> + * reset code is inherited from calxeda xgmac native driver
>> + *
>> + * Copyright 2010-2011 Calxeda, Inc.
>> + * Copyright (c) 2015 Linaro Ltd.
>> + *              www.linaro.org
>> + *
>> + * This program is free software; you can redistribute it and/or modify it
>> + * under the terms and conditions of the GNU General Public License,
>> + * version 2, as published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope it will be useful, but WITHOUT
>> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
>> + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
>> + * more details.
>> + *
>> + * You should have received a copy of the GNU General Public License along with
>> + * this program.  If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <linux/module.h>
>> +#include <linux/kernel.h>
>> +#include <linux/init.h>
>> +#include <linux/io.h>
>> +
>> +#include "vfio_platform_private.h"
>> +
>> +#define DRIVER_VERSION  "0.1"
>> +#define DRIVER_AUTHOR   "Eric Auger <eric.auger@linaro.org>"
>> +#define DRIVER_DESC     "Reset support for Calxeda xgmac vfio platform device"
>> +
>> +#define CALXEDAXGMAC_COMPAT "calxeda,hb-xgmac"
>> +
>> +/* XGMAC Register definitions */
>> +#define XGMAC_CONTROL           0x00000000      /* MAC Configuration */
>> +
>> +/* DMA Control and Status Registers */
>> +#define XGMAC_DMA_CONTROL       0x00000f18      /* Ctrl (Operational Mode) */
>> +#define XGMAC_DMA_INTR_ENA      0x00000f1c      /* Interrupt Enable */
>> +
>> +/* DMA Control registe defines */
>> +#define DMA_CONTROL_ST          0x00002000      /* Start/Stop Transmission */
>> +#define DMA_CONTROL_SR          0x00000002      /* Start/Stop Receive */
>> +
>> +/* Common MAC defines */
>> +#define MAC_ENABLE_TX           0x00000008      /* Transmitter Enable */
>> +#define MAC_ENABLE_RX           0x00000004      /* Receiver Enable */
>> +
>> +static inline void xgmac_mac_disable(void __iomem *ioaddr)
>> +{
>> +	u32 value = readl(ioaddr + XGMAC_DMA_CONTROL);
>> +
>> +	value &= ~(DMA_CONTROL_ST | DMA_CONTROL_SR);
>> +	writel(value, ioaddr + XGMAC_DMA_CONTROL);
>> +
>> +	value = readl(ioaddr + XGMAC_CONTROL);
>> +	value &= ~(MAC_ENABLE_TX | MAC_ENABLE_RX);
>> +	writel(value, ioaddr + XGMAC_CONTROL);
>> +}
>> +
>> +int vfio_platform_calxedaxgmac_reset(struct vfio_platform_device *vdev)
>> +{
>> +	struct vfio_platform_region reg = vdev->regions[0];
>> +
>> +	if (!reg.ioaddr) {
>> +		reg.ioaddr =
>> +			ioremap_nocache(reg.addr, reg.size);
>> +		if (!reg.ioaddr)
>> +			return -ENOMEM;
>> +	}
>> +
>> +	/* disable IRQ */
>> +	writel(0, reg.ioaddr + XGMAC_DMA_INTR_ENA);
>> +
>> +	/* Disable the MAC core */
>> +	xgmac_mac_disable(reg.ioaddr);
>> +
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(vfio_platform_calxedaxgmac_reset);
>> +
>> +static int __init vfio_platform_calxedaxgmac_init(void)
>> +{
>> +	int (*register_reset)(char *, struct module *,
>> +				vfio_platform_reset_fn_t);
>> +	int ret;
>> +
>> +	register_reset = symbol_get(vfio_platform_register_reset);
>> +	if (!register_reset)
>> +		return -EINVAL;
>> +
>> +	ret = register_reset(CALXEDAXGMAC_COMPAT, THIS_MODULE,
>> +			     vfio_platform_calxedaxgmac_reset);
>> +
>> +	symbol_put(vfio_platform_register_reset);
>> +
>> +	return ret;
>> +}
>> +
>> +static void __exit vfio_platform_calxedaxgmac_exit(void)
>> +{
>> +	int (*unregister_reset)(char *);
>> +	int ret;
>> +
>> +	unregister_reset = symbol_get(vfio_platform_unregister_reset);
> 
> Shouldn't we have gotten the symbol during the module init so that this
> cannot fail?
yes makes sense
> 
>> +	if (!unregister_reset)
>> +		return;
>> +
>> +	ret = unregister_reset(CALXEDAXGMAC_COMPAT);
> 
> unregister_reset() should just return void
ok
> 
>> +
>> +	symbol_put(vfio_platform_unregister_reset);
>> +}
>> +
>> +module_init(vfio_platform_calxedaxgmac_init);
>> +module_exit(vfio_platform_calxedaxgmac_exit);
>> +
>> +MODULE_VERSION(DRIVER_VERSION);
>> +MODULE_LICENSE("GPL v2");
>> +MODULE_AUTHOR(DRIVER_AUTHOR);
>> +MODULE_DESCRIPTION(DRIVER_DESC);
> 
> 
> So a user needs to manually load this module to get a working reset for
> this device?  That's never going to happen.
The reset module is devised to be in-kernel or external. There is always
the choice to include all reset modules as soon as vfio-platform /amba
is chosen. What would be your preferred approach then? Do you consider
this approach using separate reset modules is not sensible? Do you think
we should put everything in the vfio-platform driver?

Thanks for the whole review.

Best Regards

Eric
> 

Re: [PATCH 5/5] VFIO: platform: VFIO platform Calxeda xgmac reset module

[Eric Auger]

Alex,
On 05/13/2015 08:33 PM, Alex Williamson wrote:
> On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
>> This patch introduces a module that registers and implements a basic
>> reset function for the Calxeda xgmac device. This latter basically disables
>> interrupts and stops DMA transfers.
>>
>> The reset function code is inherited from the native calxeda xgmac driver.
>>
>> Signed-off-by: Eric Auger <eric.auger@linaro.org>
>> ---
>>  drivers/vfio/platform/Kconfig                      |   2 +
>>  drivers/vfio/platform/Makefile                     |   2 +
>>  drivers/vfio/platform/reset/Kconfig                |   7 ++
>>  drivers/vfio/platform/reset/Makefile               |   5 +
>>  .../platform/reset/vfio_platform_calxedaxgmac.c    | 121 +++++++++++++++++++++
>>  5 files changed, 137 insertions(+)
>>  create mode 100644 drivers/vfio/platform/reset/Kconfig
>>  create mode 100644 drivers/vfio/platform/reset/Makefile
>>  create mode 100644 drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
>>
>> diff --git a/drivers/vfio/platform/Kconfig b/drivers/vfio/platform/Kconfig
>> index 9a4403e..1df7477 100644
>> --- a/drivers/vfio/platform/Kconfig
>> +++ b/drivers/vfio/platform/Kconfig
>> @@ -18,3 +18,5 @@ config VFIO_AMBA
>>  	  framework.
>>  
>>  	  If you don't know what to do here, say N.
>> +
>> +source "drivers/vfio/platform/reset/Kconfig"
>> diff --git a/drivers/vfio/platform/Makefile b/drivers/vfio/platform/Makefile
>> index 81de144..9ce8afe 100644
>> --- a/drivers/vfio/platform/Makefile
>> +++ b/drivers/vfio/platform/Makefile
>> @@ -2,7 +2,9 @@
>>  vfio-platform-y := vfio_platform.o vfio_platform_common.o vfio_platform_irq.o
>>  
>>  obj-$(CONFIG_VFIO_PLATFORM) += vfio-platform.o
>> +obj-$(CONFIG_VFIO_PLATFORM) += reset/
>>  
>>  vfio-amba-y := vfio_amba.o
>>  
>>  obj-$(CONFIG_VFIO_AMBA) += vfio-amba.o
>> +obj-$(CONFIG_VFIO_AMBA) += reset/
>> diff --git a/drivers/vfio/platform/reset/Kconfig b/drivers/vfio/platform/reset/Kconfig
>> new file mode 100644
>> index 0000000..746b96b
>> --- /dev/null
>> +++ b/drivers/vfio/platform/reset/Kconfig
>> @@ -0,0 +1,7 @@
>> +config VFIO_PLATFORM_CALXEDAXGMAC_RESET
>> +	tristate "VFIO support for calxeda xgmac reset"
>> +	depends on VFIO_PLATFORM
>> +	help
>> +	  Enables the VFIO platform driver to handle reset for Calxeda xgmac
>> +
>> +	  If you don't know what to do here, say N.
>> diff --git a/drivers/vfio/platform/reset/Makefile b/drivers/vfio/platform/reset/Makefile
>> new file mode 100644
>> index 0000000..2a486af
>> --- /dev/null
>> +++ b/drivers/vfio/platform/reset/Makefile
>> @@ -0,0 +1,5 @@
>> +vfio-platform-calxedaxgmac-y := vfio_platform_calxedaxgmac.o
>> +
>> +ccflags-y += -Idrivers/vfio/platform
>> +
>> +obj-$(CONFIG_VFIO_PLATFORM_CALXEDAXGMAC_RESET) += vfio-platform-calxedaxgmac.o
>> diff --git a/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c b/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
>> new file mode 100644
>> index 0000000..3c6e428
>> --- /dev/null
>> +++ b/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
>> @@ -0,0 +1,121 @@
>> +/*
>> + * VFIO platform driver specialized for Calxeda xgmac reset
>> + * reset code is inherited from calxeda xgmac native driver
>> + *
>> + * Copyright 2010-2011 Calxeda, Inc.
>> + * Copyright (c) 2015 Linaro Ltd.
>> + *              www.linaro.org
>> + *
>> + * This program is free software; you can redistribute it and/or modify it
>> + * under the terms and conditions of the GNU General Public License,
>> + * version 2, as published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope it will be useful, but WITHOUT
>> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
>> + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
>> + * more details.
>> + *
>> + * You should have received a copy of the GNU General Public License along with
>> + * this program.  If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <linux/module.h>
>> +#include <linux/kernel.h>
>> +#include <linux/init.h>
>> +#include <linux/io.h>
>> +
>> +#include "vfio_platform_private.h"
>> +
>> +#define DRIVER_VERSION  "0.1"
>> +#define DRIVER_AUTHOR   "Eric Auger <eric.auger@linaro.org>"
>> +#define DRIVER_DESC     "Reset support for Calxeda xgmac vfio platform device"
>> +
>> +#define CALXEDAXGMAC_COMPAT "calxeda,hb-xgmac"
>> +
>> +/* XGMAC Register definitions */
>> +#define XGMAC_CONTROL           0x00000000      /* MAC Configuration */
>> +
>> +/* DMA Control and Status Registers */
>> +#define XGMAC_DMA_CONTROL       0x00000f18      /* Ctrl (Operational Mode) */
>> +#define XGMAC_DMA_INTR_ENA      0x00000f1c      /* Interrupt Enable */
>> +
>> +/* DMA Control registe defines */
>> +#define DMA_CONTROL_ST          0x00002000      /* Start/Stop Transmission */
>> +#define DMA_CONTROL_SR          0x00000002      /* Start/Stop Receive */
>> +
>> +/* Common MAC defines */
>> +#define MAC_ENABLE_TX           0x00000008      /* Transmitter Enable */
>> +#define MAC_ENABLE_RX           0x00000004      /* Receiver Enable */
>> +
>> +static inline void xgmac_mac_disable(void __iomem *ioaddr)
>> +{
>> +	u32 value = readl(ioaddr + XGMAC_DMA_CONTROL);
>> +
>> +	value &= ~(DMA_CONTROL_ST | DMA_CONTROL_SR);
>> +	writel(value, ioaddr + XGMAC_DMA_CONTROL);
>> +
>> +	value = readl(ioaddr + XGMAC_CONTROL);
>> +	value &= ~(MAC_ENABLE_TX | MAC_ENABLE_RX);
>> +	writel(value, ioaddr + XGMAC_CONTROL);
>> +}
>> +
>> +int vfio_platform_calxedaxgmac_reset(struct vfio_platform_device *vdev)
>> +{
>> +	struct vfio_platform_region reg = vdev->regions[0];
>> +
>> +	if (!reg.ioaddr) {
>> +		reg.ioaddr =
>> +			ioremap_nocache(reg.addr, reg.size);
>> +		if (!reg.ioaddr)
>> +			return -ENOMEM;
>> +	}
>> +
>> +	/* disable IRQ */
>> +	writel(0, reg.ioaddr + XGMAC_DMA_INTR_ENA);
>> +
>> +	/* Disable the MAC core */
>> +	xgmac_mac_disable(reg.ioaddr);
>> +
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(vfio_platform_calxedaxgmac_reset);
>> +
>> +static int __init vfio_platform_calxedaxgmac_init(void)
>> +{
>> +	int (*register_reset)(char *, struct module *,
>> +				vfio_platform_reset_fn_t);
>> +	int ret;
>> +
>> +	register_reset = symbol_get(vfio_platform_register_reset);
>> +	if (!register_reset)
>> +		return -EINVAL;
>> +
>> +	ret = register_reset(CALXEDAXGMAC_COMPAT, THIS_MODULE,
>> +			     vfio_platform_calxedaxgmac_reset);
>> +
>> +	symbol_put(vfio_platform_register_reset);
>> +
>> +	return ret;
>> +}
>> +
>> +static void __exit vfio_platform_calxedaxgmac_exit(void)
>> +{
>> +	int (*unregister_reset)(char *);
>> +	int ret;
>> +
>> +	unregister_reset = symbol_get(vfio_platform_unregister_reset);
> 
> Shouldn't we have gotten the symbol during the module init so that this
> cannot fail?
yes makes sense
> 
>> +	if (!unregister_reset)
>> +		return;
>> +
>> +	ret = unregister_reset(CALXEDAXGMAC_COMPAT);
> 
> unregister_reset() should just return void
ok
> 
>> +
>> +	symbol_put(vfio_platform_unregister_reset);
>> +}
>> +
>> +module_init(vfio_platform_calxedaxgmac_init);
>> +module_exit(vfio_platform_calxedaxgmac_exit);
>> +
>> +MODULE_VERSION(DRIVER_VERSION);
>> +MODULE_LICENSE("GPL v2");
>> +MODULE_AUTHOR(DRIVER_AUTHOR);
>> +MODULE_DESCRIPTION(DRIVER_DESC);
> 
> 
> So a user needs to manually load this module to get a working reset for
> this device?  That's never going to happen.
The reset module is devised to be in-kernel or external. There is always
the choice to include all reset modules as soon as vfio-platform /amba
is chosen. What would be your preferred approach then? Do you consider
this approach using separate reset modules is not sensible? Do you think
we should put everything in the vfio-platform driver?

Thanks for the whole review.

Best Regards

Eric
> 

[PATCH 5/5] VFIO: platform: VFIO platform Calxeda xgmac reset module

[Eric Auger]

Alex,
On 05/13/2015 08:33 PM, Alex Williamson wrote:
> On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
>> This patch introduces a module that registers and implements a basic
>> reset function for the Calxeda xgmac device. This latter basically disables
>> interrupts and stops DMA transfers.
>>
>> The reset function code is inherited from the native calxeda xgmac driver.
>>
>> Signed-off-by: Eric Auger <eric.auger@linaro.org>
>> ---
>>  drivers/vfio/platform/Kconfig                      |   2 +
>>  drivers/vfio/platform/Makefile                     |   2 +
>>  drivers/vfio/platform/reset/Kconfig                |   7 ++
>>  drivers/vfio/platform/reset/Makefile               |   5 +
>>  .../platform/reset/vfio_platform_calxedaxgmac.c    | 121 +++++++++++++++++++++
>>  5 files changed, 137 insertions(+)
>>  create mode 100644 drivers/vfio/platform/reset/Kconfig
>>  create mode 100644 drivers/vfio/platform/reset/Makefile
>>  create mode 100644 drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
>>
>> diff --git a/drivers/vfio/platform/Kconfig b/drivers/vfio/platform/Kconfig
>> index 9a4403e..1df7477 100644
>> --- a/drivers/vfio/platform/Kconfig
>> +++ b/drivers/vfio/platform/Kconfig
>> @@ -18,3 +18,5 @@ config VFIO_AMBA
>>  	  framework.
>>  
>>  	  If you don't know what to do here, say N.
>> +
>> +source "drivers/vfio/platform/reset/Kconfig"
>> diff --git a/drivers/vfio/platform/Makefile b/drivers/vfio/platform/Makefile
>> index 81de144..9ce8afe 100644
>> --- a/drivers/vfio/platform/Makefile
>> +++ b/drivers/vfio/platform/Makefile
>> @@ -2,7 +2,9 @@
>>  vfio-platform-y := vfio_platform.o vfio_platform_common.o vfio_platform_irq.o
>>  
>>  obj-$(CONFIG_VFIO_PLATFORM) += vfio-platform.o
>> +obj-$(CONFIG_VFIO_PLATFORM) += reset/
>>  
>>  vfio-amba-y := vfio_amba.o
>>  
>>  obj-$(CONFIG_VFIO_AMBA) += vfio-amba.o
>> +obj-$(CONFIG_VFIO_AMBA) += reset/
>> diff --git a/drivers/vfio/platform/reset/Kconfig b/drivers/vfio/platform/reset/Kconfig
>> new file mode 100644
>> index 0000000..746b96b
>> --- /dev/null
>> +++ b/drivers/vfio/platform/reset/Kconfig
>> @@ -0,0 +1,7 @@
>> +config VFIO_PLATFORM_CALXEDAXGMAC_RESET
>> +	tristate "VFIO support for calxeda xgmac reset"
>> +	depends on VFIO_PLATFORM
>> +	help
>> +	  Enables the VFIO platform driver to handle reset for Calxeda xgmac
>> +
>> +	  If you don't know what to do here, say N.
>> diff --git a/drivers/vfio/platform/reset/Makefile b/drivers/vfio/platform/reset/Makefile
>> new file mode 100644
>> index 0000000..2a486af
>> --- /dev/null
>> +++ b/drivers/vfio/platform/reset/Makefile
>> @@ -0,0 +1,5 @@
>> +vfio-platform-calxedaxgmac-y := vfio_platform_calxedaxgmac.o
>> +
>> +ccflags-y += -Idrivers/vfio/platform
>> +
>> +obj-$(CONFIG_VFIO_PLATFORM_CALXEDAXGMAC_RESET) += vfio-platform-calxedaxgmac.o
>> diff --git a/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c b/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
>> new file mode 100644
>> index 0000000..3c6e428
>> --- /dev/null
>> +++ b/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
>> @@ -0,0 +1,121 @@
>> +/*
>> + * VFIO platform driver specialized for Calxeda xgmac reset
>> + * reset code is inherited from calxeda xgmac native driver
>> + *
>> + * Copyright 2010-2011 Calxeda, Inc.
>> + * Copyright (c) 2015 Linaro Ltd.
>> + *              www.linaro.org
>> + *
>> + * This program is free software; you can redistribute it and/or modify it
>> + * under the terms and conditions of the GNU General Public License,
>> + * version 2, as published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope it will be useful, but WITHOUT
>> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
>> + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
>> + * more details.
>> + *
>> + * You should have received a copy of the GNU General Public License along with
>> + * this program.  If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <linux/module.h>
>> +#include <linux/kernel.h>
>> +#include <linux/init.h>
>> +#include <linux/io.h>
>> +
>> +#include "vfio_platform_private.h"
>> +
>> +#define DRIVER_VERSION  "0.1"
>> +#define DRIVER_AUTHOR   "Eric Auger <eric.auger@linaro.org>"
>> +#define DRIVER_DESC     "Reset support for Calxeda xgmac vfio platform device"
>> +
>> +#define CALXEDAXGMAC_COMPAT "calxeda,hb-xgmac"
>> +
>> +/* XGMAC Register definitions */
>> +#define XGMAC_CONTROL           0x00000000      /* MAC Configuration */
>> +
>> +/* DMA Control and Status Registers */
>> +#define XGMAC_DMA_CONTROL       0x00000f18      /* Ctrl (Operational Mode) */
>> +#define XGMAC_DMA_INTR_ENA      0x00000f1c      /* Interrupt Enable */
>> +
>> +/* DMA Control registe defines */
>> +#define DMA_CONTROL_ST          0x00002000      /* Start/Stop Transmission */
>> +#define DMA_CONTROL_SR          0x00000002      /* Start/Stop Receive */
>> +
>> +/* Common MAC defines */
>> +#define MAC_ENABLE_TX           0x00000008      /* Transmitter Enable */
>> +#define MAC_ENABLE_RX           0x00000004      /* Receiver Enable */
>> +
>> +static inline void xgmac_mac_disable(void __iomem *ioaddr)
>> +{
>> +	u32 value = readl(ioaddr + XGMAC_DMA_CONTROL);
>> +
>> +	value &= ~(DMA_CONTROL_ST | DMA_CONTROL_SR);
>> +	writel(value, ioaddr + XGMAC_DMA_CONTROL);
>> +
>> +	value = readl(ioaddr + XGMAC_CONTROL);
>> +	value &= ~(MAC_ENABLE_TX | MAC_ENABLE_RX);
>> +	writel(value, ioaddr + XGMAC_CONTROL);
>> +}
>> +
>> +int vfio_platform_calxedaxgmac_reset(struct vfio_platform_device *vdev)
>> +{
>> +	struct vfio_platform_region reg = vdev->regions[0];
>> +
>> +	if (!reg.ioaddr) {
>> +		reg.ioaddr =
>> +			ioremap_nocache(reg.addr, reg.size);
>> +		if (!reg.ioaddr)
>> +			return -ENOMEM;
>> +	}
>> +
>> +	/* disable IRQ */
>> +	writel(0, reg.ioaddr + XGMAC_DMA_INTR_ENA);
>> +
>> +	/* Disable the MAC core */
>> +	xgmac_mac_disable(reg.ioaddr);
>> +
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(vfio_platform_calxedaxgmac_reset);
>> +
>> +static int __init vfio_platform_calxedaxgmac_init(void)
>> +{
>> +	int (*register_reset)(char *, struct module *,
>> +				vfio_platform_reset_fn_t);
>> +	int ret;
>> +
>> +	register_reset = symbol_get(vfio_platform_register_reset);
>> +	if (!register_reset)
>> +		return -EINVAL;
>> +
>> +	ret = register_reset(CALXEDAXGMAC_COMPAT, THIS_MODULE,
>> +			     vfio_platform_calxedaxgmac_reset);
>> +
>> +	symbol_put(vfio_platform_register_reset);
>> +
>> +	return ret;
>> +}
>> +
>> +static void __exit vfio_platform_calxedaxgmac_exit(void)
>> +{
>> +	int (*unregister_reset)(char *);
>> +	int ret;
>> +
>> +	unregister_reset = symbol_get(vfio_platform_unregister_reset);
> 
> Shouldn't we have gotten the symbol during the module init so that this
> cannot fail?
yes makes sense
> 
>> +	if (!unregister_reset)
>> +		return;
>> +
>> +	ret = unregister_reset(CALXEDAXGMAC_COMPAT);
> 
> unregister_reset() should just return void
ok
> 
>> +
>> +	symbol_put(vfio_platform_unregister_reset);
>> +}
>> +
>> +module_init(vfio_platform_calxedaxgmac_init);
>> +module_exit(vfio_platform_calxedaxgmac_exit);
>> +
>> +MODULE_VERSION(DRIVER_VERSION);
>> +MODULE_LICENSE("GPL v2");
>> +MODULE_AUTHOR(DRIVER_AUTHOR);
>> +MODULE_DESCRIPTION(DRIVER_DESC);
> 
> 
> So a user needs to manually load this module to get a working reset for
> this device?  That's never going to happen.
The reset module is devised to be in-kernel or external. There is always
the choice to include all reset modules as soon as vfio-platform /amba
is chosen. What would be your preferred approach then? Do you consider
this approach using separate reset modules is not sensible? Do you think
we should put everything in the vfio-platform driver?

Thanks for the whole review.

Best Regards

Eric
> 

Re: [PATCH 5/5] VFIO: platform: VFIO platform Calxeda xgmac reset module

[Alex Williamson]

On Thu, 2015-05-14 at 11:06 +0200, Eric Auger wrote:
> Alex,
> On 05/13/2015 08:33 PM, Alex Williamson wrote:
> > On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
> >> This patch introduces a module that registers and implements a basic
> >> reset function for the Calxeda xgmac device. This latter basically disables
> >> interrupts and stops DMA transfers.
> >>
> >> The reset function code is inherited from the native calxeda xgmac driver.
> >>
> >> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> >> ---
> >>  drivers/vfio/platform/Kconfig                      |   2 +
> >>  drivers/vfio/platform/Makefile                     |   2 +
> >>  drivers/vfio/platform/reset/Kconfig                |   7 ++
> >>  drivers/vfio/platform/reset/Makefile               |   5 +
> >>  .../platform/reset/vfio_platform_calxedaxgmac.c    | 121 +++++++++++++++++++++
> >>  5 files changed, 137 insertions(+)
> >>  create mode 100644 drivers/vfio/platform/reset/Kconfig
> >>  create mode 100644 drivers/vfio/platform/reset/Makefile
> >>  create mode 100644 drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
> >>
> >> diff --git a/drivers/vfio/platform/Kconfig b/drivers/vfio/platform/Kconfig
> >> index 9a4403e..1df7477 100644
> >> --- a/drivers/vfio/platform/Kconfig
> >> +++ b/drivers/vfio/platform/Kconfig
> >> @@ -18,3 +18,5 @@ config VFIO_AMBA
> >>  	  framework.
> >>  
> >>  	  If you don't know what to do here, say N.
> >> +
> >> +source "drivers/vfio/platform/reset/Kconfig"
> >> diff --git a/drivers/vfio/platform/Makefile b/drivers/vfio/platform/Makefile
> >> index 81de144..9ce8afe 100644
> >> --- a/drivers/vfio/platform/Makefile
> >> +++ b/drivers/vfio/platform/Makefile
> >> @@ -2,7 +2,9 @@
> >>  vfio-platform-y := vfio_platform.o vfio_platform_common.o vfio_platform_irq.o
> >>  
> >>  obj-$(CONFIG_VFIO_PLATFORM) += vfio-platform.o
> >> +obj-$(CONFIG_VFIO_PLATFORM) += reset/
> >>  
> >>  vfio-amba-y := vfio_amba.o
> >>  
> >>  obj-$(CONFIG_VFIO_AMBA) += vfio-amba.o
> >> +obj-$(CONFIG_VFIO_AMBA) += reset/
> >> diff --git a/drivers/vfio/platform/reset/Kconfig b/drivers/vfio/platform/reset/Kconfig
> >> new file mode 100644
> >> index 0000000..746b96b
> >> --- /dev/null
> >> +++ b/drivers/vfio/platform/reset/Kconfig
> >> @@ -0,0 +1,7 @@
> >> +config VFIO_PLATFORM_CALXEDAXGMAC_RESET
> >> +	tristate "VFIO support for calxeda xgmac reset"
> >> +	depends on VFIO_PLATFORM
> >> +	help
> >> +	  Enables the VFIO platform driver to handle reset for Calxeda xgmac
> >> +
> >> +	  If you don't know what to do here, say N.
> >> diff --git a/drivers/vfio/platform/reset/Makefile b/drivers/vfio/platform/reset/Makefile
> >> new file mode 100644
> >> index 0000000..2a486af
> >> --- /dev/null
> >> +++ b/drivers/vfio/platform/reset/Makefile
> >> @@ -0,0 +1,5 @@
> >> +vfio-platform-calxedaxgmac-y := vfio_platform_calxedaxgmac.o
> >> +
> >> +ccflags-y += -Idrivers/vfio/platform
> >> +
> >> +obj-$(CONFIG_VFIO_PLATFORM_CALXEDAXGMAC_RESET) += vfio-platform-calxedaxgmac.o
> >> diff --git a/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c b/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
> >> new file mode 100644
> >> index 0000000..3c6e428
> >> --- /dev/null
> >> +++ b/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
> >> @@ -0,0 +1,121 @@
> >> +/*
> >> + * VFIO platform driver specialized for Calxeda xgmac reset
> >> + * reset code is inherited from calxeda xgmac native driver
> >> + *
> >> + * Copyright 2010-2011 Calxeda, Inc.
> >> + * Copyright (c) 2015 Linaro Ltd.
> >> + *              www.linaro.org
> >> + *
> >> + * This program is free software; you can redistribute it and/or modify it
> >> + * under the terms and conditions of the GNU General Public License,
> >> + * version 2, as published by the Free Software Foundation.
> >> + *
> >> + * This program is distributed in the hope it will be useful, but WITHOUT
> >> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
> >> + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
> >> + * more details.
> >> + *
> >> + * You should have received a copy of the GNU General Public License along with
> >> + * this program.  If not, see <http://www.gnu.org/licenses/>.
> >> + */
> >> +
> >> +#include <linux/module.h>
> >> +#include <linux/kernel.h>
> >> +#include <linux/init.h>
> >> +#include <linux/io.h>
> >> +
> >> +#include "vfio_platform_private.h"
> >> +
> >> +#define DRIVER_VERSION  "0.1"
> >> +#define DRIVER_AUTHOR   "Eric Auger <eric.auger@linaro.org>"
> >> +#define DRIVER_DESC     "Reset support for Calxeda xgmac vfio platform device"
> >> +
> >> +#define CALXEDAXGMAC_COMPAT "calxeda,hb-xgmac"
> >> +
> >> +/* XGMAC Register definitions */
> >> +#define XGMAC_CONTROL           0x00000000      /* MAC Configuration */
> >> +
> >> +/* DMA Control and Status Registers */
> >> +#define XGMAC_DMA_CONTROL       0x00000f18      /* Ctrl (Operational Mode) */
> >> +#define XGMAC_DMA_INTR_ENA      0x00000f1c      /* Interrupt Enable */
> >> +
> >> +/* DMA Control registe defines */
> >> +#define DMA_CONTROL_ST          0x00002000      /* Start/Stop Transmission */
> >> +#define DMA_CONTROL_SR          0x00000002      /* Start/Stop Receive */
> >> +
> >> +/* Common MAC defines */
> >> +#define MAC_ENABLE_TX           0x00000008      /* Transmitter Enable */
> >> +#define MAC_ENABLE_RX           0x00000004      /* Receiver Enable */
> >> +
> >> +static inline void xgmac_mac_disable(void __iomem *ioaddr)
> >> +{
> >> +	u32 value = readl(ioaddr + XGMAC_DMA_CONTROL);
> >> +
> >> +	value &= ~(DMA_CONTROL_ST | DMA_CONTROL_SR);
> >> +	writel(value, ioaddr + XGMAC_DMA_CONTROL);
> >> +
> >> +	value = readl(ioaddr + XGMAC_CONTROL);
> >> +	value &= ~(MAC_ENABLE_TX | MAC_ENABLE_RX);
> >> +	writel(value, ioaddr + XGMAC_CONTROL);
> >> +}
> >> +
> >> +int vfio_platform_calxedaxgmac_reset(struct vfio_platform_device *vdev)
> >> +{
> >> +	struct vfio_platform_region reg = vdev->regions[0];
> >> +
> >> +	if (!reg.ioaddr) {
> >> +		reg.ioaddr =
> >> +			ioremap_nocache(reg.addr, reg.size);
> >> +		if (!reg.ioaddr)
> >> +			return -ENOMEM;
> >> +	}
> >> +
> >> +	/* disable IRQ */
> >> +	writel(0, reg.ioaddr + XGMAC_DMA_INTR_ENA);
> >> +
> >> +	/* Disable the MAC core */
> >> +	xgmac_mac_disable(reg.ioaddr);
> >> +
> >> +	return 0;
> >> +}
> >> +EXPORT_SYMBOL_GPL(vfio_platform_calxedaxgmac_reset);
> >> +
> >> +static int __init vfio_platform_calxedaxgmac_init(void)
> >> +{
> >> +	int (*register_reset)(char *, struct module *,
> >> +				vfio_platform_reset_fn_t);
> >> +	int ret;
> >> +
> >> +	register_reset = symbol_get(vfio_platform_register_reset);
> >> +	if (!register_reset)
> >> +		return -EINVAL;
> >> +
> >> +	ret = register_reset(CALXEDAXGMAC_COMPAT, THIS_MODULE,
> >> +			     vfio_platform_calxedaxgmac_reset);
> >> +
> >> +	symbol_put(vfio_platform_register_reset);
> >> +
> >> +	return ret;
> >> +}
> >> +
> >> +static void __exit vfio_platform_calxedaxgmac_exit(void)
> >> +{
> >> +	int (*unregister_reset)(char *);
> >> +	int ret;
> >> +
> >> +	unregister_reset = symbol_get(vfio_platform_unregister_reset);
> > 
> > Shouldn't we have gotten the symbol during the module init so that this
> > cannot fail?
> yes makes sense
> > 
> >> +	if (!unregister_reset)
> >> +		return;
> >> +
> >> +	ret = unregister_reset(CALXEDAXGMAC_COMPAT);
> > 
> > unregister_reset() should just return void
> ok
> > 
> >> +
> >> +	symbol_put(vfio_platform_unregister_reset);
> >> +}
> >> +
> >> +module_init(vfio_platform_calxedaxgmac_init);
> >> +module_exit(vfio_platform_calxedaxgmac_exit);
> >> +
> >> +MODULE_VERSION(DRIVER_VERSION);
> >> +MODULE_LICENSE("GPL v2");
> >> +MODULE_AUTHOR(DRIVER_AUTHOR);
> >> +MODULE_DESCRIPTION(DRIVER_DESC);
> > 
> > 
> > So a user needs to manually load this module to get a working reset for
> > this device?  That's never going to happen.
> The reset module is devised to be in-kernel or external. There is always
> the choice to include all reset modules as soon as vfio-platform /amba
> is chosen. What would be your preferred approach then? Do you consider
> this approach using separate reset modules is not sensible? Do you think
> we should put everything in the vfio-platform driver?

I respect your attempt to keep the code slim and modular by making the
reset functionality loadable, but we can't expect users to figure out
what module they need to load to make their device work.  It needs to be
automatic.  One way we could achieve that and still keep your premise of
loadable reset modules would be a lookup table in vfio-platform to
translate a device compatibility ID to a reset module name.  We could
then do a request_module() to auto-load the reset module if one exists.
Maybe the ID table could even live in the reset driver, much like PCI
IDs live in those drivers.  We also have the complication that there's
no direct module dependency for the reset module(s), so they would need
to be explicitly listed to install them into an initramfs.

If we take that approach, we're quickly building infrastructure that
gets more bloated than a few reset functions buried directly into
vfio-platform.  Then we may wonder how many reset functions are we
reasonably going to get.  If you don't have more than a couple in mind,
maybe we should just embed them into vfio-platform and worry about
creating something more dynamic when the need presents itself.  Thanks,

Alex

[PATCH 5/5] VFIO: platform: VFIO platform Calxeda xgmac reset module

[Alex Williamson]

On Thu, 2015-05-14 at 11:06 +0200, Eric Auger wrote:
> Alex,
> On 05/13/2015 08:33 PM, Alex Williamson wrote:
> > On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
> >> This patch introduces a module that registers and implements a basic
> >> reset function for the Calxeda xgmac device. This latter basically disables
> >> interrupts and stops DMA transfers.
> >>
> >> The reset function code is inherited from the native calxeda xgmac driver.
> >>
> >> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> >> ---
> >>  drivers/vfio/platform/Kconfig                      |   2 +
> >>  drivers/vfio/platform/Makefile                     |   2 +
> >>  drivers/vfio/platform/reset/Kconfig                |   7 ++
> >>  drivers/vfio/platform/reset/Makefile               |   5 +
> >>  .../platform/reset/vfio_platform_calxedaxgmac.c    | 121 +++++++++++++++++++++
> >>  5 files changed, 137 insertions(+)
> >>  create mode 100644 drivers/vfio/platform/reset/Kconfig
> >>  create mode 100644 drivers/vfio/platform/reset/Makefile
> >>  create mode 100644 drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
> >>
> >> diff --git a/drivers/vfio/platform/Kconfig b/drivers/vfio/platform/Kconfig
> >> index 9a4403e..1df7477 100644
> >> --- a/drivers/vfio/platform/Kconfig
> >> +++ b/drivers/vfio/platform/Kconfig
> >> @@ -18,3 +18,5 @@ config VFIO_AMBA
> >>  	  framework.
> >>  
> >>  	  If you don't know what to do here, say N.
> >> +
> >> +source "drivers/vfio/platform/reset/Kconfig"
> >> diff --git a/drivers/vfio/platform/Makefile b/drivers/vfio/platform/Makefile
> >> index 81de144..9ce8afe 100644
> >> --- a/drivers/vfio/platform/Makefile
> >> +++ b/drivers/vfio/platform/Makefile
> >> @@ -2,7 +2,9 @@
> >>  vfio-platform-y := vfio_platform.o vfio_platform_common.o vfio_platform_irq.o
> >>  
> >>  obj-$(CONFIG_VFIO_PLATFORM) += vfio-platform.o
> >> +obj-$(CONFIG_VFIO_PLATFORM) += reset/
> >>  
> >>  vfio-amba-y := vfio_amba.o
> >>  
> >>  obj-$(CONFIG_VFIO_AMBA) += vfio-amba.o
> >> +obj-$(CONFIG_VFIO_AMBA) += reset/
> >> diff --git a/drivers/vfio/platform/reset/Kconfig b/drivers/vfio/platform/reset/Kconfig
> >> new file mode 100644
> >> index 0000000..746b96b
> >> --- /dev/null
> >> +++ b/drivers/vfio/platform/reset/Kconfig
> >> @@ -0,0 +1,7 @@
> >> +config VFIO_PLATFORM_CALXEDAXGMAC_RESET
> >> +	tristate "VFIO support for calxeda xgmac reset"
> >> +	depends on VFIO_PLATFORM
> >> +	help
> >> +	  Enables the VFIO platform driver to handle reset for Calxeda xgmac
> >> +
> >> +	  If you don't know what to do here, say N.
> >> diff --git a/drivers/vfio/platform/reset/Makefile b/drivers/vfio/platform/reset/Makefile
> >> new file mode 100644
> >> index 0000000..2a486af
> >> --- /dev/null
> >> +++ b/drivers/vfio/platform/reset/Makefile
> >> @@ -0,0 +1,5 @@
> >> +vfio-platform-calxedaxgmac-y := vfio_platform_calxedaxgmac.o
> >> +
> >> +ccflags-y += -Idrivers/vfio/platform
> >> +
> >> +obj-$(CONFIG_VFIO_PLATFORM_CALXEDAXGMAC_RESET) += vfio-platform-calxedaxgmac.o
> >> diff --git a/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c b/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
> >> new file mode 100644
> >> index 0000000..3c6e428
> >> --- /dev/null
> >> +++ b/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
> >> @@ -0,0 +1,121 @@
> >> +/*
> >> + * VFIO platform driver specialized for Calxeda xgmac reset
> >> + * reset code is inherited from calxeda xgmac native driver
> >> + *
> >> + * Copyright 2010-2011 Calxeda, Inc.
> >> + * Copyright (c) 2015 Linaro Ltd.
> >> + *              www.linaro.org
> >> + *
> >> + * This program is free software; you can redistribute it and/or modify it
> >> + * under the terms and conditions of the GNU General Public License,
> >> + * version 2, as published by the Free Software Foundation.
> >> + *
> >> + * This program is distributed in the hope it will be useful, but WITHOUT
> >> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
> >> + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
> >> + * more details.
> >> + *
> >> + * You should have received a copy of the GNU General Public License along with
> >> + * this program.  If not, see <http://www.gnu.org/licenses/>.
> >> + */
> >> +
> >> +#include <linux/module.h>
> >> +#include <linux/kernel.h>
> >> +#include <linux/init.h>
> >> +#include <linux/io.h>
> >> +
> >> +#include "vfio_platform_private.h"
> >> +
> >> +#define DRIVER_VERSION  "0.1"
> >> +#define DRIVER_AUTHOR   "Eric Auger <eric.auger@linaro.org>"
> >> +#define DRIVER_DESC     "Reset support for Calxeda xgmac vfio platform device"
> >> +
> >> +#define CALXEDAXGMAC_COMPAT "calxeda,hb-xgmac"
> >> +
> >> +/* XGMAC Register definitions */
> >> +#define XGMAC_CONTROL           0x00000000      /* MAC Configuration */
> >> +
> >> +/* DMA Control and Status Registers */
> >> +#define XGMAC_DMA_CONTROL       0x00000f18      /* Ctrl (Operational Mode) */
> >> +#define XGMAC_DMA_INTR_ENA      0x00000f1c      /* Interrupt Enable */
> >> +
> >> +/* DMA Control registe defines */
> >> +#define DMA_CONTROL_ST          0x00002000      /* Start/Stop Transmission */
> >> +#define DMA_CONTROL_SR          0x00000002      /* Start/Stop Receive */
> >> +
> >> +/* Common MAC defines */
> >> +#define MAC_ENABLE_TX           0x00000008      /* Transmitter Enable */
> >> +#define MAC_ENABLE_RX           0x00000004      /* Receiver Enable */
> >> +
> >> +static inline void xgmac_mac_disable(void __iomem *ioaddr)
> >> +{
> >> +	u32 value = readl(ioaddr + XGMAC_DMA_CONTROL);
> >> +
> >> +	value &= ~(DMA_CONTROL_ST | DMA_CONTROL_SR);
> >> +	writel(value, ioaddr + XGMAC_DMA_CONTROL);
> >> +
> >> +	value = readl(ioaddr + XGMAC_CONTROL);
> >> +	value &= ~(MAC_ENABLE_TX | MAC_ENABLE_RX);
> >> +	writel(value, ioaddr + XGMAC_CONTROL);
> >> +}
> >> +
> >> +int vfio_platform_calxedaxgmac_reset(struct vfio_platform_device *vdev)
> >> +{
> >> +	struct vfio_platform_region reg = vdev->regions[0];
> >> +
> >> +	if (!reg.ioaddr) {
> >> +		reg.ioaddr =
> >> +			ioremap_nocache(reg.addr, reg.size);
> >> +		if (!reg.ioaddr)
> >> +			return -ENOMEM;
> >> +	}
> >> +
> >> +	/* disable IRQ */
> >> +	writel(0, reg.ioaddr + XGMAC_DMA_INTR_ENA);
> >> +
> >> +	/* Disable the MAC core */
> >> +	xgmac_mac_disable(reg.ioaddr);
> >> +
> >> +	return 0;
> >> +}
> >> +EXPORT_SYMBOL_GPL(vfio_platform_calxedaxgmac_reset);
> >> +
> >> +static int __init vfio_platform_calxedaxgmac_init(void)
> >> +{
> >> +	int (*register_reset)(char *, struct module *,
> >> +				vfio_platform_reset_fn_t);
> >> +	int ret;
> >> +
> >> +	register_reset = symbol_get(vfio_platform_register_reset);
> >> +	if (!register_reset)
> >> +		return -EINVAL;
> >> +
> >> +	ret = register_reset(CALXEDAXGMAC_COMPAT, THIS_MODULE,
> >> +			     vfio_platform_calxedaxgmac_reset);
> >> +
> >> +	symbol_put(vfio_platform_register_reset);
> >> +
> >> +	return ret;
> >> +}
> >> +
> >> +static void __exit vfio_platform_calxedaxgmac_exit(void)
> >> +{
> >> +	int (*unregister_reset)(char *);
> >> +	int ret;
> >> +
> >> +	unregister_reset = symbol_get(vfio_platform_unregister_reset);
> > 
> > Shouldn't we have gotten the symbol during the module init so that this
> > cannot fail?
> yes makes sense
> > 
> >> +	if (!unregister_reset)
> >> +		return;
> >> +
> >> +	ret = unregister_reset(CALXEDAXGMAC_COMPAT);
> > 
> > unregister_reset() should just return void
> ok
> > 
> >> +
> >> +	symbol_put(vfio_platform_unregister_reset);
> >> +}
> >> +
> >> +module_init(vfio_platform_calxedaxgmac_init);
> >> +module_exit(vfio_platform_calxedaxgmac_exit);
> >> +
> >> +MODULE_VERSION(DRIVER_VERSION);
> >> +MODULE_LICENSE("GPL v2");
> >> +MODULE_AUTHOR(DRIVER_AUTHOR);
> >> +MODULE_DESCRIPTION(DRIVER_DESC);
> > 
> > 
> > So a user needs to manually load this module to get a working reset for
> > this device?  That's never going to happen.
> The reset module is devised to be in-kernel or external. There is always
> the choice to include all reset modules as soon as vfio-platform /amba
> is chosen. What would be your preferred approach then? Do you consider
> this approach using separate reset modules is not sensible? Do you think
> we should put everything in the vfio-platform driver?

I respect your attempt to keep the code slim and modular by making the
reset functionality loadable, but we can't expect users to figure out
what module they need to load to make their device work.  It needs to be
automatic.  One way we could achieve that and still keep your premise of
loadable reset modules would be a lookup table in vfio-platform to
translate a device compatibility ID to a reset module name.  We could
then do a request_module() to auto-load the reset module if one exists.
Maybe the ID table could even live in the reset driver, much like PCI
IDs live in those drivers.  We also have the complication that there's
no direct module dependency for the reset module(s), so they would need
to be explicitly listed to install them into an initramfs.

If we take that approach, we're quickly building infrastructure that
gets more bloated than a few reset functions buried directly into
vfio-platform.  Then we may wonder how many reset functions are we
reasonably going to get.  If you don't have more than a couple in mind,
maybe we should just embed them into vfio-platform and worry about
creating something more dynamic when the need presents itself.  Thanks,

Alex

Re: [PATCH 4/5] VFIO: platform: populate reset function according to compat

[Alex Williamson]

On Thu, 2015-05-14 at 10:57 +0200, Eric Auger wrote:
> On 05/13/2015 08:33 PM, Alex Williamson wrote:
> > On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
> >> Add the reset function lookup according to the device compat
> >> string. This lookup is added at different places:
> >> - on VFIO_DEVICE_GET_INFO
> >> - on VFIO_DEVICE_RESET
> >> - on device release
> >>
> >> A reference to the module implementing the reset function is taken
> >> on first reset function lookup and released on vfio platform device
> >> release.
> >>
> >> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> >> ---
> >>  drivers/vfio/platform/vfio_platform_common.c | 50 ++++++++++++++++++++++++++++
> >>  1 file changed, 50 insertions(+)
> >>
> >> diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c
> >> index 0d10018..bd7e44c 100644
> >> --- a/drivers/vfio/platform/vfio_platform_common.c
> >> +++ b/drivers/vfio/platform/vfio_platform_common.c
> >> @@ -28,6 +28,52 @@ LIST_HEAD(reset_list);
> >>  
> >>  static DEFINE_MUTEX(driver_lock);
> >>  
> >> +static vfio_platform_reset_fn_t vfio_platform_lookup_reset(const char *compat)
> >> +{
> >> +	struct vfio_platform_reset_node *iter;
> >> +
> >> +	list_for_each_entry(iter, &reset_list, link) {
> > 
> > Racy
> ok
> > 
> >> +		if (!strcmp(iter->compat, compat) &&
> >> +			try_module_get(iter->owner))
> >> +			return iter->reset;
> >> +	}
> >> +
> >> +	return ERR_PTR(-ENODEV);
> > 
> > return NULL imo
> ok
> > 
> >> +}
> >> +
> >> +static vfio_platform_reset_fn_t vfio_platform_get_reset(
> >> +					struct vfio_platform_device *vdev)
> >> +{
> >> +	struct device *dev = vdev->get_device(vdev);
> >> +	const char *compat_str_array[2];
> >> +	vfio_platform_reset_fn_t reset;
> >> +	int ret;
> >> +
> >> +	if (!IS_ERR_OR_NULL(vdev->reset))
> >> +		return vdev->reset;
> >> +
> >> +	ret = device_property_read_string_array(dev, "compatible",
> >> +						compat_str_array, 2);
> >> +	if (!ret)
> >> +		return ERR_PTR(ret);
> >> +
> >> +	reset = vfio_platform_lookup_reset(compat_str_array[0]);
> >> +	return reset;
> > 
> > Something got allocated into compat_str_array and gets leaked here.
> is there any allocation? device_property_read_string_array does not
> return -ENOMEM.

Yeah, since they're const I guess maybe it's just setting a pointer.  It
troubles me a little bit that nobody else seems to be using this
device_property_read_string_array() interface.
 
> >> +}
> >> +
> >> +static void vfio_platform_put_reset(struct vfio_platform_device *vdev)
> >> +{
> >> +	struct vfio_platform_reset_node *iter;
> >> +
> >> +	list_for_each_entry(iter, &reset_list, link) {
> > 
> > Racy
> ok
> > 
> >> +		if (iter->reset == vdev->reset) {
> >> +			module_put(iter->owner);
> >> +			vdev->reset = NULL;
> >> +			return;
> >> +		}
> >> +	}
> >> +}
> >> +
> >>  static int vfio_platform_regions_init(struct vfio_platform_device *vdev)
> >>  {
> >>  	int cnt = 0, i;
> >> @@ -103,10 +149,12 @@ static void vfio_platform_release(void *device_data)
> >>  	mutex_lock(&driver_lock);
> >>  
> >>  	if (!(--vdev->refcnt)) {
> >> +		vdev->reset = vfio_platform_get_reset(vdev);
> >>  		if (!IS_ERR_OR_NULL(vdev->reset))
> >>  			vdev->reset(vdev);
> >>  		vfio_platform_regions_cleanup(vdev);
> >>  		vfio_platform_irq_cleanup(vdev);
> >> +		vfio_platform_put_reset(vdev);
> >>  	}
> >>  
> >>  	mutex_unlock(&driver_lock);
> >> @@ -164,6 +212,7 @@ static long vfio_platform_ioctl(void *device_data,
> >>  		if (info.argsz < minsz)
> >>  			return -EINVAL;
> >>  
> >> +		vdev->reset = vfio_platform_get_reset(vdev);
> >>  		if (!IS_ERR_OR_NULL(vdev->reset))
> >>  			vdev->flags |= VFIO_DEVICE_FLAGS_RESET;
> >>  		info.flags = vdev->flags;
> >> @@ -260,6 +309,7 @@ static long vfio_platform_ioctl(void *device_data,
> >>  		return ret;
> >>  
> >>  	} else if (cmd == VFIO_DEVICE_RESET) {
> >> +		vdev->reset = vfio_platform_get_reset(vdev);
> >>  		if (!IS_ERR_OR_NULL(vdev->reset))
> >>  			return vdev->reset(vdev);
> >>  		else
> > 
> > I count 3 gets and 1 put, isn't the module reference count increase
> > showing that?
> 
> vfio_platform_get_reset simply returns if the function pointer already
> is populated so there is no systematic ref increment.

Ah, so it does.

> 
>   This looks like it hasn't been tested.
> 
> It did testing with external and in-kernel modules through
>   Why would we do a
> > get every time we want to do a reset?
> 
> My doubt were about the order of probing between the
> vfio-platform_driver and the vfio reset module? This question was the
> rationale of this implementation choice. But again the actual ref count
> increment is devised to be done once on the first entry point (iotcl or
> internal release)

I think we need to enforce the ordering; the reset function should be
set on device open via request_module() and try_module_get() to make
sure it is present and can't go away (or make it all a static part of
vfio-platform).  A user doesn't expect the reset capability advertised
in the vfio device info struct to change over time.  Thanks,

Alex

[PATCH 4/5] VFIO: platform: populate reset function according to compat

[Alex Williamson]

On Thu, 2015-05-14 at 10:57 +0200, Eric Auger wrote:
> On 05/13/2015 08:33 PM, Alex Williamson wrote:
> > On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
> >> Add the reset function lookup according to the device compat
> >> string. This lookup is added at different places:
> >> - on VFIO_DEVICE_GET_INFO
> >> - on VFIO_DEVICE_RESET
> >> - on device release
> >>
> >> A reference to the module implementing the reset function is taken
> >> on first reset function lookup and released on vfio platform device
> >> release.
> >>
> >> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> >> ---
> >>  drivers/vfio/platform/vfio_platform_common.c | 50 ++++++++++++++++++++++++++++
> >>  1 file changed, 50 insertions(+)
> >>
> >> diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c
> >> index 0d10018..bd7e44c 100644
> >> --- a/drivers/vfio/platform/vfio_platform_common.c
> >> +++ b/drivers/vfio/platform/vfio_platform_common.c
> >> @@ -28,6 +28,52 @@ LIST_HEAD(reset_list);
> >>  
> >>  static DEFINE_MUTEX(driver_lock);
> >>  
> >> +static vfio_platform_reset_fn_t vfio_platform_lookup_reset(const char *compat)
> >> +{
> >> +	struct vfio_platform_reset_node *iter;
> >> +
> >> +	list_for_each_entry(iter, &reset_list, link) {
> > 
> > Racy
> ok
> > 
> >> +		if (!strcmp(iter->compat, compat) &&
> >> +			try_module_get(iter->owner))
> >> +			return iter->reset;
> >> +	}
> >> +
> >> +	return ERR_PTR(-ENODEV);
> > 
> > return NULL imo
> ok
> > 
> >> +}
> >> +
> >> +static vfio_platform_reset_fn_t vfio_platform_get_reset(
> >> +					struct vfio_platform_device *vdev)
> >> +{
> >> +	struct device *dev = vdev->get_device(vdev);
> >> +	const char *compat_str_array[2];
> >> +	vfio_platform_reset_fn_t reset;
> >> +	int ret;
> >> +
> >> +	if (!IS_ERR_OR_NULL(vdev->reset))
> >> +		return vdev->reset;
> >> +
> >> +	ret = device_property_read_string_array(dev, "compatible",
> >> +						compat_str_array, 2);
> >> +	if (!ret)
> >> +		return ERR_PTR(ret);
> >> +
> >> +	reset = vfio_platform_lookup_reset(compat_str_array[0]);
> >> +	return reset;
> > 
> > Something got allocated into compat_str_array and gets leaked here.
> is there any allocation? device_property_read_string_array does not
> return -ENOMEM.

Yeah, since they're const I guess maybe it's just setting a pointer.  It
troubles me a little bit that nobody else seems to be using this
device_property_read_string_array() interface.
 
> >> +}
> >> +
> >> +static void vfio_platform_put_reset(struct vfio_platform_device *vdev)
> >> +{
> >> +	struct vfio_platform_reset_node *iter;
> >> +
> >> +	list_for_each_entry(iter, &reset_list, link) {
> > 
> > Racy
> ok
> > 
> >> +		if (iter->reset == vdev->reset) {
> >> +			module_put(iter->owner);
> >> +			vdev->reset = NULL;
> >> +			return;
> >> +		}
> >> +	}
> >> +}
> >> +
> >>  static int vfio_platform_regions_init(struct vfio_platform_device *vdev)
> >>  {
> >>  	int cnt = 0, i;
> >> @@ -103,10 +149,12 @@ static void vfio_platform_release(void *device_data)
> >>  	mutex_lock(&driver_lock);
> >>  
> >>  	if (!(--vdev->refcnt)) {
> >> +		vdev->reset = vfio_platform_get_reset(vdev);
> >>  		if (!IS_ERR_OR_NULL(vdev->reset))
> >>  			vdev->reset(vdev);
> >>  		vfio_platform_regions_cleanup(vdev);
> >>  		vfio_platform_irq_cleanup(vdev);
> >> +		vfio_platform_put_reset(vdev);
> >>  	}
> >>  
> >>  	mutex_unlock(&driver_lock);
> >> @@ -164,6 +212,7 @@ static long vfio_platform_ioctl(void *device_data,
> >>  		if (info.argsz < minsz)
> >>  			return -EINVAL;
> >>  
> >> +		vdev->reset = vfio_platform_get_reset(vdev);
> >>  		if (!IS_ERR_OR_NULL(vdev->reset))
> >>  			vdev->flags |= VFIO_DEVICE_FLAGS_RESET;
> >>  		info.flags = vdev->flags;
> >> @@ -260,6 +309,7 @@ static long vfio_platform_ioctl(void *device_data,
> >>  		return ret;
> >>  
> >>  	} else if (cmd == VFIO_DEVICE_RESET) {
> >> +		vdev->reset = vfio_platform_get_reset(vdev);
> >>  		if (!IS_ERR_OR_NULL(vdev->reset))
> >>  			return vdev->reset(vdev);
> >>  		else
> > 
> > I count 3 gets and 1 put, isn't the module reference count increase
> > showing that?
> 
> vfio_platform_get_reset simply returns if the function pointer already
> is populated so there is no systematic ref increment.

Ah, so it does.

> 
>   This looks like it hasn't been tested.
> 
> It did testing with external and in-kernel modules through
>   Why would we do a
> > get every time we want to do a reset?
> 
> My doubt were about the order of probing between the
> vfio-platform_driver and the vfio reset module? This question was the
> rationale of this implementation choice. But again the actual ref count
> increment is devised to be done once on the first entry point (iotcl or
> internal release)

I think we need to enforce the ordering; the reset function should be
set on device open via request_module() and try_module_get() to make
sure it is present and can't go away (or make it all a static part of
vfio-platform).  A user doesn't expect the reset capability advertised
in the vfio device info struct to change over time.  Thanks,

Alex

Re: [PATCH 1/5] VFIO: platform: add reset_list and register/unregister functions

[Alex Williamson]

On Thu, 2015-05-14 at 10:25 +0200, Eric Auger wrote:
> Hi Alex,
> On 05/13/2015 08:32 PM, Alex Williamson wrote:
> > On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
> >> vfio_platform_common now stores a lists of available reset functions.
> >> Two functions are exposed to register/unregister a reset function. A
> >> reset function is paired with a compat string.
> >>
> >> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> >> ---
> >>  drivers/vfio/platform/vfio_platform_common.c  | 63 +++++++++++++++++++++++++++
> >>  drivers/vfio/platform/vfio_platform_private.h | 13 ++++++
> >>  2 files changed, 76 insertions(+)
> >>
> >> diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c
> >> index abcff7a..edbf24c 100644
> >> --- a/drivers/vfio/platform/vfio_platform_common.c
> >> +++ b/drivers/vfio/platform/vfio_platform_common.c
> >> @@ -23,6 +23,9 @@
> >>  
> >>  #include "vfio_platform_private.h"
> >>  
> >> +struct list_head reset_list;
> >> +LIST_HEAD(reset_list);
> >> +
> > 
> > Redundant?  Static?
> static, yes
> > 
> >>  static DEFINE_MUTEX(driver_lock);
> >>  
> >>  static int vfio_platform_regions_init(struct vfio_platform_device *vdev)
> >> @@ -511,6 +514,13 @@ EXPORT_SYMBOL_GPL(vfio_platform_probe_common);
> >>  struct vfio_platform_device *vfio_platform_remove_common(struct device *dev)
> >>  {
> >>  	struct vfio_platform_device *vdev;
> >> +	struct vfio_platform_reset_node *iter, *tmp;
> >> +
> >> +	list_for_each_entry_safe(iter, tmp, &reset_list, link) {
> >> +		list_del(&iter->link);
> >> +		kfree(iter->compat);
> >> +		kfree(iter);
> >> +	}
> > 
> > 
> > This doesn't make sense.  We allow reset functions to be registered and
> > unregistered, but we forget them all when any device is released?!
> I acknowledge indeed. Can I rely on the reset module exit and associated
> unregister_reset or shall I take this action in the vfio driver itself,
> core?

If we were to load the reset modules via request_module() from
vfio-platform, I think they would stay loaded as long as vfio-platform
is loaded and automatically unload if vfio-platform is unloaded.  Our
reference via try_module_get() for an in-use reset function should
prevent the module from being unloaded while we're dependent on it.  I
think that covers everything we need.  A user is free to rmmod the reset
module, but if it's in use it shouldn't work, and we can request it
again for the next device.  

> >>  
> >>  	vdev = vfio_del_group_dev(dev);
> >>  	if (vdev)
> >> @@ -519,3 +529,56 @@ struct vfio_platform_device *vfio_platform_remove_common(struct device *dev)
> >>  	return vdev;
> >>  }
> >>  EXPORT_SYMBOL_GPL(vfio_platform_remove_common);
> >> +
> >> +int vfio_platform_register_reset(char *compat, struct module *reset_owner,
> >> +				 vfio_platform_reset_fn_t reset)
> >> +{
> >> +	struct vfio_platform_reset_node *node, *iter;
> >> +	bool found = false;
> >> +
> >> +	list_for_each_entry(iter, &reset_list, link) {
> >> +		if (!strcmp(iter->compat, compat)) {
> >> +			found = true;
> > 
> > Just return errno here
> ok
> > 
> >> +			break;
> >> +		}
> >> +	}
> >> +	if (found)
> >> +		return -EINVAL;
> >> +
> >> +	node = kmalloc(sizeof(*node), GFP_KERNEL);
> >> +	if (!node)
> >> +		return -ENOMEM;
> >> +
> >> +	node->compat = kstrdup(compat, GFP_KERNEL);
> >> +	if (!node->compat)
> > 
> > Leaking node
> ok
> > 
> >> +		return -ENOMEM;
> >> +
> >> +	node->owner = reset_owner;
> >> +	node->reset = reset;
> >> +
> >> +	list_add(&node->link, &reset_list);
> > 
> > Isn't this racy?  Don't we need some locks around the list?
> I will add a lock to protect access to the list.
> > 
> >> +	return 0;
> >> +}
> >> +EXPORT_SYMBOL_GPL(vfio_platform_register_reset);
> >> +
> >> +int vfio_platform_unregister_reset(char *compat)
> >> +{
> >> +	struct vfio_platform_reset_node *iter;
> >> +	bool found = false;
> >> +
> >> +	list_for_each_entry(iter, &reset_list, link) {
> >> +		if (!strcmp(iter->compat, compat)) {
> > 
> > Return errno here
> ok
> > 
> >> +			found = true;
> >> +			break;
> >> +		}
> >> +	}
> >> +	if (!found)
> >> +		return -EINVAL;
> >> +
> >> +	list_del(&iter->link);
> > 
> > Racy
> > 
> >> +	kfree(iter->compat);
> >> +	kfree(iter);
> >> +	return 0;
> >> +}
> >> +EXPORT_SYMBOL_GPL(vfio_platform_unregister_reset);
> >> +
> >> diff --git a/drivers/vfio/platform/vfio_platform_private.h b/drivers/vfio/platform/vfio_platform_private.h
> >> index 5d31e04..da2d60b 100644
> >> --- a/drivers/vfio/platform/vfio_platform_private.h
> >> +++ b/drivers/vfio/platform/vfio_platform_private.h
> >> @@ -69,6 +69,15 @@ struct vfio_platform_device {
> >>  	int	(*get_irq)(struct vfio_platform_device *vdev, int i);
> >>  };
> >>  
> >> +typedef int (*vfio_platform_reset_fn_t)(struct vfio_platform_device *vdev);
> > 
> > Seems like this ought to be in a non-private header if we're exporting
> > the [un]register functions.
> I considered the vfio reset modules were internal to the vfio subsystem
> but if you prefer I can expose that in vfio.h. I guess
> register/unregister should become an external API then?

The patch descriptions implied that it was intended for external reset
modules, which I took to mean a potentially broader code base.  It's
fine and probably better if we want to only make it an "internal"
interface, though we really have no ability to enforce that once the
functions are exported.

> >> +
> >> +struct vfio_platform_reset_node {
> >> +	struct list_head link;
> >> +	char *compat;
> >> +	struct module *owner;
> >> +	vfio_platform_reset_fn_t reset;
> >> +};
> >> +
> >>  extern int vfio_platform_probe_common(struct vfio_platform_device *vdev,
> >>  				      struct device *dev);
> >>  extern struct vfio_platform_device *vfio_platform_remove_common
> >> @@ -82,4 +91,8 @@ extern int vfio_platform_set_irqs_ioctl(struct vfio_platform_device *vdev,
> >>  					unsigned start, unsigned count,
> >>  					void *data);
> >>  
> >> +extern int vfio_platform_register_reset(char *compat, struct module *owner,
> >> +					vfio_platform_reset_fn_t reset);
> >> +extern int vfio_platform_unregister_reset(char *compat);
> >> +
> >>  #endif /* VFIO_PLATFORM_PRIVATE_H */
> > 
> > 
> > 
> 

[PATCH 1/5] VFIO: platform: add reset_list and register/unregister functions

[Alex Williamson]

On Thu, 2015-05-14 at 10:25 +0200, Eric Auger wrote:
> Hi Alex,
> On 05/13/2015 08:32 PM, Alex Williamson wrote:
> > On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
> >> vfio_platform_common now stores a lists of available reset functions.
> >> Two functions are exposed to register/unregister a reset function. A
> >> reset function is paired with a compat string.
> >>
> >> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> >> ---
> >>  drivers/vfio/platform/vfio_platform_common.c  | 63 +++++++++++++++++++++++++++
> >>  drivers/vfio/platform/vfio_platform_private.h | 13 ++++++
> >>  2 files changed, 76 insertions(+)
> >>
> >> diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c
> >> index abcff7a..edbf24c 100644
> >> --- a/drivers/vfio/platform/vfio_platform_common.c
> >> +++ b/drivers/vfio/platform/vfio_platform_common.c
> >> @@ -23,6 +23,9 @@
> >>  
> >>  #include "vfio_platform_private.h"
> >>  
> >> +struct list_head reset_list;
> >> +LIST_HEAD(reset_list);
> >> +
> > 
> > Redundant?  Static?
> static, yes
> > 
> >>  static DEFINE_MUTEX(driver_lock);
> >>  
> >>  static int vfio_platform_regions_init(struct vfio_platform_device *vdev)
> >> @@ -511,6 +514,13 @@ EXPORT_SYMBOL_GPL(vfio_platform_probe_common);
> >>  struct vfio_platform_device *vfio_platform_remove_common(struct device *dev)
> >>  {
> >>  	struct vfio_platform_device *vdev;
> >> +	struct vfio_platform_reset_node *iter, *tmp;
> >> +
> >> +	list_for_each_entry_safe(iter, tmp, &reset_list, link) {
> >> +		list_del(&iter->link);
> >> +		kfree(iter->compat);
> >> +		kfree(iter);
> >> +	}
> > 
> > 
> > This doesn't make sense.  We allow reset functions to be registered and
> > unregistered, but we forget them all when any device is released?!
> I acknowledge indeed. Can I rely on the reset module exit and associated
> unregister_reset or shall I take this action in the vfio driver itself,
> core?

If we were to load the reset modules via request_module() from
vfio-platform, I think they would stay loaded as long as vfio-platform
is loaded and automatically unload if vfio-platform is unloaded.  Our
reference via try_module_get() for an in-use reset function should
prevent the module from being unloaded while we're dependent on it.  I
think that covers everything we need.  A user is free to rmmod the reset
module, but if it's in use it shouldn't work, and we can request it
again for the next device.  

> >>  
> >>  	vdev = vfio_del_group_dev(dev);
> >>  	if (vdev)
> >> @@ -519,3 +529,56 @@ struct vfio_platform_device *vfio_platform_remove_common(struct device *dev)
> >>  	return vdev;
> >>  }
> >>  EXPORT_SYMBOL_GPL(vfio_platform_remove_common);
> >> +
> >> +int vfio_platform_register_reset(char *compat, struct module *reset_owner,
> >> +				 vfio_platform_reset_fn_t reset)
> >> +{
> >> +	struct vfio_platform_reset_node *node, *iter;
> >> +	bool found = false;
> >> +
> >> +	list_for_each_entry(iter, &reset_list, link) {
> >> +		if (!strcmp(iter->compat, compat)) {
> >> +			found = true;
> > 
> > Just return errno here
> ok
> > 
> >> +			break;
> >> +		}
> >> +	}
> >> +	if (found)
> >> +		return -EINVAL;
> >> +
> >> +	node = kmalloc(sizeof(*node), GFP_KERNEL);
> >> +	if (!node)
> >> +		return -ENOMEM;
> >> +
> >> +	node->compat = kstrdup(compat, GFP_KERNEL);
> >> +	if (!node->compat)
> > 
> > Leaking node
> ok
> > 
> >> +		return -ENOMEM;
> >> +
> >> +	node->owner = reset_owner;
> >> +	node->reset = reset;
> >> +
> >> +	list_add(&node->link, &reset_list);
> > 
> > Isn't this racy?  Don't we need some locks around the list?
> I will add a lock to protect access to the list.
> > 
> >> +	return 0;
> >> +}
> >> +EXPORT_SYMBOL_GPL(vfio_platform_register_reset);
> >> +
> >> +int vfio_platform_unregister_reset(char *compat)
> >> +{
> >> +	struct vfio_platform_reset_node *iter;
> >> +	bool found = false;
> >> +
> >> +	list_for_each_entry(iter, &reset_list, link) {
> >> +		if (!strcmp(iter->compat, compat)) {
> > 
> > Return errno here
> ok
> > 
> >> +			found = true;
> >> +			break;
> >> +		}
> >> +	}
> >> +	if (!found)
> >> +		return -EINVAL;
> >> +
> >> +	list_del(&iter->link);
> > 
> > Racy
> > 
> >> +	kfree(iter->compat);
> >> +	kfree(iter);
> >> +	return 0;
> >> +}
> >> +EXPORT_SYMBOL_GPL(vfio_platform_unregister_reset);
> >> +
> >> diff --git a/drivers/vfio/platform/vfio_platform_private.h b/drivers/vfio/platform/vfio_platform_private.h
> >> index 5d31e04..da2d60b 100644
> >> --- a/drivers/vfio/platform/vfio_platform_private.h
> >> +++ b/drivers/vfio/platform/vfio_platform_private.h
> >> @@ -69,6 +69,15 @@ struct vfio_platform_device {
> >>  	int	(*get_irq)(struct vfio_platform_device *vdev, int i);
> >>  };
> >>  
> >> +typedef int (*vfio_platform_reset_fn_t)(struct vfio_platform_device *vdev);
> > 
> > Seems like this ought to be in a non-private header if we're exporting
> > the [un]register functions.
> I considered the vfio reset modules were internal to the vfio subsystem
> but if you prefer I can expose that in vfio.h. I guess
> register/unregister should become an external API then?

The patch descriptions implied that it was intended for external reset
modules, which I took to mean a potentially broader code base.  It's
fine and probably better if we want to only make it an "internal"
interface, though we really have no ability to enforce that once the
functions are exported.

> >> +
> >> +struct vfio_platform_reset_node {
> >> +	struct list_head link;
> >> +	char *compat;
> >> +	struct module *owner;
> >> +	vfio_platform_reset_fn_t reset;
> >> +};
> >> +
> >>  extern int vfio_platform_probe_common(struct vfio_platform_device *vdev,
> >>  				      struct device *dev);
> >>  extern struct vfio_platform_device *vfio_platform_remove_common
> >> @@ -82,4 +91,8 @@ extern int vfio_platform_set_irqs_ioctl(struct vfio_platform_device *vdev,
> >>  					unsigned start, unsigned count,
> >>  					void *data);
> >>  
> >> +extern int vfio_platform_register_reset(char *compat, struct module *owner,
> >> +					vfio_platform_reset_fn_t reset);
> >> +extern int vfio_platform_unregister_reset(char *compat);
> >> +
> >>  #endif /* VFIO_PLATFORM_PRIVATE_H */
> > 
> > 
> > 
> 

Re: [PATCH 5/5] VFIO: platform: VFIO platform Calxeda xgmac reset module

[Eric Auger]

On 05/14/2015 05:14 PM, Alex Williamson wrote:
> On Thu, 2015-05-14 at 11:06 +0200, Eric Auger wrote:
>> Alex,
>> On 05/13/2015 08:33 PM, Alex Williamson wrote:
>>> On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
>>>> This patch introduces a module that registers and implements a basic
>>>> reset function for the Calxeda xgmac device. This latter basically disables
>>>> interrupts and stops DMA transfers.
>>>>
>>>> The reset function code is inherited from the native calxeda xgmac driver.
>>>>
>>>> Signed-off-by: Eric Auger <eric.auger@linaro.org>
>>>> ---
>>>>  drivers/vfio/platform/Kconfig                      |   2 +
>>>>  drivers/vfio/platform/Makefile                     |   2 +
>>>>  drivers/vfio/platform/reset/Kconfig                |   7 ++
>>>>  drivers/vfio/platform/reset/Makefile               |   5 +
>>>>  .../platform/reset/vfio_platform_calxedaxgmac.c    | 121 +++++++++++++++++++++
>>>>  5 files changed, 137 insertions(+)
>>>>  create mode 100644 drivers/vfio/platform/reset/Kconfig
>>>>  create mode 100644 drivers/vfio/platform/reset/Makefile
>>>>  create mode 100644 drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
>>>>
>>>> diff --git a/drivers/vfio/platform/Kconfig b/drivers/vfio/platform/Kconfig
>>>> index 9a4403e..1df7477 100644
>>>> --- a/drivers/vfio/platform/Kconfig
>>>> +++ b/drivers/vfio/platform/Kconfig
>>>> @@ -18,3 +18,5 @@ config VFIO_AMBA
>>>>  	  framework.
>>>>  
>>>>  	  If you don't know what to do here, say N.
>>>> +
>>>> +source "drivers/vfio/platform/reset/Kconfig"
>>>> diff --git a/drivers/vfio/platform/Makefile b/drivers/vfio/platform/Makefile
>>>> index 81de144..9ce8afe 100644
>>>> --- a/drivers/vfio/platform/Makefile
>>>> +++ b/drivers/vfio/platform/Makefile
>>>> @@ -2,7 +2,9 @@
>>>>  vfio-platform-y := vfio_platform.o vfio_platform_common.o vfio_platform_irq.o
>>>>  
>>>>  obj-$(CONFIG_VFIO_PLATFORM) += vfio-platform.o
>>>> +obj-$(CONFIG_VFIO_PLATFORM) += reset/
>>>>  
>>>>  vfio-amba-y := vfio_amba.o
>>>>  
>>>>  obj-$(CONFIG_VFIO_AMBA) += vfio-amba.o
>>>> +obj-$(CONFIG_VFIO_AMBA) += reset/
>>>> diff --git a/drivers/vfio/platform/reset/Kconfig b/drivers/vfio/platform/reset/Kconfig
>>>> new file mode 100644
>>>> index 0000000..746b96b
>>>> --- /dev/null
>>>> +++ b/drivers/vfio/platform/reset/Kconfig
>>>> @@ -0,0 +1,7 @@
>>>> +config VFIO_PLATFORM_CALXEDAXGMAC_RESET
>>>> +	tristate "VFIO support for calxeda xgmac reset"
>>>> +	depends on VFIO_PLATFORM
>>>> +	help
>>>> +	  Enables the VFIO platform driver to handle reset for Calxeda xgmac
>>>> +
>>>> +	  If you don't know what to do here, say N.
>>>> diff --git a/drivers/vfio/platform/reset/Makefile b/drivers/vfio/platform/reset/Makefile
>>>> new file mode 100644
>>>> index 0000000..2a486af
>>>> --- /dev/null
>>>> +++ b/drivers/vfio/platform/reset/Makefile
>>>> @@ -0,0 +1,5 @@
>>>> +vfio-platform-calxedaxgmac-y := vfio_platform_calxedaxgmac.o
>>>> +
>>>> +ccflags-y += -Idrivers/vfio/platform
>>>> +
>>>> +obj-$(CONFIG_VFIO_PLATFORM_CALXEDAXGMAC_RESET) += vfio-platform-calxedaxgmac.o
>>>> diff --git a/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c b/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
>>>> new file mode 100644
>>>> index 0000000..3c6e428
>>>> --- /dev/null
>>>> +++ b/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
>>>> @@ -0,0 +1,121 @@
>>>> +/*
>>>> + * VFIO platform driver specialized for Calxeda xgmac reset
>>>> + * reset code is inherited from calxeda xgmac native driver
>>>> + *
>>>> + * Copyright 2010-2011 Calxeda, Inc.
>>>> + * Copyright (c) 2015 Linaro Ltd.
>>>> + *              www.linaro.org
>>>> + *
>>>> + * This program is free software; you can redistribute it and/or modify it
>>>> + * under the terms and conditions of the GNU General Public License,
>>>> + * version 2, as published by the Free Software Foundation.
>>>> + *
>>>> + * This program is distributed in the hope it will be useful, but WITHOUT
>>>> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
>>>> + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
>>>> + * more details.
>>>> + *
>>>> + * You should have received a copy of the GNU General Public License along with
>>>> + * this program.  If not, see <http://www.gnu.org/licenses/>.
>>>> + */
>>>> +
>>>> +#include <linux/module.h>
>>>> +#include <linux/kernel.h>
>>>> +#include <linux/init.h>
>>>> +#include <linux/io.h>
>>>> +
>>>> +#include "vfio_platform_private.h"
>>>> +
>>>> +#define DRIVER_VERSION  "0.1"
>>>> +#define DRIVER_AUTHOR   "Eric Auger <eric.auger@linaro.org>"
>>>> +#define DRIVER_DESC     "Reset support for Calxeda xgmac vfio platform device"
>>>> +
>>>> +#define CALXEDAXGMAC_COMPAT "calxeda,hb-xgmac"
>>>> +
>>>> +/* XGMAC Register definitions */
>>>> +#define XGMAC_CONTROL           0x00000000      /* MAC Configuration */
>>>> +
>>>> +/* DMA Control and Status Registers */
>>>> +#define XGMAC_DMA_CONTROL       0x00000f18      /* Ctrl (Operational Mode) */
>>>> +#define XGMAC_DMA_INTR_ENA      0x00000f1c      /* Interrupt Enable */
>>>> +
>>>> +/* DMA Control registe defines */
>>>> +#define DMA_CONTROL_ST          0x00002000      /* Start/Stop Transmission */
>>>> +#define DMA_CONTROL_SR          0x00000002      /* Start/Stop Receive */
>>>> +
>>>> +/* Common MAC defines */
>>>> +#define MAC_ENABLE_TX           0x00000008      /* Transmitter Enable */
>>>> +#define MAC_ENABLE_RX           0x00000004      /* Receiver Enable */
>>>> +
>>>> +static inline void xgmac_mac_disable(void __iomem *ioaddr)
>>>> +{
>>>> +	u32 value = readl(ioaddr + XGMAC_DMA_CONTROL);
>>>> +
>>>> +	value &= ~(DMA_CONTROL_ST | DMA_CONTROL_SR);
>>>> +	writel(value, ioaddr + XGMAC_DMA_CONTROL);
>>>> +
>>>> +	value = readl(ioaddr + XGMAC_CONTROL);
>>>> +	value &= ~(MAC_ENABLE_TX | MAC_ENABLE_RX);
>>>> +	writel(value, ioaddr + XGMAC_CONTROL);
>>>> +}
>>>> +
>>>> +int vfio_platform_calxedaxgmac_reset(struct vfio_platform_device *vdev)
>>>> +{
>>>> +	struct vfio_platform_region reg = vdev->regions[0];
>>>> +
>>>> +	if (!reg.ioaddr) {
>>>> +		reg.ioaddr =
>>>> +			ioremap_nocache(reg.addr, reg.size);
>>>> +		if (!reg.ioaddr)
>>>> +			return -ENOMEM;
>>>> +	}
>>>> +
>>>> +	/* disable IRQ */
>>>> +	writel(0, reg.ioaddr + XGMAC_DMA_INTR_ENA);
>>>> +
>>>> +	/* Disable the MAC core */
>>>> +	xgmac_mac_disable(reg.ioaddr);
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +EXPORT_SYMBOL_GPL(vfio_platform_calxedaxgmac_reset);
>>>> +
>>>> +static int __init vfio_platform_calxedaxgmac_init(void)
>>>> +{
>>>> +	int (*register_reset)(char *, struct module *,
>>>> +				vfio_platform_reset_fn_t);
>>>> +	int ret;
>>>> +
>>>> +	register_reset = symbol_get(vfio_platform_register_reset);
>>>> +	if (!register_reset)
>>>> +		return -EINVAL;
>>>> +
>>>> +	ret = register_reset(CALXEDAXGMAC_COMPAT, THIS_MODULE,
>>>> +			     vfio_platform_calxedaxgmac_reset);
>>>> +
>>>> +	symbol_put(vfio_platform_register_reset);
>>>> +
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +static void __exit vfio_platform_calxedaxgmac_exit(void)
>>>> +{
>>>> +	int (*unregister_reset)(char *);
>>>> +	int ret;
>>>> +
>>>> +	unregister_reset = symbol_get(vfio_platform_unregister_reset);
>>>
>>> Shouldn't we have gotten the symbol during the module init so that this
>>> cannot fail?
>> yes makes sense
>>>
>>>> +	if (!unregister_reset)
>>>> +		return;
>>>> +
>>>> +	ret = unregister_reset(CALXEDAXGMAC_COMPAT);
>>>
>>> unregister_reset() should just return void
>> ok
>>>
>>>> +
>>>> +	symbol_put(vfio_platform_unregister_reset);
>>>> +}
>>>> +
>>>> +module_init(vfio_platform_calxedaxgmac_init);
>>>> +module_exit(vfio_platform_calxedaxgmac_exit);
>>>> +
>>>> +MODULE_VERSION(DRIVER_VERSION);
>>>> +MODULE_LICENSE("GPL v2");
>>>> +MODULE_AUTHOR(DRIVER_AUTHOR);
>>>> +MODULE_DESCRIPTION(DRIVER_DESC);
>>>
>>>
>>> So a user needs to manually load this module to get a working reset for
>>> this device?  That's never going to happen.
>> The reset module is devised to be in-kernel or external. There is always
>> the choice to include all reset modules as soon as vfio-platform /amba
>> is chosen. What would be your preferred approach then? Do you consider
>> this approach using separate reset modules is not sensible? Do you think
>> we should put everything in the vfio-platform driver?
> 
> I respect your attempt to keep the code slim and modular by making the
> reset functionality loadable, but we can't expect users to figure out
> what module they need to load to make their device work.  It needs to be
> automatic.  One way we could achieve that and still keep your premise of
> loadable reset modules would be a lookup table in vfio-platform to
> translate a device compatibility ID to a reset module name.  We could
> then do a request_module() to auto-load the reset module if one exists.
> Maybe the ID table could even live in the reset driver, much like PCI
> IDs live in those drivers.  We also have the complication that there's
> no direct module dependency for the reset module(s), so they would need
> to be explicitly listed to install them into an initramfs.
> 
> If we take that approach, we're quickly building infrastructure that
> gets more bloated than a few reset functions buried directly into
> vfio-platform.  Then we may wonder how many reset functions are we
> reasonably going to get.  If you don't have more than a couple in mind,
> maybe we should just embed them into vfio-platform and worry about
> creating something more dynamic when the need presents itself.  Thanks,

Hi Alex,

I will explore the approach you described above, based on
request_module(). Logically the number of reset functions should grow
rapidly as vfio-platform drivers get used.

Thanks for exchanging on this!

Best Regards

Eric
> 
> Alex
> 

Re: [PATCH 5/5] VFIO: platform: VFIO platform Calxeda xgmac reset module

[Eric Auger]

On 05/14/2015 05:14 PM, Alex Williamson wrote:
> On Thu, 2015-05-14 at 11:06 +0200, Eric Auger wrote:
>> Alex,
>> On 05/13/2015 08:33 PM, Alex Williamson wrote:
>>> On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
>>>> This patch introduces a module that registers and implements a basic
>>>> reset function for the Calxeda xgmac device. This latter basically disables
>>>> interrupts and stops DMA transfers.
>>>>
>>>> The reset function code is inherited from the native calxeda xgmac driver.
>>>>
>>>> Signed-off-by: Eric Auger <eric.auger@linaro.org>
>>>> ---
>>>>  drivers/vfio/platform/Kconfig                      |   2 +
>>>>  drivers/vfio/platform/Makefile                     |   2 +
>>>>  drivers/vfio/platform/reset/Kconfig                |   7 ++
>>>>  drivers/vfio/platform/reset/Makefile               |   5 +
>>>>  .../platform/reset/vfio_platform_calxedaxgmac.c    | 121 +++++++++++++++++++++
>>>>  5 files changed, 137 insertions(+)
>>>>  create mode 100644 drivers/vfio/platform/reset/Kconfig
>>>>  create mode 100644 drivers/vfio/platform/reset/Makefile
>>>>  create mode 100644 drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
>>>>
>>>> diff --git a/drivers/vfio/platform/Kconfig b/drivers/vfio/platform/Kconfig
>>>> index 9a4403e..1df7477 100644
>>>> --- a/drivers/vfio/platform/Kconfig
>>>> +++ b/drivers/vfio/platform/Kconfig
>>>> @@ -18,3 +18,5 @@ config VFIO_AMBA
>>>>  	  framework.
>>>>  
>>>>  	  If you don't know what to do here, say N.
>>>> +
>>>> +source "drivers/vfio/platform/reset/Kconfig"
>>>> diff --git a/drivers/vfio/platform/Makefile b/drivers/vfio/platform/Makefile
>>>> index 81de144..9ce8afe 100644
>>>> --- a/drivers/vfio/platform/Makefile
>>>> +++ b/drivers/vfio/platform/Makefile
>>>> @@ -2,7 +2,9 @@
>>>>  vfio-platform-y := vfio_platform.o vfio_platform_common.o vfio_platform_irq.o
>>>>  
>>>>  obj-$(CONFIG_VFIO_PLATFORM) += vfio-platform.o
>>>> +obj-$(CONFIG_VFIO_PLATFORM) += reset/
>>>>  
>>>>  vfio-amba-y := vfio_amba.o
>>>>  
>>>>  obj-$(CONFIG_VFIO_AMBA) += vfio-amba.o
>>>> +obj-$(CONFIG_VFIO_AMBA) += reset/
>>>> diff --git a/drivers/vfio/platform/reset/Kconfig b/drivers/vfio/platform/reset/Kconfig
>>>> new file mode 100644
>>>> index 0000000..746b96b
>>>> --- /dev/null
>>>> +++ b/drivers/vfio/platform/reset/Kconfig
>>>> @@ -0,0 +1,7 @@
>>>> +config VFIO_PLATFORM_CALXEDAXGMAC_RESET
>>>> +	tristate "VFIO support for calxeda xgmac reset"
>>>> +	depends on VFIO_PLATFORM
>>>> +	help
>>>> +	  Enables the VFIO platform driver to handle reset for Calxeda xgmac
>>>> +
>>>> +	  If you don't know what to do here, say N.
>>>> diff --git a/drivers/vfio/platform/reset/Makefile b/drivers/vfio/platform/reset/Makefile
>>>> new file mode 100644
>>>> index 0000000..2a486af
>>>> --- /dev/null
>>>> +++ b/drivers/vfio/platform/reset/Makefile
>>>> @@ -0,0 +1,5 @@
>>>> +vfio-platform-calxedaxgmac-y := vfio_platform_calxedaxgmac.o
>>>> +
>>>> +ccflags-y += -Idrivers/vfio/platform
>>>> +
>>>> +obj-$(CONFIG_VFIO_PLATFORM_CALXEDAXGMAC_RESET) += vfio-platform-calxedaxgmac.o
>>>> diff --git a/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c b/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
>>>> new file mode 100644
>>>> index 0000000..3c6e428
>>>> --- /dev/null
>>>> +++ b/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
>>>> @@ -0,0 +1,121 @@
>>>> +/*
>>>> + * VFIO platform driver specialized for Calxeda xgmac reset
>>>> + * reset code is inherited from calxeda xgmac native driver
>>>> + *
>>>> + * Copyright 2010-2011 Calxeda, Inc.
>>>> + * Copyright (c) 2015 Linaro Ltd.
>>>> + *              www.linaro.org
>>>> + *
>>>> + * This program is free software; you can redistribute it and/or modify it
>>>> + * under the terms and conditions of the GNU General Public License,
>>>> + * version 2, as published by the Free Software Foundation.
>>>> + *
>>>> + * This program is distributed in the hope it will be useful, but WITHOUT
>>>> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
>>>> + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
>>>> + * more details.
>>>> + *
>>>> + * You should have received a copy of the GNU General Public License along with
>>>> + * this program.  If not, see <http://www.gnu.org/licenses/>.
>>>> + */
>>>> +
>>>> +#include <linux/module.h>
>>>> +#include <linux/kernel.h>
>>>> +#include <linux/init.h>
>>>> +#include <linux/io.h>
>>>> +
>>>> +#include "vfio_platform_private.h"
>>>> +
>>>> +#define DRIVER_VERSION  "0.1"
>>>> +#define DRIVER_AUTHOR   "Eric Auger <eric.auger@linaro.org>"
>>>> +#define DRIVER_DESC     "Reset support for Calxeda xgmac vfio platform device"
>>>> +
>>>> +#define CALXEDAXGMAC_COMPAT "calxeda,hb-xgmac"
>>>> +
>>>> +/* XGMAC Register definitions */
>>>> +#define XGMAC_CONTROL           0x00000000      /* MAC Configuration */
>>>> +
>>>> +/* DMA Control and Status Registers */
>>>> +#define XGMAC_DMA_CONTROL       0x00000f18      /* Ctrl (Operational Mode) */
>>>> +#define XGMAC_DMA_INTR_ENA      0x00000f1c      /* Interrupt Enable */
>>>> +
>>>> +/* DMA Control registe defines */
>>>> +#define DMA_CONTROL_ST          0x00002000      /* Start/Stop Transmission */
>>>> +#define DMA_CONTROL_SR          0x00000002      /* Start/Stop Receive */
>>>> +
>>>> +/* Common MAC defines */
>>>> +#define MAC_ENABLE_TX           0x00000008      /* Transmitter Enable */
>>>> +#define MAC_ENABLE_RX           0x00000004      /* Receiver Enable */
>>>> +
>>>> +static inline void xgmac_mac_disable(void __iomem *ioaddr)
>>>> +{
>>>> +	u32 value = readl(ioaddr + XGMAC_DMA_CONTROL);
>>>> +
>>>> +	value &= ~(DMA_CONTROL_ST | DMA_CONTROL_SR);
>>>> +	writel(value, ioaddr + XGMAC_DMA_CONTROL);
>>>> +
>>>> +	value = readl(ioaddr + XGMAC_CONTROL);
>>>> +	value &= ~(MAC_ENABLE_TX | MAC_ENABLE_RX);
>>>> +	writel(value, ioaddr + XGMAC_CONTROL);
>>>> +}
>>>> +
>>>> +int vfio_platform_calxedaxgmac_reset(struct vfio_platform_device *vdev)
>>>> +{
>>>> +	struct vfio_platform_region reg = vdev->regions[0];
>>>> +
>>>> +	if (!reg.ioaddr) {
>>>> +		reg.ioaddr =
>>>> +			ioremap_nocache(reg.addr, reg.size);
>>>> +		if (!reg.ioaddr)
>>>> +			return -ENOMEM;
>>>> +	}
>>>> +
>>>> +	/* disable IRQ */
>>>> +	writel(0, reg.ioaddr + XGMAC_DMA_INTR_ENA);
>>>> +
>>>> +	/* Disable the MAC core */
>>>> +	xgmac_mac_disable(reg.ioaddr);
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +EXPORT_SYMBOL_GPL(vfio_platform_calxedaxgmac_reset);
>>>> +
>>>> +static int __init vfio_platform_calxedaxgmac_init(void)
>>>> +{
>>>> +	int (*register_reset)(char *, struct module *,
>>>> +				vfio_platform_reset_fn_t);
>>>> +	int ret;
>>>> +
>>>> +	register_reset = symbol_get(vfio_platform_register_reset);
>>>> +	if (!register_reset)
>>>> +		return -EINVAL;
>>>> +
>>>> +	ret = register_reset(CALXEDAXGMAC_COMPAT, THIS_MODULE,
>>>> +			     vfio_platform_calxedaxgmac_reset);
>>>> +
>>>> +	symbol_put(vfio_platform_register_reset);
>>>> +
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +static void __exit vfio_platform_calxedaxgmac_exit(void)
>>>> +{
>>>> +	int (*unregister_reset)(char *);
>>>> +	int ret;
>>>> +
>>>> +	unregister_reset = symbol_get(vfio_platform_unregister_reset);
>>>
>>> Shouldn't we have gotten the symbol during the module init so that this
>>> cannot fail?
>> yes makes sense
>>>
>>>> +	if (!unregister_reset)
>>>> +		return;
>>>> +
>>>> +	ret = unregister_reset(CALXEDAXGMAC_COMPAT);
>>>
>>> unregister_reset() should just return void
>> ok
>>>
>>>> +
>>>> +	symbol_put(vfio_platform_unregister_reset);
>>>> +}
>>>> +
>>>> +module_init(vfio_platform_calxedaxgmac_init);
>>>> +module_exit(vfio_platform_calxedaxgmac_exit);
>>>> +
>>>> +MODULE_VERSION(DRIVER_VERSION);
>>>> +MODULE_LICENSE("GPL v2");
>>>> +MODULE_AUTHOR(DRIVER_AUTHOR);
>>>> +MODULE_DESCRIPTION(DRIVER_DESC);
>>>
>>>
>>> So a user needs to manually load this module to get a working reset for
>>> this device?  That's never going to happen.
>> The reset module is devised to be in-kernel or external. There is always
>> the choice to include all reset modules as soon as vfio-platform /amba
>> is chosen. What would be your preferred approach then? Do you consider
>> this approach using separate reset modules is not sensible? Do you think
>> we should put everything in the vfio-platform driver?
> 
> I respect your attempt to keep the code slim and modular by making the
> reset functionality loadable, but we can't expect users to figure out
> what module they need to load to make their device work.  It needs to be
> automatic.  One way we could achieve that and still keep your premise of
> loadable reset modules would be a lookup table in vfio-platform to
> translate a device compatibility ID to a reset module name.  We could
> then do a request_module() to auto-load the reset module if one exists.
> Maybe the ID table could even live in the reset driver, much like PCI
> IDs live in those drivers.  We also have the complication that there's
> no direct module dependency for the reset module(s), so they would need
> to be explicitly listed to install them into an initramfs.
> 
> If we take that approach, we're quickly building infrastructure that
> gets more bloated than a few reset functions buried directly into
> vfio-platform.  Then we may wonder how many reset functions are we
> reasonably going to get.  If you don't have more than a couple in mind,
> maybe we should just embed them into vfio-platform and worry about
> creating something more dynamic when the need presents itself.  Thanks,

Hi Alex,

I will explore the approach you described above, based on
request_module(). Logically the number of reset functions should grow
rapidly as vfio-platform drivers get used.

Thanks for exchanging on this!

Best Regards

Eric
> 
> Alex
> 

[PATCH 5/5] VFIO: platform: VFIO platform Calxeda xgmac reset module

[Eric Auger]

On 05/14/2015 05:14 PM, Alex Williamson wrote:
> On Thu, 2015-05-14 at 11:06 +0200, Eric Auger wrote:
>> Alex,
>> On 05/13/2015 08:33 PM, Alex Williamson wrote:
>>> On Thu, 2015-05-07 at 16:27 +0200, Eric Auger wrote:
>>>> This patch introduces a module that registers and implements a basic
>>>> reset function for the Calxeda xgmac device. This latter basically disables
>>>> interrupts and stops DMA transfers.
>>>>
>>>> The reset function code is inherited from the native calxeda xgmac driver.
>>>>
>>>> Signed-off-by: Eric Auger <eric.auger@linaro.org>
>>>> ---
>>>>  drivers/vfio/platform/Kconfig                      |   2 +
>>>>  drivers/vfio/platform/Makefile                     |   2 +
>>>>  drivers/vfio/platform/reset/Kconfig                |   7 ++
>>>>  drivers/vfio/platform/reset/Makefile               |   5 +
>>>>  .../platform/reset/vfio_platform_calxedaxgmac.c    | 121 +++++++++++++++++++++
>>>>  5 files changed, 137 insertions(+)
>>>>  create mode 100644 drivers/vfio/platform/reset/Kconfig
>>>>  create mode 100644 drivers/vfio/platform/reset/Makefile
>>>>  create mode 100644 drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
>>>>
>>>> diff --git a/drivers/vfio/platform/Kconfig b/drivers/vfio/platform/Kconfig
>>>> index 9a4403e..1df7477 100644
>>>> --- a/drivers/vfio/platform/Kconfig
>>>> +++ b/drivers/vfio/platform/Kconfig
>>>> @@ -18,3 +18,5 @@ config VFIO_AMBA
>>>>  	  framework.
>>>>  
>>>>  	  If you don't know what to do here, say N.
>>>> +
>>>> +source "drivers/vfio/platform/reset/Kconfig"
>>>> diff --git a/drivers/vfio/platform/Makefile b/drivers/vfio/platform/Makefile
>>>> index 81de144..9ce8afe 100644
>>>> --- a/drivers/vfio/platform/Makefile
>>>> +++ b/drivers/vfio/platform/Makefile
>>>> @@ -2,7 +2,9 @@
>>>>  vfio-platform-y := vfio_platform.o vfio_platform_common.o vfio_platform_irq.o
>>>>  
>>>>  obj-$(CONFIG_VFIO_PLATFORM) += vfio-platform.o
>>>> +obj-$(CONFIG_VFIO_PLATFORM) += reset/
>>>>  
>>>>  vfio-amba-y := vfio_amba.o
>>>>  
>>>>  obj-$(CONFIG_VFIO_AMBA) += vfio-amba.o
>>>> +obj-$(CONFIG_VFIO_AMBA) += reset/
>>>> diff --git a/drivers/vfio/platform/reset/Kconfig b/drivers/vfio/platform/reset/Kconfig
>>>> new file mode 100644
>>>> index 0000000..746b96b
>>>> --- /dev/null
>>>> +++ b/drivers/vfio/platform/reset/Kconfig
>>>> @@ -0,0 +1,7 @@
>>>> +config VFIO_PLATFORM_CALXEDAXGMAC_RESET
>>>> +	tristate "VFIO support for calxeda xgmac reset"
>>>> +	depends on VFIO_PLATFORM
>>>> +	help
>>>> +	  Enables the VFIO platform driver to handle reset for Calxeda xgmac
>>>> +
>>>> +	  If you don't know what to do here, say N.
>>>> diff --git a/drivers/vfio/platform/reset/Makefile b/drivers/vfio/platform/reset/Makefile
>>>> new file mode 100644
>>>> index 0000000..2a486af
>>>> --- /dev/null
>>>> +++ b/drivers/vfio/platform/reset/Makefile
>>>> @@ -0,0 +1,5 @@
>>>> +vfio-platform-calxedaxgmac-y := vfio_platform_calxedaxgmac.o
>>>> +
>>>> +ccflags-y += -Idrivers/vfio/platform
>>>> +
>>>> +obj-$(CONFIG_VFIO_PLATFORM_CALXEDAXGMAC_RESET) += vfio-platform-calxedaxgmac.o
>>>> diff --git a/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c b/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
>>>> new file mode 100644
>>>> index 0000000..3c6e428
>>>> --- /dev/null
>>>> +++ b/drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c
>>>> @@ -0,0 +1,121 @@
>>>> +/*
>>>> + * VFIO platform driver specialized for Calxeda xgmac reset
>>>> + * reset code is inherited from calxeda xgmac native driver
>>>> + *
>>>> + * Copyright 2010-2011 Calxeda, Inc.
>>>> + * Copyright (c) 2015 Linaro Ltd.
>>>> + *              www.linaro.org
>>>> + *
>>>> + * This program is free software; you can redistribute it and/or modify it
>>>> + * under the terms and conditions of the GNU General Public License,
>>>> + * version 2, as published by the Free Software Foundation.
>>>> + *
>>>> + * This program is distributed in the hope it will be useful, but WITHOUT
>>>> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
>>>> + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
>>>> + * more details.
>>>> + *
>>>> + * You should have received a copy of the GNU General Public License along with
>>>> + * this program.  If not, see <http://www.gnu.org/licenses/>.
>>>> + */
>>>> +
>>>> +#include <linux/module.h>
>>>> +#include <linux/kernel.h>
>>>> +#include <linux/init.h>
>>>> +#include <linux/io.h>
>>>> +
>>>> +#include "vfio_platform_private.h"
>>>> +
>>>> +#define DRIVER_VERSION  "0.1"
>>>> +#define DRIVER_AUTHOR   "Eric Auger <eric.auger@linaro.org>"
>>>> +#define DRIVER_DESC     "Reset support for Calxeda xgmac vfio platform device"
>>>> +
>>>> +#define CALXEDAXGMAC_COMPAT "calxeda,hb-xgmac"
>>>> +
>>>> +/* XGMAC Register definitions */
>>>> +#define XGMAC_CONTROL           0x00000000      /* MAC Configuration */
>>>> +
>>>> +/* DMA Control and Status Registers */
>>>> +#define XGMAC_DMA_CONTROL       0x00000f18      /* Ctrl (Operational Mode) */
>>>> +#define XGMAC_DMA_INTR_ENA      0x00000f1c      /* Interrupt Enable */
>>>> +
>>>> +/* DMA Control registe defines */
>>>> +#define DMA_CONTROL_ST          0x00002000      /* Start/Stop Transmission */
>>>> +#define DMA_CONTROL_SR          0x00000002      /* Start/Stop Receive */
>>>> +
>>>> +/* Common MAC defines */
>>>> +#define MAC_ENABLE_TX           0x00000008      /* Transmitter Enable */
>>>> +#define MAC_ENABLE_RX           0x00000004      /* Receiver Enable */
>>>> +
>>>> +static inline void xgmac_mac_disable(void __iomem *ioaddr)
>>>> +{
>>>> +	u32 value = readl(ioaddr + XGMAC_DMA_CONTROL);
>>>> +
>>>> +	value &= ~(DMA_CONTROL_ST | DMA_CONTROL_SR);
>>>> +	writel(value, ioaddr + XGMAC_DMA_CONTROL);
>>>> +
>>>> +	value = readl(ioaddr + XGMAC_CONTROL);
>>>> +	value &= ~(MAC_ENABLE_TX | MAC_ENABLE_RX);
>>>> +	writel(value, ioaddr + XGMAC_CONTROL);
>>>> +}
>>>> +
>>>> +int vfio_platform_calxedaxgmac_reset(struct vfio_platform_device *vdev)
>>>> +{
>>>> +	struct vfio_platform_region reg = vdev->regions[0];
>>>> +
>>>> +	if (!reg.ioaddr) {
>>>> +		reg.ioaddr =
>>>> +			ioremap_nocache(reg.addr, reg.size);
>>>> +		if (!reg.ioaddr)
>>>> +			return -ENOMEM;
>>>> +	}
>>>> +
>>>> +	/* disable IRQ */
>>>> +	writel(0, reg.ioaddr + XGMAC_DMA_INTR_ENA);
>>>> +
>>>> +	/* Disable the MAC core */
>>>> +	xgmac_mac_disable(reg.ioaddr);
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +EXPORT_SYMBOL_GPL(vfio_platform_calxedaxgmac_reset);
>>>> +
>>>> +static int __init vfio_platform_calxedaxgmac_init(void)
>>>> +{
>>>> +	int (*register_reset)(char *, struct module *,
>>>> +				vfio_platform_reset_fn_t);
>>>> +	int ret;
>>>> +
>>>> +	register_reset = symbol_get(vfio_platform_register_reset);
>>>> +	if (!register_reset)
>>>> +		return -EINVAL;
>>>> +
>>>> +	ret = register_reset(CALXEDAXGMAC_COMPAT, THIS_MODULE,
>>>> +			     vfio_platform_calxedaxgmac_reset);
>>>> +
>>>> +	symbol_put(vfio_platform_register_reset);
>>>> +
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +static void __exit vfio_platform_calxedaxgmac_exit(void)
>>>> +{
>>>> +	int (*unregister_reset)(char *);
>>>> +	int ret;
>>>> +
>>>> +	unregister_reset = symbol_get(vfio_platform_unregister_reset);
>>>
>>> Shouldn't we have gotten the symbol during the module init so that this
>>> cannot fail?
>> yes makes sense
>>>
>>>> +	if (!unregister_reset)
>>>> +		return;
>>>> +
>>>> +	ret = unregister_reset(CALXEDAXGMAC_COMPAT);
>>>
>>> unregister_reset() should just return void
>> ok
>>>
>>>> +
>>>> +	symbol_put(vfio_platform_unregister_reset);
>>>> +}
>>>> +
>>>> +module_init(vfio_platform_calxedaxgmac_init);
>>>> +module_exit(vfio_platform_calxedaxgmac_exit);
>>>> +
>>>> +MODULE_VERSION(DRIVER_VERSION);
>>>> +MODULE_LICENSE("GPL v2");
>>>> +MODULE_AUTHOR(DRIVER_AUTHOR);
>>>> +MODULE_DESCRIPTION(DRIVER_DESC);
>>>
>>>
>>> So a user needs to manually load this module to get a working reset for
>>> this device?  That's never going to happen.
>> The reset module is devised to be in-kernel or external. There is always
>> the choice to include all reset modules as soon as vfio-platform /amba
>> is chosen. What would be your preferred approach then? Do you consider
>> this approach using separate reset modules is not sensible? Do you think
>> we should put everything in the vfio-platform driver?
> 
> I respect your attempt to keep the code slim and modular by making the
> reset functionality loadable, but we can't expect users to figure out
> what module they need to load to make their device work.  It needs to be
> automatic.  One way we could achieve that and still keep your premise of
> loadable reset modules would be a lookup table in vfio-platform to
> translate a device compatibility ID to a reset module name.  We could
> then do a request_module() to auto-load the reset module if one exists.
> Maybe the ID table could even live in the reset driver, much like PCI
> IDs live in those drivers.  We also have the complication that there's
> no direct module dependency for the reset module(s), so they would need
> to be explicitly listed to install them into an initramfs.
> 
> If we take that approach, we're quickly building infrastructure that
> gets more bloated than a few reset functions buried directly into
> vfio-platform.  Then we may wonder how many reset functions are we
> reasonably going to get.  If you don't have more than a couple in mind,
> maybe we should just embed them into vfio-platform and worry about
> creating something more dynamic when the need presents itself.  Thanks,

Hi Alex,

I will explore the approach you described above, based on
request_module(). Logically the number of reset functions should grow
rapidly as vfio-platform drivers get used.

Thanks for exchanging on this!

Best Regards

Eric
> 
> Alex
>