[PATCH 0/5] Implement FTW driver

[PATCH 0/5] Implement FTW driver

[Sukadev Bhattiprolu]

The Virtual Accelerator Switchboard (VAS) subsystem in the POWER9 processor
provides a low latency Core-to-core wakeup" mechanism which allows a thread
on one core the processor to efficiently send a message to a thread waiting
on another core.

This Fast thread-wakeup (FTW) driver provides user space applications an
interface to the Core-to-core wakeup mechanism. The FTW driver uses the
"external" interfaces provided by the VAS driver to interact with the VAS
hardware.

PATCH 5/5 documents the API.

The ftw-next branch on my github has some initial test cases for the
driver:

	https://github.com/sukadev/linux/tree/ftw-next

Thanks to input from Ben Herrenschmidt, Michael Ellerman, Michael
Neuling and Robert Blackmore.

Sukadev Bhattiprolu (5):
  powerpc/vas: Remove a stray line in Makefile
  powerpc/ftw: Define FTW_SETUP ioctl API
  powerpc/ftw: Implement a simple FTW driver
  powerpc/ftw: Add a couple of trace points
  powerpc/ftw: Document FTW API/usage

 Documentation/powerpc/ftw-api.txt       | 283 +++++++++++++++++++++++++
 MAINTAINERS                             |   8 +
 arch/powerpc/platforms/powernv/Makefile |   1 -
 drivers/misc/Kconfig                    |   1 +
 drivers/misc/Makefile                   |   1 +
 drivers/misc/ftw/Kconfig                |  16 ++
 drivers/misc/ftw/Makefile               |   4 +
 drivers/misc/ftw/ftw-trace.h            |  75 +++++++
 drivers/misc/ftw/ftw.c                  | 352 ++++++++++++++++++++++++++++++++
 include/uapi/misc/ftw.h                 |  31 +++
 10 files changed, 771 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/powerpc/ftw-api.txt
 create mode 100644 drivers/misc/ftw/Kconfig
 create mode 100644 drivers/misc/ftw/Makefile
 create mode 100644 drivers/misc/ftw/ftw-trace.h
 create mode 100644 drivers/misc/ftw/ftw.c
 create mode 100644 include/uapi/misc/ftw.h

-- 
2.7.4

[PATCH 1/5] powerpc/vas: Remove a stray line in Makefile

[Sukadev Bhattiprolu]

Remove a bogus line from arch/powerpc/platforms/powernv/Makefile that
was added by commit ece4e51 ("powerpc/vas: Export HVWC to debugfs").

Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
---
 arch/powerpc/platforms/powernv/Makefile | 1 -
 1 file changed, 1 deletion(-)

diff --git a/arch/powerpc/platforms/powernv/Makefile b/arch/powerpc/platforms/powernv/Makefile
index 3732118..ca94488 100644
--- a/arch/powerpc/platforms/powernv/Makefile
+++ b/arch/powerpc/platforms/powernv/Makefile
@@ -16,4 +16,3 @@ obj-$(CONFIG_OPAL_PRD)	+= opal-prd.o
 obj-$(CONFIG_PERF_EVENTS) += opal-imc.o
 obj-$(CONFIG_PPC_MEMTRACE)	+= memtrace.o
 obj-$(CONFIG_PPC_VAS)	+= vas.o vas-window.o vas-debug.o
-obj-$(CONFIG_PPC_FTW)	+= nx-ftw.o
-- 
2.7.4

Re: [1/5] powerpc/vas: Remove a stray line in Makefile

[Michael Ellerman]

On Wed, 2018-01-17 at 02:50:39 UTC, Sukadev Bhattiprolu wrote:
> Remove a bogus line from arch/powerpc/platforms/powernv/Makefile that
> was added by commit ece4e51 ("powerpc/vas: Export HVWC to debugfs").
> 
> Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>

Applied to powerpc next, thanks.

https://git.kernel.org/powerpc/c/2f65272a2a304ec6aa32ad9a45b150

cheers

[PATCH 2/5] powerpc/ftw: Define FTW_SETUP ioctl API

[Sukadev Bhattiprolu]

Define the FTW_SETUP ioctl interface for fast thread wakeup (FTW). A
follow-on patch will implement the FTW driver and ioctl.

Thanks to input from Ben Herrenschmidt, Michael Neuling, Michael Ellerman.

Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
---
Changelog[v2]
	- [Michael Neuling] Use a single VAS_FTW_SETUP ioctl and simplify
	  the interface.
---
 include/uapi/misc/ftw.h | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)
 create mode 100644 include/uapi/misc/ftw.h

diff --git a/include/uapi/misc/ftw.h b/include/uapi/misc/ftw.h
new file mode 100644
index 0000000..f233f51
--- /dev/null
+++ b/include/uapi/misc/ftw.h
@@ -0,0 +1,31 @@
+/*
+ * Copyright 2018 IBM Corp.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+
+#ifndef _UAPI_MISC_FTW_H
+#define _UAPI_MISC_FTW_H
+
+#include <linux/types.h>
+#include <linux/ioctl.h>
+
+#define FTW_FLAGS_PIN_WINDOW	0x1
+
+#define FTW_SETUP		_IOW('v', 1, struct ftw_setup_attr)
+
+struct ftw_setup_attr {
+	__s16	version;
+	__s16	vas_id;		/* specific instance of vas or -1 for default */
+	__u32	reserved;
+
+	__u64	reserved1;
+
+	__u64	flags;
+	__u64	reserved2;
+};
+
+#endif /* _UAPI_MISC_FTW_H */
-- 
2.7.4

Re: [PATCH 2/5] powerpc/ftw: Define FTW_SETUP ioctl API

[Randy Dunlap]

On 01/16/18 18:50, Sukadev Bhattiprolu wrote:
> Define the FTW_SETUP ioctl interface for fast thread wakeup (FTW). A
> follow-on patch will implement the FTW driver and ioctl.
> 
> Thanks to input from Ben Herrenschmidt, Michael Neuling, Michael Ellerman.
> 
> Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
> ---
> Changelog[v2]
> 	- [Michael Neuling] Use a single VAS_FTW_SETUP ioctl and simplify
> 	  the interface.
> ---
>  include/uapi/misc/ftw.h | 31 +++++++++++++++++++++++++++++++
>  1 file changed, 31 insertions(+)
>  create mode 100644 include/uapi/misc/ftw.h
> 
> diff --git a/include/uapi/misc/ftw.h b/include/uapi/misc/ftw.h
> new file mode 100644
> index 0000000..f233f51
> --- /dev/null
> +++ b/include/uapi/misc/ftw.h
> @@ -0,0 +1,31 @@
> +/*
> + * Copyright 2018 IBM Corp.
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU General Public License
> + * as published by the Free Software Foundation; either version
> + * 2 of the License, or (at your option) any later version.
> + */
> +
> +#ifndef _UAPI_MISC_FTW_H
> +#define _UAPI_MISC_FTW_H
> +
> +#include <linux/types.h>
> +#include <linux/ioctl.h>
> +
> +#define FTW_FLAGS_PIN_WINDOW	0x1
> +
> +#define FTW_SETUP		_IOW('v', 1, struct ftw_setup_attr)

ioctls should be documented in Documentation/ioctl/ioctl-number.txt.
Please update that file.

> +
> +struct ftw_setup_attr {
> +	__s16	version;
> +	__s16	vas_id;		/* specific instance of vas or -1 for default */
> +	__u32	reserved;
> +
> +	__u64	reserved1;
> +
> +	__u64	flags;
> +	__u64	reserved2;
> +};
> +
> +#endif /* _UAPI_MISC_FTW_H */
> 


-- 
~Randy

Re: [PATCH 2/5] powerpc/ftw: Define FTW_SETUP ioctl API

[Sukadev Bhattiprolu]

Randy Dunlap [rdunlap@infradead.org] wrote:

> > +#define FTW_FLAGS_PIN_WINDOW	0x1
> > +
> > +#define FTW_SETUP		_IOW('v', 1, struct ftw_setup_attr)
> 
> ioctls should be documented in Documentation/ioctl/ioctl-number.txt.
> Please update that file.

Ok. Here is the updated patch.

Thanks for the review.

Sukadev
---
>From 1f347c199a0b1bbc528705c8e9ddd11c825a80fc Mon Sep 17 00:00:00 2001
From: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
Date: Thu, 2 Feb 2017 06:20:07 -0500
Subject: [PATCH 2/5] powerpc/ftw: Define FTW_SETUP ioctl API

Define the FTW_SETUP ioctl interface for fast thread wakeup (FTW). A
follow-on patch will implement the FTW driver and ioctl.

Thanks to input from Ben Herrenschmidt, Michael Neuling, Michael Ellerman.

Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
---
Changelog[v2]
	- [Michael Neuling] Use a single VAS_FTW_SETUP ioctl and simplify
	  the interface.
	- [Randy Dunlap] Reserve/document the ioctl number used.
---
 Documentation/ioctl/ioctl-number.txt |  1 +
 include/uapi/misc/ftw.h              | 35 +++++++++++++++++++++++++++++++++++
 2 files changed, 36 insertions(+)
 create mode 100644 include/uapi/misc/ftw.h

diff --git a/Documentation/ioctl/ioctl-number.txt b/Documentation/ioctl/ioctl-number.txt
index 3e3fdae..b0f323c 100644
--- a/Documentation/ioctl/ioctl-number.txt
+++ b/Documentation/ioctl/ioctl-number.txt
@@ -277,6 +277,7 @@ Code  Seq#(hex)	Include File		Comments
 'v'	00-1F	linux/fs.h		conflict!
 'v'	00-0F	linux/sonypi.h		conflict!
 'v'	C0-FF	linux/meye.h		conflict!
+'v'	20-27	include/uapi/misc/ftw.h
 'w'	all				CERN SCI driver
 'y'	00-1F				packet based user level communications
 					<mailto:zapman@interlan.net>
diff --git a/include/uapi/misc/ftw.h b/include/uapi/misc/ftw.h
new file mode 100644
index 0000000..99676b2
--- /dev/null
+++ b/include/uapi/misc/ftw.h
@@ -0,0 +1,35 @@
+/*
+ * Copyright 2018 IBM Corp.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+
+#ifndef _UAPI_MISC_FTW_H
+#define _UAPI_MISC_FTW_H
+
+#include <linux/types.h>
+#include <linux/ioctl.h>
+
+#define FTW_FLAGS_PIN_WINDOW	0x1
+
+/*
+ * Note: The range 0x20-27 for letter 'v' are reserved for FTW ioctls in
+ *	 Documentation/ioctl/ioctl-number.txt.
+ */
+#define FTW_SETUP		_IOW('v', 0x20, struct ftw_setup_attr)
+
+struct ftw_setup_attr {
+	__s16	version;
+	__s16	vas_id;		/* specific instance of vas or -1 for default */
+	__u32	reserved;
+
+	__u64	reserved1;
+
+	__u64	flags;
+	__u64	reserved2;
+};
+
+#endif /* _UAPI_MISC_FTW_H */
-- 
2.7.4

[PATCH 3/5] powerpc/ftw: Implement a simple FTW driver

[Sukadev Bhattiprolu]

The Fast Thread Wake-up (FTW) driver provides user space applications an
interface to the low latency Core-to-Core wakeup functionality in POWER9.

This mechanism allows a thread on one core to efficiently send a message
to a "waiting thread" on another core on the same chip, using the Virtual
Accelrator Switchboard (VAS) subsystem.

This initial FTW driver implements the ioctl and mmap operations on an
FTW device node. Using these operations, a pair of application threads
can establish a "communication channel" and use the COPY, PASTE and WAIT
instructions to wait/wake up.

PATCH 5/5 documents the API and includes an example of the usage.

Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
---
Changelog[v2]
	- [Michael Neuling] Rename from drop "nx" from name "nx-ftw".
	- [Michael Neuling] Use a single VAS_FTW_SETUP ioctl to simplify
	  interface.
	- [Michael Ellerman] To work with paste emulation patch, mark
	  PTE dirty in ->mmap() to ensure there is no fault on paste
	  (the emulation patch must disable pagefaults when updating
	  thread reconfig registers).
	- Check return value from set_thread_tidr().
	- Move driver drivers/misc/ftw.

---
 drivers/misc/Kconfig      |   1 +
 drivers/misc/Makefile     |   1 +
 drivers/misc/ftw/Kconfig  |  16 +++
 drivers/misc/ftw/Makefile |   4 +
 drivers/misc/ftw/ftw.c    | 346 ++++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 368 insertions(+)
 create mode 100644 drivers/misc/ftw/Kconfig
 create mode 100644 drivers/misc/ftw/Makefile
 create mode 100644 drivers/misc/ftw/ftw.c

diff --git a/drivers/misc/Kconfig b/drivers/misc/Kconfig
index f1a5c23..a9b161f 100644
--- a/drivers/misc/Kconfig
+++ b/drivers/misc/Kconfig
@@ -508,4 +508,5 @@ source "drivers/misc/mic/Kconfig"
 source "drivers/misc/genwqe/Kconfig"
 source "drivers/misc/echo/Kconfig"
 source "drivers/misc/cxl/Kconfig"
+source "drivers/misc/ftw/Kconfig"
 endmenu
diff --git a/drivers/misc/Makefile b/drivers/misc/Makefile
index 5ca5f64..338668c 100644
--- a/drivers/misc/Makefile
+++ b/drivers/misc/Makefile
@@ -52,6 +52,7 @@ obj-$(CONFIG_GENWQE)		+= genwqe/
 obj-$(CONFIG_ECHO)		+= echo/
 obj-$(CONFIG_VEXPRESS_SYSCFG)	+= vexpress-syscfg.o
 obj-$(CONFIG_CXL_BASE)		+= cxl/
+obj-$(CONFIG_PPC_FTW)		+= ftw/
 obj-$(CONFIG_ASPEED_LPC_CTRL)	+= aspeed-lpc-ctrl.o
 obj-$(CONFIG_ASPEED_LPC_SNOOP)	+= aspeed-lpc-snoop.o
 obj-$(CONFIG_PCI_ENDPOINT_TEST)	+= pci_endpoint_test.o
diff --git a/drivers/misc/ftw/Kconfig b/drivers/misc/ftw/Kconfig
new file mode 100644
index 0000000..5454d40
--- /dev/null
+++ b/drivers/misc/ftw/Kconfig
@@ -0,0 +1,16 @@
+
+config PPC_FTW
+	tristate "IBM Fast Thread-Wakeup (FTW)"
+	depends on PPC_VAS
+	default n
+	help
+          This enables support for IBM Fast Thread-Wakeup driver.
+
+          The FTW driver allows applications to utilize a low overhead
+          core-to-core wake up mechansim in the IBM Virtual Accelerator
+          Switchboard (VAS) to improve performance.
+
+          VAS adapters are found in POWER9 based systems and are required
+          for the FTW driver to be operational.
+
+          If unsure, say N.
diff --git a/drivers/misc/ftw/Makefile b/drivers/misc/ftw/Makefile
new file mode 100644
index 0000000..2cfe566
--- /dev/null
+++ b/drivers/misc/ftw/Makefile
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: GPL-2.0
+ccflags-y			:= $(call cc-disable-warning, unused-const-variable)
+ccflags-$(CONFIG_PPC_WERROR)	+= -Werror
+obj-$(CONFIG_PPC_FTW)		+= ftw.o
diff --git a/drivers/misc/ftw/ftw.c b/drivers/misc/ftw/ftw.c
new file mode 100644
index 0000000..6fcb4e2
--- /dev/null
+++ b/drivers/misc/ftw/ftw.c
@@ -0,0 +1,346 @@
+/*
+ * Copyright 2018 IBM Corp.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+#define pr_fmt(fmt) "ftw: " fmt
+
+#include <linux/module.h>
+#include <linux/kernel.h>
+#include <linux/device.h>
+#include <linux/cdev.h>
+#include <linux/fs.h>
+#include <linux/mm.h>
+#include <linux/slab.h>
+#include <linux/uaccess.h>
+#include <linux/pfn_t.h>
+#include <asm/switch_to.h>
+#include <asm/vas.h>
+#include <uapi/misc/ftw.h>
+
+/*
+ * FTW is a device driver used to provide user space access to the
+ * Core-to-Core aka Fast Thread Wakeup (FTW) functionality provided by
+ * the Virtual Accelerator Subsystem (VAS) in POWER9 systems. See also
+ * arch/powerpc/platforms/powernv/vas*.
+ *
+ * The driver creates the device /dev/ftw that can be used as follows:
+ *
+ *	fd = open("/dev/ftw", O_RDWR);
+ *	rc = ioctl(fd, FTW_SETUP, &attr);
+ *	paste_addr = mmap(NULL, PAGE_SIZE, prot, MAP_SHARED, fd, 0ULL).
+ *	vas_copy(&crb, 0, 1);
+ *	vas_paste(paste_addr, 0, 1);
+ *
+ * where "vas_copy" and "vas_paste" are defined in copy-paste.h.
+ */
+
+static char		*ftw_dev_name = "ftw";
+static atomic_t		ftw_instid = ATOMIC_INIT(0);
+
+/*
+ * Wrapper object for the ftw device - there is just one instance of
+ * this node in the system.
+ */
+struct ftw_dev {
+	struct cdev cdev;
+	struct device *device;
+	char *name;
+	dev_t devt;
+	struct class *class;
+} ftw_device;
+
+/*
+ * One instance per open of a ftw device. Each ftw_instance is
+ * associated with a VAS window after the caller issues FTW_SETUP
+ * ioctl.
+ */
+struct ftw_instance {
+	int id;
+	struct vas_window *rxwin;
+	struct vas_window *txwin;
+};
+
+static char *ftw_devnode(struct device *dev, umode_t *mode)
+{
+	return kasprintf(GFP_KERNEL, "%s", dev_name(dev));
+}
+
+static int ftw_open(struct inode *inode, struct file *fp)
+{
+	struct ftw_instance *instance;
+
+	instance = kzalloc(sizeof(*instance), GFP_KERNEL);
+	if (!instance)
+		return -ENOMEM;
+
+	instance->id = atomic_inc_return(&ftw_instid);
+
+	fp->private_data = instance;
+
+	return 0;
+}
+
+static int validate_ftw_setup_attr(struct ftw_setup_attr *uattr)
+{
+	if (uattr->version != 1 || uattr->reserved || uattr->reserved1 ||
+				   uattr->reserved2)
+		return -EINVAL;
+
+	if (uattr->flags & ~FTW_FLAGS_PIN_WINDOW)
+		return -EINVAL;
+
+	if (uattr->flags & FTW_FLAGS_PIN_WINDOW && !capable(CAP_SYS_ADMIN))
+		return -EPERM;
+
+	return 0;
+}
+
+static int ftw_ioc_ftw_setup(struct file *fp, unsigned long arg)
+{
+	int rc, vasid, cop;
+	struct vas_rx_win_attr rxattr;
+	struct vas_tx_win_attr txattr;
+	struct ftw_setup_attr uattr;
+	void __user *uptr = (void *)arg;
+	struct vas_window *rxwin, *txwin;
+	struct ftw_instance *instance = fp->private_data;
+
+	rc = copy_from_user(&uattr, uptr, sizeof(uattr));
+	if (rc) {
+		pr_debug("copy_from_user() returns %d\n", rc);
+		return -EFAULT;
+	}
+
+	rc = validate_ftw_setup_attr(&uattr);
+	if (rc)
+		return rc;
+
+	cop = VAS_COP_TYPE_FTW;
+	rc = set_thread_tidr(current);
+	if (rc)
+		return rc;
+
+	vasid = uattr.vas_id;
+
+	vas_init_rx_win_attr(&rxattr, cop);
+	rxattr.lnotify_lpid = mfspr(SPRN_LPID);
+
+	/*
+	 * Only caller can own the window for now. Not sure if there is need
+	 * for process P1 to make P2 the owner of a window. If so, we need to
+	 * find P2, make sure we have permissions, get a reference etc.
+	 */
+	rxattr.lnotify_pid = mfspr(SPRN_PID);
+	rxattr.lnotify_tid = mfspr(SPRN_TIDR);
+
+	rxwin = vas_rx_win_open(vasid, cop, &rxattr);
+	if (IS_ERR(rxwin)) {
+		pr_debug("vas_rx_win_open() failed, %ld\n", PTR_ERR(rxwin));
+		return PTR_ERR(rxwin);
+	}
+
+	vas_init_tx_win_attr(&txattr, cop);
+
+	txattr.lpid = mfspr(SPRN_LPID);
+	txattr.pidr = mfspr(SPRN_PID);
+	txattr.pid = task_pid_nr(current);
+	txattr.pswid = vas_win_id(rxwin);
+
+	txwin = vas_tx_win_open(vasid, cop, &txattr);
+	if (IS_ERR(txwin)) {
+		pr_debug("vas_tx_win_open() failed, %ld\n", PTR_ERR(txwin));
+		rc = PTR_ERR(txwin);
+		goto close_rxwin;
+	}
+
+	instance->rxwin = rxwin;
+	instance->txwin = txwin;
+
+	return 0;
+
+close_rxwin:
+	vas_win_close(rxwin);
+	return rc;
+}
+
+static int ftw_release(struct inode *inode, struct file *fp)
+{
+	struct ftw_instance *instance;
+
+	instance = fp->private_data;
+
+	if (instance->txwin)
+		vas_win_close(instance->txwin);
+	if (instance->rxwin)
+		vas_win_close(instance->rxwin);
+	/*
+	 * TODO We don't know here if user has other receive windows
+	 *      open, and can't really call clear_thread_tidr(). So,
+	 *      once the process calls set_thread_tidr(), the TIDR value
+	 *      sticks around until process exits, potentially resulting
+	 *      in an unnecessary copy in restore_sprs() when even the
+	 *      process has closed its last window.
+	 */
+
+	instance->rxwin = instance->txwin = NULL;
+
+	kfree(instance);
+	fp->private_data = NULL;
+	atomic_dec(&ftw_instid);
+
+	return 0;
+}
+
+static int ftw_mmap(struct file *fp, struct vm_area_struct *vma)
+{
+	int rc;
+	pgprot_t prot;
+	u64 paste_addr;
+	unsigned long pfn;
+	struct ftw_instance *instance = fp->private_data;
+
+	if ((vma->vm_end - vma->vm_start) > PAGE_SIZE) {
+		pr_debug("size 0x%zx, PAGE_SIZE 0x%zx\n",
+				(vma->vm_end - vma->vm_start), PAGE_SIZE);
+		return -EINVAL;
+	}
+
+	/* Ensure instance has an open send window */
+	if (!instance->txwin) {
+		pr_debug("No send window open?\n");
+		return -EINVAL;
+	}
+
+	paste_addr = vas_win_paste_addr(instance->txwin);
+	pfn = paste_addr >> PAGE_SHIFT;
+
+	/* flags, page_prot from cxl_mmap(), except we want cachable */
+	vma->vm_flags |= VM_IO | VM_PFNMAP;
+	vma->vm_page_prot = pgprot_cached(vma->vm_page_prot);
+
+	/*
+	 * We must disable page faults when emulating the paste
+	 * instruction. To ensure that the page associated with
+	 * the paste address is in memory, mark it dirty.
+	 */
+	prot = __pgprot(pgprot_val(vma->vm_page_prot) | _PAGE_DIRTY);
+
+	rc = remap_pfn_range(vma, vma->vm_start, pfn + vma->vm_pgoff,
+			vma->vm_end - vma->vm_start, prot);
+
+	pr_devel("paste addr %llx at %lx, rc %d\n", paste_addr, vma->vm_start,
+			rc);
+
+	set_thread_uses_vas();
+
+	return rc;
+}
+
+static long ftw_ioctl(struct file *fp, unsigned int cmd, unsigned long arg)
+{
+	switch (cmd) {
+
+	case FTW_SETUP:
+		return ftw_ioc_ftw_setup(fp, arg);
+
+	default:
+		return -EINVAL;
+	}
+}
+
+const struct file_operations ftw_fops = {
+	.owner = THIS_MODULE,
+	.open = ftw_open,
+	.release = ftw_release,
+	.mmap = ftw_mmap,
+	.unlocked_ioctl = ftw_ioctl,
+};
+
+
+int ftw_file_init(void)
+{
+	int rc;
+	dev_t devno;
+
+	rc = alloc_chrdev_region(&ftw_device.devt, 1, 1, "ftw");
+	if (rc) {
+		pr_debug("Unable to allocate ftw major number: %i\n", rc);
+		return rc;
+	}
+
+	pr_devel("device allocated, dev [%i,%i]\n",
+			MAJOR(ftw_device.devt), MINOR(ftw_device.devt));
+
+	ftw_device.class = class_create(THIS_MODULE, "ftw");
+	if (IS_ERR(ftw_device.class)) {
+		pr_debug("Unable to create FTW class\n");
+		rc = PTR_ERR(ftw_device.class);
+		goto err;
+	}
+	ftw_device.class->devnode = ftw_devnode;
+
+	cdev_init(&ftw_device.cdev, &ftw_fops);
+
+	devno = MKDEV(MAJOR(ftw_device.devt), 0);
+	if (cdev_add(&ftw_device.cdev, devno, 1)) {
+		pr_debug("cdev_add() failed\n");
+		goto err;
+	}
+
+	ftw_device.device = device_create(ftw_device.class, NULL,
+			devno, NULL, ftw_dev_name, MINOR(devno));
+	if (IS_ERR(ftw_device.device)) {
+		pr_debug("Unable to create ftw-%d\n", MINOR(devno));
+		goto err;
+	}
+
+	pr_devel("Added dev [%d,%d]\n", MAJOR(devno), MINOR(devno));
+
+	return 0;
+
+err:
+	unregister_chrdev_region(ftw_device.devt, 1);
+	return rc;
+}
+
+void ftw_file_exit(void)
+{
+	dev_t devno;
+
+	cdev_del(&ftw_device.cdev);
+	devno = MKDEV(MAJOR(ftw_device.devt), MINOR(ftw_device.devt));
+	device_destroy(ftw_device.class, devno);
+
+	class_destroy(ftw_device.class);
+	unregister_chrdev_region(ftw_device.devt, 1);
+}
+
+int __init ftw_init(void)
+{
+	int rc;
+
+	rc = ftw_file_init();
+	if (rc)
+		return rc;
+
+	pr_info("Device initialized\n");
+
+	return 0;
+}
+
+void __init ftw_exit(void)
+{
+	pr_devel("Device exiting\n");
+	ftw_file_exit();
+}
+
+module_init(ftw_init);
+module_exit(ftw_exit);
+
+MODULE_DESCRIPTION("IBM NX Fast Thread Wakeup Device");
+MODULE_AUTHOR("Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>");
+MODULE_LICENSE("GPL");
-- 
2.7.4

Re: [PATCH 3/5] powerpc/ftw: Implement a simple FTW driver

[Randy Dunlap]

On 01/16/18 18:50, Sukadev Bhattiprolu wrote:
> The Fast Thread Wake-up (FTW) driver provides user space applications an
> interface to the low latency Core-to-Core wakeup functionality in POWER9.
> 
> This mechanism allows a thread on one core to efficiently send a message
> to a "waiting thread" on another core on the same chip, using the Virtual
> Accelrator Switchboard (VAS) subsystem.
> 
> This initial FTW driver implements the ioctl and mmap operations on an
> FTW device node. Using these operations, a pair of application threads
> can establish a "communication channel" and use the COPY, PASTE and WAIT
> instructions to wait/wake up.
> 
> PATCH 5/5 documents the API and includes an example of the usage.
> 
> Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
> ---
> Changelog[v2]
> 	- [Michael Neuling] Rename from drop "nx" from name "nx-ftw".
> 	- [Michael Neuling] Use a single VAS_FTW_SETUP ioctl to simplify
> 	  interface.
> 	- [Michael Ellerman] To work with paste emulation patch, mark
> 	  PTE dirty in ->mmap() to ensure there is no fault on paste
> 	  (the emulation patch must disable pagefaults when updating
> 	  thread reconfig registers).
> 	- Check return value from set_thread_tidr().
> 	- Move driver drivers/misc/ftw.
> 
> ---
>  drivers/misc/Kconfig      |   1 +
>  drivers/misc/Makefile     |   1 +
>  drivers/misc/ftw/Kconfig  |  16 +++
>  drivers/misc/ftw/Makefile |   4 +
>  drivers/misc/ftw/ftw.c    | 346 ++++++++++++++++++++++++++++++++++++++++++++++
>  5 files changed, 368 insertions(+)
>  create mode 100644 drivers/misc/ftw/Kconfig
>  create mode 100644 drivers/misc/ftw/Makefile
>  create mode 100644 drivers/misc/ftw/ftw.c

> +static long ftw_ioctl(struct file *fp, unsigned int cmd, unsigned long arg)
> +{
> +	switch (cmd) {
> +
> +	case FTW_SETUP:
> +		return ftw_ioc_ftw_setup(fp, arg);
> +
> +	default:
> +		return -EINVAL;
> +	}
> +}

Nit:  some versions of gcc (or maybe clang) complain about a typed function
not always having a return value in code like above, so it is often done as:

> +static long ftw_ioctl(struct file *fp, unsigned int cmd, unsigned long arg)
> +{
> +	switch (cmd) {
> +
> +	case FTW_SETUP:
> +		return ftw_ioc_ftw_setup(fp, arg);
> +
> +	default:
> +		break;
> +	}

	return -EINVAL;
> +}

Do you expect to implement more ioctls?  If not, just change the switch to
an if ().

-- 
~Randy

Re: [PATCH 3/5] powerpc/ftw: Implement a simple FTW driver

[Sukadev Bhattiprolu]

Randy Dunlap [rdunlap@infradead.org] wrote:
> > +
> > +	default:
> > +		return -EINVAL;
> > +	}
> > +}
> 
> Nit:  some versions of gcc (or maybe clang) complain about a typed function
> not always having a return value in code like above, so it is often done as:

Ok.
> 
> > +static long ftw_ioctl(struct file *fp, unsigned int cmd, unsigned long arg)
> > +{
> > +	switch (cmd) {
> > +
> > +	case FTW_SETUP:
> > +		return ftw_ioc_ftw_setup(fp, arg);
> > +
> > +	default:
> > +		break;
> > +	}
> 
> 	return -EINVAL;
> > +}
> 
> Do you expect to implement more ioctls?  If not, just change the switch to
> an if ().
Maybe a couple more but changed it to an 'if' for now (and fixed an
error handling issue in ftw_file_init()).

Here is the updated patch.

---
>From 344ffbcc2cd1e64dd87249d508cf6000e6e41a0c Mon Sep 17 00:00:00 2001
From: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
Date: Fri, 4 Aug 2017 16:45:34 -0500
Subject: [PATCH 3/5] powerpc/ftw: Implement a simple FTW driver

The Fast Thread Wake-up (FTW) driver provides user space applications an
interface to the low latency Core-to-Core wakeup functionality in POWER9.

This mechanism allows a thread on one core to efficiently send a message
to a "waiting thread" on another core on the same chip, using the Virtual
Accelrator Switchboard (VAS) subsystem.

This initial FTW driver implements the ioctl and mmap operations on an
FTW device node. Using these operations, a pair of application threads
can establish a "communication channel" and use the COPY, PASTE and WAIT
instructions to wait/wake up.

PATCH 5/5 documents the API and includes an example of the usage.

Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
---
Changelog[v2]
	- [Michael Neuling] Rename from drop "nx" from name "nx-ftw".
	- [Michael Neuling] Use a single VAS_FTW_SETUP ioctl to simplify
	  interface.
	- [Michael Ellerman] To work with paste emulation patch, mark
	  PTE dirty in ->mmap() to ensure there is no fault on paste
	  (the emulation patch must disable pagefaults when updating
	  thread reconfig registers).
	- [Randy Dunlap] Minor cleanup in ftw_ioctl().
	- Fix cleanup code in ftw_file_init()
	- Check return value from set_thread_tidr().
	- Move driver drivers/misc/ftw.
---
 drivers/misc/Kconfig      |   1 +
 drivers/misc/Makefile     |   1 +
 drivers/misc/ftw/Kconfig  |  16 +++
 drivers/misc/ftw/Makefile |   4 +
 drivers/misc/ftw/ftw.c    | 346 ++++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 368 insertions(+)
 create mode 100644 drivers/misc/ftw/Kconfig
 create mode 100644 drivers/misc/ftw/Makefile
 create mode 100644 drivers/misc/ftw/ftw.c

diff --git a/drivers/misc/Kconfig b/drivers/misc/Kconfig
index f1a5c23..a9b161f 100644
--- a/drivers/misc/Kconfig
+++ b/drivers/misc/Kconfig
@@ -508,4 +508,5 @@ source "drivers/misc/mic/Kconfig"
 source "drivers/misc/genwqe/Kconfig"
 source "drivers/misc/echo/Kconfig"
 source "drivers/misc/cxl/Kconfig"
+source "drivers/misc/ftw/Kconfig"
 endmenu
diff --git a/drivers/misc/Makefile b/drivers/misc/Makefile
index 5ca5f64..338668c 100644
--- a/drivers/misc/Makefile
+++ b/drivers/misc/Makefile
@@ -52,6 +52,7 @@ obj-$(CONFIG_GENWQE)		+= genwqe/
 obj-$(CONFIG_ECHO)		+= echo/
 obj-$(CONFIG_VEXPRESS_SYSCFG)	+= vexpress-syscfg.o
 obj-$(CONFIG_CXL_BASE)		+= cxl/
+obj-$(CONFIG_PPC_FTW)		+= ftw/
 obj-$(CONFIG_ASPEED_LPC_CTRL)	+= aspeed-lpc-ctrl.o
 obj-$(CONFIG_ASPEED_LPC_SNOOP)	+= aspeed-lpc-snoop.o
 obj-$(CONFIG_PCI_ENDPOINT_TEST)	+= pci_endpoint_test.o
diff --git a/drivers/misc/ftw/Kconfig b/drivers/misc/ftw/Kconfig
new file mode 100644
index 0000000..5454d40
--- /dev/null
+++ b/drivers/misc/ftw/Kconfig
@@ -0,0 +1,16 @@
+
+config PPC_FTW
+	tristate "IBM Fast Thread-Wakeup (FTW)"
+	depends on PPC_VAS
+	default n
+	help
+          This enables support for IBM Fast Thread-Wakeup driver.
+
+          The FTW driver allows applications to utilize a low overhead
+          core-to-core wake up mechansim in the IBM Virtual Accelerator
+          Switchboard (VAS) to improve performance.
+
+          VAS adapters are found in POWER9 based systems and are required
+          for the FTW driver to be operational.
+
+          If unsure, say N.
diff --git a/drivers/misc/ftw/Makefile b/drivers/misc/ftw/Makefile
new file mode 100644
index 0000000..2cfe566
--- /dev/null
+++ b/drivers/misc/ftw/Makefile
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: GPL-2.0
+ccflags-y			:= $(call cc-disable-warning, unused-const-variable)
+ccflags-$(CONFIG_PPC_WERROR)	+= -Werror
+obj-$(CONFIG_PPC_FTW)		+= ftw.o
diff --git a/drivers/misc/ftw/ftw.c b/drivers/misc/ftw/ftw.c
new file mode 100644
index 0000000..ea02a19
--- /dev/null
+++ b/drivers/misc/ftw/ftw.c
@@ -0,0 +1,346 @@
+/*
+ * Copyright 2018 IBM Corp.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+#define pr_fmt(fmt) "ftw: " fmt
+
+#include <linux/module.h>
+#include <linux/kernel.h>
+#include <linux/device.h>
+#include <linux/cdev.h>
+#include <linux/fs.h>
+#include <linux/mm.h>
+#include <linux/slab.h>
+#include <linux/uaccess.h>
+#include <linux/pfn_t.h>
+#include <asm/switch_to.h>
+#include <asm/vas.h>
+#include <uapi/misc/ftw.h>
+
+/*
+ * FTW is a device driver used to provide user space access to the
+ * Core-to-Core aka Fast Thread Wakeup (FTW) functionality provided by
+ * the Virtual Accelerator Subsystem (VAS) in POWER9 systems. See also
+ * arch/powerpc/platforms/powernv/vas*.
+ *
+ * The driver creates the device /dev/ftw that can be used as follows:
+ *
+ *	fd = open("/dev/ftw", O_RDWR);
+ *	rc = ioctl(fd, FTW_SETUP, &attr);
+ *	paste_addr = mmap(NULL, PAGE_SIZE, prot, MAP_SHARED, fd, 0ULL).
+ *	vas_copy(&crb, 0, 1);
+ *	vas_paste(paste_addr, 0, 1);
+ *
+ * where "vas_copy" and "vas_paste" are defined in copy-paste.h.
+ */
+
+static char		*ftw_dev_name = "ftw";
+static atomic_t		ftw_instid = ATOMIC_INIT(0);
+
+/*
+ * Wrapper object for the ftw device - there is just one instance of
+ * this node in the system.
+ */
+struct ftw_dev {
+	struct cdev cdev;
+	struct device *device;
+	char *name;
+	dev_t devt;
+	struct class *class;
+} ftw_device;
+
+/*
+ * One instance per open of a ftw device. Each ftw_instance is
+ * associated with a VAS window after the caller issues FTW_SETUP
+ * ioctl.
+ */
+struct ftw_instance {
+	int id;
+	struct vas_window *rxwin;
+	struct vas_window *txwin;
+};
+
+static char *ftw_devnode(struct device *dev, umode_t *mode)
+{
+	return kasprintf(GFP_KERNEL, "%s", dev_name(dev));
+}
+
+static int ftw_open(struct inode *inode, struct file *fp)
+{
+	struct ftw_instance *instance;
+
+	instance = kzalloc(sizeof(*instance), GFP_KERNEL);
+	if (!instance)
+		return -ENOMEM;
+
+	instance->id = atomic_inc_return(&ftw_instid);
+
+	fp->private_data = instance;
+
+	return 0;
+}
+
+static int validate_ftw_setup_attr(struct ftw_setup_attr *uattr)
+{
+	if (uattr->version != 1 || uattr->reserved || uattr->reserved1 ||
+				   uattr->reserved2)
+		return -EINVAL;
+
+	if (uattr->flags & ~FTW_FLAGS_PIN_WINDOW)
+		return -EINVAL;
+
+	if (uattr->flags & FTW_FLAGS_PIN_WINDOW && !capable(CAP_SYS_ADMIN))
+		return -EPERM;
+
+	return 0;
+}
+
+static int ftw_ioc_ftw_setup(struct file *fp, unsigned long arg)
+{
+	int rc, vasid, cop;
+	struct vas_rx_win_attr rxattr;
+	struct vas_tx_win_attr txattr;
+	struct ftw_setup_attr uattr;
+	void __user *uptr = (void *)arg;
+	struct vas_window *rxwin, *txwin;
+	struct ftw_instance *instance = fp->private_data;
+
+	rc = copy_from_user(&uattr, uptr, sizeof(uattr));
+	if (rc) {
+		pr_debug("copy_from_user() returns %d\n", rc);
+		return -EFAULT;
+	}
+
+	rc = validate_ftw_setup_attr(&uattr);
+	if (rc)
+		return rc;
+
+	cop = VAS_COP_TYPE_FTW;
+	rc = set_thread_tidr(current);
+	if (rc)
+		return rc;
+
+	vasid = uattr.vas_id;
+
+	vas_init_rx_win_attr(&rxattr, cop);
+	rxattr.lnotify_lpid = mfspr(SPRN_LPID);
+
+	/*
+	 * Only caller can own the window for now. Not sure if there is need
+	 * for process P1 to make P2 the owner of a window. If so, we need to
+	 * find P2, make sure we have permissions, get a reference etc.
+	 */
+	rxattr.lnotify_pid = mfspr(SPRN_PID);
+	rxattr.lnotify_tid = mfspr(SPRN_TIDR);
+
+	rxwin = vas_rx_win_open(vasid, cop, &rxattr);
+	if (IS_ERR(rxwin)) {
+		pr_debug("vas_rx_win_open() failed, %ld\n", PTR_ERR(rxwin));
+		return PTR_ERR(rxwin);
+	}
+
+	vas_init_tx_win_attr(&txattr, cop);
+
+	txattr.lpid = mfspr(SPRN_LPID);
+	txattr.pidr = mfspr(SPRN_PID);
+	txattr.pid = task_pid_nr(current);
+	txattr.pswid = vas_win_id(rxwin);
+
+	txwin = vas_tx_win_open(vasid, cop, &txattr);
+	if (IS_ERR(txwin)) {
+		pr_debug("vas_tx_win_open() failed, %ld\n", PTR_ERR(txwin));
+		rc = PTR_ERR(txwin);
+		goto close_rxwin;
+	}
+
+	instance->rxwin = rxwin;
+	instance->txwin = txwin;
+
+	return 0;
+
+close_rxwin:
+	vas_win_close(rxwin);
+	return rc;
+}
+
+static int ftw_release(struct inode *inode, struct file *fp)
+{
+	struct ftw_instance *instance;
+
+	instance = fp->private_data;
+
+	if (instance->txwin)
+		vas_win_close(instance->txwin);
+	if (instance->rxwin)
+		vas_win_close(instance->rxwin);
+	/*
+	 * TODO We don't know here if user has other receive windows
+	 *      open, and can't really call clear_thread_tidr(). So,
+	 *      once the process calls set_thread_tidr(), the TIDR value
+	 *      sticks around until process exits, potentially resulting
+	 *      in an unnecessary copy in restore_sprs() when even the
+	 *      process has closed its last window.
+	 */
+
+	instance->rxwin = instance->txwin = NULL;
+
+	kfree(instance);
+	fp->private_data = NULL;
+	atomic_dec(&ftw_instid);
+
+	return 0;
+}
+
+static int ftw_mmap(struct file *fp, struct vm_area_struct *vma)
+{
+	int rc;
+	pgprot_t prot;
+	u64 paste_addr;
+	unsigned long pfn;
+	struct ftw_instance *instance = fp->private_data;
+
+	if ((vma->vm_end - vma->vm_start) > PAGE_SIZE) {
+		pr_debug("size 0x%zx, PAGE_SIZE 0x%zx\n",
+				(vma->vm_end - vma->vm_start), PAGE_SIZE);
+		return -EINVAL;
+	}
+
+	/* Ensure instance has an open send window */
+	if (!instance->txwin) {
+		pr_debug("No send window open?\n");
+		return -EINVAL;
+	}
+
+	paste_addr = vas_win_paste_addr(instance->txwin);
+	pfn = paste_addr >> PAGE_SHIFT;
+
+	/* flags, page_prot from cxl_mmap(), except we want cachable */
+	vma->vm_flags |= VM_IO | VM_PFNMAP;
+	vma->vm_page_prot = pgprot_cached(vma->vm_page_prot);
+
+	/*
+	 * We must disable page faults when emulating the paste
+	 * instruction. To ensure that the page associated with
+	 * the paste address is in memory, mark it dirty.
+	 */
+	prot = __pgprot(pgprot_val(vma->vm_page_prot) | _PAGE_DIRTY);
+
+	rc = remap_pfn_range(vma, vma->vm_start, pfn + vma->vm_pgoff,
+			vma->vm_end - vma->vm_start, prot);
+
+	pr_devel("paste addr %llx at %lx, rc %d\n", paste_addr, vma->vm_start,
+			rc);
+
+	set_thread_uses_vas();
+
+	return rc;
+}
+
+static long ftw_ioctl(struct file *fp, unsigned int cmd, unsigned long arg)
+{
+	if (cmd == FTW_SETUP)
+		return ftw_ioc_ftw_setup(fp, arg);
+
+	return -EINVAL;
+}
+
+const struct file_operations ftw_fops = {
+	.owner = THIS_MODULE,
+	.open = ftw_open,
+	.release = ftw_release,
+	.mmap = ftw_mmap,
+	.unlocked_ioctl = ftw_ioctl,
+};
+
+
+int ftw_file_init(void)
+{
+	int rc;
+	dev_t devno;
+
+	rc = alloc_chrdev_region(&ftw_device.devt, 0, 1, "ftw");
+	if (rc) {
+		pr_debug("Unable to allocate ftw major number: %i\n", rc);
+		return rc;
+	}
+
+	pr_devel("device allocated, dev [%i,%i]\n",
+			MAJOR(ftw_device.devt), MINOR(ftw_device.devt));
+
+	ftw_device.class = class_create(THIS_MODULE, "ftw");
+	if (IS_ERR(ftw_device.class)) {
+		pr_debug("Unable to create FTW class\n");
+		rc = PTR_ERR(ftw_device.class);
+		goto free_chrdev;
+	}
+	ftw_device.class->devnode = ftw_devnode;
+
+	cdev_init(&ftw_device.cdev, &ftw_fops);
+
+	devno = MKDEV(MAJOR(ftw_device.devt), 0);
+	if (cdev_add(&ftw_device.cdev, devno, 1)) {
+		pr_debug("cdev_add() failed\n");
+		goto free_class;
+	}
+
+	ftw_device.device = device_create(ftw_device.class, NULL,
+			devno, NULL, ftw_dev_name, MINOR(devno));
+	if (IS_ERR(ftw_device.device)) {
+		pr_debug("Unable to create ftw-%d\n", MINOR(devno));
+		goto free_cdev;
+	}
+
+	pr_devel("Added dev [%d,%d]\n", MAJOR(devno), MINOR(devno));
+
+	return 0;
+
+free_cdev:
+	cdev_del(&ftw_device.cdev);
+free_class:
+	class_destroy(ftw_device.class);
+free_chrdev:
+	unregister_chrdev_region(ftw_device.devt, 1);
+	return rc;
+}
+
+void ftw_file_exit(void)
+{
+	dev_t devno;
+
+	cdev_del(&ftw_device.cdev);
+	devno = MKDEV(MAJOR(ftw_device.devt), MINOR(ftw_device.devt));
+	device_destroy(ftw_device.class, devno);
+
+	class_destroy(ftw_device.class);
+	unregister_chrdev_region(ftw_device.devt, 1);
+}
+
+int __init ftw_init(void)
+{
+	int rc;
+
+	rc = ftw_file_init();
+	if (rc)
+		return rc;
+
+	pr_info("Device initialized\n");
+
+	return 0;
+}
+
+void __init ftw_exit(void)
+{
+	pr_devel("Device exiting\n");
+	ftw_file_exit();
+}
+
+module_init(ftw_init);
+module_exit(ftw_exit);
+
+MODULE_DESCRIPTION("IBM NX Fast Thread Wakeup Device");
+MODULE_AUTHOR("Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>");
+MODULE_LICENSE("GPL");
-- 
2.7.4

[PATCH 4/5] powerpc/ftw: Add a couple of trace points

[Sukadev Bhattiprolu]

Add a couple of trace points in the FTW driver

Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
---
 drivers/misc/ftw/ftw-trace.h | 75 ++++++++++++++++++++++++++++++++++++++++++++
 drivers/misc/ftw/ftw.c       |  6 ++++
 2 files changed, 81 insertions(+)
 create mode 100644 drivers/misc/ftw/ftw-trace.h

diff --git a/drivers/misc/ftw/ftw-trace.h b/drivers/misc/ftw/ftw-trace.h
new file mode 100644
index 0000000..0d96046
--- /dev/null
+++ b/drivers/misc/ftw/ftw-trace.h
@@ -0,0 +1,75 @@
+/*
+ * Copyright 2018 IBM Corp.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM	ftw
+
+#if !defined(_FTW_TRACE_H) || defined(TRACE_HEADER_MULTI_READ)
+
+#define _FTW_TRACE_H
+#include <linux/tracepoint.h>
+#include <linux/sched.h>
+
+TRACE_EVENT(	ftw_open_event,
+
+		TP_PROTO(struct task_struct *tsk,
+			 int instid),
+
+		TP_ARGS(tsk, instid),
+
+		TP_STRUCT__entry(
+			__field(struct task_struct *, tsk)
+			__field(int, instid)
+			__field(int, pid)
+		),
+
+		TP_fast_assign(
+			__entry->pid = tsk->pid;
+			__entry->instid = instid;
+		),
+
+		TP_printk("pid=%d, inst=%d", __entry->pid, __entry->instid)
+);
+
+TRACE_EVENT(	ftw_mmap_event,
+
+		TP_PROTO(struct task_struct *tsk,
+			 int instid,
+			 unsigned long paste_addr,
+			 unsigned long vma_start),
+
+		TP_ARGS(tsk, instid, paste_addr, vma_start),
+
+		TP_STRUCT__entry(
+			__field(struct task_struct *, tsk)
+			__field(int, pid)
+			__field(int, instid)
+			__field(unsigned long, paste_addr)
+			__field(unsigned long, vma_start)
+		),
+
+		TP_fast_assign(
+			__entry->pid = tsk->pid;
+			__entry->instid = instid;
+			__entry->paste_addr = paste_addr;
+			__entry->vma_start = vma_start;
+		),
+
+		TP_printk(
+			"pid=%d, inst=%d, pasteaddr=0x%16lx, vma_start=0x%16lx",
+			__entry->pid, __entry->instid, __entry->paste_addr,
+			__entry->vma_start)
+);
+
+#endif /* _FTW_TRACE_H */
+
+#undef TRACE_INCLUDE_PATH
+#define TRACE_INCLUDE_PATH .
+#define TRACE_INCLUDE_FILE ftw-trace
+#include <trace/define_trace.h>
diff --git a/drivers/misc/ftw/ftw.c b/drivers/misc/ftw/ftw.c
index 6fcb4e2..a01c9e6 100644
--- a/drivers/misc/ftw/ftw.c
+++ b/drivers/misc/ftw/ftw.c
@@ -21,6 +21,9 @@
 #include <asm/vas.h>
 #include <uapi/misc/ftw.h>
 
+#define CREATE_TRACE_POINTS
+#include "ftw-trace.h"
+
 /*
  * FTW is a device driver used to provide user space access to the
  * Core-to-Core aka Fast Thread Wakeup (FTW) functionality provided by
@@ -81,6 +84,8 @@ static int ftw_open(struct inode *inode, struct file *fp)
 
 	fp->private_data = instance;
 
+	trace_ftw_open_event(current, instance->id);
+
 	return 0;
 }
 
@@ -234,6 +239,7 @@ static int ftw_mmap(struct file *fp, struct vm_area_struct *vma)
 
 	pr_devel("paste addr %llx at %lx, rc %d\n", paste_addr, vma->vm_start,
 			rc);
+	trace_ftw_mmap_event(current, instance->id, paste_addr, vma->vm_start);
 
 	set_thread_uses_vas();
 
-- 
2.7.4

[PATCH 5/5] powerpc/ftw: Document FTW API/usage

[Sukadev Bhattiprolu]

Document the usage of the VAS Fast thread-wakeup API and add an entry in
MAINTAINERS file.

Thanks for input/comments from Benjamin Herrenschmidt, Michael Neuling,
Michael Ellerman, Robert Blackmore, Ian Munsie, Haren Myneni and Paul
Mackerras.

Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
---

Changelog[v2]
	- [Michael Neuling] Update API to use a single, VAS_FTW_SEUTP ioctl
	  rather than two ioctls.
	- [Michael Neuling] Drop "nx" from name "nx-ftw".

---
 Documentation/powerpc/ftw-api.txt | 283 ++++++++++++++++++++++++++++++++++++++
 MAINTAINERS                       |   8 ++
 2 files changed, 291 insertions(+)
 create mode 100644 Documentation/powerpc/ftw-api.txt

diff --git a/Documentation/powerpc/ftw-api.txt b/Documentation/powerpc/ftw-api.txt
new file mode 100644
index 0000000..a107628
--- /dev/null
+++ b/Documentation/powerpc/ftw-api.txt
@@ -0,0 +1,283 @@
+Virtual Accelerator Switchboard and Fast Thread-Wakeup API
+
+    Power9 processor supports a hardware subystem known as the Virtual
+    Accelerator Switchboard (VAS) which allows two entities in the Power9
+    system to efficiently exchange messages. Messages must be formatted as
+    Coprocessor Request Blocks (CRB) and be submitted using the COPY/PASTE
+    instructions (new in Power9).
+
+    Usage of VAS depends on the entities exchanging the messages and
+    currently two usages have been identified.
+
+    First usage of VAS, referred to as VAS/NX involves a software thread
+    submitting data compression requests to a co-processor (hardware/nest
+    accelerator) aka NX engine. This usage is not yet available to user
+    applications.
+
+    Alternatively, VAS can be used by two software threads to efficiently
+    exchange messages. Initially, this mechanism is intended to wake up a
+    waiting thread quickly - i.e "fast thread wake-up (FTW)". This document
+    describes the user API for this VAS/FTW mechanism.
+
+    Application access to the FTW mechanism is provided through the FTW
+    device node (/dev/ftw) implemented by the FTW device driver.
+
+    A multi-threaded software processes that intends to use the FTW
+    mechanism must first setup a channel (consisting of a pair of VAS
+    windows) for the waiting and waking threads to communicate. The
+    channel is set up by opening the FTW device and issuing the FTW_SETUP
+    ioctl. Upon successful return from the ioctl, the waiting side of
+    channel is complete and a thread can issue the "Wait" instruction
+    to wait for an event.
+
+    After the successful return from the FTW_SETUP ioctl, the waking
+    thread must use mmap() system call on the same file descriptor and
+    obtain a virtual address known as the "paste address".
+
+    Once the mmap() call succeeds the setup of "waking" side of the channel
+    is complete. To wake up a waiting thread, the waking thread should use
+    the "COPY" and "PASTE" instructions to write a zero-filled CRB to the
+    paste-address.
+
+    The wait and wake up operations can be repeated as long as the paste
+    address and the FTW file descriptor are valid (i.e until munmap() of
+    the paste address or a close() of the FTW fd).
+
+1. FTW Device Node
+
+    There is one /dev/ftw node in the system and it provides access to the
+    VAS/FTW functionality.
+
+    The only valid operations (system calls) on the FTW node are:
+
+        - open() the device for read and write.
+
+        - issue the FTW_SETUP ioctl to set up a channel.
+
+        - mmap() the file descriptor
+
+        - close the device node.
+
+    Other file operations on the FTW node are undefined.
+
+    Note that the COPY and PASTE operations go directly to the hardware
+    and do not involve system calls or go through the FTW device.
+
+    Although a system may have several instances of the VAS in the system
+    (typically, one per P9 chip) there is just one FTW device node in
+    the system.
+
+    When the FTW device node is opened, the kernel assigns a suitable
+    instance of VAS to the process. Kernel will make a best-effort attempt
+    to assign an optimal instance of VAS for the process - based on the CPU/
+    chip that the process is running on. In the initial release, the kernel
+    does not support migrating the VAS instance if the process migrates from
+    a CPU on one chip to a CPU on another chip.
+
+    Applications may chose a specific instance of the VAS using the 'vas_id'
+    field in the FTW_SETUP ioctl as detailed below.
+
+2. Open FTW node
+
+    The device should be opened for read and write. No special privileges
+    are needed to open the device. The device may be opened multiple times.
+
+    Each open() of the FTW device is associated with one channel of
+    communication. There is a system-wide limit (currently 64K windows per
+    chip and since some are reserved for hardware, there are about 32K
+    channels per chip). If no more channels are available, the open() system
+    call will fail.
+
+    See open(2) system call man pages for other details such as return
+    values, error codes and restrictions.
+
+3. Setup a communication channel (FTW_SETUP ioctl)
+
+    A process that intends to use the Fast Thread-wakeup mechanism must
+    first setup a channel by issuing the FTW_SETUP ioctl.
+
+        #include <misc/ftw.h>
+
+        struct ftw_setup_attr ftwattr;
+
+        rc = ioctl(fd, FTW_SETUP, &ftwattr);
+
+    The attributes of ftwattr are as follows:
+
+        struct ftw_setup_attr {
+                int16_t       version;
+                int16_t       vas_id;
+                uint32_t      reserved;
+
+                int64_t       reserved1;
+                int64_t       flags;
+                int64_t       reserved2;
+        };
+
+    The version field identifies the version of the API and must currently
+    be set to 1.
+
+    The vas_id field identifies a specific instance of the VAS that the
+    application wishes to access. See section on VAS ID below.
+
+    The reserved fields must all be set to zeroes.
+
+    The flags field specifies additional attributes to the channel. The
+    only valid bit in the flags for Fast thread-wakeup usage are:
+
+        FTW_FLAGS_PIN_WINDOW    if set, indicates that the channel should be
+                                pinned in cache. This flag is restricted
+                                to privileged users. See Pinning windows
+                                below.
+
+    All the other bits in the flags field must be set to 0.
+
+    Return value:
+
+    The FTW_SETUP ioctl returns 0 on success. On error, it returns -1
+    and sets the errno variable to indicate the error.
+
+    Error codes:
+
+        EINVAL      version is invalid
+
+        EINVAL      vas_id is invalid
+
+        EINVAL      fd does not refer to a valid VAS device.
+
+        ENOSPC      System has too many active channels (windows) open,
+
+        EPERM       FTW_FLAGS_PIN_WINDOW is set in 'flags' field and process
+                    is not privileged.
+
+        EINVAL      reserved fields are not set to 0.
+
+    See the ioctl(2) man page for more details, error codes and restrictions.
+
+4. mmap() FTW device fd
+
+    The mmap() system call for a FTW device fd returns a "paste address"
+    that the application can use to COPY/PASTE a CRB to the waiting thread.
+
+        paste_addr = mmap(NULL, size, prot, flags, fd, offset);
+
+    Only restrictions on mmap for a FTW device fd are:
+
+        - size parameter should be one page size
+
+        - offset parameter should be 0ULL.
+
+    Refer to mmap(2) man page for additional details/restrictions.
+
+    In addition to the error conditions listed on the mmap(2) man page,
+    mmap() can also fail with one of following error codes:
+
+        EINVAL      fd is not associated with an open channel (window)
+                    (i.e mmap() does not follow a successful call to the
+                    FTW_SETUP ioctl).
+
+        EINVAL      offset field is not 0ULL.
+
+
+5. VAS ID
+
+    A system may have several instances of VAS in the hardware, typically
+    one per POWER 9 chip. The choice of a specific instance of VAS can have
+    significant impact on the performance, specially if the application
+    migrates from one CPU to another. Applications can specify a vas_id
+    using the FTW_SETUP ioctl and should be prudent in choosing an
+    instance of VAS.
+
+    The vas_id for each instance of VAS is listed as the device tree
+    property 'ibm,vas-id'. Determining the specific vas_id to use for
+    a specific application thread is beyond the scope of this API.
+
+    If the application has no preference, the vas_id field may be set to
+    -1 and the kernel will choose a suitable instance of the VAS engine.
+
+6. COPY/PASTE operations:
+
+    Applications should use the COPY and PASTE instructions defined in
+    the RFC to copy/paste the CRB. For VAS/FTW usage, the contents of
+    CRB, are ignored and can be zero, but CRB should point to a valid buffer
+
+7. Interrupt completion and signal handling
+
+    No VAS-specific signals will be generated to the application threads
+    with the VAS/FTW usage.
+
+8. Example/Proposed usage of the VAS/FTW API
+
+    In the following example we use two threads that use the VAS/FTW API.
+    Thread T1 sets up the channel and uses the WAIT instruction to wait for
+    an event. Thread T2 uses copy/paste instructions to wake up T1.
+    Note that the pthread_cond_wait() calls must be in a loop for spurious
+    wake ups, but are simplified here.
+
+    Common interfaces:
+
+        static bool paste_done;
+
+        #define WAIT    .long (0x7C00003C)
+
+        static inline int do_wait(void)
+        {
+                __asm__ __volatile(stringify_in_c(WAIT)";");
+        }
+
+        /*
+         * Check if paste_done is true
+         */
+        static bool is_paste_done(void)
+        {
+                return __sync_bool_compare_and_swap(&paste_done, 1, 0);
+
+        }
+
+        /*
+         * Set paste_done to true
+         */
+        static inline void set_paste_done(void)
+        {
+                __sync_bool_compare_and_swap(&paste_done, 0, 1);
+        }
+
+
+        int fd = -1;        // global, visible to both T1 and T2
+
+    Thread T1:
+
+        struct ftw_setup_attr ftwattr;
+
+        fd = open("/dev/ftw", O_RDWR);
+
+        memset(&rxattr, 0, sizeof(rxattr));
+        ftwattr.version = 1;
+        ftwattr.vas_id = -1;
+
+        rc = ioctl(fd, FTW_SETUP, &ftwattr);
+
+        /* Tell T2 that waiter side of channel is ready */
+        pthread_cond_signal(&rx_win_ready);
+
+        /* Rx set up done */
+
+        /* later, wait for an event to occur */
+
+        while(!is_paste_done())
+            do_wait();
+
+    Thread T2:
+
+        /* Wait for waiter side of channel to be set up first */
+        pthread_cond_wait(&rx_win_ready);
+
+        prot = PROT_READ|PROT_WRITE;
+        paste_addr = mmap(NULL, 4096, prot, MAP_SHARED, fd, 0ULL);
+
+        /* Tx setup done */
+
+        /* later ... */
+
+        set_paste_done();           /* ... event occurred */
+        write_empty_crb(paste_addr); /* wake up T1 */
diff --git a/MAINTAINERS b/MAINTAINERS
index 1899480..cb4b0f7 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4244,6 +4244,14 @@ L:	linux-i2c@vger.kernel.org
 S:	Maintained
 F:	drivers/i2c/busses/i2c-diolan-u2c.c
 
+FAST THREAD-WAKEUP DRIVER
+M:	Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
+L:	linuxppc-dev@lists.ozlabs.org
+S:	Maintained
+F:	drivers/misc/ftw/
+F:	include/uapi/misc/ftw.h
+F:	Documentation/powerpc/ftw-api.txt
+
 FILESYSTEM DIRECT ACCESS (DAX)
 M:	Matthew Wilcox <mawilcox@microsoft.com>
 M:	Ross Zwisler <ross.zwisler@linux.intel.com>
-- 
2.7.4

Re: [PATCH 5/5] powerpc/ftw: Document FTW API/usage

[Randy Dunlap]

On 01/16/2018 06:50 PM, Sukadev Bhattiprolu wrote:
> Document the usage of the VAS Fast thread-wakeup API and add an entry in
> MAINTAINERS file.
> 
> Thanks for input/comments from Benjamin Herrenschmidt, Michael Neuling,
> Michael Ellerman, Robert Blackmore, Ian Munsie, Haren Myneni and Paul
> Mackerras.
> 
> Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
> ---
> 
> Changelog[v2]
> 	- [Michael Neuling] Update API to use a single, VAS_FTW_SEUTP ioctl
> 	  rather than two ioctls.
> 	- [Michael Neuling] Drop "nx" from name "nx-ftw".
> 
> ---
>  Documentation/powerpc/ftw-api.txt | 283 ++++++++++++++++++++++++++++++++++++++
>  MAINTAINERS                       |   8 ++
>  2 files changed, 291 insertions(+)
>  create mode 100644 Documentation/powerpc/ftw-api.txt
> 
> diff --git a/Documentation/powerpc/ftw-api.txt b/Documentation/powerpc/ftw-api.txt
> new file mode 100644
> index 0000000..a107628
> --- /dev/null
> +++ b/Documentation/powerpc/ftw-api.txt
> @@ -0,0 +1,283 @@
> +Virtual Accelerator Switchboard and Fast Thread-Wakeup API
> +
...
> +
> +    Application access to the FTW mechanism is provided through the FTW
> +    device node (/dev/ftw) implemented by the FTW device driver.
> +
> +    A multi-threaded software processes that intends to use the FTW

                                 process

> +    mechanism must first setup a channel (consisting of a pair of VAS
> +    windows) for the waiting and waking threads to communicate. The
> +    channel is set up by opening the FTW device and issuing the FTW_SETUP
> +    ioctl. Upon successful return from the ioctl, the waiting side of
> +    channel is complete and a thread can issue the "Wait" instruction
> +    to wait for an event.
> +
> +    After the successful return from the FTW_SETUP ioctl, the waking
> +    thread must use mmap() system call on the same file descriptor and
> +    obtain a virtual address known as the "paste address".
> +
> +    Once the mmap() call succeeds the setup of "waking" side of the channel
> +    is complete. To wake up a waiting thread, the waking thread should use
> +    the "COPY" and "PASTE" instructions to write a zero-filled CRB to the
> +    paste-address.
> +
> +    The wait and wake up operations can be repeated as long as the paste
> +    address and the FTW file descriptor are valid (i.e until munmap() of
> +    the paste address or a close() of the FTW fd).
> +
> +1. FTW Device Node
> +
> +    There is one /dev/ftw node in the system and it provides access to the
> +    VAS/FTW functionality.
> +
> +    The only valid operations (system calls) on the FTW node are:
> +
> +        - open() the device for read and write.
> +
> +        - issue the FTW_SETUP ioctl to set up a channel.
> +
> +        - mmap() the file descriptor
> +
> +        - close the device node.
> +
> +    Other file operations on the FTW node are undefined.
> +
> +    Note that the COPY and PASTE operations go directly to the hardware
> +    and do not involve system calls or go through the FTW device.
> +
> +    Although a system may have several instances of the VAS in the system
> +    (typically, one per P9 chip) there is just one FTW device node in
> +    the system.
> +
> +    When the FTW device node is opened, the kernel assigns a suitable
> +    instance of VAS to the process. Kernel will make a best-effort attempt
> +    to assign an optimal instance of VAS for the process - based on the CPU/
> +    chip that the process is running on. In the initial release, the kernel
> +    does not support migrating the VAS instance if the process migrates from
> +    a CPU on one chip to a CPU on another chip.
> +
> +    Applications may chose a specific instance of the VAS using the 'vas_id'

                        choose

> +    field in the FTW_SETUP ioctl as detailed below.
> +
> +2. Open FTW node
> +
> +    The device should be opened for read and write. No special privileges
> +    are needed to open the device. The device may be opened multiple times.
> +
> +    Each open() of the FTW device is associated with one channel of
> +    communication. There is a system-wide limit (currently 64K windows per
> +    chip and since some are reserved for hardware, there are about 32K
> +    channels per chip). If no more channels are available, the open() system
> +    call will fail.
> +
> +    See open(2) system call man pages for other details such as return
> +    values, error codes and restrictions.
> +
> +3. Setup a communication channel (FTW_SETUP ioctl)
> +
> +    A process that intends to use the Fast Thread-wakeup mechanism must
> +    first setup a channel by issuing the FTW_SETUP ioctl.
> +
> +        #include <misc/ftw.h>
> +
> +        struct ftw_setup_attr ftwattr;
> +
> +        rc = ioctl(fd, FTW_SETUP, &ftwattr);
> +
> +    The attributes of ftwattr are as follows:
> +
> +        struct ftw_setup_attr {
> +                int16_t       version;
> +                int16_t       vas_id;
> +                uint32_t      reserved;
> +
> +                int64_t       reserved1;
> +                int64_t       flags;
> +                int64_t       reserved2;
> +        };
> +
> +    The version field identifies the version of the API and must currently
> +    be set to 1.
> +
> +    The vas_id field identifies a specific instance of the VAS that the
> +    application wishes to access. See section on VAS ID below.
> +
> +    The reserved fields must all be set to zeroes.
> +
> +    The flags field specifies additional attributes to the channel. The
> +    only valid bit in the flags for Fast thread-wakeup usage are:
> +
> +        FTW_FLAGS_PIN_WINDOW    if set, indicates that the channel should be
> +                                pinned in cache. This flag is restricted
> +                                to privileged users. See Pinning windows
> +                                below.
> +
> +    All the other bits in the flags field must be set to 0.
> +
> +    Return value:
> +
> +    The FTW_SETUP ioctl returns 0 on success. On error, it returns -1
> +    and sets the errno variable to indicate the error.
> +
> +    Error codes:
> +
> +        EINVAL      version is invalid
> +
> +        EINVAL      vas_id is invalid
> +
> +        EINVAL      fd does not refer to a valid VAS device.
> +
> +        ENOSPC      System has too many active channels (windows) open,

s/,/./ or s/,//

> +
> +        EPERM       FTW_FLAGS_PIN_WINDOW is set in 'flags' field and process
> +                    is not privileged.
> +
> +        EINVAL      reserved fields are not set to 0.
> +
> +    See the ioctl(2) man page for more details, error codes and restrictions.
> +
> +4. mmap() FTW device fd
> +
> +    The mmap() system call for a FTW device fd returns a "paste address"
> +    that the application can use to COPY/PASTE a CRB to the waiting thread.
> +
> +        paste_addr = mmap(NULL, size, prot, flags, fd, offset);
> +
> +    Only restrictions on mmap for a FTW device fd are:
> +
> +        - size parameter should be one page size
> +
> +        - offset parameter should be 0ULL.
> +
> +    Refer to mmap(2) man page for additional details/restrictions.
> +
> +    In addition to the error conditions listed on the mmap(2) man page,
> +    mmap() can also fail with one of following error codes:
> +
> +        EINVAL      fd is not associated with an open channel (window)
> +                    (i.e mmap() does not follow a successful call to the
> +                    FTW_SETUP ioctl).
> +
> +        EINVAL      offset field is not 0ULL.
> +
> +
> +5. VAS ID
...
> +
> +6. COPY/PASTE operations:
> +
> +    Applications should use the COPY and PASTE instructions defined in
> +    the RFC to copy/paste the CRB. For VAS/FTW usage, the contents of
> +    CRB, are ignored and can be zero, but CRB should point to a valid buffer

       CRB are ignored                                                   buffer.

> +
> +7. Interrupt completion and signal handling
> +
> +    No VAS-specific signals will be generated to the application threads
> +    with the VAS/FTW usage.
> +
> +8. Example/Proposed usage of the VAS/FTW API
> +
> +    In the following example we use two threads that use the VAS/FTW API.
> +    Thread T1 sets up the channel and uses the WAIT instruction to wait for
> +    an event. Thread T2 uses copy/paste instructions to wake up T1.
> +    Note that the pthread_cond_wait() calls must be in a loop for spurious
> +    wake ups, but are simplified here.
> +
> +    Common interfaces:
> +
> +        static bool paste_done;
> +
> +        #define WAIT    .long (0x7C00003C)
> +
> +        static inline int do_wait(void)
> +        {
> +                __asm__ __volatile(stringify_in_c(WAIT)";");
> +        }
> +
> +        /*
> +         * Check if paste_done is true
> +         */
> +        static bool is_paste_done(void)
> +        {
> +                return __sync_bool_compare_and_swap(&paste_done, 1, 0);
> +
> +        }
> +
> +        /*
> +         * Set paste_done to true
> +         */
> +        static inline void set_paste_done(void)
> +        {
> +                __sync_bool_compare_and_swap(&paste_done, 0, 1);
> +        }
> +
> +
> +        int fd = -1;        // global, visible to both T1 and T2
> +
> +    Thread T1:
> +
> +        struct ftw_setup_attr ftwattr;
> +
> +        fd = open("/dev/ftw", O_RDWR);
> +
> +        memset(&rxattr, 0, sizeof(rxattr));

Is that supposed to be ftwattr (2x above)?

> +        ftwattr.version = 1;
> +        ftwattr.vas_id = -1;
> +
> +        rc = ioctl(fd, FTW_SETUP, &ftwattr);
> +
> +        /* Tell T2 that waiter side of channel is ready */
> +        pthread_cond_signal(&rx_win_ready);
> +
> +        /* Rx set up done */
> +
> +        /* later, wait for an event to occur */
> +
> +        while(!is_paste_done())
> +            do_wait();
> +
> +    Thread T2:
> +
> +        /* Wait for waiter side of channel to be set up first */
> +        pthread_cond_wait(&rx_win_ready);
> +
> +        prot = PROT_READ|PROT_WRITE;
> +        paste_addr = mmap(NULL, 4096, prot, MAP_SHARED, fd, 0ULL);
> +
> +        /* Tx setup done */
> +
> +        /* later ... */
> +
> +        set_paste_done();           /* ... event occurred */
> +        write_empty_crb(paste_addr); /* wake up T1 */


-- 
~Randy

Re: [PATCH 5/5] powerpc/ftw: Document FTW API/usage

[Sukadev Bhattiprolu]

Randy Dunlap [rdunlap@infradead.org] wrote:

> > +        struct ftw_setup_attr ftwattr;
> > +
> > +        fd = open("/dev/ftw", O_RDWR);
> > +
> > +        memset(&rxattr, 0, sizeof(rxattr));
> 
> Is that supposed to be ftwattr (2x above)?

Yes. I agree with your other comments as well and will send a new version.

Thanks for the detailed review.

Sukadev