[PATCH v3 00/10] Enable VAS

[PATCH v3 00/10] Enable VAS

[Sukadev Bhattiprolu]

Power9 introduces a hardware subsystem referred to as the Virtual
Accelerator Switchboard (VAS). VAS allows kernel subsystems and user
space processes to directly access the Nest Accelerator (NX) engines
which implement compression and encryption algorithms in the hardware.

NX has been in Power processors since Power7+, but access to the NX
engines was through the 'icswx' instruction which is only available
to the kernel/hypervisor. Starting with Power9, access to the NX
engines is provided to both kernel and user space processes through
VAS.

The switchboard (i.e VAS) multiplexes accesses between "receivers" and
"senders", where the "receivers" are typically the NX engines and
"senders" are the kernel subsystems and user processors that wish to
access the receivers (NX engines).  Once a sender is "connected" to
a receiver through the switchboard, the senders can submit compression/
encryption requests to the hardware using the new (PowerISA 3.0)
"copy" and "paste" instructions.

In the initial OPAL and PowerNV kernel patchsets, the "senders" can
only be kernel subsystems (eg NX-842 driver). A follow-on patch set 
will allow senders to be user-space processes.

This kernel patch set configures the VAS subsystems and provides
kernel interfaces to drivers like NX-842 to open receive and send
windows in VAS and to submit requests to the NX engine.

This patch set that has been tested in a Simics Power9 environment using
a modified NX-842 kernel driver and a compression self-test module from
Power8. The corresponding OPAL patchset for VAS support was posted to
skiboot mailing list:

	https://lists.ozlabs.org/pipermail/skiboot/2017-January/006193.html
	
OPAL and kernel patchsets for NX-842 driver will be posted separately.
All four patchsets are needed to effectively use VAS/NX in Power9.

Thanks to input from Ben Herrenschmidt, Michael Neuling, Michael Ellerman
and Haren Myneni.

Changelog[v3]
	- Rebase to v4.11-rc1
	- Add interfaces to initialize send/receive window attributes to
	  defaults that drivers can use (see arch/powerpc/include/asm/vas.h)
	- Modify interface vas_paste() to return 0 or error code
	- Fix a bug in setting Translation Control Mode (0b11 not 0x11)
	- Enable send-window-credit checking 
	- Reorg code  in vas_win_close()
	- Minor reorgs and tweaks to register field settings to make it
	  easier to add support for user space windows.
	- Skip writing to read-only registers
	- Start window indexing from 0 rather than 1

Changelog[v2]
	- Use vas-id, HVWC, UWC and paste address, entries from device tree
	  rather than defining/computing them in kernel and reorg code.

Sukadev Bhattiprolu (10):
  VAS: Define macros, register fields and structures
  Move GET_FIELD/SET_FIELD to vas.h
  VAS: Define vas_init() and vas_exit()
  VAS: Define helpers for access MMIO regions
  VAS: Define helpers to init window context
  VAS: Define helpers to alloc/free windows
  VAS: Define vas_rx_win_open() interface
  VAS: Define vas_win_close() interface
  VAS: Define vas_tx_win_open()
  VAS: Define copy/paste interfaces

 MAINTAINERS                        |   14 +-
 arch/powerpc/include/asm/reg.h     |    1 +
 arch/powerpc/include/asm/vas.h     |  157 ++++++
 drivers/crypto/nx/nx-842-powernv.c |    7 +-
 drivers/crypto/nx/nx-842.h         |    5 -
 drivers/misc/Kconfig               |    1 +
 drivers/misc/Makefile              |    1 +
 drivers/misc/vas/Kconfig           |   21 +
 drivers/misc/vas/Makefile          |    3 +
 drivers/misc/vas/copy-paste.h      |   74 +++
 drivers/misc/vas/vas-internal.h    |  473 ++++++++++++++++
 drivers/misc/vas/vas-window.c      | 1076 ++++++++++++++++++++++++++++++++++++
 drivers/misc/vas/vas.c             |  155 ++++++
 13 files changed, 1978 insertions(+), 10 deletions(-)
 create mode 100644 arch/powerpc/include/asm/vas.h
 create mode 100644 drivers/misc/vas/Kconfig
 create mode 100644 drivers/misc/vas/Makefile
 create mode 100644 drivers/misc/vas/copy-paste.h
 create mode 100644 drivers/misc/vas/vas-internal.h
 create mode 100644 drivers/misc/vas/vas-window.c
 create mode 100644 drivers/misc/vas/vas.c

-- 
2.7.4

[PATCH v3 01/10] VAS: Define macros, register fields and structures

[Sukadev Bhattiprolu]

Define macros for the VAS hardware registers and bit-fields as well
as couple of data structures needed by the VAS driver.

Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
---
Changelog[v3]
	- Rename winctx->pid to winctx->pidr to reflect that its a value
	  from the PID register (SPRN_PID), not the linux process id.
	- Make it easier to split header into kernel/user parts
	- To keep user interface simple, use macros rather than enum for
	  the threshold-control modes.
	- Add a pid field to struct vas_window - needed for user space
	  send windows.

Changelog[v2]
	- Add an overview of VAS in vas-internal.h
	- Get window context parameters from device tree and drop
	  unnecessary macros.
---
 MAINTAINERS                     |   6 +
 arch/powerpc/include/asm/vas.h  |  43 +++++
 drivers/misc/vas/vas-internal.h | 392 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 441 insertions(+)
 create mode 100644 arch/powerpc/include/asm/vas.h
 create mode 100644 drivers/misc/vas/vas-internal.h

diff --git a/MAINTAINERS b/MAINTAINERS
index c265a5f..2a910c9 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -13213,6 +13213,12 @@ S:	Maintained
 F:	Documentation/fb/uvesafb.txt
 F:	drivers/video/fbdev/uvesafb.*
 
+VAS (IBM Virtual Accelerator Switchboard) DRIVER
+M:	Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
+L:	linuxppc-dev@lists.ozlabs.org
+S:	Supported
+F:	drivers/misc/vas/*
+
 VF610 NAND DRIVER
 M:	Stefan Agner <stefan@agner.ch>
 L:	linux-mtd@lists.infradead.org
diff --git a/arch/powerpc/include/asm/vas.h b/arch/powerpc/include/asm/vas.h
new file mode 100644
index 0000000..6d35ce6
--- /dev/null
+++ b/arch/powerpc/include/asm/vas.h
@@ -0,0 +1,43 @@
+/*
+ * Copyright 2016 IBM Corp.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+
+#ifndef VAS_H
+#define VAS_H
+
+/*
+ * Threshold Control Mode: Have paste operation fail if the number of
+ * requests in receive FIFO exceeds a threshold.
+ *
+ * NOTE: No special error code yet if paste is rejected because of these
+ *	 limits. So users can't distinguish between this and other errors.
+ */
+#define VAS_THRESH_DISABLED		0
+#define VAS_THRESH_FIFO_GT_HALF_FULL	1
+#define VAS_THRESH_FIFO_GT_QTR_FULL	2
+#define VAS_THRESH_FIFO_GT_EIGHTH_FULL	3
+
+#ifdef __KERNEL__
+
+#define VAS_RX_FIFO_SIZE_MAX	(8 << 20)	/* 8MB */
+/*
+ * Co-processor Engine type.
+ */
+enum vas_cop_type {
+	VAS_COP_TYPE_FAULT,
+	VAS_COP_TYPE_842,
+	VAS_COP_TYPE_842_HIPRI,
+	VAS_COP_TYPE_GZIP,
+	VAS_COP_TYPE_GZIP_HIPRI,
+	VAS_COP_TYPE_MAX,
+};
+
+
+#endif	/* __KERNEL__ */
+
+#endif
diff --git a/drivers/misc/vas/vas-internal.h b/drivers/misc/vas/vas-internal.h
new file mode 100644
index 0000000..ce48f14
--- /dev/null
+++ b/drivers/misc/vas/vas-internal.h
@@ -0,0 +1,392 @@
+/*
+ * Copyright 2016 IBM Corp.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+
+#ifndef VAS_INTERNAL_H
+#define VAS_INTERNAL_H
+#include <linux/atomic.h>
+#include <linux/idr.h>
+#include <asm/vas.h>
+
+#ifdef CONFIG_PPC_4K_PAGES
+#	error "TODO: Compute RMA/Paste-address for 4K pages."
+#else
+#ifndef CONFIG_PPC_64K_PAGES
+#	error "Unexpected Page size."
+#endif
+#endif
+
+/*
+ * Overview of Virtual Accelerator Switchboard (VAS).
+ *
+ * VAS is a hardware "switchboard" that allows senders and receivers to
+ * exchange messages with _minimal_ kernel involvment. The receivers are
+ * typically NX coprocessor engines that perform compression or encryption
+ * in hardware, but receivers can also be other software threads.
+ *
+ * Senders are user/kernel threads that submit compression/encryption or
+ * other requests to the receivers. Senders must format their messages as
+ * Coprocessor Request Blocks (CRB)s and submit them using the instructions
+ * "copy" and "paste" which were introduced in Power9.
+ *
+ * A Power node can have (upto?) 8 Power chips. There is one instance of
+ * VAS in each Power9 chip. Each instance of VAS has 64K windows or ports,
+ * Senders and receivers must each connect to a separate window before they
+ * can exchange messages through the switchboard.
+ *
+ * Each window is described by two types of window contexts:
+ *
+ *	Hypervisor Window Context (HVWC) of size VAS_HVWC_SIZE bytes
+ *
+ *	OS/User Window Context (UWC) of size VAS_UWC_SIZE bytes.
+ *
+ * A window context can be viewed as a set of 64-bit registers. The settings
+ * in these registers configure/control/determine the behavior of the VAS
+ * hardware when messages are sent/received through the window. The registers
+ * in the HVWC are configured by the kernel while the registers in the UWC can
+ * be configured by the kernel or by the user space application that is using
+ * the window.
+ *
+ * The HVWCs for all windows on a specific instance of VAS are in a contiguous
+ * range of hardware addresses or Base address region (BAR) referred to as the
+ * HVWC BAR for the instance. Similarly the UWCs for all windows on an instance
+ * are referred to as the UWC BAR for the instance. The two BARs for each
+ * instance are defined Power9 MMIO Ranges spreadsheet and available to the
+ * kernel the device tree as follows:
+ *
+ *	/proc/device-tree/xscom@.../vas@.../hvwc-bar-start
+ *	/proc/device-tree/xscom@.../vas@.../hvwc-bar-size
+ *	/proc/device-tree/xscom@.../vas@.../uwc-bar-start
+ *	/proc/device-tree/xscom@.../vas@.../uwc-bar-size
+ *
+ * The kernel maps these two hardware address regions into the kernel address
+ * space (hvwc_map and uwc_map) and accesses the window contexts of a specific
+ * window using:
+ *
+ *	 hvwc = hvwc_map + winid * VAS_HVWC_SIZE.
+ *	 uwc = uwc_map + winid * VAS_UWC_SIZE.
+ *
+ * where winid is the window index (0..64K).
+ *
+ * Note that the window contexts are used to "configure" the windows. In
+ * addition to this configuration address, each _send_ window also has a
+ * unique hardware address, referred to as the "paste-address" to which the
+ * sender must "paste" the message (CRB) they wish to submit. This hardware
+ * paste address for window can be computed from the following nodes in the
+ * device tree:
+ *
+ *	/proc/device-tree/xscom@.../vas@.../window-base
+ *	/proc/device-tree/xscom@.../vas@.../window-shift
+ *
+ * Thus if 'base' and 'shift' give the values of the above nodes for a given
+ * instance of VAS, the paste address for a window can be computed using:
+ *
+ *	paste_addr = base + ((winid << shift))
+ *
+ * The kernel maps this hardware address into the sender's address space after
+ * which they can use the 'paste' instruction to send a message (submit a
+ * request).
+ *
+ * NOTE: In the initial version, senders can only in-kernel drivers/threads.
+ *	 Support for user space threads will be added in follow-on patches.
+ *
+ * TODO: Do we need to map the UWC into user address space so they can return
+ *	 credits? Its NA for NX but may be needed for other receive windows.
+ *
+ */
+
+/* TODO: Increase to 64K after initial development */
+#define VAS_MAX_WINDOWS_PER_CHIP       64
+
+/*
+ * Hypervisor and OS/USer Window Context sizes
+ */
+#define VAS_HVWC_SIZE			512
+#define VAS_UWC_SIZE			PAGE_SIZE
+
+/*
+ * Initial per-process credits.
+ * Max send window credits:    4K-1 (12-bits in VAS_TX_WCRED)
+ * Max receive window credits: 64K-1 (16 bits in VAS_LRX_WCRED)
+ *
+ * TODO: Needs tuning for per-process credits
+ */
+#define VAS_WCREDS_MIN			16
+#define VAS_WCREDS_MAX			((64 << 10) - 1)
+#define VAS_WCREDS_DEFAULT		(1 << 10)
+
+/*
+ * VAS Window Context Register Offsets and bitmasks.
+ * See Section 3.1.4 of VAS Work book
+ */
+#define VAS_LPID_OFFSET			0x010
+#define VAS_LPID			PPC_BITMASK(0, 11)
+
+#define VAS_PID_OFFSET			0x018
+#define VAS_PID_ID			PPC_BITMASK(0, 19)
+
+#define VAS_XLATE_MSR_OFFSET		0x020
+#define VAS_XLATE_MSR_DR		PPC_BIT(0)
+#define VAS_XLATE_MSR_TA		PPC_BIT(1)
+#define VAS_XLATE_MSR_PR		PPC_BIT(2)
+#define VAS_XLATE_MSR_US		PPC_BIT(3)
+#define VAS_XLATE_MSR_HV		PPC_BIT(4)
+#define VAS_XLATE_MSR_SF		PPC_BIT(5)
+#define VAS_XLATE_MSR_UV		PPC_BIT(6)
+
+#define VAS_XLATE_LPCR_OFFSET		0x028
+#define VAS_XLATE_LPCR_PAGE_SIZE	PPC_BITMASK(0, 2)
+#define VAS_XLATE_LPCR_ISL		PPC_BIT(3)
+#define VAS_XLATE_LPCR_TC		PPC_BIT(4)
+#define VAS_XLATE_LPCR_SC		PPC_BIT(5)
+
+#define VAS_XLATE_CTL_OFFSET		0x030
+#define VAS_XLATE_MODE			PPC_BITMASK(0, 1)
+
+#define VAS_AMR_OFFSET			0x040
+#define VAS_AMR				PPC_BITMASK(0, 63)
+
+#define VAS_SEIDR_OFFSET		0x048
+#define VAS_SEIDR			PPC_BITMASK(0, 63)
+
+#define VAS_FAULT_TX_WIN_OFFSET		0x050
+#define VAS_FAULT_TX_WIN		PPC_BITMASK(48, 63)
+
+#define VAS_OSU_INTR_SRC_RA_OFFSET	0x060
+#define VAS_OSU_INTR_SRC_RA		PPC_BITMASK(8, 63)
+
+#define VAS_HV_INTR_SRC_RA_OFFSET	0x070
+#define VAS_HV_INTR_SRC_RA		PPC_BITMASK(8, 63)
+
+#define VAS_PSWID_OFFSET		0x078
+#define VAS_PSWID_EA_HANDLE		PPC_BITMASK(0, 31)
+
+#define VAS_SPARE1_OFFSET		0x080
+#define VAS_SPARE2_OFFSET		0x088
+#define VAS_SPARE3_OFFSET		0x090
+#define VAS_SPARE4_OFFSET		0x130
+#define VAS_SPARE5_OFFSET		0x160
+#define VAS_SPARE6_OFFSET		0x188
+
+#define VAS_LFIFO_BAR_OFFSET		0x0A0
+#define VAS_LFIFO_BAR			PPC_BITMASK(8, 53)
+#define VAS_PAGE_MIGRATION_SELECT	PPC_BITMASK(54, 56)
+
+#define VAS_LDATA_STAMP_CTL_OFFSET	0x0A8
+#define VAS_LDATA_STAMP			PPC_BITMASK(0, 1)
+#define VAS_XTRA_WRITE			PPC_BIT(2)
+
+#define VAS_LDMA_CACHE_CTL_OFFSET	0x0B0
+#define VAS_LDMA_TYPE			PPC_BITMASK(0, 1)
+
+#define VAS_LRFIFO_PUSH_OFFSET		0x0B8
+#define VAS_LRFIFO_PUSH			PPC_BITMASK(0, 15)
+
+#define VAS_CURR_MSG_COUNT_OFFSET	0x0C0
+#define VAS_CURR_MSG_COUNT		PPC_BITMASK(0, 7)
+
+#define VAS_LNOTIFY_AFTER_COUNT_OFFSET	0x0C8
+#define VAS_LNOTIFY_AFTER_COUNT		PPC_BITMASK(0, 7)
+
+#define VAS_LRX_WCRED_OFFSET		0x0E0
+#define VAS_LRX_WCRED			PPC_BITMASK(0, 15)
+
+#define VAS_LRX_WCRED_ADDER_OFFSET	0x190
+#define VAS_LRX_WCRED_ADDER		PPC_BITMASK(0, 15)
+
+#define VAS_TX_WCRED_OFFSET		0x0F0
+#define VAS_TX_WCRED			PPC_BITMASK(4, 15)
+
+#define VAS_TX_WCRED_ADDER_OFFSET	0x1A0
+#define VAS_TX_WCRED_ADDER		PPC_BITMASK(4, 15)
+
+#define VAS_LFIFO_SIZE_OFFSET		0x100
+#define VAS_LFIFO_SIZE			PPC_BITMASK(0, 3)
+
+#define VAS_WINCTL_OFFSET		0x108
+#define VAS_WINCTL_OPEN			PPC_BIT(0)
+#define VAS_WINCTL_REJ_NO_CREDIT	PPC_BIT(1)
+#define VAS_WINCTL_PIN			PPC_BIT(2)
+#define VAS_WINCTL_TX_WCRED_MODE	PPC_BIT(3)
+#define VAS_WINCTL_RX_WCRED_MODE	PPC_BIT(4)
+#define VAS_WINCTL_TX_WORD_MODE		PPC_BIT(5)
+#define VAS_WINCTL_RX_WORD_MODE		PPC_BIT(6)
+#define VAS_WINCTL_RSVD_TXBUF		PPC_BIT(7)
+#define VAS_WINCTL_THRESH_CTL		PPC_BITMASK(8, 9)
+#define VAS_WINCTL_FAULT_WIN		PPC_BIT(10)
+#define VAS_WINCTL_NX_WIN		PPC_BIT(11)
+
+#define VAS_WIN_STATUS_OFFSET		0x110
+#define VAS_WIN_BUSY			PPC_BIT(1)
+
+#define VAS_WIN_CTX_CACHING_CTL_OFFSET	0x118
+#define VAS_CASTOUT_REQ			PPC_BIT(0)
+#define VAS_PUSH_TO_MEM			PPC_BIT(1)
+#define VAS_WIN_CACHE_STATUS		PPC_BIT(4)
+
+#define VAS_TX_RSVD_BUF_COUNT_OFFSET	0x120
+#define VAS_RXVD_BUF_COUNT		PPC_BITMASK(58, 63)
+
+#define VAS_LRFIFO_WIN_PTR_OFFSET	0x128
+#define VAS_LRX_WIN_ID			PPC_BITMASK(0, 15)
+
+/*
+ * Local Notification Control Register controls what happens in _response_
+ * to a paste command and hence applies only to receive windows.
+ */
+#define VAS_LNOTIFY_CTL_OFFSET		0x138
+#define VAS_NOTIFY_DISABLE		PPC_BIT(0)
+#define VAS_INTR_DISABLE		PPC_BIT(1)
+#define VAS_NOTIFY_EARLY		PPC_BIT(2)
+#define VAS_NOTIFY_OSU_INTR		PPC_BIT(3)
+
+#define VAS_LNOTIFY_PID_OFFSET		0x140
+#define VAS_LNOTIFY_PID			PPC_BITMASK(0, 19)
+
+#define VAS_LNOTIFY_LPID_OFFSET		0x148
+#define VAS_LNOTIFY_LPID		PPC_BITMASK(0, 11)
+
+#define VAS_LNOTIFY_TID_OFFSET		0x150
+#define VAS_LNOTIFY_TID			PPC_BITMASK(0, 15)
+
+#define VAS_LNOTIFY_SCOPE_OFFSET	0x158
+#define VAS_LNOTIFY_MIN_SCOPE		PPC_BITMASK(0, 1)
+#define VAS_LNOTIFY_MAX_SCOPE		PPC_BITMASK(2, 3)
+
+#define VAS_NX_UTIL_OFFSET		0x1B0
+#define VAS_NX_UTIL			PPC_BITMASK(0, 63)
+
+/* SE: Side effects */
+#define VAS_NX_UTIL_SE_OFFSET		0x1B8
+#define VAS_NX_UTIL_SE			PPC_BITMASK(0, 63)
+
+#define VAS_NX_UTIL_ADDER_OFFSET	0x180
+#define VAS_NX_UTIL_ADDER		PPC_BITMASK(32, 63)
+
+/*
+ * Local Notify Scope Control Register. (Receive windows only).
+ */
+enum vas_notify_scope {
+	VAS_SCOPE_LOCAL,
+	VAS_SCOPE_GROUP,
+	VAS_SCOPE_VECTORED_GROUP,
+	VAS_SCOPE_UNUSED,
+};
+
+/*
+ * Local DMA Cache Control Register (Receive windows only).
+ */
+enum vas_dma_type {
+	VAS_DMA_TYPE_INJECT,
+	VAS_DMA_TYPE_WRITE,
+};
+
+/*
+ * Local Notify Scope Control Register. (Receive windows only).
+ * Not applicable to NX receive windows.
+ */
+enum vas_notify_after_count {
+	VAS_NOTIFY_AFTER_256 = 0,
+	VAS_NOTIFY_NONE,
+	VAS_NOTIFY_AFTER_2
+};
+
+/*
+ * One per instance of VAS. Each instance will have a separate set of
+ * receive windows, one per coprocessor type.
+ */
+struct vas_instance {
+	int vas_id;
+	struct ida ida;
+
+	u64 hvwc_bar_start;
+	u64 hvwc_bar_len;
+	u64 uwc_bar_start;
+	u64 uwc_bar_len;
+	u64 win_base_addr;
+	u64 win_id_shift;
+
+	struct mutex mutex;
+	struct vas_window *rxwin[VAS_COP_TYPE_MAX];
+};
+
+/*
+ * In-kernel data structure for a VAS window. One per window.
+ */
+struct vas_window {
+	/* Fields common to Send and receive windows */
+	struct vas_instance *vinst;
+	int winid;
+	bool tx_win;		/* True if send window */
+	bool nx_win;		/* True if NX window */
+	void *hvwc_map;		/* HV window context */
+	void *uwc_map;		/* OS/User window context */
+
+	/* Fields applicable only to send windows */
+	pid_t pid;		/* linux process id of sender, if applicable */
+	void *paste_kaddr;
+	char *paste_addr_name;
+	struct vas_window *rxwin;
+
+	/* Feilds applicable only to receive windows */
+	enum vas_cop_type cop;
+	atomic_t num_txwins;
+
+	int32_t hwirq;
+	uint64_t irq_port;
+};
+
+/*
+ * A VAS Window context is a 512-byte area in the hardware that contains
+ * a set of 64-bit registers. Individual bit-fields in these registers
+ * determine the configuration/operation of the hardware. struct vas_winctx
+ * is a container for the register fields in the window context.
+ * One per window.
+ */
+struct vas_winctx {
+	void *rx_fifo;
+	int rx_fifo_size;
+	int wcreds_max;
+	int rsvd_txbuf_count;
+
+	bool user_win;
+	bool nx_win;
+	bool fault_win;
+	bool rsvd_txbuf_enable;
+	bool pin_win;
+	bool rej_no_credit;
+	bool tx_wcred_mode;
+	bool rx_wcred_mode;
+	bool tx_word_mode;
+	bool rx_word_mode;
+	bool data_stamp;
+	bool xtra_write;
+	bool notify_disable;
+	bool intr_disable;
+	bool notify_early;
+	bool notify_os_intr_reg;
+
+	int lpid;
+	int pidr;		/* value from SPRN_PID, not linux pid */
+	int lnotify_lpid;
+	int lnotify_pid;
+	int lnotify_tid;
+	int pswid;
+	int rx_win_id;
+	int fault_win_id;
+	int tc_mode;
+
+	uint64_t irq_port;
+
+	enum vas_dma_type dma_type;
+	enum vas_notify_scope min_scope;
+	enum vas_notify_scope max_scope;
+	enum vas_notify_after_count notify_after_count;
+};
+
+#endif
-- 
2.7.4

[PATCH v3 02/10] Move GET_FIELD/SET_FIELD to vas.h

[Sukadev Bhattiprolu]

Move the GET_FIELD and SET_FIELD macros to vas.h as VAS and other
users of VAS, including NX-842 can use those macros.

There is a lot of related code between the VAS/NX kernel drivers
and skiboot. For consistency switch the order of parameters in
SET_FIELD to match the order in skiboot.

Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
---

Changelog[v3]
	- Fix order of parameters in nx-842 driver.
---
 arch/powerpc/include/asm/vas.h     | 8 +++++++-
 drivers/crypto/nx/nx-842-powernv.c | 7 ++++---
 drivers/crypto/nx/nx-842.h         | 5 -----
 3 files changed, 11 insertions(+), 9 deletions(-)

diff --git a/arch/powerpc/include/asm/vas.h b/arch/powerpc/include/asm/vas.h
index 6d35ce6..184eeb2 100644
--- a/arch/powerpc/include/asm/vas.h
+++ b/arch/powerpc/include/asm/vas.h
@@ -37,7 +37,13 @@ enum vas_cop_type {
 	VAS_COP_TYPE_MAX,
 };
 
+/*
+ * Get/Set bit fields
+ */
+#define GET_FIELD(m, v)		(((v) & (m)) >> MASK_LSH(m))
+#define MASK_LSH(m)		(__builtin_ffsl(m) - 1)
+#define SET_FIELD(m, v, val)	\
+		(((v) & ~(m)) | ((((typeof(v))(val)) << MASK_LSH(m)) & (m)))
 
 #endif	/* __KERNEL__ */
-
 #endif
diff --git a/drivers/crypto/nx/nx-842-powernv.c b/drivers/crypto/nx/nx-842-powernv.c
index 1710f80..3abb045 100644
--- a/drivers/crypto/nx/nx-842-powernv.c
+++ b/drivers/crypto/nx/nx-842-powernv.c
@@ -22,6 +22,7 @@
 
 #include <asm/prom.h>
 #include <asm/icswx.h>
+#include <asm/vas.h>
 
 MODULE_LICENSE("GPL");
 MODULE_AUTHOR("Dan Streetman <ddstreet@ieee.org>");
@@ -424,9 +425,9 @@ static int nx842_powernv_function(const unsigned char *in, unsigned int inlen,
 
 	/* set up CCW */
 	ccw = 0;
-	ccw = SET_FIELD(ccw, CCW_CT, nx842_ct);
-	ccw = SET_FIELD(ccw, CCW_CI_842, 0); /* use 0 for hw auto-selection */
-	ccw = SET_FIELD(ccw, CCW_FC_842, fc);
+	ccw = SET_FIELD(CCW_CT, ccw, nx842_ct);
+	ccw = SET_FIELD(CCW_CI_842, ccw, 0); /* use 0 for hw auto-selection */
+	ccw = SET_FIELD(CCW_FC_842, ccw, fc);
 
 	/* set up CRB's CSB addr */
 	csb_addr = nx842_get_pa(csb) & CRB_CSB_ADDRESS;
diff --git a/drivers/crypto/nx/nx-842.h b/drivers/crypto/nx/nx-842.h
index a4eee3b..30929bd 100644
--- a/drivers/crypto/nx/nx-842.h
+++ b/drivers/crypto/nx/nx-842.h
@@ -100,11 +100,6 @@ static inline unsigned long nx842_get_pa(void *addr)
 	return page_to_phys(vmalloc_to_page(addr)) + offset_in_page(addr);
 }
 
-/* Get/Set bit fields */
-#define MASK_LSH(m)		(__builtin_ffsl(m) - 1)
-#define GET_FIELD(v, m)		(((v) & (m)) >> MASK_LSH(m))
-#define SET_FIELD(v, m, val)	(((v) & ~(m)) | (((val) << MASK_LSH(m)) & (m)))
-
 /**
  * This provides the driver's constraints.  Different nx842 implementations
  * may have varying requirements.  The constraints are:
-- 
2.7.4

[PATCH v3 03/10] VAS: Define vas_init() and vas_exit()

[Sukadev Bhattiprolu]

Implement vas_init() and vas_exit() functions for a new VAS module.
This VAS module is essentially a library for other device drivers
and kernel users of the NX coprocessors like NX-842 and NX-GZIP.

Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
---
Changelog[v3]:
	- Zero vas_instances memory on allocation
	- [Haren Myneni] Fix description in Kconfig
Changelog[v2]:
	- Get HVWC, UWC and window address parameters from device tree.
---
 MAINTAINERS                     |   8 ++-
 arch/powerpc/include/asm/reg.h  |   1 +
 drivers/misc/Kconfig            |   1 +
 drivers/misc/Makefile           |   1 +
 drivers/misc/vas/Kconfig        |  21 ++++++
 drivers/misc/vas/Makefile       |   3 +
 drivers/misc/vas/vas-internal.h |   3 +
 drivers/misc/vas/vas-window.c   |  19 +++++
 drivers/misc/vas/vas.c          | 155 ++++++++++++++++++++++++++++++++++++++++
 9 files changed, 210 insertions(+), 2 deletions(-)
 create mode 100644 drivers/misc/vas/Kconfig
 create mode 100644 drivers/misc/vas/Makefile
 create mode 100644 drivers/misc/vas/vas-window.c
 create mode 100644 drivers/misc/vas/vas.c

diff --git a/MAINTAINERS b/MAINTAINERS
index 2a910c9..4037252 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -3673,8 +3673,6 @@ F:	arch/powerpc/platforms/powernv/pci-cxl.c
 F:	drivers/misc/cxl/
 F:	include/misc/cxl*
 F:	include/uapi/misc/cxl.h
-F:	Documentation/powerpc/cxl.txt
-F:	Documentation/ABI/testing/sysfs-class-cxl
 
 CXLFLASH (IBM Coherent Accelerator Processor Interface CAPI Flash) SCSI DRIVER
 M:	Manoj N. Kumar <manoj@linux.vnet.ibm.com>
@@ -3686,6 +3684,12 @@ F:	drivers/scsi/cxlflash/
 F:	include/uapi/scsi/cxlflash_ioctls.h
 F:	Documentation/powerpc/cxlflash.txt
 
+VAS (IBM Virtual Accelerator Switch) DRIVER
+M:	Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
+L:	linuxppc-dev@lists.ozlabs.org
+S:	Supported
+F:	drivers/misc/vas/
+
 STMMAC ETHERNET DRIVER
 M:	Giuseppe Cavallaro <peppe.cavallaro@st.com>
 M:	Alexandre Torgue <alexandre.torgue@st.com>
diff --git a/arch/powerpc/include/asm/reg.h b/arch/powerpc/include/asm/reg.h
index fc879fd..7a45ff7 100644
--- a/arch/powerpc/include/asm/reg.h
+++ b/arch/powerpc/include/asm/reg.h
@@ -1225,6 +1225,7 @@
 #define PVR_POWER8E	0x004B
 #define PVR_POWER8NVL	0x004C
 #define PVR_POWER8	0x004D
+#define PVR_POWER9	0x004E
 #define PVR_BE		0x0070
 #define PVR_PA6T	0x0090
 
diff --git a/drivers/misc/Kconfig b/drivers/misc/Kconfig
index c290990..97d652e 100644
--- a/drivers/misc/Kconfig
+++ b/drivers/misc/Kconfig
@@ -783,4 +783,5 @@ source "drivers/misc/mic/Kconfig"
 source "drivers/misc/genwqe/Kconfig"
 source "drivers/misc/echo/Kconfig"
 source "drivers/misc/cxl/Kconfig"
+source "drivers/misc/vas/Kconfig"
 endmenu
diff --git a/drivers/misc/Makefile b/drivers/misc/Makefile
index 7a3ea89..5201ffd 100644
--- a/drivers/misc/Makefile
+++ b/drivers/misc/Makefile
@@ -53,6 +53,7 @@ obj-$(CONFIG_GENWQE)		+= genwqe/
 obj-$(CONFIG_ECHO)		+= echo/
 obj-$(CONFIG_VEXPRESS_SYSCFG)	+= vexpress-syscfg.o
 obj-$(CONFIG_CXL_BASE)		+= cxl/
+obj-$(CONFIG_VAS)		+= vas/
 obj-$(CONFIG_PANEL)             += panel.o
 
 lkdtm-$(CONFIG_LKDTM)		+= lkdtm_core.o
diff --git a/drivers/misc/vas/Kconfig b/drivers/misc/vas/Kconfig
new file mode 100644
index 0000000..43cedda
--- /dev/null
+++ b/drivers/misc/vas/Kconfig
@@ -0,0 +1,21 @@
+#
+# IBM Virtual Accelarator Switchboard (VAS) compatible devices
+#depends on PPC_POWERNV && PCI_MSI && EEH
+#
+
+config VAS
+	tristate "Support for IBM Virtual Accelerator Switchboard (VAS)"
+	depends on PPC_POWERNV
+	default n
+	help
+	  Select this option to enable driver support for IBM Virtual
+	  Accelerator Switchboard (VAS).
+
+	  VAS allows accelerators in co processors like NX-842 to be
+	  directly available to a user process. This driver enables
+	  userspace programs to access these accelerators via device
+	  nodes like /dev/crypto/nx-gzip.
+
+	  VAS adapters are found in POWER9 based systems.
+
+	  If unsure, say N.
diff --git a/drivers/misc/vas/Makefile b/drivers/misc/vas/Makefile
new file mode 100644
index 0000000..7dd7139
--- /dev/null
+++ b/drivers/misc/vas/Makefile
@@ -0,0 +1,3 @@
+ccflags-y			:= $(call cc-disable-warning, unused-const-variable)
+ccflags-$(CONFIG_PPC_WERROR)	+= -Werror
+obj-$(CONFIG_VAS)		+= vas.o vas-window.o
diff --git a/drivers/misc/vas/vas-internal.h b/drivers/misc/vas/vas-internal.h
index ce48f14..15b62e0 100644
--- a/drivers/misc/vas/vas-internal.h
+++ b/drivers/misc/vas/vas-internal.h
@@ -389,4 +389,7 @@ struct vas_winctx {
 	enum vas_notify_after_count notify_after_count;
 };
 
+extern int vas_initialized;
+extern int vas_window_reset(struct vas_instance *vinst, int winid);
+extern struct vas_instance *find_vas_instance(int vasid);
 #endif
diff --git a/drivers/misc/vas/vas-window.c b/drivers/misc/vas/vas-window.c
new file mode 100644
index 0000000..468f3bf
--- /dev/null
+++ b/drivers/misc/vas/vas-window.c
@@ -0,0 +1,19 @@
+/*
+ * Copyright 2016 IBM Corp.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+
+#include <linux/types.h>
+#include <linux/mutex.h>
+#include <asm/vas.h>
+#include "vas-internal.h"
+
+/* stub for now */
+int vas_window_reset(struct vas_instance *vinst, int winid)
+{
+	return 0;
+}
diff --git a/drivers/misc/vas/vas.c b/drivers/misc/vas/vas.c
new file mode 100644
index 0000000..75e55f9
--- /dev/null
+++ b/drivers/misc/vas/vas.c
@@ -0,0 +1,155 @@
+/*
+ * Copyright 2016 IBM Corp.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+
+#include <linux/module.h>
+#include <linux/kernel.h>
+#include <linux/export.h>
+#include <linux/types.h>
+#include <linux/slab.h>
+#include <linux/of.h>
+#include <asm/vas.h>
+#include "vas-internal.h"
+
+int vas_initialized;
+int vas_num_instances;
+struct vas_instance *vas_instances;
+
+static void init_vas_chip(struct vas_instance *vinst)
+{
+	int i;
+
+	for (i = 0; i < VAS_MAX_WINDOWS_PER_CHIP; i++)
+		vas_window_reset(vinst, i);
+}
+
+static int init_vas_instance(struct device_node *dn,
+				struct vas_instance *vinst)
+{
+	int rc;
+	const __be32 *p;
+
+	ida_init(&vinst->ida);
+	mutex_init(&vinst->mutex);
+
+	p = of_get_property(dn, "vas-id", NULL);
+	if (!p) {
+		pr_err("VAS: NULL vas-id? %p\n", p);
+		return -ENODEV;
+	}
+
+	vinst->vas_id = of_read_number(p, 1);
+
+	rc = of_property_read_u64(dn, "hvwc-bar-start", &vinst->hvwc_bar_start);
+	if (rc)
+		return rc;
+
+	rc = of_property_read_u64(dn, "hvwc-bar-size", &vinst->hvwc_bar_len);
+	if (rc)
+		return rc;
+
+	rc = of_property_read_u64(dn, "uwc-bar-start", &vinst->uwc_bar_start);
+	if (rc)
+		return rc;
+
+	rc = of_property_read_u64(dn, "uwc-bar-size", &vinst->uwc_bar_len);
+	if (rc)
+		return rc;
+
+	rc = of_property_read_u64(dn, "window-base", &vinst->win_base_addr);
+	if (rc)
+		return rc;
+
+	rc = of_property_read_u64(dn, "window-shift", &vinst->win_id_shift);
+	if (rc)
+		return rc;
+
+	init_vas_chip(vinst);
+
+	return 0;
+}
+
+/*
+ * Although this is read/used multiple times, it is written to only
+ * during initialization.
+ */
+struct vas_instance *find_vas_instance(int vasid)
+{
+	int i;
+	struct vas_instance *vinst;
+
+	for (i = 0; i < vas_num_instances; i++) {
+		vinst = &vas_instances[i];
+		if (vinst->vas_id == vasid)
+			return vinst;
+	}
+	pr_err("VAS instance for vas-id %d not found\n", vasid);
+	WARN_ON_ONCE(1);
+	return NULL;
+}
+
+
+int vas_init(void)
+{
+	int rc;
+	struct device_node *dn;
+	struct vas_instance *vinst;
+
+	if (!pvr_version_is(PVR_POWER9))
+		return -ENODEV;
+
+	vas_num_instances = 0;
+	for_each_node_by_name(dn, "vas")
+		vas_num_instances++;
+
+	if (!vas_num_instances)
+		return -ENODEV;
+
+	vas_instances = kcalloc(vas_num_instances, sizeof(*vinst), GFP_KERNEL);
+	if (!vas_instances)
+		return -ENOMEM;
+
+	vinst = &vas_instances[0];
+	for_each_node_by_name(dn, "vas") {
+		rc = init_vas_instance(dn, vinst);
+		if (rc) {
+			pr_err("Error %d initializing VAS instance %ld\n", rc,
+					(vinst-&vas_instances[0]));
+			goto cleanup;
+		}
+		vinst++;
+	}
+
+	rc = -ENODEV;
+	if (vinst == &vas_instances[0]) {
+		/* Should not happen as we saw some above. */
+		pr_err("VAS: Did not find any VAS DT nodes now!\n");
+		goto cleanup;
+	}
+
+	pr_devel("VAS: Initialized %d instances\n", vas_num_instances);
+	vas_initialized = 1;
+
+	return 0;
+
+cleanup:
+	kfree(vas_instances);
+	return rc;
+}
+
+void vas_exit(void)
+{
+	vas_initialized = 0;
+	kfree(vas_instances);
+}
+
+module_init(vas_init);
+module_exit(vas_exit);
+MODULE_DESCRIPTION("IBM Virtual Accelerator Switchboard");
+MODULE_AUTHOR("Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>");
+MODULE_LICENSE("GPL");
-- 
2.7.4

[PATCH v3 04/10] VAS: Define helpers for access MMIO regions

[Sukadev Bhattiprolu]

Define some helper functions to access the MMIO regions. We use these
in a follow-on patches to read/write VAS hardware registers. These
helpers are also used to later issue 'paste' instructions to submit
requests to the NX hardware engines.

Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
---
Changelog [v3]:
	- Minor reorg/cleanup of map/unmap functions

Changelog [v2]:
	- Get HVWC, UWC and paste addresses from window->vinst (i.e DT)
	  rather than kernel macros.
---
 drivers/misc/vas/vas-window.c | 126 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 126 insertions(+)

diff --git a/drivers/misc/vas/vas-window.c b/drivers/misc/vas/vas-window.c
index 468f3bf..32dd1d0 100644
--- a/drivers/misc/vas/vas-window.c
+++ b/drivers/misc/vas/vas-window.c
@@ -9,9 +9,135 @@
 
 #include <linux/types.h>
 #include <linux/mutex.h>
+#include <linux/slab.h>
+#include <linux/io.h>
 #include <asm/vas.h>
 #include "vas-internal.h"
 
+/*
+ * Compute the paste address region for the window @window using the
+ * ->win_base_addr and ->win_id_shift we got from device tree.
+ */
+void compute_paste_address(struct vas_window *window, uint64_t *addr, int *len)
+{
+	uint64_t base, shift;
+	int winid;
+
+	base = window->vinst->win_base_addr;
+	shift = window->vinst->win_id_shift;
+	winid = window->winid;
+
+	*addr  = base + (winid << shift);
+	*len = PAGE_SIZE;
+
+	pr_debug("Txwin #%d: Paste addr 0x%llx\n", winid, *addr);
+}
+
+static inline void get_hvwc_mmio_bar(struct vas_window *window,
+			uint64_t *start, int *len)
+{
+	uint64_t pbaddr;
+
+	pbaddr = window->vinst->hvwc_bar_start;
+	*start = pbaddr + window->winid * VAS_HVWC_SIZE;
+	*len = VAS_HVWC_SIZE;
+}
+
+static inline void get_uwc_mmio_bar(struct vas_window *window,
+			uint64_t *start, int *len)
+{
+	uint64_t pbaddr;
+
+	pbaddr = window->vinst->uwc_bar_start;
+	*start = pbaddr + window->winid * VAS_UWC_SIZE;
+	*len = VAS_UWC_SIZE;
+}
+
+static void *map_mmio_region(char *name, uint64_t start, int len)
+{
+	void *map;
+
+	if (!request_mem_region(start, len, name)) {
+		pr_devel("%s(): request_mem_region(0x%llx, %d) failed\n",
+				__func__, start, len);
+		return NULL;
+	}
+
+	map = __ioremap(start, len, pgprot_val(pgprot_cached(__pgprot(0))));
+	if (!map) {
+		pr_devel("%s(): ioremap(0x%llx, %d) failed\n", __func__, start,
+				len);
+		return NULL;
+	}
+
+	return map;
+}
+
+/*
+ * Unmap the MMIO regions for a window.
+ */
+static void unmap_wc_paste_kaddr(struct vas_window *window)
+{
+	int len;
+	uint64_t busaddr_start;
+
+	if (window->paste_kaddr) {
+		iounmap(window->paste_kaddr);
+		compute_paste_address(window, &busaddr_start, &len);
+		release_mem_region((phys_addr_t)busaddr_start, len);
+		window->paste_kaddr = NULL;
+	}
+
+}
+
+static void unmap_wc_mmio_bars(struct vas_window *window)
+{
+	int len;
+	uint64_t busaddr_start;
+
+	unmap_wc_paste_kaddr(window);
+
+	if (window->hvwc_map) {
+		iounmap(window->hvwc_map);
+		get_hvwc_mmio_bar(window, &busaddr_start, &len);
+		release_mem_region((phys_addr_t)busaddr_start, len);
+		window->hvwc_map = NULL;
+	}
+
+	if (window->uwc_map) {
+		iounmap(window->uwc_map);
+		get_uwc_mmio_bar(window, &busaddr_start, &len);
+		release_mem_region((phys_addr_t)busaddr_start, len);
+		window->uwc_map = NULL;
+	}
+}
+
+/*
+ * Find the Hypervisor Window Context (HVWC) MMIO Base Address Region and the
+ * OS/User Window Context (UWC) MMIO Base Address Region for the given window.
+ * Map these bus addresses and save the mapped kernel addresses in @window.
+ */
+int map_wc_mmio_bars(struct vas_window *window)
+{
+	int len;
+	uint64_t start;
+
+	window->paste_kaddr = window->hvwc_map = window->uwc_map = NULL;
+
+	get_hvwc_mmio_bar(window, &start, &len);
+	window->hvwc_map = map_mmio_region("HVWCM_Window", start, len);
+
+	get_uwc_mmio_bar(window, &start, &len);
+	window->uwc_map = map_mmio_region("UWCM_Window", start, len);
+
+	if (!window->hvwc_map || !window->uwc_map) {
+		unmap_wc_mmio_bars(window);
+		return -1;
+	}
+
+	return 0;
+}
+
 /* stub for now */
 int vas_window_reset(struct vas_instance *vinst, int winid)
 {
-- 
2.7.4

[PATCH v3 05/10] VAS: Define helpers to init window context

[Sukadev Bhattiprolu]

Define helpers to initialize window context registers of the VAS
hardware. These will be used in follow-on patches when opening/closing
VAS windows.

Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
---
Changelog[v3]
	- Have caller, rather than init_xlate_regs() reset window regs
	  so we don't reset any settings caller may already have set.
	- Translation mode should be 0x3 (0b11) not 0x11.
	- Skip initilaizing read-only registers NX_UTIL and NX_UTIL_SE
	- Skip initializing adder registers from UWC - they are already
	  initialized from the HVWC.
	- Check winctx->user_win when setting translation registers
---
 drivers/misc/vas/vas-internal.h |  59 ++++++-
 drivers/misc/vas/vas-window.c   | 334 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 390 insertions(+), 3 deletions(-)

diff --git a/drivers/misc/vas/vas-internal.h b/drivers/misc/vas/vas-internal.h
index 15b62e0..8e721df 100644
--- a/drivers/misc/vas/vas-internal.h
+++ b/drivers/misc/vas/vas-internal.h
@@ -11,6 +11,7 @@
 #define VAS_INTERNAL_H
 #include <linux/atomic.h>
 #include <linux/idr.h>
+#include <linux/io.h>
 #include <asm/vas.h>
 
 #ifdef CONFIG_PPC_4K_PAGES
@@ -336,9 +337,6 @@ struct vas_window {
 	/* Feilds applicable only to receive windows */
 	enum vas_cop_type cop;
 	atomic_t num_txwins;
-
-	int32_t hwirq;
-	uint64_t irq_port;
 };
 
 /*
@@ -392,4 +390,59 @@ struct vas_winctx {
 extern int vas_initialized;
 extern int vas_window_reset(struct vas_instance *vinst, int winid);
 extern struct vas_instance *find_vas_instance(int vasid);
+
+/*
+ * VREG(x):
+ * Expand a register's short name (eg: LPID) into two parameters:
+ *	- the register's short name in string form ("LPID"), and
+ *	- the name of the macro (eg: VAS_LPID_OFFSET), defining the
+ *	  register's offset in the window context
+ */
+#define VREG_SFX(n, s)	__stringify(n), VAS_##n##s
+#define VREG(r)		VREG_SFX(r, _OFFSET)
+
+#ifndef vas_debug
+static inline void vas_log_write(struct vas_window *win, char *name,
+			void *regptr, uint64_t val)
+{
+	if (val)
+		pr_err("%swin #%d: %s reg %p, val 0x%llx\n",
+				win->tx_win ? "Tx" : "Rx", win->winid, name,
+				regptr, val);
+}
+
+#else	/* vas_debug */
+
+#define vas_log_write(win, name, reg, val)
+
+#endif	/* vas_debug */
+
+static inline void write_uwc_reg(struct vas_window *win, char *name,
+			int32_t reg, uint64_t val)
+{
+	void *regptr;
+
+	regptr = win->uwc_map + reg;
+	vas_log_write(win, name, regptr, val);
+
+	out_be64(regptr, val);
+}
+
+static inline void write_hvwc_reg(struct vas_window *win, char *name,
+			int32_t reg, uint64_t val)
+{
+	void *regptr;
+
+	regptr = win->hvwc_map + reg;
+	vas_log_write(win, name, regptr, val);
+
+	out_be64(regptr, val);
+}
+
+static inline uint64_t read_hvwc_reg(struct vas_window *win,
+			char *name __maybe_unused, int32_t reg)
+{
+	return in_be64(win->hvwc_map+reg);
+}
+
 #endif
diff --git a/drivers/misc/vas/vas-window.c b/drivers/misc/vas/vas-window.c
index 32dd1d0..edf5c9f 100644
--- a/drivers/misc/vas/vas-window.c
+++ b/drivers/misc/vas/vas-window.c
@@ -14,6 +14,8 @@
 #include <asm/vas.h>
 #include "vas-internal.h"
 
+static int fault_winid;
+
 /*
  * Compute the paste address region for the window @window using the
  * ->win_base_addr and ->win_id_shift we got from device tree.
@@ -138,6 +140,338 @@ int map_wc_mmio_bars(struct vas_window *window)
 	return 0;
 }
 
+/*
+ * Reset all valid registers in the HV and OS/User Window Contexts for
+ * the window identified by @window.
+ *
+ * NOTE: We cannot really use a for loop to reset window context. Not all
+ *	 offsets in a window context are valid registers and the valid
+ *	 registers are not sequential. And, we can only write to offsets
+ *	 with valid registers (or is that only in Simics?).
+ */
+void reset_window_regs(struct vas_window *window)
+{
+	write_hvwc_reg(window, VREG(LPID), 0ULL);
+	write_hvwc_reg(window, VREG(PID), 0ULL);
+	write_hvwc_reg(window, VREG(XLATE_MSR), 0ULL);
+	write_hvwc_reg(window, VREG(XLATE_LPCR), 0ULL);
+	write_hvwc_reg(window, VREG(XLATE_CTL), 0ULL);
+	write_hvwc_reg(window, VREG(AMR), 0ULL);
+	write_hvwc_reg(window, VREG(SEIDR), 0ULL);
+	write_hvwc_reg(window, VREG(FAULT_TX_WIN), 0ULL);
+	write_hvwc_reg(window, VREG(OSU_INTR_SRC_RA), 0ULL);
+	write_hvwc_reg(window, VREG(HV_INTR_SRC_RA), 0ULL);
+	write_hvwc_reg(window, VREG(PSWID), 0ULL);
+	write_hvwc_reg(window, VREG(SPARE1), 0ULL);
+	write_hvwc_reg(window, VREG(SPARE2), 0ULL);
+	write_hvwc_reg(window, VREG(SPARE3), 0ULL);
+	write_hvwc_reg(window, VREG(SPARE4), 0ULL);
+	write_hvwc_reg(window, VREG(SPARE5), 0ULL);
+	write_hvwc_reg(window, VREG(SPARE6), 0ULL);
+	write_hvwc_reg(window, VREG(LFIFO_BAR), 0ULL);
+	write_hvwc_reg(window, VREG(LDATA_STAMP_CTL), 0ULL);
+	write_hvwc_reg(window, VREG(LDMA_CACHE_CTL), 0ULL);
+	write_hvwc_reg(window, VREG(LRFIFO_PUSH), 0ULL);
+	write_hvwc_reg(window, VREG(CURR_MSG_COUNT), 0ULL);
+	write_hvwc_reg(window, VREG(LNOTIFY_AFTER_COUNT), 0ULL);
+	write_hvwc_reg(window, VREG(LRX_WCRED), 0ULL);
+	write_hvwc_reg(window, VREG(LRX_WCRED_ADDER), 0ULL);
+	write_hvwc_reg(window, VREG(TX_WCRED), 0ULL);
+	write_hvwc_reg(window, VREG(TX_WCRED_ADDER), 0ULL);
+	write_hvwc_reg(window, VREG(LFIFO_SIZE), 0ULL);
+	write_hvwc_reg(window, VREG(WINCTL), 0ULL);
+	write_hvwc_reg(window, VREG(WIN_STATUS), 0ULL);
+	write_hvwc_reg(window, VREG(WIN_CTX_CACHING_CTL), 0ULL);
+	write_hvwc_reg(window, VREG(TX_RSVD_BUF_COUNT), 0ULL);
+	write_hvwc_reg(window, VREG(LRFIFO_WIN_PTR), 0ULL);
+	write_hvwc_reg(window, VREG(LNOTIFY_CTL), 0ULL);
+	write_hvwc_reg(window, VREG(LNOTIFY_PID), 0ULL);
+	write_hvwc_reg(window, VREG(LNOTIFY_LPID), 0ULL);
+	write_hvwc_reg(window, VREG(LNOTIFY_TID), 0ULL);
+	write_hvwc_reg(window, VREG(LNOTIFY_SCOPE), 0ULL);
+	write_hvwc_reg(window, VREG(NX_UTIL_ADDER), 0ULL);
+
+	/* Skip read-only registers: NX_UTIL and NX_UTIL_SE */
+
+	/*
+	 * The send and receive window credit adder registers are also
+	 * accessible from HVWC and have been initialized above. We don't
+	 * need to initialize from the OS/User Window Context, so skip
+	 * following calls:
+	 *
+	 *	write_uwc_reg(window, VREG(TX_WCRED_ADDER), 0ULL);
+	 *	write_uwc_reg(window, VREG(LRX_WCRED_ADDER), 0ULL);
+	 */
+}
+
+/*
+ * Initialize window context registers related to Address Translation.
+ * These registers are common to send/receive windows although they
+ * differ for user/kernel windows. As we resolve the TODOs we may
+ * want to add fields to vas_winctx and move the initialization to
+ * init_vas_winctx_regs().
+ */
+static void init_xlate_regs(struct vas_window *window, bool user_win)
+{
+	uint64_t lpcr, msr, val;
+
+	msr = mfmsr();
+	WARN_ON_ONCE(!(msr & MSR_SF));
+	val = 0ULL;
+	if (user_win) {
+		val = SET_FIELD(VAS_XLATE_MSR_DR, val, true);
+		val = SET_FIELD(VAS_XLATE_MSR_TA, val, false);
+		val = SET_FIELD(VAS_XLATE_MSR_PR, val, true);
+		val = SET_FIELD(VAS_XLATE_MSR_US, val, false);
+		val = SET_FIELD(VAS_XLATE_MSR_HV, val, true);
+		val = SET_FIELD(VAS_XLATE_MSR_SF, val, true);
+		val = SET_FIELD(VAS_XLATE_MSR_UV, val, false);
+	} else {
+		val = SET_FIELD(VAS_XLATE_MSR_DR, val, false);
+		val = SET_FIELD(VAS_XLATE_MSR_TA, val, false);
+		val = SET_FIELD(VAS_XLATE_MSR_PR, val, msr & MSR_PR);
+		val = SET_FIELD(VAS_XLATE_MSR_US, val, false);
+		val = SET_FIELD(VAS_XLATE_MSR_HV, val, true);
+		val = SET_FIELD(VAS_XLATE_MSR_SF, val, true);
+		val = SET_FIELD(VAS_XLATE_MSR_UV, val, false);
+	}
+	write_hvwc_reg(window, VREG(XLATE_MSR), val);
+
+	lpcr = mfspr(SPRN_LPCR);
+	val = 0ULL;
+	/*
+	 * NOTE: From Section 5.7.6.1 Segment Lookaside Buffer of the
+	 *	 Power ISA, v2.07, Page size encoding is 0 = 4KB, 5 = 64KB.
+	 *
+	 * NOTE: From Section 1.3.1, Address Translation Context of the
+	 *	 Nest MMU Workbook, LPCR_SC should be 0 for Power9.
+	 */
+	val = SET_FIELD(VAS_XLATE_LPCR_PAGE_SIZE, val, 5);
+	val = SET_FIELD(VAS_XLATE_LPCR_ISL, val, lpcr & LPCR_ISL);
+	val = SET_FIELD(VAS_XLATE_LPCR_TC, val, lpcr & LPCR_TC);
+	val = SET_FIELD(VAS_XLATE_LPCR_SC, val, 0);
+	write_hvwc_reg(window, VREG(XLATE_LPCR), val);
+
+	/*
+	 * Section 1.3.1 (Address translation Context) of NMMU workbook.
+	 *	0b00	Hashed Page Table mode
+	 *	0b01	Reserved
+	 *	0b10	Radix on HPT - not supported in P9
+	 *	0b11	Radix on Radix (only mode supported in Linux on P9).
+	 */
+	val = 0ULL;
+	val = SET_FIELD(VAS_XLATE_MODE, val, 0x3);
+	write_hvwc_reg(window, VREG(XLATE_CTL), val);
+
+	/*
+	 * TODO: Can we mfspr(AMR) even for user windows?
+	 */
+	val = 0ULL;
+	val = SET_FIELD(VAS_AMR, val, mfspr(SPRN_AMR));
+	write_hvwc_reg(window, VREG(AMR), val);
+
+	/*
+	 * TODO: Assuming Secure Executable ID Register (SEIDR) is only used
+	 *	 in the ultravisor mode. Since MSR(UV) is 0 for now, set SEIDR
+	 *	 to 0 as well, although we should 'mfspr(SEIDR)' at some point.
+	 */
+	val = 0ULL;
+	val = SET_FIELD(VAS_SEIDR, val, 0);
+	write_hvwc_reg(window, VREG(SEIDR), val);
+}
+
+/*
+ * Initialize Reserved Send Buffer Count for the send window. It involves
+ * writing to the register, reading it back to confirm that the hardware
+ * has enough buffers to reserve. See section 1.3.1.2.1 of VAS workbook.
+ *
+ * Since we can only make a best-effort attempt to fulfill the request,
+ * we don't return any errors if we cannot.
+ *
+ * TODO: Reserved (aka dedicated) send buffers are not supported yet.
+ */
+static void init_rsvd_tx_buf_count(struct vas_window *txwin,
+				struct vas_winctx *winctx)
+{
+	write_hvwc_reg(txwin, VREG(TX_RSVD_BUF_COUNT), 0ULL);
+}
+
+/*
+ * Compute the log2() of the FIFO size expressed as kilobytes. It is intended
+ * to be used to initialize the Local FIFO Size Register defined in Section
+ * 3.14.25 of the VAS Workbook.
+ */
+static int map_fifo_size_to_reg(int fifo_size)
+{
+	int kb;
+	int map;
+
+	kb = fifo_size / 1024;
+	if (!kb)
+		kb = 1;
+
+	map = -1;
+	while (kb) {
+		kb >>= 1;
+		map++;
+	}
+
+	return map;
+}
+
+/*
+ * init_winctx_regs()
+ *	Initialize window context registers for a receive window.
+ *	Except for caching control and marking window open, the registers
+ *	are initialized in the order listed in Section 3.1.4 (Window Context
+ *	Cache Register Details) of the VAS workbook although they don't need
+ *	to be.
+ *
+ * Design note: For NX receive windows, NX allocates the FIFO buffer in OPAL
+ *	(so that it can get a large contiguous area) and passes that buffer
+ *	to kernel via device tree. We now write that buffer address to the
+ *	FIFO BAR. Would it make sense to do this all in OPAL? i.e have OPAL
+ *	write the per-chip RX FIFO addresses to the windows during boot-up
+ *	as a one-time task? That could work for NX but what about other
+ *	receivers?  Let the receivers tell us the rx-fifo buffers for now.
+ */
+int init_winctx_regs(struct vas_window *window, struct vas_winctx *winctx)
+{
+	uint64_t val;
+	int fifo_size;
+
+	reset_window_regs(window);
+
+	val = 0ULL;
+	val = SET_FIELD(VAS_LPID, val, winctx->lpid);
+	write_hvwc_reg(window, VREG(LPID), val);
+
+	val = 0ULL;
+	val = SET_FIELD(VAS_PID_ID, val, winctx->pidr);
+	write_hvwc_reg(window, VREG(PID), val);
+
+	init_xlate_regs(window, winctx->user_win);
+
+	val = 0ULL;
+	val = SET_FIELD(VAS_FAULT_TX_WIN, val, fault_winid);
+	write_hvwc_reg(window, VREG(FAULT_TX_WIN), val);
+
+	/* In PowerNV, interrupts go to HV. */
+	write_hvwc_reg(window, VREG(OSU_INTR_SRC_RA), 0ULL);
+
+	val = 0ULL;
+	val = SET_FIELD(VAS_HV_INTR_SRC_RA, val, winctx->irq_port);
+	write_hvwc_reg(window, VREG(HV_INTR_SRC_RA), val);
+
+	val = 0ULL;
+	val = SET_FIELD(VAS_PSWID_EA_HANDLE, val, winctx->pswid);
+	write_hvwc_reg(window, VREG(PSWID), val);
+
+	write_hvwc_reg(window, VREG(SPARE1), 0ULL);
+	write_hvwc_reg(window, VREG(SPARE2), 0ULL);
+	write_hvwc_reg(window, VREG(SPARE3), 0ULL);
+
+	/* See also: Design note in function header */
+	val = 0ULL;
+	val = SET_FIELD(VAS_LFIFO_BAR, val, __pa(winctx->rx_fifo));
+	val = SET_FIELD(VAS_PAGE_MIGRATION_SELECT, val, 0);
+	write_hvwc_reg(window, VREG(LFIFO_BAR), val);
+
+	val = 0ULL;
+	val = SET_FIELD(VAS_LDATA_STAMP, val, winctx->data_stamp);
+	write_hvwc_reg(window, VREG(LDATA_STAMP_CTL), val);
+
+	val = 0ULL;
+	val = SET_FIELD(VAS_LDMA_TYPE, val, winctx->dma_type);
+	write_hvwc_reg(window, VREG(LDMA_CACHE_CTL), val);
+
+	write_hvwc_reg(window, VREG(LRFIFO_PUSH), 0ULL);
+	write_hvwc_reg(window, VREG(CURR_MSG_COUNT), 0ULL);
+	write_hvwc_reg(window, VREG(LNOTIFY_AFTER_COUNT), 0ULL);
+
+	val = 0ULL;
+	val = SET_FIELD(VAS_LRX_WCRED, val, winctx->wcreds_max);
+	write_hvwc_reg(window, VREG(LRX_WCRED), val);
+
+	val = 0ULL;
+	val = SET_FIELD(VAS_TX_WCRED, val, winctx->wcreds_max);
+	write_hvwc_reg(window, VREG(TX_WCRED), val);
+
+	write_hvwc_reg(window, VREG(LRX_WCRED_ADDER), 0ULL);
+	write_hvwc_reg(window, VREG(TX_WCRED_ADDER), 0ULL);
+
+	val = 0ULL;
+	fifo_size = winctx->rx_fifo_size;
+	val = SET_FIELD(VAS_LFIFO_SIZE, val, map_fifo_size_to_reg(fifo_size));
+	write_hvwc_reg(window, VREG(LFIFO_SIZE), val);
+
+	/* Update window control and caching control registers last so
+	 * we mark the window open only after fully initializing it and
+	 * pushing context to cache.
+	 */
+
+	write_hvwc_reg(window, VREG(WIN_STATUS), 0ULL);
+
+	init_rsvd_tx_buf_count(window, winctx);
+
+	/* for a send window, point to the matching receive window */
+	val = 0ULL;
+	val = SET_FIELD(VAS_LRX_WIN_ID, val, winctx->rx_win_id);
+	write_hvwc_reg(window, VREG(LRFIFO_WIN_PTR), val);
+
+	write_hvwc_reg(window, VREG(SPARE4), 0ULL);
+
+	val = 0ULL;
+	val = SET_FIELD(VAS_NOTIFY_DISABLE, val, winctx->notify_disable);
+	val = SET_FIELD(VAS_INTR_DISABLE, val, winctx->intr_disable);
+	val = SET_FIELD(VAS_NOTIFY_EARLY, val, winctx->notify_early);
+	val = SET_FIELD(VAS_NOTIFY_OSU_INTR, val, winctx->notify_os_intr_reg);
+	write_hvwc_reg(window, VREG(LNOTIFY_CTL), val);
+
+	val = 0ULL;
+	val = SET_FIELD(VAS_LNOTIFY_PID, val, winctx->lnotify_pid);
+	write_hvwc_reg(window, VREG(LNOTIFY_PID), val);
+
+	val = 0ULL;
+	val = SET_FIELD(VAS_LNOTIFY_LPID, val, winctx->lnotify_lpid);
+	write_hvwc_reg(window, VREG(LNOTIFY_LPID), val);
+
+	val = 0ULL;
+	val = SET_FIELD(VAS_LNOTIFY_TID, val, winctx->lnotify_tid);
+	write_hvwc_reg(window, VREG(LNOTIFY_TID), val);
+
+	val = 0ULL;
+	val = SET_FIELD(VAS_LNOTIFY_MIN_SCOPE, val, winctx->min_scope);
+	val = SET_FIELD(VAS_LNOTIFY_MAX_SCOPE, val, winctx->max_scope);
+	write_hvwc_reg(window, VREG(LNOTIFY_SCOPE), val);
+
+	/* Skip read-only registers NX_UTIL and NX_UTIL_SE */
+
+	write_hvwc_reg(window, VREG(SPARE5), 0ULL);
+	write_hvwc_reg(window, VREG(NX_UTIL_ADDER), 0ULL);
+	write_hvwc_reg(window, VREG(SPARE6), 0ULL);
+
+	/* Finally, push window context to memory and... */
+	val = 0ULL;
+	val = SET_FIELD(VAS_PUSH_TO_MEM, val, 1);
+	write_hvwc_reg(window, VREG(WIN_CTX_CACHING_CTL), val);
+
+	/* ... mark the window open for business */
+	val = 0ULL;
+	val = SET_FIELD(VAS_WINCTL_REJ_NO_CREDIT, val, winctx->rej_no_credit);
+	val = SET_FIELD(VAS_WINCTL_PIN, val, winctx->pin_win);
+	val = SET_FIELD(VAS_WINCTL_TX_WCRED_MODE, val, winctx->tx_wcred_mode);
+	val = SET_FIELD(VAS_WINCTL_RX_WCRED_MODE, val, winctx->rx_wcred_mode);
+	val = SET_FIELD(VAS_WINCTL_TX_WORD_MODE, val, winctx->tx_word_mode);
+	val = SET_FIELD(VAS_WINCTL_RX_WORD_MODE, val, winctx->rx_word_mode);
+	val = SET_FIELD(VAS_WINCTL_FAULT_WIN, val, winctx->fault_win);
+	val = SET_FIELD(VAS_WINCTL_NX_WIN, val, winctx->nx_win);
+	val = SET_FIELD(VAS_WINCTL_OPEN, val, 1);
+	write_hvwc_reg(window, VREG(WINCTL), val);
+
+	return 0;
+}
+
 /* stub for now */
 int vas_window_reset(struct vas_instance *vinst, int winid)
 {
-- 
2.7.4

[PATCH v3 06/10] VAS: Define helpers to alloc/free windows

[Sukadev Bhattiprolu]

Define helpers to allocate/free VAS window objects. These will
be used in follow-on patches when opening/closing windows.

Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
---
 drivers/misc/vas/vas-window.c | 74 +++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 72 insertions(+), 2 deletions(-)

diff --git a/drivers/misc/vas/vas-window.c b/drivers/misc/vas/vas-window.c
index edf5c9f..9233bf5 100644
--- a/drivers/misc/vas/vas-window.c
+++ b/drivers/misc/vas/vas-window.c
@@ -119,7 +119,7 @@ static void unmap_wc_mmio_bars(struct vas_window *window)
  * OS/User Window Context (UWC) MMIO Base Address Region for the given window.
  * Map these bus addresses and save the mapped kernel addresses in @window.
  */
-int map_wc_mmio_bars(struct vas_window *window)
+static int map_wc_mmio_bars(struct vas_window *window)
 {
 	int len;
 	uint64_t start;
@@ -472,8 +472,78 @@ int init_winctx_regs(struct vas_window *window, struct vas_winctx *winctx)
 	return 0;
 }
 
-/* stub for now */
+DEFINE_SPINLOCK(vas_ida_lock);
+
+void vas_release_window_id(struct ida *ida, int winid)
+{
+	spin_lock(&vas_ida_lock);
+	ida_remove(ida, winid);
+	spin_unlock(&vas_ida_lock);
+}
+
+int vas_assign_window_id(struct ida *ida)
+{
+	int rc, winid;
+
+	rc = ida_pre_get(ida, GFP_KERNEL);
+	if (!rc)
+		return -EAGAIN;
+
+	spin_lock(&vas_ida_lock);
+	rc = ida_get_new_above(ida, 0, &winid);
+	spin_unlock(&vas_ida_lock);
+
+	if (rc)
+		return rc;
+
+	if (winid > VAS_MAX_WINDOWS_PER_CHIP) {
+		pr_err("VAS: Too many (%d) open windows\n", winid);
+		vas_release_window_id(ida, winid);
+		return -EAGAIN;
+	}
+
+	return winid;
+}
+
+static void vas_window_free(struct vas_window *window)
+{
+	unmap_wc_mmio_bars(window);
+	kfree(window->paste_addr_name);
+	kfree(window);
+}
+
+static struct vas_window *vas_window_alloc(struct vas_instance *vinst, int id)
+{
+	struct vas_window *window;
+
+	window = kzalloc(sizeof(*window), GFP_KERNEL);
+	if (!window)
+		return NULL;
+
+	window->vinst = vinst;
+	window->winid = id;
+
+	if (map_wc_mmio_bars(window))
+		goto out_free;
+
+	return window;
+
+out_free:
+	kfree(window);
+	return NULL;
+}
+
 int vas_window_reset(struct vas_instance *vinst, int winid)
 {
+	struct vas_window *window;
+
+	window = vas_window_alloc(vinst, winid);
+	if (!window)
+		return -ENOMEM;
+
+	reset_window_regs(window);
+
+	vas_window_free(window);
+
 	return 0;
 }
-- 
2.7.4

[PATCH v3 07/10] VAS: Define vas_rx_win_open() interface

[Sukadev Bhattiprolu]

Define the vas_rx_win_open() interface. This interface is intended to be
used by the Nest Accelerator (NX) driver(s) to setup receive windows for
one or more NX engines (which implement compression/encryption algorithms
in the hardware).

Follow-on patches will provide an interface to close the window and to open
a send window that kenrel subsystems can use to access the NX engines.

The interface to open a receive window is expected to be invoked for each
instance of VAS in the system.

Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
---

Changelog[v3]:
	- Fault receive windows must enable interrupts and disable
	  notifications. NX Windows are opposite.
	- Use macros rather than enum for threshold-control mode
	- Ignore irq_ports for in-kernel windows. They are needed for
	  user space windows and will be added later
---
 arch/powerpc/include/asm/vas.h  |  46 +++++++++
 drivers/misc/vas/vas-internal.h |  11 +++
 drivers/misc/vas/vas-window.c   | 200 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 257 insertions(+)

diff --git a/arch/powerpc/include/asm/vas.h b/arch/powerpc/include/asm/vas.h
index 184eeb2..d49b05b 100644
--- a/arch/powerpc/include/asm/vas.h
+++ b/arch/powerpc/include/asm/vas.h
@@ -38,6 +38,51 @@ enum vas_cop_type {
 };
 
 /*
+ * Receive window attributes specified by the (in-kernel) owner of window.
+ */
+struct vas_rx_win_attr {
+	void *rx_fifo;
+	int rx_fifo_size;
+	int wcreds_max;
+
+	bool pin_win;
+	bool rej_no_credit;
+	bool tx_wcred_mode;
+	bool rx_wcred_mode;
+	bool tx_win_ord_mode;
+	bool rx_win_ord_mode;
+	bool data_stamp;
+	bool nx_win;
+	bool fault_win;
+	bool notify_disable;
+	bool intr_disable;
+	bool notify_early;
+
+	int lnotify_lpid;
+	int lnotify_pid;
+	int lnotify_tid;
+	int pswid;
+
+	int tc_mode;
+};
+
+/*
+ * Helper to initialize receive window attributes to defaults for an
+ * NX window.
+ */
+extern void vas_init_rx_win_attr(struct vas_rx_win_attr *rxattr,
+				enum vas_cop_type cop);
+
+/*
+ * Open a VAS receive window for the instance of VAS identified by @vasid
+ * Use @attr to initialize the attributes of the window.
+ *
+ * Return a handle to the window or ERR_PTR() on error.
+ */
+extern struct vas_window *vas_rx_win_open(int vasid, enum vas_cop_type cop,
+			struct vas_rx_win_attr *attr);
+
+/*
  * Get/Set bit fields
  */
 #define GET_FIELD(m, v)		(((v) & (m)) >> MASK_LSH(m))
@@ -46,4 +91,5 @@ enum vas_cop_type {
 		(((v) & ~(m)) | ((((typeof(v))(val)) << MASK_LSH(m)) & (m)))
 
 #endif	/* __KERNEL__ */
+
 #endif
diff --git a/drivers/misc/vas/vas-internal.h b/drivers/misc/vas/vas-internal.h
index 8e721df..1e5c94b 100644
--- a/drivers/misc/vas/vas-internal.h
+++ b/drivers/misc/vas/vas-internal.h
@@ -402,6 +402,16 @@ extern struct vas_instance *find_vas_instance(int vasid);
 #define VREG(r)		VREG_SFX(r, _OFFSET)
 
 #ifndef vas_debug
+static inline void dump_rx_win_attr(struct vas_rx_win_attr *attr)
+{
+	pr_err("VAS: fault %d, notify %d, intr %d early %d\n",
+			attr->fault_win, attr->notify_disable,
+			attr->intr_disable, attr->notify_early);
+
+	pr_err("VAS: rx_fifo_size %d, max value %d\n",
+				attr->rx_fifo_size, VAS_RX_FIFO_SIZE_MAX);
+}
+
 static inline void vas_log_write(struct vas_window *win, char *name,
 			void *regptr, uint64_t val)
 {
@@ -414,6 +424,7 @@ static inline void vas_log_write(struct vas_window *win, char *name,
 #else	/* vas_debug */
 
 #define vas_log_write(win, name, reg, val)
+#define dump_rx_win_attr(attr)
 
 #endif	/* vas_debug */
 
diff --git a/drivers/misc/vas/vas-window.c b/drivers/misc/vas/vas-window.c
index 9233bf5..605a2fa 100644
--- a/drivers/misc/vas/vas-window.c
+++ b/drivers/misc/vas/vas-window.c
@@ -547,3 +547,203 @@ int vas_window_reset(struct vas_instance *vinst, int winid)
 
 	return 0;
 }
+
+struct vas_window *get_vinstance_rxwin(struct vas_instance *vinst,
+			enum vas_cop_type cop)
+{
+	struct vas_window *rxwin;
+
+	mutex_lock(&vinst->mutex);
+
+	rxwin = vinst->rxwin[cop];
+	if (rxwin)
+		atomic_inc(&rxwin->num_txwins);
+
+	mutex_unlock(&vinst->mutex);
+
+	return rxwin;
+}
+
+static void set_vinstance_rxwin(struct vas_instance *vinst,
+			enum vas_cop_type cop, struct vas_window *window)
+{
+	mutex_lock(&vinst->mutex);
+
+	/*
+	 * There should only be one receive window for a coprocessor type.
+	 */
+	WARN_ON_ONCE(vinst->rxwin[cop]);
+	vinst->rxwin[cop] = window;
+
+	mutex_unlock(&vinst->mutex);
+}
+
+static void init_winctx_for_rxwin(struct vas_window *rxwin,
+			struct vas_rx_win_attr *rxattr,
+			struct vas_winctx *winctx)
+{
+	/*
+	 * We first zero (memset()) all fields and only set non-zero fields.
+	 * Following fields are 0/false but maybe deserve a comment:
+	 *
+	 *	->user_win		No support for user Rx windows yet
+	 *	->notify_os_intr_reg	In powerNV, send intrs to HV
+	 *	->notify_disable	False for NX windows
+	 *	->intr_disable		False for Fault Windows
+	 *	->xtra_write		False for NX windows
+	 *	->notify_early		NA for NX windows
+	 *	->rsvd_txbuf_count	NA for Rx windows
+	 *	->lpid, ->pid, ->tid	NA for Rx windows
+	 */
+
+	memset(winctx, 0, sizeof(struct vas_winctx));
+
+	winctx->rx_fifo = rxattr->rx_fifo;
+	winctx->rx_fifo_size = rxattr->rx_fifo_size;
+	winctx->wcreds_max = rxattr->wcreds_max ?: VAS_WCREDS_DEFAULT;
+	winctx->pin_win = rxattr->pin_win;
+
+	winctx->nx_win = rxattr->nx_win;
+	winctx->fault_win = rxattr->fault_win;
+	winctx->rx_word_mode = rxattr->rx_win_ord_mode;
+	winctx->tx_word_mode = rxattr->tx_win_ord_mode;
+	winctx->rx_wcred_mode = rxattr->rx_wcred_mode;
+	winctx->tx_wcred_mode = rxattr->tx_wcred_mode;
+
+	winctx->fault_win_id = fault_winid;
+
+	if (winctx->nx_win) {
+		winctx->data_stamp = true;
+		winctx->intr_disable = true;
+		winctx->pin_win = true;
+
+		WARN_ON_ONCE(winctx->fault_win);
+		WARN_ON_ONCE(!winctx->rx_word_mode);
+		WARN_ON_ONCE(!winctx->tx_word_mode);
+		WARN_ON_ONCE(winctx->notify_after_count);
+	} else if (winctx->fault_win) {
+		winctx->notify_disable = true;
+	}
+
+	winctx->lnotify_lpid = rxattr->lnotify_lpid;
+	winctx->lnotify_pid = rxattr->lnotify_pid;
+	winctx->lnotify_tid = rxattr->lnotify_tid;
+	winctx->pswid = rxattr->pswid;
+	winctx->dma_type = VAS_DMA_TYPE_INJECT;
+	winctx->tc_mode = rxattr->tc_mode;
+
+	winctx->min_scope = VAS_SCOPE_LOCAL;
+	winctx->max_scope = VAS_SCOPE_VECTORED_GROUP;
+}
+
+static bool rx_win_args_valid(enum vas_cop_type cop,
+			struct vas_rx_win_attr *attr)
+{
+	dump_rx_win_attr(attr);
+
+	if (cop >= VAS_COP_TYPE_MAX)
+		return false;
+
+	if (attr->rx_fifo_size > VAS_RX_FIFO_SIZE_MAX)
+		return false;
+
+	if (attr->nx_win) {
+		/* cannot be both fault and nx */
+		if (attr->fault_win)
+			return false;
+		/*
+		 * Section 3.1.4.32: NX Windows must not disable notification,
+		 *	and must not enable interrupts or early notification.
+		 */
+		if (attr->notify_disable || !attr->intr_disable ||
+				attr->notify_early)
+			return false;
+	} else if (attr->fault_win) {
+		/*
+		 * Section 3.1.4.32: Fault windows must disable notification
+		 *	but not interrupts.
+		 */
+		if (!attr->notify_disable || attr->intr_disable)
+			return false;
+	} else {
+		/* Rx window must be either NX or Fault window for now.  */
+		return false;
+	}
+
+	return true;
+}
+
+void vas_init_rx_win_attr(struct vas_rx_win_attr *rxattr, enum vas_cop_type cop)
+{
+	memset(rxattr, 0, sizeof(*rxattr));
+
+	if (cop == VAS_COP_TYPE_842 || cop == VAS_COP_TYPE_842_HIPRI) {
+		rxattr->pin_win = true;
+		rxattr->nx_win = true;
+		rxattr->fault_win = false;
+		rxattr->intr_disable = true;
+		rxattr->rx_wcred_mode = true;
+		rxattr->tx_wcred_mode = true;
+		rxattr->rx_win_ord_mode = true;
+		rxattr->tx_win_ord_mode = true;
+	} else if (cop == VAS_COP_TYPE_FAULT) {
+		rxattr->pin_win = true;
+		rxattr->fault_win = true;
+		rxattr->notify_disable = true;
+		rxattr->rx_wcred_mode = true;
+		rxattr->tx_wcred_mode = true;
+		rxattr->rx_win_ord_mode = true;
+		rxattr->tx_win_ord_mode = true;
+	}
+}
+
+struct vas_window *vas_rx_win_open(int vasid, enum vas_cop_type cop,
+			struct vas_rx_win_attr *rxattr)
+{
+	int rc, winid;
+	struct vas_instance *vinst;
+	struct vas_window *rxwin;
+	struct vas_winctx winctx;
+
+	if (!vas_initialized)
+		return ERR_PTR(-EAGAIN);
+
+	if (!rx_win_args_valid(cop, rxattr))
+		return ERR_PTR(-EINVAL);
+
+	vinst = find_vas_instance(vasid);
+	if (!vinst) {
+		pr_devel("VAS: vasid %d not found!\n", vasid);
+		return ERR_PTR(-EINVAL);
+	}
+	pr_devel("VAS: Found instance %d\n", vasid);
+
+	winid = vas_assign_window_id(&vinst->ida);
+	if (winid < 0)
+		return ERR_PTR(winid);
+
+	rc = -ENOMEM;
+	rxwin = vas_window_alloc(vinst, winid);
+	if (!rxwin) {
+		pr_devel("VAS: Unable to allocate memory for Rx window\n");
+		goto release_winid;
+	}
+
+	rxwin->tx_win = false;
+	rxwin->cop = cop;
+
+	init_winctx_for_rxwin(rxwin, rxattr, &winctx);
+	rxwin->nx_win = winctx.nx_win;
+	init_winctx_regs(rxwin, &winctx);
+
+	set_vinstance_rxwin(vinst, cop, rxwin);
+
+	if (winctx.fault_win)
+		fault_winid = winid;
+
+	return rxwin;
+
+release_winid:
+	vas_release_window_id(&vinst->ida, rxwin->winid);
+	return ERR_PTR(rc);
+}
-- 
2.7.4

[PATCH v3 08/10] VAS: Define vas_win_close() interface

[Sukadev Bhattiprolu]

Define the vas_win_close() interface which should be used to close a
send or receive windows.

While the hardware configurations required to open send and receive windows
differ, the configuration to close a window is the same for both. So we use
a single interface to close the window.

Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
---
Changelog[v3]:
	- Fix order of parameters in GET_FIELD().
	- Update references and sequence for closing/quiescing a window.
---
 arch/powerpc/include/asm/vas.h |   7 +++
 drivers/misc/vas/vas-window.c  | 120 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 127 insertions(+)

diff --git a/arch/powerpc/include/asm/vas.h b/arch/powerpc/include/asm/vas.h
index d49b05b..fc00249 100644
--- a/arch/powerpc/include/asm/vas.h
+++ b/arch/powerpc/include/asm/vas.h
@@ -83,6 +83,13 @@ extern struct vas_window *vas_rx_win_open(int vasid, enum vas_cop_type cop,
 			struct vas_rx_win_attr *attr);
 
 /*
+ * Close the send or receive window identified by @win. For receive windows
+ * return -EAGAIN if there are active send windows attached to this receive
+ * window.
+ */
+int vas_win_close(struct vas_window *win);
+
+/*
  * Get/Set bit fields
  */
 #define GET_FIELD(m, v)		(((v) & (m)) >> MASK_LSH(m))
diff --git a/drivers/misc/vas/vas-window.c b/drivers/misc/vas/vas-window.c
index 605a2fa..40e4f7d 100644
--- a/drivers/misc/vas/vas-window.c
+++ b/drivers/misc/vas/vas-window.c
@@ -548,6 +548,14 @@ int vas_window_reset(struct vas_instance *vinst, int winid)
 	return 0;
 }
 
+static void put_rx_win(struct vas_window *rxwin)
+{
+	/* Better not be a send window! */
+	WARN_ON_ONCE(rxwin->tx_win);
+
+	atomic_dec(&rxwin->num_txwins);
+}
+
 struct vas_window *get_vinstance_rxwin(struct vas_instance *vinst,
 			enum vas_cop_type cop)
 {
@@ -747,3 +755,115 @@ struct vas_window *vas_rx_win_open(int vasid, enum vas_cop_type cop,
 	vas_release_window_id(&vinst->ida, rxwin->winid);
 	return ERR_PTR(rc);
 }
+
+static void poll_window_busy_state(struct vas_window *window)
+{
+	int busy;
+	uint64_t val;
+
+retry:
+	/*
+	 * Poll Window Busy flag
+	 */
+	val = read_hvwc_reg(window, VREG(WIN_STATUS));
+	busy = GET_FIELD(VAS_WIN_BUSY, val);
+	if (busy) {
+		val = 0;
+		schedule_timeout(2000);
+		goto retry;
+	}
+}
+
+static void poll_window_credits_return(struct vas_window *window)
+{
+	int credits;
+	uint64_t val;
+
+retry:
+	/*
+	 * Poll Window Credits
+	 */
+	if (window->tx_win) {
+		val = read_hvwc_reg(window, VREG(TX_WCRED));
+		credits = GET_FIELD(VAS_TX_WCRED, val);
+	} else {
+		val = read_hvwc_reg(window, VREG(LRX_WCRED));
+		credits = GET_FIELD(VAS_LRX_WCRED, val);
+	}
+
+	if (credits != VAS_WCREDS_DEFAULT) {
+		val = 0;
+		schedule_timeout(2000);
+		goto retry;
+	}
+}
+
+static void poll_window_castout(struct vas_window *window)
+{
+	int cached;
+	uint64_t val;
+
+	/* Cast window context out of the cache */
+retry:
+	val = read_hvwc_reg(window, VREG(WIN_CTX_CACHING_CTL));
+	cached = GET_FIELD(VAS_WIN_CACHE_STATUS, val);
+	if (cached) {
+		val = 0ULL;
+		val = SET_FIELD(VAS_CASTOUT_REQ, val, 1);
+		val = SET_FIELD(VAS_PUSH_TO_MEM, val, 0);
+		write_hvwc_reg(window, VREG(WIN_CTX_CACHING_CTL), val);
+
+		schedule_timeout(2000);
+		goto retry;
+	}
+}
+
+/*
+ * Close a window.
+ *
+ * See Section 1.12.1 of VAS workbook v1.05 for details on closing window:
+ *	- disable new paste operations (unmap paste address)
+ *	- Poll for the "Window Busy" bit to be cleared
+ *	- Clear the Open/Enable bit for the Window.
+ *	- Poll for return of window Credits (implies FIFO empty for Rx win?)
+ *	- Unpin and cast window context out of cache
+ *
+ * Besides the hardware, kernel has some bookkeeping of course.
+ */
+int vas_win_close(struct vas_window *window)
+{
+	uint64_t val;
+
+	if (!window)
+		return 0;
+
+	if (!window->tx_win && atomic_read(&window->num_txwins) != 0) {
+		pr_devel("VAS: Attempting to close an active Rx window!\n");
+		WARN_ON_ONCE(1);
+		return -EAGAIN;
+	}
+
+	unmap_wc_paste_kaddr(window);
+
+	poll_window_busy_state(window);
+
+	/* Unpin window from cache and close it */
+	val = read_hvwc_reg(window, VREG(WINCTL));
+	val = SET_FIELD(VAS_WINCTL_PIN, val, 0);
+	val = SET_FIELD(VAS_WINCTL_OPEN, val, 0);
+	write_hvwc_reg(window, VREG(WINCTL), val);
+
+	poll_window_credits_return(window);
+
+	poll_window_castout(window);
+
+	/* if send window, drop reference to matching receive window */
+	if (window->tx_win)
+		put_rx_win(window->rxwin);
+
+	vas_release_window_id(&window->vinst->ida, window->winid);
+
+	vas_window_free(window);
+
+	return 0;
+}
-- 
2.7.4

[PATCH v3 09/10] VAS: Define vas_tx_win_open()

[Sukadev Bhattiprolu]

Define an interface to open a VAS send window. This interface is
intended to be used the Nest Accelerator (NX) driver(s) to open
a send window and use it to submit compression/encryption requests
to a VAS receive window.

The receive window, identified by the [node, chip, cop] parameters,
must already be open in VAS (i.e connected to an NX engine).

Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>

---
Changelog [v3]:
- Distinguish between hardware PID (SPRN_PID) and Linux pid.
- Use macros rather than enum for threshold-control mode
- Set the pid of send window from attr (needed for user space
  send windows).
- Ignore irq port setting for now. They are needed for user space
  windows and will be added later
---
 arch/powerpc/include/asm/vas.h |  42 +++++++++++
 drivers/misc/vas/vas-window.c  | 157 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 199 insertions(+)

diff --git a/arch/powerpc/include/asm/vas.h b/arch/powerpc/include/asm/vas.h
index fc00249..ff8da98 100644
--- a/arch/powerpc/include/asm/vas.h
+++ b/arch/powerpc/include/asm/vas.h
@@ -67,6 +67,29 @@ struct vas_rx_win_attr {
 };
 
 /*
+ * Window attributes specified by the in-kernel owner of a send window.
+ */
+struct vas_tx_win_attr {
+	enum vas_cop_type cop;
+	int wcreds_max;
+	int lpid;
+	int pidr;		/* hardware PID (from SPRN_PID) */
+	int pid;		/* linux process id */
+	int pswid;
+	int rsvd_txbuf_count;
+	int tc_mode;
+
+	bool user_win;
+	bool pin_win;
+	bool rej_no_credit;
+	bool rsvd_txbuf_enable;
+	bool tx_wcred_mode;
+	bool rx_wcred_mode;
+	bool tx_win_ord_mode;
+	bool rx_win_ord_mode;
+};
+
+/*
  * Helper to initialize receive window attributes to defaults for an
  * NX window.
  */
@@ -83,6 +106,25 @@ extern struct vas_window *vas_rx_win_open(int vasid, enum vas_cop_type cop,
 			struct vas_rx_win_attr *attr);
 
 /*
+ * Helper to initialize send window attributes to defaults for an NX window.
+ */
+extern void vas_init_tx_win_attr(struct vas_tx_win_attr *txattr,
+			enum vas_cop_type cop);
+
+/*
+ * Open a VAS send window for the instance of VAS identified by @vasid
+ * and the co-processor type @cop. Use @attr to initialize attributes
+ * of the window.
+ *
+ * Note: The instance of VAS must already have an open receive window for
+ * the coprocessor type @cop.
+ *
+ * Return a handle to the send window or ERR_PTR() on error.
+ */
+struct vas_window *vas_tx_win_open(int vasid, enum vas_cop_type cop,
+			struct vas_tx_win_attr *attr);
+
+/*
  * Close the send or receive window identified by @win. For receive windows
  * return -EAGAIN if there are active send windows attached to this receive
  * window.
diff --git a/drivers/misc/vas/vas-window.c b/drivers/misc/vas/vas-window.c
index 40e4f7d..9caf10b 100644
--- a/drivers/misc/vas/vas-window.c
+++ b/drivers/misc/vas/vas-window.c
@@ -756,6 +756,163 @@ struct vas_window *vas_rx_win_open(int vasid, enum vas_cop_type cop,
 	return ERR_PTR(rc);
 }
 
+void vas_init_tx_win_attr(struct vas_tx_win_attr *txattr, enum vas_cop_type cop)
+{
+	memset(txattr, 0, sizeof(*txattr));
+
+	if (cop == VAS_COP_TYPE_842 || cop == VAS_COP_TYPE_842_HIPRI) {
+		txattr->rej_no_credit = false;
+		txattr->rx_wcred_mode = true;
+		txattr->tx_wcred_mode = true;
+		txattr->rx_win_ord_mode = true;
+		txattr->tx_win_ord_mode = true;
+	}
+}
+
+static void init_winctx_for_txwin(struct vas_window *txwin,
+			struct vas_tx_win_attr *txattr,
+			struct vas_winctx *winctx)
+{
+	/*
+	 * We first zero all fields and only set non-zero ones. Following
+	 * are some fields set to 0/false for the stated reason:
+	 *
+	 *	->notify_os_intr_reg	In powerNV, send intrs to HV
+	 *	->rsvd_txbuf_count	Not supported yet.
+	 *	->notify_disable	False for NX windows
+	 *	->xtra_write		False for NX windows
+	 *	->notify_early		NA for NX windows
+	 *	->lnotify_lpid		NA for Tx windows
+	 *	->lnotify_pid		NA for Tx windows
+	 *	->lnotify_tid		NA for Tx windows
+	 *	->tx_win_cred_mode	Ignore for now for NX windows
+	 *	->rx_win_cred_mode	Ignore for now for NX windows
+	 */
+	memset(winctx, 0, sizeof(struct vas_winctx));
+
+	winctx->wcreds_max = txattr->wcreds_max ?: VAS_WCREDS_DEFAULT;
+
+	winctx->user_win = txattr->user_win;
+	winctx->nx_win = txwin->rxwin->nx_win;
+	winctx->pin_win = txattr->pin_win;
+
+	winctx->rx_wcred_mode = txattr->rx_wcred_mode;
+	winctx->tx_wcred_mode = txattr->tx_wcred_mode;
+	winctx->rx_word_mode = txattr->rx_win_ord_mode;
+	winctx->tx_word_mode = txattr->tx_win_ord_mode;
+
+	if (winctx->nx_win) {
+		winctx->data_stamp = true;
+		winctx->intr_disable = true;
+	}
+
+	winctx->lpid = txattr->lpid;
+	winctx->pidr = txattr->pidr;
+	winctx->rx_win_id = txwin->rxwin->winid;
+	winctx->fault_win_id = fault_winid;
+
+	winctx->dma_type = VAS_DMA_TYPE_INJECT;
+	winctx->tc_mode = txattr->tc_mode;
+	winctx->min_scope = VAS_SCOPE_LOCAL;
+	winctx->max_scope = VAS_SCOPE_VECTORED_GROUP;
+}
+
+static bool tx_win_args_valid(enum vas_cop_type cop,
+			struct vas_tx_win_attr *attr)
+{
+	if (attr->tc_mode != VAS_THRESH_DISABLED)
+		return false;
+
+	if (cop > VAS_COP_TYPE_MAX)
+		return false;
+
+	if (attr->user_win) {
+		if (cop != VAS_COP_TYPE_GZIP && cop != VAS_COP_TYPE_GZIP_HIPRI)
+			return false;
+
+		if (attr->rsvd_txbuf_count != 0)
+			return false;
+	}
+
+	return true;
+}
+
+struct vas_window *vas_tx_win_open(int vasid, enum vas_cop_type cop,
+			struct vas_tx_win_attr *attr)
+{
+	int rc, winid;
+	struct vas_instance *vinst;
+	struct vas_window *txwin;
+	struct vas_window *rxwin;
+	struct vas_winctx winctx;
+	int size;
+	char *name;
+	uint64_t paste_busaddr;
+
+	if (!vas_initialized)
+		return ERR_PTR(-EAGAIN);
+
+	if (!tx_win_args_valid(cop, attr))
+		return ERR_PTR(-EINVAL);
+
+	vinst = find_vas_instance(vasid);
+	if (!vinst) {
+		pr_devel("VAS: vasid %d not found!\n", vasid);
+		return ERR_PTR(-EINVAL);
+	}
+
+	rxwin = get_vinstance_rxwin(vinst, cop);
+	if (!rxwin) {
+		pr_devel("VAS: No RxWin for vasid %d, cop %d\n", vasid, cop);
+		return ERR_PTR(-EINVAL);
+	}
+
+	rc = -EAGAIN;
+	winid = vas_assign_window_id(&vinst->ida);
+	if (winid < 0)
+		goto put_rxwin;
+
+	rc = -ENOMEM;
+	txwin = vas_window_alloc(vinst, winid);
+	if (!txwin)
+		goto release_winid;
+
+	txwin->tx_win = 1;
+	txwin->rxwin = rxwin;
+	txwin->nx_win = txwin->rxwin->nx_win;
+	txwin->pid = attr->pid;
+
+	init_winctx_for_txwin(txwin, attr, &winctx);
+
+	init_winctx_regs(txwin, &winctx);
+
+	name = kasprintf(GFP_KERNEL, "window-v%d-w%d", vasid, winid);
+	if (!name)
+		goto release_winid;
+
+	txwin->paste_addr_name = name;
+	compute_paste_address(txwin, &paste_busaddr, &size);
+
+	txwin->paste_kaddr = map_mmio_region(name, paste_busaddr, size);
+	if (!txwin->paste_kaddr)
+		goto free_name;
+
+	pr_devel("VAS: mapped paste addr 0x%llx to kaddr 0x%p\n",
+				paste_busaddr, txwin->paste_kaddr);
+	return txwin;
+
+free_name:
+	kfree(txwin->paste_addr_name);
+
+release_winid:
+	vas_release_window_id(&vinst->ida, txwin->winid);
+
+put_rxwin:
+	put_rx_win(rxwin);
+	return ERR_PTR(rc);
+
+}
+
 static void poll_window_busy_state(struct vas_window *window)
 {
 	int busy;
-- 
2.7.4

[PATCH v3 10/10] VAS: Define copy/paste interfaces

[Sukadev Bhattiprolu]

Define interfaces (wrappers) to the 'copy' and 'paste' instructions
(which are new in PowerISA 3.0). These are intended to be used to
by NX driver(s) to submit Coprocessor Request Blocks (CRBs) to the
NX hardware engines.

Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>

---
Changelog[v3]
	- Map raw CR value from paste instruction into an error code.
---
 arch/powerpc/include/asm/vas.h  | 13 ++++++++
 drivers/misc/vas/copy-paste.h   | 74 +++++++++++++++++++++++++++++++++++++++++
 drivers/misc/vas/vas-internal.h | 14 ++++++++
 drivers/misc/vas/vas-window.c   | 50 ++++++++++++++++++++++++++++
 4 files changed, 151 insertions(+)
 create mode 100644 drivers/misc/vas/copy-paste.h

diff --git a/arch/powerpc/include/asm/vas.h b/arch/powerpc/include/asm/vas.h
index ff8da98..1ef81ed 100644
--- a/arch/powerpc/include/asm/vas.h
+++ b/arch/powerpc/include/asm/vas.h
@@ -132,6 +132,19 @@ struct vas_window *vas_tx_win_open(int vasid, enum vas_cop_type cop,
 int vas_win_close(struct vas_window *win);
 
 /*
+ * Copy the co-processor request block (CRB) @crb into the local L2 cache.
+ * For now, @offset must be 0 and @first must be true.
+ */
+extern int vas_copy_crb(void *crb, int offset, bool first);
+
+/*
+ * Paste a previously copied CRB (see vas_copy_crb()) from the L2 cache to
+ * the hardware address associated with the window @win. For now, @off must
+ * 0 and @last must be true. @re is expected/assumed to be true for NX windows.
+ */
+extern int vas_paste_crb(struct vas_window *win, int off, bool last, bool re);
+
+/*
  * Get/Set bit fields
  */
 #define GET_FIELD(m, v)		(((v) & (m)) >> MASK_LSH(m))
diff --git a/drivers/misc/vas/copy-paste.h b/drivers/misc/vas/copy-paste.h
new file mode 100644
index 0000000..7783bb8
--- /dev/null
+++ b/drivers/misc/vas/copy-paste.h
@@ -0,0 +1,74 @@
+/*
+ * Copyright 2016 IBM Corp.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+
+/*
+ * Macros taken from tools/testing/selftests/powerpc/context_switch/cp_abort.c
+ */
+#define PASTE(RA, RB, L, RC) \
+	.long (0x7c00070c | (RA) << (31-15) | (RB) << (31-20) \
+			  | (L) << (31-10) | (RC) << (31-31))
+
+#define COPY(RA, RB, L) \
+	.long (0x7c00060c | (RA) << (31-15) | (RB) << (31-20) \
+			  | (L) << (31-10))
+
+#define CR0_FXM		"0x80"
+#define CR0_SHIFT	28
+#define CR0_MASK	0xF
+/*
+ * Copy/paste instructions:
+ *
+ *	copy RA,RB,L
+ *		Copy contents of address (RA) + effective_address(RB)
+ *		to internal copy-buffer.
+ *
+ *		L == 1 indicates this is the first copy.
+ *
+ *		L == 0 indicates its a continuation of a prior first copy.
+ *
+ *	paste RA,RB,L
+ *		Paste contents of internal copy-buffer to the address
+ *		(RA) + effective_address(RB)
+ *
+ *		L == 0 indicates its a continuation of a prior paste. i.e.
+ *		don't wait for the completion or update status.
+ *
+ *		L == 1 indicates this is the last paste in the group (i.e.
+ *		wait for the group to complete and update status in CR0).
+ *
+ *	For Power9, the L bit must be 'true' in both copy and paste.
+ */
+
+static inline int vas_copy(void *crb, int offset, int first)
+{
+	WARN_ON_ONCE(!first);
+
+	__asm__ __volatile(stringify_in_c(COPY(%0, %1, %2))";"
+		:
+		: "b" (offset), "b" (crb), "i" (1)
+		: "memory");
+
+	return 0;
+}
+
+static inline int vas_paste(void *paste_address, int offset, int last)
+{
+	unsigned long long cr;
+
+	WARN_ON_ONCE(!last);
+
+	cr = 0;
+	__asm__ __volatile(stringify_in_c(PASTE(%1, %2, 1, 1))";"
+		"mfocrf %0," CR0_FXM ";"
+		: "=r" (cr)
+		: "b" (paste_address), "b" (offset)
+		: "memory");
+
+	return cr;
+}
diff --git a/drivers/misc/vas/vas-internal.h b/drivers/misc/vas/vas-internal.h
index 1e5c94b..54e2a31 100644
--- a/drivers/misc/vas/vas-internal.h
+++ b/drivers/misc/vas/vas-internal.h
@@ -456,4 +456,18 @@ static inline uint64_t read_hvwc_reg(struct vas_window *win,
 	return in_be64(win->hvwc_map+reg);
 }
 
+#ifdef vas_debug
+
+static void print_fifo_msg_count(struct vas_window *txwin)
+{
+	uint64_t read_hvwc_reg(struct vas_window *w, char *n, uint64_t o);
+	pr_devel("Winid %d, Msg count %llu\n", txwin->winid,
+			(uint64_t)read_hvwc_reg(txwin, VREG(LRFIFO_PUSH)));
+}
+#else	/* vas_debug */
+
+#define print_fifo_msg_count(window)
+
+#endif	/* vas_debug */
+
 #endif
diff --git a/drivers/misc/vas/vas-window.c b/drivers/misc/vas/vas-window.c
index 9caf10b..fa2dd72 100644
--- a/drivers/misc/vas/vas-window.c
+++ b/drivers/misc/vas/vas-window.c
@@ -13,6 +13,7 @@
 #include <linux/io.h>
 #include <asm/vas.h>
 #include "vas-internal.h"
+#include "copy-paste.h"
 
 static int fault_winid;
 
@@ -913,6 +914,55 @@ struct vas_window *vas_tx_win_open(int vasid, enum vas_cop_type cop,
 
 }
 
+int vas_copy_crb(void *crb, int offset, bool first)
+{
+	if (!vas_initialized)
+		return -1;
+
+	return vas_copy(crb, offset, first);
+}
+
+#define RMA_LSMP_REPORT_ENABLE PPC_BIT(53)
+int vas_paste_crb(struct vas_window *txwin, int offset, bool last, bool re)
+{
+	int rc;
+	uint64_t val;
+	void *addr;
+
+	if (!vas_initialized)
+		return -1;
+	/*
+	 * Only NX windows are supported for now and hardware assumes
+	 * report-enable flag is set for NX windows. Ensure software
+	 * complies too.
+	 */
+	WARN_ON_ONCE(!re);
+
+	addr = txwin->paste_kaddr;
+	if (re) {
+		/*
+		 * Set the REPORT_ENABLE bit (equivalent to writing
+		 * to 1K offset of the paste address)
+		 */
+		val = SET_FIELD(RMA_LSMP_REPORT_ENABLE, 0ULL, 1);
+		addr += val;
+	}
+
+	/*
+	 * Map the raw CR value from vas_paste() to an error code (there
+	 * is just pass or fail for now though).
+	 */
+	rc = vas_paste(addr, offset, last);
+	if (rc == 0x20000000)
+		rc = 0;
+	else
+		rc = -EINVAL;
+
+	print_fifo_msg_count(txwin);
+
+	return rc;
+}
+
 static void poll_window_busy_state(struct vas_window *window)
 {
 	int busy;
-- 
2.7.4

Re: [PATCH v3 02/10] Move GET_FIELD/SET_FIELD to vas.h

[Dan Streetman]

On Thu, Mar 16, 2017 at 11:33 PM, Sukadev Bhattiprolu
<sukadev@linux.vnet.ibm.com> wrote:
> Move the GET_FIELD and SET_FIELD macros to vas.h as VAS and other
> users of VAS, including NX-842 can use those macros.
>
> There is a lot of related code between the VAS/NX kernel drivers
> and skiboot. For consistency switch the order of parameters in
> SET_FIELD to match the order in skiboot.
>
> Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>

Reviewed-by: Dan Streetman <ddstreet@ieee.org>

also, can you change the MAINTAINERS file owner of the IBM Power 842
accelerator to yourself, or someone else at IBM?  I no longer have
access to any of the ppc hardware, so it hardly makes sense for me to
remain the maintainer of it.

> ---
>
> Changelog[v3]
>         - Fix order of parameters in nx-842 driver.
> ---
>  arch/powerpc/include/asm/vas.h     | 8 +++++++-
>  drivers/crypto/nx/nx-842-powernv.c | 7 ++++---
>  drivers/crypto/nx/nx-842.h         | 5 -----
>  3 files changed, 11 insertions(+), 9 deletions(-)
>
> diff --git a/arch/powerpc/include/asm/vas.h b/arch/powerpc/include/asm/vas.h
> index 6d35ce6..184eeb2 100644
> --- a/arch/powerpc/include/asm/vas.h
> +++ b/arch/powerpc/include/asm/vas.h
> @@ -37,7 +37,13 @@ enum vas_cop_type {
>         VAS_COP_TYPE_MAX,
>  };
>
> +/*
> + * Get/Set bit fields
> + */
> +#define GET_FIELD(m, v)                (((v) & (m)) >> MASK_LSH(m))
> +#define MASK_LSH(m)            (__builtin_ffsl(m) - 1)
> +#define SET_FIELD(m, v, val)   \
> +               (((v) & ~(m)) | ((((typeof(v))(val)) << MASK_LSH(m)) & (m)))
>
>  #endif /* __KERNEL__ */
> -
>  #endif
> diff --git a/drivers/crypto/nx/nx-842-powernv.c b/drivers/crypto/nx/nx-842-powernv.c
> index 1710f80..3abb045 100644
> --- a/drivers/crypto/nx/nx-842-powernv.c
> +++ b/drivers/crypto/nx/nx-842-powernv.c
> @@ -22,6 +22,7 @@
>
>  #include <asm/prom.h>
>  #include <asm/icswx.h>
> +#include <asm/vas.h>
>
>  MODULE_LICENSE("GPL");
>  MODULE_AUTHOR("Dan Streetman <ddstreet@ieee.org>");
> @@ -424,9 +425,9 @@ static int nx842_powernv_function(const unsigned char *in, unsigned int inlen,
>
>         /* set up CCW */
>         ccw = 0;
> -       ccw = SET_FIELD(ccw, CCW_CT, nx842_ct);
> -       ccw = SET_FIELD(ccw, CCW_CI_842, 0); /* use 0 for hw auto-selection */
> -       ccw = SET_FIELD(ccw, CCW_FC_842, fc);
> +       ccw = SET_FIELD(CCW_CT, ccw, nx842_ct);
> +       ccw = SET_FIELD(CCW_CI_842, ccw, 0); /* use 0 for hw auto-selection */
> +       ccw = SET_FIELD(CCW_FC_842, ccw, fc);
>
>         /* set up CRB's CSB addr */
>         csb_addr = nx842_get_pa(csb) & CRB_CSB_ADDRESS;
> diff --git a/drivers/crypto/nx/nx-842.h b/drivers/crypto/nx/nx-842.h
> index a4eee3b..30929bd 100644
> --- a/drivers/crypto/nx/nx-842.h
> +++ b/drivers/crypto/nx/nx-842.h
> @@ -100,11 +100,6 @@ static inline unsigned long nx842_get_pa(void *addr)
>         return page_to_phys(vmalloc_to_page(addr)) + offset_in_page(addr);
>  }
>
> -/* Get/Set bit fields */
> -#define MASK_LSH(m)            (__builtin_ffsl(m) - 1)
> -#define GET_FIELD(v, m)                (((v) & (m)) >> MASK_LSH(m))
> -#define SET_FIELD(v, m, val)   (((v) & ~(m)) | (((val) << MASK_LSH(m)) & (m)))
> -
>  /**
>   * This provides the driver's constraints.  Different nx842 implementations
>   * may have varying requirements.  The constraints are:
> --
> 2.7.4
>

Re: [PATCH v3 03/10] VAS: Define vas_init() and vas_exit()

[Michael Neuling]

On Thu, 2017-03-16 at 20:33 -0700, Sukadev Bhattiprolu wrote:
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -3673,8 +3673,6 @@ F:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0a=
rch/powerpc/platforms/powernv/pci-cxl.c
> =C2=A0F:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0drivers/misc/cxl/
> =C2=A0F:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0include/misc/cxl*
> =C2=A0F:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0include/uapi/misc/cxl.h
> -F:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Documentation/powerpc/cxl.txt
> -F:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Documentation/ABI/testing/sysfs-class-cx=
l
>=20

This shouldn't be deleted.

Mikey

Re: [PATCH v3 01/10] VAS: Define macros, register fields and structures

[Michael Neuling]

On Thu, 2017-03-16 at 20:33 -0700, Sukadev Bhattiprolu wrote:
> Define macros for the VAS hardware registers and bit-fields as well
> as couple of data structures needed by the VAS driver.
>=20
> > Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
> ---
> Changelog[v3]
> 	- Rename winctx->pid to winctx->pidr to reflect that its a value
> 	=C2=A0=C2=A0from the PID register (SPRN_PID), not the linux process id.
> 	- Make it easier to split header into kernel/user parts
> 	- To keep user interface simple, use macros rather than enum for
> 	=C2=A0=C2=A0the threshold-control modes.
> 	- Add a pid field to struct vas_window - needed for user space
> 	=C2=A0=C2=A0send windows.
>=20
> Changelog[v2]
> 	- Add an overview of VAS in vas-internal.h
> 	- Get window context parameters from device tree and drop
> 	=C2=A0=C2=A0unnecessary macros.
> ---
> =C2=A0MAINTAINERS=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0|=
=C2=A0=C2=A0=C2=A06 +
> =C2=A0arch/powerpc/include/asm/vas.h=C2=A0=C2=A0|=C2=A0=C2=A043 +++++
> =C2=A0drivers/misc/vas/vas-internal.h | 392 +++++++++++++++++++++++++++++=
+++++++++++

This is going to have to go through gregkh/lkml if it's drivers/misc.  you'=
ll at
least need gregkh's ack/ok before mpe will take them (which is what we did =
for
CAPI).

We might want to keep this in arch/powerpc but I'm not sure.

> =C2=A03 files changed, 441 insertions(+)
> =C2=A0create mode 100644 arch/powerpc/include/asm/vas.h
> =C2=A0create mode 100644 drivers/misc/vas/vas-internal.h
>=20
<snip>
> +
> +/*
> + * Overview of Virtual Accelerator Switchboard (VAS).
> + *
> + * VAS is a hardware "switchboard" that allows senders and receivers to
> + * exchange messages with _minimal_ kernel involvment. The receivers are
> + * typically NX coprocessor engines that perform compression or encrypti=
on
> + * in hardware, but receivers can also be other software threads.
> + *
> + * Senders are user/kernel threads that submit compression/encryption or
> + * other requests to the receivers. Senders must format their messages a=
s
> + * Coprocessor Request Blocks (CRB)s and submit them using the instructi=
ons
> + * "copy" and "paste" which were introduced in Power9.
> + *
> + * A Power node can have (upto?) 8 Power chips. There is one instance of
> + * VAS in each Power9 chip. Each instance of VAS has 64K windows or port=
s,
> + * Senders and receivers must each connect to a separate window before t=
hey
> + * can exchange messages through the switchboard.
> + *
> + * Each window is described by two types of window contexts:
> + *
> > + *	Hypervisor Window Context (HVWC) of size VAS_HVWC_SIZE bytes
> + *
> > + *	OS/User Window Context (UWC) of size VAS_UWC_SIZE bytes.
> + *
> + * A window context can be viewed as a set of 64-bit registers. The sett=
ings
> + * in these registers configure/control/determine the behavior of the VA=
S
> + * hardware when messages are sent/received through the window. The regi=
sters
> + * in the HVWC are configured by the kernel while the registers in the U=
WC can
> + * be configured by the kernel or by the user space application that is =
using
> + * the window.
> + *
> + * The HVWCs for all windows on a specific instance of VAS are in a cont=
iguous
> + * range of hardware addresses or Base address region (BAR) referred to =
as the
> + * HVWC BAR for the instance. Similarly the UWCs for all windows on an i=
nstance
> + * are referred to as the UWC BAR for the instance. The two BARs for eac=
h
> + * instance are defined Power9 MMIO Ranges spreadsheet and available to =
the
> + * kernel the device tree as follows:
> + *
> > + *	/proc/device-tree/xscom@.../vas@.../hvwc-bar-start
> > + *	/proc/device-tree/xscom@.../vas@.../hvwc-bar-size
> > + *	/proc/device-tree/xscom@.../vas@.../uwc-bar-start
> + *	/proc/device-tree/xscom@.../vas@.../uwc-bar-size

should these just be reg properties?

> + *
> + * The kernel maps these two hardware address regions into the kernel ad=
dress
> + * space (hvwc_map and uwc_map) and accesses the window contexts of a sp=
ecific
> + * window using:
> + *
> > + *	=C2=A0hvwc =3D hvwc_map + winid * VAS_HVWC_SIZE.
> > + *	=C2=A0uwc =3D uwc_map + winid * VAS_UWC_SIZE.
> + *
> + * where winid is the window index (0..64K).
> + *
> + * Note that the window contexts are used to "configure" the windows. In
> + * addition to this configuration address, each _send_ window also has a
> + * unique hardware address, referred to as the "paste-address" to which =
the
> + * sender must "paste" the message (CRB) they wish to submit. This hardw=
are
> + * paste address for window can be computed from the following nodes in =
the
> + * device tree:
> + *
> > + *	/proc/device-tree/xscom@.../vas@.../window-base
> + *	/proc/device-tree/xscom@.../vas@.../window-shift

Same here with reg properties.

<snip>=20
> +struct vas_winctx {
<snip>
> > +	int lpid;
> +	int pidr;		/* value from SPRN_PID, not linux pid */

I'm surprised we have a copy of these here.  They should be accessed from t=
he
context we are attaching to, rather than copied here... but I've not looked=
 at
the rest of the code yet...

Re: [PATCH v3 03/10] VAS: Define vas_init() and vas_exit()

[Sukadev Bhattiprolu]

Michael Neuling [michael.neuling@au1.ibm.com] wrote:
> On Thu, 2017-03-16 at 20:33 -0700, Sukadev Bhattiprolu wrote:
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -3673,8 +3673,6 @@ F:=A0=A0=A0=A0=A0=A0=A0=A0arch/powerpc/platforms/=
powernv/pci-cxl.c
> > =A0F:=A0=A0=A0=A0=A0drivers/misc/cxl/
> > =A0F:=A0=A0=A0=A0=A0include/misc/cxl*
> > =A0F:=A0=A0=A0=A0=A0include/uapi/misc/cxl.h
> > -F:=A0=A0=A0=A0=A0Documentation/powerpc/cxl.txt
> > -F:=A0=A0=A0=A0=A0Documentation/ABI/testing/sysfs-class-cxl
> >=20
>=20
> This shouldn't be deleted.

Yes, I will fix it.

Thanks

Sukadev
>=20
> Mikey

Re: [PATCH v3 03/10] VAS: Define vas_init() and vas_exit()

[Michael Neuling]

On Thu, 2017-03-16 at 20:33 -0700, Sukadev Bhattiprolu wrote:
> Implement vas_init() and vas_exit() functions for a new VAS module.
> This VAS module is essentially a library for other device drivers
> and kernel users of the NX coprocessors like NX-842 and NX-GZIP.
>=20
> Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
> ---
> Changelog[v3]:
> 	- Zero vas_instances memory on allocation
> 	- [Haren Myneni] Fix description in Kconfig
> Changelog[v2]:
> 	- Get HVWC, UWC and window address parameters from device tree.
> ---
> =C2=A0MAINTAINERS=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0|=
=C2=A0=C2=A0=C2=A08 ++-
> =C2=A0arch/powerpc/include/asm/reg.h=C2=A0=C2=A0|=C2=A0=C2=A0=C2=A01 +
> =C2=A0drivers/misc/Kconfig=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0|=C2=A0=C2=A0=C2=A01 +
> =C2=A0drivers/misc/Makefile=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0|=C2=A0=C2=A0=C2=A01 +
> =C2=A0drivers/misc/vas/Kconfig=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0|=C2=A0=C2=A021 ++++++
> =C2=A0drivers/misc/vas/Makefile=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
|=C2=A0=C2=A0=C2=A03 +
> =C2=A0drivers/misc/vas/vas-internal.h |=C2=A0=C2=A0=C2=A03 +
> =C2=A0drivers/misc/vas/vas-window.c=C2=A0=C2=A0=C2=A0|=C2=A0=C2=A019 ++++=
+
> =C2=A0drivers/misc/vas/vas.c=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0| 155
> ++++++++++++++++++++++++++++++++++++++++
> =C2=A09 files changed, 210 insertions(+), 2 deletions(-)
> =C2=A0create mode 100644 drivers/misc/vas/Kconfig
> =C2=A0create mode 100644 drivers/misc/vas/Makefile
> =C2=A0create mode 100644 drivers/misc/vas/vas-window.c
> =C2=A0create mode 100644 drivers/misc/vas/vas.c
>=20
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 2a910c9..4037252 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -3673,8 +3673,6 @@ F:	arch/powerpc/platforms/powernv/pci-cxl.c
> =C2=A0F:	drivers/misc/cxl/
> =C2=A0F:	include/misc/cxl*
> =C2=A0F:	include/uapi/misc/cxl.h
> -F:	Documentation/powerpc/cxl.txt
> -F:	Documentation/ABI/testing/sysfs-class-cxl

err?

> =C2=A0CXLFLASH (IBM Coherent Accelerator Processor Interface CAPI Flash) =
SCSI
> DRIVER
> =C2=A0M:	Manoj N. Kumar <manoj@linux.vnet.ibm.com>
> @@ -3686,6 +3684,12 @@ F:	drivers/scsi/cxlflash/
> =C2=A0F:	include/uapi/scsi/cxlflash_ioctls.h
> =C2=A0F:	Documentation/powerpc/cxlflash.txt
> =C2=A0
> +VAS (IBM Virtual Accelerator Switch) DRIVER
> +M:	Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
> +L:	linuxppc-dev@lists.ozlabs.org
> +S:	Supported
> +F:	drivers/misc/vas/
> +

This was already added in patch 1.

> =C2=A0STMMAC ETHERNET DRIVER
> =C2=A0M:	Giuseppe Cavallaro <peppe.cavallaro@st.com>
> =C2=A0M:	Alexandre Torgue <alexandre.torgue@st.com>
> diff --git a/arch/powerpc/include/asm/reg.h b/arch/powerpc/include/asm/re=
g.h
> index fc879fd..7a45ff7 100644
> --- a/arch/powerpc/include/asm/reg.h
> +++ b/arch/powerpc/include/asm/reg.h
> @@ -1225,6 +1225,7 @@
> =C2=A0#define PVR_POWER8E	0x004B
> =C2=A0#define PVR_POWER8NVL	0x004C
> =C2=A0#define PVR_POWER8	0x004D
> +#define PVR_POWER9	0x004E

Can you send this up separately? =20

> =C2=A0#define PVR_BE		0x0070
> =C2=A0#define PVR_PA6T	0x0090
> =C2=A0
> diff --git a/drivers/misc/Kconfig b/drivers/misc/Kconfig
> index c290990..97d652e 100644
> --- a/drivers/misc/Kconfig
> +++ b/drivers/misc/Kconfig
> @@ -783,4 +783,5 @@ source "drivers/misc/mic/Kconfig"
> =C2=A0source "drivers/misc/genwqe/Kconfig"
> =C2=A0source "drivers/misc/echo/Kconfig"
> =C2=A0source "drivers/misc/cxl/Kconfig"
> +source "drivers/misc/vas/Kconfig"
> =C2=A0endmenu
> diff --git a/drivers/misc/Makefile b/drivers/misc/Makefile
> index 7a3ea89..5201ffd 100644
> --- a/drivers/misc/Makefile
> +++ b/drivers/misc/Makefile
> @@ -53,6 +53,7 @@ obj-$(CONFIG_GENWQE)		+=3D genwqe/
> =C2=A0obj-$(CONFIG_ECHO)		+=3D echo/
> =C2=A0obj-$(CONFIG_VEXPRESS_SYSCFG)	+=3D vexpress-syscfg.o
> =C2=A0obj-$(CONFIG_CXL_BASE)		+=3D cxl/
> +obj-$(CONFIG_VAS)		+=3D vas/
> =C2=A0obj-$(CONFIG_PANEL)=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0+=3D panel.o
> =C2=A0
> =C2=A0lkdtm-$(CONFIG_LKDTM)		+=3D lkdtm_core.o
> diff --git a/drivers/misc/vas/Kconfig b/drivers/misc/vas/Kconfig
> new file mode 100644
> index 0000000..43cedda
> --- /dev/null
> +++ b/drivers/misc/vas/Kconfig
> @@ -0,0 +1,21 @@
> +#
> +# IBM Virtual Accelarator Switchboard (VAS) compatible devices
> +#depends on PPC_POWERNV && PCI_MSI && EEH
> +#
> +
> +config VAS
> +	tristate "Support for IBM Virtual Accelerator Switchboard (VAS)"
> +	depends on PPC_POWERNV
> +	default n
> +	help
> +	=C2=A0=C2=A0Select this option to enable driver support for IBM Virtual
> +	=C2=A0=C2=A0Accelerator Switchboard (VAS).
> +
> +	=C2=A0=C2=A0VAS allows accelerators in co processors like NX-842 to be
> +	=C2=A0=C2=A0directly available to a user process. This driver enables
> +	=C2=A0=C2=A0userspace programs to access these accelerators via device
> +	=C2=A0=C2=A0nodes like /dev/crypto/nx-gzip.

I though this was kernel only users for now?

> +
> +	=C2=A0=C2=A0VAS adapters are found in POWER9 based systems.
> +
> +	=C2=A0=C2=A0If unsure, say N.
> diff --git a/drivers/misc/vas/Makefile b/drivers/misc/vas/Makefile
> new file mode 100644
> index 0000000..7dd7139
> --- /dev/null
> +++ b/drivers/misc/vas/Makefile
> @@ -0,0 +1,3 @@
> +ccflags-y			:=3D $(call cc-disable-warning, unused-const-
> variable)
> +ccflags-$(CONFIG_PPC_WERROR)	+=3D -Werror
> +obj-$(CONFIG_VAS)		+=3D vas.o vas-window.o
> diff --git a/drivers/misc/vas/vas-internal.h b/drivers/misc/vas/vas-inter=
nal.h
> index ce48f14..15b62e0 100644
> --- a/drivers/misc/vas/vas-internal.h
> +++ b/drivers/misc/vas/vas-internal.h

Just call this file vas.h.=20

Internal one is here.  Kernel api one lives in include/misc.  User API live=
s in
./include/uapi/misc. =C2=A0

eg. CXL does this:

% find . -name cxl.h
./include/uapi/misc/cxl.h
./include/misc/cxl.h
./drivers/misc/cxl/cxl.h


> @@ -389,4 +389,7 @@ struct vas_winctx {
> =C2=A0	enum vas_notify_after_count notify_after_count;
> =C2=A0};
> =C2=A0
> +extern int vas_initialized;

I would probably make this a function.

> +extern int vas_window_reset(struct vas_instance *vinst, int winid);
> +extern struct vas_instance *find_vas_instance(int vasid);
> =C2=A0#endif
> diff --git a/drivers/misc/vas/vas-window.c b/drivers/misc/vas/vas-window.=
c
> new file mode 100644
> index 0000000..468f3bf
> --- /dev/null
> +++ b/drivers/misc/vas/vas-window.c
> @@ -0,0 +1,19 @@
> +/*
> + * Copyright 2016 IBM Corp.
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU General Public License
> + * as published by the Free Software Foundation; either version
> + * 2 of the License, or (at your option) any later version.
> + */
> +
> +#include <linux/types.h>
> +#include <linux/mutex.h>
> +#include <asm/vas.h>
> +#include "vas-internal.h"
> +
> +/* stub for now */
> +int vas_window_reset(struct vas_instance *vinst, int winid)
> +{
> +	return 0;

No biggy, but best to return an error if it's just a stub in case sometime =
tries
to use it.

> +}
> diff --git a/drivers/misc/vas/vas.c b/drivers/misc/vas/vas.c
> new file mode 100644
> index 0000000..75e55f9
> --- /dev/null
> +++ b/drivers/misc/vas/vas.c
> @@ -0,0 +1,155 @@
> +/*
> + * Copyright 2016 IBM Corp.
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU General Public License
> + * as published by the Free Software Foundation; either version
> + * 2 of the License, or (at your option) any later version.
> + */
> +
> +#include <linux/module.h>
> +#include <linux/kernel.h>
> +#include <linux/export.h>
> +#include <linux/types.h>
> +#include <linux/slab.h>
> +#include <linux/of.h>
> +#include <asm/vas.h>
> +#include "vas-internal.h"
> +
> +int vas_initialized;
> +int vas_num_instances;
> +struct vas_instance *vas_instances;
> +
> +static void init_vas_chip(struct vas_instance *vinst)
> +{
> +	int i;
> +
> +	for (i =3D 0; i < VAS_MAX_WINDOWS_PER_CHIP; i++)
> +		vas_window_reset(vinst, i);
> +}
> +
> +static int init_vas_instance(struct device_node *dn,
> +				struct vas_instance *vinst)
> +{
> +	int rc;
> +	const __be32 *p;
> +
> +	ida_init(&vinst->ida);
> +	mutex_init(&vinst->mutex);
> +
> +	p =3D of_get_property(dn, "vas-id", NULL);
> +	if (!p) {
> +		pr_err("VAS: NULL vas-id? %p\n", p);
> +		return -ENODEV;
> +	}
> +
> +	vinst->vas_id =3D of_read_number(p, 1);
> +
> +	rc =3D of_property_read_u64(dn, "hvwc-bar-start", &vinst-
> >hvwc_bar_start);
> +	if (rc)
> +		return rc;
> +
> +	rc =3D of_property_read_u64(dn, "hvwc-bar-size", &vinst->hvwc_bar_len);
> +	if (rc)
> +		return rc;
> +
> +	rc =3D of_property_read_u64(dn, "uwc-bar-start", &vinst-
> >uwc_bar_start);
> +	if (rc)
> +		return rc;
> +
> +	rc =3D of_property_read_u64(dn, "uwc-bar-size", &vinst->uwc_bar_len);
> +	if (rc)
> +		return rc;
> +
> +	rc =3D of_property_read_u64(dn, "window-base", &vinst->win_base_addr);
> +	if (rc)
> +		return rc;
> +
> +	rc =3D of_property_read_u64(dn, "window-shift", &vinst->win_id_shift);
> +	if (rc)
> +		return rc;
> +
> +	init_vas_chip(vinst);
> +
> +	return 0;
> +}
> +
> +/*
> + * Although this is read/used multiple times, it is written to only
> + * during initialization.
> + */
> +struct vas_instance *find_vas_instance(int vasid)
> +{
> +	int i;
> +	struct vas_instance *vinst;
> +
> +	for (i =3D 0; i < vas_num_instances; i++) {
> +		vinst =3D &vas_instances[i];
> +		if (vinst->vas_id =3D=3D vasid)
> +			return vinst;
> +	}
> +	pr_err("VAS instance for vas-id %d not found\n", vasid);
> +	WARN_ON_ONCE(1);
> +	return NULL;
> +}
> +
> +
> +int vas_init(void)
> +{
> +	int rc;
> +	struct device_node *dn;
> +	struct vas_instance *vinst;
> +
> +	if (!pvr_version_is(PVR_POWER9))
> +		return -ENODEV;
> +
> +	vas_num_instances =3D 0;
> +	for_each_node_by_name(dn, "vas")
> +		vas_num_instances++;
> +
> +	if (!vas_num_instances)
> +		return -ENODEV;
> +
> +	vas_instances =3D kcalloc(vas_num_instances, sizeof(*vinst),
> GFP_KERNEL);
> +	if (!vas_instances)
> +		return -ENOMEM;
> +
> +	vinst =3D &vas_instances[0];
> +	for_each_node_by_name(dn, "vas") {
> +		rc =3D init_vas_instance(dn, vinst);
> +		if (rc) {
> +			pr_err("Error %d initializing VAS instance %ld\n",
> rc,
> +					(vinst-&vas_instances[0]));
> +			goto cleanup;
> +		}
> +		vinst++;
> +	}
> +
> +	rc =3D -ENODEV;
> +	if (vinst =3D=3D &vas_instances[0]) {
> +		/* Should not happen as we saw some above. */
> +		pr_err("VAS: Did not find any VAS DT nodes now!\n");
> +		goto cleanup;
> +	}
> +
> +	pr_devel("VAS: Initialized %d instances\n", vas_num_instances);
> +	vas_initialized =3D 1;
> +
> +	return 0;
> +
> +cleanup:
> +	kfree(vas_instances);
> +	return rc;
> +}
> +
> +void vas_exit(void)
> +{
> +	vas_initialized =3D 0;
> +	kfree(vas_instances);
> +}
> +
> +module_init(vas_init);
> +module_exit(vas_exit);
> +MODULE_DESCRIPTION("IBM Virtual Accelerator Switchboard");
> +MODULE_AUTHOR("Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>");
> +MODULE_LICENSE("GPL");

Re: [PATCH v3 04/10] VAS: Define helpers for access MMIO regions

[Michael Neuling]

On Thu, 2017-03-16 at 20:33 -0700, Sukadev Bhattiprolu wrote:
> Define some helper functions to access the MMIO regions. We use these
> in a follow-on patches to read/write VAS hardware registers. These
> helpers are also used to later issue 'paste' instructions to submit
> requests to the NX hardware engines.
>=20
> Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
> ---
> Changelog [v3]:
> 	- Minor reorg/cleanup of map/unmap functions
>=20
> Changelog [v2]:
> 	- Get HVWC, UWC and paste addresses from window->vinst (i.e DT)
> 	=C2=A0=C2=A0rather than kernel macros.
> ---
> =C2=A0drivers/misc/vas/vas-window.c | 126
> ++++++++++++++++++++++++++++++++++++++++++
> =C2=A01 file changed, 126 insertions(+)
>=20
> diff --git a/drivers/misc/vas/vas-window.c b/drivers/misc/vas/vas-window.=
c
> index 468f3bf..32dd1d0 100644
> --- a/drivers/misc/vas/vas-window.c
> +++ b/drivers/misc/vas/vas-window.c
> @@ -9,9 +9,135 @@
> =C2=A0
> =C2=A0#include <linux/types.h>
> =C2=A0#include <linux/mutex.h>
> +#include <linux/slab.h>
> +#include <linux/io.h>
> =C2=A0#include <asm/vas.h>
> =C2=A0#include "vas-internal.h"
> =C2=A0
> +/*
> + * Compute the paste address region for the window @window using the
> + * ->win_base_addr and ->win_id_shift we got from device tree.
> + */
> +void compute_paste_address(struct vas_window *window, uint64_t *addr, in=
t
> *len)
> +{
> +	uint64_t base, shift;
> +	int winid;
> +
> +	base =3D window->vinst->win_base_addr;
> +	shift =3D window->vinst->win_id_shift;
> +	winid =3D window->winid;
> +
> +	*addr=C2=A0=C2=A0=3D base + (winid << shift);
> +	*len =3D PAGE_SIZE;
> +
> +	pr_debug("Txwin #%d: Paste addr 0x%llx\n", winid, *addr);
> +}
> +
> +static inline void get_hvwc_mmio_bar(struct vas_window *window,
> +			uint64_t *start, int *len)
> +{
> +	uint64_t pbaddr;
> +
> +	pbaddr =3D window->vinst->hvwc_bar_start;
> +	*start =3D pbaddr + window->winid * VAS_HVWC_SIZE;
> +	*len =3D VAS_HVWC_SIZE;
> +}
> +
> +static inline void get_uwc_mmio_bar(struct vas_window *window,
> +			uint64_t *start, int *len)
> +{
> +	uint64_t pbaddr;
> +
> +	pbaddr =3D window->vinst->uwc_bar_start;
> +	*start =3D pbaddr + window->winid * VAS_UWC_SIZE;
> +	*len =3D VAS_UWC_SIZE;

I'm not sure this works for 4K pages since VAS_UWC_SIZE =3D PAGE_SIZE but i=
n
reality I think it's always 64K.  Right?

Seem like we are mixing pages sizes and hardware sizes here.

(I realise 4K isn't supported yet, but....)

> +}
> +
> +static void *map_mmio_region(char *name, uint64_t start, int len)
> +{
> +	void *map;
> +
> +	if (!request_mem_region(start, len, name)) {
> +		pr_devel("%s(): request_mem_region(0x%llx, %d) failed\n",
> +				__func__, start, len);
> +		return NULL;
> +	}
> +
> +	map =3D __ioremap(start, len, pgprot_val(pgprot_cached(__pgprot(0))));
> +	if (!map) {
> +		pr_devel("%s(): ioremap(0x%llx, %d) failed\n", __func__,
> start,
> +				len);
> +		return NULL;
> +	}
> +
> +	return map;
> +}
> +
> +/*
> + * Unmap the MMIO regions for a window.
> + */
> +static void unmap_wc_paste_kaddr(struct vas_window *window)
> +{
> +	int len;
> +	uint64_t busaddr_start;
> +
> +	if (window->paste_kaddr) {
> +		iounmap(window->paste_kaddr);
> +		compute_paste_address(window, &busaddr_start, &len);
> +		release_mem_region((phys_addr_t)busaddr_start, len);
> +		window->paste_kaddr =3D NULL;
> +	}
> +
> +}
> +
> +static void unmap_wc_mmio_bars(struct vas_window *window)
> +{
> +	int len;
> +	uint64_t busaddr_start;
> +
> +	unmap_wc_paste_kaddr(window);
> +
> +	if (window->hvwc_map) {
> +		iounmap(window->hvwc_map);
> +		get_hvwc_mmio_bar(window, &busaddr_start, &len);
> +		release_mem_region((phys_addr_t)busaddr_start, len);
> +		window->hvwc_map =3D NULL;
> +	}
> +
> +	if (window->uwc_map) {
> +		iounmap(window->uwc_map);
> +		get_uwc_mmio_bar(window, &busaddr_start, &len);
> +		release_mem_region((phys_addr_t)busaddr_start, len);
> +		window->uwc_map =3D NULL;
> +	}
> +}
> +
> +/*
> + * Find the Hypervisor Window Context (HVWC) MMIO Base Address Region an=
d the
> + * OS/User Window Context (UWC) MMIO Base Address Region for the given
> window.
> + * Map these bus addresses and save the mapped kernel addresses in @wind=
ow.
> + */
> +int map_wc_mmio_bars(struct vas_window *window)
> +{
> +	int len;
> +	uint64_t start;
> +
> +	window->paste_kaddr =3D window->hvwc_map =3D window->uwc_map =3D NULL;
> +
> +	get_hvwc_mmio_bar(window, &start, &len);
> +	window->hvwc_map =3D map_mmio_region("HVWCM_Window", start, len);
> +
> +	get_uwc_mmio_bar(window, &start, &len);
> +	window->uwc_map =3D map_mmio_region("UWCM_Window", start, len);
> +
> +	if (!window->hvwc_map || !window->uwc_map) {
> +		unmap_wc_mmio_bars(window);
> +		return -1;
> +	}
> +
> +	return 0;
> +}
> +
> =C2=A0/* stub for now */
> =C2=A0int vas_window_reset(struct vas_instance *vinst, int winid)
> =C2=A0{

Re: [PATCH v3 05/10] VAS: Define helpers to init window context

[Michael Neuling]

On Thu, 2017-03-16 at 20:33 -0700, Sukadev Bhattiprolu wrote:
> Define helpers to initialize window context registers of the VAS
> hardware. These will be used in follow-on patches when opening/closing
> VAS windows.
>=20
> Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
> ---
> Changelog[v3]
> 	- Have caller, rather than init_xlate_regs() reset window regs
> 	=C2=A0=C2=A0so we don't reset any settings caller may already have set.
> 	- Translation mode should be 0x3 (0b11) not 0x11.
> 	- Skip initilaizing read-only registers NX_UTIL and NX_UTIL_SE
> 	- Skip initializing adder registers from UWC - they are already
> 	=C2=A0=C2=A0initialized from the HVWC.
> 	- Check winctx->user_win when setting translation registers
> ---
> =C2=A0drivers/misc/vas/vas-internal.h |=C2=A0=C2=A059 ++++++-
> =C2=A0drivers/misc/vas/vas-window.c=C2=A0=C2=A0=C2=A0| 334
> ++++++++++++++++++++++++++++++++++++++++
> =C2=A02 files changed, 390 insertions(+), 3 deletions(-)
>=20
> diff --git a/drivers/misc/vas/vas-internal.h b/drivers/misc/vas/vas-inter=
nal.h
> index 15b62e0..8e721df 100644
> --- a/drivers/misc/vas/vas-internal.h
> +++ b/drivers/misc/vas/vas-internal.h
> @@ -11,6 +11,7 @@
> =C2=A0#define VAS_INTERNAL_H
> =C2=A0#include <linux/atomic.h>
> =C2=A0#include <linux/idr.h>
> +#include <linux/io.h>
> =C2=A0#include <asm/vas.h>
> =C2=A0
> =C2=A0#ifdef CONFIG_PPC_4K_PAGES
> @@ -336,9 +337,6 @@ struct vas_window {
> =C2=A0	/* Feilds applicable only to receive windows */
> =C2=A0	enum vas_cop_type cop;
> =C2=A0	atomic_t num_txwins;
> -
> -	int32_t hwirq;
> -	uint64_t irq_port;

We are removing things already? :-)

> =C2=A0};
> =C2=A0
> =C2=A0/*
> @@ -392,4 +390,59 @@ struct vas_winctx {
> =C2=A0extern int vas_initialized;
> =C2=A0extern int vas_window_reset(struct vas_instance *vinst, int winid);
> =C2=A0extern struct vas_instance *find_vas_instance(int vasid);
> +
> +/*
> + * VREG(x):
> + * Expand a register's short name (eg: LPID) into two parameters:
> + *	- the register's short name in string form ("LPID"), and
> + *	- the name of the macro (eg: VAS_LPID_OFFSET), defining the
> + *	=C2=A0=C2=A0register's offset in the window context
> + */
> +#define VREG_SFX(n, s)	__stringify(n), VAS_##n##s
> +#define VREG(r)		VREG_SFX(r, _OFFSET)
> +
> +#ifndef vas_debug
> +static inline void vas_log_write(struct vas_window *win, char *name,
> +			void *regptr, uint64_t val)
> +{
> +	if (val)
> +		pr_err("%swin #%d: %s reg %p, val 0x%llx\n",
> +				win->tx_win ? "Tx" : "Rx", win->winid, name,
> +				regptr, val);
> +}
> +
> +#else	/* vas_debug */
> +
> +#define vas_log_write(win, name, reg, val)
> +
> +#endif	/* vas_debug */
> +
> +static inline void write_uwc_reg(struct vas_window *win, char *name,
> +			int32_t reg, uint64_t val)
> +{
> +	void *regptr;
> +
> +	regptr =3D win->uwc_map + reg;
> +	vas_log_write(win, name, regptr, val);
> +
> +	out_be64(regptr, val);
> +}
> +
> +static inline void write_hvwc_reg(struct vas_window *win, char *name,
> +			int32_t reg, uint64_t val)
> +{
> +	void *regptr;
> +
> +	regptr =3D win->hvwc_map + reg;
> +	vas_log_write(win, name, regptr, val);
> +
> +	out_be64(regptr, val);
> +}
> +
> +static inline uint64_t read_hvwc_reg(struct vas_window *win,
> +			char *name __maybe_unused, int32_t reg)
> +{
> +	return in_be64(win->hvwc_map+reg);
> +}
> +
> =C2=A0#endif
> diff --git a/drivers/misc/vas/vas-window.c b/drivers/misc/vas/vas-window.=
c
> index 32dd1d0..edf5c9f 100644
> --- a/drivers/misc/vas/vas-window.c
> +++ b/drivers/misc/vas/vas-window.c
> @@ -14,6 +14,8 @@
> =C2=A0#include <asm/vas.h>
> =C2=A0#include "vas-internal.h"
> =C2=A0
> +static int fault_winid;
> +
> =C2=A0/*
> =C2=A0 * Compute the paste address region for the window @window using th=
e
> =C2=A0 * ->win_base_addr and ->win_id_shift we got from device tree.
> @@ -138,6 +140,338 @@ int map_wc_mmio_bars(struct vas_window *window)
> =C2=A0	return 0;
> =C2=A0}
> =C2=A0
> +/*
> + * Reset all valid registers in the HV and OS/User Window Contexts for
> + * the window identified by @window.
> + *
> + * NOTE: We cannot really use a for loop to reset window context. Not al=
l
> + *	=C2=A0offsets in a window context are valid registers and the valid
> + *	=C2=A0registers are not sequential. And, we can only write to offsets
> + *	=C2=A0with valid registers (or is that only in Simics?).
> + */
> +void reset_window_regs(struct vas_window *window)
> +{
> +	write_hvwc_reg(window, VREG(LPID), 0ULL);
> +	write_hvwc_reg(window, VREG(PID), 0ULL);
> +	write_hvwc_reg(window, VREG(XLATE_MSR), 0ULL);
> +	write_hvwc_reg(window, VREG(XLATE_LPCR), 0ULL);
> +	write_hvwc_reg(window, VREG(XLATE_CTL), 0ULL);
> +	write_hvwc_reg(window, VREG(AMR), 0ULL);
> +	write_hvwc_reg(window, VREG(SEIDR), 0ULL);
> +	write_hvwc_reg(window, VREG(FAULT_TX_WIN), 0ULL);
> +	write_hvwc_reg(window, VREG(OSU_INTR_SRC_RA), 0ULL);
> +	write_hvwc_reg(window, VREG(HV_INTR_SRC_RA), 0ULL);
> +	write_hvwc_reg(window, VREG(PSWID), 0ULL);
> +	write_hvwc_reg(window, VREG(SPARE1), 0ULL);
> +	write_hvwc_reg(window, VREG(SPARE2), 0ULL);
> +	write_hvwc_reg(window, VREG(SPARE3), 0ULL);
> +	write_hvwc_reg(window, VREG(SPARE4), 0ULL);
> +	write_hvwc_reg(window, VREG(SPARE5), 0ULL);
> +	write_hvwc_reg(window, VREG(SPARE6), 0ULL);
> +	write_hvwc_reg(window, VREG(LFIFO_BAR), 0ULL);
> +	write_hvwc_reg(window, VREG(LDATA_STAMP_CTL), 0ULL);
> +	write_hvwc_reg(window, VREG(LDMA_CACHE_CTL), 0ULL);
> +	write_hvwc_reg(window, VREG(LRFIFO_PUSH), 0ULL);
> +	write_hvwc_reg(window, VREG(CURR_MSG_COUNT), 0ULL);
> +	write_hvwc_reg(window, VREG(LNOTIFY_AFTER_COUNT), 0ULL);
> +	write_hvwc_reg(window, VREG(LRX_WCRED), 0ULL);
> +	write_hvwc_reg(window, VREG(LRX_WCRED_ADDER), 0ULL);
> +	write_hvwc_reg(window, VREG(TX_WCRED), 0ULL);
> +	write_hvwc_reg(window, VREG(TX_WCRED_ADDER), 0ULL);
> +	write_hvwc_reg(window, VREG(LFIFO_SIZE), 0ULL);
> +	write_hvwc_reg(window, VREG(WINCTL), 0ULL);
> +	write_hvwc_reg(window, VREG(WIN_STATUS), 0ULL);
> +	write_hvwc_reg(window, VREG(WIN_CTX_CACHING_CTL), 0ULL);
> +	write_hvwc_reg(window, VREG(TX_RSVD_BUF_COUNT), 0ULL);
> +	write_hvwc_reg(window, VREG(LRFIFO_WIN_PTR), 0ULL);
> +	write_hvwc_reg(window, VREG(LNOTIFY_CTL), 0ULL);
> +	write_hvwc_reg(window, VREG(LNOTIFY_PID), 0ULL);
> +	write_hvwc_reg(window, VREG(LNOTIFY_LPID), 0ULL);
> +	write_hvwc_reg(window, VREG(LNOTIFY_TID), 0ULL);
> +	write_hvwc_reg(window, VREG(LNOTIFY_SCOPE), 0ULL);
> +	write_hvwc_reg(window, VREG(NX_UTIL_ADDER), 0ULL);
> +
> +	/* Skip read-only registers: NX_UTIL and NX_UTIL_SE */
> +
> +	/*
> +	=C2=A0* The send and receive window credit adder registers are also
> +	=C2=A0* accessible from HVWC and have been initialized above. We don't
> +	=C2=A0* need to initialize from the OS/User Window Context, so skip
> +	=C2=A0* following calls:
> +	=C2=A0*
> +	=C2=A0*	write_uwc_reg(window, VREG(TX_WCRED_ADDER), 0ULL);
> +	=C2=A0*	write_uwc_reg(window, VREG(LRX_WCRED_ADDER), 0ULL);
> +	=C2=A0*/
> +}
> +
> +/*
> + * Initialize window context registers related to Address Translation.
> + * These registers are common to send/receive windows although they
> + * differ for user/kernel windows. As we resolve the TODOs we may
> + * want to add fields to vas_winctx and move the initialization to
> + * init_vas_winctx_regs().
> + */
> +static void init_xlate_regs(struct vas_window *window, bool user_win)
> +{
> +	uint64_t lpcr, msr, val;
> +
> +	msr =3D mfmsr();
> +	WARN_ON_ONCE(!(msr & MSR_SF));

We don't support 32 bit userspace?  I would return an error rather than thi=
s.

> +
> +	val =3D 0ULL;
> +	if (user_win) {
> +		val =3D SET_FIELD(VAS_XLATE_MSR_DR, val, true);
> +		val =3D SET_FIELD(VAS_XLATE_MSR_TA, val, false);
> +		val =3D SET_FIELD(VAS_XLATE_MSR_PR, val, true);
> +		val =3D SET_FIELD(VAS_XLATE_MSR_US, val, false);
> +		val =3D SET_FIELD(VAS_XLATE_MSR_HV, val, true);
> +		val =3D SET_FIELD(VAS_XLATE_MSR_SF, val, true);
> +		val =3D SET_FIELD(VAS_XLATE_MSR_UV, val, false);
> +	} else {
> +		val =3D SET_FIELD(VAS_XLATE_MSR_DR, val, false);

kernel contexts don't go through the nestmmu?

> +		val =3D SET_FIELD(VAS_XLATE_MSR_TA, val, false);
> +		val =3D SET_FIELD(VAS_XLATE_MSR_PR, val, msr & MSR_PR);

I don't understand this.  It should just be 0 for the kernel.

> +		val =3D SET_FIELD(VAS_XLATE_MSR_US, val, false);
> +		val =3D SET_FIELD(VAS_XLATE_MSR_HV, val, true);
> +		val =3D SET_FIELD(VAS_XLATE_MSR_SF, val, true);
> +		val =3D SET_FIELD(VAS_XLATE_MSR_UV, val, false);
> +	}
> +	write_hvwc_reg(window, VREG(XLATE_MSR), val);
> +
> +	lpcr =3D mfspr(SPRN_LPCR);
> +	val =3D 0ULL;
> +	/*
> +	=C2=A0* NOTE: From Section 5.7.6.1 Segment Lookaside Buffer of the
> +	=C2=A0*	=C2=A0Power ISA, v2.07, Page size encoding is 0 =3D 4KB, 5 =3D =
64KB.
> +	=C2=A0*
> +	=C2=A0* NOTE: From Section 1.3.1, Address Translation Context of the
> +	=C2=A0*	=C2=A0Nest MMU Workbook, LPCR_SC should be 0 for Power9.
> +	=C2=A0*/
> +	val =3D SET_FIELD(VAS_XLATE_LPCR_PAGE_SIZE, val, 5);
> +	val =3D SET_FIELD(VAS_XLATE_LPCR_ISL, val, lpcr & LPCR_ISL);
> +	val =3D SET_FIELD(VAS_XLATE_LPCR_TC, val, lpcr & LPCR_TC);
> +	val =3D SET_FIELD(VAS_XLATE_LPCR_SC, val, 0);
> +	write_hvwc_reg(window, VREG(XLATE_LPCR), val);
> +
> +	/*
> +	=C2=A0* Section 1.3.1 (Address translation Context) of NMMU workbook.
> +	=C2=A0*	0b00	Hashed Page Table mode
> +	=C2=A0*	0b01	Reserved
> +	=C2=A0*	0b10	Radix on HPT - not supported in P9
> +	=C2=A0*	0b11	Radix on Radix (only mode supported in Linux on
> P9).

Linux supports hash on P9.

Does VAS only support radix?  If so you should error out if we are booted h=
ash.

> +	=C2=A0*/
> +	val =3D 0ULL;
> +	val =3D SET_FIELD(VAS_XLATE_MODE, val, 0x3);

You can use radix_enabled() to set this for hash vs radix.

> +	write_hvwc_reg(window, VREG(XLATE_CTL), val);
> +
> +	/*
> +	=C2=A0* TODO: Can we mfspr(AMR) even for user windows?
> +	=C2=A0*/
> +	val =3D 0ULL;
> +	val =3D SET_FIELD(VAS_AMR, val, mfspr(SPRN_AMR));
> +	write_hvwc_reg(window, VREG(AMR), val);
> +
> +	/*
> +	=C2=A0* TODO: Assuming Secure Executable ID Register (SEIDR) is only us=
ed
> +	=C2=A0*	=C2=A0in the ultravisor mode. Since MSR(UV) is 0 for now, set
> SEIDR
> +	=C2=A0*	=C2=A0to 0 as well, although we should 'mfspr(SEIDR)' at some
> point.
> +	=C2=A0*/
> +	val =3D 0ULL;
> +	val =3D SET_FIELD(VAS_SEIDR, val, 0);
> +	write_hvwc_reg(window, VREG(SEIDR), val);
> +}
> +
> +/*
> + * Initialize Reserved Send Buffer Count for the send window. It involve=
s
> + * writing to the register, reading it back to confirm that the hardware
> + * has enough buffers to reserve. See section 1.3.1.2.1 of VAS workbook.
> + *
> + * Since we can only make a best-effort attempt to fulfill the request,
> + * we don't return any errors if we cannot.
> + *
> + * TODO: Reserved (aka dedicated) send buffers are not supported yet.
> + */
> +static void init_rsvd_tx_buf_count(struct vas_window *txwin,
> +				struct vas_winctx *winctx)
> +{
> +	write_hvwc_reg(txwin, VREG(TX_RSVD_BUF_COUNT), 0ULL);
> +}
> +
> +/*
> + * Compute the log2() of the FIFO size expressed as kilobytes. It is int=
ended
> + * to be used to initialize the Local FIFO Size Register defined in Sect=
ion
> + * 3.14.25 of the VAS Workbook.

There is a ilog2() function..

> + */
> +static int map_fifo_size_to_reg(int fifo_size)
> +{
> +	int kb;
> +	int map;
> +
> +	kb =3D fifo_size / 1024;
> +	if (!kb)
> +		kb =3D 1;
> +
> +	map =3D -1;
> +	while (kb) {
> +		kb >>=3D 1;
> +		map++;
> +	}
> +
> +	return map;
> +}
> +

Re: [PATCH v3 06/10] VAS: Define helpers to alloc/free windows

[Michael Neuling]

On Thu, 2017-03-16 at 20:33 -0700, Sukadev Bhattiprolu wrote:
> Define helpers to allocate/free VAS window objects. These will
> be used in follow-on patches when opening/closing windows.
>=20
> Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
> ---
> =C2=A0drivers/misc/vas/vas-window.c | 74 ++++++++++++++++++++++++++++++++=
+++++++++-
> -
> =C2=A01 file changed, 72 insertions(+), 2 deletions(-)
>=20
> diff --git a/drivers/misc/vas/vas-window.c b/drivers/misc/vas/vas-window.=
c
> index edf5c9f..9233bf5 100644
> --- a/drivers/misc/vas/vas-window.c
> +++ b/drivers/misc/vas/vas-window.c
> @@ -119,7 +119,7 @@ static void unmap_wc_mmio_bars(struct vas_window *win=
dow)
> =C2=A0 * OS/User Window Context (UWC) MMIO Base Address Region for the gi=
ven
> window.
> =C2=A0 * Map these bus addresses and save the mapped kernel addresses in =
@window.
> =C2=A0 */
> -int map_wc_mmio_bars(struct vas_window *window)
> +static int map_wc_mmio_bars(struct vas_window *window)
> =C2=A0{
> =C2=A0	int len;
> =C2=A0	uint64_t start;
> @@ -472,8 +472,78 @@ int init_winctx_regs(struct vas_window *window, stru=
ct
> vas_winctx *winctx)
> =C2=A0	return 0;
> =C2=A0}
> =C2=A0
> -/* stub for now */
> +DEFINE_SPINLOCK(vas_ida_lock);
> +
> +void vas_release_window_id(struct ida *ida, int winid)
> +{
> +	spin_lock(&vas_ida_lock);
> +	ida_remove(ida, winid);
> +	spin_unlock(&vas_ida_lock);
> +}
> +
> +int vas_assign_window_id(struct ida *ida)
> +{
> +	int rc, winid;
> +
> +	rc =3D ida_pre_get(ida, GFP_KERNEL);
> +	if (!rc)
> +		return -EAGAIN;
> +
> +	spin_lock(&vas_ida_lock);
> +	rc =3D ida_get_new_above(ida, 0, &winid);
> +	spin_unlock(&vas_ida_lock);
> +
> +	if (rc)
> +		return rc;
> +
> +	if (winid > VAS_MAX_WINDOWS_PER_CHIP) {
> +		pr_err("VAS: Too many (%d) open windows\n", winid);
> +		vas_release_window_id(ida, winid);
> +		return -EAGAIN;
> +	}
> +
> +	return winid;
> +}
> +
> +static void vas_window_free(struct vas_window *window)
> +{
> +	unmap_wc_mmio_bars(window);
> +	kfree(window->paste_addr_name);
> +	kfree(window);
> +}
> +
> +static struct vas_window *vas_window_alloc(struct vas_instance *vinst, i=
nt
> id)
> +{
> +	struct vas_window *window;
> +
> +	window =3D kzalloc(sizeof(*window), GFP_KERNEL);
> +	if (!window)
> +		return NULL;
> +
> +	window->vinst =3D vinst;
> +	window->winid =3D id;
> +
> +	if (map_wc_mmio_bars(window))
> +		goto out_free;
> +
> +	return window;
> +
> +out_free:
> +	kfree(window);
> +	return NULL;
> +}
> +
> =C2=A0int vas_window_reset(struct vas_instance *vinst, int winid)
>=20

This interface seems a little weird to me. Needing an alloc in a hardware r=
eset
path seems a bit strange.

Maybe the data structures are the issue.  A window is a hardware construct.=
=20
Something that uses it should probably be called something else like a cont=
ext.=20
Something that references a window should just be the vas_instance + winid.=
=20

You should be able to reset this hardware window by referencing structures
already allocated.  Something associated with the struct vas_instance.

Mikey

> =C2=A0{
> +	struct vas_window *window;
> +
> +	window =3D vas_window_alloc(vinst, winid);
> +	if (!window)
> +		return -ENOMEM;
> +
> +	reset_window_regs(window);
> +
> +	vas_window_free(window);
> +
> =C2=A0	return 0;
> =C2=A0}

Re: [PATCH v3 04/10] VAS: Define helpers for access MMIO regions

[Sukadev Bhattiprolu]

Michael Neuling [michael.neuling@au1.ibm.com] wrote:
> > +static inline void get_uwc_mmio_bar(struct vas_window *window,
> > +			uint64_t *start, int *len)
> > +{
> > +	uint64_t pbaddr;
> > +
> > +	pbaddr = window->vinst->uwc_bar_start;
> > +	*start = pbaddr + window->winid * VAS_UWC_SIZE;
> > +	*len = VAS_UWC_SIZE;
> 
> I'm not sure this works for 4K pages since VAS_UWC_SIZE = PAGE_SIZE but in
> reality I think it's always 64K.  Right?

I believe the idea is that each user process will have its OS/User Window
context on a separate page - to provide proper protection of the window
context between processes. So, the OS/User Window context size _is_ 
dependent on page size.

The page size is set to 64K in init_north_ctl() in skiboot but can be
set to 4K as well. The amount of memory allocated for the OS/User Window
Context MMIO BAR (init_uwcm() in skiboot) also depends on the page size
256MB for 4K or 4GB for 64K.

> 
> Seem like we are mixing pages sizes and hardware sizes here.
> 
> (I realise 4K isn't supported yet, but....)

Yes, only 64K page-size for now.

Thanks,

Sukadev

Re: [PATCH v3 03/10] VAS: Define vas_init() and vas_exit()

[Sukadev Bhattiprolu]

Michael Neuling [michael.neuling@au1.ibm.com] wrote:
> On Thu, 2017-03-16 at 20:33 -0700, Sukadev Bhattiprolu wrote:
> > Implement vas_init() and vas_exit() functions for a new VAS module.
> > This VAS module is essentially a library for other device drivers
> > and kernel users of the NX coprocessors like NX-842 and NX-GZIP.
> >=20
> > Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
> > ---
> > Changelog[v3]:
> > 	- Zero vas_instances memory on allocation
> > 	- [Haren Myneni] Fix description in Kconfig
> > Changelog[v2]:
> > 	- Get HVWC, UWC and window address parameters from device tree.
> > ---
> > =A0MAINTAINERS=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=
=A0=A0|=A0=A0=A08 ++-
> > =A0arch/powerpc/include/asm/reg.h=A0=A0|=A0=A0=A01 +
> > =A0drivers/misc/Kconfig=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0|=A0=A0=A01 +
> > =A0drivers/misc/Makefile=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0|=A0=A0=A01 +
> > =A0drivers/misc/vas/Kconfig=A0=A0=A0=A0=A0=A0=A0=A0|=A0=A021 ++++++
> > =A0drivers/misc/vas/Makefile=A0=A0=A0=A0=A0=A0=A0|=A0=A0=A03 +
> > =A0drivers/misc/vas/vas-internal.h |=A0=A0=A03 +
> > =A0drivers/misc/vas/vas-window.c=A0=A0=A0|=A0=A019 +++++
> > =A0drivers/misc/vas/vas.c=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0| 155
> > ++++++++++++++++++++++++++++++++++++++++
> > =A09 files changed, 210 insertions(+), 2 deletions(-)
> > =A0create mode 100644 drivers/misc/vas/Kconfig
> > =A0create mode 100644 drivers/misc/vas/Makefile
> > =A0create mode 100644 drivers/misc/vas/vas-window.c
> > =A0create mode 100644 drivers/misc/vas/vas.c
> >=20
> > diff --git a/MAINTAINERS b/MAINTAINERS
> > index 2a910c9..4037252 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -3673,8 +3673,6 @@ F:	arch/powerpc/platforms/powernv/pci-cxl.c
> > =A0F:	drivers/misc/cxl/
> > =A0F:	include/misc/cxl*
> > =A0F:	include/uapi/misc/cxl.h
> > -F:	Documentation/powerpc/cxl.txt
> > -F:	Documentation/ABI/testing/sysfs-class-cxl
>=20
> err?

Yeah, something got messed up here and

>=20
> > =A0CXLFLASH (IBM Coherent Accelerator Processor Interface CAPI Flash) S=
CSI
> > DRIVER
> > =A0M:	Manoj N. Kumar <manoj@linux.vnet.ibm.com>
> > @@ -3686,6 +3684,12 @@ F:	drivers/scsi/cxlflash/
> > =A0F:	include/uapi/scsi/cxlflash_ioctls.h
> > =A0F:	Documentation/powerpc/cxlflash.txt
> > =A0
> > +VAS (IBM Virtual Accelerator Switch) DRIVER
> > +M:	Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
> > +L:	linuxppc-dev@lists.ozlabs.org
> > +S:	Supported
> > +F:	drivers/misc/vas/
> > +
>=20
> This was already added in patch 1.

here. Will fix

>=20
> > =A0STMMAC ETHERNET DRIVER
> > =A0M:	Giuseppe Cavallaro <peppe.cavallaro@st.com>
> > =A0M:	Alexandre Torgue <alexandre.torgue@st.com>
> > diff --git a/arch/powerpc/include/asm/reg.h b/arch/powerpc/include/asm/=
reg.h
> > index fc879fd..7a45ff7 100644
> > --- a/arch/powerpc/include/asm/reg.h
> > +++ b/arch/powerpc/include/asm/reg.h
> > @@ -1225,6 +1225,7 @@
> > =A0#define PVR_POWER8E	0x004B
> > =A0#define PVR_POWER8NVL	0x004C
> > =A0#define PVR_POWER8	0x004D
> > +#define PVR_POWER9	0x004E
>=20
> Can you send this up separately? =20

Sure.

>=20
> > =A0#define PVR_BE		0x0070
> > =A0#define PVR_PA6T	0x0090
> > =A0
> > diff --git a/drivers/misc/Kconfig b/drivers/misc/Kconfig
> > index c290990..97d652e 100644
> > --- a/drivers/misc/Kconfig
> > +++ b/drivers/misc/Kconfig
> > @@ -783,4 +783,5 @@ source "drivers/misc/mic/Kconfig"
> > =A0source "drivers/misc/genwqe/Kconfig"
> > =A0source "drivers/misc/echo/Kconfig"
> > =A0source "drivers/misc/cxl/Kconfig"
> > +source "drivers/misc/vas/Kconfig"
> > =A0endmenu
> > diff --git a/drivers/misc/Makefile b/drivers/misc/Makefile
> > index 7a3ea89..5201ffd 100644
> > --- a/drivers/misc/Makefile
> > +++ b/drivers/misc/Makefile
> > @@ -53,6 +53,7 @@ obj-$(CONFIG_GENWQE)		+=3D genwqe/
> > =A0obj-$(CONFIG_ECHO)		+=3D echo/
> > =A0obj-$(CONFIG_VEXPRESS_SYSCFG)	+=3D vexpress-syscfg.o
> > =A0obj-$(CONFIG_CXL_BASE)		+=3D cxl/
> > +obj-$(CONFIG_VAS)		+=3D vas/
> > =A0obj-$(CONFIG_PANEL)=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0+=3D panel=
=2Eo
> > =A0
> > =A0lkdtm-$(CONFIG_LKDTM)		+=3D lkdtm_core.o
> > diff --git a/drivers/misc/vas/Kconfig b/drivers/misc/vas/Kconfig
> > new file mode 100644
> > index 0000000..43cedda
> > --- /dev/null
> > +++ b/drivers/misc/vas/Kconfig
> > @@ -0,0 +1,21 @@
> > +#
> > +# IBM Virtual Accelarator Switchboard (VAS) compatible devices
> > +#depends on PPC_POWERNV && PCI_MSI && EEH
> > +#
> > +
> > +config VAS
> > +	tristate "Support for IBM Virtual Accelerator Switchboard (VAS)"
> > +	depends on PPC_POWERNV
> > +	default n
> > +	help
> > +	=A0=A0Select this option to enable driver support for IBM Virtual
> > +	=A0=A0Accelerator Switchboard (VAS).
> > +
> > +	=A0=A0VAS allows accelerators in co processors like NX-842 to be
> > +	=A0=A0directly available to a user process. This driver enables
> > +	=A0=A0userspace programs to access these accelerators via device
> > +	=A0=A0nodes like /dev/crypto/nx-gzip.
>=20
> I though this was kernel only users for now?

Yes, its only kernel for now. Will drop the last sentence.
>=20
> > +
> > +	=A0=A0VAS adapters are found in POWER9 based systems.
> > +
> > +	=A0=A0If unsure, say N.
> > diff --git a/drivers/misc/vas/Makefile b/drivers/misc/vas/Makefile
> > new file mode 100644
> > index 0000000..7dd7139
> > --- /dev/null
> > +++ b/drivers/misc/vas/Makefile
> > @@ -0,0 +1,3 @@
> > +ccflags-y			:=3D $(call cc-disable-warning, unused-const-
> > variable)
> > +ccflags-$(CONFIG_PPC_WERROR)	+=3D -Werror
> > +obj-$(CONFIG_VAS)		+=3D vas.o vas-window.o
> > diff --git a/drivers/misc/vas/vas-internal.h b/drivers/misc/vas/vas-int=
ernal.h
> > index ce48f14..15b62e0 100644
> > --- a/drivers/misc/vas/vas-internal.h
> > +++ b/drivers/misc/vas/vas-internal.h
>=20
> Just call this file vas.h.=20
>=20
> Internal one is here.  Kernel api one lives in include/misc.  User API li=
ves in
> ./include/uapi/misc. =A0
>=20
> eg. CXL does this:
>=20
> % find . -name cxl.h
> ./include/uapi/misc/cxl.h
> ./include/misc/cxl.h
> ./drivers/misc/cxl/cxl.h

Sure, makes sense.

>=20
>=20
> > @@ -389,4 +389,7 @@ struct vas_winctx {
> > =A0	enum vas_notify_after_count notify_after_count;
> > =A0};
> > =A0
> > +extern int vas_initialized;
>=20
> I would probably make this a function.

Ok.

>=20
> > +extern int vas_window_reset(struct vas_instance *vinst, int winid);
> > +extern struct vas_instance *find_vas_instance(int vasid);
> > =A0#endif
> > diff --git a/drivers/misc/vas/vas-window.c b/drivers/misc/vas/vas-windo=
w.c
> > new file mode 100644
> > index 0000000..468f3bf
> > --- /dev/null
> > +++ b/drivers/misc/vas/vas-window.c
> > @@ -0,0 +1,19 @@
> > +/*
> > + * Copyright 2016 IBM Corp.
> > + *
> > + * This program is free software; you can redistribute it and/or
> > + * modify it under the terms of the GNU General Public License
> > + * as published by the Free Software Foundation; either version
> > + * 2 of the License, or (at your option) any later version.
> > + */
> > +
> > +#include <linux/types.h>
> > +#include <linux/mutex.h>
> > +#include <asm/vas.h>
> > +#include "vas-internal.h"
> > +
> > +/* stub for now */
> > +int vas_window_reset(struct vas_instance *vinst, int winid)
> > +{
> > +	return 0;
>=20
> No biggy, but best to return an error if it's just a stub in case sometim=
e tries
> to use it.

Sure.

Thanks

Sukadev

Re: [PATCH v3 01/10] VAS: Define macros, register fields and structures

[Sukadev Bhattiprolu]

Michael Neuling [mikey@neuling.org] wrote:
> On Thu, 2017-03-16 at 20:33 -0700, Sukadev Bhattiprolu wrote:
> > Define macros for the VAS hardware registers and bit-fields as well
> > as couple of data structures needed by the VAS driver.
> >=20
> > > Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
> > ---
> > Changelog[v3]
> > 	- Rename winctx->pid to winctx->pidr to reflect that its a value
> > 	=A0=A0from the PID register (SPRN_PID), not the linux process id.
> > 	- Make it easier to split header into kernel/user parts
> > 	- To keep user interface simple, use macros rather than enum for
> > 	=A0=A0the threshold-control modes.
> > 	- Add a pid field to struct vas_window - needed for user space
> > 	=A0=A0send windows.
> >=20
> > Changelog[v2]
> > 	- Add an overview of VAS in vas-internal.h
> > 	- Get window context parameters from device tree and drop
> > 	=A0=A0unnecessary macros.
> > ---
> > =A0MAINTAINERS=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=
=A0=A0|=A0=A0=A06 +
> > =A0arch/powerpc/include/asm/vas.h=A0=A0|=A0=A043 +++++
> > =A0drivers/misc/vas/vas-internal.h | 392 ++++++++++++++++++++++++++++++=
++++++++++
>=20
> This is going to have to go through gregkh/lkml if it's drivers/misc.  yo=
u'll at
> least need gregkh's ack/ok before mpe will take them (which is what we di=
d for
> CAPI).
>=20
> We might want to keep this in arch/powerpc but I'm not sure.
>=20

We will have device nodes accessible to user space so put it here and can
copy Gregkh next time. But let me know if we should move to arch/powerpc.

> > =A03 files changed, 441 insertions(+)
> > =A0create mode 100644 arch/powerpc/include/asm/vas.h
> > =A0create mode 100644 drivers/misc/vas/vas-internal.h
> >=20
> <snip>
> > +
> > +/*
> > + * Overview of Virtual Accelerator Switchboard (VAS).
> > + *
> > + * VAS is a hardware "switchboard" that allows senders and receivers to
> > + * exchange messages with _minimal_ kernel involvment. The receivers a=
re
> > + * typically NX coprocessor engines that perform compression or encryp=
tion
> > + * in hardware, but receivers can also be other software threads.
> > + *
> > + * Senders are user/kernel threads that submit compression/encryption =
or
> > + * other requests to the receivers. Senders must format their messages=
 as
> > + * Coprocessor Request Blocks (CRB)s and submit them using the instruc=
tions
> > + * "copy" and "paste" which were introduced in Power9.
> > + *
> > + * A Power node can have (upto?) 8 Power chips. There is one instance =
of
> > + * VAS in each Power9 chip. Each instance of VAS has 64K windows or po=
rts,
> > + * Senders and receivers must each connect to a separate window before=
 they
> > + * can exchange messages through the switchboard.
> > + *
> > + * Each window is described by two types of window contexts:
> > + *
> > > + *	Hypervisor Window Context (HVWC) of size VAS_HVWC_SIZE bytes
> > + *
> > > + *	OS/User Window Context (UWC) of size VAS_UWC_SIZE bytes.
> > + *
> > + * A window context can be viewed as a set of 64-bit registers. The se=
ttings
> > + * in these registers configure/control/determine the behavior of the =
VAS
> > + * hardware when messages are sent/received through the window. The re=
gisters
> > + * in the HVWC are configured by the kernel while the registers in the=
 UWC can
> > + * be configured by the kernel or by the user space application that i=
s using
> > + * the window.
> > + *
> > + * The HVWCs for all windows on a specific instance of VAS are in a co=
ntiguous
> > + * range of hardware addresses or Base address region (BAR) referred t=
o as the
> > + * HVWC BAR for the instance. Similarly the UWCs for all windows on an=
 instance
> > + * are referred to as the UWC BAR for the instance. The two BARs for e=
ach
> > + * instance are defined Power9 MMIO Ranges spreadsheet and available t=
o the
> > + * kernel the device tree as follows:
> > + *
> > > + *	/proc/device-tree/xscom@.../vas@.../hvwc-bar-start
> > > + *	/proc/device-tree/xscom@.../vas@.../hvwc-bar-size
> > > + *	/proc/device-tree/xscom@.../vas@.../uwc-bar-start
> > + *	/proc/device-tree/xscom@.../vas@.../uwc-bar-size
>=20
> should these just be reg properties?

I guess they could. Will try that
>=20
> > + *
> > + * The kernel maps these two hardware address regions into the kernel =
address
> > + * space (hvwc_map and uwc_map) and accesses the window contexts of a =
specific
> > + * window using:
> > + *
> > > + *	=A0hvwc =3D hvwc_map + winid * VAS_HVWC_SIZE.
> > > + *	=A0uwc =3D uwc_map + winid * VAS_UWC_SIZE.
> > + *
> > + * where winid is the window index (0..64K).
> > + *
> > + * Note that the window contexts are used to "configure" the windows. =
In
> > + * addition to this configuration address, each _send_ window also has=
 a
> > + * unique hardware address, referred to as the "paste-address" to whic=
h the
> > + * sender must "paste" the message (CRB) they wish to submit. This har=
dware
> > + * paste address for window can be computed from the following nodes i=
n the
> > + * device tree:
> > + *
> > > + *	/proc/device-tree/xscom@.../vas@.../window-base
> > + *	/proc/device-tree/xscom@.../vas@.../window-shift
>=20
> Same here with reg properties.

ok

>=20
> <snip>=20
> > +struct vas_winctx {
> <snip>
> > > +	int lpid;
> > +	int pidr;		/* value from SPRN_PID, not linux pid */
>=20
> I'm surprised we have a copy of these here.  They should be accessed from=
 the
> context we are attaching to, rather than copied here... but I've not look=
ed at
> the rest of the code yet...

struct vas_winctx is just a convenient container for the window context.
It gets initialized based on type/use of window (user/kernel send/receive).
The lpid/pid are set by callers of VAS kernel interfaces.

Sukadev

Re: [PATCH v3 05/10] VAS: Define helpers to init window context

[Sukadev Bhattiprolu]

Michael Neuling [mikey@neuling.org] wrote:
> On Thu, 2017-03-16 at 20:33 -0700, Sukadev Bhattiprolu wrote:
> > =A0#ifdef CONFIG_PPC_4K_PAGES
> > @@ -336,9 +337,6 @@ struct vas_window {
> > =A0	/* Feilds applicable only to receive windows */
> > =A0	enum vas_cop_type cop;
> > =A0	atomic_t num_txwins;
> > -
> > -	int32_t hwirq;
> > -	uint64_t irq_port;
>=20
> We are removing things already? :-)

:-) They are needed when we add support for user windows will
remove them in the earlier patch.=20

>=20
> > =A0};
> > =A0
> > =A0/*
> > @@ -392,4 +390,59 @@ struct vas_winctx {
> > =A0extern int vas_initialized;
> > =A0extern int vas_window_reset(struct vas_instance *vinst, int winid);
> > =A0extern struct vas_instance *find_vas_instance(int vasid);
> > +
> > +/*
> > + * VREG(x):
> > + * Expand a register's short name (eg: LPID) into two parameters:
> > + *	- the register's short name in string form ("LPID"), and
> > + *	- the name of the macro (eg: VAS_LPID_OFFSET), defining the
> > + *	=A0=A0register's offset in the window context
> > + */
> > +#define VREG_SFX(n, s)	__stringify(n), VAS_##n##s
> > +#define VREG(r)		VREG_SFX(r, _OFFSET)
> > +
> > +#ifndef vas_debug
> > +static inline void vas_log_write(struct vas_window *win, char *name,
> > +			void *regptr, uint64_t val)
> > +{
> > +	if (val)
> > +		pr_err("%swin #%d: %s reg %p, val 0x%llx\n",
> > +				win->tx_win ? "Tx" : "Rx", win->winid, name,
> > +				regptr, val);
> > +}
> > +
> > +#else	/* vas_debug */
> > +
> > +#define vas_log_write(win, name, reg, val)
> > +
> > +#endif	/* vas_debug */
> > +
> > +static inline void write_uwc_reg(struct vas_window *win, char *name,
> > +			int32_t reg, uint64_t val)
> > +{
> > +	void *regptr;
> > +
> > +	regptr =3D win->uwc_map + reg;
> > +	vas_log_write(win, name, regptr, val);
> > +
> > +	out_be64(regptr, val);
> > +}
> > +
> > +static inline void write_hvwc_reg(struct vas_window *win, char *name,
> > +			int32_t reg, uint64_t val)
> > +{
> > +	void *regptr;
> > +
> > +	regptr =3D win->hvwc_map + reg;
> > +	vas_log_write(win, name, regptr, val);
> > +
> > +	out_be64(regptr, val);
> > +}
> > +
> > +static inline uint64_t read_hvwc_reg(struct vas_window *win,
> > +			char *name __maybe_unused, int32_t reg)
> > +{
> > +	return in_be64(win->hvwc_map+reg);
> > +}
> > +
> > =A0#endif
> > diff --git a/drivers/misc/vas/vas-window.c b/drivers/misc/vas/vas-windo=
w.c
> > index 32dd1d0..edf5c9f 100644
> > --- a/drivers/misc/vas/vas-window.c
> > +++ b/drivers/misc/vas/vas-window.c
> > @@ -14,6 +14,8 @@
> > =A0#include <asm/vas.h>
> > =A0#include "vas-internal.h"
> > =A0
> > +static int fault_winid;
> > +
> > =A0/*
> > =A0 * Compute the paste address region for the window @window using the
> > =A0 * ->win_base_addr and ->win_id_shift we got from device tree.
> > @@ -138,6 +140,338 @@ int map_wc_mmio_bars(struct vas_window *window)
> > =A0	return 0;
> > =A0}
> > =A0
> > +/*
> > + * Reset all valid registers in the HV and OS/User Window Contexts for
> > + * the window identified by @window.
> > + *
> > + * NOTE: We cannot really use a for loop to reset window context. Not =
all
> > + *	=A0offsets in a window context are valid registers and the valid
> > + *	=A0registers are not sequential. And, we can only write to offsets
> > + *	=A0with valid registers (or is that only in Simics?).
> > + */
> > +void reset_window_regs(struct vas_window *window)
> > +{
> > +	write_hvwc_reg(window, VREG(LPID), 0ULL);
> > +	write_hvwc_reg(window, VREG(PID), 0ULL);
> > +	write_hvwc_reg(window, VREG(XLATE_MSR), 0ULL);
> > +	write_hvwc_reg(window, VREG(XLATE_LPCR), 0ULL);
> > +	write_hvwc_reg(window, VREG(XLATE_CTL), 0ULL);
> > +	write_hvwc_reg(window, VREG(AMR), 0ULL);
> > +	write_hvwc_reg(window, VREG(SEIDR), 0ULL);
> > +	write_hvwc_reg(window, VREG(FAULT_TX_WIN), 0ULL);
> > +	write_hvwc_reg(window, VREG(OSU_INTR_SRC_RA), 0ULL);
> > +	write_hvwc_reg(window, VREG(HV_INTR_SRC_RA), 0ULL);
> > +	write_hvwc_reg(window, VREG(PSWID), 0ULL);
> > +	write_hvwc_reg(window, VREG(SPARE1), 0ULL);
> > +	write_hvwc_reg(window, VREG(SPARE2), 0ULL);
> > +	write_hvwc_reg(window, VREG(SPARE3), 0ULL);
> > +	write_hvwc_reg(window, VREG(SPARE4), 0ULL);
> > +	write_hvwc_reg(window, VREG(SPARE5), 0ULL);
> > +	write_hvwc_reg(window, VREG(SPARE6), 0ULL);
> > +	write_hvwc_reg(window, VREG(LFIFO_BAR), 0ULL);
> > +	write_hvwc_reg(window, VREG(LDATA_STAMP_CTL), 0ULL);
> > +	write_hvwc_reg(window, VREG(LDMA_CACHE_CTL), 0ULL);
> > +	write_hvwc_reg(window, VREG(LRFIFO_PUSH), 0ULL);
> > +	write_hvwc_reg(window, VREG(CURR_MSG_COUNT), 0ULL);
> > +	write_hvwc_reg(window, VREG(LNOTIFY_AFTER_COUNT), 0ULL);
> > +	write_hvwc_reg(window, VREG(LRX_WCRED), 0ULL);
> > +	write_hvwc_reg(window, VREG(LRX_WCRED_ADDER), 0ULL);
> > +	write_hvwc_reg(window, VREG(TX_WCRED), 0ULL);
> > +	write_hvwc_reg(window, VREG(TX_WCRED_ADDER), 0ULL);
> > +	write_hvwc_reg(window, VREG(LFIFO_SIZE), 0ULL);
> > +	write_hvwc_reg(window, VREG(WINCTL), 0ULL);
> > +	write_hvwc_reg(window, VREG(WIN_STATUS), 0ULL);
> > +	write_hvwc_reg(window, VREG(WIN_CTX_CACHING_CTL), 0ULL);
> > +	write_hvwc_reg(window, VREG(TX_RSVD_BUF_COUNT), 0ULL);
> > +	write_hvwc_reg(window, VREG(LRFIFO_WIN_PTR), 0ULL);
> > +	write_hvwc_reg(window, VREG(LNOTIFY_CTL), 0ULL);
> > +	write_hvwc_reg(window, VREG(LNOTIFY_PID), 0ULL);
> > +	write_hvwc_reg(window, VREG(LNOTIFY_LPID), 0ULL);
> > +	write_hvwc_reg(window, VREG(LNOTIFY_TID), 0ULL);
> > +	write_hvwc_reg(window, VREG(LNOTIFY_SCOPE), 0ULL);
> > +	write_hvwc_reg(window, VREG(NX_UTIL_ADDER), 0ULL);
> > +
> > +	/* Skip read-only registers: NX_UTIL and NX_UTIL_SE */
> > +
> > +	/*
> > +	=A0* The send and receive window credit adder registers are also
> > +	=A0* accessible from HVWC and have been initialized above. We don't
> > +	=A0* need to initialize from the OS/User Window Context, so skip
> > +	=A0* following calls:
> > +	=A0*
> > +	=A0*	write_uwc_reg(window, VREG(TX_WCRED_ADDER), 0ULL);
> > +	=A0*	write_uwc_reg(window, VREG(LRX_WCRED_ADDER), 0ULL);
> > +	=A0*/
> > +}
> > +
> > +/*
> > + * Initialize window context registers related to Address Translation.
> > + * These registers are common to send/receive windows although they
> > + * differ for user/kernel windows. As we resolve the TODOs we may
> > + * want to add fields to vas_winctx and move the initialization to
> > + * init_vas_winctx_regs().
> > + */
> > +static void init_xlate_regs(struct vas_window *window, bool user_win)
> > +{
> > +	uint64_t lpcr, msr, val;
> > +
> > +	msr =3D mfmsr();
> > +	WARN_ON_ONCE(!(msr & MSR_SF));
>=20
> We don't support 32 bit userspace?  I would return an error rather than t=
his.

I will return an error and add a TODO for 32-bit.

>=20
> > +
> > +	val =3D 0ULL;
> > +	if (user_win) {
> > +		val =3D SET_FIELD(VAS_XLATE_MSR_DR, val, true);
> > +		val =3D SET_FIELD(VAS_XLATE_MSR_TA, val, false);
> > +		val =3D SET_FIELD(VAS_XLATE_MSR_PR, val, true);
> > +		val =3D SET_FIELD(VAS_XLATE_MSR_US, val, false);
> > +		val =3D SET_FIELD(VAS_XLATE_MSR_HV, val, true);
> > +		val =3D SET_FIELD(VAS_XLATE_MSR_SF, val, true);
> > +		val =3D SET_FIELD(VAS_XLATE_MSR_UV, val, false);
> > +	} else {
> > +		val =3D SET_FIELD(VAS_XLATE_MSR_DR, val, false);
>=20
> kernel contexts don't go through the nestmmu?

I think so, but will check with Alistair/Ben.

>=20
> > +		val =3D SET_FIELD(VAS_XLATE_MSR_TA, val, false);
> > +		val =3D SET_FIELD(VAS_XLATE_MSR_PR, val, msr & MSR_PR);
>=20
> I don't understand this.  It should just be 0 for the kernel.

Ok.

>=20
> > +		val =3D SET_FIELD(VAS_XLATE_MSR_US, val, false);
> > +		val =3D SET_FIELD(VAS_XLATE_MSR_HV, val, true);
> > +		val =3D SET_FIELD(VAS_XLATE_MSR_SF, val, true);
> > +		val =3D SET_FIELD(VAS_XLATE_MSR_UV, val, false);
> > +	}
> > +	write_hvwc_reg(window, VREG(XLATE_MSR), val);
> > +
> > +	lpcr =3D mfspr(SPRN_LPCR);
> > +	val =3D 0ULL;
> > +	/*
> > +	=A0* NOTE: From Section 5.7.6.1 Segment Lookaside Buffer of the
> > +	=A0*	=A0Power ISA, v2.07, Page size encoding is 0 =3D 4KB, 5 =3D 64KB.
> > +	=A0*
> > +	=A0* NOTE: From Section 1.3.1, Address Translation Context of the
> > +	=A0*	=A0Nest MMU Workbook, LPCR_SC should be 0 for Power9.
> > +	=A0*/
> > +	val =3D SET_FIELD(VAS_XLATE_LPCR_PAGE_SIZE, val, 5);
> > +	val =3D SET_FIELD(VAS_XLATE_LPCR_ISL, val, lpcr & LPCR_ISL);
> > +	val =3D SET_FIELD(VAS_XLATE_LPCR_TC, val, lpcr & LPCR_TC);
> > +	val =3D SET_FIELD(VAS_XLATE_LPCR_SC, val, 0);
> > +	write_hvwc_reg(window, VREG(XLATE_LPCR), val);
> > +
> > +	/*
> > +	=A0* Section 1.3.1 (Address translation Context) of NMMU workbook.
> > +	=A0*	0b00	Hashed Page Table mode
> > +	=A0*	0b01	Reserved
> > +	=A0*	0b10	Radix on HPT - not supported in P9
> > +	=A0*	0b11	Radix on Radix (only mode supported in Linux on
> > P9).
>=20
> Linux supports hash on P9.

Ok. I was quoting from the Nest MMU spec. Will remove the text in ()s.

>=20
> Does VAS only support radix?  If so you should error out if we are booted=
 hash.

I think VAS just offloads the translation to NestMMU and doesn't really
depend on radix/HPT. I was going by the comment in the spec.

>=20
> > +	=A0*/
> > +	val =3D 0ULL;
> > +	val =3D SET_FIELD(VAS_XLATE_MODE, val, 0x3);
>=20
> You can use radix_enabled() to set this for hash vs radix.

Ok.

>=20
> > +	write_hvwc_reg(window, VREG(XLATE_CTL), val);
> > +
> > +	/*
> > +	=A0* TODO: Can we mfspr(AMR) even for user windows?
> > +	=A0*/
> > +	val =3D 0ULL;
> > +	val =3D SET_FIELD(VAS_AMR, val, mfspr(SPRN_AMR));
> > +	write_hvwc_reg(window, VREG(AMR), val);
> > +
> > +	/*
> > +	=A0* TODO: Assuming Secure Executable ID Register (SEIDR) is only used
> > +	=A0*	=A0in the ultravisor mode. Since MSR(UV) is 0 for now, set
> > SEIDR
> > +	=A0*	=A0to 0 as well, although we should 'mfspr(SEIDR)' at some
> > point.
> > +	=A0*/
> > +	val =3D 0ULL;
> > +	val =3D SET_FIELD(VAS_SEIDR, val, 0);
> > +	write_hvwc_reg(window, VREG(SEIDR), val);
> > +}
> > +
> > +/*
> > + * Initialize Reserved Send Buffer Count for the send window. It invol=
ves
> > + * writing to the register, reading it back to confirm that the hardwa=
re
> > + * has enough buffers to reserve. See section 1.3.1.2.1 of VAS workboo=
k.
> > + *
> > + * Since we can only make a best-effort attempt to fulfill the request,
> > + * we don't return any errors if we cannot.
> > + *
> > + * TODO: Reserved (aka dedicated) send buffers are not supported yet.
> > + */
> > +static void init_rsvd_tx_buf_count(struct vas_window *txwin,
> > +				struct vas_winctx *winctx)
> > +{
> > +	write_hvwc_reg(txwin, VREG(TX_RSVD_BUF_COUNT), 0ULL);
> > +}
> > +
> > +/*
> > + * Compute the log2() of the FIFO size expressed as kilobytes. It is i=
ntended
> > + * to be used to initialize the Local FIFO Size Register defined in Se=
ction
> > + * 3.14.25 of the VAS Workbook.
>=20
> There is a ilog2() function..

Ok, will use that.

Thanks,

Suka

Re: [PATCH v3 06/10] VAS: Define helpers to alloc/free windows

[Sukadev Bhattiprolu]

Michael Neuling [michael.neuling@au1.ibm.com] wrote:
> On Thu, 2017-03-16 at 20:33 -0700, Sukadev Bhattiprolu wrote:
> > Define helpers to allocate/free VAS window objects. These will
> > be used in follow-on patches when opening/closing windows.
> >=20
> > Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
> > ---
> > =A0drivers/misc/vas/vas-window.c | 74 +++++++++++++++++++++++++++++++++=
++++++++-
> > -
> > =A01 file changed, 72 insertions(+), 2 deletions(-)
> >=20
> > diff --git a/drivers/misc/vas/vas-window.c b/drivers/misc/vas/vas-windo=
w.c
> > index edf5c9f..9233bf5 100644
> > --- a/drivers/misc/vas/vas-window.c
> > +++ b/drivers/misc/vas/vas-window.c
> > @@ -119,7 +119,7 @@ static void unmap_wc_mmio_bars(struct vas_window *w=
indow)
> > =A0 * OS/User Window Context (UWC) MMIO Base Address Region for the giv=
en
> > window.
> > =A0 * Map these bus addresses and save the mapped kernel addresses in @=
window.
> > =A0 */
> > -int map_wc_mmio_bars(struct vas_window *window)
> > +static int map_wc_mmio_bars(struct vas_window *window)
> > =A0{
> > =A0	int len;
> > =A0	uint64_t start;
> > @@ -472,8 +472,78 @@ int init_winctx_regs(struct vas_window *window, st=
ruct
> > vas_winctx *winctx)
> > =A0	return 0;
> > =A0}
> > =A0
> > -/* stub for now */
> > +DEFINE_SPINLOCK(vas_ida_lock);
> > +
> > +void vas_release_window_id(struct ida *ida, int winid)
> > +{
> > +	spin_lock(&vas_ida_lock);
> > +	ida_remove(ida, winid);
> > +	spin_unlock(&vas_ida_lock);
> > +}
> > +
> > +int vas_assign_window_id(struct ida *ida)
> > +{
> > +	int rc, winid;
> > +
> > +	rc =3D ida_pre_get(ida, GFP_KERNEL);
> > +	if (!rc)
> > +		return -EAGAIN;
> > +
> > +	spin_lock(&vas_ida_lock);
> > +	rc =3D ida_get_new_above(ida, 0, &winid);
> > +	spin_unlock(&vas_ida_lock);
> > +
> > +	if (rc)
> > +		return rc;
> > +
> > +	if (winid > VAS_MAX_WINDOWS_PER_CHIP) {
> > +		pr_err("VAS: Too many (%d) open windows\n", winid);
> > +		vas_release_window_id(ida, winid);
> > +		return -EAGAIN;
> > +	}
> > +
> > +	return winid;
> > +}
> > +
> > +static void vas_window_free(struct vas_window *window)
> > +{
> > +	unmap_wc_mmio_bars(window);
> > +	kfree(window->paste_addr_name);
> > +	kfree(window);
> > +}
> > +
> > +static struct vas_window *vas_window_alloc(struct vas_instance *vinst,=
 int
> > id)
> > +{
> > +	struct vas_window *window;
> > +
> > +	window =3D kzalloc(sizeof(*window), GFP_KERNEL);
> > +	if (!window)
> > +		return NULL;
> > +
> > +	window->vinst =3D vinst;
> > +	window->winid =3D id;
> > +
> > +	if (map_wc_mmio_bars(window))
> > +		goto out_free;
> > +
> > +	return window;
> > +
> > +out_free:
> > +	kfree(window);
> > +	return NULL;
> > +}
> > +
> > =A0int vas_window_reset(struct vas_instance *vinst, int winid)
> >=20
>=20
> This interface seems a little weird to me. Needing an alloc in a hardware=
 reset
> path seems a bit strange.

Yeah, the name alloc in this interface is awkward.

I probably can drop this interface. Its used only during start up to clear
the window contexts. But since we must and do clear each window context
before using, we don't to do this during start up.

>=20
> Maybe the data structures are the issue.  A window is a hardware construc=
t.=20
> Something that uses it should probably be called something else like a co=
ntext.=20
> Something that references a window should just be the vas_instance + wini=
d.=20
>=20
> You should be able to reset this hardware window by referencing structures
> already allocated.  Something associated with the struct vas_instance.
>=20

'struct vas_winctx' is the window context (register fields associated
with the window) 'struct vas_window' is a container for the kernel state
associated with a window.

> Mikey

Thanks for the review.

Sukadev

Re: [PATCH v3 05/10] VAS: Define helpers to init window context

[Michael Neuling]

> > > +	val =3D 0ULL;
> > > +	if (user_win) {
> > > +		val =3D SET_FIELD(VAS_XLATE_MSR_DR, val, true);
> > > +		val =3D SET_FIELD(VAS_XLATE_MSR_TA, val, false);
> > > +		val =3D SET_FIELD(VAS_XLATE_MSR_PR, val, true);
> > > +		val =3D SET_FIELD(VAS_XLATE_MSR_US, val, false);
> > > +		val =3D SET_FIELD(VAS_XLATE_MSR_HV, val, true);
> > > +		val =3D SET_FIELD(VAS_XLATE_MSR_SF, val, true);
> > > +		val =3D SET_FIELD(VAS_XLATE_MSR_UV, val, false);
> > > +	} else {
> > > +		val =3D SET_FIELD(VAS_XLATE_MSR_DR, val, false);
> >=20
> > kernel contexts don't go through the nestmmu?
>=20
> I think so, but will check with Alistair/Ben.

Well it's kinda up to you.  Do you want to use the kernel mapping or not?  =
I
probably would unless there are issues.

Mikey

Re: [PATCH v3 01/10] VAS: Define macros, register fields and structures

[Sukadev Bhattiprolu]

Sukadev Bhattiprolu [sukadev@linux.vnet.ibm.com] wrote:
> Michael Neuling [mikey@neuling.org] wrote:
> > On Thu, 2017-03-16 at 20:33 -0700, Sukadev Bhattiprolu wrote:
> > > Define macros for the VAS hardware registers and bit-fields as well
> > > as couple of data structures needed by the VAS driver.
> > >=20
> > > > Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
> > > ---
> > > Changelog[v3]
> > > 	- Rename winctx->pid to winctx->pidr to reflect that its a value
> > > 	=A0=A0from the PID register (SPRN_PID), not the linux process id.
> > > 	- Make it easier to split header into kernel/user parts
> > > 	- To keep user interface simple, use macros rather than enum for
> > > 	=A0=A0the threshold-control modes.
> > > 	- Add a pid field to struct vas_window - needed for user space
> > > 	=A0=A0send windows.
> > >=20
> > > Changelog[v2]
> > > 	- Add an overview of VAS in vas-internal.h
> > > 	- Get window context parameters from device tree and drop
> > > 	=A0=A0unnecessary macros.
> > > ---
> > > =A0MAINTAINERS=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=
=A0=A0=A0|=A0=A0=A06 +
> > > =A0arch/powerpc/include/asm/vas.h=A0=A0|=A0=A043 +++++
> > > =A0drivers/misc/vas/vas-internal.h | 392 ++++++++++++++++++++++++++++=
++++++++++++
> >=20
> > This is going to have to go through gregkh/lkml if it's drivers/misc.  =
you'll at
> > least need gregkh's ack/ok before mpe will take them (which is what we =
did for
> > CAPI).
> >=20
> > We might want to keep this in arch/powerpc but I'm not sure.
> >=20
>=20
> We will have device nodes accessible to user space so put it here and can
> copy Gregkh next time. But let me know if we should move to arch/powerpc.
>=20

Thinking about this some more, the VAS module itself does not provide
the device nodes. Rather, the drivers that use VAS, like NX-GZIP, will
provide them. So, I am moving the VAS code into arch/powerpc/kernel.

Please let me know of any comments/concerns.

Thanks
Suka