linux-pci.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v8 0/9] vfio/hisilicon: add ACC live migration driver
@ 2022-03-03 23:01 Shameer Kolothum
  2022-03-03 23:01 ` [PATCH v8 1/9] crypto: hisilicon/qm: Move the QM header to include/linux Shameer Kolothum
                   ` (8 more replies)
  0 siblings, 9 replies; 56+ messages in thread
From: Shameer Kolothum @ 2022-03-03 23:01 UTC (permalink / raw)
  To: kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	linuxarm, liulongfang, prime.zeng, jonathan.cameron, wangzhou1

Hi,

This series attempts to add vfio live migration support for HiSilicon
ACC VF devices based on the new v2 migration protocol definition and
mlx5 v9 series discussed here[0].

v7 --> v8
 - Dropped PRE_COPY support and early compatibility checking based on
   the discussion here[1].
 - Addressed comments from John, Jason & Alex (Thanks!).

This is sanity tested on a HiSilicon platform using the Qemu branch
provided here[2].

Please take a look and let me know your feedback.

Thanks,
Shameer
[0] https://lore.kernel.org/kvm/20220224142024.147653-1-yishaih@nvidia.com/
[1] https://lore.kernel.org/kvm/20220302133159.3c803f56.alex.williamson@redhat.com/
[2] https://github.com/jgunthorpe/qemu/commits/vfio_migration_v2

v6 --> v7
 -Renamed MIG_PRECOPY ioctl name and struct name. Updated ioctl descriptions
  regarding ioctl validity (patch #7).
- Adressed comments from Jason and Alex on PRE_COPY read() and ioctl() fns
  (patch #9).
- Moved only VF PCI ids to pci_ids.h(patch #3).

v5 --> v6
 -Report PRE_COPY support and use that for early compatibility check
  between src and dst devices.
 -For generic PRE_COPY support, included patch #7 from Jason(Thanks!).
 -Addressed comments from Alex(Thanks!).
 -Added the QM state register update to QM driver(patch #8) since that
  is being used in migration driver to decide whether the device is
  ready to save the state.

RFCv4 --> v5
  - Dropped RFC tag as v2 migration APIs are more stable now.
  - Addressed review comments from Jason and Alex (Thanks!).

v3 --> RFCv4
-Based on migration v2 protocol and mlx5 v7 series.
-Added RFC tag again as migration v2 protocol is still under discussion.
-Added new patch #6 to retrieve the PF QM data.
-PRE_COPY compatibility check is now done after the migration data
 transfer. This is not ideal and needs discussion.

RFC v2 --> v3
 -Dropped RFC tag as the vfio_pci_core subsystem framework is now
  part of 5.15-rc1.
 -Added override methods for vfio_device_ops read/write/mmap calls
  to limit the access within the functional register space.
 -Patches 1 to 3 are code refactoring to move the common ACC QM
  definitions and header around.

RFCv1 --> RFCv2

 -Adds a new vendor-specific vfio_pci driver(hisi-acc-vfio-pci)
  for HiSilicon ACC VF devices based on the new vfio-pci-core
  framework proposal.

 -Since HiSilicon ACC VF device MMIO space contains both the
  functional register space and migration control register space,
  override the vfio_device_ops ioctl method to report only the
  functional space to VMs.

 -For a successful migration, we still need access to VF dev
  functional register space mainly to read the status registers.
  But accessing these while the Guest vCPUs are running may leave
  a security hole. To avoid any potential security issues, we
  map/unmap the MMIO regions on a need basis and is safe to do so.
  (Please see hisi_acc_vf_ioremap/unmap() fns in patch #4).
 
 -Dropped debugfs support for now.
 -Uses common QM functions for mailbox access(patch #3).

Longfang Liu (3):
  crypto: hisilicon/qm: Move few definitions to common header
  crypto: hisilicon/qm: Set the VF QM state register
  hisi_acc_vfio_pci: Add support for VFIO live migration

Shameer Kolothum (6):
  crypto: hisilicon/qm: Move the QM header to include/linux
  hisi_acc_qm: Move VF PCI device IDs to common header
  hisi_acc_vfio_pci: add new vfio_pci driver for HiSilicon ACC devices
  hisi_acc_vfio_pci: Restrict access to VF dev BAR2 migration region
  hisi_acc_vfio_pci: Add helper to retrieve the struct pci_driver
  hisi_acc_vfio_pci: Use its own PCI reset_done error handler

 drivers/crypto/hisilicon/hpre/hpre.h          |    2 +-
 drivers/crypto/hisilicon/hpre/hpre_main.c     |   19 +-
 drivers/crypto/hisilicon/qm.c                 |   68 +-
 drivers/crypto/hisilicon/sec2/sec.h           |    2 +-
 drivers/crypto/hisilicon/sec2/sec_main.c      |   21 +-
 drivers/crypto/hisilicon/sgl.c                |    2 +-
 drivers/crypto/hisilicon/zip/zip.h            |    2 +-
 drivers/crypto/hisilicon/zip/zip_main.c       |   17 +-
 drivers/vfio/pci/Kconfig                      |    2 +
 drivers/vfio/pci/Makefile                     |    2 +
 drivers/vfio/pci/hisilicon/Kconfig            |   17 +
 drivers/vfio/pci/hisilicon/Makefile           |    4 +
 .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    | 1319 +++++++++++++++++
 .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h    |  116 ++
 .../qm.h => include/linux/hisi_acc_qm.h       |   49 +
 include/linux/pci_ids.h                       |    3 +
 16 files changed, 1579 insertions(+), 66 deletions(-)
 create mode 100644 drivers/vfio/pci/hisilicon/Kconfig
 create mode 100644 drivers/vfio/pci/hisilicon/Makefile
 create mode 100644 drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
 create mode 100644 drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
 rename drivers/crypto/hisilicon/qm.h => include/linux/hisi_acc_qm.h (87%)

-- 
2.25.1


^ permalink raw reply	[flat|nested] 56+ messages in thread

* [PATCH v8 1/9] crypto: hisilicon/qm: Move the QM header to include/linux
  2022-03-03 23:01 [PATCH v8 0/9] vfio/hisilicon: add ACC live migration driver Shameer Kolothum
@ 2022-03-03 23:01 ` Shameer Kolothum
  2022-03-04  9:03   ` Zhou Wang
                     ` (2 more replies)
  2022-03-03 23:01 ` [PATCH v8 2/9] crypto: hisilicon/qm: Move few definitions to common header Shameer Kolothum
                   ` (7 subsequent siblings)
  8 siblings, 3 replies; 56+ messages in thread
From: Shameer Kolothum @ 2022-03-03 23:01 UTC (permalink / raw)
  To: kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	linuxarm, liulongfang, prime.zeng, jonathan.cameron, wangzhou1

Since we are going to introduce VFIO PCI HiSilicon ACC driver for live
migration in subsequent patches, move the ACC QM header file to a
common include dir.

Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
---
 drivers/crypto/hisilicon/hpre/hpre.h                         | 2 +-
 drivers/crypto/hisilicon/qm.c                                | 2 +-
 drivers/crypto/hisilicon/sec2/sec.h                          | 2 +-
 drivers/crypto/hisilicon/sgl.c                               | 2 +-
 drivers/crypto/hisilicon/zip/zip.h                           | 2 +-
 drivers/crypto/hisilicon/qm.h => include/linux/hisi_acc_qm.h | 0
 6 files changed, 5 insertions(+), 5 deletions(-)
 rename drivers/crypto/hisilicon/qm.h => include/linux/hisi_acc_qm.h (100%)

diff --git a/drivers/crypto/hisilicon/hpre/hpre.h b/drivers/crypto/hisilicon/hpre/hpre.h
index e0b4a1982ee9..9a0558ed82f9 100644
--- a/drivers/crypto/hisilicon/hpre/hpre.h
+++ b/drivers/crypto/hisilicon/hpre/hpre.h
@@ -4,7 +4,7 @@
 #define __HISI_HPRE_H
 
 #include <linux/list.h>
-#include "../qm.h"
+#include <linux/hisi_acc_qm.h>
 
 #define HPRE_SQE_SIZE			sizeof(struct hpre_sqe)
 #define HPRE_PF_DEF_Q_NUM		64
diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c
index c5b84a5ea350..ed23e1d3fa27 100644
--- a/drivers/crypto/hisilicon/qm.c
+++ b/drivers/crypto/hisilicon/qm.c
@@ -15,7 +15,7 @@
 #include <linux/uacce.h>
 #include <linux/uaccess.h>
 #include <uapi/misc/uacce/hisi_qm.h>
-#include "qm.h"
+#include <linux/hisi_acc_qm.h>
 
 /* eq/aeq irq enable */
 #define QM_VF_AEQ_INT_SOURCE		0x0
diff --git a/drivers/crypto/hisilicon/sec2/sec.h b/drivers/crypto/hisilicon/sec2/sec.h
index d97cf02b1df7..c2e9b01187a7 100644
--- a/drivers/crypto/hisilicon/sec2/sec.h
+++ b/drivers/crypto/hisilicon/sec2/sec.h
@@ -4,7 +4,7 @@
 #ifndef __HISI_SEC_V2_H
 #define __HISI_SEC_V2_H
 
-#include "../qm.h"
+#include <linux/hisi_acc_qm.h>
 #include "sec_crypto.h"
 
 /* Algorithm resource per hardware SEC queue */
diff --git a/drivers/crypto/hisilicon/sgl.c b/drivers/crypto/hisilicon/sgl.c
index 057273769f26..f7efc02b065f 100644
--- a/drivers/crypto/hisilicon/sgl.c
+++ b/drivers/crypto/hisilicon/sgl.c
@@ -1,9 +1,9 @@
 // SPDX-License-Identifier: GPL-2.0
 /* Copyright (c) 2019 HiSilicon Limited. */
 #include <linux/dma-mapping.h>
+#include <linux/hisi_acc_qm.h>
 #include <linux/module.h>
 #include <linux/slab.h>
-#include "qm.h"
 
 #define HISI_ACC_SGL_SGE_NR_MIN		1
 #define HISI_ACC_SGL_NR_MAX		256
diff --git a/drivers/crypto/hisilicon/zip/zip.h b/drivers/crypto/hisilicon/zip/zip.h
index 517fdbdff3ea..3dfd3bac5a33 100644
--- a/drivers/crypto/hisilicon/zip/zip.h
+++ b/drivers/crypto/hisilicon/zip/zip.h
@@ -7,7 +7,7 @@
 #define pr_fmt(fmt)	"hisi_zip: " fmt
 
 #include <linux/list.h>
-#include "../qm.h"
+#include <linux/hisi_acc_qm.h>
 
 enum hisi_zip_error_type {
 	/* negative compression */
diff --git a/drivers/crypto/hisilicon/qm.h b/include/linux/hisi_acc_qm.h
similarity index 100%
rename from drivers/crypto/hisilicon/qm.h
rename to include/linux/hisi_acc_qm.h
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 56+ messages in thread

* [PATCH v8 2/9] crypto: hisilicon/qm: Move few definitions to common header
  2022-03-03 23:01 [PATCH v8 0/9] vfio/hisilicon: add ACC live migration driver Shameer Kolothum
  2022-03-03 23:01 ` [PATCH v8 1/9] crypto: hisilicon/qm: Move the QM header to include/linux Shameer Kolothum
@ 2022-03-03 23:01 ` Shameer Kolothum
  2022-03-04  9:06   ` Zhou Wang
  2022-03-03 23:01 ` [PATCH v8 3/9] hisi_acc_qm: Move VF PCI device IDs " Shameer Kolothum
                   ` (6 subsequent siblings)
  8 siblings, 1 reply; 56+ messages in thread
From: Shameer Kolothum @ 2022-03-03 23:01 UTC (permalink / raw)
  To: kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	linuxarm, liulongfang, prime.zeng, jonathan.cameron, wangzhou1

From: Longfang Liu <liulongfang@huawei.com>

Move Doorbell and Mailbox definitions to common header file.
Also export QM mailbox functions.

This will be useful when we introduce VFIO PCI HiSilicon ACC live
migration driver.

Signed-off-by: Longfang Liu <liulongfang@huawei.com>
Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
---
 drivers/crypto/hisilicon/qm.c | 58 +++++++++++------------------------
 include/linux/hisi_acc_qm.h   | 38 +++++++++++++++++++++++
 2 files changed, 56 insertions(+), 40 deletions(-)

diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c
index ed23e1d3fa27..c88e013371af 100644
--- a/drivers/crypto/hisilicon/qm.c
+++ b/drivers/crypto/hisilicon/qm.c
@@ -33,23 +33,6 @@
 #define QM_ABNORMAL_EVENT_IRQ_VECTOR	3
 
 /* mailbox */
-#define QM_MB_CMD_SQC			0x0
-#define QM_MB_CMD_CQC			0x1
-#define QM_MB_CMD_EQC			0x2
-#define QM_MB_CMD_AEQC			0x3
-#define QM_MB_CMD_SQC_BT		0x4
-#define QM_MB_CMD_CQC_BT		0x5
-#define QM_MB_CMD_SQC_VFT_V2		0x6
-#define QM_MB_CMD_STOP_QP		0x8
-#define QM_MB_CMD_SRC			0xc
-#define QM_MB_CMD_DST			0xd
-
-#define QM_MB_CMD_SEND_BASE		0x300
-#define QM_MB_EVENT_SHIFT		8
-#define QM_MB_BUSY_SHIFT		13
-#define QM_MB_OP_SHIFT			14
-#define QM_MB_CMD_DATA_ADDR_L		0x304
-#define QM_MB_CMD_DATA_ADDR_H		0x308
 #define QM_MB_PING_ALL_VFS		0xffff
 #define QM_MB_CMD_DATA_SHIFT		32
 #define QM_MB_CMD_DATA_MASK		GENMASK(31, 0)
@@ -103,19 +86,12 @@
 #define QM_DB_CMD_SHIFT_V1		16
 #define QM_DB_INDEX_SHIFT_V1		32
 #define QM_DB_PRIORITY_SHIFT_V1		48
-#define QM_DOORBELL_SQ_CQ_BASE_V2	0x1000
-#define QM_DOORBELL_EQ_AEQ_BASE_V2	0x2000
 #define QM_QUE_ISO_CFG_V		0x0030
 #define QM_PAGE_SIZE			0x0034
 #define QM_QUE_ISO_EN			0x100154
 #define QM_CAPBILITY			0x100158
 #define QM_QP_NUN_MASK			GENMASK(10, 0)
 #define QM_QP_DB_INTERVAL		0x10000
-#define QM_QP_MAX_NUM_SHIFT		11
-#define QM_DB_CMD_SHIFT_V2		12
-#define QM_DB_RAND_SHIFT_V2		16
-#define QM_DB_INDEX_SHIFT_V2		32
-#define QM_DB_PRIORITY_SHIFT_V2		48
 
 #define QM_MEM_START_INIT		0x100040
 #define QM_MEM_INIT_DONE		0x100044
@@ -693,7 +669,7 @@ static void qm_mb_pre_init(struct qm_mailbox *mailbox, u8 cmd,
 }
 
 /* return 0 mailbox ready, -ETIMEDOUT hardware timeout */
-static int qm_wait_mb_ready(struct hisi_qm *qm)
+int hisi_qm_wait_mb_ready(struct hisi_qm *qm)
 {
 	u32 val;
 
@@ -701,6 +677,7 @@ static int qm_wait_mb_ready(struct hisi_qm *qm)
 					  val, !((val >> QM_MB_BUSY_SHIFT) &
 					  0x1), POLL_PERIOD, POLL_TIMEOUT);
 }
+EXPORT_SYMBOL_GPL(hisi_qm_wait_mb_ready);
 
 /* 128 bit should be written to hardware at one time to trigger a mailbox */
 static void qm_mb_write(struct hisi_qm *qm, const void *src)
@@ -726,14 +703,14 @@ static void qm_mb_write(struct hisi_qm *qm, const void *src)
 
 static int qm_mb_nolock(struct hisi_qm *qm, struct qm_mailbox *mailbox)
 {
-	if (unlikely(qm_wait_mb_ready(qm))) {
+	if (unlikely(hisi_qm_wait_mb_ready(qm))) {
 		dev_err(&qm->pdev->dev, "QM mailbox is busy to start!\n");
 		goto mb_busy;
 	}
 
 	qm_mb_write(qm, mailbox);
 
-	if (unlikely(qm_wait_mb_ready(qm))) {
+	if (unlikely(hisi_qm_wait_mb_ready(qm))) {
 		dev_err(&qm->pdev->dev, "QM mailbox operation timeout!\n");
 		goto mb_busy;
 	}
@@ -745,8 +722,8 @@ static int qm_mb_nolock(struct hisi_qm *qm, struct qm_mailbox *mailbox)
 	return -EBUSY;
 }
 
-static int qm_mb(struct hisi_qm *qm, u8 cmd, dma_addr_t dma_addr, u16 queue,
-		 bool op)
+int hisi_qm_mb(struct hisi_qm *qm, u8 cmd, dma_addr_t dma_addr, u16 queue,
+	       bool op)
 {
 	struct qm_mailbox mailbox;
 	int ret;
@@ -762,6 +739,7 @@ static int qm_mb(struct hisi_qm *qm, u8 cmd, dma_addr_t dma_addr, u16 queue,
 
 	return ret;
 }
+EXPORT_SYMBOL_GPL(hisi_qm_mb);
 
 static void qm_db_v1(struct hisi_qm *qm, u16 qn, u8 cmd, u16 index, u8 priority)
 {
@@ -1351,7 +1329,7 @@ static int qm_get_vft_v2(struct hisi_qm *qm, u32 *base, u32 *number)
 	u64 sqc_vft;
 	int ret;
 
-	ret = qm_mb(qm, QM_MB_CMD_SQC_VFT_V2, 0, 0, 1);
+	ret = hisi_qm_mb(qm, QM_MB_CMD_SQC_VFT_V2, 0, 0, 1);
 	if (ret)
 		return ret;
 
@@ -1725,12 +1703,12 @@ static int dump_show(struct hisi_qm *qm, void *info,
 
 static int qm_dump_sqc_raw(struct hisi_qm *qm, dma_addr_t dma_addr, u16 qp_id)
 {
-	return qm_mb(qm, QM_MB_CMD_SQC, dma_addr, qp_id, 1);
+	return hisi_qm_mb(qm, QM_MB_CMD_SQC, dma_addr, qp_id, 1);
 }
 
 static int qm_dump_cqc_raw(struct hisi_qm *qm, dma_addr_t dma_addr, u16 qp_id)
 {
-	return qm_mb(qm, QM_MB_CMD_CQC, dma_addr, qp_id, 1);
+	return hisi_qm_mb(qm, QM_MB_CMD_CQC, dma_addr, qp_id, 1);
 }
 
 static int qm_sqc_dump(struct hisi_qm *qm, const char *s)
@@ -1842,7 +1820,7 @@ static int qm_eqc_aeqc_dump(struct hisi_qm *qm, char *s, size_t size,
 	if (IS_ERR(xeqc))
 		return PTR_ERR(xeqc);
 
-	ret = qm_mb(qm, cmd, xeqc_dma, 0, 1);
+	ret = hisi_qm_mb(qm, cmd, xeqc_dma, 0, 1);
 	if (ret)
 		goto err_free_ctx;
 
@@ -2495,7 +2473,7 @@ static int qm_ping_pf(struct hisi_qm *qm, u64 cmd)
 
 static int qm_stop_qp(struct hisi_qp *qp)
 {
-	return qm_mb(qp->qm, QM_MB_CMD_STOP_QP, 0, qp->qp_id, 0);
+	return hisi_qm_mb(qp->qm, QM_MB_CMD_STOP_QP, 0, qp->qp_id, 0);
 }
 
 static int qm_set_msi(struct hisi_qm *qm, bool set)
@@ -2763,7 +2741,7 @@ static int qm_sq_ctx_cfg(struct hisi_qp *qp, int qp_id, u32 pasid)
 		return -ENOMEM;
 	}
 
-	ret = qm_mb(qm, QM_MB_CMD_SQC, sqc_dma, qp_id, 0);
+	ret = hisi_qm_mb(qm, QM_MB_CMD_SQC, sqc_dma, qp_id, 0);
 	dma_unmap_single(dev, sqc_dma, sizeof(struct qm_sqc), DMA_TO_DEVICE);
 	kfree(sqc);
 
@@ -2804,7 +2782,7 @@ static int qm_cq_ctx_cfg(struct hisi_qp *qp, int qp_id, u32 pasid)
 		return -ENOMEM;
 	}
 
-	ret = qm_mb(qm, QM_MB_CMD_CQC, cqc_dma, qp_id, 0);
+	ret = hisi_qm_mb(qm, QM_MB_CMD_CQC, cqc_dma, qp_id, 0);
 	dma_unmap_single(dev, cqc_dma, sizeof(struct qm_cqc), DMA_TO_DEVICE);
 	kfree(cqc);
 
@@ -3655,7 +3633,7 @@ static int qm_eq_ctx_cfg(struct hisi_qm *qm)
 		return -ENOMEM;
 	}
 
-	ret = qm_mb(qm, QM_MB_CMD_EQC, eqc_dma, 0, 0);
+	ret = hisi_qm_mb(qm, QM_MB_CMD_EQC, eqc_dma, 0, 0);
 	dma_unmap_single(dev, eqc_dma, sizeof(struct qm_eqc), DMA_TO_DEVICE);
 	kfree(eqc);
 
@@ -3684,7 +3662,7 @@ static int qm_aeq_ctx_cfg(struct hisi_qm *qm)
 		return -ENOMEM;
 	}
 
-	ret = qm_mb(qm, QM_MB_CMD_AEQC, aeqc_dma, 0, 0);
+	ret = hisi_qm_mb(qm, QM_MB_CMD_AEQC, aeqc_dma, 0, 0);
 	dma_unmap_single(dev, aeqc_dma, sizeof(struct qm_aeqc), DMA_TO_DEVICE);
 	kfree(aeqc);
 
@@ -3723,11 +3701,11 @@ static int __hisi_qm_start(struct hisi_qm *qm)
 	if (ret)
 		return ret;
 
-	ret = qm_mb(qm, QM_MB_CMD_SQC_BT, qm->sqc_dma, 0, 0);
+	ret = hisi_qm_mb(qm, QM_MB_CMD_SQC_BT, qm->sqc_dma, 0, 0);
 	if (ret)
 		return ret;
 
-	ret = qm_mb(qm, QM_MB_CMD_CQC_BT, qm->cqc_dma, 0, 0);
+	ret = hisi_qm_mb(qm, QM_MB_CMD_CQC_BT, qm->cqc_dma, 0, 0);
 	if (ret)
 		return ret;
 
diff --git a/include/linux/hisi_acc_qm.h b/include/linux/hisi_acc_qm.h
index 3068093229a5..6a6477c34666 100644
--- a/include/linux/hisi_acc_qm.h
+++ b/include/linux/hisi_acc_qm.h
@@ -34,6 +34,40 @@
 #define QM_WUSER_M_CFG_ENABLE		0x1000a8
 #define WUSER_M_CFG_ENABLE		0xffffffff
 
+/* mailbox */
+#define QM_MB_CMD_SQC                   0x0
+#define QM_MB_CMD_CQC                   0x1
+#define QM_MB_CMD_EQC                   0x2
+#define QM_MB_CMD_AEQC                  0x3
+#define QM_MB_CMD_SQC_BT                0x4
+#define QM_MB_CMD_CQC_BT                0x5
+#define QM_MB_CMD_SQC_VFT_V2            0x6
+#define QM_MB_CMD_STOP_QP               0x8
+#define QM_MB_CMD_SRC                   0xc
+#define QM_MB_CMD_DST                   0xd
+
+#define QM_MB_CMD_SEND_BASE		0x300
+#define QM_MB_EVENT_SHIFT               8
+#define QM_MB_BUSY_SHIFT		13
+#define QM_MB_OP_SHIFT			14
+#define QM_MB_CMD_DATA_ADDR_L		0x304
+#define QM_MB_CMD_DATA_ADDR_H		0x308
+#define QM_MB_MAX_WAIT_CNT		6000
+
+/* doorbell */
+#define QM_DOORBELL_CMD_SQ              0
+#define QM_DOORBELL_CMD_CQ              1
+#define QM_DOORBELL_CMD_EQ              2
+#define QM_DOORBELL_CMD_AEQ             3
+
+#define QM_DOORBELL_SQ_CQ_BASE_V2	0x1000
+#define QM_DOORBELL_EQ_AEQ_BASE_V2	0x2000
+#define QM_QP_MAX_NUM_SHIFT             11
+#define QM_DB_CMD_SHIFT_V2		12
+#define QM_DB_RAND_SHIFT_V2		16
+#define QM_DB_INDEX_SHIFT_V2		32
+#define QM_DB_PRIORITY_SHIFT_V2		48
+
 /* qm cache */
 #define QM_CACHE_CTL			0x100050
 #define SQC_CACHE_ENABLE		BIT(0)
@@ -414,6 +448,10 @@ pci_ers_result_t hisi_qm_dev_slot_reset(struct pci_dev *pdev);
 void hisi_qm_reset_prepare(struct pci_dev *pdev);
 void hisi_qm_reset_done(struct pci_dev *pdev);
 
+int hisi_qm_wait_mb_ready(struct hisi_qm *qm);
+int hisi_qm_mb(struct hisi_qm *qm, u8 cmd, dma_addr_t dma_addr, u16 queue,
+	       bool op);
+
 struct hisi_acc_sgl_pool;
 struct hisi_acc_hw_sgl *hisi_acc_sg_buf_map_to_hw_sgl(struct device *dev,
 	struct scatterlist *sgl, struct hisi_acc_sgl_pool *pool,
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 56+ messages in thread

* [PATCH v8 3/9] hisi_acc_qm: Move VF PCI device IDs to common header
  2022-03-03 23:01 [PATCH v8 0/9] vfio/hisilicon: add ACC live migration driver Shameer Kolothum
  2022-03-03 23:01 ` [PATCH v8 1/9] crypto: hisilicon/qm: Move the QM header to include/linux Shameer Kolothum
  2022-03-03 23:01 ` [PATCH v8 2/9] crypto: hisilicon/qm: Move few definitions to common header Shameer Kolothum
@ 2022-03-03 23:01 ` Shameer Kolothum
  2022-03-04  9:34   ` Zhou Wang
                     ` (3 more replies)
  2022-03-03 23:01 ` [PATCH v8 4/9] hisi_acc_vfio_pci: add new vfio_pci driver for HiSilicon ACC devices Shameer Kolothum
                   ` (5 subsequent siblings)
  8 siblings, 4 replies; 56+ messages in thread
From: Shameer Kolothum @ 2022-03-03 23:01 UTC (permalink / raw)
  To: kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	linuxarm, liulongfang, prime.zeng, jonathan.cameron, wangzhou1

Move the PCI Device IDs of HiSilicon ACC VF devices to a common header
and also use a uniform naming convention.

This will be useful when we introduce the vfio PCI HiSilicon ACC live
migration driver in subsequent patches.

Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
---
 drivers/crypto/hisilicon/hpre/hpre_main.c | 13 ++++++-------
 drivers/crypto/hisilicon/sec2/sec_main.c  | 15 +++++++--------
 drivers/crypto/hisilicon/zip/zip_main.c   | 11 +++++------
 include/linux/pci_ids.h                   |  3 +++
 4 files changed, 21 insertions(+), 21 deletions(-)

diff --git a/drivers/crypto/hisilicon/hpre/hpre_main.c b/drivers/crypto/hisilicon/hpre/hpre_main.c
index ebfab3e14499..3589d8879b5e 100644
--- a/drivers/crypto/hisilicon/hpre/hpre_main.c
+++ b/drivers/crypto/hisilicon/hpre/hpre_main.c
@@ -68,8 +68,7 @@
 #define HPRE_REG_RD_INTVRL_US		10
 #define HPRE_REG_RD_TMOUT_US		1000
 #define HPRE_DBGFS_VAL_MAX_LEN		20
-#define HPRE_PCI_DEVICE_ID		0xa258
-#define HPRE_PCI_VF_DEVICE_ID		0xa259
+#define PCI_DEVICE_ID_HUAWEI_HPRE_PF	0xa258
 #define HPRE_QM_USR_CFG_MASK		GENMASK(31, 1)
 #define HPRE_QM_AXI_CFG_MASK		GENMASK(15, 0)
 #define HPRE_QM_VFG_AX_MASK		GENMASK(7, 0)
@@ -111,8 +110,8 @@
 static const char hpre_name[] = "hisi_hpre";
 static struct dentry *hpre_debugfs_root;
 static const struct pci_device_id hpre_dev_ids[] = {
-	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, HPRE_PCI_DEVICE_ID) },
-	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, HPRE_PCI_VF_DEVICE_ID) },
+	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_HPRE_PF) },
+	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_HPRE_VF) },
 	{ 0, }
 };
 
@@ -242,7 +241,7 @@ MODULE_PARM_DESC(uacce_mode, UACCE_MODE_DESC);
 
 static int pf_q_num_set(const char *val, const struct kernel_param *kp)
 {
-	return q_num_set(val, kp, HPRE_PCI_DEVICE_ID);
+	return q_num_set(val, kp, PCI_DEVICE_ID_HUAWEI_HPRE_PF);
 }
 
 static const struct kernel_param_ops hpre_pf_q_num_ops = {
@@ -921,7 +920,7 @@ static int hpre_debugfs_init(struct hisi_qm *qm)
 	qm->debug.sqe_mask_len = HPRE_SQE_MASK_LEN;
 	hisi_qm_debug_init(qm);
 
-	if (qm->pdev->device == HPRE_PCI_DEVICE_ID) {
+	if (qm->pdev->device == PCI_DEVICE_ID_HUAWEI_HPRE_PF) {
 		ret = hpre_ctrl_debug_init(qm);
 		if (ret)
 			goto failed_to_create;
@@ -958,7 +957,7 @@ static int hpre_qm_init(struct hisi_qm *qm, struct pci_dev *pdev)
 	qm->sqe_size = HPRE_SQE_SIZE;
 	qm->dev_name = hpre_name;
 
-	qm->fun_type = (pdev->device == HPRE_PCI_DEVICE_ID) ?
+	qm->fun_type = (pdev->device == PCI_DEVICE_ID_HUAWEI_HPRE_PF) ?
 			QM_HW_PF : QM_HW_VF;
 	if (qm->fun_type == QM_HW_PF) {
 		qm->qp_base = HPRE_PF_DEF_Q_BASE;
diff --git a/drivers/crypto/hisilicon/sec2/sec_main.c b/drivers/crypto/hisilicon/sec2/sec_main.c
index 26d3ab1d308b..311a8747b5bf 100644
--- a/drivers/crypto/hisilicon/sec2/sec_main.c
+++ b/drivers/crypto/hisilicon/sec2/sec_main.c
@@ -20,8 +20,7 @@
 
 #define SEC_VF_NUM			63
 #define SEC_QUEUE_NUM_V1		4096
-#define SEC_PF_PCI_DEVICE_ID		0xa255
-#define SEC_VF_PCI_DEVICE_ID		0xa256
+#define PCI_DEVICE_ID_HUAWEI_SEC_PF	0xa255
 
 #define SEC_BD_ERR_CHK_EN0		0xEFFFFFFF
 #define SEC_BD_ERR_CHK_EN1		0x7ffff7fd
@@ -225,7 +224,7 @@ static const struct debugfs_reg32 sec_dfx_regs[] = {
 
 static int sec_pf_q_num_set(const char *val, const struct kernel_param *kp)
 {
-	return q_num_set(val, kp, SEC_PF_PCI_DEVICE_ID);
+	return q_num_set(val, kp, PCI_DEVICE_ID_HUAWEI_SEC_PF);
 }
 
 static const struct kernel_param_ops sec_pf_q_num_ops = {
@@ -313,8 +312,8 @@ module_param_cb(uacce_mode, &sec_uacce_mode_ops, &uacce_mode, 0444);
 MODULE_PARM_DESC(uacce_mode, UACCE_MODE_DESC);
 
 static const struct pci_device_id sec_dev_ids[] = {
-	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, SEC_PF_PCI_DEVICE_ID) },
-	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, SEC_VF_PCI_DEVICE_ID) },
+	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_SEC_PF) },
+	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_SEC_VF) },
 	{ 0, }
 };
 MODULE_DEVICE_TABLE(pci, sec_dev_ids);
@@ -717,7 +716,7 @@ static int sec_core_debug_init(struct hisi_qm *qm)
 	regset->base = qm->io_base;
 	regset->dev = dev;
 
-	if (qm->pdev->device == SEC_PF_PCI_DEVICE_ID)
+	if (qm->pdev->device == PCI_DEVICE_ID_HUAWEI_SEC_PF)
 		debugfs_create_file("regs", 0444, tmp_d, regset, &sec_regs_fops);
 
 	for (i = 0; i < ARRAY_SIZE(sec_dfx_labels); i++) {
@@ -735,7 +734,7 @@ static int sec_debug_init(struct hisi_qm *qm)
 	struct sec_dev *sec = container_of(qm, struct sec_dev, qm);
 	int i;
 
-	if (qm->pdev->device == SEC_PF_PCI_DEVICE_ID) {
+	if (qm->pdev->device == PCI_DEVICE_ID_HUAWEI_SEC_PF) {
 		for (i = SEC_CLEAR_ENABLE; i < SEC_DEBUG_FILE_NUM; i++) {
 			spin_lock_init(&sec->debug.files[i].lock);
 			sec->debug.files[i].index = i;
@@ -877,7 +876,7 @@ static int sec_qm_init(struct hisi_qm *qm, struct pci_dev *pdev)
 	qm->sqe_size = SEC_SQE_SIZE;
 	qm->dev_name = sec_name;
 
-	qm->fun_type = (pdev->device == SEC_PF_PCI_DEVICE_ID) ?
+	qm->fun_type = (pdev->device == PCI_DEVICE_ID_HUAWEI_SEC_PF) ?
 			QM_HW_PF : QM_HW_VF;
 	if (qm->fun_type == QM_HW_PF) {
 		qm->qp_base = SEC_PF_DEF_Q_BASE;
diff --git a/drivers/crypto/hisilicon/zip/zip_main.c b/drivers/crypto/hisilicon/zip/zip_main.c
index 678f8b58ec42..66decfe07282 100644
--- a/drivers/crypto/hisilicon/zip/zip_main.c
+++ b/drivers/crypto/hisilicon/zip/zip_main.c
@@ -15,8 +15,7 @@
 #include <linux/uacce.h>
 #include "zip.h"
 
-#define PCI_DEVICE_ID_ZIP_PF		0xa250
-#define PCI_DEVICE_ID_ZIP_VF		0xa251
+#define PCI_DEVICE_ID_HUAWEI_ZIP_PF	0xa250
 
 #define HZIP_QUEUE_NUM_V1		4096
 
@@ -246,7 +245,7 @@ MODULE_PARM_DESC(uacce_mode, UACCE_MODE_DESC);
 
 static int pf_q_num_set(const char *val, const struct kernel_param *kp)
 {
-	return q_num_set(val, kp, PCI_DEVICE_ID_ZIP_PF);
+	return q_num_set(val, kp, PCI_DEVICE_ID_HUAWEI_ZIP_PF);
 }
 
 static const struct kernel_param_ops pf_q_num_ops = {
@@ -268,8 +267,8 @@ module_param_cb(vfs_num, &vfs_num_ops, &vfs_num, 0444);
 MODULE_PARM_DESC(vfs_num, "Number of VFs to enable(1-63), 0(default)");
 
 static const struct pci_device_id hisi_zip_dev_ids[] = {
-	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_ZIP_PF) },
-	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_ZIP_VF) },
+	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_ZIP_PF) },
+	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_ZIP_VF) },
 	{ 0, }
 };
 MODULE_DEVICE_TABLE(pci, hisi_zip_dev_ids);
@@ -838,7 +837,7 @@ static int hisi_zip_qm_init(struct hisi_qm *qm, struct pci_dev *pdev)
 	qm->sqe_size = HZIP_SQE_SIZE;
 	qm->dev_name = hisi_zip_name;
 
-	qm->fun_type = (pdev->device == PCI_DEVICE_ID_ZIP_PF) ?
+	qm->fun_type = (pdev->device == PCI_DEVICE_ID_HUAWEI_ZIP_PF) ?
 			QM_HW_PF : QM_HW_VF;
 	if (qm->fun_type == QM_HW_PF) {
 		qm->qp_base = HZIP_PF_DEF_Q_BASE;
diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
index aad54c666407..31dee2b65a62 100644
--- a/include/linux/pci_ids.h
+++ b/include/linux/pci_ids.h
@@ -2529,6 +2529,9 @@
 #define PCI_DEVICE_ID_KORENIX_JETCARDF3	0x17ff
 
 #define PCI_VENDOR_ID_HUAWEI		0x19e5
+#define PCI_DEVICE_ID_HUAWEI_ZIP_VF	0xa251
+#define PCI_DEVICE_ID_HUAWEI_SEC_VF	0xa256
+#define PCI_DEVICE_ID_HUAWEI_HPRE_VF	0xa259
 
 #define PCI_VENDOR_ID_NETRONOME		0x19ee
 #define PCI_DEVICE_ID_NETRONOME_NFP4000	0x4000
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 56+ messages in thread

* [PATCH v8 4/9] hisi_acc_vfio_pci: add new vfio_pci driver for HiSilicon ACC devices
  2022-03-03 23:01 [PATCH v8 0/9] vfio/hisilicon: add ACC live migration driver Shameer Kolothum
                   ` (2 preceding siblings ...)
  2022-03-03 23:01 ` [PATCH v8 3/9] hisi_acc_qm: Move VF PCI device IDs " Shameer Kolothum
@ 2022-03-03 23:01 ` Shameer Kolothum
  2022-03-03 23:01 ` [PATCH v8 5/9] hisi_acc_vfio_pci: Restrict access to VF dev BAR2 migration region Shameer Kolothum
                   ` (4 subsequent siblings)
  8 siblings, 0 replies; 56+ messages in thread
From: Shameer Kolothum @ 2022-03-03 23:01 UTC (permalink / raw)
  To: kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	linuxarm, liulongfang, prime.zeng, jonathan.cameron, wangzhou1

Add a vendor-specific vfio_pci driver for HiSilicon ACC devices.
This will be extended in subsequent patches to add support for VFIO
live migration feature.

Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
---
 drivers/vfio/pci/Kconfig                      |   2 +
 drivers/vfio/pci/Makefile                     |   2 +
 drivers/vfio/pci/hisilicon/Kconfig            |  10 ++
 drivers/vfio/pci/hisilicon/Makefile           |   4 +
 .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    | 100 ++++++++++++++++++
 5 files changed, 118 insertions(+)
 create mode 100644 drivers/vfio/pci/hisilicon/Kconfig
 create mode 100644 drivers/vfio/pci/hisilicon/Makefile
 create mode 100644 drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c

diff --git a/drivers/vfio/pci/Kconfig b/drivers/vfio/pci/Kconfig
index 187b9c259944..4da1914425e1 100644
--- a/drivers/vfio/pci/Kconfig
+++ b/drivers/vfio/pci/Kconfig
@@ -46,4 +46,6 @@ endif
 
 source "drivers/vfio/pci/mlx5/Kconfig"
 
+source "drivers/vfio/pci/hisilicon/Kconfig"
+
 endif
diff --git a/drivers/vfio/pci/Makefile b/drivers/vfio/pci/Makefile
index ed9d6f2e0555..7052ebd893e0 100644
--- a/drivers/vfio/pci/Makefile
+++ b/drivers/vfio/pci/Makefile
@@ -9,3 +9,5 @@ vfio-pci-$(CONFIG_VFIO_PCI_IGD) += vfio_pci_igd.o
 obj-$(CONFIG_VFIO_PCI) += vfio-pci.o
 
 obj-$(CONFIG_MLX5_VFIO_PCI)           += mlx5/
+
+obj-$(CONFIG_HISI_ACC_VFIO_PCI) += hisilicon/
diff --git a/drivers/vfio/pci/hisilicon/Kconfig b/drivers/vfio/pci/hisilicon/Kconfig
new file mode 100644
index 000000000000..dc723bad05c2
--- /dev/null
+++ b/drivers/vfio/pci/hisilicon/Kconfig
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config HISI_ACC_VFIO_PCI
+	tristate "VFIO PCI support for HiSilicon ACC devices"
+	depends on ARM64 || (COMPILE_TEST && 64BIT)
+	depends on VFIO_PCI_CORE
+	help
+	  This provides generic PCI support for HiSilicon ACC devices
+	  using the VFIO framework.
+
+	  If you don't know what to do here, say N.
diff --git a/drivers/vfio/pci/hisilicon/Makefile b/drivers/vfio/pci/hisilicon/Makefile
new file mode 100644
index 000000000000..c66b3783f2f9
--- /dev/null
+++ b/drivers/vfio/pci/hisilicon/Makefile
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: GPL-2.0-only
+obj-$(CONFIG_HISI_ACC_VFIO_PCI) += hisi-acc-vfio-pci.o
+hisi-acc-vfio-pci-y := hisi_acc_vfio_pci.o
+
diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
new file mode 100644
index 000000000000..8129c3457b3b
--- /dev/null
+++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
@@ -0,0 +1,100 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2021, HiSilicon Ltd.
+ */
+
+#include <linux/device.h>
+#include <linux/eventfd.h>
+#include <linux/file.h>
+#include <linux/hisi_acc_qm.h>
+#include <linux/interrupt.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/vfio.h>
+#include <linux/vfio_pci_core.h>
+
+static int hisi_acc_vfio_pci_open_device(struct vfio_device *core_vdev)
+{
+	struct vfio_pci_core_device *vdev =
+		container_of(core_vdev, struct vfio_pci_core_device, vdev);
+	int ret;
+
+	ret = vfio_pci_core_enable(vdev);
+	if (ret)
+		return ret;
+
+	vfio_pci_core_finish_enable(vdev);
+
+	return 0;
+}
+
+static const struct vfio_device_ops hisi_acc_vfio_pci_ops = {
+	.name = "hisi-acc-vfio-pci",
+	.open_device = hisi_acc_vfio_pci_open_device,
+	.close_device = vfio_pci_core_close_device,
+	.ioctl = vfio_pci_core_ioctl,
+	.device_feature = vfio_pci_core_ioctl_feature,
+	.read = vfio_pci_core_read,
+	.write = vfio_pci_core_write,
+	.mmap = vfio_pci_core_mmap,
+	.request = vfio_pci_core_request,
+	.match = vfio_pci_core_match,
+};
+
+static int hisi_acc_vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
+{
+	struct vfio_pci_core_device *vdev;
+	int ret;
+
+	vdev = kzalloc(sizeof(*vdev), GFP_KERNEL);
+	if (!vdev)
+		return -ENOMEM;
+
+	vfio_pci_core_init_device(vdev, pdev, &hisi_acc_vfio_pci_ops);
+
+	ret = vfio_pci_core_register_device(vdev);
+	if (ret)
+		goto out_free;
+
+	dev_set_drvdata(&pdev->dev, vdev);
+
+	return 0;
+
+out_free:
+	vfio_pci_core_uninit_device(vdev);
+	kfree(vdev);
+	return ret;
+}
+
+static void hisi_acc_vfio_pci_remove(struct pci_dev *pdev)
+{
+	struct vfio_pci_core_device *vdev = dev_get_drvdata(&pdev->dev);
+
+	vfio_pci_core_unregister_device(vdev);
+	vfio_pci_core_uninit_device(vdev);
+	kfree(vdev);
+}
+
+static const struct pci_device_id hisi_acc_vfio_pci_table[] = {
+	{ PCI_DRIVER_OVERRIDE_DEVICE_VFIO(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_SEC_VF) },
+	{ PCI_DRIVER_OVERRIDE_DEVICE_VFIO(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_HPRE_VF) },
+	{ PCI_DRIVER_OVERRIDE_DEVICE_VFIO(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_ZIP_VF) },
+	{ }
+};
+
+MODULE_DEVICE_TABLE(pci, hisi_acc_vfio_pci_table);
+
+static struct pci_driver hisi_acc_vfio_pci_driver = {
+	.name = KBUILD_MODNAME,
+	.id_table = hisi_acc_vfio_pci_table,
+	.probe = hisi_acc_vfio_pci_probe,
+	.remove = hisi_acc_vfio_pci_remove,
+	.err_handler = &vfio_pci_core_err_handlers,
+};
+
+module_pci_driver(hisi_acc_vfio_pci_driver);
+
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR("Liu Longfang <liulongfang@huawei.com>");
+MODULE_AUTHOR("Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>");
+MODULE_DESCRIPTION("HiSilicon VFIO PCI - Generic VFIO PCI driver for HiSilicon ACC device family");
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 56+ messages in thread

* [PATCH v8 5/9] hisi_acc_vfio_pci: Restrict access to VF dev BAR2 migration region
  2022-03-03 23:01 [PATCH v8 0/9] vfio/hisilicon: add ACC live migration driver Shameer Kolothum
                   ` (3 preceding siblings ...)
  2022-03-03 23:01 ` [PATCH v8 4/9] hisi_acc_vfio_pci: add new vfio_pci driver for HiSilicon ACC devices Shameer Kolothum
@ 2022-03-03 23:01 ` Shameer Kolothum
  2022-03-08  1:11   ` liulongfang
  2022-03-08  6:23   ` Tian, Kevin
  2022-03-03 23:01 ` [PATCH v8 6/9] hisi_acc_vfio_pci: Add helper to retrieve the struct pci_driver Shameer Kolothum
                   ` (3 subsequent siblings)
  8 siblings, 2 replies; 56+ messages in thread
From: Shameer Kolothum @ 2022-03-03 23:01 UTC (permalink / raw)
  To: kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	linuxarm, liulongfang, prime.zeng, jonathan.cameron, wangzhou1

HiSilicon ACC VF device BAR2 region consists of both functional
register space and migration control register space. From a
security point of view, it's not advisable to export the migration
control region to Guest.

Hence, introduce a separate struct vfio_device_ops for migration
support which will override the ioctl/read/write/mmap methods to
hide the migration region and limit the access only to the
functional register space.

This will be used in subsequent patches when we add migration
support to the driver.

Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
---
 .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    | 126 ++++++++++++++++++
 1 file changed, 126 insertions(+)

diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
index 8129c3457b3b..582ee4fa4109 100644
--- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
+++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
@@ -13,6 +13,119 @@
 #include <linux/vfio.h>
 #include <linux/vfio_pci_core.h>
 
+static int hisi_acc_pci_rw_access_check(struct vfio_device *core_vdev,
+					size_t count, loff_t *ppos,
+					size_t *new_count)
+{
+	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
+	struct vfio_pci_core_device *vdev =
+		container_of(core_vdev, struct vfio_pci_core_device, vdev);
+
+	if (index == VFIO_PCI_BAR2_REGION_INDEX) {
+		loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK;
+		resource_size_t end = pci_resource_len(vdev->pdev, index) / 2;
+
+		/* Check if access is for migration control region */
+		if (pos >= end)
+			return -EINVAL;
+
+		*new_count = min(count, (size_t)(end - pos));
+	}
+
+	return 0;
+}
+
+static int hisi_acc_vfio_pci_mmap(struct vfio_device *core_vdev,
+				  struct vm_area_struct *vma)
+{
+	struct vfio_pci_core_device *vdev =
+		container_of(core_vdev, struct vfio_pci_core_device, vdev);
+	unsigned int index;
+
+	index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT);
+	if (index == VFIO_PCI_BAR2_REGION_INDEX) {
+		u64 req_len, pgoff, req_start;
+		resource_size_t end = pci_resource_len(vdev->pdev, index) / 2;
+
+		req_len = vma->vm_end - vma->vm_start;
+		pgoff = vma->vm_pgoff &
+			((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1);
+		req_start = pgoff << PAGE_SHIFT;
+
+		if (req_start + req_len > end)
+			return -EINVAL;
+	}
+
+	return vfio_pci_core_mmap(core_vdev, vma);
+}
+
+static ssize_t hisi_acc_vfio_pci_write(struct vfio_device *core_vdev,
+				       const char __user *buf, size_t count,
+				       loff_t *ppos)
+{
+	size_t new_count = count;
+	int ret;
+
+	ret = hisi_acc_pci_rw_access_check(core_vdev, count, ppos, &new_count);
+	if (ret)
+		return ret;
+
+	return vfio_pci_core_write(core_vdev, buf, new_count, ppos);
+}
+
+static ssize_t hisi_acc_vfio_pci_read(struct vfio_device *core_vdev,
+				      char __user *buf, size_t count,
+				      loff_t *ppos)
+{
+	size_t new_count = count;
+	int ret;
+
+	ret = hisi_acc_pci_rw_access_check(core_vdev, count, ppos, &new_count);
+	if (ret)
+		return ret;
+
+	return vfio_pci_core_read(core_vdev, buf, new_count, ppos);
+}
+
+static long hisi_acc_vfio_pci_ioctl(struct vfio_device *core_vdev, unsigned int cmd,
+				    unsigned long arg)
+{
+	if (cmd == VFIO_DEVICE_GET_REGION_INFO) {
+		struct vfio_pci_core_device *vdev =
+			container_of(core_vdev, struct vfio_pci_core_device, vdev);
+		struct pci_dev *pdev = vdev->pdev;
+		struct vfio_region_info info;
+		unsigned long minsz;
+
+		minsz = offsetofend(struct vfio_region_info, offset);
+
+		if (copy_from_user(&info, (void __user *)arg, minsz))
+			return -EFAULT;
+
+		if (info.argsz < minsz)
+			return -EINVAL;
+
+		if (info.index == VFIO_PCI_BAR2_REGION_INDEX) {
+			info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);
+
+			/*
+			 * ACC VF dev BAR2 region consists of both functional
+			 * register space and migration control register space.
+			 * Report only the functional region to Guest.
+			 */
+			info.size = pci_resource_len(pdev, info.index) / 2;
+
+			info.flags = VFIO_REGION_INFO_FLAG_READ |
+					VFIO_REGION_INFO_FLAG_WRITE |
+					VFIO_REGION_INFO_FLAG_MMAP;
+
+			return copy_to_user((void __user *)arg, &info, minsz) ?
+					    -EFAULT : 0;
+		}
+	}
+	return vfio_pci_core_ioctl(core_vdev, cmd, arg);
+}
+
 static int hisi_acc_vfio_pci_open_device(struct vfio_device *core_vdev)
 {
 	struct vfio_pci_core_device *vdev =
@@ -28,6 +141,19 @@ static int hisi_acc_vfio_pci_open_device(struct vfio_device *core_vdev)
 	return 0;
 }
 
+static const struct vfio_device_ops hisi_acc_vfio_pci_migrn_ops = {
+	.name = "hisi-acc-vfio-pci-migration",
+	.open_device = hisi_acc_vfio_pci_open_device,
+	.close_device = vfio_pci_core_close_device,
+	.ioctl = hisi_acc_vfio_pci_ioctl,
+	.device_feature = vfio_pci_core_ioctl_feature,
+	.read = hisi_acc_vfio_pci_read,
+	.write = hisi_acc_vfio_pci_write,
+	.mmap = hisi_acc_vfio_pci_mmap,
+	.request = vfio_pci_core_request,
+	.match = vfio_pci_core_match,
+};
+
 static const struct vfio_device_ops hisi_acc_vfio_pci_ops = {
 	.name = "hisi-acc-vfio-pci",
 	.open_device = hisi_acc_vfio_pci_open_device,
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 56+ messages in thread

* [PATCH v8 6/9] hisi_acc_vfio_pci: Add helper to retrieve the struct pci_driver
  2022-03-03 23:01 [PATCH v8 0/9] vfio/hisilicon: add ACC live migration driver Shameer Kolothum
                   ` (4 preceding siblings ...)
  2022-03-03 23:01 ` [PATCH v8 5/9] hisi_acc_vfio_pci: Restrict access to VF dev BAR2 migration region Shameer Kolothum
@ 2022-03-03 23:01 ` Shameer Kolothum
  2022-03-04  9:40   ` Zhou Wang
                     ` (2 more replies)
  2022-03-03 23:01 ` [PATCH v8 7/9] crypto: hisilicon/qm: Set the VF QM state register Shameer Kolothum
                   ` (2 subsequent siblings)
  8 siblings, 3 replies; 56+ messages in thread
From: Shameer Kolothum @ 2022-03-03 23:01 UTC (permalink / raw)
  To: kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	linuxarm, liulongfang, prime.zeng, jonathan.cameron, wangzhou1

struct pci_driver pointer is an input into the pci_iov_get_pf_drvdata().
Introduce helpers to retrieve the ACC PF dev struct pci_driver pointers
as we use this in ACC vfio migration driver.

Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
---
 drivers/crypto/hisilicon/hpre/hpre_main.c | 6 ++++++
 drivers/crypto/hisilicon/sec2/sec_main.c  | 6 ++++++
 drivers/crypto/hisilicon/zip/zip_main.c   | 6 ++++++
 include/linux/hisi_acc_qm.h               | 5 +++++
 4 files changed, 23 insertions(+)

diff --git a/drivers/crypto/hisilicon/hpre/hpre_main.c b/drivers/crypto/hisilicon/hpre/hpre_main.c
index 3589d8879b5e..36ab30e9e654 100644
--- a/drivers/crypto/hisilicon/hpre/hpre_main.c
+++ b/drivers/crypto/hisilicon/hpre/hpre_main.c
@@ -1190,6 +1190,12 @@ static struct pci_driver hpre_pci_driver = {
 	.driver.pm		= &hpre_pm_ops,
 };
 
+struct pci_driver *hisi_hpre_get_pf_driver(void)
+{
+	return &hpre_pci_driver;
+}
+EXPORT_SYMBOL_GPL(hisi_hpre_get_pf_driver);
+
 static void hpre_register_debugfs(void)
 {
 	if (!debugfs_initialized())
diff --git a/drivers/crypto/hisilicon/sec2/sec_main.c b/drivers/crypto/hisilicon/sec2/sec_main.c
index 311a8747b5bf..421a405ca337 100644
--- a/drivers/crypto/hisilicon/sec2/sec_main.c
+++ b/drivers/crypto/hisilicon/sec2/sec_main.c
@@ -1088,6 +1088,12 @@ static struct pci_driver sec_pci_driver = {
 	.driver.pm = &sec_pm_ops,
 };
 
+struct pci_driver *hisi_sec_get_pf_driver(void)
+{
+	return &sec_pci_driver;
+}
+EXPORT_SYMBOL_GPL(hisi_sec_get_pf_driver);
+
 static void sec_register_debugfs(void)
 {
 	if (!debugfs_initialized())
diff --git a/drivers/crypto/hisilicon/zip/zip_main.c b/drivers/crypto/hisilicon/zip/zip_main.c
index 66decfe07282..4534e1e107d1 100644
--- a/drivers/crypto/hisilicon/zip/zip_main.c
+++ b/drivers/crypto/hisilicon/zip/zip_main.c
@@ -1012,6 +1012,12 @@ static struct pci_driver hisi_zip_pci_driver = {
 	.driver.pm		= &hisi_zip_pm_ops,
 };
 
+struct pci_driver *hisi_zip_get_pf_driver(void)
+{
+	return &hisi_zip_pci_driver;
+}
+EXPORT_SYMBOL_GPL(hisi_zip_get_pf_driver);
+
 static void hisi_zip_register_debugfs(void)
 {
 	if (!debugfs_initialized())
diff --git a/include/linux/hisi_acc_qm.h b/include/linux/hisi_acc_qm.h
index 6a6477c34666..00f2a4db8723 100644
--- a/include/linux/hisi_acc_qm.h
+++ b/include/linux/hisi_acc_qm.h
@@ -476,4 +476,9 @@ void hisi_qm_pm_init(struct hisi_qm *qm);
 int hisi_qm_get_dfx_access(struct hisi_qm *qm);
 void hisi_qm_put_dfx_access(struct hisi_qm *qm);
 void hisi_qm_regs_dump(struct seq_file *s, struct debugfs_regset32 *regset);
+
+/* Used by VFIO ACC live migration driver */
+struct pci_driver *hisi_sec_get_pf_driver(void);
+struct pci_driver *hisi_hpre_get_pf_driver(void);
+struct pci_driver *hisi_zip_get_pf_driver(void);
 #endif
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 56+ messages in thread

* [PATCH v8 7/9] crypto: hisilicon/qm: Set the VF QM state register
  2022-03-03 23:01 [PATCH v8 0/9] vfio/hisilicon: add ACC live migration driver Shameer Kolothum
                   ` (5 preceding siblings ...)
  2022-03-03 23:01 ` [PATCH v8 6/9] hisi_acc_vfio_pci: Add helper to retrieve the struct pci_driver Shameer Kolothum
@ 2022-03-03 23:01 ` Shameer Kolothum
  2022-03-04  9:43   ` Zhou Wang
  2022-03-08  6:31   ` Tian, Kevin
  2022-03-03 23:01 ` [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live migration Shameer Kolothum
  2022-03-03 23:01 ` [PATCH v8 9/9] hisi_acc_vfio_pci: Use its own PCI reset_done error handler Shameer Kolothum
  8 siblings, 2 replies; 56+ messages in thread
From: Shameer Kolothum @ 2022-03-03 23:01 UTC (permalink / raw)
  To: kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	linuxarm, liulongfang, prime.zeng, jonathan.cameron, wangzhou1

From: Longfang Liu <liulongfang@huawei.com>

We use VF QM state register to record the status of the QM configuration
state. This will be used in the ACC migration driver to determine whether
we can safely save and restore the QM data.

Signed-off-by: Longfang Liu <liulongfang@huawei.com>
Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
---
 drivers/crypto/hisilicon/qm.c | 8 ++++++++
 include/linux/hisi_acc_qm.h   | 6 ++++++
 2 files changed, 14 insertions(+)

diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c
index c88e013371af..6a8776db38b5 100644
--- a/drivers/crypto/hisilicon/qm.c
+++ b/drivers/crypto/hisilicon/qm.c
@@ -3492,6 +3492,12 @@ static void hisi_qm_pci_uninit(struct hisi_qm *qm)
 	pci_disable_device(pdev);
 }
 
+static void hisi_qm_set_state(struct hisi_qm *qm, u8 state)
+{
+	if (qm->ver > QM_HW_V2 && qm->fun_type == QM_HW_VF)
+		writel(state, qm->io_base + QM_VF_STATE);
+}
+
 /**
  * hisi_qm_uninit() - Uninitialize qm.
  * @qm: The qm needed uninit.
@@ -3520,6 +3526,7 @@ void hisi_qm_uninit(struct hisi_qm *qm)
 		dma_free_coherent(dev, qm->qdma.size,
 				  qm->qdma.va, qm->qdma.dma);
 	}
+	hisi_qm_set_state(qm, QM_NOT_READY);
 	up_write(&qm->qps_lock);
 
 	qm_irq_unregister(qm);
@@ -3745,6 +3752,7 @@ int hisi_qm_start(struct hisi_qm *qm)
 	if (!ret)
 		atomic_set(&qm->status.flags, QM_START);
 
+	hisi_qm_set_state(qm, QM_READY);
 err_unlock:
 	up_write(&qm->qps_lock);
 	return ret;
diff --git a/include/linux/hisi_acc_qm.h b/include/linux/hisi_acc_qm.h
index 00f2a4db8723..177f7b7cd414 100644
--- a/include/linux/hisi_acc_qm.h
+++ b/include/linux/hisi_acc_qm.h
@@ -67,6 +67,7 @@
 #define QM_DB_RAND_SHIFT_V2		16
 #define QM_DB_INDEX_SHIFT_V2		32
 #define QM_DB_PRIORITY_SHIFT_V2		48
+#define QM_VF_STATE			0x60
 
 /* qm cache */
 #define QM_CACHE_CTL			0x100050
@@ -162,6 +163,11 @@ enum qm_debug_file {
 	DEBUG_FILE_NUM,
 };
 
+enum qm_vf_state {
+	QM_READY = 0,
+	QM_NOT_READY,
+};
+
 struct qm_dfx {
 	atomic64_t err_irq_cnt;
 	atomic64_t aeq_irq_cnt;
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 56+ messages in thread

* [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live migration
  2022-03-03 23:01 [PATCH v8 0/9] vfio/hisilicon: add ACC live migration driver Shameer Kolothum
                   ` (6 preceding siblings ...)
  2022-03-03 23:01 ` [PATCH v8 7/9] crypto: hisilicon/qm: Set the VF QM state register Shameer Kolothum
@ 2022-03-03 23:01 ` Shameer Kolothum
  2022-03-04  8:48   ` Shameerali Kolothum Thodi
                     ` (2 more replies)
  2022-03-03 23:01 ` [PATCH v8 9/9] hisi_acc_vfio_pci: Use its own PCI reset_done error handler Shameer Kolothum
  8 siblings, 3 replies; 56+ messages in thread
From: Shameer Kolothum @ 2022-03-03 23:01 UTC (permalink / raw)
  To: kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	linuxarm, liulongfang, prime.zeng, jonathan.cameron, wangzhou1

From: Longfang Liu <liulongfang@huawei.com>

VMs assigned with HiSilicon ACC VF devices can now perform live migration
if the VF devices are bind to the hisi_acc_vfio_pci driver.

Signed-off-by: Longfang Liu <liulongfang@huawei.com>
Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
---
 drivers/vfio/pci/hisilicon/Kconfig            |    7 +
 .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    | 1078 ++++++++++++++++-
 .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h    |  114 ++
 3 files changed, 1181 insertions(+), 18 deletions(-)
 create mode 100644 drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h

diff --git a/drivers/vfio/pci/hisilicon/Kconfig b/drivers/vfio/pci/hisilicon/Kconfig
index dc723bad05c2..2a68d39f339f 100644
--- a/drivers/vfio/pci/hisilicon/Kconfig
+++ b/drivers/vfio/pci/hisilicon/Kconfig
@@ -3,6 +3,13 @@ config HISI_ACC_VFIO_PCI
 	tristate "VFIO PCI support for HiSilicon ACC devices"
 	depends on ARM64 || (COMPILE_TEST && 64BIT)
 	depends on VFIO_PCI_CORE
+	depends on PCI && PCI_MSI
+	depends on UACCE || UACCE=n
+	depends on ACPI
+	select CRYPTO_DEV_HISI_QM
+	select CRYPTO_DEV_HISI_HPRE
+	select CRYPTO_DEV_HISI_SEC2
+	select CRYPTO_DEV_HISI_ZIP
 	help
 	  This provides generic PCI support for HiSilicon ACC devices
 	  using the VFIO framework.
diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
index 582ee4fa4109..aa2e4b6bf598 100644
--- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
+++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
@@ -12,6 +12,993 @@
 #include <linux/pci.h>
 #include <linux/vfio.h>
 #include <linux/vfio_pci_core.h>
+#include <linux/anon_inodes.h>
+
+#include "hisi_acc_vfio_pci.h"
+
+/* return 0 on VM acc device ready, -ETIMEDOUT hardware timeout */
+static int qm_wait_dev_not_ready(struct hisi_qm *qm)
+{
+	u32 val;
+
+	return readl_relaxed_poll_timeout(qm->io_base + QM_VF_STATE,
+				val, !(val & 0x1), MB_POLL_PERIOD_US,
+				MB_POLL_TIMEOUT_US);
+}
+
+/*
+ * Each state Reg is checked 100 times,
+ * with a delay of 100 microseconds after each check
+ */
+static u32 acc_check_reg_state(struct hisi_qm *qm, u32 regs)
+{
+	int check_times = 0;
+	u32 state;
+
+	state = readl(qm->io_base + regs);
+	while (state && check_times < ERROR_CHECK_TIMEOUT) {
+		udelay(CHECK_DELAY_TIME);
+		state = readl(qm->io_base + regs);
+		check_times++;
+	}
+
+	return state;
+}
+
+/* Check the PF's RAS state and Function INT state */
+static int qm_check_int_state(struct hisi_acc_vf_core_device *hisi_acc_vdev)
+{
+	struct hisi_qm *vfqm = &hisi_acc_vdev->vf_qm;
+	struct hisi_qm *qm = hisi_acc_vdev->pf_qm;
+	struct pci_dev *vf_pdev = hisi_acc_vdev->vf_dev;
+	struct device *dev = &qm->pdev->dev;
+	u32 state;
+
+	/* Check RAS state */
+	state = acc_check_reg_state(qm, QM_ABNORMAL_INT_STATUS);
+	if (state) {
+		dev_err(dev, "failed to check QM RAS state!\n");
+		return -EBUSY;
+	}
+
+	/* Check Function Communication state between PF and VF */
+	state = acc_check_reg_state(vfqm, QM_IFC_INT_STATUS);
+	if (state) {
+		dev_err(dev, "failed to check QM IFC INT state!\n");
+		return -EBUSY;
+	}
+	state = acc_check_reg_state(vfqm, QM_IFC_INT_SET_V);
+	if (state) {
+		dev_err(dev, "failed to check QM IFC INT SET state!\n");
+		return -EBUSY;
+	}
+
+	/* Check submodule task state */
+	switch (vf_pdev->device) {
+	case PCI_DEVICE_ID_HUAWEI_SEC_VF:
+		state = acc_check_reg_state(qm, SEC_CORE_INT_STATUS);
+		if (state) {
+			dev_err(dev, "failed to check QM SEC Core INT state!\n");
+			return -EBUSY;
+		}
+		return 0;
+	case PCI_DEVICE_ID_HUAWEI_HPRE_VF:
+		state = acc_check_reg_state(qm, HPRE_HAC_INT_STATUS);
+		if (state) {
+			dev_err(dev, "failed to check QM HPRE HAC INT state!\n");
+			return -EBUSY;
+		}
+		return 0;
+	case PCI_DEVICE_ID_HUAWEI_ZIP_VF:
+		state = acc_check_reg_state(qm, HZIP_CORE_INT_STATUS);
+		if (state) {
+			dev_err(dev, "failed to check QM ZIP Core INT state!\n");
+			return -EBUSY;
+		}
+		return 0;
+	default:
+		dev_err(dev, "failed to detect acc module type!\n");
+		return -EINVAL;
+	}
+}
+
+static int qm_read_reg(struct hisi_qm *qm, u32 reg_addr,
+		       u32 *data, u8 nums)
+{
+	int i;
+
+	if (nums < 1 || nums > QM_REGS_MAX_LEN)
+		return -EINVAL;
+
+	for (i = 0; i < nums; i++) {
+		data[i] = readl(qm->io_base + reg_addr);
+		reg_addr += QM_REG_ADDR_OFFSET;
+	}
+
+	return 0;
+}
+
+static int qm_write_reg(struct hisi_qm *qm, u32 reg,
+			u32 *data, u8 nums)
+{
+	int i;
+
+	if (nums < 1 || nums > QM_REGS_MAX_LEN)
+		return -EINVAL;
+
+	for (i = 0; i < nums; i++)
+		writel(data[i], qm->io_base + reg + i * QM_REG_ADDR_OFFSET);
+
+	return 0;
+}
+
+static int qm_get_vft(struct hisi_qm *qm, u32 *base)
+{
+	u64 sqc_vft;
+	u32 qp_num;
+	int ret;
+
+	ret = hisi_qm_mb(qm, QM_MB_CMD_SQC_VFT_V2, 0, 0, 1);
+	if (ret)
+		return ret;
+
+	sqc_vft = readl(qm->io_base + QM_MB_CMD_DATA_ADDR_L) |
+		  ((u64)readl(qm->io_base + QM_MB_CMD_DATA_ADDR_H) <<
+		  QM_XQC_ADDR_OFFSET);
+	*base = QM_SQC_VFT_BASE_MASK_V2 & (sqc_vft >> QM_SQC_VFT_BASE_SHIFT_V2);
+	qp_num = (QM_SQC_VFT_NUM_MASK_V2 &
+		  (sqc_vft >> QM_SQC_VFT_NUM_SHIFT_V2)) + 1;
+
+	return qp_num;
+}
+
+static int qm_get_sqc(struct hisi_qm *qm, u64 *addr)
+{
+	int ret;
+
+	ret = hisi_qm_mb(qm, QM_MB_CMD_SQC_BT, 0, 0, 1);
+	if (ret)
+		return ret;
+
+	*addr = readl(qm->io_base + QM_MB_CMD_DATA_ADDR_L) |
+		  ((u64)readl(qm->io_base + QM_MB_CMD_DATA_ADDR_H) <<
+		  QM_XQC_ADDR_OFFSET);
+
+	return 0;
+}
+
+static int qm_get_cqc(struct hisi_qm *qm, u64 *addr)
+{
+	int ret;
+
+	ret = hisi_qm_mb(qm, QM_MB_CMD_CQC_BT, 0, 0, 1);
+	if (ret)
+		return ret;
+
+	*addr = readl(qm->io_base + QM_MB_CMD_DATA_ADDR_L) |
+		  ((u64)readl(qm->io_base + QM_MB_CMD_DATA_ADDR_H) <<
+		  QM_XQC_ADDR_OFFSET);
+
+	return 0;
+}
+
+static int qm_rw_regs_read(struct hisi_qm *qm, struct acc_vf_data *vf_data)
+{
+	struct device *dev = &qm->pdev->dev;
+	int ret;
+
+	ret = qm_read_reg(qm, QM_VF_AEQ_INT_MASK, &vf_data->aeq_int_mask, 1);
+	if (ret) {
+		dev_err(dev, "failed to read QM_VF_AEQ_INT_MASK\n");
+		return ret;
+	}
+
+	ret = qm_read_reg(qm, QM_VF_EQ_INT_MASK, &vf_data->eq_int_mask, 1);
+	if (ret) {
+		dev_err(dev, "failed to read QM_VF_EQ_INT_MASK\n");
+		return ret;
+	}
+
+	ret = qm_read_reg(qm, QM_IFC_INT_SOURCE_V,
+			  &vf_data->ifc_int_source, 1);
+	if (ret) {
+		dev_err(dev, "failed to read QM_IFC_INT_SOURCE_V\n");
+		return ret;
+	}
+
+	ret = qm_read_reg(qm, QM_IFC_INT_MASK, &vf_data->ifc_int_mask, 1);
+	if (ret) {
+		dev_err(dev, "failed to read QM_IFC_INT_MASK\n");
+		return ret;
+	}
+
+	ret = qm_read_reg(qm, QM_IFC_INT_SET_V, &vf_data->ifc_int_set, 1);
+	if (ret) {
+		dev_err(dev, "failed to read QM_IFC_INT_SET_V\n");
+		return ret;
+	}
+
+	ret = qm_read_reg(qm, QM_PAGE_SIZE, &vf_data->page_size, 1);
+	if (ret) {
+		dev_err(dev, "failed to read QM_PAGE_SIZE\n");
+		return ret;
+	}
+
+	/* QM_EQC_DW has 7 regs */
+	ret = qm_read_reg(qm, QM_EQC_DW0, vf_data->qm_eqc_dw, 7);
+	if (ret) {
+		dev_err(dev, "failed to read QM_EQC_DW\n");
+		return ret;
+	}
+
+	/* QM_AEQC_DW has 7 regs */
+	ret = qm_read_reg(qm, QM_AEQC_DW0, vf_data->qm_aeqc_dw, 7);
+	if (ret) {
+		dev_err(dev, "failed to read QM_AEQC_DW\n");
+		return ret;
+	}
+
+	return 0;
+}
+
+static int qm_rw_regs_write(struct hisi_qm *qm, struct acc_vf_data *vf_data)
+{
+	struct device *dev = &qm->pdev->dev;
+	int ret;
+
+	/* check VF state */
+	if (unlikely(hisi_qm_wait_mb_ready(qm))) {
+		dev_err(&qm->pdev->dev, "QM device is not ready to write\n");
+		return -EBUSY;
+	}
+
+	ret = qm_write_reg(qm, QM_VF_AEQ_INT_MASK, &vf_data->aeq_int_mask, 1);
+	if (ret) {
+		dev_err(dev, "failed to write QM_VF_AEQ_INT_MASK\n");
+		return ret;
+	}
+
+	ret = qm_write_reg(qm, QM_VF_EQ_INT_MASK, &vf_data->eq_int_mask, 1);
+	if (ret) {
+		dev_err(dev, "failed to write QM_VF_EQ_INT_MASK\n");
+		return ret;
+	}
+
+	ret = qm_write_reg(qm, QM_IFC_INT_SOURCE_V,
+			   &vf_data->ifc_int_source, 1);
+	if (ret) {
+		dev_err(dev, "failed to write QM_IFC_INT_SOURCE_V\n");
+		return ret;
+	}
+
+	ret = qm_write_reg(qm, QM_IFC_INT_MASK, &vf_data->ifc_int_mask, 1);
+	if (ret) {
+		dev_err(dev, "failed to write QM_IFC_INT_MASK\n");
+		return ret;
+	}
+
+	ret = qm_write_reg(qm, QM_IFC_INT_SET_V, &vf_data->ifc_int_set, 1);
+	if (ret) {
+		dev_err(dev, "failed to write QM_IFC_INT_SET_V\n");
+		return ret;
+	}
+
+	ret = qm_write_reg(qm, QM_QUE_ISO_CFG_V, &vf_data->que_iso_cfg, 1);
+	if (ret) {
+		dev_err(dev, "failed to write QM_QUE_ISO_CFG_V\n");
+		return ret;
+	}
+
+	ret = qm_write_reg(qm, QM_PAGE_SIZE, &vf_data->page_size, 1);
+	if (ret) {
+		dev_err(dev, "failed to write QM_PAGE_SIZE\n");
+		return ret;
+	}
+
+	/* QM_EQC_DW has 7 regs */
+	ret = qm_write_reg(qm, QM_EQC_DW0, vf_data->qm_eqc_dw, 7);
+	if (ret) {
+		dev_err(dev, "failed to write QM_EQC_DW\n");
+		return ret;
+	}
+
+	/* QM_AEQC_DW has 7 regs */
+	ret = qm_write_reg(qm, QM_AEQC_DW0, vf_data->qm_aeqc_dw, 7);
+	if (ret) {
+		dev_err(dev, "failed to write QM_AEQC_DW\n");
+		return ret;
+	}
+
+	return 0;
+}
+
+static void qm_db(struct hisi_qm *qm, u16 qn, u8 cmd,
+		  u16 index, u8 priority)
+{
+	u64 doorbell;
+	u64 dbase;
+	u16 randata = 0;
+
+	if (cmd == QM_DOORBELL_CMD_SQ || cmd == QM_DOORBELL_CMD_CQ)
+		dbase = QM_DOORBELL_SQ_CQ_BASE_V2;
+	else
+		dbase = QM_DOORBELL_EQ_AEQ_BASE_V2;
+
+	doorbell = qn | ((u64)cmd << QM_DB_CMD_SHIFT_V2) |
+		   ((u64)randata << QM_DB_RAND_SHIFT_V2) |
+		   ((u64)index << QM_DB_INDEX_SHIFT_V2)	 |
+		   ((u64)priority << QM_DB_PRIORITY_SHIFT_V2);
+
+	writeq(doorbell, qm->io_base + dbase);
+}
+
+static int pf_qm_get_qp_num(struct hisi_qm *qm, int vf_id, u32 *rbase)
+{
+	unsigned int val;
+	u64 sqc_vft;
+	u32 qp_num;
+	int ret;
+
+	ret = readl_relaxed_poll_timeout(qm->io_base + QM_VFT_CFG_RDY, val,
+					 val & BIT(0), MB_POLL_PERIOD_US,
+					 MB_POLL_TIMEOUT_US);
+	if (ret)
+		return ret;
+
+	writel(0x1, qm->io_base + QM_VFT_CFG_OP_WR);
+	/* 0 mean SQC VFT */
+	writel(0x0, qm->io_base + QM_VFT_CFG_TYPE);
+	writel(vf_id, qm->io_base + QM_VFT_CFG);
+
+	writel(0x0, qm->io_base + QM_VFT_CFG_RDY);
+	writel(0x1, qm->io_base + QM_VFT_CFG_OP_ENABLE);
+
+	ret = readl_relaxed_poll_timeout(qm->io_base + QM_VFT_CFG_RDY, val,
+					 val & BIT(0), MB_POLL_PERIOD_US,
+					 MB_POLL_TIMEOUT_US);
+	if (ret)
+		return ret;
+
+	sqc_vft = readl(qm->io_base + QM_VFT_CFG_DATA_L) |
+		  ((u64)readl(qm->io_base + QM_VFT_CFG_DATA_H) <<
+		  QM_XQC_ADDR_OFFSET);
+	*rbase = QM_SQC_VFT_BASE_MASK_V2 &
+		  (sqc_vft >> QM_SQC_VFT_BASE_SHIFT_V2);
+	qp_num = (QM_SQC_VFT_NUM_MASK_V2 &
+		  (sqc_vft >> QM_SQC_VFT_NUM_SHIFT_V2)) + 1;
+
+	return qp_num;
+}
+
+static void qm_dev_cmd_init(struct hisi_qm *qm)
+{
+	/* Clear VF communication status registers. */
+	writel(0x1, qm->io_base + QM_IFC_INT_SOURCE_V);
+
+	/* Enable pf and vf communication. */
+	writel(0x0, qm->io_base + QM_IFC_INT_MASK);
+}
+
+static int vf_qm_cache_wb(struct hisi_qm *qm)
+{
+	unsigned int val;
+
+	writel(0x1, qm->io_base + QM_CACHE_WB_START);
+	if (readl_relaxed_poll_timeout(qm->io_base + QM_CACHE_WB_DONE,
+				       val, val & BIT(0), MB_POLL_PERIOD_US,
+				       MB_POLL_TIMEOUT_US)) {
+		dev_err(&qm->pdev->dev, "vf QM writeback sqc cache fail\n");
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void vf_qm_fun_reset(struct hisi_acc_vf_core_device *hisi_acc_vdev,
+			    struct hisi_qm *qm)
+{
+	int i;
+
+	for (i = 0; i < qm->qp_num; i++)
+		qm_db(qm, i, QM_DOORBELL_CMD_SQ, 0, 1);
+}
+
+static int vf_qm_func_stop(struct hisi_qm *qm)
+{
+	return hisi_qm_mb(qm, QM_MB_CMD_PAUSE_QM, 0, 0, 0);
+}
+
+static int vf_qm_check_match(struct hisi_acc_vf_core_device *hisi_acc_vdev,
+			     struct hisi_acc_vf_migration_file *migf)
+{
+	struct acc_vf_data *vf_data = &migf->vf_data;
+	struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
+	struct hisi_qm *pf_qm = &hisi_acc_vdev->vf_qm;
+	struct device *dev = &vf_qm->pdev->dev;
+	u32 que_iso_state;
+	int ret;
+
+	if (migf->total_length < QM_MATCH_SIZE)
+		return -EINVAL;
+
+	if (vf_data->acc_magic != ACC_DEV_MAGIC) {
+		dev_err(dev, "failed to match ACC_DEV_MAGIC\n");
+		return -EINVAL;
+	}
+
+	if (vf_data->dev_id != hisi_acc_vdev->vf_dev->device) {
+		dev_err(dev, "failed to match VF devices\n");
+		return -EINVAL;
+	}
+
+	/* vf qp num check */
+	ret = qm_get_vft(vf_qm, &vf_qm->qp_base);
+	if (ret <= 0) {
+		dev_err(dev, "failed to get vft qp nums\n");
+		return -EINVAL;
+	}
+
+	if (ret != vf_data->qp_num) {
+		dev_err(dev, "failed to match VF qp num\n");
+		return -EINVAL;
+	}
+
+	vf_qm->qp_num = ret;
+
+	/* vf isolation state check */
+	ret = qm_read_reg(pf_qm, QM_QUE_ISO_CFG_V, &que_iso_state, 1);
+	if (ret) {
+		dev_err(dev, "failed to read QM_QUE_ISO_CFG_V\n");
+		return ret;
+	}
+
+	if (vf_data->que_iso_cfg != que_iso_state) {
+		dev_err(dev, "failed to match isolation state\n");
+		return ret;
+	}
+
+	ret = qm_write_reg(vf_qm, QM_VF_STATE, &vf_data->vf_qm_state, 1);
+	if (ret) {
+		dev_err(dev, "failed to write QM_VF_STATE\n");
+		return ret;
+	}
+
+	hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state;
+	return 0;
+}
+
+static int vf_qm_get_match_data(struct hisi_acc_vf_core_device *hisi_acc_vdev,
+				struct acc_vf_data *vf_data)
+{
+	struct hisi_qm *pf_qm = hisi_acc_vdev->pf_qm;
+	struct device *dev = &pf_qm->pdev->dev;
+	int vf_id = hisi_acc_vdev->vf_id;
+	int ret;
+
+	vf_data->acc_magic = ACC_DEV_MAGIC;
+	/* save device id */
+	vf_data->dev_id = hisi_acc_vdev->vf_dev->device;
+
+	/* vf qp num save from PF */
+	ret = pf_qm_get_qp_num(pf_qm, vf_id, &vf_data->qp_base);
+	if (ret <= 0) {
+		dev_err(dev, "failed to get vft qp nums!\n");
+		return -EINVAL;
+	}
+
+	vf_data->qp_num = ret;
+
+	/* VF isolation state save from PF */
+	ret = qm_read_reg(pf_qm, QM_QUE_ISO_CFG_V, &vf_data->que_iso_cfg, 1);
+	if (ret) {
+		dev_err(dev, "failed to read QM_QUE_ISO_CFG_V!\n");
+		return ret;
+	}
+
+	return 0;
+}
+
+static int vf_qm_load_data(struct hisi_acc_vf_core_device *hisi_acc_vdev,
+			   struct hisi_acc_vf_migration_file *migf)
+{
+	struct hisi_qm *qm = &hisi_acc_vdev->vf_qm;
+	struct device *dev = &qm->pdev->dev;
+	struct acc_vf_data *vf_data = &migf->vf_data;
+	int ret;
+
+	/* Return if only match data was transferred */
+	if (migf->total_length == QM_MATCH_SIZE)
+		return 0;
+
+	if (migf->total_length < sizeof(struct acc_vf_data))
+		return -EINVAL;
+
+	qm->eqe_dma = vf_data->eqe_dma;
+	qm->aeqe_dma = vf_data->aeqe_dma;
+	qm->sqc_dma = vf_data->sqc_dma;
+	qm->cqc_dma = vf_data->cqc_dma;
+
+	qm->qp_base = vf_data->qp_base;
+	qm->qp_num = vf_data->qp_num;
+
+	ret = qm_rw_regs_write(qm, vf_data);
+	if (ret) {
+		dev_err(dev, "Set VF regs failed\n");
+		return ret;
+	}
+
+	ret = hisi_qm_mb(qm, QM_MB_CMD_SQC_BT, qm->sqc_dma, 0, 0);
+	if (ret) {
+		dev_err(dev, "Set sqc failed\n");
+		return ret;
+	}
+
+	ret = hisi_qm_mb(qm, QM_MB_CMD_CQC_BT, qm->cqc_dma, 0, 0);
+	if (ret) {
+		dev_err(dev, "Set cqc failed\n");
+		return ret;
+	}
+
+	qm_dev_cmd_init(qm);
+	return 0;
+}
+
+static int vf_qm_state_save(struct hisi_acc_vf_core_device *hisi_acc_vdev,
+			    struct hisi_acc_vf_migration_file *migf)
+{
+	struct acc_vf_data *vf_data = &migf->vf_data;
+	struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
+	struct device *dev = &vf_qm->pdev->dev;
+	int ret;
+
+	ret = vf_qm_get_match_data(hisi_acc_vdev, vf_data);
+	if (ret)
+		return ret;
+
+	if (unlikely(qm_wait_dev_not_ready(vf_qm))) {
+		/* Update state and return with match data */
+		vf_data->vf_qm_state = QM_NOT_READY;
+		hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state;
+		migf->total_length = QM_MATCH_SIZE;
+		return 0;
+	}
+
+	vf_data->vf_qm_state = QM_READY;
+	hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state;
+
+	ret = vf_qm_cache_wb(vf_qm);
+	if (ret) {
+		dev_err(dev, "failed to writeback QM Cache!\n");
+		return ret;
+	}
+
+	ret = qm_rw_regs_read(vf_qm, vf_data);
+	if (ret)
+		return -EINVAL;
+
+	/* Every reg is 32 bit, the dma address is 64 bit. */
+	vf_data->eqe_dma = vf_data->qm_eqc_dw[2];
+	vf_data->eqe_dma <<= QM_XQC_ADDR_OFFSET;
+	vf_data->eqe_dma |= vf_data->qm_eqc_dw[1];
+	vf_data->aeqe_dma = vf_data->qm_aeqc_dw[2];
+	vf_data->aeqe_dma <<= QM_XQC_ADDR_OFFSET;
+	vf_data->aeqe_dma |= vf_data->qm_aeqc_dw[1];
+
+	/* Through SQC_BT/CQC_BT to get sqc and cqc address */
+	ret = qm_get_sqc(vf_qm, &vf_data->sqc_dma);
+	if (ret) {
+		dev_err(dev, "failed to read SQC addr!\n");
+		return -EINVAL;
+	}
+
+	ret = qm_get_cqc(vf_qm, &vf_data->cqc_dma);
+	if (ret) {
+		dev_err(dev, "failed to read CQC addr!\n");
+		return -EINVAL;
+	}
+
+	migf->total_length = sizeof(struct acc_vf_data);
+	return 0;
+}
+
+static void hisi_acc_vf_disable_fd(struct hisi_acc_vf_migration_file *migf)
+{
+	mutex_lock(&migf->lock);
+	migf->disabled = true;
+	migf->total_length = 0;
+	migf->filp->f_pos = 0;
+	mutex_unlock(&migf->lock);
+}
+
+static void hisi_acc_vf_disable_fds(struct hisi_acc_vf_core_device *hisi_acc_vdev)
+{
+	if (hisi_acc_vdev->resuming_migf) {
+		hisi_acc_vf_disable_fd(hisi_acc_vdev->resuming_migf);
+		fput(hisi_acc_vdev->resuming_migf->filp);
+		hisi_acc_vdev->resuming_migf = NULL;
+	}
+
+	if (hisi_acc_vdev->saving_migf) {
+		hisi_acc_vf_disable_fd(hisi_acc_vdev->saving_migf);
+		fput(hisi_acc_vdev->saving_migf->filp);
+		hisi_acc_vdev->saving_migf = NULL;
+	}
+}
+
+static void hisi_acc_vf_start_device(struct hisi_acc_vf_core_device *hisi_acc_vdev)
+{
+	struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
+
+	if (hisi_acc_vdev->vf_qm_state != QM_READY)
+		return;
+
+	vf_qm_fun_reset(hisi_acc_vdev, vf_qm);
+}
+
+static int hisi_acc_vf_load_state(struct hisi_acc_vf_core_device *hisi_acc_vdev)
+{
+	struct device *dev = &hisi_acc_vdev->vf_dev->dev;
+	struct hisi_acc_vf_migration_file *migf = hisi_acc_vdev->resuming_migf;
+	int ret;
+
+	/* Check dev compatibility */
+	ret = vf_qm_check_match(hisi_acc_vdev, migf);
+	if (ret) {
+		dev_err(dev, "failed to match the VF!\n");
+		return ret;
+	}
+	/* Recover data to VF */
+	ret = vf_qm_load_data(hisi_acc_vdev, migf);
+	if (ret) {
+		dev_err(dev, "failed to recover the VF!\n");
+		return ret;
+	}
+
+	return 0;
+}
+
+static int hisi_acc_vf_release_file(struct inode *inode, struct file *filp)
+{
+	struct hisi_acc_vf_migration_file *migf = filp->private_data;
+
+	hisi_acc_vf_disable_fd(migf);
+	mutex_destroy(&migf->lock);
+	kfree(migf);
+	return 0;
+}
+
+static ssize_t hisi_acc_vf_resume_write(struct file *filp, const char __user *buf,
+					size_t len, loff_t *pos)
+{
+	struct hisi_acc_vf_migration_file *migf = filp->private_data;
+	loff_t requested_length;
+	ssize_t done = 0;
+	int ret;
+
+	if (pos)
+		return -ESPIPE;
+	pos = &filp->f_pos;
+
+	if (*pos < 0 ||
+	    check_add_overflow((loff_t)len, *pos, &requested_length))
+		return -EINVAL;
+
+	if (requested_length > sizeof(struct acc_vf_data))
+		return -ENOMEM;
+
+	mutex_lock(&migf->lock);
+	if (migf->disabled) {
+		done = -ENODEV;
+		goto out_unlock;
+	}
+
+	ret = copy_from_user(&migf->vf_data, buf, len);
+	if (ret) {
+		done = -EFAULT;
+		goto out_unlock;
+	}
+	*pos += len;
+	done = len;
+	migf->total_length += len;
+out_unlock:
+	mutex_unlock(&migf->lock);
+	return done;
+}
+
+static const struct file_operations hisi_acc_vf_resume_fops = {
+	.owner = THIS_MODULE,
+	.write = hisi_acc_vf_resume_write,
+	.release = hisi_acc_vf_release_file,
+	.llseek = no_llseek,
+};
+
+static struct hisi_acc_vf_migration_file *
+hisi_acc_vf_pci_resume(struct hisi_acc_vf_core_device *hisi_acc_vdev)
+{
+	struct hisi_acc_vf_migration_file *migf;
+
+	migf = kzalloc(sizeof(*migf), GFP_KERNEL);
+	if (!migf)
+		return ERR_PTR(-ENOMEM);
+
+	migf->filp = anon_inode_getfile("hisi_acc_vf_mig", &hisi_acc_vf_resume_fops, migf,
+					O_WRONLY);
+	if (IS_ERR(migf->filp)) {
+		int err = PTR_ERR(migf->filp);
+
+		kfree(migf);
+		return ERR_PTR(err);
+	}
+
+	stream_open(migf->filp->f_inode, migf->filp);
+	mutex_init(&migf->lock);
+	return migf;
+}
+
+static ssize_t hisi_acc_vf_save_read(struct file *filp, char __user *buf, size_t len,
+				     loff_t *pos)
+{
+	struct hisi_acc_vf_migration_file *migf = filp->private_data;
+	ssize_t done = 0;
+	int ret;
+
+	if (pos)
+		return -ESPIPE;
+	pos = &filp->f_pos;
+
+	mutex_lock(&migf->lock);
+	if (*pos > migf->total_length) {
+		done = -EINVAL;
+		goto out_unlock;
+	}
+
+	if (migf->disabled) {
+		done = -ENODEV;
+		goto out_unlock;
+	}
+
+	len = min_t(size_t, migf->total_length - *pos, len);
+	if (len) {
+		ret = copy_to_user(buf, &migf->vf_data, len);
+		if (ret) {
+			done = -EFAULT;
+			goto out_unlock;
+		}
+		*pos += len;
+		done = len;
+	}
+out_unlock:
+	mutex_unlock(&migf->lock);
+	return done;
+}
+
+static const struct file_operations hisi_acc_vf_save_fops = {
+	.owner = THIS_MODULE,
+	.read = hisi_acc_vf_save_read,
+	.release = hisi_acc_vf_release_file,
+	.llseek = no_llseek,
+};
+
+static struct hisi_acc_vf_migration_file *
+hisi_acc_vf_stop_copy(struct hisi_acc_vf_core_device *hisi_acc_vdev)
+{
+	struct hisi_acc_vf_migration_file *migf;
+	int ret;
+
+	migf = kzalloc(sizeof(*migf), GFP_KERNEL);
+	if (!migf)
+		return ERR_PTR(-ENOMEM);
+
+	migf->filp = anon_inode_getfile("hisi_acc_vf_mig", &hisi_acc_vf_save_fops, migf,
+					O_RDONLY);
+	if (IS_ERR(migf->filp)) {
+		int err = PTR_ERR(migf->filp);
+
+		kfree(migf);
+		return ERR_PTR(err);
+	}
+
+	stream_open(migf->filp->f_inode, migf->filp);
+	mutex_init(&migf->lock);
+
+	ret = vf_qm_state_save(hisi_acc_vdev, migf);
+	if (ret) {
+		fput(migf->filp);
+		return ERR_PTR(ret);
+	}
+
+	return migf;
+}
+
+static int hisi_acc_vf_stop_device(struct hisi_acc_vf_core_device *hisi_acc_vdev)
+{
+	struct device *dev = &hisi_acc_vdev->vf_dev->dev;
+	struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
+	int ret;
+
+	ret = vf_qm_func_stop(vf_qm);
+	if (ret) {
+		dev_err(dev, "failed to stop QM VF function!\n");
+		return ret;
+	}
+
+	ret = qm_check_int_state(hisi_acc_vdev);
+	if (ret) {
+		dev_err(dev, "failed to check QM INT state!\n");
+		return ret;
+	}
+	return 0;
+}
+
+static struct file *
+hisi_acc_vf_set_device_state(struct hisi_acc_vf_core_device *hisi_acc_vdev,
+			     u32 new)
+{
+	u32 cur = hisi_acc_vdev->mig_state;
+	int ret;
+
+	if (cur == VFIO_DEVICE_STATE_RUNNING && new == VFIO_DEVICE_STATE_STOP) {
+		ret = hisi_acc_vf_stop_device(hisi_acc_vdev);
+		if (ret)
+			return ERR_PTR(ret);
+		return NULL;
+	}
+
+	if (cur == VFIO_DEVICE_STATE_STOP && new == VFIO_DEVICE_STATE_STOP_COPY) {
+		struct hisi_acc_vf_migration_file *migf;
+
+		migf = hisi_acc_vf_stop_copy(hisi_acc_vdev);
+		if (IS_ERR(migf))
+			return ERR_CAST(migf);
+		get_file(migf->filp);
+		hisi_acc_vdev->saving_migf = migf;
+		return migf->filp;
+	}
+
+	if ((cur == VFIO_DEVICE_STATE_STOP_COPY && new == VFIO_DEVICE_STATE_STOP)) {
+		hisi_acc_vf_disable_fds(hisi_acc_vdev);
+		return NULL;
+	}
+
+	if (cur == VFIO_DEVICE_STATE_STOP && new == VFIO_DEVICE_STATE_RESUMING) {
+		struct hisi_acc_vf_migration_file *migf;
+
+		migf = hisi_acc_vf_pci_resume(hisi_acc_vdev);
+		if (IS_ERR(migf))
+			return ERR_CAST(migf);
+		get_file(migf->filp);
+		hisi_acc_vdev->resuming_migf = migf;
+		return migf->filp;
+	}
+
+	if (cur == VFIO_DEVICE_STATE_RESUMING && new == VFIO_DEVICE_STATE_STOP) {
+		ret = hisi_acc_vf_load_state(hisi_acc_vdev);
+		if (ret)
+			return ERR_PTR(ret);
+		hisi_acc_vf_disable_fds(hisi_acc_vdev);
+		return NULL;
+	}
+
+	if (cur == VFIO_DEVICE_STATE_STOP && new == VFIO_DEVICE_STATE_RUNNING) {
+		hisi_acc_vf_start_device(hisi_acc_vdev);
+		return NULL;
+	}
+
+	/*
+	 * vfio_mig_get_next_state() does not use arcs other than the above
+	 */
+	WARN_ON(true);
+	return ERR_PTR(-EINVAL);
+}
+
+static struct file *
+hisi_acc_vfio_pci_set_device_state(struct vfio_device *vdev,
+				   enum vfio_device_mig_state new_state)
+{
+	struct hisi_acc_vf_core_device *hisi_acc_vdev = container_of(vdev,
+			struct hisi_acc_vf_core_device, core_device.vdev);
+	enum vfio_device_mig_state next_state;
+	struct file *res = NULL;
+	int ret;
+
+	mutex_lock(&hisi_acc_vdev->state_mutex);
+	while (new_state != hisi_acc_vdev->mig_state) {
+		ret = vfio_mig_get_next_state(vdev,
+					      hisi_acc_vdev->mig_state,
+					      new_state, &next_state);
+		if (ret) {
+			res = ERR_PTR(-EINVAL);
+			break;
+		}
+
+		res = hisi_acc_vf_set_device_state(hisi_acc_vdev, next_state);
+		if (IS_ERR(res))
+			break;
+		hisi_acc_vdev->mig_state = next_state;
+		if (WARN_ON(res && new_state != hisi_acc_vdev->mig_state)) {
+			fput(res);
+			res = ERR_PTR(-EINVAL);
+			break;
+		}
+	}
+	mutex_unlock(&hisi_acc_vdev->state_mutex);
+	return res;
+}
+
+static int
+hisi_acc_vfio_pci_get_device_state(struct vfio_device *vdev,
+				   enum vfio_device_mig_state *curr_state)
+{
+	struct hisi_acc_vf_core_device *hisi_acc_vdev = container_of(vdev,
+			struct hisi_acc_vf_core_device, core_device.vdev);
+
+	mutex_lock(&hisi_acc_vdev->state_mutex);
+	*curr_state = hisi_acc_vdev->mig_state;
+	mutex_unlock(&hisi_acc_vdev->state_mutex);
+	return 0;
+}
+
+static int hisi_acc_vf_qm_init(struct hisi_acc_vf_core_device *hisi_acc_vdev)
+{
+	struct vfio_pci_core_device *vdev = &hisi_acc_vdev->core_device;
+	struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
+	struct pci_dev *vf_dev = vdev->pdev;
+
+	/*
+	 * ACC VF dev BAR2 region consists of both functional register space
+	 * and migration control register space. For migration to work, we
+	 * need access to both. Hence, we map the entire BAR2 region here.
+	 * But from a security point of view, we restrict access to the
+	 * migration control space from Guest(Please see mmap/ioctl/read/write
+	 * override functions).
+	 *
+	 * Also the HiSilicon ACC VF devices supported by this driver on
+	 * HiSilicon hardware platforms are integrated end point devices
+	 * and has no capability to perform PCIe P2P.
+	 */
+
+	vf_qm->io_base =
+		ioremap(pci_resource_start(vf_dev, VFIO_PCI_BAR2_REGION_INDEX),
+			pci_resource_len(vf_dev, VFIO_PCI_BAR2_REGION_INDEX));
+	if (!vf_qm->io_base)
+		return -EIO;
+
+	vf_qm->fun_type = QM_HW_VF;
+	vf_qm->pdev = vf_dev;
+	mutex_init(&vf_qm->mailbox_lock);
+
+	return 0;
+}
+
+static struct hisi_qm *hisi_acc_get_pf_qm(struct pci_dev *pdev)
+{
+	struct hisi_qm	*pf_qm;
+	struct pci_driver *pf_driver;
+
+	if (!pdev->is_virtfn)
+		return NULL;
+
+	switch (pdev->device) {
+	case PCI_DEVICE_ID_HUAWEI_SEC_VF:
+		pf_driver = hisi_sec_get_pf_driver();
+		break;
+	case PCI_DEVICE_ID_HUAWEI_HPRE_VF:
+		pf_driver = hisi_hpre_get_pf_driver();
+		break;
+	case PCI_DEVICE_ID_HUAWEI_ZIP_VF:
+		pf_driver = hisi_zip_get_pf_driver();
+		break;
+	default:
+		return NULL;
+	}
+
+	if (!pf_driver)
+		return NULL;
+
+	pf_qm = pci_iov_get_pf_drvdata(pdev, pf_driver);
+
+	return !IS_ERR(pf_qm) ? pf_qm : NULL;
+}
 
 static int hisi_acc_pci_rw_access_check(struct vfio_device *core_vdev,
 					size_t count, loff_t *ppos,
@@ -128,23 +1115,42 @@ static long hisi_acc_vfio_pci_ioctl(struct vfio_device *core_vdev, unsigned int
 
 static int hisi_acc_vfio_pci_open_device(struct vfio_device *core_vdev)
 {
-	struct vfio_pci_core_device *vdev =
-		container_of(core_vdev, struct vfio_pci_core_device, vdev);
+	struct hisi_acc_vf_core_device *hisi_acc_vdev = container_of(core_vdev,
+			struct hisi_acc_vf_core_device, core_device.vdev);
+	struct vfio_pci_core_device *vdev = &hisi_acc_vdev->core_device;
 	int ret;
 
 	ret = vfio_pci_core_enable(vdev);
 	if (ret)
 		return ret;
 
-	vfio_pci_core_finish_enable(vdev);
+	if (core_vdev->ops->migration_set_state) {
+		ret = hisi_acc_vf_qm_init(hisi_acc_vdev);
+		if (ret) {
+			vfio_pci_core_disable(vdev);
+			return ret;
+		}
+		hisi_acc_vdev->mig_state = VFIO_DEVICE_STATE_RUNNING;
+	}
 
+	vfio_pci_core_finish_enable(vdev);
 	return 0;
 }
 
+static void hisi_acc_vfio_pci_close_device(struct vfio_device *core_vdev)
+{
+	struct hisi_acc_vf_core_device *hisi_acc_vdev = container_of(core_vdev,
+			struct hisi_acc_vf_core_device, core_device.vdev);
+	struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
+
+	iounmap(vf_qm->io_base);
+	vfio_pci_core_close_device(core_vdev);
+}
+
 static const struct vfio_device_ops hisi_acc_vfio_pci_migrn_ops = {
 	.name = "hisi-acc-vfio-pci-migration",
 	.open_device = hisi_acc_vfio_pci_open_device,
-	.close_device = vfio_pci_core_close_device,
+	.close_device = hisi_acc_vfio_pci_close_device,
 	.ioctl = hisi_acc_vfio_pci_ioctl,
 	.device_feature = vfio_pci_core_ioctl_feature,
 	.read = hisi_acc_vfio_pci_read,
@@ -152,6 +1158,8 @@ static const struct vfio_device_ops hisi_acc_vfio_pci_migrn_ops = {
 	.mmap = hisi_acc_vfio_pci_mmap,
 	.request = vfio_pci_core_request,
 	.match = vfio_pci_core_match,
+	.migration_set_state = hisi_acc_vfio_pci_set_device_state,
+	.migration_get_state = hisi_acc_vfio_pci_get_device_state,
 };
 
 static const struct vfio_device_ops hisi_acc_vfio_pci_ops = {
@@ -167,38 +1175,72 @@ static const struct vfio_device_ops hisi_acc_vfio_pci_ops = {
 	.match = vfio_pci_core_match,
 };
 
+static int
+hisi_acc_vfio_pci_migrn_init(struct hisi_acc_vf_core_device *hisi_acc_vdev,
+			     struct pci_dev *pdev, struct hisi_qm *pf_qm)
+{
+	int vf_id;
+
+	vf_id = pci_iov_vf_id(pdev);
+	if (vf_id < 0)
+		return vf_id;
+
+	hisi_acc_vdev->vf_id = vf_id + 1;
+	hisi_acc_vdev->core_device.vdev.migration_flags =
+					VFIO_MIGRATION_STOP_COPY;
+	hisi_acc_vdev->pf_qm = pf_qm;
+	hisi_acc_vdev->vf_dev = pdev;
+	mutex_init(&hisi_acc_vdev->state_mutex);
+
+	return 0;
+}
+
 static int hisi_acc_vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 {
-	struct vfio_pci_core_device *vdev;
+	struct hisi_acc_vf_core_device *hisi_acc_vdev;
+	struct hisi_qm *pf_qm;
 	int ret;
 
-	vdev = kzalloc(sizeof(*vdev), GFP_KERNEL);
-	if (!vdev)
+	hisi_acc_vdev = kzalloc(sizeof(*hisi_acc_vdev), GFP_KERNEL);
+	if (!hisi_acc_vdev)
 		return -ENOMEM;
 
-	vfio_pci_core_init_device(vdev, pdev, &hisi_acc_vfio_pci_ops);
+	pf_qm = hisi_acc_get_pf_qm(pdev);
+	if (pf_qm && pf_qm->ver >= QM_HW_V3) {
+		ret = hisi_acc_vfio_pci_migrn_init(hisi_acc_vdev, pdev, pf_qm);
+		if (!ret) {
+			vfio_pci_core_init_device(&hisi_acc_vdev->core_device, pdev,
+						  &hisi_acc_vfio_pci_migrn_ops);
+		} else {
+			pci_warn(pdev, "migration support failed, continue with generic interface\n");
+			vfio_pci_core_init_device(&hisi_acc_vdev->core_device, pdev,
+						  &hisi_acc_vfio_pci_ops);
+		}
+	} else {
+		vfio_pci_core_init_device(&hisi_acc_vdev->core_device, pdev,
+					  &hisi_acc_vfio_pci_ops);
+	}
 
-	ret = vfio_pci_core_register_device(vdev);
+	ret = vfio_pci_core_register_device(&hisi_acc_vdev->core_device);
 	if (ret)
 		goto out_free;
 
-	dev_set_drvdata(&pdev->dev, vdev);
-
+	dev_set_drvdata(&pdev->dev, hisi_acc_vdev);
 	return 0;
 
 out_free:
-	vfio_pci_core_uninit_device(vdev);
-	kfree(vdev);
+	vfio_pci_core_uninit_device(&hisi_acc_vdev->core_device);
+	kfree(hisi_acc_vdev);
 	return ret;
 }
 
 static void hisi_acc_vfio_pci_remove(struct pci_dev *pdev)
 {
-	struct vfio_pci_core_device *vdev = dev_get_drvdata(&pdev->dev);
+	struct hisi_acc_vf_core_device *hisi_acc_vdev = dev_get_drvdata(&pdev->dev);
 
-	vfio_pci_core_unregister_device(vdev);
-	vfio_pci_core_uninit_device(vdev);
-	kfree(vdev);
+	vfio_pci_core_unregister_device(&hisi_acc_vdev->core_device);
+	vfio_pci_core_uninit_device(&hisi_acc_vdev->core_device);
+	kfree(hisi_acc_vdev);
 }
 
 static const struct pci_device_id hisi_acc_vfio_pci_table[] = {
@@ -223,4 +1265,4 @@ module_pci_driver(hisi_acc_vfio_pci_driver);
 MODULE_LICENSE("GPL v2");
 MODULE_AUTHOR("Liu Longfang <liulongfang@huawei.com>");
 MODULE_AUTHOR("Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>");
-MODULE_DESCRIPTION("HiSilicon VFIO PCI - Generic VFIO PCI driver for HiSilicon ACC device family");
+MODULE_DESCRIPTION("HiSilicon VFIO PCI - VFIO PCI driver with live migration support for HiSilicon ACC device family");
diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
new file mode 100644
index 000000000000..1c7d75408790
--- /dev/null
+++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
@@ -0,0 +1,114 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2021 HiSilicon Ltd. */
+
+#ifndef HISI_ACC_VFIO_PCI_H
+#define HISI_ACC_VFIO_PCI_H
+
+#include <linux/hisi_acc_qm.h>
+
+#define MB_POLL_PERIOD_US		10
+#define MB_POLL_TIMEOUT_US		1000
+#define QM_CACHE_WB_START		0x204
+#define QM_CACHE_WB_DONE		0x208
+#define QM_MB_CMD_PAUSE_QM		0xe
+#define QM_ABNORMAL_INT_STATUS		0x100008
+#define QM_IFC_INT_STATUS		0x0028
+#define SEC_CORE_INT_STATUS		0x301008
+#define HPRE_HAC_INT_STATUS		0x301800
+#define HZIP_CORE_INT_STATUS		0x3010AC
+#define QM_QUE_ISO_CFG			0x301154
+
+#define QM_VFT_CFG_RDY			0x10006c
+#define QM_VFT_CFG_OP_WR		0x100058
+#define QM_VFT_CFG_TYPE			0x10005c
+#define QM_VFT_CFG			0x100060
+#define QM_VFT_CFG_OP_ENABLE		0x100054
+#define QM_VFT_CFG_DATA_L		0x100064
+#define QM_VFT_CFG_DATA_H		0x100068
+
+#define ERROR_CHECK_TIMEOUT		100
+#define CHECK_DELAY_TIME		100
+
+#define QM_SQC_VFT_BASE_SHIFT_V2	28
+#define QM_SQC_VFT_BASE_MASK_V2		GENMASK(15, 0)
+#define QM_SQC_VFT_NUM_SHIFT_V2		45
+#define QM_SQC_VFT_NUM_MASK_V2		GENMASK(9, 0)
+
+/* RW regs */
+#define QM_REGS_MAX_LEN		7
+#define QM_REG_ADDR_OFFSET	0x0004
+
+#define QM_XQC_ADDR_OFFSET	32U
+#define QM_VF_AEQ_INT_MASK	0x0004
+#define QM_VF_EQ_INT_MASK	0x000c
+#define QM_IFC_INT_SOURCE_V	0x0020
+#define QM_IFC_INT_MASK		0x0024
+#define QM_IFC_INT_SET_V	0x002c
+#define QM_QUE_ISO_CFG_V	0x0030
+#define QM_PAGE_SIZE		0x0034
+
+#define QM_EQC_DW0		0X8000
+#define QM_AEQC_DW0		0X8020
+
+struct acc_vf_data {
+#define QM_MATCH_SIZE offsetofend(struct acc_vf_data, qm_rsv_state)
+	/* QM match information */
+#define ACC_DEV_MAGIC	0XCDCDCDCDFEEDAACC
+	u64 acc_magic;
+	u32 qp_num;
+	u32 dev_id;
+	u32 que_iso_cfg;
+	u32 qp_base;
+	u32 vf_qm_state;
+	/* QM reserved match information */
+	u32 qm_rsv_state[3];
+
+	/* QM RW regs */
+	u32 aeq_int_mask;
+	u32 eq_int_mask;
+	u32 ifc_int_source;
+	u32 ifc_int_mask;
+	u32 ifc_int_set;
+	u32 page_size;
+
+	/* QM_EQC_DW has 7 regs */
+	u32 qm_eqc_dw[7];
+
+	/* QM_AEQC_DW has 7 regs */
+	u32 qm_aeqc_dw[7];
+
+	/* QM reserved 5 regs */
+	u32 qm_rsv_regs[5];
+	u32 padding;
+	/* qm memory init information */
+	u64 eqe_dma;
+	u64 aeqe_dma;
+	u64 sqc_dma;
+	u64 cqc_dma;
+};
+
+struct hisi_acc_vf_migration_file {
+	struct file *filp;
+	struct mutex lock;
+	bool disabled;
+
+	struct acc_vf_data vf_data;
+	size_t total_length;
+};
+
+struct hisi_acc_vf_core_device {
+	struct vfio_pci_core_device core_device;
+	/* for migration state */
+	struct mutex state_mutex;
+	enum vfio_device_mig_state mig_state;
+	struct pci_dev *pf_dev;
+	struct pci_dev *vf_dev;
+	struct hisi_qm *pf_qm;
+	struct hisi_qm vf_qm;
+	u32 vf_qm_state;
+	int vf_id;
+
+	struct hisi_acc_vf_migration_file *resuming_migf;
+	struct hisi_acc_vf_migration_file *saving_migf;
+};
+#endif /* HISI_ACC_VFIO_PCI_H */
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 56+ messages in thread

* [PATCH v8 9/9] hisi_acc_vfio_pci: Use its own PCI reset_done error handler
  2022-03-03 23:01 [PATCH v8 0/9] vfio/hisilicon: add ACC live migration driver Shameer Kolothum
                   ` (7 preceding siblings ...)
  2022-03-03 23:01 ` [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live migration Shameer Kolothum
@ 2022-03-03 23:01 ` Shameer Kolothum
  2022-03-04 20:54   ` Jason Gunthorpe
  2022-03-08  1:14   ` liulongfang
  8 siblings, 2 replies; 56+ messages in thread
From: Shameer Kolothum @ 2022-03-03 23:01 UTC (permalink / raw)
  To: kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	linuxarm, liulongfang, prime.zeng, jonathan.cameron, wangzhou1

Register private handler for pci_error_handlers.reset_done and update
state accordingly.

Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
---
 .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    | 57 ++++++++++++++++++-
 .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h    |  4 +-
 2 files changed, 57 insertions(+), 4 deletions(-)

diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
index aa2e4b6bf598..53e4c5cb3a71 100644
--- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
+++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
@@ -625,6 +625,27 @@ static void hisi_acc_vf_disable_fds(struct hisi_acc_vf_core_device *hisi_acc_vde
 	}
 }
 
+/*
+ * This function is called in all state_mutex unlock cases to
+ * handle a 'deferred_reset' if exists.
+ */
+static void
+hisi_acc_vf_state_mutex_unlock(struct hisi_acc_vf_core_device *hisi_acc_vdev)
+{
+again:
+	spin_lock(&hisi_acc_vdev->reset_lock);
+	if (hisi_acc_vdev->deferred_reset) {
+		hisi_acc_vdev->deferred_reset = false;
+		spin_unlock(&hisi_acc_vdev->reset_lock);
+		hisi_acc_vdev->vf_qm_state = QM_NOT_READY;
+		hisi_acc_vdev->mig_state = VFIO_DEVICE_STATE_RUNNING;
+		hisi_acc_vf_disable_fds(hisi_acc_vdev);
+		goto again;
+	}
+	mutex_unlock(&hisi_acc_vdev->state_mutex);
+	spin_unlock(&hisi_acc_vdev->reset_lock);
+}
+
 static void hisi_acc_vf_start_device(struct hisi_acc_vf_core_device *hisi_acc_vdev)
 {
 	struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
@@ -921,7 +942,7 @@ hisi_acc_vfio_pci_set_device_state(struct vfio_device *vdev,
 			break;
 		}
 	}
-	mutex_unlock(&hisi_acc_vdev->state_mutex);
+	hisi_acc_vf_state_mutex_unlock(hisi_acc_vdev);
 	return res;
 }
 
@@ -934,10 +955,35 @@ hisi_acc_vfio_pci_get_device_state(struct vfio_device *vdev,
 
 	mutex_lock(&hisi_acc_vdev->state_mutex);
 	*curr_state = hisi_acc_vdev->mig_state;
-	mutex_unlock(&hisi_acc_vdev->state_mutex);
+	hisi_acc_vf_state_mutex_unlock(hisi_acc_vdev);
 	return 0;
 }
 
+static void hisi_acc_vf_pci_aer_reset_done(struct pci_dev *pdev)
+{
+	struct hisi_acc_vf_core_device *hisi_acc_vdev = dev_get_drvdata(&pdev->dev);
+
+	if (hisi_acc_vdev->core_device.vdev.migration_flags !=
+				VFIO_MIGRATION_STOP_COPY)
+		return;
+
+	/*
+	 * As the higher VFIO layers are holding locks across reset and using
+	 * those same locks with the mm_lock we need to prevent ABBA deadlock
+	 * with the state_mutex and mm_lock.
+	 * In case the state_mutex was taken already we defer the cleanup work
+	 * to the unlock flow of the other running context.
+	 */
+	spin_lock(&hisi_acc_vdev->reset_lock);
+	hisi_acc_vdev->deferred_reset = true;
+	if (!mutex_trylock(&hisi_acc_vdev->state_mutex)) {
+		spin_unlock(&hisi_acc_vdev->reset_lock);
+		return;
+	}
+	spin_unlock(&hisi_acc_vdev->reset_lock);
+	hisi_acc_vf_state_mutex_unlock(hisi_acc_vdev);
+}
+
 static int hisi_acc_vf_qm_init(struct hisi_acc_vf_core_device *hisi_acc_vdev)
 {
 	struct vfio_pci_core_device *vdev = &hisi_acc_vdev->core_device;
@@ -1252,12 +1298,17 @@ static const struct pci_device_id hisi_acc_vfio_pci_table[] = {
 
 MODULE_DEVICE_TABLE(pci, hisi_acc_vfio_pci_table);
 
+static const struct pci_error_handlers hisi_acc_vf_err_handlers = {
+	.reset_done = hisi_acc_vf_pci_aer_reset_done,
+	.error_detected = vfio_pci_core_aer_err_detected,
+};
+
 static struct pci_driver hisi_acc_vfio_pci_driver = {
 	.name = KBUILD_MODNAME,
 	.id_table = hisi_acc_vfio_pci_table,
 	.probe = hisi_acc_vfio_pci_probe,
 	.remove = hisi_acc_vfio_pci_remove,
-	.err_handler = &vfio_pci_core_err_handlers,
+	.err_handler = &hisi_acc_vf_err_handlers,
 };
 
 module_pci_driver(hisi_acc_vfio_pci_driver);
diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
index 1c7d75408790..5494f4983bbe 100644
--- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
+++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
@@ -98,6 +98,7 @@ struct hisi_acc_vf_migration_file {
 
 struct hisi_acc_vf_core_device {
 	struct vfio_pci_core_device core_device;
+	u8 deferred_reset:1;
 	/* for migration state */
 	struct mutex state_mutex;
 	enum vfio_device_mig_state mig_state;
@@ -107,7 +108,8 @@ struct hisi_acc_vf_core_device {
 	struct hisi_qm vf_qm;
 	u32 vf_qm_state;
 	int vf_id;
-
+	/* for reset handler */
+	spinlock_t reset_lock;
 	struct hisi_acc_vf_migration_file *resuming_migf;
 	struct hisi_acc_vf_migration_file *saving_migf;
 };
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 56+ messages in thread

* RE: [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live migration
  2022-03-03 23:01 ` [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live migration Shameer Kolothum
@ 2022-03-04  8:48   ` Shameerali Kolothum Thodi
  2022-03-04 19:44     ` Alex Williamson
  2022-03-04 20:57   ` Jason Gunthorpe
  2022-03-08  7:41   ` Tian, Kevin
  2 siblings, 1 reply; 56+ messages in thread
From: Shameerali Kolothum Thodi @ 2022-03-04  8:48 UTC (permalink / raw)
  To: kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	liulongfang, Zengtao (B), Jonathan Cameron, Wangzhou (B)

Hi Alex,

> -----Original Message-----
> From: Shameerali Kolothum Thodi
> Sent: 03 March 2022 23:02
> To: kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-crypto@vger.kernel.org
> Cc: linux-pci@vger.kernel.org; alex.williamson@redhat.com; jgg@nvidia.com;
> cohuck@redhat.com; mgurtovoy@nvidia.com; yishaih@nvidia.com; Linuxarm
> <linuxarm@huawei.com>; liulongfang <liulongfang@huawei.com>; Zengtao (B)
> <prime.zeng@hisilicon.com>; Jonathan Cameron
> <jonathan.cameron@huawei.com>; Wangzhou (B) <wangzhou1@hisilicon.com>
> Subject: [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live migration
> 
> From: Longfang Liu <liulongfang@huawei.com>
> 
> VMs assigned with HiSilicon ACC VF devices can now perform live migration if
> the VF devices are bind to the hisi_acc_vfio_pci driver.
> 
> Signed-off-by: Longfang Liu <liulongfang@huawei.com>
> Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>

[...]
> +
> +static int vf_qm_check_match(struct hisi_acc_vf_core_device *hisi_acc_vdev,
> +			     struct hisi_acc_vf_migration_file *migf) {
> +	struct acc_vf_data *vf_data = &migf->vf_data;
> +	struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
> +	struct hisi_qm *pf_qm = &hisi_acc_vdev->vf_qm;

Oops, the above has to be,
  struct hisi_qm *pf_qm = hisi_acc_vdev->pf_qm;

This was actually fixed in v6, but now that I rebased mainly to v5, missed it.
Please let me know if you want a re-spin with the above fix(in case there are no further
comments) or this is something you can take care.

Thanks,
Shameer

> +	struct device *dev = &vf_qm->pdev->dev;
> +	u32 que_iso_state;
> +	int ret;
> +
> +	if (migf->total_length < QM_MATCH_SIZE)
> +		return -EINVAL;
> +
> +	if (vf_data->acc_magic != ACC_DEV_MAGIC) {
> +		dev_err(dev, "failed to match ACC_DEV_MAGIC\n");
> +		return -EINVAL;
> +	}
> +
> +	if (vf_data->dev_id != hisi_acc_vdev->vf_dev->device) {
> +		dev_err(dev, "failed to match VF devices\n");
> +		return -EINVAL;
> +	}
> +
> +	/* vf qp num check */
> +	ret = qm_get_vft(vf_qm, &vf_qm->qp_base);
> +	if (ret <= 0) {
> +		dev_err(dev, "failed to get vft qp nums\n");
> +		return -EINVAL;
> +	}
> +
> +	if (ret != vf_data->qp_num) {
> +		dev_err(dev, "failed to match VF qp num\n");
> +		return -EINVAL;
> +	}
> +
> +	vf_qm->qp_num = ret;
> +
> +	/* vf isolation state check */
> +	ret = qm_read_reg(pf_qm, QM_QUE_ISO_CFG_V, &que_iso_state, 1);
> +	if (ret) {
> +		dev_err(dev, "failed to read QM_QUE_ISO_CFG_V\n");
> +		return ret;
> +	}
> +
> +	if (vf_data->que_iso_cfg != que_iso_state) {
> +		dev_err(dev, "failed to match isolation state\n");
> +		return ret;
> +	}
> +
> +	ret = qm_write_reg(vf_qm, QM_VF_STATE, &vf_data->vf_qm_state, 1);
> +	if (ret) {
> +		dev_err(dev, "failed to write QM_VF_STATE\n");
> +		return ret;
> +	}
> +
> +	hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state;
> +	return 0;
> +}
> +
> +static int vf_qm_get_match_data(struct hisi_acc_vf_core_device
> *hisi_acc_vdev,
> +				struct acc_vf_data *vf_data)
> +{
> +	struct hisi_qm *pf_qm = hisi_acc_vdev->pf_qm;
> +	struct device *dev = &pf_qm->pdev->dev;
> +	int vf_id = hisi_acc_vdev->vf_id;
> +	int ret;
> +
> +	vf_data->acc_magic = ACC_DEV_MAGIC;
> +	/* save device id */
> +	vf_data->dev_id = hisi_acc_vdev->vf_dev->device;
> +
> +	/* vf qp num save from PF */
> +	ret = pf_qm_get_qp_num(pf_qm, vf_id, &vf_data->qp_base);
> +	if (ret <= 0) {
> +		dev_err(dev, "failed to get vft qp nums!\n");
> +		return -EINVAL;
> +	}
> +
> +	vf_data->qp_num = ret;
> +
> +	/* VF isolation state save from PF */
> +	ret = qm_read_reg(pf_qm, QM_QUE_ISO_CFG_V, &vf_data->que_iso_cfg,
> 1);
> +	if (ret) {
> +		dev_err(dev, "failed to read QM_QUE_ISO_CFG_V!\n");
> +		return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static int vf_qm_load_data(struct hisi_acc_vf_core_device *hisi_acc_vdev,
> +			   struct hisi_acc_vf_migration_file *migf) {
> +	struct hisi_qm *qm = &hisi_acc_vdev->vf_qm;
> +	struct device *dev = &qm->pdev->dev;
> +	struct acc_vf_data *vf_data = &migf->vf_data;
> +	int ret;
> +
> +	/* Return if only match data was transferred */
> +	if (migf->total_length == QM_MATCH_SIZE)
> +		return 0;
> +
> +	if (migf->total_length < sizeof(struct acc_vf_data))
> +		return -EINVAL;
> +
> +	qm->eqe_dma = vf_data->eqe_dma;
> +	qm->aeqe_dma = vf_data->aeqe_dma;
> +	qm->sqc_dma = vf_data->sqc_dma;
> +	qm->cqc_dma = vf_data->cqc_dma;
> +
> +	qm->qp_base = vf_data->qp_base;
> +	qm->qp_num = vf_data->qp_num;
> +
> +	ret = qm_rw_regs_write(qm, vf_data);
> +	if (ret) {
> +		dev_err(dev, "Set VF regs failed\n");
> +		return ret;
> +	}
> +
> +	ret = hisi_qm_mb(qm, QM_MB_CMD_SQC_BT, qm->sqc_dma, 0, 0);
> +	if (ret) {
> +		dev_err(dev, "Set sqc failed\n");
> +		return ret;
> +	}
> +
> +	ret = hisi_qm_mb(qm, QM_MB_CMD_CQC_BT, qm->cqc_dma, 0, 0);
> +	if (ret) {
> +		dev_err(dev, "Set cqc failed\n");
> +		return ret;
> +	}
> +
> +	qm_dev_cmd_init(qm);
> +	return 0;
> +}
> +
> +static int vf_qm_state_save(struct hisi_acc_vf_core_device *hisi_acc_vdev,
> +			    struct hisi_acc_vf_migration_file *migf) {
> +	struct acc_vf_data *vf_data = &migf->vf_data;
> +	struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
> +	struct device *dev = &vf_qm->pdev->dev;
> +	int ret;
> +
> +	ret = vf_qm_get_match_data(hisi_acc_vdev, vf_data);
> +	if (ret)
> +		return ret;
> +
> +	if (unlikely(qm_wait_dev_not_ready(vf_qm))) {
> +		/* Update state and return with match data */
> +		vf_data->vf_qm_state = QM_NOT_READY;
> +		hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state;
> +		migf->total_length = QM_MATCH_SIZE;
> +		return 0;
> +	}
> +
> +	vf_data->vf_qm_state = QM_READY;
> +	hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state;
> +
> +	ret = vf_qm_cache_wb(vf_qm);
> +	if (ret) {
> +		dev_err(dev, "failed to writeback QM Cache!\n");
> +		return ret;
> +	}
> +
> +	ret = qm_rw_regs_read(vf_qm, vf_data);
> +	if (ret)
> +		return -EINVAL;
> +
> +	/* Every reg is 32 bit, the dma address is 64 bit. */
> +	vf_data->eqe_dma = vf_data->qm_eqc_dw[2];
> +	vf_data->eqe_dma <<= QM_XQC_ADDR_OFFSET;
> +	vf_data->eqe_dma |= vf_data->qm_eqc_dw[1];
> +	vf_data->aeqe_dma = vf_data->qm_aeqc_dw[2];
> +	vf_data->aeqe_dma <<= QM_XQC_ADDR_OFFSET;
> +	vf_data->aeqe_dma |= vf_data->qm_aeqc_dw[1];
> +
> +	/* Through SQC_BT/CQC_BT to get sqc and cqc address */
> +	ret = qm_get_sqc(vf_qm, &vf_data->sqc_dma);
> +	if (ret) {
> +		dev_err(dev, "failed to read SQC addr!\n");
> +		return -EINVAL;
> +	}
> +
> +	ret = qm_get_cqc(vf_qm, &vf_data->cqc_dma);
> +	if (ret) {
> +		dev_err(dev, "failed to read CQC addr!\n");
> +		return -EINVAL;
> +	}
> +
> +	migf->total_length = sizeof(struct acc_vf_data);
> +	return 0;
> +}
> +
> +static void hisi_acc_vf_disable_fd(struct hisi_acc_vf_migration_file
> +*migf) {
> +	mutex_lock(&migf->lock);
> +	migf->disabled = true;
> +	migf->total_length = 0;
> +	migf->filp->f_pos = 0;
> +	mutex_unlock(&migf->lock);
> +}
> +
> +static void hisi_acc_vf_disable_fds(struct hisi_acc_vf_core_device
> +*hisi_acc_vdev) {
> +	if (hisi_acc_vdev->resuming_migf) {
> +		hisi_acc_vf_disable_fd(hisi_acc_vdev->resuming_migf);
> +		fput(hisi_acc_vdev->resuming_migf->filp);
> +		hisi_acc_vdev->resuming_migf = NULL;
> +	}
> +
> +	if (hisi_acc_vdev->saving_migf) {
> +		hisi_acc_vf_disable_fd(hisi_acc_vdev->saving_migf);
> +		fput(hisi_acc_vdev->saving_migf->filp);
> +		hisi_acc_vdev->saving_migf = NULL;
> +	}
> +}
> +
> +static void hisi_acc_vf_start_device(struct hisi_acc_vf_core_device
> +*hisi_acc_vdev) {
> +	struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
> +
> +	if (hisi_acc_vdev->vf_qm_state != QM_READY)
> +		return;
> +
> +	vf_qm_fun_reset(hisi_acc_vdev, vf_qm); }
> +
> +static int hisi_acc_vf_load_state(struct hisi_acc_vf_core_device
> +*hisi_acc_vdev) {
> +	struct device *dev = &hisi_acc_vdev->vf_dev->dev;
> +	struct hisi_acc_vf_migration_file *migf = hisi_acc_vdev->resuming_migf;
> +	int ret;
> +
> +	/* Check dev compatibility */
> +	ret = vf_qm_check_match(hisi_acc_vdev, migf);
> +	if (ret) {
> +		dev_err(dev, "failed to match the VF!\n");
> +		return ret;
> +	}
> +	/* Recover data to VF */
> +	ret = vf_qm_load_data(hisi_acc_vdev, migf);
> +	if (ret) {
> +		dev_err(dev, "failed to recover the VF!\n");
> +		return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static int hisi_acc_vf_release_file(struct inode *inode, struct file
> +*filp) {
> +	struct hisi_acc_vf_migration_file *migf = filp->private_data;
> +
> +	hisi_acc_vf_disable_fd(migf);
> +	mutex_destroy(&migf->lock);
> +	kfree(migf);
> +	return 0;
> +}
> +
> +static ssize_t hisi_acc_vf_resume_write(struct file *filp, const char __user
> *buf,
> +					size_t len, loff_t *pos)
> +{
> +	struct hisi_acc_vf_migration_file *migf = filp->private_data;
> +	loff_t requested_length;
> +	ssize_t done = 0;
> +	int ret;
> +
> +	if (pos)
> +		return -ESPIPE;
> +	pos = &filp->f_pos;
> +
> +	if (*pos < 0 ||
> +	    check_add_overflow((loff_t)len, *pos, &requested_length))
> +		return -EINVAL;
> +
> +	if (requested_length > sizeof(struct acc_vf_data))
> +		return -ENOMEM;
> +
> +	mutex_lock(&migf->lock);
> +	if (migf->disabled) {
> +		done = -ENODEV;
> +		goto out_unlock;
> +	}
> +
> +	ret = copy_from_user(&migf->vf_data, buf, len);
> +	if (ret) {
> +		done = -EFAULT;
> +		goto out_unlock;
> +	}
> +	*pos += len;
> +	done = len;
> +	migf->total_length += len;
> +out_unlock:
> +	mutex_unlock(&migf->lock);
> +	return done;
> +}
> +
> +static const struct file_operations hisi_acc_vf_resume_fops = {
> +	.owner = THIS_MODULE,
> +	.write = hisi_acc_vf_resume_write,
> +	.release = hisi_acc_vf_release_file,
> +	.llseek = no_llseek,
> +};
> +
> +static struct hisi_acc_vf_migration_file *
> +hisi_acc_vf_pci_resume(struct hisi_acc_vf_core_device *hisi_acc_vdev) {
> +	struct hisi_acc_vf_migration_file *migf;
> +
> +	migf = kzalloc(sizeof(*migf), GFP_KERNEL);
> +	if (!migf)
> +		return ERR_PTR(-ENOMEM);
> +
> +	migf->filp = anon_inode_getfile("hisi_acc_vf_mig",
> &hisi_acc_vf_resume_fops, migf,
> +					O_WRONLY);
> +	if (IS_ERR(migf->filp)) {
> +		int err = PTR_ERR(migf->filp);
> +
> +		kfree(migf);
> +		return ERR_PTR(err);
> +	}
> +
> +	stream_open(migf->filp->f_inode, migf->filp);
> +	mutex_init(&migf->lock);
> +	return migf;
> +}
> +
> +static ssize_t hisi_acc_vf_save_read(struct file *filp, char __user *buf, size_t
> len,
> +				     loff_t *pos)
> +{
> +	struct hisi_acc_vf_migration_file *migf = filp->private_data;
> +	ssize_t done = 0;
> +	int ret;
> +
> +	if (pos)
> +		return -ESPIPE;
> +	pos = &filp->f_pos;
> +
> +	mutex_lock(&migf->lock);
> +	if (*pos > migf->total_length) {
> +		done = -EINVAL;
> +		goto out_unlock;
> +	}
> +
> +	if (migf->disabled) {
> +		done = -ENODEV;
> +		goto out_unlock;
> +	}
> +
> +	len = min_t(size_t, migf->total_length - *pos, len);
> +	if (len) {
> +		ret = copy_to_user(buf, &migf->vf_data, len);
> +		if (ret) {
> +			done = -EFAULT;
> +			goto out_unlock;
> +		}
> +		*pos += len;
> +		done = len;
> +	}
> +out_unlock:
> +	mutex_unlock(&migf->lock);
> +	return done;
> +}
> +
> +static const struct file_operations hisi_acc_vf_save_fops = {
> +	.owner = THIS_MODULE,
> +	.read = hisi_acc_vf_save_read,
> +	.release = hisi_acc_vf_release_file,
> +	.llseek = no_llseek,
> +};
> +
> +static struct hisi_acc_vf_migration_file * hisi_acc_vf_stop_copy(struct
> +hisi_acc_vf_core_device *hisi_acc_vdev) {
> +	struct hisi_acc_vf_migration_file *migf;
> +	int ret;
> +
> +	migf = kzalloc(sizeof(*migf), GFP_KERNEL);
> +	if (!migf)
> +		return ERR_PTR(-ENOMEM);
> +
> +	migf->filp = anon_inode_getfile("hisi_acc_vf_mig",
> &hisi_acc_vf_save_fops, migf,
> +					O_RDONLY);
> +	if (IS_ERR(migf->filp)) {
> +		int err = PTR_ERR(migf->filp);
> +
> +		kfree(migf);
> +		return ERR_PTR(err);
> +	}
> +
> +	stream_open(migf->filp->f_inode, migf->filp);
> +	mutex_init(&migf->lock);
> +
> +	ret = vf_qm_state_save(hisi_acc_vdev, migf);
> +	if (ret) {
> +		fput(migf->filp);
> +		return ERR_PTR(ret);
> +	}
> +
> +	return migf;
> +}
> +
> +static int hisi_acc_vf_stop_device(struct hisi_acc_vf_core_device
> +*hisi_acc_vdev) {
> +	struct device *dev = &hisi_acc_vdev->vf_dev->dev;
> +	struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
> +	int ret;
> +
> +	ret = vf_qm_func_stop(vf_qm);
> +	if (ret) {
> +		dev_err(dev, "failed to stop QM VF function!\n");
> +		return ret;
> +	}
> +
> +	ret = qm_check_int_state(hisi_acc_vdev);
> +	if (ret) {
> +		dev_err(dev, "failed to check QM INT state!\n");
> +		return ret;
> +	}
> +	return 0;
> +}
> +
> +static struct file *
> +hisi_acc_vf_set_device_state(struct hisi_acc_vf_core_device *hisi_acc_vdev,
> +			     u32 new)
> +{
> +	u32 cur = hisi_acc_vdev->mig_state;
> +	int ret;
> +
> +	if (cur == VFIO_DEVICE_STATE_RUNNING && new ==
> VFIO_DEVICE_STATE_STOP) {
> +		ret = hisi_acc_vf_stop_device(hisi_acc_vdev);
> +		if (ret)
> +			return ERR_PTR(ret);
> +		return NULL;
> +	}
> +
> +	if (cur == VFIO_DEVICE_STATE_STOP && new ==
> VFIO_DEVICE_STATE_STOP_COPY) {
> +		struct hisi_acc_vf_migration_file *migf;
> +
> +		migf = hisi_acc_vf_stop_copy(hisi_acc_vdev);
> +		if (IS_ERR(migf))
> +			return ERR_CAST(migf);
> +		get_file(migf->filp);
> +		hisi_acc_vdev->saving_migf = migf;
> +		return migf->filp;
> +	}
> +
> +	if ((cur == VFIO_DEVICE_STATE_STOP_COPY && new ==
> VFIO_DEVICE_STATE_STOP)) {
> +		hisi_acc_vf_disable_fds(hisi_acc_vdev);
> +		return NULL;
> +	}
> +
> +	if (cur == VFIO_DEVICE_STATE_STOP && new ==
> VFIO_DEVICE_STATE_RESUMING) {
> +		struct hisi_acc_vf_migration_file *migf;
> +
> +		migf = hisi_acc_vf_pci_resume(hisi_acc_vdev);
> +		if (IS_ERR(migf))
> +			return ERR_CAST(migf);
> +		get_file(migf->filp);
> +		hisi_acc_vdev->resuming_migf = migf;
> +		return migf->filp;
> +	}
> +
> +	if (cur == VFIO_DEVICE_STATE_RESUMING && new ==
> VFIO_DEVICE_STATE_STOP) {
> +		ret = hisi_acc_vf_load_state(hisi_acc_vdev);
> +		if (ret)
> +			return ERR_PTR(ret);
> +		hisi_acc_vf_disable_fds(hisi_acc_vdev);
> +		return NULL;
> +	}
> +
> +	if (cur == VFIO_DEVICE_STATE_STOP && new ==
> VFIO_DEVICE_STATE_RUNNING) {
> +		hisi_acc_vf_start_device(hisi_acc_vdev);
> +		return NULL;
> +	}
> +
> +	/*
> +	 * vfio_mig_get_next_state() does not use arcs other than the above
> +	 */
> +	WARN_ON(true);
> +	return ERR_PTR(-EINVAL);
> +}
> +
> +static struct file *
> +hisi_acc_vfio_pci_set_device_state(struct vfio_device *vdev,
> +				   enum vfio_device_mig_state new_state) {
> +	struct hisi_acc_vf_core_device *hisi_acc_vdev = container_of(vdev,
> +			struct hisi_acc_vf_core_device, core_device.vdev);
> +	enum vfio_device_mig_state next_state;
> +	struct file *res = NULL;
> +	int ret;
> +
> +	mutex_lock(&hisi_acc_vdev->state_mutex);
> +	while (new_state != hisi_acc_vdev->mig_state) {
> +		ret = vfio_mig_get_next_state(vdev,
> +					      hisi_acc_vdev->mig_state,
> +					      new_state, &next_state);
> +		if (ret) {
> +			res = ERR_PTR(-EINVAL);
> +			break;
> +		}
> +
> +		res = hisi_acc_vf_set_device_state(hisi_acc_vdev, next_state);
> +		if (IS_ERR(res))
> +			break;
> +		hisi_acc_vdev->mig_state = next_state;
> +		if (WARN_ON(res && new_state != hisi_acc_vdev->mig_state)) {
> +			fput(res);
> +			res = ERR_PTR(-EINVAL);
> +			break;
> +		}
> +	}
> +	mutex_unlock(&hisi_acc_vdev->state_mutex);
> +	return res;
> +}
> +
> +static int
> +hisi_acc_vfio_pci_get_device_state(struct vfio_device *vdev,
> +				   enum vfio_device_mig_state *curr_state) {
> +	struct hisi_acc_vf_core_device *hisi_acc_vdev = container_of(vdev,
> +			struct hisi_acc_vf_core_device, core_device.vdev);
> +
> +	mutex_lock(&hisi_acc_vdev->state_mutex);
> +	*curr_state = hisi_acc_vdev->mig_state;
> +	mutex_unlock(&hisi_acc_vdev->state_mutex);
> +	return 0;
> +}
> +
> +static int hisi_acc_vf_qm_init(struct hisi_acc_vf_core_device
> +*hisi_acc_vdev) {
> +	struct vfio_pci_core_device *vdev = &hisi_acc_vdev->core_device;
> +	struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
> +	struct pci_dev *vf_dev = vdev->pdev;
> +
> +	/*
> +	 * ACC VF dev BAR2 region consists of both functional register space
> +	 * and migration control register space. For migration to work, we
> +	 * need access to both. Hence, we map the entire BAR2 region here.
> +	 * But from a security point of view, we restrict access to the
> +	 * migration control space from Guest(Please see mmap/ioctl/read/write
> +	 * override functions).
> +	 *
> +	 * Also the HiSilicon ACC VF devices supported by this driver on
> +	 * HiSilicon hardware platforms are integrated end point devices
> +	 * and has no capability to perform PCIe P2P.
> +	 */
> +
> +	vf_qm->io_base =
> +		ioremap(pci_resource_start(vf_dev, VFIO_PCI_BAR2_REGION_INDEX),
> +			pci_resource_len(vf_dev, VFIO_PCI_BAR2_REGION_INDEX));
> +	if (!vf_qm->io_base)
> +		return -EIO;
> +
> +	vf_qm->fun_type = QM_HW_VF;
> +	vf_qm->pdev = vf_dev;
> +	mutex_init(&vf_qm->mailbox_lock);
> +
> +	return 0;
> +}
> +
> +static struct hisi_qm *hisi_acc_get_pf_qm(struct pci_dev *pdev) {
> +	struct hisi_qm	*pf_qm;
> +	struct pci_driver *pf_driver;
> +
> +	if (!pdev->is_virtfn)
> +		return NULL;
> +
> +	switch (pdev->device) {
> +	case PCI_DEVICE_ID_HUAWEI_SEC_VF:
> +		pf_driver = hisi_sec_get_pf_driver();
> +		break;
> +	case PCI_DEVICE_ID_HUAWEI_HPRE_VF:
> +		pf_driver = hisi_hpre_get_pf_driver();
> +		break;
> +	case PCI_DEVICE_ID_HUAWEI_ZIP_VF:
> +		pf_driver = hisi_zip_get_pf_driver();
> +		break;
> +	default:
> +		return NULL;
> +	}
> +
> +	if (!pf_driver)
> +		return NULL;
> +
> +	pf_qm = pci_iov_get_pf_drvdata(pdev, pf_driver);
> +
> +	return !IS_ERR(pf_qm) ? pf_qm : NULL;
> +}
> 
>  static int hisi_acc_pci_rw_access_check(struct vfio_device *core_vdev,
>  					size_t count, loff_t *ppos,
> @@ -128,23 +1115,42 @@ static long hisi_acc_vfio_pci_ioctl(struct
> vfio_device *core_vdev, unsigned int
> 
>  static int hisi_acc_vfio_pci_open_device(struct vfio_device *core_vdev)  {
> -	struct vfio_pci_core_device *vdev =
> -		container_of(core_vdev, struct vfio_pci_core_device, vdev);
> +	struct hisi_acc_vf_core_device *hisi_acc_vdev = container_of(core_vdev,
> +			struct hisi_acc_vf_core_device, core_device.vdev);
> +	struct vfio_pci_core_device *vdev = &hisi_acc_vdev->core_device;
>  	int ret;
> 
>  	ret = vfio_pci_core_enable(vdev);
>  	if (ret)
>  		return ret;
> 
> -	vfio_pci_core_finish_enable(vdev);
> +	if (core_vdev->ops->migration_set_state) {
> +		ret = hisi_acc_vf_qm_init(hisi_acc_vdev);
> +		if (ret) {
> +			vfio_pci_core_disable(vdev);
> +			return ret;
> +		}
> +		hisi_acc_vdev->mig_state = VFIO_DEVICE_STATE_RUNNING;
> +	}
> 
> +	vfio_pci_core_finish_enable(vdev);
>  	return 0;
>  }
> 
> +static void hisi_acc_vfio_pci_close_device(struct vfio_device
> +*core_vdev) {
> +	struct hisi_acc_vf_core_device *hisi_acc_vdev = container_of(core_vdev,
> +			struct hisi_acc_vf_core_device, core_device.vdev);
> +	struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
> +
> +	iounmap(vf_qm->io_base);
> +	vfio_pci_core_close_device(core_vdev);
> +}
> +
>  static const struct vfio_device_ops hisi_acc_vfio_pci_migrn_ops = {
>  	.name = "hisi-acc-vfio-pci-migration",
>  	.open_device = hisi_acc_vfio_pci_open_device,
> -	.close_device = vfio_pci_core_close_device,
> +	.close_device = hisi_acc_vfio_pci_close_device,
>  	.ioctl = hisi_acc_vfio_pci_ioctl,
>  	.device_feature = vfio_pci_core_ioctl_feature,
>  	.read = hisi_acc_vfio_pci_read,
> @@ -152,6 +1158,8 @@ static const struct vfio_device_ops
> hisi_acc_vfio_pci_migrn_ops = {
>  	.mmap = hisi_acc_vfio_pci_mmap,
>  	.request = vfio_pci_core_request,
>  	.match = vfio_pci_core_match,
> +	.migration_set_state = hisi_acc_vfio_pci_set_device_state,
> +	.migration_get_state = hisi_acc_vfio_pci_get_device_state,
>  };
> 
>  static const struct vfio_device_ops hisi_acc_vfio_pci_ops = { @@ -167,38
> +1175,72 @@ static const struct vfio_device_ops hisi_acc_vfio_pci_ops = {
>  	.match = vfio_pci_core_match,
>  };
> 
> +static int
> +hisi_acc_vfio_pci_migrn_init(struct hisi_acc_vf_core_device *hisi_acc_vdev,
> +			     struct pci_dev *pdev, struct hisi_qm *pf_qm) {
> +	int vf_id;
> +
> +	vf_id = pci_iov_vf_id(pdev);
> +	if (vf_id < 0)
> +		return vf_id;
> +
> +	hisi_acc_vdev->vf_id = vf_id + 1;
> +	hisi_acc_vdev->core_device.vdev.migration_flags =
> +					VFIO_MIGRATION_STOP_COPY;
> +	hisi_acc_vdev->pf_qm = pf_qm;
> +	hisi_acc_vdev->vf_dev = pdev;
> +	mutex_init(&hisi_acc_vdev->state_mutex);
> +
> +	return 0;
> +}
> +
>  static int hisi_acc_vfio_pci_probe(struct pci_dev *pdev, const struct
> pci_device_id *id)  {
> -	struct vfio_pci_core_device *vdev;
> +	struct hisi_acc_vf_core_device *hisi_acc_vdev;
> +	struct hisi_qm *pf_qm;
>  	int ret;
> 
> -	vdev = kzalloc(sizeof(*vdev), GFP_KERNEL);
> -	if (!vdev)
> +	hisi_acc_vdev = kzalloc(sizeof(*hisi_acc_vdev), GFP_KERNEL);
> +	if (!hisi_acc_vdev)
>  		return -ENOMEM;
> 
> -	vfio_pci_core_init_device(vdev, pdev, &hisi_acc_vfio_pci_ops);
> +	pf_qm = hisi_acc_get_pf_qm(pdev);
> +	if (pf_qm && pf_qm->ver >= QM_HW_V3) {
> +		ret = hisi_acc_vfio_pci_migrn_init(hisi_acc_vdev, pdev, pf_qm);
> +		if (!ret) {
> +			vfio_pci_core_init_device(&hisi_acc_vdev->core_device, pdev,
> +						  &hisi_acc_vfio_pci_migrn_ops);
> +		} else {
> +			pci_warn(pdev, "migration support failed, continue with generic
> interface\n");
> +			vfio_pci_core_init_device(&hisi_acc_vdev->core_device, pdev,
> +						  &hisi_acc_vfio_pci_ops);
> +		}
> +	} else {
> +		vfio_pci_core_init_device(&hisi_acc_vdev->core_device, pdev,
> +					  &hisi_acc_vfio_pci_ops);
> +	}
> 
> -	ret = vfio_pci_core_register_device(vdev);
> +	ret = vfio_pci_core_register_device(&hisi_acc_vdev->core_device);
>  	if (ret)
>  		goto out_free;
> 
> -	dev_set_drvdata(&pdev->dev, vdev);
> -
> +	dev_set_drvdata(&pdev->dev, hisi_acc_vdev);
>  	return 0;
> 
>  out_free:
> -	vfio_pci_core_uninit_device(vdev);
> -	kfree(vdev);
> +	vfio_pci_core_uninit_device(&hisi_acc_vdev->core_device);
> +	kfree(hisi_acc_vdev);
>  	return ret;
>  }
> 
>  static void hisi_acc_vfio_pci_remove(struct pci_dev *pdev)  {
> -	struct vfio_pci_core_device *vdev = dev_get_drvdata(&pdev->dev);
> +	struct hisi_acc_vf_core_device *hisi_acc_vdev =
> +dev_get_drvdata(&pdev->dev);
> 
> -	vfio_pci_core_unregister_device(vdev);
> -	vfio_pci_core_uninit_device(vdev);
> -	kfree(vdev);
> +	vfio_pci_core_unregister_device(&hisi_acc_vdev->core_device);
> +	vfio_pci_core_uninit_device(&hisi_acc_vdev->core_device);
> +	kfree(hisi_acc_vdev);
>  }
> 
>  static const struct pci_device_id hisi_acc_vfio_pci_table[] = { @@ -223,4
> +1265,4 @@ module_pci_driver(hisi_acc_vfio_pci_driver);
>  MODULE_LICENSE("GPL v2");
>  MODULE_AUTHOR("Liu Longfang <liulongfang@huawei.com>");
> MODULE_AUTHOR("Shameer Kolothum
> <shameerali.kolothum.thodi@huawei.com>");
> -MODULE_DESCRIPTION("HiSilicon VFIO PCI - Generic VFIO PCI driver for
> HiSilicon ACC device family");
> +MODULE_DESCRIPTION("HiSilicon VFIO PCI - VFIO PCI driver with live
> +migration support for HiSilicon ACC device family");
> diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
> b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
> new file mode 100644
> index 000000000000..1c7d75408790
> --- /dev/null
> +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
> @@ -0,0 +1,114 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/* Copyright (c) 2021 HiSilicon Ltd. */
> +
> +#ifndef HISI_ACC_VFIO_PCI_H
> +#define HISI_ACC_VFIO_PCI_H
> +
> +#include <linux/hisi_acc_qm.h>
> +
> +#define MB_POLL_PERIOD_US		10
> +#define MB_POLL_TIMEOUT_US		1000
> +#define QM_CACHE_WB_START		0x204
> +#define QM_CACHE_WB_DONE		0x208
> +#define QM_MB_CMD_PAUSE_QM		0xe
> +#define QM_ABNORMAL_INT_STATUS		0x100008
> +#define QM_IFC_INT_STATUS		0x0028
> +#define SEC_CORE_INT_STATUS		0x301008
> +#define HPRE_HAC_INT_STATUS		0x301800
> +#define HZIP_CORE_INT_STATUS		0x3010AC
> +#define QM_QUE_ISO_CFG			0x301154
> +
> +#define QM_VFT_CFG_RDY			0x10006c
> +#define QM_VFT_CFG_OP_WR		0x100058
> +#define QM_VFT_CFG_TYPE			0x10005c
> +#define QM_VFT_CFG			0x100060
> +#define QM_VFT_CFG_OP_ENABLE		0x100054
> +#define QM_VFT_CFG_DATA_L		0x100064
> +#define QM_VFT_CFG_DATA_H		0x100068
> +
> +#define ERROR_CHECK_TIMEOUT		100
> +#define CHECK_DELAY_TIME		100
> +
> +#define QM_SQC_VFT_BASE_SHIFT_V2	28
> +#define QM_SQC_VFT_BASE_MASK_V2		GENMASK(15, 0)
> +#define QM_SQC_VFT_NUM_SHIFT_V2		45
> +#define QM_SQC_VFT_NUM_MASK_V2		GENMASK(9, 0)
> +
> +/* RW regs */
> +#define QM_REGS_MAX_LEN		7
> +#define QM_REG_ADDR_OFFSET	0x0004
> +
> +#define QM_XQC_ADDR_OFFSET	32U
> +#define QM_VF_AEQ_INT_MASK	0x0004
> +#define QM_VF_EQ_INT_MASK	0x000c
> +#define QM_IFC_INT_SOURCE_V	0x0020
> +#define QM_IFC_INT_MASK		0x0024
> +#define QM_IFC_INT_SET_V	0x002c
> +#define QM_QUE_ISO_CFG_V	0x0030
> +#define QM_PAGE_SIZE		0x0034
> +
> +#define QM_EQC_DW0		0X8000
> +#define QM_AEQC_DW0		0X8020
> +
> +struct acc_vf_data {
> +#define QM_MATCH_SIZE offsetofend(struct acc_vf_data, qm_rsv_state)
> +	/* QM match information */
> +#define ACC_DEV_MAGIC	0XCDCDCDCDFEEDAACC
> +	u64 acc_magic;
> +	u32 qp_num;
> +	u32 dev_id;
> +	u32 que_iso_cfg;
> +	u32 qp_base;
> +	u32 vf_qm_state;
> +	/* QM reserved match information */
> +	u32 qm_rsv_state[3];
> +
> +	/* QM RW regs */
> +	u32 aeq_int_mask;
> +	u32 eq_int_mask;
> +	u32 ifc_int_source;
> +	u32 ifc_int_mask;
> +	u32 ifc_int_set;
> +	u32 page_size;
> +
> +	/* QM_EQC_DW has 7 regs */
> +	u32 qm_eqc_dw[7];
> +
> +	/* QM_AEQC_DW has 7 regs */
> +	u32 qm_aeqc_dw[7];
> +
> +	/* QM reserved 5 regs */
> +	u32 qm_rsv_regs[5];
> +	u32 padding;
> +	/* qm memory init information */
> +	u64 eqe_dma;
> +	u64 aeqe_dma;
> +	u64 sqc_dma;
> +	u64 cqc_dma;
> +};
> +
> +struct hisi_acc_vf_migration_file {
> +	struct file *filp;
> +	struct mutex lock;
> +	bool disabled;
> +
> +	struct acc_vf_data vf_data;
> +	size_t total_length;
> +};
> +
> +struct hisi_acc_vf_core_device {
> +	struct vfio_pci_core_device core_device;
> +	/* for migration state */
> +	struct mutex state_mutex;
> +	enum vfio_device_mig_state mig_state;
> +	struct pci_dev *pf_dev;
> +	struct pci_dev *vf_dev;
> +	struct hisi_qm *pf_qm;
> +	struct hisi_qm vf_qm;
> +	u32 vf_qm_state;
> +	int vf_id;
> +
> +	struct hisi_acc_vf_migration_file *resuming_migf;
> +	struct hisi_acc_vf_migration_file *saving_migf; }; #endif /*
> +HISI_ACC_VFIO_PCI_H */
> --
> 2.25.1


^ permalink raw reply	[flat|nested] 56+ messages in thread

* Re: [PATCH v8 1/9] crypto: hisilicon/qm: Move the QM header to include/linux
  2022-03-03 23:01 ` [PATCH v8 1/9] crypto: hisilicon/qm: Move the QM header to include/linux Shameer Kolothum
@ 2022-03-04  9:03   ` Zhou Wang
  2022-03-04 11:33     ` Shameerali Kolothum Thodi
  2022-03-08  1:07   ` liulongfang
  2022-03-08 10:27   ` yekai(A)
  2 siblings, 1 reply; 56+ messages in thread
From: Zhou Wang @ 2022-03-04  9:03 UTC (permalink / raw)
  To: Shameer Kolothum, kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	linuxarm, liulongfang, prime.zeng, jonathan.cameron

> Since we are going to introduce VFIO PCI HiSilicon ACC driver for live
> migration in subsequent patches, move the ACC QM header file to a
> common include dir.
> 
> Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>

Hi Shameer,

It looks good to me for this movement.

Acked-by: Zhou Wang <wangzhou1@hisilicon.com>

Thanks,
Zhou

> ---
>  drivers/crypto/hisilicon/hpre/hpre.h                         | 2 +-
>  drivers/crypto/hisilicon/qm.c                                | 2 +-
>  drivers/crypto/hisilicon/sec2/sec.h                          | 2 +-
>  drivers/crypto/hisilicon/sgl.c                               | 2 +-
>  drivers/crypto/hisilicon/zip/zip.h                           | 2 +-
>  drivers/crypto/hisilicon/qm.h => include/linux/hisi_acc_qm.h | 0
>  6 files changed, 5 insertions(+), 5 deletions(-)
>  rename drivers/crypto/hisilicon/qm.h => include/linux/hisi_acc_qm.h (100%)
> 
> diff --git a/drivers/crypto/hisilicon/hpre/hpre.h b/drivers/crypto/hisilicon/hpre/hpre.h
> index e0b4a1982ee9..9a0558ed82f9 100644
> --- a/drivers/crypto/hisilicon/hpre/hpre.h
> +++ b/drivers/crypto/hisilicon/hpre/hpre.h
> @@ -4,7 +4,7 @@
>  #define __HISI_HPRE_H
>  
>  #include <linux/list.h>
> -#include "../qm.h"
> +#include <linux/hisi_acc_qm.h>
>  
>  #define HPRE_SQE_SIZE			sizeof(struct hpre_sqe)
>  #define HPRE_PF_DEF_Q_NUM		64
> diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c
> index c5b84a5ea350..ed23e1d3fa27 100644
> --- a/drivers/crypto/hisilicon/qm.c
> +++ b/drivers/crypto/hisilicon/qm.c
> @@ -15,7 +15,7 @@
>  #include <linux/uacce.h>
>  #include <linux/uaccess.h>
>  #include <uapi/misc/uacce/hisi_qm.h>
> -#include "qm.h"
> +#include <linux/hisi_acc_qm.h>
>  
>  /* eq/aeq irq enable */
>  #define QM_VF_AEQ_INT_SOURCE		0x0
> diff --git a/drivers/crypto/hisilicon/sec2/sec.h b/drivers/crypto/hisilicon/sec2/sec.h
> index d97cf02b1df7..c2e9b01187a7 100644
> --- a/drivers/crypto/hisilicon/sec2/sec.h
> +++ b/drivers/crypto/hisilicon/sec2/sec.h
> @@ -4,7 +4,7 @@
>  #ifndef __HISI_SEC_V2_H
>  #define __HISI_SEC_V2_H
>  
> -#include "../qm.h"
> +#include <linux/hisi_acc_qm.h>
>  #include "sec_crypto.h"
>  
>  /* Algorithm resource per hardware SEC queue */
> diff --git a/drivers/crypto/hisilicon/sgl.c b/drivers/crypto/hisilicon/sgl.c
> index 057273769f26..f7efc02b065f 100644
> --- a/drivers/crypto/hisilicon/sgl.c
> +++ b/drivers/crypto/hisilicon/sgl.c
> @@ -1,9 +1,9 @@
>  // SPDX-License-Identifier: GPL-2.0
>  /* Copyright (c) 2019 HiSilicon Limited. */
>  #include <linux/dma-mapping.h>
> +#include <linux/hisi_acc_qm.h>
>  #include <linux/module.h>
>  #include <linux/slab.h>
> -#include "qm.h"
>  
>  #define HISI_ACC_SGL_SGE_NR_MIN		1
>  #define HISI_ACC_SGL_NR_MAX		256
> diff --git a/drivers/crypto/hisilicon/zip/zip.h b/drivers/crypto/hisilicon/zip/zip.h
> index 517fdbdff3ea..3dfd3bac5a33 100644
> --- a/drivers/crypto/hisilicon/zip/zip.h
> +++ b/drivers/crypto/hisilicon/zip/zip.h
> @@ -7,7 +7,7 @@
>  #define pr_fmt(fmt)	"hisi_zip: " fmt
>  
>  #include <linux/list.h>
> -#include "../qm.h"
> +#include <linux/hisi_acc_qm.h>
>  
>  enum hisi_zip_error_type {
>  	/* negative compression */
> diff --git a/drivers/crypto/hisilicon/qm.h b/include/linux/hisi_acc_qm.h
> similarity index 100%
> rename from drivers/crypto/hisilicon/qm.h
> rename to include/linux/hisi_acc_qm.h
> 

^ permalink raw reply	[flat|nested] 56+ messages in thread

* Re: [PATCH v8 2/9] crypto: hisilicon/qm: Move few definitions to common header
  2022-03-03 23:01 ` [PATCH v8 2/9] crypto: hisilicon/qm: Move few definitions to common header Shameer Kolothum
@ 2022-03-04  9:06   ` Zhou Wang
  0 siblings, 0 replies; 56+ messages in thread
From: Zhou Wang @ 2022-03-04  9:06 UTC (permalink / raw)
  To: Shameer Kolothum, kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	linuxarm, liulongfang, prime.zeng, jonathan.cameron

> From: Longfang Liu <liulongfang@huawei.com>>
> Move Doorbell and Mailbox definitions to common header file.
> Also export QM mailbox functions.
> 
> This will be useful when we introduce VFIO PCI HiSilicon ACC live
> migration driver.
> 
> Signed-off-by: Longfang Liu <liulongfang@huawei.com>
> Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>

Looks good to me, so:

Acked-by: Zhou Wang <wangzhou1@hisilicon.com>

Thanks,
Zhou

> ---
>  drivers/crypto/hisilicon/qm.c | 58 +++++++++++------------------------
>  include/linux/hisi_acc_qm.h   | 38 +++++++++++++++++++++++
>  2 files changed, 56 insertions(+), 40 deletions(-)
> 
> diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c
> index ed23e1d3fa27..c88e013371af 100644
> --- a/drivers/crypto/hisilicon/qm.c
> +++ b/drivers/crypto/hisilicon/qm.c
> @@ -33,23 +33,6 @@
>  #define QM_ABNORMAL_EVENT_IRQ_VECTOR	3
>  
>  /* mailbox */
> -#define QM_MB_CMD_SQC			0x0
> -#define QM_MB_CMD_CQC			0x1
> -#define QM_MB_CMD_EQC			0x2
> -#define QM_MB_CMD_AEQC			0x3
> -#define QM_MB_CMD_SQC_BT		0x4
> -#define QM_MB_CMD_CQC_BT		0x5
> -#define QM_MB_CMD_SQC_VFT_V2		0x6
> -#define QM_MB_CMD_STOP_QP		0x8
> -#define QM_MB_CMD_SRC			0xc
> -#define QM_MB_CMD_DST			0xd
> -
> -#define QM_MB_CMD_SEND_BASE		0x300
> -#define QM_MB_EVENT_SHIFT		8
> -#define QM_MB_BUSY_SHIFT		13
> -#define QM_MB_OP_SHIFT			14
> -#define QM_MB_CMD_DATA_ADDR_L		0x304
> -#define QM_MB_CMD_DATA_ADDR_H		0x308
>  #define QM_MB_PING_ALL_VFS		0xffff
>  #define QM_MB_CMD_DATA_SHIFT		32
>  #define QM_MB_CMD_DATA_MASK		GENMASK(31, 0)
> @@ -103,19 +86,12 @@
>  #define QM_DB_CMD_SHIFT_V1		16
>  #define QM_DB_INDEX_SHIFT_V1		32
>  #define QM_DB_PRIORITY_SHIFT_V1		48
> -#define QM_DOORBELL_SQ_CQ_BASE_V2	0x1000
> -#define QM_DOORBELL_EQ_AEQ_BASE_V2	0x2000
>  #define QM_QUE_ISO_CFG_V		0x0030
>  #define QM_PAGE_SIZE			0x0034
>  #define QM_QUE_ISO_EN			0x100154
>  #define QM_CAPBILITY			0x100158
>  #define QM_QP_NUN_MASK			GENMASK(10, 0)
>  #define QM_QP_DB_INTERVAL		0x10000
> -#define QM_QP_MAX_NUM_SHIFT		11
> -#define QM_DB_CMD_SHIFT_V2		12
> -#define QM_DB_RAND_SHIFT_V2		16
> -#define QM_DB_INDEX_SHIFT_V2		32
> -#define QM_DB_PRIORITY_SHIFT_V2		48
>  
>  #define QM_MEM_START_INIT		0x100040
>  #define QM_MEM_INIT_DONE		0x100044
> @@ -693,7 +669,7 @@ static void qm_mb_pre_init(struct qm_mailbox *mailbox, u8 cmd,
>  }
>  
>  /* return 0 mailbox ready, -ETIMEDOUT hardware timeout */
> -static int qm_wait_mb_ready(struct hisi_qm *qm)
> +int hisi_qm_wait_mb_ready(struct hisi_qm *qm)
>  {
>  	u32 val;
>  
> @@ -701,6 +677,7 @@ static int qm_wait_mb_ready(struct hisi_qm *qm)
>  					  val, !((val >> QM_MB_BUSY_SHIFT) &
>  					  0x1), POLL_PERIOD, POLL_TIMEOUT);
>  }
> +EXPORT_SYMBOL_GPL(hisi_qm_wait_mb_ready);
>  
>  /* 128 bit should be written to hardware at one time to trigger a mailbox */
>  static void qm_mb_write(struct hisi_qm *qm, const void *src)
> @@ -726,14 +703,14 @@ static void qm_mb_write(struct hisi_qm *qm, const void *src)
>  
>  static int qm_mb_nolock(struct hisi_qm *qm, struct qm_mailbox *mailbox)
>  {
> -	if (unlikely(qm_wait_mb_ready(qm))) {
> +	if (unlikely(hisi_qm_wait_mb_ready(qm))) {
>  		dev_err(&qm->pdev->dev, "QM mailbox is busy to start!\n");
>  		goto mb_busy;
>  	}
>  
>  	qm_mb_write(qm, mailbox);
>  
> -	if (unlikely(qm_wait_mb_ready(qm))) {
> +	if (unlikely(hisi_qm_wait_mb_ready(qm))) {
>  		dev_err(&qm->pdev->dev, "QM mailbox operation timeout!\n");
>  		goto mb_busy;
>  	}
> @@ -745,8 +722,8 @@ static int qm_mb_nolock(struct hisi_qm *qm, struct qm_mailbox *mailbox)
>  	return -EBUSY;
>  }
>  
> -static int qm_mb(struct hisi_qm *qm, u8 cmd, dma_addr_t dma_addr, u16 queue,
> -		 bool op)
> +int hisi_qm_mb(struct hisi_qm *qm, u8 cmd, dma_addr_t dma_addr, u16 queue,
> +	       bool op)
>  {
>  	struct qm_mailbox mailbox;
>  	int ret;
> @@ -762,6 +739,7 @@ static int qm_mb(struct hisi_qm *qm, u8 cmd, dma_addr_t dma_addr, u16 queue,
>  
>  	return ret;
>  }
> +EXPORT_SYMBOL_GPL(hisi_qm_mb);
>  
>  static void qm_db_v1(struct hisi_qm *qm, u16 qn, u8 cmd, u16 index, u8 priority)
>  {
> @@ -1351,7 +1329,7 @@ static int qm_get_vft_v2(struct hisi_qm *qm, u32 *base, u32 *number)
>  	u64 sqc_vft;
>  	int ret;
>  
> -	ret = qm_mb(qm, QM_MB_CMD_SQC_VFT_V2, 0, 0, 1);
> +	ret = hisi_qm_mb(qm, QM_MB_CMD_SQC_VFT_V2, 0, 0, 1);
>  	if (ret)
>  		return ret;
>  
> @@ -1725,12 +1703,12 @@ static int dump_show(struct hisi_qm *qm, void *info,
>  
>  static int qm_dump_sqc_raw(struct hisi_qm *qm, dma_addr_t dma_addr, u16 qp_id)
>  {
> -	return qm_mb(qm, QM_MB_CMD_SQC, dma_addr, qp_id, 1);
> +	return hisi_qm_mb(qm, QM_MB_CMD_SQC, dma_addr, qp_id, 1);
>  }
>  
>  static int qm_dump_cqc_raw(struct hisi_qm *qm, dma_addr_t dma_addr, u16 qp_id)
>  {
> -	return qm_mb(qm, QM_MB_CMD_CQC, dma_addr, qp_id, 1);
> +	return hisi_qm_mb(qm, QM_MB_CMD_CQC, dma_addr, qp_id, 1);
>  }
>  
>  static int qm_sqc_dump(struct hisi_qm *qm, const char *s)
> @@ -1842,7 +1820,7 @@ static int qm_eqc_aeqc_dump(struct hisi_qm *qm, char *s, size_t size,
>  	if (IS_ERR(xeqc))
>  		return PTR_ERR(xeqc);
>  
> -	ret = qm_mb(qm, cmd, xeqc_dma, 0, 1);
> +	ret = hisi_qm_mb(qm, cmd, xeqc_dma, 0, 1);
>  	if (ret)
>  		goto err_free_ctx;
>  
> @@ -2495,7 +2473,7 @@ static int qm_ping_pf(struct hisi_qm *qm, u64 cmd)
>  
>  static int qm_stop_qp(struct hisi_qp *qp)
>  {
> -	return qm_mb(qp->qm, QM_MB_CMD_STOP_QP, 0, qp->qp_id, 0);
> +	return hisi_qm_mb(qp->qm, QM_MB_CMD_STOP_QP, 0, qp->qp_id, 0);
>  }
>  
>  static int qm_set_msi(struct hisi_qm *qm, bool set)
> @@ -2763,7 +2741,7 @@ static int qm_sq_ctx_cfg(struct hisi_qp *qp, int qp_id, u32 pasid)
>  		return -ENOMEM;
>  	}
>  
> -	ret = qm_mb(qm, QM_MB_CMD_SQC, sqc_dma, qp_id, 0);
> +	ret = hisi_qm_mb(qm, QM_MB_CMD_SQC, sqc_dma, qp_id, 0);
>  	dma_unmap_single(dev, sqc_dma, sizeof(struct qm_sqc), DMA_TO_DEVICE);
>  	kfree(sqc);
>  
> @@ -2804,7 +2782,7 @@ static int qm_cq_ctx_cfg(struct hisi_qp *qp, int qp_id, u32 pasid)
>  		return -ENOMEM;
>  	}
>  
> -	ret = qm_mb(qm, QM_MB_CMD_CQC, cqc_dma, qp_id, 0);
> +	ret = hisi_qm_mb(qm, QM_MB_CMD_CQC, cqc_dma, qp_id, 0);
>  	dma_unmap_single(dev, cqc_dma, sizeof(struct qm_cqc), DMA_TO_DEVICE);
>  	kfree(cqc);
>  
> @@ -3655,7 +3633,7 @@ static int qm_eq_ctx_cfg(struct hisi_qm *qm)
>  		return -ENOMEM;
>  	}
>  
> -	ret = qm_mb(qm, QM_MB_CMD_EQC, eqc_dma, 0, 0);
> +	ret = hisi_qm_mb(qm, QM_MB_CMD_EQC, eqc_dma, 0, 0);
>  	dma_unmap_single(dev, eqc_dma, sizeof(struct qm_eqc), DMA_TO_DEVICE);
>  	kfree(eqc);
>  
> @@ -3684,7 +3662,7 @@ static int qm_aeq_ctx_cfg(struct hisi_qm *qm)
>  		return -ENOMEM;
>  	}
>  
> -	ret = qm_mb(qm, QM_MB_CMD_AEQC, aeqc_dma, 0, 0);
> +	ret = hisi_qm_mb(qm, QM_MB_CMD_AEQC, aeqc_dma, 0, 0);
>  	dma_unmap_single(dev, aeqc_dma, sizeof(struct qm_aeqc), DMA_TO_DEVICE);
>  	kfree(aeqc);
>  
> @@ -3723,11 +3701,11 @@ static int __hisi_qm_start(struct hisi_qm *qm)
>  	if (ret)
>  		return ret;
>  
> -	ret = qm_mb(qm, QM_MB_CMD_SQC_BT, qm->sqc_dma, 0, 0);
> +	ret = hisi_qm_mb(qm, QM_MB_CMD_SQC_BT, qm->sqc_dma, 0, 0);
>  	if (ret)
>  		return ret;
>  
> -	ret = qm_mb(qm, QM_MB_CMD_CQC_BT, qm->cqc_dma, 0, 0);
> +	ret = hisi_qm_mb(qm, QM_MB_CMD_CQC_BT, qm->cqc_dma, 0, 0);
>  	if (ret)
>  		return ret;
>  
> diff --git a/include/linux/hisi_acc_qm.h b/include/linux/hisi_acc_qm.h
> index 3068093229a5..6a6477c34666 100644
> --- a/include/linux/hisi_acc_qm.h
> +++ b/include/linux/hisi_acc_qm.h
> @@ -34,6 +34,40 @@
>  #define QM_WUSER_M_CFG_ENABLE		0x1000a8
>  #define WUSER_M_CFG_ENABLE		0xffffffff
>  
> +/* mailbox */
> +#define QM_MB_CMD_SQC                   0x0
> +#define QM_MB_CMD_CQC                   0x1
> +#define QM_MB_CMD_EQC                   0x2
> +#define QM_MB_CMD_AEQC                  0x3
> +#define QM_MB_CMD_SQC_BT                0x4
> +#define QM_MB_CMD_CQC_BT                0x5
> +#define QM_MB_CMD_SQC_VFT_V2            0x6
> +#define QM_MB_CMD_STOP_QP               0x8
> +#define QM_MB_CMD_SRC                   0xc
> +#define QM_MB_CMD_DST                   0xd
> +
> +#define QM_MB_CMD_SEND_BASE		0x300
> +#define QM_MB_EVENT_SHIFT               8
> +#define QM_MB_BUSY_SHIFT		13
> +#define QM_MB_OP_SHIFT			14
> +#define QM_MB_CMD_DATA_ADDR_L		0x304
> +#define QM_MB_CMD_DATA_ADDR_H		0x308
> +#define QM_MB_MAX_WAIT_CNT		6000
> +
> +/* doorbell */
> +#define QM_DOORBELL_CMD_SQ              0
> +#define QM_DOORBELL_CMD_CQ              1
> +#define QM_DOORBELL_CMD_EQ              2
> +#define QM_DOORBELL_CMD_AEQ             3
> +
> +#define QM_DOORBELL_SQ_CQ_BASE_V2	0x1000
> +#define QM_DOORBELL_EQ_AEQ_BASE_V2	0x2000
> +#define QM_QP_MAX_NUM_SHIFT             11
> +#define QM_DB_CMD_SHIFT_V2		12
> +#define QM_DB_RAND_SHIFT_V2		16
> +#define QM_DB_INDEX_SHIFT_V2		32
> +#define QM_DB_PRIORITY_SHIFT_V2		48
> +
>  /* qm cache */
>  #define QM_CACHE_CTL			0x100050
>  #define SQC_CACHE_ENABLE		BIT(0)
> @@ -414,6 +448,10 @@ pci_ers_result_t hisi_qm_dev_slot_reset(struct pci_dev *pdev);
>  void hisi_qm_reset_prepare(struct pci_dev *pdev);
>  void hisi_qm_reset_done(struct pci_dev *pdev);
>  
> +int hisi_qm_wait_mb_ready(struct hisi_qm *qm);
> +int hisi_qm_mb(struct hisi_qm *qm, u8 cmd, dma_addr_t dma_addr, u16 queue,
> +	       bool op);
> +
>  struct hisi_acc_sgl_pool;
>  struct hisi_acc_hw_sgl *hisi_acc_sg_buf_map_to_hw_sgl(struct device *dev,
>  	struct scatterlist *sgl, struct hisi_acc_sgl_pool *pool,
> 

^ permalink raw reply	[flat|nested] 56+ messages in thread

* Re: [PATCH v8 3/9] hisi_acc_qm: Move VF PCI device IDs to common header
  2022-03-03 23:01 ` [PATCH v8 3/9] hisi_acc_qm: Move VF PCI device IDs " Shameer Kolothum
@ 2022-03-04  9:34   ` Zhou Wang
  2022-03-04 11:35     ` Shameerali Kolothum Thodi
  2022-03-07 17:53   ` Alex Williamson
                     ` (2 subsequent siblings)
  3 siblings, 1 reply; 56+ messages in thread
From: Zhou Wang @ 2022-03-04  9:34 UTC (permalink / raw)
  To: Shameer Kolothum, kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	linuxarm, liulongfang, prime.zeng, jonathan.cameron

> Move the PCI Device IDs of HiSilicon ACC VF devices to a common heade> and also use a uniform naming convention.
> 
> This will be useful when we introduce the vfio PCI HiSilicon ACC live
> migration driver in subsequent patches.
> 
> Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>

Acked-by: Zhou Wang <wangzhou1@hisilicon.com>

Best,
Zhou

> ---
>  drivers/crypto/hisilicon/hpre/hpre_main.c | 13 ++++++-------
>  drivers/crypto/hisilicon/sec2/sec_main.c  | 15 +++++++--------
>  drivers/crypto/hisilicon/zip/zip_main.c   | 11 +++++------
>  include/linux/pci_ids.h                   |  3 +++
>  4 files changed, 21 insertions(+), 21 deletions(-)
> 
> diff --git a/drivers/crypto/hisilicon/hpre/hpre_main.c b/drivers/crypto/hisilicon/hpre/hpre_main.c
> index ebfab3e14499..3589d8879b5e 100644
> --- a/drivers/crypto/hisilicon/hpre/hpre_main.c
> +++ b/drivers/crypto/hisilicon/hpre/hpre_main.c
> @@ -68,8 +68,7 @@
>  #define HPRE_REG_RD_INTVRL_US		10
>  #define HPRE_REG_RD_TMOUT_US		1000
>  #define HPRE_DBGFS_VAL_MAX_LEN		20
> -#define HPRE_PCI_DEVICE_ID		0xa258
> -#define HPRE_PCI_VF_DEVICE_ID		0xa259
> +#define PCI_DEVICE_ID_HUAWEI_HPRE_PF	0xa258
>  #define HPRE_QM_USR_CFG_MASK		GENMASK(31, 1)
>  #define HPRE_QM_AXI_CFG_MASK		GENMASK(15, 0)
>  #define HPRE_QM_VFG_AX_MASK		GENMASK(7, 0)
> @@ -111,8 +110,8 @@
>  static const char hpre_name[] = "hisi_hpre";
>  static struct dentry *hpre_debugfs_root;
>  static const struct pci_device_id hpre_dev_ids[] = {
> -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, HPRE_PCI_DEVICE_ID) },
> -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, HPRE_PCI_VF_DEVICE_ID) },
> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_HPRE_PF) },
> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_HPRE_VF) },
>  	{ 0, }
>  };
>  
> @@ -242,7 +241,7 @@ MODULE_PARM_DESC(uacce_mode, UACCE_MODE_DESC);
>  
>  static int pf_q_num_set(const char *val, const struct kernel_param *kp)
>  {
> -	return q_num_set(val, kp, HPRE_PCI_DEVICE_ID);
> +	return q_num_set(val, kp, PCI_DEVICE_ID_HUAWEI_HPRE_PF);
>  }
>  
>  static const struct kernel_param_ops hpre_pf_q_num_ops = {
> @@ -921,7 +920,7 @@ static int hpre_debugfs_init(struct hisi_qm *qm)
>  	qm->debug.sqe_mask_len = HPRE_SQE_MASK_LEN;
>  	hisi_qm_debug_init(qm);
>  
> -	if (qm->pdev->device == HPRE_PCI_DEVICE_ID) {
> +	if (qm->pdev->device == PCI_DEVICE_ID_HUAWEI_HPRE_PF) {
>  		ret = hpre_ctrl_debug_init(qm);
>  		if (ret)
>  			goto failed_to_create;
> @@ -958,7 +957,7 @@ static int hpre_qm_init(struct hisi_qm *qm, struct pci_dev *pdev)
>  	qm->sqe_size = HPRE_SQE_SIZE;
>  	qm->dev_name = hpre_name;
>  
> -	qm->fun_type = (pdev->device == HPRE_PCI_DEVICE_ID) ?
> +	qm->fun_type = (pdev->device == PCI_DEVICE_ID_HUAWEI_HPRE_PF) ?
>  			QM_HW_PF : QM_HW_VF;
>  	if (qm->fun_type == QM_HW_PF) {
>  		qm->qp_base = HPRE_PF_DEF_Q_BASE;
> diff --git a/drivers/crypto/hisilicon/sec2/sec_main.c b/drivers/crypto/hisilicon/sec2/sec_main.c
> index 26d3ab1d308b..311a8747b5bf 100644
> --- a/drivers/crypto/hisilicon/sec2/sec_main.c
> +++ b/drivers/crypto/hisilicon/sec2/sec_main.c
> @@ -20,8 +20,7 @@
>  
>  #define SEC_VF_NUM			63
>  #define SEC_QUEUE_NUM_V1		4096
> -#define SEC_PF_PCI_DEVICE_ID		0xa255
> -#define SEC_VF_PCI_DEVICE_ID		0xa256
> +#define PCI_DEVICE_ID_HUAWEI_SEC_PF	0xa255
>  
>  #define SEC_BD_ERR_CHK_EN0		0xEFFFFFFF
>  #define SEC_BD_ERR_CHK_EN1		0x7ffff7fd
> @@ -225,7 +224,7 @@ static const struct debugfs_reg32 sec_dfx_regs[] = {
>  
>  static int sec_pf_q_num_set(const char *val, const struct kernel_param *kp)
>  {
> -	return q_num_set(val, kp, SEC_PF_PCI_DEVICE_ID);
> +	return q_num_set(val, kp, PCI_DEVICE_ID_HUAWEI_SEC_PF);
>  }
>  
>  static const struct kernel_param_ops sec_pf_q_num_ops = {
> @@ -313,8 +312,8 @@ module_param_cb(uacce_mode, &sec_uacce_mode_ops, &uacce_mode, 0444);
>  MODULE_PARM_DESC(uacce_mode, UACCE_MODE_DESC);
>  
>  static const struct pci_device_id sec_dev_ids[] = {
> -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, SEC_PF_PCI_DEVICE_ID) },
> -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, SEC_VF_PCI_DEVICE_ID) },
> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_SEC_PF) },
> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_SEC_VF) },
>  	{ 0, }
>  };
>  MODULE_DEVICE_TABLE(pci, sec_dev_ids);
> @@ -717,7 +716,7 @@ static int sec_core_debug_init(struct hisi_qm *qm)
>  	regset->base = qm->io_base;
>  	regset->dev = dev;
>  
> -	if (qm->pdev->device == SEC_PF_PCI_DEVICE_ID)
> +	if (qm->pdev->device == PCI_DEVICE_ID_HUAWEI_SEC_PF)
>  		debugfs_create_file("regs", 0444, tmp_d, regset, &sec_regs_fops);
>  
>  	for (i = 0; i < ARRAY_SIZE(sec_dfx_labels); i++) {
> @@ -735,7 +734,7 @@ static int sec_debug_init(struct hisi_qm *qm)
>  	struct sec_dev *sec = container_of(qm, struct sec_dev, qm);
>  	int i;
>  
> -	if (qm->pdev->device == SEC_PF_PCI_DEVICE_ID) {
> +	if (qm->pdev->device == PCI_DEVICE_ID_HUAWEI_SEC_PF) {
>  		for (i = SEC_CLEAR_ENABLE; i < SEC_DEBUG_FILE_NUM; i++) {
>  			spin_lock_init(&sec->debug.files[i].lock);
>  			sec->debug.files[i].index = i;
> @@ -877,7 +876,7 @@ static int sec_qm_init(struct hisi_qm *qm, struct pci_dev *pdev)
>  	qm->sqe_size = SEC_SQE_SIZE;
>  	qm->dev_name = sec_name;
>  
> -	qm->fun_type = (pdev->device == SEC_PF_PCI_DEVICE_ID) ?
> +	qm->fun_type = (pdev->device == PCI_DEVICE_ID_HUAWEI_SEC_PF) ?
>  			QM_HW_PF : QM_HW_VF;
>  	if (qm->fun_type == QM_HW_PF) {
>  		qm->qp_base = SEC_PF_DEF_Q_BASE;
> diff --git a/drivers/crypto/hisilicon/zip/zip_main.c b/drivers/crypto/hisilicon/zip/zip_main.c
> index 678f8b58ec42..66decfe07282 100644
> --- a/drivers/crypto/hisilicon/zip/zip_main.c
> +++ b/drivers/crypto/hisilicon/zip/zip_main.c
> @@ -15,8 +15,7 @@
>  #include <linux/uacce.h>
>  #include "zip.h"
>  
> -#define PCI_DEVICE_ID_ZIP_PF		0xa250
> -#define PCI_DEVICE_ID_ZIP_VF		0xa251
> +#define PCI_DEVICE_ID_HUAWEI_ZIP_PF	0xa250
>  
>  #define HZIP_QUEUE_NUM_V1		4096
>  
> @@ -246,7 +245,7 @@ MODULE_PARM_DESC(uacce_mode, UACCE_MODE_DESC);
>  
>  static int pf_q_num_set(const char *val, const struct kernel_param *kp)
>  {
> -	return q_num_set(val, kp, PCI_DEVICE_ID_ZIP_PF);
> +	return q_num_set(val, kp, PCI_DEVICE_ID_HUAWEI_ZIP_PF);
>  }
>  
>  static const struct kernel_param_ops pf_q_num_ops = {
> @@ -268,8 +267,8 @@ module_param_cb(vfs_num, &vfs_num_ops, &vfs_num, 0444);
>  MODULE_PARM_DESC(vfs_num, "Number of VFs to enable(1-63), 0(default)");
>  
>  static const struct pci_device_id hisi_zip_dev_ids[] = {
> -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_ZIP_PF) },
> -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_ZIP_VF) },
> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_ZIP_PF) },
> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_ZIP_VF) },
>  	{ 0, }
>  };
>  MODULE_DEVICE_TABLE(pci, hisi_zip_dev_ids);
> @@ -838,7 +837,7 @@ static int hisi_zip_qm_init(struct hisi_qm *qm, struct pci_dev *pdev)
>  	qm->sqe_size = HZIP_SQE_SIZE;
>  	qm->dev_name = hisi_zip_name;
>  
> -	qm->fun_type = (pdev->device == PCI_DEVICE_ID_ZIP_PF) ?
> +	qm->fun_type = (pdev->device == PCI_DEVICE_ID_HUAWEI_ZIP_PF) ?
>  			QM_HW_PF : QM_HW_VF;
>  	if (qm->fun_type == QM_HW_PF) {
>  		qm->qp_base = HZIP_PF_DEF_Q_BASE;
> diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
> index aad54c666407..31dee2b65a62 100644
> --- a/include/linux/pci_ids.h
> +++ b/include/linux/pci_ids.h
> @@ -2529,6 +2529,9 @@
>  #define PCI_DEVICE_ID_KORENIX_JETCARDF3	0x17ff
>  
>  #define PCI_VENDOR_ID_HUAWEI		0x19e5
> +#define PCI_DEVICE_ID_HUAWEI_ZIP_VF	0xa251
> +#define PCI_DEVICE_ID_HUAWEI_SEC_VF	0xa256
> +#define PCI_DEVICE_ID_HUAWEI_HPRE_VF	0xa259
>  
>  #define PCI_VENDOR_ID_NETRONOME		0x19ee
>  #define PCI_DEVICE_ID_NETRONOME_NFP4000	0x4000
> 

^ permalink raw reply	[flat|nested] 56+ messages in thread

* Re: [PATCH v8 6/9] hisi_acc_vfio_pci: Add helper to retrieve the struct pci_driver
  2022-03-03 23:01 ` [PATCH v8 6/9] hisi_acc_vfio_pci: Add helper to retrieve the struct pci_driver Shameer Kolothum
@ 2022-03-04  9:40   ` Zhou Wang
  2022-03-04 11:31     ` Shameerali Kolothum Thodi
  2022-03-08 10:28   ` yekai(A)
  2022-03-08 12:02   ` liulongfang
  2 siblings, 1 reply; 56+ messages in thread
From: Zhou Wang @ 2022-03-04  9:40 UTC (permalink / raw)
  To: Shameer Kolothum, kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	linuxarm, liulongfang, prime.zeng, jonathan.cameron

> struct pci_driver pointer is an input into the pci_iov_get_pf_drvdata().> Introduce helpers to retrieve the ACC PF dev struct pci_driver pointers
> as we use this in ACC vfio migration driver.
> 
> Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>

Acked-by: Zhou Wang <wangzhou1@hisilicon.com>

Best,
Zhou

> ---
>  drivers/crypto/hisilicon/hpre/hpre_main.c | 6 ++++++
>  drivers/crypto/hisilicon/sec2/sec_main.c  | 6 ++++++
>  drivers/crypto/hisilicon/zip/zip_main.c   | 6 ++++++
>  include/linux/hisi_acc_qm.h               | 5 +++++
>  4 files changed, 23 insertions(+)
> 
> diff --git a/drivers/crypto/hisilicon/hpre/hpre_main.c b/drivers/crypto/hisilicon/hpre/hpre_main.c
> index 3589d8879b5e..36ab30e9e654 100644
> --- a/drivers/crypto/hisilicon/hpre/hpre_main.c
> +++ b/drivers/crypto/hisilicon/hpre/hpre_main.c
> @@ -1190,6 +1190,12 @@ static struct pci_driver hpre_pci_driver = {
>  	.driver.pm		= &hpre_pm_ops,
>  };
>  
> +struct pci_driver *hisi_hpre_get_pf_driver(void)
> +{
> +	return &hpre_pci_driver;
> +}
> +EXPORT_SYMBOL_GPL(hisi_hpre_get_pf_driver);
> +
>  static void hpre_register_debugfs(void)
>  {
>  	if (!debugfs_initialized())
> diff --git a/drivers/crypto/hisilicon/sec2/sec_main.c b/drivers/crypto/hisilicon/sec2/sec_main.c
> index 311a8747b5bf..421a405ca337 100644
> --- a/drivers/crypto/hisilicon/sec2/sec_main.c
> +++ b/drivers/crypto/hisilicon/sec2/sec_main.c
> @@ -1088,6 +1088,12 @@ static struct pci_driver sec_pci_driver = {
>  	.driver.pm = &sec_pm_ops,
>  };
>  
> +struct pci_driver *hisi_sec_get_pf_driver(void)
> +{
> +	return &sec_pci_driver;
> +}
> +EXPORT_SYMBOL_GPL(hisi_sec_get_pf_driver);
> +
>  static void sec_register_debugfs(void)
>  {
>  	if (!debugfs_initialized())
> diff --git a/drivers/crypto/hisilicon/zip/zip_main.c b/drivers/crypto/hisilicon/zip/zip_main.c
> index 66decfe07282..4534e1e107d1 100644
> --- a/drivers/crypto/hisilicon/zip/zip_main.c
> +++ b/drivers/crypto/hisilicon/zip/zip_main.c
> @@ -1012,6 +1012,12 @@ static struct pci_driver hisi_zip_pci_driver = {
>  	.driver.pm		= &hisi_zip_pm_ops,
>  };
>  
> +struct pci_driver *hisi_zip_get_pf_driver(void)
> +{
> +	return &hisi_zip_pci_driver;
> +}
> +EXPORT_SYMBOL_GPL(hisi_zip_get_pf_driver);
> +
>  static void hisi_zip_register_debugfs(void)
>  {
>  	if (!debugfs_initialized())
> diff --git a/include/linux/hisi_acc_qm.h b/include/linux/hisi_acc_qm.h
> index 6a6477c34666..00f2a4db8723 100644
> --- a/include/linux/hisi_acc_qm.h
> +++ b/include/linux/hisi_acc_qm.h
> @@ -476,4 +476,9 @@ void hisi_qm_pm_init(struct hisi_qm *qm);
>  int hisi_qm_get_dfx_access(struct hisi_qm *qm);
>  void hisi_qm_put_dfx_access(struct hisi_qm *qm);
>  void hisi_qm_regs_dump(struct seq_file *s, struct debugfs_regset32 *regset);
> +
> +/* Used by VFIO ACC live migration driver */
> +struct pci_driver *hisi_sec_get_pf_driver(void);
> +struct pci_driver *hisi_hpre_get_pf_driver(void);
> +struct pci_driver *hisi_zip_get_pf_driver(void);
>  #endif
> 

^ permalink raw reply	[flat|nested] 56+ messages in thread

* Re: [PATCH v8 7/9] crypto: hisilicon/qm: Set the VF QM state register
  2022-03-03 23:01 ` [PATCH v8 7/9] crypto: hisilicon/qm: Set the VF QM state register Shameer Kolothum
@ 2022-03-04  9:43   ` Zhou Wang
  2022-03-08  6:31   ` Tian, Kevin
  1 sibling, 0 replies; 56+ messages in thread
From: Zhou Wang @ 2022-03-04  9:43 UTC (permalink / raw)
  To: Shameer Kolothum, kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	linuxarm, liulongfang, prime.zeng, jonathan.cameron

> From: Longfang Liu <liulongfang@huawei.com>>
> We use VF QM state register to record the status of the QM configuration
> state. This will be used in the ACC migration driver to determine whether
> we can safely save and restore the QM data.
> 
> Signed-off-by: Longfang Liu <liulongfang@huawei.com>
> Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>

Acked-by: Zhou Wang <wangzhou1@hisilicon.com>

Best,
Zhou

> ---
>  drivers/crypto/hisilicon/qm.c | 8 ++++++++
>  include/linux/hisi_acc_qm.h   | 6 ++++++
>  2 files changed, 14 insertions(+)
> 
> diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c
> index c88e013371af..6a8776db38b5 100644
> --- a/drivers/crypto/hisilicon/qm.c
> +++ b/drivers/crypto/hisilicon/qm.c
> @@ -3492,6 +3492,12 @@ static void hisi_qm_pci_uninit(struct hisi_qm *qm)
>  	pci_disable_device(pdev);
>  }
>  
> +static void hisi_qm_set_state(struct hisi_qm *qm, u8 state)
> +{
> +	if (qm->ver > QM_HW_V2 && qm->fun_type == QM_HW_VF)
> +		writel(state, qm->io_base + QM_VF_STATE);
> +}
> +
>  /**
>   * hisi_qm_uninit() - Uninitialize qm.
>   * @qm: The qm needed uninit.
> @@ -3520,6 +3526,7 @@ void hisi_qm_uninit(struct hisi_qm *qm)
>  		dma_free_coherent(dev, qm->qdma.size,
>  				  qm->qdma.va, qm->qdma.dma);
>  	}
> +	hisi_qm_set_state(qm, QM_NOT_READY);
>  	up_write(&qm->qps_lock);
>  
>  	qm_irq_unregister(qm);
> @@ -3745,6 +3752,7 @@ int hisi_qm_start(struct hisi_qm *qm)
>  	if (!ret)
>  		atomic_set(&qm->status.flags, QM_START);
>  
> +	hisi_qm_set_state(qm, QM_READY);
>  err_unlock:
>  	up_write(&qm->qps_lock);
>  	return ret;
> diff --git a/include/linux/hisi_acc_qm.h b/include/linux/hisi_acc_qm.h
> index 00f2a4db8723..177f7b7cd414 100644
> --- a/include/linux/hisi_acc_qm.h
> +++ b/include/linux/hisi_acc_qm.h
> @@ -67,6 +67,7 @@
>  #define QM_DB_RAND_SHIFT_V2		16
>  #define QM_DB_INDEX_SHIFT_V2		32
>  #define QM_DB_PRIORITY_SHIFT_V2		48
> +#define QM_VF_STATE			0x60
>  
>  /* qm cache */
>  #define QM_CACHE_CTL			0x100050
> @@ -162,6 +163,11 @@ enum qm_debug_file {
>  	DEBUG_FILE_NUM,
>  };
>  
> +enum qm_vf_state {
> +	QM_READY = 0,
> +	QM_NOT_READY,
> +};
> +
>  struct qm_dfx {
>  	atomic64_t err_irq_cnt;
>  	atomic64_t aeq_irq_cnt;
> 

^ permalink raw reply	[flat|nested] 56+ messages in thread

* RE: [PATCH v8 6/9] hisi_acc_vfio_pci: Add helper to retrieve the struct pci_driver
  2022-03-04  9:40   ` Zhou Wang
@ 2022-03-04 11:31     ` Shameerali Kolothum Thodi
  0 siblings, 0 replies; 56+ messages in thread
From: Shameerali Kolothum Thodi @ 2022-03-04 11:31 UTC (permalink / raw)
  To: Wangzhou (B), kvm, linux-kernel, linux-crypto, Xu Zaibo
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	Linuxarm, liulongfang, Zengtao (B),
	Jonathan Cameron


> -----Original Message-----
> From: Wangzhou (B)
> Sent: 04 March 2022 09:41
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>;
> kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-crypto@vger.kernel.org
> Cc: linux-pci@vger.kernel.org; alex.williamson@redhat.com; jgg@nvidia.com;
> cohuck@redhat.com; mgurtovoy@nvidia.com; yishaih@nvidia.com; Linuxarm
> <linuxarm@huawei.com>; liulongfang <liulongfang@huawei.com>; Zengtao (B)
> <prime.zeng@hisilicon.com>; Jonathan Cameron
> <jonathan.cameron@huawei.com>
> Subject: Re: [PATCH v8 6/9] hisi_acc_vfio_pci: Add helper to retrieve the struct
> pci_driver
> 
> > struct pci_driver pointer is an input into the pci_iov_get_pf_drvdata().>
> Introduce helpers to retrieve the ACC PF dev struct pci_driver pointers
> > as we use this in ACC vfio migration driver.
> >
> > Signed-off-by: Shameer Kolothum
> <shameerali.kolothum.thodi@huawei.com>
> 
> Acked-by: Zhou Wang <wangzhou1@hisilicon.com>

Thanks.

[+cc Zaibo] for hpre/sec part

> 
> Best,
> Zhou
> 
> > ---
> >  drivers/crypto/hisilicon/hpre/hpre_main.c | 6 ++++++
> >  drivers/crypto/hisilicon/sec2/sec_main.c  | 6 ++++++
> >  drivers/crypto/hisilicon/zip/zip_main.c   | 6 ++++++
> >  include/linux/hisi_acc_qm.h               | 5 +++++
> >  4 files changed, 23 insertions(+)
> >
> > diff --git a/drivers/crypto/hisilicon/hpre/hpre_main.c
> b/drivers/crypto/hisilicon/hpre/hpre_main.c
> > index 3589d8879b5e..36ab30e9e654 100644
> > --- a/drivers/crypto/hisilicon/hpre/hpre_main.c
> > +++ b/drivers/crypto/hisilicon/hpre/hpre_main.c
> > @@ -1190,6 +1190,12 @@ static struct pci_driver hpre_pci_driver = {
> >  	.driver.pm		= &hpre_pm_ops,
> >  };
> >
> > +struct pci_driver *hisi_hpre_get_pf_driver(void)
> > +{
> > +	return &hpre_pci_driver;
> > +}
> > +EXPORT_SYMBOL_GPL(hisi_hpre_get_pf_driver);
> > +
> >  static void hpre_register_debugfs(void)
> >  {
> >  	if (!debugfs_initialized())
> > diff --git a/drivers/crypto/hisilicon/sec2/sec_main.c
> b/drivers/crypto/hisilicon/sec2/sec_main.c
> > index 311a8747b5bf..421a405ca337 100644
> > --- a/drivers/crypto/hisilicon/sec2/sec_main.c
> > +++ b/drivers/crypto/hisilicon/sec2/sec_main.c
> > @@ -1088,6 +1088,12 @@ static struct pci_driver sec_pci_driver = {
> >  	.driver.pm = &sec_pm_ops,
> >  };
> >
> > +struct pci_driver *hisi_sec_get_pf_driver(void)
> > +{
> > +	return &sec_pci_driver;
> > +}
> > +EXPORT_SYMBOL_GPL(hisi_sec_get_pf_driver);
> > +
> >  static void sec_register_debugfs(void)
> >  {
> >  	if (!debugfs_initialized())
> > diff --git a/drivers/crypto/hisilicon/zip/zip_main.c
> b/drivers/crypto/hisilicon/zip/zip_main.c
> > index 66decfe07282..4534e1e107d1 100644
> > --- a/drivers/crypto/hisilicon/zip/zip_main.c
> > +++ b/drivers/crypto/hisilicon/zip/zip_main.c
> > @@ -1012,6 +1012,12 @@ static struct pci_driver hisi_zip_pci_driver = {
> >  	.driver.pm		= &hisi_zip_pm_ops,
> >  };
> >
> > +struct pci_driver *hisi_zip_get_pf_driver(void)
> > +{
> > +	return &hisi_zip_pci_driver;
> > +}
> > +EXPORT_SYMBOL_GPL(hisi_zip_get_pf_driver);
> > +
> >  static void hisi_zip_register_debugfs(void)
> >  {
> >  	if (!debugfs_initialized())
> > diff --git a/include/linux/hisi_acc_qm.h b/include/linux/hisi_acc_qm.h
> > index 6a6477c34666..00f2a4db8723 100644
> > --- a/include/linux/hisi_acc_qm.h
> > +++ b/include/linux/hisi_acc_qm.h
> > @@ -476,4 +476,9 @@ void hisi_qm_pm_init(struct hisi_qm *qm);
> >  int hisi_qm_get_dfx_access(struct hisi_qm *qm);
> >  void hisi_qm_put_dfx_access(struct hisi_qm *qm);
> >  void hisi_qm_regs_dump(struct seq_file *s, struct debugfs_regset32
> *regset);
> > +
> > +/* Used by VFIO ACC live migration driver */
> > +struct pci_driver *hisi_sec_get_pf_driver(void);
> > +struct pci_driver *hisi_hpre_get_pf_driver(void);
> > +struct pci_driver *hisi_zip_get_pf_driver(void);
> >  #endif
> >

^ permalink raw reply	[flat|nested] 56+ messages in thread

* RE: [PATCH v8 1/9] crypto: hisilicon/qm: Move the QM header to include/linux
  2022-03-04  9:03   ` Zhou Wang
@ 2022-03-04 11:33     ` Shameerali Kolothum Thodi
  0 siblings, 0 replies; 56+ messages in thread
From: Shameerali Kolothum Thodi @ 2022-03-04 11:33 UTC (permalink / raw)
  To: Wangzhou (B), kvm, linux-kernel, linux-crypto, Xu Zaibo
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	Linuxarm, liulongfang, Zengtao (B),
	Jonathan Cameron

[+ Zaibo]

> -----Original Message-----
> From: Wangzhou (B)
> Sent: 04 March 2022 09:03
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>;
> kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-crypto@vger.kernel.org
> Cc: linux-pci@vger.kernel.org; alex.williamson@redhat.com; jgg@nvidia.com;
> cohuck@redhat.com; mgurtovoy@nvidia.com; yishaih@nvidia.com; Linuxarm
> <linuxarm@huawei.com>; liulongfang <liulongfang@huawei.com>; Zengtao (B)
> <prime.zeng@hisilicon.com>; Jonathan Cameron
> <jonathan.cameron@huawei.com>
> Subject: Re: [PATCH v8 1/9] crypto: hisilicon/qm: Move the QM header to
> include/linux
> 
> > Since we are going to introduce VFIO PCI HiSilicon ACC driver for live
> > migration in subsequent patches, move the ACC QM header file to a
> > common include dir.
> >
> > Signed-off-by: Shameer Kolothum
> <shameerali.kolothum.thodi@huawei.com>
> 
> Hi Shameer,
> 
> It looks good to me for this movement.
> 
> Acked-by: Zhou Wang <wangzhou1@hisilicon.com>

Thanks. [+cc Zaibo] for hpre/sec part.


> 
> > ---
> >  drivers/crypto/hisilicon/hpre/hpre.h                         | 2 +-
> >  drivers/crypto/hisilicon/qm.c                                | 2 +-
> >  drivers/crypto/hisilicon/sec2/sec.h                          | 2 +-
> >  drivers/crypto/hisilicon/sgl.c                               | 2 +-
> >  drivers/crypto/hisilicon/zip/zip.h                           | 2 +-
> >  drivers/crypto/hisilicon/qm.h => include/linux/hisi_acc_qm.h | 0
> >  6 files changed, 5 insertions(+), 5 deletions(-)
> >  rename drivers/crypto/hisilicon/qm.h => include/linux/hisi_acc_qm.h
> (100%)
> >
> > diff --git a/drivers/crypto/hisilicon/hpre/hpre.h
> b/drivers/crypto/hisilicon/hpre/hpre.h
> > index e0b4a1982ee9..9a0558ed82f9 100644
> > --- a/drivers/crypto/hisilicon/hpre/hpre.h
> > +++ b/drivers/crypto/hisilicon/hpre/hpre.h
> > @@ -4,7 +4,7 @@
> >  #define __HISI_HPRE_H
> >
> >  #include <linux/list.h>
> > -#include "../qm.h"
> > +#include <linux/hisi_acc_qm.h>
> >
> >  #define HPRE_SQE_SIZE			sizeof(struct hpre_sqe)
> >  #define HPRE_PF_DEF_Q_NUM		64
> > diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c
> > index c5b84a5ea350..ed23e1d3fa27 100644
> > --- a/drivers/crypto/hisilicon/qm.c
> > +++ b/drivers/crypto/hisilicon/qm.c
> > @@ -15,7 +15,7 @@
> >  #include <linux/uacce.h>
> >  #include <linux/uaccess.h>
> >  #include <uapi/misc/uacce/hisi_qm.h>
> > -#include "qm.h"
> > +#include <linux/hisi_acc_qm.h>
> >
> >  /* eq/aeq irq enable */
> >  #define QM_VF_AEQ_INT_SOURCE		0x0
> > diff --git a/drivers/crypto/hisilicon/sec2/sec.h
> b/drivers/crypto/hisilicon/sec2/sec.h
> > index d97cf02b1df7..c2e9b01187a7 100644
> > --- a/drivers/crypto/hisilicon/sec2/sec.h
> > +++ b/drivers/crypto/hisilicon/sec2/sec.h
> > @@ -4,7 +4,7 @@
> >  #ifndef __HISI_SEC_V2_H
> >  #define __HISI_SEC_V2_H
> >
> > -#include "../qm.h"
> > +#include <linux/hisi_acc_qm.h>
> >  #include "sec_crypto.h"
> >
> >  /* Algorithm resource per hardware SEC queue */
> > diff --git a/drivers/crypto/hisilicon/sgl.c b/drivers/crypto/hisilicon/sgl.c
> > index 057273769f26..f7efc02b065f 100644
> > --- a/drivers/crypto/hisilicon/sgl.c
> > +++ b/drivers/crypto/hisilicon/sgl.c
> > @@ -1,9 +1,9 @@
> >  // SPDX-License-Identifier: GPL-2.0
> >  /* Copyright (c) 2019 HiSilicon Limited. */
> >  #include <linux/dma-mapping.h>
> > +#include <linux/hisi_acc_qm.h>
> >  #include <linux/module.h>
> >  #include <linux/slab.h>
> > -#include "qm.h"
> >
> >  #define HISI_ACC_SGL_SGE_NR_MIN		1
> >  #define HISI_ACC_SGL_NR_MAX		256
> > diff --git a/drivers/crypto/hisilicon/zip/zip.h
> b/drivers/crypto/hisilicon/zip/zip.h
> > index 517fdbdff3ea..3dfd3bac5a33 100644
> > --- a/drivers/crypto/hisilicon/zip/zip.h
> > +++ b/drivers/crypto/hisilicon/zip/zip.h
> > @@ -7,7 +7,7 @@
> >  #define pr_fmt(fmt)	"hisi_zip: " fmt
> >
> >  #include <linux/list.h>
> > -#include "../qm.h"
> > +#include <linux/hisi_acc_qm.h>
> >
> >  enum hisi_zip_error_type {
> >  	/* negative compression */
> > diff --git a/drivers/crypto/hisilicon/qm.h b/include/linux/hisi_acc_qm.h
> > similarity index 100%
> > rename from drivers/crypto/hisilicon/qm.h
> > rename to include/linux/hisi_acc_qm.h
> >

^ permalink raw reply	[flat|nested] 56+ messages in thread

* RE: [PATCH v8 3/9] hisi_acc_qm: Move VF PCI device IDs to common header
  2022-03-04  9:34   ` Zhou Wang
@ 2022-03-04 11:35     ` Shameerali Kolothum Thodi
  0 siblings, 0 replies; 56+ messages in thread
From: Shameerali Kolothum Thodi @ 2022-03-04 11:35 UTC (permalink / raw)
  To: Wangzhou (B), kvm, linux-kernel, linux-crypto, Xu Zaibo
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	Linuxarm, liulongfang, Zengtao (B),
	Jonathan Cameron

[+Zaibo]

> -----Original Message-----
> From: Wangzhou (B)
> Sent: 04 March 2022 09:35
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>;
> kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-crypto@vger.kernel.org
> Cc: linux-pci@vger.kernel.org; alex.williamson@redhat.com; jgg@nvidia.com;
> cohuck@redhat.com; mgurtovoy@nvidia.com; yishaih@nvidia.com; Linuxarm
> <linuxarm@huawei.com>; liulongfang <liulongfang@huawei.com>; Zengtao (B)
> <prime.zeng@hisilicon.com>; Jonathan Cameron
> <jonathan.cameron@huawei.com>
> Subject: Re: [PATCH v8 3/9] hisi_acc_qm: Move VF PCI device IDs to common
> header
> 
> > Move the PCI Device IDs of HiSilicon ACC VF devices to a common heade>
> and also use a uniform naming convention.
> >
> > This will be useful when we introduce the vfio PCI HiSilicon ACC live
> > migration driver in subsequent patches.
> >
> > Signed-off-by: Shameer Kolothum
> <shameerali.kolothum.thodi@huawei.com>
> 
> Acked-by: Zhou Wang <wangzhou1@hisilicon.com>

Thanks. Added Zaibo for hpre/sec parts.

> > ---
> >  drivers/crypto/hisilicon/hpre/hpre_main.c | 13 ++++++-------
> >  drivers/crypto/hisilicon/sec2/sec_main.c  | 15 +++++++--------
> >  drivers/crypto/hisilicon/zip/zip_main.c   | 11 +++++------
> >  include/linux/pci_ids.h                   |  3 +++
> >  4 files changed, 21 insertions(+), 21 deletions(-)
> >
> > diff --git a/drivers/crypto/hisilicon/hpre/hpre_main.c
> b/drivers/crypto/hisilicon/hpre/hpre_main.c
> > index ebfab3e14499..3589d8879b5e 100644
> > --- a/drivers/crypto/hisilicon/hpre/hpre_main.c
> > +++ b/drivers/crypto/hisilicon/hpre/hpre_main.c
> > @@ -68,8 +68,7 @@
> >  #define HPRE_REG_RD_INTVRL_US		10
> >  #define HPRE_REG_RD_TMOUT_US		1000
> >  #define HPRE_DBGFS_VAL_MAX_LEN		20
> > -#define HPRE_PCI_DEVICE_ID		0xa258
> > -#define HPRE_PCI_VF_DEVICE_ID		0xa259
> > +#define PCI_DEVICE_ID_HUAWEI_HPRE_PF	0xa258
> >  #define HPRE_QM_USR_CFG_MASK		GENMASK(31, 1)
> >  #define HPRE_QM_AXI_CFG_MASK		GENMASK(15, 0)
> >  #define HPRE_QM_VFG_AX_MASK		GENMASK(7, 0)
> > @@ -111,8 +110,8 @@
> >  static const char hpre_name[] = "hisi_hpre";
> >  static struct dentry *hpre_debugfs_root;
> >  static const struct pci_device_id hpre_dev_ids[] = {
> > -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, HPRE_PCI_DEVICE_ID) },
> > -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, HPRE_PCI_VF_DEVICE_ID) },
> > +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI,
> PCI_DEVICE_ID_HUAWEI_HPRE_PF) },
> > +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI,
> PCI_DEVICE_ID_HUAWEI_HPRE_VF) },
> >  	{ 0, }
> >  };
> >
> > @@ -242,7 +241,7 @@ MODULE_PARM_DESC(uacce_mode,
> UACCE_MODE_DESC);
> >
> >  static int pf_q_num_set(const char *val, const struct kernel_param *kp)
> >  {
> > -	return q_num_set(val, kp, HPRE_PCI_DEVICE_ID);
> > +	return q_num_set(val, kp, PCI_DEVICE_ID_HUAWEI_HPRE_PF);
> >  }
> >
> >  static const struct kernel_param_ops hpre_pf_q_num_ops = {
> > @@ -921,7 +920,7 @@ static int hpre_debugfs_init(struct hisi_qm *qm)
> >  	qm->debug.sqe_mask_len = HPRE_SQE_MASK_LEN;
> >  	hisi_qm_debug_init(qm);
> >
> > -	if (qm->pdev->device == HPRE_PCI_DEVICE_ID) {
> > +	if (qm->pdev->device == PCI_DEVICE_ID_HUAWEI_HPRE_PF) {
> >  		ret = hpre_ctrl_debug_init(qm);
> >  		if (ret)
> >  			goto failed_to_create;
> > @@ -958,7 +957,7 @@ static int hpre_qm_init(struct hisi_qm *qm, struct
> pci_dev *pdev)
> >  	qm->sqe_size = HPRE_SQE_SIZE;
> >  	qm->dev_name = hpre_name;
> >
> > -	qm->fun_type = (pdev->device == HPRE_PCI_DEVICE_ID) ?
> > +	qm->fun_type = (pdev->device == PCI_DEVICE_ID_HUAWEI_HPRE_PF) ?
> >  			QM_HW_PF : QM_HW_VF;
> >  	if (qm->fun_type == QM_HW_PF) {
> >  		qm->qp_base = HPRE_PF_DEF_Q_BASE;
> > diff --git a/drivers/crypto/hisilicon/sec2/sec_main.c
> b/drivers/crypto/hisilicon/sec2/sec_main.c
> > index 26d3ab1d308b..311a8747b5bf 100644
> > --- a/drivers/crypto/hisilicon/sec2/sec_main.c
> > +++ b/drivers/crypto/hisilicon/sec2/sec_main.c
> > @@ -20,8 +20,7 @@
> >
> >  #define SEC_VF_NUM			63
> >  #define SEC_QUEUE_NUM_V1		4096
> > -#define SEC_PF_PCI_DEVICE_ID		0xa255
> > -#define SEC_VF_PCI_DEVICE_ID		0xa256
> > +#define PCI_DEVICE_ID_HUAWEI_SEC_PF	0xa255
> >
> >  #define SEC_BD_ERR_CHK_EN0		0xEFFFFFFF
> >  #define SEC_BD_ERR_CHK_EN1		0x7ffff7fd
> > @@ -225,7 +224,7 @@ static const struct debugfs_reg32 sec_dfx_regs[] = {
> >
> >  static int sec_pf_q_num_set(const char *val, const struct kernel_param
> *kp)
> >  {
> > -	return q_num_set(val, kp, SEC_PF_PCI_DEVICE_ID);
> > +	return q_num_set(val, kp, PCI_DEVICE_ID_HUAWEI_SEC_PF);
> >  }
> >
> >  static const struct kernel_param_ops sec_pf_q_num_ops = {
> > @@ -313,8 +312,8 @@ module_param_cb(uacce_mode,
> &sec_uacce_mode_ops, &uacce_mode, 0444);
> >  MODULE_PARM_DESC(uacce_mode, UACCE_MODE_DESC);
> >
> >  static const struct pci_device_id sec_dev_ids[] = {
> > -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, SEC_PF_PCI_DEVICE_ID) },
> > -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, SEC_VF_PCI_DEVICE_ID) },
> > +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI,
> PCI_DEVICE_ID_HUAWEI_SEC_PF) },
> > +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI,
> PCI_DEVICE_ID_HUAWEI_SEC_VF) },
> >  	{ 0, }
> >  };
> >  MODULE_DEVICE_TABLE(pci, sec_dev_ids);
> > @@ -717,7 +716,7 @@ static int sec_core_debug_init(struct hisi_qm *qm)
> >  	regset->base = qm->io_base;
> >  	regset->dev = dev;
> >
> > -	if (qm->pdev->device == SEC_PF_PCI_DEVICE_ID)
> > +	if (qm->pdev->device == PCI_DEVICE_ID_HUAWEI_SEC_PF)
> >  		debugfs_create_file("regs", 0444, tmp_d, regset, &sec_regs_fops);
> >
> >  	for (i = 0; i < ARRAY_SIZE(sec_dfx_labels); i++) {
> > @@ -735,7 +734,7 @@ static int sec_debug_init(struct hisi_qm *qm)
> >  	struct sec_dev *sec = container_of(qm, struct sec_dev, qm);
> >  	int i;
> >
> > -	if (qm->pdev->device == SEC_PF_PCI_DEVICE_ID) {
> > +	if (qm->pdev->device == PCI_DEVICE_ID_HUAWEI_SEC_PF) {
> >  		for (i = SEC_CLEAR_ENABLE; i < SEC_DEBUG_FILE_NUM; i++) {
> >  			spin_lock_init(&sec->debug.files[i].lock);
> >  			sec->debug.files[i].index = i;
> > @@ -877,7 +876,7 @@ static int sec_qm_init(struct hisi_qm *qm, struct
> pci_dev *pdev)
> >  	qm->sqe_size = SEC_SQE_SIZE;
> >  	qm->dev_name = sec_name;
> >
> > -	qm->fun_type = (pdev->device == SEC_PF_PCI_DEVICE_ID) ?
> > +	qm->fun_type = (pdev->device == PCI_DEVICE_ID_HUAWEI_SEC_PF) ?
> >  			QM_HW_PF : QM_HW_VF;
> >  	if (qm->fun_type == QM_HW_PF) {
> >  		qm->qp_base = SEC_PF_DEF_Q_BASE;
> > diff --git a/drivers/crypto/hisilicon/zip/zip_main.c
> b/drivers/crypto/hisilicon/zip/zip_main.c
> > index 678f8b58ec42..66decfe07282 100644
> > --- a/drivers/crypto/hisilicon/zip/zip_main.c
> > +++ b/drivers/crypto/hisilicon/zip/zip_main.c
> > @@ -15,8 +15,7 @@
> >  #include <linux/uacce.h>
> >  #include "zip.h"
> >
> > -#define PCI_DEVICE_ID_ZIP_PF		0xa250
> > -#define PCI_DEVICE_ID_ZIP_VF		0xa251
> > +#define PCI_DEVICE_ID_HUAWEI_ZIP_PF	0xa250
> >
> >  #define HZIP_QUEUE_NUM_V1		4096
> >
> > @@ -246,7 +245,7 @@ MODULE_PARM_DESC(uacce_mode,
> UACCE_MODE_DESC);
> >
> >  static int pf_q_num_set(const char *val, const struct kernel_param *kp)
> >  {
> > -	return q_num_set(val, kp, PCI_DEVICE_ID_ZIP_PF);
> > +	return q_num_set(val, kp, PCI_DEVICE_ID_HUAWEI_ZIP_PF);
> >  }
> >
> >  static const struct kernel_param_ops pf_q_num_ops = {
> > @@ -268,8 +267,8 @@ module_param_cb(vfs_num, &vfs_num_ops,
> &vfs_num, 0444);
> >  MODULE_PARM_DESC(vfs_num, "Number of VFs to enable(1-63),
> 0(default)");
> >
> >  static const struct pci_device_id hisi_zip_dev_ids[] = {
> > -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_ZIP_PF) },
> > -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_ZIP_VF) },
> > +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI,
> PCI_DEVICE_ID_HUAWEI_ZIP_PF) },
> > +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI,
> PCI_DEVICE_ID_HUAWEI_ZIP_VF) },
> >  	{ 0, }
> >  };
> >  MODULE_DEVICE_TABLE(pci, hisi_zip_dev_ids);
> > @@ -838,7 +837,7 @@ static int hisi_zip_qm_init(struct hisi_qm *qm, struct
> pci_dev *pdev)
> >  	qm->sqe_size = HZIP_SQE_SIZE;
> >  	qm->dev_name = hisi_zip_name;
> >
> > -	qm->fun_type = (pdev->device == PCI_DEVICE_ID_ZIP_PF) ?
> > +	qm->fun_type = (pdev->device == PCI_DEVICE_ID_HUAWEI_ZIP_PF) ?
> >  			QM_HW_PF : QM_HW_VF;
> >  	if (qm->fun_type == QM_HW_PF) {
> >  		qm->qp_base = HZIP_PF_DEF_Q_BASE;
> > diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
> > index aad54c666407..31dee2b65a62 100644
> > --- a/include/linux/pci_ids.h
> > +++ b/include/linux/pci_ids.h
> > @@ -2529,6 +2529,9 @@
> >  #define PCI_DEVICE_ID_KORENIX_JETCARDF3	0x17ff
> >
> >  #define PCI_VENDOR_ID_HUAWEI		0x19e5
> > +#define PCI_DEVICE_ID_HUAWEI_ZIP_VF	0xa251
> > +#define PCI_DEVICE_ID_HUAWEI_SEC_VF	0xa256
> > +#define PCI_DEVICE_ID_HUAWEI_HPRE_VF	0xa259
> >
> >  #define PCI_VENDOR_ID_NETRONOME		0x19ee
> >  #define PCI_DEVICE_ID_NETRONOME_NFP4000	0x4000
> >

^ permalink raw reply	[flat|nested] 56+ messages in thread

* Re: [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live migration
  2022-03-04  8:48   ` Shameerali Kolothum Thodi
@ 2022-03-04 19:44     ` Alex Williamson
  2022-03-04 20:36       ` Shameerali Kolothum Thodi
  0 siblings, 1 reply; 56+ messages in thread
From: Alex Williamson @ 2022-03-04 19:44 UTC (permalink / raw)
  To: Shameerali Kolothum Thodi
  Cc: kvm, linux-kernel, linux-crypto, linux-pci, jgg, cohuck,
	mgurtovoy, yishaih, liulongfang, Zengtao (B),
	Jonathan Cameron, Wangzhou (B)

On Fri, 4 Mar 2022 08:48:27 +0000
Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com> wrote:

> Hi Alex,
> 
> > -----Original Message-----
> > From: Shameerali Kolothum Thodi
> > Sent: 03 March 2022 23:02
> > To: kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> > linux-crypto@vger.kernel.org
> > Cc: linux-pci@vger.kernel.org; alex.williamson@redhat.com; jgg@nvidia.com;
> > cohuck@redhat.com; mgurtovoy@nvidia.com; yishaih@nvidia.com; Linuxarm
> > <linuxarm@huawei.com>; liulongfang <liulongfang@huawei.com>; Zengtao (B)
> > <prime.zeng@hisilicon.com>; Jonathan Cameron
> > <jonathan.cameron@huawei.com>; Wangzhou (B) <wangzhou1@hisilicon.com>
> > Subject: [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live migration
> > 
> > From: Longfang Liu <liulongfang@huawei.com>
> > 
> > VMs assigned with HiSilicon ACC VF devices can now perform live migration if
> > the VF devices are bind to the hisi_acc_vfio_pci driver.
> > 
> > Signed-off-by: Longfang Liu <liulongfang@huawei.com>
> > Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>  
> 
> [...]
> > +
> > +static int vf_qm_check_match(struct hisi_acc_vf_core_device *hisi_acc_vdev,
> > +			     struct hisi_acc_vf_migration_file *migf) {
> > +	struct acc_vf_data *vf_data = &migf->vf_data;
> > +	struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
> > +	struct hisi_qm *pf_qm = &hisi_acc_vdev->vf_qm;  
> 
> Oops, the above has to be,
>   struct hisi_qm *pf_qm = hisi_acc_vdev->pf_qm;
> 
> This was actually fixed in v6, but now that I rebased mainly to v5, missed it.
> Please let me know if you want a re-spin with the above fix(in case there are no further
> comments) or this is something you can take care.

To confirm, you're looking for this change:

diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
index aa2e4b6bf598..f2a0c046413f 100644
--- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
+++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
@@ -413,7 +413,7 @@ static int vf_qm_check_match(struct hisi_acc_vf_core_device *hisi_acc_vdev,
 {
 	struct acc_vf_data *vf_data = &migf->vf_data;
 	struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
-	struct hisi_qm *pf_qm = &hisi_acc_vdev->vf_qm;
+	struct hisi_qm *pf_qm = &hisi_acc_vdev->pf_qm;
 	struct device *dev = &vf_qm->pdev->dev;
 	u32 que_iso_state;
 	int ret;

Right?  I can roll that in assuming there are no further comments that
would generate a respin.  Thanks,

Alex


^ permalink raw reply related	[flat|nested] 56+ messages in thread

* RE: [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live migration
  2022-03-04 19:44     ` Alex Williamson
@ 2022-03-04 20:36       ` Shameerali Kolothum Thodi
  2022-03-04 20:40         ` Alex Williamson
  0 siblings, 1 reply; 56+ messages in thread
From: Shameerali Kolothum Thodi @ 2022-03-04 20:36 UTC (permalink / raw)
  To: Alex Williamson
  Cc: kvm, linux-kernel, linux-crypto, linux-pci, jgg, cohuck,
	mgurtovoy, yishaih, liulongfang, Zengtao (B),
	Jonathan Cameron, Wangzhou (B),
	Linuxarm



> -----Original Message-----
> From: Alex Williamson [mailto:alex.williamson@redhat.com]
> Sent: 04 March 2022 19:44
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> Cc: kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-crypto@vger.kernel.org; linux-pci@vger.kernel.org; jgg@nvidia.com;
> cohuck@redhat.com; mgurtovoy@nvidia.com; yishaih@nvidia.com;
> liulongfang <liulongfang@huawei.com>; Zengtao (B)
> <prime.zeng@hisilicon.com>; Jonathan Cameron
> <jonathan.cameron@huawei.com>; Wangzhou (B) <wangzhou1@hisilicon.com>
> Subject: Re: [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live
> migration
> 
> On Fri, 4 Mar 2022 08:48:27 +0000
> Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> wrote:
> 
> > Hi Alex,
> >
> > > -----Original Message-----
> > > From: Shameerali Kolothum Thodi
> > > Sent: 03 March 2022 23:02
> > > To: kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> > > linux-crypto@vger.kernel.org
> > > Cc: linux-pci@vger.kernel.org; alex.williamson@redhat.com;
> jgg@nvidia.com;
> > > cohuck@redhat.com; mgurtovoy@nvidia.com; yishaih@nvidia.com;
> Linuxarm
> > > <linuxarm@huawei.com>; liulongfang <liulongfang@huawei.com>;
> Zengtao (B)
> > > <prime.zeng@hisilicon.com>; Jonathan Cameron
> > > <jonathan.cameron@huawei.com>; Wangzhou (B)
> <wangzhou1@hisilicon.com>
> > > Subject: [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live
> migration
> > >
> > > From: Longfang Liu <liulongfang@huawei.com>
> > >
> > > VMs assigned with HiSilicon ACC VF devices can now perform live
> migration if
> > > the VF devices are bind to the hisi_acc_vfio_pci driver.
> > >
> > > Signed-off-by: Longfang Liu <liulongfang@huawei.com>
> > > Signed-off-by: Shameer Kolothum
> <shameerali.kolothum.thodi@huawei.com>
> >
> > [...]
> > > +
> > > +static int vf_qm_check_match(struct hisi_acc_vf_core_device
> *hisi_acc_vdev,
> > > +			     struct hisi_acc_vf_migration_file *migf) {
> > > +	struct acc_vf_data *vf_data = &migf->vf_data;
> > > +	struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
> > > +	struct hisi_qm *pf_qm = &hisi_acc_vdev->vf_qm;
> >
> > Oops, the above has to be,
> >   struct hisi_qm *pf_qm = hisi_acc_vdev->pf_qm;
> >
> > This was actually fixed in v6, but now that I rebased mainly to v5, missed it.
> > Please let me know if you want a re-spin with the above fix(in case there are
> no further
> > comments) or this is something you can take care.
> 
> To confirm, you're looking for this change:
> 
> diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
> b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
> index aa2e4b6bf598..f2a0c046413f 100644
> --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
> +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
> @@ -413,7 +413,7 @@ static int vf_qm_check_match(struct
> hisi_acc_vf_core_device *hisi_acc_vdev,
>  {
>  	struct acc_vf_data *vf_data = &migf->vf_data;
>  	struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
> -	struct hisi_qm *pf_qm = &hisi_acc_vdev->vf_qm;
> +	struct hisi_qm *pf_qm = &hisi_acc_vdev->pf_qm;
>  	struct device *dev = &vf_qm->pdev->dev;
>  	u32 que_iso_state;
>  	int ret;
> 
> Right? 

Not really. pf_qm is a pointer. This is the change,

diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
index 53e4c5cb3a71..54813772a071 100644
--- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
+++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
@@ -413,7 +413,7 @@ static int vf_qm_check_match(struct hisi_acc_vf_core_device *hisi_acc_vdev,
 {
 	struct acc_vf_data *vf_data = &migf->vf_data;
 	struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
-	struct hisi_qm *pf_qm = &hisi_acc_vdev->vf_qm;
+	struct hisi_qm *pf_qm = hisi_acc_vdev->pf_qm;
 	struct device *dev = &vf_qm->pdev->dev;
 	u32 que_iso_state;
 	int ret;

 I can roll that in assuming there are no further comments that
> would generate a respin.  Thanks,
> 
Thanks,
Shameer


^ permalink raw reply related	[flat|nested] 56+ messages in thread

* Re: [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live migration
  2022-03-04 20:36       ` Shameerali Kolothum Thodi
@ 2022-03-04 20:40         ` Alex Williamson
  0 siblings, 0 replies; 56+ messages in thread
From: Alex Williamson @ 2022-03-04 20:40 UTC (permalink / raw)
  To: Shameerali Kolothum Thodi
  Cc: kvm, linux-kernel, linux-crypto, linux-pci, jgg, cohuck,
	mgurtovoy, yishaih, liulongfang, Zengtao (B),
	Jonathan Cameron, Wangzhou (B),
	Linuxarm

On Fri, 4 Mar 2022 20:36:24 +0000
Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com> wrote:

> > -----Original Message-----
> > From: Alex Williamson [mailto:alex.williamson@redhat.com]
> > Sent: 04 March 2022 19:44
> > To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> > Cc: kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> > linux-crypto@vger.kernel.org; linux-pci@vger.kernel.org; jgg@nvidia.com;
> > cohuck@redhat.com; mgurtovoy@nvidia.com; yishaih@nvidia.com;
> > liulongfang <liulongfang@huawei.com>; Zengtao (B)
> > <prime.zeng@hisilicon.com>; Jonathan Cameron
> > <jonathan.cameron@huawei.com>; Wangzhou (B) <wangzhou1@hisilicon.com>
> > Subject: Re: [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live
> > migration
> > 
> > On Fri, 4 Mar 2022 08:48:27 +0000
> > Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> > wrote:
> >   
> > > Hi Alex,
> > >  
> > > > -----Original Message-----
> > > > From: Shameerali Kolothum Thodi
> > > > Sent: 03 March 2022 23:02
> > > > To: kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> > > > linux-crypto@vger.kernel.org
> > > > Cc: linux-pci@vger.kernel.org; alex.williamson@redhat.com;  
> > jgg@nvidia.com;  
> > > > cohuck@redhat.com; mgurtovoy@nvidia.com; yishaih@nvidia.com;  
> > Linuxarm  
> > > > <linuxarm@huawei.com>; liulongfang <liulongfang@huawei.com>;  
> > Zengtao (B)  
> > > > <prime.zeng@hisilicon.com>; Jonathan Cameron
> > > > <jonathan.cameron@huawei.com>; Wangzhou (B)  
> > <wangzhou1@hisilicon.com>  
> > > > Subject: [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live  
> > migration  
> > > >
> > > > From: Longfang Liu <liulongfang@huawei.com>
> > > >
> > > > VMs assigned with HiSilicon ACC VF devices can now perform live  
> > migration if  
> > > > the VF devices are bind to the hisi_acc_vfio_pci driver.
> > > >
> > > > Signed-off-by: Longfang Liu <liulongfang@huawei.com>
> > > > Signed-off-by: Shameer Kolothum  
> > <shameerali.kolothum.thodi@huawei.com>  
> > >
> > > [...]  
> > > > +
> > > > +static int vf_qm_check_match(struct hisi_acc_vf_core_device  
> > *hisi_acc_vdev,  
> > > > +			     struct hisi_acc_vf_migration_file *migf) {
> > > > +	struct acc_vf_data *vf_data = &migf->vf_data;
> > > > +	struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
> > > > +	struct hisi_qm *pf_qm = &hisi_acc_vdev->vf_qm;  
> > >
> > > Oops, the above has to be,
> > >   struct hisi_qm *pf_qm = hisi_acc_vdev->pf_qm;
> > >
> > > This was actually fixed in v6, but now that I rebased mainly to v5, missed it.
> > > Please let me know if you want a re-spin with the above fix(in case there are  
> > no further  
> > > comments) or this is something you can take care.  
> > 
> > To confirm, you're looking for this change:
> > 
> > diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
> > b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
> > index aa2e4b6bf598..f2a0c046413f 100644
> > --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
> > +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
> > @@ -413,7 +413,7 @@ static int vf_qm_check_match(struct
> > hisi_acc_vf_core_device *hisi_acc_vdev,
> >  {
> >  	struct acc_vf_data *vf_data = &migf->vf_data;
> >  	struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
> > -	struct hisi_qm *pf_qm = &hisi_acc_vdev->vf_qm;
> > +	struct hisi_qm *pf_qm = &hisi_acc_vdev->pf_qm;
> >  	struct device *dev = &vf_qm->pdev->dev;
> >  	u32 que_iso_state;
> >  	int ret;
> > 
> > Right?   
> 
> Not really. pf_qm is a pointer. This is the change,
> 
> diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
> index 53e4c5cb3a71..54813772a071 100644
> --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
> +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
> @@ -413,7 +413,7 @@ static int vf_qm_check_match(struct hisi_acc_vf_core_device *hisi_acc_vdev,
>  {
>  	struct acc_vf_data *vf_data = &migf->vf_data;
>  	struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
> -	struct hisi_qm *pf_qm = &hisi_acc_vdev->vf_qm;
> +	struct hisi_qm *pf_qm = hisi_acc_vdev->pf_qm;
>  	struct device *dev = &vf_qm->pdev->dev;
>  	u32 que_iso_state;
>  	int ret;
> 

Got it.  Thanks,

Alex


^ permalink raw reply	[flat|nested] 56+ messages in thread

* Re: [PATCH v8 9/9] hisi_acc_vfio_pci: Use its own PCI reset_done error handler
  2022-03-03 23:01 ` [PATCH v8 9/9] hisi_acc_vfio_pci: Use its own PCI reset_done error handler Shameer Kolothum
@ 2022-03-04 20:54   ` Jason Gunthorpe
  2022-03-08  1:14   ` liulongfang
  1 sibling, 0 replies; 56+ messages in thread
From: Jason Gunthorpe @ 2022-03-04 20:54 UTC (permalink / raw)
  To: Shameer Kolothum
  Cc: kvm, linux-kernel, linux-crypto, linux-pci, alex.williamson,
	cohuck, mgurtovoy, yishaih, linuxarm, liulongfang, prime.zeng,
	jonathan.cameron, wangzhou1

On Thu, Mar 03, 2022 at 11:01:31PM +0000, Shameer Kolothum wrote:
> Register private handler for pci_error_handlers.reset_done and update
> state accordingly.
> 
> Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
> ---
>  .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    | 57 ++++++++++++++++++-
>  .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h    |  4 +-
>  2 files changed, 57 insertions(+), 4 deletions(-)

It looks OK to me

Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>

Jason

^ permalink raw reply	[flat|nested] 56+ messages in thread

* Re: [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live migration
  2022-03-03 23:01 ` [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live migration Shameer Kolothum
  2022-03-04  8:48   ` Shameerali Kolothum Thodi
@ 2022-03-04 20:57   ` Jason Gunthorpe
  2022-03-07 19:05     ` Alex Williamson
  2022-03-08  7:41   ` Tian, Kevin
  2 siblings, 1 reply; 56+ messages in thread
From: Jason Gunthorpe @ 2022-03-04 20:57 UTC (permalink / raw)
  To: Shameer Kolothum
  Cc: kvm, linux-kernel, linux-crypto, linux-pci, alex.williamson,
	cohuck, mgurtovoy, yishaih, linuxarm, liulongfang, prime.zeng,
	jonathan.cameron, wangzhou1

On Thu, Mar 03, 2022 at 11:01:30PM +0000, Shameer Kolothum wrote:
> From: Longfang Liu <liulongfang@huawei.com>
> 
> VMs assigned with HiSilicon ACC VF devices can now perform live migration
> if the VF devices are bind to the hisi_acc_vfio_pci driver.
> 
> Signed-off-by: Longfang Liu <liulongfang@huawei.com>
> Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
> ---
>  drivers/vfio/pci/hisilicon/Kconfig            |    7 +
>  .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    | 1078 ++++++++++++++++-
>  .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h    |  114 ++
>  3 files changed, 1181 insertions(+), 18 deletions(-)
>  create mode 100644 drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
> 
> diff --git a/drivers/vfio/pci/hisilicon/Kconfig b/drivers/vfio/pci/hisilicon/Kconfig
> index dc723bad05c2..2a68d39f339f 100644
> --- a/drivers/vfio/pci/hisilicon/Kconfig
> +++ b/drivers/vfio/pci/hisilicon/Kconfig
> @@ -3,6 +3,13 @@ config HISI_ACC_VFIO_PCI
>  	tristate "VFIO PCI support for HiSilicon ACC devices"
>  	depends on ARM64 || (COMPILE_TEST && 64BIT)
>  	depends on VFIO_PCI_CORE
> +	depends on PCI && PCI_MSI

PCI is already in the depends from the 2nd line in
drivers/vfio/pci/Kconfig, but it is harmless

> +	depends on UACCE || UACCE=n
> +	depends on ACPI

Scratching my head a bit on why we have these

Looks OK though:

Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>

Jason

^ permalink raw reply	[flat|nested] 56+ messages in thread

* Re: [PATCH v8 3/9] hisi_acc_qm: Move VF PCI device IDs to common header
  2022-03-03 23:01 ` [PATCH v8 3/9] hisi_acc_qm: Move VF PCI device IDs " Shameer Kolothum
  2022-03-04  9:34   ` Zhou Wang
@ 2022-03-07 17:53   ` Alex Williamson
  2022-03-10 13:55     ` Shameerali Kolothum Thodi
  2022-03-08  1:08   ` liulongfang
  2022-03-08 10:28   ` yekai(A)
  3 siblings, 1 reply; 56+ messages in thread
From: Alex Williamson @ 2022-03-07 17:53 UTC (permalink / raw)
  To: Shameer Kolothum, Bjorn Helgaas
  Cc: kvm, linux-kernel, linux-crypto, linux-pci, jgg, cohuck,
	mgurtovoy, yishaih, linuxarm, liulongfang, prime.zeng,
	jonathan.cameron, wangzhou1

Hi Bjorn,

Here's the respin of this patch that adds only the VF device IDs to
pci_ids.h.  The next patch in the series[1] adds a consumer of these
IDs as a vfio-pci vendor driver.  Thanks,

Alex

[1]https://lore.kernel.org/all/20220303230131.2103-5-shameerali.kolothum.thodi@huawei.com/

On Thu, 3 Mar 2022 23:01:25 +0000
Shameer Kolothum <shameerali.kolothum.thodi@huawei.com> wrote:

> Move the PCI Device IDs of HiSilicon ACC VF devices to a common header
> and also use a uniform naming convention.
> 
> This will be useful when we introduce the vfio PCI HiSilicon ACC live
> migration driver in subsequent patches.
> 
> Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
> ---
>  drivers/crypto/hisilicon/hpre/hpre_main.c | 13 ++++++-------
>  drivers/crypto/hisilicon/sec2/sec_main.c  | 15 +++++++--------
>  drivers/crypto/hisilicon/zip/zip_main.c   | 11 +++++------
>  include/linux/pci_ids.h                   |  3 +++
>  4 files changed, 21 insertions(+), 21 deletions(-)
> 
> diff --git a/drivers/crypto/hisilicon/hpre/hpre_main.c b/drivers/crypto/hisilicon/hpre/hpre_main.c
> index ebfab3e14499..3589d8879b5e 100644
> --- a/drivers/crypto/hisilicon/hpre/hpre_main.c
> +++ b/drivers/crypto/hisilicon/hpre/hpre_main.c
> @@ -68,8 +68,7 @@
>  #define HPRE_REG_RD_INTVRL_US		10
>  #define HPRE_REG_RD_TMOUT_US		1000
>  #define HPRE_DBGFS_VAL_MAX_LEN		20
> -#define HPRE_PCI_DEVICE_ID		0xa258
> -#define HPRE_PCI_VF_DEVICE_ID		0xa259
> +#define PCI_DEVICE_ID_HUAWEI_HPRE_PF	0xa258
>  #define HPRE_QM_USR_CFG_MASK		GENMASK(31, 1)
>  #define HPRE_QM_AXI_CFG_MASK		GENMASK(15, 0)
>  #define HPRE_QM_VFG_AX_MASK		GENMASK(7, 0)
> @@ -111,8 +110,8 @@
>  static const char hpre_name[] = "hisi_hpre";
>  static struct dentry *hpre_debugfs_root;
>  static const struct pci_device_id hpre_dev_ids[] = {
> -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, HPRE_PCI_DEVICE_ID) },
> -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, HPRE_PCI_VF_DEVICE_ID) },
> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_HPRE_PF) },
> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_HPRE_VF) },
>  	{ 0, }
>  };
>  
> @@ -242,7 +241,7 @@ MODULE_PARM_DESC(uacce_mode, UACCE_MODE_DESC);
>  
>  static int pf_q_num_set(const char *val, const struct kernel_param *kp)
>  {
> -	return q_num_set(val, kp, HPRE_PCI_DEVICE_ID);
> +	return q_num_set(val, kp, PCI_DEVICE_ID_HUAWEI_HPRE_PF);
>  }
>  
>  static const struct kernel_param_ops hpre_pf_q_num_ops = {
> @@ -921,7 +920,7 @@ static int hpre_debugfs_init(struct hisi_qm *qm)
>  	qm->debug.sqe_mask_len = HPRE_SQE_MASK_LEN;
>  	hisi_qm_debug_init(qm);
>  
> -	if (qm->pdev->device == HPRE_PCI_DEVICE_ID) {
> +	if (qm->pdev->device == PCI_DEVICE_ID_HUAWEI_HPRE_PF) {
>  		ret = hpre_ctrl_debug_init(qm);
>  		if (ret)
>  			goto failed_to_create;
> @@ -958,7 +957,7 @@ static int hpre_qm_init(struct hisi_qm *qm, struct pci_dev *pdev)
>  	qm->sqe_size = HPRE_SQE_SIZE;
>  	qm->dev_name = hpre_name;
>  
> -	qm->fun_type = (pdev->device == HPRE_PCI_DEVICE_ID) ?
> +	qm->fun_type = (pdev->device == PCI_DEVICE_ID_HUAWEI_HPRE_PF) ?
>  			QM_HW_PF : QM_HW_VF;
>  	if (qm->fun_type == QM_HW_PF) {
>  		qm->qp_base = HPRE_PF_DEF_Q_BASE;
> diff --git a/drivers/crypto/hisilicon/sec2/sec_main.c b/drivers/crypto/hisilicon/sec2/sec_main.c
> index 26d3ab1d308b..311a8747b5bf 100644
> --- a/drivers/crypto/hisilicon/sec2/sec_main.c
> +++ b/drivers/crypto/hisilicon/sec2/sec_main.c
> @@ -20,8 +20,7 @@
>  
>  #define SEC_VF_NUM			63
>  #define SEC_QUEUE_NUM_V1		4096
> -#define SEC_PF_PCI_DEVICE_ID		0xa255
> -#define SEC_VF_PCI_DEVICE_ID		0xa256
> +#define PCI_DEVICE_ID_HUAWEI_SEC_PF	0xa255
>  
>  #define SEC_BD_ERR_CHK_EN0		0xEFFFFFFF
>  #define SEC_BD_ERR_CHK_EN1		0x7ffff7fd
> @@ -225,7 +224,7 @@ static const struct debugfs_reg32 sec_dfx_regs[] = {
>  
>  static int sec_pf_q_num_set(const char *val, const struct kernel_param *kp)
>  {
> -	return q_num_set(val, kp, SEC_PF_PCI_DEVICE_ID);
> +	return q_num_set(val, kp, PCI_DEVICE_ID_HUAWEI_SEC_PF);
>  }
>  
>  static const struct kernel_param_ops sec_pf_q_num_ops = {
> @@ -313,8 +312,8 @@ module_param_cb(uacce_mode, &sec_uacce_mode_ops, &uacce_mode, 0444);
>  MODULE_PARM_DESC(uacce_mode, UACCE_MODE_DESC);
>  
>  static const struct pci_device_id sec_dev_ids[] = {
> -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, SEC_PF_PCI_DEVICE_ID) },
> -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, SEC_VF_PCI_DEVICE_ID) },
> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_SEC_PF) },
> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_SEC_VF) },
>  	{ 0, }
>  };
>  MODULE_DEVICE_TABLE(pci, sec_dev_ids);
> @@ -717,7 +716,7 @@ static int sec_core_debug_init(struct hisi_qm *qm)
>  	regset->base = qm->io_base;
>  	regset->dev = dev;
>  
> -	if (qm->pdev->device == SEC_PF_PCI_DEVICE_ID)
> +	if (qm->pdev->device == PCI_DEVICE_ID_HUAWEI_SEC_PF)
>  		debugfs_create_file("regs", 0444, tmp_d, regset, &sec_regs_fops);
>  
>  	for (i = 0; i < ARRAY_SIZE(sec_dfx_labels); i++) {
> @@ -735,7 +734,7 @@ static int sec_debug_init(struct hisi_qm *qm)
>  	struct sec_dev *sec = container_of(qm, struct sec_dev, qm);
>  	int i;
>  
> -	if (qm->pdev->device == SEC_PF_PCI_DEVICE_ID) {
> +	if (qm->pdev->device == PCI_DEVICE_ID_HUAWEI_SEC_PF) {
>  		for (i = SEC_CLEAR_ENABLE; i < SEC_DEBUG_FILE_NUM; i++) {
>  			spin_lock_init(&sec->debug.files[i].lock);
>  			sec->debug.files[i].index = i;
> @@ -877,7 +876,7 @@ static int sec_qm_init(struct hisi_qm *qm, struct pci_dev *pdev)
>  	qm->sqe_size = SEC_SQE_SIZE;
>  	qm->dev_name = sec_name;
>  
> -	qm->fun_type = (pdev->device == SEC_PF_PCI_DEVICE_ID) ?
> +	qm->fun_type = (pdev->device == PCI_DEVICE_ID_HUAWEI_SEC_PF) ?
>  			QM_HW_PF : QM_HW_VF;
>  	if (qm->fun_type == QM_HW_PF) {
>  		qm->qp_base = SEC_PF_DEF_Q_BASE;
> diff --git a/drivers/crypto/hisilicon/zip/zip_main.c b/drivers/crypto/hisilicon/zip/zip_main.c
> index 678f8b58ec42..66decfe07282 100644
> --- a/drivers/crypto/hisilicon/zip/zip_main.c
> +++ b/drivers/crypto/hisilicon/zip/zip_main.c
> @@ -15,8 +15,7 @@
>  #include <linux/uacce.h>
>  #include "zip.h"
>  
> -#define PCI_DEVICE_ID_ZIP_PF		0xa250
> -#define PCI_DEVICE_ID_ZIP_VF		0xa251
> +#define PCI_DEVICE_ID_HUAWEI_ZIP_PF	0xa250
>  
>  #define HZIP_QUEUE_NUM_V1		4096
>  
> @@ -246,7 +245,7 @@ MODULE_PARM_DESC(uacce_mode, UACCE_MODE_DESC);
>  
>  static int pf_q_num_set(const char *val, const struct kernel_param *kp)
>  {
> -	return q_num_set(val, kp, PCI_DEVICE_ID_ZIP_PF);
> +	return q_num_set(val, kp, PCI_DEVICE_ID_HUAWEI_ZIP_PF);
>  }
>  
>  static const struct kernel_param_ops pf_q_num_ops = {
> @@ -268,8 +267,8 @@ module_param_cb(vfs_num, &vfs_num_ops, &vfs_num, 0444);
>  MODULE_PARM_DESC(vfs_num, "Number of VFs to enable(1-63), 0(default)");
>  
>  static const struct pci_device_id hisi_zip_dev_ids[] = {
> -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_ZIP_PF) },
> -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_ZIP_VF) },
> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_ZIP_PF) },
> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_ZIP_VF) },
>  	{ 0, }
>  };
>  MODULE_DEVICE_TABLE(pci, hisi_zip_dev_ids);
> @@ -838,7 +837,7 @@ static int hisi_zip_qm_init(struct hisi_qm *qm, struct pci_dev *pdev)
>  	qm->sqe_size = HZIP_SQE_SIZE;
>  	qm->dev_name = hisi_zip_name;
>  
> -	qm->fun_type = (pdev->device == PCI_DEVICE_ID_ZIP_PF) ?
> +	qm->fun_type = (pdev->device == PCI_DEVICE_ID_HUAWEI_ZIP_PF) ?
>  			QM_HW_PF : QM_HW_VF;
>  	if (qm->fun_type == QM_HW_PF) {
>  		qm->qp_base = HZIP_PF_DEF_Q_BASE;
> diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
> index aad54c666407..31dee2b65a62 100644
> --- a/include/linux/pci_ids.h
> +++ b/include/linux/pci_ids.h
> @@ -2529,6 +2529,9 @@
>  #define PCI_DEVICE_ID_KORENIX_JETCARDF3	0x17ff
>  
>  #define PCI_VENDOR_ID_HUAWEI		0x19e5
> +#define PCI_DEVICE_ID_HUAWEI_ZIP_VF	0xa251
> +#define PCI_DEVICE_ID_HUAWEI_SEC_VF	0xa256
> +#define PCI_DEVICE_ID_HUAWEI_HPRE_VF	0xa259
>  
>  #define PCI_VENDOR_ID_NETRONOME		0x19ee
>  #define PCI_DEVICE_ID_NETRONOME_NFP4000	0x4000


^ permalink raw reply	[flat|nested] 56+ messages in thread

* Re: [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live migration
  2022-03-04 20:57   ` Jason Gunthorpe
@ 2022-03-07 19:05     ` Alex Williamson
  2022-03-07 19:29       ` Shameerali Kolothum Thodi
  0 siblings, 1 reply; 56+ messages in thread
From: Alex Williamson @ 2022-03-07 19:05 UTC (permalink / raw)
  To: Jason Gunthorpe
  Cc: Shameer Kolothum, kvm, linux-kernel, linux-crypto, linux-pci,
	cohuck, mgurtovoy, yishaih, linuxarm, liulongfang, prime.zeng,
	jonathan.cameron, wangzhou1

On Fri, 4 Mar 2022 16:57:20 -0400
Jason Gunthorpe <jgg@nvidia.com> wrote:

> On Thu, Mar 03, 2022 at 11:01:30PM +0000, Shameer Kolothum wrote:
> > From: Longfang Liu <liulongfang@huawei.com>
> > 
> > VMs assigned with HiSilicon ACC VF devices can now perform live migration
> > if the VF devices are bind to the hisi_acc_vfio_pci driver.
> > 
> > Signed-off-by: Longfang Liu <liulongfang@huawei.com>
> > Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
> > ---
> >  drivers/vfio/pci/hisilicon/Kconfig            |    7 +
> >  .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    | 1078 ++++++++++++++++-
> >  .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h    |  114 ++
> >  3 files changed, 1181 insertions(+), 18 deletions(-)
> >  create mode 100644 drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
> > 
> > diff --git a/drivers/vfio/pci/hisilicon/Kconfig b/drivers/vfio/pci/hisilicon/Kconfig
> > index dc723bad05c2..2a68d39f339f 100644
> > --- a/drivers/vfio/pci/hisilicon/Kconfig
> > +++ b/drivers/vfio/pci/hisilicon/Kconfig
> > @@ -3,6 +3,13 @@ config HISI_ACC_VFIO_PCI
> >  	tristate "VFIO PCI support for HiSilicon ACC devices"
> >  	depends on ARM64 || (COMPILE_TEST && 64BIT)
> >  	depends on VFIO_PCI_CORE
> > +	depends on PCI && PCI_MSI  
> 
> PCI is already in the depends from the 2nd line in
> drivers/vfio/pci/Kconfig, but it is harmless
> 
> > +	depends on UACCE || UACCE=n
> > +	depends on ACPI  
> 
> Scratching my head a bit on why we have these

Same curiosity from me, each of the CRYPTO_DEV_HISI_* options selected
also depend on these so they seem redundant.

I think we still require acks from Bjorn and Zaibo for select patches
in this series.

From me, I would request a MAINTAINERS entry similar to the one the
mlx5 folks added for their driver.  This should be in patch 4/9 where
the driver is originally added.  Thanks,

Alex


^ permalink raw reply	[flat|nested] 56+ messages in thread

* RE: [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live migration
  2022-03-07 19:05     ` Alex Williamson
@ 2022-03-07 19:29       ` Shameerali Kolothum Thodi
  2022-03-07 19:52         ` Alex Williamson
  0 siblings, 1 reply; 56+ messages in thread
From: Shameerali Kolothum Thodi @ 2022-03-07 19:29 UTC (permalink / raw)
  To: Alex Williamson, Jason Gunthorpe
  Cc: kvm, linux-kernel, linux-crypto, linux-pci, cohuck, mgurtovoy,
	yishaih, Linuxarm, liulongfang, Zengtao (B),
	Jonathan Cameron, Wangzhou (B),
	Xu Zaibo



> -----Original Message-----
> From: Alex Williamson [mailto:alex.williamson@redhat.com]
> Sent: 07 March 2022 19:05
> To: Jason Gunthorpe <jgg@nvidia.com>
> Cc: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>;
> kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-crypto@vger.kernel.org; linux-pci@vger.kernel.org; cohuck@redhat.com;
> mgurtovoy@nvidia.com; yishaih@nvidia.com; Linuxarm
> <linuxarm@huawei.com>; liulongfang <liulongfang@huawei.com>; Zengtao (B)
> <prime.zeng@hisilicon.com>; Jonathan Cameron
> <jonathan.cameron@huawei.com>; Wangzhou (B) <wangzhou1@hisilicon.com>
> Subject: Re: [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live
> migration
> 
> On Fri, 4 Mar 2022 16:57:20 -0400
> Jason Gunthorpe <jgg@nvidia.com> wrote:
> 
> > On Thu, Mar 03, 2022 at 11:01:30PM +0000, Shameer Kolothum wrote:
> > > From: Longfang Liu <liulongfang@huawei.com>
> > >
> > > VMs assigned with HiSilicon ACC VF devices can now perform live
> migration
> > > if the VF devices are bind to the hisi_acc_vfio_pci driver.
> > >
> > > Signed-off-by: Longfang Liu <liulongfang@huawei.com>
> > > Signed-off-by: Shameer Kolothum
> <shameerali.kolothum.thodi@huawei.com>
> > > ---
> > >  drivers/vfio/pci/hisilicon/Kconfig            |    7 +
> > >  .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    | 1078 ++++++++++++++++-
> > >  .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h    |  114 ++
> > >  3 files changed, 1181 insertions(+), 18 deletions(-)
> > >  create mode 100644 drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
> > >
> > > diff --git a/drivers/vfio/pci/hisilicon/Kconfig
> b/drivers/vfio/pci/hisilicon/Kconfig
> > > index dc723bad05c2..2a68d39f339f 100644
> > > --- a/drivers/vfio/pci/hisilicon/Kconfig
> > > +++ b/drivers/vfio/pci/hisilicon/Kconfig
> > > @@ -3,6 +3,13 @@ config HISI_ACC_VFIO_PCI
> > >  	tristate "VFIO PCI support for HiSilicon ACC devices"
> > >  	depends on ARM64 || (COMPILE_TEST && 64BIT)
> > >  	depends on VFIO_PCI_CORE
> > > +	depends on PCI && PCI_MSI
> >
> > PCI is already in the depends from the 2nd line in
> > drivers/vfio/pci/Kconfig, but it is harmless
> >
> > > +	depends on UACCE || UACCE=n
> > > +	depends on ACPI
> >
> > Scratching my head a bit on why we have these
> 
> Same curiosity from me, each of the CRYPTO_DEV_HISI_* options selected
> also depend on these so they seem redundant.

Yes, they are redundant now since we have added CRYPTO_DEV_HISI_ drivers
as "depends" now. I will remove that.
 
> I think we still require acks from Bjorn and Zaibo for select patches
> in this series.

I checked with Ziabo. He moved projects and is no longer looking into crypto stuff.
Wangzhou and LiuLongfang now take care of this. Received acks from Wangzhou
already and I will request Longfang to provide his. Hope that's ok.

> 
> From me, I would request a MAINTAINERS entry similar to the one the
> mlx5 folks added for their driver.  This should be in patch 4/9 where
> the driver is originally added.  Thanks,

Ok I will do that. I will pick up the R-by tags as well and send out v9 by
tomorrow.

Thanks,
Shameer 


^ permalink raw reply	[flat|nested] 56+ messages in thread

* Re: [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live migration
  2022-03-07 19:29       ` Shameerali Kolothum Thodi
@ 2022-03-07 19:52         ` Alex Williamson
  2022-03-08  8:11           ` Tian, Kevin
  2022-03-08  9:46           ` liulongfang
  0 siblings, 2 replies; 56+ messages in thread
From: Alex Williamson @ 2022-03-07 19:52 UTC (permalink / raw)
  To: Shameerali Kolothum Thodi
  Cc: Jason Gunthorpe, kvm, linux-kernel, linux-crypto, linux-pci,
	cohuck, mgurtovoy, yishaih, Linuxarm, liulongfang, Zengtao (B),
	Jonathan Cameron, Wangzhou (B),
	Xu Zaibo

On Mon, 7 Mar 2022 19:29:06 +0000
Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com> wrote:

> > -----Original Message-----
> > From: Alex Williamson [mailto:alex.williamson@redhat.com]
> > Sent: 07 March 2022 19:05
> > To: Jason Gunthorpe <jgg@nvidia.com>
> > Cc: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>;
> > kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> > linux-crypto@vger.kernel.org; linux-pci@vger.kernel.org; cohuck@redhat.com;
> > mgurtovoy@nvidia.com; yishaih@nvidia.com; Linuxarm
> > <linuxarm@huawei.com>; liulongfang <liulongfang@huawei.com>; Zengtao (B)
> > <prime.zeng@hisilicon.com>; Jonathan Cameron
> > <jonathan.cameron@huawei.com>; Wangzhou (B) <wangzhou1@hisilicon.com>
> > Subject: Re: [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live
> > migration
> > 
> > On Fri, 4 Mar 2022 16:57:20 -0400
> > Jason Gunthorpe <jgg@nvidia.com> wrote:
> >   
> > > On Thu, Mar 03, 2022 at 11:01:30PM +0000, Shameer Kolothum wrote:  
> > > > From: Longfang Liu <liulongfang@huawei.com>
> > > >
> > > > VMs assigned with HiSilicon ACC VF devices can now perform live  
> > migration  
> > > > if the VF devices are bind to the hisi_acc_vfio_pci driver.
> > > >
> > > > Signed-off-by: Longfang Liu <liulongfang@huawei.com>
> > > > Signed-off-by: Shameer Kolothum  
> > <shameerali.kolothum.thodi@huawei.com>  
> > > > ---
> > > >  drivers/vfio/pci/hisilicon/Kconfig            |    7 +
> > > >  .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    | 1078 ++++++++++++++++-
> > > >  .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h    |  114 ++
> > > >  3 files changed, 1181 insertions(+), 18 deletions(-)
> > > >  create mode 100644 drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
> > > >
> > > > diff --git a/drivers/vfio/pci/hisilicon/Kconfig  
> > b/drivers/vfio/pci/hisilicon/Kconfig  
> > > > index dc723bad05c2..2a68d39f339f 100644
> > > > --- a/drivers/vfio/pci/hisilicon/Kconfig
> > > > +++ b/drivers/vfio/pci/hisilicon/Kconfig
> > > > @@ -3,6 +3,13 @@ config HISI_ACC_VFIO_PCI
> > > >  	tristate "VFIO PCI support for HiSilicon ACC devices"
> > > >  	depends on ARM64 || (COMPILE_TEST && 64BIT)
> > > >  	depends on VFIO_PCI_CORE
> > > > +	depends on PCI && PCI_MSI  
> > >
> > > PCI is already in the depends from the 2nd line in
> > > drivers/vfio/pci/Kconfig, but it is harmless
> > >  
> > > > +	depends on UACCE || UACCE=n
> > > > +	depends on ACPI  
> > >
> > > Scratching my head a bit on why we have these  
> > 
> > Same curiosity from me, each of the CRYPTO_DEV_HISI_* options selected
> > also depend on these so they seem redundant.  
> 
> Yes, they are redundant now since we have added CRYPTO_DEV_HISI_ drivers
> as "depends" now. I will remove that.
>  
> > I think we still require acks from Bjorn and Zaibo for select patches
> > in this series.  
> 
> I checked with Ziabo. He moved projects and is no longer looking into crypto stuff.
> Wangzhou and LiuLongfang now take care of this. Received acks from Wangzhou
> already and I will request Longfang to provide his. Hope that's ok.

Maybe a good time to have them update MAINTAINERS as well.  Thanks,

Alex


^ permalink raw reply	[flat|nested] 56+ messages in thread

* Re: [PATCH v8 1/9] crypto: hisilicon/qm: Move the QM header to include/linux
  2022-03-03 23:01 ` [PATCH v8 1/9] crypto: hisilicon/qm: Move the QM header to include/linux Shameer Kolothum
  2022-03-04  9:03   ` Zhou Wang
@ 2022-03-08  1:07   ` liulongfang
  2022-03-08 10:27   ` yekai(A)
  2 siblings, 0 replies; 56+ messages in thread
From: liulongfang @ 2022-03-08  1:07 UTC (permalink / raw)
  To: Shameer Kolothum, kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	linuxarm, prime.zeng, jonathan.cameron, wangzhou1

On 2022/3/4 7:01, Shameer Kolothum Wrote:
> Since we are going to introduce VFIO PCI HiSilicon ACC driver for live
> migration in subsequent patches, move the ACC QM header file to a
> common include dir.
> 
> Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
> ---
>  drivers/crypto/hisilicon/hpre/hpre.h                         | 2 +-
>  drivers/crypto/hisilicon/qm.c                                | 2 +-
>  drivers/crypto/hisilicon/sec2/sec.h                          | 2 +-
>  drivers/crypto/hisilicon/sgl.c                               | 2 +-
>  drivers/crypto/hisilicon/zip/zip.h                           | 2 +-
>  drivers/crypto/hisilicon/qm.h => include/linux/hisi_acc_qm.h | 0
>  6 files changed, 5 insertions(+), 5 deletions(-)
>  rename drivers/crypto/hisilicon/qm.h => include/linux/hisi_acc_qm.h (100%)
> 
> diff --git a/drivers/crypto/hisilicon/hpre/hpre.h b/drivers/crypto/hisilicon/hpre/hpre.h
> index e0b4a1982ee9..9a0558ed82f9 100644
> --- a/drivers/crypto/hisilicon/hpre/hpre.h
> +++ b/drivers/crypto/hisilicon/hpre/hpre.h
> @@ -4,7 +4,7 @@
>  #define __HISI_HPRE_H
>  
>  #include <linux/list.h>
> -#include "../qm.h"
> +#include <linux/hisi_acc_qm.h>
>  
>  #define HPRE_SQE_SIZE			sizeof(struct hpre_sqe)
>  #define HPRE_PF_DEF_Q_NUM		64
> diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c
> index c5b84a5ea350..ed23e1d3fa27 100644
> --- a/drivers/crypto/hisilicon/qm.c
> +++ b/drivers/crypto/hisilicon/qm.c
> @@ -15,7 +15,7 @@
>  #include <linux/uacce.h>
>  #include <linux/uaccess.h>
>  #include <uapi/misc/uacce/hisi_qm.h>
> -#include "qm.h"
> +#include <linux/hisi_acc_qm.h>
>  
>  /* eq/aeq irq enable */
>  #define QM_VF_AEQ_INT_SOURCE		0x0
> diff --git a/drivers/crypto/hisilicon/sec2/sec.h b/drivers/crypto/hisilicon/sec2/sec.h
> index d97cf02b1df7..c2e9b01187a7 100644
> --- a/drivers/crypto/hisilicon/sec2/sec.h
> +++ b/drivers/crypto/hisilicon/sec2/sec.h
> @@ -4,7 +4,7 @@
>  #ifndef __HISI_SEC_V2_H
>  #define __HISI_SEC_V2_H
>  
> -#include "../qm.h"
> +#include <linux/hisi_acc_qm.h>
>  #include "sec_crypto.h"
>  
>  /* Algorithm resource per hardware SEC queue */
> diff --git a/drivers/crypto/hisilicon/sgl.c b/drivers/crypto/hisilicon/sgl.c
> index 057273769f26..f7efc02b065f 100644
> --- a/drivers/crypto/hisilicon/sgl.c
> +++ b/drivers/crypto/hisilicon/sgl.c
> @@ -1,9 +1,9 @@
>  // SPDX-License-Identifier: GPL-2.0
>  /* Copyright (c) 2019 HiSilicon Limited. */
>  #include <linux/dma-mapping.h>
> +#include <linux/hisi_acc_qm.h>
>  #include <linux/module.h>
>  #include <linux/slab.h>
> -#include "qm.h"
>  
>  #define HISI_ACC_SGL_SGE_NR_MIN		1
>  #define HISI_ACC_SGL_NR_MAX		256
> diff --git a/drivers/crypto/hisilicon/zip/zip.h b/drivers/crypto/hisilicon/zip/zip.h
> index 517fdbdff3ea..3dfd3bac5a33 100644
> --- a/drivers/crypto/hisilicon/zip/zip.h
> +++ b/drivers/crypto/hisilicon/zip/zip.h
> @@ -7,7 +7,7 @@
>  #define pr_fmt(fmt)	"hisi_zip: " fmt
>  
>  #include <linux/list.h>
> -#include "../qm.h"
> +#include <linux/hisi_acc_qm.h>
>  
>  enum hisi_zip_error_type {
>  	/* negative compression */
> diff --git a/drivers/crypto/hisilicon/qm.h b/include/linux/hisi_acc_qm.h
> similarity index 100%
> rename from drivers/crypto/hisilicon/qm.h
> rename to include/linux/hisi_acc_qm.h
> 
Acked-by: Longfang Liu <liulongfang@huawei.com>

Thanks,
Longfang

^ permalink raw reply	[flat|nested] 56+ messages in thread

* Re: [PATCH v8 3/9] hisi_acc_qm: Move VF PCI device IDs to common header
  2022-03-03 23:01 ` [PATCH v8 3/9] hisi_acc_qm: Move VF PCI device IDs " Shameer Kolothum
  2022-03-04  9:34   ` Zhou Wang
  2022-03-07 17:53   ` Alex Williamson
@ 2022-03-08  1:08   ` liulongfang
  2022-03-08 10:28   ` yekai(A)
  3 siblings, 0 replies; 56+ messages in thread
From: liulongfang @ 2022-03-08  1:08 UTC (permalink / raw)
  To: Shameer Kolothum, kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	linuxarm, prime.zeng, jonathan.cameron, wangzhou1

On 2022/3/4 7:01, Shameer Kolothum wrote:
> Move the PCI Device IDs of HiSilicon ACC VF devices to a common header
> and also use a uniform naming convention.
> 
> This will be useful when we introduce the vfio PCI HiSilicon ACC live
> migration driver in subsequent patches.
> 
> Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
> ---
>  drivers/crypto/hisilicon/hpre/hpre_main.c | 13 ++++++-------
>  drivers/crypto/hisilicon/sec2/sec_main.c  | 15 +++++++--------
>  drivers/crypto/hisilicon/zip/zip_main.c   | 11 +++++------
>  include/linux/pci_ids.h                   |  3 +++
>  4 files changed, 21 insertions(+), 21 deletions(-)
> 
> diff --git a/drivers/crypto/hisilicon/hpre/hpre_main.c b/drivers/crypto/hisilicon/hpre/hpre_main.c
> index ebfab3e14499..3589d8879b5e 100644
> --- a/drivers/crypto/hisilicon/hpre/hpre_main.c
> +++ b/drivers/crypto/hisilicon/hpre/hpre_main.c
> @@ -68,8 +68,7 @@
>  #define HPRE_REG_RD_INTVRL_US		10
>  #define HPRE_REG_RD_TMOUT_US		1000
>  #define HPRE_DBGFS_VAL_MAX_LEN		20
> -#define HPRE_PCI_DEVICE_ID		0xa258
> -#define HPRE_PCI_VF_DEVICE_ID		0xa259
> +#define PCI_DEVICE_ID_HUAWEI_HPRE_PF	0xa258
>  #define HPRE_QM_USR_CFG_MASK		GENMASK(31, 1)
>  #define HPRE_QM_AXI_CFG_MASK		GENMASK(15, 0)
>  #define HPRE_QM_VFG_AX_MASK		GENMASK(7, 0)
> @@ -111,8 +110,8 @@
>  static const char hpre_name[] = "hisi_hpre";
>  static struct dentry *hpre_debugfs_root;
>  static const struct pci_device_id hpre_dev_ids[] = {
> -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, HPRE_PCI_DEVICE_ID) },
> -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, HPRE_PCI_VF_DEVICE_ID) },
> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_HPRE_PF) },
> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_HPRE_VF) },
>  	{ 0, }
>  };
>  
> @@ -242,7 +241,7 @@ MODULE_PARM_DESC(uacce_mode, UACCE_MODE_DESC);
>  
>  static int pf_q_num_set(const char *val, const struct kernel_param *kp)
>  {
> -	return q_num_set(val, kp, HPRE_PCI_DEVICE_ID);
> +	return q_num_set(val, kp, PCI_DEVICE_ID_HUAWEI_HPRE_PF);
>  }
>  
>  static const struct kernel_param_ops hpre_pf_q_num_ops = {
> @@ -921,7 +920,7 @@ static int hpre_debugfs_init(struct hisi_qm *qm)
>  	qm->debug.sqe_mask_len = HPRE_SQE_MASK_LEN;
>  	hisi_qm_debug_init(qm);
>  
> -	if (qm->pdev->device == HPRE_PCI_DEVICE_ID) {
> +	if (qm->pdev->device == PCI_DEVICE_ID_HUAWEI_HPRE_PF) {
>  		ret = hpre_ctrl_debug_init(qm);
>  		if (ret)
>  			goto failed_to_create;
> @@ -958,7 +957,7 @@ static int hpre_qm_init(struct hisi_qm *qm, struct pci_dev *pdev)
>  	qm->sqe_size = HPRE_SQE_SIZE;
>  	qm->dev_name = hpre_name;
>  
> -	qm->fun_type = (pdev->device == HPRE_PCI_DEVICE_ID) ?
> +	qm->fun_type = (pdev->device == PCI_DEVICE_ID_HUAWEI_HPRE_PF) ?
>  			QM_HW_PF : QM_HW_VF;
>  	if (qm->fun_type == QM_HW_PF) {
>  		qm->qp_base = HPRE_PF_DEF_Q_BASE;
> diff --git a/drivers/crypto/hisilicon/sec2/sec_main.c b/drivers/crypto/hisilicon/sec2/sec_main.c
> index 26d3ab1d308b..311a8747b5bf 100644
> --- a/drivers/crypto/hisilicon/sec2/sec_main.c
> +++ b/drivers/crypto/hisilicon/sec2/sec_main.c
> @@ -20,8 +20,7 @@
>  
>  #define SEC_VF_NUM			63
>  #define SEC_QUEUE_NUM_V1		4096
> -#define SEC_PF_PCI_DEVICE_ID		0xa255
> -#define SEC_VF_PCI_DEVICE_ID		0xa256
> +#define PCI_DEVICE_ID_HUAWEI_SEC_PF	0xa255
>  
>  #define SEC_BD_ERR_CHK_EN0		0xEFFFFFFF
>  #define SEC_BD_ERR_CHK_EN1		0x7ffff7fd
> @@ -225,7 +224,7 @@ static const struct debugfs_reg32 sec_dfx_regs[] = {
>  
>  static int sec_pf_q_num_set(const char *val, const struct kernel_param *kp)
>  {
> -	return q_num_set(val, kp, SEC_PF_PCI_DEVICE_ID);
> +	return q_num_set(val, kp, PCI_DEVICE_ID_HUAWEI_SEC_PF);
>  }
>  
>  static const struct kernel_param_ops sec_pf_q_num_ops = {
> @@ -313,8 +312,8 @@ module_param_cb(uacce_mode, &sec_uacce_mode_ops, &uacce_mode, 0444);
>  MODULE_PARM_DESC(uacce_mode, UACCE_MODE_DESC);
>  
>  static const struct pci_device_id sec_dev_ids[] = {
> -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, SEC_PF_PCI_DEVICE_ID) },
> -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, SEC_VF_PCI_DEVICE_ID) },
> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_SEC_PF) },
> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_SEC_VF) },
>  	{ 0, }
>  };
>  MODULE_DEVICE_TABLE(pci, sec_dev_ids);
> @@ -717,7 +716,7 @@ static int sec_core_debug_init(struct hisi_qm *qm)
>  	regset->base = qm->io_base;
>  	regset->dev = dev;
>  
> -	if (qm->pdev->device == SEC_PF_PCI_DEVICE_ID)
> +	if (qm->pdev->device == PCI_DEVICE_ID_HUAWEI_SEC_PF)
>  		debugfs_create_file("regs", 0444, tmp_d, regset, &sec_regs_fops);
>  
>  	for (i = 0; i < ARRAY_SIZE(sec_dfx_labels); i++) {
> @@ -735,7 +734,7 @@ static int sec_debug_init(struct hisi_qm *qm)
>  	struct sec_dev *sec = container_of(qm, struct sec_dev, qm);
>  	int i;
>  
> -	if (qm->pdev->device == SEC_PF_PCI_DEVICE_ID) {
> +	if (qm->pdev->device == PCI_DEVICE_ID_HUAWEI_SEC_PF) {
>  		for (i = SEC_CLEAR_ENABLE; i < SEC_DEBUG_FILE_NUM; i++) {
>  			spin_lock_init(&sec->debug.files[i].lock);
>  			sec->debug.files[i].index = i;
> @@ -877,7 +876,7 @@ static int sec_qm_init(struct hisi_qm *qm, struct pci_dev *pdev)
>  	qm->sqe_size = SEC_SQE_SIZE;
>  	qm->dev_name = sec_name;
>  
> -	qm->fun_type = (pdev->device == SEC_PF_PCI_DEVICE_ID) ?
> +	qm->fun_type = (pdev->device == PCI_DEVICE_ID_HUAWEI_SEC_PF) ?
>  			QM_HW_PF : QM_HW_VF;
>  	if (qm->fun_type == QM_HW_PF) {
>  		qm->qp_base = SEC_PF_DEF_Q_BASE;
> diff --git a/drivers/crypto/hisilicon/zip/zip_main.c b/drivers/crypto/hisilicon/zip/zip_main.c
> index 678f8b58ec42..66decfe07282 100644
> --- a/drivers/crypto/hisilicon/zip/zip_main.c
> +++ b/drivers/crypto/hisilicon/zip/zip_main.c
> @@ -15,8 +15,7 @@
>  #include <linux/uacce.h>
>  #include "zip.h"
>  
> -#define PCI_DEVICE_ID_ZIP_PF		0xa250
> -#define PCI_DEVICE_ID_ZIP_VF		0xa251
> +#define PCI_DEVICE_ID_HUAWEI_ZIP_PF	0xa250
>  
>  #define HZIP_QUEUE_NUM_V1		4096
>  
> @@ -246,7 +245,7 @@ MODULE_PARM_DESC(uacce_mode, UACCE_MODE_DESC);
>  
>  static int pf_q_num_set(const char *val, const struct kernel_param *kp)
>  {
> -	return q_num_set(val, kp, PCI_DEVICE_ID_ZIP_PF);
> +	return q_num_set(val, kp, PCI_DEVICE_ID_HUAWEI_ZIP_PF);
>  }
>  
>  static const struct kernel_param_ops pf_q_num_ops = {
> @@ -268,8 +267,8 @@ module_param_cb(vfs_num, &vfs_num_ops, &vfs_num, 0444);
>  MODULE_PARM_DESC(vfs_num, "Number of VFs to enable(1-63), 0(default)");
>  
>  static const struct pci_device_id hisi_zip_dev_ids[] = {
> -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_ZIP_PF) },
> -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_ZIP_VF) },
> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_ZIP_PF) },
> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_ZIP_VF) },
>  	{ 0, }
>  };
>  MODULE_DEVICE_TABLE(pci, hisi_zip_dev_ids);
> @@ -838,7 +837,7 @@ static int hisi_zip_qm_init(struct hisi_qm *qm, struct pci_dev *pdev)
>  	qm->sqe_size = HZIP_SQE_SIZE;
>  	qm->dev_name = hisi_zip_name;
>  
> -	qm->fun_type = (pdev->device == PCI_DEVICE_ID_ZIP_PF) ?
> +	qm->fun_type = (pdev->device == PCI_DEVICE_ID_HUAWEI_ZIP_PF) ?
>  			QM_HW_PF : QM_HW_VF;
>  	if (qm->fun_type == QM_HW_PF) {
>  		qm->qp_base = HZIP_PF_DEF_Q_BASE;
> diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
> index aad54c666407..31dee2b65a62 100644
> --- a/include/linux/pci_ids.h
> +++ b/include/linux/pci_ids.h
> @@ -2529,6 +2529,9 @@
>  #define PCI_DEVICE_ID_KORENIX_JETCARDF3	0x17ff
>  
>  #define PCI_VENDOR_ID_HUAWEI		0x19e5
> +#define PCI_DEVICE_ID_HUAWEI_ZIP_VF	0xa251
> +#define PCI_DEVICE_ID_HUAWEI_SEC_VF	0xa256
> +#define PCI_DEVICE_ID_HUAWEI_HPRE_VF	0xa259
>  
>  #define PCI_VENDOR_ID_NETRONOME		0x19ee
>  #define PCI_DEVICE_ID_NETRONOME_NFP4000	0x4000
> 
Acked-by: Longfang Liu <liulongfang@huawei.com>

Thanks,
Longfang

^ permalink raw reply	[flat|nested] 56+ messages in thread

* Re: [PATCH v8 5/9] hisi_acc_vfio_pci: Restrict access to VF dev BAR2 migration region
  2022-03-03 23:01 ` [PATCH v8 5/9] hisi_acc_vfio_pci: Restrict access to VF dev BAR2 migration region Shameer Kolothum
@ 2022-03-08  1:11   ` liulongfang
  2022-03-08  6:23   ` Tian, Kevin
  1 sibling, 0 replies; 56+ messages in thread
From: liulongfang @ 2022-03-08  1:11 UTC (permalink / raw)
  To: Shameer Kolothum, kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	linuxarm, prime.zeng, jonathan.cameron, wangzhou1

On 2022/3/4 7:01, Shameer Kolothum wrote:
> HiSilicon ACC VF device BAR2 region consists of both functional
> register space and migration control register space. From a
> security point of view, it's not advisable to export the migration
> control region to Guest.
> 
> Hence, introduce a separate struct vfio_device_ops for migration
> support which will override the ioctl/read/write/mmap methods to
> hide the migration region and limit the access only to the
> functional register space.
> 
> This will be used in subsequent patches when we add migration
> support to the driver.
> 
> Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
Reviewed-by: Longfang Liu <liulongfang@huawei.com>

Thanks,
Longfang
> ---
>  .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    | 126 ++++++++++++++++++
>  1 file changed, 126 insertions(+)
> 
> diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
> index 8129c3457b3b..582ee4fa4109 100644
> --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
> +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
> @@ -13,6 +13,119 @@
>  #include <linux/vfio.h>
>  #include <linux/vfio_pci_core.h>
>  
> +static int hisi_acc_pci_rw_access_check(struct vfio_device *core_vdev,
> +					size_t count, loff_t *ppos,
> +					size_t *new_count)
> +{
> +	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
> +	struct vfio_pci_core_device *vdev =
> +		container_of(core_vdev, struct vfio_pci_core_device, vdev);
> +
> +	if (index == VFIO_PCI_BAR2_REGION_INDEX) {
> +		loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK;
> +		resource_size_t end = pci_resource_len(vdev->pdev, index) / 2;
> +
> +		/* Check if access is for migration control region */
> +		if (pos >= end)
> +			return -EINVAL;
> +
> +		*new_count = min(count, (size_t)(end - pos));
> +	}
> +
> +	return 0;
> +}
> +
> +static int hisi_acc_vfio_pci_mmap(struct vfio_device *core_vdev,
> +				  struct vm_area_struct *vma)
> +{
> +	struct vfio_pci_core_device *vdev =
> +		container_of(core_vdev, struct vfio_pci_core_device, vdev);
> +	unsigned int index;
> +
> +	index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT);
> +	if (index == VFIO_PCI_BAR2_REGION_INDEX) {
> +		u64 req_len, pgoff, req_start;
> +		resource_size_t end = pci_resource_len(vdev->pdev, index) / 2;
> +
> +		req_len = vma->vm_end - vma->vm_start;
> +		pgoff = vma->vm_pgoff &
> +			((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1);
> +		req_start = pgoff << PAGE_SHIFT;
> +
> +		if (req_start + req_len > end)
> +			return -EINVAL;
> +	}
> +
> +	return vfio_pci_core_mmap(core_vdev, vma);
> +}
> +
> +static ssize_t hisi_acc_vfio_pci_write(struct vfio_device *core_vdev,
> +				       const char __user *buf, size_t count,
> +				       loff_t *ppos)
> +{
> +	size_t new_count = count;
> +	int ret;
> +
> +	ret = hisi_acc_pci_rw_access_check(core_vdev, count, ppos, &new_count);
> +	if (ret)
> +		return ret;
> +
> +	return vfio_pci_core_write(core_vdev, buf, new_count, ppos);
> +}
> +
> +static ssize_t hisi_acc_vfio_pci_read(struct vfio_device *core_vdev,
> +				      char __user *buf, size_t count,
> +				      loff_t *ppos)
> +{
> +	size_t new_count = count;
> +	int ret;
> +
> +	ret = hisi_acc_pci_rw_access_check(core_vdev, count, ppos, &new_count);
> +	if (ret)
> +		return ret;
> +
> +	return vfio_pci_core_read(core_vdev, buf, new_count, ppos);
> +}
> +
> +static long hisi_acc_vfio_pci_ioctl(struct vfio_device *core_vdev, unsigned int cmd,
> +				    unsigned long arg)
> +{
> +	if (cmd == VFIO_DEVICE_GET_REGION_INFO) {
> +		struct vfio_pci_core_device *vdev =
> +			container_of(core_vdev, struct vfio_pci_core_device, vdev);
> +		struct pci_dev *pdev = vdev->pdev;
> +		struct vfio_region_info info;
> +		unsigned long minsz;
> +
> +		minsz = offsetofend(struct vfio_region_info, offset);
> +
> +		if (copy_from_user(&info, (void __user *)arg, minsz))
> +			return -EFAULT;
> +
> +		if (info.argsz < minsz)
> +			return -EINVAL;
> +
> +		if (info.index == VFIO_PCI_BAR2_REGION_INDEX) {
> +			info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);
> +
> +			/*
> +			 * ACC VF dev BAR2 region consists of both functional
> +			 * register space and migration control register space.
> +			 * Report only the functional region to Guest.
> +			 */
> +			info.size = pci_resource_len(pdev, info.index) / 2;
> +
> +			info.flags = VFIO_REGION_INFO_FLAG_READ |
> +					VFIO_REGION_INFO_FLAG_WRITE |
> +					VFIO_REGION_INFO_FLAG_MMAP;
> +
> +			return copy_to_user((void __user *)arg, &info, minsz) ?
> +					    -EFAULT : 0;
> +		}
> +	}
> +	return vfio_pci_core_ioctl(core_vdev, cmd, arg);
> +}
> +
>  static int hisi_acc_vfio_pci_open_device(struct vfio_device *core_vdev)
>  {
>  	struct vfio_pci_core_device *vdev =
> @@ -28,6 +141,19 @@ static int hisi_acc_vfio_pci_open_device(struct vfio_device *core_vdev)
>  	return 0;
>  }
>  
> +static const struct vfio_device_ops hisi_acc_vfio_pci_migrn_ops = {
> +	.name = "hisi-acc-vfio-pci-migration",
> +	.open_device = hisi_acc_vfio_pci_open_device,
> +	.close_device = vfio_pci_core_close_device,
> +	.ioctl = hisi_acc_vfio_pci_ioctl,
> +	.device_feature = vfio_pci_core_ioctl_feature,
> +	.read = hisi_acc_vfio_pci_read,
> +	.write = hisi_acc_vfio_pci_write,
> +	.mmap = hisi_acc_vfio_pci_mmap,
> +	.request = vfio_pci_core_request,
> +	.match = vfio_pci_core_match,
> +};
> +
>  static const struct vfio_device_ops hisi_acc_vfio_pci_ops = {
>  	.name = "hisi-acc-vfio-pci",
>  	.open_device = hisi_acc_vfio_pci_open_device,
> 

^ permalink raw reply	[flat|nested] 56+ messages in thread

* Re: [PATCH v8 9/9] hisi_acc_vfio_pci: Use its own PCI reset_done error handler
  2022-03-03 23:01 ` [PATCH v8 9/9] hisi_acc_vfio_pci: Use its own PCI reset_done error handler Shameer Kolothum
  2022-03-04 20:54   ` Jason Gunthorpe
@ 2022-03-08  1:14   ` liulongfang
  1 sibling, 0 replies; 56+ messages in thread
From: liulongfang @ 2022-03-08  1:14 UTC (permalink / raw)
  To: Shameer Kolothum, kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	linuxarm, prime.zeng, jonathan.cameron, wangzhou1

On 2022/3/4 7:01, Shameer Kolothum wrote:
> Register private handler for pci_error_handlers.reset_done and update
> state accordingly.
> 
> Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
> ---
>  .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    | 57 ++++++++++++++++++-
>  .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h    |  4 +-
>  2 files changed, 57 insertions(+), 4 deletions(-)
> 
Reviewed-by: Longfang Liu <liulongfang@huawei.com>

Thanks,
Longfang
> diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
> index aa2e4b6bf598..53e4c5cb3a71 100644
> --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
> +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
> @@ -625,6 +625,27 @@ static void hisi_acc_vf_disable_fds(struct hisi_acc_vf_core_device *hisi_acc_vde
>  	}
>  }
>  
> +/*
> + * This function is called in all state_mutex unlock cases to
> + * handle a 'deferred_reset' if exists.
> + */
> +static void
> +hisi_acc_vf_state_mutex_unlock(struct hisi_acc_vf_core_device *hisi_acc_vdev)
> +{
> +again:
> +	spin_lock(&hisi_acc_vdev->reset_lock);
> +	if (hisi_acc_vdev->deferred_reset) {
> +		hisi_acc_vdev->deferred_reset = false;
> +		spin_unlock(&hisi_acc_vdev->reset_lock);
> +		hisi_acc_vdev->vf_qm_state = QM_NOT_READY;
> +		hisi_acc_vdev->mig_state = VFIO_DEVICE_STATE_RUNNING;
> +		hisi_acc_vf_disable_fds(hisi_acc_vdev);
> +		goto again;
> +	}
> +	mutex_unlock(&hisi_acc_vdev->state_mutex);
> +	spin_unlock(&hisi_acc_vdev->reset_lock);
> +}
> +
>  static void hisi_acc_vf_start_device(struct hisi_acc_vf_core_device *hisi_acc_vdev)
>  {
>  	struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
> @@ -921,7 +942,7 @@ hisi_acc_vfio_pci_set_device_state(struct vfio_device *vdev,
>  			break;
>  		}
>  	}
> -	mutex_unlock(&hisi_acc_vdev->state_mutex);
> +	hisi_acc_vf_state_mutex_unlock(hisi_acc_vdev);
>  	return res;
>  }
>  
> @@ -934,10 +955,35 @@ hisi_acc_vfio_pci_get_device_state(struct vfio_device *vdev,
>  
>  	mutex_lock(&hisi_acc_vdev->state_mutex);
>  	*curr_state = hisi_acc_vdev->mig_state;
> -	mutex_unlock(&hisi_acc_vdev->state_mutex);
> +	hisi_acc_vf_state_mutex_unlock(hisi_acc_vdev);
>  	return 0;
>  }
>  
> +static void hisi_acc_vf_pci_aer_reset_done(struct pci_dev *pdev)
> +{
> +	struct hisi_acc_vf_core_device *hisi_acc_vdev = dev_get_drvdata(&pdev->dev);
> +
> +	if (hisi_acc_vdev->core_device.vdev.migration_flags !=
> +				VFIO_MIGRATION_STOP_COPY)
> +		return;
> +
> +	/*
> +	 * As the higher VFIO layers are holding locks across reset and using
> +	 * those same locks with the mm_lock we need to prevent ABBA deadlock
> +	 * with the state_mutex and mm_lock.
> +	 * In case the state_mutex was taken already we defer the cleanup work
> +	 * to the unlock flow of the other running context.
> +	 */
> +	spin_lock(&hisi_acc_vdev->reset_lock);
> +	hisi_acc_vdev->deferred_reset = true;
> +	if (!mutex_trylock(&hisi_acc_vdev->state_mutex)) {
> +		spin_unlock(&hisi_acc_vdev->reset_lock);
> +		return;
> +	}
> +	spin_unlock(&hisi_acc_vdev->reset_lock);
> +	hisi_acc_vf_state_mutex_unlock(hisi_acc_vdev);
> +}
> +
>  static int hisi_acc_vf_qm_init(struct hisi_acc_vf_core_device *hisi_acc_vdev)
>  {
>  	struct vfio_pci_core_device *vdev = &hisi_acc_vdev->core_device;
> @@ -1252,12 +1298,17 @@ static const struct pci_device_id hisi_acc_vfio_pci_table[] = {
>  
>  MODULE_DEVICE_TABLE(pci, hisi_acc_vfio_pci_table);
>  
> +static const struct pci_error_handlers hisi_acc_vf_err_handlers = {
> +	.reset_done = hisi_acc_vf_pci_aer_reset_done,
> +	.error_detected = vfio_pci_core_aer_err_detected,
> +};
> +
>  static struct pci_driver hisi_acc_vfio_pci_driver = {
>  	.name = KBUILD_MODNAME,
>  	.id_table = hisi_acc_vfio_pci_table,
>  	.probe = hisi_acc_vfio_pci_probe,
>  	.remove = hisi_acc_vfio_pci_remove,
> -	.err_handler = &vfio_pci_core_err_handlers,
> +	.err_handler = &hisi_acc_vf_err_handlers,
>  };
>  
>  module_pci_driver(hisi_acc_vfio_pci_driver);
> diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
> index 1c7d75408790..5494f4983bbe 100644
> --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
> +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
> @@ -98,6 +98,7 @@ struct hisi_acc_vf_migration_file {
>  
>  struct hisi_acc_vf_core_device {
>  	struct vfio_pci_core_device core_device;
> +	u8 deferred_reset:1;
>  	/* for migration state */
>  	struct mutex state_mutex;
>  	enum vfio_device_mig_state mig_state;
> @@ -107,7 +108,8 @@ struct hisi_acc_vf_core_device {
>  	struct hisi_qm vf_qm;
>  	u32 vf_qm_state;
>  	int vf_id;
> -
> +	/* for reset handler */
> +	spinlock_t reset_lock;
>  	struct hisi_acc_vf_migration_file *resuming_migf;
>  	struct hisi_acc_vf_migration_file *saving_migf;
>  };
> 

^ permalink raw reply	[flat|nested] 56+ messages in thread

* RE: [PATCH v8 5/9] hisi_acc_vfio_pci: Restrict access to VF dev BAR2 migration region
  2022-03-03 23:01 ` [PATCH v8 5/9] hisi_acc_vfio_pci: Restrict access to VF dev BAR2 migration region Shameer Kolothum
  2022-03-08  1:11   ` liulongfang
@ 2022-03-08  6:23   ` Tian, Kevin
  2022-03-08  8:33     ` Shameerali Kolothum Thodi
  1 sibling, 1 reply; 56+ messages in thread
From: Tian, Kevin @ 2022-03-08  6:23 UTC (permalink / raw)
  To: Shameer Kolothum, kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	linuxarm, liulongfang, prime.zeng, jonathan.cameron, wangzhou1

Hi, Shameer,

> From: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
> Sent: Friday, March 4, 2022 7:01 AM
> 
> HiSilicon ACC VF device BAR2 region consists of both functional
> register space and migration control register space. From a
> security point of view, it's not advisable to export the migration
> control region to Guest.
> 
> Hence, introduce a separate struct vfio_device_ops for migration
> support which will override the ioctl/read/write/mmap methods to
> hide the migration region and limit the access only to the
> functional register space.
> 
> This will be used in subsequent patches when we add migration
> support to the driver.

As a security concern the migration control region should be always
disabled regardless of whether migration support is added to the
driver for such device... It sounds like we should first fix this security
hole for acc device assignment and then add the migration support
atop (at least organize the series in this way).

Thanks
Kevin

^ permalink raw reply	[flat|nested] 56+ messages in thread

* RE: [PATCH v8 7/9] crypto: hisilicon/qm: Set the VF QM state register
  2022-03-03 23:01 ` [PATCH v8 7/9] crypto: hisilicon/qm: Set the VF QM state register Shameer Kolothum
  2022-03-04  9:43   ` Zhou Wang
@ 2022-03-08  6:31   ` Tian, Kevin
  2022-03-08  8:46     ` Shameerali Kolothum Thodi
  1 sibling, 1 reply; 56+ messages in thread
From: Tian, Kevin @ 2022-03-08  6:31 UTC (permalink / raw)
  To: Shameer Kolothum, kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	linuxarm, liulongfang, prime.zeng, jonathan.cameron, wangzhou1

> From: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
> Sent: Friday, March 4, 2022 7:01 AM
> 
> From: Longfang Liu <liulongfang@huawei.com>
> 
> We use VF QM state register to record the status of the QM configuration
> state. This will be used in the ACC migration driver to determine whether
> we can safely save and restore the QM data.

Can you say something about what QM is and how it is related to the VF state
to be migrated? It might be obvious to acc driver people but not so to to 
others in vfio community like me. 😊

Thanks
Kevin

^ permalink raw reply	[flat|nested] 56+ messages in thread

* RE: [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live migration
  2022-03-03 23:01 ` [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live migration Shameer Kolothum
  2022-03-04  8:48   ` Shameerali Kolothum Thodi
  2022-03-04 20:57   ` Jason Gunthorpe
@ 2022-03-08  7:41   ` Tian, Kevin
  2022-03-08  8:52     ` Shameerali Kolothum Thodi
  2 siblings, 1 reply; 56+ messages in thread
From: Tian, Kevin @ 2022-03-08  7:41 UTC (permalink / raw)
  To: Shameer Kolothum, kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	linuxarm, liulongfang, prime.zeng, jonathan.cameron, wangzhou1

> From: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
> Sent: Friday, March 4, 2022 7:02 AM
> +/*
> + * Each state Reg is checked 100 times,
> + * with a delay of 100 microseconds after each check
> + */
> +static u32 acc_check_reg_state(struct hisi_qm *qm, u32 regs)

qm_check_reg_state() given the 1st argument is qm

> +/* Check the PF's RAS state and Function INT state */
> +static int qm_check_int_state(struct hisi_acc_vf_core_device *hisi_acc_vdev)

then this should be acc_check_int_state() given the input is an acc device?

anyway please have a consistent naming convention here.

> +static int qm_read_reg(struct hisi_qm *qm, u32 reg_addr,
> +		       u32 *data, u8 nums)

qm_read_regs() to reflect that multiple registers are processed.

> +
> +static int qm_write_reg(struct hisi_qm *qm, u32 reg,
> +			u32 *data, u8 nums)

qm_write_regs()

> +
> +static int qm_rw_regs_read(struct hisi_qm *qm, struct acc_vf_data *vf_data)

qm_load_regs(). It's confusing to have both 'rw' and 'read'.

> +
> +static int qm_rw_regs_write(struct hisi_qm *qm, struct acc_vf_data
> *vf_data)

qm_save_regs()

> +static int hisi_acc_vf_qm_init(struct hisi_acc_vf_core_device *hisi_acc_vdev)
> +{
> +	struct vfio_pci_core_device *vdev = &hisi_acc_vdev->core_device;
> +	struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
> +	struct pci_dev *vf_dev = vdev->pdev;
> +
> +	/*
> +	 * ACC VF dev BAR2 region consists of both functional register space
> +	 * and migration control register space. For migration to work, we
> +	 * need access to both. Hence, we map the entire BAR2 region here.
> +	 * But from a security point of view, we restrict access to the
> +	 * migration control space from Guest(Please see
> mmap/ioctl/read/write
> +	 * override functions).

(Please see hisi_acc_vfio_pci_migrn_ops)

> +	 *
> +	 * Also the HiSilicon ACC VF devices supported by this driver on
> +	 * HiSilicon hardware platforms are integrated end point devices
> +	 * and has no capability to perform PCIe P2P.

According to v5 discussion I think it is the platform which lacks of the
P2P capability instead of the device. Current writing is read to the latter.

better clarify it accurately. 😊

>  static int hisi_acc_vfio_pci_probe(struct pci_dev *pdev, const struct
> pci_device_id *id)
>  {
> -	struct vfio_pci_core_device *vdev;
> +	struct hisi_acc_vf_core_device *hisi_acc_vdev;
> +	struct hisi_qm *pf_qm;
>  	int ret;
> 
> -	vdev = kzalloc(sizeof(*vdev), GFP_KERNEL);
> -	if (!vdev)
> +	hisi_acc_vdev = kzalloc(sizeof(*hisi_acc_vdev), GFP_KERNEL);
> +	if (!hisi_acc_vdev)
>  		return -ENOMEM;
> 
> -	vfio_pci_core_init_device(vdev, pdev, &hisi_acc_vfio_pci_ops);
> +	pf_qm = hisi_acc_get_pf_qm(pdev);
> +	if (pf_qm && pf_qm->ver >= QM_HW_V3) {
> +		ret = hisi_acc_vfio_pci_migrn_init(hisi_acc_vdev, pdev,
> pf_qm);
> +		if (!ret) {
> +			vfio_pci_core_init_device(&hisi_acc_vdev-
> >core_device, pdev,
> +
> &hisi_acc_vfio_pci_migrn_ops);
> +		} else {
> +			pci_warn(pdev, "migration support failed, continue
> with generic interface\n");
> +			vfio_pci_core_init_device(&hisi_acc_vdev-
> >core_device, pdev,
> +						  &hisi_acc_vfio_pci_ops);
> +		}

This logic looks weird. Earlier you state that the migration control 
region must be hidden from the userspace as a security requirement,
but above logic reads like if the driver fails to initialize migration 
support then we just fall back to the default ops which grants the
user the full access to the entire MMIO bar.

> +	} else {
> +		vfio_pci_core_init_device(&hisi_acc_vdev->core_device, pdev,
> +					  &hisi_acc_vfio_pci_ops);
> +	}

If the hardware itself doesn't support the migration capability, can we
just move it out of the id table and let vfio-pci to drive it?

Thanks
Kevin

^ permalink raw reply	[flat|nested] 56+ messages in thread

* RE: [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live migration
  2022-03-07 19:52         ` Alex Williamson
@ 2022-03-08  8:11           ` Tian, Kevin
  2022-03-08 19:33             ` Alex Williamson
  2022-03-08  9:46           ` liulongfang
  1 sibling, 1 reply; 56+ messages in thread
From: Tian, Kevin @ 2022-03-08  8:11 UTC (permalink / raw)
  To: Alex Williamson, Shameerali Kolothum Thodi
  Cc: Jason Gunthorpe, kvm, linux-kernel, linux-crypto, linux-pci,
	cohuck, mgurtovoy, yishaih, Linuxarm, liulongfang, Zengtao (B),
	Jonathan Cameron, Wangzhou (B),
	Xu Zaibo

> From: Alex Williamson <alex.williamson@redhat.com>
> Sent: Tuesday, March 8, 2022 3:53 AM
> >
> > > I think we still require acks from Bjorn and Zaibo for select patches
> > > in this series.
> >
> > I checked with Ziabo. He moved projects and is no longer looking into
> crypto stuff.
> > Wangzhou and LiuLongfang now take care of this. Received acks from
> Wangzhou
> > already and I will request Longfang to provide his. Hope that's ok.
> 
> Maybe a good time to have them update MAINTAINERS as well.  Thanks,
> 

I have one question here (similar to what we discussed for mdev before).

Now we are adding vendor specific drivers under /drivers/vfio. Two drivers
on radar and more will come. Then what would be the criteria for 
accepting such a driver? Do we prefer to a model in which the author should
provide enough background for vfio community to understand how it works 
or as done here just rely on the PF driver owner to cover device specific
code?

If the former we may need document some process for what information
is necessary and also need secure increased review bandwidth from key
reviewers in vfio community.

If the latter then how can we guarantee no corner case overlooked by both
sides (i.e. how to know the coverage of total reviews)? Another open is who
from the PF driver sub-system should be considered as the one to give the
green signal. If the sub-system maintainer trusts the PF driver owner and
just pulls commits from him then having the r-b from the PF driver owner is
sufficient. But if the sub-system maintainer wants to review detail change
in every underlying driver then we probably also want to get the ack from
the maintainer.

Overall I didn't mean to slow down the progress of this series. But above
does be some puzzle occurred in my review. 😊

Thanks
Kevin

^ permalink raw reply	[flat|nested] 56+ messages in thread

* RE: [PATCH v8 5/9] hisi_acc_vfio_pci: Restrict access to VF dev BAR2 migration region
  2022-03-08  6:23   ` Tian, Kevin
@ 2022-03-08  8:33     ` Shameerali Kolothum Thodi
  2022-03-08 10:09       ` Tian, Kevin
  0 siblings, 1 reply; 56+ messages in thread
From: Shameerali Kolothum Thodi @ 2022-03-08  8:33 UTC (permalink / raw)
  To: Tian, Kevin, kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	Linuxarm, liulongfang, Zengtao (B),
	Jonathan Cameron, Wangzhou (B)

Hi Kevin,

> -----Original Message-----
> From: Tian, Kevin [mailto:kevin.tian@intel.com]
> Sent: 08 March 2022 06:23
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>;
> kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-crypto@vger.kernel.org
> Cc: linux-pci@vger.kernel.org; alex.williamson@redhat.com; jgg@nvidia.com;
> cohuck@redhat.com; mgurtovoy@nvidia.com; yishaih@nvidia.com; Linuxarm
> <linuxarm@huawei.com>; liulongfang <liulongfang@huawei.com>; Zengtao (B)
> <prime.zeng@hisilicon.com>; Jonathan Cameron
> <jonathan.cameron@huawei.com>; Wangzhou (B) <wangzhou1@hisilicon.com>
> Subject: RE: [PATCH v8 5/9] hisi_acc_vfio_pci: Restrict access to VF dev BAR2
> migration region
> 
> Hi, Shameer,
> 
> > From: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
> > Sent: Friday, March 4, 2022 7:01 AM
> >
> > HiSilicon ACC VF device BAR2 region consists of both functional
> > register space and migration control register space. From a
> > security point of view, it's not advisable to export the migration
> > control region to Guest.
> >
> > Hence, introduce a separate struct vfio_device_ops for migration
> > support which will override the ioctl/read/write/mmap methods to
> > hide the migration region and limit the access only to the
> > functional register space.
> >
> > This will be used in subsequent patches when we add migration
> > support to the driver.
> 
> As a security concern the migration control region should be always
> disabled regardless of whether migration support is added to the
> driver for such device... It sounds like we should first fix this security
> hole for acc device assignment and then add the migration support
> atop (at least organize the series in this way).

By exposing the migration BAR region, there is a possibility that a malicious
Guest can prevent migration from happening by manipulating the migration
BAR region. I don't think there are any other security concerns now especially
since we only support the STOP_COPY state.  And the approach has been that
we only restrict this if migration support is enabled. I think I can change the
above "security concern" description to "malicious Guest can prevent migration"
to make it more clear.

Hope this is fine.

Thanks,
Shameer

^ permalink raw reply	[flat|nested] 56+ messages in thread

* RE: [PATCH v8 7/9] crypto: hisilicon/qm: Set the VF QM state register
  2022-03-08  6:31   ` Tian, Kevin
@ 2022-03-08  8:46     ` Shameerali Kolothum Thodi
  2022-03-08 10:10       ` Tian, Kevin
  0 siblings, 1 reply; 56+ messages in thread
From: Shameerali Kolothum Thodi @ 2022-03-08  8:46 UTC (permalink / raw)
  To: Tian, Kevin, kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	Linuxarm, liulongfang, Zengtao (B),
	Jonathan Cameron, Wangzhou (B)



> -----Original Message-----
> From: Tian, Kevin [mailto:kevin.tian@intel.com]
> Sent: 08 March 2022 06:31
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>;
> kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-crypto@vger.kernel.org
> Cc: linux-pci@vger.kernel.org; alex.williamson@redhat.com; jgg@nvidia.com;
> cohuck@redhat.com; mgurtovoy@nvidia.com; yishaih@nvidia.com; Linuxarm
> <linuxarm@huawei.com>; liulongfang <liulongfang@huawei.com>; Zengtao (B)
> <prime.zeng@hisilicon.com>; Jonathan Cameron
> <jonathan.cameron@huawei.com>; Wangzhou (B) <wangzhou1@hisilicon.com>
> Subject: RE: [PATCH v8 7/9] crypto: hisilicon/qm: Set the VF QM state register
> 
> > From: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
> > Sent: Friday, March 4, 2022 7:01 AM
> >
> > From: Longfang Liu <liulongfang@huawei.com>
> >
> > We use VF QM state register to record the status of the QM configuration
> > state. This will be used in the ACC migration driver to determine whether
> > we can safely save and restore the QM data.
> 
> Can you say something about what QM is and how it is related to the VF state
> to be migrated? It might be obvious to acc driver people but not so to to
> others in vfio community like me. 😊

Ok, understand that. I will lift a description from the QM driver patch :).

QM stands for Queue Management which is a generic IP used by ACC devices.
It provides a general PCIe interface for the CPU and the ACC devices to share
a group of queues.

QM integrated into an accelerator provides queue management service.
Queues can be assigned to PF and VFs, and queues can be controlled by
unified mailboxes and doorbells. The QM driver(drivers/crypto/hisilicon/qm.c)
provides generic interfaces to ACC drivers to manage the QM.

I will add some of this in either this commit msg or in the actual file somewhere.

Thanks,
Shameer

^ permalink raw reply	[flat|nested] 56+ messages in thread

* RE: [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live migration
  2022-03-08  7:41   ` Tian, Kevin
@ 2022-03-08  8:52     ` Shameerali Kolothum Thodi
  2022-03-08 10:17       ` Tian, Kevin
  0 siblings, 1 reply; 56+ messages in thread
From: Shameerali Kolothum Thodi @ 2022-03-08  8:52 UTC (permalink / raw)
  To: Tian, Kevin, kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	Linuxarm, liulongfang, Zengtao (B),
	Jonathan Cameron, Wangzhou (B)



> -----Original Message-----
> From: Tian, Kevin [mailto:kevin.tian@intel.com]
> Sent: 08 March 2022 07:42
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>;
> kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-crypto@vger.kernel.org
> Cc: linux-pci@vger.kernel.org; alex.williamson@redhat.com; jgg@nvidia.com;
> cohuck@redhat.com; mgurtovoy@nvidia.com; yishaih@nvidia.com; Linuxarm
> <linuxarm@huawei.com>; liulongfang <liulongfang@huawei.com>; Zengtao (B)
> <prime.zeng@hisilicon.com>; Jonathan Cameron
> <jonathan.cameron@huawei.com>; Wangzhou (B) <wangzhou1@hisilicon.com>
> Subject: RE: [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live
> migration
> 
> > From: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
> > Sent: Friday, March 4, 2022 7:02 AM
> > +/*
> > + * Each state Reg is checked 100 times,
> > + * with a delay of 100 microseconds after each check  */ static u32
> > +acc_check_reg_state(struct hisi_qm *qm, u32 regs)
> 
> qm_check_reg_state() given the 1st argument is qm
> 
> > +/* Check the PF's RAS state and Function INT state */ static int
> > +qm_check_int_state(struct hisi_acc_vf_core_device *hisi_acc_vdev)
> 
> then this should be acc_check_int_state() given the input is an acc device?
> 
> anyway please have a consistent naming convention here.
> 
> > +static int qm_read_reg(struct hisi_qm *qm, u32 reg_addr,
> > +		       u32 *data, u8 nums)
> 
> qm_read_regs() to reflect that multiple registers are processed.
> 
> > +
> > +static int qm_write_reg(struct hisi_qm *qm, u32 reg,
> > +			u32 *data, u8 nums)
> 
> qm_write_regs()
> 
> > +
> > +static int qm_rw_regs_read(struct hisi_qm *qm, struct acc_vf_data
> > +*vf_data)
> 
> qm_load_regs(). It's confusing to have both 'rw' and 'read'.
> 
> > +
> > +static int qm_rw_regs_write(struct hisi_qm *qm, struct acc_vf_data
> > *vf_data)
> 
> qm_save_regs()

Right. I am Ok with the above suggestions.

> 
> > +static int hisi_acc_vf_qm_init(struct hisi_acc_vf_core_device
> > +*hisi_acc_vdev) {
> > +	struct vfio_pci_core_device *vdev = &hisi_acc_vdev->core_device;
> > +	struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
> > +	struct pci_dev *vf_dev = vdev->pdev;
> > +
> > +	/*
> > +	 * ACC VF dev BAR2 region consists of both functional register space
> > +	 * and migration control register space. For migration to work, we
> > +	 * need access to both. Hence, we map the entire BAR2 region here.
> > +	 * But from a security point of view, we restrict access to the
> > +	 * migration control space from Guest(Please see
> > mmap/ioctl/read/write
> > +	 * override functions).
> 
> (Please see hisi_acc_vfio_pci_migrn_ops)
> 
> > +	 *
> > +	 * Also the HiSilicon ACC VF devices supported by this driver on
> > +	 * HiSilicon hardware platforms are integrated end point devices
> > +	 * and has no capability to perform PCIe P2P.
> 
> According to v5 discussion I think it is the platform which lacks of the P2P
> capability instead of the device. Current writing is read to the latter.
> 
> better clarify it accurately. 😊

That’s right. It is the platform.

> 
> >  static int hisi_acc_vfio_pci_probe(struct pci_dev *pdev, const struct
> > pci_device_id *id)  {
> > -	struct vfio_pci_core_device *vdev;
> > +	struct hisi_acc_vf_core_device *hisi_acc_vdev;
> > +	struct hisi_qm *pf_qm;
> >  	int ret;
> >
> > -	vdev = kzalloc(sizeof(*vdev), GFP_KERNEL);
> > -	if (!vdev)
> > +	hisi_acc_vdev = kzalloc(sizeof(*hisi_acc_vdev), GFP_KERNEL);
> > +	if (!hisi_acc_vdev)
> >  		return -ENOMEM;
> >
> > -	vfio_pci_core_init_device(vdev, pdev, &hisi_acc_vfio_pci_ops);
> > +	pf_qm = hisi_acc_get_pf_qm(pdev);
> > +	if (pf_qm && pf_qm->ver >= QM_HW_V3) {
> > +		ret = hisi_acc_vfio_pci_migrn_init(hisi_acc_vdev, pdev,
> > pf_qm);
> > +		if (!ret) {
> > +			vfio_pci_core_init_device(&hisi_acc_vdev-
> > >core_device, pdev,
> > +
> > &hisi_acc_vfio_pci_migrn_ops);
> > +		} else {
> > +			pci_warn(pdev, "migration support failed, continue
> > with generic interface\n");
> > +			vfio_pci_core_init_device(&hisi_acc_vdev-
> > >core_device, pdev,
> > +						  &hisi_acc_vfio_pci_ops);
> > +		}
> 
> This logic looks weird. Earlier you state that the migration control region must
> be hidden from the userspace as a security requirement, but above logic reads
> like if the driver fails to initialize migration support then we just fall back to the
> default ops which grants the user the full access to the entire MMIO bar.

As I explained previously the risk of exposing migration BAR is only limited to migration
use case. So if for some reason we can't get the migration working, we default to the
generic vfio-pci like behavior.
 
> 
> > +	} else {
> > +		vfio_pci_core_init_device(&hisi_acc_vdev->core_device, pdev,
> > +					  &hisi_acc_vfio_pci_ops);
> > +	}
> 
> If the hardware itself doesn't support the migration capability, can we just
> move it out of the id table and let vfio-pci to drive it?
> 
 But the above is just like vfio-pci driving it, right?

Thanks,
Shameer

^ permalink raw reply	[flat|nested] 56+ messages in thread

* Re: [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live migration
  2022-03-07 19:52         ` Alex Williamson
  2022-03-08  8:11           ` Tian, Kevin
@ 2022-03-08  9:46           ` liulongfang
  1 sibling, 0 replies; 56+ messages in thread
From: liulongfang @ 2022-03-08  9:46 UTC (permalink / raw)
  To: Alex Williamson, Shameerali Kolothum Thodi
  Cc: Jason Gunthorpe, kvm, linux-kernel, linux-crypto, linux-pci,
	cohuck, mgurtovoy, yishaih, Linuxarm, Zengtao (B),
	Jonathan Cameron, Wangzhou (B),
	Xu Zaibo

On 2022/3/8 3:52, Alex Williamson wrote:
> On Mon, 7 Mar 2022 19:29:06 +0000
> Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com> wrote:
> 
>>> -----Original Message-----
>>> From: Alex Williamson [mailto:alex.williamson@redhat.com]
>>> Sent: 07 March 2022 19:05
>>> To: Jason Gunthorpe <jgg@nvidia.com>
>>> Cc: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>;
>>> kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
>>> linux-crypto@vger.kernel.org; linux-pci@vger.kernel.org; cohuck@redhat.com;
>>> mgurtovoy@nvidia.com; yishaih@nvidia.com; Linuxarm
>>> <linuxarm@huawei.com>; liulongfang <liulongfang@huawei.com>; Zengtao (B)
>>> <prime.zeng@hisilicon.com>; Jonathan Cameron
>>> <jonathan.cameron@huawei.com>; Wangzhou (B) <wangzhou1@hisilicon.com>
>>> Subject: Re: [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live
>>> migration
>>>
>>> On Fri, 4 Mar 2022 16:57:20 -0400
>>> Jason Gunthorpe <jgg@nvidia.com> wrote:
>>>   
>>>> On Thu, Mar 03, 2022 at 11:01:30PM +0000, Shameer Kolothum wrote:  
>>>>> From: Longfang Liu <liulongfang@huawei.com>
>>>>>
>>>>> VMs assigned with HiSilicon ACC VF devices can now perform live  
>>> migration  
>>>>> if the VF devices are bind to the hisi_acc_vfio_pci driver.
>>>>>
>>>>> Signed-off-by: Longfang Liu <liulongfang@huawei.com>
>>>>> Signed-off-by: Shameer Kolothum  
>>> <shameerali.kolothum.thodi@huawei.com>  
>>>>> ---
>>>>>  drivers/vfio/pci/hisilicon/Kconfig            |    7 +
>>>>>  .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    | 1078 ++++++++++++++++-
>>>>>  .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h    |  114 ++
>>>>>  3 files changed, 1181 insertions(+), 18 deletions(-)
>>>>>  create mode 100644 drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
>>>>>
>>>>> diff --git a/drivers/vfio/pci/hisilicon/Kconfig  
>>> b/drivers/vfio/pci/hisilicon/Kconfig  
>>>>> index dc723bad05c2..2a68d39f339f 100644
>>>>> --- a/drivers/vfio/pci/hisilicon/Kconfig
>>>>> +++ b/drivers/vfio/pci/hisilicon/Kconfig
>>>>> @@ -3,6 +3,13 @@ config HISI_ACC_VFIO_PCI
>>>>>  	tristate "VFIO PCI support for HiSilicon ACC devices"
>>>>>  	depends on ARM64 || (COMPILE_TEST && 64BIT)
>>>>>  	depends on VFIO_PCI_CORE
>>>>> +	depends on PCI && PCI_MSI  
>>>>
>>>> PCI is already in the depends from the 2nd line in
>>>> drivers/vfio/pci/Kconfig, but it is harmless
>>>>  
>>>>> +	depends on UACCE || UACCE=n
>>>>> +	depends on ACPI  
>>>>
>>>> Scratching my head a bit on why we have these  
>>>
>>> Same curiosity from me, each of the CRYPTO_DEV_HISI_* options selected
>>> also depend on these so they seem redundant.  
>>
>> Yes, they are redundant now since we have added CRYPTO_DEV_HISI_ drivers
>> as "depends" now. I will remove that.
>>  
>>> I think we still require acks from Bjorn and Zaibo for select patches
>>> in this series.  
>>
>> I checked with Ziabo. He moved projects and is no longer looking into crypto stuff.
>> Wangzhou and LiuLongfang now take care of this. Received acks from Wangzhou
>> already and I will request Longfang to provide his. Hope that's ok.
> 
> Maybe a good time to have them update MAINTAINERS as well.  Thanks,
> 
> Alex
> 
OK, we have discussed it internally, I will send a patch to update
MAINTAINERS of crypto stuff.
Thanks.
Longfang
> .
> 

^ permalink raw reply	[flat|nested] 56+ messages in thread

* RE: [PATCH v8 5/9] hisi_acc_vfio_pci: Restrict access to VF dev BAR2 migration region
  2022-03-08  8:33     ` Shameerali Kolothum Thodi
@ 2022-03-08 10:09       ` Tian, Kevin
  2022-03-08 11:02         ` Shameerali Kolothum Thodi
  0 siblings, 1 reply; 56+ messages in thread
From: Tian, Kevin @ 2022-03-08 10:09 UTC (permalink / raw)
  To: Shameerali Kolothum Thodi, kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	Linuxarm, liulongfang, Zengtao (B),
	Jonathan Cameron, Wangzhou (B)

> From: Shameerali Kolothum Thodi
> <shameerali.kolothum.thodi@huawei.com>
> Sent: Tuesday, March 8, 2022 4:33 PM
> 
> Hi Kevin,
> 
> > -----Original Message-----
> > From: Tian, Kevin [mailto:kevin.tian@intel.com]
> > Sent: 08 March 2022 06:23
> > To: Shameerali Kolothum Thodi
> <shameerali.kolothum.thodi@huawei.com>;
> > kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> > linux-crypto@vger.kernel.org
> > Cc: linux-pci@vger.kernel.org; alex.williamson@redhat.com;
> jgg@nvidia.com;
> > cohuck@redhat.com; mgurtovoy@nvidia.com; yishaih@nvidia.com;
> Linuxarm
> > <linuxarm@huawei.com>; liulongfang <liulongfang@huawei.com>;
> Zengtao (B)
> > <prime.zeng@hisilicon.com>; Jonathan Cameron
> > <jonathan.cameron@huawei.com>; Wangzhou (B)
> <wangzhou1@hisilicon.com>
> > Subject: RE: [PATCH v8 5/9] hisi_acc_vfio_pci: Restrict access to VF dev
> BAR2
> > migration region
> >
> > Hi, Shameer,
> >
> > > From: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
> > > Sent: Friday, March 4, 2022 7:01 AM
> > >
> > > HiSilicon ACC VF device BAR2 region consists of both functional
> > > register space and migration control register space. From a
> > > security point of view, it's not advisable to export the migration
> > > control region to Guest.
> > >
> > > Hence, introduce a separate struct vfio_device_ops for migration
> > > support which will override the ioctl/read/write/mmap methods to
> > > hide the migration region and limit the access only to the
> > > functional register space.
> > >
> > > This will be used in subsequent patches when we add migration
> > > support to the driver.
> >
> > As a security concern the migration control region should be always
> > disabled regardless of whether migration support is added to the
> > driver for such device... It sounds like we should first fix this security
> > hole for acc device assignment and then add the migration support
> > atop (at least organize the series in this way).
> 
> By exposing the migration BAR region, there is a possibility that a malicious
> Guest can prevent migration from happening by manipulating the migration
> BAR region. I don't think there are any other security concerns now especially
> since we only support the STOP_COPY state.  And the approach has been
> that
> we only restrict this if migration support is enabled. I think I can change the
> above "security concern" description to "malicious Guest can prevent
> migration"
> to make it more clear.
> 

In concept migrated device state may include both the state directly 
touched by the guest driver and also the one that is configured by 
the PF driver. Unless there is guarantee that the state managed via 
the migration control interface only touches the former (which implies
the latter managed via the PF driver) this security concern will hold
even for normal device assignment.

If the acc device has such guarantee it's worth of a clarification here.

Thanks
Kevin

^ permalink raw reply	[flat|nested] 56+ messages in thread

* RE: [PATCH v8 7/9] crypto: hisilicon/qm: Set the VF QM state register
  2022-03-08  8:46     ` Shameerali Kolothum Thodi
@ 2022-03-08 10:10       ` Tian, Kevin
  0 siblings, 0 replies; 56+ messages in thread
From: Tian, Kevin @ 2022-03-08 10:10 UTC (permalink / raw)
  To: Shameerali Kolothum Thodi, kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	Linuxarm, liulongfang, Zengtao (B),
	Jonathan Cameron, Wangzhou (B)

> From: Shameerali Kolothum Thodi
> <shameerali.kolothum.thodi@huawei.com>
> Sent: Tuesday, March 8, 2022 4:46 PM
> 
> 
> 
> > -----Original Message-----
> > From: Tian, Kevin [mailto:kevin.tian@intel.com]
> > Sent: 08 March 2022 06:31
> > To: Shameerali Kolothum Thodi
> <shameerali.kolothum.thodi@huawei.com>;
> > kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> > linux-crypto@vger.kernel.org
> > Cc: linux-pci@vger.kernel.org; alex.williamson@redhat.com;
> jgg@nvidia.com;
> > cohuck@redhat.com; mgurtovoy@nvidia.com; yishaih@nvidia.com;
> Linuxarm
> > <linuxarm@huawei.com>; liulongfang <liulongfang@huawei.com>;
> Zengtao (B)
> > <prime.zeng@hisilicon.com>; Jonathan Cameron
> > <jonathan.cameron@huawei.com>; Wangzhou (B)
> <wangzhou1@hisilicon.com>
> > Subject: RE: [PATCH v8 7/9] crypto: hisilicon/qm: Set the VF QM state
> register
> >
> > > From: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
> > > Sent: Friday, March 4, 2022 7:01 AM
> > >
> > > From: Longfang Liu <liulongfang@huawei.com>
> > >
> > > We use VF QM state register to record the status of the QM configuration
> > > state. This will be used in the ACC migration driver to determine whether
> > > we can safely save and restore the QM data.
> >
> > Can you say something about what QM is and how it is related to the VF
> state
> > to be migrated? It might be obvious to acc driver people but not so to to
> > others in vfio community like me. 😊
> 
> Ok, understand that. I will lift a description from the QM driver patch :).
> 
> QM stands for Queue Management which is a generic IP used by ACC devices.
> It provides a general PCIe interface for the CPU and the ACC devices to share
> a group of queues.
> 
> QM integrated into an accelerator provides queue management service.
> Queues can be assigned to PF and VFs, and queues can be controlled by
> unified mailboxes and doorbells. The QM
> driver(drivers/crypto/hisilicon/qm.c)
> provides generic interfaces to ACC drivers to manage the QM.
> 
> I will add some of this in either this commit msg or in the actual file
> somewhere.
> 

Yes, such information is helpful

^ permalink raw reply	[flat|nested] 56+ messages in thread

* RE: [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live migration
  2022-03-08  8:52     ` Shameerali Kolothum Thodi
@ 2022-03-08 10:17       ` Tian, Kevin
  0 siblings, 0 replies; 56+ messages in thread
From: Tian, Kevin @ 2022-03-08 10:17 UTC (permalink / raw)
  To: Shameerali Kolothum Thodi, kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	Linuxarm, liulongfang, Zengtao (B),
	Jonathan Cameron, Wangzhou (B)

> From: Shameerali Kolothum Thodi
> <shameerali.kolothum.thodi@huawei.com>
> Sent: Tuesday, March 8, 2022 4:53 PM
> > > +		} else {
> > > +			pci_warn(pdev, "migration support failed, continue
> > > with generic interface\n");
> > > +			vfio_pci_core_init_device(&hisi_acc_vdev-
> > > >core_device, pdev,
> > > +						  &hisi_acc_vfio_pci_ops);
> > > +		}
> >
> > This logic looks weird. Earlier you state that the migration control region
> must
> > be hidden from the userspace as a security requirement, but above logic
> reads
> > like if the driver fails to initialize migration support then we just fall back to
> the
> > default ops which grants the user the full access to the entire MMIO bar.
> 
> As I explained previously the risk of exposing migration BAR is only limited to
> migration
> use case. So if for some reason we can't get the migration working, we
> default to the
> generic vfio-pci like behavior.

Yes, as long as there is guarantee that exposing such region doesn't allow 
the guest to escape any configuration enforced by the PF driver.

> 
> >
> > > +	} else {
> > > +		vfio_pci_core_init_device(&hisi_acc_vdev->core_device, pdev,
> > > +					  &hisi_acc_vfio_pci_ops);
> > > +	}
> >
> > If the hardware itself doesn't support the migration capability, can we just
> > move it out of the id table and let vfio-pci to drive it?
> >
>  But the above is just like vfio-pci driving it, right?
> 

Yes, earlier I thought this driver may only want to drive devices which
support migration capability. But my opinion on this is not strong
and having it drive all acc devices is not harmful anyway...

Thanks
Kevin

^ permalink raw reply	[flat|nested] 56+ messages in thread

* Re: [PATCH v8 1/9] crypto: hisilicon/qm: Move the QM header to include/linux
  2022-03-03 23:01 ` [PATCH v8 1/9] crypto: hisilicon/qm: Move the QM header to include/linux Shameer Kolothum
  2022-03-04  9:03   ` Zhou Wang
  2022-03-08  1:07   ` liulongfang
@ 2022-03-08 10:27   ` yekai(A)
  2 siblings, 0 replies; 56+ messages in thread
From: yekai(A) @ 2022-03-08 10:27 UTC (permalink / raw)
  To: Shameer Kolothum, kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	linuxarm, liulongfang, prime.zeng, jonathan.cameron, wangzhou1



On 2022/3/4 7:01, Shameer Kolothum wrote:
> Since we are going to introduce VFIO PCI HiSilicon ACC driver for live
> migration in subsequent patches, move the ACC QM header file to a
> common include dir.
>
> Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
> ---
>  drivers/crypto/hisilicon/hpre/hpre.h                         | 2 +-
>  drivers/crypto/hisilicon/qm.c                                | 2 +-
>  drivers/crypto/hisilicon/sec2/sec.h                          | 2 +-
>  drivers/crypto/hisilicon/sgl.c                               | 2 +-
>  drivers/crypto/hisilicon/zip/zip.h                           | 2 +-
>  drivers/crypto/hisilicon/qm.h => include/linux/hisi_acc_qm.h | 0
>  6 files changed, 5 insertions(+), 5 deletions(-)
>  rename drivers/crypto/hisilicon/qm.h => include/linux/hisi_acc_qm.h (100%)
>
> diff --git a/drivers/crypto/hisilicon/hpre/hpre.h b/drivers/crypto/hisilicon/hpre/hpre.h
> index e0b4a1982ee9..9a0558ed82f9 100644
> --- a/drivers/crypto/hisilicon/hpre/hpre.h
> +++ b/drivers/crypto/hisilicon/hpre/hpre.h
> @@ -4,7 +4,7 @@
>  #define __HISI_HPRE_H
>
>  #include <linux/list.h>
> -#include "../qm.h"
> +#include <linux/hisi_acc_qm.h>
>
>  #define HPRE_SQE_SIZE			sizeof(struct hpre_sqe)
>  #define HPRE_PF_DEF_Q_NUM		64
> diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c
> index c5b84a5ea350..ed23e1d3fa27 100644
> --- a/drivers/crypto/hisilicon/qm.c
> +++ b/drivers/crypto/hisilicon/qm.c
> @@ -15,7 +15,7 @@
>  #include <linux/uacce.h>
>  #include <linux/uaccess.h>
>  #include <uapi/misc/uacce/hisi_qm.h>
> -#include "qm.h"
> +#include <linux/hisi_acc_qm.h>
>
>  /* eq/aeq irq enable */
>  #define QM_VF_AEQ_INT_SOURCE		0x0
> diff --git a/drivers/crypto/hisilicon/sec2/sec.h b/drivers/crypto/hisilicon/sec2/sec.h
> index d97cf02b1df7..c2e9b01187a7 100644
> --- a/drivers/crypto/hisilicon/sec2/sec.h
> +++ b/drivers/crypto/hisilicon/sec2/sec.h
> @@ -4,7 +4,7 @@
>  #ifndef __HISI_SEC_V2_H
>  #define __HISI_SEC_V2_H
>
> -#include "../qm.h"
> +#include <linux/hisi_acc_qm.h>
>  #include "sec_crypto.h"
>
>  /* Algorithm resource per hardware SEC queue */
> diff --git a/drivers/crypto/hisilicon/sgl.c b/drivers/crypto/hisilicon/sgl.c
> index 057273769f26..f7efc02b065f 100644
> --- a/drivers/crypto/hisilicon/sgl.c
> +++ b/drivers/crypto/hisilicon/sgl.c
> @@ -1,9 +1,9 @@
>  // SPDX-License-Identifier: GPL-2.0
>  /* Copyright (c) 2019 HiSilicon Limited. */
>  #include <linux/dma-mapping.h>
> +#include <linux/hisi_acc_qm.h>
>  #include <linux/module.h>
>  #include <linux/slab.h>
> -#include "qm.h"
>
>  #define HISI_ACC_SGL_SGE_NR_MIN		1
>  #define HISI_ACC_SGL_NR_MAX		256
> diff --git a/drivers/crypto/hisilicon/zip/zip.h b/drivers/crypto/hisilicon/zip/zip.h
> index 517fdbdff3ea..3dfd3bac5a33 100644
> --- a/drivers/crypto/hisilicon/zip/zip.h
> +++ b/drivers/crypto/hisilicon/zip/zip.h
> @@ -7,7 +7,7 @@
>  #define pr_fmt(fmt)	"hisi_zip: " fmt
>
>  #include <linux/list.h>
> -#include "../qm.h"
> +#include <linux/hisi_acc_qm.h>
>
>  enum hisi_zip_error_type {
>  	/* negative compression */
> diff --git a/drivers/crypto/hisilicon/qm.h b/include/linux/hisi_acc_qm.h
> similarity index 100%
> rename from drivers/crypto/hisilicon/qm.h
> rename to include/linux/hisi_acc_qm.h
>

Hi Shameer,

It looks good to me for this movement.

Acked-by:  Kai Ye <yekai13@huawei.com>

Thanks,
Kai

^ permalink raw reply	[flat|nested] 56+ messages in thread

* Re: [PATCH v8 3/9] hisi_acc_qm: Move VF PCI device IDs to common header
  2022-03-03 23:01 ` [PATCH v8 3/9] hisi_acc_qm: Move VF PCI device IDs " Shameer Kolothum
                     ` (2 preceding siblings ...)
  2022-03-08  1:08   ` liulongfang
@ 2022-03-08 10:28   ` yekai(A)
  3 siblings, 0 replies; 56+ messages in thread
From: yekai(A) @ 2022-03-08 10:28 UTC (permalink / raw)
  To: Shameer Kolothum, kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	linuxarm, liulongfang, prime.zeng, jonathan.cameron, wangzhou1



On 2022/3/4 7:01, Shameer Kolothum wrote:
> Move the PCI Device IDs of HiSilicon ACC VF devices to a common header
> and also use a uniform naming convention.
>
> This will be useful when we introduce the vfio PCI HiSilicon ACC live
> migration driver in subsequent patches.
>
> Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
> ---
>  drivers/crypto/hisilicon/hpre/hpre_main.c | 13 ++++++-------
>  drivers/crypto/hisilicon/sec2/sec_main.c  | 15 +++++++--------
>  drivers/crypto/hisilicon/zip/zip_main.c   | 11 +++++------
>  include/linux/pci_ids.h                   |  3 +++
>  4 files changed, 21 insertions(+), 21 deletions(-)
>
> diff --git a/drivers/crypto/hisilicon/hpre/hpre_main.c b/drivers/crypto/hisilicon/hpre/hpre_main.c
> index ebfab3e14499..3589d8879b5e 100644
> --- a/drivers/crypto/hisilicon/hpre/hpre_main.c
> +++ b/drivers/crypto/hisilicon/hpre/hpre_main.c
> @@ -68,8 +68,7 @@
>  #define HPRE_REG_RD_INTVRL_US		10
>  #define HPRE_REG_RD_TMOUT_US		1000
>  #define HPRE_DBGFS_VAL_MAX_LEN		20
> -#define HPRE_PCI_DEVICE_ID		0xa258
> -#define HPRE_PCI_VF_DEVICE_ID		0xa259
> +#define PCI_DEVICE_ID_HUAWEI_HPRE_PF	0xa258
>  #define HPRE_QM_USR_CFG_MASK		GENMASK(31, 1)
>  #define HPRE_QM_AXI_CFG_MASK		GENMASK(15, 0)
>  #define HPRE_QM_VFG_AX_MASK		GENMASK(7, 0)
> @@ -111,8 +110,8 @@
>  static const char hpre_name[] = "hisi_hpre";
>  static struct dentry *hpre_debugfs_root;
>  static const struct pci_device_id hpre_dev_ids[] = {
> -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, HPRE_PCI_DEVICE_ID) },
> -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, HPRE_PCI_VF_DEVICE_ID) },
> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_HPRE_PF) },
> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_HPRE_VF) },
>  	{ 0, }
>  };
>
> @@ -242,7 +241,7 @@ MODULE_PARM_DESC(uacce_mode, UACCE_MODE_DESC);
>
>  static int pf_q_num_set(const char *val, const struct kernel_param *kp)
>  {
> -	return q_num_set(val, kp, HPRE_PCI_DEVICE_ID);
> +	return q_num_set(val, kp, PCI_DEVICE_ID_HUAWEI_HPRE_PF);
>  }
>
>  static const struct kernel_param_ops hpre_pf_q_num_ops = {
> @@ -921,7 +920,7 @@ static int hpre_debugfs_init(struct hisi_qm *qm)
>  	qm->debug.sqe_mask_len = HPRE_SQE_MASK_LEN;
>  	hisi_qm_debug_init(qm);
>
> -	if (qm->pdev->device == HPRE_PCI_DEVICE_ID) {
> +	if (qm->pdev->device == PCI_DEVICE_ID_HUAWEI_HPRE_PF) {
>  		ret = hpre_ctrl_debug_init(qm);
>  		if (ret)
>  			goto failed_to_create;
> @@ -958,7 +957,7 @@ static int hpre_qm_init(struct hisi_qm *qm, struct pci_dev *pdev)
>  	qm->sqe_size = HPRE_SQE_SIZE;
>  	qm->dev_name = hpre_name;
>
> -	qm->fun_type = (pdev->device == HPRE_PCI_DEVICE_ID) ?
> +	qm->fun_type = (pdev->device == PCI_DEVICE_ID_HUAWEI_HPRE_PF) ?
>  			QM_HW_PF : QM_HW_VF;
>  	if (qm->fun_type == QM_HW_PF) {
>  		qm->qp_base = HPRE_PF_DEF_Q_BASE;
> diff --git a/drivers/crypto/hisilicon/sec2/sec_main.c b/drivers/crypto/hisilicon/sec2/sec_main.c
> index 26d3ab1d308b..311a8747b5bf 100644
> --- a/drivers/crypto/hisilicon/sec2/sec_main.c
> +++ b/drivers/crypto/hisilicon/sec2/sec_main.c
> @@ -20,8 +20,7 @@
>
>  #define SEC_VF_NUM			63
>  #define SEC_QUEUE_NUM_V1		4096
> -#define SEC_PF_PCI_DEVICE_ID		0xa255
> -#define SEC_VF_PCI_DEVICE_ID		0xa256
> +#define PCI_DEVICE_ID_HUAWEI_SEC_PF	0xa255
>
>  #define SEC_BD_ERR_CHK_EN0		0xEFFFFFFF
>  #define SEC_BD_ERR_CHK_EN1		0x7ffff7fd
> @@ -225,7 +224,7 @@ static const struct debugfs_reg32 sec_dfx_regs[] = {
>
>  static int sec_pf_q_num_set(const char *val, const struct kernel_param *kp)
>  {
> -	return q_num_set(val, kp, SEC_PF_PCI_DEVICE_ID);
> +	return q_num_set(val, kp, PCI_DEVICE_ID_HUAWEI_SEC_PF);
>  }
>
>  static const struct kernel_param_ops sec_pf_q_num_ops = {
> @@ -313,8 +312,8 @@ module_param_cb(uacce_mode, &sec_uacce_mode_ops, &uacce_mode, 0444);
>  MODULE_PARM_DESC(uacce_mode, UACCE_MODE_DESC);
>
>  static const struct pci_device_id sec_dev_ids[] = {
> -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, SEC_PF_PCI_DEVICE_ID) },
> -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, SEC_VF_PCI_DEVICE_ID) },
> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_SEC_PF) },
> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_SEC_VF) },
>  	{ 0, }
>  };
>  MODULE_DEVICE_TABLE(pci, sec_dev_ids);
> @@ -717,7 +716,7 @@ static int sec_core_debug_init(struct hisi_qm *qm)
>  	regset->base = qm->io_base;
>  	regset->dev = dev;
>
> -	if (qm->pdev->device == SEC_PF_PCI_DEVICE_ID)
> +	if (qm->pdev->device == PCI_DEVICE_ID_HUAWEI_SEC_PF)
>  		debugfs_create_file("regs", 0444, tmp_d, regset, &sec_regs_fops);
>
>  	for (i = 0; i < ARRAY_SIZE(sec_dfx_labels); i++) {
> @@ -735,7 +734,7 @@ static int sec_debug_init(struct hisi_qm *qm)
>  	struct sec_dev *sec = container_of(qm, struct sec_dev, qm);
>  	int i;
>
> -	if (qm->pdev->device == SEC_PF_PCI_DEVICE_ID) {
> +	if (qm->pdev->device == PCI_DEVICE_ID_HUAWEI_SEC_PF) {
>  		for (i = SEC_CLEAR_ENABLE; i < SEC_DEBUG_FILE_NUM; i++) {
>  			spin_lock_init(&sec->debug.files[i].lock);
>  			sec->debug.files[i].index = i;
> @@ -877,7 +876,7 @@ static int sec_qm_init(struct hisi_qm *qm, struct pci_dev *pdev)
>  	qm->sqe_size = SEC_SQE_SIZE;
>  	qm->dev_name = sec_name;
>
> -	qm->fun_type = (pdev->device == SEC_PF_PCI_DEVICE_ID) ?
> +	qm->fun_type = (pdev->device == PCI_DEVICE_ID_HUAWEI_SEC_PF) ?
>  			QM_HW_PF : QM_HW_VF;
>  	if (qm->fun_type == QM_HW_PF) {
>  		qm->qp_base = SEC_PF_DEF_Q_BASE;
> diff --git a/drivers/crypto/hisilicon/zip/zip_main.c b/drivers/crypto/hisilicon/zip/zip_main.c
> index 678f8b58ec42..66decfe07282 100644
> --- a/drivers/crypto/hisilicon/zip/zip_main.c
> +++ b/drivers/crypto/hisilicon/zip/zip_main.c
> @@ -15,8 +15,7 @@
>  #include <linux/uacce.h>
>  #include "zip.h"
>
> -#define PCI_DEVICE_ID_ZIP_PF		0xa250
> -#define PCI_DEVICE_ID_ZIP_VF		0xa251
> +#define PCI_DEVICE_ID_HUAWEI_ZIP_PF	0xa250
>
>  #define HZIP_QUEUE_NUM_V1		4096
>
> @@ -246,7 +245,7 @@ MODULE_PARM_DESC(uacce_mode, UACCE_MODE_DESC);
>
>  static int pf_q_num_set(const char *val, const struct kernel_param *kp)
>  {
> -	return q_num_set(val, kp, PCI_DEVICE_ID_ZIP_PF);
> +	return q_num_set(val, kp, PCI_DEVICE_ID_HUAWEI_ZIP_PF);
>  }
>
>  static const struct kernel_param_ops pf_q_num_ops = {
> @@ -268,8 +267,8 @@ module_param_cb(vfs_num, &vfs_num_ops, &vfs_num, 0444);
>  MODULE_PARM_DESC(vfs_num, "Number of VFs to enable(1-63), 0(default)");
>
>  static const struct pci_device_id hisi_zip_dev_ids[] = {
> -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_ZIP_PF) },
> -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_ZIP_VF) },
> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_ZIP_PF) },
> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_ZIP_VF) },
>  	{ 0, }
>  };
>  MODULE_DEVICE_TABLE(pci, hisi_zip_dev_ids);
> @@ -838,7 +837,7 @@ static int hisi_zip_qm_init(struct hisi_qm *qm, struct pci_dev *pdev)
>  	qm->sqe_size = HZIP_SQE_SIZE;
>  	qm->dev_name = hisi_zip_name;
>
> -	qm->fun_type = (pdev->device == PCI_DEVICE_ID_ZIP_PF) ?
> +	qm->fun_type = (pdev->device == PCI_DEVICE_ID_HUAWEI_ZIP_PF) ?
>  			QM_HW_PF : QM_HW_VF;
>  	if (qm->fun_type == QM_HW_PF) {
>  		qm->qp_base = HZIP_PF_DEF_Q_BASE;
> diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
> index aad54c666407..31dee2b65a62 100644
> --- a/include/linux/pci_ids.h
> +++ b/include/linux/pci_ids.h
> @@ -2529,6 +2529,9 @@
>  #define PCI_DEVICE_ID_KORENIX_JETCARDF3	0x17ff
>
>  #define PCI_VENDOR_ID_HUAWEI		0x19e5
> +#define PCI_DEVICE_ID_HUAWEI_ZIP_VF	0xa251
> +#define PCI_DEVICE_ID_HUAWEI_SEC_VF	0xa256
> +#define PCI_DEVICE_ID_HUAWEI_HPRE_VF	0xa259
>
>  #define PCI_VENDOR_ID_NETRONOME		0x19ee
>  #define PCI_DEVICE_ID_NETRONOME_NFP4000	0x4000
>


Hi Shameer,

It looks good to me for this movement.

Acked-by:  Kai Ye <yekai13@huawei.com>

Thanks,
Kai

^ permalink raw reply	[flat|nested] 56+ messages in thread

* Re: [PATCH v8 6/9] hisi_acc_vfio_pci: Add helper to retrieve the struct pci_driver
  2022-03-03 23:01 ` [PATCH v8 6/9] hisi_acc_vfio_pci: Add helper to retrieve the struct pci_driver Shameer Kolothum
  2022-03-04  9:40   ` Zhou Wang
@ 2022-03-08 10:28   ` yekai(A)
  2022-03-08 12:02   ` liulongfang
  2 siblings, 0 replies; 56+ messages in thread
From: yekai(A) @ 2022-03-08 10:28 UTC (permalink / raw)
  To: Shameer Kolothum, kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	linuxarm, liulongfang, prime.zeng, jonathan.cameron, wangzhou1



On 2022/3/4 7:01, Shameer Kolothum wrote:
> struct pci_driver pointer is an input into the pci_iov_get_pf_drvdata().
> Introduce helpers to retrieve the ACC PF dev struct pci_driver pointers
> as we use this in ACC vfio migration driver.
>
> Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
> ---
>  drivers/crypto/hisilicon/hpre/hpre_main.c | 6 ++++++
>  drivers/crypto/hisilicon/sec2/sec_main.c  | 6 ++++++
>  drivers/crypto/hisilicon/zip/zip_main.c   | 6 ++++++
>  include/linux/hisi_acc_qm.h               | 5 +++++
>  4 files changed, 23 insertions(+)
>
> diff --git a/drivers/crypto/hisilicon/hpre/hpre_main.c b/drivers/crypto/hisilicon/hpre/hpre_main.c
> index 3589d8879b5e..36ab30e9e654 100644
> --- a/drivers/crypto/hisilicon/hpre/hpre_main.c
> +++ b/drivers/crypto/hisilicon/hpre/hpre_main.c
> @@ -1190,6 +1190,12 @@ static struct pci_driver hpre_pci_driver = {
>  	.driver.pm		= &hpre_pm_ops,
>  };
>
> +struct pci_driver *hisi_hpre_get_pf_driver(void)
> +{
> +	return &hpre_pci_driver;
> +}
> +EXPORT_SYMBOL_GPL(hisi_hpre_get_pf_driver);
> +
>  static void hpre_register_debugfs(void)
>  {
>  	if (!debugfs_initialized())
> diff --git a/drivers/crypto/hisilicon/sec2/sec_main.c b/drivers/crypto/hisilicon/sec2/sec_main.c
> index 311a8747b5bf..421a405ca337 100644
> --- a/drivers/crypto/hisilicon/sec2/sec_main.c
> +++ b/drivers/crypto/hisilicon/sec2/sec_main.c
> @@ -1088,6 +1088,12 @@ static struct pci_driver sec_pci_driver = {
>  	.driver.pm = &sec_pm_ops,
>  };
>
> +struct pci_driver *hisi_sec_get_pf_driver(void)
> +{
> +	return &sec_pci_driver;
> +}
> +EXPORT_SYMBOL_GPL(hisi_sec_get_pf_driver);
> +
>  static void sec_register_debugfs(void)
>  {
>  	if (!debugfs_initialized())
> diff --git a/drivers/crypto/hisilicon/zip/zip_main.c b/drivers/crypto/hisilicon/zip/zip_main.c
> index 66decfe07282..4534e1e107d1 100644
> --- a/drivers/crypto/hisilicon/zip/zip_main.c
> +++ b/drivers/crypto/hisilicon/zip/zip_main.c
> @@ -1012,6 +1012,12 @@ static struct pci_driver hisi_zip_pci_driver = {
>  	.driver.pm		= &hisi_zip_pm_ops,
>  };
>
> +struct pci_driver *hisi_zip_get_pf_driver(void)
> +{
> +	return &hisi_zip_pci_driver;
> +}
> +EXPORT_SYMBOL_GPL(hisi_zip_get_pf_driver);
> +
>  static void hisi_zip_register_debugfs(void)
>  {
>  	if (!debugfs_initialized())
> diff --git a/include/linux/hisi_acc_qm.h b/include/linux/hisi_acc_qm.h
> index 6a6477c34666..00f2a4db8723 100644
> --- a/include/linux/hisi_acc_qm.h
> +++ b/include/linux/hisi_acc_qm.h
> @@ -476,4 +476,9 @@ void hisi_qm_pm_init(struct hisi_qm *qm);
>  int hisi_qm_get_dfx_access(struct hisi_qm *qm);
>  void hisi_qm_put_dfx_access(struct hisi_qm *qm);
>  void hisi_qm_regs_dump(struct seq_file *s, struct debugfs_regset32 *regset);
> +
> +/* Used by VFIO ACC live migration driver */
> +struct pci_driver *hisi_sec_get_pf_driver(void);
> +struct pci_driver *hisi_hpre_get_pf_driver(void);
> +struct pci_driver *hisi_zip_get_pf_driver(void);
>  #endif
>

Hi Shameer,

It looks good to me for this movement.

Acked-by:  Kai Ye <yekai13@huawei.com>

Thanks,
Kai

^ permalink raw reply	[flat|nested] 56+ messages in thread

* RE: [PATCH v8 5/9] hisi_acc_vfio_pci: Restrict access to VF dev BAR2 migration region
  2022-03-08 10:09       ` Tian, Kevin
@ 2022-03-08 11:02         ` Shameerali Kolothum Thodi
  0 siblings, 0 replies; 56+ messages in thread
From: Shameerali Kolothum Thodi @ 2022-03-08 11:02 UTC (permalink / raw)
  To: Tian, Kevin, kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	Linuxarm, liulongfang, Zengtao (B),
	Jonathan Cameron, Wangzhou (B)



> -----Original Message-----
> From: Tian, Kevin [mailto:kevin.tian@intel.com]
> Sent: 08 March 2022 10:09
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>;
> kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-crypto@vger.kernel.org
> Cc: linux-pci@vger.kernel.org; alex.williamson@redhat.com; jgg@nvidia.com;
> cohuck@redhat.com; mgurtovoy@nvidia.com; yishaih@nvidia.com; Linuxarm
> <linuxarm@huawei.com>; liulongfang <liulongfang@huawei.com>; Zengtao (B)
> <prime.zeng@hisilicon.com>; Jonathan Cameron
> <jonathan.cameron@huawei.com>; Wangzhou (B) <wangzhou1@hisilicon.com>
> Subject: RE: [PATCH v8 5/9] hisi_acc_vfio_pci: Restrict access to VF dev BAR2
> migration region
> 
> > From: Shameerali Kolothum Thodi
> > <shameerali.kolothum.thodi@huawei.com>
> > Sent: Tuesday, March 8, 2022 4:33 PM
> >
> > Hi Kevin,
> >
> > > -----Original Message-----
> > > From: Tian, Kevin [mailto:kevin.tian@intel.com]
> > > Sent: 08 March 2022 06:23
> > > To: Shameerali Kolothum Thodi
> > <shameerali.kolothum.thodi@huawei.com>;
> > > kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> > > linux-crypto@vger.kernel.org
> > > Cc: linux-pci@vger.kernel.org; alex.williamson@redhat.com;
> > jgg@nvidia.com;
> > > cohuck@redhat.com; mgurtovoy@nvidia.com; yishaih@nvidia.com;
> > Linuxarm
> > > <linuxarm@huawei.com>; liulongfang <liulongfang@huawei.com>;
> > Zengtao (B)
> > > <prime.zeng@hisilicon.com>; Jonathan Cameron
> > > <jonathan.cameron@huawei.com>; Wangzhou (B)
> > <wangzhou1@hisilicon.com>
> > > Subject: RE: [PATCH v8 5/9] hisi_acc_vfio_pci: Restrict access to VF dev
> > BAR2
> > > migration region
> > >
> > > Hi, Shameer,
> > >
> > > > From: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
> > > > Sent: Friday, March 4, 2022 7:01 AM
> > > >
> > > > HiSilicon ACC VF device BAR2 region consists of both functional
> > > > register space and migration control register space. From a
> > > > security point of view, it's not advisable to export the migration
> > > > control region to Guest.
> > > >
> > > > Hence, introduce a separate struct vfio_device_ops for migration
> > > > support which will override the ioctl/read/write/mmap methods to
> > > > hide the migration region and limit the access only to the
> > > > functional register space.
> > > >
> > > > This will be used in subsequent patches when we add migration
> > > > support to the driver.
> > >
> > > As a security concern the migration control region should be always
> > > disabled regardless of whether migration support is added to the
> > > driver for such device... It sounds like we should first fix this security
> > > hole for acc device assignment and then add the migration support
> > > atop (at least organize the series in this way).
> >
> > By exposing the migration BAR region, there is a possibility that a malicious
> > Guest can prevent migration from happening by manipulating the migration
> > BAR region. I don't think there are any other security concerns now
> especially
> > since we only support the STOP_COPY state.  And the approach has been
> > that
> > we only restrict this if migration support is enabled. I think I can change the
> > above "security concern" description to "malicious Guest can prevent
> > migration"
> > to make it more clear.
> >
> 
> In concept migrated device state may include both the state directly
> touched by the guest driver and also the one that is configured by
> the PF driver. Unless there is guarantee that the state managed via
> the migration control interface only touches the former (which implies
> the latter managed via the PF driver) this security concern will hold
> even for normal device assignment.
> 
> If the acc device has such guarantee it's worth of a clarification here.

I just double-checked with our ACC team and the VF migration region 
manipulations will not affect the PF configurations. I will add a clarification
here to make it clear.

Thanks,
Shameer

^ permalink raw reply	[flat|nested] 56+ messages in thread

* Re: [PATCH v8 6/9] hisi_acc_vfio_pci: Add helper to retrieve the struct pci_driver
  2022-03-03 23:01 ` [PATCH v8 6/9] hisi_acc_vfio_pci: Add helper to retrieve the struct pci_driver Shameer Kolothum
  2022-03-04  9:40   ` Zhou Wang
  2022-03-08 10:28   ` yekai(A)
@ 2022-03-08 12:02   ` liulongfang
  2 siblings, 0 replies; 56+ messages in thread
From: liulongfang @ 2022-03-08 12:02 UTC (permalink / raw)
  To: Shameer Kolothum, kvm, linux-kernel, linux-crypto
  Cc: linux-pci, alex.williamson, jgg, cohuck, mgurtovoy, yishaih,
	linuxarm, prime.zeng, jonathan.cameron, wangzhou1

On 2022/3/4 7:01, Shameer Kolothum wrote:
> struct pci_driver pointer is an input into the pci_iov_get_pf_drvdata().
> Introduce helpers to retrieve the ACC PF dev struct pci_driver pointers
> as we use this in ACC vfio migration driver.
> 
> Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
> ---
>  drivers/crypto/hisilicon/hpre/hpre_main.c | 6 ++++++
>  drivers/crypto/hisilicon/sec2/sec_main.c  | 6 ++++++
>  drivers/crypto/hisilicon/zip/zip_main.c   | 6 ++++++
>  include/linux/hisi_acc_qm.h               | 5 +++++
>  4 files changed, 23 insertions(+)
> 
Acked-by: Longfang Liu <liulongfang@huawei.com>

Thanks,
Longfang.
> diff --git a/drivers/crypto/hisilicon/hpre/hpre_main.c b/drivers/crypto/hisilicon/hpre/hpre_main.c
> index 3589d8879b5e..36ab30e9e654 100644
> --- a/drivers/crypto/hisilicon/hpre/hpre_main.c
> +++ b/drivers/crypto/hisilicon/hpre/hpre_main.c
> @@ -1190,6 +1190,12 @@ static struct pci_driver hpre_pci_driver = {
>  	.driver.pm		= &hpre_pm_ops,
>  };
>  
> +struct pci_driver *hisi_hpre_get_pf_driver(void)
> +{
> +	return &hpre_pci_driver;
> +}
> +EXPORT_SYMBOL_GPL(hisi_hpre_get_pf_driver);
> +
>  static void hpre_register_debugfs(void)
>  {
>  	if (!debugfs_initialized())
> diff --git a/drivers/crypto/hisilicon/sec2/sec_main.c b/drivers/crypto/hisilicon/sec2/sec_main.c
> index 311a8747b5bf..421a405ca337 100644
> --- a/drivers/crypto/hisilicon/sec2/sec_main.c
> +++ b/drivers/crypto/hisilicon/sec2/sec_main.c
> @@ -1088,6 +1088,12 @@ static struct pci_driver sec_pci_driver = {
>  	.driver.pm = &sec_pm_ops,
>  };
>  
> +struct pci_driver *hisi_sec_get_pf_driver(void)
> +{
> +	return &sec_pci_driver;
> +}
> +EXPORT_SYMBOL_GPL(hisi_sec_get_pf_driver);
> +
>  static void sec_register_debugfs(void)
>  {
>  	if (!debugfs_initialized())
> diff --git a/drivers/crypto/hisilicon/zip/zip_main.c b/drivers/crypto/hisilicon/zip/zip_main.c
> index 66decfe07282..4534e1e107d1 100644
> --- a/drivers/crypto/hisilicon/zip/zip_main.c
> +++ b/drivers/crypto/hisilicon/zip/zip_main.c
> @@ -1012,6 +1012,12 @@ static struct pci_driver hisi_zip_pci_driver = {
>  	.driver.pm		= &hisi_zip_pm_ops,
>  };
>  
> +struct pci_driver *hisi_zip_get_pf_driver(void)
> +{
> +	return &hisi_zip_pci_driver;
> +}
> +EXPORT_SYMBOL_GPL(hisi_zip_get_pf_driver);
> +
>  static void hisi_zip_register_debugfs(void)
>  {
>  	if (!debugfs_initialized())
> diff --git a/include/linux/hisi_acc_qm.h b/include/linux/hisi_acc_qm.h
> index 6a6477c34666..00f2a4db8723 100644
> --- a/include/linux/hisi_acc_qm.h
> +++ b/include/linux/hisi_acc_qm.h
> @@ -476,4 +476,9 @@ void hisi_qm_pm_init(struct hisi_qm *qm);
>  int hisi_qm_get_dfx_access(struct hisi_qm *qm);
>  void hisi_qm_put_dfx_access(struct hisi_qm *qm);
>  void hisi_qm_regs_dump(struct seq_file *s, struct debugfs_regset32 *regset);
> +
> +/* Used by VFIO ACC live migration driver */
> +struct pci_driver *hisi_sec_get_pf_driver(void);
> +struct pci_driver *hisi_hpre_get_pf_driver(void);
> +struct pci_driver *hisi_zip_get_pf_driver(void);
>  #endif
> 

^ permalink raw reply	[flat|nested] 56+ messages in thread

* Re: [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live migration
  2022-03-08  8:11           ` Tian, Kevin
@ 2022-03-08 19:33             ` Alex Williamson
  2022-03-09 10:11               ` Tian, Kevin
  0 siblings, 1 reply; 56+ messages in thread
From: Alex Williamson @ 2022-03-08 19:33 UTC (permalink / raw)
  To: Tian, Kevin
  Cc: Shameerali Kolothum Thodi, Jason Gunthorpe, kvm, linux-kernel,
	linux-crypto, linux-pci, cohuck, mgurtovoy, yishaih, Linuxarm,
	liulongfang, Zengtao (B), Jonathan Cameron, Wangzhou (B),
	Xu Zaibo

On Tue, 8 Mar 2022 08:11:11 +0000
"Tian, Kevin" <kevin.tian@intel.com> wrote:

> > From: Alex Williamson <alex.williamson@redhat.com>
> > Sent: Tuesday, March 8, 2022 3:53 AM  
> > >  
> > > > I think we still require acks from Bjorn and Zaibo for select patches
> > > > in this series.  
> > >
> > > I checked with Ziabo. He moved projects and is no longer looking into  
> > crypto stuff.  
> > > Wangzhou and LiuLongfang now take care of this. Received acks from  
> > Wangzhou  
> > > already and I will request Longfang to provide his. Hope that's ok.  
> > 
> > Maybe a good time to have them update MAINTAINERS as well.  Thanks,
> >   
> 
> I have one question here (similar to what we discussed for mdev before).
> 
> Now we are adding vendor specific drivers under /drivers/vfio. Two drivers
> on radar and more will come. Then what would be the criteria for 
> accepting such a driver? Do we prefer to a model in which the author should
> provide enough background for vfio community to understand how it works 
> or as done here just rely on the PF driver owner to cover device specific
> code?
> 
> If the former we may need document some process for what information
> is necessary and also need secure increased review bandwidth from key
> reviewers in vfio community.
> 
> If the latter then how can we guarantee no corner case overlooked by both
> sides (i.e. how to know the coverage of total reviews)? Another open is who
> from the PF driver sub-system should be considered as the one to give the
> green signal. If the sub-system maintainer trusts the PF driver owner and
> just pulls commits from him then having the r-b from the PF driver owner is
> sufficient. But if the sub-system maintainer wants to review detail change
> in every underlying driver then we probably also want to get the ack from
> the maintainer.
> 
> Overall I didn't mean to slow down the progress of this series. But above
> does be some puzzle occurred in my review. 😊

Hi Kevin,

Good questions, I'd like a better understanding of expectations as
well.  I think the intentions are the same as any other sub-system, the
drivers make use of shared interfaces and extensions and the role of
the sub-system should be to make sure those interfaces are used
correctly and extensions fit well within the overall design.  However,
just as the network maintainer isn't expected to fully understand every
NIC driver, I think/hope we have the same expectations here.  It's
certainly a benefit to the community and perceived trustworthiness if
each driver outlines its operating model and security nuances, but
those are only ever going to be the nuances identified by the people
who have the access and energy to evaluate the device.

It's going to be up to the community to try to determine that any new
drivers are seriously considering security and not opening any new gaps
relative to behavior using the base vfio-pci driver.  For the driver
examples we have, this seems a bit easier than evaluating an entire
mdev device because they're largely providing direct access to the
device rather than trying to multiplex a shared physical device.  We
can therefore focus on incremental functionality, as both drivers have
done, implementing a boilerplate vendor driver, then adding migration
support.  I imagine this won't always be the case though and some
drivers will re-implement much of the core to support further emulation
and shared resources.

So how do we as a community want to handle this?  I wouldn't mind, I'd
actually welcome, some sort of review requirement for new vfio vendor
driver variants.  Is that reasonable?  What would be the criteria?
Approval from the PF driver owner, if different/necessary, and at least
one unaffiliated reviewer (preferably an active vfio reviewer or
existing vfio variant driver owner/contributor)?  Ideas welcome.
Thanks,

Alex


^ permalink raw reply	[flat|nested] 56+ messages in thread

* RE: [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live migration
  2022-03-08 19:33             ` Alex Williamson
@ 2022-03-09 10:11               ` Tian, Kevin
  2022-03-10 20:49                 ` Alex Williamson
  0 siblings, 1 reply; 56+ messages in thread
From: Tian, Kevin @ 2022-03-09 10:11 UTC (permalink / raw)
  To: Alex Williamson
  Cc: Shameerali Kolothum Thodi, Jason Gunthorpe, kvm, linux-kernel,
	linux-crypto, linux-pci, cohuck, mgurtovoy, yishaih, Linuxarm,
	liulongfang, Zengtao (B), Jonathan Cameron, Wangzhou (B),
	Xu Zaibo

> From: Alex Williamson <alex.williamson@redhat.com>
> Sent: Wednesday, March 9, 2022 3:33 AM
> 
> On Tue, 8 Mar 2022 08:11:11 +0000
> "Tian, Kevin" <kevin.tian@intel.com> wrote:
> 
> > > From: Alex Williamson <alex.williamson@redhat.com>
> > > Sent: Tuesday, March 8, 2022 3:53 AM
> > > >
> > > > > I think we still require acks from Bjorn and Zaibo for select patches
> > > > > in this series.
> > > >
> > > > I checked with Ziabo. He moved projects and is no longer looking into
> > > crypto stuff.
> > > > Wangzhou and LiuLongfang now take care of this. Received acks from
> > > Wangzhou
> > > > already and I will request Longfang to provide his. Hope that's ok.
> > >
> > > Maybe a good time to have them update MAINTAINERS as well.  Thanks,
> > >
> >
> > I have one question here (similar to what we discussed for mdev before).
> >
> > Now we are adding vendor specific drivers under /drivers/vfio. Two drivers
> > on radar and more will come. Then what would be the criteria for
> > accepting such a driver? Do we prefer to a model in which the author
> should
> > provide enough background for vfio community to understand how it
> works
> > or as done here just rely on the PF driver owner to cover device specific
> > code?
> >
> > If the former we may need document some process for what information
> > is necessary and also need secure increased review bandwidth from key
> > reviewers in vfio community.
> >
> > If the latter then how can we guarantee no corner case overlooked by both
> > sides (i.e. how to know the coverage of total reviews)? Another open is
> who
> > from the PF driver sub-system should be considered as the one to give the
> > green signal. If the sub-system maintainer trusts the PF driver owner and
> > just pulls commits from him then having the r-b from the PF driver owner is
> > sufficient. But if the sub-system maintainer wants to review detail change
> > in every underlying driver then we probably also want to get the ack from
> > the maintainer.
> >
> > Overall I didn't mean to slow down the progress of this series. But above
> > does be some puzzle occurred in my review. 😊
> 
> Hi Kevin,
> 
> Good questions, I'd like a better understanding of expectations as
> well.  I think the intentions are the same as any other sub-system, the
> drivers make use of shared interfaces and extensions and the role of
> the sub-system should be to make sure those interfaces are used
> correctly and extensions fit well within the overall design.  However,
> just as the network maintainer isn't expected to fully understand every
> NIC driver, I think/hope we have the same expectations here.  It's
> certainly a benefit to the community and perceived trustworthiness if
> each driver outlines its operating model and security nuances, but
> those are only ever going to be the nuances identified by the people
> who have the access and energy to evaluate the device.
> 
> It's going to be up to the community to try to determine that any new
> drivers are seriously considering security and not opening any new gaps
> relative to behavior using the base vfio-pci driver.  For the driver
> examples we have, this seems a bit easier than evaluating an entire
> mdev device because they're largely providing direct access to the
> device rather than trying to multiplex a shared physical device.  We
> can therefore focus on incremental functionality, as both drivers have
> done, implementing a boilerplate vendor driver, then adding migration
> support.  I imagine this won't always be the case though and some
> drivers will re-implement much of the core to support further emulation
> and shared resources.
> 
> So how do we as a community want to handle this?  I wouldn't mind, I'd
> actually welcome, some sort of review requirement for new vfio vendor
> driver variants.  Is that reasonable?  What would be the criteria?
> Approval from the PF driver owner, if different/necessary, and at least
> one unaffiliated reviewer (preferably an active vfio reviewer or
> existing vfio variant driver owner/contributor)?  Ideas welcome.
> Thanks,
> 

Yes, and the criteria is the hard part. In the end it largely depend on 
the expectations of the reviewers.  

If the unaffiliated reviewer only cares about the usage of shared 
interfaces or extensions as you said then what this series does is
just fine. Such type of review can be easily done via reading code 
and doesn't require detail device knowledge.

On the other hand if the reviewer wants to do a full functional
review of how migration is actually supported for such device, 
whatever information (patch description, code comment, kdoc,
etc.) necessary to build a standalone migration story would be
appreciated, e.g.:

  - What composes the device state?
  - Which portion of the device state is exposed to and managed
    by the user and which is hidden from the user (i.e. controlled
    by the PF driver)?
  - Interface between the vfio driver and the device (and/or PF
    driver) to manage the device state;
  - Rich functional-level comments for the reviewer to dive into
    the migration flow;
  - ...

I guess we don't want to force one model over the other. Just
from my impression the more information the driver can 
provide the more time I'd like to spend on the review. Otherwise
it has to trend to the minimal form i.e. the first model.

and currently I don't have a concrete idea how the 2nd model will
work. maybe it will get clear only when a future driver attracts 
people to do thorough review...

Thanks
Kevin

^ permalink raw reply	[flat|nested] 56+ messages in thread

* RE: [PATCH v8 3/9] hisi_acc_qm: Move VF PCI device IDs to common header
  2022-03-07 17:53   ` Alex Williamson
@ 2022-03-10 13:55     ` Shameerali Kolothum Thodi
  0 siblings, 0 replies; 56+ messages in thread
From: Shameerali Kolothum Thodi @ 2022-03-10 13:55 UTC (permalink / raw)
  To: Alex Williamson, Bjorn Helgaas, helgaas
  Cc: kvm, linux-kernel, linux-crypto, linux-pci, jgg, cohuck,
	mgurtovoy, yishaih, Linuxarm, liulongfang, Zengtao (B),
	Jonathan Cameron, Wangzhou (B)

Hi Bjorn,

> -----Original Message-----
> From: Alex Williamson [mailto:alex.williamson@redhat.com]
> Sent: 07 March 2022 17:54
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>;
> Bjorn Helgaas <bhelgaas@google.com>
> Cc: kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-crypto@vger.kernel.org; linux-pci@vger.kernel.org; jgg@nvidia.com;
> cohuck@redhat.com; mgurtovoy@nvidia.com; yishaih@nvidia.com; Linuxarm
> <linuxarm@huawei.com>; liulongfang <liulongfang@huawei.com>; Zengtao (B)
> <prime.zeng@hisilicon.com>; Jonathan Cameron
> <jonathan.cameron@huawei.com>; Wangzhou (B) <wangzhou1@hisilicon.com>
> Subject: Re: [PATCH v8 3/9] hisi_acc_qm: Move VF PCI device IDs to common
> header
> 
> Hi Bjorn,
> 
> Here's the respin of this patch that adds only the VF device IDs to
> pci_ids.h.  The next patch in the series[1] adds a consumer of these
> IDs as a vfio-pci vendor driver.  Thanks,

Just a gentle ping on this. Also the latest respin is now at v9 and can be
found here.

https://lore.kernel.org/kvm/20220308184902.2242-4-shameerali.kolothum.thodi@huawei.com/

Thanks,
Shameer

> Alex
> 
> [1]https://lore.kernel.org/all/20220303230131.2103-5-shameerali.kolothum.t
> hodi@huawei.com/
> 
> On Thu, 3 Mar 2022 23:01:25 +0000
> Shameer Kolothum <shameerali.kolothum.thodi@huawei.com> wrote:
> 
> > Move the PCI Device IDs of HiSilicon ACC VF devices to a common header
> > and also use a uniform naming convention.
> >
> > This will be useful when we introduce the vfio PCI HiSilicon ACC live
> > migration driver in subsequent patches.
> >
> > Signed-off-by: Shameer Kolothum
> <shameerali.kolothum.thodi@huawei.com>
> > ---
> >  drivers/crypto/hisilicon/hpre/hpre_main.c | 13 ++++++-------
> >  drivers/crypto/hisilicon/sec2/sec_main.c  | 15 +++++++--------
> >  drivers/crypto/hisilicon/zip/zip_main.c   | 11 +++++------
> >  include/linux/pci_ids.h                   |  3 +++
> >  4 files changed, 21 insertions(+), 21 deletions(-)
> >
> > diff --git a/drivers/crypto/hisilicon/hpre/hpre_main.c
> b/drivers/crypto/hisilicon/hpre/hpre_main.c
> > index ebfab3e14499..3589d8879b5e 100644
> > --- a/drivers/crypto/hisilicon/hpre/hpre_main.c
> > +++ b/drivers/crypto/hisilicon/hpre/hpre_main.c
> > @@ -68,8 +68,7 @@
> >  #define HPRE_REG_RD_INTVRL_US		10
> >  #define HPRE_REG_RD_TMOUT_US		1000
> >  #define HPRE_DBGFS_VAL_MAX_LEN		20
> > -#define HPRE_PCI_DEVICE_ID		0xa258
> > -#define HPRE_PCI_VF_DEVICE_ID		0xa259
> > +#define PCI_DEVICE_ID_HUAWEI_HPRE_PF	0xa258
> >  #define HPRE_QM_USR_CFG_MASK		GENMASK(31, 1)
> >  #define HPRE_QM_AXI_CFG_MASK		GENMASK(15, 0)
> >  #define HPRE_QM_VFG_AX_MASK		GENMASK(7, 0)
> > @@ -111,8 +110,8 @@
> >  static const char hpre_name[] = "hisi_hpre";
> >  static struct dentry *hpre_debugfs_root;
> >  static const struct pci_device_id hpre_dev_ids[] = {
> > -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, HPRE_PCI_DEVICE_ID) },
> > -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, HPRE_PCI_VF_DEVICE_ID) },
> > +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI,
> PCI_DEVICE_ID_HUAWEI_HPRE_PF) },
> > +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI,
> PCI_DEVICE_ID_HUAWEI_HPRE_VF) },
> >  	{ 0, }
> >  };
> >
> > @@ -242,7 +241,7 @@ MODULE_PARM_DESC(uacce_mode,
> UACCE_MODE_DESC);
> >
> >  static int pf_q_num_set(const char *val, const struct kernel_param *kp)
> >  {
> > -	return q_num_set(val, kp, HPRE_PCI_DEVICE_ID);
> > +	return q_num_set(val, kp, PCI_DEVICE_ID_HUAWEI_HPRE_PF);
> >  }
> >
> >  static const struct kernel_param_ops hpre_pf_q_num_ops = {
> > @@ -921,7 +920,7 @@ static int hpre_debugfs_init(struct hisi_qm *qm)
> >  	qm->debug.sqe_mask_len = HPRE_SQE_MASK_LEN;
> >  	hisi_qm_debug_init(qm);
> >
> > -	if (qm->pdev->device == HPRE_PCI_DEVICE_ID) {
> > +	if (qm->pdev->device == PCI_DEVICE_ID_HUAWEI_HPRE_PF) {
> >  		ret = hpre_ctrl_debug_init(qm);
> >  		if (ret)
> >  			goto failed_to_create;
> > @@ -958,7 +957,7 @@ static int hpre_qm_init(struct hisi_qm *qm, struct
> pci_dev *pdev)
> >  	qm->sqe_size = HPRE_SQE_SIZE;
> >  	qm->dev_name = hpre_name;
> >
> > -	qm->fun_type = (pdev->device == HPRE_PCI_DEVICE_ID) ?
> > +	qm->fun_type = (pdev->device == PCI_DEVICE_ID_HUAWEI_HPRE_PF) ?
> >  			QM_HW_PF : QM_HW_VF;
> >  	if (qm->fun_type == QM_HW_PF) {
> >  		qm->qp_base = HPRE_PF_DEF_Q_BASE;
> > diff --git a/drivers/crypto/hisilicon/sec2/sec_main.c
> b/drivers/crypto/hisilicon/sec2/sec_main.c
> > index 26d3ab1d308b..311a8747b5bf 100644
> > --- a/drivers/crypto/hisilicon/sec2/sec_main.c
> > +++ b/drivers/crypto/hisilicon/sec2/sec_main.c
> > @@ -20,8 +20,7 @@
> >
> >  #define SEC_VF_NUM			63
> >  #define SEC_QUEUE_NUM_V1		4096
> > -#define SEC_PF_PCI_DEVICE_ID		0xa255
> > -#define SEC_VF_PCI_DEVICE_ID		0xa256
> > +#define PCI_DEVICE_ID_HUAWEI_SEC_PF	0xa255
> >
> >  #define SEC_BD_ERR_CHK_EN0		0xEFFFFFFF
> >  #define SEC_BD_ERR_CHK_EN1		0x7ffff7fd
> > @@ -225,7 +224,7 @@ static const struct debugfs_reg32 sec_dfx_regs[] = {
> >
> >  static int sec_pf_q_num_set(const char *val, const struct kernel_param
> *kp)
> >  {
> > -	return q_num_set(val, kp, SEC_PF_PCI_DEVICE_ID);
> > +	return q_num_set(val, kp, PCI_DEVICE_ID_HUAWEI_SEC_PF);
> >  }
> >
> >  static const struct kernel_param_ops sec_pf_q_num_ops = {
> > @@ -313,8 +312,8 @@ module_param_cb(uacce_mode,
> &sec_uacce_mode_ops, &uacce_mode, 0444);
> >  MODULE_PARM_DESC(uacce_mode, UACCE_MODE_DESC);
> >
> >  static const struct pci_device_id sec_dev_ids[] = {
> > -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, SEC_PF_PCI_DEVICE_ID) },
> > -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, SEC_VF_PCI_DEVICE_ID) },
> > +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI,
> PCI_DEVICE_ID_HUAWEI_SEC_PF) },
> > +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI,
> PCI_DEVICE_ID_HUAWEI_SEC_VF) },
> >  	{ 0, }
> >  };
> >  MODULE_DEVICE_TABLE(pci, sec_dev_ids);
> > @@ -717,7 +716,7 @@ static int sec_core_debug_init(struct hisi_qm *qm)
> >  	regset->base = qm->io_base;
> >  	regset->dev = dev;
> >
> > -	if (qm->pdev->device == SEC_PF_PCI_DEVICE_ID)
> > +	if (qm->pdev->device == PCI_DEVICE_ID_HUAWEI_SEC_PF)
> >  		debugfs_create_file("regs", 0444, tmp_d, regset, &sec_regs_fops);
> >
> >  	for (i = 0; i < ARRAY_SIZE(sec_dfx_labels); i++) {
> > @@ -735,7 +734,7 @@ static int sec_debug_init(struct hisi_qm *qm)
> >  	struct sec_dev *sec = container_of(qm, struct sec_dev, qm);
> >  	int i;
> >
> > -	if (qm->pdev->device == SEC_PF_PCI_DEVICE_ID) {
> > +	if (qm->pdev->device == PCI_DEVICE_ID_HUAWEI_SEC_PF) {
> >  		for (i = SEC_CLEAR_ENABLE; i < SEC_DEBUG_FILE_NUM; i++) {
> >  			spin_lock_init(&sec->debug.files[i].lock);
> >  			sec->debug.files[i].index = i;
> > @@ -877,7 +876,7 @@ static int sec_qm_init(struct hisi_qm *qm, struct
> pci_dev *pdev)
> >  	qm->sqe_size = SEC_SQE_SIZE;
> >  	qm->dev_name = sec_name;
> >
> > -	qm->fun_type = (pdev->device == SEC_PF_PCI_DEVICE_ID) ?
> > +	qm->fun_type = (pdev->device == PCI_DEVICE_ID_HUAWEI_SEC_PF) ?
> >  			QM_HW_PF : QM_HW_VF;
> >  	if (qm->fun_type == QM_HW_PF) {
> >  		qm->qp_base = SEC_PF_DEF_Q_BASE;
> > diff --git a/drivers/crypto/hisilicon/zip/zip_main.c
> b/drivers/crypto/hisilicon/zip/zip_main.c
> > index 678f8b58ec42..66decfe07282 100644
> > --- a/drivers/crypto/hisilicon/zip/zip_main.c
> > +++ b/drivers/crypto/hisilicon/zip/zip_main.c
> > @@ -15,8 +15,7 @@
> >  #include <linux/uacce.h>
> >  #include "zip.h"
> >
> > -#define PCI_DEVICE_ID_ZIP_PF		0xa250
> > -#define PCI_DEVICE_ID_ZIP_VF		0xa251
> > +#define PCI_DEVICE_ID_HUAWEI_ZIP_PF	0xa250
> >
> >  #define HZIP_QUEUE_NUM_V1		4096
> >
> > @@ -246,7 +245,7 @@ MODULE_PARM_DESC(uacce_mode,
> UACCE_MODE_DESC);
> >
> >  static int pf_q_num_set(const char *val, const struct kernel_param *kp)
> >  {
> > -	return q_num_set(val, kp, PCI_DEVICE_ID_ZIP_PF);
> > +	return q_num_set(val, kp, PCI_DEVICE_ID_HUAWEI_ZIP_PF);
> >  }
> >
> >  static const struct kernel_param_ops pf_q_num_ops = {
> > @@ -268,8 +267,8 @@ module_param_cb(vfs_num, &vfs_num_ops,
> &vfs_num, 0444);
> >  MODULE_PARM_DESC(vfs_num, "Number of VFs to enable(1-63),
> 0(default)");
> >
> >  static const struct pci_device_id hisi_zip_dev_ids[] = {
> > -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_ZIP_PF) },
> > -	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_ZIP_VF) },
> > +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI,
> PCI_DEVICE_ID_HUAWEI_ZIP_PF) },
> > +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI,
> PCI_DEVICE_ID_HUAWEI_ZIP_VF) },
> >  	{ 0, }
> >  };
> >  MODULE_DEVICE_TABLE(pci, hisi_zip_dev_ids);
> > @@ -838,7 +837,7 @@ static int hisi_zip_qm_init(struct hisi_qm *qm, struct
> pci_dev *pdev)
> >  	qm->sqe_size = HZIP_SQE_SIZE;
> >  	qm->dev_name = hisi_zip_name;
> >
> > -	qm->fun_type = (pdev->device == PCI_DEVICE_ID_ZIP_PF) ?
> > +	qm->fun_type = (pdev->device == PCI_DEVICE_ID_HUAWEI_ZIP_PF) ?
> >  			QM_HW_PF : QM_HW_VF;
> >  	if (qm->fun_type == QM_HW_PF) {
> >  		qm->qp_base = HZIP_PF_DEF_Q_BASE;
> > diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
> > index aad54c666407..31dee2b65a62 100644
> > --- a/include/linux/pci_ids.h
> > +++ b/include/linux/pci_ids.h
> > @@ -2529,6 +2529,9 @@
> >  #define PCI_DEVICE_ID_KORENIX_JETCARDF3	0x17ff
> >
> >  #define PCI_VENDOR_ID_HUAWEI		0x19e5
> > +#define PCI_DEVICE_ID_HUAWEI_ZIP_VF	0xa251
> > +#define PCI_DEVICE_ID_HUAWEI_SEC_VF	0xa256
> > +#define PCI_DEVICE_ID_HUAWEI_HPRE_VF	0xa259
> >
> >  #define PCI_VENDOR_ID_NETRONOME		0x19ee
> >  #define PCI_DEVICE_ID_NETRONOME_NFP4000	0x4000


^ permalink raw reply	[flat|nested] 56+ messages in thread

* Re: [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live migration
  2022-03-09 10:11               ` Tian, Kevin
@ 2022-03-10 20:49                 ` Alex Williamson
  2022-03-11  8:52                   ` Cornelia Huck
                                     ` (3 more replies)
  0 siblings, 4 replies; 56+ messages in thread
From: Alex Williamson @ 2022-03-10 20:49 UTC (permalink / raw)
  To: Tian, Kevin
  Cc: Shameerali Kolothum Thodi, Jason Gunthorpe, kvm, linux-kernel,
	linux-crypto, linux-pci, cohuck, mgurtovoy, yishaih, Linuxarm,
	liulongfang, Zengtao (B), Jonathan Cameron, Wangzhou (B),
	Xu Zaibo

On Wed, 9 Mar 2022 10:11:06 +0000
"Tian, Kevin" <kevin.tian@intel.com> wrote:

> > From: Alex Williamson <alex.williamson@redhat.com>
> > Sent: Wednesday, March 9, 2022 3:33 AM
> > 
> > On Tue, 8 Mar 2022 08:11:11 +0000
> > "Tian, Kevin" <kevin.tian@intel.com> wrote:
> >   
> > > > From: Alex Williamson <alex.williamson@redhat.com>
> > > > Sent: Tuesday, March 8, 2022 3:53 AM  
> > > > >  
> > > > > > I think we still require acks from Bjorn and Zaibo for select patches
> > > > > > in this series.  
> > > > >
> > > > > I checked with Ziabo. He moved projects and is no longer looking into  
> > > > crypto stuff.  
> > > > > Wangzhou and LiuLongfang now take care of this. Received acks from  
> > > > Wangzhou  
> > > > > already and I will request Longfang to provide his. Hope that's ok.  
> > > >
> > > > Maybe a good time to have them update MAINTAINERS as well.  Thanks,
> > > >  
> > >
> > > I have one question here (similar to what we discussed for mdev before).
> > >
> > > Now we are adding vendor specific drivers under /drivers/vfio. Two drivers
> > > on radar and more will come. Then what would be the criteria for
> > > accepting such a driver? Do we prefer to a model in which the author  
> > should  
> > > provide enough background for vfio community to understand how it  
> > works  
> > > or as done here just rely on the PF driver owner to cover device specific
> > > code?
> > >
> > > If the former we may need document some process for what information
> > > is necessary and also need secure increased review bandwidth from key
> > > reviewers in vfio community.
> > >
> > > If the latter then how can we guarantee no corner case overlooked by both
> > > sides (i.e. how to know the coverage of total reviews)? Another open is  
> > who  
> > > from the PF driver sub-system should be considered as the one to give the
> > > green signal. If the sub-system maintainer trusts the PF driver owner and
> > > just pulls commits from him then having the r-b from the PF driver owner is
> > > sufficient. But if the sub-system maintainer wants to review detail change
> > > in every underlying driver then we probably also want to get the ack from
> > > the maintainer.
> > >
> > > Overall I didn't mean to slow down the progress of this series. But above
> > > does be some puzzle occurred in my review. 😊  
> > 
> > Hi Kevin,
> > 
> > Good questions, I'd like a better understanding of expectations as
> > well.  I think the intentions are the same as any other sub-system, the
> > drivers make use of shared interfaces and extensions and the role of
> > the sub-system should be to make sure those interfaces are used
> > correctly and extensions fit well within the overall design.  However,
> > just as the network maintainer isn't expected to fully understand every
> > NIC driver, I think/hope we have the same expectations here.  It's
> > certainly a benefit to the community and perceived trustworthiness if
> > each driver outlines its operating model and security nuances, but
> > those are only ever going to be the nuances identified by the people
> > who have the access and energy to evaluate the device.
> > 
> > It's going to be up to the community to try to determine that any new
> > drivers are seriously considering security and not opening any new gaps
> > relative to behavior using the base vfio-pci driver.  For the driver
> > examples we have, this seems a bit easier than evaluating an entire
> > mdev device because they're largely providing direct access to the
> > device rather than trying to multiplex a shared physical device.  We
> > can therefore focus on incremental functionality, as both drivers have
> > done, implementing a boilerplate vendor driver, then adding migration
> > support.  I imagine this won't always be the case though and some
> > drivers will re-implement much of the core to support further emulation
> > and shared resources.
> > 
> > So how do we as a community want to handle this?  I wouldn't mind, I'd
> > actually welcome, some sort of review requirement for new vfio vendor
> > driver variants.  Is that reasonable?  What would be the criteria?
> > Approval from the PF driver owner, if different/necessary, and at least
> > one unaffiliated reviewer (preferably an active vfio reviewer or
> > existing vfio variant driver owner/contributor)?  Ideas welcome.
> > Thanks,
> >   
> 
> Yes, and the criteria is the hard part. In the end it largely depend on 
> the expectations of the reviewers.  
> 
> If the unaffiliated reviewer only cares about the usage of shared 
> interfaces or extensions as you said then what this series does is
> just fine. Such type of review can be easily done via reading code 
> and doesn't require detail device knowledge.
> 
> On the other hand if the reviewer wants to do a full functional
> review of how migration is actually supported for such device, 
> whatever information (patch description, code comment, kdoc,
> etc.) necessary to build a standalone migration story would be
> appreciated, e.g.:
> 
>   - What composes the device state?
>   - Which portion of the device state is exposed to and managed
>     by the user and which is hidden from the user (i.e. controlled
>     by the PF driver)?
>   - Interface between the vfio driver and the device (and/or PF
>     driver) to manage the device state;
>   - Rich functional-level comments for the reviewer to dive into
>     the migration flow;
>   - ...
> 
> I guess we don't want to force one model over the other. Just
> from my impression the more information the driver can 
> provide the more time I'd like to spend on the review. Otherwise
> it has to trend to the minimal form i.e. the first model.
> 
> and currently I don't have a concrete idea how the 2nd model will
> work. maybe it will get clear only when a future driver attracts 
> people to do thorough review...

Do you think we should go so far as to formalize this via a MAINTAINERS
entry, for example:

diff --git a/Documentation/vfio/vfio-pci-vendor-driver-acceptance.rst b/Documentation/vfio/vfio-pci-vendor-driver-acceptance.rst
new file mode 100644
index 000000000000..54ebafcdd735
--- /dev/null
+++ b/Documentation/vfio/vfio-pci-vendor-driver-acceptance.rst
@@ -0,0 +1,35 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+Acceptance criteria for vfio-pci device specific driver variants
+================================================================
+
+Overview
+--------
+The vfio-pci driver exists as a device agnostic driver using the
+system IOMMU and relying on the robustness of platform fault
+handling to provide isolated device access to userspace.  While the
+vfio-pci driver does include some device specific support, further
+extensions for yet more advanced device specific features are not
+sustainable.  The vfio-pci driver has therefore split out
+vfio-pci-core as a library that may be reused to implement features
+requiring device specific knowledge, ex. saving and loading device
+state for the purposes of supporting migration.
+
+In support of such features, it's expected that some device specific
+variants may interact with parent devices (ex. SR-IOV PF in support of
+a user assigned VF) or other extensions that may not be otherwise
+accessible via the vfio-pci base driver.  Authors of such drivers
+should be diligent not to create exploitable interfaces via such
+interactions or allow unchecked userspace data to have an effect
+beyond the scope of the assigned device.
+
+New driver submissions are therefore requested to have approval via
+Sign-off for any interactions with parent drivers.  Additionally,
+drivers should make an attempt to provide sufficient documentation
+for reviewers to understand the device specific extensions, for
+example in the case of migration data, how is the device state
+composed and consumed, which portions are not otherwise available to
+the user via vfio-pci, what safeguards exist to validate the data,
+etc.  To that extent, authors should additionally expect to require
+reviews from at least one of the listed reviewers, in addition to the
+overall vfio maintainer.
diff --git a/MAINTAINERS b/MAINTAINERS
index 4322b5321891..4f7d26f9aac6 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -20314,6 +20314,13 @@ F:	drivers/vfio/mdev/
 F:	include/linux/mdev.h
 F:	samples/vfio-mdev/
 
+VFIO PCI VENDOR DRIVERS
+R:	Your Name <your.name@here.com>
+L:	kvm@vger.kernel.org
+S:	Maintained
+P:	Documentation/vfio/vfio-pci-vendor-driver-acceptance.rst
+F:	drivers/vfio/pci/*/
+
 VFIO PLATFORM DRIVER
 M:	Eric Auger <eric.auger@redhat.com>
 L:	kvm@vger.kernel.org

Ideally we'd have at least Yishai, Shameer, Jason, and yourself listed
as reviewers (Connie and I are included via the higher level entry).
Thoughts from anyone?  Volunteers for reviewers if we want to press
forward with this as formal acceptance criteria?  Thanks,

Alex


^ permalink raw reply related	[flat|nested] 56+ messages in thread

* Re: [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live migration
  2022-03-10 20:49                 ` Alex Williamson
@ 2022-03-11  8:52                   ` Cornelia Huck
  2022-03-11 13:21                   ` Shameerali Kolothum Thodi
                                     ` (2 subsequent siblings)
  3 siblings, 0 replies; 56+ messages in thread
From: Cornelia Huck @ 2022-03-11  8:52 UTC (permalink / raw)
  To: Alex Williamson, Tian, Kevin
  Cc: Shameerali Kolothum Thodi, Jason Gunthorpe, kvm, linux-kernel,
	linux-crypto, linux-pci, mgurtovoy, yishaih, Linuxarm,
	liulongfang, Zengtao (B), Jonathan Cameron, Wangzhou (B),
	Xu Zaibo

On Thu, Mar 10 2022, Alex Williamson <alex.williamson@redhat.com> wrote:

> Do you think we should go so far as to formalize this via a MAINTAINERS
> entry, for example:
>
> diff --git a/Documentation/vfio/vfio-pci-vendor-driver-acceptance.rst b/Documentation/vfio/vfio-pci-vendor-driver-acceptance.rst
> new file mode 100644
> index 000000000000..54ebafcdd735
> --- /dev/null
> +++ b/Documentation/vfio/vfio-pci-vendor-driver-acceptance.rst
> @@ -0,0 +1,35 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +Acceptance criteria for vfio-pci device specific driver variants
> +================================================================
> +
> +Overview
> +--------
> +The vfio-pci driver exists as a device agnostic driver using the
> +system IOMMU and relying on the robustness of platform fault
> +handling to provide isolated device access to userspace.  While the
> +vfio-pci driver does include some device specific support, further
> +extensions for yet more advanced device specific features are not
> +sustainable.  The vfio-pci driver has therefore split out
> +vfio-pci-core as a library that may be reused to implement features
> +requiring device specific knowledge, ex. saving and loading device
> +state for the purposes of supporting migration.
> +
> +In support of such features, it's expected that some device specific
> +variants may interact with parent devices (ex. SR-IOV PF in support of
> +a user assigned VF) or other extensions that may not be otherwise
> +accessible via the vfio-pci base driver.  Authors of such drivers
> +should be diligent not to create exploitable interfaces via such
> +interactions or allow unchecked userspace data to have an effect
> +beyond the scope of the assigned device.
> +
> +New driver submissions are therefore requested to have approval via
> +Sign-off for any interactions with parent drivers.  Additionally,
> +drivers should make an attempt to provide sufficient documentation
> +for reviewers to understand the device specific extensions, for
> +example in the case of migration data, how is the device state
> +composed and consumed, which portions are not otherwise available to
> +the user via vfio-pci, what safeguards exist to validate the data,
> +etc.  To that extent, authors should additionally expect to require
> +reviews from at least one of the listed reviewers, in addition to the
> +overall vfio maintainer.
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 4322b5321891..4f7d26f9aac6 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -20314,6 +20314,13 @@ F:	drivers/vfio/mdev/
>  F:	include/linux/mdev.h
>  F:	samples/vfio-mdev/
>  
> +VFIO PCI VENDOR DRIVERS
> +R:	Your Name <your.name@here.com>
> +L:	kvm@vger.kernel.org
> +S:	Maintained
> +P:	Documentation/vfio/vfio-pci-vendor-driver-acceptance.rst
> +F:	drivers/vfio/pci/*/

This works as long as the only subdirectories are for vendor drivers;
should something else come up, we'd need to add an exclude statement, so
no biggie.

> +
>  VFIO PLATFORM DRIVER
>  M:	Eric Auger <eric.auger@redhat.com>
>  L:	kvm@vger.kernel.org
>
> Ideally we'd have at least Yishai, Shameer, Jason, and yourself listed
> as reviewers (Connie and I are included via the higher level entry).
> Thoughts from anyone?  Volunteers for reviewers if we want to press
> forward with this as formal acceptance criteria?  Thanks,
>
> Alex

I like having this formalized. More eyeballs are good (especially as
getting good review is one of the worst bottlenecks), and I'd trust
people having worked on other vendor drivers having a better grip on
issues that have not been my priority.


^ permalink raw reply	[flat|nested] 56+ messages in thread

* RE: [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live migration
  2022-03-10 20:49                 ` Alex Williamson
  2022-03-11  8:52                   ` Cornelia Huck
@ 2022-03-11 13:21                   ` Shameerali Kolothum Thodi
  2022-03-14  3:40                   ` Tian, Kevin
  2022-03-14 15:03                   ` Jason Gunthorpe
  3 siblings, 0 replies; 56+ messages in thread
From: Shameerali Kolothum Thodi @ 2022-03-11 13:21 UTC (permalink / raw)
  To: Alex Williamson, Tian, Kevin
  Cc: Jason Gunthorpe, kvm, linux-kernel, linux-crypto, linux-pci,
	cohuck, mgurtovoy, yishaih, Linuxarm, liulongfang, Zengtao (B),
	Jonathan Cameron, Wangzhou (B),
	Xu Zaibo



> -----Original Message-----
> From: Alex Williamson [mailto:alex.williamson@redhat.com]
> Sent: 10 March 2022 20:50
> To: Tian, Kevin <kevin.tian@intel.com>
> Cc: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>;
> Jason Gunthorpe <jgg@nvidia.com>; kvm@vger.kernel.org;
> linux-kernel@vger.kernel.org; linux-crypto@vger.kernel.org;
> linux-pci@vger.kernel.org; cohuck@redhat.com; mgurtovoy@nvidia.com;
> yishaih@nvidia.com; Linuxarm <linuxarm@huawei.com>; liulongfang
> <liulongfang@huawei.com>; Zengtao (B) <prime.zeng@hisilicon.com>;
> Jonathan Cameron <jonathan.cameron@huawei.com>; Wangzhou (B)
> <wangzhou1@hisilicon.com>; Xu Zaibo <xuzaibo@huawei.com>
> Subject: Re: [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live
> migration
> 
> On Wed, 9 Mar 2022 10:11:06 +0000
> "Tian, Kevin" <kevin.tian@intel.com> wrote:
> 
> > > From: Alex Williamson <alex.williamson@redhat.com>
> > > Sent: Wednesday, March 9, 2022 3:33 AM
> > >
> > > On Tue, 8 Mar 2022 08:11:11 +0000
> > > "Tian, Kevin" <kevin.tian@intel.com> wrote:
> > >
> > > > > From: Alex Williamson <alex.williamson@redhat.com>
> > > > > Sent: Tuesday, March 8, 2022 3:53 AM
> > > > > >
> > > > > > > I think we still require acks from Bjorn and Zaibo for select patches
> > > > > > > in this series.
> > > > > >
> > > > > > I checked with Ziabo. He moved projects and is no longer looking into
> > > > > crypto stuff.
> > > > > > Wangzhou and LiuLongfang now take care of this. Received acks from
> > > > > Wangzhou
> > > > > > already and I will request Longfang to provide his. Hope that's ok.
> > > > >
> > > > > Maybe a good time to have them update MAINTAINERS as well.
> Thanks,
> > > > >
> > > >
> > > > I have one question here (similar to what we discussed for mdev before).
> > > >
> > > > Now we are adding vendor specific drivers under /drivers/vfio. Two
> drivers
> > > > on radar and more will come. Then what would be the criteria for
> > > > accepting such a driver? Do we prefer to a model in which the author
> > > should
> > > > provide enough background for vfio community to understand how it
> > > works
> > > > or as done here just rely on the PF driver owner to cover device specific
> > > > code?
> > > >
> > > > If the former we may need document some process for what information
> > > > is necessary and also need secure increased review bandwidth from key
> > > > reviewers in vfio community.
> > > >
> > > > If the latter then how can we guarantee no corner case overlooked by
> both
> > > > sides (i.e. how to know the coverage of total reviews)? Another open is
> > > who
> > > > from the PF driver sub-system should be considered as the one to give
> the
> > > > green signal. If the sub-system maintainer trusts the PF driver owner and
> > > > just pulls commits from him then having the r-b from the PF driver owner
> is
> > > > sufficient. But if the sub-system maintainer wants to review detail change
> > > > in every underlying driver then we probably also want to get the ack
> from
> > > > the maintainer.
> > > >
> > > > Overall I didn't mean to slow down the progress of this series. But above
> > > > does be some puzzle occurred in my review. 😊
> > >
> > > Hi Kevin,
> > >
> > > Good questions, I'd like a better understanding of expectations as
> > > well.  I think the intentions are the same as any other sub-system, the
> > > drivers make use of shared interfaces and extensions and the role of
> > > the sub-system should be to make sure those interfaces are used
> > > correctly and extensions fit well within the overall design.  However,
> > > just as the network maintainer isn't expected to fully understand every
> > > NIC driver, I think/hope we have the same expectations here.  It's
> > > certainly a benefit to the community and perceived trustworthiness if
> > > each driver outlines its operating model and security nuances, but
> > > those are only ever going to be the nuances identified by the people
> > > who have the access and energy to evaluate the device.
> > >
> > > It's going to be up to the community to try to determine that any new
> > > drivers are seriously considering security and not opening any new gaps
> > > relative to behavior using the base vfio-pci driver.  For the driver
> > > examples we have, this seems a bit easier than evaluating an entire
> > > mdev device because they're largely providing direct access to the
> > > device rather than trying to multiplex a shared physical device.  We
> > > can therefore focus on incremental functionality, as both drivers have
> > > done, implementing a boilerplate vendor driver, then adding migration
> > > support.  I imagine this won't always be the case though and some
> > > drivers will re-implement much of the core to support further emulation
> > > and shared resources.
> > >
> > > So how do we as a community want to handle this?  I wouldn't mind, I'd
> > > actually welcome, some sort of review requirement for new vfio vendor
> > > driver variants.  Is that reasonable?  What would be the criteria?
> > > Approval from the PF driver owner, if different/necessary, and at least
> > > one unaffiliated reviewer (preferably an active vfio reviewer or
> > > existing vfio variant driver owner/contributor)?  Ideas welcome.
> > > Thanks,
> > >
> >
> > Yes, and the criteria is the hard part. In the end it largely depend on
> > the expectations of the reviewers.
> >
> > If the unaffiliated reviewer only cares about the usage of shared
> > interfaces or extensions as you said then what this series does is
> > just fine. Such type of review can be easily done via reading code
> > and doesn't require detail device knowledge.
> >
> > On the other hand if the reviewer wants to do a full functional
> > review of how migration is actually supported for such device,
> > whatever information (patch description, code comment, kdoc,
> > etc.) necessary to build a standalone migration story would be
> > appreciated, e.g.:
> >
> >   - What composes the device state?
> >   - Which portion of the device state is exposed to and managed
> >     by the user and which is hidden from the user (i.e. controlled
> >     by the PF driver)?
> >   - Interface between the vfio driver and the device (and/or PF
> >     driver) to manage the device state;
> >   - Rich functional-level comments for the reviewer to dive into
> >     the migration flow;
> >   - ...
> >
> > I guess we don't want to force one model over the other. Just
> > from my impression the more information the driver can
> > provide the more time I'd like to spend on the review. Otherwise
> > it has to trend to the minimal form i.e. the first model.
> >
> > and currently I don't have a concrete idea how the 2nd model will
> > work. maybe it will get clear only when a future driver attracts
> > people to do thorough review...
> 
> Do you think we should go so far as to formalize this via a MAINTAINERS
> entry, for example:
> 
> diff --git a/Documentation/vfio/vfio-pci-vendor-driver-acceptance.rst
> b/Documentation/vfio/vfio-pci-vendor-driver-acceptance.rst
> new file mode 100644
> index 000000000000..54ebafcdd735
> --- /dev/null
> +++ b/Documentation/vfio/vfio-pci-vendor-driver-acceptance.rst
> @@ -0,0 +1,35 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +Acceptance criteria for vfio-pci device specific driver variants
> +================================================================
> +
> +Overview
> +--------
> +The vfio-pci driver exists as a device agnostic driver using the
> +system IOMMU and relying on the robustness of platform fault
> +handling to provide isolated device access to userspace.  While the
> +vfio-pci driver does include some device specific support, further
> +extensions for yet more advanced device specific features are not
> +sustainable.  The vfio-pci driver has therefore split out
> +vfio-pci-core as a library that may be reused to implement features
> +requiring device specific knowledge, ex. saving and loading device
> +state for the purposes of supporting migration.
> +
> +In support of such features, it's expected that some device specific
> +variants may interact with parent devices (ex. SR-IOV PF in support of
> +a user assigned VF) or other extensions that may not be otherwise
> +accessible via the vfio-pci base driver.  Authors of such drivers
> +should be diligent not to create exploitable interfaces via such
> +interactions or allow unchecked userspace data to have an effect
> +beyond the scope of the assigned device.
> +
> +New driver submissions are therefore requested to have approval via
> +Sign-off for any interactions with parent drivers.  Additionally,
> +drivers should make an attempt to provide sufficient documentation
> +for reviewers to understand the device specific extensions, for
> +example in the case of migration data, how is the device state
> +composed and consumed, which portions are not otherwise available to
> +the user via vfio-pci, what safeguards exist to validate the data,
> +etc.  To that extent, authors should additionally expect to require
> +reviews from at least one of the listed reviewers, in addition to the
> +overall vfio maintainer.
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 4322b5321891..4f7d26f9aac6 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -20314,6 +20314,13 @@ F:	drivers/vfio/mdev/
>  F:	include/linux/mdev.h
>  F:	samples/vfio-mdev/
> 
> +VFIO PCI VENDOR DRIVERS
> +R:	Your Name <your.name@here.com>
> +L:	kvm@vger.kernel.org
> +S:	Maintained
> +P:	Documentation/vfio/vfio-pci-vendor-driver-acceptance.rst
> +F:	drivers/vfio/pci/*/
> +
>  VFIO PLATFORM DRIVER
>  M:	Eric Auger <eric.auger@redhat.com>
>  L:	kvm@vger.kernel.org
> 
> Ideally we'd have at least Yishai, Shameer, Jason, and yourself listed
> as reviewers (Connie and I are included via the higher level entry).
> Thoughts from anyone?  Volunteers for reviewers if we want to press
> forward with this as formal acceptance criteria?  Thanks,
> 

Sure. Happy to help with reviews, verifications etc.

Thanks,
Shameer

^ permalink raw reply	[flat|nested] 56+ messages in thread

* RE: [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live migration
  2022-03-10 20:49                 ` Alex Williamson
  2022-03-11  8:52                   ` Cornelia Huck
  2022-03-11 13:21                   ` Shameerali Kolothum Thodi
@ 2022-03-14  3:40                   ` Tian, Kevin
  2022-03-14 15:03                   ` Jason Gunthorpe
  3 siblings, 0 replies; 56+ messages in thread
From: Tian, Kevin @ 2022-03-14  3:40 UTC (permalink / raw)
  To: Alex Williamson
  Cc: Shameerali Kolothum Thodi, Jason Gunthorpe, kvm, linux-kernel,
	linux-crypto, linux-pci, cohuck, mgurtovoy, yishaih, Linuxarm,
	liulongfang, Zengtao (B), Jonathan Cameron, Wangzhou (B),
	Xu Zaibo

> From: Alex Williamson <alex.williamson@redhat.com>
> Sent: Friday, March 11, 2022 4:50 AM
> 
> On Wed, 9 Mar 2022 10:11:06 +0000
> "Tian, Kevin" <kevin.tian@intel.com> wrote:
> 
> > > From: Alex Williamson <alex.williamson@redhat.com>
> > > Sent: Wednesday, March 9, 2022 3:33 AM
> > >
> > > On Tue, 8 Mar 2022 08:11:11 +0000
> > > "Tian, Kevin" <kevin.tian@intel.com> wrote:
> > >
> > > > > From: Alex Williamson <alex.williamson@redhat.com>
> > > > > Sent: Tuesday, March 8, 2022 3:53 AM
> > > > > >
> > > > > > > I think we still require acks from Bjorn and Zaibo for select patches
> > > > > > > in this series.
> > > > > >
> > > > > > I checked with Ziabo. He moved projects and is no longer looking
> into
> > > > > crypto stuff.
> > > > > > Wangzhou and LiuLongfang now take care of this. Received acks
> from
> > > > > Wangzhou
> > > > > > already and I will request Longfang to provide his. Hope that's ok.
> > > > >
> > > > > Maybe a good time to have them update MAINTAINERS as well.
> Thanks,
> > > > >
> > > >
> > > > I have one question here (similar to what we discussed for mdev
> before).
> > > >
> > > > Now we are adding vendor specific drivers under /drivers/vfio. Two
> drivers
> > > > on radar and more will come. Then what would be the criteria for
> > > > accepting such a driver? Do we prefer to a model in which the author
> > > should
> > > > provide enough background for vfio community to understand how it
> > > works
> > > > or as done here just rely on the PF driver owner to cover device specific
> > > > code?
> > > >
> > > > If the former we may need document some process for what
> information
> > > > is necessary and also need secure increased review bandwidth from key
> > > > reviewers in vfio community.
> > > >
> > > > If the latter then how can we guarantee no corner case overlooked by
> both
> > > > sides (i.e. how to know the coverage of total reviews)? Another open is
> > > who
> > > > from the PF driver sub-system should be considered as the one to give
> the
> > > > green signal. If the sub-system maintainer trusts the PF driver owner
> and
> > > > just pulls commits from him then having the r-b from the PF driver
> owner is
> > > > sufficient. But if the sub-system maintainer wants to review detail
> change
> > > > in every underlying driver then we probably also want to get the ack
> from
> > > > the maintainer.
> > > >
> > > > Overall I didn't mean to slow down the progress of this series. But
> above
> > > > does be some puzzle occurred in my review. 😊
> > >
> > > Hi Kevin,
> > >
> > > Good questions, I'd like a better understanding of expectations as
> > > well.  I think the intentions are the same as any other sub-system, the
> > > drivers make use of shared interfaces and extensions and the role of
> > > the sub-system should be to make sure those interfaces are used
> > > correctly and extensions fit well within the overall design.  However,
> > > just as the network maintainer isn't expected to fully understand every
> > > NIC driver, I think/hope we have the same expectations here.  It's
> > > certainly a benefit to the community and perceived trustworthiness if
> > > each driver outlines its operating model and security nuances, but
> > > those are only ever going to be the nuances identified by the people
> > > who have the access and energy to evaluate the device.
> > >
> > > It's going to be up to the community to try to determine that any new
> > > drivers are seriously considering security and not opening any new gaps
> > > relative to behavior using the base vfio-pci driver.  For the driver
> > > examples we have, this seems a bit easier than evaluating an entire
> > > mdev device because they're largely providing direct access to the
> > > device rather than trying to multiplex a shared physical device.  We
> > > can therefore focus on incremental functionality, as both drivers have
> > > done, implementing a boilerplate vendor driver, then adding migration
> > > support.  I imagine this won't always be the case though and some
> > > drivers will re-implement much of the core to support further emulation
> > > and shared resources.
> > >
> > > So how do we as a community want to handle this?  I wouldn't mind, I'd
> > > actually welcome, some sort of review requirement for new vfio vendor
> > > driver variants.  Is that reasonable?  What would be the criteria?
> > > Approval from the PF driver owner, if different/necessary, and at least
> > > one unaffiliated reviewer (preferably an active vfio reviewer or
> > > existing vfio variant driver owner/contributor)?  Ideas welcome.
> > > Thanks,
> > >
> >
> > Yes, and the criteria is the hard part. In the end it largely depend on
> > the expectations of the reviewers.
> >
> > If the unaffiliated reviewer only cares about the usage of shared
> > interfaces or extensions as you said then what this series does is
> > just fine. Such type of review can be easily done via reading code
> > and doesn't require detail device knowledge.
> >
> > On the other hand if the reviewer wants to do a full functional
> > review of how migration is actually supported for such device,
> > whatever information (patch description, code comment, kdoc,
> > etc.) necessary to build a standalone migration story would be
> > appreciated, e.g.:
> >
> >   - What composes the device state?
> >   - Which portion of the device state is exposed to and managed
> >     by the user and which is hidden from the user (i.e. controlled
> >     by the PF driver)?
> >   - Interface between the vfio driver and the device (and/or PF
> >     driver) to manage the device state;
> >   - Rich functional-level comments for the reviewer to dive into
> >     the migration flow;
> >   - ...
> >
> > I guess we don't want to force one model over the other. Just
> > from my impression the more information the driver can
> > provide the more time I'd like to spend on the review. Otherwise
> > it has to trend to the minimal form i.e. the first model.
> >
> > and currently I don't have a concrete idea how the 2nd model will
> > work. maybe it will get clear only when a future driver attracts
> > people to do thorough review...
> 
> Do you think we should go so far as to formalize this via a MAINTAINERS
> entry, for example:
> 
> diff --git a/Documentation/vfio/vfio-pci-vendor-driver-acceptance.rst
> b/Documentation/vfio/vfio-pci-vendor-driver-acceptance.rst
> new file mode 100644
> index 000000000000..54ebafcdd735
> --- /dev/null
> +++ b/Documentation/vfio/vfio-pci-vendor-driver-acceptance.rst
> @@ -0,0 +1,35 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +Acceptance criteria for vfio-pci device specific driver variants
> +===============================================================
> =
> +
> +Overview
> +--------
> +The vfio-pci driver exists as a device agnostic driver using the
> +system IOMMU and relying on the robustness of platform fault
> +handling to provide isolated device access to userspace.  While the
> +vfio-pci driver does include some device specific support, further
> +extensions for yet more advanced device specific features are not
> +sustainable.  The vfio-pci driver has therefore split out
> +vfio-pci-core as a library that may be reused to implement features
> +requiring device specific knowledge, ex. saving and loading device
> +state for the purposes of supporting migration.
> +
> +In support of such features, it's expected that some device specific
> +variants may interact with parent devices (ex. SR-IOV PF in support of
> +a user assigned VF) or other extensions that may not be otherwise
> +accessible via the vfio-pci base driver.  Authors of such drivers
> +should be diligent not to create exploitable interfaces via such
> +interactions or allow unchecked userspace data to have an effect
> +beyond the scope of the assigned device.
> +
> +New driver submissions are therefore requested to have approval via
> +Sign-off for any interactions with parent drivers.  Additionally,
> +drivers should make an attempt to provide sufficient documentation
> +for reviewers to understand the device specific extensions, for
> +example in the case of migration data, how is the device state
> +composed and consumed, which portions are not otherwise available to
> +the user via vfio-pci, what safeguards exist to validate the data,
> +etc.  To that extent, authors should additionally expect to require
> +reviews from at least one of the listed reviewers, in addition to the
> +overall vfio maintainer.
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 4322b5321891..4f7d26f9aac6 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -20314,6 +20314,13 @@ F:	drivers/vfio/mdev/
>  F:	include/linux/mdev.h
>  F:	samples/vfio-mdev/
> 
> +VFIO PCI VENDOR DRIVERS
> +R:	Your Name <your.name@here.com>
> +L:	kvm@vger.kernel.org
> +S:	Maintained
> +P:	Documentation/vfio/vfio-pci-vendor-driver-acceptance.rst
> +F:	drivers/vfio/pci/*/
> +
>  VFIO PLATFORM DRIVER
>  M:	Eric Auger <eric.auger@redhat.com>
>  L:	kvm@vger.kernel.org
> 
> Ideally we'd have at least Yishai, Shameer, Jason, and yourself listed
> as reviewers (Connie and I are included via the higher level entry).
> Thoughts from anyone?  Volunteers for reviewers if we want to press
> forward with this as formal acceptance criteria?  Thanks,
> 

Yes, this works for me. Moving forward the kdoc may be expanded
to include certain template/example to demonstrate necessary 
information to be provided by vendor drivers if common review
gaps are repeatedly identified from practice.

Thanks
Kevin

^ permalink raw reply	[flat|nested] 56+ messages in thread

* Re: [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live migration
  2022-03-10 20:49                 ` Alex Williamson
                                     ` (2 preceding siblings ...)
  2022-03-14  3:40                   ` Tian, Kevin
@ 2022-03-14 15:03                   ` Jason Gunthorpe
  3 siblings, 0 replies; 56+ messages in thread
From: Jason Gunthorpe @ 2022-03-14 15:03 UTC (permalink / raw)
  To: Alex Williamson
  Cc: Tian, Kevin, Shameerali Kolothum Thodi, kvm, linux-kernel,
	linux-crypto, linux-pci, cohuck, mgurtovoy, yishaih, Linuxarm,
	liulongfang, Zengtao (B), Jonathan Cameron, Wangzhou (B),
	Xu Zaibo

On Thu, Mar 10, 2022 at 01:49:54PM -0700, Alex Williamson wrote:

> +New driver submissions are therefore requested to have approval via
> +Sign-off for any interactions with parent drivers.  Additionally,

Acked-by/etc.

> +drivers should make an attempt to provide sufficient documentation
> +for reviewers to understand the device specific extensions, for
> +example in the case of migration data, how is the device state
> +composed and consumed, which portions are not otherwise available to
> +the user via vfio-pci, what safeguards exist to validate the data,
> +etc.  To that extent, authors should additionally expect to require
> +reviews from at least one of the listed reviewers, in addition to the
> +overall vfio maintainer.
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 4322b5321891..4f7d26f9aac6 100644
> +++ b/MAINTAINERS
> @@ -20314,6 +20314,13 @@ F:	drivers/vfio/mdev/
>  F:	include/linux/mdev.h
>  F:	samples/vfio-mdev/
>  
> +VFIO PCI VENDOR DRIVERS
> +R:	Your Name <your.name@here.com>
> +L:	kvm@vger.kernel.org
> +S:	Maintained
> +P:	Documentation/vfio/vfio-pci-vendor-driver-acceptance.rst
> +F:	drivers/vfio/pci/*/
> +
>  VFIO PLATFORM DRIVER
>  M:	Eric Auger <eric.auger@redhat.com>
>  L:	kvm@vger.kernel.org
> 
> Ideally we'd have at least Yishai, Shameer, Jason, and yourself listed
> as reviewers (Connie and I are included via the higher level entry).
> Thoughts from anyone?  Volunteers for reviewers if we want to press
> forward with this as formal acceptance criteria?  Thanks,

I've not seen such formalism before in the kernel, but seems like it
might be an interesting experiment. You can add my name

Jason

^ permalink raw reply	[flat|nested] 56+ messages in thread

end of thread, other threads:[~2022-03-14 15:03 UTC | newest]

Thread overview: 56+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-03-03 23:01 [PATCH v8 0/9] vfio/hisilicon: add ACC live migration driver Shameer Kolothum
2022-03-03 23:01 ` [PATCH v8 1/9] crypto: hisilicon/qm: Move the QM header to include/linux Shameer Kolothum
2022-03-04  9:03   ` Zhou Wang
2022-03-04 11:33     ` Shameerali Kolothum Thodi
2022-03-08  1:07   ` liulongfang
2022-03-08 10:27   ` yekai(A)
2022-03-03 23:01 ` [PATCH v8 2/9] crypto: hisilicon/qm: Move few definitions to common header Shameer Kolothum
2022-03-04  9:06   ` Zhou Wang
2022-03-03 23:01 ` [PATCH v8 3/9] hisi_acc_qm: Move VF PCI device IDs " Shameer Kolothum
2022-03-04  9:34   ` Zhou Wang
2022-03-04 11:35     ` Shameerali Kolothum Thodi
2022-03-07 17:53   ` Alex Williamson
2022-03-10 13:55     ` Shameerali Kolothum Thodi
2022-03-08  1:08   ` liulongfang
2022-03-08 10:28   ` yekai(A)
2022-03-03 23:01 ` [PATCH v8 4/9] hisi_acc_vfio_pci: add new vfio_pci driver for HiSilicon ACC devices Shameer Kolothum
2022-03-03 23:01 ` [PATCH v8 5/9] hisi_acc_vfio_pci: Restrict access to VF dev BAR2 migration region Shameer Kolothum
2022-03-08  1:11   ` liulongfang
2022-03-08  6:23   ` Tian, Kevin
2022-03-08  8:33     ` Shameerali Kolothum Thodi
2022-03-08 10:09       ` Tian, Kevin
2022-03-08 11:02         ` Shameerali Kolothum Thodi
2022-03-03 23:01 ` [PATCH v8 6/9] hisi_acc_vfio_pci: Add helper to retrieve the struct pci_driver Shameer Kolothum
2022-03-04  9:40   ` Zhou Wang
2022-03-04 11:31     ` Shameerali Kolothum Thodi
2022-03-08 10:28   ` yekai(A)
2022-03-08 12:02   ` liulongfang
2022-03-03 23:01 ` [PATCH v8 7/9] crypto: hisilicon/qm: Set the VF QM state register Shameer Kolothum
2022-03-04  9:43   ` Zhou Wang
2022-03-08  6:31   ` Tian, Kevin
2022-03-08  8:46     ` Shameerali Kolothum Thodi
2022-03-08 10:10       ` Tian, Kevin
2022-03-03 23:01 ` [PATCH v8 8/9] hisi_acc_vfio_pci: Add support for VFIO live migration Shameer Kolothum
2022-03-04  8:48   ` Shameerali Kolothum Thodi
2022-03-04 19:44     ` Alex Williamson
2022-03-04 20:36       ` Shameerali Kolothum Thodi
2022-03-04 20:40         ` Alex Williamson
2022-03-04 20:57   ` Jason Gunthorpe
2022-03-07 19:05     ` Alex Williamson
2022-03-07 19:29       ` Shameerali Kolothum Thodi
2022-03-07 19:52         ` Alex Williamson
2022-03-08  8:11           ` Tian, Kevin
2022-03-08 19:33             ` Alex Williamson
2022-03-09 10:11               ` Tian, Kevin
2022-03-10 20:49                 ` Alex Williamson
2022-03-11  8:52                   ` Cornelia Huck
2022-03-11 13:21                   ` Shameerali Kolothum Thodi
2022-03-14  3:40                   ` Tian, Kevin
2022-03-14 15:03                   ` Jason Gunthorpe
2022-03-08  9:46           ` liulongfang
2022-03-08  7:41   ` Tian, Kevin
2022-03-08  8:52     ` Shameerali Kolothum Thodi
2022-03-08 10:17       ` Tian, Kevin
2022-03-03 23:01 ` [PATCH v8 9/9] hisi_acc_vfio_pci: Use its own PCI reset_done error handler Shameer Kolothum
2022-03-04 20:54   ` Jason Gunthorpe
2022-03-08  1:14   ` liulongfang

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).