[PATCH V4 00/11] mmc: Add Command Queue support

[PATCH V4 00/11] mmc: Add Command Queue support

[Adrian Hunter]

Hi

Here is V4 of the hardware command queue patches without the software
command queue patches.


Changes since V3:
	Adjusted ...blk_end_request...() for new block status codes
	Fixed CQHCI transaction descriptor for "no DCMD" case

Changes since V2:
	Dropped patches that have been applied.
	Re-based
	Added "mmc: sdhci-pci: Add CQHCI support for Intel GLK"

Changes since V1:

	"Share mmc request array between partitions" is dependent
	on changes in "Introduce queue semantics", so added that
	and block fixes:

	Added "Fix is_waiting_last_req set incorrectly"
	Added "Fix cmd error reset failure path"
	Added "Use local var for mqrq_cur"
	Added "Introduce queue semantics"

Changes since RFC:

	Re-based on next.
	Added comment about command queue priority.
	Added some acks and reviews.


Adrian Hunter (10):
      mmc: core: Add mmc_retune_hold_now()
      mmc: core: Add members to mmc_request and mmc_data for CQE's
      mmc: host: Add CQE interface
      mmc: core: Turn off CQE before sending commands
      mmc: core: Add support for handling CQE requests
      mmc: mmc: Enable Command Queuing
      mmc: mmc: Enable CQE's
      mmc: block: Prepare CQE data
      mmc: block: Add CQE support
      mmc: sdhci-pci: Add CQHCI support for Intel GLK

Venkat Gopalakrishnan (1):
      mmc: cqhci: support for command queue enabled host

 drivers/mmc/core/block.c          |  240 +++++++-
 drivers/mmc/core/block.h          |    7 +
 drivers/mmc/core/bus.c            |    7 +
 drivers/mmc/core/core.c           |  156 ++++-
 drivers/mmc/core/host.c           |    6 +
 drivers/mmc/core/host.h           |    1 +
 drivers/mmc/core/mmc.c            |   30 +-
 drivers/mmc/core/queue.c          |  273 ++++++++-
 drivers/mmc/core/queue.h          |   42 +-
 drivers/mmc/host/Kconfig          |   14 +
 drivers/mmc/host/Makefile         |    1 +
 drivers/mmc/host/cqhci.c          | 1146 +++++++++++++++++++++++++++++++++++++
 drivers/mmc/host/cqhci.h          |  240 ++++++++
 drivers/mmc/host/sdhci-pci-core.c |  153 ++++-
 include/linux/mmc/core.h          |   18 +-
 include/linux/mmc/host.h          |   24 +
 include/trace/events/mmc.h        |   36 +-
 17 files changed, 2354 insertions(+), 40 deletions(-)
 create mode 100644 drivers/mmc/host/cqhci.c
 create mode 100644 drivers/mmc/host/cqhci.h


Regards
Adrian

[PATCH V4 01/11] mmc: core: Add mmc_retune_hold_now()

[Adrian Hunter]

In preparation for CQE support.

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
---
 drivers/mmc/core/host.c | 6 ++++++
 drivers/mmc/core/host.h | 1 +
 2 files changed, 7 insertions(+)

diff --git a/drivers/mmc/core/host.c b/drivers/mmc/core/host.c
index 1503412f826c..ad88deb2e8f3 100644
--- a/drivers/mmc/core/host.c
+++ b/drivers/mmc/core/host.c
@@ -111,6 +111,12 @@ void mmc_retune_hold(struct mmc_host *host)
 	host->hold_retune += 1;
 }
 
+void mmc_retune_hold_now(struct mmc_host *host)
+{
+	host->retune_now = 0;
+	host->hold_retune += 1;
+}
+
 void mmc_retune_release(struct mmc_host *host)
 {
 	if (host->hold_retune)
diff --git a/drivers/mmc/core/host.h b/drivers/mmc/core/host.h
index fb6a76a03833..77d6f60d1bf9 100644
--- a/drivers/mmc/core/host.h
+++ b/drivers/mmc/core/host.h
@@ -19,6 +19,7 @@
 void mmc_retune_enable(struct mmc_host *host);
 void mmc_retune_disable(struct mmc_host *host);
 void mmc_retune_hold(struct mmc_host *host);
+void mmc_retune_hold_now(struct mmc_host *host);
 void mmc_retune_release(struct mmc_host *host);
 int mmc_retune(struct mmc_host *host);
 void mmc_retune_pause(struct mmc_host *host);
-- 
1.9.1

[PATCH V4 02/11] mmc: core: Add members to mmc_request and mmc_data for CQE's

[Adrian Hunter]

Most of the information needed to issue requests to a CQE is already in
struct mmc_request and struct mmc_data. Add data block address, some flags,
and the task id (tag), and allow for cmd being NULL which it is for CQE
tasks.

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
---
 include/linux/mmc/core.h   | 13 +++++++++++--
 include/trace/events/mmc.h | 36 +++++++++++++++++++++++-------------
 2 files changed, 34 insertions(+), 15 deletions(-)

diff --git a/include/linux/mmc/core.h b/include/linux/mmc/core.h
index a0c63ea28796..bf1788a224e6 100644
--- a/include/linux/mmc/core.h
+++ b/include/linux/mmc/core.h
@@ -122,11 +122,18 @@ struct mmc_data {
 	unsigned int		timeout_clks;	/* data timeout (in clocks) */
 	unsigned int		blksz;		/* data block size */
 	unsigned int		blocks;		/* number of blocks */
+	unsigned int		blk_addr;	/* block address */
 	int			error;		/* data error */
 	unsigned int		flags;
 
-#define MMC_DATA_WRITE	(1 << 8)
-#define MMC_DATA_READ	(1 << 9)
+#define MMC_DATA_WRITE		BIT(8)
+#define MMC_DATA_READ		BIT(9)
+/* Extra flags used by CQE */
+#define MMC_DATA_QBR		BIT(10)		/* CQE queue barrier*/
+#define MMC_DATA_PRIO		BIT(11)		/* CQE high priority */
+#define MMC_DATA_REL_WR		BIT(12)		/* Reliable write */
+#define MMC_DATA_DAT_TAG	BIT(13)		/* Tag request */
+#define MMC_DATA_FORCED_PRG	BIT(14)		/* Forced programming */
 
 	unsigned int		bytes_xfered;
 
@@ -153,6 +160,8 @@ struct mmc_request {
 
 	/* Allow other commands during this ongoing data transfer or busy wait */
 	bool			cap_cmd_during_tfr;
+
+	int			tag;
 };
 
 struct mmc_card;
diff --git a/include/trace/events/mmc.h b/include/trace/events/mmc.h
index a72f9b94c80b..f30a99ac65b6 100644
--- a/include/trace/events/mmc.h
+++ b/include/trace/events/mmc.h
@@ -29,8 +29,10 @@
 		__field(unsigned int,		sbc_flags)
 		__field(unsigned int,		sbc_retries)
 		__field(unsigned int,		blocks)
+		__field(unsigned int,		blk_addr)
 		__field(unsigned int,		blksz)
 		__field(unsigned int,		data_flags)
+		__field(int,			tag)
 		__field(unsigned int,		can_retune)
 		__field(unsigned int,		doing_retune)
 		__field(unsigned int,		retune_now)
@@ -42,10 +44,10 @@
 	),
 
 	TP_fast_assign(
-		__entry->cmd_opcode = mrq->cmd->opcode;
-		__entry->cmd_arg = mrq->cmd->arg;
-		__entry->cmd_flags = mrq->cmd->flags;
-		__entry->cmd_retries = mrq->cmd->retries;
+		__entry->cmd_opcode = mrq->cmd ? mrq->cmd->opcode : 0;
+		__entry->cmd_arg = mrq->cmd ? mrq->cmd->arg : 0;
+		__entry->cmd_flags = mrq->cmd ? mrq->cmd->flags : 0;
+		__entry->cmd_retries = mrq->cmd ? mrq->cmd->retries : 0;
 		__entry->stop_opcode = mrq->stop ? mrq->stop->opcode : 0;
 		__entry->stop_arg = mrq->stop ? mrq->stop->arg : 0;
 		__entry->stop_flags = mrq->stop ? mrq->stop->flags : 0;
@@ -56,7 +58,9 @@
 		__entry->sbc_retries = mrq->sbc ? mrq->sbc->retries : 0;
 		__entry->blksz = mrq->data ? mrq->data->blksz : 0;
 		__entry->blocks = mrq->data ? mrq->data->blocks : 0;
+		__entry->blk_addr = mrq->data ? mrq->data->blk_addr : 0;
 		__entry->data_flags = mrq->data ? mrq->data->flags : 0;
+		__entry->tag = mrq->tag;
 		__entry->can_retune = host->can_retune;
 		__entry->doing_retune = host->doing_retune;
 		__entry->retune_now = host->retune_now;
@@ -71,8 +75,8 @@
 		  "cmd_opcode=%u cmd_arg=0x%x cmd_flags=0x%x cmd_retries=%u "
 		  "stop_opcode=%u stop_arg=0x%x stop_flags=0x%x stop_retries=%u "
 		  "sbc_opcode=%u sbc_arg=0x%x sbc_flags=0x%x sbc_retires=%u "
-		  "blocks=%u block_size=%u data_flags=0x%x "
-		  "can_retune=%u doing_retune=%u retune_now=%u "
+		  "blocks=%u block_size=%u blk_addr=%u data_flags=0x%x "
+		  "tag=%d can_retune=%u doing_retune=%u retune_now=%u "
 		  "need_retune=%d hold_retune=%d retune_period=%u",
 		  __get_str(name), __entry->mrq,
 		  __entry->cmd_opcode, __entry->cmd_arg,
@@ -81,7 +85,8 @@
 		  __entry->stop_flags, __entry->stop_retries,
 		  __entry->sbc_opcode, __entry->sbc_arg,
 		  __entry->sbc_flags, __entry->sbc_retries,
-		  __entry->blocks, __entry->blksz, __entry->data_flags,
+		  __entry->blocks, __entry->blk_addr,
+		  __entry->blksz, __entry->data_flags, __entry->tag,
 		  __entry->can_retune, __entry->doing_retune,
 		  __entry->retune_now, __entry->need_retune,
 		  __entry->hold_retune, __entry->retune_period)
@@ -108,6 +113,7 @@
 		__field(unsigned int,		sbc_retries)
 		__field(unsigned int,		bytes_xfered)
 		__field(int,			data_err)
+		__field(int,			tag)
 		__field(unsigned int,		can_retune)
 		__field(unsigned int,		doing_retune)
 		__field(unsigned int,		retune_now)
@@ -119,10 +125,13 @@
 	),
 
 	TP_fast_assign(
-		__entry->cmd_opcode = mrq->cmd->opcode;
-		__entry->cmd_err = mrq->cmd->error;
-		memcpy(__entry->cmd_resp, mrq->cmd->resp, 4);
-		__entry->cmd_retries = mrq->cmd->retries;
+		__entry->cmd_opcode = mrq->cmd ? mrq->cmd->opcode : 0;
+		__entry->cmd_err = mrq->cmd ? mrq->cmd->error : 0;
+		__entry->cmd_resp[0] = mrq->cmd ? mrq->cmd->resp[0] : 0;
+		__entry->cmd_resp[1] = mrq->cmd ? mrq->cmd->resp[1] : 0;
+		__entry->cmd_resp[2] = mrq->cmd ? mrq->cmd->resp[2] : 0;
+		__entry->cmd_resp[3] = mrq->cmd ? mrq->cmd->resp[3] : 0;
+		__entry->cmd_retries = mrq->cmd ? mrq->cmd->retries : 0;
 		__entry->stop_opcode = mrq->stop ? mrq->stop->opcode : 0;
 		__entry->stop_err = mrq->stop ? mrq->stop->error : 0;
 		__entry->stop_resp[0] = mrq->stop ? mrq->stop->resp[0] : 0;
@@ -139,6 +148,7 @@
 		__entry->sbc_retries = mrq->sbc ? mrq->sbc->retries : 0;
 		__entry->bytes_xfered = mrq->data ? mrq->data->bytes_xfered : 0;
 		__entry->data_err = mrq->data ? mrq->data->error : 0;
+		__entry->tag = mrq->tag;
 		__entry->can_retune = host->can_retune;
 		__entry->doing_retune = host->doing_retune;
 		__entry->retune_now = host->retune_now;
@@ -154,7 +164,7 @@
 		  "cmd_retries=%u stop_opcode=%u stop_err=%d "
 		  "stop_resp=0x%x 0x%x 0x%x 0x%x stop_retries=%u "
 		  "sbc_opcode=%u sbc_err=%d sbc_resp=0x%x 0x%x 0x%x 0x%x "
-		  "sbc_retries=%u bytes_xfered=%u data_err=%d "
+		  "sbc_retries=%u bytes_xfered=%u data_err=%d tag=%d "
 		  "can_retune=%u doing_retune=%u retune_now=%u need_retune=%d "
 		  "hold_retune=%d retune_period=%u",
 		  __get_str(name), __entry->mrq,
@@ -170,7 +180,7 @@
 		  __entry->sbc_resp[0], __entry->sbc_resp[1],
 		  __entry->sbc_resp[2], __entry->sbc_resp[3],
 		  __entry->sbc_retries,
-		  __entry->bytes_xfered, __entry->data_err,
+		  __entry->bytes_xfered, __entry->data_err, __entry->tag,
 		  __entry->can_retune, __entry->doing_retune,
 		  __entry->retune_now, __entry->need_retune,
 		  __entry->hold_retune, __entry->retune_period)
-- 
1.9.1

[PATCH V4 03/11] mmc: host: Add CQE interface

[Adrian Hunter]

Add CQE host operations, capabilities, and host members.

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
---
 include/linux/mmc/host.h | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/include/linux/mmc/host.h b/include/linux/mmc/host.h
index ebd1cebbef0c..4dd7ada9b4b9 100644
--- a/include/linux/mmc/host.h
+++ b/include/linux/mmc/host.h
@@ -162,6 +162,19 @@ struct mmc_host_ops {
 				  unsigned int direction, int blk_size);
 };
 
+struct mmc_cqe_ops {
+	int	(*cqe_enable)(struct mmc_host *host, struct mmc_card *card);
+	void	(*cqe_disable)(struct mmc_host *host);
+	int	(*cqe_request)(struct mmc_host *host, struct mmc_request *mrq);
+	void	(*cqe_post_req)(struct mmc_host *host, struct mmc_request *mrq);
+	void	(*cqe_off)(struct mmc_host *host);
+	int	(*cqe_wait_for_idle)(struct mmc_host *host);
+	bool	(*cqe_timeout)(struct mmc_host *host, struct mmc_request *mrq,
+			       bool *recovery_needed);
+	void	(*cqe_recovery_start)(struct mmc_host *host);
+	void	(*cqe_recovery_finish)(struct mmc_host *host);
+};
+
 struct mmc_async_req {
 	/* active mmc request */
 	struct mmc_request	*mrq;
@@ -307,6 +320,8 @@ struct mmc_host {
 #define MMC_CAP2_HS400_ES	(1 << 20)	/* Host supports enhanced strobe */
 #define MMC_CAP2_NO_SD		(1 << 21)	/* Do not send SD commands during initialization */
 #define MMC_CAP2_NO_MMC		(1 << 22)	/* Do not send (e)MMC commands during initialization */
+#define MMC_CAP2_CQE		(1 << 23)	/* Has eMMC command queue engine */
+#define MMC_CAP2_CQE_DCMD	(1 << 24)	/* CQE can issue a direct command */
 
 	mmc_pm_flag_t		pm_caps;	/* supported pm features */
 
@@ -393,6 +408,15 @@ struct mmc_host {
 	int			dsr_req;	/* DSR value is valid */
 	u32			dsr;	/* optional driver stage (DSR) value */
 
+	/* Command Queue Engine (CQE) support */
+	const struct mmc_cqe_ops *cqe_ops;
+	void			*cqe_private;
+	void			(*cqe_recovery_notifier)(struct mmc_host *,
+							 struct mmc_request *);
+	int			cqe_qdepth;
+	bool			cqe_enabled;
+	bool			cqe_on;
+
 	unsigned long		private[0] ____cacheline_aligned;
 };
 
-- 
1.9.1

[PATCH V4 04/11] mmc: core: Turn off CQE before sending commands

[Adrian Hunter]

Turn off the CQE before sending commands, and ensure it is off in any reset
or power management paths, or re-tuning.

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
---
 drivers/mmc/core/core.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/mmc/core/core.c b/drivers/mmc/core/core.c
index 26431267a3e2..b0af9db18eef 100644
--- a/drivers/mmc/core/core.c
+++ b/drivers/mmc/core/core.c
@@ -260,6 +260,9 @@ static void __mmc_start_request(struct mmc_host *host, struct mmc_request *mrq)
 
 	trace_mmc_request_start(host, mrq);
 
+	if (host->cqe_on)
+		host->cqe_ops->cqe_off(host);
+
 	host->ops->request(host, mrq);
 }
 
@@ -982,6 +985,9 @@ int mmc_execute_tuning(struct mmc_card *card)
 	if (!host->ops->execute_tuning)
 		return 0;
 
+	if (host->cqe_on)
+		host->cqe_ops->cqe_off(host);
+
 	if (mmc_card_mmc(card))
 		opcode = MMC_SEND_TUNING_BLOCK_HS200;
 	else
@@ -1021,6 +1027,9 @@ void mmc_set_bus_width(struct mmc_host *host, unsigned int width)
  */
 void mmc_set_initial_state(struct mmc_host *host)
 {
+	if (host->cqe_on)
+		host->cqe_ops->cqe_off(host);
+
 	mmc_retune_disable(host);
 
 	if (mmc_host_is_spi(host))
-- 
1.9.1

[PATCH V4 05/11] mmc: core: Add support for handling CQE requests

[Adrian Hunter]

Add core support for handling CQE requests, including starting, completing
and recovering.

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
---
 drivers/mmc/core/core.c  | 147 +++++++++++++++++++++++++++++++++++++++++++++--
 include/linux/mmc/core.h |   5 ++
 2 files changed, 147 insertions(+), 5 deletions(-)

diff --git a/drivers/mmc/core/core.c b/drivers/mmc/core/core.c
index b0af9db18eef..5a9d837599a1 100644
--- a/drivers/mmc/core/core.c
+++ b/drivers/mmc/core/core.c
@@ -266,7 +266,8 @@ static void __mmc_start_request(struct mmc_host *host, struct mmc_request *mrq)
 	host->ops->request(host, mrq);
 }
 
-static void mmc_mrq_pr_debug(struct mmc_host *host, struct mmc_request *mrq)
+static void mmc_mrq_pr_debug(struct mmc_host *host, struct mmc_request *mrq,
+			     bool cqe)
 {
 	if (mrq->sbc) {
 		pr_debug("<%s: starting CMD%u arg %08x flags %08x>\n",
@@ -275,9 +276,12 @@ static void mmc_mrq_pr_debug(struct mmc_host *host, struct mmc_request *mrq)
 	}
 
 	if (mrq->cmd) {
-		pr_debug("%s: starting CMD%u arg %08x flags %08x\n",
-			 mmc_hostname(host), mrq->cmd->opcode, mrq->cmd->arg,
-			 mrq->cmd->flags);
+		pr_debug("%s: starting %sCMD%u arg %08x flags %08x\n",
+			 mmc_hostname(host), cqe ? "CQE direct " : "",
+			 mrq->cmd->opcode, mrq->cmd->arg, mrq->cmd->flags);
+	} else if (cqe) {
+		pr_debug("%s: starting CQE transfer for tag %d blkaddr %u\n",
+			 mmc_hostname(host), mrq->tag, mrq->data->blk_addr);
 	}
 
 	if (mrq->data) {
@@ -345,7 +349,7 @@ static int mmc_start_request(struct mmc_host *host, struct mmc_request *mrq)
 	if (mmc_card_removed(host->card))
 		return -ENOMEDIUM;
 
-	mmc_mrq_pr_debug(host, mrq);
+	mmc_mrq_pr_debug(host, mrq, false);
 
 	WARN_ON(!host->claimed);
 
@@ -485,6 +489,139 @@ void mmc_wait_for_req_done(struct mmc_host *host, struct mmc_request *mrq)
 }
 EXPORT_SYMBOL(mmc_wait_for_req_done);
 
+int mmc_cqe_start_req(struct mmc_host *host, struct mmc_request *mrq)
+{
+	int err;
+
+	/* Caller must hold retuning while CQE is in use */
+	err = mmc_retune(host);
+	if (err)
+		goto out_err;
+
+	mrq->host = host;
+
+	mmc_mrq_pr_debug(host, mrq, true);
+
+	err = mmc_mrq_prep(host, mrq);
+	if (err)
+		goto out_err;
+
+	err = host->cqe_ops->cqe_request(host, mrq);
+	if (err)
+		goto out_err;
+
+	trace_mmc_request_start(host, mrq);
+
+	return 0;
+
+out_err:
+	if (mrq->cmd) {
+		pr_debug("%s: failed to start CQE direct CMD%u, error %d\n",
+			 mmc_hostname(host), mrq->cmd->opcode, err);
+	} else {
+		pr_debug("%s: failed to start CQE transfer for tag %d, error %d\n",
+			 mmc_hostname(host), mrq->tag, err);
+	}
+	return err;
+}
+EXPORT_SYMBOL(mmc_cqe_start_req);
+
+static void __mmc_cqe_request_done(struct mmc_host *host,
+				   struct mmc_request *mrq)
+{
+	mmc_should_fail_request(host, mrq);
+
+	/* Flag re-tuning needed on CRC errors */
+	if ((mrq->cmd && mrq->cmd->error == -EILSEQ) ||
+	    (mrq->data && mrq->data->error == -EILSEQ))
+		mmc_retune_needed(host);
+
+	trace_mmc_request_done(host, mrq);
+
+	if (mrq->cmd) {
+		pr_debug("%s: CQE req done (direct CMD%u): %d\n",
+			 mmc_hostname(host), mrq->cmd->opcode, mrq->cmd->error);
+	} else {
+		pr_debug("%s: CQE transfer done tag %d\n",
+			 mmc_hostname(host), mrq->tag);
+	}
+
+	if (mrq->data) {
+		pr_debug("%s:     %d bytes transferred: %d\n",
+			 mmc_hostname(host),
+			 mrq->data->bytes_xfered, mrq->data->error);
+	}
+}
+
+/**
+ *	mmc_cqe_request_done - CQE has finished processing an MMC request
+ *	@host: MMC host which completed request
+ *	@mrq: MMC request which completed
+ *
+ *	CQE drivers should call this function when they have completed
+ *	their processing of a request.
+ */
+void mmc_cqe_request_done(struct mmc_host *host, struct mmc_request *mrq)
+{
+	__mmc_cqe_request_done(host, mrq);
+
+	mrq->done(mrq);
+}
+EXPORT_SYMBOL(mmc_cqe_request_done);
+
+/**
+ *	mmc_cqe_post_req - CQE post process of a completed MMC request
+ *	@host: MMC host
+ *	@mrq: MMC request to be processed
+ */
+void mmc_cqe_post_req(struct mmc_host *host, struct mmc_request *mrq)
+{
+	if (host->cqe_ops->cqe_post_req)
+		host->cqe_ops->cqe_post_req(host, mrq);
+}
+EXPORT_SYMBOL(mmc_cqe_post_req);
+
+/* Arbitrary 1 second timeout */
+#define MMC_CQE_RECOVERY_TIMEOUT	1000
+
+int mmc_cqe_recovery(struct mmc_host *host)
+{
+	struct mmc_command cmd;
+	int err;
+
+	mmc_retune_hold_now(host);
+
+	/*
+	 * Recovery is expected seldom, if at all, but it reduces performance,
+	 * so make sure it is not completely silent.
+	 */
+	pr_warn("%s: running CQE recovery\n", mmc_hostname(host));
+
+	host->cqe_ops->cqe_recovery_start(host);
+
+	memset(&cmd, 0, sizeof(cmd));
+	cmd.opcode       = MMC_STOP_TRANSMISSION,
+	cmd.flags        = MMC_RSP_R1B | MMC_CMD_AC,
+	cmd.flags       &= ~MMC_RSP_CRC; /* Ignore CRC */
+	cmd.busy_timeout = MMC_CQE_RECOVERY_TIMEOUT,
+	mmc_wait_for_cmd(host, &cmd, 0);
+
+	memset(&cmd, 0, sizeof(cmd));
+	cmd.opcode       = MMC_CMDQ_TASK_MGMT;
+	cmd.arg          = 1; /* Discard entire queue */
+	cmd.flags        = MMC_RSP_R1B | MMC_CMD_AC;
+	cmd.flags       &= ~MMC_RSP_CRC; /* Ignore CRC */
+	cmd.busy_timeout = MMC_CQE_RECOVERY_TIMEOUT,
+	err = mmc_wait_for_cmd(host, &cmd, 0);
+
+	host->cqe_ops->cqe_recovery_finish(host);
+
+	mmc_retune_release(host);
+
+	return err;
+}
+EXPORT_SYMBOL(mmc_cqe_recovery);
+
 /**
  *	mmc_is_req_done - Determine if a 'cap_cmd_during_tfr' request is done
  *	@host: MMC host
diff --git a/include/linux/mmc/core.h b/include/linux/mmc/core.h
index bf1788a224e6..1974fcfd4284 100644
--- a/include/linux/mmc/core.h
+++ b/include/linux/mmc/core.h
@@ -174,6 +174,11 @@ struct mmc_async_req *mmc_start_areq(struct mmc_host *host,
 int mmc_wait_for_cmd(struct mmc_host *host, struct mmc_command *cmd,
 		int retries);
 
+int mmc_cqe_start_req(struct mmc_host *host, struct mmc_request *mrq);
+void mmc_cqe_request_done(struct mmc_host *host, struct mmc_request *mrq);
+void mmc_cqe_post_req(struct mmc_host *host, struct mmc_request *mrq);
+int mmc_cqe_recovery(struct mmc_host *host);
+
 int mmc_hw_reset(struct mmc_host *host);
 void mmc_set_data_timeout(struct mmc_data *data, const struct mmc_card *card);
 
-- 
1.9.1

[PATCH V4 06/11] mmc: mmc: Enable Command Queuing

[Adrian Hunter]

Enable the Command Queue if the host controller supports i a command queue
engine. It is not compatible with Packed Commands, so do not enable that
at the same time.

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
---
 drivers/mmc/core/mmc.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)

diff --git a/drivers/mmc/core/mmc.c b/drivers/mmc/core/mmc.c
index 4ffea14b7eb6..2ff0caf92bc8 100644
--- a/drivers/mmc/core/mmc.c
+++ b/drivers/mmc/core/mmc.c
@@ -1783,6 +1783,20 @@ static int mmc_init_card(struct mmc_host *host, u32 ocr,
 		}
 	}
 
+	/* Enable Command Queue if supported */
+	card->ext_csd.cmdq_en = false;
+	if (card->ext_csd.cmdq_support && host->caps2 & MMC_CAP2_CQE) {
+		err = mmc_cmdq_enable(card);
+		if (err && err != -EBADMSG)
+			goto free_card;
+		if (err) {
+			pr_warn("%s: Enabling CMDQ failed\n",
+				mmc_hostname(card->host));
+			card->ext_csd.cmdq_support = false;
+			card->ext_csd.cmdq_depth = 0;
+			err = 0;
+		}
+	}
 	/*
 	 * In some cases (e.g. RPMB or mmc_test), the Command Queue must be
 	 * disabled for a time, so a flag is needed to indicate to re-enable the
@@ -1796,7 +1810,8 @@ static int mmc_init_card(struct mmc_host *host, u32 ocr,
 	 */
 	if (card->ext_csd.max_packed_writes >= 3 &&
 	    card->ext_csd.max_packed_reads >= 5 &&
-	    host->caps2 & MMC_CAP2_PACKED_CMD) {
+	    host->caps2 & MMC_CAP2_PACKED_CMD &&
+	    !card->ext_csd.cmdq_en) {
 		err = mmc_switch(card, EXT_CSD_CMD_SET_NORMAL,
 				EXT_CSD_EXP_EVENTS_CTRL,
 				EXT_CSD_PACKED_EVENT_EN,
-- 
1.9.1

[PATCH V4 07/11] mmc: mmc: Enable CQE's

[Adrian Hunter]

Enable or disable CQE when a card is added or removed respectively.

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
---
 drivers/mmc/core/bus.c |  7 +++++++
 drivers/mmc/core/mmc.c | 13 +++++++++++++
 2 files changed, 20 insertions(+)

diff --git a/drivers/mmc/core/bus.c b/drivers/mmc/core/bus.c
index 301246513a37..a4b49e25fe96 100644
--- a/drivers/mmc/core/bus.c
+++ b/drivers/mmc/core/bus.c
@@ -369,10 +369,17 @@ int mmc_add_card(struct mmc_card *card)
  */
 void mmc_remove_card(struct mmc_card *card)
 {
+	struct mmc_host *host = card->host;
+
 #ifdef CONFIG_DEBUG_FS
 	mmc_remove_card_debugfs(card);
 #endif
 
+	if (host->cqe_enabled) {
+		host->cqe_ops->cqe_disable(host);
+		host->cqe_enabled = false;
+	}
+
 	if (mmc_card_present(card)) {
 		if (mmc_host_is_spi(card->host)) {
 			pr_info("%s: SPI card removed\n",
diff --git a/drivers/mmc/core/mmc.c b/drivers/mmc/core/mmc.c
index 2ff0caf92bc8..92c6167d64e0 100644
--- a/drivers/mmc/core/mmc.c
+++ b/drivers/mmc/core/mmc.c
@@ -1804,6 +1804,19 @@ static int mmc_init_card(struct mmc_host *host, u32 ocr,
 	 */
 	card->reenable_cmdq = card->ext_csd.cmdq_en;
 
+	if (card->ext_csd.cmdq_en && (host->caps2 & MMC_CAP2_CQE) &&
+	    !host->cqe_enabled) {
+		err = host->cqe_ops->cqe_enable(host, card);
+		if (err) {
+			pr_err("%s: Failed to enable CQE, error %d\n",
+				mmc_hostname(host), err);
+		} else {
+			host->cqe_enabled = true;
+			pr_info("%s: Command Queue Engine enabled\n",
+				mmc_hostname(host));
+		}
+	}
+
 	/*
 	 * The mandatory minimum values are defined for packed command.
 	 * read: 5, write: 3
-- 
1.9.1

[PATCH V4 08/11] mmc: block: Prepare CQE data

[Adrian Hunter]

Enhance mmc_blk_data_prep() to support CQE requests.

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
---
 drivers/mmc/core/block.c | 45 ++++++++++++++++++++++++++++++++++-----------
 1 file changed, 34 insertions(+), 11 deletions(-)

diff --git a/drivers/mmc/core/block.c b/drivers/mmc/core/block.c
index 40f0d596ed54..915290c74363 100644
--- a/drivers/mmc/core/block.c
+++ b/drivers/mmc/core/block.c
@@ -36,6 +36,7 @@
 #include <linux/compat.h>
 #include <linux/pm_runtime.h>
 #include <linux/idr.h>
+#include <linux/ioprio.h>
 
 #include <linux/mmc/ioctl.h>
 #include <linux/mmc/card.h>
@@ -1485,25 +1486,27 @@ static enum mmc_blk_status mmc_blk_err_check(struct mmc_card *card,
 }
 
 static void mmc_blk_data_prep(struct mmc_queue *mq, struct mmc_queue_req *mqrq,
-			      int disable_multi, bool *do_rel_wr,
-			      bool *do_data_tag)
+			      int disable_multi, bool *do_rel_wr_p,
+			      bool *do_data_tag_p)
 {
 	struct mmc_blk_data *md = mq->blkdata;
 	struct mmc_card *card = md->queue.card;
 	struct mmc_blk_request *brq = &mqrq->brq;
 	struct request *req = mmc_queue_req_to_req(mqrq);
+	bool do_rel_wr, do_data_tag;
 
 	/*
 	 * Reliable writes are used to implement Forced Unit Access and
 	 * are supported only on MMCs.
 	 */
-	*do_rel_wr = (req->cmd_flags & REQ_FUA) &&
-		     rq_data_dir(req) == WRITE &&
-		     (md->flags & MMC_BLK_REL_WR);
+	do_rel_wr = (req->cmd_flags & REQ_FUA) &&
+		    rq_data_dir(req) == WRITE &&
+		    (md->flags & MMC_BLK_REL_WR);
 
 	memset(brq, 0, sizeof(struct mmc_blk_request));
 
 	brq->mrq.data = &brq->data;
+	brq->mrq.tag = req->tag;
 
 	brq->stop.opcode = MMC_STOP_TRANSMISSION;
 	brq->stop.arg = 0;
@@ -1518,6 +1521,15 @@ static void mmc_blk_data_prep(struct mmc_queue *mq, struct mmc_queue_req *mqrq,
 
 	brq->data.blksz = 512;
 	brq->data.blocks = blk_rq_sectors(req);
+	brq->data.blk_addr = blk_rq_pos(req);
+
+	/*
+	 * The command queue supports 2 priorities: "high" (1) and "simple" (0).
+	 * The eMMC will give "high" priority tasks priority over "simple"
+	 * priority tasks. Here we give priority to IOPRIO_CLASS_RT.
+	 */
+	if (IOPRIO_PRIO_CLASS(req_get_ioprio(req)) == IOPRIO_CLASS_RT)
+		brq->data.flags |= MMC_DATA_PRIO;
 
 	/*
 	 * The block layer doesn't support all sector count
@@ -1547,18 +1559,23 @@ static void mmc_blk_data_prep(struct mmc_queue *mq, struct mmc_queue_req *mqrq,
 						brq->data.blocks);
 	}
 
-	if (*do_rel_wr)
+	if (do_rel_wr) {
 		mmc_apply_rel_rw(brq, card, req);
+		brq->data.flags |= MMC_DATA_REL_WR;
+	}
 
 	/*
 	 * Data tag is used only during writing meta data to speed
 	 * up write and any subsequent read of this meta data
 	 */
-	*do_data_tag = card->ext_csd.data_tag_unit_size &&
-		       (req->cmd_flags & REQ_META) &&
-		       (rq_data_dir(req) == WRITE) &&
-		       ((brq->data.blocks * brq->data.blksz) >=
-			card->ext_csd.data_tag_unit_size);
+	do_data_tag = card->ext_csd.data_tag_unit_size &&
+		      (req->cmd_flags & REQ_META) &&
+		      (rq_data_dir(req) == WRITE) &&
+		      ((brq->data.blocks * brq->data.blksz) >=
+		       card->ext_csd.data_tag_unit_size);
+
+	if (do_data_tag)
+		brq->data.flags |= MMC_DATA_DAT_TAG;
 
 	mmc_set_data_timeout(&brq->data, card);
 
@@ -1587,6 +1604,12 @@ static void mmc_blk_data_prep(struct mmc_queue *mq, struct mmc_queue_req *mqrq,
 	mqrq->areq.mrq = &brq->mrq;
 
 	mmc_queue_bounce_pre(mqrq);
+
+	if (do_rel_wr_p)
+		*do_rel_wr_p = do_rel_wr;
+
+	if (do_data_tag_p)
+		*do_data_tag_p = do_data_tag;
 }
 
 static void mmc_blk_rw_rq_prep(struct mmc_queue_req *mqrq,
-- 
1.9.1

[PATCH V4 09/11] mmc: block: Add CQE support

[Adrian Hunter]

Add CQE support to the block driver, including:
	- optionally using DCMD for flush requests
	- manually issuing discard requests
	- issuing read / write requests to the CQE
	- supporting block-layer timeouts
	- handling recovery
	- supporting re-tuning

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
---
 drivers/mmc/core/block.c | 195 ++++++++++++++++++++++++++++++++-
 drivers/mmc/core/block.h |   7 ++
 drivers/mmc/core/queue.c | 273 ++++++++++++++++++++++++++++++++++++++++++++++-
 drivers/mmc/core/queue.h |  42 +++++++-
 4 files changed, 510 insertions(+), 7 deletions(-)

diff --git a/drivers/mmc/core/block.c b/drivers/mmc/core/block.c
index 915290c74363..2d25115637b7 100644
--- a/drivers/mmc/core/block.c
+++ b/drivers/mmc/core/block.c
@@ -109,6 +109,7 @@ struct mmc_blk_data {
 #define MMC_BLK_WRITE		BIT(1)
 #define MMC_BLK_DISCARD		BIT(2)
 #define MMC_BLK_SECDISCARD	BIT(3)
+#define MMC_BLK_CQE_RECOVERY	BIT(4)
 
 	/*
 	 * Only set in main mmc_blk_data associated
@@ -1612,6 +1613,198 @@ static void mmc_blk_data_prep(struct mmc_queue *mq, struct mmc_queue_req *mqrq,
 		*do_data_tag_p = do_data_tag;
 }
 
+#define MMC_CQE_RETRIES 2
+
+void mmc_blk_cqe_complete_rq(struct request *req)
+{
+	struct mmc_queue_req *mqrq = req_to_mmc_queue_req(req);
+	struct mmc_request *mrq = &mqrq->brq.mrq;
+	struct request_queue *q = req->q;
+	struct mmc_queue *mq = q->queuedata;
+	struct mmc_host *host = mq->card->host;
+	unsigned long flags;
+	bool put_card;
+	int err;
+
+	mmc_cqe_post_req(host, mrq);
+
+	spin_lock_irqsave(q->queue_lock, flags);
+
+	mq->cqe_in_flight[mmc_cqe_issue_type(host, req)] -= 1;
+
+	put_card = mmc_cqe_tot_in_flight(mq) == 0;
+
+	if (mrq->cmd && mrq->cmd->error)
+		err = mrq->cmd->error;
+	else if (mrq->data && mrq->data->error)
+		err = mrq->data->error;
+	else
+		err = 0;
+
+	if (err) {
+		if (mqrq->retries++ < MMC_CQE_RETRIES)
+			blk_requeue_request(q, req);
+		else
+			__blk_end_request_all(req, BLK_STS_IOERR);
+	} else if (mrq->data) {
+		if (__blk_end_request(req, BLK_STS_OK, mrq->data->bytes_xfered))
+			blk_requeue_request(q, req);
+	} else {
+		__blk_end_request_all(req, BLK_STS_OK);
+	}
+
+	mmc_cqe_kick_queue(mq);
+
+	spin_unlock_irqrestore(q->queue_lock, flags);
+
+	if (put_card)
+		mmc_put_card(mq->card);
+}
+
+void mmc_blk_cqe_recovery(struct mmc_queue *mq)
+{
+	struct mmc_card *card = mq->card;
+	struct mmc_host *host = card->host;
+	int err;
+
+	mmc_get_card(card);
+
+	pr_debug("%s: CQE recovery start\n", mmc_hostname(host));
+
+	mq->cqe_in_recovery = true;
+
+	err = mmc_cqe_recovery(host);
+	if (err)
+		mmc_blk_reset(mq->blkdata, host, MMC_BLK_CQE_RECOVERY);
+	else
+		mmc_blk_reset_success(mq->blkdata, MMC_BLK_CQE_RECOVERY);
+
+	mq->cqe_in_recovery = false;
+
+	pr_debug("%s: CQE recovery done\n", mmc_hostname(host));
+
+	mmc_put_card(card);
+}
+
+static void mmc_blk_cqe_req_done(struct mmc_request *mrq)
+{
+	struct mmc_queue_req *mqrq = container_of(mrq, struct mmc_queue_req,
+						  brq.mrq);
+	struct request *req = mmc_queue_req_to_req(mqrq);
+	struct request_queue *q = req->q;
+	struct mmc_queue *mq = q->queuedata;
+
+	/*
+	 * Block layer timeouts race with completions which means the normal
+	 * completion path cannot be used during recovery.
+	 */
+	if (mq->cqe_in_recovery)
+		mmc_blk_cqe_complete_rq(req);
+	else
+		blk_complete_request(req);
+}
+
+static int mmc_blk_cqe_start_req(struct mmc_host *host, struct mmc_request *mrq)
+{
+	mrq->done = mmc_blk_cqe_req_done;
+	return mmc_cqe_start_req(host, mrq);
+}
+
+static struct mmc_request *mmc_blk_cqe_prep_dcmd(struct mmc_queue_req *mqrq,
+						 struct request *req)
+{
+	struct mmc_blk_request *brq = &mqrq->brq;
+
+	memset(brq, 0, sizeof(*brq));
+
+	brq->mrq.cmd = &brq->cmd;
+	brq->mrq.tag = req->tag;
+
+	return &brq->mrq;
+}
+
+static int mmc_blk_cqe_issue_flush(struct mmc_queue *mq, struct request *req)
+{
+	struct mmc_queue_req *mqrq = req_to_mmc_queue_req(req);
+	struct mmc_request *mrq = mmc_blk_cqe_prep_dcmd(mqrq, req);
+
+	mrq->cmd->opcode = MMC_SWITCH;
+	mrq->cmd->arg = (MMC_SWITCH_MODE_WRITE_BYTE << 24) |
+			(EXT_CSD_FLUSH_CACHE << 16) |
+			(1 << 8) |
+			EXT_CSD_CMD_SET_NORMAL;
+	mrq->cmd->flags = MMC_CMD_AC | MMC_RSP_R1B;
+
+	return mmc_blk_cqe_start_req(mq->card->host, mrq);
+}
+
+static int mmc_blk_cqe_issue_rw_rq(struct mmc_queue *mq, struct request *req)
+{
+	struct mmc_queue_req *mqrq = req_to_mmc_queue_req(req);
+
+	mmc_blk_data_prep(mq, mqrq, 0, NULL, NULL);
+
+	return mmc_blk_cqe_start_req(mq->card->host, &mqrq->brq.mrq);
+}
+
+enum mmc_issued mmc_blk_cqe_issue_rq(struct mmc_queue *mq, struct request *req)
+{
+	struct mmc_blk_data *md = mq->blkdata;
+	struct mmc_card *card = md->queue.card;
+	struct mmc_host *host = card->host;
+	int ret;
+
+	ret = mmc_blk_part_switch(card, md);
+	if (ret)
+		return MMC_REQ_FAILED_TO_START;
+
+	switch (mmc_cqe_issue_type(host, req)) {
+	case MMC_ISSUE_SYNC:
+		ret = host->cqe_ops->cqe_wait_for_idle(host);
+		if (ret)
+			return MMC_REQ_BUSY;
+		switch (req_op(req)) {
+		case REQ_OP_DRV_IN:
+		case REQ_OP_DRV_OUT:
+			mmc_blk_issue_drv_op(mq, req);
+			break;
+		case REQ_OP_DISCARD:
+			mmc_blk_issue_discard_rq(mq, req);
+			break;
+		case REQ_OP_SECURE_ERASE:
+			mmc_blk_issue_secdiscard_rq(mq, req);
+			break;
+		case REQ_OP_FLUSH:
+			mmc_blk_issue_flush(mq, req);
+			break;
+		default:
+			WARN_ON_ONCE(1);
+			return MMC_REQ_FAILED_TO_START;
+		}
+		return MMC_REQ_FINISHED;
+	case MMC_ISSUE_DCMD:
+	case MMC_ISSUE_ASYNC:
+		switch (req_op(req)) {
+		case REQ_OP_FLUSH:
+			ret = mmc_blk_cqe_issue_flush(mq, req);
+			break;
+		case REQ_OP_READ:
+		case REQ_OP_WRITE:
+			ret = mmc_blk_cqe_issue_rw_rq(mq, req);
+			break;
+		default:
+			WARN_ON_ONCE(1);
+			ret = -EINVAL;
+		}
+		if (!ret)
+			return MMC_REQ_STARTED;
+		return ret == -EBUSY ? MMC_REQ_BUSY : MMC_REQ_FAILED_TO_START;
+	default:
+		WARN_ON_ONCE(1);
+		return MMC_REQ_FAILED_TO_START;
+	}
+}
+
 static void mmc_blk_rw_rq_prep(struct mmc_queue_req *mqrq,
 			       struct mmc_card *card,
 			       int disable_multi,
@@ -2035,7 +2228,7 @@ static struct mmc_blk_data *mmc_blk_alloc_req(struct mmc_card *card,
 	INIT_LIST_HEAD(&md->part);
 	md->usage = 1;
 
-	ret = mmc_init_queue(&md->queue, card, &md->lock, subname);
+	ret = mmc_init_queue(&md->queue, card, &md->lock, subname, area_type);
 	if (ret)
 		goto err_putdisk;
 
diff --git a/drivers/mmc/core/block.h b/drivers/mmc/core/block.h
index 860ca7c8df86..d7b3d7008b00 100644
--- a/drivers/mmc/core/block.h
+++ b/drivers/mmc/core/block.h
@@ -6,4 +6,11 @@
 
 void mmc_blk_issue_rq(struct mmc_queue *mq, struct request *req);
 
+enum mmc_issued;
+
+enum mmc_issued mmc_blk_cqe_issue_rq(struct mmc_queue *mq,
+				     struct request *req);
+void mmc_blk_cqe_complete_rq(struct request *rq);
+void mmc_blk_cqe_recovery(struct mmc_queue *mq);
+
 #endif
diff --git a/drivers/mmc/core/queue.c b/drivers/mmc/core/queue.c
index affa7370ba82..0cb7b0e8ee58 100644
--- a/drivers/mmc/core/queue.c
+++ b/drivers/mmc/core/queue.c
@@ -36,10 +36,254 @@ static int mmc_prep_request(struct request_queue *q, struct request *req)
 		return BLKPREP_KILL;
 
 	req->rq_flags |= RQF_DONTPREP;
+	req_to_mmc_queue_req(req)->retries = 0;
 
 	return BLKPREP_OK;
 }
 
+static void mmc_cqe_request_fn(struct request_queue *q)
+{
+	struct mmc_queue *mq = q->queuedata;
+	struct request *req;
+
+	if (!mq) {
+		while ((req = blk_fetch_request(q)) != NULL) {
+			req->rq_flags |= RQF_QUIET;
+			__blk_end_request_all(req, BLK_STS_IOERR);
+		}
+		return;
+	}
+
+	if (mq->asleep && !mq->cqe_busy)
+		wake_up_process(mq->thread);
+}
+
+static inline bool mmc_cqe_dcmd_busy(struct mmc_queue *mq)
+{
+	/* Allow only 1 DCMD at a time */
+	return mq->cqe_in_flight[MMC_ISSUE_DCMD];
+}
+
+void mmc_cqe_kick_queue(struct mmc_queue *mq)
+{
+	if ((mq->cqe_busy & MMC_CQE_DCMD_BUSY) && !mmc_cqe_dcmd_busy(mq))
+		mq->cqe_busy &= ~MMC_CQE_DCMD_BUSY;
+
+	mq->cqe_busy &= ~MMC_CQE_QUEUE_FULL;
+
+	if (mq->asleep && !mq->cqe_busy)
+		__blk_run_queue(mq->queue);
+}
+
+static inline bool mmc_cqe_can_dcmd(struct mmc_host *host)
+{
+	return host->caps2 & MMC_CAP2_CQE_DCMD;
+}
+
+enum mmc_issue_type mmc_cqe_issue_type(struct mmc_host *host,
+				       struct request *req)
+{
+	switch (req_op(req)) {
+	case REQ_OP_DRV_IN:
+	case REQ_OP_DRV_OUT:
+	case REQ_OP_DISCARD:
+	case REQ_OP_SECURE_ERASE:
+		return MMC_ISSUE_SYNC;
+	case REQ_OP_FLUSH:
+		return mmc_cqe_can_dcmd(host) ? MMC_ISSUE_DCMD : MMC_ISSUE_SYNC;
+	default:
+		return MMC_ISSUE_ASYNC;
+	}
+}
+
+static void __mmc_cqe_recovery_notifier(struct mmc_queue *mq)
+{
+	if (!mq->cqe_recovery_needed) {
+		mq->cqe_recovery_needed = true;
+		wake_up_process(mq->thread);
+	}
+}
+
+static void mmc_cqe_recovery_notifier(struct mmc_host *host,
+				      struct mmc_request *mrq)
+{
+	struct mmc_queue_req *mqrq = container_of(mrq, struct mmc_queue_req,
+						  brq.mrq);
+	struct request *req = mmc_queue_req_to_req(mqrq);
+	struct request_queue *q = req->q;
+	struct mmc_queue *mq = q->queuedata;
+	unsigned long flags;
+
+	spin_lock_irqsave(q->queue_lock, flags);
+	__mmc_cqe_recovery_notifier(mq);
+	spin_unlock_irqrestore(q->queue_lock, flags);
+}
+
+static int mmc_cqe_thread(void *d)
+{
+	struct mmc_queue *mq = d;
+	struct request_queue *q = mq->queue;
+	struct mmc_card *card = mq->card;
+	struct mmc_host *host = card->host;
+	unsigned long flags;
+	int get_put = 0;
+
+	current->flags |= PF_MEMALLOC;
+
+	down(&mq->thread_sem);
+	spin_lock_irqsave(q->queue_lock, flags);
+	while (1) {
+		struct request *req = NULL;
+		enum mmc_issue_type issue_type;
+		bool retune_ok = false;
+
+		if (mq->cqe_recovery_needed) {
+			spin_unlock_irqrestore(q->queue_lock, flags);
+			mmc_blk_cqe_recovery(mq);
+			spin_lock_irqsave(q->queue_lock, flags);
+			mq->cqe_recovery_needed = false;
+		}
+
+		set_current_state(TASK_INTERRUPTIBLE);
+
+		if (!kthread_should_stop())
+			req = blk_peek_request(q);
+
+		if (req) {
+			issue_type = mmc_cqe_issue_type(host, req);
+			switch (issue_type) {
+			case MMC_ISSUE_DCMD:
+				if (mmc_cqe_dcmd_busy(mq)) {
+					mq->cqe_busy |= MMC_CQE_DCMD_BUSY;
+					req = NULL;
+					break;
+				}
+				/* Fall through */
+			case MMC_ISSUE_ASYNC:
+				if (blk_queue_start_tag(q, req)) {
+					mq->cqe_busy |= MMC_CQE_QUEUE_FULL;
+					req = NULL;
+				}
+				break;
+			default:
+				/*
+				 * Timeouts are handled by mmc core, so set a
+				 * large value to avoid races.
+				 */
+				req->timeout = 600 * HZ;
+				blk_start_request(req);
+				break;
+			}
+			if (req) {
+				mq->cqe_in_flight[issue_type] += 1;
+				if (mmc_cqe_tot_in_flight(mq) == 1)
+					get_put += 1;
+				if (mmc_cqe_qcnt(mq) == 1)
+					retune_ok = true;
+			}
+		}
+
+		mq->asleep = !req;
+
+		spin_unlock_irqrestore(q->queue_lock, flags);
+
+		if (req) {
+			enum mmc_issued issued;
+
+			set_current_state(TASK_RUNNING);
+
+			if (get_put) {
+				get_put = 0;
+				mmc_get_card(card);
+			}
+
+			if (host->need_retune && retune_ok &&
+			    !host->hold_retune)
+				host->retune_now = true;
+			else
+				host->retune_now = false;
+
+			issued = mmc_blk_cqe_issue_rq(mq, req);
+
+			cond_resched();
+
+			spin_lock_irqsave(q->queue_lock, flags);
+
+			switch (issued) {
+			case MMC_REQ_STARTED:
+				break;
+			case MMC_REQ_BUSY:
+				blk_requeue_request(q, req);
+				goto finished;
+			case MMC_REQ_FAILED_TO_START:
+				__blk_end_request_all(req, BLK_STS_IOERR);
+				/* Fall through */
+			case MMC_REQ_FINISHED:
+finished:
+				mq->cqe_in_flight[issue_type] -= 1;
+				if (mmc_cqe_tot_in_flight(mq) == 0)
+					get_put = -1;
+			}
+		} else {
+			if (get_put < 0) {
+				get_put = 0;
+				mmc_put_card(card);
+			}
+			/*
+			 * Do not stop with requests in flight in case recovery
+			 * is needed.
+			 */
+			if (kthread_should_stop() &&
+			    !mmc_cqe_tot_in_flight(mq)) {
+				set_current_state(TASK_RUNNING);
+				break;
+			}
+			up(&mq->thread_sem);
+			schedule();
+			down(&mq->thread_sem);
+			spin_lock_irqsave(q->queue_lock, flags);
+		}
+	} /* loop */
+	up(&mq->thread_sem);
+
+	return 0;
+}
+
+static enum blk_eh_timer_return __mmc_cqe_timed_out(struct request *req)
+{
+	struct mmc_queue_req *mqrq = req_to_mmc_queue_req(req);
+	struct mmc_request *mrq = &mqrq->brq.mrq;
+	struct mmc_queue *mq = req->q->queuedata;
+	struct mmc_host *host = mq->card->host;
+	enum mmc_issue_type issue_type = mmc_cqe_issue_type(host, req);
+	bool recovery_needed = false;
+
+	switch (issue_type) {
+	case MMC_ISSUE_ASYNC:
+	case MMC_ISSUE_DCMD:
+		if (host->cqe_ops->cqe_timeout(host, mrq, &recovery_needed)) {
+			if (recovery_needed)
+				__mmc_cqe_recovery_notifier(mq);
+			return BLK_EH_RESET_TIMER;
+		}
+		/* No timeout */
+		return BLK_EH_HANDLED;
+	default:
+		/* Timeout is handled by mmc core */
+		return BLK_EH_RESET_TIMER;
+	}
+}
+
+static enum blk_eh_timer_return mmc_cqe_timed_out(struct request *req)
+{
+	struct mmc_queue *mq = req->q->queuedata;
+
+	if (mq->cqe_recovery_needed)
+		return BLK_EH_RESET_TIMER;
+
+	return __mmc_cqe_timed_out(req);
+}
+
 static int mmc_queue_thread(void *d)
 {
 	struct mmc_queue *mq = d;
@@ -233,11 +477,12 @@ static void mmc_exit_request(struct request_queue *q, struct request *req)
  * Initialise a MMC card request queue.
  */
 int mmc_init_queue(struct mmc_queue *mq, struct mmc_card *card,
-		   spinlock_t *lock, const char *subname)
+		   spinlock_t *lock, const char *subname, int area_type)
 {
 	struct mmc_host *host = card->host;
 	u64 limit = BLK_BOUNCE_HIGH;
 	int ret = -ENOMEM;
+	bool use_cqe = host->cqe_enabled && area_type != MMC_BLK_DATA_AREA_RPMB;
 
 	if (mmc_dev(host)->dma_mask && *mmc_dev(host)->dma_mask)
 		limit = (u64)dma_max_pfn(mmc_dev(host)) << PAGE_SHIFT;
@@ -247,7 +492,7 @@ int mmc_init_queue(struct mmc_queue *mq, struct mmc_card *card,
 	if (!mq->queue)
 		return -ENOMEM;
 	mq->queue->queue_lock = lock;
-	mq->queue->request_fn = mmc_request_fn;
+	mq->queue->request_fn = use_cqe ? mmc_cqe_request_fn : mmc_request_fn;
 	mq->queue->init_rq_fn = mmc_init_request;
 	mq->queue->exit_rq_fn = mmc_exit_request;
 	mq->queue->cmd_size = sizeof(struct mmc_queue_req);
@@ -259,6 +504,24 @@ int mmc_init_queue(struct mmc_queue *mq, struct mmc_card *card,
 		return ret;
 	}
 
+	if (use_cqe) {
+		int q_depth = card->ext_csd.cmdq_depth;
+
+		if (q_depth > host->cqe_qdepth)
+			q_depth = host->cqe_qdepth;
+
+		ret = blk_queue_init_tags(mq->queue, q_depth, NULL,
+					  BLK_TAG_ALLOC_FIFO);
+		if (ret)
+			goto cleanup_queue;
+
+		blk_queue_softirq_done(mq->queue, mmc_blk_cqe_complete_rq);
+		blk_queue_rq_timed_out(mq->queue, mmc_cqe_timed_out);
+		blk_queue_rq_timeout(mq->queue, 60 * HZ);
+
+		host->cqe_recovery_notifier = mmc_cqe_recovery_notifier;
+	}
+
 	blk_queue_prep_rq(mq->queue, mmc_prep_request);
 	queue_flag_set_unlocked(QUEUE_FLAG_NONROT, mq->queue);
 	queue_flag_clear_unlocked(QUEUE_FLAG_ADD_RANDOM, mq->queue);
@@ -280,9 +543,9 @@ int mmc_init_queue(struct mmc_queue *mq, struct mmc_card *card,
 
 	sema_init(&mq->thread_sem, 1);
 
-	mq->thread = kthread_run(mmc_queue_thread, mq, "mmcqd/%d%s",
-		host->index, subname ? subname : "");
-
+	mq->thread = kthread_run(use_cqe ? mmc_cqe_thread : mmc_queue_thread,
+				 mq, "mmcqd/%d%s", host->index,
+				 subname ? subname : "");
 	if (IS_ERR(mq->thread)) {
 		ret = PTR_ERR(mq->thread);
 		goto cleanup_queue;
diff --git a/drivers/mmc/core/queue.h b/drivers/mmc/core/queue.h
index 361b46408e0f..8e9273d977c0 100644
--- a/drivers/mmc/core/queue.h
+++ b/drivers/mmc/core/queue.h
@@ -7,6 +7,20 @@
 #include <linux/mmc/core.h>
 #include <linux/mmc/host.h>
 
+enum mmc_issued {
+	MMC_REQ_STARTED,
+	MMC_REQ_BUSY,
+	MMC_REQ_FAILED_TO_START,
+	MMC_REQ_FINISHED,
+};
+
+enum mmc_issue_type {
+	MMC_ISSUE_SYNC,
+	MMC_ISSUE_DCMD,
+	MMC_ISSUE_ASYNC,
+	MMC_ISSUE_MAX,
+};
+
 static inline struct mmc_queue_req *req_to_mmc_queue_req(struct request *rq)
 {
 	return blk_mq_rq_to_pdu(rq);
@@ -53,6 +67,7 @@ struct mmc_queue_req {
 	int			drv_op_result;
 	struct mmc_blk_ioc_data	**idata;
 	unsigned int		ioc_count;
+	int			retries;
 };
 
 struct mmc_queue {
@@ -70,10 +85,17 @@ struct mmc_queue {
 	 * associated mmc_queue_req data.
 	 */
 	int			qcnt;
+	/* Following are defined for a Command Queue Engine */
+	int			cqe_in_flight[MMC_ISSUE_MAX];
+	unsigned int		cqe_busy;
+	bool			cqe_recovery_needed;
+	bool			cqe_in_recovery;
+#define MMC_CQE_DCMD_BUSY	BIT(0)
+#define MMC_CQE_QUEUE_FULL	BIT(1)
 };
 
 extern int mmc_init_queue(struct mmc_queue *, struct mmc_card *, spinlock_t *,
-			  const char *);
+			  const char *, int);
 extern void mmc_cleanup_queue(struct mmc_queue *);
 extern void mmc_queue_suspend(struct mmc_queue *);
 extern void mmc_queue_resume(struct mmc_queue *);
@@ -85,4 +107,22 @@ extern unsigned int mmc_queue_map_sg(struct mmc_queue *,
 
 extern int mmc_access_rpmb(struct mmc_queue *);
 
+void mmc_cqe_kick_queue(struct mmc_queue *mq);
+
+enum mmc_issue_type mmc_cqe_issue_type(struct mmc_host *host,
+				       struct request *req);
+
+static inline int mmc_cqe_tot_in_flight(struct mmc_queue *mq)
+{
+	return mq->cqe_in_flight[MMC_ISSUE_SYNC] +
+	       mq->cqe_in_flight[MMC_ISSUE_DCMD] +
+	       mq->cqe_in_flight[MMC_ISSUE_ASYNC];
+}
+
+static inline int mmc_cqe_qcnt(struct mmc_queue *mq)
+{
+	return mq->cqe_in_flight[MMC_ISSUE_DCMD] +
+	       mq->cqe_in_flight[MMC_ISSUE_ASYNC];
+}
+
 #endif
-- 
1.9.1

[PATCH V4 10/11] mmc: cqhci: support for command queue enabled host

[Adrian Hunter]

From: Venkat Gopalakrishnan <venkatg@codeaurora.org>

This patch adds CMDQ support for command-queue compatible
hosts.

Command queue is added in eMMC-5.1 specification. This
enables the controller to process upto 32 requests at
a time.

Adrian Hunter contributed renaming to cqhci, recovery, suspend
and resume, cqhci_off, cqhci_wait_for_idle, and external timeout
handling.

Signed-off-by: Asutosh Das <asutoshd@codeaurora.org>
Signed-off-by: Sujit Reddy Thumma <sthumma@codeaurora.org>
Signed-off-by: Konstantin Dorfman <kdorfman@codeaurora.org>
Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: Ritesh Harjani <riteshh@codeaurora.org>
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
---
 drivers/mmc/host/Kconfig  |   13 +
 drivers/mmc/host/Makefile |    1 +
 drivers/mmc/host/cqhci.c  | 1146 +++++++++++++++++++++++++++++++++++++++++++++
 drivers/mmc/host/cqhci.h  |  240 ++++++++++
 4 files changed, 1400 insertions(+)
 create mode 100644 drivers/mmc/host/cqhci.c
 create mode 100644 drivers/mmc/host/cqhci.h

diff --git a/drivers/mmc/host/Kconfig b/drivers/mmc/host/Kconfig
index 2242633550df..3a164a03f2bc 100644
--- a/drivers/mmc/host/Kconfig
+++ b/drivers/mmc/host/Kconfig
@@ -834,6 +834,19 @@ config MMC_SUNXI
 	  This selects support for the SD/MMC Host Controller on
 	  Allwinner sunxi SoCs.
 
+config MMC_CQHCI
+	tristate "Command Queue Host Controller Interface support"
+	depends on HAS_DMA
+	help
+	  This selects the Command Queue Host Controller Interface (CQHCI)
+	  support present in host controllers of Qualcomm Technologies, Inc
+	  amongst others.
+	  This controller supports eMMC devices with command queue support.
+
+	  If you have a controller with this interface, say Y or M here.
+
+	  If unsure, say N.
+
 config MMC_TOSHIBA_PCI
 	tristate "Toshiba Type A SD/MMC Card Interface Driver"
 	depends on PCI
diff --git a/drivers/mmc/host/Makefile b/drivers/mmc/host/Makefile
index 8c46766c000c..3ae71e006890 100644
--- a/drivers/mmc/host/Makefile
+++ b/drivers/mmc/host/Makefile
@@ -88,6 +88,7 @@ obj-$(CONFIG_MMC_SDHCI_MSM)		+= sdhci-msm.o
 obj-$(CONFIG_MMC_SDHCI_ST)		+= sdhci-st.o
 obj-$(CONFIG_MMC_SDHCI_MICROCHIP_PIC32)	+= sdhci-pic32.o
 obj-$(CONFIG_MMC_SDHCI_BRCMSTB)		+= sdhci-brcmstb.o
+obj-$(CONFIG_MMC_CQHCI)			+= cqhci.o
 
 ifeq ($(CONFIG_CB710_DEBUG),y)
 	CFLAGS-cb710-mmc	+= -DDEBUG
diff --git a/drivers/mmc/host/cqhci.c b/drivers/mmc/host/cqhci.c
new file mode 100644
index 000000000000..302421a26230
--- /dev/null
+++ b/drivers/mmc/host/cqhci.c
@@ -0,0 +1,1146 @@
+/* Copyright (c) 2015, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/delay.h>
+#include <linux/highmem.h>
+#include <linux/io.h>
+#include <linux/module.h>
+#include <linux/dma-mapping.h>
+#include <linux/slab.h>
+#include <linux/scatterlist.h>
+#include <linux/platform_device.h>
+#include <linux/ktime.h>
+
+#include <linux/mmc/mmc.h>
+#include <linux/mmc/host.h>
+#include <linux/mmc/card.h>
+
+#include "cqhci.h"
+
+#define DCMD_SLOT 31
+#define NUM_SLOTS 32
+
+struct cqhci_slot {
+	struct mmc_request *mrq;
+	unsigned int flags;
+#define CQHCI_EXTERNAL_TIMEOUT	BIT(0)
+#define CQHCI_COMPLETED		BIT(1)
+#define CQHCI_HOST_CRC		BIT(2)
+#define CQHCI_HOST_TIMEOUT	BIT(3)
+#define CQHCI_HOST_OTHER	BIT(4)
+};
+
+static inline u8 *get_desc(struct cqhci_host *cq_host, u8 tag)
+{
+	return cq_host->desc_base + (tag * cq_host->slot_sz);
+}
+
+static inline u8 *get_link_desc(struct cqhci_host *cq_host, u8 tag)
+{
+	u8 *desc = get_desc(cq_host, tag);
+
+	return desc + cq_host->task_desc_len;
+}
+
+static inline dma_addr_t get_trans_desc_dma(struct cqhci_host *cq_host, u8 tag)
+{
+	return cq_host->trans_desc_dma_base +
+		(cq_host->mmc->max_segs * tag *
+		 cq_host->trans_desc_len);
+}
+
+static inline u8 *get_trans_desc(struct cqhci_host *cq_host, u8 tag)
+{
+	return cq_host->trans_desc_base +
+		(cq_host->trans_desc_len * cq_host->mmc->max_segs * tag);
+}
+
+static void setup_trans_desc(struct cqhci_host *cq_host, u8 tag)
+{
+	u8 *link_temp;
+	dma_addr_t trans_temp;
+
+	link_temp = get_link_desc(cq_host, tag);
+	trans_temp = get_trans_desc_dma(cq_host, tag);
+
+	memset(link_temp, 0, cq_host->link_desc_len);
+	if (cq_host->link_desc_len > 8)
+		*(link_temp + 8) = 0;
+
+	if (tag == DCMD_SLOT && (cq_host->mmc->caps2 & MMC_CAP2_CQE_DCMD)) {
+		*link_temp = CQHCI_VALID(0) | CQHCI_ACT(0) | CQHCI_END(1);
+		return;
+	}
+
+	*link_temp = CQHCI_VALID(1) | CQHCI_ACT(0x6) | CQHCI_END(0);
+
+	if (cq_host->dma64) {
+		__le64 *data_addr = (__le64 __force *)(link_temp + 4);
+
+		data_addr[0] = cpu_to_le64(trans_temp);
+	} else {
+		__le32 *data_addr = (__le32 __force *)(link_temp + 4);
+
+		data_addr[0] = cpu_to_le32(trans_temp);
+	}
+}
+
+static void cqhci_set_irqs(struct cqhci_host *cq_host, u32 set)
+{
+	u32 ier;
+
+	ier = cqhci_readl(cq_host, CQHCI_ISTE);
+	ier |= set;
+	cqhci_writel(cq_host, ier, CQHCI_ISTE);
+	cqhci_writel(cq_host, ier, CQHCI_ISGE);
+}
+
+#define DRV_NAME "cqhci"
+
+#define CQHCI_DUMP(f, x...) \
+	pr_err("%s: " DRV_NAME ": " f, mmc_hostname(mmc), ## x)
+
+static void cqhci_dumpregs(struct cqhci_host *cq_host)
+{
+	struct mmc_host *mmc = cq_host->mmc;
+
+	CQHCI_DUMP("============ CQHCI REGISTER DUMP ===========\n");
+
+	CQHCI_DUMP("Caps:      0x%08x | Version:  0x%08x\n",
+		   cqhci_readl(cq_host, CQHCI_CAP),
+		   cqhci_readl(cq_host, CQHCI_VER));
+	CQHCI_DUMP("Config:    0x%08x | Control:  0x%08x\n",
+		   cqhci_readl(cq_host, CQHCI_CFG),
+		   cqhci_readl(cq_host, CQHCI_CTL));
+	CQHCI_DUMP("Int stat:  0x%08x | Int enab: 0x%08x\n",
+		   cqhci_readl(cq_host, CQHCI_IS),
+		   cqhci_readl(cq_host, CQHCI_ISTE));
+	CQHCI_DUMP("Int sig:   0x%08x | Int Coal: 0x%08x\n",
+		   cqhci_readl(cq_host, CQHCI_ISGE),
+		   cqhci_readl(cq_host, CQHCI_IC));
+	CQHCI_DUMP("TDL base:  0x%08x | TDL up32: 0x%08x\n",
+		   cqhci_readl(cq_host, CQHCI_TDLBA),
+		   cqhci_readl(cq_host, CQHCI_TDLBAU));
+	CQHCI_DUMP("Doorbell:  0x%08x | TCN:      0x%08x\n",
+		   cqhci_readl(cq_host, CQHCI_TDBR),
+		   cqhci_readl(cq_host, CQHCI_TCN));
+	CQHCI_DUMP("Dev queue: 0x%08x | Dev Pend: 0x%08x\n",
+		   cqhci_readl(cq_host, CQHCI_DQS),
+		   cqhci_readl(cq_host, CQHCI_DPT));
+	CQHCI_DUMP("Task clr:  0x%08x | SSC1:     0x%08x\n",
+		   cqhci_readl(cq_host, CQHCI_TCLR),
+		   cqhci_readl(cq_host, CQHCI_SSC1));
+	CQHCI_DUMP("SSC2:      0x%08x | DCMD rsp: 0x%08x\n",
+		   cqhci_readl(cq_host, CQHCI_SSC2),
+		   cqhci_readl(cq_host, CQHCI_CRDCT));
+	CQHCI_DUMP("RED mask:  0x%08x | TERRI:    0x%08x\n",
+		   cqhci_readl(cq_host, CQHCI_RMEM),
+		   cqhci_readl(cq_host, CQHCI_TERRI));
+	CQHCI_DUMP("Resp idx:  0x%08x | Resp arg: 0x%08x\n",
+		   cqhci_readl(cq_host, CQHCI_CRI),
+		   cqhci_readl(cq_host, CQHCI_CRA));
+
+	if (cq_host->ops->dumpregs)
+		cq_host->ops->dumpregs(mmc);
+	else
+		CQHCI_DUMP(": ===========================================\n");
+}
+
+/**
+ * The allocated descriptor table for task, link & transfer descritors
+ * looks like:
+ * |----------|
+ * |task desc |  |->|----------|
+ * |----------|  |  |trans desc|
+ * |link desc-|->|  |----------|
+ * |----------|          .
+ *      .                .
+ *  no. of slots      max-segs
+ *      .           |----------|
+ * |----------|
+ * The idea here is to create the [task+trans] table and mark & point the
+ * link desc to the transfer desc table on a per slot basis.
+ */
+static int cqhci_host_alloc_tdl(struct cqhci_host *cq_host)
+{
+	int i = 0;
+
+	/* task descriptor can be 64/128 bit irrespective of arch */
+	if (cq_host->caps & CQHCI_TASK_DESC_SZ_128) {
+		cqhci_writel(cq_host, cqhci_readl(cq_host, CQHCI_CFG) |
+			       CQHCI_TASK_DESC_SZ, CQHCI_CFG);
+		cq_host->task_desc_len = 16;
+	} else {
+		cq_host->task_desc_len = 8;
+	}
+
+	/*
+	 * 96 bits length of transfer desc instead of 128 bits which means
+	 * ADMA would expect next valid descriptor at the 96th bit
+	 * or 128th bit
+	 */
+	if (cq_host->dma64) {
+		if (cq_host->quirks & CQHCI_QUIRK_SHORT_TXFR_DESC_SZ)
+			cq_host->trans_desc_len = 12;
+		else
+			cq_host->trans_desc_len = 16;
+		cq_host->link_desc_len = 16;
+	} else {
+		cq_host->trans_desc_len = 8;
+		cq_host->link_desc_len = 8;
+	}
+
+	/* total size of a slot: 1 task & 1 transfer (link) */
+	cq_host->slot_sz = cq_host->task_desc_len + cq_host->link_desc_len;
+
+	cq_host->desc_size = cq_host->slot_sz * cq_host->num_slots;
+
+	cq_host->data_size = cq_host->trans_desc_len * cq_host->mmc->max_segs *
+		(cq_host->num_slots - 1);
+
+	pr_debug("%s: cqhci: desc_size: %zu data_sz: %zu slot-sz: %d\n",
+		 mmc_hostname(cq_host->mmc), cq_host->desc_size, cq_host->data_size,
+		 cq_host->slot_sz);
+
+	/*
+	 * allocate a dma-mapped chunk of memory for the descriptors
+	 * allocate a dma-mapped chunk of memory for link descriptors
+	 * setup each link-desc memory offset per slot-number to
+	 * the descriptor table.
+	 */
+	cq_host->desc_base = dmam_alloc_coherent(mmc_dev(cq_host->mmc),
+						 cq_host->desc_size,
+						 &cq_host->desc_dma_base,
+						 GFP_KERNEL);
+	cq_host->trans_desc_base = dmam_alloc_coherent(mmc_dev(cq_host->mmc),
+					      cq_host->data_size,
+					      &cq_host->trans_desc_dma_base,
+					      GFP_KERNEL);
+	if (!cq_host->desc_base || !cq_host->trans_desc_base)
+		return -ENOMEM;
+
+	pr_debug("%s: cqhci: desc-base: 0x%p trans-base: 0x%p\n desc_dma 0x%llx trans_dma: 0x%llx\n",
+		 mmc_hostname(cq_host->mmc), cq_host->desc_base, cq_host->trans_desc_base,
+		(unsigned long long)cq_host->desc_dma_base,
+		(unsigned long long)cq_host->trans_desc_dma_base);
+
+	for (; i < (cq_host->num_slots); i++)
+		setup_trans_desc(cq_host, i);
+
+	return 0;
+}
+
+static void __cqhci_enable(struct cqhci_host *cq_host)
+{
+	struct mmc_host *mmc = cq_host->mmc;
+	u32 cqcfg;
+
+	cqcfg = cqhci_readl(cq_host, CQHCI_CFG);
+
+	/* Configuration must not be changed while enabled */
+	if (cqcfg & CQHCI_ENABLE) {
+		cqcfg &= ~CQHCI_ENABLE;
+		cqhci_writel(cq_host, cqcfg, CQHCI_CFG);
+	}
+
+	cqcfg &= ~(CQHCI_DCMD | CQHCI_TASK_DESC_SZ);
+
+	if (mmc->caps2 & MMC_CAP2_CQE_DCMD)
+		cqcfg |= CQHCI_DCMD;
+
+	if (cq_host->caps & CQHCI_TASK_DESC_SZ_128)
+		cqcfg |= CQHCI_TASK_DESC_SZ;
+
+	cqhci_writel(cq_host, cqcfg, CQHCI_CFG);
+
+	cqhci_writel(cq_host, lower_32_bits(cq_host->desc_dma_base),
+		     CQHCI_TDLBA);
+	cqhci_writel(cq_host, upper_32_bits(cq_host->desc_dma_base),
+		     CQHCI_TDLBAU);
+
+	cqhci_writel(cq_host, cq_host->rca, CQHCI_SSC2);
+
+	cqhci_set_irqs(cq_host, 0);
+
+	cqcfg |= CQHCI_ENABLE;
+
+	cqhci_writel(cq_host, cqcfg, CQHCI_CFG);
+
+	mmc->cqe_on = true;
+
+	if (cq_host->ops->enable)
+		cq_host->ops->enable(mmc);
+
+	/* Ensure all writes are done before interrupts are enabled */
+	wmb();
+
+	cqhci_set_irqs(cq_host, CQHCI_IS_MASK);
+
+	cq_host->activated = true;
+}
+
+static void __cqhci_disable(struct cqhci_host *cq_host)
+{
+	u32 cqcfg;
+
+	cqcfg = cqhci_readl(cq_host, CQHCI_CFG);
+	cqcfg &= ~CQHCI_ENABLE;
+	cqhci_writel(cq_host, cqcfg, CQHCI_CFG);
+
+	cq_host->mmc->cqe_on = false;
+
+	cq_host->activated = false;
+}
+
+int cqhci_suspend(struct mmc_host *mmc)
+{
+	struct cqhci_host *cq_host = mmc->cqe_private;
+
+	if (cq_host->enabled)
+		__cqhci_disable(cq_host);
+
+	return 0;
+}
+EXPORT_SYMBOL(cqhci_suspend);
+
+int cqhci_resume(struct mmc_host *mmc)
+{
+	/* Re-enable is done upon first request */
+	return 0;
+}
+EXPORT_SYMBOL(cqhci_resume);
+
+static int cqhci_enable(struct mmc_host *mmc, struct mmc_card *card)
+{
+	struct cqhci_host *cq_host = mmc->cqe_private;
+	int err;
+
+	if (cq_host->enabled)
+		return 0;
+
+	cq_host->rca = card->rca;
+
+	err = cqhci_host_alloc_tdl(cq_host);
+	if (err)
+		return err;
+
+	__cqhci_enable(cq_host);
+
+	cq_host->enabled = true;
+
+#ifdef DEBUG
+	cqhci_dumpregs(cq_host);
+#endif
+	return 0;
+}
+
+/* CQHCI is idle and should halt immediately, so set a small timeout */
+#define CQHCI_OFF_TIMEOUT 100
+
+static void cqhci_off(struct mmc_host *mmc)
+{
+	struct cqhci_host *cq_host = mmc->cqe_private;
+	ktime_t timeout;
+	bool timed_out;
+	u32 reg;
+
+	if (!cq_host->enabled || !mmc->cqe_on || cq_host->recovery_halt)
+		return;
+
+	if (cq_host->ops->disable)
+		cq_host->ops->disable(mmc, false);
+
+	cqhci_writel(cq_host, CQHCI_HALT, CQHCI_CTL);
+
+	timeout = ktime_add_us(ktime_get(), CQHCI_OFF_TIMEOUT);
+	while (1) {
+		timed_out = ktime_compare(ktime_get(), timeout) > 0;
+		reg = cqhci_readl(cq_host, CQHCI_CTL);
+		if ((reg & CQHCI_HALT) || timed_out)
+			break;
+	}
+
+	if (timed_out)
+		pr_err("%s: cqhci: CQE stuck on\n", mmc_hostname(mmc));
+	else
+		pr_debug("%s: cqhci: CQE off\n", mmc_hostname(mmc));
+
+	mmc->cqe_on = false;
+}
+
+static void cqhci_disable(struct mmc_host *mmc)
+{
+	struct cqhci_host *cq_host = mmc->cqe_private;
+
+	if (!cq_host->enabled)
+		return;
+
+	cqhci_off(mmc);
+
+	__cqhci_disable(cq_host);
+
+	dmam_free_coherent(mmc_dev(mmc), cq_host->data_size,
+			   cq_host->trans_desc_base,
+			   cq_host->trans_desc_dma_base);
+
+	dmam_free_coherent(mmc_dev(mmc), cq_host->desc_size,
+			   cq_host->desc_base,
+			   cq_host->desc_dma_base);
+
+	cq_host->trans_desc_base = NULL;
+	cq_host->desc_base = NULL;
+
+	cq_host->enabled = false;
+}
+
+static void cqhci_prep_task_desc(struct mmc_request *mrq,
+					u64 *data, bool intr)
+{
+	u32 req_flags = mrq->data->flags;
+
+	*data = CQHCI_VALID(1) |
+		CQHCI_END(1) |
+		CQHCI_INT(intr) |
+		CQHCI_ACT(0x5) |
+		CQHCI_FORCED_PROG(!!(req_flags & MMC_DATA_FORCED_PRG)) |
+		CQHCI_DATA_TAG(!!(req_flags & MMC_DATA_DAT_TAG)) |
+		CQHCI_DATA_DIR(!!(req_flags & MMC_DATA_READ)) |
+		CQHCI_PRIORITY(!!(req_flags & MMC_DATA_PRIO)) |
+		CQHCI_QBAR(!!(req_flags & MMC_DATA_QBR)) |
+		CQHCI_REL_WRITE(!!(req_flags & MMC_DATA_REL_WR)) |
+		CQHCI_BLK_COUNT(mrq->data->blocks) |
+		CQHCI_BLK_ADDR((u64)mrq->data->blk_addr);
+
+	pr_debug("%s: cqhci: tag %d task descriptor 0x016%llx\n",
+		 mmc_hostname(mrq->host), mrq->tag, (unsigned long long)*data);
+}
+
+static int cqhci_dma_map(struct mmc_host *host, struct mmc_request *mrq)
+{
+	int sg_count;
+	struct mmc_data *data = mrq->data;
+
+	if (!data)
+		return -EINVAL;
+
+	sg_count = dma_map_sg(mmc_dev(host), data->sg,
+			      data->sg_len,
+			      (data->flags & MMC_DATA_WRITE) ?
+			      DMA_TO_DEVICE : DMA_FROM_DEVICE);
+	if (!sg_count) {
+		pr_err("%s: sg-len: %d\n", __func__, data->sg_len);
+		return -ENOMEM;
+	}
+
+	return sg_count;
+}
+
+static void cqhci_set_tran_desc(u8 *desc,
+				 dma_addr_t addr, int len, bool end)
+{
+	__le64 *dataddr = (__le64 __force *)(desc + 4);
+	__le32 *attr = (__le32 __force *)desc;
+
+	*attr = (CQHCI_VALID(1) |
+		 CQHCI_END(end ? 1 : 0) |
+		 CQHCI_INT(0) |
+		 CQHCI_ACT(0x4) |
+		 CQHCI_DAT_LENGTH(len));
+
+	dataddr[0] = cpu_to_le64(addr);
+}
+
+static int cqhci_prep_tran_desc(struct mmc_request *mrq,
+			       struct cqhci_host *cq_host, int tag)
+{
+	struct mmc_data *data = mrq->data;
+	int i, sg_count, len;
+	bool end = false;
+	dma_addr_t addr;
+	u8 *desc;
+	struct scatterlist *sg;
+
+	sg_count = cqhci_dma_map(mrq->host, mrq);
+	if (sg_count < 0) {
+		pr_err("%s: %s: unable to map sg lists, %d\n",
+				mmc_hostname(mrq->host), __func__, sg_count);
+		return sg_count;
+	}
+
+	desc = get_trans_desc(cq_host, tag);
+
+	for_each_sg(data->sg, sg, sg_count, i) {
+		addr = sg_dma_address(sg);
+		len = sg_dma_len(sg);
+
+		if ((i+1) == sg_count)
+			end = true;
+		cqhci_set_tran_desc(desc, addr, len, end);
+		desc += cq_host->trans_desc_len;
+	}
+
+	return 0;
+}
+
+static void cqhci_prep_dcmd_desc(struct mmc_host *mmc,
+				   struct mmc_request *mrq)
+{
+	u64 *task_desc = NULL;
+	u64 data = 0;
+	u8 resp_type;
+	u8 *desc;
+	__le64 *dataddr;
+	struct cqhci_host *cq_host = mmc->cqe_private;
+	u8 timing;
+
+	if (!(mrq->cmd->flags & MMC_RSP_PRESENT)) {
+		resp_type = 0x0;
+		timing = 0x1;
+	} else {
+		if (mrq->cmd->flags & MMC_RSP_R1B) {
+			resp_type = 0x3;
+			timing = 0x0;
+		} else {
+			resp_type = 0x2;
+			timing = 0x1;
+		}
+	}
+
+	task_desc = (__le64 __force *)get_desc(cq_host, cq_host->dcmd_slot);
+	memset(task_desc, 0, cq_host->task_desc_len);
+	data |= (CQHCI_VALID(1) |
+		 CQHCI_END(1) |
+		 CQHCI_INT(1) |
+		 CQHCI_QBAR(1) |
+		 CQHCI_ACT(0x5) |
+		 CQHCI_CMD_INDEX(mrq->cmd->opcode) |
+		 CQHCI_CMD_TIMING(timing) | CQHCI_RESP_TYPE(resp_type));
+	*task_desc |= data;
+	desc = (u8 *)task_desc;
+	pr_debug("%s: cqhci: dcmd: cmd: %d timing: %d resp: %d\n",
+		 mmc_hostname(mmc), mrq->cmd->opcode, timing, resp_type);
+	dataddr = (__le64 __force *)(desc + 4);
+	dataddr[0] = cpu_to_le64((u64)mrq->cmd->arg);
+
+}
+
+static void cqhci_post_req(struct mmc_host *host, struct mmc_request *mrq)
+{
+	struct mmc_data *data = mrq->data;
+
+	if (data) {
+		dma_unmap_sg(mmc_dev(host), data->sg, data->sg_len,
+			     (data->flags & MMC_DATA_READ) ?
+			     DMA_FROM_DEVICE : DMA_TO_DEVICE);
+	}
+}
+
+static inline int cqhci_tag(struct mmc_request *mrq)
+{
+	return mrq->cmd ? DCMD_SLOT : mrq->tag;
+}
+
+static int cqhci_request(struct mmc_host *mmc, struct mmc_request *mrq)
+{
+	int err = 0;
+	u64 data = 0;
+	u64 *task_desc = NULL;
+	int tag = cqhci_tag(mrq);
+	struct cqhci_host *cq_host = mmc->cqe_private;
+	unsigned long flags;
+
+	if (!cq_host->enabled) {
+		pr_err("%s: cqhci: not enabled\n", mmc_hostname(mmc));
+		return -EINVAL;
+	}
+
+	/* First request after resume has to re-enable */
+	if (!cq_host->activated)
+		__cqhci_enable(cq_host);
+
+	if (!mmc->cqe_on) {
+		cqhci_writel(cq_host, 0, CQHCI_CTL);
+		mmc->cqe_on = true;
+		pr_debug("%s: cqhci: CQE on\n", mmc_hostname(mmc));
+		if (cqhci_readl(cq_host, CQHCI_CTL) && CQHCI_HALT) {
+			pr_err("%s: cqhci: CQE failed to exit halt state\n",
+			       mmc_hostname(mmc));
+		}
+		if (cq_host->ops->enable)
+			cq_host->ops->enable(mmc);
+	}
+
+	if (mrq->data) {
+		task_desc = (__le64 __force *)get_desc(cq_host, tag);
+		cqhci_prep_task_desc(mrq, &data, 1);
+		*task_desc = cpu_to_le64(data);
+		err = cqhci_prep_tran_desc(mrq, cq_host, tag);
+		if (err) {
+			pr_err("%s: cqhci: failed to setup tx desc: %d\n",
+			       mmc_hostname(mmc), err);
+			return err;
+		}
+	} else {
+		cqhci_prep_dcmd_desc(mmc, mrq);
+	}
+
+	spin_lock_irqsave(&cq_host->lock, flags);
+
+	if (cq_host->recovery_halt) {
+		err = -EBUSY;
+		goto out_unlock;
+	}
+
+	cq_host->slot[tag].mrq = mrq;
+	cq_host->slot[tag].flags = 0;
+
+	cq_host->qcnt += 1;
+
+	cqhci_writel(cq_host, 1 << tag, CQHCI_TDBR);
+	if (!(cqhci_readl(cq_host, CQHCI_TDBR) & (1 << tag)))
+		pr_debug("%s: cqhci: doorbell not set for tag %d\n",
+			 mmc_hostname(mmc), tag);
+out_unlock:
+	spin_unlock_irqrestore(&cq_host->lock, flags);
+
+	if (err)
+		cqhci_post_req(mmc, mrq);
+
+	return err;
+}
+
+static void cqhci_recovery_needed(struct mmc_host *mmc, struct mmc_request *mrq,
+				  bool notify)
+{
+	struct cqhci_host *cq_host = mmc->cqe_private;
+
+	if (!cq_host->recovery_halt) {
+		cq_host->recovery_halt = true;
+		pr_debug("%s: cqhci: recovery needed\n", mmc_hostname(mmc));
+		wake_up(&cq_host->wait_queue);
+		if (notify && mmc->cqe_recovery_notifier)
+			mmc->cqe_recovery_notifier(mmc, mrq);
+	}
+}
+
+static unsigned int cqhci_error_flags(int error1, int error2)
+{
+	int error = error1 ? error1 : error2;
+
+	switch (error) {
+	case -EILSEQ:
+		return CQHCI_HOST_CRC;
+	case -ETIMEDOUT:
+		return CQHCI_HOST_TIMEOUT;
+	default:
+		return CQHCI_HOST_OTHER;
+	}
+}
+
+static void cqhci_error_irq(struct mmc_host *mmc, u32 status, int cmd_error,
+			    int data_error)
+{
+	struct cqhci_host *cq_host = mmc->cqe_private;
+	struct cqhci_slot *slot;
+	u32 terri;
+	int tag;
+
+	spin_lock(&cq_host->lock);
+
+	terri = cqhci_readl(cq_host, CQHCI_TERRI);
+
+	pr_debug("%s: cqhci: error IRQ status: 0x%08x cmd error %d data error %d TERRI: 0x%08x\n",
+		 mmc_hostname(mmc), status, cmd_error, data_error, terri);
+
+	/* Forget about errors when recovery has already been triggered */
+	if (cq_host->recovery_halt)
+		goto out_unlock;
+
+	if (!cq_host->qcnt) {
+		WARN_ONCE(1, "%s: cqhci: error when idle. IRQ status: 0x%08x cmd error %d data error %d TERRI: 0x%08x\n",
+			  mmc_hostname(mmc), status, cmd_error, data_error,
+			  terri);
+		goto out_unlock;
+	}
+
+	if (CQHCI_TERRI_C_VALID(terri)) {
+		tag = CQHCI_TERRI_C_TASK(terri);
+		slot = &cq_host->slot[tag];
+		if (slot->mrq) {
+			slot->flags = cqhci_error_flags(cmd_error, data_error);
+			cqhci_recovery_needed(mmc, slot->mrq, true);
+		}
+	}
+
+	if (CQHCI_TERRI_D_VALID(terri)) {
+		tag = CQHCI_TERRI_D_TASK(terri);
+		slot = &cq_host->slot[tag];
+		if (slot->mrq) {
+			slot->flags = cqhci_error_flags(data_error, cmd_error);
+			cqhci_recovery_needed(mmc, slot->mrq, true);
+		}
+	}
+
+	if (!cq_host->recovery_halt) {
+		/*
+		 * The only way to guarantee forward progress is to mark at
+		 * least one task in error, so if none is indicated, pick one.
+		 */
+		for (tag = 0; tag < NUM_SLOTS; tag++) {
+			slot = &cq_host->slot[tag];
+			if (!slot->mrq)
+				continue;
+			slot->flags = cqhci_error_flags(data_error, cmd_error);
+			cqhci_recovery_needed(mmc, slot->mrq, true);
+			break;
+		}
+	}
+
+out_unlock:
+	spin_unlock(&cq_host->lock);
+}
+
+static void cqhci_finish_mrq(struct mmc_host *mmc, unsigned int tag)
+{
+	struct cqhci_host *cq_host = mmc->cqe_private;
+	struct cqhci_slot *slot = &cq_host->slot[tag];
+	struct mmc_request *mrq = slot->mrq;
+	struct mmc_data *data;
+
+	if (!mrq) {
+		WARN_ONCE(1, "%s: cqhci: spurious TCN for tag %d\n",
+			  mmc_hostname(mmc), tag);
+		return;
+	}
+
+	/* No completions allowed during recovery */
+	if (cq_host->recovery_halt) {
+		slot->flags |= CQHCI_COMPLETED;
+		return;
+	}
+
+	slot->mrq = NULL;
+
+	cq_host->qcnt -= 1;
+
+	data = mrq->data;
+	if (data) {
+		if (data->error)
+			data->bytes_xfered = 0;
+		else
+			data->bytes_xfered = data->blksz * data->blocks;
+	}
+
+	mmc_cqe_request_done(mmc, mrq);
+}
+
+irqreturn_t cqhci_irq(struct mmc_host *mmc, u32 intmask, int cmd_error,
+		      int data_error)
+{
+	u32 status;
+	unsigned long tag = 0, comp_status;
+	struct cqhci_host *cq_host = mmc->cqe_private;
+
+	status = cqhci_readl(cq_host, CQHCI_IS);
+	cqhci_writel(cq_host, status, CQHCI_IS);
+
+	pr_debug("%s: cqhci: IRQ status: 0x%08x\n", mmc_hostname(mmc), status);
+
+	if ((status & CQHCI_IS_RED) || cmd_error || data_error)
+		cqhci_error_irq(mmc, status, cmd_error, data_error);
+
+	if (status & CQHCI_IS_TCC) {
+		/* read TCN and complete the request */
+		comp_status = cqhci_readl(cq_host, CQHCI_TCN);
+		cqhci_writel(cq_host, comp_status, CQHCI_TCN);
+		pr_debug("%s: cqhci: TCN: 0x%08lx\n",
+			 mmc_hostname(mmc), comp_status);
+
+		spin_lock(&cq_host->lock);
+
+		for_each_set_bit(tag, &comp_status, cq_host->num_slots) {
+			/* complete the corresponding mrq */
+			pr_debug("%s: cqhci: completing tag %lu\n",
+				 mmc_hostname(mmc), tag);
+			cqhci_finish_mrq(mmc, tag);
+		}
+
+		if (cq_host->waiting_for_idle && !cq_host->qcnt) {
+			cq_host->waiting_for_idle = false;
+			wake_up(&cq_host->wait_queue);
+		}
+
+		spin_unlock(&cq_host->lock);
+	}
+
+	if (status & CQHCI_IS_TCL)
+		wake_up(&cq_host->wait_queue);
+
+	if (status & CQHCI_IS_HAC)
+		wake_up(&cq_host->wait_queue);
+
+	return IRQ_HANDLED;
+}
+EXPORT_SYMBOL(cqhci_irq);
+
+static bool cqhci_is_idle(struct cqhci_host *cq_host, int *ret)
+{
+	unsigned long flags;
+	bool is_idle;
+
+	spin_lock_irqsave(&cq_host->lock, flags);
+	is_idle = !cq_host->qcnt || cq_host->recovery_halt;
+	*ret = cq_host->recovery_halt ? -EBUSY : 0;
+	cq_host->waiting_for_idle = !is_idle;
+	spin_unlock_irqrestore(&cq_host->lock, flags);
+
+	return is_idle;
+}
+
+static int cqhci_wait_for_idle(struct mmc_host *mmc)
+{
+	struct cqhci_host *cq_host = mmc->cqe_private;
+	int ret;
+
+	wait_event(cq_host->wait_queue, cqhci_is_idle(cq_host, &ret));
+
+	return ret;
+}
+
+static bool cqhci_timeout(struct mmc_host *mmc, struct mmc_request *mrq,
+			  bool *recovery_needed)
+{
+	struct cqhci_host *cq_host = mmc->cqe_private;
+	int tag = cqhci_tag(mrq);
+	struct cqhci_slot *slot = &cq_host->slot[tag];
+	unsigned long flags;
+	bool timed_out;
+
+	spin_lock_irqsave(&cq_host->lock, flags);
+	timed_out = slot->mrq == mrq;
+	if (timed_out) {
+		slot->flags |= CQHCI_EXTERNAL_TIMEOUT;
+		cqhci_recovery_needed(mmc, mrq, false);
+		*recovery_needed = cq_host->recovery_halt;
+	}
+	spin_unlock_irqrestore(&cq_host->lock, flags);
+
+	if (timed_out) {
+		pr_err("%s: cqhci: timeout for tag %d\n",
+		       mmc_hostname(mmc), tag);
+		cqhci_dumpregs(cq_host);
+	}
+
+	return timed_out;
+}
+
+static bool cqhci_tasks_cleared(struct cqhci_host *cq_host)
+{
+	return !(cqhci_readl(cq_host, CQHCI_CTL) & CQHCI_CLEAR_ALL_TASKS);
+}
+
+static bool cqhci_clear_all_tasks(struct mmc_host *mmc, unsigned int timeout)
+{
+	struct cqhci_host *cq_host = mmc->cqe_private;
+	bool ret;
+	u32 ctl;
+
+	cqhci_set_irqs(cq_host, CQHCI_IS_TCL);
+
+	ctl = cqhci_readl(cq_host, CQHCI_CTL);
+	ctl |= CQHCI_CLEAR_ALL_TASKS;
+	cqhci_writel(cq_host, ctl, CQHCI_CTL);
+
+	wait_event_timeout(cq_host->wait_queue, cqhci_tasks_cleared(cq_host),
+			   msecs_to_jiffies(timeout) + 1);
+
+	cqhci_set_irqs(cq_host, 0);
+
+	ret = cqhci_tasks_cleared(cq_host);
+
+	if (!ret)
+		pr_debug("%s: cqhci: Failed to clear tasks\n",
+			 mmc_hostname(mmc));
+
+	return ret;
+}
+
+static bool cqhci_halted(struct cqhci_host *cq_host)
+{
+	return cqhci_readl(cq_host, CQHCI_CTL) & CQHCI_HALT;
+}
+
+static bool cqhci_halt(struct mmc_host *mmc, unsigned int timeout)
+{
+	struct cqhci_host *cq_host = mmc->cqe_private;
+	bool ret;
+	u32 ctl;
+
+	if (cqhci_halted(cq_host))
+		return true;
+
+	cqhci_set_irqs(cq_host, CQHCI_IS_HAC);
+
+	ctl = cqhci_readl(cq_host, CQHCI_CTL);
+	ctl |= CQHCI_HALT;
+	cqhci_writel(cq_host, ctl, CQHCI_CTL);
+
+	wait_event_timeout(cq_host->wait_queue, cqhci_halted(cq_host),
+			   msecs_to_jiffies(timeout) + 1);
+
+	cqhci_set_irqs(cq_host, 0);
+
+	ret = cqhci_halted(cq_host);
+
+	if (!ret)
+		pr_debug("%s: cqhci: Failed to halt\n", mmc_hostname(mmc));
+
+	return ret;
+}
+
+/*
+ * After halting we expect to be able to use the command line. We interpret the
+ * failure to halt to mean the data lines might still be in use (and the upper
+ * layers will need to send a STOP command), so we set the timeout based on a
+ * generous command timeout.
+ */
+#define CQHCI_START_HALT_TIMEOUT	5
+
+static void cqhci_recovery_start(struct mmc_host *mmc)
+{
+	struct cqhci_host *cq_host = mmc->cqe_private;
+
+	pr_debug("%s: cqhci: %s\n", mmc_hostname(mmc), __func__);
+
+	WARN_ON(!cq_host->recovery_halt);
+
+	cqhci_halt(mmc, CQHCI_START_HALT_TIMEOUT);
+
+	if (cq_host->ops->disable)
+		cq_host->ops->disable(mmc, true);
+
+	mmc->cqe_on = false;
+}
+
+static int cqhci_error_from_flags(unsigned int flags)
+{
+	if (!flags)
+		return 0;
+
+	/* CRC errors might indicate re-tuning so prefer to report that */
+	if (flags & CQHCI_HOST_CRC)
+		return -EILSEQ;
+
+	if (flags & (CQHCI_EXTERNAL_TIMEOUT | CQHCI_HOST_TIMEOUT))
+		return -ETIMEDOUT;
+
+	return -EIO;
+}
+
+static void cqhci_recover_mrq(struct cqhci_host *cq_host, unsigned int tag)
+{
+	struct cqhci_slot *slot = &cq_host->slot[tag];
+	struct mmc_request *mrq = slot->mrq;
+	struct mmc_data *data;
+
+	if (!mrq)
+		return;
+
+	slot->mrq = NULL;
+
+	cq_host->qcnt -= 1;
+
+	data = mrq->data;
+	if (data) {
+		data->bytes_xfered = 0;
+		data->error = cqhci_error_from_flags(slot->flags);
+	} else {
+		mrq->cmd->error = cqhci_error_from_flags(slot->flags);
+	}
+
+	mmc_cqe_request_done(cq_host->mmc, mrq);
+}
+
+static void cqhci_recover_mrqs(struct cqhci_host *cq_host)
+{
+	int i;
+
+	for (i = 0; i < cq_host->num_slots; i++)
+		cqhci_recover_mrq(cq_host, i);
+}
+
+/*
+ * By now the command and data lines should be unused so there is no reason for
+ * CQHCI to take a long time to halt, but if it doesn't halt there could be
+ * problems clearing tasks, so be generous.
+ */
+#define CQHCI_FINISH_HALT_TIMEOUT	20
+
+/* CQHCI could be expected to clear it's internal state pretty quickly */
+#define CQHCI_CLEAR_TIMEOUT		20
+
+static void cqhci_recovery_finish(struct mmc_host *mmc)
+{
+	struct cqhci_host *cq_host = mmc->cqe_private;
+	unsigned long flags;
+	u32 cqcfg;
+	bool ok;
+
+	pr_debug("%s: cqhci: %s\n", mmc_hostname(mmc), __func__);
+
+	WARN_ON(!cq_host->recovery_halt);
+
+	ok = cqhci_halt(mmc, CQHCI_FINISH_HALT_TIMEOUT);
+
+	if (!cqhci_clear_all_tasks(mmc, CQHCI_CLEAR_TIMEOUT))
+		ok = false;
+
+	/*
+	 * The specification contradicts itself, by saying that tasks cannot be
+	 * cleared if CQHCI does not halt, but if CQHCI does not halt, it should
+	 * be disabled/re-enabled, but not to disable before clearing tasks.
+	 * Have a go anyway.
+	 */
+	if (!ok) {
+		pr_debug("%s: cqhci: disable / re-enable\n", mmc_hostname(mmc));
+		cqcfg = cqhci_readl(cq_host, CQHCI_CFG);
+		cqcfg &= ~CQHCI_ENABLE;
+		cqhci_writel(cq_host, cqcfg, CQHCI_CFG);
+		cqcfg |= CQHCI_ENABLE;
+		cqhci_writel(cq_host, cqcfg, CQHCI_CFG);
+		/* Be sure that there are no tasks */
+		ok = cqhci_halt(mmc, CQHCI_FINISH_HALT_TIMEOUT);
+		if (!cqhci_clear_all_tasks(mmc, CQHCI_CLEAR_TIMEOUT))
+			ok = false;
+		WARN_ON(!ok);
+	}
+
+	cqhci_recover_mrqs(cq_host);
+
+	WARN_ON(cq_host->qcnt);
+
+	spin_lock_irqsave(&cq_host->lock, flags);
+	cq_host->qcnt = 0;
+	cq_host->recovery_halt = false;
+	mmc->cqe_on = false;
+	spin_unlock_irqrestore(&cq_host->lock, flags);
+
+	/* Ensure all writes are done before interrupts are re-enabled */
+	wmb();
+
+	cqhci_writel(cq_host, CQHCI_IS_HAC | CQHCI_IS_TCL, CQHCI_IS);
+
+	cqhci_set_irqs(cq_host, CQHCI_IS_MASK);
+
+	pr_debug("%s: cqhci: recovery done\n", mmc_hostname(mmc));
+}
+
+static const struct mmc_cqe_ops cqhci_cqe_ops = {
+	.cqe_enable = cqhci_enable,
+	.cqe_disable = cqhci_disable,
+	.cqe_request = cqhci_request,
+	.cqe_post_req = cqhci_post_req,
+	.cqe_off = cqhci_off,
+	.cqe_wait_for_idle = cqhci_wait_for_idle,
+	.cqe_timeout = cqhci_timeout,
+	.cqe_recovery_start = cqhci_recovery_start,
+	.cqe_recovery_finish = cqhci_recovery_finish,
+};
+
+struct cqhci_host *cqhci_pltfm_init(struct platform_device *pdev)
+{
+	struct cqhci_host *cq_host;
+	struct resource *cqhci_memres = NULL;
+
+	/* check and setup CMDQ interface */
+	cqhci_memres = platform_get_resource_byname(pdev, IORESOURCE_MEM,
+						   "cqhci_mem");
+	if (!cqhci_memres) {
+		dev_dbg(&pdev->dev, "CMDQ not supported\n");
+		return ERR_PTR(-EINVAL);
+	}
+
+	cq_host = devm_kzalloc(&pdev->dev, sizeof(*cq_host), GFP_KERNEL);
+	if (!cq_host)
+		return ERR_PTR(-ENOMEM);
+	cq_host->mmio = devm_ioremap(&pdev->dev,
+				     cqhci_memres->start,
+				     resource_size(cqhci_memres));
+	if (!cq_host->mmio) {
+		dev_err(&pdev->dev, "failed to remap cqhci regs\n");
+		return ERR_PTR(-EBUSY);
+	}
+	dev_dbg(&pdev->dev, "CMDQ ioremap: done\n");
+
+	return cq_host;
+}
+EXPORT_SYMBOL(cqhci_pltfm_init);
+
+static unsigned int cqhci_ver_major(struct cqhci_host *cq_host)
+{
+	return CQHCI_VER_MAJOR(cqhci_readl(cq_host, CQHCI_VER));
+}
+
+static unsigned int cqhci_ver_minor(struct cqhci_host *cq_host)
+{
+	u32 ver = cqhci_readl(cq_host, CQHCI_VER);
+
+	return CQHCI_VER_MINOR1(ver) * 10 + CQHCI_VER_MINOR2(ver);
+}
+
+int cqhci_init(struct cqhci_host *cq_host, struct mmc_host *mmc,
+	      bool dma64)
+{
+	int err;
+
+	cq_host->dma64 = dma64;
+	cq_host->mmc = mmc;
+	cq_host->mmc->cqe_private = cq_host;
+
+	cq_host->num_slots = NUM_SLOTS;
+	cq_host->dcmd_slot = DCMD_SLOT;
+
+	mmc->cqe_ops = &cqhci_cqe_ops;
+
+	mmc->cqe_qdepth = NUM_SLOTS;
+	if (mmc->caps2 & MMC_CAP2_CQE_DCMD)
+		mmc->cqe_qdepth -= 1;
+
+	cq_host->slot = devm_kcalloc(mmc_dev(mmc), cq_host->num_slots,
+				     sizeof(*cq_host->slot), GFP_KERNEL);
+	if (!cq_host->slot) {
+		err = -ENOMEM;
+		goto out_err;
+	}
+
+	spin_lock_init(&cq_host->lock);
+
+	init_completion(&cq_host->halt_comp);
+	init_waitqueue_head(&cq_host->wait_queue);
+
+	pr_info("%s: CQHCI version %u.%02u\n",
+		mmc_hostname(mmc), cqhci_ver_major(cq_host),
+		cqhci_ver_minor(cq_host));
+
+	return 0;
+
+out_err:
+	pr_err("%s: CQHCI version %u.%02u failed to initialize, error %d\n",
+	       mmc_hostname(mmc), cqhci_ver_major(cq_host),
+	       cqhci_ver_minor(cq_host), err);
+	return err;
+}
+EXPORT_SYMBOL(cqhci_init);
+
+MODULE_AUTHOR("Venkat Gopalakrishnan <venkatg@codeaurora.org>");
+MODULE_DESCRIPTION("Command Queue Host Controller Interface driver");
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/mmc/host/cqhci.h b/drivers/mmc/host/cqhci.h
new file mode 100644
index 000000000000..2d39d361b322
--- /dev/null
+++ b/drivers/mmc/host/cqhci.h
@@ -0,0 +1,240 @@
+/* Copyright (c) 2015, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+#ifndef LINUX_MMC_CQHCI_H
+#define LINUX_MMC_CQHCI_H
+
+#include <linux/compiler.h>
+#include <linux/bitops.h>
+#include <linux/spinlock_types.h>
+#include <linux/types.h>
+#include <linux/completion.h>
+#include <linux/wait.h>
+#include <linux/irqreturn.h>
+#include <asm/io.h>
+
+/* registers */
+/* version */
+#define CQHCI_VER			0x00
+#define CQHCI_VER_MAJOR(x)		(((x) & GENMASK(11, 8)) >> 8)
+#define CQHCI_VER_MINOR1(x)		(((x) & GENMASK(7, 4)) >> 4)
+#define CQHCI_VER_MINOR2(x)		((x) & GENMASK(3, 0))
+
+/* capabilities */
+#define CQHCI_CAP			0x04
+/* configuration */
+#define CQHCI_CFG			0x08
+#define CQHCI_DCMD			0x00001000
+#define CQHCI_TASK_DESC_SZ		0x00000100
+#define CQHCI_ENABLE			0x00000001
+
+/* control */
+#define CQHCI_CTL			0x0C
+#define CQHCI_CLEAR_ALL_TASKS		0x00000100
+#define CQHCI_HALT			0x00000001
+
+/* interrupt status */
+#define CQHCI_IS			0x10
+#define CQHCI_IS_HAC			BIT(0)
+#define CQHCI_IS_TCC			BIT(1)
+#define CQHCI_IS_RED			BIT(2)
+#define CQHCI_IS_TCL			BIT(3)
+
+#define CQHCI_IS_MASK (CQHCI_IS_TCC | CQHCI_IS_RED)
+
+/* interrupt status enable */
+#define CQHCI_ISTE			0x14
+
+/* interrupt signal enable */
+#define CQHCI_ISGE			0x18
+
+/* interrupt coalescing */
+#define CQHCI_IC			0x1C
+#define CQHCI_IC_ENABLE			BIT(31)
+#define CQHCI_IC_RESET			BIT(16)
+#define CQHCI_IC_ICCTHWEN		BIT(15)
+#define CQHCI_IC_ICCTH(x)		((x & 0x1F) << 8)
+#define CQHCI_IC_ICTOVALWEN		BIT(7)
+#define CQHCI_IC_ICTOVAL(x)		(x & 0x7F)
+
+/* task list base address */
+#define CQHCI_TDLBA			0x20
+
+/* task list base address upper */
+#define CQHCI_TDLBAU			0x24
+
+/* door-bell */
+#define CQHCI_TDBR			0x28
+
+/* task completion notification */
+#define CQHCI_TCN			0x2C
+
+/* device queue status */
+#define CQHCI_DQS			0x30
+
+/* device pending tasks */
+#define CQHCI_DPT			0x34
+
+/* task clear */
+#define CQHCI_TCLR			0x38
+
+/* send status config 1 */
+#define CQHCI_SSC1			0x40
+
+/* send status config 2 */
+#define CQHCI_SSC2			0x44
+
+/* response for dcmd */
+#define CQHCI_CRDCT			0x48
+
+/* response mode error mask */
+#define CQHCI_RMEM			0x50
+
+/* task error info */
+#define CQHCI_TERRI			0x54
+
+#define CQHCI_TERRI_C_INDEX(x)		((x) & GENMASK(5, 0))
+#define CQHCI_TERRI_C_TASK(x)		(((x) & GENMASK(12, 8)) >> 8)
+#define CQHCI_TERRI_C_VALID(x)		((x) & BIT(15))
+#define CQHCI_TERRI_D_INDEX(x)		(((x) & GENMASK(21, 16)) >> 16)
+#define CQHCI_TERRI_D_TASK(x)		(((x) & GENMASK(28, 24)) >> 24)
+#define CQHCI_TERRI_D_VALID(x)		((x) & BIT(31))
+
+/* command response index */
+#define CQHCI_CRI			0x58
+
+/* command response argument */
+#define CQHCI_CRA			0x5C
+
+#define CQHCI_INT_ALL			0xF
+#define CQHCI_IC_DEFAULT_ICCTH		31
+#define CQHCI_IC_DEFAULT_ICTOVAL	1
+
+/* attribute fields */
+#define CQHCI_VALID(x)			((x & 1) << 0)
+#define CQHCI_END(x)			((x & 1) << 1)
+#define CQHCI_INT(x)			((x & 1) << 2)
+#define CQHCI_ACT(x)			((x & 0x7) << 3)
+
+/* data command task descriptor fields */
+#define CQHCI_FORCED_PROG(x)		((x & 1) << 6)
+#define CQHCI_CONTEXT(x)		((x & 0xF) << 7)
+#define CQHCI_DATA_TAG(x)		((x & 1) << 11)
+#define CQHCI_DATA_DIR(x)		((x & 1) << 12)
+#define CQHCI_PRIORITY(x)		((x & 1) << 13)
+#define CQHCI_QBAR(x)			((x & 1) << 14)
+#define CQHCI_REL_WRITE(x)		((x & 1) << 15)
+#define CQHCI_BLK_COUNT(x)		((x & 0xFFFF) << 16)
+#define CQHCI_BLK_ADDR(x)		((x & 0xFFFFFFFF) << 32)
+
+/* direct command task descriptor fields */
+#define CQHCI_CMD_INDEX(x)		((x & 0x3F) << 16)
+#define CQHCI_CMD_TIMING(x)		((x & 1) << 22)
+#define CQHCI_RESP_TYPE(x)		((x & 0x3) << 23)
+
+/* transfer descriptor fields */
+#define CQHCI_DAT_LENGTH(x)		((x & 0xFFFF) << 16)
+#define CQHCI_DAT_ADDR_LO(x)		((x & 0xFFFFFFFF) << 32)
+#define CQHCI_DAT_ADDR_HI(x)		((x & 0xFFFFFFFF) << 0)
+
+struct cqhci_host_ops;
+struct mmc_host;
+struct cqhci_slot;
+
+struct cqhci_host {
+	const struct cqhci_host_ops *ops;
+	void __iomem *mmio;
+	struct mmc_host *mmc;
+
+	spinlock_t lock;
+
+	/* relative card address of device */
+	unsigned int rca;
+
+	/* 64 bit DMA */
+	bool dma64;
+	int num_slots;
+	int qcnt;
+
+	u32 dcmd_slot;
+	u32 caps;
+#define CQHCI_TASK_DESC_SZ_128		0x1
+
+	u32 quirks;
+#define CQHCI_QUIRK_SHORT_TXFR_DESC_SZ	0x1
+
+	bool enabled;
+	bool halted;
+	bool init_done;
+	bool activated;
+	bool waiting_for_idle;
+	bool recovery_halt;
+
+	size_t desc_size;
+	size_t data_size;
+
+	u8 *desc_base;
+
+	/* total descriptor size */
+	u8 slot_sz;
+
+	/* 64/128 bit depends on CQHCI_CFG */
+	u8 task_desc_len;
+
+	/* 64 bit on 32-bit arch, 128 bit on 64-bit */
+	u8 link_desc_len;
+
+	u8 *trans_desc_base;
+	/* same length as transfer descriptor */
+	u8 trans_desc_len;
+
+	dma_addr_t desc_dma_base;
+	dma_addr_t trans_desc_dma_base;
+
+	struct completion halt_comp;
+	wait_queue_head_t wait_queue;
+	struct cqhci_slot *slot;
+};
+
+struct cqhci_host_ops {
+	void (*dumpregs)(struct mmc_host *mmc);
+	void (*write_l)(struct cqhci_host *host, u32 val, int reg);
+	u32 (*read_l)(struct cqhci_host *host, int reg);
+	void (*enable)(struct mmc_host *mmc);
+	void (*disable)(struct mmc_host *mmc, bool recovery);
+};
+
+static inline void cqhci_writel(struct cqhci_host *host, u32 val, int reg)
+{
+	if (unlikely(host->ops->write_l))
+		host->ops->write_l(host, val, reg);
+	else
+		writel_relaxed(val, host->mmio + reg);
+}
+
+static inline u32 cqhci_readl(struct cqhci_host *host, int reg)
+{
+	if (unlikely(host->ops->read_l))
+		return host->ops->read_l(host, reg);
+	else
+		return readl_relaxed(host->mmio + reg);
+}
+
+struct platform_device;
+
+irqreturn_t cqhci_irq(struct mmc_host *mmc, u32 intmask, int cmd_error,
+		      int data_error);
+int cqhci_init(struct cqhci_host *cq_host, struct mmc_host *mmc, bool dma64);
+struct cqhci_host *cqhci_pltfm_init(struct platform_device *pdev);
+int cqhci_suspend(struct mmc_host *mmc);
+int cqhci_resume(struct mmc_host *mmc);
+
+#endif
-- 
1.9.1

[PATCH V4 11/11] mmc: sdhci-pci: Add CQHCI support for Intel GLK

[Adrian Hunter]

Add CQHCI initialization and implement CQHCI operations for Intel GLK.

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
---
 drivers/mmc/host/Kconfig          |   1 +
 drivers/mmc/host/sdhci-pci-core.c | 153 +++++++++++++++++++++++++++++++++++++-
 2 files changed, 153 insertions(+), 1 deletion(-)

diff --git a/drivers/mmc/host/Kconfig b/drivers/mmc/host/Kconfig
index 3a164a03f2bc..ac365ac5fc76 100644
--- a/drivers/mmc/host/Kconfig
+++ b/drivers/mmc/host/Kconfig
@@ -72,6 +72,7 @@ config MMC_SDHCI_BIG_ENDIAN_32BIT_BYTE_SWAPPER
 config MMC_SDHCI_PCI
 	tristate "SDHCI support on PCI bus"
 	depends on MMC_SDHCI && PCI
+	select MMC_CQHCI
 	help
 	  This selects the PCI Secure Digital Host Controller Interface.
 	  Most controllers found today are PCI devices.
diff --git a/drivers/mmc/host/sdhci-pci-core.c b/drivers/mmc/host/sdhci-pci-core.c
index e1721ac37919..45e507487031 100644
--- a/drivers/mmc/host/sdhci-pci-core.c
+++ b/drivers/mmc/host/sdhci-pci-core.c
@@ -30,6 +30,8 @@
 #include <linux/mmc/sdhci-pci-data.h>
 #include <linux/acpi.h>
 
+#include "cqhci.h"
+
 #include "sdhci.h"
 #include "sdhci-pci.h"
 #include "sdhci-pci-o2micro.h"
@@ -118,6 +120,28 @@ int sdhci_pci_resume_host(struct sdhci_pci_chip *chip)
 
 	return 0;
 }
+
+static int sdhci_cqhci_suspend(struct sdhci_pci_chip *chip)
+{
+	int ret;
+
+	ret = cqhci_suspend(chip->slots[0]->host->mmc);
+	if (ret)
+		return ret;
+
+	return sdhci_pci_suspend_host(chip);
+}
+
+static int sdhci_cqhci_resume(struct sdhci_pci_chip *chip)
+{
+	int ret;
+
+	ret = sdhci_pci_resume_host(chip);
+	if (ret)
+		return ret;
+
+	return cqhci_resume(chip->slots[0]->host->mmc);
+}
 #endif
 
 #ifdef CONFIG_PM
@@ -168,8 +192,48 @@ static int sdhci_pci_runtime_resume_host(struct sdhci_pci_chip *chip)
 
 	return 0;
 }
+
+static int sdhci_cqhci_runtime_suspend(struct sdhci_pci_chip *chip)
+{
+	int ret;
+
+	ret = cqhci_suspend(chip->slots[0]->host->mmc);
+	if (ret)
+		return ret;
+
+	return sdhci_pci_runtime_suspend_host(chip);
+}
+
+static int sdhci_cqhci_runtime_resume(struct sdhci_pci_chip *chip)
+{
+	int ret;
+
+	ret = sdhci_pci_runtime_resume_host(chip);
+	if (ret)
+		return ret;
+
+	return cqhci_resume(chip->slots[0]->host->mmc);
+}
 #endif
 
+static u32 sdhci_cqhci_irq(struct sdhci_host *host, u32 intmask)
+{
+	int cmd_error = 0;
+	int data_error = 0;
+
+	if (!sdhci_cqe_irq(host, intmask, &cmd_error, &data_error))
+		return intmask;
+
+	cqhci_irq(host->mmc, intmask, cmd_error, data_error);
+
+	return 0;
+}
+
+static void sdhci_pci_dumpregs(struct mmc_host *mmc)
+{
+	sdhci_dumpregs(mmc_priv(mmc));
+}
+
 /*****************************************************************************\
  *                                                                           *
  * Hardware specific quirk handling                                          *
@@ -568,6 +632,17 @@ static void intel_hs400_enhanced_strobe(struct mmc_host *mmc,
 	.hw_reset		= sdhci_pci_hw_reset,
 };
 
+static const struct sdhci_ops sdhci_intel_glk_ops = {
+	.set_clock		= sdhci_set_clock,
+	.set_power		= sdhci_intel_set_power,
+	.enable_dma		= sdhci_pci_enable_dma,
+	.set_bus_width		= sdhci_pci_set_bus_width,
+	.reset			= sdhci_reset,
+	.set_uhs_signaling	= sdhci_set_uhs_signaling,
+	.hw_reset		= sdhci_pci_hw_reset,
+	.irq			= sdhci_cqhci_irq,
+};
+
 static void byt_read_dsm(struct sdhci_pci_slot *slot)
 {
 	struct intel_host *intel_host = sdhci_pci_priv(slot);
@@ -597,6 +672,8 @@ static int glk_emmc_probe_slot(struct sdhci_pci_slot *slot)
 {
 	int ret = byt_emmc_probe_slot(slot);
 
+	slot->host->mmc->caps2 |= MMC_CAP2_CQE;
+
 	if (slot->chip->pdev->device != PCI_DEVICE_ID_INTEL_GLK_EMMC) {
 		slot->host->mmc->caps2 |= MMC_CAP2_HS400_ES,
 		slot->host->mmc_host_ops.hs400_enhanced_strobe =
@@ -606,6 +683,71 @@ static int glk_emmc_probe_slot(struct sdhci_pci_slot *slot)
 	return ret;
 }
 
+static void glk_cqe_enable(struct mmc_host *mmc)
+{
+	struct sdhci_host *host = mmc_priv(mmc);
+	u32 reg;
+
+	/*
+	 * CQE gets stuck if it sees Buffer Read Enable bit set, which can be
+	 * the case after tuning, so ensure the buffer is drained.
+	 */
+	reg = sdhci_readl(host, SDHCI_PRESENT_STATE);
+	while (reg & SDHCI_DATA_AVAILABLE) {
+		sdhci_readl(host, SDHCI_BUFFER);
+		reg = sdhci_readl(host, SDHCI_PRESENT_STATE);
+	}
+
+	sdhci_cqe_enable(mmc);
+}
+
+static const struct cqhci_host_ops glk_cqhci_ops = {
+	.enable		= glk_cqe_enable,
+	.disable	= sdhci_cqe_disable,
+	.dumpregs	= sdhci_pci_dumpregs,
+};
+
+static int glk_emmc_add_host(struct sdhci_pci_slot *slot)
+{
+	struct device *dev = &slot->chip->pdev->dev;
+	struct sdhci_host *host = slot->host;
+	struct cqhci_host *cq_host;
+	bool dma64;
+	int ret;
+
+	ret = sdhci_setup_host(host);
+	if (ret)
+		return ret;
+
+	cq_host = devm_kzalloc(dev, sizeof(*cq_host), GFP_KERNEL);
+	if (!cq_host) {
+		ret = -ENOMEM;
+		goto cleanup;
+	}
+
+	cq_host->mmio = host->ioaddr + 0x200;
+	cq_host->quirks |= CQHCI_QUIRK_SHORT_TXFR_DESC_SZ;
+	cq_host->ops = &glk_cqhci_ops;
+
+	dma64 = host->flags & SDHCI_USE_64_BIT_DMA;
+	if (dma64)
+		cq_host->caps |= CQHCI_TASK_DESC_SZ_128;
+
+	ret = cqhci_init(cq_host, host->mmc, dma64);
+	if (ret)
+		goto cleanup;
+
+	ret = __sdhci_add_host(host);
+	if (ret)
+		goto cleanup;
+
+	return 0;
+
+cleanup:
+	sdhci_cleanup_host(host);
+	return ret;
+}
+
 #ifdef CONFIG_ACPI
 static int ni_set_max_freq(struct sdhci_pci_slot *slot)
 {
@@ -684,11 +826,20 @@ static int byt_sd_probe_slot(struct sdhci_pci_slot *slot)
 static const struct sdhci_pci_fixes sdhci_intel_glk_emmc = {
 	.allow_runtime_pm	= true,
 	.probe_slot		= glk_emmc_probe_slot,
+	.add_host		= glk_emmc_add_host,
+#ifdef CONFIG_PM_SLEEP
+	.suspend		= sdhci_cqhci_suspend,
+	.resume			= sdhci_cqhci_resume,
+#endif
+#ifdef CONFIG_PM
+	.runtime_suspend	= sdhci_cqhci_runtime_suspend,
+	.runtime_resume		= sdhci_cqhci_runtime_resume,
+#endif
 	.quirks			= SDHCI_QUIRK_NO_ENDATTR_IN_NOPDESC,
 	.quirks2		= SDHCI_QUIRK2_PRESET_VALUE_BROKEN |
 				  SDHCI_QUIRK2_CAPS_BIT63_FOR_HS400 |
 				  SDHCI_QUIRK2_STOP_WITH_TC,
-	.ops			= &sdhci_intel_byt_ops,
+	.ops			= &sdhci_intel_glk_ops,
 	.priv_size		= sizeof(struct intel_host),
 };
 
-- 
1.9.1

Re: [PATCH V4 09/11] mmc: block: Add CQE support

[Shawn Lin]

Hi Adrian,

On 2017/7/21 17:49, Adrian Hunter wrote:
> Add CQE support to the block driver, including:
> 	- optionally using DCMD for flush requests

That doesn't seem to be fact when I was looking into
the patch 10, as it always prepare DCMD with QBR
whenever not finding mrq->data. Do I miss something?

> 	- manually issuing discard requests
> 	- issuing read / write requests to the CQE
> 	- supporting block-layer timeouts
> 	- handling recovery
> 	- supporting re-tuning
>
> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
> ---
>  drivers/mmc/core/block.c | 195 ++++++++++++++++++++++++++++++++-
>  drivers/mmc/core/block.h |   7 ++
>  drivers/mmc/core/queue.c | 273 ++++++++++++++++++++++++++++++++++++++++++++++-
>  drivers/mmc/core/queue.h |  42 +++++++-
>  4 files changed, 510 insertions(+), 7 deletions(-)
>
> diff --git a/drivers/mmc/core/block.c b/drivers/mmc/core/block.c
> index 915290c74363..2d25115637b7 100644
> --- a/drivers/mmc/core/block.c
> +++ b/drivers/mmc/core/block.c
> @@ -109,6 +109,7 @@ struct mmc_blk_data {
>  #define MMC_BLK_WRITE		BIT(1)
>  #define MMC_BLK_DISCARD		BIT(2)
>  #define MMC_BLK_SECDISCARD	BIT(3)
> +#define MMC_BLK_CQE_RECOVERY	BIT(4)
>
>  	/*
>  	 * Only set in main mmc_blk_data associated
> @@ -1612,6 +1613,198 @@ static void mmc_blk_data_prep(struct mmc_queue *mq, struct mmc_queue_req *mqrq,
>  		*do_data_tag_p = do_data_tag;
>  }
>
> +#define MMC_CQE_RETRIES 2
> +
> +void mmc_blk_cqe_complete_rq(struct request *req)
> +{
> +	struct mmc_queue_req *mqrq = req_to_mmc_queue_req(req);
> +	struct mmc_request *mrq = &mqrq->brq.mrq;
> +	struct request_queue *q = req->q;
> +	struct mmc_queue *mq = q->queuedata;
> +	struct mmc_host *host = mq->card->host;
> +	unsigned long flags;
> +	bool put_card;
> +	int err;
> +
> +	mmc_cqe_post_req(host, mrq);
> +
> +	spin_lock_irqsave(q->queue_lock, flags);
> +
> +	mq->cqe_in_flight[mmc_cqe_issue_type(host, req)] -= 1;
> +
> +	put_card = mmc_cqe_tot_in_flight(mq) == 0;
> +
> +	if (mrq->cmd && mrq->cmd->error)
> +		err = mrq->cmd->error;
> +	else if (mrq->data && mrq->data->error)
> +		err = mrq->data->error;
> +	else
> +		err = 0;
> +
> +	if (err) {
> +		if (mqrq->retries++ < MMC_CQE_RETRIES)
> +			blk_requeue_request(q, req);
> +		else
> +			__blk_end_request_all(req, BLK_STS_IOERR);
> +	} else if (mrq->data) {
> +		if (__blk_end_request(req, BLK_STS_OK, mrq->data->bytes_xfered))
> +			blk_requeue_request(q, req);
> +	} else {
> +		__blk_end_request_all(req, BLK_STS_OK);
> +	}
> +
> +	mmc_cqe_kick_queue(mq);
> +
> +	spin_unlock_irqrestore(q->queue_lock, flags);
> +
> +	if (put_card)
> +		mmc_put_card(mq->card);
> +}
> +
> +void mmc_blk_cqe_recovery(struct mmc_queue *mq)
> +{
> +	struct mmc_card *card = mq->card;
> +	struct mmc_host *host = card->host;
> +	int err;
> +
> +	mmc_get_card(card);
> +
> +	pr_debug("%s: CQE recovery start\n", mmc_hostname(host));
> +
> +	mq->cqe_in_recovery = true;
> +
> +	err = mmc_cqe_recovery(host);
> +	if (err)
> +		mmc_blk_reset(mq->blkdata, host, MMC_BLK_CQE_RECOVERY);
> +	else
> +		mmc_blk_reset_success(mq->blkdata, MMC_BLK_CQE_RECOVERY);
> +
> +	mq->cqe_in_recovery = false;
> +
> +	pr_debug("%s: CQE recovery done\n", mmc_hostname(host));
> +
> +	mmc_put_card(card);
> +}
> +
> +static void mmc_blk_cqe_req_done(struct mmc_request *mrq)
> +{
> +	struct mmc_queue_req *mqrq = container_of(mrq, struct mmc_queue_req,
> +						  brq.mrq);
> +	struct request *req = mmc_queue_req_to_req(mqrq);
> +	struct request_queue *q = req->q;
> +	struct mmc_queue *mq = q->queuedata;
> +
> +	/*
> +	 * Block layer timeouts race with completions which means the normal
> +	 * completion path cannot be used during recovery.
> +	 */
> +	if (mq->cqe_in_recovery)
> +		mmc_blk_cqe_complete_rq(req);
> +	else
> +		blk_complete_request(req);
> +}
> +
> +static int mmc_blk_cqe_start_req(struct mmc_host *host, struct mmc_request *mrq)
> +{
> +	mrq->done = mmc_blk_cqe_req_done;
> +	return mmc_cqe_start_req(host, mrq);
> +}
> +
> +static struct mmc_request *mmc_blk_cqe_prep_dcmd(struct mmc_queue_req *mqrq,
> +						 struct request *req)
> +{
> +	struct mmc_blk_request *brq = &mqrq->brq;
> +
> +	memset(brq, 0, sizeof(*brq));
> +
> +	brq->mrq.cmd = &brq->cmd;
> +	brq->mrq.tag = req->tag;
> +
> +	return &brq->mrq;
> +}
> +
> +static int mmc_blk_cqe_issue_flush(struct mmc_queue *mq, struct request *req)
> +{
> +	struct mmc_queue_req *mqrq = req_to_mmc_queue_req(req);
> +	struct mmc_request *mrq = mmc_blk_cqe_prep_dcmd(mqrq, req);
> +
> +	mrq->cmd->opcode = MMC_SWITCH;
> +	mrq->cmd->arg = (MMC_SWITCH_MODE_WRITE_BYTE << 24) |
> +			(EXT_CSD_FLUSH_CACHE << 16) |
> +			(1 << 8) |
> +			EXT_CSD_CMD_SET_NORMAL;
> +	mrq->cmd->flags = MMC_CMD_AC | MMC_RSP_R1B;
> +
> +	return mmc_blk_cqe_start_req(mq->card->host, mrq);
> +}
> +
> +static int mmc_blk_cqe_issue_rw_rq(struct mmc_queue *mq, struct request *req)
> +{
> +	struct mmc_queue_req *mqrq = req_to_mmc_queue_req(req);
> +
> +	mmc_blk_data_prep(mq, mqrq, 0, NULL, NULL);
> +
> +	return mmc_blk_cqe_start_req(mq->card->host, &mqrq->brq.mrq);
> +}
> +
> +enum mmc_issued mmc_blk_cqe_issue_rq(struct mmc_queue *mq, struct request *req)
> +{
> +	struct mmc_blk_data *md = mq->blkdata;
> +	struct mmc_card *card = md->queue.card;
> +	struct mmc_host *host = card->host;
> +	int ret;
> +
> +	ret = mmc_blk_part_switch(card, md);
> +	if (ret)
> +		return MMC_REQ_FAILED_TO_START;
> +
> +	switch (mmc_cqe_issue_type(host, req)) {
> +	case MMC_ISSUE_SYNC:
> +		ret = host->cqe_ops->cqe_wait_for_idle(host);
> +		if (ret)
> +			return MMC_REQ_BUSY;
> +		switch (req_op(req)) {
> +		case REQ_OP_DRV_IN:
> +		case REQ_OP_DRV_OUT:
> +			mmc_blk_issue_drv_op(mq, req);
> +			break;
> +		case REQ_OP_DISCARD:
> +			mmc_blk_issue_discard_rq(mq, req);
> +			break;
> +		case REQ_OP_SECURE_ERASE:
> +			mmc_blk_issue_secdiscard_rq(mq, req);
> +			break;
> +		case REQ_OP_FLUSH:
> +			mmc_blk_issue_flush(mq, req);
> +			break;
> +		default:
> +			WARN_ON_ONCE(1);
> +			return MMC_REQ_FAILED_TO_START;
> +		}
> +		return MMC_REQ_FINISHED;
> +	case MMC_ISSUE_DCMD:
> +	case MMC_ISSUE_ASYNC:
> +		switch (req_op(req)) {
> +		case REQ_OP_FLUSH:
> +			ret = mmc_blk_cqe_issue_flush(mq, req);
> +			break;
> +		case REQ_OP_READ:
> +		case REQ_OP_WRITE:
> +			ret = mmc_blk_cqe_issue_rw_rq(mq, req);
> +			break;
> +		default:
> +			WARN_ON_ONCE(1);
> +			ret = -EINVAL;
> +		}
> +		if (!ret)
> +			return MMC_REQ_STARTED;
> +		return ret == -EBUSY ? MMC_REQ_BUSY : MMC_REQ_FAILED_TO_START;
> +	default:
> +		WARN_ON_ONCE(1);
> +		return MMC_REQ_FAILED_TO_START;
> +	}
> +}
> +
>  static void mmc_blk_rw_rq_prep(struct mmc_queue_req *mqrq,
>  			       struct mmc_card *card,
>  			       int disable_multi,
> @@ -2035,7 +2228,7 @@ static struct mmc_blk_data *mmc_blk_alloc_req(struct mmc_card *card,
>  	INIT_LIST_HEAD(&md->part);
>  	md->usage = 1;
>
> -	ret = mmc_init_queue(&md->queue, card, &md->lock, subname);
> +	ret = mmc_init_queue(&md->queue, card, &md->lock, subname, area_type);
>  	if (ret)
>  		goto err_putdisk;
>
> diff --git a/drivers/mmc/core/block.h b/drivers/mmc/core/block.h
> index 860ca7c8df86..d7b3d7008b00 100644
> --- a/drivers/mmc/core/block.h
> +++ b/drivers/mmc/core/block.h
> @@ -6,4 +6,11 @@
>
>  void mmc_blk_issue_rq(struct mmc_queue *mq, struct request *req);
>
> +enum mmc_issued;
> +
> +enum mmc_issued mmc_blk_cqe_issue_rq(struct mmc_queue *mq,
> +				     struct request *req);
> +void mmc_blk_cqe_complete_rq(struct request *rq);
> +void mmc_blk_cqe_recovery(struct mmc_queue *mq);
> +
>  #endif
> diff --git a/drivers/mmc/core/queue.c b/drivers/mmc/core/queue.c
> index affa7370ba82..0cb7b0e8ee58 100644
> --- a/drivers/mmc/core/queue.c
> +++ b/drivers/mmc/core/queue.c
> @@ -36,10 +36,254 @@ static int mmc_prep_request(struct request_queue *q, struct request *req)
>  		return BLKPREP_KILL;
>
>  	req->rq_flags |= RQF_DONTPREP;
> +	req_to_mmc_queue_req(req)->retries = 0;
>
>  	return BLKPREP_OK;
>  }
>
> +static void mmc_cqe_request_fn(struct request_queue *q)
> +{
> +	struct mmc_queue *mq = q->queuedata;
> +	struct request *req;
> +
> +	if (!mq) {
> +		while ((req = blk_fetch_request(q)) != NULL) {
> +			req->rq_flags |= RQF_QUIET;
> +			__blk_end_request_all(req, BLK_STS_IOERR);
> +		}
> +		return;
> +	}
> +
> +	if (mq->asleep && !mq->cqe_busy)
> +		wake_up_process(mq->thread);
> +}
> +
> +static inline bool mmc_cqe_dcmd_busy(struct mmc_queue *mq)
> +{
> +	/* Allow only 1 DCMD at a time */
> +	return mq->cqe_in_flight[MMC_ISSUE_DCMD];
> +}
> +
> +void mmc_cqe_kick_queue(struct mmc_queue *mq)
> +{
> +	if ((mq->cqe_busy & MMC_CQE_DCMD_BUSY) && !mmc_cqe_dcmd_busy(mq))
> +		mq->cqe_busy &= ~MMC_CQE_DCMD_BUSY;
> +
> +	mq->cqe_busy &= ~MMC_CQE_QUEUE_FULL;
> +
> +	if (mq->asleep && !mq->cqe_busy)
> +		__blk_run_queue(mq->queue);
> +}
> +
> +static inline bool mmc_cqe_can_dcmd(struct mmc_host *host)
> +{
> +	return host->caps2 & MMC_CAP2_CQE_DCMD;
> +}
> +
> +enum mmc_issue_type mmc_cqe_issue_type(struct mmc_host *host,
> +				       struct request *req)
> +{
> +	switch (req_op(req)) {
> +	case REQ_OP_DRV_IN:
> +	case REQ_OP_DRV_OUT:
> +	case REQ_OP_DISCARD:
> +	case REQ_OP_SECURE_ERASE:
> +		return MMC_ISSUE_SYNC;
> +	case REQ_OP_FLUSH:
> +		return mmc_cqe_can_dcmd(host) ? MMC_ISSUE_DCMD : MMC_ISSUE_SYNC;
> +	default:
> +		return MMC_ISSUE_ASYNC;
> +	}
> +}
> +
> +static void __mmc_cqe_recovery_notifier(struct mmc_queue *mq)
> +{
> +	if (!mq->cqe_recovery_needed) {
> +		mq->cqe_recovery_needed = true;
> +		wake_up_process(mq->thread);
> +	}
> +}
> +
> +static void mmc_cqe_recovery_notifier(struct mmc_host *host,
> +				      struct mmc_request *mrq)
> +{
> +	struct mmc_queue_req *mqrq = container_of(mrq, struct mmc_queue_req,
> +						  brq.mrq);
> +	struct request *req = mmc_queue_req_to_req(mqrq);
> +	struct request_queue *q = req->q;
> +	struct mmc_queue *mq = q->queuedata;
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(q->queue_lock, flags);
> +	__mmc_cqe_recovery_notifier(mq);
> +	spin_unlock_irqrestore(q->queue_lock, flags);
> +}
> +
> +static int mmc_cqe_thread(void *d)
> +{
> +	struct mmc_queue *mq = d;
> +	struct request_queue *q = mq->queue;
> +	struct mmc_card *card = mq->card;
> +	struct mmc_host *host = card->host;
> +	unsigned long flags;
> +	int get_put = 0;
> +
> +	current->flags |= PF_MEMALLOC;
> +
> +	down(&mq->thread_sem);
> +	spin_lock_irqsave(q->queue_lock, flags);
> +	while (1) {
> +		struct request *req = NULL;
> +		enum mmc_issue_type issue_type;
> +		bool retune_ok = false;
> +
> +		if (mq->cqe_recovery_needed) {
> +			spin_unlock_irqrestore(q->queue_lock, flags);
> +			mmc_blk_cqe_recovery(mq);
> +			spin_lock_irqsave(q->queue_lock, flags);
> +			mq->cqe_recovery_needed = false;
> +		}
> +
> +		set_current_state(TASK_INTERRUPTIBLE);
> +
> +		if (!kthread_should_stop())
> +			req = blk_peek_request(q);
> +
> +		if (req) {
> +			issue_type = mmc_cqe_issue_type(host, req);
> +			switch (issue_type) {
> +			case MMC_ISSUE_DCMD:
> +				if (mmc_cqe_dcmd_busy(mq)) {
> +					mq->cqe_busy |= MMC_CQE_DCMD_BUSY;
> +					req = NULL;
> +					break;
> +				}
> +				/* Fall through */
> +			case MMC_ISSUE_ASYNC:
> +				if (blk_queue_start_tag(q, req)) {
> +					mq->cqe_busy |= MMC_CQE_QUEUE_FULL;
> +					req = NULL;
> +				}
> +				break;
> +			default:
> +				/*
> +				 * Timeouts are handled by mmc core, so set a
> +				 * large value to avoid races.
> +				 */
> +				req->timeout = 600 * HZ;
> +				blk_start_request(req);
> +				break;
> +			}
> +			if (req) {
> +				mq->cqe_in_flight[issue_type] += 1;
> +				if (mmc_cqe_tot_in_flight(mq) == 1)
> +					get_put += 1;
> +				if (mmc_cqe_qcnt(mq) == 1)
> +					retune_ok = true;
> +			}
> +		}
> +
> +		mq->asleep = !req;
> +
> +		spin_unlock_irqrestore(q->queue_lock, flags);
> +
> +		if (req) {
> +			enum mmc_issued issued;
> +
> +			set_current_state(TASK_RUNNING);
> +
> +			if (get_put) {
> +				get_put = 0;
> +				mmc_get_card(card);
> +			}
> +
> +			if (host->need_retune && retune_ok &&
> +			    !host->hold_retune)
> +				host->retune_now = true;
> +			else
> +				host->retune_now = false;
> +
> +			issued = mmc_blk_cqe_issue_rq(mq, req);
> +
> +			cond_resched();
> +
> +			spin_lock_irqsave(q->queue_lock, flags);
> +
> +			switch (issued) {
> +			case MMC_REQ_STARTED:
> +				break;
> +			case MMC_REQ_BUSY:
> +				blk_requeue_request(q, req);
> +				goto finished;
> +			case MMC_REQ_FAILED_TO_START:
> +				__blk_end_request_all(req, BLK_STS_IOERR);
> +				/* Fall through */
> +			case MMC_REQ_FINISHED:
> +finished:
> +				mq->cqe_in_flight[issue_type] -= 1;
> +				if (mmc_cqe_tot_in_flight(mq) == 0)
> +					get_put = -1;
> +			}
> +		} else {
> +			if (get_put < 0) {
> +				get_put = 0;
> +				mmc_put_card(card);
> +			}
> +			/*
> +			 * Do not stop with requests in flight in case recovery
> +			 * is needed.
> +			 */
> +			if (kthread_should_stop() &&
> +			    !mmc_cqe_tot_in_flight(mq)) {
> +				set_current_state(TASK_RUNNING);
> +				break;
> +			}
> +			up(&mq->thread_sem);
> +			schedule();
> +			down(&mq->thread_sem);
> +			spin_lock_irqsave(q->queue_lock, flags);
> +		}
> +	} /* loop */
> +	up(&mq->thread_sem);
> +
> +	return 0;
> +}
> +
> +static enum blk_eh_timer_return __mmc_cqe_timed_out(struct request *req)
> +{
> +	struct mmc_queue_req *mqrq = req_to_mmc_queue_req(req);
> +	struct mmc_request *mrq = &mqrq->brq.mrq;
> +	struct mmc_queue *mq = req->q->queuedata;
> +	struct mmc_host *host = mq->card->host;
> +	enum mmc_issue_type issue_type = mmc_cqe_issue_type(host, req);
> +	bool recovery_needed = false;
> +
> +	switch (issue_type) {
> +	case MMC_ISSUE_ASYNC:
> +	case MMC_ISSUE_DCMD:
> +		if (host->cqe_ops->cqe_timeout(host, mrq, &recovery_needed)) {
> +			if (recovery_needed)
> +				__mmc_cqe_recovery_notifier(mq);
> +			return BLK_EH_RESET_TIMER;
> +		}
> +		/* No timeout */
> +		return BLK_EH_HANDLED;
> +	default:
> +		/* Timeout is handled by mmc core */
> +		return BLK_EH_RESET_TIMER;
> +	}
> +}
> +
> +static enum blk_eh_timer_return mmc_cqe_timed_out(struct request *req)
> +{
> +	struct mmc_queue *mq = req->q->queuedata;
> +
> +	if (mq->cqe_recovery_needed)
> +		return BLK_EH_RESET_TIMER;
> +
> +	return __mmc_cqe_timed_out(req);
> +}
> +
>  static int mmc_queue_thread(void *d)
>  {
>  	struct mmc_queue *mq = d;
> @@ -233,11 +477,12 @@ static void mmc_exit_request(struct request_queue *q, struct request *req)
>   * Initialise a MMC card request queue.
>   */
>  int mmc_init_queue(struct mmc_queue *mq, struct mmc_card *card,
> -		   spinlock_t *lock, const char *subname)
> +		   spinlock_t *lock, const char *subname, int area_type)
>  {
>  	struct mmc_host *host = card->host;
>  	u64 limit = BLK_BOUNCE_HIGH;
>  	int ret = -ENOMEM;
> +	bool use_cqe = host->cqe_enabled && area_type != MMC_BLK_DATA_AREA_RPMB;
>
>  	if (mmc_dev(host)->dma_mask && *mmc_dev(host)->dma_mask)
>  		limit = (u64)dma_max_pfn(mmc_dev(host)) << PAGE_SHIFT;
> @@ -247,7 +492,7 @@ int mmc_init_queue(struct mmc_queue *mq, struct mmc_card *card,
>  	if (!mq->queue)
>  		return -ENOMEM;
>  	mq->queue->queue_lock = lock;
> -	mq->queue->request_fn = mmc_request_fn;
> +	mq->queue->request_fn = use_cqe ? mmc_cqe_request_fn : mmc_request_fn;
>  	mq->queue->init_rq_fn = mmc_init_request;
>  	mq->queue->exit_rq_fn = mmc_exit_request;
>  	mq->queue->cmd_size = sizeof(struct mmc_queue_req);
> @@ -259,6 +504,24 @@ int mmc_init_queue(struct mmc_queue *mq, struct mmc_card *card,
>  		return ret;
>  	}
>
> +	if (use_cqe) {
> +		int q_depth = card->ext_csd.cmdq_depth;
> +
> +		if (q_depth > host->cqe_qdepth)
> +			q_depth = host->cqe_qdepth;
> +
> +		ret = blk_queue_init_tags(mq->queue, q_depth, NULL,
> +					  BLK_TAG_ALLOC_FIFO);
> +		if (ret)
> +			goto cleanup_queue;
> +
> +		blk_queue_softirq_done(mq->queue, mmc_blk_cqe_complete_rq);
> +		blk_queue_rq_timed_out(mq->queue, mmc_cqe_timed_out);
> +		blk_queue_rq_timeout(mq->queue, 60 * HZ);
> +
> +		host->cqe_recovery_notifier = mmc_cqe_recovery_notifier;
> +	}
> +
>  	blk_queue_prep_rq(mq->queue, mmc_prep_request);
>  	queue_flag_set_unlocked(QUEUE_FLAG_NONROT, mq->queue);
>  	queue_flag_clear_unlocked(QUEUE_FLAG_ADD_RANDOM, mq->queue);
> @@ -280,9 +543,9 @@ int mmc_init_queue(struct mmc_queue *mq, struct mmc_card *card,
>
>  	sema_init(&mq->thread_sem, 1);
>
> -	mq->thread = kthread_run(mmc_queue_thread, mq, "mmcqd/%d%s",
> -		host->index, subname ? subname : "");
> -
> +	mq->thread = kthread_run(use_cqe ? mmc_cqe_thread : mmc_queue_thread,
> +				 mq, "mmcqd/%d%s", host->index,
> +				 subname ? subname : "");
>  	if (IS_ERR(mq->thread)) {
>  		ret = PTR_ERR(mq->thread);
>  		goto cleanup_queue;
> diff --git a/drivers/mmc/core/queue.h b/drivers/mmc/core/queue.h
> index 361b46408e0f..8e9273d977c0 100644
> --- a/drivers/mmc/core/queue.h
> +++ b/drivers/mmc/core/queue.h
> @@ -7,6 +7,20 @@
>  #include <linux/mmc/core.h>
>  #include <linux/mmc/host.h>
>
> +enum mmc_issued {
> +	MMC_REQ_STARTED,
> +	MMC_REQ_BUSY,
> +	MMC_REQ_FAILED_TO_START,
> +	MMC_REQ_FINISHED,
> +};
> +
> +enum mmc_issue_type {
> +	MMC_ISSUE_SYNC,
> +	MMC_ISSUE_DCMD,
> +	MMC_ISSUE_ASYNC,
> +	MMC_ISSUE_MAX,
> +};
> +
>  static inline struct mmc_queue_req *req_to_mmc_queue_req(struct request *rq)
>  {
>  	return blk_mq_rq_to_pdu(rq);
> @@ -53,6 +67,7 @@ struct mmc_queue_req {
>  	int			drv_op_result;
>  	struct mmc_blk_ioc_data	**idata;
>  	unsigned int		ioc_count;
> +	int			retries;
>  };
>
>  struct mmc_queue {
> @@ -70,10 +85,17 @@ struct mmc_queue {
>  	 * associated mmc_queue_req data.
>  	 */
>  	int			qcnt;
> +	/* Following are defined for a Command Queue Engine */
> +	int			cqe_in_flight[MMC_ISSUE_MAX];
> +	unsigned int		cqe_busy;
> +	bool			cqe_recovery_needed;
> +	bool			cqe_in_recovery;
> +#define MMC_CQE_DCMD_BUSY	BIT(0)
> +#define MMC_CQE_QUEUE_FULL	BIT(1)
>  };
>
>  extern int mmc_init_queue(struct mmc_queue *, struct mmc_card *, spinlock_t *,
> -			  const char *);
> +			  const char *, int);
>  extern void mmc_cleanup_queue(struct mmc_queue *);
>  extern void mmc_queue_suspend(struct mmc_queue *);
>  extern void mmc_queue_resume(struct mmc_queue *);
> @@ -85,4 +107,22 @@ extern unsigned int mmc_queue_map_sg(struct mmc_queue *,
>
>  extern int mmc_access_rpmb(struct mmc_queue *);
>
> +void mmc_cqe_kick_queue(struct mmc_queue *mq);
> +
> +enum mmc_issue_type mmc_cqe_issue_type(struct mmc_host *host,
> +				       struct request *req);
> +
> +static inline int mmc_cqe_tot_in_flight(struct mmc_queue *mq)
> +{
> +	return mq->cqe_in_flight[MMC_ISSUE_SYNC] +
> +	       mq->cqe_in_flight[MMC_ISSUE_DCMD] +
> +	       mq->cqe_in_flight[MMC_ISSUE_ASYNC];
> +}
> +
> +static inline int mmc_cqe_qcnt(struct mmc_queue *mq)
> +{
> +	return mq->cqe_in_flight[MMC_ISSUE_DCMD] +
> +	       mq->cqe_in_flight[MMC_ISSUE_ASYNC];
> +}
> +
>  #endif
>


-- 
Best Regards
Shawn Lin

Re: [PATCH V4 09/11] mmc: block: Add CQE support

[Shawn Lin]

Hi Adrian,

On 2017/7/21 17:49, Adrian Hunter wrote:
> Add CQE support to the block driver, including:
> 	- optionally using DCMD for flush requests

That doesn't seem to be fact when I was looking into
patch 10 as it always fire DCMD with QBR whenever seeing
!mrq->data. Do I miss something?

> 	- manually issuing discard requests
> 	- issuing read / write requests to the CQE
> 	- supporting block-layer timeouts
> 	- handling recovery
> 	- supporting re-tuning
>
> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
> ---
>  drivers/mmc/core/block.c | 195 ++++++++++++++++++++++++++++++++-
>  drivers/mmc/core/block.h |   7 ++
>  drivers/mmc/core/queue.c | 273 ++++++++++++++++++++++++++++++++++++++++++++++-
>  drivers/mmc/core/queue.h |  42 +++++++-
>  4 files changed, 510 insertions(+), 7 deletions(-)
>
> diff --git a/drivers/mmc/core/block.c b/drivers/mmc/core/block.c
> index 915290c74363..2d25115637b7 100644
> --- a/drivers/mmc/core/block.c
> +++ b/drivers/mmc/core/block.c
> @@ -109,6 +109,7 @@ struct mmc_blk_data {
>  #define MMC_BLK_WRITE		BIT(1)
>  #define MMC_BLK_DISCARD		BIT(2)
>  #define MMC_BLK_SECDISCARD	BIT(3)
> +#define MMC_BLK_CQE_RECOVERY	BIT(4)
>
>  	/*
>  	 * Only set in main mmc_blk_data associated
> @@ -1612,6 +1613,198 @@ static void mmc_blk_data_prep(struct mmc_queue *mq, struct mmc_queue_req *mqrq,
>  		*do_data_tag_p = do_data_tag;
>  }
>
> +#define MMC_CQE_RETRIES 2
> +
> +void mmc_blk_cqe_complete_rq(struct request *req)
> +{
> +	struct mmc_queue_req *mqrq = req_to_mmc_queue_req(req);
> +	struct mmc_request *mrq = &mqrq->brq.mrq;
> +	struct request_queue *q = req->q;
> +	struct mmc_queue *mq = q->queuedata;
> +	struct mmc_host *host = mq->card->host;
> +	unsigned long flags;
> +	bool put_card;
> +	int err;
> +
> +	mmc_cqe_post_req(host, mrq);
> +
> +	spin_lock_irqsave(q->queue_lock, flags);
> +
> +	mq->cqe_in_flight[mmc_cqe_issue_type(host, req)] -= 1;
> +
> +	put_card = mmc_cqe_tot_in_flight(mq) == 0;
> +
> +	if (mrq->cmd && mrq->cmd->error)
> +		err = mrq->cmd->error;
> +	else if (mrq->data && mrq->data->error)
> +		err = mrq->data->error;
> +	else
> +		err = 0;
> +
> +	if (err) {
> +		if (mqrq->retries++ < MMC_CQE_RETRIES)
> +			blk_requeue_request(q, req);
> +		else
> +			__blk_end_request_all(req, BLK_STS_IOERR);
> +	} else if (mrq->data) {
> +		if (__blk_end_request(req, BLK_STS_OK, mrq->data->bytes_xfered))
> +			blk_requeue_request(q, req);
> +	} else {
> +		__blk_end_request_all(req, BLK_STS_OK);
> +	}
> +
> +	mmc_cqe_kick_queue(mq);
> +
> +	spin_unlock_irqrestore(q->queue_lock, flags);
> +
> +	if (put_card)
> +		mmc_put_card(mq->card);
> +}
> +
> +void mmc_blk_cqe_recovery(struct mmc_queue *mq)
> +{
> +	struct mmc_card *card = mq->card;
> +	struct mmc_host *host = card->host;
> +	int err;
> +
> +	mmc_get_card(card);
> +
> +	pr_debug("%s: CQE recovery start\n", mmc_hostname(host));
> +
> +	mq->cqe_in_recovery = true;
> +
> +	err = mmc_cqe_recovery(host);
> +	if (err)
> +		mmc_blk_reset(mq->blkdata, host, MMC_BLK_CQE_RECOVERY);
> +	else
> +		mmc_blk_reset_success(mq->blkdata, MMC_BLK_CQE_RECOVERY);
> +
> +	mq->cqe_in_recovery = false;
> +
> +	pr_debug("%s: CQE recovery done\n", mmc_hostname(host));
> +
> +	mmc_put_card(card);
> +}
> +
> +static void mmc_blk_cqe_req_done(struct mmc_request *mrq)
> +{
> +	struct mmc_queue_req *mqrq = container_of(mrq, struct mmc_queue_req,
> +						  brq.mrq);
> +	struct request *req = mmc_queue_req_to_req(mqrq);
> +	struct request_queue *q = req->q;
> +	struct mmc_queue *mq = q->queuedata;
> +
> +	/*
> +	 * Block layer timeouts race with completions which means the normal
> +	 * completion path cannot be used during recovery.
> +	 */
> +	if (mq->cqe_in_recovery)
> +		mmc_blk_cqe_complete_rq(req);
> +	else
> +		blk_complete_request(req);
> +}
> +
> +static int mmc_blk_cqe_start_req(struct mmc_host *host, struct mmc_request *mrq)
> +{
> +	mrq->done = mmc_blk_cqe_req_done;
> +	return mmc_cqe_start_req(host, mrq);
> +}
> +
> +static struct mmc_request *mmc_blk_cqe_prep_dcmd(struct mmc_queue_req *mqrq,
> +						 struct request *req)
> +{
> +	struct mmc_blk_request *brq = &mqrq->brq;
> +
> +	memset(brq, 0, sizeof(*brq));
> +
> +	brq->mrq.cmd = &brq->cmd;
> +	brq->mrq.tag = req->tag;
> +
> +	return &brq->mrq;
> +}
> +
> +static int mmc_blk_cqe_issue_flush(struct mmc_queue *mq, struct request *req)
> +{
> +	struct mmc_queue_req *mqrq = req_to_mmc_queue_req(req);
> +	struct mmc_request *mrq = mmc_blk_cqe_prep_dcmd(mqrq, req);
> +
> +	mrq->cmd->opcode = MMC_SWITCH;
> +	mrq->cmd->arg = (MMC_SWITCH_MODE_WRITE_BYTE << 24) |
> +			(EXT_CSD_FLUSH_CACHE << 16) |
> +			(1 << 8) |
> +			EXT_CSD_CMD_SET_NORMAL;
> +	mrq->cmd->flags = MMC_CMD_AC | MMC_RSP_R1B;
> +
> +	return mmc_blk_cqe_start_req(mq->card->host, mrq);
> +}
> +
> +static int mmc_blk_cqe_issue_rw_rq(struct mmc_queue *mq, struct request *req)
> +{
> +	struct mmc_queue_req *mqrq = req_to_mmc_queue_req(req);
> +
> +	mmc_blk_data_prep(mq, mqrq, 0, NULL, NULL);
> +
> +	return mmc_blk_cqe_start_req(mq->card->host, &mqrq->brq.mrq);
> +}
> +
> +enum mmc_issued mmc_blk_cqe_issue_rq(struct mmc_queue *mq, struct request *req)
> +{
> +	struct mmc_blk_data *md = mq->blkdata;
> +	struct mmc_card *card = md->queue.card;
> +	struct mmc_host *host = card->host;
> +	int ret;
> +
> +	ret = mmc_blk_part_switch(card, md);
> +	if (ret)
> +		return MMC_REQ_FAILED_TO_START;
> +
> +	switch (mmc_cqe_issue_type(host, req)) {
> +	case MMC_ISSUE_SYNC:
> +		ret = host->cqe_ops->cqe_wait_for_idle(host);
> +		if (ret)
> +			return MMC_REQ_BUSY;
> +		switch (req_op(req)) {
> +		case REQ_OP_DRV_IN:
> +		case REQ_OP_DRV_OUT:
> +			mmc_blk_issue_drv_op(mq, req);
> +			break;
> +		case REQ_OP_DISCARD:
> +			mmc_blk_issue_discard_rq(mq, req);
> +			break;
> +		case REQ_OP_SECURE_ERASE:
> +			mmc_blk_issue_secdiscard_rq(mq, req);
> +			break;
> +		case REQ_OP_FLUSH:
> +			mmc_blk_issue_flush(mq, req);
> +			break;
> +		default:
> +			WARN_ON_ONCE(1);
> +			return MMC_REQ_FAILED_TO_START;
> +		}
> +		return MMC_REQ_FINISHED;
> +	case MMC_ISSUE_DCMD:
> +	case MMC_ISSUE_ASYNC:
> +		switch (req_op(req)) {
> +		case REQ_OP_FLUSH:
> +			ret = mmc_blk_cqe_issue_flush(mq, req);
> +			break;
> +		case REQ_OP_READ:
> +		case REQ_OP_WRITE:
> +			ret = mmc_blk_cqe_issue_rw_rq(mq, req);
> +			break;
> +		default:
> +			WARN_ON_ONCE(1);
> +			ret = -EINVAL;
> +		}
> +		if (!ret)
> +			return MMC_REQ_STARTED;
> +		return ret == -EBUSY ? MMC_REQ_BUSY : MMC_REQ_FAILED_TO_START;
> +	default:
> +		WARN_ON_ONCE(1);
> +		return MMC_REQ_FAILED_TO_START;
> +	}
> +}
> +
>  static void mmc_blk_rw_rq_prep(struct mmc_queue_req *mqrq,
>  			       struct mmc_card *card,
>  			       int disable_multi,
> @@ -2035,7 +2228,7 @@ static struct mmc_blk_data *mmc_blk_alloc_req(struct mmc_card *card,
>  	INIT_LIST_HEAD(&md->part);
>  	md->usage = 1;
>
> -	ret = mmc_init_queue(&md->queue, card, &md->lock, subname);
> +	ret = mmc_init_queue(&md->queue, card, &md->lock, subname, area_type);
>  	if (ret)
>  		goto err_putdisk;
>
> diff --git a/drivers/mmc/core/block.h b/drivers/mmc/core/block.h
> index 860ca7c8df86..d7b3d7008b00 100644
> --- a/drivers/mmc/core/block.h
> +++ b/drivers/mmc/core/block.h
> @@ -6,4 +6,11 @@
>
>  void mmc_blk_issue_rq(struct mmc_queue *mq, struct request *req);
>
> +enum mmc_issued;
> +
> +enum mmc_issued mmc_blk_cqe_issue_rq(struct mmc_queue *mq,
> +				     struct request *req);
> +void mmc_blk_cqe_complete_rq(struct request *rq);
> +void mmc_blk_cqe_recovery(struct mmc_queue *mq);
> +
>  #endif
> diff --git a/drivers/mmc/core/queue.c b/drivers/mmc/core/queue.c
> index affa7370ba82..0cb7b0e8ee58 100644
> --- a/drivers/mmc/core/queue.c
> +++ b/drivers/mmc/core/queue.c
> @@ -36,10 +36,254 @@ static int mmc_prep_request(struct request_queue *q, struct request *req)
>  		return BLKPREP_KILL;
>
>  	req->rq_flags |= RQF_DONTPREP;
> +	req_to_mmc_queue_req(req)->retries = 0;
>
>  	return BLKPREP_OK;
>  }
>
> +static void mmc_cqe_request_fn(struct request_queue *q)
> +{
> +	struct mmc_queue *mq = q->queuedata;
> +	struct request *req;
> +
> +	if (!mq) {
> +		while ((req = blk_fetch_request(q)) != NULL) {
> +			req->rq_flags |= RQF_QUIET;
> +			__blk_end_request_all(req, BLK_STS_IOERR);
> +		}
> +		return;
> +	}
> +
> +	if (mq->asleep && !mq->cqe_busy)
> +		wake_up_process(mq->thread);
> +}
> +
> +static inline bool mmc_cqe_dcmd_busy(struct mmc_queue *mq)
> +{
> +	/* Allow only 1 DCMD at a time */
> +	return mq->cqe_in_flight[MMC_ISSUE_DCMD];
> +}
> +
> +void mmc_cqe_kick_queue(struct mmc_queue *mq)
> +{
> +	if ((mq->cqe_busy & MMC_CQE_DCMD_BUSY) && !mmc_cqe_dcmd_busy(mq))
> +		mq->cqe_busy &= ~MMC_CQE_DCMD_BUSY;
> +
> +	mq->cqe_busy &= ~MMC_CQE_QUEUE_FULL;
> +
> +	if (mq->asleep && !mq->cqe_busy)
> +		__blk_run_queue(mq->queue);
> +}
> +
> +static inline bool mmc_cqe_can_dcmd(struct mmc_host *host)
> +{
> +	return host->caps2 & MMC_CAP2_CQE_DCMD;
> +}
> +
> +enum mmc_issue_type mmc_cqe_issue_type(struct mmc_host *host,
> +				       struct request *req)
> +{
> +	switch (req_op(req)) {
> +	case REQ_OP_DRV_IN:
> +	case REQ_OP_DRV_OUT:
> +	case REQ_OP_DISCARD:
> +	case REQ_OP_SECURE_ERASE:
> +		return MMC_ISSUE_SYNC;
> +	case REQ_OP_FLUSH:
> +		return mmc_cqe_can_dcmd(host) ? MMC_ISSUE_DCMD : MMC_ISSUE_SYNC;
> +	default:
> +		return MMC_ISSUE_ASYNC;
> +	}
> +}
> +
> +static void __mmc_cqe_recovery_notifier(struct mmc_queue *mq)
> +{
> +	if (!mq->cqe_recovery_needed) {
> +		mq->cqe_recovery_needed = true;
> +		wake_up_process(mq->thread);
> +	}
> +}
> +
> +static void mmc_cqe_recovery_notifier(struct mmc_host *host,
> +				      struct mmc_request *mrq)
> +{
> +	struct mmc_queue_req *mqrq = container_of(mrq, struct mmc_queue_req,
> +						  brq.mrq);
> +	struct request *req = mmc_queue_req_to_req(mqrq);
> +	struct request_queue *q = req->q;
> +	struct mmc_queue *mq = q->queuedata;
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(q->queue_lock, flags);
> +	__mmc_cqe_recovery_notifier(mq);
> +	spin_unlock_irqrestore(q->queue_lock, flags);
> +}
> +
> +static int mmc_cqe_thread(void *d)
> +{
> +	struct mmc_queue *mq = d;
> +	struct request_queue *q = mq->queue;
> +	struct mmc_card *card = mq->card;
> +	struct mmc_host *host = card->host;
> +	unsigned long flags;
> +	int get_put = 0;
> +
> +	current->flags |= PF_MEMALLOC;
> +
> +	down(&mq->thread_sem);
> +	spin_lock_irqsave(q->queue_lock, flags);
> +	while (1) {
> +		struct request *req = NULL;
> +		enum mmc_issue_type issue_type;
> +		bool retune_ok = false;
> +
> +		if (mq->cqe_recovery_needed) {
> +			spin_unlock_irqrestore(q->queue_lock, flags);
> +			mmc_blk_cqe_recovery(mq);
> +			spin_lock_irqsave(q->queue_lock, flags);
> +			mq->cqe_recovery_needed = false;
> +		}
> +
> +		set_current_state(TASK_INTERRUPTIBLE);
> +
> +		if (!kthread_should_stop())
> +			req = blk_peek_request(q);
> +
> +		if (req) {
> +			issue_type = mmc_cqe_issue_type(host, req);
> +			switch (issue_type) {
> +			case MMC_ISSUE_DCMD:
> +				if (mmc_cqe_dcmd_busy(mq)) {
> +					mq->cqe_busy |= MMC_CQE_DCMD_BUSY;
> +					req = NULL;
> +					break;
> +				}
> +				/* Fall through */
> +			case MMC_ISSUE_ASYNC:
> +				if (blk_queue_start_tag(q, req)) {
> +					mq->cqe_busy |= MMC_CQE_QUEUE_FULL;
> +					req = NULL;
> +				}
> +				break;
> +			default:
> +				/*
> +				 * Timeouts are handled by mmc core, so set a
> +				 * large value to avoid races.
> +				 */
> +				req->timeout = 600 * HZ;
> +				blk_start_request(req);
> +				break;
> +			}
> +			if (req) {
> +				mq->cqe_in_flight[issue_type] += 1;
> +				if (mmc_cqe_tot_in_flight(mq) == 1)
> +					get_put += 1;
> +				if (mmc_cqe_qcnt(mq) == 1)
> +					retune_ok = true;
> +			}
> +		}
> +
> +		mq->asleep = !req;
> +
> +		spin_unlock_irqrestore(q->queue_lock, flags);
> +
> +		if (req) {
> +			enum mmc_issued issued;
> +
> +			set_current_state(TASK_RUNNING);
> +
> +			if (get_put) {
> +				get_put = 0;
> +				mmc_get_card(card);
> +			}
> +
> +			if (host->need_retune && retune_ok &&
> +			    !host->hold_retune)
> +				host->retune_now = true;
> +			else
> +				host->retune_now = false;
> +
> +			issued = mmc_blk_cqe_issue_rq(mq, req);
> +
> +			cond_resched();
> +
> +			spin_lock_irqsave(q->queue_lock, flags);
> +
> +			switch (issued) {
> +			case MMC_REQ_STARTED:
> +				break;
> +			case MMC_REQ_BUSY:
> +				blk_requeue_request(q, req);
> +				goto finished;
> +			case MMC_REQ_FAILED_TO_START:
> +				__blk_end_request_all(req, BLK_STS_IOERR);
> +				/* Fall through */
> +			case MMC_REQ_FINISHED:
> +finished:
> +				mq->cqe_in_flight[issue_type] -= 1;
> +				if (mmc_cqe_tot_in_flight(mq) == 0)
> +					get_put = -1;
> +			}
> +		} else {
> +			if (get_put < 0) {
> +				get_put = 0;
> +				mmc_put_card(card);
> +			}
> +			/*
> +			 * Do not stop with requests in flight in case recovery
> +			 * is needed.
> +			 */
> +			if (kthread_should_stop() &&
> +			    !mmc_cqe_tot_in_flight(mq)) {
> +				set_current_state(TASK_RUNNING);
> +				break;
> +			}
> +			up(&mq->thread_sem);
> +			schedule();
> +			down(&mq->thread_sem);
> +			spin_lock_irqsave(q->queue_lock, flags);
> +		}
> +	} /* loop */
> +	up(&mq->thread_sem);
> +
> +	return 0;
> +}
> +
> +static enum blk_eh_timer_return __mmc_cqe_timed_out(struct request *req)
> +{
> +	struct mmc_queue_req *mqrq = req_to_mmc_queue_req(req);
> +	struct mmc_request *mrq = &mqrq->brq.mrq;
> +	struct mmc_queue *mq = req->q->queuedata;
> +	struct mmc_host *host = mq->card->host;
> +	enum mmc_issue_type issue_type = mmc_cqe_issue_type(host, req);
> +	bool recovery_needed = false;
> +
> +	switch (issue_type) {
> +	case MMC_ISSUE_ASYNC:
> +	case MMC_ISSUE_DCMD:
> +		if (host->cqe_ops->cqe_timeout(host, mrq, &recovery_needed)) {
> +			if (recovery_needed)
> +				__mmc_cqe_recovery_notifier(mq);
> +			return BLK_EH_RESET_TIMER;
> +		}
> +		/* No timeout */
> +		return BLK_EH_HANDLED;
> +	default:
> +		/* Timeout is handled by mmc core */
> +		return BLK_EH_RESET_TIMER;
> +	}
> +}
> +
> +static enum blk_eh_timer_return mmc_cqe_timed_out(struct request *req)
> +{
> +	struct mmc_queue *mq = req->q->queuedata;
> +
> +	if (mq->cqe_recovery_needed)
> +		return BLK_EH_RESET_TIMER;
> +
> +	return __mmc_cqe_timed_out(req);
> +}
> +
>  static int mmc_queue_thread(void *d)
>  {
>  	struct mmc_queue *mq = d;
> @@ -233,11 +477,12 @@ static void mmc_exit_request(struct request_queue *q, struct request *req)
>   * Initialise a MMC card request queue.
>   */
>  int mmc_init_queue(struct mmc_queue *mq, struct mmc_card *card,
> -		   spinlock_t *lock, const char *subname)
> +		   spinlock_t *lock, const char *subname, int area_type)
>  {
>  	struct mmc_host *host = card->host;
>  	u64 limit = BLK_BOUNCE_HIGH;
>  	int ret = -ENOMEM;
> +	bool use_cqe = host->cqe_enabled && area_type != MMC_BLK_DATA_AREA_RPMB;
>
>  	if (mmc_dev(host)->dma_mask && *mmc_dev(host)->dma_mask)
>  		limit = (u64)dma_max_pfn(mmc_dev(host)) << PAGE_SHIFT;
> @@ -247,7 +492,7 @@ int mmc_init_queue(struct mmc_queue *mq, struct mmc_card *card,
>  	if (!mq->queue)
>  		return -ENOMEM;
>  	mq->queue->queue_lock = lock;
> -	mq->queue->request_fn = mmc_request_fn;
> +	mq->queue->request_fn = use_cqe ? mmc_cqe_request_fn : mmc_request_fn;
>  	mq->queue->init_rq_fn = mmc_init_request;
>  	mq->queue->exit_rq_fn = mmc_exit_request;
>  	mq->queue->cmd_size = sizeof(struct mmc_queue_req);
> @@ -259,6 +504,24 @@ int mmc_init_queue(struct mmc_queue *mq, struct mmc_card *card,
>  		return ret;
>  	}
>
> +	if (use_cqe) {
> +		int q_depth = card->ext_csd.cmdq_depth;
> +
> +		if (q_depth > host->cqe_qdepth)
> +			q_depth = host->cqe_qdepth;
> +
> +		ret = blk_queue_init_tags(mq->queue, q_depth, NULL,
> +					  BLK_TAG_ALLOC_FIFO);
> +		if (ret)
> +			goto cleanup_queue;
> +
> +		blk_queue_softirq_done(mq->queue, mmc_blk_cqe_complete_rq);
> +		blk_queue_rq_timed_out(mq->queue, mmc_cqe_timed_out);
> +		blk_queue_rq_timeout(mq->queue, 60 * HZ);
> +
> +		host->cqe_recovery_notifier = mmc_cqe_recovery_notifier;
> +	}
> +
>  	blk_queue_prep_rq(mq->queue, mmc_prep_request);
>  	queue_flag_set_unlocked(QUEUE_FLAG_NONROT, mq->queue);
>  	queue_flag_clear_unlocked(QUEUE_FLAG_ADD_RANDOM, mq->queue);
> @@ -280,9 +543,9 @@ int mmc_init_queue(struct mmc_queue *mq, struct mmc_card *card,
>
>  	sema_init(&mq->thread_sem, 1);
>
> -	mq->thread = kthread_run(mmc_queue_thread, mq, "mmcqd/%d%s",
> -		host->index, subname ? subname : "");
> -
> +	mq->thread = kthread_run(use_cqe ? mmc_cqe_thread : mmc_queue_thread,
> +				 mq, "mmcqd/%d%s", host->index,
> +				 subname ? subname : "");
>  	if (IS_ERR(mq->thread)) {
>  		ret = PTR_ERR(mq->thread);
>  		goto cleanup_queue;
> diff --git a/drivers/mmc/core/queue.h b/drivers/mmc/core/queue.h
> index 361b46408e0f..8e9273d977c0 100644
> --- a/drivers/mmc/core/queue.h
> +++ b/drivers/mmc/core/queue.h
> @@ -7,6 +7,20 @@
>  #include <linux/mmc/core.h>
>  #include <linux/mmc/host.h>
>
> +enum mmc_issued {
> +	MMC_REQ_STARTED,
> +	MMC_REQ_BUSY,
> +	MMC_REQ_FAILED_TO_START,
> +	MMC_REQ_FINISHED,
> +};
> +
> +enum mmc_issue_type {
> +	MMC_ISSUE_SYNC,
> +	MMC_ISSUE_DCMD,
> +	MMC_ISSUE_ASYNC,
> +	MMC_ISSUE_MAX,
> +};
> +
>  static inline struct mmc_queue_req *req_to_mmc_queue_req(struct request *rq)
>  {
>  	return blk_mq_rq_to_pdu(rq);
> @@ -53,6 +67,7 @@ struct mmc_queue_req {
>  	int			drv_op_result;
>  	struct mmc_blk_ioc_data	**idata;
>  	unsigned int		ioc_count;
> +	int			retries;
>  };
>
>  struct mmc_queue {
> @@ -70,10 +85,17 @@ struct mmc_queue {
>  	 * associated mmc_queue_req data.
>  	 */
>  	int			qcnt;
> +	/* Following are defined for a Command Queue Engine */
> +	int			cqe_in_flight[MMC_ISSUE_MAX];
> +	unsigned int		cqe_busy;
> +	bool			cqe_recovery_needed;
> +	bool			cqe_in_recovery;
> +#define MMC_CQE_DCMD_BUSY	BIT(0)
> +#define MMC_CQE_QUEUE_FULL	BIT(1)
>  };
>
>  extern int mmc_init_queue(struct mmc_queue *, struct mmc_card *, spinlock_t *,
> -			  const char *);
> +			  const char *, int);
>  extern void mmc_cleanup_queue(struct mmc_queue *);
>  extern void mmc_queue_suspend(struct mmc_queue *);
>  extern void mmc_queue_resume(struct mmc_queue *);
> @@ -85,4 +107,22 @@ extern unsigned int mmc_queue_map_sg(struct mmc_queue *,
>
>  extern int mmc_access_rpmb(struct mmc_queue *);
>
> +void mmc_cqe_kick_queue(struct mmc_queue *mq);
> +
> +enum mmc_issue_type mmc_cqe_issue_type(struct mmc_host *host,
> +				       struct request *req);
> +
> +static inline int mmc_cqe_tot_in_flight(struct mmc_queue *mq)
> +{
> +	return mq->cqe_in_flight[MMC_ISSUE_SYNC] +
> +	       mq->cqe_in_flight[MMC_ISSUE_DCMD] +
> +	       mq->cqe_in_flight[MMC_ISSUE_ASYNC];
> +}
> +
> +static inline int mmc_cqe_qcnt(struct mmc_queue *mq)
> +{
> +	return mq->cqe_in_flight[MMC_ISSUE_DCMD] +
> +	       mq->cqe_in_flight[MMC_ISSUE_ASYNC];
> +}
> +
>  #endif
>


-- 
Best Regards
Shawn Lin

Re: [PATCH V4 10/11] mmc: cqhci: support for command queue enabled host

[Shawn Lin]

Hi Adrian,

On 2017/7/21 17:49, Adrian Hunter wrote:
> From: Venkat Gopalakrishnan <venkatg@codeaurora.org>
>
> This patch adds CMDQ support for command-queue compatible
> hosts.
>
> Command queue is added in eMMC-5.1 specification. This
> enables the controller to process upto 32 requests at
> a time.
>
> Adrian Hunter contributed renaming to cqhci, recovery, suspend
> and resume, cqhci_off, cqhci_wait_for_idle, and external timeout
> handling.
>
> Signed-off-by: Asutosh Das <asutoshd@codeaurora.org>
> Signed-off-by: Sujit Reddy Thumma <sthumma@codeaurora.org>
> Signed-off-by: Konstantin Dorfman <kdorfman@codeaurora.org>
> Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
> Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
> Signed-off-by: Ritesh Harjani <riteshh@codeaurora.org>
> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
> ---
>  drivers/mmc/host/Kconfig  |   13 +
>  drivers/mmc/host/Makefile |    1 +
>  drivers/mmc/host/cqhci.c  | 1146 +++++++++++++++++++++++++++++++++++++++++++++
>  drivers/mmc/host/cqhci.h  |  240 ++++++++++
>  4 files changed, 1400 insertions(+)
>  create mode 100644 drivers/mmc/host/cqhci.c
>  create mode 100644 drivers/mmc/host/cqhci.h
>

[...]

> +
> +static void cqhci_prep_task_desc(struct mmc_request *mrq,
> +					u64 *data, bool intr)
> +{
> +	u32 req_flags = mrq->data->flags;
> +
> +	*data = CQHCI_VALID(1) |
> +		CQHCI_END(1) |
> +		CQHCI_INT(intr) |
> +		CQHCI_ACT(0x5) |
> +		CQHCI_FORCED_PROG(!!(req_flags & MMC_DATA_FORCED_PRG)) |
> +		CQHCI_DATA_TAG(!!(req_flags & MMC_DATA_DAT_TAG)) |
> +		CQHCI_DATA_DIR(!!(req_flags & MMC_DATA_READ)) |
> +		CQHCI_PRIORITY(!!(req_flags & MMC_DATA_PRIO)) |
> +		CQHCI_QBAR(!!(req_flags & MMC_DATA_QBR)) |
> +		CQHCI_REL_WRITE(!!(req_flags & MMC_DATA_REL_WR)) |
> +		CQHCI_BLK_COUNT(mrq->data->blocks) |
> +		CQHCI_BLK_ADDR((u64)mrq->data->blk_addr);
> +

I don't find where MMC_DATA_FORCED_PRG is set and MMC_DATA_QBR is set
only for DCMD(erase, flush etc). So I have some silly questions :)

(1) Should MMC_DATA_FORCED_PRG be used for secure erase/trim, etc?
(2) How could the overall code guarantee the read-after-write issue? for
instance, is it possible that two TDLs contain one read for LBA x and a
write for same LBA x, so how to make sure the devices' execute order?


> +int cqhci_resume(struct mmc_host *mmc);
> +
> +#endif
>


-- 
Best Regards
Shawn Lin

Re: [PATCH V4 09/11] mmc: block: Add CQE support

[Adrian Hunter]

On 22/07/17 12:26, Shawn Lin wrote:
> Hi Adrian,
> 
> On 2017/7/21 17:49, Adrian Hunter wrote:
>> Add CQE support to the block driver, including:
>>     - optionally using DCMD for flush requests
> 
> That doesn't seem to be fact when I was looking into
> patch 10 as it always fire DCMD with QBR whenever seeing
> !mrq->data. Do I miss something?

The function that decides how to issue a request is mmc_cqe_issue_type()
which checks mmc_cqe_can_dcmd().

<SNIP>

>> +static inline bool mmc_cqe_can_dcmd(struct mmc_host *host)
>> +{
>> +    return host->caps2 & MMC_CAP2_CQE_DCMD;
>> +}
>> +
>> +enum mmc_issue_type mmc_cqe_issue_type(struct mmc_host *host,
>> +                       struct request *req)
>> +{
>> +    switch (req_op(req)) {
>> +    case REQ_OP_DRV_IN:
>> +    case REQ_OP_DRV_OUT:
>> +    case REQ_OP_DISCARD:
>> +    case REQ_OP_SECURE_ERASE:
>> +        return MMC_ISSUE_SYNC;
>> +    case REQ_OP_FLUSH:
>> +        return mmc_cqe_can_dcmd(host) ? MMC_ISSUE_DCMD : MMC_ISSUE_SYNC;
>> +    default:
>> +        return MMC_ISSUE_ASYNC;
>> +    }
>> +}

Re: [PATCH V4 10/11] mmc: cqhci: support for command queue enabled host

[Adrian Hunter]

On 22/07/17 12:39, Shawn Lin wrote:
> Hi Adrian,
> 
> On 2017/7/21 17:49, Adrian Hunter wrote:
>> From: Venkat Gopalakrishnan <venkatg@codeaurora.org>
>>
>> This patch adds CMDQ support for command-queue compatible
>> hosts.
>>
>> Command queue is added in eMMC-5.1 specification. This
>> enables the controller to process upto 32 requests at
>> a time.
>>
>> Adrian Hunter contributed renaming to cqhci, recovery, suspend
>> and resume, cqhci_off, cqhci_wait_for_idle, and external timeout
>> handling.
>>
>> Signed-off-by: Asutosh Das <asutoshd@codeaurora.org>
>> Signed-off-by: Sujit Reddy Thumma <sthumma@codeaurora.org>
>> Signed-off-by: Konstantin Dorfman <kdorfman@codeaurora.org>
>> Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
>> Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
>> Signed-off-by: Ritesh Harjani <riteshh@codeaurora.org>
>> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
>> ---
>>  drivers/mmc/host/Kconfig  |   13 +
>>  drivers/mmc/host/Makefile |    1 +
>>  drivers/mmc/host/cqhci.c  | 1146
>> +++++++++++++++++++++++++++++++++++++++++++++
>>  drivers/mmc/host/cqhci.h  |  240 ++++++++++
>>  4 files changed, 1400 insertions(+)
>>  create mode 100644 drivers/mmc/host/cqhci.c
>>  create mode 100644 drivers/mmc/host/cqhci.h
>>
> 
> [...]
> 
>> +
>> +static void cqhci_prep_task_desc(struct mmc_request *mrq,
>> +                    u64 *data, bool intr)
>> +{
>> +    u32 req_flags = mrq->data->flags;
>> +
>> +    *data = CQHCI_VALID(1) |
>> +        CQHCI_END(1) |
>> +        CQHCI_INT(intr) |
>> +        CQHCI_ACT(0x5) |
>> +        CQHCI_FORCED_PROG(!!(req_flags & MMC_DATA_FORCED_PRG)) |
>> +        CQHCI_DATA_TAG(!!(req_flags & MMC_DATA_DAT_TAG)) |
>> +        CQHCI_DATA_DIR(!!(req_flags & MMC_DATA_READ)) |
>> +        CQHCI_PRIORITY(!!(req_flags & MMC_DATA_PRIO)) |
>> +        CQHCI_QBAR(!!(req_flags & MMC_DATA_QBR)) |
>> +        CQHCI_REL_WRITE(!!(req_flags & MMC_DATA_REL_WR)) |
>> +        CQHCI_BLK_COUNT(mrq->data->blocks) |
>> +        CQHCI_BLK_ADDR((u64)mrq->data->blk_addr);
>> +
> 
> I don't find where MMC_DATA_FORCED_PRG is set and MMC_DATA_QBR is set

Forced Programming is not used at present.  FUA results in Reliable Write
requests, at least because Reliable Write pre-dates Forced Programming.
Refer mmc_blk_data_prep().  Note CQE is the same as non-CQE in this regard.

WRT QBR, the block layer does not support the concept of barriers.  However
the code is setting QBR always for DCMD, refer cqhci_prep_dcmd_desc().  That
is just because it feels safer and more predictable for flushes but it might
be more efficient to take it off.

> only for DCMD(erase, flush etc). So I have some silly questions :)

DCMD is not used for erase.  It is only used for flush.  That is because
erase requires 3 commands so it is non-trivial to put it into DCMD.  Also it
is not clear whether there would be a benefit from using DCMD for erase.

> 
> (1) Should MMC_DATA_FORCED_PRG be used for secure erase/trim, etc?

Forced Programming applies only to write requests.  Also it is not a
CQE-only feature - refer CMD23.

> (2) How could the overall code guarantee the read-after-write issue? for
> instance, is it possible that two TDLs contain one read for LBA x and a
> write for same LBA x, so how to make sure the devices' execute order?

That issue is the responsibility of the upper layers.  Typically file
systems make use of the page cache where pages are locked during update.

RE: [PATCH V4 10/11] mmc: cqhci: support for command queue enabled host

[Bough Chen]

[...]

> -----Original Message-----
> From: Adrian Hunter [mailto:adrian.hunter@intel.com]
> Sent: Friday, July 21, 2017 5:50 PM
> To: Ulf Hansson <ulf.hansson@linaro.org>
> Cc: linux-mmc <linux-mmc@vger.kernel.org>; Bough Chen
> <haibo.chen@nxp.com>; Alex Lemberg <alex.lemberg@sandisk.com>;
> Mateusz Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung <jh80.chung@samsung.com>;
> Dong Aisheng <dongas86@gmail.com>; Das Asutosh
> <asutoshd@codeaurora.org>; Zhangfei Gao <zhangfei.gao@gmail.com>;
> Dorfman Konstantin <kdorfman@codeaurora.org>; David Griego
> <david.griego@linaro.org>; Sahitya Tummala <stummala@codeaurora.org>;
> Harjani Ritesh <riteshh@codeaurora.org>; Venu Byravarasu
> <vbyravarasu@nvidia.com>; Linus Walleij <linus.walleij@linaro.org>; Shawn Lin
> <shawn.lin@rock-chips.com>
> Subject: [PATCH V4 10/11] mmc: cqhci: support for command queue enabled
> host
> 
> From: Venkat Gopalakrishnan <venkatg@codeaurora.org>
> 
> This patch adds CMDQ support for command-queue compatible hosts.
> 
> Command queue is added in eMMC-5.1 specification. This enables the
> controller to process upto 32 requests at a time.
> 
> Adrian Hunter contributed renaming to cqhci, recovery, suspend and resume,
> cqhci_off, cqhci_wait_for_idle, and external timeout handling.
> 
> Signed-off-by: Asutosh Das <asutoshd@codeaurora.org>
> Signed-off-by: Sujit Reddy Thumma <sthumma@codeaurora.org>
> Signed-off-by: Konstantin Dorfman <kdorfman@codeaurora.org>
> Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
> Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
> Signed-off-by: Ritesh Harjani <riteshh@codeaurora.org>
> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
> ---
>  drivers/mmc/host/Kconfig  |   13 +
>  drivers/mmc/host/Makefile |    1 +
>  drivers/mmc/host/cqhci.c  | 1146
> +++++++++++++++++++++++++++++++++++++++++++++
>  drivers/mmc/host/cqhci.h  |  240 ++++++++++
>  4 files changed, 1400 insertions(+)
>  create mode 100644 drivers/mmc/host/cqhci.c  create mode 100644
> drivers/mmc/host/cqhci.h
> 
> diff --git a/drivers/mmc/host/Kconfig b/drivers/mmc/host/Kconfig index
> 2242633550df..3a164a03f2bc 100644
> --- a/drivers/mmc/host/Kconfig
> +++ b/drivers/mmc/host/Kconfig
> @@ -834,6 +834,19 @@ config MMC_SUNXI
>  	  This selects support for the SD/MMC Host Controller on
>  	  Allwinner sunxi SoCs.
> 
> +config MMC_CQHCI
> +	tristate "Command Queue Host Controller Interface support"
> +	depends on HAS_DMA
> +	help
> +	  This selects the Command Queue Host Controller Interface (CQHCI)
> +	  support present in host controllers of Qualcomm Technologies, Inc
> +	  amongst others.
> +	  This controller supports eMMC devices with command queue support.
> +
> +	  If you have a controller with this interface, say Y or M here.
> +
> +	  If unsure, say N.
> +
>  config MMC_TOSHIBA_PCI
>  	tristate "Toshiba Type A SD/MMC Card Interface Driver"
>  	depends on PCI
> diff --git a/drivers/mmc/host/Makefile b/drivers/mmc/host/Makefile index
> 8c46766c000c..3ae71e006890 100644
> --- a/drivers/mmc/host/Makefile
> +++ b/drivers/mmc/host/Makefile
> @@ -88,6 +88,7 @@ obj-$(CONFIG_MMC_SDHCI_MSM)		+=
> sdhci-msm.o
>  obj-$(CONFIG_MMC_SDHCI_ST)		+= sdhci-st.o
>  obj-$(CONFIG_MMC_SDHCI_MICROCHIP_PIC32)	+= sdhci-pic32.o
>  obj-$(CONFIG_MMC_SDHCI_BRCMSTB)		+= sdhci-brcmstb.o
> +obj-$(CONFIG_MMC_CQHCI)			+= cqhci.o
> 
>  ifeq ($(CONFIG_CB710_DEBUG),y)
>  	CFLAGS-cb710-mmc	+= -DDEBUG
> diff --git a/drivers/mmc/host/cqhci.c b/drivers/mmc/host/cqhci.c new file
> mode 100644 index 000000000000..302421a26230
> --- /dev/null
> +++ b/drivers/mmc/host/cqhci.c
> @@ -0,0 +1,1146 @@

<SNIP>

> +
> +static int cqhci_request(struct mmc_host *mmc, struct mmc_request *mrq)
> +{
> +	int err = 0;
> +	u64 data = 0;
> +	u64 *task_desc = NULL;
> +	int tag = cqhci_tag(mrq);
> +	struct cqhci_host *cq_host = mmc->cqe_private;
> +	unsigned long flags;
> +
> +	if (!cq_host->enabled) {
> +		pr_err("%s: cqhci: not enabled\n", mmc_hostname(mmc));
> +		return -EINVAL;
> +	}
> +
> +	/* First request after resume has to re-enable */
> +	if (!cq_host->activated)
> +		__cqhci_enable(cq_host);
> +
> +	if (!mmc->cqe_on) {
> +		cqhci_writel(cq_host, 0, CQHCI_CTL);
> +		mmc->cqe_on = true;
> +		pr_debug("%s: cqhci: CQE on\n", mmc_hostname(mmc));
> +		if (cqhci_readl(cq_host, CQHCI_CTL) && CQHCI_HALT) {
> +			pr_err("%s: cqhci: CQE failed to exit halt state\n",
> +			       mmc_hostname(mmc));
> +		}
> +		if (cq_host->ops->enable)
> +			cq_host->ops->enable(mmc);
> +	}
> +
> +	if (mrq->data) {
> +		task_desc = (__le64 __force *)get_desc(cq_host, tag);
> +		cqhci_prep_task_desc(mrq, &data, 1);
> +		*task_desc = cpu_to_le64(data);
> +		err = cqhci_prep_tran_desc(mrq, cq_host, tag);
> +		if (err) {
> +			pr_err("%s: cqhci: failed to setup tx desc: %d\n",
> +			       mmc_hostname(mmc), err);
> +			return err;
> +		}
> +	} else {
> +		cqhci_prep_dcmd_desc(mmc, mrq);
> +	}
> +
Hi Adrian,

For cqhci data request, I think we need to also config SDHCI_TRNS_BLK_CNT_EN and SDHCI_TRNS_DMA,  just like function sdhci_set_transfer_mode(): set SDHCI_TRNS_BLK_CNT_EN and SDHCI_TRNS_DMA in every request. 

Currently, we do not do this for cqhci request, if support Runtime PM, when runtime resume, 
sdhci_runtime_resume_host() --> sdhci_init(host, 0) --> sdhci_do_reset(host, SDHCI_RESET_ALL)
sdhci_reset() will do software reset for all, this will clear some SDHCI register, including SDHCI_TRNS_BLK_CNT_EN and SDHCI_TRNS_DMA, then when use cqhci request, error happens. I meet cqhci wait timeout error on our i.MX8. After set these two bits, this issue gone.

Best Regards,
Haibo Chen

> +	spin_lock_irqsave(&cq_host->lock, flags);
> +
> +	if (cq_host->recovery_halt) {
> +		err = -EBUSY;
> +		goto out_unlock;
> +	}
> +
> +	cq_host->slot[tag].mrq = mrq;
> +	cq_host->slot[tag].flags = 0;
> +
> +	cq_host->qcnt += 1;
> +
> +	cqhci_writel(cq_host, 1 << tag, CQHCI_TDBR);
> +	if (!(cqhci_readl(cq_host, CQHCI_TDBR) & (1 << tag)))
> +		pr_debug("%s: cqhci: doorbell not set for tag %d\n",
> +			 mmc_hostname(mmc), tag);
> +out_unlock:
> +	spin_unlock_irqrestore(&cq_host->lock, flags);
> +
> +	if (err)
> +		cqhci_post_req(mmc, mrq);
> +
> +	return err;
> +}
> +
> +static void cqhci_recovery_needed(struct mmc_host *mmc, struct
> mmc_request *mrq,
> +				  bool notify)
> +{
> +	struct cqhci_host *cq_host = mmc->cqe_private;
> +
> +	if (!cq_host->recovery_halt) {
> +		cq_host->recovery_halt = true;
> +		pr_debug("%s: cqhci: recovery needed\n",
> mmc_hostname(mmc));
> +		wake_up(&cq_host->wait_queue);
> +		if (notify && mmc->cqe_recovery_notifier)
> +			mmc->cqe_recovery_notifier(mmc, mrq);
> +	}
> +}
> +
> +static unsigned int cqhci_error_flags(int error1, int error2) {
> +	int error = error1 ? error1 : error2;
> +
> +	switch (error) {
> +	case -EILSEQ:
> +		return CQHCI_HOST_CRC;
> +	case -ETIMEDOUT:
> +		return CQHCI_HOST_TIMEOUT;
> +	default:
> +		return CQHCI_HOST_OTHER;
> +	}
> +}
> +
> +static void cqhci_error_irq(struct mmc_host *mmc, u32 status, int cmd_error,
> +			    int data_error)
> +{
> +	struct cqhci_host *cq_host = mmc->cqe_private;
> +	struct cqhci_slot *slot;
> +	u32 terri;
> +	int tag;
> +
> +	spin_lock(&cq_host->lock);
> +
> +	terri = cqhci_readl(cq_host, CQHCI_TERRI);
> +
> +	pr_debug("%s: cqhci: error IRQ status: 0x%08x cmd error %d data
> error %d TERRI: 0x%08x\n",
> +		 mmc_hostname(mmc), status, cmd_error, data_error, terri);
> +
> +	/* Forget about errors when recovery has already been triggered */
> +	if (cq_host->recovery_halt)
> +		goto out_unlock;
> +
> +	if (!cq_host->qcnt) {
> +		WARN_ONCE(1, "%s: cqhci: error when idle. IRQ status: 0x%08x
> cmd error %d data error %d TERRI: 0x%08x\n",
> +			  mmc_hostname(mmc), status, cmd_error, data_error,
> +			  terri);
> +		goto out_unlock;
> +	}
> +
> +	if (CQHCI_TERRI_C_VALID(terri)) {
> +		tag = CQHCI_TERRI_C_TASK(terri);
> +		slot = &cq_host->slot[tag];
> +		if (slot->mrq) {
> +			slot->flags = cqhci_error_flags(cmd_error, data_error);
> +			cqhci_recovery_needed(mmc, slot->mrq, true);
> +		}
> +	}
> +
> +	if (CQHCI_TERRI_D_VALID(terri)) {
> +		tag = CQHCI_TERRI_D_TASK(terri);
> +		slot = &cq_host->slot[tag];
> +		if (slot->mrq) {
> +			slot->flags = cqhci_error_flags(data_error, cmd_error);
> +			cqhci_recovery_needed(mmc, slot->mrq, true);
> +		}
> +	}
> +
> +	if (!cq_host->recovery_halt) {
> +		/*
> +		 * The only way to guarantee forward progress is to mark at
> +		 * least one task in error, so if none is indicated, pick one.
> +		 */
> +		for (tag = 0; tag < NUM_SLOTS; tag++) {
> +			slot = &cq_host->slot[tag];
> +			if (!slot->mrq)
> +				continue;
> +			slot->flags = cqhci_error_flags(data_error, cmd_error);
> +			cqhci_recovery_needed(mmc, slot->mrq, true);
> +			break;
> +		}
> +	}
> +
> +out_unlock:
> +	spin_unlock(&cq_host->lock);
> +}
> +
> +static void cqhci_finish_mrq(struct mmc_host *mmc, unsigned int tag) {
> +	struct cqhci_host *cq_host = mmc->cqe_private;
> +	struct cqhci_slot *slot = &cq_host->slot[tag];
> +	struct mmc_request *mrq = slot->mrq;
> +	struct mmc_data *data;
> +
> +	if (!mrq) {
> +		WARN_ONCE(1, "%s: cqhci: spurious TCN for tag %d\n",
> +			  mmc_hostname(mmc), tag);
> +		return;
> +	}
> +
> +	/* No completions allowed during recovery */
> +	if (cq_host->recovery_halt) {
> +		slot->flags |= CQHCI_COMPLETED;
> +		return;
> +	}
> +
> +	slot->mrq = NULL;
> +
> +	cq_host->qcnt -= 1;
> +
> +	data = mrq->data;
> +	if (data) {
> +		if (data->error)
> +			data->bytes_xfered = 0;
> +		else
> +			data->bytes_xfered = data->blksz * data->blocks;
> +	}
> +
> +	mmc_cqe_request_done(mmc, mrq);
> +}
> +
> +irqreturn_t cqhci_irq(struct mmc_host *mmc, u32 intmask, int cmd_error,
> +		      int data_error)
> +{
> +	u32 status;
> +	unsigned long tag = 0, comp_status;
> +	struct cqhci_host *cq_host = mmc->cqe_private;
> +
> +	status = cqhci_readl(cq_host, CQHCI_IS);
> +	cqhci_writel(cq_host, status, CQHCI_IS);
> +
> +	pr_debug("%s: cqhci: IRQ status: 0x%08x\n", mmc_hostname(mmc),
> +status);
> +
> +	if ((status & CQHCI_IS_RED) || cmd_error || data_error)
> +		cqhci_error_irq(mmc, status, cmd_error, data_error);
> +
> +	if (status & CQHCI_IS_TCC) {
> +		/* read TCN and complete the request */
> +		comp_status = cqhci_readl(cq_host, CQHCI_TCN);
> +		cqhci_writel(cq_host, comp_status, CQHCI_TCN);
> +		pr_debug("%s: cqhci: TCN: 0x%08lx\n",
> +			 mmc_hostname(mmc), comp_status);
> +
> +		spin_lock(&cq_host->lock);
> +
> +		for_each_set_bit(tag, &comp_status, cq_host->num_slots) {
> +			/* complete the corresponding mrq */
> +			pr_debug("%s: cqhci: completing tag %lu\n",
> +				 mmc_hostname(mmc), tag);
> +			cqhci_finish_mrq(mmc, tag);
> +		}
> +
> +		if (cq_host->waiting_for_idle && !cq_host->qcnt) {
> +			cq_host->waiting_for_idle = false;
> +			wake_up(&cq_host->wait_queue);
> +		}
> +
> +		spin_unlock(&cq_host->lock);
> +	}
> +
> +	if (status & CQHCI_IS_TCL)
> +		wake_up(&cq_host->wait_queue);
> +
> +	if (status & CQHCI_IS_HAC)
> +		wake_up(&cq_host->wait_queue);
> +
> +	return IRQ_HANDLED;
> +}
> +EXPORT_SYMBOL(cqhci_irq);
> +
> +static bool cqhci_is_idle(struct cqhci_host *cq_host, int *ret) {
> +	unsigned long flags;
> +	bool is_idle;
> +
> +	spin_lock_irqsave(&cq_host->lock, flags);
> +	is_idle = !cq_host->qcnt || cq_host->recovery_halt;
> +	*ret = cq_host->recovery_halt ? -EBUSY : 0;
> +	cq_host->waiting_for_idle = !is_idle;
> +	spin_unlock_irqrestore(&cq_host->lock, flags);
> +
> +	return is_idle;
> +}
> +
> +static int cqhci_wait_for_idle(struct mmc_host *mmc) {
> +	struct cqhci_host *cq_host = mmc->cqe_private;
> +	int ret;
> +
> +	wait_event(cq_host->wait_queue, cqhci_is_idle(cq_host, &ret));
> +
> +	return ret;
> +}
> +
> +static bool cqhci_timeout(struct mmc_host *mmc, struct mmc_request *mrq,
> +			  bool *recovery_needed)
> +{
> +	struct cqhci_host *cq_host = mmc->cqe_private;
> +	int tag = cqhci_tag(mrq);
> +	struct cqhci_slot *slot = &cq_host->slot[tag];
> +	unsigned long flags;
> +	bool timed_out;
> +
> +	spin_lock_irqsave(&cq_host->lock, flags);
> +	timed_out = slot->mrq == mrq;
> +	if (timed_out) {
> +		slot->flags |= CQHCI_EXTERNAL_TIMEOUT;
> +		cqhci_recovery_needed(mmc, mrq, false);
> +		*recovery_needed = cq_host->recovery_halt;
> +	}
> +	spin_unlock_irqrestore(&cq_host->lock, flags);
> +
> +	if (timed_out) {
> +		pr_err("%s: cqhci: timeout for tag %d\n",
> +		       mmc_hostname(mmc), tag);
> +		cqhci_dumpregs(cq_host);
> +	}
> +
> +	return timed_out;
> +}
> +
> +static bool cqhci_tasks_cleared(struct cqhci_host *cq_host) {
> +	return !(cqhci_readl(cq_host, CQHCI_CTL) &
> CQHCI_CLEAR_ALL_TASKS); }
> +
> +static bool cqhci_clear_all_tasks(struct mmc_host *mmc, unsigned int
> +timeout) {
> +	struct cqhci_host *cq_host = mmc->cqe_private;
> +	bool ret;
> +	u32 ctl;
> +
> +	cqhci_set_irqs(cq_host, CQHCI_IS_TCL);
> +
> +	ctl = cqhci_readl(cq_host, CQHCI_CTL);
> +	ctl |= CQHCI_CLEAR_ALL_TASKS;
> +	cqhci_writel(cq_host, ctl, CQHCI_CTL);
> +
> +	wait_event_timeout(cq_host->wait_queue,
> cqhci_tasks_cleared(cq_host),
> +			   msecs_to_jiffies(timeout) + 1);
> +
> +	cqhci_set_irqs(cq_host, 0);
> +
> +	ret = cqhci_tasks_cleared(cq_host);
> +
> +	if (!ret)
> +		pr_debug("%s: cqhci: Failed to clear tasks\n",
> +			 mmc_hostname(mmc));
> +
> +	return ret;
> +}
> +
> +static bool cqhci_halted(struct cqhci_host *cq_host) {
> +	return cqhci_readl(cq_host, CQHCI_CTL) & CQHCI_HALT; }
> +
> +static bool cqhci_halt(struct mmc_host *mmc, unsigned int timeout) {
> +	struct cqhci_host *cq_host = mmc->cqe_private;
> +	bool ret;
> +	u32 ctl;
> +
> +	if (cqhci_halted(cq_host))
> +		return true;
> +
> +	cqhci_set_irqs(cq_host, CQHCI_IS_HAC);
> +
> +	ctl = cqhci_readl(cq_host, CQHCI_CTL);
> +	ctl |= CQHCI_HALT;
> +	cqhci_writel(cq_host, ctl, CQHCI_CTL);
> +
> +	wait_event_timeout(cq_host->wait_queue, cqhci_halted(cq_host),
> +			   msecs_to_jiffies(timeout) + 1);
> +
> +	cqhci_set_irqs(cq_host, 0);
> +
> +	ret = cqhci_halted(cq_host);
> +
> +	if (!ret)
> +		pr_debug("%s: cqhci: Failed to halt\n", mmc_hostname(mmc));
> +
> +	return ret;
> +}
> +
> +/*
> + * After halting we expect to be able to use the command line. We
> +interpret the
> + * failure to halt to mean the data lines might still be in use (and
> +the upper
> + * layers will need to send a STOP command), so we set the timeout
> +based on a
> + * generous command timeout.
> + */
> +#define CQHCI_START_HALT_TIMEOUT	5
> +
> +static void cqhci_recovery_start(struct mmc_host *mmc) {
> +	struct cqhci_host *cq_host = mmc->cqe_private;
> +
> +	pr_debug("%s: cqhci: %s\n", mmc_hostname(mmc), __func__);
> +
> +	WARN_ON(!cq_host->recovery_halt);
> +
> +	cqhci_halt(mmc, CQHCI_START_HALT_TIMEOUT);
> +
> +	if (cq_host->ops->disable)
> +		cq_host->ops->disable(mmc, true);
> +
> +	mmc->cqe_on = false;
> +}
> +
> +static int cqhci_error_from_flags(unsigned int flags) {
> +	if (!flags)
> +		return 0;
> +
> +	/* CRC errors might indicate re-tuning so prefer to report that */
> +	if (flags & CQHCI_HOST_CRC)
> +		return -EILSEQ;
> +
> +	if (flags & (CQHCI_EXTERNAL_TIMEOUT | CQHCI_HOST_TIMEOUT))
> +		return -ETIMEDOUT;
> +
> +	return -EIO;
> +}
> +
> +static void cqhci_recover_mrq(struct cqhci_host *cq_host, unsigned int
> +tag) {
> +	struct cqhci_slot *slot = &cq_host->slot[tag];
> +	struct mmc_request *mrq = slot->mrq;
> +	struct mmc_data *data;
> +
> +	if (!mrq)
> +		return;
> +
> +	slot->mrq = NULL;
> +
> +	cq_host->qcnt -= 1;
> +
> +	data = mrq->data;
> +	if (data) {
> +		data->bytes_xfered = 0;
> +		data->error = cqhci_error_from_flags(slot->flags);
> +	} else {
> +		mrq->cmd->error = cqhci_error_from_flags(slot->flags);
> +	}
> +
> +	mmc_cqe_request_done(cq_host->mmc, mrq); }
> +
> +static void cqhci_recover_mrqs(struct cqhci_host *cq_host) {
> +	int i;
> +
> +	for (i = 0; i < cq_host->num_slots; i++)
> +		cqhci_recover_mrq(cq_host, i);
> +}
> +
> +/*
> + * By now the command and data lines should be unused so there is no
> +reason for
> + * CQHCI to take a long time to halt, but if it doesn't halt there
> +could be
> + * problems clearing tasks, so be generous.
> + */
> +#define CQHCI_FINISH_HALT_TIMEOUT	20
> +
> +/* CQHCI could be expected to clear it's internal state pretty quickly */
> +#define CQHCI_CLEAR_TIMEOUT		20
> +
> +static void cqhci_recovery_finish(struct mmc_host *mmc) {
> +	struct cqhci_host *cq_host = mmc->cqe_private;
> +	unsigned long flags;
> +	u32 cqcfg;
> +	bool ok;
> +
> +	pr_debug("%s: cqhci: %s\n", mmc_hostname(mmc), __func__);
> +
> +	WARN_ON(!cq_host->recovery_halt);
> +
> +	ok = cqhci_halt(mmc, CQHCI_FINISH_HALT_TIMEOUT);
> +
> +	if (!cqhci_clear_all_tasks(mmc, CQHCI_CLEAR_TIMEOUT))
> +		ok = false;
> +
> +	/*
> +	 * The specification contradicts itself, by saying that tasks cannot be
> +	 * cleared if CQHCI does not halt, but if CQHCI does not halt, it should
> +	 * be disabled/re-enabled, but not to disable before clearing tasks.
> +	 * Have a go anyway.
> +	 */
> +	if (!ok) {
> +		pr_debug("%s: cqhci: disable / re-enable\n",
> mmc_hostname(mmc));
> +		cqcfg = cqhci_readl(cq_host, CQHCI_CFG);
> +		cqcfg &= ~CQHCI_ENABLE;
> +		cqhci_writel(cq_host, cqcfg, CQHCI_CFG);
> +		cqcfg |= CQHCI_ENABLE;
> +		cqhci_writel(cq_host, cqcfg, CQHCI_CFG);
> +		/* Be sure that there are no tasks */
> +		ok = cqhci_halt(mmc, CQHCI_FINISH_HALT_TIMEOUT);
> +		if (!cqhci_clear_all_tasks(mmc, CQHCI_CLEAR_TIMEOUT))
> +			ok = false;
> +		WARN_ON(!ok);
> +	}
> +
> +	cqhci_recover_mrqs(cq_host);
> +
> +	WARN_ON(cq_host->qcnt);
> +
> +	spin_lock_irqsave(&cq_host->lock, flags);
> +	cq_host->qcnt = 0;
> +	cq_host->recovery_halt = false;
> +	mmc->cqe_on = false;
> +	spin_unlock_irqrestore(&cq_host->lock, flags);
> +
> +	/* Ensure all writes are done before interrupts are re-enabled */
> +	wmb();
> +
> +	cqhci_writel(cq_host, CQHCI_IS_HAC | CQHCI_IS_TCL, CQHCI_IS);
> +
> +	cqhci_set_irqs(cq_host, CQHCI_IS_MASK);
> +
> +	pr_debug("%s: cqhci: recovery done\n", mmc_hostname(mmc)); }
> +
> +static const struct mmc_cqe_ops cqhci_cqe_ops = {
> +	.cqe_enable = cqhci_enable,
> +	.cqe_disable = cqhci_disable,
> +	.cqe_request = cqhci_request,
> +	.cqe_post_req = cqhci_post_req,
> +	.cqe_off = cqhci_off,
> +	.cqe_wait_for_idle = cqhci_wait_for_idle,
> +	.cqe_timeout = cqhci_timeout,
> +	.cqe_recovery_start = cqhci_recovery_start,
> +	.cqe_recovery_finish = cqhci_recovery_finish, };
> +
> +struct cqhci_host *cqhci_pltfm_init(struct platform_device *pdev) {
> +	struct cqhci_host *cq_host;
> +	struct resource *cqhci_memres = NULL;
> +
> +	/* check and setup CMDQ interface */
> +	cqhci_memres = platform_get_resource_byname(pdev,
> IORESOURCE_MEM,
> +						   "cqhci_mem");
> +	if (!cqhci_memres) {
> +		dev_dbg(&pdev->dev, "CMDQ not supported\n");
> +		return ERR_PTR(-EINVAL);
> +	}
> +
> +	cq_host = devm_kzalloc(&pdev->dev, sizeof(*cq_host), GFP_KERNEL);
> +	if (!cq_host)
> +		return ERR_PTR(-ENOMEM);
> +	cq_host->mmio = devm_ioremap(&pdev->dev,
> +				     cqhci_memres->start,
> +				     resource_size(cqhci_memres));
> +	if (!cq_host->mmio) {
> +		dev_err(&pdev->dev, "failed to remap cqhci regs\n");
> +		return ERR_PTR(-EBUSY);
> +	}
> +	dev_dbg(&pdev->dev, "CMDQ ioremap: done\n");
> +
> +	return cq_host;
> +}
> +EXPORT_SYMBOL(cqhci_pltfm_init);
> +
> +static unsigned int cqhci_ver_major(struct cqhci_host *cq_host) {
> +	return CQHCI_VER_MAJOR(cqhci_readl(cq_host, CQHCI_VER)); }
> +
> +static unsigned int cqhci_ver_minor(struct cqhci_host *cq_host) {
> +	u32 ver = cqhci_readl(cq_host, CQHCI_VER);
> +
> +	return CQHCI_VER_MINOR1(ver) * 10 + CQHCI_VER_MINOR2(ver); }
> +
> +int cqhci_init(struct cqhci_host *cq_host, struct mmc_host *mmc,
> +	      bool dma64)
> +{
> +	int err;
> +
> +	cq_host->dma64 = dma64;
> +	cq_host->mmc = mmc;
> +	cq_host->mmc->cqe_private = cq_host;
> +
> +	cq_host->num_slots = NUM_SLOTS;
> +	cq_host->dcmd_slot = DCMD_SLOT;
> +
> +	mmc->cqe_ops = &cqhci_cqe_ops;
> +
> +	mmc->cqe_qdepth = NUM_SLOTS;
> +	if (mmc->caps2 & MMC_CAP2_CQE_DCMD)
> +		mmc->cqe_qdepth -= 1;
> +
> +	cq_host->slot = devm_kcalloc(mmc_dev(mmc), cq_host->num_slots,
> +				     sizeof(*cq_host->slot), GFP_KERNEL);
> +	if (!cq_host->slot) {
> +		err = -ENOMEM;
> +		goto out_err;
> +	}
> +
> +	spin_lock_init(&cq_host->lock);
> +
> +	init_completion(&cq_host->halt_comp);
> +	init_waitqueue_head(&cq_host->wait_queue);
> +
> +	pr_info("%s: CQHCI version %u.%02u\n",
> +		mmc_hostname(mmc), cqhci_ver_major(cq_host),
> +		cqhci_ver_minor(cq_host));
> +
> +	return 0;
> +
> +out_err:
> +	pr_err("%s: CQHCI version %u.%02u failed to initialize, error %d\n",
> +	       mmc_hostname(mmc), cqhci_ver_major(cq_host),
> +	       cqhci_ver_minor(cq_host), err);
> +	return err;
> +}
> +EXPORT_SYMBOL(cqhci_init);
> +
> +MODULE_AUTHOR("Venkat Gopalakrishnan <venkatg@codeaurora.org>");
> +MODULE_DESCRIPTION("Command Queue Host Controller Interface driver");
> +MODULE_LICENSE("GPL v2");
> diff --git a/drivers/mmc/host/cqhci.h b/drivers/mmc/host/cqhci.h new file
> mode 100644 index 000000000000..2d39d361b322
> --- /dev/null
> +++ b/drivers/mmc/host/cqhci.h
> @@ -0,0 +1,240 @@
> +/* Copyright (c) 2015, The Linux Foundation. All rights reserved.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 and
> + * only version 2 as published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + */
> +#ifndef LINUX_MMC_CQHCI_H
> +#define LINUX_MMC_CQHCI_H
> +
> +#include <linux/compiler.h>
> +#include <linux/bitops.h>
> +#include <linux/spinlock_types.h>
> +#include <linux/types.h>
> +#include <linux/completion.h>
> +#include <linux/wait.h>
> +#include <linux/irqreturn.h>
> +#include <asm/io.h>
> +
> +/* registers */
> +/* version */
> +#define CQHCI_VER			0x00
> +#define CQHCI_VER_MAJOR(x)		(((x) & GENMASK(11, 8)) >> 8)
> +#define CQHCI_VER_MINOR1(x)		(((x) & GENMASK(7, 4)) >> 4)
> +#define CQHCI_VER_MINOR2(x)		((x) & GENMASK(3, 0))
> +
> +/* capabilities */
> +#define CQHCI_CAP			0x04
> +/* configuration */
> +#define CQHCI_CFG			0x08
> +#define CQHCI_DCMD			0x00001000
> +#define CQHCI_TASK_DESC_SZ		0x00000100
> +#define CQHCI_ENABLE			0x00000001
> +
> +/* control */
> +#define CQHCI_CTL			0x0C
> +#define CQHCI_CLEAR_ALL_TASKS		0x00000100
> +#define CQHCI_HALT			0x00000001
> +
> +/* interrupt status */
> +#define CQHCI_IS			0x10
> +#define CQHCI_IS_HAC			BIT(0)
> +#define CQHCI_IS_TCC			BIT(1)
> +#define CQHCI_IS_RED			BIT(2)
> +#define CQHCI_IS_TCL			BIT(3)
> +
> +#define CQHCI_IS_MASK (CQHCI_IS_TCC | CQHCI_IS_RED)
> +
> +/* interrupt status enable */
> +#define CQHCI_ISTE			0x14
> +
> +/* interrupt signal enable */
> +#define CQHCI_ISGE			0x18
> +
> +/* interrupt coalescing */
> +#define CQHCI_IC			0x1C
> +#define CQHCI_IC_ENABLE			BIT(31)
> +#define CQHCI_IC_RESET			BIT(16)
> +#define CQHCI_IC_ICCTHWEN		BIT(15)
> +#define CQHCI_IC_ICCTH(x)		((x & 0x1F) << 8)
> +#define CQHCI_IC_ICTOVALWEN		BIT(7)
> +#define CQHCI_IC_ICTOVAL(x)		(x & 0x7F)
> +
> +/* task list base address */
> +#define CQHCI_TDLBA			0x20
> +
> +/* task list base address upper */
> +#define CQHCI_TDLBAU			0x24
> +
> +/* door-bell */
> +#define CQHCI_TDBR			0x28
> +
> +/* task completion notification */
> +#define CQHCI_TCN			0x2C
> +
> +/* device queue status */
> +#define CQHCI_DQS			0x30
> +
> +/* device pending tasks */
> +#define CQHCI_DPT			0x34
> +
> +/* task clear */
> +#define CQHCI_TCLR			0x38
> +
> +/* send status config 1 */
> +#define CQHCI_SSC1			0x40
> +
> +/* send status config 2 */
> +#define CQHCI_SSC2			0x44
> +
> +/* response for dcmd */
> +#define CQHCI_CRDCT			0x48
> +
> +/* response mode error mask */
> +#define CQHCI_RMEM			0x50
> +
> +/* task error info */
> +#define CQHCI_TERRI			0x54
> +
> +#define CQHCI_TERRI_C_INDEX(x)		((x) & GENMASK(5, 0))
> +#define CQHCI_TERRI_C_TASK(x)		(((x) & GENMASK(12, 8)) >> 8)
> +#define CQHCI_TERRI_C_VALID(x)		((x) & BIT(15))
> +#define CQHCI_TERRI_D_INDEX(x)		(((x) & GENMASK(21, 16)) >> 16)
> +#define CQHCI_TERRI_D_TASK(x)		(((x) & GENMASK(28, 24)) >> 24)
> +#define CQHCI_TERRI_D_VALID(x)		((x) & BIT(31))
> +
> +/* command response index */
> +#define CQHCI_CRI			0x58
> +
> +/* command response argument */
> +#define CQHCI_CRA			0x5C
> +
> +#define CQHCI_INT_ALL			0xF
> +#define CQHCI_IC_DEFAULT_ICCTH		31
> +#define CQHCI_IC_DEFAULT_ICTOVAL	1
> +
> +/* attribute fields */
> +#define CQHCI_VALID(x)			((x & 1) << 0)
> +#define CQHCI_END(x)			((x & 1) << 1)
> +#define CQHCI_INT(x)			((x & 1) << 2)
> +#define CQHCI_ACT(x)			((x & 0x7) << 3)
> +
> +/* data command task descriptor fields */
> +#define CQHCI_FORCED_PROG(x)		((x & 1) << 6)
> +#define CQHCI_CONTEXT(x)		((x & 0xF) << 7)
> +#define CQHCI_DATA_TAG(x)		((x & 1) << 11)
> +#define CQHCI_DATA_DIR(x)		((x & 1) << 12)
> +#define CQHCI_PRIORITY(x)		((x & 1) << 13)
> +#define CQHCI_QBAR(x)			((x & 1) << 14)
> +#define CQHCI_REL_WRITE(x)		((x & 1) << 15)
> +#define CQHCI_BLK_COUNT(x)		((x & 0xFFFF) << 16)
> +#define CQHCI_BLK_ADDR(x)		((x & 0xFFFFFFFF) << 32)
> +
> +/* direct command task descriptor fields */
> +#define CQHCI_CMD_INDEX(x)		((x & 0x3F) << 16)
> +#define CQHCI_CMD_TIMING(x)		((x & 1) << 22)
> +#define CQHCI_RESP_TYPE(x)		((x & 0x3) << 23)
> +
> +/* transfer descriptor fields */
> +#define CQHCI_DAT_LENGTH(x)		((x & 0xFFFF) << 16)
> +#define CQHCI_DAT_ADDR_LO(x)		((x & 0xFFFFFFFF) << 32)
> +#define CQHCI_DAT_ADDR_HI(x)		((x & 0xFFFFFFFF) << 0)
> +
> +struct cqhci_host_ops;
> +struct mmc_host;
> +struct cqhci_slot;
> +
> +struct cqhci_host {
> +	const struct cqhci_host_ops *ops;
> +	void __iomem *mmio;
> +	struct mmc_host *mmc;
> +
> +	spinlock_t lock;
> +
> +	/* relative card address of device */
> +	unsigned int rca;
> +
> +	/* 64 bit DMA */
> +	bool dma64;
> +	int num_slots;
> +	int qcnt;
> +
> +	u32 dcmd_slot;
> +	u32 caps;
> +#define CQHCI_TASK_DESC_SZ_128		0x1
> +
> +	u32 quirks;
> +#define CQHCI_QUIRK_SHORT_TXFR_DESC_SZ	0x1
> +
> +	bool enabled;
> +	bool halted;
> +	bool init_done;
> +	bool activated;
> +	bool waiting_for_idle;
> +	bool recovery_halt;
> +
> +	size_t desc_size;
> +	size_t data_size;
> +
> +	u8 *desc_base;
> +
> +	/* total descriptor size */
> +	u8 slot_sz;
> +
> +	/* 64/128 bit depends on CQHCI_CFG */
> +	u8 task_desc_len;
> +
> +	/* 64 bit on 32-bit arch, 128 bit on 64-bit */
> +	u8 link_desc_len;
> +
> +	u8 *trans_desc_base;
> +	/* same length as transfer descriptor */
> +	u8 trans_desc_len;
> +
> +	dma_addr_t desc_dma_base;
> +	dma_addr_t trans_desc_dma_base;
> +
> +	struct completion halt_comp;
> +	wait_queue_head_t wait_queue;
> +	struct cqhci_slot *slot;
> +};
> +
> +struct cqhci_host_ops {
> +	void (*dumpregs)(struct mmc_host *mmc);
> +	void (*write_l)(struct cqhci_host *host, u32 val, int reg);
> +	u32 (*read_l)(struct cqhci_host *host, int reg);
> +	void (*enable)(struct mmc_host *mmc);
> +	void (*disable)(struct mmc_host *mmc, bool recovery); };
> +
> +static inline void cqhci_writel(struct cqhci_host *host, u32 val, int
> +reg) {
> +	if (unlikely(host->ops->write_l))
> +		host->ops->write_l(host, val, reg);
> +	else
> +		writel_relaxed(val, host->mmio + reg); }
> +
> +static inline u32 cqhci_readl(struct cqhci_host *host, int reg) {
> +	if (unlikely(host->ops->read_l))
> +		return host->ops->read_l(host, reg);
> +	else
> +		return readl_relaxed(host->mmio + reg); }
> +
> +struct platform_device;
> +
> +irqreturn_t cqhci_irq(struct mmc_host *mmc, u32 intmask, int cmd_error,
> +		      int data_error);
> +int cqhci_init(struct cqhci_host *cq_host, struct mmc_host *mmc, bool
> +dma64); struct cqhci_host *cqhci_pltfm_init(struct platform_device
> +*pdev); int cqhci_suspend(struct mmc_host *mmc); int
> +cqhci_resume(struct mmc_host *mmc);
> +
> +#endif
> --
> 1.9.1

Re: [PATCH V4 00/11] mmc: Add Command Queue support

[Shawn Lin]

Hi Adrian,

On 2017/7/21 17:49, Adrian Hunter wrote:
> Hi
> 
> Here is V4 of the hardware command queue patches without the software
> command queue patches.
> 

I see this log for the reboot test last weekend. One of my boards
seems to easily trigger this WARN_ONCE.

[   10.012778] mmc1: cqhci: spurious TCN for tag 31
[   10.013332] ------------[ cut here ]------------
[   10.013779] WARNING: CPU: 0 PID: 2596 at drivers/mmc/host/cqhci.c:721 
cqhci_irq+0x2cc/0x42c
[   10.014520] Modules linked in:
[   10.014837] CPU: 0 PID: 2596 Comm: sed Not tainted 
4.13.0-rc1-next-20170720-00012-g9d9bf45 #33
[   10.015595] Hardware name: Firefly-RK3399 Board (DT)
[   10.016043] task: ffff80007b510000 task.stack: ffff80007a160000
[   10.016575] PC is at cqhci_irq+0x2cc/0x42c
[   10.016946] LR is at cqhci_irq+0x2cc/0x42c
[   10.017321] pc : [<ffff0000087f7c3c>] lr : [<ffff0000087f7c3c>] 
pstate: 600001c5
[   10.017974] sp : ffff80007ef11db0
[   10.018275] x29: ffff80007ef11db0 x28: 0000000000000001
[   10.018759] x27: ffff80007ba7d600 x26: 0000000000000001
[   10.019242] x25: ffff000008d279a8 x24: ffff80007b8f4130
[   10.019725] x23: ffff000008fb0d00 x22: 0000000000000002
[   10.020208] x21: ffff80007ba7d000 x20: ffff80007b8f4118
[   10.020690] x19: 000000000000001f x18: 0000000000000000
[   10.021172] x17: 0000000000000000 x16: 0000000000000000
[   10.021654] x15: 0000000000000000 x14: 00000000f7b60065
[   10.022138] x13: 00000000ffc54040 x12: 0000000000000000
[   10.022624] x11: 000000000000081b x10: 0000000000000006
[   10.023105] x9 : 000000000000081c x8 : ffff00000853de20
[   10.023587] x7 : ffff000008ec9df8 x6 : ffff00000900651b
[   10.024070] x5 : 0000000000000000 x4 : 0000000000000000
[   10.024553] x3 : 0000000000000000 x2 : ffff80007ef19970
[   10.025035] x1 : ffff80007b510000 x0 : 0000000000000024
[   10.025519] Call trace:
[   10.025761] Exception stack(0xffff80007ef11be0 to 0xffff80007ef11d10)
[   10.026342] 1be0: 000000000000001f 0001000000000000 ffff80007ef11db0 
ffff0000087f7c3c
[   10.027040] 1c00: ffff000008d279a8 0000000000000001 ffff80007ba7d600 
0000000000000001
[   10.027738] 1c20: 0000000000000002 0000000000000000 ffff80007ef11db0 
ffff80007ef11db0
[   10.028437] 1c40: ffff80007ef11d70 00000000ffffffc8 ffff80007ef11c80 
ffff00000810f980
[   10.029135] 1c60: ffff80007ef11db0 ffff80007ef11db0 ffff80007ef11d70 
00000000ffffffc8
[   10.029833] 1c80: 0000000000000024 ffff80007b510000 ffff80007ef19970 
0000000000000000
[   10.030533] 1ca0: 0000000000000000 0000000000000000 ffff00000900651b 
ffff000008ec9df8
[   10.031230] 1cc0: ffff00000853de20 000000000000081c 0000000000000006 
000000000000081b
[   10.031928] 1ce0: 0000000000000000 00000000ffc54040 00000000f7b60065 
0000000000000000
[   10.032624] 1d00: 0000000000000000 0000000000000000
[   10.033068] [<ffff0000087f7c3c>] cqhci_irq+0x2cc/0x42c
[   10.033541] [<ffff0000087f2c90>] sdhci_arasan_cqhci_irq+0x44/0x58
[   10.034092] [<ffff0000087dec28>] sdhci_irq+0xb0/0xb74
[   10.034554] [<ffff00000811096c>] __handle_irq_event_percpu+0xa0/0x128
[   10.035133] [<ffff000008110a10>] handle_irq_event_percpu+0x1c/0x54
[   10.035687] [<ffff000008110a8c>] handle_irq_event+0x44/0x74
[   10.036192] [<ffff000008114040>] handle_fasteoi_irq+0x9c/0x154
[   10.036717] [<ffff00000810fa7c>] generic_handle_irq+0x24/0x38
[   10.037234] [<ffff0000081100fc>] __handle_domain_irq+0x60/0xac
[   10.037759] [<ffff00000808173c>] gic_handle_irq+0xd4/0x17c

> 
> Changes since V3:
> 	Adjusted ...blk_end_request...() for new block status codes
> 	Fixed CQHCI transaction descriptor for "no DCMD" case
> 
> Changes since V2:
> 	Dropped patches that have been applied.
> 	Re-based
> 	Added "mmc: sdhci-pci: Add CQHCI support for Intel GLK"
> 
> Changes since V1:
> 
> 	"Share mmc request array between partitions" is dependent
> 	on changes in "Introduce queue semantics", so added that
> 	and block fixes:
> 
> 	Added "Fix is_waiting_last_req set incorrectly"
> 	Added "Fix cmd error reset failure path"
> 	Added "Use local var for mqrq_cur"
> 	Added "Introduce queue semantics"
> 
> Changes since RFC:
> 
> 	Re-based on next.
> 	Added comment about command queue priority.
> 	Added some acks and reviews.
> 
> 
> Adrian Hunter (10):
>        mmc: core: Add mmc_retune_hold_now()
>        mmc: core: Add members to mmc_request and mmc_data for CQE's
>        mmc: host: Add CQE interface
>        mmc: core: Turn off CQE before sending commands
>        mmc: core: Add support for handling CQE requests
>        mmc: mmc: Enable Command Queuing
>        mmc: mmc: Enable CQE's
>        mmc: block: Prepare CQE data
>        mmc: block: Add CQE support
>        mmc: sdhci-pci: Add CQHCI support for Intel GLK
> 
> Venkat Gopalakrishnan (1):
>        mmc: cqhci: support for command queue enabled host
> 
>   drivers/mmc/core/block.c          |  240 +++++++-
>   drivers/mmc/core/block.h          |    7 +
>   drivers/mmc/core/bus.c            |    7 +
>   drivers/mmc/core/core.c           |  156 ++++-
>   drivers/mmc/core/host.c           |    6 +
>   drivers/mmc/core/host.h           |    1 +
>   drivers/mmc/core/mmc.c            |   30 +-
>   drivers/mmc/core/queue.c          |  273 ++++++++-
>   drivers/mmc/core/queue.h          |   42 +-
>   drivers/mmc/host/Kconfig          |   14 +
>   drivers/mmc/host/Makefile         |    1 +
>   drivers/mmc/host/cqhci.c          | 1146 +++++++++++++++++++++++++++++++++++++
>   drivers/mmc/host/cqhci.h          |  240 ++++++++
>   drivers/mmc/host/sdhci-pci-core.c |  153 ++++-
>   include/linux/mmc/core.h          |   18 +-
>   include/linux/mmc/host.h          |   24 +
>   include/trace/events/mmc.h        |   36 +-
>   17 files changed, 2354 insertions(+), 40 deletions(-)
>   create mode 100644 drivers/mmc/host/cqhci.c
>   create mode 100644 drivers/mmc/host/cqhci.h
> 
> 
> Regards
> Adrian
> 
> 
> 

Re: [PATCH V4 00/11] mmc: Add Command Queue support

[Adrian Hunter]

On 24/07/17 12:17, Shawn Lin wrote:
> Hi Adrian,
> 
> On 2017/7/21 17:49, Adrian Hunter wrote:
>> Hi
>>
>> Here is V4 of the hardware command queue patches without the software
>> command queue patches.
>>
> 
> I see this log for the reboot test last weekend. One of my boards
> seems to easily trigger this WARN_ONCE.
> 
> [   10.012778] mmc1: cqhci: spurious TCN for tag 31

Tag 31 is DCMD.  DCMD can be problematic which is one of the reasons it is
optional.  In this case it looks like you are getting a TCN before the DCMD
is actually finished.  You should be able to see if that is the case by
dumping the registers when it happens, although you will need to stop
messages going to the console because that will be too slow.

> [   10.013332] ------------[ cut here ]------------
> [   10.013779] WARNING: CPU: 0 PID: 2596 at drivers/mmc/host/cqhci.c:721
> cqhci_irq+0x2cc/0x42c
> [   10.014520] Modules linked in:
> [   10.014837] CPU: 0 PID: 2596 Comm: sed Not tainted
> 4.13.0-rc1-next-20170720-00012-g9d9bf45 #33
> [   10.015595] Hardware name: Firefly-RK3399 Board (DT)
> [   10.016043] task: ffff80007b510000 task.stack: ffff80007a160000
> [   10.016575] PC is at cqhci_irq+0x2cc/0x42c
> [   10.016946] LR is at cqhci_irq+0x2cc/0x42c
> [   10.017321] pc : [<ffff0000087f7c3c>] lr : [<ffff0000087f7c3c>] pstate:
> 600001c5
> [   10.017974] sp : ffff80007ef11db0
> [   10.018275] x29: ffff80007ef11db0 x28: 0000000000000001
> [   10.018759] x27: ffff80007ba7d600 x26: 0000000000000001
> [   10.019242] x25: ffff000008d279a8 x24: ffff80007b8f4130
> [   10.019725] x23: ffff000008fb0d00 x22: 0000000000000002
> [   10.020208] x21: ffff80007ba7d000 x20: ffff80007b8f4118
> [   10.020690] x19: 000000000000001f x18: 0000000000000000
> [   10.021172] x17: 0000000000000000 x16: 0000000000000000
> [   10.021654] x15: 0000000000000000 x14: 00000000f7b60065
> [   10.022138] x13: 00000000ffc54040 x12: 0000000000000000
> [   10.022624] x11: 000000000000081b x10: 0000000000000006
> [   10.023105] x9 : 000000000000081c x8 : ffff00000853de20
> [   10.023587] x7 : ffff000008ec9df8 x6 : ffff00000900651b
> [   10.024070] x5 : 0000000000000000 x4 : 0000000000000000
> [   10.024553] x3 : 0000000000000000 x2 : ffff80007ef19970
> [   10.025035] x1 : ffff80007b510000 x0 : 0000000000000024
> [   10.025519] Call trace:
> [   10.025761] Exception stack(0xffff80007ef11be0 to 0xffff80007ef11d10)
> [   10.026342] 1be0: 000000000000001f 0001000000000000 ffff80007ef11db0
> ffff0000087f7c3c
> [   10.027040] 1c00: ffff000008d279a8 0000000000000001 ffff80007ba7d600
> 0000000000000001
> [   10.027738] 1c20: 0000000000000002 0000000000000000 ffff80007ef11db0
> ffff80007ef11db0
> [   10.028437] 1c40: ffff80007ef11d70 00000000ffffffc8 ffff80007ef11c80
> ffff00000810f980
> [   10.029135] 1c60: ffff80007ef11db0 ffff80007ef11db0 ffff80007ef11d70
> 00000000ffffffc8
> [   10.029833] 1c80: 0000000000000024 ffff80007b510000 ffff80007ef19970
> 0000000000000000
> [   10.030533] 1ca0: 0000000000000000 0000000000000000 ffff00000900651b
> ffff000008ec9df8
> [   10.031230] 1cc0: ffff00000853de20 000000000000081c 0000000000000006
> 000000000000081b
> [   10.031928] 1ce0: 0000000000000000 00000000ffc54040 00000000f7b60065
> 0000000000000000
> [   10.032624] 1d00: 0000000000000000 0000000000000000
> [   10.033068] [<ffff0000087f7c3c>] cqhci_irq+0x2cc/0x42c
> [   10.033541] [<ffff0000087f2c90>] sdhci_arasan_cqhci_irq+0x44/0x58
> [   10.034092] [<ffff0000087dec28>] sdhci_irq+0xb0/0xb74
> [   10.034554] [<ffff00000811096c>] __handle_irq_event_percpu+0xa0/0x128
> [   10.035133] [<ffff000008110a10>] handle_irq_event_percpu+0x1c/0x54
> [   10.035687] [<ffff000008110a8c>] handle_irq_event+0x44/0x74
> [   10.036192] [<ffff000008114040>] handle_fasteoi_irq+0x9c/0x154
> [   10.036717] [<ffff00000810fa7c>] generic_handle_irq+0x24/0x38
> [   10.037234] [<ffff0000081100fc>] __handle_domain_irq+0x60/0xac
> [   10.037759] [<ffff00000808173c>] gic_handle_irq+0xd4/0x17c
> 
>>
>> Changes since V3:
>>     Adjusted ...blk_end_request...() for new block status codes
>>     Fixed CQHCI transaction descriptor for "no DCMD" case
>>
>> Changes since V2:
>>     Dropped patches that have been applied.
>>     Re-based
>>     Added "mmc: sdhci-pci: Add CQHCI support for Intel GLK"
>>
>> Changes since V1:
>>
>>     "Share mmc request array between partitions" is dependent
>>     on changes in "Introduce queue semantics", so added that
>>     and block fixes:
>>
>>     Added "Fix is_waiting_last_req set incorrectly"
>>     Added "Fix cmd error reset failure path"
>>     Added "Use local var for mqrq_cur"
>>     Added "Introduce queue semantics"
>>
>> Changes since RFC:
>>
>>     Re-based on next.
>>     Added comment about command queue priority.
>>     Added some acks and reviews.
>>
>>
>> Adrian Hunter (10):
>>        mmc: core: Add mmc_retune_hold_now()
>>        mmc: core: Add members to mmc_request and mmc_data for CQE's
>>        mmc: host: Add CQE interface
>>        mmc: core: Turn off CQE before sending commands
>>        mmc: core: Add support for handling CQE requests
>>        mmc: mmc: Enable Command Queuing
>>        mmc: mmc: Enable CQE's
>>        mmc: block: Prepare CQE data
>>        mmc: block: Add CQE support
>>        mmc: sdhci-pci: Add CQHCI support for Intel GLK
>>
>> Venkat Gopalakrishnan (1):
>>        mmc: cqhci: support for command queue enabled host
>>
>>   drivers/mmc/core/block.c          |  240 +++++++-
>>   drivers/mmc/core/block.h          |    7 +
>>   drivers/mmc/core/bus.c            |    7 +
>>   drivers/mmc/core/core.c           |  156 ++++-
>>   drivers/mmc/core/host.c           |    6 +
>>   drivers/mmc/core/host.h           |    1 +
>>   drivers/mmc/core/mmc.c            |   30 +-
>>   drivers/mmc/core/queue.c          |  273 ++++++++-
>>   drivers/mmc/core/queue.h          |   42 +-
>>   drivers/mmc/host/Kconfig          |   14 +
>>   drivers/mmc/host/Makefile         |    1 +
>>   drivers/mmc/host/cqhci.c          | 1146
>> +++++++++++++++++++++++++++++++++++++
>>   drivers/mmc/host/cqhci.h          |  240 ++++++++
>>   drivers/mmc/host/sdhci-pci-core.c |  153 ++++-
>>   include/linux/mmc/core.h          |   18 +-
>>   include/linux/mmc/host.h          |   24 +
>>   include/trace/events/mmc.h        |   36 +-
>>   17 files changed, 2354 insertions(+), 40 deletions(-)
>>   create mode 100644 drivers/mmc/host/cqhci.c
>>   create mode 100644 drivers/mmc/host/cqhci.h
>>
>>
>> Regards
>> Adrian
>>
>>
>>
> 
> 

Re: [PATCH V4 10/11] mmc: cqhci: support for command queue enabled host

[Adrian Hunter]

On 24/07/17 11:52, Bough Chen wrote:
> [...]
> 
>> -----Original Message-----
>> From: Adrian Hunter [mailto:adrian.hunter@intel.com]
>> Sent: Friday, July 21, 2017 5:50 PM
>> To: Ulf Hansson <ulf.hansson@linaro.org>
>> Cc: linux-mmc <linux-mmc@vger.kernel.org>; Bough Chen
>> <haibo.chen@nxp.com>; Alex Lemberg <alex.lemberg@sandisk.com>;
>> Mateusz Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
>> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung <jh80.chung@samsung.com>;
>> Dong Aisheng <dongas86@gmail.com>; Das Asutosh
>> <asutoshd@codeaurora.org>; Zhangfei Gao <zhangfei.gao@gmail.com>;
>> Dorfman Konstantin <kdorfman@codeaurora.org>; David Griego
>> <david.griego@linaro.org>; Sahitya Tummala <stummala@codeaurora.org>;
>> Harjani Ritesh <riteshh@codeaurora.org>; Venu Byravarasu
>> <vbyravarasu@nvidia.com>; Linus Walleij <linus.walleij@linaro.org>; Shawn Lin
>> <shawn.lin@rock-chips.com>
>> Subject: [PATCH V4 10/11] mmc: cqhci: support for command queue enabled
>> host
>>
>> From: Venkat Gopalakrishnan <venkatg@codeaurora.org>
>>
>> This patch adds CMDQ support for command-queue compatible hosts.
>>
>> Command queue is added in eMMC-5.1 specification. This enables the
>> controller to process upto 32 requests at a time.
>>
>> Adrian Hunter contributed renaming to cqhci, recovery, suspend and resume,
>> cqhci_off, cqhci_wait_for_idle, and external timeout handling.
>>
>> Signed-off-by: Asutosh Das <asutoshd@codeaurora.org>
>> Signed-off-by: Sujit Reddy Thumma <sthumma@codeaurora.org>
>> Signed-off-by: Konstantin Dorfman <kdorfman@codeaurora.org>
>> Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
>> Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
>> Signed-off-by: Ritesh Harjani <riteshh@codeaurora.org>
>> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
>> ---
>>  drivers/mmc/host/Kconfig  |   13 +
>>  drivers/mmc/host/Makefile |    1 +
>>  drivers/mmc/host/cqhci.c  | 1146
>> +++++++++++++++++++++++++++++++++++++++++++++
>>  drivers/mmc/host/cqhci.h  |  240 ++++++++++
>>  4 files changed, 1400 insertions(+)
>>  create mode 100644 drivers/mmc/host/cqhci.c  create mode 100644
>> drivers/mmc/host/cqhci.h
>>
>> diff --git a/drivers/mmc/host/Kconfig b/drivers/mmc/host/Kconfig index
>> 2242633550df..3a164a03f2bc 100644
>> --- a/drivers/mmc/host/Kconfig
>> +++ b/drivers/mmc/host/Kconfig
>> @@ -834,6 +834,19 @@ config MMC_SUNXI
>>  	  This selects support for the SD/MMC Host Controller on
>>  	  Allwinner sunxi SoCs.
>>
>> +config MMC_CQHCI
>> +	tristate "Command Queue Host Controller Interface support"
>> +	depends on HAS_DMA
>> +	help
>> +	  This selects the Command Queue Host Controller Interface (CQHCI)
>> +	  support present in host controllers of Qualcomm Technologies, Inc
>> +	  amongst others.
>> +	  This controller supports eMMC devices with command queue support.
>> +
>> +	  If you have a controller with this interface, say Y or M here.
>> +
>> +	  If unsure, say N.
>> +
>>  config MMC_TOSHIBA_PCI
>>  	tristate "Toshiba Type A SD/MMC Card Interface Driver"
>>  	depends on PCI
>> diff --git a/drivers/mmc/host/Makefile b/drivers/mmc/host/Makefile index
>> 8c46766c000c..3ae71e006890 100644
>> --- a/drivers/mmc/host/Makefile
>> +++ b/drivers/mmc/host/Makefile
>> @@ -88,6 +88,7 @@ obj-$(CONFIG_MMC_SDHCI_MSM)		+=
>> sdhci-msm.o
>>  obj-$(CONFIG_MMC_SDHCI_ST)		+= sdhci-st.o
>>  obj-$(CONFIG_MMC_SDHCI_MICROCHIP_PIC32)	+= sdhci-pic32.o
>>  obj-$(CONFIG_MMC_SDHCI_BRCMSTB)		+= sdhci-brcmstb.o
>> +obj-$(CONFIG_MMC_CQHCI)			+= cqhci.o
>>
>>  ifeq ($(CONFIG_CB710_DEBUG),y)
>>  	CFLAGS-cb710-mmc	+= -DDEBUG
>> diff --git a/drivers/mmc/host/cqhci.c b/drivers/mmc/host/cqhci.c new file
>> mode 100644 index 000000000000..302421a26230
>> --- /dev/null
>> +++ b/drivers/mmc/host/cqhci.c
>> @@ -0,0 +1,1146 @@
> 
> <SNIP>
> 
>> +
>> +static int cqhci_request(struct mmc_host *mmc, struct mmc_request *mrq)
>> +{
>> +	int err = 0;
>> +	u64 data = 0;
>> +	u64 *task_desc = NULL;
>> +	int tag = cqhci_tag(mrq);
>> +	struct cqhci_host *cq_host = mmc->cqe_private;
>> +	unsigned long flags;
>> +
>> +	if (!cq_host->enabled) {
>> +		pr_err("%s: cqhci: not enabled\n", mmc_hostname(mmc));
>> +		return -EINVAL;
>> +	}
>> +
>> +	/* First request after resume has to re-enable */
>> +	if (!cq_host->activated)
>> +		__cqhci_enable(cq_host);
>> +
>> +	if (!mmc->cqe_on) {
>> +		cqhci_writel(cq_host, 0, CQHCI_CTL);
>> +		mmc->cqe_on = true;
>> +		pr_debug("%s: cqhci: CQE on\n", mmc_hostname(mmc));
>> +		if (cqhci_readl(cq_host, CQHCI_CTL) && CQHCI_HALT) {
>> +			pr_err("%s: cqhci: CQE failed to exit halt state\n",
>> +			       mmc_hostname(mmc));
>> +		}
>> +		if (cq_host->ops->enable)
>> +			cq_host->ops->enable(mmc);
>> +	}
>> +
>> +	if (mrq->data) {
>> +		task_desc = (__le64 __force *)get_desc(cq_host, tag);
>> +		cqhci_prep_task_desc(mrq, &data, 1);
>> +		*task_desc = cpu_to_le64(data);
>> +		err = cqhci_prep_tran_desc(mrq, cq_host, tag);
>> +		if (err) {
>> +			pr_err("%s: cqhci: failed to setup tx desc: %d\n",
>> +			       mmc_hostname(mmc), err);
>> +			return err;
>> +		}
>> +	} else {
>> +		cqhci_prep_dcmd_desc(mmc, mrq);
>> +	}
>> +
> Hi Adrian,
> 
> For cqhci data request, I think we need to also config SDHCI_TRNS_BLK_CNT_EN and SDHCI_TRNS_DMA,  just like function sdhci_set_transfer_mode(): set SDHCI_TRNS_BLK_CNT_EN and SDHCI_TRNS_DMA in every request. 
> 
> Currently, we do not do this for cqhci request, if support Runtime PM, when runtime resume, 
> sdhci_runtime_resume_host() --> sdhci_init(host, 0) --> sdhci_do_reset(host, SDHCI_RESET_ALL)
> sdhci_reset() will do software reset for all, this will clear some SDHCI register, including SDHCI_TRNS_BLK_CNT_EN and SDHCI_TRNS_DMA, then when use cqhci request, error happens. I meet cqhci wait timeout error on our i.MX8. After set these two bits, this issue gone.

CQE anyway needs to write the transfer mode bit-4 so it is a bit strange
that it doesn't write the other bits too.  Nevertheless anything you need to
set can be done in the ->enable() callback.  For example see glk_cqe_enable().

Re: [PATCH V4 00/11] mmc: Add Command Queue support

[Shawn Lin]

Hi

On 2017/7/24 18:09, Adrian Hunter wrote:
> On 24/07/17 12:17, Shawn Lin wrote:
>> Hi Adrian,
>>
>> On 2017/7/21 17:49, Adrian Hunter wrote:
>>> Hi
>>>
>>> Here is V4 of the hardware command queue patches without the software
>>> command queue patches.
>>>
>>
>> I see this log for the reboot test last weekend. One of my boards
>> seems to easily trigger this WARN_ONCE.
>>
>> [   10.012778] mmc1: cqhci: spurious TCN for tag 31
> 
> Tag 31 is DCMD.  DCMD can be problematic which is one of the reasons it is
> optional.  In this case it looks like you are getting a TCN before the DCMD
> is actually finished.  You should be able to see if that is the case by

yes, it is that case you refer to.

> dumping the registers when it happens, although you will need to stop
> messages going to the console because that will be too slow.

okay, I will remove DCMD caps for the test and see what we could
do to make DCMD more robust later.

> 
>> [   10.013332] ------------[ cut here ]------------
>> [   10.013779] WARNING: CPU: 0 PID: 2596 at drivers/mmc/host/cqhci.c:721
>> cqhci_irq+0x2cc/0x42c
>> [   10.014520] Modules linked in:
>> [   10.014837] CPU: 0 PID: 2596 Comm: sed Not tainted
>> 4.13.0-rc1-next-20170720-00012-g9d9bf45 #33
>> [   10.015595] Hardware name: Firefly-RK3399 Board (DT)
>> [   10.016043] task: ffff80007b510000 task.stack: ffff80007a160000
>> [   10.016575] PC is at cqhci_irq+0x2cc/0x42c
>> [   10.016946] LR is at cqhci_irq+0x2cc/0x42c
>> [   10.017321] pc : [<ffff0000087f7c3c>] lr : [<ffff0000087f7c3c>] pstate:
>> 600001c5
>> [   10.017974] sp : ffff80007ef11db0
>> [   10.018275] x29: ffff80007ef11db0 x28: 0000000000000001
>> [   10.018759] x27: ffff80007ba7d600 x26: 0000000000000001
>> [   10.019242] x25: ffff000008d279a8 x24: ffff80007b8f4130
>> [   10.019725] x23: ffff000008fb0d00 x22: 0000000000000002
>> [   10.020208] x21: ffff80007ba7d000 x20: ffff80007b8f4118
>> [   10.020690] x19: 000000000000001f x18: 0000000000000000
>> [   10.021172] x17: 0000000000000000 x16: 0000000000000000
>> [   10.021654] x15: 0000000000000000 x14: 00000000f7b60065
>> [   10.022138] x13: 00000000ffc54040 x12: 0000000000000000
>> [   10.022624] x11: 000000000000081b x10: 0000000000000006
>> [   10.023105] x9 : 000000000000081c x8 : ffff00000853de20
>> [   10.023587] x7 : ffff000008ec9df8 x6 : ffff00000900651b
>> [   10.024070] x5 : 0000000000000000 x4 : 0000000000000000
>> [   10.024553] x3 : 0000000000000000 x2 : ffff80007ef19970
>> [   10.025035] x1 : ffff80007b510000 x0 : 0000000000000024
>> [   10.025519] Call trace:
>> [   10.025761] Exception stack(0xffff80007ef11be0 to 0xffff80007ef11d10)
>> [   10.026342] 1be0: 000000000000001f 0001000000000000 ffff80007ef11db0
>> ffff0000087f7c3c
>> [   10.027040] 1c00: ffff000008d279a8 0000000000000001 ffff80007ba7d600
>> 0000000000000001
>> [   10.027738] 1c20: 0000000000000002 0000000000000000 ffff80007ef11db0
>> ffff80007ef11db0
>> [   10.028437] 1c40: ffff80007ef11d70 00000000ffffffc8 ffff80007ef11c80
>> ffff00000810f980
>> [   10.029135] 1c60: ffff80007ef11db0 ffff80007ef11db0 ffff80007ef11d70
>> 00000000ffffffc8
>> [   10.029833] 1c80: 0000000000000024 ffff80007b510000 ffff80007ef19970
>> 0000000000000000
>> [   10.030533] 1ca0: 0000000000000000 0000000000000000 ffff00000900651b
>> ffff000008ec9df8
>> [   10.031230] 1cc0: ffff00000853de20 000000000000081c 0000000000000006
>> 000000000000081b
>> [   10.031928] 1ce0: 0000000000000000 00000000ffc54040 00000000f7b60065
>> 0000000000000000
>> [   10.032624] 1d00: 0000000000000000 0000000000000000
>> [   10.033068] [<ffff0000087f7c3c>] cqhci_irq+0x2cc/0x42c
>> [   10.033541] [<ffff0000087f2c90>] sdhci_arasan_cqhci_irq+0x44/0x58
>> [   10.034092] [<ffff0000087dec28>] sdhci_irq+0xb0/0xb74
>> [   10.034554] [<ffff00000811096c>] __handle_irq_event_percpu+0xa0/0x128
>> [   10.035133] [<ffff000008110a10>] handle_irq_event_percpu+0x1c/0x54
>> [   10.035687] [<ffff000008110a8c>] handle_irq_event+0x44/0x74
>> [   10.036192] [<ffff000008114040>] handle_fasteoi_irq+0x9c/0x154
>> [   10.036717] [<ffff00000810fa7c>] generic_handle_irq+0x24/0x38
>> [   10.037234] [<ffff0000081100fc>] __handle_domain_irq+0x60/0xac
>> [   10.037759] [<ffff00000808173c>] gic_handle_irq+0xd4/0x17c
>>
>>>
>>> Changes since V3:
>>>      Adjusted ...blk_end_request...() for new block status codes
>>>      Fixed CQHCI transaction descriptor for "no DCMD" case
>>>
>>> Changes since V2:
>>>      Dropped patches that have been applied.
>>>      Re-based
>>>      Added "mmc: sdhci-pci: Add CQHCI support for Intel GLK"
>>>
>>> Changes since V1:
>>>
>>>      "Share mmc request array between partitions" is dependent
>>>      on changes in "Introduce queue semantics", so added that
>>>      and block fixes:
>>>
>>>      Added "Fix is_waiting_last_req set incorrectly"
>>>      Added "Fix cmd error reset failure path"
>>>      Added "Use local var for mqrq_cur"
>>>      Added "Introduce queue semantics"
>>>
>>> Changes since RFC:
>>>
>>>      Re-based on next.
>>>      Added comment about command queue priority.
>>>      Added some acks and reviews.
>>>
>>>
>>> Adrian Hunter (10):
>>>         mmc: core: Add mmc_retune_hold_now()
>>>         mmc: core: Add members to mmc_request and mmc_data for CQE's
>>>         mmc: host: Add CQE interface
>>>         mmc: core: Turn off CQE before sending commands
>>>         mmc: core: Add support for handling CQE requests
>>>         mmc: mmc: Enable Command Queuing
>>>         mmc: mmc: Enable CQE's
>>>         mmc: block: Prepare CQE data
>>>         mmc: block: Add CQE support
>>>         mmc: sdhci-pci: Add CQHCI support for Intel GLK
>>>
>>> Venkat Gopalakrishnan (1):
>>>         mmc: cqhci: support for command queue enabled host
>>>
>>>    drivers/mmc/core/block.c          |  240 +++++++-
>>>    drivers/mmc/core/block.h          |    7 +
>>>    drivers/mmc/core/bus.c            |    7 +
>>>    drivers/mmc/core/core.c           |  156 ++++-
>>>    drivers/mmc/core/host.c           |    6 +
>>>    drivers/mmc/core/host.h           |    1 +
>>>    drivers/mmc/core/mmc.c            |   30 +-
>>>    drivers/mmc/core/queue.c          |  273 ++++++++-
>>>    drivers/mmc/core/queue.h          |   42 +-
>>>    drivers/mmc/host/Kconfig          |   14 +
>>>    drivers/mmc/host/Makefile         |    1 +
>>>    drivers/mmc/host/cqhci.c          | 1146
>>> +++++++++++++++++++++++++++++++++++++
>>>    drivers/mmc/host/cqhci.h          |  240 ++++++++
>>>    drivers/mmc/host/sdhci-pci-core.c |  153 ++++-
>>>    include/linux/mmc/core.h          |   18 +-
>>>    include/linux/mmc/host.h          |   24 +
>>>    include/trace/events/mmc.h        |   36 +-
>>>    17 files changed, 2354 insertions(+), 40 deletions(-)
>>>    create mode 100644 drivers/mmc/host/cqhci.c
>>>    create mode 100644 drivers/mmc/host/cqhci.h
>>>
>>>
>>> Regards
>>> Adrian
>>>
>>>
>>>
>>
>>
> 
> 
> 
> 

Re: [PATCH V4 10/11] mmc: cqhci: support for command queue enabled host

[Adrian Hunter]

On 24/07/17 13:21, Adrian Hunter wrote:
> On 24/07/17 11:52, Bough Chen wrote:
>> [...]
>>
>>> -----Original Message-----
>>> From: Adrian Hunter [mailto:adrian.hunter@intel.com]
>>> Sent: Friday, July 21, 2017 5:50 PM
>>> To: Ulf Hansson <ulf.hansson@linaro.org>
>>> Cc: linux-mmc <linux-mmc@vger.kernel.org>; Bough Chen
>>> <haibo.chen@nxp.com>; Alex Lemberg <alex.lemberg@sandisk.com>;
>>> Mateusz Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
>>> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung <jh80.chung@samsung.com>;
>>> Dong Aisheng <dongas86@gmail.com>; Das Asutosh
>>> <asutoshd@codeaurora.org>; Zhangfei Gao <zhangfei.gao@gmail.com>;
>>> Dorfman Konstantin <kdorfman@codeaurora.org>; David Griego
>>> <david.griego@linaro.org>; Sahitya Tummala <stummala@codeaurora.org>;
>>> Harjani Ritesh <riteshh@codeaurora.org>; Venu Byravarasu
>>> <vbyravarasu@nvidia.com>; Linus Walleij <linus.walleij@linaro.org>; Shawn Lin
>>> <shawn.lin@rock-chips.com>
>>> Subject: [PATCH V4 10/11] mmc: cqhci: support for command queue enabled
>>> host
>>>
>>> From: Venkat Gopalakrishnan <venkatg@codeaurora.org>
>>>
>>> This patch adds CMDQ support for command-queue compatible hosts.
>>>
>>> Command queue is added in eMMC-5.1 specification. This enables the
>>> controller to process upto 32 requests at a time.
>>>
>>> Adrian Hunter contributed renaming to cqhci, recovery, suspend and resume,
>>> cqhci_off, cqhci_wait_for_idle, and external timeout handling.
>>>
>>> Signed-off-by: Asutosh Das <asutoshd@codeaurora.org>
>>> Signed-off-by: Sujit Reddy Thumma <sthumma@codeaurora.org>
>>> Signed-off-by: Konstantin Dorfman <kdorfman@codeaurora.org>
>>> Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
>>> Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
>>> Signed-off-by: Ritesh Harjani <riteshh@codeaurora.org>
>>> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
>>> ---
>>>  drivers/mmc/host/Kconfig  |   13 +
>>>  drivers/mmc/host/Makefile |    1 +
>>>  drivers/mmc/host/cqhci.c  | 1146
>>> +++++++++++++++++++++++++++++++++++++++++++++
>>>  drivers/mmc/host/cqhci.h  |  240 ++++++++++
>>>  4 files changed, 1400 insertions(+)
>>>  create mode 100644 drivers/mmc/host/cqhci.c  create mode 100644
>>> drivers/mmc/host/cqhci.h
>>>
>>> diff --git a/drivers/mmc/host/Kconfig b/drivers/mmc/host/Kconfig index
>>> 2242633550df..3a164a03f2bc 100644
>>> --- a/drivers/mmc/host/Kconfig
>>> +++ b/drivers/mmc/host/Kconfig
>>> @@ -834,6 +834,19 @@ config MMC_SUNXI
>>>  	  This selects support for the SD/MMC Host Controller on
>>>  	  Allwinner sunxi SoCs.
>>>
>>> +config MMC_CQHCI
>>> +	tristate "Command Queue Host Controller Interface support"
>>> +	depends on HAS_DMA
>>> +	help
>>> +	  This selects the Command Queue Host Controller Interface (CQHCI)
>>> +	  support present in host controllers of Qualcomm Technologies, Inc
>>> +	  amongst others.
>>> +	  This controller supports eMMC devices with command queue support.
>>> +
>>> +	  If you have a controller with this interface, say Y or M here.
>>> +
>>> +	  If unsure, say N.
>>> +
>>>  config MMC_TOSHIBA_PCI
>>>  	tristate "Toshiba Type A SD/MMC Card Interface Driver"
>>>  	depends on PCI
>>> diff --git a/drivers/mmc/host/Makefile b/drivers/mmc/host/Makefile index
>>> 8c46766c000c..3ae71e006890 100644
>>> --- a/drivers/mmc/host/Makefile
>>> +++ b/drivers/mmc/host/Makefile
>>> @@ -88,6 +88,7 @@ obj-$(CONFIG_MMC_SDHCI_MSM)		+=
>>> sdhci-msm.o
>>>  obj-$(CONFIG_MMC_SDHCI_ST)		+= sdhci-st.o
>>>  obj-$(CONFIG_MMC_SDHCI_MICROCHIP_PIC32)	+= sdhci-pic32.o
>>>  obj-$(CONFIG_MMC_SDHCI_BRCMSTB)		+= sdhci-brcmstb.o
>>> +obj-$(CONFIG_MMC_CQHCI)			+= cqhci.o
>>>
>>>  ifeq ($(CONFIG_CB710_DEBUG),y)
>>>  	CFLAGS-cb710-mmc	+= -DDEBUG
>>> diff --git a/drivers/mmc/host/cqhci.c b/drivers/mmc/host/cqhci.c new file
>>> mode 100644 index 000000000000..302421a26230
>>> --- /dev/null
>>> +++ b/drivers/mmc/host/cqhci.c
>>> @@ -0,0 +1,1146 @@
>>
>> <SNIP>
>>
>>> +
>>> +static int cqhci_request(struct mmc_host *mmc, struct mmc_request *mrq)
>>> +{
>>> +	int err = 0;
>>> +	u64 data = 0;
>>> +	u64 *task_desc = NULL;
>>> +	int tag = cqhci_tag(mrq);
>>> +	struct cqhci_host *cq_host = mmc->cqe_private;
>>> +	unsigned long flags;
>>> +
>>> +	if (!cq_host->enabled) {
>>> +		pr_err("%s: cqhci: not enabled\n", mmc_hostname(mmc));
>>> +		return -EINVAL;
>>> +	}
>>> +
>>> +	/* First request after resume has to re-enable */
>>> +	if (!cq_host->activated)
>>> +		__cqhci_enable(cq_host);
>>> +
>>> +	if (!mmc->cqe_on) {
>>> +		cqhci_writel(cq_host, 0, CQHCI_CTL);
>>> +		mmc->cqe_on = true;
>>> +		pr_debug("%s: cqhci: CQE on\n", mmc_hostname(mmc));
>>> +		if (cqhci_readl(cq_host, CQHCI_CTL) && CQHCI_HALT) {
>>> +			pr_err("%s: cqhci: CQE failed to exit halt state\n",
>>> +			       mmc_hostname(mmc));
>>> +		}
>>> +		if (cq_host->ops->enable)
>>> +			cq_host->ops->enable(mmc);
>>> +	}
>>> +
>>> +	if (mrq->data) {
>>> +		task_desc = (__le64 __force *)get_desc(cq_host, tag);
>>> +		cqhci_prep_task_desc(mrq, &data, 1);
>>> +		*task_desc = cpu_to_le64(data);
>>> +		err = cqhci_prep_tran_desc(mrq, cq_host, tag);
>>> +		if (err) {
>>> +			pr_err("%s: cqhci: failed to setup tx desc: %d\n",
>>> +			       mmc_hostname(mmc), err);
>>> +			return err;
>>> +		}
>>> +	} else {
>>> +		cqhci_prep_dcmd_desc(mmc, mrq);
>>> +	}
>>> +
>> Hi Adrian,
>>
>> For cqhci data request, I think we need to also config SDHCI_TRNS_BLK_CNT_EN and SDHCI_TRNS_DMA,  just like function sdhci_set_transfer_mode(): set SDHCI_TRNS_BLK_CNT_EN and SDHCI_TRNS_DMA in every request. 
>>
>> Currently, we do not do this for cqhci request, if support Runtime PM, when runtime resume, 
>> sdhci_runtime_resume_host() --> sdhci_init(host, 0) --> sdhci_do_reset(host, SDHCI_RESET_ALL)
>> sdhci_reset() will do software reset for all, this will clear some SDHCI register, including SDHCI_TRNS_BLK_CNT_EN and SDHCI_TRNS_DMA, then when use cqhci request, error happens. I meet cqhci wait timeout error on our i.MX8. After set these two bits, this issue gone.
> 
> CQE anyway needs to write the transfer mode bit-4 so it is a bit strange
> that it doesn't write the other bits too.  Nevertheless anything you need to
> set can be done in the ->enable() callback.  For example see glk_cqe_enable().
> 

How is it going?

Re: [PATCH V4 00/11] mmc: Add Command Queue support

[Adrian Hunter]

On 25/07/17 03:34, Shawn Lin wrote:
> Hi
> 
> On 2017/7/24 18:09, Adrian Hunter wrote:
>> On 24/07/17 12:17, Shawn Lin wrote:
>>> Hi Adrian,
>>>
>>> On 2017/7/21 17:49, Adrian Hunter wrote:
>>>> Hi
>>>>
>>>> Here is V4 of the hardware command queue patches without the software
>>>> command queue patches.
>>>>
>>>
>>> I see this log for the reboot test last weekend. One of my boards
>>> seems to easily trigger this WARN_ONCE.
>>>
>>> [   10.012778] mmc1: cqhci: spurious TCN for tag 31
>>
>> Tag 31 is DCMD.  DCMD can be problematic which is one of the reasons it is
>> optional.  In this case it looks like you are getting a TCN before the DCMD
>> is actually finished.  You should be able to see if that is the case by
> 
> yes, it is that case you refer to.
> 
>> dumping the registers when it happens, although you will need to stop
>> messages going to the console because that will be too slow.
> 
> okay, I will remove DCMD caps for the test and see what we could
> do to make DCMD more robust later.
> 

Are you going to do more review of CQ?

RE: [PATCH V4 10/11] mmc: cqhci: support for command queue enabled host

[Bough Chen]

> -----Original Message-----
> From: linux-mmc-owner@vger.kernel.org [mailto:linux-mmc-
> owner@vger.kernel.org] On Behalf Of Adrian Hunter
> Sent: Monday, July 31, 2017 2:41 PM
> To: Bough Chen <haibo.chen@nxp.com>; Ulf Hansson <ulf.hansson@linaro.org>
> Cc: linux-mmc <linux-mmc@vger.kernel.org>; Alex Lemberg
> <alex.lemberg@sandisk.com>; Mateusz Nowak <mateusz.nowak@intel.com>;
> Yuliy Izrailov <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung
> <jh80.chung@samsung.com>; Dong Aisheng <dongas86@gmail.com>; Das
> Asutosh <asutoshd@codeaurora.org>; Zhangfei Gao
> <zhangfei.gao@gmail.com>; Dorfman Konstantin
> <kdorfman@codeaurora.org>; Sahitya Tummala <stummala@codeaurora.org>;
> Harjani Ritesh <riteshh@codeaurora.org>; Venu Byravarasu
> <vbyravarasu@nvidia.com>; Linus Walleij <linus.walleij@linaro.org>; Shawn Lin
> <shawn.lin@rock-chips.com>
> Subject: Re: [PATCH V4 10/11] mmc: cqhci: support for command queue
> enabled host
> 
> On 24/07/17 13:21, Adrian Hunter wrote:
> > On 24/07/17 11:52, Bough Chen wrote:
> >> [...]
> >>
> >>> -----Original Message-----
> >>> From: Adrian Hunter [mailto:adrian.hunter@intel.com]
> >>> Sent: Friday, July 21, 2017 5:50 PM
> >>> To: Ulf Hansson <ulf.hansson@linaro.org>
> >>> Cc: linux-mmc <linux-mmc@vger.kernel.org>; Bough Chen
> >>> <haibo.chen@nxp.com>; Alex Lemberg <alex.lemberg@sandisk.com>;
> >>> Mateusz Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
> >>> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung
> >>> <jh80.chung@samsung.com>; Dong Aisheng <dongas86@gmail.com>; Das
> >>> Asutosh <asutoshd@codeaurora.org>; Zhangfei Gao
> >>> <zhangfei.gao@gmail.com>; Dorfman Konstantin
> >>> <kdorfman@codeaurora.org>; David Griego <david.griego@linaro.org>;
> >>> Sahitya Tummala <stummala@codeaurora.org>; Harjani Ritesh
> >>> <riteshh@codeaurora.org>; Venu Byravarasu <vbyravarasu@nvidia.com>;
> >>> Linus Walleij <linus.walleij@linaro.org>; Shawn Lin
> >>> <shawn.lin@rock-chips.com>
> >>> Subject: [PATCH V4 10/11] mmc: cqhci: support for command queue
> >>> enabled host
> >>>
> >>> From: Venkat Gopalakrishnan <venkatg@codeaurora.org>
> >>>
> >>> This patch adds CMDQ support for command-queue compatible hosts.
> >>>
> >>> Command queue is added in eMMC-5.1 specification. This enables the
> >>> controller to process upto 32 requests at a time.
> >>>
> >>> Adrian Hunter contributed renaming to cqhci, recovery, suspend and
> >>> resume, cqhci_off, cqhci_wait_for_idle, and external timeout handling.
> >>>
> >>> Signed-off-by: Asutosh Das <asutoshd@codeaurora.org>
> >>> Signed-off-by: Sujit Reddy Thumma <sthumma@codeaurora.org>
> >>> Signed-off-by: Konstantin Dorfman <kdorfman@codeaurora.org>
> >>> Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
> >>> Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
> >>> Signed-off-by: Ritesh Harjani <riteshh@codeaurora.org>
> >>> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
> >>> ---
> >>>  drivers/mmc/host/Kconfig  |   13 +
> >>>  drivers/mmc/host/Makefile |    1 +
> >>>  drivers/mmc/host/cqhci.c  | 1146
> >>> +++++++++++++++++++++++++++++++++++++++++++++
> >>>  drivers/mmc/host/cqhci.h  |  240 ++++++++++
> >>>  4 files changed, 1400 insertions(+)  create mode 100644
> >>> drivers/mmc/host/cqhci.c  create mode 100644
> >>> drivers/mmc/host/cqhci.h
> >>>
> >>> diff --git a/drivers/mmc/host/Kconfig b/drivers/mmc/host/Kconfig
> >>> index 2242633550df..3a164a03f2bc 100644
> >>> --- a/drivers/mmc/host/Kconfig
> >>> +++ b/drivers/mmc/host/Kconfig
> >>> @@ -834,6 +834,19 @@ config MMC_SUNXI
> >>>  	  This selects support for the SD/MMC Host Controller on
> >>>  	  Allwinner sunxi SoCs.
> >>>
> >>> +config MMC_CQHCI
> >>> +	tristate "Command Queue Host Controller Interface support"
> >>> +	depends on HAS_DMA
> >>> +	help
> >>> +	  This selects the Command Queue Host Controller Interface (CQHCI)
> >>> +	  support present in host controllers of Qualcomm Technologies, Inc
> >>> +	  amongst others.
> >>> +	  This controller supports eMMC devices with command queue support.
> >>> +
> >>> +	  If you have a controller with this interface, say Y or M here.
> >>> +
> >>> +	  If unsure, say N.
> >>> +
> >>>  config MMC_TOSHIBA_PCI
> >>>  	tristate "Toshiba Type A SD/MMC Card Interface Driver"
> >>>  	depends on PCI
> >>> diff --git a/drivers/mmc/host/Makefile b/drivers/mmc/host/Makefile
> >>> index
> >>> 8c46766c000c..3ae71e006890 100644
> >>> --- a/drivers/mmc/host/Makefile
> >>> +++ b/drivers/mmc/host/Makefile
> >>> @@ -88,6 +88,7 @@ obj-$(CONFIG_MMC_SDHCI_MSM)		+=
> >>> sdhci-msm.o
> >>>  obj-$(CONFIG_MMC_SDHCI_ST)		+= sdhci-st.o
> >>>  obj-$(CONFIG_MMC_SDHCI_MICROCHIP_PIC32)	+= sdhci-pic32.o
> >>>  obj-$(CONFIG_MMC_SDHCI_BRCMSTB)		+= sdhci-brcmstb.o
> >>> +obj-$(CONFIG_MMC_CQHCI)			+= cqhci.o
> >>>
> >>>  ifeq ($(CONFIG_CB710_DEBUG),y)
> >>>  	CFLAGS-cb710-mmc	+= -DDEBUG
> >>> diff --git a/drivers/mmc/host/cqhci.c b/drivers/mmc/host/cqhci.c new
> >>> file mode 100644 index 000000000000..302421a26230
> >>> --- /dev/null
> >>> +++ b/drivers/mmc/host/cqhci.c
> >>> @@ -0,0 +1,1146 @@
> >>
> >> <SNIP>
> >>
> >>> +
> >>> +static int cqhci_request(struct mmc_host *mmc, struct mmc_request
> >>> +*mrq) {
> >>> +	int err = 0;
> >>> +	u64 data = 0;
> >>> +	u64 *task_desc = NULL;
> >>> +	int tag = cqhci_tag(mrq);
> >>> +	struct cqhci_host *cq_host = mmc->cqe_private;
> >>> +	unsigned long flags;
> >>> +
> >>> +	if (!cq_host->enabled) {
> >>> +		pr_err("%s: cqhci: not enabled\n", mmc_hostname(mmc));
> >>> +		return -EINVAL;
> >>> +	}
> >>> +
> >>> +	/* First request after resume has to re-enable */
> >>> +	if (!cq_host->activated)
> >>> +		__cqhci_enable(cq_host);
> >>> +
> >>> +	if (!mmc->cqe_on) {
> >>> +		cqhci_writel(cq_host, 0, CQHCI_CTL);
> >>> +		mmc->cqe_on = true;
> >>> +		pr_debug("%s: cqhci: CQE on\n", mmc_hostname(mmc));
> >>> +		if (cqhci_readl(cq_host, CQHCI_CTL) && CQHCI_HALT) {
> >>> +			pr_err("%s: cqhci: CQE failed to exit halt state\n",
> >>> +			       mmc_hostname(mmc));
> >>> +		}
> >>> +		if (cq_host->ops->enable)
> >>> +			cq_host->ops->enable(mmc);
> >>> +	}
> >>> +
> >>> +	if (mrq->data) {
> >>> +		task_desc = (__le64 __force *)get_desc(cq_host, tag);
> >>> +		cqhci_prep_task_desc(mrq, &data, 1);
> >>> +		*task_desc = cpu_to_le64(data);
> >>> +		err = cqhci_prep_tran_desc(mrq, cq_host, tag);
> >>> +		if (err) {
> >>> +			pr_err("%s: cqhci: failed to setup tx desc: %d\n",
> >>> +			       mmc_hostname(mmc), err);
> >>> +			return err;
> >>> +		}
> >>> +	} else {
> >>> +		cqhci_prep_dcmd_desc(mmc, mrq);
> >>> +	}
> >>> +
> >> Hi Adrian,
> >>
> >> For cqhci data request, I think we need to also config
> SDHCI_TRNS_BLK_CNT_EN and SDHCI_TRNS_DMA,  just like function
> sdhci_set_transfer_mode(): set SDHCI_TRNS_BLK_CNT_EN and
> SDHCI_TRNS_DMA in every request.
> >>
> >> Currently, we do not do this for cqhci request, if support Runtime
> >> PM, when runtime resume,
> >> sdhci_runtime_resume_host() --> sdhci_init(host, 0) -->
> >> sdhci_do_reset(host, SDHCI_RESET_ALL)
> >> sdhci_reset() will do software reset for all, this will clear some SDHCI register,
> including SDHCI_TRNS_BLK_CNT_EN and SDHCI_TRNS_DMA, then when use
> cqhci request, error happens. I meet cqhci wait timeout error on our i.MX8.
> After set these two bits, this issue gone.
> >
> > CQE anyway needs to write the transfer mode bit-4 so it is a bit
> > strange that it doesn't write the other bits too.  Nevertheless
> > anything you need to set can be done in the ->enable() callback.  For example
> see glk_cqe_enable().
> >
> 
> How is it going?

Yes, the ->enable() callback is a good place to set the i.mx special config, now CMDQ can works well
On our i.MX8.

> --
> To unsubscribe from this list: send the line "unsubscribe linux-mmc" in the
> body of a message to majordomo@vger.kernel.org More majordomo info at
> http://vger.kernel.org/majordomo-info.html

Re: [PATCH V4 10/11] mmc: cqhci: support for command queue enabled host

[Adrian Hunter]

On 31/07/17 10:03, Bough Chen wrote:
>> -----Original Message-----
>> From: linux-mmc-owner@vger.kernel.org [mailto:linux-mmc-
>> owner@vger.kernel.org] On Behalf Of Adrian Hunter
>> Sent: Monday, July 31, 2017 2:41 PM
>> To: Bough Chen <haibo.chen@nxp.com>; Ulf Hansson <ulf.hansson@linaro.org>
>> Cc: linux-mmc <linux-mmc@vger.kernel.org>; Alex Lemberg
>> <alex.lemberg@sandisk.com>; Mateusz Nowak <mateusz.nowak@intel.com>;
>> Yuliy Izrailov <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung
>> <jh80.chung@samsung.com>; Dong Aisheng <dongas86@gmail.com>; Das
>> Asutosh <asutoshd@codeaurora.org>; Zhangfei Gao
>> <zhangfei.gao@gmail.com>; Dorfman Konstantin
>> <kdorfman@codeaurora.org>; Sahitya Tummala <stummala@codeaurora.org>;
>> Harjani Ritesh <riteshh@codeaurora.org>; Venu Byravarasu
>> <vbyravarasu@nvidia.com>; Linus Walleij <linus.walleij@linaro.org>; Shawn Lin
>> <shawn.lin@rock-chips.com>
>> Subject: Re: [PATCH V4 10/11] mmc: cqhci: support for command queue
>> enabled host
>>
>> On 24/07/17 13:21, Adrian Hunter wrote:
>>> On 24/07/17 11:52, Bough Chen wrote:
>>>> [...]
>>>>
>>>>> -----Original Message-----
>>>>> From: Adrian Hunter [mailto:adrian.hunter@intel.com]
>>>>> Sent: Friday, July 21, 2017 5:50 PM
>>>>> To: Ulf Hansson <ulf.hansson@linaro.org>
>>>>> Cc: linux-mmc <linux-mmc@vger.kernel.org>; Bough Chen
>>>>> <haibo.chen@nxp.com>; Alex Lemberg <alex.lemberg@sandisk.com>;
>>>>> Mateusz Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
>>>>> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung
>>>>> <jh80.chung@samsung.com>; Dong Aisheng <dongas86@gmail.com>; Das
>>>>> Asutosh <asutoshd@codeaurora.org>; Zhangfei Gao
>>>>> <zhangfei.gao@gmail.com>; Dorfman Konstantin
>>>>> <kdorfman@codeaurora.org>; David Griego <david.griego@linaro.org>;
>>>>> Sahitya Tummala <stummala@codeaurora.org>; Harjani Ritesh
>>>>> <riteshh@codeaurora.org>; Venu Byravarasu <vbyravarasu@nvidia.com>;
>>>>> Linus Walleij <linus.walleij@linaro.org>; Shawn Lin
>>>>> <shawn.lin@rock-chips.com>
>>>>> Subject: [PATCH V4 10/11] mmc: cqhci: support for command queue
>>>>> enabled host
>>>>>
>>>>> From: Venkat Gopalakrishnan <venkatg@codeaurora.org>
>>>>>
>>>>> This patch adds CMDQ support for command-queue compatible hosts.
>>>>>
>>>>> Command queue is added in eMMC-5.1 specification. This enables the
>>>>> controller to process upto 32 requests at a time.
>>>>>
>>>>> Adrian Hunter contributed renaming to cqhci, recovery, suspend and
>>>>> resume, cqhci_off, cqhci_wait_for_idle, and external timeout handling.
>>>>>
>>>>> Signed-off-by: Asutosh Das <asutoshd@codeaurora.org>
>>>>> Signed-off-by: Sujit Reddy Thumma <sthumma@codeaurora.org>
>>>>> Signed-off-by: Konstantin Dorfman <kdorfman@codeaurora.org>
>>>>> Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
>>>>> Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
>>>>> Signed-off-by: Ritesh Harjani <riteshh@codeaurora.org>
>>>>> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
>>>>> ---
>>>>>  drivers/mmc/host/Kconfig  |   13 +
>>>>>  drivers/mmc/host/Makefile |    1 +
>>>>>  drivers/mmc/host/cqhci.c  | 1146
>>>>> +++++++++++++++++++++++++++++++++++++++++++++
>>>>>  drivers/mmc/host/cqhci.h  |  240 ++++++++++
>>>>>  4 files changed, 1400 insertions(+)  create mode 100644
>>>>> drivers/mmc/host/cqhci.c  create mode 100644
>>>>> drivers/mmc/host/cqhci.h
>>>>>
>>>>> diff --git a/drivers/mmc/host/Kconfig b/drivers/mmc/host/Kconfig
>>>>> index 2242633550df..3a164a03f2bc 100644
>>>>> --- a/drivers/mmc/host/Kconfig
>>>>> +++ b/drivers/mmc/host/Kconfig
>>>>> @@ -834,6 +834,19 @@ config MMC_SUNXI
>>>>>  	  This selects support for the SD/MMC Host Controller on
>>>>>  	  Allwinner sunxi SoCs.
>>>>>
>>>>> +config MMC_CQHCI
>>>>> +	tristate "Command Queue Host Controller Interface support"
>>>>> +	depends on HAS_DMA
>>>>> +	help
>>>>> +	  This selects the Command Queue Host Controller Interface (CQHCI)
>>>>> +	  support present in host controllers of Qualcomm Technologies, Inc
>>>>> +	  amongst others.
>>>>> +	  This controller supports eMMC devices with command queue support.
>>>>> +
>>>>> +	  If you have a controller with this interface, say Y or M here.
>>>>> +
>>>>> +	  If unsure, say N.
>>>>> +
>>>>>  config MMC_TOSHIBA_PCI
>>>>>  	tristate "Toshiba Type A SD/MMC Card Interface Driver"
>>>>>  	depends on PCI
>>>>> diff --git a/drivers/mmc/host/Makefile b/drivers/mmc/host/Makefile
>>>>> index
>>>>> 8c46766c000c..3ae71e006890 100644
>>>>> --- a/drivers/mmc/host/Makefile
>>>>> +++ b/drivers/mmc/host/Makefile
>>>>> @@ -88,6 +88,7 @@ obj-$(CONFIG_MMC_SDHCI_MSM)		+=
>>>>> sdhci-msm.o
>>>>>  obj-$(CONFIG_MMC_SDHCI_ST)		+= sdhci-st.o
>>>>>  obj-$(CONFIG_MMC_SDHCI_MICROCHIP_PIC32)	+= sdhci-pic32.o
>>>>>  obj-$(CONFIG_MMC_SDHCI_BRCMSTB)		+= sdhci-brcmstb.o
>>>>> +obj-$(CONFIG_MMC_CQHCI)			+= cqhci.o
>>>>>
>>>>>  ifeq ($(CONFIG_CB710_DEBUG),y)
>>>>>  	CFLAGS-cb710-mmc	+= -DDEBUG
>>>>> diff --git a/drivers/mmc/host/cqhci.c b/drivers/mmc/host/cqhci.c new
>>>>> file mode 100644 index 000000000000..302421a26230
>>>>> --- /dev/null
>>>>> +++ b/drivers/mmc/host/cqhci.c
>>>>> @@ -0,0 +1,1146 @@
>>>>
>>>> <SNIP>
>>>>
>>>>> +
>>>>> +static int cqhci_request(struct mmc_host *mmc, struct mmc_request
>>>>> +*mrq) {
>>>>> +	int err = 0;
>>>>> +	u64 data = 0;
>>>>> +	u64 *task_desc = NULL;
>>>>> +	int tag = cqhci_tag(mrq);
>>>>> +	struct cqhci_host *cq_host = mmc->cqe_private;
>>>>> +	unsigned long flags;
>>>>> +
>>>>> +	if (!cq_host->enabled) {
>>>>> +		pr_err("%s: cqhci: not enabled\n", mmc_hostname(mmc));
>>>>> +		return -EINVAL;
>>>>> +	}
>>>>> +
>>>>> +	/* First request after resume has to re-enable */
>>>>> +	if (!cq_host->activated)
>>>>> +		__cqhci_enable(cq_host);
>>>>> +
>>>>> +	if (!mmc->cqe_on) {
>>>>> +		cqhci_writel(cq_host, 0, CQHCI_CTL);
>>>>> +		mmc->cqe_on = true;
>>>>> +		pr_debug("%s: cqhci: CQE on\n", mmc_hostname(mmc));
>>>>> +		if (cqhci_readl(cq_host, CQHCI_CTL) && CQHCI_HALT) {
>>>>> +			pr_err("%s: cqhci: CQE failed to exit halt state\n",
>>>>> +			       mmc_hostname(mmc));
>>>>> +		}
>>>>> +		if (cq_host->ops->enable)
>>>>> +			cq_host->ops->enable(mmc);
>>>>> +	}
>>>>> +
>>>>> +	if (mrq->data) {
>>>>> +		task_desc = (__le64 __force *)get_desc(cq_host, tag);
>>>>> +		cqhci_prep_task_desc(mrq, &data, 1);
>>>>> +		*task_desc = cpu_to_le64(data);
>>>>> +		err = cqhci_prep_tran_desc(mrq, cq_host, tag);
>>>>> +		if (err) {
>>>>> +			pr_err("%s: cqhci: failed to setup tx desc: %d\n",
>>>>> +			       mmc_hostname(mmc), err);
>>>>> +			return err;
>>>>> +		}
>>>>> +	} else {
>>>>> +		cqhci_prep_dcmd_desc(mmc, mrq);
>>>>> +	}
>>>>> +
>>>> Hi Adrian,
>>>>
>>>> For cqhci data request, I think we need to also config
>> SDHCI_TRNS_BLK_CNT_EN and SDHCI_TRNS_DMA,  just like function
>> sdhci_set_transfer_mode(): set SDHCI_TRNS_BLK_CNT_EN and
>> SDHCI_TRNS_DMA in every request.
>>>>
>>>> Currently, we do not do this for cqhci request, if support Runtime
>>>> PM, when runtime resume,
>>>> sdhci_runtime_resume_host() --> sdhci_init(host, 0) -->
>>>> sdhci_do_reset(host, SDHCI_RESET_ALL)
>>>> sdhci_reset() will do software reset for all, this will clear some SDHCI register,
>> including SDHCI_TRNS_BLK_CNT_EN and SDHCI_TRNS_DMA, then when use
>> cqhci request, error happens. I meet cqhci wait timeout error on our i.MX8.
>> After set these two bits, this issue gone.
>>>
>>> CQE anyway needs to write the transfer mode bit-4 so it is a bit
>>> strange that it doesn't write the other bits too.  Nevertheless
>>> anything you need to set can be done in the ->enable() callback.  For example
>> see glk_cqe_enable().
>>>
>>
>> How is it going?
> 
> Yes, the ->enable() callback is a good place to set the i.mx special config, now CMDQ can works well
> On our i.MX8.

So your tested-by still applies, also to V4?

Re: [PATCH V4 00/11] mmc: Add Command Queue support

[Shawn Lin]

Hi

On 2017/7/31 14:54, Adrian Hunter wrote:
> On 25/07/17 03:34, Shawn Lin wrote:
>> Hi
>>
>> On 2017/7/24 18:09, Adrian Hunter wrote:
>>> On 24/07/17 12:17, Shawn Lin wrote:
>>>> Hi Adrian,
>>>>
>>>> On 2017/7/21 17:49, Adrian Hunter wrote:
>>>>> Hi
>>>>>
>>>>> Here is V4 of the hardware command queue patches without the software
>>>>> command queue patches.
>>>>>
>>>>
>>>> I see this log for the reboot test last weekend. One of my boards
>>>> seems to easily trigger this WARN_ONCE.
>>>>
>>>> [   10.012778] mmc1: cqhci: spurious TCN for tag 31
>>>
>>> Tag 31 is DCMD.  DCMD can be problematic which is one of the reasons it is
>>> optional.  In this case it looks like you are getting a TCN before the DCMD
>>> is actually finished.  You should be able to see if that is the case by
>>
>> yes, it is that case you refer to.
>>
>>> dumping the registers when it happens, although you will need to stop
>>> messages going to the console because that will be too slow.
>>
>> okay, I will remove DCMD caps for the test and see what we could
>> do to make DCMD more robust later.
>>
> 
> Are you going to do more review of CQ?

Yes, I am reviewing it :)

And I have been testing it for more than one week for various stress
test and all things work fine except for the the spurious TCN for DCMD.
But that seems much likely to be the IP relevant, so I won't going to
add caps for DCMD for my controller right now, until I find the root
cause.



> 
> 
> 
> 

RE: [PATCH V4 10/11] mmc: cqhci: support for command queue enabled host

[Bough Chen]

> -----Original Message-----
> From: Adrian Hunter [mailto:adrian.hunter@intel.com]
> Sent: Monday, July 31, 2017 3:04 PM
> To: Bough Chen <haibo.chen@nxp.com>; Ulf Hansson <ulf.hansson@linaro.org>
> Cc: linux-mmc <linux-mmc@vger.kernel.org>; Alex Lemberg
> <alex.lemberg@sandisk.com>; Mateusz Nowak <mateusz.nowak@intel.com>;
> Yuliy Izrailov <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung
> <jh80.chung@samsung.com>; Dong Aisheng <dongas86@gmail.com>; Das
> Asutosh <asutoshd@codeaurora.org>; Zhangfei Gao
> <zhangfei.gao@gmail.com>; Dorfman Konstantin
> <kdorfman@codeaurora.org>; Sahitya Tummala <stummala@codeaurora.org>;
> Harjani Ritesh <riteshh@codeaurora.org>; Venu Byravarasu
> <vbyravarasu@nvidia.com>; Linus Walleij <linus.walleij@linaro.org>; Shawn Lin
> <shawn.lin@rock-chips.com>
> Subject: Re: [PATCH V4 10/11] mmc: cqhci: support for command queue
> enabled host
> 
> On 31/07/17 10:03, Bough Chen wrote:
> >> -----Original Message-----
> >> From: linux-mmc-owner@vger.kernel.org [mailto:linux-mmc-
> >> owner@vger.kernel.org] On Behalf Of Adrian Hunter
> >> Sent: Monday, July 31, 2017 2:41 PM
> >> To: Bough Chen <haibo.chen@nxp.com>; Ulf Hansson
> >> <ulf.hansson@linaro.org>
> >> Cc: linux-mmc <linux-mmc@vger.kernel.org>; Alex Lemberg
> >> <alex.lemberg@sandisk.com>; Mateusz Nowak
> <mateusz.nowak@intel.com>;
> >> Yuliy Izrailov <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung
> >> <jh80.chung@samsung.com>; Dong Aisheng <dongas86@gmail.com>; Das
> >> Asutosh <asutoshd@codeaurora.org>; Zhangfei Gao
> >> <zhangfei.gao@gmail.com>; Dorfman Konstantin
> >> <kdorfman@codeaurora.org>; Sahitya Tummala
> <stummala@codeaurora.org>;
> >> Harjani Ritesh <riteshh@codeaurora.org>; Venu Byravarasu
> >> <vbyravarasu@nvidia.com>; Linus Walleij <linus.walleij@linaro.org>;
> >> Shawn Lin <shawn.lin@rock-chips.com>
> >> Subject: Re: [PATCH V4 10/11] mmc: cqhci: support for command queue
> >> enabled host
> >>
> >> On 24/07/17 13:21, Adrian Hunter wrote:
> >>> On 24/07/17 11:52, Bough Chen wrote:
> >>>> [...]
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Adrian Hunter [mailto:adrian.hunter@intel.com]
> >>>>> Sent: Friday, July 21, 2017 5:50 PM
> >>>>> To: Ulf Hansson <ulf.hansson@linaro.org>
> >>>>> Cc: linux-mmc <linux-mmc@vger.kernel.org>; Bough Chen
> >>>>> <haibo.chen@nxp.com>; Alex Lemberg <alex.lemberg@sandisk.com>;
> >>>>> Mateusz Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
> >>>>> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung
> >>>>> <jh80.chung@samsung.com>; Dong Aisheng <dongas86@gmail.com>;
> Das
> >>>>> Asutosh <asutoshd@codeaurora.org>; Zhangfei Gao
> >>>>> <zhangfei.gao@gmail.com>; Dorfman Konstantin
> >>>>> <kdorfman@codeaurora.org>; David Griego <david.griego@linaro.org>;
> >>>>> Sahitya Tummala <stummala@codeaurora.org>; Harjani Ritesh
> >>>>> <riteshh@codeaurora.org>; Venu Byravarasu
> >>>>> <vbyravarasu@nvidia.com>; Linus Walleij
> >>>>> <linus.walleij@linaro.org>; Shawn Lin <shawn.lin@rock-chips.com>
> >>>>> Subject: [PATCH V4 10/11] mmc: cqhci: support for command queue
> >>>>> enabled host
> >>>>>
> >>>>> From: Venkat Gopalakrishnan <venkatg@codeaurora.org>
> >>>>>
> >>>>> This patch adds CMDQ support for command-queue compatible hosts.
> >>>>>
> >>>>> Command queue is added in eMMC-5.1 specification. This enables the
> >>>>> controller to process upto 32 requests at a time.
> >>>>>
> >>>>> Adrian Hunter contributed renaming to cqhci, recovery, suspend and
> >>>>> resume, cqhci_off, cqhci_wait_for_idle, and external timeout handling.
> >>>>>
> >>>>> Signed-off-by: Asutosh Das <asutoshd@codeaurora.org>
> >>>>> Signed-off-by: Sujit Reddy Thumma <sthumma@codeaurora.org>
> >>>>> Signed-off-by: Konstantin Dorfman <kdorfman@codeaurora.org>
> >>>>> Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
> >>>>> Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
> >>>>> Signed-off-by: Ritesh Harjani <riteshh@codeaurora.org>
> >>>>> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
> >>>>> ---
> >>>>>  drivers/mmc/host/Kconfig  |   13 +
> >>>>>  drivers/mmc/host/Makefile |    1 +
> >>>>>  drivers/mmc/host/cqhci.c  | 1146
> >>>>> +++++++++++++++++++++++++++++++++++++++++++++
> >>>>>  drivers/mmc/host/cqhci.h  |  240 ++++++++++
> >>>>>  4 files changed, 1400 insertions(+)  create mode 100644
> >>>>> drivers/mmc/host/cqhci.c  create mode 100644
> >>>>> drivers/mmc/host/cqhci.h
> >>>>>
> >>>>> diff --git a/drivers/mmc/host/Kconfig b/drivers/mmc/host/Kconfig
> >>>>> index 2242633550df..3a164a03f2bc 100644
> >>>>> --- a/drivers/mmc/host/Kconfig
> >>>>> +++ b/drivers/mmc/host/Kconfig
> >>>>> @@ -834,6 +834,19 @@ config MMC_SUNXI
> >>>>>  	  This selects support for the SD/MMC Host Controller on
> >>>>>  	  Allwinner sunxi SoCs.
> >>>>>
> >>>>> +config MMC_CQHCI
> >>>>> +	tristate "Command Queue Host Controller Interface support"
> >>>>> +	depends on HAS_DMA
> >>>>> +	help
> >>>>> +	  This selects the Command Queue Host Controller Interface
> (CQHCI)
> >>>>> +	  support present in host controllers of Qualcomm Technologies,
> Inc
> >>>>> +	  amongst others.
> >>>>> +	  This controller supports eMMC devices with command queue
> support.
> >>>>> +
> >>>>> +	  If you have a controller with this interface, say Y or M here.
> >>>>> +
> >>>>> +	  If unsure, say N.
> >>>>> +
> >>>>>  config MMC_TOSHIBA_PCI
> >>>>>  	tristate "Toshiba Type A SD/MMC Card Interface Driver"
> >>>>>  	depends on PCI
> >>>>> diff --git a/drivers/mmc/host/Makefile b/drivers/mmc/host/Makefile
> >>>>> index
> >>>>> 8c46766c000c..3ae71e006890 100644
> >>>>> --- a/drivers/mmc/host/Makefile
> >>>>> +++ b/drivers/mmc/host/Makefile
> >>>>> @@ -88,6 +88,7 @@ obj-$(CONFIG_MMC_SDHCI_MSM)
> 	+=
> >>>>> sdhci-msm.o
> >>>>>  obj-$(CONFIG_MMC_SDHCI_ST)		+= sdhci-st.o
> >>>>>  obj-$(CONFIG_MMC_SDHCI_MICROCHIP_PIC32)	+= sdhci-pic32.o
> >>>>>  obj-$(CONFIG_MMC_SDHCI_BRCMSTB)		+= sdhci-
> brcmstb.o
> >>>>> +obj-$(CONFIG_MMC_CQHCI)			+= cqhci.o
> >>>>>
> >>>>>  ifeq ($(CONFIG_CB710_DEBUG),y)
> >>>>>  	CFLAGS-cb710-mmc	+= -DDEBUG
> >>>>> diff --git a/drivers/mmc/host/cqhci.c b/drivers/mmc/host/cqhci.c
> >>>>> new file mode 100644 index 000000000000..302421a26230
> >>>>> --- /dev/null
> >>>>> +++ b/drivers/mmc/host/cqhci.c
> >>>>> @@ -0,0 +1,1146 @@
> >>>>
> >>>> <SNIP>
> >>>>
> >>>>> +
> >>>>> +static int cqhci_request(struct mmc_host *mmc, struct mmc_request
> >>>>> +*mrq) {
> >>>>> +	int err = 0;
> >>>>> +	u64 data = 0;
> >>>>> +	u64 *task_desc = NULL;
> >>>>> +	int tag = cqhci_tag(mrq);
> >>>>> +	struct cqhci_host *cq_host = mmc->cqe_private;
> >>>>> +	unsigned long flags;
> >>>>> +
> >>>>> +	if (!cq_host->enabled) {
> >>>>> +		pr_err("%s: cqhci: not enabled\n",
> mmc_hostname(mmc));
> >>>>> +		return -EINVAL;
> >>>>> +	}
> >>>>> +
> >>>>> +	/* First request after resume has to re-enable */
> >>>>> +	if (!cq_host->activated)
> >>>>> +		__cqhci_enable(cq_host);
> >>>>> +
> >>>>> +	if (!mmc->cqe_on) {
> >>>>> +		cqhci_writel(cq_host, 0, CQHCI_CTL);
> >>>>> +		mmc->cqe_on = true;
> >>>>> +		pr_debug("%s: cqhci: CQE on\n",
> mmc_hostname(mmc));
> >>>>> +		if (cqhci_readl(cq_host, CQHCI_CTL) && CQHCI_HALT) {
> >>>>> +			pr_err("%s: cqhci: CQE failed to exit halt
> state\n",
> >>>>> +			       mmc_hostname(mmc));
> >>>>> +		}
> >>>>> +		if (cq_host->ops->enable)
> >>>>> +			cq_host->ops->enable(mmc);
> >>>>> +	}
> >>>>> +
> >>>>> +	if (mrq->data) {
> >>>>> +		task_desc = (__le64 __force *)get_desc(cq_host, tag);
> >>>>> +		cqhci_prep_task_desc(mrq, &data, 1);
> >>>>> +		*task_desc = cpu_to_le64(data);
> >>>>> +		err = cqhci_prep_tran_desc(mrq, cq_host, tag);
> >>>>> +		if (err) {
> >>>>> +			pr_err("%s: cqhci: failed to setup tx
> desc: %d\n",
> >>>>> +			       mmc_hostname(mmc), err);
> >>>>> +			return err;
> >>>>> +		}
> >>>>> +	} else {
> >>>>> +		cqhci_prep_dcmd_desc(mmc, mrq);
> >>>>> +	}
> >>>>> +
> >>>> Hi Adrian,
> >>>>
> >>>> For cqhci data request, I think we need to also config
> >> SDHCI_TRNS_BLK_CNT_EN and SDHCI_TRNS_DMA,  just like function
> >> sdhci_set_transfer_mode(): set SDHCI_TRNS_BLK_CNT_EN and
> >> SDHCI_TRNS_DMA in every request.
> >>>>
> >>>> Currently, we do not do this for cqhci request, if support Runtime
> >>>> PM, when runtime resume,
> >>>> sdhci_runtime_resume_host() --> sdhci_init(host, 0) -->
> >>>> sdhci_do_reset(host, SDHCI_RESET_ALL)
> >>>> sdhci_reset() will do software reset for all, this will clear some
> >>>> SDHCI register,
> >> including SDHCI_TRNS_BLK_CNT_EN and SDHCI_TRNS_DMA, then when
> use
> >> cqhci request, error happens. I meet cqhci wait timeout error on our i.MX8.
> >> After set these two bits, this issue gone.
> >>>
> >>> CQE anyway needs to write the transfer mode bit-4 so it is a bit
> >>> strange that it doesn't write the other bits too.  Nevertheless
> >>> anything you need to set can be done in the ->enable() callback.
> >>> For example
> >> see glk_cqe_enable().
> >>>
> >>
> >> How is it going?
> >
> > Yes, the ->enable() callback is a good place to set the i.mx special
> > config, now CMDQ can works well On our i.MX8.
> 
> So your tested-by still applies, also to V4?

Yes, sure.

Re: [PATCH V4 10/11] mmc: cqhci: support for command queue enabled host

[Adrian Hunter]

On 31/07/17 10:18, Bough Chen wrote:
>> -----Original Message-----
>> From: Adrian Hunter [mailto:adrian.hunter@intel.com]
>> Sent: Monday, July 31, 2017 3:04 PM
>> To: Bough Chen <haibo.chen@nxp.com>; Ulf Hansson <ulf.hansson@linaro.org>
>> Cc: linux-mmc <linux-mmc@vger.kernel.org>; Alex Lemberg
>> <alex.lemberg@sandisk.com>; Mateusz Nowak <mateusz.nowak@intel.com>;
>> Yuliy Izrailov <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung
>> <jh80.chung@samsung.com>; Dong Aisheng <dongas86@gmail.com>; Das
>> Asutosh <asutoshd@codeaurora.org>; Zhangfei Gao
>> <zhangfei.gao@gmail.com>; Dorfman Konstantin
>> <kdorfman@codeaurora.org>; Sahitya Tummala <stummala@codeaurora.org>;
>> Harjani Ritesh <riteshh@codeaurora.org>; Venu Byravarasu
>> <vbyravarasu@nvidia.com>; Linus Walleij <linus.walleij@linaro.org>; Shawn Lin
>> <shawn.lin@rock-chips.com>
>> Subject: Re: [PATCH V4 10/11] mmc: cqhci: support for command queue
>> enabled host
>>
>> On 31/07/17 10:03, Bough Chen wrote:
>>>> -----Original Message-----
>>>> From: linux-mmc-owner@vger.kernel.org [mailto:linux-mmc-
>>>> owner@vger.kernel.org] On Behalf Of Adrian Hunter
>>>> Sent: Monday, July 31, 2017 2:41 PM
>>>> To: Bough Chen <haibo.chen@nxp.com>; Ulf Hansson
>>>> <ulf.hansson@linaro.org>
>>>> Cc: linux-mmc <linux-mmc@vger.kernel.org>; Alex Lemberg
>>>> <alex.lemberg@sandisk.com>; Mateusz Nowak
>> <mateusz.nowak@intel.com>;
>>>> Yuliy Izrailov <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung
>>>> <jh80.chung@samsung.com>; Dong Aisheng <dongas86@gmail.com>; Das
>>>> Asutosh <asutoshd@codeaurora.org>; Zhangfei Gao
>>>> <zhangfei.gao@gmail.com>; Dorfman Konstantin
>>>> <kdorfman@codeaurora.org>; Sahitya Tummala
>> <stummala@codeaurora.org>;
>>>> Harjani Ritesh <riteshh@codeaurora.org>; Venu Byravarasu
>>>> <vbyravarasu@nvidia.com>; Linus Walleij <linus.walleij@linaro.org>;
>>>> Shawn Lin <shawn.lin@rock-chips.com>
>>>> Subject: Re: [PATCH V4 10/11] mmc: cqhci: support for command queue
>>>> enabled host
>>>>
>>>>
>>>> How is it going?
>>>
>>> Yes, the ->enable() callback is a good place to set the i.mx special
>>> config, now CMDQ can works well On our i.MX8.
>>
>> So your tested-by still applies, also to V4?
> 
> Yes, sure.
> 

Thank you!

Re: [PATCH V4 09/11] mmc: block: Add CQE support

[Shawn Lin]

Hi Adrian,

On 2017/7/21 17:49, Adrian Hunter wrote:
> Add CQE support to the block driver, including:
> 	- optionally using DCMD for flush requests
> 	- manually issuing discard requests
> 	- issuing read / write requests to the CQE
> 	- supporting block-layer timeouts
> 	- handling recovery
> 	- supporting re-tuning
> 
> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
> ---
>   drivers/mmc/core/block.c | 195 ++++++++++++++++++++++++++++++++-
>   drivers/mmc/core/block.h |   7 ++
>   drivers/mmc/core/queue.c | 273 ++++++++++++++++++++++++++++++++++++++++++++++-
>   drivers/mmc/core/queue.h |  42 +++++++-
>   4 files changed, 510 insertions(+), 7 deletions(-)
> 
> diff --git a/drivers/mmc/core/block.c b/drivers/mmc/core/block.c
> index 915290c74363..2d25115637b7 100644
> --- a/drivers/mmc/core/block.c
> +++ b/drivers/mmc/core/block.c
> @@ -109,6 +109,7 @@ struct mmc_blk_data {
>   #define MMC_BLK_WRITE		BIT(1)
>   #define MMC_BLK_DISCARD		BIT(2)
>   #define MMC_BLK_SECDISCARD	BIT(3)
> +#define MMC_BLK_CQE_RECOVERY	BIT(4)
>   
>   	/*
>   	 * Only set in main mmc_blk_data associated
> @@ -1612,6 +1613,198 @@ static void mmc_blk_data_prep(struct mmc_queue *mq, struct mmc_queue_req *mqrq,
>   		*do_data_tag_p = do_data_tag;
>   }
>   
> +#define MMC_CQE_RETRIES 2
> +
> +void mmc_blk_cqe_complete_rq(struct request *req)
> +{
> +	struct mmc_queue_req *mqrq = req_to_mmc_queue_req(req);
> +	struct mmc_request *mrq = &mqrq->brq.mrq;
> +	struct request_queue *q = req->q;
> +	struct mmc_queue *mq = q->queuedata;
> +	struct mmc_host *host = mq->card->host;
> +	unsigned long flags;
> +	bool put_card;
> +	int err;
> +
> +	mmc_cqe_post_req(host, mrq);
> +
> +	spin_lock_irqsave(q->queue_lock, flags);
> +
> +	mq->cqe_in_flight[mmc_cqe_issue_type(host, req)] -= 1;
> +
> +	put_card = mmc_cqe_tot_in_flight(mq) == 0;
> +
> +	if (mrq->cmd && mrq->cmd->error)
> +		err = mrq->cmd->error;
> +	else if (mrq->data && mrq->data->error)
> +		err = mrq->data->error;
> +	else
> +		err = 0;
> +
> +	if (err) {
> +		if (mqrq->retries++ < MMC_CQE_RETRIES)
> +			blk_requeue_request(q, req);
> +		else
> +			__blk_end_request_all(req, BLK_STS_IOERR);
> +	} else if (mrq->data) {
> +		if (__blk_end_request(req, BLK_STS_OK, mrq->data->bytes_xfered))
> +			blk_requeue_request(q, req);
> +	} else {
> +		__blk_end_request_all(req, BLK_STS_OK);
> +	}
> +
> +	mmc_cqe_kick_queue(mq);
> +
> +	spin_unlock_irqrestore(q->queue_lock, flags);
> +
> +	if (put_card)
> +		mmc_put_card(mq->card);
> +}
> +
> +void mmc_blk_cqe_recovery(struct mmc_queue *mq)
> +{
> +	struct mmc_card *card = mq->card;
> +	struct mmc_host *host = card->host;
> +	int err;
> +
> +	mmc_get_card(card);
> +
> +	pr_debug("%s: CQE recovery start\n", mmc_hostname(host));
> +
> +	mq->cqe_in_recovery = true;
> +
> +	err = mmc_cqe_recovery(host);
> +	if (err)
> +		mmc_blk_reset(mq->blkdata, host, MMC_BLK_CQE_RECOVERY);
> +	else
> +		mmc_blk_reset_success(mq->blkdata, MMC_BLK_CQE_RECOVERY);
> +
> +	mq->cqe_in_recovery = false;
> +
> +	pr_debug("%s: CQE recovery done\n", mmc_hostname(host));
> +
> +	mmc_put_card(card);
> +}
> +
> +static void mmc_blk_cqe_req_done(struct mmc_request *mrq)
> +{
> +	struct mmc_queue_req *mqrq = container_of(mrq, struct mmc_queue_req,
> +						  brq.mrq);
> +	struct request *req = mmc_queue_req_to_req(mqrq);
> +	struct request_queue *q = req->q;
> +	struct mmc_queue *mq = q->queuedata;
> +
> +	/*
> +	 * Block layer timeouts race with completions which means the normal
> +	 * completion path cannot be used during recovery.
> +	 */
> +	if (mq->cqe_in_recovery)
> +		mmc_blk_cqe_complete_rq(req);
> +	else
> +		blk_complete_request(req);
> +}
> +
> +static int mmc_blk_cqe_start_req(struct mmc_host *host, struct mmc_request *mrq)
> +{
> +	mrq->done = mmc_blk_cqe_req_done;
> +	return mmc_cqe_start_req(host, mrq);
> +}
> +
> +static struct mmc_request *mmc_blk_cqe_prep_dcmd(struct mmc_queue_req *mqrq,
> +						 struct request *req)
> +{
> +	struct mmc_blk_request *brq = &mqrq->brq;
> +
> +	memset(brq, 0, sizeof(*brq));
> +
> +	brq->mrq.cmd = &brq->cmd;
> +	brq->mrq.tag = req->tag;
> +
> +	return &brq->mrq;
> +}
> +
> +static int mmc_blk_cqe_issue_flush(struct mmc_queue *mq, struct request *req)
> +{
> +	struct mmc_queue_req *mqrq = req_to_mmc_queue_req(req);
> +	struct mmc_request *mrq = mmc_blk_cqe_prep_dcmd(mqrq, req);
> +
> +	mrq->cmd->opcode = MMC_SWITCH;
> +	mrq->cmd->arg = (MMC_SWITCH_MODE_WRITE_BYTE << 24) |
> +			(EXT_CSD_FLUSH_CACHE << 16) |
> +			(1 << 8) |
> +			EXT_CSD_CMD_SET_NORMAL;
> +	mrq->cmd->flags = MMC_CMD_AC | MMC_RSP_R1B;
> +
> +	return mmc_blk_cqe_start_req(mq->card->host, mrq);
> +}
> +
> +static int mmc_blk_cqe_issue_rw_rq(struct mmc_queue *mq, struct request *req)
> +{
> +	struct mmc_queue_req *mqrq = req_to_mmc_queue_req(req);
> +
> +	mmc_blk_data_prep(mq, mqrq, 0, NULL, NULL);
> +
> +	return mmc_blk_cqe_start_req(mq->card->host, &mqrq->brq.mrq);
> +}
> +
> +enum mmc_issued mmc_blk_cqe_issue_rq(struct mmc_queue *mq, struct request *req)
> +{
> +	struct mmc_blk_data *md = mq->blkdata;
> +	struct mmc_card *card = md->queue.card;
> +	struct mmc_host *host = card->host;
> +	int ret;
> +
> +	ret = mmc_blk_part_switch(card, md);
> +	if (ret)
> +		return MMC_REQ_FAILED_TO_START;
> +
> +	switch (mmc_cqe_issue_type(host, req)) {
> +	case MMC_ISSUE_SYNC:
> +		ret = host->cqe_ops->cqe_wait_for_idle(host);
> +		if (ret)
> +			return MMC_REQ_BUSY;
> +		switch (req_op(req)) {
> +		case REQ_OP_DRV_IN:
> +		case REQ_OP_DRV_OUT:
> +			mmc_blk_issue_drv_op(mq, req);
> +			break;
> +		case REQ_OP_DISCARD:
> +			mmc_blk_issue_discard_rq(mq, req);
> +			break;
> +		case REQ_OP_SECURE_ERASE:
> +			mmc_blk_issue_secdiscard_rq(mq, req);
> +			break;
> +		case REQ_OP_FLUSH:
> +			mmc_blk_issue_flush(mq, req);
> +			break;
> +		default:
> +			WARN_ON_ONCE(1);
> +			return MMC_REQ_FAILED_TO_START;
> +		}
> +		return MMC_REQ_FINISHED;
> +	case MMC_ISSUE_DCMD:
> +	case MMC_ISSUE_ASYNC:
> +		switch (req_op(req)) {
> +		case REQ_OP_FLUSH:
> +			ret = mmc_blk_cqe_issue_flush(mq, req);
> +			break;
> +		case REQ_OP_READ:
> +		case REQ_OP_WRITE:
> +			ret = mmc_blk_cqe_issue_rw_rq(mq, req);
> +			break;
> +		default:
> +			WARN_ON_ONCE(1);
> +			ret = -EINVAL;
> +		}
> +		if (!ret)
> +			return MMC_REQ_STARTED;
> +		return ret == -EBUSY ? MMC_REQ_BUSY : MMC_REQ_FAILED_TO_START;
> +	default:
> +		WARN_ON_ONCE(1);
> +		return MMC_REQ_FAILED_TO_START;
> +	}
> +}
> +
>   static void mmc_blk_rw_rq_prep(struct mmc_queue_req *mqrq,
>   			       struct mmc_card *card,
>   			       int disable_multi,
> @@ -2035,7 +2228,7 @@ static struct mmc_blk_data *mmc_blk_alloc_req(struct mmc_card *card,
>   	INIT_LIST_HEAD(&md->part);
>   	md->usage = 1;
>   
> -	ret = mmc_init_queue(&md->queue, card, &md->lock, subname);
> +	ret = mmc_init_queue(&md->queue, card, &md->lock, subname, area_type);
>   	if (ret)
>   		goto err_putdisk;
>   
> diff --git a/drivers/mmc/core/block.h b/drivers/mmc/core/block.h
> index 860ca7c8df86..d7b3d7008b00 100644
> --- a/drivers/mmc/core/block.h
> +++ b/drivers/mmc/core/block.h
> @@ -6,4 +6,11 @@
>   
>   void mmc_blk_issue_rq(struct mmc_queue *mq, struct request *req);
>   
> +enum mmc_issued;
> +
> +enum mmc_issued mmc_blk_cqe_issue_rq(struct mmc_queue *mq,
> +				     struct request *req);
> +void mmc_blk_cqe_complete_rq(struct request *rq);
> +void mmc_blk_cqe_recovery(struct mmc_queue *mq);
> +
>   #endif
> diff --git a/drivers/mmc/core/queue.c b/drivers/mmc/core/queue.c
> index affa7370ba82..0cb7b0e8ee58 100644
> --- a/drivers/mmc/core/queue.c
> +++ b/drivers/mmc/core/queue.c
> @@ -36,10 +36,254 @@ static int mmc_prep_request(struct request_queue *q, struct request *req)
>   		return BLKPREP_KILL;
>   
>   	req->rq_flags |= RQF_DONTPREP;
> +	req_to_mmc_queue_req(req)->retries = 0;
>   
>   	return BLKPREP_OK;
>   }
>   
> +static void mmc_cqe_request_fn(struct request_queue *q)
> +{
> +	struct mmc_queue *mq = q->queuedata;
> +	struct request *req;
> +
> +	if (!mq) {
> +		while ((req = blk_fetch_request(q)) != NULL) {
> +			req->rq_flags |= RQF_QUIET;
> +			__blk_end_request_all(req, BLK_STS_IOERR);
> +		}
> +		return;
> +	}
> +
> +	if (mq->asleep && !mq->cqe_busy)
> +		wake_up_process(mq->thread);
> +}
> +
> +static inline bool mmc_cqe_dcmd_busy(struct mmc_queue *mq)
> +{
> +	/* Allow only 1 DCMD at a time */
> +	return mq->cqe_in_flight[MMC_ISSUE_DCMD];
> +}
> +
> +void mmc_cqe_kick_queue(struct mmc_queue *mq)
> +{
> +	if ((mq->cqe_busy & MMC_CQE_DCMD_BUSY) && !mmc_cqe_dcmd_busy(mq))
> +		mq->cqe_busy &= ~MMC_CQE_DCMD_BUSY;
> +
> +	mq->cqe_busy &= ~MMC_CQE_QUEUE_FULL;
> +
> +	if (mq->asleep && !mq->cqe_busy)
> +		__blk_run_queue(mq->queue);
> +}
> +
> +static inline bool mmc_cqe_can_dcmd(struct mmc_host *host)
> +{
> +	return host->caps2 & MMC_CAP2_CQE_DCMD;
> +}
> +
> +enum mmc_issue_type mmc_cqe_issue_type(struct mmc_host *host,
> +				       struct request *req)
> +{
> +	switch (req_op(req)) {
> +	case REQ_OP_DRV_IN:
> +	case REQ_OP_DRV_OUT:
> +	case REQ_OP_DISCARD:
> +	case REQ_OP_SECURE_ERASE:
> +		return MMC_ISSUE_SYNC;
> +	case REQ_OP_FLUSH:
> +		return mmc_cqe_can_dcmd(host) ? MMC_ISSUE_DCMD : MMC_ISSUE_SYNC;
> +	default:
> +		return MMC_ISSUE_ASYNC;
> +	}
> +}
> +
> +static void __mmc_cqe_recovery_notifier(struct mmc_queue *mq)
> +{
> +	if (!mq->cqe_recovery_needed) {
> +		mq->cqe_recovery_needed = true;
> +		wake_up_process(mq->thread);
> +	}
> +}
> +
> +static void mmc_cqe_recovery_notifier(struct mmc_host *host,
> +				      struct mmc_request *mrq)
> +{
> +	struct mmc_queue_req *mqrq = container_of(mrq, struct mmc_queue_req,
> +						  brq.mrq);
> +	struct request *req = mmc_queue_req_to_req(mqrq);
> +	struct request_queue *q = req->q;
> +	struct mmc_queue *mq = q->queuedata;
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(q->queue_lock, flags);
> +	__mmc_cqe_recovery_notifier(mq);
> +	spin_unlock_irqrestore(q->queue_lock, flags);
> +}
> +
> +static int mmc_cqe_thread(void *d)
> +{
> +	struct mmc_queue *mq = d;
> +	struct request_queue *q = mq->queue;
> +	struct mmc_card *card = mq->card;
> +	struct mmc_host *host = card->host;
> +	unsigned long flags;
> +	int get_put = 0;
> +
> +	current->flags |= PF_MEMALLOC;
> +
> +	down(&mq->thread_sem);
> +	spin_lock_irqsave(q->queue_lock, flags);
> +	while (1) {
> +		struct request *req = NULL;
> +		enum mmc_issue_type issue_type;
> +		bool retune_ok = false;
> +
> +		if (mq->cqe_recovery_needed) {
> +			spin_unlock_irqrestore(q->queue_lock, flags);
> +			mmc_blk_cqe_recovery(mq);
> +			spin_lock_irqsave(q->queue_lock, flags);
> +			mq->cqe_recovery_needed = false;
> +		}
> +
> +		set_current_state(TASK_INTERRUPTIBLE);
> +
> +		if (!kthread_should_stop())
> +			req = blk_peek_request(q);
> +
> +		if (req) {
> +			issue_type = mmc_cqe_issue_type(host, req);
> +			switch (issue_type) {
> +			case MMC_ISSUE_DCMD:
> +				if (mmc_cqe_dcmd_busy(mq)) {
> +					mq->cqe_busy |= MMC_CQE_DCMD_BUSY;
> +					req = NULL;
> +					break;
> +				}
> +				/* Fall through */
> +			case MMC_ISSUE_ASYNC:
> +				if (blk_queue_start_tag(q, req)) {
> +					mq->cqe_busy |= MMC_CQE_QUEUE_FULL;
> +					req = NULL;
> +				}
> +				break;
> +			default:
> +				/*
> +				 * Timeouts are handled by mmc core, so set a
> +				 * large value to avoid races.
> +				 */
> +				req->timeout = 600 * HZ;
> +				blk_start_request(req);
> +				break;
> +			}
> +			if (req) {
> +				mq->cqe_in_flight[issue_type] += 1;
> +				if (mmc_cqe_tot_in_flight(mq) == 1)
> +					get_put += 1;
> +				if (mmc_cqe_qcnt(mq) == 1)
> +					retune_ok = true;
> +			}
> +		}
> +

Just a thought that mmc_cq_thread is a little heavy for manage in-flight
request, so could we kick the check back to blk layer like this in the
prepare and unprepare hook?


--- a/drivers/mmc/core/queue.c
+++ b/drivers/mmc/core/queue.c
@@ -25,22 +25,59 @@

  #define MMC_QUEUE_BOUNCESZ	65536

+static inline bool mmc_cqe_dcmd_busy(struct mmc_queue *mq);
+static inline bool mmc_cqe_can_dcmd(struct mmc_host *host);
+enum mmc_issue_type mmc_cqe_issue_type(struct mmc_host *host,
+				       struct request *req);
+
  /*
   * Prepare a MMC request. This just filters out odd stuff.
   */
  static int mmc_prep_request(struct request_queue *q, struct request *req)
  {
  	struct mmc_queue *mq = q->queuedata;
+	enum mmc_issue_type issue_type;

  	if (mq && (mmc_card_removed(mq->card) || mmc_access_rpmb(mq)))
  		return BLKPREP_KILL;

+	if (mq->card->host->cqe_enabled) {
+		issue_type = mmc_cqe_issue_type(mq->card->host, req);
+		if ((issue_type == MMC_ISSUE_DCMD && mmc_cqe_dcmd_busy(mq))) {
+			mq->cqe_busy |= MMC_CQE_DCMD_BUSY;
+			return BLKPREP_DEFER;
+		} else if (issue_type == MMC_ISSUE_ASYNC &&
+				   mq->cqe_in_flight[issue_type] >=
+				   mq->queue->queue_tags->max_depth) {
+			mq->cqe_busy |= MMC_CQE_QUEUE_FULL;
+			return BLKPREP_DEFER;
+		}
+	}
+
  	req->rq_flags |= RQF_DONTPREP;
  	req_to_mmc_queue_req(req)->retries = 0;

  	return BLKPREP_OK;
  }

+static void mmc_unprep_request(struct request_queue *q, struct request 
*req)
+{
+	struct mmc_queue *mq = q->queuedata;
+	enum mmc_issue_type issue_type;
+
+	if (!mq->card->host->cqe_enabled)
+		return;
+
+	issue_type = mmc_cqe_issue_type(mq->card->host, req);
+	if (issue_type == MMC_ISSUE_DCMD &&
+		(mq->cqe_busy & MMC_CQE_DCMD_BUSY) &&
+		!mmc_cqe_dcmd_busy(mq))
+		mq->cqe_busy &= ~MMC_CQE_DCMD_BUSY;
+	else if (issue_type == MMC_ISSUE_ASYNC)
+		mq->cqe_busy &= ~MMC_CQE_QUEUE_FULL;
+}
+
+
  static void mmc_cqe_request_fn(struct request_queue *q)
  {
  	struct mmc_queue *mq = q->queuedata;
@@ -66,11 +103,6 @@ static inline bool mmc_cqe_dcmd_busy(struct 
mmc_queue *mq)

  void mmc_cqe_kick_queue(struct mmc_queue *mq)
  {
-	if ((mq->cqe_busy & MMC_CQE_DCMD_BUSY) && !mmc_cqe_dcmd_busy(mq))
-		mq->cqe_busy &= ~MMC_CQE_DCMD_BUSY;
-
-	mq->cqe_busy &= ~MMC_CQE_QUEUE_FULL;
-
  	if (mq->asleep && !mq->cqe_busy)
  		__blk_run_queue(mq->queue);
  }
@@ -151,21 +183,9 @@ static int mmc_cqe_thread(void *d)

  		if (req) {
  			issue_type = mmc_cqe_issue_type(host, req);
-			switch (issue_type) {
-			case MMC_ISSUE_DCMD:
-				if (mmc_cqe_dcmd_busy(mq)) {
-					mq->cqe_busy |= MMC_CQE_DCMD_BUSY;
-					req = NULL;
-					break;
-				}
-				/* Fall through */
-			case MMC_ISSUE_ASYNC:
-				if (blk_queue_start_tag(q, req)) {
-					mq->cqe_busy |= MMC_CQE_QUEUE_FULL;
-					req = NULL;
-				}
-				break;
-			default:
+			if (issue_type == MMC_ISSUE_DCMD || issue_type == MMC_ISSUE_ASYNC) {
+				blk_queue_start_tag(q, req);
+			} else {
  				/*
  				 * Timeouts are handled by mmc core, so set a
  				 * large value to avoid races.
@@ -174,13 +194,12 @@ static int mmc_cqe_thread(void *d)
  				blk_start_request(req);
  				break;
  			}
-			if (req) {
-				mq->cqe_in_flight[issue_type] += 1;
-				if (mmc_cqe_tot_in_flight(mq) == 1)
-					get_put += 1;
-				if (mmc_cqe_qcnt(mq) == 1)
-					retune_ok = true;
-			}
+
+			mq->cqe_in_flight[issue_type] += 1;
+			if (mmc_cqe_tot_in_flight(mq) == 1)
+				get_put += 1;
+			if (mmc_cqe_qcnt(mq) == 1)
+				retune_ok = true;
  		}

  		mq->asleep = !req;
@@ -523,6 +542,7 @@ int mmc_init_queue(struct mmc_queue *mq, struct 
mmc_card *card,
  	}

  	blk_queue_prep_rq(mq->queue, mmc_prep_request);
+	blk_queue_unprep_rq(mq->queue, mmc_unprep_request);
  	queue_flag_set_unlocked(QUEUE_FLAG_NONROT, mq->queue);
  	queue_flag_clear_unlocked(QUEUE_FLAG_ADD_RANDOM, mq->queue);
  	if (mmc_can_erase(card))



> +		mq->asleep = !req;
> +
> +		spin_unlock_irqrestore(q->queue_lock, flags);
> +
> +		if (req) {
> +			enum mmc_issued issued;
> +
> +			set_current_state(TASK_RUNNING);
> +
> +			if (get_put) {
> +				get_put = 0;
> +				mmc_get_card(card);
> +			}
> +
> +			if (host->need_retune && retune_ok &&
> +			    !host->hold_retune)
> +				host->retune_now = true;
> +			else
> +				host->retune_now = false;
> +
> +			issued = mmc_blk_cqe_issue_rq(mq, req);
> +
> +			cond_resched();
> +
> +			spin_lock_irqsave(q->queue_lock, flags);
> +
> +			switch (issued) {
> +			case MMC_REQ_STARTED:
> +				break;
> +			case MMC_REQ_BUSY:
> +				blk_requeue_request(q, req);
> +				goto finished;
> +			case MMC_REQ_FAILED_TO_START:
> +				__blk_end_request_all(req, BLK_STS_IOERR);
> +				/* Fall through */
> +			case MMC_REQ_FINISHED:
> +finished:
> +				mq->cqe_in_flight[issue_type] -= 1;
> +				if (mmc_cqe_tot_in_flight(mq) == 0)
> +					get_put = -1;
> +			}
> +		} else {
> +			if (get_put < 0) {
> +				get_put = 0;
> +				mmc_put_card(card);
> +			}
> +			/*
> +			 * Do not stop with requests in flight in case recovery
> +			 * is needed.
> +			 */
> +			if (kthread_should_stop() &&
> +			    !mmc_cqe_tot_in_flight(mq)) {
> +				set_current_state(TASK_RUNNING);
> +				break;
> +			}
> +			up(&mq->thread_sem);
> +			schedule();
> +			down(&mq->thread_sem);
> +			spin_lock_irqsave(q->queue_lock, flags);
> +		}
> +	} /* loop */
> +	up(&mq->thread_sem);
> +
> +	return 0;
> +}
> +
> +static enum blk_eh_timer_return __mmc_cqe_timed_out(struct request *req)
> +{
> +	struct mmc_queue_req *mqrq = req_to_mmc_queue_req(req);
> +	struct mmc_request *mrq = &mqrq->brq.mrq;
> +	struct mmc_queue *mq = req->q->queuedata;
> +	struct mmc_host *host = mq->card->host;
> +	enum mmc_issue_type issue_type = mmc_cqe_issue_type(host, req);
> +	bool recovery_needed = false;
> +
> +	switch (issue_type) {
> +	case MMC_ISSUE_ASYNC:
> +	case MMC_ISSUE_DCMD:
> +		if (host->cqe_ops->cqe_timeout(host, mrq, &recovery_needed)) {
> +			if (recovery_needed)
> +				__mmc_cqe_recovery_notifier(mq);
> +			return BLK_EH_RESET_TIMER;
> +		}
> +		/* No timeout */
> +		return BLK_EH_HANDLED;
> +	default:
> +		/* Timeout is handled by mmc core */
> +		return BLK_EH_RESET_TIMER;
> +	}
> +}
> +
> +static enum blk_eh_timer_return mmc_cqe_timed_out(struct request *req)
> +{
> +	struct mmc_queue *mq = req->q->queuedata;
> +
> +	if (mq->cqe_recovery_needed)
> +		return BLK_EH_RESET_TIMER;
> +
> +	return __mmc_cqe_timed_out(req);
> +}
> +
>   static int mmc_queue_thread(void *d)
>   {
>   	struct mmc_queue *mq = d;
> @@ -233,11 +477,12 @@ static void mmc_exit_request(struct request_queue *q, struct request *req)
>    * Initialise a MMC card request queue.
>    */
>   int mmc_init_queue(struct mmc_queue *mq, struct mmc_card *card,
> -		   spinlock_t *lock, const char *subname)
> +		   spinlock_t *lock, const char *subname, int area_type)
>   {
>   	struct mmc_host *host = card->host;
>   	u64 limit = BLK_BOUNCE_HIGH;
>   	int ret = -ENOMEM;
> +	bool use_cqe = host->cqe_enabled && area_type != MMC_BLK_DATA_AREA_RPMB;
>   
>   	if (mmc_dev(host)->dma_mask && *mmc_dev(host)->dma_mask)
>   		limit = (u64)dma_max_pfn(mmc_dev(host)) << PAGE_SHIFT;
> @@ -247,7 +492,7 @@ int mmc_init_queue(struct mmc_queue *mq, struct mmc_card *card,
>   	if (!mq->queue)
>   		return -ENOMEM;
>   	mq->queue->queue_lock = lock;
> -	mq->queue->request_fn = mmc_request_fn;
> +	mq->queue->request_fn = use_cqe ? mmc_cqe_request_fn : mmc_request_fn;
>   	mq->queue->init_rq_fn = mmc_init_request;
>   	mq->queue->exit_rq_fn = mmc_exit_request;
>   	mq->queue->cmd_size = sizeof(struct mmc_queue_req);
> @@ -259,6 +504,24 @@ int mmc_init_queue(struct mmc_queue *mq, struct mmc_card *card,
>   		return ret;
>   	}
>   
> +	if (use_cqe) {
> +		int q_depth = card->ext_csd.cmdq_depth;
> +
> +		if (q_depth > host->cqe_qdepth)
> +			q_depth = host->cqe_qdepth;
> +
> +		ret = blk_queue_init_tags(mq->queue, q_depth, NULL,
> +					  BLK_TAG_ALLOC_FIFO);
> +		if (ret)
> +			goto cleanup_queue;
> +
> +		blk_queue_softirq_done(mq->queue, mmc_blk_cqe_complete_rq);
> +		blk_queue_rq_timed_out(mq->queue, mmc_cqe_timed_out);
> +		blk_queue_rq_timeout(mq->queue, 60 * HZ);
> +
> +		host->cqe_recovery_notifier = mmc_cqe_recovery_notifier;
> +	}
> +
>   	blk_queue_prep_rq(mq->queue, mmc_prep_request);
>   	queue_flag_set_unlocked(QUEUE_FLAG_NONROT, mq->queue);
>   	queue_flag_clear_unlocked(QUEUE_FLAG_ADD_RANDOM, mq->queue);
> @@ -280,9 +543,9 @@ int mmc_init_queue(struct mmc_queue *mq, struct mmc_card *card,
>   
>   	sema_init(&mq->thread_sem, 1);
>   
> -	mq->thread = kthread_run(mmc_queue_thread, mq, "mmcqd/%d%s",
> -		host->index, subname ? subname : "");
> -
> +	mq->thread = kthread_run(use_cqe ? mmc_cqe_thread : mmc_queue_thread,
> +				 mq, "mmcqd/%d%s", host->index,
> +				 subname ? subname : "");
>   	if (IS_ERR(mq->thread)) {
>   		ret = PTR_ERR(mq->thread);
>   		goto cleanup_queue;
> diff --git a/drivers/mmc/core/queue.h b/drivers/mmc/core/queue.h
> index 361b46408e0f..8e9273d977c0 100644
> --- a/drivers/mmc/core/queue.h
> +++ b/drivers/mmc/core/queue.h
> @@ -7,6 +7,20 @@
>   #include <linux/mmc/core.h>
>   #include <linux/mmc/host.h>
>   
> +enum mmc_issued {
> +	MMC_REQ_STARTED,
> +	MMC_REQ_BUSY,
> +	MMC_REQ_FAILED_TO_START,
> +	MMC_REQ_FINISHED,
> +};
> +
> +enum mmc_issue_type {
> +	MMC_ISSUE_SYNC,
> +	MMC_ISSUE_DCMD,
> +	MMC_ISSUE_ASYNC,
> +	MMC_ISSUE_MAX,
> +};
> +
>   static inline struct mmc_queue_req *req_to_mmc_queue_req(struct request *rq)
>   {
>   	return blk_mq_rq_to_pdu(rq);
> @@ -53,6 +67,7 @@ struct mmc_queue_req {
>   	int			drv_op_result;
>   	struct mmc_blk_ioc_data	**idata;
>   	unsigned int		ioc_count;
> +	int			retries;
>   };
>   
>   struct mmc_queue {
> @@ -70,10 +85,17 @@ struct mmc_queue {
>   	 * associated mmc_queue_req data.
>   	 */
>   	int			qcnt;
> +	/* Following are defined for a Command Queue Engine */
> +	int			cqe_in_flight[MMC_ISSUE_MAX];
> +	unsigned int		cqe_busy;
> +	bool			cqe_recovery_needed;
> +	bool			cqe_in_recovery;
> +#define MMC_CQE_DCMD_BUSY	BIT(0)
> +#define MMC_CQE_QUEUE_FULL	BIT(1)
>   };
>   
>   extern int mmc_init_queue(struct mmc_queue *, struct mmc_card *, spinlock_t *,
> -			  const char *);
> +			  const char *, int);
>   extern void mmc_cleanup_queue(struct mmc_queue *);
>   extern void mmc_queue_suspend(struct mmc_queue *);
>   extern void mmc_queue_resume(struct mmc_queue *);
> @@ -85,4 +107,22 @@ extern unsigned int mmc_queue_map_sg(struct mmc_queue *,
>   
>   extern int mmc_access_rpmb(struct mmc_queue *);
>   
> +void mmc_cqe_kick_queue(struct mmc_queue *mq);
> +
> +enum mmc_issue_type mmc_cqe_issue_type(struct mmc_host *host,
> +				       struct request *req);
> +
> +static inline int mmc_cqe_tot_in_flight(struct mmc_queue *mq)
> +{
> +	return mq->cqe_in_flight[MMC_ISSUE_SYNC] +
> +	       mq->cqe_in_flight[MMC_ISSUE_DCMD] +
> +	       mq->cqe_in_flight[MMC_ISSUE_ASYNC];
> +}
> +
> +static inline int mmc_cqe_qcnt(struct mmc_queue *mq)
> +{
> +	return mq->cqe_in_flight[MMC_ISSUE_DCMD] +
> +	       mq->cqe_in_flight[MMC_ISSUE_ASYNC];
> +}
> +
>   #endif
> 

Re: [PATCH V4 09/11] mmc: block: Add CQE support

[Adrian Hunter]

On 01/08/17 11:57, Shawn Lin wrote:
> Hi Adrian,
> 
> On 2017/7/21 17:49, Adrian Hunter wrote:
>> Add CQE support to the block driver, including:
>>     - optionally using DCMD for flush requests
>>     - manually issuing discard requests
>>     - issuing read / write requests to the CQE
>>     - supporting block-layer timeouts
>>     - handling recovery
>>     - supporting re-tuning
>>
>> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
>> ---
>>   drivers/mmc/core/block.c | 195 ++++++++++++++++++++++++++++++++-
>>   drivers/mmc/core/block.h |   7 ++
>>   drivers/mmc/core/queue.c | 273
>> ++++++++++++++++++++++++++++++++++++++++++++++-
>>   drivers/mmc/core/queue.h |  42 +++++++-
>>   4 files changed, 510 insertions(+), 7 deletions(-)
>>
>> diff --git a/drivers/mmc/core/block.c b/drivers/mmc/core/block.c
>> index 915290c74363..2d25115637b7 100644
>> --- a/drivers/mmc/core/block.c
>> +++ b/drivers/mmc/core/block.c
>> @@ -109,6 +109,7 @@ struct mmc_blk_data {
>>   #define MMC_BLK_WRITE        BIT(1)
>>   #define MMC_BLK_DISCARD        BIT(2)
>>   #define MMC_BLK_SECDISCARD    BIT(3)
>> +#define MMC_BLK_CQE_RECOVERY    BIT(4)
>>         /*
>>        * Only set in main mmc_blk_data associated
>> @@ -1612,6 +1613,198 @@ static void mmc_blk_data_prep(struct mmc_queue
>> *mq, struct mmc_queue_req *mqrq,
>>           *do_data_tag_p = do_data_tag;
>>   }
>>   +#define MMC_CQE_RETRIES 2
>> +
>> +void mmc_blk_cqe_complete_rq(struct request *req)
>> +{
>> +    struct mmc_queue_req *mqrq = req_to_mmc_queue_req(req);
>> +    struct mmc_request *mrq = &mqrq->brq.mrq;
>> +    struct request_queue *q = req->q;
>> +    struct mmc_queue *mq = q->queuedata;
>> +    struct mmc_host *host = mq->card->host;
>> +    unsigned long flags;
>> +    bool put_card;
>> +    int err;
>> +
>> +    mmc_cqe_post_req(host, mrq);
>> +
>> +    spin_lock_irqsave(q->queue_lock, flags);
>> +
>> +    mq->cqe_in_flight[mmc_cqe_issue_type(host, req)] -= 1;
>> +
>> +    put_card = mmc_cqe_tot_in_flight(mq) == 0;
>> +
>> +    if (mrq->cmd && mrq->cmd->error)
>> +        err = mrq->cmd->error;
>> +    else if (mrq->data && mrq->data->error)
>> +        err = mrq->data->error;
>> +    else
>> +        err = 0;
>> +
>> +    if (err) {
>> +        if (mqrq->retries++ < MMC_CQE_RETRIES)
>> +            blk_requeue_request(q, req);
>> +        else
>> +            __blk_end_request_all(req, BLK_STS_IOERR);
>> +    } else if (mrq->data) {
>> +        if (__blk_end_request(req, BLK_STS_OK, mrq->data->bytes_xfered))
>> +            blk_requeue_request(q, req);
>> +    } else {
>> +        __blk_end_request_all(req, BLK_STS_OK);
>> +    }
>> +
>> +    mmc_cqe_kick_queue(mq);
>> +
>> +    spin_unlock_irqrestore(q->queue_lock, flags);
>> +
>> +    if (put_card)
>> +        mmc_put_card(mq->card);
>> +}
>> +
>> +void mmc_blk_cqe_recovery(struct mmc_queue *mq)
>> +{
>> +    struct mmc_card *card = mq->card;
>> +    struct mmc_host *host = card->host;
>> +    int err;
>> +
>> +    mmc_get_card(card);
>> +
>> +    pr_debug("%s: CQE recovery start\n", mmc_hostname(host));
>> +
>> +    mq->cqe_in_recovery = true;
>> +
>> +    err = mmc_cqe_recovery(host);
>> +    if (err)
>> +        mmc_blk_reset(mq->blkdata, host, MMC_BLK_CQE_RECOVERY);
>> +    else
>> +        mmc_blk_reset_success(mq->blkdata, MMC_BLK_CQE_RECOVERY);
>> +
>> +    mq->cqe_in_recovery = false;
>> +
>> +    pr_debug("%s: CQE recovery done\n", mmc_hostname(host));
>> +
>> +    mmc_put_card(card);
>> +}
>> +
>> +static void mmc_blk_cqe_req_done(struct mmc_request *mrq)
>> +{
>> +    struct mmc_queue_req *mqrq = container_of(mrq, struct mmc_queue_req,
>> +                          brq.mrq);
>> +    struct request *req = mmc_queue_req_to_req(mqrq);
>> +    struct request_queue *q = req->q;
>> +    struct mmc_queue *mq = q->queuedata;
>> +
>> +    /*
>> +     * Block layer timeouts race with completions which means the normal
>> +     * completion path cannot be used during recovery.
>> +     */
>> +    if (mq->cqe_in_recovery)
>> +        mmc_blk_cqe_complete_rq(req);
>> +    else
>> +        blk_complete_request(req);
>> +}
>> +
>> +static int mmc_blk_cqe_start_req(struct mmc_host *host, struct
>> mmc_request *mrq)
>> +{
>> +    mrq->done = mmc_blk_cqe_req_done;
>> +    return mmc_cqe_start_req(host, mrq);
>> +}
>> +
>> +static struct mmc_request *mmc_blk_cqe_prep_dcmd(struct mmc_queue_req *mqrq,
>> +                         struct request *req)
>> +{
>> +    struct mmc_blk_request *brq = &mqrq->brq;
>> +
>> +    memset(brq, 0, sizeof(*brq));
>> +
>> +    brq->mrq.cmd = &brq->cmd;
>> +    brq->mrq.tag = req->tag;
>> +
>> +    return &brq->mrq;
>> +}
>> +
>> +static int mmc_blk_cqe_issue_flush(struct mmc_queue *mq, struct request
>> *req)
>> +{
>> +    struct mmc_queue_req *mqrq = req_to_mmc_queue_req(req);
>> +    struct mmc_request *mrq = mmc_blk_cqe_prep_dcmd(mqrq, req);
>> +
>> +    mrq->cmd->opcode = MMC_SWITCH;
>> +    mrq->cmd->arg = (MMC_SWITCH_MODE_WRITE_BYTE << 24) |
>> +            (EXT_CSD_FLUSH_CACHE << 16) |
>> +            (1 << 8) |
>> +            EXT_CSD_CMD_SET_NORMAL;
>> +    mrq->cmd->flags = MMC_CMD_AC | MMC_RSP_R1B;
>> +
>> +    return mmc_blk_cqe_start_req(mq->card->host, mrq);
>> +}
>> +
>> +static int mmc_blk_cqe_issue_rw_rq(struct mmc_queue *mq, struct request
>> *req)
>> +{
>> +    struct mmc_queue_req *mqrq = req_to_mmc_queue_req(req);
>> +
>> +    mmc_blk_data_prep(mq, mqrq, 0, NULL, NULL);
>> +
>> +    return mmc_blk_cqe_start_req(mq->card->host, &mqrq->brq.mrq);
>> +}
>> +
>> +enum mmc_issued mmc_blk_cqe_issue_rq(struct mmc_queue *mq, struct request
>> *req)
>> +{
>> +    struct mmc_blk_data *md = mq->blkdata;
>> +    struct mmc_card *card = md->queue.card;
>> +    struct mmc_host *host = card->host;
>> +    int ret;
>> +
>> +    ret = mmc_blk_part_switch(card, md);
>> +    if (ret)
>> +        return MMC_REQ_FAILED_TO_START;
>> +
>> +    switch (mmc_cqe_issue_type(host, req)) {
>> +    case MMC_ISSUE_SYNC:
>> +        ret = host->cqe_ops->cqe_wait_for_idle(host);
>> +        if (ret)
>> +            return MMC_REQ_BUSY;
>> +        switch (req_op(req)) {
>> +        case REQ_OP_DRV_IN:
>> +        case REQ_OP_DRV_OUT:
>> +            mmc_blk_issue_drv_op(mq, req);
>> +            break;
>> +        case REQ_OP_DISCARD:
>> +            mmc_blk_issue_discard_rq(mq, req);
>> +            break;
>> +        case REQ_OP_SECURE_ERASE:
>> +            mmc_blk_issue_secdiscard_rq(mq, req);
>> +            break;
>> +        case REQ_OP_FLUSH:
>> +            mmc_blk_issue_flush(mq, req);
>> +            break;
>> +        default:
>> +            WARN_ON_ONCE(1);
>> +            return MMC_REQ_FAILED_TO_START;
>> +        }
>> +        return MMC_REQ_FINISHED;
>> +    case MMC_ISSUE_DCMD:
>> +    case MMC_ISSUE_ASYNC:
>> +        switch (req_op(req)) {
>> +        case REQ_OP_FLUSH:
>> +            ret = mmc_blk_cqe_issue_flush(mq, req);
>> +            break;
>> +        case REQ_OP_READ:
>> +        case REQ_OP_WRITE:
>> +            ret = mmc_blk_cqe_issue_rw_rq(mq, req);
>> +            break;
>> +        default:
>> +            WARN_ON_ONCE(1);
>> +            ret = -EINVAL;
>> +        }
>> +        if (!ret)
>> +            return MMC_REQ_STARTED;
>> +        return ret == -EBUSY ? MMC_REQ_BUSY : MMC_REQ_FAILED_TO_START;
>> +    default:
>> +        WARN_ON_ONCE(1);
>> +        return MMC_REQ_FAILED_TO_START;
>> +    }
>> +}
>> +
>>   static void mmc_blk_rw_rq_prep(struct mmc_queue_req *mqrq,
>>                      struct mmc_card *card,
>>                      int disable_multi,
>> @@ -2035,7 +2228,7 @@ static struct mmc_blk_data *mmc_blk_alloc_req(struct
>> mmc_card *card,
>>       INIT_LIST_HEAD(&md->part);
>>       md->usage = 1;
>>   -    ret = mmc_init_queue(&md->queue, card, &md->lock, subname);
>> +    ret = mmc_init_queue(&md->queue, card, &md->lock, subname, area_type);
>>       if (ret)
>>           goto err_putdisk;
>>   diff --git a/drivers/mmc/core/block.h b/drivers/mmc/core/block.h
>> index 860ca7c8df86..d7b3d7008b00 100644
>> --- a/drivers/mmc/core/block.h
>> +++ b/drivers/mmc/core/block.h
>> @@ -6,4 +6,11 @@
>>     void mmc_blk_issue_rq(struct mmc_queue *mq, struct request *req);
>>   +enum mmc_issued;
>> +
>> +enum mmc_issued mmc_blk_cqe_issue_rq(struct mmc_queue *mq,
>> +                     struct request *req);
>> +void mmc_blk_cqe_complete_rq(struct request *rq);
>> +void mmc_blk_cqe_recovery(struct mmc_queue *mq);
>> +
>>   #endif
>> diff --git a/drivers/mmc/core/queue.c b/drivers/mmc/core/queue.c
>> index affa7370ba82..0cb7b0e8ee58 100644
>> --- a/drivers/mmc/core/queue.c
>> +++ b/drivers/mmc/core/queue.c
>> @@ -36,10 +36,254 @@ static int mmc_prep_request(struct request_queue *q,
>> struct request *req)
>>           return BLKPREP_KILL;
>>         req->rq_flags |= RQF_DONTPREP;
>> +    req_to_mmc_queue_req(req)->retries = 0;
>>         return BLKPREP_OK;
>>   }
>>   +static void mmc_cqe_request_fn(struct request_queue *q)
>> +{
>> +    struct mmc_queue *mq = q->queuedata;
>> +    struct request *req;
>> +
>> +    if (!mq) {
>> +        while ((req = blk_fetch_request(q)) != NULL) {
>> +            req->rq_flags |= RQF_QUIET;
>> +            __blk_end_request_all(req, BLK_STS_IOERR);
>> +        }
>> +        return;
>> +    }
>> +
>> +    if (mq->asleep && !mq->cqe_busy)
>> +        wake_up_process(mq->thread);
>> +}
>> +
>> +static inline bool mmc_cqe_dcmd_busy(struct mmc_queue *mq)
>> +{
>> +    /* Allow only 1 DCMD at a time */
>> +    return mq->cqe_in_flight[MMC_ISSUE_DCMD];
>> +}
>> +
>> +void mmc_cqe_kick_queue(struct mmc_queue *mq)
>> +{
>> +    if ((mq->cqe_busy & MMC_CQE_DCMD_BUSY) && !mmc_cqe_dcmd_busy(mq))
>> +        mq->cqe_busy &= ~MMC_CQE_DCMD_BUSY;
>> +
>> +    mq->cqe_busy &= ~MMC_CQE_QUEUE_FULL;
>> +
>> +    if (mq->asleep && !mq->cqe_busy)
>> +        __blk_run_queue(mq->queue);
>> +}
>> +
>> +static inline bool mmc_cqe_can_dcmd(struct mmc_host *host)
>> +{
>> +    return host->caps2 & MMC_CAP2_CQE_DCMD;
>> +}
>> +
>> +enum mmc_issue_type mmc_cqe_issue_type(struct mmc_host *host,
>> +                       struct request *req)
>> +{
>> +    switch (req_op(req)) {
>> +    case REQ_OP_DRV_IN:
>> +    case REQ_OP_DRV_OUT:
>> +    case REQ_OP_DISCARD:
>> +    case REQ_OP_SECURE_ERASE:
>> +        return MMC_ISSUE_SYNC;
>> +    case REQ_OP_FLUSH:
>> +        return mmc_cqe_can_dcmd(host) ? MMC_ISSUE_DCMD : MMC_ISSUE_SYNC;
>> +    default:
>> +        return MMC_ISSUE_ASYNC;
>> +    }
>> +}
>> +
>> +static void __mmc_cqe_recovery_notifier(struct mmc_queue *mq)
>> +{
>> +    if (!mq->cqe_recovery_needed) {
>> +        mq->cqe_recovery_needed = true;
>> +        wake_up_process(mq->thread);
>> +    }
>> +}
>> +
>> +static void mmc_cqe_recovery_notifier(struct mmc_host *host,
>> +                      struct mmc_request *mrq)
>> +{
>> +    struct mmc_queue_req *mqrq = container_of(mrq, struct mmc_queue_req,
>> +                          brq.mrq);
>> +    struct request *req = mmc_queue_req_to_req(mqrq);
>> +    struct request_queue *q = req->q;
>> +    struct mmc_queue *mq = q->queuedata;
>> +    unsigned long flags;
>> +
>> +    spin_lock_irqsave(q->queue_lock, flags);
>> +    __mmc_cqe_recovery_notifier(mq);
>> +    spin_unlock_irqrestore(q->queue_lock, flags);
>> +}
>> +
>> +static int mmc_cqe_thread(void *d)
>> +{
>> +    struct mmc_queue *mq = d;
>> +    struct request_queue *q = mq->queue;
>> +    struct mmc_card *card = mq->card;
>> +    struct mmc_host *host = card->host;
>> +    unsigned long flags;
>> +    int get_put = 0;
>> +
>> +    current->flags |= PF_MEMALLOC;
>> +
>> +    down(&mq->thread_sem);
>> +    spin_lock_irqsave(q->queue_lock, flags);
>> +    while (1) {
>> +        struct request *req = NULL;
>> +        enum mmc_issue_type issue_type;
>> +        bool retune_ok = false;
>> +
>> +        if (mq->cqe_recovery_needed) {
>> +            spin_unlock_irqrestore(q->queue_lock, flags);
>> +            mmc_blk_cqe_recovery(mq);
>> +            spin_lock_irqsave(q->queue_lock, flags);
>> +            mq->cqe_recovery_needed = false;
>> +        }
>> +
>> +        set_current_state(TASK_INTERRUPTIBLE);
>> +
>> +        if (!kthread_should_stop())
>> +            req = blk_peek_request(q);
>> +
>> +        if (req) {
>> +            issue_type = mmc_cqe_issue_type(host, req);
>> +            switch (issue_type) {
>> +            case MMC_ISSUE_DCMD:
>> +                if (mmc_cqe_dcmd_busy(mq)) {
>> +                    mq->cqe_busy |= MMC_CQE_DCMD_BUSY;
>> +                    req = NULL;
>> +                    break;
>> +                }
>> +                /* Fall through */
>> +            case MMC_ISSUE_ASYNC:
>> +                if (blk_queue_start_tag(q, req)) {
>> +                    mq->cqe_busy |= MMC_CQE_QUEUE_FULL;
>> +                    req = NULL;
>> +                }
>> +                break;
>> +            default:
>> +                /*
>> +                 * Timeouts are handled by mmc core, so set a
>> +                 * large value to avoid races.
>> +                 */
>> +                req->timeout = 600 * HZ;
>> +                blk_start_request(req);
>> +                break;
>> +            }
>> +            if (req) {
>> +                mq->cqe_in_flight[issue_type] += 1;
>> +                if (mmc_cqe_tot_in_flight(mq) == 1)
>> +                    get_put += 1;
>> +                if (mmc_cqe_qcnt(mq) == 1)
>> +                    retune_ok = true;
>> +            }
>> +        }
>> +
> 
> Just a thought that mmc_cq_thread is a little heavy for manage in-flight
> request, so could we kick the check back to blk layer like this in the
> prepare and unprepare hook?

I am not sure what you are aiming at.  The prepare function is called by
blk_peek_request() so the work is anyway being done by the thread.  However
the prepare function isn't always called, for example if the request has
already been prepared and then later re-queued.  So it looks to me as though
it won't work that way.

> 
> 
> --- a/drivers/mmc/core/queue.c
> +++ b/drivers/mmc/core/queue.c
> @@ -25,22 +25,59 @@
> 
>  #define MMC_QUEUE_BOUNCESZ    65536
> 
> +static inline bool mmc_cqe_dcmd_busy(struct mmc_queue *mq);
> +static inline bool mmc_cqe_can_dcmd(struct mmc_host *host);
> +enum mmc_issue_type mmc_cqe_issue_type(struct mmc_host *host,
> +                       struct request *req);
> +
>  /*
>   * Prepare a MMC request. This just filters out odd stuff.
>   */
>  static int mmc_prep_request(struct request_queue *q, struct request *req)
>  {
>      struct mmc_queue *mq = q->queuedata;
> +    enum mmc_issue_type issue_type;
> 
>      if (mq && (mmc_card_removed(mq->card) || mmc_access_rpmb(mq)))
>          return BLKPREP_KILL;
> 
> +    if (mq->card->host->cqe_enabled) {
> +        issue_type = mmc_cqe_issue_type(mq->card->host, req);
> +        if ((issue_type == MMC_ISSUE_DCMD && mmc_cqe_dcmd_busy(mq))) {
> +            mq->cqe_busy |= MMC_CQE_DCMD_BUSY;
> +            return BLKPREP_DEFER;
> +        } else if (issue_type == MMC_ISSUE_ASYNC &&
> +                   mq->cqe_in_flight[issue_type] >=
> +                   mq->queue->queue_tags->max_depth) {
> +            mq->cqe_busy |= MMC_CQE_QUEUE_FULL;
> +            return BLKPREP_DEFER;
> +        }
> +    }
> +
>      req->rq_flags |= RQF_DONTPREP;
>      req_to_mmc_queue_req(req)->retries = 0;
> 
>      return BLKPREP_OK;
>  }
> 
> +static void mmc_unprep_request(struct request_queue *q, struct request *req)
> +{
> +    struct mmc_queue *mq = q->queuedata;
> +    enum mmc_issue_type issue_type;
> +
> +    if (!mq->card->host->cqe_enabled)
> +        return;
> +
> +    issue_type = mmc_cqe_issue_type(mq->card->host, req);
> +    if (issue_type == MMC_ISSUE_DCMD &&
> +        (mq->cqe_busy & MMC_CQE_DCMD_BUSY) &&
> +        !mmc_cqe_dcmd_busy(mq))
> +        mq->cqe_busy &= ~MMC_CQE_DCMD_BUSY;
> +    else if (issue_type == MMC_ISSUE_ASYNC)
> +        mq->cqe_busy &= ~MMC_CQE_QUEUE_FULL;
> +}
> +
> +
>  static void mmc_cqe_request_fn(struct request_queue *q)
>  {
>      struct mmc_queue *mq = q->queuedata;
> @@ -66,11 +103,6 @@ static inline bool mmc_cqe_dcmd_busy(struct mmc_queue *mq)
> 
>  void mmc_cqe_kick_queue(struct mmc_queue *mq)
>  {
> -    if ((mq->cqe_busy & MMC_CQE_DCMD_BUSY) && !mmc_cqe_dcmd_busy(mq))
> -        mq->cqe_busy &= ~MMC_CQE_DCMD_BUSY;
> -
> -    mq->cqe_busy &= ~MMC_CQE_QUEUE_FULL;
> -
>      if (mq->asleep && !mq->cqe_busy)
>          __blk_run_queue(mq->queue);
>  }
> @@ -151,21 +183,9 @@ static int mmc_cqe_thread(void *d)
> 
>          if (req) {
>              issue_type = mmc_cqe_issue_type(host, req);
> -            switch (issue_type) {
> -            case MMC_ISSUE_DCMD:
> -                if (mmc_cqe_dcmd_busy(mq)) {
> -                    mq->cqe_busy |= MMC_CQE_DCMD_BUSY;
> -                    req = NULL;
> -                    break;
> -                }
> -                /* Fall through */
> -            case MMC_ISSUE_ASYNC:
> -                if (blk_queue_start_tag(q, req)) {
> -                    mq->cqe_busy |= MMC_CQE_QUEUE_FULL;
> -                    req = NULL;
> -                }
> -                break;
> -            default:
> +            if (issue_type == MMC_ISSUE_DCMD || issue_type ==
> MMC_ISSUE_ASYNC) {
> +                blk_queue_start_tag(q, req);
> +            } else {
>                  /*
>                   * Timeouts are handled by mmc core, so set a
>                   * large value to avoid races.
> @@ -174,13 +194,12 @@ static int mmc_cqe_thread(void *d)
>                  blk_start_request(req);
>                  break;
>              }
> -            if (req) {
> -                mq->cqe_in_flight[issue_type] += 1;
> -                if (mmc_cqe_tot_in_flight(mq) == 1)
> -                    get_put += 1;
> -                if (mmc_cqe_qcnt(mq) == 1)
> -                    retune_ok = true;
> -            }
> +
> +            mq->cqe_in_flight[issue_type] += 1;
> +            if (mmc_cqe_tot_in_flight(mq) == 1)
> +                get_put += 1;
> +            if (mmc_cqe_qcnt(mq) == 1)
> +                retune_ok = true;
>          }
> 
>          mq->asleep = !req;
> @@ -523,6 +542,7 @@ int mmc_init_queue(struct mmc_queue *mq, struct mmc_card
> *card,
>      }
> 
>      blk_queue_prep_rq(mq->queue, mmc_prep_request);
> +    blk_queue_unprep_rq(mq->queue, mmc_unprep_request);
>      queue_flag_set_unlocked(QUEUE_FLAG_NONROT, mq->queue);
>      queue_flag_clear_unlocked(QUEUE_FLAG_ADD_RANDOM, mq->queue);
>      if (mmc_can_erase(card))
> 
> 
> 
>> +        mq->asleep = !req;
>> +
>> +        spin_unlock_irqrestore(q->queue_lock, flags);
>> +
>> +        if (req) {
>> +            enum mmc_issued issued;
>> +
>> +            set_current_state(TASK_RUNNING);
>> +
>> +            if (get_put) {
>> +                get_put = 0;
>> +                mmc_get_card(card);
>> +            }
>> +
>> +            if (host->need_retune && retune_ok &&
>> +                !host->hold_retune)
>> +                host->retune_now = true;
>> +            else
>> +                host->retune_now = false;
>> +
>> +            issued = mmc_blk_cqe_issue_rq(mq, req);
>> +
>> +            cond_resched();
>> +
>> +            spin_lock_irqsave(q->queue_lock, flags);
>> +
>> +            switch (issued) {
>> +            case MMC_REQ_STARTED:
>> +                break;
>> +            case MMC_REQ_BUSY:
>> +                blk_requeue_request(q, req);
>> +                goto finished;
>> +            case MMC_REQ_FAILED_TO_START:
>> +                __blk_end_request_all(req, BLK_STS_IOERR);
>> +                /* Fall through */
>> +            case MMC_REQ_FINISHED:
>> +finished:
>> +                mq->cqe_in_flight[issue_type] -= 1;
>> +                if (mmc_cqe_tot_in_flight(mq) == 0)
>> +                    get_put = -1;
>> +            }
>> +        } else {
>> +            if (get_put < 0) {
>> +                get_put = 0;
>> +                mmc_put_card(card);
>> +            }
>> +            /*
>> +             * Do not stop with requests in flight in case recovery
>> +             * is needed.
>> +             */
>> +            if (kthread_should_stop() &&
>> +                !mmc_cqe_tot_in_flight(mq)) {
>> +                set_current_state(TASK_RUNNING);
>> +                break;
>> +            }
>> +            up(&mq->thread_sem);
>> +            schedule();
>> +            down(&mq->thread_sem);
>> +            spin_lock_irqsave(q->queue_lock, flags);
>> +        }
>> +    } /* loop */
>> +    up(&mq->thread_sem);
>> +
>> +    return 0;
>> +}
>> +
>> +static enum blk_eh_timer_return __mmc_cqe_timed_out(struct request *req)
>> +{
>> +    struct mmc_queue_req *mqrq = req_to_mmc_queue_req(req);
>> +    struct mmc_request *mrq = &mqrq->brq.mrq;
>> +    struct mmc_queue *mq = req->q->queuedata;
>> +    struct mmc_host *host = mq->card->host;
>> +    enum mmc_issue_type issue_type = mmc_cqe_issue_type(host, req);
>> +    bool recovery_needed = false;
>> +
>> +    switch (issue_type) {
>> +    case MMC_ISSUE_ASYNC:
>> +    case MMC_ISSUE_DCMD:
>> +        if (host->cqe_ops->cqe_timeout(host, mrq, &recovery_needed)) {
>> +            if (recovery_needed)
>> +                __mmc_cqe_recovery_notifier(mq);
>> +            return BLK_EH_RESET_TIMER;
>> +        }
>> +        /* No timeout */
>> +        return BLK_EH_HANDLED;
>> +    default:
>> +        /* Timeout is handled by mmc core */
>> +        return BLK_EH_RESET_TIMER;
>> +    }
>> +}
>> +
>> +static enum blk_eh_timer_return mmc_cqe_timed_out(struct request *req)
>> +{
>> +    struct mmc_queue *mq = req->q->queuedata;
>> +
>> +    if (mq->cqe_recovery_needed)
>> +        return BLK_EH_RESET_TIMER;
>> +
>> +    return __mmc_cqe_timed_out(req);
>> +}
>> +
>>   static int mmc_queue_thread(void *d)
>>   {
>>       struct mmc_queue *mq = d;
>> @@ -233,11 +477,12 @@ static void mmc_exit_request(struct request_queue
>> *q, struct request *req)
>>    * Initialise a MMC card request queue.
>>    */
>>   int mmc_init_queue(struct mmc_queue *mq, struct mmc_card *card,
>> -           spinlock_t *lock, const char *subname)
>> +           spinlock_t *lock, const char *subname, int area_type)
>>   {
>>       struct mmc_host *host = card->host;
>>       u64 limit = BLK_BOUNCE_HIGH;
>>       int ret = -ENOMEM;
>> +    bool use_cqe = host->cqe_enabled && area_type != MMC_BLK_DATA_AREA_RPMB;
>>         if (mmc_dev(host)->dma_mask && *mmc_dev(host)->dma_mask)
>>           limit = (u64)dma_max_pfn(mmc_dev(host)) << PAGE_SHIFT;
>> @@ -247,7 +492,7 @@ int mmc_init_queue(struct mmc_queue *mq, struct
>> mmc_card *card,
>>       if (!mq->queue)
>>           return -ENOMEM;
>>       mq->queue->queue_lock = lock;
>> -    mq->queue->request_fn = mmc_request_fn;
>> +    mq->queue->request_fn = use_cqe ? mmc_cqe_request_fn : mmc_request_fn;
>>       mq->queue->init_rq_fn = mmc_init_request;
>>       mq->queue->exit_rq_fn = mmc_exit_request;
>>       mq->queue->cmd_size = sizeof(struct mmc_queue_req);
>> @@ -259,6 +504,24 @@ int mmc_init_queue(struct mmc_queue *mq, struct
>> mmc_card *card,
>>           return ret;
>>       }
>>   +    if (use_cqe) {
>> +        int q_depth = card->ext_csd.cmdq_depth;
>> +
>> +        if (q_depth > host->cqe_qdepth)
>> +            q_depth = host->cqe_qdepth;
>> +
>> +        ret = blk_queue_init_tags(mq->queue, q_depth, NULL,
>> +                      BLK_TAG_ALLOC_FIFO);
>> +        if (ret)
>> +            goto cleanup_queue;
>> +
>> +        blk_queue_softirq_done(mq->queue, mmc_blk_cqe_complete_rq);
>> +        blk_queue_rq_timed_out(mq->queue, mmc_cqe_timed_out);
>> +        blk_queue_rq_timeout(mq->queue, 60 * HZ);
>> +
>> +        host->cqe_recovery_notifier = mmc_cqe_recovery_notifier;
>> +    }
>> +
>>       blk_queue_prep_rq(mq->queue, mmc_prep_request);
>>       queue_flag_set_unlocked(QUEUE_FLAG_NONROT, mq->queue);
>>       queue_flag_clear_unlocked(QUEUE_FLAG_ADD_RANDOM, mq->queue);
>> @@ -280,9 +543,9 @@ int mmc_init_queue(struct mmc_queue *mq, struct
>> mmc_card *card,
>>         sema_init(&mq->thread_sem, 1);
>>   -    mq->thread = kthread_run(mmc_queue_thread, mq, "mmcqd/%d%s",
>> -        host->index, subname ? subname : "");
>> -
>> +    mq->thread = kthread_run(use_cqe ? mmc_cqe_thread : mmc_queue_thread,
>> +                 mq, "mmcqd/%d%s", host->index,
>> +                 subname ? subname : "");
>>       if (IS_ERR(mq->thread)) {
>>           ret = PTR_ERR(mq->thread);
>>           goto cleanup_queue;
>> diff --git a/drivers/mmc/core/queue.h b/drivers/mmc/core/queue.h
>> index 361b46408e0f..8e9273d977c0 100644
>> --- a/drivers/mmc/core/queue.h
>> +++ b/drivers/mmc/core/queue.h
>> @@ -7,6 +7,20 @@
>>   #include <linux/mmc/core.h>
>>   #include <linux/mmc/host.h>
>>   +enum mmc_issued {
>> +    MMC_REQ_STARTED,
>> +    MMC_REQ_BUSY,
>> +    MMC_REQ_FAILED_TO_START,
>> +    MMC_REQ_FINISHED,
>> +};
>> +
>> +enum mmc_issue_type {
>> +    MMC_ISSUE_SYNC,
>> +    MMC_ISSUE_DCMD,
>> +    MMC_ISSUE_ASYNC,
>> +    MMC_ISSUE_MAX,
>> +};
>> +
>>   static inline struct mmc_queue_req *req_to_mmc_queue_req(struct request
>> *rq)
>>   {
>>       return blk_mq_rq_to_pdu(rq);
>> @@ -53,6 +67,7 @@ struct mmc_queue_req {
>>       int            drv_op_result;
>>       struct mmc_blk_ioc_data    **idata;
>>       unsigned int        ioc_count;
>> +    int            retries;
>>   };
>>     struct mmc_queue {
>> @@ -70,10 +85,17 @@ struct mmc_queue {
>>        * associated mmc_queue_req data.
>>        */
>>       int            qcnt;
>> +    /* Following are defined for a Command Queue Engine */
>> +    int            cqe_in_flight[MMC_ISSUE_MAX];
>> +    unsigned int        cqe_busy;
>> +    bool            cqe_recovery_needed;
>> +    bool            cqe_in_recovery;
>> +#define MMC_CQE_DCMD_BUSY    BIT(0)
>> +#define MMC_CQE_QUEUE_FULL    BIT(1)
>>   };
>>     extern int mmc_init_queue(struct mmc_queue *, struct mmc_card *,
>> spinlock_t *,
>> -              const char *);
>> +              const char *, int);
>>   extern void mmc_cleanup_queue(struct mmc_queue *);
>>   extern void mmc_queue_suspend(struct mmc_queue *);
>>   extern void mmc_queue_resume(struct mmc_queue *);
>> @@ -85,4 +107,22 @@ extern unsigned int mmc_queue_map_sg(struct mmc_queue *,
>>     extern int mmc_access_rpmb(struct mmc_queue *);
>>   +void mmc_cqe_kick_queue(struct mmc_queue *mq);
>> +
>> +enum mmc_issue_type mmc_cqe_issue_type(struct mmc_host *host,
>> +                       struct request *req);
>> +
>> +static inline int mmc_cqe_tot_in_flight(struct mmc_queue *mq)
>> +{
>> +    return mq->cqe_in_flight[MMC_ISSUE_SYNC] +
>> +           mq->cqe_in_flight[MMC_ISSUE_DCMD] +
>> +           mq->cqe_in_flight[MMC_ISSUE_ASYNC];
>> +}
>> +
>> +static inline int mmc_cqe_qcnt(struct mmc_queue *mq)
>> +{
>> +    return mq->cqe_in_flight[MMC_ISSUE_DCMD] +
>> +           mq->cqe_in_flight[MMC_ISSUE_ASYNC];
>> +}
>> +
>>   #endif
>>
> 
> 

Re: [PATCH V4 09/11] mmc: block: Add CQE support

[Shawn Lin]

On 2017/8/1 18:06, Adrian Hunter wrote:
> On 01/08/17 11:57, Shawn Lin wrote:
>> Hi Adrian,
>>
>> On 2017/7/21 17:49, Adrian Hunter wrote:
>>> Add CQE support to the block driver, including:
>>>      - optionally using DCMD for flush requests
>>>      - manually issuing discard requests
>>>      - issuing read / write requests to the CQE
>>>      - supporting block-layer timeouts
>>>      - handling recovery
>>>      - supporting re-tuning
>>>
>>> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
>>> ---
>>>    drivers/mmc/core/block.c | 195 ++++++++++++++++++++++++++++++++-
>>>    drivers/mmc/core/block.h |   7 ++
>>>    drivers/mmc/core/queue.c | 273
>>> ++++++++++++++++++++++++++++++++++++++++++++++-
>>>    drivers/mmc/core/queue.h |  42 +++++++-
>>>    4 files changed, 510 insertions(+), 7 deletions(-)
>>>
>>> diff --git a/drivers/mmc/core/block.c b/drivers/mmc/core/block.c
>>> index 915290c74363..2d25115637b7 100644
>>> --- a/drivers/mmc/core/block.c
>>> +++ b/drivers/mmc/core/block.c
>>> @@ -109,6 +109,7 @@ struct mmc_blk_data {
>>>    #define MMC_BLK_WRITE        BIT(1)
>>>    #define MMC_BLK_DISCARD        BIT(2)
>>>    #define MMC_BLK_SECDISCARD    BIT(3)
>>> +#define MMC_BLK_CQE_RECOVERY    BIT(4)
>>>          /*
>>>         * Only set in main mmc_blk_data associated
>>> @@ -1612,6 +1613,198 @@ static void mmc_blk_data_prep(struct mmc_queue
>>> *mq, struct mmc_queue_req *mqrq,
>>>            *do_data_tag_p = do_data_tag;
>>>    }
>>>    +#define MMC_CQE_RETRIES 2
>>> +
>>> +void mmc_blk_cqe_complete_rq(struct request *req)
>>> +{
>>> +    struct mmc_queue_req *mqrq = req_to_mmc_queue_req(req);
>>> +    struct mmc_request *mrq = &mqrq->brq.mrq;
>>> +    struct request_queue *q = req->q;
>>> +    struct mmc_queue *mq = q->queuedata;
>>> +    struct mmc_host *host = mq->card->host;
>>> +    unsigned long flags;
>>> +    bool put_card;
>>> +    int err;
>>> +
>>> +    mmc_cqe_post_req(host, mrq);
>>> +
>>> +    spin_lock_irqsave(q->queue_lock, flags);
>>> +
>>> +    mq->cqe_in_flight[mmc_cqe_issue_type(host, req)] -= 1;
>>> +
>>> +    put_card = mmc_cqe_tot_in_flight(mq) == 0;
>>> +
>>> +    if (mrq->cmd && mrq->cmd->error)
>>> +        err = mrq->cmd->error;
>>> +    else if (mrq->data && mrq->data->error)
>>> +        err = mrq->data->error;
>>> +    else
>>> +        err = 0;
>>> +
>>> +    if (err) {
>>> +        if (mqrq->retries++ < MMC_CQE_RETRIES)
>>> +            blk_requeue_request(q, req);
>>> +        else
>>> +            __blk_end_request_all(req, BLK_STS_IOERR);
>>> +    } else if (mrq->data) {
>>> +        if (__blk_end_request(req, BLK_STS_OK, mrq->data->bytes_xfered))
>>> +            blk_requeue_request(q, req);
>>> +    } else {
>>> +        __blk_end_request_all(req, BLK_STS_OK);
>>> +    }
>>> +
>>> +    mmc_cqe_kick_queue(mq);
>>> +
>>> +    spin_unlock_irqrestore(q->queue_lock, flags);
>>> +
>>> +    if (put_card)
>>> +        mmc_put_card(mq->card);
>>> +}
>>> +
>>> +void mmc_blk_cqe_recovery(struct mmc_queue *mq)
>>> +{
>>> +    struct mmc_card *card = mq->card;
>>> +    struct mmc_host *host = card->host;
>>> +    int err;
>>> +
>>> +    mmc_get_card(card);
>>> +
>>> +    pr_debug("%s: CQE recovery start\n", mmc_hostname(host));
>>> +
>>> +    mq->cqe_in_recovery = true;
>>> +
>>> +    err = mmc_cqe_recovery(host);
>>> +    if (err)
>>> +        mmc_blk_reset(mq->blkdata, host, MMC_BLK_CQE_RECOVERY);
>>> +    else
>>> +        mmc_blk_reset_success(mq->blkdata, MMC_BLK_CQE_RECOVERY);
>>> +
>>> +    mq->cqe_in_recovery = false;
>>> +
>>> +    pr_debug("%s: CQE recovery done\n", mmc_hostname(host));
>>> +
>>> +    mmc_put_card(card);
>>> +}
>>> +
>>> +static void mmc_blk_cqe_req_done(struct mmc_request *mrq)
>>> +{
>>> +    struct mmc_queue_req *mqrq = container_of(mrq, struct mmc_queue_req,
>>> +                          brq.mrq);
>>> +    struct request *req = mmc_queue_req_to_req(mqrq);
>>> +    struct request_queue *q = req->q;
>>> +    struct mmc_queue *mq = q->queuedata;
>>> +
>>> +    /*
>>> +     * Block layer timeouts race with completions which means the normal
>>> +     * completion path cannot be used during recovery.
>>> +     */
>>> +    if (mq->cqe_in_recovery)
>>> +        mmc_blk_cqe_complete_rq(req);
>>> +    else
>>> +        blk_complete_request(req);
>>> +}
>>> +
>>> +static int mmc_blk_cqe_start_req(struct mmc_host *host, struct
>>> mmc_request *mrq)
>>> +{
>>> +    mrq->done = mmc_blk_cqe_req_done;
>>> +    return mmc_cqe_start_req(host, mrq);
>>> +}
>>> +
>>> +static struct mmc_request *mmc_blk_cqe_prep_dcmd(struct mmc_queue_req *mqrq,
>>> +                         struct request *req)
>>> +{
>>> +    struct mmc_blk_request *brq = &mqrq->brq;
>>> +
>>> +    memset(brq, 0, sizeof(*brq));
>>> +
>>> +    brq->mrq.cmd = &brq->cmd;
>>> +    brq->mrq.tag = req->tag;
>>> +
>>> +    return &brq->mrq;
>>> +}
>>> +
>>> +static int mmc_blk_cqe_issue_flush(struct mmc_queue *mq, struct request
>>> *req)
>>> +{
>>> +    struct mmc_queue_req *mqrq = req_to_mmc_queue_req(req);
>>> +    struct mmc_request *mrq = mmc_blk_cqe_prep_dcmd(mqrq, req);
>>> +
>>> +    mrq->cmd->opcode = MMC_SWITCH;
>>> +    mrq->cmd->arg = (MMC_SWITCH_MODE_WRITE_BYTE << 24) |
>>> +            (EXT_CSD_FLUSH_CACHE << 16) |
>>> +            (1 << 8) |
>>> +            EXT_CSD_CMD_SET_NORMAL;
>>> +    mrq->cmd->flags = MMC_CMD_AC | MMC_RSP_R1B;
>>> +
>>> +    return mmc_blk_cqe_start_req(mq->card->host, mrq);
>>> +}
>>> +
>>> +static int mmc_blk_cqe_issue_rw_rq(struct mmc_queue *mq, struct request
>>> *req)
>>> +{
>>> +    struct mmc_queue_req *mqrq = req_to_mmc_queue_req(req);
>>> +
>>> +    mmc_blk_data_prep(mq, mqrq, 0, NULL, NULL);
>>> +
>>> +    return mmc_blk_cqe_start_req(mq->card->host, &mqrq->brq.mrq);
>>> +}
>>> +
>>> +enum mmc_issued mmc_blk_cqe_issue_rq(struct mmc_queue *mq, struct request
>>> *req)
>>> +{
>>> +    struct mmc_blk_data *md = mq->blkdata;
>>> +    struct mmc_card *card = md->queue.card;
>>> +    struct mmc_host *host = card->host;
>>> +    int ret;
>>> +
>>> +    ret = mmc_blk_part_switch(card, md);
>>> +    if (ret)
>>> +        return MMC_REQ_FAILED_TO_START;
>>> +
>>> +    switch (mmc_cqe_issue_type(host, req)) {
>>> +    case MMC_ISSUE_SYNC:
>>> +        ret = host->cqe_ops->cqe_wait_for_idle(host);
>>> +        if (ret)
>>> +            return MMC_REQ_BUSY;
>>> +        switch (req_op(req)) {
>>> +        case REQ_OP_DRV_IN:
>>> +        case REQ_OP_DRV_OUT:
>>> +            mmc_blk_issue_drv_op(mq, req);
>>> +            break;
>>> +        case REQ_OP_DISCARD:
>>> +            mmc_blk_issue_discard_rq(mq, req);
>>> +            break;
>>> +        case REQ_OP_SECURE_ERASE:
>>> +            mmc_blk_issue_secdiscard_rq(mq, req);
>>> +            break;
>>> +        case REQ_OP_FLUSH:
>>> +            mmc_blk_issue_flush(mq, req);
>>> +            break;
>>> +        default:
>>> +            WARN_ON_ONCE(1);
>>> +            return MMC_REQ_FAILED_TO_START;
>>> +        }
>>> +        return MMC_REQ_FINISHED;
>>> +    case MMC_ISSUE_DCMD:
>>> +    case MMC_ISSUE_ASYNC:
>>> +        switch (req_op(req)) {
>>> +        case REQ_OP_FLUSH:
>>> +            ret = mmc_blk_cqe_issue_flush(mq, req);
>>> +            break;
>>> +        case REQ_OP_READ:
>>> +        case REQ_OP_WRITE:
>>> +            ret = mmc_blk_cqe_issue_rw_rq(mq, req);
>>> +            break;
>>> +        default:
>>> +            WARN_ON_ONCE(1);
>>> +            ret = -EINVAL;
>>> +        }
>>> +        if (!ret)
>>> +            return MMC_REQ_STARTED;
>>> +        return ret == -EBUSY ? MMC_REQ_BUSY : MMC_REQ_FAILED_TO_START;
>>> +    default:
>>> +        WARN_ON_ONCE(1);
>>> +        return MMC_REQ_FAILED_TO_START;
>>> +    }
>>> +}
>>> +
>>>    static void mmc_blk_rw_rq_prep(struct mmc_queue_req *mqrq,
>>>                       struct mmc_card *card,
>>>                       int disable_multi,
>>> @@ -2035,7 +2228,7 @@ static struct mmc_blk_data *mmc_blk_alloc_req(struct
>>> mmc_card *card,
>>>        INIT_LIST_HEAD(&md->part);
>>>        md->usage = 1;
>>>    -    ret = mmc_init_queue(&md->queue, card, &md->lock, subname);
>>> +    ret = mmc_init_queue(&md->queue, card, &md->lock, subname, area_type);
>>>        if (ret)
>>>            goto err_putdisk;
>>>    diff --git a/drivers/mmc/core/block.h b/drivers/mmc/core/block.h
>>> index 860ca7c8df86..d7b3d7008b00 100644
>>> --- a/drivers/mmc/core/block.h
>>> +++ b/drivers/mmc/core/block.h
>>> @@ -6,4 +6,11 @@
>>>      void mmc_blk_issue_rq(struct mmc_queue *mq, struct request *req);
>>>    +enum mmc_issued;
>>> +
>>> +enum mmc_issued mmc_blk_cqe_issue_rq(struct mmc_queue *mq,
>>> +                     struct request *req);
>>> +void mmc_blk_cqe_complete_rq(struct request *rq);
>>> +void mmc_blk_cqe_recovery(struct mmc_queue *mq);
>>> +
>>>    #endif
>>> diff --git a/drivers/mmc/core/queue.c b/drivers/mmc/core/queue.c
>>> index affa7370ba82..0cb7b0e8ee58 100644
>>> --- a/drivers/mmc/core/queue.c
>>> +++ b/drivers/mmc/core/queue.c
>>> @@ -36,10 +36,254 @@ static int mmc_prep_request(struct request_queue *q,
>>> struct request *req)
>>>            return BLKPREP_KILL;
>>>          req->rq_flags |= RQF_DONTPREP;
>>> +    req_to_mmc_queue_req(req)->retries = 0;
>>>          return BLKPREP_OK;
>>>    }
>>>    +static void mmc_cqe_request_fn(struct request_queue *q)
>>> +{
>>> +    struct mmc_queue *mq = q->queuedata;
>>> +    struct request *req;
>>> +
>>> +    if (!mq) {
>>> +        while ((req = blk_fetch_request(q)) != NULL) {
>>> +            req->rq_flags |= RQF_QUIET;
>>> +            __blk_end_request_all(req, BLK_STS_IOERR);
>>> +        }
>>> +        return;
>>> +    }
>>> +
>>> +    if (mq->asleep && !mq->cqe_busy)
>>> +        wake_up_process(mq->thread);
>>> +}
>>> +
>>> +static inline bool mmc_cqe_dcmd_busy(struct mmc_queue *mq)
>>> +{
>>> +    /* Allow only 1 DCMD at a time */
>>> +    return mq->cqe_in_flight[MMC_ISSUE_DCMD];
>>> +}
>>> +
>>> +void mmc_cqe_kick_queue(struct mmc_queue *mq)
>>> +{
>>> +    if ((mq->cqe_busy & MMC_CQE_DCMD_BUSY) && !mmc_cqe_dcmd_busy(mq))
>>> +        mq->cqe_busy &= ~MMC_CQE_DCMD_BUSY;
>>> +
>>> +    mq->cqe_busy &= ~MMC_CQE_QUEUE_FULL;
>>> +
>>> +    if (mq->asleep && !mq->cqe_busy)
>>> +        __blk_run_queue(mq->queue);
>>> +}
>>> +
>>> +static inline bool mmc_cqe_can_dcmd(struct mmc_host *host)
>>> +{
>>> +    return host->caps2 & MMC_CAP2_CQE_DCMD;
>>> +}
>>> +
>>> +enum mmc_issue_type mmc_cqe_issue_type(struct mmc_host *host,
>>> +                       struct request *req)
>>> +{
>>> +    switch (req_op(req)) {
>>> +    case REQ_OP_DRV_IN:
>>> +    case REQ_OP_DRV_OUT:
>>> +    case REQ_OP_DISCARD:
>>> +    case REQ_OP_SECURE_ERASE:
>>> +        return MMC_ISSUE_SYNC;
>>> +    case REQ_OP_FLUSH:
>>> +        return mmc_cqe_can_dcmd(host) ? MMC_ISSUE_DCMD : MMC_ISSUE_SYNC;
>>> +    default:
>>> +        return MMC_ISSUE_ASYNC;
>>> +    }
>>> +}
>>> +
>>> +static void __mmc_cqe_recovery_notifier(struct mmc_queue *mq)
>>> +{
>>> +    if (!mq->cqe_recovery_needed) {
>>> +        mq->cqe_recovery_needed = true;
>>> +        wake_up_process(mq->thread);
>>> +    }
>>> +}
>>> +
>>> +static void mmc_cqe_recovery_notifier(struct mmc_host *host,
>>> +                      struct mmc_request *mrq)
>>> +{
>>> +    struct mmc_queue_req *mqrq = container_of(mrq, struct mmc_queue_req,
>>> +                          brq.mrq);
>>> +    struct request *req = mmc_queue_req_to_req(mqrq);
>>> +    struct request_queue *q = req->q;
>>> +    struct mmc_queue *mq = q->queuedata;
>>> +    unsigned long flags;
>>> +
>>> +    spin_lock_irqsave(q->queue_lock, flags);
>>> +    __mmc_cqe_recovery_notifier(mq);
>>> +    spin_unlock_irqrestore(q->queue_lock, flags);
>>> +}
>>> +
>>> +static int mmc_cqe_thread(void *d)
>>> +{
>>> +    struct mmc_queue *mq = d;
>>> +    struct request_queue *q = mq->queue;
>>> +    struct mmc_card *card = mq->card;
>>> +    struct mmc_host *host = card->host;
>>> +    unsigned long flags;
>>> +    int get_put = 0;
>>> +
>>> +    current->flags |= PF_MEMALLOC;
>>> +
>>> +    down(&mq->thread_sem);
>>> +    spin_lock_irqsave(q->queue_lock, flags);
>>> +    while (1) {
>>> +        struct request *req = NULL;
>>> +        enum mmc_issue_type issue_type;
>>> +        bool retune_ok = false;
>>> +
>>> +        if (mq->cqe_recovery_needed) {
>>> +            spin_unlock_irqrestore(q->queue_lock, flags);
>>> +            mmc_blk_cqe_recovery(mq);
>>> +            spin_lock_irqsave(q->queue_lock, flags);
>>> +            mq->cqe_recovery_needed = false;
>>> +        }
>>> +
>>> +        set_current_state(TASK_INTERRUPTIBLE);
>>> +
>>> +        if (!kthread_should_stop())
>>> +            req = blk_peek_request(q);
>>> +
>>> +        if (req) {
>>> +            issue_type = mmc_cqe_issue_type(host, req);
>>> +            switch (issue_type) {
>>> +            case MMC_ISSUE_DCMD:
>>> +                if (mmc_cqe_dcmd_busy(mq)) {
>>> +                    mq->cqe_busy |= MMC_CQE_DCMD_BUSY;
>>> +                    req = NULL;
>>> +                    break;
>>> +                }
>>> +                /* Fall through */
>>> +            case MMC_ISSUE_ASYNC:
>>> +                if (blk_queue_start_tag(q, req)) {
>>> +                    mq->cqe_busy |= MMC_CQE_QUEUE_FULL;
>>> +                    req = NULL;
>>> +                }
>>> +                break;
>>> +            default:
>>> +                /*
>>> +                 * Timeouts are handled by mmc core, so set a
>>> +                 * large value to avoid races.
>>> +                 */
>>> +                req->timeout = 600 * HZ;
>>> +                blk_start_request(req);
>>> +                break;
>>> +            }
>>> +            if (req) {
>>> +                mq->cqe_in_flight[issue_type] += 1;
>>> +                if (mmc_cqe_tot_in_flight(mq) == 1)
>>> +                    get_put += 1;
>>> +                if (mmc_cqe_qcnt(mq) == 1)
>>> +                    retune_ok = true;
>>> +            }
>>> +        }
>>> +
>>
>> Just a thought that mmc_cq_thread is a little heavy for manage in-flight
>> request, so could we kick the check back to blk layer like this in the
>> prepare and unprepare hook?
> 
> I am not sure what you are aiming at.  The prepare function is called by
> blk_peek_request() so the work is anyway being done by the thread.  However
> the prepare function isn't always called, for example if the request has
> already been prepared and then later re-queued.  So it looks to me as though
> it won't work that way.

yes, prepare is called when calling blk_peek_request. I was trying to
simplify the code the mmc_cqe_thread looks like. yes, I missed the
re-queued case. Thanks for the explaination. :)

> 
>>
>>
>> --- a/drivers/mmc/core/queue.c
>> +++ b/drivers/mmc/core/queue.c
>> @@ -25,22 +25,59 @@
>>
>>   #define MMC_QUEUE_BOUNCESZ    65536
>>
>> +static inline bool mmc_cqe_dcmd_busy(struct mmc_queue *mq);
>> +static inline bool mmc_cqe_can_dcmd(struct mmc_host *host);
>> +enum mmc_issue_type mmc_cqe_issue_type(struct mmc_host *host,
>> +                       struct request *req);
>> +
>>   /*
>>    * Prepare a MMC request. This just filters out odd stuff.
>>    */
>>   static int mmc_prep_request(struct request_queue *q, struct request *req)
>>   {
>>       struct mmc_queue *mq = q->queuedata;
>> +    enum mmc_issue_type issue_type;
>>
>>       if (mq && (mmc_card_removed(mq->card) || mmc_access_rpmb(mq)))
>>           return BLKPREP_KILL;
>>
>> +    if (mq->card->host->cqe_enabled) {
>> +        issue_type = mmc_cqe_issue_type(mq->card->host, req);
>> +        if ((issue_type == MMC_ISSUE_DCMD && mmc_cqe_dcmd_busy(mq))) {
>> +            mq->cqe_busy |= MMC_CQE_DCMD_BUSY;
>> +            return BLKPREP_DEFER;
>> +        } else if (issue_type == MMC_ISSUE_ASYNC &&
>> +                   mq->cqe_in_flight[issue_type] >=
>> +                   mq->queue->queue_tags->max_depth) {
>> +            mq->cqe_busy |= MMC_CQE_QUEUE_FULL;
>> +            return BLKPREP_DEFER;
>> +        }
>> +    }
>> +
>>       req->rq_flags |= RQF_DONTPREP;
>>       req_to_mmc_queue_req(req)->retries = 0;
>>
>>       return BLKPREP_OK;
>>   }
>>
>> +static void mmc_unprep_request(struct request_queue *q, struct request *req)
>> +{
>> +    struct mmc_queue *mq = q->queuedata;
>> +    enum mmc_issue_type issue_type;
>> +
>> +    if (!mq->card->host->cqe_enabled)
>> +        return;
>> +
>> +    issue_type = mmc_cqe_issue_type(mq->card->host, req);
>> +    if (issue_type == MMC_ISSUE_DCMD &&
>> +        (mq->cqe_busy & MMC_CQE_DCMD_BUSY) &&
>> +        !mmc_cqe_dcmd_busy(mq))
>> +        mq->cqe_busy &= ~MMC_CQE_DCMD_BUSY;
>> +    else if (issue_type == MMC_ISSUE_ASYNC)
>> +        mq->cqe_busy &= ~MMC_CQE_QUEUE_FULL;
>> +}
>> +
>> +
>>   static void mmc_cqe_request_fn(struct request_queue *q)
>>   {
>>       struct mmc_queue *mq = q->queuedata;
>> @@ -66,11 +103,6 @@ static inline bool mmc_cqe_dcmd_busy(struct mmc_queue *mq)
>>
>>   void mmc_cqe_kick_queue(struct mmc_queue *mq)
>>   {
>> -    if ((mq->cqe_busy & MMC_CQE_DCMD_BUSY) && !mmc_cqe_dcmd_busy(mq))
>> -        mq->cqe_busy &= ~MMC_CQE_DCMD_BUSY;
>> -
>> -    mq->cqe_busy &= ~MMC_CQE_QUEUE_FULL;
>> -
>>       if (mq->asleep && !mq->cqe_busy)
>>           __blk_run_queue(mq->queue);
>>   }
>> @@ -151,21 +183,9 @@ static int mmc_cqe_thread(void *d)
>>
>>           if (req) {
>>               issue_type = mmc_cqe_issue_type(host, req);
>> -            switch (issue_type) {
>> -            case MMC_ISSUE_DCMD:
>> -                if (mmc_cqe_dcmd_busy(mq)) {
>> -                    mq->cqe_busy |= MMC_CQE_DCMD_BUSY;
>> -                    req = NULL;
>> -                    break;
>> -                }
>> -                /* Fall through */
>> -            case MMC_ISSUE_ASYNC:
>> -                if (blk_queue_start_tag(q, req)) {
>> -                    mq->cqe_busy |= MMC_CQE_QUEUE_FULL;
>> -                    req = NULL;
>> -                }
>> -                break;
>> -            default:
>> +            if (issue_type == MMC_ISSUE_DCMD || issue_type ==
>> MMC_ISSUE_ASYNC) {
>> +                blk_queue_start_tag(q, req);
>> +            } else {
>>                   /*
>>                    * Timeouts are handled by mmc core, so set a
>>                    * large value to avoid races.
>> @@ -174,13 +194,12 @@ static int mmc_cqe_thread(void *d)
>>                   blk_start_request(req);
>>                   break;
>>               }
>> -            if (req) {
>> -                mq->cqe_in_flight[issue_type] += 1;
>> -                if (mmc_cqe_tot_in_flight(mq) == 1)
>> -                    get_put += 1;
>> -                if (mmc_cqe_qcnt(mq) == 1)
>> -                    retune_ok = true;
>> -            }
>> +
>> +            mq->cqe_in_flight[issue_type] += 1;
>> +            if (mmc_cqe_tot_in_flight(mq) == 1)
>> +                get_put += 1;
>> +            if (mmc_cqe_qcnt(mq) == 1)
>> +                retune_ok = true;
>>           }
>>
>>           mq->asleep = !req;
>> @@ -523,6 +542,7 @@ int mmc_init_queue(struct mmc_queue *mq, struct mmc_card
>> *card,
>>       }
>>
>>       blk_queue_prep_rq(mq->queue, mmc_prep_request);
>> +    blk_queue_unprep_rq(mq->queue, mmc_unprep_request);
>>       queue_flag_set_unlocked(QUEUE_FLAG_NONROT, mq->queue);
>>       queue_flag_clear_unlocked(QUEUE_FLAG_ADD_RANDOM, mq->queue);
>>       if (mmc_can_erase(card))
>>
>>
>>
>>> +        mq->asleep = !req;
>>> +
>>> +        spin_unlock_irqrestore(q->queue_lock, flags);
>>> +
>>> +        if (req) {
>>> +            enum mmc_issued issued;
>>> +
>>> +            set_current_state(TASK_RUNNING);
>>> +
>>> +            if (get_put) {
>>> +                get_put = 0;
>>> +                mmc_get_card(card);
>>> +            }
>>> +
>>> +            if (host->need_retune && retune_ok &&
>>> +                !host->hold_retune)
>>> +                host->retune_now = true;
>>> +            else
>>> +                host->retune_now = false;
>>> +
>>> +            issued = mmc_blk_cqe_issue_rq(mq, req);
>>> +
>>> +            cond_resched();
>>> +
>>> +            spin_lock_irqsave(q->queue_lock, flags);
>>> +
>>> +            switch (issued) {
>>> +            case MMC_REQ_STARTED:
>>> +                break;
>>> +            case MMC_REQ_BUSY:
>>> +                blk_requeue_request(q, req);
>>> +                goto finished;
>>> +            case MMC_REQ_FAILED_TO_START:
>>> +                __blk_end_request_all(req, BLK_STS_IOERR);
>>> +                /* Fall through */
>>> +            case MMC_REQ_FINISHED:
>>> +finished:
>>> +                mq->cqe_in_flight[issue_type] -= 1;
>>> +                if (mmc_cqe_tot_in_flight(mq) == 0)
>>> +                    get_put = -1;
>>> +            }
>>> +        } else {
>>> +            if (get_put < 0) {
>>> +                get_put = 0;
>>> +                mmc_put_card(card);
>>> +            }
>>> +            /*
>>> +             * Do not stop with requests in flight in case recovery
>>> +             * is needed.
>>> +             */
>>> +            if (kthread_should_stop() &&
>>> +                !mmc_cqe_tot_in_flight(mq)) {
>>> +                set_current_state(TASK_RUNNING);
>>> +                break;
>>> +            }
>>> +            up(&mq->thread_sem);
>>> +            schedule();
>>> +            down(&mq->thread_sem);
>>> +            spin_lock_irqsave(q->queue_lock, flags);
>>> +        }
>>> +    } /* loop */
>>> +    up(&mq->thread_sem);
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static enum blk_eh_timer_return __mmc_cqe_timed_out(struct request *req)
>>> +{
>>> +    struct mmc_queue_req *mqrq = req_to_mmc_queue_req(req);
>>> +    struct mmc_request *mrq = &mqrq->brq.mrq;
>>> +    struct mmc_queue *mq = req->q->queuedata;
>>> +    struct mmc_host *host = mq->card->host;
>>> +    enum mmc_issue_type issue_type = mmc_cqe_issue_type(host, req);
>>> +    bool recovery_needed = false;
>>> +
>>> +    switch (issue_type) {
>>> +    case MMC_ISSUE_ASYNC:
>>> +    case MMC_ISSUE_DCMD:
>>> +        if (host->cqe_ops->cqe_timeout(host, mrq, &recovery_needed)) {
>>> +            if (recovery_needed)
>>> +                __mmc_cqe_recovery_notifier(mq);
>>> +            return BLK_EH_RESET_TIMER;
>>> +        }
>>> +        /* No timeout */
>>> +        return BLK_EH_HANDLED;
>>> +    default:
>>> +        /* Timeout is handled by mmc core */
>>> +        return BLK_EH_RESET_TIMER;
>>> +    }
>>> +}
>>> +
>>> +static enum blk_eh_timer_return mmc_cqe_timed_out(struct request *req)
>>> +{
>>> +    struct mmc_queue *mq = req->q->queuedata;
>>> +
>>> +    if (mq->cqe_recovery_needed)
>>> +        return BLK_EH_RESET_TIMER;
>>> +
>>> +    return __mmc_cqe_timed_out(req);
>>> +}
>>> +
>>>    static int mmc_queue_thread(void *d)
>>>    {
>>>        struct mmc_queue *mq = d;
>>> @@ -233,11 +477,12 @@ static void mmc_exit_request(struct request_queue
>>> *q, struct request *req)
>>>     * Initialise a MMC card request queue.
>>>     */
>>>    int mmc_init_queue(struct mmc_queue *mq, struct mmc_card *card,
>>> -           spinlock_t *lock, const char *subname)
>>> +           spinlock_t *lock, const char *subname, int area_type)
>>>    {
>>>        struct mmc_host *host = card->host;
>>>        u64 limit = BLK_BOUNCE_HIGH;
>>>        int ret = -ENOMEM;
>>> +    bool use_cqe = host->cqe_enabled && area_type != MMC_BLK_DATA_AREA_RPMB;
>>>          if (mmc_dev(host)->dma_mask && *mmc_dev(host)->dma_mask)
>>>            limit = (u64)dma_max_pfn(mmc_dev(host)) << PAGE_SHIFT;
>>> @@ -247,7 +492,7 @@ int mmc_init_queue(struct mmc_queue *mq, struct
>>> mmc_card *card,
>>>        if (!mq->queue)
>>>            return -ENOMEM;
>>>        mq->queue->queue_lock = lock;
>>> -    mq->queue->request_fn = mmc_request_fn;
>>> +    mq->queue->request_fn = use_cqe ? mmc_cqe_request_fn : mmc_request_fn;
>>>        mq->queue->init_rq_fn = mmc_init_request;
>>>        mq->queue->exit_rq_fn = mmc_exit_request;
>>>        mq->queue->cmd_size = sizeof(struct mmc_queue_req);
>>> @@ -259,6 +504,24 @@ int mmc_init_queue(struct mmc_queue *mq, struct
>>> mmc_card *card,
>>>            return ret;
>>>        }
>>>    +    if (use_cqe) {
>>> +        int q_depth = card->ext_csd.cmdq_depth;
>>> +
>>> +        if (q_depth > host->cqe_qdepth)
>>> +            q_depth = host->cqe_qdepth;
>>> +
>>> +        ret = blk_queue_init_tags(mq->queue, q_depth, NULL,
>>> +                      BLK_TAG_ALLOC_FIFO);
>>> +        if (ret)
>>> +            goto cleanup_queue;
>>> +
>>> +        blk_queue_softirq_done(mq->queue, mmc_blk_cqe_complete_rq);
>>> +        blk_queue_rq_timed_out(mq->queue, mmc_cqe_timed_out);
>>> +        blk_queue_rq_timeout(mq->queue, 60 * HZ);
>>> +
>>> +        host->cqe_recovery_notifier = mmc_cqe_recovery_notifier;
>>> +    }
>>> +
>>>        blk_queue_prep_rq(mq->queue, mmc_prep_request);
>>>        queue_flag_set_unlocked(QUEUE_FLAG_NONROT, mq->queue);
>>>        queue_flag_clear_unlocked(QUEUE_FLAG_ADD_RANDOM, mq->queue);
>>> @@ -280,9 +543,9 @@ int mmc_init_queue(struct mmc_queue *mq, struct
>>> mmc_card *card,
>>>          sema_init(&mq->thread_sem, 1);
>>>    -    mq->thread = kthread_run(mmc_queue_thread, mq, "mmcqd/%d%s",
>>> -        host->index, subname ? subname : "");
>>> -
>>> +    mq->thread = kthread_run(use_cqe ? mmc_cqe_thread : mmc_queue_thread,
>>> +                 mq, "mmcqd/%d%s", host->index,
>>> +                 subname ? subname : "");
>>>        if (IS_ERR(mq->thread)) {
>>>            ret = PTR_ERR(mq->thread);
>>>            goto cleanup_queue;
>>> diff --git a/drivers/mmc/core/queue.h b/drivers/mmc/core/queue.h
>>> index 361b46408e0f..8e9273d977c0 100644
>>> --- a/drivers/mmc/core/queue.h
>>> +++ b/drivers/mmc/core/queue.h
>>> @@ -7,6 +7,20 @@
>>>    #include <linux/mmc/core.h>
>>>    #include <linux/mmc/host.h>
>>>    +enum mmc_issued {
>>> +    MMC_REQ_STARTED,
>>> +    MMC_REQ_BUSY,
>>> +    MMC_REQ_FAILED_TO_START,
>>> +    MMC_REQ_FINISHED,
>>> +};
>>> +
>>> +enum mmc_issue_type {
>>> +    MMC_ISSUE_SYNC,
>>> +    MMC_ISSUE_DCMD,
>>> +    MMC_ISSUE_ASYNC,
>>> +    MMC_ISSUE_MAX,
>>> +};
>>> +
>>>    static inline struct mmc_queue_req *req_to_mmc_queue_req(struct request
>>> *rq)
>>>    {
>>>        return blk_mq_rq_to_pdu(rq);
>>> @@ -53,6 +67,7 @@ struct mmc_queue_req {
>>>        int            drv_op_result;
>>>        struct mmc_blk_ioc_data    **idata;
>>>        unsigned int        ioc_count;
>>> +    int            retries;
>>>    };
>>>      struct mmc_queue {
>>> @@ -70,10 +85,17 @@ struct mmc_queue {
>>>         * associated mmc_queue_req data.
>>>         */
>>>        int            qcnt;
>>> +    /* Following are defined for a Command Queue Engine */
>>> +    int            cqe_in_flight[MMC_ISSUE_MAX];
>>> +    unsigned int        cqe_busy;
>>> +    bool            cqe_recovery_needed;
>>> +    bool            cqe_in_recovery;
>>> +#define MMC_CQE_DCMD_BUSY    BIT(0)
>>> +#define MMC_CQE_QUEUE_FULL    BIT(1)
>>>    };
>>>      extern int mmc_init_queue(struct mmc_queue *, struct mmc_card *,
>>> spinlock_t *,
>>> -              const char *);
>>> +              const char *, int);
>>>    extern void mmc_cleanup_queue(struct mmc_queue *);
>>>    extern void mmc_queue_suspend(struct mmc_queue *);
>>>    extern void mmc_queue_resume(struct mmc_queue *);
>>> @@ -85,4 +107,22 @@ extern unsigned int mmc_queue_map_sg(struct mmc_queue *,
>>>      extern int mmc_access_rpmb(struct mmc_queue *);
>>>    +void mmc_cqe_kick_queue(struct mmc_queue *mq);
>>> +
>>> +enum mmc_issue_type mmc_cqe_issue_type(struct mmc_host *host,
>>> +                       struct request *req);
>>> +
>>> +static inline int mmc_cqe_tot_in_flight(struct mmc_queue *mq)
>>> +{
>>> +    return mq->cqe_in_flight[MMC_ISSUE_SYNC] +
>>> +           mq->cqe_in_flight[MMC_ISSUE_DCMD] +
>>> +           mq->cqe_in_flight[MMC_ISSUE_ASYNC];
>>> +}
>>> +
>>> +static inline int mmc_cqe_qcnt(struct mmc_queue *mq)
>>> +{
>>> +    return mq->cqe_in_flight[MMC_ISSUE_DCMD] +
>>> +           mq->cqe_in_flight[MMC_ISSUE_ASYNC];
>>> +}
>>> +
>>>    #endif
>>>
>>
>>
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-mmc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
> 

Re: [PATCH V4 00/11] mmc: Add Command Queue support

[Shawn Lin]

Hi Adrian,

On 2017/7/21 17:49, Adrian Hunter wrote:
> Hi
> 
> Here is V4 of the hardware command queue patches without the software
> command queue patches.
> 
> 

After looking into the changes in details for some days, it looks
awsome to me. And my boards are still alive after more than one week's
test. Can't wait to enable it for 4.14 :)

Reviewed-by: Shawn Lin <shawn.lin@rock-chips.com>
Tested-by: Shawn Lin <shawn.lin@rock-chips.com>

> Changes since V3:
> 	Adjusted ...blk_end_request...() for new block status codes
> 	Fixed CQHCI transaction descriptor for "no DCMD" case
> 
> Changes since V2:
> 	Dropped patches that have been applied.
> 	Re-based
> 	Added "mmc: sdhci-pci: Add CQHCI support for Intel GLK"
> 
> Changes since V1:
> 
> 	"Share mmc request array between partitions" is dependent
> 	on changes in "Introduce queue semantics", so added that
> 	and block fixes:
> 
> 	Added "Fix is_waiting_last_req set incorrectly"
> 	Added "Fix cmd error reset failure path"
> 	Added "Use local var for mqrq_cur"
> 	Added "Introduce queue semantics"
> 
> Changes since RFC:
> 
> 	Re-based on next.
> 	Added comment about command queue priority.
> 	Added some acks and reviews.
> 
> 
> Adrian Hunter (10):
>        mmc: core: Add mmc_retune_hold_now()
>        mmc: core: Add members to mmc_request and mmc_data for CQE's
>        mmc: host: Add CQE interface
>        mmc: core: Turn off CQE before sending commands
>        mmc: core: Add support for handling CQE requests
>        mmc: mmc: Enable Command Queuing
>        mmc: mmc: Enable CQE's
>        mmc: block: Prepare CQE data
>        mmc: block: Add CQE support
>        mmc: sdhci-pci: Add CQHCI support for Intel GLK
> 
> Venkat Gopalakrishnan (1):
>        mmc: cqhci: support for command queue enabled host
> 
>   drivers/mmc/core/block.c          |  240 +++++++-
>   drivers/mmc/core/block.h          |    7 +
>   drivers/mmc/core/bus.c            |    7 +
>   drivers/mmc/core/core.c           |  156 ++++-
>   drivers/mmc/core/host.c           |    6 +
>   drivers/mmc/core/host.h           |    1 +
>   drivers/mmc/core/mmc.c            |   30 +-
>   drivers/mmc/core/queue.c          |  273 ++++++++-
>   drivers/mmc/core/queue.h          |   42 +-
>   drivers/mmc/host/Kconfig          |   14 +
>   drivers/mmc/host/Makefile         |    1 +
>   drivers/mmc/host/cqhci.c          | 1146 +++++++++++++++++++++++++++++++++++++
>   drivers/mmc/host/cqhci.h          |  240 ++++++++
>   drivers/mmc/host/sdhci-pci-core.c |  153 ++++-
>   include/linux/mmc/core.h          |   18 +-
>   include/linux/mmc/host.h          |   24 +
>   include/trace/events/mmc.h        |   36 +-
>   17 files changed, 2354 insertions(+), 40 deletions(-)
>   create mode 100644 drivers/mmc/host/cqhci.c
>   create mode 100644 drivers/mmc/host/cqhci.h
> 
> 
> Regards
> Adrian
> 
> 
> 

Re: [PATCH V4 00/11] mmc: Add Command Queue support

[Ulf Hansson]

On 21 July 2017 at 11:49, Adrian Hunter <adrian.hunter@intel.com> wrote:
> Hi
>
> Here is V4 of the hardware command queue patches without the software
> command queue patches.

Adrian, again apologize for the delay.

I am reviewing the series now and my intent is to provide comments on
each change separately during the week.

However, a couple of overall thoughts:

*) Could you please post some fresh performance measurements, such we
can get some real proof on why CMDQ is worth to be merged? If not
increased throughput, perhaps we can show some decreased I/O requests
latency!?

**) I have spoken to Linus W offlist - and he is still working on the
blkmq port. Although we first need to continue with the
re-factorization of the mmc block/core code, to minimize the bad
impact of our big mmc claim host lock. We should expect a
re-submission of his series quite soonish. However, it's also likely
that we need yet another round of re-factoring, before we can complete
the port to blkmq.

***) The reason for bringing up **), is of course because I think
deploying CMDQ support would be better done on top of the blkmq
interface as it's better suited for these kind of device types. My
goal is to reach a better maintenance situation, using more modern mmc
block code. We have spoken about this before, however of course I also
don't want to delay the CMDQ series. Anyway, let's move forward and
see what path we end up taking.

>
>
> Changes since V3:
>         Adjusted ...blk_end_request...() for new block status codes
>         Fixed CQHCI transaction descriptor for "no DCMD" case
>
> Changes since V2:
>         Dropped patches that have been applied.
>         Re-based
>         Added "mmc: sdhci-pci: Add CQHCI support for Intel GLK"
>
> Changes since V1:
>
>         "Share mmc request array between partitions" is dependent
>         on changes in "Introduce queue semantics", so added that
>         and block fixes:
>
>         Added "Fix is_waiting_last_req set incorrectly"
>         Added "Fix cmd error reset failure path"
>         Added "Use local var for mqrq_cur"
>         Added "Introduce queue semantics"
>
> Changes since RFC:
>
>         Re-based on next.
>         Added comment about command queue priority.
>         Added some acks and reviews.
>
>
> Adrian Hunter (10):
>       mmc: core: Add mmc_retune_hold_now()
>       mmc: core: Add members to mmc_request and mmc_data for CQE's
>       mmc: host: Add CQE interface
>       mmc: core: Turn off CQE before sending commands
>       mmc: core: Add support for handling CQE requests
>       mmc: mmc: Enable Command Queuing
>       mmc: mmc: Enable CQE's
>       mmc: block: Prepare CQE data
>       mmc: block: Add CQE support
>       mmc: sdhci-pci: Add CQHCI support for Intel GLK
>
> Venkat Gopalakrishnan (1):
>       mmc: cqhci: support for command queue enabled host
>
>  drivers/mmc/core/block.c          |  240 +++++++-
>  drivers/mmc/core/block.h          |    7 +
>  drivers/mmc/core/bus.c            |    7 +
>  drivers/mmc/core/core.c           |  156 ++++-
>  drivers/mmc/core/host.c           |    6 +
>  drivers/mmc/core/host.h           |    1 +
>  drivers/mmc/core/mmc.c            |   30 +-
>  drivers/mmc/core/queue.c          |  273 ++++++++-
>  drivers/mmc/core/queue.h          |   42 +-
>  drivers/mmc/host/Kconfig          |   14 +
>  drivers/mmc/host/Makefile         |    1 +
>  drivers/mmc/host/cqhci.c          | 1146 +++++++++++++++++++++++++++++++++++++
>  drivers/mmc/host/cqhci.h          |  240 ++++++++
>  drivers/mmc/host/sdhci-pci-core.c |  153 ++++-
>  include/linux/mmc/core.h          |   18 +-
>  include/linux/mmc/host.h          |   24 +
>  include/trace/events/mmc.h        |   36 +-
>  17 files changed, 2354 insertions(+), 40 deletions(-)
>  create mode 100644 drivers/mmc/host/cqhci.c
>  create mode 100644 drivers/mmc/host/cqhci.h
>
>
> Regards
> Adrian

Kind regards
Uffe

Re: [PATCH V4 01/11] mmc: core: Add mmc_retune_hold_now()

[Ulf Hansson]

On 21 July 2017 at 11:49, Adrian Hunter <adrian.hunter@intel.com> wrote:
> In preparation for CQE support.

Can you please elaborate a bit on this. Perhaps explaining when/why a
host needs to call this new host API.

>
> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
> ---
>  drivers/mmc/core/host.c | 6 ++++++
>  drivers/mmc/core/host.h | 1 +
>  2 files changed, 7 insertions(+)
>
> diff --git a/drivers/mmc/core/host.c b/drivers/mmc/core/host.c
> index 1503412f826c..ad88deb2e8f3 100644
> --- a/drivers/mmc/core/host.c
> +++ b/drivers/mmc/core/host.c
> @@ -111,6 +111,12 @@ void mmc_retune_hold(struct mmc_host *host)
>         host->hold_retune += 1;
>  }
>
> +void mmc_retune_hold_now(struct mmc_host *host)
> +{
> +       host->retune_now = 0;
> +       host->hold_retune += 1;
> +}
> +
>  void mmc_retune_release(struct mmc_host *host)
>  {
>         if (host->hold_retune)
> diff --git a/drivers/mmc/core/host.h b/drivers/mmc/core/host.h
> index fb6a76a03833..77d6f60d1bf9 100644
> --- a/drivers/mmc/core/host.h
> +++ b/drivers/mmc/core/host.h
> @@ -19,6 +19,7 @@
>  void mmc_retune_enable(struct mmc_host *host);
>  void mmc_retune_disable(struct mmc_host *host);
>  void mmc_retune_hold(struct mmc_host *host);
> +void mmc_retune_hold_now(struct mmc_host *host);
>  void mmc_retune_release(struct mmc_host *host);
>  int mmc_retune(struct mmc_host *host);
>  void mmc_retune_pause(struct mmc_host *host);
> --
> 1.9.1
>

Kind regards
Uffe

Re: [PATCH V4 02/11] mmc: core: Add members to mmc_request and mmc_data for CQE's

[Ulf Hansson]

On 21 July 2017 at 11:49, Adrian Hunter <adrian.hunter@intel.com> wrote:
> Most of the information needed to issue requests to a CQE is already in
> struct mmc_request and struct mmc_data. Add data block address, some flags,
> and the task id (tag), and allow for cmd being NULL which it is for CQE
> tasks.
>
> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
> ---
>  include/linux/mmc/core.h   | 13 +++++++++++--
>  include/trace/events/mmc.h | 36 +++++++++++++++++++++++-------------
>  2 files changed, 34 insertions(+), 15 deletions(-)
>
> diff --git a/include/linux/mmc/core.h b/include/linux/mmc/core.h
> index a0c63ea28796..bf1788a224e6 100644
> --- a/include/linux/mmc/core.h
> +++ b/include/linux/mmc/core.h
> @@ -122,11 +122,18 @@ struct mmc_data {
>         unsigned int            timeout_clks;   /* data timeout (in clocks) */
>         unsigned int            blksz;          /* data block size */
>         unsigned int            blocks;         /* number of blocks */
> +       unsigned int            blk_addr;       /* block address */
>         int                     error;          /* data error */
>         unsigned int            flags;
>
> -#define MMC_DATA_WRITE (1 << 8)
> -#define MMC_DATA_READ  (1 << 9)
> +#define MMC_DATA_WRITE         BIT(8)
> +#define MMC_DATA_READ          BIT(9)
> +/* Extra flags used by CQE */
> +#define MMC_DATA_QBR           BIT(10)         /* CQE queue barrier*/
> +#define MMC_DATA_PRIO          BIT(11)         /* CQE high priority */

Regarding high priority data requests. I think this can be debated a
bit on how it should be implemented.

If possible, may I suggest we move adding the support for this to the
end of the series? In that way we can discuss it separately and it
won't block us moving forward.

> +#define MMC_DATA_REL_WR                BIT(12)         /* Reliable write */

What has this bit to do with CQE?

> +#define MMC_DATA_DAT_TAG       BIT(13)         /* Tag request */
> +#define MMC_DATA_FORCED_PRG    BIT(14)         /* Forced programming */
>
>         unsigned int            bytes_xfered;
>
> @@ -153,6 +160,8 @@ struct mmc_request {
>
>         /* Allow other commands during this ongoing data transfer or busy wait */
>         bool                    cap_cmd_during_tfr;
> +
> +       int                     tag;
>  };
>
>  struct mmc_card;
> diff --git a/include/trace/events/mmc.h b/include/trace/events/mmc.h
> index a72f9b94c80b..f30a99ac65b6 100644
> --- a/include/trace/events/mmc.h
> +++ b/include/trace/events/mmc.h
> @@ -29,8 +29,10 @@
>                 __field(unsigned int,           sbc_flags)
>                 __field(unsigned int,           sbc_retries)
>                 __field(unsigned int,           blocks)
> +               __field(unsigned int,           blk_addr)
>                 __field(unsigned int,           blksz)
>                 __field(unsigned int,           data_flags)
> +               __field(int,                    tag)
>                 __field(unsigned int,           can_retune)
>                 __field(unsigned int,           doing_retune)
>                 __field(unsigned int,           retune_now)
> @@ -42,10 +44,10 @@
>         ),
>
>         TP_fast_assign(
> -               __entry->cmd_opcode = mrq->cmd->opcode;
> -               __entry->cmd_arg = mrq->cmd->arg;
> -               __entry->cmd_flags = mrq->cmd->flags;
> -               __entry->cmd_retries = mrq->cmd->retries;
> +               __entry->cmd_opcode = mrq->cmd ? mrq->cmd->opcode : 0;
> +               __entry->cmd_arg = mrq->cmd ? mrq->cmd->arg : 0;
> +               __entry->cmd_flags = mrq->cmd ? mrq->cmd->flags : 0;
> +               __entry->cmd_retries = mrq->cmd ? mrq->cmd->retries : 0;
>                 __entry->stop_opcode = mrq->stop ? mrq->stop->opcode : 0;
>                 __entry->stop_arg = mrq->stop ? mrq->stop->arg : 0;
>                 __entry->stop_flags = mrq->stop ? mrq->stop->flags : 0;
> @@ -56,7 +58,9 @@
>                 __entry->sbc_retries = mrq->sbc ? mrq->sbc->retries : 0;
>                 __entry->blksz = mrq->data ? mrq->data->blksz : 0;
>                 __entry->blocks = mrq->data ? mrq->data->blocks : 0;
> +               __entry->blk_addr = mrq->data ? mrq->data->blk_addr : 0;
>                 __entry->data_flags = mrq->data ? mrq->data->flags : 0;
> +               __entry->tag = mrq->tag;
>                 __entry->can_retune = host->can_retune;
>                 __entry->doing_retune = host->doing_retune;
>                 __entry->retune_now = host->retune_now;
> @@ -71,8 +75,8 @@
>                   "cmd_opcode=%u cmd_arg=0x%x cmd_flags=0x%x cmd_retries=%u "
>                   "stop_opcode=%u stop_arg=0x%x stop_flags=0x%x stop_retries=%u "
>                   "sbc_opcode=%u sbc_arg=0x%x sbc_flags=0x%x sbc_retires=%u "
> -                 "blocks=%u block_size=%u data_flags=0x%x "
> -                 "can_retune=%u doing_retune=%u retune_now=%u "
> +                 "blocks=%u block_size=%u blk_addr=%u data_flags=0x%x "
> +                 "tag=%d can_retune=%u doing_retune=%u retune_now=%u "
>                   "need_retune=%d hold_retune=%d retune_period=%u",
>                   __get_str(name), __entry->mrq,
>                   __entry->cmd_opcode, __entry->cmd_arg,
> @@ -81,7 +85,8 @@
>                   __entry->stop_flags, __entry->stop_retries,
>                   __entry->sbc_opcode, __entry->sbc_arg,
>                   __entry->sbc_flags, __entry->sbc_retries,
> -                 __entry->blocks, __entry->blksz, __entry->data_flags,
> +                 __entry->blocks, __entry->blk_addr,
> +                 __entry->blksz, __entry->data_flags, __entry->tag,
>                   __entry->can_retune, __entry->doing_retune,
>                   __entry->retune_now, __entry->need_retune,
>                   __entry->hold_retune, __entry->retune_period)
> @@ -108,6 +113,7 @@
>                 __field(unsigned int,           sbc_retries)
>                 __field(unsigned int,           bytes_xfered)
>                 __field(int,                    data_err)
> +               __field(int,                    tag)
>                 __field(unsigned int,           can_retune)
>                 __field(unsigned int,           doing_retune)
>                 __field(unsigned int,           retune_now)
> @@ -119,10 +125,13 @@
>         ),
>
>         TP_fast_assign(
> -               __entry->cmd_opcode = mrq->cmd->opcode;
> -               __entry->cmd_err = mrq->cmd->error;
> -               memcpy(__entry->cmd_resp, mrq->cmd->resp, 4);
> -               __entry->cmd_retries = mrq->cmd->retries;
> +               __entry->cmd_opcode = mrq->cmd ? mrq->cmd->opcode : 0;
> +               __entry->cmd_err = mrq->cmd ? mrq->cmd->error : 0;
> +               __entry->cmd_resp[0] = mrq->cmd ? mrq->cmd->resp[0] : 0;
> +               __entry->cmd_resp[1] = mrq->cmd ? mrq->cmd->resp[1] : 0;
> +               __entry->cmd_resp[2] = mrq->cmd ? mrq->cmd->resp[2] : 0;
> +               __entry->cmd_resp[3] = mrq->cmd ? mrq->cmd->resp[3] : 0;
> +               __entry->cmd_retries = mrq->cmd ? mrq->cmd->retries : 0;
>                 __entry->stop_opcode = mrq->stop ? mrq->stop->opcode : 0;
>                 __entry->stop_err = mrq->stop ? mrq->stop->error : 0;
>                 __entry->stop_resp[0] = mrq->stop ? mrq->stop->resp[0] : 0;
> @@ -139,6 +148,7 @@
>                 __entry->sbc_retries = mrq->sbc ? mrq->sbc->retries : 0;
>                 __entry->bytes_xfered = mrq->data ? mrq->data->bytes_xfered : 0;
>                 __entry->data_err = mrq->data ? mrq->data->error : 0;
> +               __entry->tag = mrq->tag;
>                 __entry->can_retune = host->can_retune;
>                 __entry->doing_retune = host->doing_retune;
>                 __entry->retune_now = host->retune_now;
> @@ -154,7 +164,7 @@
>                   "cmd_retries=%u stop_opcode=%u stop_err=%d "
>                   "stop_resp=0x%x 0x%x 0x%x 0x%x stop_retries=%u "
>                   "sbc_opcode=%u sbc_err=%d sbc_resp=0x%x 0x%x 0x%x 0x%x "
> -                 "sbc_retries=%u bytes_xfered=%u data_err=%d "
> +                 "sbc_retries=%u bytes_xfered=%u data_err=%d tag=%d "
>                   "can_retune=%u doing_retune=%u retune_now=%u need_retune=%d "
>                   "hold_retune=%d retune_period=%u",
>                   __get_str(name), __entry->mrq,
> @@ -170,7 +180,7 @@
>                   __entry->sbc_resp[0], __entry->sbc_resp[1],
>                   __entry->sbc_resp[2], __entry->sbc_resp[3],
>                   __entry->sbc_retries,
> -                 __entry->bytes_xfered, __entry->data_err,
> +                 __entry->bytes_xfered, __entry->data_err, __entry->tag,
>                   __entry->can_retune, __entry->doing_retune,
>                   __entry->retune_now, __entry->need_retune,
>                   __entry->hold_retune, __entry->retune_period)
> --
> 1.9.1
>

Kind regards
Uffe

Re: [PATCH V4 03/11] mmc: host: Add CQE interface

[Ulf Hansson]

On 21 July 2017 at 11:49, Adrian Hunter <adrian.hunter@intel.com> wrote:
> Add CQE host operations, capabilities, and host members.

I think adding these new interfaces deserves a bit more descriptive changelog.

>
> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
> ---
>  include/linux/mmc/host.h | 24 ++++++++++++++++++++++++
>  1 file changed, 24 insertions(+)
>
> diff --git a/include/linux/mmc/host.h b/include/linux/mmc/host.h
> index ebd1cebbef0c..4dd7ada9b4b9 100644
> --- a/include/linux/mmc/host.h
> +++ b/include/linux/mmc/host.h
> @@ -162,6 +162,19 @@ struct mmc_host_ops {
>                                   unsigned int direction, int blk_size);
>  };
>
> +struct mmc_cqe_ops {

Could you please add a small comment to each new callback. Even if the
names gives a good hint on they are used, some additional information
would be nice.

> +       int     (*cqe_enable)(struct mmc_host *host, struct mmc_card *card);
> +       void    (*cqe_disable)(struct mmc_host *host);
> +       int     (*cqe_request)(struct mmc_host *host, struct mmc_request *mrq);
> +       void    (*cqe_post_req)(struct mmc_host *host, struct mmc_request *mrq);
> +       void    (*cqe_off)(struct mmc_host *host);

What differs between ->cqe_off() and ->cqe_disable()? Do we need both?

> +       int     (*cqe_wait_for_idle)(struct mmc_host *host);

The name sounds like you will poll the host to understand when the
card becomes idle.

Is that so? Then when is this needed?

> +       bool    (*cqe_timeout)(struct mmc_host *host, struct mmc_request *mrq,
> +                              bool *recovery_needed);
> +       void    (*cqe_recovery_start)(struct mmc_host *host);
> +       void    (*cqe_recovery_finish)(struct mmc_host *host);
> +};
> +
>  struct mmc_async_req {
>         /* active mmc request */
>         struct mmc_request      *mrq;
> @@ -307,6 +320,8 @@ struct mmc_host {
>  #define MMC_CAP2_HS400_ES      (1 << 20)       /* Host supports enhanced strobe */
>  #define MMC_CAP2_NO_SD         (1 << 21)       /* Do not send SD commands during initialization */
>  #define MMC_CAP2_NO_MMC                (1 << 22)       /* Do not send (e)MMC commands during initialization */
> +#define MMC_CAP2_CQE           (1 << 23)       /* Has eMMC command queue engine */
> +#define MMC_CAP2_CQE_DCMD      (1 << 24)       /* CQE can issue a direct command */
>
>         mmc_pm_flag_t           pm_caps;        /* supported pm features */
>
> @@ -393,6 +408,15 @@ struct mmc_host {
>         int                     dsr_req;        /* DSR value is valid */
>         u32                     dsr;    /* optional driver stage (DSR) value */
>
> +       /* Command Queue Engine (CQE) support */
> +       const struct mmc_cqe_ops *cqe_ops;
> +       void                    *cqe_private;
> +       void                    (*cqe_recovery_notifier)(struct mmc_host *,
> +                                                        struct mmc_request *);

Please add a comment here to understand how this callback is going to be used.

> +       int                     cqe_qdepth;
> +       bool                    cqe_enabled;
> +       bool                    cqe_on;
> +
>         unsigned long           private[0] ____cacheline_aligned;
>  };
>
> --
> 1.9.1
>

Kind regards
Uffe

Re: [PATCH V4 04/11] mmc: core: Turn off CQE before sending commands

[Ulf Hansson]

On 21 July 2017 at 11:49, Adrian Hunter <adrian.hunter@intel.com> wrote:
> Turn off the CQE before sending commands, and ensure it is off in any reset
> or power management paths, or re-tuning.

Please explain a bit why this is needed.

Also, it's not clear how it becomes re-enabled. Or it doesn't have to?

>
> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
> ---
>  drivers/mmc/core/core.c | 9 +++++++++
>  1 file changed, 9 insertions(+)
>
> diff --git a/drivers/mmc/core/core.c b/drivers/mmc/core/core.c
> index 26431267a3e2..b0af9db18eef 100644
> --- a/drivers/mmc/core/core.c
> +++ b/drivers/mmc/core/core.c
> @@ -260,6 +260,9 @@ static void __mmc_start_request(struct mmc_host *host, struct mmc_request *mrq)
>
>         trace_mmc_request_start(host, mrq);
>
> +       if (host->cqe_on)
> +               host->cqe_ops->cqe_off(host);
> +

Why can't you use the ->cqe_disable() callback here instead?

I guess what puzzles me is that there is no corresponding ->cqe_on()
callback. When does cqe gets turned on again?

>         host->ops->request(host, mrq);
>  }
>
> @@ -982,6 +985,9 @@ int mmc_execute_tuning(struct mmc_card *card)
>         if (!host->ops->execute_tuning)
>                 return 0;
>
> +       if (host->cqe_on)
> +               host->cqe_ops->cqe_off(host);
> +
>         if (mmc_card_mmc(card))
>                 opcode = MMC_SEND_TUNING_BLOCK_HS200;
>         else
> @@ -1021,6 +1027,9 @@ void mmc_set_bus_width(struct mmc_host *host, unsigned int width)
>   */
>  void mmc_set_initial_state(struct mmc_host *host)
>  {
> +       if (host->cqe_on)
> +               host->cqe_ops->cqe_off(host);
> +
>         mmc_retune_disable(host);
>
>         if (mmc_host_is_spi(host))
> --
> 1.9.1
>

Kind regards
Uffe

Re: [PATCH V4 05/11] mmc: core: Add support for handling CQE requests

[Ulf Hansson]

On 21 July 2017 at 11:49, Adrian Hunter <adrian.hunter@intel.com> wrote:
> Add core support for handling CQE requests, including starting, completing
> and recovering.
>
> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
> ---
>  drivers/mmc/core/core.c  | 147 +++++++++++++++++++++++++++++++++++++++++++++--
>  include/linux/mmc/core.h |   5 ++
>  2 files changed, 147 insertions(+), 5 deletions(-)
>
> diff --git a/drivers/mmc/core/core.c b/drivers/mmc/core/core.c
> index b0af9db18eef..5a9d837599a1 100644
> --- a/drivers/mmc/core/core.c
> +++ b/drivers/mmc/core/core.c
> @@ -266,7 +266,8 @@ static void __mmc_start_request(struct mmc_host *host, struct mmc_request *mrq)
>         host->ops->request(host, mrq);
>  }
>
> -static void mmc_mrq_pr_debug(struct mmc_host *host, struct mmc_request *mrq)
> +static void mmc_mrq_pr_debug(struct mmc_host *host, struct mmc_request *mrq,
> +                            bool cqe)
>  {
>         if (mrq->sbc) {
>                 pr_debug("<%s: starting CMD%u arg %08x flags %08x>\n",
> @@ -275,9 +276,12 @@ static void mmc_mrq_pr_debug(struct mmc_host *host, struct mmc_request *mrq)
>         }
>
>         if (mrq->cmd) {
> -               pr_debug("%s: starting CMD%u arg %08x flags %08x\n",
> -                        mmc_hostname(host), mrq->cmd->opcode, mrq->cmd->arg,
> -                        mrq->cmd->flags);
> +               pr_debug("%s: starting %sCMD%u arg %08x flags %08x\n",
> +                        mmc_hostname(host), cqe ? "CQE direct " : "",
> +                        mrq->cmd->opcode, mrq->cmd->arg, mrq->cmd->flags);
> +       } else if (cqe) {
> +               pr_debug("%s: starting CQE transfer for tag %d blkaddr %u\n",
> +                        mmc_hostname(host), mrq->tag, mrq->data->blk_addr);
>         }
>
>         if (mrq->data) {
> @@ -345,7 +349,7 @@ static int mmc_start_request(struct mmc_host *host, struct mmc_request *mrq)
>         if (mmc_card_removed(host->card))
>                 return -ENOMEDIUM;
>
> -       mmc_mrq_pr_debug(host, mrq);
> +       mmc_mrq_pr_debug(host, mrq, false);
>
>         WARN_ON(!host->claimed);
>
> @@ -485,6 +489,139 @@ void mmc_wait_for_req_done(struct mmc_host *host, struct mmc_request *mrq)
>  }
>  EXPORT_SYMBOL(mmc_wait_for_req_done);
>

As this is an exported function, could you please add some function
header information.

> +int mmc_cqe_start_req(struct mmc_host *host, struct mmc_request *mrq)
> +{
> +       int err;
> +
> +       /* Caller must hold retuning while CQE is in use */

Would be nice to get some more clarification on *why* this is needed.
Can you please extend the comment a bit on that?

> +       err = mmc_retune(host);
> +       if (err)
> +               goto out_err;
> +
> +       mrq->host = host;
> +
> +       mmc_mrq_pr_debug(host, mrq, true);
> +
> +       err = mmc_mrq_prep(host, mrq);
> +       if (err)
> +               goto out_err;
> +
> +       err = host->cqe_ops->cqe_request(host, mrq);
> +       if (err)
> +               goto out_err;
> +
> +       trace_mmc_request_start(host, mrq);
> +
> +       return 0;
> +
> +out_err:
> +       if (mrq->cmd) {
> +               pr_debug("%s: failed to start CQE direct CMD%u, error %d\n",
> +                        mmc_hostname(host), mrq->cmd->opcode, err);
> +       } else {
> +               pr_debug("%s: failed to start CQE transfer for tag %d, error %d\n",
> +                        mmc_hostname(host), mrq->tag, err);
> +       }
> +       return err;
> +}
> +EXPORT_SYMBOL(mmc_cqe_start_req);
> +
> +static void __mmc_cqe_request_done(struct mmc_host *host,
> +                                  struct mmc_request *mrq)
> +{
> +       mmc_should_fail_request(host, mrq);
> +
> +       /* Flag re-tuning needed on CRC errors */
> +       if ((mrq->cmd && mrq->cmd->error == -EILSEQ) ||
> +           (mrq->data && mrq->data->error == -EILSEQ))
> +               mmc_retune_needed(host);
> +
> +       trace_mmc_request_done(host, mrq);
> +
> +       if (mrq->cmd) {
> +               pr_debug("%s: CQE req done (direct CMD%u): %d\n",
> +                        mmc_hostname(host), mrq->cmd->opcode, mrq->cmd->error);
> +       } else {
> +               pr_debug("%s: CQE transfer done tag %d\n",
> +                        mmc_hostname(host), mrq->tag);
> +       }
> +
> +       if (mrq->data) {
> +               pr_debug("%s:     %d bytes transferred: %d\n",
> +                        mmc_hostname(host),
> +                        mrq->data->bytes_xfered, mrq->data->error);
> +       }
> +}
> +
> +/**
> + *     mmc_cqe_request_done - CQE has finished processing an MMC request
> + *     @host: MMC host which completed request
> + *     @mrq: MMC request which completed
> + *
> + *     CQE drivers should call this function when they have completed
> + *     their processing of a request.
> + */
> +void mmc_cqe_request_done(struct mmc_host *host, struct mmc_request *mrq)
> +{
> +       __mmc_cqe_request_done(host, mrq);
> +
> +       mrq->done(mrq);
> +}
> +EXPORT_SYMBOL(mmc_cqe_request_done);
> +
> +/**
> + *     mmc_cqe_post_req - CQE post process of a completed MMC request
> + *     @host: MMC host
> + *     @mrq: MMC request to be processed
> + */
> +void mmc_cqe_post_req(struct mmc_host *host, struct mmc_request *mrq)
> +{
> +       if (host->cqe_ops->cqe_post_req)
> +               host->cqe_ops->cqe_post_req(host, mrq);
> +}
> +EXPORT_SYMBOL(mmc_cqe_post_req);
> +
> +/* Arbitrary 1 second timeout */
> +#define MMC_CQE_RECOVERY_TIMEOUT       1000

Why 1 second?

Nitpick:
I am bit reluctant to use the terminology of "recovery". In then end
it's about a request timeout and then act by doing a graceful abort.

> +

Please add function header.

> +int mmc_cqe_recovery(struct mmc_host *host)
> +{
> +       struct mmc_command cmd;
> +       int err;
> +
> +       mmc_retune_hold_now(host);
> +
> +       /*
> +        * Recovery is expected seldom, if at all, but it reduces performance,
> +        * so make sure it is not completely silent.
> +        */
> +       pr_warn("%s: running CQE recovery\n", mmc_hostname(host));
> +
> +       host->cqe_ops->cqe_recovery_start(host);

What is expected to be done by the host from this callback?

I have been thinking about deploying a common data request timeout
behavior in the mmc core. In principle calling
wait_for_completion_timeout() instead of just wait_for_completion().
To deal with that, one would also need to invent a new host ops
callback to allow the host driver to restore its HW to a know state.
Very similar to what you also seems to need here for CQE.

I was just thinking that part of this code/callbacks could be re-used
for other non-CQE data requests that timeouts. Do you think that could
makes sense?

> +
> +       memset(&cmd, 0, sizeof(cmd));
> +       cmd.opcode       = MMC_STOP_TRANSMISSION,
> +       cmd.flags        = MMC_RSP_R1B | MMC_CMD_AC,
> +       cmd.flags       &= ~MMC_RSP_CRC; /* Ignore CRC */
> +       cmd.busy_timeout = MMC_CQE_RECOVERY_TIMEOUT,
> +       mmc_wait_for_cmd(host, &cmd, 0);
> +
> +       memset(&cmd, 0, sizeof(cmd));
> +       cmd.opcode       = MMC_CMDQ_TASK_MGMT;
> +       cmd.arg          = 1; /* Discard entire queue */
> +       cmd.flags        = MMC_RSP_R1B | MMC_CMD_AC;
> +       cmd.flags       &= ~MMC_RSP_CRC; /* Ignore CRC */
> +       cmd.busy_timeout = MMC_CQE_RECOVERY_TIMEOUT,
> +       err = mmc_wait_for_cmd(host, &cmd, 0);
> +
> +       host->cqe_ops->cqe_recovery_finish(host);
> +
> +       mmc_retune_release(host);
> +
> +       return err;
> +}
> +EXPORT_SYMBOL(mmc_cqe_recovery);
> +
>  /**
>   *     mmc_is_req_done - Determine if a 'cap_cmd_during_tfr' request is done
>   *     @host: MMC host
> diff --git a/include/linux/mmc/core.h b/include/linux/mmc/core.h
> index bf1788a224e6..1974fcfd4284 100644
> --- a/include/linux/mmc/core.h
> +++ b/include/linux/mmc/core.h
> @@ -174,6 +174,11 @@ struct mmc_async_req *mmc_start_areq(struct mmc_host *host,
>  int mmc_wait_for_cmd(struct mmc_host *host, struct mmc_command *cmd,
>                 int retries);
>
> +int mmc_cqe_start_req(struct mmc_host *host, struct mmc_request *mrq);
> +void mmc_cqe_request_done(struct mmc_host *host, struct mmc_request *mrq);
> +void mmc_cqe_post_req(struct mmc_host *host, struct mmc_request *mrq);
> +int mmc_cqe_recovery(struct mmc_host *host);
> +
>  int mmc_hw_reset(struct mmc_host *host);
>  void mmc_set_data_timeout(struct mmc_data *data, const struct mmc_card *card);
>
> --
> 1.9.1
>

Kind regards
Uffe

Re: [PATCH V4 06/11] mmc: mmc: Enable Command Queuing

[Ulf Hansson]

On 21 July 2017 at 11:49, Adrian Hunter <adrian.hunter@intel.com> wrote:
> Enable the Command Queue if the host controller supports i a command queue

Perhaps clarify the above to:
Enable the eMMC Command Queue if the card supports it and if the host
controller also supports the command queue engine.

> engine. It is not compatible with Packed Commands, so do not enable that
> at the same time.
>
> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
> ---
>  drivers/mmc/core/mmc.c | 17 ++++++++++++++++-
>  1 file changed, 16 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/mmc/core/mmc.c b/drivers/mmc/core/mmc.c
> index 4ffea14b7eb6..2ff0caf92bc8 100644
> --- a/drivers/mmc/core/mmc.c
> +++ b/drivers/mmc/core/mmc.c
> @@ -1783,6 +1783,20 @@ static int mmc_init_card(struct mmc_host *host, u32 ocr,
>                 }
>         }
>
> +       /* Enable Command Queue if supported */
> +       card->ext_csd.cmdq_en = false;
> +       if (card->ext_csd.cmdq_support && host->caps2 & MMC_CAP2_CQE) {
> +               err = mmc_cmdq_enable(card);
> +               if (err && err != -EBADMSG)
> +                       goto free_card;
> +               if (err) {
> +                       pr_warn("%s: Enabling CMDQ failed\n",
> +                               mmc_hostname(card->host));
> +                       card->ext_csd.cmdq_support = false;
> +                       card->ext_csd.cmdq_depth = 0;
> +                       err = 0;
> +               }
> +       }
>         /*
>          * In some cases (e.g. RPMB or mmc_test), the Command Queue must be
>          * disabled for a time, so a flag is needed to indicate to re-enable the
> @@ -1796,7 +1810,8 @@ static int mmc_init_card(struct mmc_host *host, u32 ocr,
>          */
>         if (card->ext_csd.max_packed_writes >= 3 &&
>             card->ext_csd.max_packed_reads >= 5 &&
> -           host->caps2 & MMC_CAP2_PACKED_CMD) {
> +           host->caps2 & MMC_CAP2_PACKED_CMD &&

There are currently no host using MMC_CAP2_PACKED_CMD and we have
recently removed the support for it in the mmc block device layer.
It turned out we still have some leftovers. Would you mind folding in
a cleanup patch removing the support for packed command altogether and
adopt this change to it?

I guess adding a comment that CMDQ can't be used together with packed
command is still relevant. Especially if we would consider re-adding
the support for it.

> +           !card->ext_csd.cmdq_en) {
>                 err = mmc_switch(card, EXT_CSD_CMD_SET_NORMAL,
>                                 EXT_CSD_EXP_EVENTS_CTRL,
>                                 EXT_CSD_PACKED_EVENT_EN,
> --
> 1.9.1
>

Kind regards
Uffe

Re: [PATCH V4 07/11] mmc: mmc: Enable CQE's

[Ulf Hansson]

On 21 July 2017 at 11:49, Adrian Hunter <adrian.hunter@intel.com> wrote:
> Enable or disable CQE when a card is added or removed respectively.

As a standalone change, this is hard to understand.

Perhaps if you squash this with some the patche calling ->cqe_off()
and the one actually setting ext_csd.cmdq_en, I can get a better
picture.

>
> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
> ---
>  drivers/mmc/core/bus.c |  7 +++++++
>  drivers/mmc/core/mmc.c | 13 +++++++++++++
>  2 files changed, 20 insertions(+)
>
> diff --git a/drivers/mmc/core/bus.c b/drivers/mmc/core/bus.c
> index 301246513a37..a4b49e25fe96 100644
> --- a/drivers/mmc/core/bus.c
> +++ b/drivers/mmc/core/bus.c
> @@ -369,10 +369,17 @@ int mmc_add_card(struct mmc_card *card)
>   */
>  void mmc_remove_card(struct mmc_card *card)
>  {
> +       struct mmc_host *host = card->host;
> +
>  #ifdef CONFIG_DEBUG_FS
>         mmc_remove_card_debugfs(card);
>  #endif
>
> +       if (host->cqe_enabled) {
> +               host->cqe_ops->cqe_disable(host);
> +               host->cqe_enabled = false;
> +       }
> +

This doesn't feel like the correct place to disable cqe.

Primarily because I don't think you enable cqe in mmc_add_card(), so
this isn't consistent.

>         if (mmc_card_present(card)) {
>                 if (mmc_host_is_spi(card->host)) {
>                         pr_info("%s: SPI card removed\n",
> diff --git a/drivers/mmc/core/mmc.c b/drivers/mmc/core/mmc.c
> index 2ff0caf92bc8..92c6167d64e0 100644
> --- a/drivers/mmc/core/mmc.c
> +++ b/drivers/mmc/core/mmc.c
> @@ -1804,6 +1804,19 @@ static int mmc_init_card(struct mmc_host *host, u32 ocr,
>          */
>         card->reenable_cmdq = card->ext_csd.cmdq_en;
>
> +       if (card->ext_csd.cmdq_en && (host->caps2 & MMC_CAP2_CQE) &&
> +           !host->cqe_enabled) {

I assume card->ext_csd.cmdq_en can't be set, unless the MMC_CAP2_CQE
bit also is set. So there should be no reason to check it here again?

Also, can ever host->cqe_enabled be true in this path? If so, isn't
that wrong by itself?

> +               err = host->cqe_ops->cqe_enable(host, card);
> +               if (err) {
> +                       pr_err("%s: Failed to enable CQE, error %d\n",
> +                               mmc_hostname(host), err);
> +               } else {
> +                       host->cqe_enabled = true;
> +                       pr_info("%s: Command Queue Engine enabled\n",
> +                               mmc_hostname(host));
> +               }
> +       }
> +
>         /*
>          * The mandatory minimum values are defined for packed command.
>          * read: 5, write: 3
> --
> 1.9.1
>

Kind regards
Uffe

Re: [PATCH V4 08/11] mmc: block: Prepare CQE data

[Ulf Hansson]

On 21 July 2017 at 11:49, Adrian Hunter <adrian.hunter@intel.com> wrote:
> Enhance mmc_blk_data_prep() to support CQE requests.

Please try to be a bit more descriptive.

>
> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
> ---
>  drivers/mmc/core/block.c | 45 ++++++++++++++++++++++++++++++++++-----------
>  1 file changed, 34 insertions(+), 11 deletions(-)
>
> diff --git a/drivers/mmc/core/block.c b/drivers/mmc/core/block.c
> index 40f0d596ed54..915290c74363 100644
> --- a/drivers/mmc/core/block.c
> +++ b/drivers/mmc/core/block.c
> @@ -36,6 +36,7 @@
>  #include <linux/compat.h>
>  #include <linux/pm_runtime.h>
>  #include <linux/idr.h>
> +#include <linux/ioprio.h>
>
>  #include <linux/mmc/ioctl.h>
>  #include <linux/mmc/card.h>
> @@ -1485,25 +1486,27 @@ static enum mmc_blk_status mmc_blk_err_check(struct mmc_card *card,
>  }
>
>  static void mmc_blk_data_prep(struct mmc_queue *mq, struct mmc_queue_req *mqrq,
> -                             int disable_multi, bool *do_rel_wr,
> -                             bool *do_data_tag)
> +                             int disable_multi, bool *do_rel_wr_p,
> +                             bool *do_data_tag_p)
>  {
>         struct mmc_blk_data *md = mq->blkdata;
>         struct mmc_card *card = md->queue.card;
>         struct mmc_blk_request *brq = &mqrq->brq;
>         struct request *req = mmc_queue_req_to_req(mqrq);
> +       bool do_rel_wr, do_data_tag;
>
>         /*
>          * Reliable writes are used to implement Forced Unit Access and
>          * are supported only on MMCs.
>          */
> -       *do_rel_wr = (req->cmd_flags & REQ_FUA) &&
> -                    rq_data_dir(req) == WRITE &&
> -                    (md->flags & MMC_BLK_REL_WR);
> +       do_rel_wr = (req->cmd_flags & REQ_FUA) &&
> +                   rq_data_dir(req) == WRITE &&
> +                   (md->flags & MMC_BLK_REL_WR);
>
>         memset(brq, 0, sizeof(struct mmc_blk_request));
>
>         brq->mrq.data = &brq->data;
> +       brq->mrq.tag = req->tag;
>
>         brq->stop.opcode = MMC_STOP_TRANSMISSION;
>         brq->stop.arg = 0;
> @@ -1518,6 +1521,15 @@ static void mmc_blk_data_prep(struct mmc_queue *mq, struct mmc_queue_req *mqrq,
>
>         brq->data.blksz = 512;
>         brq->data.blocks = blk_rq_sectors(req);
> +       brq->data.blk_addr = blk_rq_pos(req);
> +
> +       /*
> +        * The command queue supports 2 priorities: "high" (1) and "simple" (0).
> +        * The eMMC will give "high" priority tasks priority over "simple"
> +        * priority tasks. Here we give priority to IOPRIO_CLASS_RT.
> +        */
> +       if (IOPRIO_PRIO_CLASS(req_get_ioprio(req)) == IOPRIO_CLASS_RT)
> +               brq->data.flags |= MMC_DATA_PRIO;

As I said earlier. I suggest you move the things related to priority
last in the series. Then we can discuss them separately, not blocking
anything else.

>
>         /*
>          * The block layer doesn't support all sector count
> @@ -1547,18 +1559,23 @@ static void mmc_blk_data_prep(struct mmc_queue *mq, struct mmc_queue_req *mqrq,
>                                                 brq->data.blocks);
>         }
>
> -       if (*do_rel_wr)
> +       if (do_rel_wr) {
>                 mmc_apply_rel_rw(brq, card, req);
> +               brq->data.flags |= MMC_DATA_REL_WR;
> +       }
>
>         /*
>          * Data tag is used only during writing meta data to speed
>          * up write and any subsequent read of this meta data
>          */
> -       *do_data_tag = card->ext_csd.data_tag_unit_size &&
> -                      (req->cmd_flags & REQ_META) &&
> -                      (rq_data_dir(req) == WRITE) &&
> -                      ((brq->data.blocks * brq->data.blksz) >=
> -                       card->ext_csd.data_tag_unit_size);
> +       do_data_tag = card->ext_csd.data_tag_unit_size &&
> +                     (req->cmd_flags & REQ_META) &&
> +                     (rq_data_dir(req) == WRITE) &&
> +                     ((brq->data.blocks * brq->data.blksz) >=
> +                      card->ext_csd.data_tag_unit_size);
> +
> +       if (do_data_tag)
> +               brq->data.flags |= MMC_DATA_DAT_TAG;
>
>         mmc_set_data_timeout(&brq->data, card);
>
> @@ -1587,6 +1604,12 @@ static void mmc_blk_data_prep(struct mmc_queue *mq, struct mmc_queue_req *mqrq,
>         mqrq->areq.mrq = &brq->mrq;
>
>         mmc_queue_bounce_pre(mqrq);
> +
> +       if (do_rel_wr_p)

I guess you don't need to make this conditional, let's just do the
below assignment.

> +               *do_rel_wr_p = do_rel_wr;
> +
> +       if (do_data_tag_p)

Ditto.

> +               *do_data_tag_p = do_data_tag;
>  }
>
>  static void mmc_blk_rw_rq_prep(struct mmc_queue_req *mqrq,
> --
> 1.9.1
>

I like the clarification this change introduces, as to assign the two
out parameters the values of the two local variables. However, could
you perhaps make that change being a separate patch, which should come
prior $subject patch in the series. That would make it more evident on
what changes are really for CQE.

Kind regards
Uffe

Re: [PATCH V4 00/11] mmc: Add Command Queue support

[Adrian Hunter]

On 07/08/17 16:41, Ulf Hansson wrote:
> On 21 July 2017 at 11:49, Adrian Hunter <adrian.hunter@intel.com> wrote:
>> Hi
>>
>> Here is V4 of the hardware command queue patches without the software
>> command queue patches.
> 
> Adrian, again apologize for the delay.
> 
> I am reviewing the series now and my intent is to provide comments on
> each change separately during the week.
> 
> However, a couple of overall thoughts:
> 
> *) Could you please post some fresh performance measurements, such we
> can get some real proof on why CMDQ is worth to be merged? If not
> increased throughput, perhaps we can show some decreased I/O requests
> latency!?

HW CMDQ offers 25% - 50% better random multi-threaded I/O.  I see a slight
2% drop in sequential read speed but no change to sequential write.

> 
> **) I have spoken to Linus W offlist - and he is still working on the
> blkmq port. Although we first need to continue with the
> re-factorization of the mmc block/core code, to minimize the bad
> impact of our big mmc claim host lock. We should expect a
> re-submission of his series quite soonish. However, it's also likely
> that we need yet another round of re-factoring, before we can complete
> the port to blkmq.
> 
> ***) The reason for bringing up **), is of course because I think
> deploying CMDQ support would be better done on top of the blkmq
> interface as it's better suited for these kind of device types. My
> goal is to reach a better maintenance situation, using more modern mmc
> block code. We have spoken about this before, however of course I also
> don't want to delay the CMDQ series. Anyway, let's move forward and
> see what path we end up taking.

There is no advantage to holding up HW CMDQ.

> 
>>
>>
>> Changes since V3:
>>         Adjusted ...blk_end_request...() for new block status codes
>>         Fixed CQHCI transaction descriptor for "no DCMD" case
>>
>> Changes since V2:
>>         Dropped patches that have been applied.
>>         Re-based
>>         Added "mmc: sdhci-pci: Add CQHCI support for Intel GLK"
>>
>> Changes since V1:
>>
>>         "Share mmc request array between partitions" is dependent
>>         on changes in "Introduce queue semantics", so added that
>>         and block fixes:
>>
>>         Added "Fix is_waiting_last_req set incorrectly"
>>         Added "Fix cmd error reset failure path"
>>         Added "Use local var for mqrq_cur"
>>         Added "Introduce queue semantics"
>>
>> Changes since RFC:
>>
>>         Re-based on next.
>>         Added comment about command queue priority.
>>         Added some acks and reviews.
>>
>>
>> Adrian Hunter (10):
>>       mmc: core: Add mmc_retune_hold_now()
>>       mmc: core: Add members to mmc_request and mmc_data for CQE's
>>       mmc: host: Add CQE interface
>>       mmc: core: Turn off CQE before sending commands
>>       mmc: core: Add support for handling CQE requests
>>       mmc: mmc: Enable Command Queuing
>>       mmc: mmc: Enable CQE's
>>       mmc: block: Prepare CQE data
>>       mmc: block: Add CQE support
>>       mmc: sdhci-pci: Add CQHCI support for Intel GLK
>>
>> Venkat Gopalakrishnan (1):
>>       mmc: cqhci: support for command queue enabled host
>>
>>  drivers/mmc/core/block.c          |  240 +++++++-
>>  drivers/mmc/core/block.h          |    7 +
>>  drivers/mmc/core/bus.c            |    7 +
>>  drivers/mmc/core/core.c           |  156 ++++-
>>  drivers/mmc/core/host.c           |    6 +
>>  drivers/mmc/core/host.h           |    1 +
>>  drivers/mmc/core/mmc.c            |   30 +-
>>  drivers/mmc/core/queue.c          |  273 ++++++++-
>>  drivers/mmc/core/queue.h          |   42 +-
>>  drivers/mmc/host/Kconfig          |   14 +
>>  drivers/mmc/host/Makefile         |    1 +
>>  drivers/mmc/host/cqhci.c          | 1146 +++++++++++++++++++++++++++++++++++++
>>  drivers/mmc/host/cqhci.h          |  240 ++++++++
>>  drivers/mmc/host/sdhci-pci-core.c |  153 ++++-
>>  include/linux/mmc/core.h          |   18 +-
>>  include/linux/mmc/host.h          |   24 +
>>  include/trace/events/mmc.h        |   36 +-
>>  17 files changed, 2354 insertions(+), 40 deletions(-)
>>  create mode 100644 drivers/mmc/host/cqhci.c
>>  create mode 100644 drivers/mmc/host/cqhci.h
>>
>>
>> Regards
>> Adrian
> 
> Kind regards
> Uffe
> 

Re: [PATCH V4 00/11] mmc: Add Command Queue support

[Ulf Hansson]

On 8 August 2017 at 11:26, Adrian Hunter <adrian.hunter@intel.com> wrote:
> On 07/08/17 16:41, Ulf Hansson wrote:
>> On 21 July 2017 at 11:49, Adrian Hunter <adrian.hunter@intel.com> wrote:
>>> Hi
>>>
>>> Here is V4 of the hardware command queue patches without the software
>>> command queue patches.
>>
>> Adrian, again apologize for the delay.
>>
>> I am reviewing the series now and my intent is to provide comments on
>> each change separately during the week.
>>
>> However, a couple of overall thoughts:
>>
>> *) Could you please post some fresh performance measurements, such we
>> can get some real proof on why CMDQ is worth to be merged? If not
>> increased throughput, perhaps we can show some decreased I/O requests
>> latency!?
>
> HW CMDQ offers 25% - 50% better random multi-threaded I/O.  I see a slight
> 2% drop in sequential read speed but no change to sequential write.

Great, that satisfies my request! Could you please fold in this
information in one of the change-logs, which adds supports for the
CQE?

>
>>
>> **) I have spoken to Linus W offlist - and he is still working on the
>> blkmq port. Although we first need to continue with the
>> re-factorization of the mmc block/core code, to minimize the bad
>> impact of our big mmc claim host lock. We should expect a
>> re-submission of his series quite soonish. However, it's also likely
>> that we need yet another round of re-factoring, before we can complete
>> the port to blkmq.
>>
>> ***) The reason for bringing up **), is of course because I think
>> deploying CMDQ support would be better done on top of the blkmq
>> interface as it's better suited for these kind of device types. My
>> goal is to reach a better maintenance situation, using more modern mmc
>> block code. We have spoken about this before, however of course I also
>> don't want to delay the CMDQ series. Anyway, let's move forward and
>> see what path we end up taking.
>
> There is no advantage to holding up HW CMDQ.

Maybe not.

I haven't completed the review yet, but I guess you understand my
concerns on the impact of the blkmq port.

If I can get some promises about yours support (testing/reviewing/etc)
of the blkmq port series when going forward, that would of course
impact my decision on what path to pick.

Does that sounds reasonable to you?

[...]

Kind regards
Uffe

Re: [PATCH V4 00/11] mmc: Add Command Queue support

[Adrian Hunter]

On 08/08/17 13:36, Ulf Hansson wrote:
> On 8 August 2017 at 11:26, Adrian Hunter <adrian.hunter@intel.com> wrote:
>> On 07/08/17 16:41, Ulf Hansson wrote:
>>> On 21 July 2017 at 11:49, Adrian Hunter <adrian.hunter@intel.com> wrote:
>>>> Hi
>>>>
>>>> Here is V4 of the hardware command queue patches without the software
>>>> command queue patches.
>>>
>>> Adrian, again apologize for the delay.
>>>
>>> I am reviewing the series now and my intent is to provide comments on
>>> each change separately during the week.
>>>
>>> However, a couple of overall thoughts:
>>>
>>> *) Could you please post some fresh performance measurements, such we
>>> can get some real proof on why CMDQ is worth to be merged? If not
>>> increased throughput, perhaps we can show some decreased I/O requests
>>> latency!?
>>
>> HW CMDQ offers 25% - 50% better random multi-threaded I/O.  I see a slight
>> 2% drop in sequential read speed but no change to sequential write.
> 
> Great, that satisfies my request! Could you please fold in this
> information in one of the change-logs, which adds supports for the
> CQE?

Sure

> 
>>
>>>
>>> **) I have spoken to Linus W offlist - and he is still working on the
>>> blkmq port. Although we first need to continue with the
>>> re-factorization of the mmc block/core code, to minimize the bad
>>> impact of our big mmc claim host lock. We should expect a
>>> re-submission of his series quite soonish. However, it's also likely
>>> that we need yet another round of re-factoring, before we can complete
>>> the port to blkmq.
>>>
>>> ***) The reason for bringing up **), is of course because I think
>>> deploying CMDQ support would be better done on top of the blkmq
>>> interface as it's better suited for these kind of device types. My
>>> goal is to reach a better maintenance situation, using more modern mmc
>>> block code. We have spoken about this before, however of course I also
>>> don't want to delay the CMDQ series. Anyway, let's move forward and
>>> see what path we end up taking.
>>
>> There is no advantage to holding up HW CMDQ.
> 
> Maybe not.
> 
> I haven't completed the review yet, but I guess you understand my
> concerns on the impact of the blkmq port.

If the blkmq port were ready there would be no problem.  But I don't think
it should be rushed.  It would be better to throw more light on the issues.

> 
> If I can get some promises about yours support (testing/reviewing/etc)
> of the blkmq port series when going forward, that would of course
> impact my decision on what path to pick.
> 
> Does that sounds reasonable to you?

Sure

Re: [PATCH V4 02/11] mmc: core: Add members to mmc_request and mmc_data for CQE's

[Adrian Hunter]

On 07/08/17 16:51, Ulf Hansson wrote:
> On 21 July 2017 at 11:49, Adrian Hunter <adrian.hunter@intel.com> wrote:
>> Most of the information needed to issue requests to a CQE is already in
>> struct mmc_request and struct mmc_data. Add data block address, some flags,
>> and the task id (tag), and allow for cmd being NULL which it is for CQE
>> tasks.
>>
>> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
>> ---
>>  include/linux/mmc/core.h   | 13 +++++++++++--
>>  include/trace/events/mmc.h | 36 +++++++++++++++++++++++-------------
>>  2 files changed, 34 insertions(+), 15 deletions(-)
>>
>> diff --git a/include/linux/mmc/core.h b/include/linux/mmc/core.h
>> index a0c63ea28796..bf1788a224e6 100644
>> --- a/include/linux/mmc/core.h
>> +++ b/include/linux/mmc/core.h
>> @@ -122,11 +122,18 @@ struct mmc_data {
>>         unsigned int            timeout_clks;   /* data timeout (in clocks) */
>>         unsigned int            blksz;          /* data block size */
>>         unsigned int            blocks;         /* number of blocks */
>> +       unsigned int            blk_addr;       /* block address */
>>         int                     error;          /* data error */
>>         unsigned int            flags;
>>
>> -#define MMC_DATA_WRITE (1 << 8)
>> -#define MMC_DATA_READ  (1 << 9)
>> +#define MMC_DATA_WRITE         BIT(8)
>> +#define MMC_DATA_READ          BIT(9)
>> +/* Extra flags used by CQE */
>> +#define MMC_DATA_QBR           BIT(10)         /* CQE queue barrier*/
>> +#define MMC_DATA_PRIO          BIT(11)         /* CQE high priority */
> 
> Regarding high priority data requests. I think this can be debated a
> bit on how it should be implemented.
> 
> If possible, may I suggest we move adding the support for this to the
> end of the series? In that way we can discuss it separately and it
> won't block us moving forward.

Yes, there is no reason we need to make use of the priority to start with.
> 
>> +#define MMC_DATA_REL_WR                BIT(12)         /* Reliable write */
> 
> What has this bit to do with CQE?

CQE processes read / write requests based on the information in struct
mmc_data, so we have to tell it whether the request is a reliable write.

> 
>> +#define MMC_DATA_DAT_TAG       BIT(13)         /* Tag request */
>> +#define MMC_DATA_FORCED_PRG    BIT(14)         /* Forced programming */
>>
>>         unsigned int            bytes_xfered;
>>
>> @@ -153,6 +160,8 @@ struct mmc_request {
>>
>>         /* Allow other commands during this ongoing data transfer or busy wait */
>>         bool                    cap_cmd_during_tfr;
>> +
>> +       int                     tag;
>>  };
>>
>>  struct mmc_card;
>> diff --git a/include/trace/events/mmc.h b/include/trace/events/mmc.h
>> index a72f9b94c80b..f30a99ac65b6 100644
>> --- a/include/trace/events/mmc.h
>> +++ b/include/trace/events/mmc.h
>> @@ -29,8 +29,10 @@
>>                 __field(unsigned int,           sbc_flags)
>>                 __field(unsigned int,           sbc_retries)
>>                 __field(unsigned int,           blocks)
>> +               __field(unsigned int,           blk_addr)
>>                 __field(unsigned int,           blksz)
>>                 __field(unsigned int,           data_flags)
>> +               __field(int,                    tag)
>>                 __field(unsigned int,           can_retune)
>>                 __field(unsigned int,           doing_retune)
>>                 __field(unsigned int,           retune_now)
>> @@ -42,10 +44,10 @@
>>         ),
>>
>>         TP_fast_assign(
>> -               __entry->cmd_opcode = mrq->cmd->opcode;
>> -               __entry->cmd_arg = mrq->cmd->arg;
>> -               __entry->cmd_flags = mrq->cmd->flags;
>> -               __entry->cmd_retries = mrq->cmd->retries;
>> +               __entry->cmd_opcode = mrq->cmd ? mrq->cmd->opcode : 0;
>> +               __entry->cmd_arg = mrq->cmd ? mrq->cmd->arg : 0;
>> +               __entry->cmd_flags = mrq->cmd ? mrq->cmd->flags : 0;
>> +               __entry->cmd_retries = mrq->cmd ? mrq->cmd->retries : 0;
>>                 __entry->stop_opcode = mrq->stop ? mrq->stop->opcode : 0;
>>                 __entry->stop_arg = mrq->stop ? mrq->stop->arg : 0;
>>                 __entry->stop_flags = mrq->stop ? mrq->stop->flags : 0;
>> @@ -56,7 +58,9 @@
>>                 __entry->sbc_retries = mrq->sbc ? mrq->sbc->retries : 0;
>>                 __entry->blksz = mrq->data ? mrq->data->blksz : 0;
>>                 __entry->blocks = mrq->data ? mrq->data->blocks : 0;
>> +               __entry->blk_addr = mrq->data ? mrq->data->blk_addr : 0;
>>                 __entry->data_flags = mrq->data ? mrq->data->flags : 0;
>> +               __entry->tag = mrq->tag;
>>                 __entry->can_retune = host->can_retune;
>>                 __entry->doing_retune = host->doing_retune;
>>                 __entry->retune_now = host->retune_now;
>> @@ -71,8 +75,8 @@
>>                   "cmd_opcode=%u cmd_arg=0x%x cmd_flags=0x%x cmd_retries=%u "
>>                   "stop_opcode=%u stop_arg=0x%x stop_flags=0x%x stop_retries=%u "
>>                   "sbc_opcode=%u sbc_arg=0x%x sbc_flags=0x%x sbc_retires=%u "
>> -                 "blocks=%u block_size=%u data_flags=0x%x "
>> -                 "can_retune=%u doing_retune=%u retune_now=%u "
>> +                 "blocks=%u block_size=%u blk_addr=%u data_flags=0x%x "
>> +                 "tag=%d can_retune=%u doing_retune=%u retune_now=%u "
>>                   "need_retune=%d hold_retune=%d retune_period=%u",
>>                   __get_str(name), __entry->mrq,
>>                   __entry->cmd_opcode, __entry->cmd_arg,
>> @@ -81,7 +85,8 @@
>>                   __entry->stop_flags, __entry->stop_retries,
>>                   __entry->sbc_opcode, __entry->sbc_arg,
>>                   __entry->sbc_flags, __entry->sbc_retries,
>> -                 __entry->blocks, __entry->blksz, __entry->data_flags,
>> +                 __entry->blocks, __entry->blk_addr,
>> +                 __entry->blksz, __entry->data_flags, __entry->tag,
>>                   __entry->can_retune, __entry->doing_retune,
>>                   __entry->retune_now, __entry->need_retune,
>>                   __entry->hold_retune, __entry->retune_period)
>> @@ -108,6 +113,7 @@
>>                 __field(unsigned int,           sbc_retries)
>>                 __field(unsigned int,           bytes_xfered)
>>                 __field(int,                    data_err)
>> +               __field(int,                    tag)
>>                 __field(unsigned int,           can_retune)
>>                 __field(unsigned int,           doing_retune)
>>                 __field(unsigned int,           retune_now)
>> @@ -119,10 +125,13 @@
>>         ),
>>
>>         TP_fast_assign(
>> -               __entry->cmd_opcode = mrq->cmd->opcode;
>> -               __entry->cmd_err = mrq->cmd->error;
>> -               memcpy(__entry->cmd_resp, mrq->cmd->resp, 4);
>> -               __entry->cmd_retries = mrq->cmd->retries;
>> +               __entry->cmd_opcode = mrq->cmd ? mrq->cmd->opcode : 0;
>> +               __entry->cmd_err = mrq->cmd ? mrq->cmd->error : 0;
>> +               __entry->cmd_resp[0] = mrq->cmd ? mrq->cmd->resp[0] : 0;
>> +               __entry->cmd_resp[1] = mrq->cmd ? mrq->cmd->resp[1] : 0;
>> +               __entry->cmd_resp[2] = mrq->cmd ? mrq->cmd->resp[2] : 0;
>> +               __entry->cmd_resp[3] = mrq->cmd ? mrq->cmd->resp[3] : 0;
>> +               __entry->cmd_retries = mrq->cmd ? mrq->cmd->retries : 0;
>>                 __entry->stop_opcode = mrq->stop ? mrq->stop->opcode : 0;
>>                 __entry->stop_err = mrq->stop ? mrq->stop->error : 0;
>>                 __entry->stop_resp[0] = mrq->stop ? mrq->stop->resp[0] : 0;
>> @@ -139,6 +148,7 @@
>>                 __entry->sbc_retries = mrq->sbc ? mrq->sbc->retries : 0;
>>                 __entry->bytes_xfered = mrq->data ? mrq->data->bytes_xfered : 0;
>>                 __entry->data_err = mrq->data ? mrq->data->error : 0;
>> +               __entry->tag = mrq->tag;
>>                 __entry->can_retune = host->can_retune;
>>                 __entry->doing_retune = host->doing_retune;
>>                 __entry->retune_now = host->retune_now;
>> @@ -154,7 +164,7 @@
>>                   "cmd_retries=%u stop_opcode=%u stop_err=%d "
>>                   "stop_resp=0x%x 0x%x 0x%x 0x%x stop_retries=%u "
>>                   "sbc_opcode=%u sbc_err=%d sbc_resp=0x%x 0x%x 0x%x 0x%x "
>> -                 "sbc_retries=%u bytes_xfered=%u data_err=%d "
>> +                 "sbc_retries=%u bytes_xfered=%u data_err=%d tag=%d "
>>                   "can_retune=%u doing_retune=%u retune_now=%u need_retune=%d "
>>                   "hold_retune=%d retune_period=%u",
>>                   __get_str(name), __entry->mrq,
>> @@ -170,7 +180,7 @@
>>                   __entry->sbc_resp[0], __entry->sbc_resp[1],
>>                   __entry->sbc_resp[2], __entry->sbc_resp[3],
>>                   __entry->sbc_retries,
>> -                 __entry->bytes_xfered, __entry->data_err,
>> +                 __entry->bytes_xfered, __entry->data_err, __entry->tag,
>>                   __entry->can_retune, __entry->doing_retune,
>>                   __entry->retune_now, __entry->need_retune,
>>                   __entry->hold_retune, __entry->retune_period)
>> --
>> 1.9.1
>>
> 
> Kind regards
> Uffe
> 

Re: [PATCH V4 03/11] mmc: host: Add CQE interface

[Adrian Hunter]

On 07/08/17 16:55, Ulf Hansson wrote:
> On 21 July 2017 at 11:49, Adrian Hunter <adrian.hunter@intel.com> wrote:
>> Add CQE host operations, capabilities, and host members.
> 
> I think adding these new interfaces deserves a bit more descriptive changelog.

OK

> 
>>
>> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
>> ---
>>  include/linux/mmc/host.h | 24 ++++++++++++++++++++++++
>>  1 file changed, 24 insertions(+)
>>
>> diff --git a/include/linux/mmc/host.h b/include/linux/mmc/host.h
>> index ebd1cebbef0c..4dd7ada9b4b9 100644
>> --- a/include/linux/mmc/host.h
>> +++ b/include/linux/mmc/host.h
>> @@ -162,6 +162,19 @@ struct mmc_host_ops {
>>                                   unsigned int direction, int blk_size);
>>  };
>>
>> +struct mmc_cqe_ops {
> 
> Could you please add a small comment to each new callback. Even if the
> names gives a good hint on they are used, some additional information
> would be nice.
> 
>> +       int     (*cqe_enable)(struct mmc_host *host, struct mmc_card *card);
>> +       void    (*cqe_disable)(struct mmc_host *host);
>> +       int     (*cqe_request)(struct mmc_host *host, struct mmc_request *mrq);
>> +       void    (*cqe_post_req)(struct mmc_host *host, struct mmc_request *mrq);
>> +       void    (*cqe_off)(struct mmc_host *host);
> 
> What differs between ->cqe_off() and ->cqe_disable()? Do we need both?

Yes we need both.  CQE must be set up, which is what ->cqe_enable() and
->cqe_disable() take care of.  But CQE can only process read / write
requests, so it must be "halted" (in CQHCI terminology) to allow the host
controller to send other commands (like discards and flushes).  So
->cqe_off() switches from CQE mode to legacy mode.  There is not a
->cqe_on() because we can simply let the ->cqe_request() path do that.  We
could let the host controller do ->cqe_off() too but it is simpler if the
core does it.

> 
>> +       int     (*cqe_wait_for_idle)(struct mmc_host *host);
> 
> The name sounds like you will poll the host to understand when the
> card becomes idle.
> 
> Is that so? Then when is this needed?

No, it is not to do with the card.  It is to wait for the CQE to become
idle.  As I mentioned above, we have to ->cqe_off() to send non-read/write
commands but that won't work if CQE is busy.  We have to wait for CQE to
become idle and handle any errors that might result from its ongoing
requests, before we start other commands.

> 
>> +       bool    (*cqe_timeout)(struct mmc_host *host, struct mmc_request *mrq,
>> +                              bool *recovery_needed);
>> +       void    (*cqe_recovery_start)(struct mmc_host *host);
>> +       void    (*cqe_recovery_finish)(struct mmc_host *host);
>> +};
>> +
>>  struct mmc_async_req {
>>         /* active mmc request */
>>         struct mmc_request      *mrq;
>> @@ -307,6 +320,8 @@ struct mmc_host {
>>  #define MMC_CAP2_HS400_ES      (1 << 20)       /* Host supports enhanced strobe */
>>  #define MMC_CAP2_NO_SD         (1 << 21)       /* Do not send SD commands during initialization */
>>  #define MMC_CAP2_NO_MMC                (1 << 22)       /* Do not send (e)MMC commands during initialization */
>> +#define MMC_CAP2_CQE           (1 << 23)       /* Has eMMC command queue engine */
>> +#define MMC_CAP2_CQE_DCMD      (1 << 24)       /* CQE can issue a direct command */
>>
>>         mmc_pm_flag_t           pm_caps;        /* supported pm features */
>>
>> @@ -393,6 +408,15 @@ struct mmc_host {
>>         int                     dsr_req;        /* DSR value is valid */
>>         u32                     dsr;    /* optional driver stage (DSR) value */
>>
>> +       /* Command Queue Engine (CQE) support */
>> +       const struct mmc_cqe_ops *cqe_ops;
>> +       void                    *cqe_private;
>> +       void                    (*cqe_recovery_notifier)(struct mmc_host *,
>> +                                                        struct mmc_request *);
> 
> Please add a comment here to understand how this callback is going to be used.

OK

> 
>> +       int                     cqe_qdepth;
>> +       bool                    cqe_enabled;
>> +       bool                    cqe_on;
>> +
>>         unsigned long           private[0] ____cacheline_aligned;
>>  };
>>
>> --
>> 1.9.1
>>
> 
> Kind regards
> Uffe
> 

Re: [PATCH V4 04/11] mmc: core: Turn off CQE before sending commands

[Adrian Hunter]

On 07/08/17 16:59, Ulf Hansson wrote:
> On 21 July 2017 at 11:49, Adrian Hunter <adrian.hunter@intel.com> wrote:
>> Turn off the CQE before sending commands, and ensure it is off in any reset
>> or power management paths, or re-tuning.
> 
> Please explain a bit why this is needed.
> 
> Also, it's not clear how it becomes re-enabled. Or it doesn't have to?

OK, I answered that in the previous email.  I can add more explanation here.

> 
>>
>> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
>> ---
>>  drivers/mmc/core/core.c | 9 +++++++++
>>  1 file changed, 9 insertions(+)
>>
>> diff --git a/drivers/mmc/core/core.c b/drivers/mmc/core/core.c
>> index 26431267a3e2..b0af9db18eef 100644
>> --- a/drivers/mmc/core/core.c
>> +++ b/drivers/mmc/core/core.c
>> @@ -260,6 +260,9 @@ static void __mmc_start_request(struct mmc_host *host, struct mmc_request *mrq)
>>
>>         trace_mmc_request_start(host, mrq);
>>
>> +       if (host->cqe_on)
>> +               host->cqe_ops->cqe_off(host);
>> +
> 
> Why can't you use the ->cqe_disable() callback here instead?
> 
> I guess what puzzles me is that there is no corresponding ->cqe_on()
> callback. When does cqe gets turned on again?

That was covered in the previous email too.

> 
>>         host->ops->request(host, mrq);
>>  }
>>
>> @@ -982,6 +985,9 @@ int mmc_execute_tuning(struct mmc_card *card)
>>         if (!host->ops->execute_tuning)
>>                 return 0;
>>
>> +       if (host->cqe_on)
>> +               host->cqe_ops->cqe_off(host);
>> +
>>         if (mmc_card_mmc(card))
>>                 opcode = MMC_SEND_TUNING_BLOCK_HS200;
>>         else
>> @@ -1021,6 +1027,9 @@ void mmc_set_bus_width(struct mmc_host *host, unsigned int width)
>>   */
>>  void mmc_set_initial_state(struct mmc_host *host)
>>  {
>> +       if (host->cqe_on)
>> +               host->cqe_ops->cqe_off(host);
>> +
>>         mmc_retune_disable(host);
>>
>>         if (mmc_host_is_spi(host))
>> --
>> 1.9.1
>>
> 
> Kind regards
> Uffe
> 

RE: [PATCH V4 09/11] mmc: block: Add CQE support

[Bough Chen]

> -----Original Message-----
> From: Adrian Hunter [mailto:adrian.hunter@intel.com]
> Sent: Friday, July 21, 2017 5:50 PM
> To: Ulf Hansson <ulf.hansson@linaro.org>
> Cc: linux-mmc <linux-mmc@vger.kernel.org>; Bough Chen
> <haibo.chen@nxp.com>; Alex Lemberg <alex.lemberg@sandisk.com>;
> Mateusz Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung <jh80.chung@samsung.com>;
> Dong Aisheng <dongas86@gmail.com>; Das Asutosh
> <asutoshd@codeaurora.org>; Zhangfei Gao <zhangfei.gao@gmail.com>;
> Dorfman Konstantin <kdorfman@codeaurora.org>; David Griego
> <david.griego@linaro.org>; Sahitya Tummala <stummala@codeaurora.org>;
> Harjani Ritesh <riteshh@codeaurora.org>; Venu Byravarasu
> <vbyravarasu@nvidia.com>; Linus Walleij <linus.walleij@linaro.org>; Shawn Lin
> <shawn.lin@rock-chips.com>
> Subject: [PATCH V4 09/11] mmc: block: Add CQE support
> 
> Add CQE support to the block driver, including:
> 	- optionally using DCMD for flush requests
> 	- manually issuing discard requests
> 	- issuing read / write requests to the CQE
> 	- supporting block-layer timeouts
> 	- handling recovery
> 	- supporting re-tuning
> 
> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
> ---
>  drivers/mmc/core/block.c | 195 ++++++++++++++++++++++++++++++++-
>  drivers/mmc/core/block.h |   7 ++
>  drivers/mmc/core/queue.c | 273
> ++++++++++++++++++++++++++++++++++++++++++++++-
>  drivers/mmc/core/queue.h |  42 +++++++-
>  4 files changed, 510 insertions(+), 7 deletions(-)
> 
> diff --git a/drivers/mmc/core/block.c b/drivers/mmc/core/block.c index
> 915290c74363..2d25115637b7 100644
> --- a/drivers/mmc/core/block.c
> +++ b/drivers/mmc/core/block.c
> @@ -109,6 +109,7 @@ struct mmc_blk_data {
>  #define MMC_BLK_WRITE		BIT(1)
>  #define MMC_BLK_DISCARD		BIT(2)
>  #define MMC_BLK_SECDISCARD	BIT(3)
> +#define MMC_BLK_CQE_RECOVERY	BIT(4)
> 
>  	/*
>  	 * Only set in main mmc_blk_data associated @@ -1612,6 +1613,198
> @@ static void mmc_blk_data_prep(struct mmc_queue *mq, struct
> mmc_queue_req *mqrq,
>  		*do_data_tag_p = do_data_tag;
>  }
> 
> +#define MMC_CQE_RETRIES 2
> +
> +void mmc_blk_cqe_complete_rq(struct request *req) {
> +	struct mmc_queue_req *mqrq = req_to_mmc_queue_req(req);
> +	struct mmc_request *mrq = &mqrq->brq.mrq;
> +	struct request_queue *q = req->q;
> +	struct mmc_queue *mq = q->queuedata;
> +	struct mmc_host *host = mq->card->host;
> +	unsigned long flags;
> +	bool put_card;
> +	int err;
> +
> +	mmc_cqe_post_req(host, mrq);
> +
> +	spin_lock_irqsave(q->queue_lock, flags);
> +
> +	mq->cqe_in_flight[mmc_cqe_issue_type(host, req)] -= 1;
> +
> +	put_card = mmc_cqe_tot_in_flight(mq) == 0;
> +
> +	if (mrq->cmd && mrq->cmd->error)
> +		err = mrq->cmd->error;
> +	else if (mrq->data && mrq->data->error)
> +		err = mrq->data->error;
> +	else
> +		err = 0;
> +
> +	if (err) {
> +		if (mqrq->retries++ < MMC_CQE_RETRIES)
> +			blk_requeue_request(q, req);
> +		else
> +			__blk_end_request_all(req, BLK_STS_IOERR);
> +	} else if (mrq->data) {
> +		if (__blk_end_request(req, BLK_STS_OK, mrq->data-
> >bytes_xfered))
> +			blk_requeue_request(q, req);
> +	} else {
> +		__blk_end_request_all(req, BLK_STS_OK);
> +	}
> +
> +	mmc_cqe_kick_queue(mq);
> +
> +	spin_unlock_irqrestore(q->queue_lock, flags);
> +
> +	if (put_card)
> +		mmc_put_card(mq->card);
> +}
> +
> +void mmc_blk_cqe_recovery(struct mmc_queue *mq) {
> +	struct mmc_card *card = mq->card;
> +	struct mmc_host *host = card->host;
> +	int err;
> +
> +	mmc_get_card(card);
> +
> +	pr_debug("%s: CQE recovery start\n", mmc_hostname(host));
> +
> +	mq->cqe_in_recovery = true;
> +
> +	err = mmc_cqe_recovery(host);
> +	if (err)
> +		mmc_blk_reset(mq->blkdata, host,
> MMC_BLK_CQE_RECOVERY);
> +	else
> +		mmc_blk_reset_success(mq->blkdata,
> MMC_BLK_CQE_RECOVERY);
> +
> +	mq->cqe_in_recovery = false;
> +
> +	pr_debug("%s: CQE recovery done\n", mmc_hostname(host));
> +
> +	mmc_put_card(card);
> +}
> +
> +static void mmc_blk_cqe_req_done(struct mmc_request *mrq) {
> +	struct mmc_queue_req *mqrq = container_of(mrq, struct
> mmc_queue_req,
> +						  brq.mrq);
> +	struct request *req = mmc_queue_req_to_req(mqrq);
> +	struct request_queue *q = req->q;
> +	struct mmc_queue *mq = q->queuedata;
> +
> +	/*
> +	 * Block layer timeouts race with completions which means the normal
> +	 * completion path cannot be used during recovery.
> +	 */
> +	if (mq->cqe_in_recovery)
> +		mmc_blk_cqe_complete_rq(req);
> +	else
> +		blk_complete_request(req);
> +}
> +
> +static int mmc_blk_cqe_start_req(struct mmc_host *host, struct
> +mmc_request *mrq) {
> +	mrq->done = mmc_blk_cqe_req_done;
> +	return mmc_cqe_start_req(host, mrq);
> +}
> +
> +static struct mmc_request *mmc_blk_cqe_prep_dcmd(struct
> mmc_queue_req *mqrq,
> +						 struct request *req)
> +{
> +	struct mmc_blk_request *brq = &mqrq->brq;
> +
> +	memset(brq, 0, sizeof(*brq));
> +
> +	brq->mrq.cmd = &brq->cmd;
> +	brq->mrq.tag = req->tag;
> +
> +	return &brq->mrq;
> +}
> +
> +static int mmc_blk_cqe_issue_flush(struct mmc_queue *mq, struct request
> +*req) {
> +	struct mmc_queue_req *mqrq = req_to_mmc_queue_req(req);
> +	struct mmc_request *mrq = mmc_blk_cqe_prep_dcmd(mqrq, req);
> +
> +	mrq->cmd->opcode = MMC_SWITCH;
> +	mrq->cmd->arg = (MMC_SWITCH_MODE_WRITE_BYTE << 24) |
> +			(EXT_CSD_FLUSH_CACHE << 16) |
> +			(1 << 8) |
> +			EXT_CSD_CMD_SET_NORMAL;
> +	mrq->cmd->flags = MMC_CMD_AC | MMC_RSP_R1B;
> +
> +	return mmc_blk_cqe_start_req(mq->card->host, mrq); }
> +
> +static int mmc_blk_cqe_issue_rw_rq(struct mmc_queue *mq, struct request
> +*req) {
> +	struct mmc_queue_req *mqrq = req_to_mmc_queue_req(req);
> +
> +	mmc_blk_data_prep(mq, mqrq, 0, NULL, NULL);
> +
> +	return mmc_blk_cqe_start_req(mq->card->host, &mqrq->brq.mrq); }
> +
> +enum mmc_issued mmc_blk_cqe_issue_rq(struct mmc_queue *mq, struct
> +request *req) {
> +	struct mmc_blk_data *md = mq->blkdata;
> +	struct mmc_card *card = md->queue.card;
> +	struct mmc_host *host = card->host;
> +	int ret;
> +
> +	ret = mmc_blk_part_switch(card, md);
> +	if (ret)
> +		return MMC_REQ_FAILED_TO_START;
> +
> +	switch (mmc_cqe_issue_type(host, req)) {
> +	case MMC_ISSUE_SYNC:
> +		ret = host->cqe_ops->cqe_wait_for_idle(host);
> +		if (ret)
> +			return MMC_REQ_BUSY;
> +		switch (req_op(req)) {
> +		case REQ_OP_DRV_IN:
> +		case REQ_OP_DRV_OUT:
> +			mmc_blk_issue_drv_op(mq, req);
> +			break;
> +		case REQ_OP_DISCARD:
> +			mmc_blk_issue_discard_rq(mq, req);
> +			break;
> +		case REQ_OP_SECURE_ERASE:
> +			mmc_blk_issue_secdiscard_rq(mq, req);
> +			break;
> +		case REQ_OP_FLUSH:
> +			mmc_blk_issue_flush(mq, req);
> +			break;
> +		default:
> +			WARN_ON_ONCE(1);
> +			return MMC_REQ_FAILED_TO_START;
> +		}
> +		return MMC_REQ_FINISHED;
> +	case MMC_ISSUE_DCMD:
> +	case MMC_ISSUE_ASYNC:
> +		switch (req_op(req)) {
> +		case REQ_OP_FLUSH:
> +			ret = mmc_blk_cqe_issue_flush(mq, req);
> +			break;
> +		case REQ_OP_READ:
> +		case REQ_OP_WRITE:
> +			ret = mmc_blk_cqe_issue_rw_rq(mq, req);
> +			break;
> +		default:
> +			WARN_ON_ONCE(1);
> +			ret = -EINVAL;
> +		}
> +		if (!ret)
> +			return MMC_REQ_STARTED;
> +		return ret == -EBUSY ? MMC_REQ_BUSY :
> MMC_REQ_FAILED_TO_START;
> +	default:
> +		WARN_ON_ONCE(1);
> +		return MMC_REQ_FAILED_TO_START;
> +	}
> +}
> +
>  static void mmc_blk_rw_rq_prep(struct mmc_queue_req *mqrq,
>  			       struct mmc_card *card,
>  			       int disable_multi,
> @@ -2035,7 +2228,7 @@ static struct mmc_blk_data
> *mmc_blk_alloc_req(struct mmc_card *card,
>  	INIT_LIST_HEAD(&md->part);
>  	md->usage = 1;
> 
> -	ret = mmc_init_queue(&md->queue, card, &md->lock, subname);
> +	ret = mmc_init_queue(&md->queue, card, &md->lock, subname,
> area_type);
>  	if (ret)
>  		goto err_putdisk;
> 
> diff --git a/drivers/mmc/core/block.h b/drivers/mmc/core/block.h index
> 860ca7c8df86..d7b3d7008b00 100644
> --- a/drivers/mmc/core/block.h
> +++ b/drivers/mmc/core/block.h
> @@ -6,4 +6,11 @@
> 
>  void mmc_blk_issue_rq(struct mmc_queue *mq, struct request *req);
> 
> +enum mmc_issued;
> +
> +enum mmc_issued mmc_blk_cqe_issue_rq(struct mmc_queue *mq,
> +				     struct request *req);
> +void mmc_blk_cqe_complete_rq(struct request *rq); void
> +mmc_blk_cqe_recovery(struct mmc_queue *mq);
> +
>  #endif
> diff --git a/drivers/mmc/core/queue.c b/drivers/mmc/core/queue.c index
> affa7370ba82..0cb7b0e8ee58 100644
> --- a/drivers/mmc/core/queue.c
> +++ b/drivers/mmc/core/queue.c
> @@ -36,10 +36,254 @@ static int mmc_prep_request(struct request_queue *q,
> struct request *req)
>  		return BLKPREP_KILL;
> 
>  	req->rq_flags |= RQF_DONTPREP;
> +	req_to_mmc_queue_req(req)->retries = 0;
> 
>  	return BLKPREP_OK;
>  }
> 
> +static void mmc_cqe_request_fn(struct request_queue *q) {
> +	struct mmc_queue *mq = q->queuedata;
> +	struct request *req;
> +
> +	if (!mq) {
> +		while ((req = blk_fetch_request(q)) != NULL) {
> +			req->rq_flags |= RQF_QUIET;
> +			__blk_end_request_all(req, BLK_STS_IOERR);
> +		}
> +		return;
> +	}
> +
> +	if (mq->asleep && !mq->cqe_busy)
> +		wake_up_process(mq->thread);
> +}
> +
> +static inline bool mmc_cqe_dcmd_busy(struct mmc_queue *mq) {
> +	/* Allow only 1 DCMD at a time */
> +	return mq->cqe_in_flight[MMC_ISSUE_DCMD];
> +}
> +
> +void mmc_cqe_kick_queue(struct mmc_queue *mq) {
> +	if ((mq->cqe_busy & MMC_CQE_DCMD_BUSY)
> && !mmc_cqe_dcmd_busy(mq))
> +		mq->cqe_busy &= ~MMC_CQE_DCMD_BUSY;
> +
> +	mq->cqe_busy &= ~MMC_CQE_QUEUE_FULL;
> +
> +	if (mq->asleep && !mq->cqe_busy)
> +		__blk_run_queue(mq->queue);
> +}
> +
> +static inline bool mmc_cqe_can_dcmd(struct mmc_host *host) {
> +	return host->caps2 & MMC_CAP2_CQE_DCMD; }
> +
> +enum mmc_issue_type mmc_cqe_issue_type(struct mmc_host *host,
> +				       struct request *req)
> +{
> +	switch (req_op(req)) {
> +	case REQ_OP_DRV_IN:
> +	case REQ_OP_DRV_OUT:
> +	case REQ_OP_DISCARD:
> +	case REQ_OP_SECURE_ERASE:
> +		return MMC_ISSUE_SYNC;
> +	case REQ_OP_FLUSH:
> +		return mmc_cqe_can_dcmd(host) ? MMC_ISSUE_DCMD :
> MMC_ISSUE_SYNC;
> +	default:
> +		return MMC_ISSUE_ASYNC;
> +	}
> +}
> +
> +static void __mmc_cqe_recovery_notifier(struct mmc_queue *mq) {
> +	if (!mq->cqe_recovery_needed) {
> +		mq->cqe_recovery_needed = true;
> +		wake_up_process(mq->thread);
> +	}
> +}
> +
> +static void mmc_cqe_recovery_notifier(struct mmc_host *host,
> +				      struct mmc_request *mrq)
> +{
> +	struct mmc_queue_req *mqrq = container_of(mrq, struct
> mmc_queue_req,
> +						  brq.mrq);
> +	struct request *req = mmc_queue_req_to_req(mqrq);
> +	struct request_queue *q = req->q;
> +	struct mmc_queue *mq = q->queuedata;
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(q->queue_lock, flags);
> +	__mmc_cqe_recovery_notifier(mq);
> +	spin_unlock_irqrestore(q->queue_lock, flags); }
> +
> +static int mmc_cqe_thread(void *d)
> +{
> +	struct mmc_queue *mq = d;
> +	struct request_queue *q = mq->queue;
> +	struct mmc_card *card = mq->card;
> +	struct mmc_host *host = card->host;
> +	unsigned long flags;
> +	int get_put = 0;
> +
> +	current->flags |= PF_MEMALLOC;
> +
> +	down(&mq->thread_sem);
> +	spin_lock_irqsave(q->queue_lock, flags);
> +	while (1) {
> +		struct request *req = NULL;
> +		enum mmc_issue_type issue_type;
> +		bool retune_ok = false;
> +
> +		if (mq->cqe_recovery_needed) {
> +			spin_unlock_irqrestore(q->queue_lock, flags);
> +			mmc_blk_cqe_recovery(mq);
> +			spin_lock_irqsave(q->queue_lock, flags);
> +			mq->cqe_recovery_needed = false;
> +		}
> +
> +		set_current_state(TASK_INTERRUPTIBLE);
> +
> +		if (!kthread_should_stop())
> +			req = blk_peek_request(q);
> +
> +		if (req) {
> +			issue_type = mmc_cqe_issue_type(host, req);
> +			switch (issue_type) {
> +			case MMC_ISSUE_DCMD:
> +				if (mmc_cqe_dcmd_busy(mq)) {
> +					mq->cqe_busy |=
> MMC_CQE_DCMD_BUSY;
> +					req = NULL;
> +					break;
> +				}
> +				/* Fall through */
> +			case MMC_ISSUE_ASYNC:
> +				if (blk_queue_start_tag(q, req)) {
> +					mq->cqe_busy |=
> MMC_CQE_QUEUE_FULL;
> +					req = NULL;
> +				}
> +				break;
> +			default:
> +				/*
> +				 * Timeouts are handled by mmc core, so set a
> +				 * large value to avoid races.
> +				 */
> +				req->timeout = 600 * HZ;
> +				blk_start_request(req);
> +				break;
> +			}
> +			if (req) {
> +				mq->cqe_in_flight[issue_type] += 1;
> +				if (mmc_cqe_tot_in_flight(mq) == 1)
> +					get_put += 1;
> +				if (mmc_cqe_qcnt(mq) == 1)
> +					retune_ok = true;
> +			}
> +		}
> +
> +		mq->asleep = !req;
> +
> +		spin_unlock_irqrestore(q->queue_lock, flags);
> +
> +		if (req) {
> +			enum mmc_issued issued;
> +
> +			set_current_state(TASK_RUNNING);
> +
> +			if (get_put) {
> +				get_put = 0;
> +				mmc_get_card(card);
> +			}
> +
> +			if (host->need_retune && retune_ok &&
> +			    !host->hold_retune)
> +				host->retune_now = true;
> +			else
> +				host->retune_now = false;
> +
> +			issued = mmc_blk_cqe_issue_rq(mq, req);
> +
> +			cond_resched();
> +
> +			spin_lock_irqsave(q->queue_lock, flags);
> +
> +			switch (issued) {
> +			case MMC_REQ_STARTED:
> +				break;
> +			case MMC_REQ_BUSY:
> +				blk_requeue_request(q, req);
> +				goto finished;
> +			case MMC_REQ_FAILED_TO_START:
> +				__blk_end_request_all(req, BLK_STS_IOERR);
> +				/* Fall through */
> +			case MMC_REQ_FINISHED:
> +finished:
> +				mq->cqe_in_flight[issue_type] -= 1;
> +				if (mmc_cqe_tot_in_flight(mq) == 0)
> +					get_put = -1;
> +			}
> +		} else {
> +			if (get_put < 0) {
> +				get_put = 0;
> +				mmc_put_card(card);
> +			}
> +			/*
> +			 * Do not stop with requests in flight in case recovery
> +			 * is needed.
> +			 */
> +			if (kthread_should_stop() &&
> +			    !mmc_cqe_tot_in_flight(mq)) {
> +				set_current_state(TASK_RUNNING);
> +				break;
> +			}
> +			up(&mq->thread_sem);
> +			schedule();
> +			down(&mq->thread_sem);
> +			spin_lock_irqsave(q->queue_lock, flags);
> +		}
> +	} /* loop */
> +	up(&mq->thread_sem);
> +
> +	return 0;
> +}
> +
> +static enum blk_eh_timer_return __mmc_cqe_timed_out(struct request
> +*req) {
> +	struct mmc_queue_req *mqrq = req_to_mmc_queue_req(req);
> +	struct mmc_request *mrq = &mqrq->brq.mrq;
> +	struct mmc_queue *mq = req->q->queuedata;
> +	struct mmc_host *host = mq->card->host;
> +	enum mmc_issue_type issue_type = mmc_cqe_issue_type(host, req);
> +	bool recovery_needed = false;
> +
> +	switch (issue_type) {
> +	case MMC_ISSUE_ASYNC:
> +	case MMC_ISSUE_DCMD:
> +		if (host->cqe_ops->cqe_timeout(host, mrq,
> &recovery_needed)) {
> +			if (recovery_needed)
> +				__mmc_cqe_recovery_notifier(mq);
> +			return BLK_EH_RESET_TIMER;
> +		}
> +		/* No timeout */
> +		return BLK_EH_HANDLED;
> +	default:
> +		/* Timeout is handled by mmc core */
> +		return BLK_EH_RESET_TIMER;
> +	}
> +}
> +
> +static enum blk_eh_timer_return mmc_cqe_timed_out(struct request *req)
> +{
> +	struct mmc_queue *mq = req->q->queuedata;
> +
> +	if (mq->cqe_recovery_needed)
> +		return BLK_EH_RESET_TIMER;
> +
> +	return __mmc_cqe_timed_out(req);
> +}
> +
>  static int mmc_queue_thread(void *d)
>  {
>  	struct mmc_queue *mq = d;
> @@ -233,11 +477,12 @@ static void mmc_exit_request(struct request_queue
> *q, struct request *req)
>   * Initialise a MMC card request queue.
>   */
>  int mmc_init_queue(struct mmc_queue *mq, struct mmc_card *card,
> -		   spinlock_t *lock, const char *subname)
> +		   spinlock_t *lock, const char *subname, int area_type)
>  {
>  	struct mmc_host *host = card->host;
>  	u64 limit = BLK_BOUNCE_HIGH;
>  	int ret = -ENOMEM;
> +	bool use_cqe = host->cqe_enabled && area_type !=
> +MMC_BLK_DATA_AREA_RPMB;
> 
>  	if (mmc_dev(host)->dma_mask && *mmc_dev(host)->dma_mask)
>  		limit = (u64)dma_max_pfn(mmc_dev(host)) << PAGE_SHIFT;
> @@ -247,7 +492,7 @@ int mmc_init_queue(struct mmc_queue *mq, struct
> mmc_card *card,
>  	if (!mq->queue)
>  		return -ENOMEM;
>  	mq->queue->queue_lock = lock;
> -	mq->queue->request_fn = mmc_request_fn;
> +	mq->queue->request_fn = use_cqe ? mmc_cqe_request_fn :
> mmc_request_fn;
>  	mq->queue->init_rq_fn = mmc_init_request;
>  	mq->queue->exit_rq_fn = mmc_exit_request;
>  	mq->queue->cmd_size = sizeof(struct mmc_queue_req); @@ -259,6
> +504,24 @@ int mmc_init_queue(struct mmc_queue *mq, struct mmc_card
> *card,
>  		return ret;
>  	}
> 
> +	if (use_cqe) {
> +		int q_depth = card->ext_csd.cmdq_depth;
> +
> +		if (q_depth > host->cqe_qdepth)
> +			q_depth = host->cqe_qdepth;
> +
> +		ret = blk_queue_init_tags(mq->queue, q_depth, NULL,
> +					  BLK_TAG_ALLOC_FIFO);
> +		if (ret)
> +			goto cleanup_queue;
> +
> +		blk_queue_softirq_done(mq->queue,
> mmc_blk_cqe_complete_rq);
> +		blk_queue_rq_timed_out(mq->queue, mmc_cqe_timed_out);
> +		blk_queue_rq_timeout(mq->queue, 60 * HZ);

Hi Adrian, 

These days I'm doing CMDQ stress test, and find one issue.
On our i.MX8QXP-ARM2 board, the RAM is 3GB. eMMC is 32GB. 
I use command 'free -m' get the total memory is 2800M, and the free memory is 2500M.

I use 'mkfs.ext4' to format ext4 file system on the eMMC under HS400ES CMDQ mode, works fine.

When I use the following command to stress test CMDQ, it works fine.
bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 2048 -r 1024

But when I change to use a large file size to do the same stress test, using 
bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 4096 -r 2048
or
bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 5600

I get the following dump message.  According to the log, mmc_cqe_timed_out() was trigged.
Seems mmc was blocked in somewhere.
Then I try to debug this issue, and open MMC_DEBUG in config, do the same test, print the detail
Command sending information on the console, but finally can't reproduce.

Shawn,
Can you have a try on your side?


[  738.385610] mmc0: cqhci: timeout for tag 1
[  738.389719] mmc0: cqhci: ============ CQHCI REGISTER DUMP ===========
[  738.396164] mmc0: cqhci: Caps:      0x0000310a | Version:  0x00000510
[  738.402601] mmc0: cqhci: Config:    0x00001001 | Control:  0x00000000
[  738.409038] mmc0: cqhci: Int stat:  0x00000000 | Int enab: 0x00000006
[  738.415475] mmc0: cqhci: Int sig:   0x00000006 | Int Coal: 0x00000000
[  738.421913] mmc0: cqhci: TDL base:  0x9007a000 | TDL up32: 0x00000000
[  738.428350] mmc0: cqhci: Doorbell:  0x1fffffff | TCN:      0x00000000
[  738.434788] mmc0: cqhci: Dev queue: 0x1f7fffff | Dev Pend: 0x1fffefff
[  738.441226] mmc0: cqhci: Task clr:  0x00000000 | SSC1:     0x00011000
[  738.447663] mmc0: cqhci: SSC2:      0x00000001 | DCMD rsp: 0x00000800
[  738.454100] mmc0: cqhci: RED mask:  0xfdf9a080 | TERRI:    0x00000000
[  738.460538] mmc0: cqhci: Resp idx:  0x0000002f | Resp arg: 0x00000900
[  738.466975] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
[  738.473414] mmc0: sdhci: Sys addr:  0xb6512000 | Version:  0x00000002
[  738.479850] mmc0: sdhci: Blk size:  0x00000200 | Blk cnt:  0x00000400
[  738.486288] mmc0: sdhci: Argument:  0x000c0400 | Trn mode: 0x00000023
[  738.492725] mmc0: sdhci: Present:   0x01fd858f | Host ctl: 0x00000030
[  738.499162] mmc0: sdhci: Power:     0x00000002 | Blk gap:  0x00000080
[  738.505600] mmc0: sdhci: Wake-up:   0x00000008 | Clock:    0x0000000f
[  738.512037] mmc0: sdhci: Timeout:   0x0000008f | Int stat: 0x00000000
[  738.518475] mmc0: sdhci: Int enab:  0x107f4000 | Sig enab: 0x107f4000
[  738.524912] mmc0: sdhci: AC12 err:  0x00000000 | Slot int: 0x00000502
[  738.531350] mmc0: sdhci: Caps:      0x07eb0000 | Caps_1:   0x8000b407
[  738.537787] mmc0: sdhci: Cmd:       0x00002c1a | Max curr: 0x00ffffff
[  738.544225] mmc0: sdhci: Resp[0]:   0x00000900 | Resp[1]:  0xffffffff
[  738.550662] mmc0: sdhci: Resp[2]:   0x328f5903 | Resp[3]:  0x00d02700
[  738.557099] mmc0: sdhci: Host ctl2: 0x00000008
[  738.561540] mmc0: sdhci: ADMA Err:  0x00000009 | ADMA Ptr: 0x90098400
[  738.567975] mmc0: sdhci: ============================================
[  738.574449] mmc0: running CQE recovery
[  738.593643] mmc0: Unexpected interrupt 0x00004000.
[  738.598436] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
[  738.604881] mmc0: sdhci: Sys addr:  0x00000000 | Version:  0x00000002
[  738.611318] mmc0: sdhci: Blk size:  0x00000200 | Blk cnt:  0x00000400
[  738.617756] mmc0: sdhci: Argument:  0x01af6800 | Trn mode: 0x00000023
[  738.624193] mmc0: sdhci: Present:   0x01fd8009 | Host ctl: 0x00000031
[  738.630630] mmc0: sdhci: Power:     0x00000002 | Blk gap:  0x00000080
[  738.637068] mmc0: sdhci: Wake-up:   0x00000008 | Clock:    0x0000000f
[  738.643505] mmc0: sdhci: Timeout:   0x0000008f | Int stat: 0x00004000
[  738.649943] mmc0: sdhci: Int enab:  0x007f1003 | Sig enab: 0x007f1003
[  738.656380] mmc0: sdhci: AC12 err:  0x00000000 | Slot int: 0x00000502
[  738.662818] mmc0: sdhci: Caps:      0x07eb0000 | Caps_1:   0x8000b407
[  738.669255] mmc0: sdhci: Cmd:       0x00002d12 | Max curr: 0x00ffffff
[  738.675693] mmc0: sdhci: Resp[0]:   0x00000c00 | Resp[1]:  0xffffffff
[  738.682130] mmc0: sdhci: Resp[2]:   0x328f5903 | Resp[3]:  0x00d02700
[  738.688566] mmc0: sdhci: Host ctl2: 0x00000008
[  738.693008] mmc0: sdhci: ADMA Err:  0x00000000 | ADMA Ptr: 0x00000000
[  738.699443] mmc0: sdhci: ============================================
[  738.715999] mmc0: Controller never released inhibit bit(s).
[  738.721573] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
[  738.728018] mmc0: sdhci: Sys addr:  0x00000000 | Version:  0x00000002
[  738.734455] mmc0: sdhci: Blk size:  0x00000200 | Blk cnt:  0x00000400
[  738.740892] mmc0: sdhci: Argument:  0x01af6800 | Trn mode: 0x00000023
[  738.747330] mmc0: sdhci: Present:   0x01fd8009 | Host ctl: 0x00000031
[  738.753767] mmc0: sdhci: Power:     0x00000002 | Blk gap:  0x00000080
[  738.760204] mmc0: sdhci: Wake-up:   0x00000008 | Clock:    0x0000000f
[  738.766642] mmc0: sdhci: Timeout:   0x0000008f | Int stat: 0x00004000
[  738.773079] mmc0: sdhci: Int enab:  0x007f1003 | Sig enab: 0x007f1003
[  738.779517] mmc0: sdhci: AC12 err:  0x00000000 | Slot int: 0x00000502
[  738.785955] mmc0: sdhci: Caps:      0x07eb0000 | Caps_1:   0x8000b407
[  738.792392] mmc0: sdhci: Cmd:       0x00002d12 | Max curr: 0x00ffffff
[  738.798829] mmc0: sdhci: Resp[0]:   0x00000c00 | Resp[1]:  0xffffffff
[  738.805266] mmc0: sdhci: Resp[2]:   0x328f5903 | Resp[3]:  0x00d02700
[  738.811703] mmc0: sdhci: Host ctl2: 0x00000008
[  738.816144] mmc0: sdhci: ADMA Err:  0x00000000 | ADMA Ptr: 0x00000000
[  738.822579] mmc0: sdhci: ============================================
[  748.881580] mmc0: Timeout waiting for hardware interrupt.
......


> +
> +		host->cqe_recovery_notifier = mmc_cqe_recovery_notifier;
> +	}
> +
>  	blk_queue_prep_rq(mq->queue, mmc_prep_request);
>  	queue_flag_set_unlocked(QUEUE_FLAG_NONROT, mq->queue);
>  	queue_flag_clear_unlocked(QUEUE_FLAG_ADD_RANDOM, mq-
> >queue); @@ -280,9 +543,9 @@ int mmc_init_queue(struct mmc_queue *mq,
> struct mmc_card *card,
> 
>  	sema_init(&mq->thread_sem, 1);
> 
> -	mq->thread = kthread_run(mmc_queue_thread, mq, "mmcqd/%d%s",
> -		host->index, subname ? subname : "");
> -
> +	mq->thread = kthread_run(use_cqe ? mmc_cqe_thread :
> mmc_queue_thread,
> +				 mq, "mmcqd/%d%s", host->index,
> +				 subname ? subname : "");
>  	if (IS_ERR(mq->thread)) {
>  		ret = PTR_ERR(mq->thread);
>  		goto cleanup_queue;
> diff --git a/drivers/mmc/core/queue.h b/drivers/mmc/core/queue.h index
> 361b46408e0f..8e9273d977c0 100644
> --- a/drivers/mmc/core/queue.h
> +++ b/drivers/mmc/core/queue.h
> @@ -7,6 +7,20 @@
>  #include <linux/mmc/core.h>
>  #include <linux/mmc/host.h>
> 
> +enum mmc_issued {
> +	MMC_REQ_STARTED,
> +	MMC_REQ_BUSY,
> +	MMC_REQ_FAILED_TO_START,
> +	MMC_REQ_FINISHED,
> +};
> +
> +enum mmc_issue_type {
> +	MMC_ISSUE_SYNC,
> +	MMC_ISSUE_DCMD,
> +	MMC_ISSUE_ASYNC,
> +	MMC_ISSUE_MAX,
> +};
> +
>  static inline struct mmc_queue_req *req_to_mmc_queue_req(struct request
> *rq)  {
>  	return blk_mq_rq_to_pdu(rq);
> @@ -53,6 +67,7 @@ struct mmc_queue_req {
>  	int			drv_op_result;
>  	struct mmc_blk_ioc_data	**idata;
>  	unsigned int		ioc_count;
> +	int			retries;
>  };
> 
>  struct mmc_queue {
> @@ -70,10 +85,17 @@ struct mmc_queue {
>  	 * associated mmc_queue_req data.
>  	 */
>  	int			qcnt;
> +	/* Following are defined for a Command Queue Engine */
> +	int			cqe_in_flight[MMC_ISSUE_MAX];
> +	unsigned int		cqe_busy;
> +	bool			cqe_recovery_needed;
> +	bool			cqe_in_recovery;
> +#define MMC_CQE_DCMD_BUSY	BIT(0)
> +#define MMC_CQE_QUEUE_FULL	BIT(1)
>  };
> 
>  extern int mmc_init_queue(struct mmc_queue *, struct mmc_card *,
> spinlock_t *,
> -			  const char *);
> +			  const char *, int);
>  extern void mmc_cleanup_queue(struct mmc_queue *);  extern void
> mmc_queue_suspend(struct mmc_queue *);  extern void
> mmc_queue_resume(struct mmc_queue *); @@ -85,4 +107,22 @@ extern
> unsigned int mmc_queue_map_sg(struct mmc_queue *,
> 
>  extern int mmc_access_rpmb(struct mmc_queue *);
> 
> +void mmc_cqe_kick_queue(struct mmc_queue *mq);
> +
> +enum mmc_issue_type mmc_cqe_issue_type(struct mmc_host *host,
> +				       struct request *req);
> +
> +static inline int mmc_cqe_tot_in_flight(struct mmc_queue *mq) {
> +	return mq->cqe_in_flight[MMC_ISSUE_SYNC] +
> +	       mq->cqe_in_flight[MMC_ISSUE_DCMD] +
> +	       mq->cqe_in_flight[MMC_ISSUE_ASYNC];
> +}
> +
> +static inline int mmc_cqe_qcnt(struct mmc_queue *mq) {
> +	return mq->cqe_in_flight[MMC_ISSUE_DCMD] +
> +	       mq->cqe_in_flight[MMC_ISSUE_ASYNC];
> +}
> +
>  #endif
> --
> 1.9.1

Re: [PATCH V4 09/11] mmc: block: Add CQE support

[Shawn Lin]

Hi,

On 2017/8/8 20:07, Bough Chen wrote:
>> -----Original Message-----
>> From: Adrian Hunter [mailto:adrian.hunter@intel.com]
>> Sent: Friday, July 21, 2017 5:50 PM
>> To: Ulf Hansson <ulf.hansson@linaro.org>
>> Cc: linux-mmc <linux-mmc@vger.kernel.org>; Bough Chen
>> <haibo.chen@nxp.com>; Alex Lemberg <alex.lemberg@sandisk.com>;
>> Mateusz Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
>> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung <jh80.chung@samsung.com>;
>> Dong Aisheng <dongas86@gmail.com>; Das Asutosh
>> <asutoshd@codeaurora.org>; Zhangfei Gao <zhangfei.gao@gmail.com>;
>> Dorfman Konstantin <kdorfman@codeaurora.org>; David Griego
>> <david.griego@linaro.org>; Sahitya Tummala <stummala@codeaurora.org>;
>> Harjani Ritesh <riteshh@codeaurora.org>; Venu Byravarasu
>> <vbyravarasu@nvidia.com>; Linus Walleij <linus.walleij@linaro.org>; Shawn Lin
>> <shawn.lin@rock-chips.com>
>> Subject: [PATCH V4 09/11] mmc: block: Add CQE support
>>
>> Add CQE support to the block driver, including:
>> 	- optionally using DCMD for flush requests
>> 	- manually issuing discard requests
>> 	- issuing read / write requests to the CQE
>> 	- supporting block-layer timeouts
>> 	- handling recovery
>> 	- supporting re-tuning
>>
>> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
>> ---
>>   drivers/mmc/core/block.c | 195 ++++++++++++++++++++++++++++++++-
>>   drivers/mmc/core/block.h |   7 ++
>>   drivers/mmc/core/queue.c | 273
>> ++++++++++++++++++++++++++++++++++++++++++++++-
>>   drivers/mmc/core/queue.h |  42 +++++++-
>>   4 files changed, 510 insertions(+), 7 deletions(-)
>>
>> diff --git a/drivers/mmc/core/block.c b/drivers/mmc/core/block.c index
>> 915290c74363..2d25115637b7 100644
>> --- a/drivers/mmc/core/block.c
>> +++ b/drivers/mmc/core/block.c
>> @@ -109,6 +109,7 @@ struct mmc_blk_data {
>>   #define MMC_BLK_WRITE		BIT(1)
>>   #define MMC_BLK_DISCARD		BIT(2)
>>   #define MMC_BLK_SECDISCARD	BIT(3)
>> +#define MMC_BLK_CQE_RECOVERY	BIT(4)
>>
>>   	/*
>>   	 * Only set in main mmc_blk_data associated @@ -1612,6 +1613,198
>> @@ static void mmc_blk_data_prep(struct mmc_queue *mq, struct
>> mmc_queue_req *mqrq,
>>   		*do_data_tag_p = do_data_tag;
>>   }
>>
>> +#define MMC_CQE_RETRIES 2


>> +		blk_queue_rq_timed_out(mq->queue, mmc_cqe_timed_out);
>> +		blk_queue_rq_timeout(mq->queue, 60 * HZ);
> 

------8<-------

> Hi Adrian,
> 
> These days I'm doing CMDQ stress test, and find one issue.
> On our i.MX8QXP-ARM2 board, the RAM is 3GB. eMMC is 32GB.
> I use command 'free -m' get the total memory is 2800M, and the free memory is 2500M.
> 
> I use 'mkfs.ext4' to format ext4 file system on the eMMC under HS400ES CMDQ mode, works fine.
> 
> When I use the following command to stress test CMDQ, it works fine.
> bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 2048 -r 1024
> 
> But when I change to use a large file size to do the same stress test, using
> bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 4096 -r 2048
> or
> bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 5600
> 
> I get the following dump message.  According to the log, mmc_cqe_timed_out() was trigged.
> Seems mmc was blocked in somewhere.
> Then I try to debug this issue, and open MMC_DEBUG in config, do the same test, print the detail
> Command sending information on the console, but finally can't reproduce.
> 
> Shawn,
> Can you have a try on your side?
> 

I think bonnie++ is almost the same disk test tool as iozone or fio. I
didn't saw this when testing CMDQ by fio, but I will try bonnie++ late
this week.


> 
> [  738.385610] mmc0: cqhci: timeout for tag 1
> [  738.389719] mmc0: cqhci: ============ CQHCI REGISTER DUMP ===========
> [  738.396164] mmc0: cqhci: Caps:      0x0000310a | Version:  0x00000510
> [  738.402601] mmc0: cqhci: Config:    0x00001001 | Control:  0x00000000
> [  738.409038] mmc0: cqhci: Int stat:  0x00000000 | Int enab: 0x00000006
> [  738.415475] mmc0: cqhci: Int sig:   0x00000006 | Int Coal: 0x00000000
> [  738.421913] mmc0: cqhci: TDL base:  0x9007a000 | TDL up32: 0x00000000
> [  738.428350] mmc0: cqhci: Doorbell:  0x1fffffff | TCN:      0x00000000
> [  738.434788] mmc0: cqhci: Dev queue: 0x1f7fffff | Dev Pend: 0x1fffefff
> [  738.441226] mmc0: cqhci: Task clr:  0x00000000 | SSC1:     0x00011000
> [  738.447663] mmc0: cqhci: SSC2:      0x00000001 | DCMD rsp: 0x00000800
> [  738.454100] mmc0: cqhci: RED mask:  0xfdf9a080 | TERRI:    0x00000000
> [  738.460538] mmc0: cqhci: Resp idx:  0x0000002f | Resp arg: 0x00000900
> [  738.466975] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
> [  738.473414] mmc0: sdhci: Sys addr:  0xb6512000 | Version:  0x00000002
> [  738.479850] mmc0: sdhci: Blk size:  0x00000200 | Blk cnt:  0x00000400
> [  738.486288] mmc0: sdhci: Argument:  0x000c0400 | Trn mode: 0x00000023
> [  738.492725] mmc0: sdhci: Present:   0x01fd858f | Host ctl: 0x00000030
> [  738.499162] mmc0: sdhci: Power:     0x00000002 | Blk gap:  0x00000080
> [  738.505600] mmc0: sdhci: Wake-up:   0x00000008 | Clock:    0x0000000f
> [  738.512037] mmc0: sdhci: Timeout:   0x0000008f | Int stat: 0x00000000
> [  738.518475] mmc0: sdhci: Int enab:  0x107f4000 | Sig enab: 0x107f4000
> [  738.524912] mmc0: sdhci: AC12 err:  0x00000000 | Slot int: 0x00000502
> [  738.531350] mmc0: sdhci: Caps:      0x07eb0000 | Caps_1:   0x8000b407
> [  738.537787] mmc0: sdhci: Cmd:       0x00002c1a | Max curr: 0x00ffffff
> [  738.544225] mmc0: sdhci: Resp[0]:   0x00000900 | Resp[1]:  0xffffffff
> [  738.550662] mmc0: sdhci: Resp[2]:   0x328f5903 | Resp[3]:  0x00d02700
> [  738.557099] mmc0: sdhci: Host ctl2: 0x00000008
> [  738.561540] mmc0: sdhci: ADMA Err:  0x00000009 | ADMA Ptr: 0x90098400
> [  738.567975] mmc0: sdhci: ============================================
> [  738.574449] mmc0: running CQE recovery
> [  738.593643] mmc0: Unexpected interrupt 0x00004000.
> [  738.598436] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
> [  738.604881] mmc0: sdhci: Sys addr:  0x00000000 | Version:  0x00000002
> [  738.611318] mmc0: sdhci: Blk size:  0x00000200 | Blk cnt:  0x00000400
> [  738.617756] mmc0: sdhci: Argument:  0x01af6800 | Trn mode: 0x00000023
> [  738.624193] mmc0: sdhci: Present:   0x01fd8009 | Host ctl: 0x00000031
> [  738.630630] mmc0: sdhci: Power:     0x00000002 | Blk gap:  0x00000080
> [  738.637068] mmc0: sdhci: Wake-up:   0x00000008 | Clock:    0x0000000f
> [  738.643505] mmc0: sdhci: Timeout:   0x0000008f | Int stat: 0x00004000
> [  738.649943] mmc0: sdhci: Int enab:  0x007f1003 | Sig enab: 0x007f1003
> [  738.656380] mmc0: sdhci: AC12 err:  0x00000000 | Slot int: 0x00000502
> [  738.662818] mmc0: sdhci: Caps:      0x07eb0000 | Caps_1:   0x8000b407
> [  738.669255] mmc0: sdhci: Cmd:       0x00002d12 | Max curr: 0x00ffffff
> [  738.675693] mmc0: sdhci: Resp[0]:   0x00000c00 | Resp[1]:  0xffffffff
> [  738.682130] mmc0: sdhci: Resp[2]:   0x328f5903 | Resp[3]:  0x00d02700
> [  738.688566] mmc0: sdhci: Host ctl2: 0x00000008
> [  738.693008] mmc0: sdhci: ADMA Err:  0x00000000 | ADMA Ptr: 0x00000000
> [  738.699443] mmc0: sdhci: ============================================
> [  738.715999] mmc0: Controller never released inhibit bit(s).
> [  738.721573] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
> [  738.728018] mmc0: sdhci: Sys addr:  0x00000000 | Version:  0x00000002
> [  738.734455] mmc0: sdhci: Blk size:  0x00000200 | Blk cnt:  0x00000400
> [  738.740892] mmc0: sdhci: Argument:  0x01af6800 | Trn mode: 0x00000023
> [  738.747330] mmc0: sdhci: Present:   0x01fd8009 | Host ctl: 0x00000031
> [  738.753767] mmc0: sdhci: Power:     0x00000002 | Blk gap:  0x00000080
> [  738.760204] mmc0: sdhci: Wake-up:   0x00000008 | Clock:    0x0000000f
> [  738.766642] mmc0: sdhci: Timeout:   0x0000008f | Int stat: 0x00004000
> [  738.773079] mmc0: sdhci: Int enab:  0x007f1003 | Sig enab: 0x007f1003
> [  738.779517] mmc0: sdhci: AC12 err:  0x00000000 | Slot int: 0x00000502
> [  738.785955] mmc0: sdhci: Caps:      0x07eb0000 | Caps_1:   0x8000b407
> [  738.792392] mmc0: sdhci: Cmd:       0x00002d12 | Max curr: 0x00ffffff
> [  738.798829] mmc0: sdhci: Resp[0]:   0x00000c00 | Resp[1]:  0xffffffff
> [  738.805266] mmc0: sdhci: Resp[2]:   0x328f5903 | Resp[3]:  0x00d02700
> [  738.811703] mmc0: sdhci: Host ctl2: 0x00000008
> [  738.816144] mmc0: sdhci: ADMA Err:  0x00000000 | ADMA Ptr: 0x00000000
> [  738.822579] mmc0: sdhci: ============================================
> [  748.881580] mmc0: Timeout waiting for hardware interrupt.
> ......
> 
> 
>> +
>> +		host->cqe_recovery_notifier = mmc_cqe_recovery_notifier;
>> +	}
>> +
>>   	blk_queue_prep_rq(mq->queue, mmc_prep_request);
>>   	queue_flag_set_unlocked(QUEUE_FLAG_NONROT, mq->queue);
>>   	queue_flag_clear_unlocked(QUEUE_FLAG_ADD_RANDOM, mq-
>>> queue); @@ -280,9 +543,9 @@ int mmc_init_queue(struct mmc_queue *mq,
>> struct mmc_card *card,
>>
>>   	sema_init(&mq->thread_sem, 1);
>>
>> -	mq->thread = kthread_run(mmc_queue_thread, mq, "mmcqd/%d%s",
>> -		host->index, subname ? subname : "");
>> -
>> +	mq->thread = kthread_run(use_cqe ? mmc_cqe_thread :
>> mmc_queue_thread,
>> +				 mq, "mmcqd/%d%s", host->index,
>> +				 subname ? subname : "");
>>   	if (IS_ERR(mq->thread)) {
>>   		ret = PTR_ERR(mq->thread);
>>   		goto cleanup_queue;
>> diff --git a/drivers/mmc/core/queue.h b/drivers/mmc/core/queue.h index
>> 361b46408e0f..8e9273d977c0 100644
>> --- a/drivers/mmc/core/queue.h
>> +++ b/drivers/mmc/core/queue.h
>> @@ -7,6 +7,20 @@
>>   #include <linux/mmc/core.h>
>>   #include <linux/mmc/host.h>
>>
>> +enum mmc_issued {
>> +	MMC_REQ_STARTED,
>> +	MMC_REQ_BUSY,
>> +	MMC_REQ_FAILED_TO_START,
>> +	MMC_REQ_FINISHED,
>> +};
>> +
>> +enum mmc_issue_type {
>> +	MMC_ISSUE_SYNC,
>> +	MMC_ISSUE_DCMD,
>> +	MMC_ISSUE_ASYNC,
>> +	MMC_ISSUE_MAX,
>> +};
>> +
>>   static inline struct mmc_queue_req *req_to_mmc_queue_req(struct request
>> *rq)  {
>>   	return blk_mq_rq_to_pdu(rq);
>> @@ -53,6 +67,7 @@ struct mmc_queue_req {
>>   	int			drv_op_result;
>>   	struct mmc_blk_ioc_data	**idata;
>>   	unsigned int		ioc_count;
>> +	int			retries;
>>   };
>>
>>   struct mmc_queue {
>> @@ -70,10 +85,17 @@ struct mmc_queue {
>>   	 * associated mmc_queue_req data.
>>   	 */
>>   	int			qcnt;
>> +	/* Following are defined for a Command Queue Engine */
>> +	int			cqe_in_flight[MMC_ISSUE_MAX];
>> +	unsigned int		cqe_busy;
>> +	bool			cqe_recovery_needed;
>> +	bool			cqe_in_recovery;
>> +#define MMC_CQE_DCMD_BUSY	BIT(0)
>> +#define MMC_CQE_QUEUE_FULL	BIT(1)
>>   };
>>
>>   extern int mmc_init_queue(struct mmc_queue *, struct mmc_card *,
>> spinlock_t *,
>> -			  const char *);
>> +			  const char *, int);
>>   extern void mmc_cleanup_queue(struct mmc_queue *);  extern void
>> mmc_queue_suspend(struct mmc_queue *);  extern void
>> mmc_queue_resume(struct mmc_queue *); @@ -85,4 +107,22 @@ extern
>> unsigned int mmc_queue_map_sg(struct mmc_queue *,
>>
>>   extern int mmc_access_rpmb(struct mmc_queue *);
>>
>> +void mmc_cqe_kick_queue(struct mmc_queue *mq);
>> +
>> +enum mmc_issue_type mmc_cqe_issue_type(struct mmc_host *host,
>> +				       struct request *req);
>> +
>> +static inline int mmc_cqe_tot_in_flight(struct mmc_queue *mq) {
>> +	return mq->cqe_in_flight[MMC_ISSUE_SYNC] +
>> +	       mq->cqe_in_flight[MMC_ISSUE_DCMD] +
>> +	       mq->cqe_in_flight[MMC_ISSUE_ASYNC];
>> +}
>> +
>> +static inline int mmc_cqe_qcnt(struct mmc_queue *mq) {
>> +	return mq->cqe_in_flight[MMC_ISSUE_DCMD] +
>> +	       mq->cqe_in_flight[MMC_ISSUE_ASYNC];
>> +}
>> +
>>   #endif
>> --
>> 1.9.1
> 
> 
> 
> 

Re: [PATCH V4 09/11] mmc: block: Add CQE support

[Adrian Hunter]

On 09/08/17 03:55, Shawn Lin wrote:
> Hi,
> 
> On 2017/8/8 20:07, Bough Chen wrote:
>>> -----Original Message-----
>>> From: Adrian Hunter [mailto:adrian.hunter@intel.com]
>>> Sent: Friday, July 21, 2017 5:50 PM
>>> To: Ulf Hansson <ulf.hansson@linaro.org>
>>> Cc: linux-mmc <linux-mmc@vger.kernel.org>; Bough Chen
>>> <haibo.chen@nxp.com>; Alex Lemberg <alex.lemberg@sandisk.com>;
>>> Mateusz Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
>>> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung <jh80.chung@samsung.com>;
>>> Dong Aisheng <dongas86@gmail.com>; Das Asutosh
>>> <asutoshd@codeaurora.org>; Zhangfei Gao <zhangfei.gao@gmail.com>;
>>> Dorfman Konstantin <kdorfman@codeaurora.org>; David Griego
>>> <david.griego@linaro.org>; Sahitya Tummala <stummala@codeaurora.org>;
>>> Harjani Ritesh <riteshh@codeaurora.org>; Venu Byravarasu
>>> <vbyravarasu@nvidia.com>; Linus Walleij <linus.walleij@linaro.org>; Shawn
>>> Lin
>>> <shawn.lin@rock-chips.com>
>>> Subject: [PATCH V4 09/11] mmc: block: Add CQE support
>>>
>>> Add CQE support to the block driver, including:
>>>     - optionally using DCMD for flush requests
>>>     - manually issuing discard requests
>>>     - issuing read / write requests to the CQE
>>>     - supporting block-layer timeouts
>>>     - handling recovery
>>>     - supporting re-tuning
>>>
>>> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
>>> ---
>>>   drivers/mmc/core/block.c | 195 ++++++++++++++++++++++++++++++++-
>>>   drivers/mmc/core/block.h |   7 ++
>>>   drivers/mmc/core/queue.c | 273
>>> ++++++++++++++++++++++++++++++++++++++++++++++-
>>>   drivers/mmc/core/queue.h |  42 +++++++-
>>>   4 files changed, 510 insertions(+), 7 deletions(-)
>>>
>>> diff --git a/drivers/mmc/core/block.c b/drivers/mmc/core/block.c index
>>> 915290c74363..2d25115637b7 100644
>>> --- a/drivers/mmc/core/block.c
>>> +++ b/drivers/mmc/core/block.c
>>> @@ -109,6 +109,7 @@ struct mmc_blk_data {
>>>   #define MMC_BLK_WRITE        BIT(1)
>>>   #define MMC_BLK_DISCARD        BIT(2)
>>>   #define MMC_BLK_SECDISCARD    BIT(3)
>>> +#define MMC_BLK_CQE_RECOVERY    BIT(4)
>>>
>>>       /*
>>>        * Only set in main mmc_blk_data associated @@ -1612,6 +1613,198
>>> @@ static void mmc_blk_data_prep(struct mmc_queue *mq, struct
>>> mmc_queue_req *mqrq,
>>>           *do_data_tag_p = do_data_tag;
>>>   }
>>>
>>> +#define MMC_CQE_RETRIES 2
> 
> 
>>> +        blk_queue_rq_timed_out(mq->queue, mmc_cqe_timed_out);
>>> +        blk_queue_rq_timeout(mq->queue, 60 * HZ);
>>
> 
> ------8<-------
> 
>> Hi Adrian,
>>
>> These days I'm doing CMDQ stress test, and find one issue.
>> On our i.MX8QXP-ARM2 board, the RAM is 3GB. eMMC is 32GB.
>> I use command 'free -m' get the total memory is 2800M, and the free memory
>> is 2500M.
>>
>> I use 'mkfs.ext4' to format ext4 file system on the eMMC under HS400ES
>> CMDQ mode, works fine.
>>
>> When I use the following command to stress test CMDQ, it works fine.
>> bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 2048 -r 1024
>>
>> But when I change to use a large file size to do the same stress test, using
>> bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 4096 -r 2048
>> or
>> bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 5600
>>
>> I get the following dump message.  According to the log,
>> mmc_cqe_timed_out() was trigged.
>> Seems mmc was blocked in somewhere.
>> Then I try to debug this issue, and open MMC_DEBUG in config, do the same
>> test, print the detail
>> Command sending information on the console, but finally can't reproduce.

mmc_cqe_timed_out() is a 60 second timeout provided by the block layer.
Refer "blk_queue_rq_timeout(mq->queue, 60 * HZ)" in mmc_init_queue().  60s
is quite a long time so I would first want to determine if the task was
really queued that long.  I would instrument some code into cqhci_request()
to record the start time on struct mmc_request, and then print the time
taken when there is a problem.

>>
>> Shawn,
>> Can you have a try on your side?
>>
> 
> I think bonnie++ is almost the same disk test tool as iozone or fio. I
> didn't saw this when testing CMDQ by fio, but I will try bonnie++ late
> this week.
> 
> 
>>
>> [  738.385610] mmc0: cqhci: timeout for tag 1
>> [  738.389719] mmc0: cqhci: ============ CQHCI REGISTER DUMP ===========
>> [  738.396164] mmc0: cqhci: Caps:      0x0000310a | Version:  0x00000510
>> [  738.402601] mmc0: cqhci: Config:    0x00001001 | Control:  0x00000000
>> [  738.409038] mmc0: cqhci: Int stat:  0x00000000 | Int enab: 0x00000006
>> [  738.415475] mmc0: cqhci: Int sig:   0x00000006 | Int Coal: 0x00000000
>> [  738.421913] mmc0: cqhci: TDL base:  0x9007a000 | TDL up32: 0x00000000
>> [  738.428350] mmc0: cqhci: Doorbell:  0x1fffffff | TCN:      0x00000000
>> [  738.434788] mmc0: cqhci: Dev queue: 0x1f7fffff | Dev Pend: 0x1fffefff
>> [  738.441226] mmc0: cqhci: Task clr:  0x00000000 | SSC1:     0x00011000
>> [  738.447663] mmc0: cqhci: SSC2:      0x00000001 | DCMD rsp: 0x00000800
>> [  738.454100] mmc0: cqhci: RED mask:  0xfdf9a080 | TERRI:    0x00000000
>> [  738.460538] mmc0: cqhci: Resp idx:  0x0000002f | Resp arg: 0x00000900
>> [  738.466975] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
>> [  738.473414] mmc0: sdhci: Sys addr:  0xb6512000 | Version:  0x00000002
>> [  738.479850] mmc0: sdhci: Blk size:  0x00000200 | Blk cnt:  0x00000400
>> [  738.486288] mmc0: sdhci: Argument:  0x000c0400 | Trn mode: 0x00000023
>> [  738.492725] mmc0: sdhci: Present:   0x01fd858f | Host ctl: 0x00000030
>> [  738.499162] mmc0: sdhci: Power:     0x00000002 | Blk gap:  0x00000080
>> [  738.505600] mmc0: sdhci: Wake-up:   0x00000008 | Clock:    0x0000000f
>> [  738.512037] mmc0: sdhci: Timeout:   0x0000008f | Int stat: 0x00000000
>> [  738.518475] mmc0: sdhci: Int enab:  0x107f4000 | Sig enab: 0x107f4000
>> [  738.524912] mmc0: sdhci: AC12 err:  0x00000000 | Slot int: 0x00000502
>> [  738.531350] mmc0: sdhci: Caps:      0x07eb0000 | Caps_1:   0x8000b407
>> [  738.537787] mmc0: sdhci: Cmd:       0x00002c1a | Max curr: 0x00ffffff
>> [  738.544225] mmc0: sdhci: Resp[0]:   0x00000900 | Resp[1]:  0xffffffff
>> [  738.550662] mmc0: sdhci: Resp[2]:   0x328f5903 | Resp[3]:  0x00d02700
>> [  738.557099] mmc0: sdhci: Host ctl2: 0x00000008
>> [  738.561540] mmc0: sdhci: ADMA Err:  0x00000009 | ADMA Ptr: 0x90098400
>> [  738.567975] mmc0: sdhci: ============================================
>> [  738.574449] mmc0: running CQE recovery
>> [  738.593643] mmc0: Unexpected interrupt 0x00004000.
>> [  738.598436] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
>> [  738.604881] mmc0: sdhci: Sys addr:  0x00000000 | Version:  0x00000002
>> [  738.611318] mmc0: sdhci: Blk size:  0x00000200 | Blk cnt:  0x00000400
>> [  738.617756] mmc0: sdhci: Argument:  0x01af6800 | Trn mode: 0x00000023
>> [  738.624193] mmc0: sdhci: Present:   0x01fd8009 | Host ctl: 0x00000031
>> [  738.630630] mmc0: sdhci: Power:     0x00000002 | Blk gap:  0x00000080
>> [  738.637068] mmc0: sdhci: Wake-up:   0x00000008 | Clock:    0x0000000f
>> [  738.643505] mmc0: sdhci: Timeout:   0x0000008f | Int stat: 0x00004000
>> [  738.649943] mmc0: sdhci: Int enab:  0x007f1003 | Sig enab: 0x007f1003
>> [  738.656380] mmc0: sdhci: AC12 err:  0x00000000 | Slot int: 0x00000502
>> [  738.662818] mmc0: sdhci: Caps:      0x07eb0000 | Caps_1:   0x8000b407
>> [  738.669255] mmc0: sdhci: Cmd:       0x00002d12 | Max curr: 0x00ffffff
>> [  738.675693] mmc0: sdhci: Resp[0]:   0x00000c00 | Resp[1]:  0xffffffff
>> [  738.682130] mmc0: sdhci: Resp[2]:   0x328f5903 | Resp[3]:  0x00d02700
>> [  738.688566] mmc0: sdhci: Host ctl2: 0x00000008
>> [  738.693008] mmc0: sdhci: ADMA Err:  0x00000000 | ADMA Ptr: 0x00000000
>> [  738.699443] mmc0: sdhci: ============================================
>> [  738.715999] mmc0: Controller never released inhibit bit(s).
>> [  738.721573] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
>> [  738.728018] mmc0: sdhci: Sys addr:  0x00000000 | Version:  0x00000002
>> [  738.734455] mmc0: sdhci: Blk size:  0x00000200 | Blk cnt:  0x00000400
>> [  738.740892] mmc0: sdhci: Argument:  0x01af6800 | Trn mode: 0x00000023
>> [  738.747330] mmc0: sdhci: Present:   0x01fd8009 | Host ctl: 0x00000031
>> [  738.753767] mmc0: sdhci: Power:     0x00000002 | Blk gap:  0x00000080
>> [  738.760204] mmc0: sdhci: Wake-up:   0x00000008 | Clock:    0x0000000f
>> [  738.766642] mmc0: sdhci: Timeout:   0x0000008f | Int stat: 0x00004000
>> [  738.773079] mmc0: sdhci: Int enab:  0x007f1003 | Sig enab: 0x007f1003
>> [  738.779517] mmc0: sdhci: AC12 err:  0x00000000 | Slot int: 0x00000502
>> [  738.785955] mmc0: sdhci: Caps:      0x07eb0000 | Caps_1:   0x8000b407
>> [  738.792392] mmc0: sdhci: Cmd:       0x00002d12 | Max curr: 0x00ffffff
>> [  738.798829] mmc0: sdhci: Resp[0]:   0x00000c00 | Resp[1]:  0xffffffff
>> [  738.805266] mmc0: sdhci: Resp[2]:   0x328f5903 | Resp[3]:  0x00d02700
>> [  738.811703] mmc0: sdhci: Host ctl2: 0x00000008
>> [  738.816144] mmc0: sdhci: ADMA Err:  0x00000000 | ADMA Ptr: 0x00000000
>> [  738.822579] mmc0: sdhci: ============================================
>> [  748.881580] mmc0: Timeout waiting for hardware interrupt.
>> ......
>>
>>
>>> +
>>> +        host->cqe_recovery_notifier = mmc_cqe_recovery_notifier;
>>> +    }
>>> +
>>>       blk_queue_prep_rq(mq->queue, mmc_prep_request);
>>>       queue_flag_set_unlocked(QUEUE_FLAG_NONROT, mq->queue);
>>>       queue_flag_clear_unlocked(QUEUE_FLAG_ADD_RANDOM, mq-
>>>> queue); @@ -280,9 +543,9 @@ int mmc_init_queue(struct mmc_queue *mq,
>>> struct mmc_card *card,
>>>
>>>       sema_init(&mq->thread_sem, 1);
>>>
>>> -    mq->thread = kthread_run(mmc_queue_thread, mq, "mmcqd/%d%s",
>>> -        host->index, subname ? subname : "");
>>> -
>>> +    mq->thread = kthread_run(use_cqe ? mmc_cqe_thread :
>>> mmc_queue_thread,
>>> +                 mq, "mmcqd/%d%s", host->index,
>>> +                 subname ? subname : "");
>>>       if (IS_ERR(mq->thread)) {
>>>           ret = PTR_ERR(mq->thread);
>>>           goto cleanup_queue;
>>> diff --git a/drivers/mmc/core/queue.h b/drivers/mmc/core/queue.h index
>>> 361b46408e0f..8e9273d977c0 100644
>>> --- a/drivers/mmc/core/queue.h
>>> +++ b/drivers/mmc/core/queue.h
>>> @@ -7,6 +7,20 @@
>>>   #include <linux/mmc/core.h>
>>>   #include <linux/mmc/host.h>
>>>
>>> +enum mmc_issued {
>>> +    MMC_REQ_STARTED,
>>> +    MMC_REQ_BUSY,
>>> +    MMC_REQ_FAILED_TO_START,
>>> +    MMC_REQ_FINISHED,
>>> +};
>>> +
>>> +enum mmc_issue_type {
>>> +    MMC_ISSUE_SYNC,
>>> +    MMC_ISSUE_DCMD,
>>> +    MMC_ISSUE_ASYNC,
>>> +    MMC_ISSUE_MAX,
>>> +};
>>> +
>>>   static inline struct mmc_queue_req *req_to_mmc_queue_req(struct request
>>> *rq)  {
>>>       return blk_mq_rq_to_pdu(rq);
>>> @@ -53,6 +67,7 @@ struct mmc_queue_req {
>>>       int            drv_op_result;
>>>       struct mmc_blk_ioc_data    **idata;
>>>       unsigned int        ioc_count;
>>> +    int            retries;
>>>   };
>>>
>>>   struct mmc_queue {
>>> @@ -70,10 +85,17 @@ struct mmc_queue {
>>>        * associated mmc_queue_req data.
>>>        */
>>>       int            qcnt;
>>> +    /* Following are defined for a Command Queue Engine */
>>> +    int            cqe_in_flight[MMC_ISSUE_MAX];
>>> +    unsigned int        cqe_busy;
>>> +    bool            cqe_recovery_needed;
>>> +    bool            cqe_in_recovery;
>>> +#define MMC_CQE_DCMD_BUSY    BIT(0)
>>> +#define MMC_CQE_QUEUE_FULL    BIT(1)
>>>   };
>>>
>>>   extern int mmc_init_queue(struct mmc_queue *, struct mmc_card *,
>>> spinlock_t *,
>>> -              const char *);
>>> +              const char *, int);
>>>   extern void mmc_cleanup_queue(struct mmc_queue *);  extern void
>>> mmc_queue_suspend(struct mmc_queue *);  extern void
>>> mmc_queue_resume(struct mmc_queue *); @@ -85,4 +107,22 @@ extern
>>> unsigned int mmc_queue_map_sg(struct mmc_queue *,
>>>
>>>   extern int mmc_access_rpmb(struct mmc_queue *);
>>>
>>> +void mmc_cqe_kick_queue(struct mmc_queue *mq);
>>> +
>>> +enum mmc_issue_type mmc_cqe_issue_type(struct mmc_host *host,
>>> +                       struct request *req);
>>> +
>>> +static inline int mmc_cqe_tot_in_flight(struct mmc_queue *mq) {
>>> +    return mq->cqe_in_flight[MMC_ISSUE_SYNC] +
>>> +           mq->cqe_in_flight[MMC_ISSUE_DCMD] +
>>> +           mq->cqe_in_flight[MMC_ISSUE_ASYNC];
>>> +}
>>> +
>>> +static inline int mmc_cqe_qcnt(struct mmc_queue *mq) {
>>> +    return mq->cqe_in_flight[MMC_ISSUE_DCMD] +
>>> +           mq->cqe_in_flight[MMC_ISSUE_ASYNC];
>>> +}
>>> +
>>>   #endif
>>> -- 
>>> 1.9.1
>>
>>
>>
>>
> 
> 

RE: [PATCH V4 09/11] mmc: block: Add CQE support

[Bough Chen]

> -----Original Message-----
> From: linux-mmc-owner@vger.kernel.org [mailto:linux-mmc-
> owner@vger.kernel.org] On Behalf Of Adrian Hunter
> Sent: Wednesday, August 09, 2017 1:58 PM
> To: Shawn Lin <shawn.lin@rock-chips.com>; Bough Chen
> <haibo.chen@nxp.com>
> Cc: Ulf Hansson <ulf.hansson@linaro.org>; linux-mmc <linux-
> mmc@vger.kernel.org>; Alex Lemberg <alex.lemberg@sandisk.com>; Mateusz
> Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung <jh80.chung@samsung.com>;
> Dong Aisheng <dongas86@gmail.com>; Das Asutosh
> <asutoshd@codeaurora.org>; Zhangfei Gao <zhangfei.gao@gmail.com>;
> Dorfman Konstantin <kdorfman@codeaurora.org>; Sahitya Tummala
> <stummala@codeaurora.org>; Harjani Ritesh <riteshh@codeaurora.org>; Venu
> Byravarasu <vbyravarasu@nvidia.com>; Linus Walleij <linus.walleij@linaro.org>
> Subject: Re: [PATCH V4 09/11] mmc: block: Add CQE support
> 
> On 09/08/17 03:55, Shawn Lin wrote:
> > Hi,
> >
> > On 2017/8/8 20:07, Bough Chen wrote:
> >>> -----Original Message-----
> >>> From: Adrian Hunter [mailto:adrian.hunter@intel.com]
> >>> Sent: Friday, July 21, 2017 5:50 PM
> >>> To: Ulf Hansson <ulf.hansson@linaro.org>
> >>> Cc: linux-mmc <linux-mmc@vger.kernel.org>; Bough Chen
> >>> <haibo.chen@nxp.com>; Alex Lemberg <alex.lemberg@sandisk.com>;
> >>> Mateusz Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
> >>> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung
> >>> <jh80.chung@samsung.com>; Dong Aisheng <dongas86@gmail.com>; Das
> >>> Asutosh <asutoshd@codeaurora.org>; Zhangfei Gao
> >>> <zhangfei.gao@gmail.com>; Dorfman Konstantin
> >>> <kdorfman@codeaurora.org>; David Griego <david.griego@linaro.org>;
> >>> Sahitya Tummala <stummala@codeaurora.org>; Harjani Ritesh
> >>> <riteshh@codeaurora.org>; Venu Byravarasu <vbyravarasu@nvidia.com>;
> >>> Linus Walleij <linus.walleij@linaro.org>; Shawn Lin
> >>> <shawn.lin@rock-chips.com>
> >>> Subject: [PATCH V4 09/11] mmc: block: Add CQE support
> >>>
> >>> Add CQE support to the block driver, including:
> >>>     - optionally using DCMD for flush requests
> >>>     - manually issuing discard requests
> >>>     - issuing read / write requests to the CQE
> >>>     - supporting block-layer timeouts
> >>>     - handling recovery
> >>>     - supporting re-tuning
> >>>
> >>> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
> >>> ---
> >>>   drivers/mmc/core/block.c | 195
> ++++++++++++++++++++++++++++++++-
> >>>   drivers/mmc/core/block.h |   7 ++
> >>>   drivers/mmc/core/queue.c | 273
> >>> ++++++++++++++++++++++++++++++++++++++++++++++-
> >>>   drivers/mmc/core/queue.h |  42 +++++++-
> >>>   4 files changed, 510 insertions(+), 7 deletions(-)
> >>>
> >>> diff --git a/drivers/mmc/core/block.c b/drivers/mmc/core/block.c
> >>> index
> >>> 915290c74363..2d25115637b7 100644
> >>> --- a/drivers/mmc/core/block.c
> >>> +++ b/drivers/mmc/core/block.c
> >>> @@ -109,6 +109,7 @@ struct mmc_blk_data {
> >>>   #define MMC_BLK_WRITE        BIT(1)
> >>>   #define MMC_BLK_DISCARD        BIT(2)
> >>>   #define MMC_BLK_SECDISCARD    BIT(3)
> >>> +#define MMC_BLK_CQE_RECOVERY    BIT(4)
> >>>
> >>>       /*
> >>>        * Only set in main mmc_blk_data associated @@ -1612,6
> >>> +1613,198 @@ static void mmc_blk_data_prep(struct mmc_queue *mq,
> >>> struct mmc_queue_req *mqrq,
> >>>           *do_data_tag_p = do_data_tag;
> >>>   }
> >>>
> >>> +#define MMC_CQE_RETRIES 2
> >
> >
> >>> +        blk_queue_rq_timed_out(mq->queue, mmc_cqe_timed_out);
> >>> +        blk_queue_rq_timeout(mq->queue, 60 * HZ);
> >>
> >
> > ------8<-------
> >
> >> Hi Adrian,
> >>
> >> These days I'm doing CMDQ stress test, and find one issue.
> >> On our i.MX8QXP-ARM2 board, the RAM is 3GB. eMMC is 32GB.
> >> I use command 'free -m' get the total memory is 2800M, and the free
> >> memory is 2500M.
> >>
> >> I use 'mkfs.ext4' to format ext4 file system on the eMMC under
> >> HS400ES CMDQ mode, works fine.
> >>
> >> When I use the following command to stress test CMDQ, it works fine.
> >> bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 2048 -r 1024
> >>
> >> But when I change to use a large file size to do the same stress
> >> test, using
> >> bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 4096 -r 2048
> >> or
> >> bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 5600
> >>
> >> I get the following dump message.  According to the log,
> >> mmc_cqe_timed_out() was trigged.
> >> Seems mmc was blocked in somewhere.
> >> Then I try to debug this issue, and open MMC_DEBUG in config, do the
> >> same test, print the detail Command sending information on the
> >> console, but finally can't reproduce.
> 
> mmc_cqe_timed_out() is a 60 second timeout provided by the block layer.
> Refer "blk_queue_rq_timeout(mq->queue, 60 * HZ)" in mmc_init_queue().
> 60s is quite a long time so I would first want to determine if the task was really
> queued that long.  I would instrument some code into cqhci_request() to
> record the start time on struct mmc_request, and then print the time taken
> when there is a problem.
> 

Hi Adrian, 

According to your suggestion, I add the following code to print the time.
When issue happens, seems the request really pending for over 60s!

done
Writing intelligently...[  689.209548] mmc0: cqhci: timeout for tag 9
[  689.213658] the mrq all use 62123742 us
[  689.217487] mmc0: cqhci: ============ CQHCI REGISTER DUMP ===========
[  689.223927] mmc0: cqhci: Caps:      0x0000310a | Version:  0x00000510
[  689.230363] mmc0: cqhci: Config:    0x00001001 | Control:  0x00000000
[  689.236800] mmc0: cqhci: Int stat:  0x00000000 | Int enab: 0x00000006
[  689.243238] mmc0: cqhci: Int sig:   0x00000006 | Int Coal: 0x00000000
[  689.249675] mmc0: cqhci: TDL base:  0x90079000 | TDL up32: 0x00000000
[  689.256113] mmc0: cqhci: Doorbell:  0x1fffffff | TCN:      0x00000000
[  689.262550] mmc0: cqhci: Dev queue: 0x1fffefff | Dev Pend: 0x1fff7fff
[  689.268988] mmc0: cqhci: Task clr:  0x00000000 | SSC1:     0x00011000
[  689.275425] mmc0: cqhci: SSC2:      0x00000001 | DCMD rsp: 0x00000800
[  689.281862] mmc0: cqhci: RED mask:  0xfdf9a080 | TERRI:    0x00000000
[  689.288300] mmc0: cqhci: Resp idx:  0x0000002f | Resp arg: 0x00000900
[  689.294737] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
[  689.301176] mmc0: sdhci: Sys addr:  0xb602f000 | Version:  0x00000002
[  689.307612] mmc0: sdhci: Blk size:  0x00000200 | Blk cnt:  0x00000400
[  689.314050] mmc0: sdhci: Argument:  0x000f0400 | Trn mode: 0x00000023
[  689.320487] mmc0: sdhci: Present:   0x01fd858f | Host ctl: 0x00000030
[  689.326925] mmc0: sdhci: Power:     0x00000002 | Blk gap:  0x00000080
[  689.333362] mmc0: sdhci: Wake-up:   0x00000008 | Clock:    0x0000000f
[  689.339800] mmc0: sdhci: Timeout:   0x0000008f | Int stat: 0x00000000
[  689.346237] mmc0: sdhci: Int enab:  0x107f4000 | Sig enab: 0x107f4000
[  689.352674] mmc0: sdhci: AC12 err:  0x00000000 | Slot int: 0x00000502
[  689.359113] mmc0: sdhci: Caps:      0x07eb0000 | Caps_1:   0x8000b407
[  689.365549] mmc0: sdhci: Cmd:       0x00002c1a | Max curr: 0x00ffffff
[  689.371987] mmc0: sdhci: Resp[0]:   0x00000900 | Resp[1]:  0xffffffff
[  689.378424] mmc0: sdhci: Resp[2]:   0x328f5903 | Resp[3]:  0x00d02700
[  689.384861] mmc0: sdhci: Host ctl2: 0x00000008
[  689.389302] mmc0: sdhci: ADMA Err:  0x00000009 | ADMA Ptr: 0x9009a400
[  689.395737] mmc0: sdhci: ============================================
[  689.402212] mmc0: running CQE recovery

diff --git a/drivers/mmc/host/cqhci.c b/drivers/mmc/host/cqhci.c
index 1b56d03..7359895 100644
--- a/drivers/mmc/host/cqhci.c
+++ b/drivers/mmc/host/cqhci.c
@@ -556,6 +556,7 @@ static int cqhci_request(struct mmc_host *mmc, struct mmc_request *mrq)
        u64 *task_desc = NULL;
        int tag = cqhci_tag(mrq);
        struct cqhci_host *cq_host = mmc->cqe_private;
+       struct timeval *start_time = &mrq->start_time;
        unsigned long flags;

        if (!cq_host->enabled) {
@@ -605,6 +606,8 @@ static int cqhci_request(struct mmc_host *mmc, struct mmc_request *mrq)

        cq_host->qcnt += 1;

+       do_gettimeofday(start_time);
+
        cqhci_writel(cq_host, 1 << tag, CQHCI_TDBR);
        if (!(cqhci_readl(cq_host, CQHCI_TDBR) & (1 << tag)))
                pr_debug("%s: cqhci: doorbell not set for tag %d\n",
@@ -822,6 +825,8 @@ static bool cqhci_timeout(struct mmc_host *mmc, struct mmc_request *mrq,
        struct cqhci_host *cq_host = mmc->cqe_private;
        int tag = cqhci_tag(mrq);
        struct cqhci_slot *slot = &cq_host->slot[tag];
+       struct timeval *end_time = &mrq->end_time;
+       struct timeval *start_time = &mrq->start_time;
        unsigned long flags;
        bool timed_out;

@@ -835,8 +840,11 @@ static bool cqhci_timeout(struct mmc_host *mmc, struct mmc_request *mrq,
        spin_unlock_irqrestore(&cq_host->lock, flags);

        if (timed_out) {
+               do_gettimeofday(end_time);
+
                pr_err("%s: cqhci: timeout for tag %d\n",
                       mmc_hostname(mmc), tag);
+               pr_err("the mrq all use %ld us\n", (end_time->tv_sec - start_time->tv_sec) * 1000000 + end_time->tv_usec - start_time->tv_usec);
                cqhci_dumpregs(cq_host);
        }

diff --git a/include/linux/mmc/core.h b/include/linux/mmc/core.h
index 80f36b7..15093aa 100644
--- a/include/linux/mmc/core.h
+++ b/include/linux/mmc/core.h
@@ -10,6 +10,7 @@

 #include <linux/completion.h>
 #include <linux/types.h>
+#include <linux/time.h>

 struct mmc_data;
 struct mmc_request;
@@ -157,6 +158,8 @@ struct mmc_request {
        struct completion       cmd_completion;
        void                    (*done)(struct mmc_request *);/* completion function */
        struct mmc_host         *host;
+       struct timeval          start_time;
+       struct timeval          end_time;

        /* Allow other commands during this ongoing data transfer or busy wait */
        bool                    cap_cmd_during_tfr;


> >>
> >> Shawn,
> >> Can you have a try on your side?
> >>
> >
> > I think bonnie++ is almost the same disk test tool as iozone or fio. I
> > didn't saw this when testing CMDQ by fio, but I will try bonnie++ late
> > this week.
> >
> >
> >>
> >> [  738.385610] mmc0: cqhci: timeout for tag 1 [  738.389719] mmc0:
> >> cqhci: ============ CQHCI REGISTER DUMP ===========
> >> [  738.396164] mmc0: cqhci: Caps:      0x0000310a | Version:  0x00000510
> >> [  738.402601] mmc0: cqhci: Config:    0x00001001 | Control:  0x00000000
> >> [  738.409038] mmc0: cqhci: Int stat:  0x00000000 | Int enab: 0x00000006
> >> [  738.415475] mmc0: cqhci: Int sig:   0x00000006 | Int Coal: 0x00000000
> >> [  738.421913] mmc0: cqhci: TDL base:  0x9007a000 | TDL up32: 0x00000000
> >> [  738.428350] mmc0: cqhci: Doorbell:  0x1fffffff | TCN:      0x00000000
> >> [  738.434788] mmc0: cqhci: Dev queue: 0x1f7fffff | Dev Pend: 0x1fffefff
> >> [  738.441226] mmc0: cqhci: Task clr:  0x00000000 | SSC1:     0x00011000
> >> [  738.447663] mmc0: cqhci: SSC2:      0x00000001 | DCMD rsp: 0x00000800
> >> [  738.454100] mmc0: cqhci: RED mask:  0xfdf9a080 | TERRI:    0x00000000
> >> [  738.460538] mmc0: cqhci: Resp idx:  0x0000002f | Resp arg:
> >> 0x00000900 [  738.466975] mmc0: sdhci: ============ SDHCI REGISTER
> >> DUMP =========== [  738.473414] mmc0: sdhci: Sys addr:  0xb6512000 |
> >> Version:  0x00000002 [  738.479850] mmc0: sdhci: Blk size:
> >> 0x00000200 | Blk cnt:  0x00000400 [  738.486288] mmc0: sdhci: Argument:
> 0x000c0400 | Trn mode: 0x00000023
> >> [  738.492725] mmc0: sdhci: Present:   0x01fd858f | Host ctl: 0x00000030
> >> [  738.499162] mmc0: sdhci: Power:     0x00000002 | Blk gap:  0x00000080
> >> [  738.505600] mmc0: sdhci: Wake-up:   0x00000008 | Clock:    0x0000000f
> >> [  738.512037] mmc0: sdhci: Timeout:   0x0000008f | Int stat: 0x00000000
> >> [  738.518475] mmc0: sdhci: Int enab:  0x107f4000 | Sig enab:
> >> 0x107f4000 [  738.524912] mmc0: sdhci: AC12 err:  0x00000000 | Slot int:
> 0x00000502
> >> [  738.531350] mmc0: sdhci: Caps:      0x07eb0000 | Caps_1:   0x8000b407
> >> [  738.537787] mmc0: sdhci: Cmd:       0x00002c1a | Max curr: 0x00ffffff
> >> [  738.544225] mmc0: sdhci: Resp[0]:   0x00000900 | Resp[1]:  0xffffffff
> >> [  738.550662] mmc0: sdhci: Resp[2]:   0x328f5903 | Resp[3]:  0x00d02700
> >> [  738.557099] mmc0: sdhci: Host ctl2: 0x00000008 [  738.561540]
> >> mmc0: sdhci: ADMA Err:  0x00000009 | ADMA Ptr: 0x90098400 [
> >> 738.567975] mmc0: sdhci:
> ============================================
> >> [  738.574449] mmc0: running CQE recovery [  738.593643] mmc0:
> >> Unexpected interrupt 0x00004000.
> >> [  738.598436] mmc0: sdhci: ============ SDHCI REGISTER DUMP
> >> =========== [  738.604881] mmc0: sdhci: Sys addr:  0x00000000 |
> >> Version:  0x00000002 [  738.611318] mmc0: sdhci: Blk size:
> >> 0x00000200 | Blk cnt:  0x00000400 [  738.617756] mmc0: sdhci: Argument:
> 0x01af6800 | Trn mode: 0x00000023
> >> [  738.624193] mmc0: sdhci: Present:   0x01fd8009 | Host ctl: 0x00000031
> >> [  738.630630] mmc0: sdhci: Power:     0x00000002 | Blk gap:  0x00000080
> >> [  738.637068] mmc0: sdhci: Wake-up:   0x00000008 | Clock:    0x0000000f
> >> [  738.643505] mmc0: sdhci: Timeout:   0x0000008f | Int stat: 0x00004000
> >> [  738.649943] mmc0: sdhci: Int enab:  0x007f1003 | Sig enab:
> >> 0x007f1003 [  738.656380] mmc0: sdhci: AC12 err:  0x00000000 | Slot int:
> 0x00000502
> >> [  738.662818] mmc0: sdhci: Caps:      0x07eb0000 | Caps_1:   0x8000b407
> >> [  738.669255] mmc0: sdhci: Cmd:       0x00002d12 | Max curr: 0x00ffffff
> >> [  738.675693] mmc0: sdhci: Resp[0]:   0x00000c00 | Resp[1]:  0xffffffff
> >> [  738.682130] mmc0: sdhci: Resp[2]:   0x328f5903 | Resp[3]:  0x00d02700
> >> [  738.688566] mmc0: sdhci: Host ctl2: 0x00000008 [  738.693008]
> >> mmc0: sdhci: ADMA Err:  0x00000000 | ADMA Ptr: 0x00000000 [
> >> 738.699443] mmc0: sdhci:
> ============================================
> >> [  738.715999] mmc0: Controller never released inhibit bit(s).
> >> [  738.721573] mmc0: sdhci: ============ SDHCI REGISTER DUMP
> >> =========== [  738.728018] mmc0: sdhci: Sys addr:  0x00000000 |
> >> Version:  0x00000002 [  738.734455] mmc0: sdhci: Blk size:
> >> 0x00000200 | Blk cnt:  0x00000400 [  738.740892] mmc0: sdhci: Argument:
> 0x01af6800 | Trn mode: 0x00000023
> >> [  738.747330] mmc0: sdhci: Present:   0x01fd8009 | Host ctl: 0x00000031
> >> [  738.753767] mmc0: sdhci: Power:     0x00000002 | Blk gap:  0x00000080
> >> [  738.760204] mmc0: sdhci: Wake-up:   0x00000008 | Clock:    0x0000000f
> >> [  738.766642] mmc0: sdhci: Timeout:   0x0000008f | Int stat: 0x00004000
> >> [  738.773079] mmc0: sdhci: Int enab:  0x007f1003 | Sig enab:
> >> 0x007f1003 [  738.779517] mmc0: sdhci: AC12 err:  0x00000000 | Slot int:
> 0x00000502
> >> [  738.785955] mmc0: sdhci: Caps:      0x07eb0000 | Caps_1:   0x8000b407
> >> [  738.792392] mmc0: sdhci: Cmd:       0x00002d12 | Max curr: 0x00ffffff
> >> [  738.798829] mmc0: sdhci: Resp[0]:   0x00000c00 | Resp[1]:  0xffffffff
> >> [  738.805266] mmc0: sdhci: Resp[2]:   0x328f5903 | Resp[3]:  0x00d02700
> >> [  738.811703] mmc0: sdhci: Host ctl2: 0x00000008 [  738.816144]
> >> mmc0: sdhci: ADMA Err:  0x00000000 | ADMA Ptr: 0x00000000 [
> >> 738.822579] mmc0: sdhci:
> ============================================
> >> [  748.881580] mmc0: Timeout waiting for hardware interrupt.
> >> ......
> >>
> >>
> >>> +
> >>> +        host->cqe_recovery_notifier = mmc_cqe_recovery_notifier;
> >>> +    }
> >>> +
> >>>       blk_queue_prep_rq(mq->queue, mmc_prep_request);
> >>>       queue_flag_set_unlocked(QUEUE_FLAG_NONROT, mq->queue);
> >>>       queue_flag_clear_unlocked(QUEUE_FLAG_ADD_RANDOM, mq-
> >>>> queue); @@ -280,9 +543,9 @@ int mmc_init_queue(struct mmc_queue
> >>>> *mq,
> >>> struct mmc_card *card,
> >>>
> >>>       sema_init(&mq->thread_sem, 1);
> >>>
> >>> -    mq->thread = kthread_run(mmc_queue_thread, mq, "mmcqd/%d%s",
> >>> -        host->index, subname ? subname : "");
> >>> -
> >>> +    mq->thread = kthread_run(use_cqe ? mmc_cqe_thread :
> >>> mmc_queue_thread,
> >>> +                 mq, "mmcqd/%d%s", host->index,
> >>> +                 subname ? subname : "");
> >>>       if (IS_ERR(mq->thread)) {
> >>>           ret = PTR_ERR(mq->thread);
> >>>           goto cleanup_queue;
> >>> diff --git a/drivers/mmc/core/queue.h b/drivers/mmc/core/queue.h
> >>> index
> >>> 361b46408e0f..8e9273d977c0 100644
> >>> --- a/drivers/mmc/core/queue.h
> >>> +++ b/drivers/mmc/core/queue.h
> >>> @@ -7,6 +7,20 @@
> >>>   #include <linux/mmc/core.h>
> >>>   #include <linux/mmc/host.h>
> >>>
> >>> +enum mmc_issued {
> >>> +    MMC_REQ_STARTED,
> >>> +    MMC_REQ_BUSY,
> >>> +    MMC_REQ_FAILED_TO_START,
> >>> +    MMC_REQ_FINISHED,
> >>> +};
> >>> +
> >>> +enum mmc_issue_type {
> >>> +    MMC_ISSUE_SYNC,
> >>> +    MMC_ISSUE_DCMD,
> >>> +    MMC_ISSUE_ASYNC,
> >>> +    MMC_ISSUE_MAX,
> >>> +};
> >>> +
> >>>   static inline struct mmc_queue_req *req_to_mmc_queue_req(struct
> >>> request
> >>> *rq)  {
> >>>       return blk_mq_rq_to_pdu(rq);
> >>> @@ -53,6 +67,7 @@ struct mmc_queue_req {
> >>>       int            drv_op_result;
> >>>       struct mmc_blk_ioc_data    **idata;
> >>>       unsigned int        ioc_count;
> >>> +    int            retries;
> >>>   };
> >>>
> >>>   struct mmc_queue {
> >>> @@ -70,10 +85,17 @@ struct mmc_queue {
> >>>        * associated mmc_queue_req data.
> >>>        */
> >>>       int            qcnt;
> >>> +    /* Following are defined for a Command Queue Engine */
> >>> +    int            cqe_in_flight[MMC_ISSUE_MAX];
> >>> +    unsigned int        cqe_busy;
> >>> +    bool            cqe_recovery_needed;
> >>> +    bool            cqe_in_recovery;
> >>> +#define MMC_CQE_DCMD_BUSY    BIT(0)
> >>> +#define MMC_CQE_QUEUE_FULL    BIT(1)
> >>>   };
> >>>
> >>>   extern int mmc_init_queue(struct mmc_queue *, struct mmc_card *,
> >>> spinlock_t *,
> >>> -              const char *);
> >>> +              const char *, int);
> >>>   extern void mmc_cleanup_queue(struct mmc_queue *);  extern void
> >>> mmc_queue_suspend(struct mmc_queue *);  extern void
> >>> mmc_queue_resume(struct mmc_queue *); @@ -85,4 +107,22 @@
> extern
> >>> unsigned int mmc_queue_map_sg(struct mmc_queue *,
> >>>
> >>>   extern int mmc_access_rpmb(struct mmc_queue *);
> >>>
> >>> +void mmc_cqe_kick_queue(struct mmc_queue *mq);
> >>> +
> >>> +enum mmc_issue_type mmc_cqe_issue_type(struct mmc_host *host,
> >>> +                       struct request *req);
> >>> +
> >>> +static inline int mmc_cqe_tot_in_flight(struct mmc_queue *mq) {
> >>> +    return mq->cqe_in_flight[MMC_ISSUE_SYNC] +
> >>> +           mq->cqe_in_flight[MMC_ISSUE_DCMD] +
> >>> +           mq->cqe_in_flight[MMC_ISSUE_ASYNC];
> >>> +}
> >>> +
> >>> +static inline int mmc_cqe_qcnt(struct mmc_queue *mq) {
> >>> +    return mq->cqe_in_flight[MMC_ISSUE_DCMD] +
> >>> +           mq->cqe_in_flight[MMC_ISSUE_ASYNC];
> >>> +}
> >>> +
> >>>   #endif
> >>> --
> >>> 1.9.1
> >>
> >>
> >>
> >>
> >
> >
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-mmc" in the
> body of a message to majordomo@vger.kernel.org More majordomo info at
> http://vger.kernel.org/majordomo-info.html

Re: [PATCH V4 09/11] mmc: block: Add CQE support

[Adrian Hunter]

On 09/08/17 10:57, Bough Chen wrote:
>> -----Original Message-----
>> From: linux-mmc-owner@vger.kernel.org [mailto:linux-mmc-
>> owner@vger.kernel.org] On Behalf Of Adrian Hunter
>> Sent: Wednesday, August 09, 2017 1:58 PM
>> To: Shawn Lin <shawn.lin@rock-chips.com>; Bough Chen
>> <haibo.chen@nxp.com>
>> Cc: Ulf Hansson <ulf.hansson@linaro.org>; linux-mmc <linux-
>> mmc@vger.kernel.org>; Alex Lemberg <alex.lemberg@sandisk.com>; Mateusz
>> Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
>> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung <jh80.chung@samsung.com>;
>> Dong Aisheng <dongas86@gmail.com>; Das Asutosh
>> <asutoshd@codeaurora.org>; Zhangfei Gao <zhangfei.gao@gmail.com>;
>> Dorfman Konstantin <kdorfman@codeaurora.org>; Sahitya Tummala
>> <stummala@codeaurora.org>; Harjani Ritesh <riteshh@codeaurora.org>; Venu
>> Byravarasu <vbyravarasu@nvidia.com>; Linus Walleij <linus.walleij@linaro.org>
>> Subject: Re: [PATCH V4 09/11] mmc: block: Add CQE support
>>
>> On 09/08/17 03:55, Shawn Lin wrote:
>>> Hi,
>>>
>>> On 2017/8/8 20:07, Bough Chen wrote:
>>>>> -----Original Message-----
>>>>> From: Adrian Hunter [mailto:adrian.hunter@intel.com]
>>>>> Sent: Friday, July 21, 2017 5:50 PM
>>>>> To: Ulf Hansson <ulf.hansson@linaro.org>
>>>>> Cc: linux-mmc <linux-mmc@vger.kernel.org>; Bough Chen
>>>>> <haibo.chen@nxp.com>; Alex Lemberg <alex.lemberg@sandisk.com>;
>>>>> Mateusz Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
>>>>> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung
>>>>> <jh80.chung@samsung.com>; Dong Aisheng <dongas86@gmail.com>; Das
>>>>> Asutosh <asutoshd@codeaurora.org>; Zhangfei Gao
>>>>> <zhangfei.gao@gmail.com>; Dorfman Konstantin
>>>>> <kdorfman@codeaurora.org>; David Griego <david.griego@linaro.org>;
>>>>> Sahitya Tummala <stummala@codeaurora.org>; Harjani Ritesh
>>>>> <riteshh@codeaurora.org>; Venu Byravarasu <vbyravarasu@nvidia.com>;
>>>>> Linus Walleij <linus.walleij@linaro.org>; Shawn Lin
>>>>> <shawn.lin@rock-chips.com>
>>>>> Subject: [PATCH V4 09/11] mmc: block: Add CQE support
>>>>>
>>>>> Add CQE support to the block driver, including:
>>>>>     - optionally using DCMD for flush requests
>>>>>     - manually issuing discard requests
>>>>>     - issuing read / write requests to the CQE
>>>>>     - supporting block-layer timeouts
>>>>>     - handling recovery
>>>>>     - supporting re-tuning
>>>>>
>>>>> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
>>>>> ---
>>>>>   drivers/mmc/core/block.c | 195
>> ++++++++++++++++++++++++++++++++-
>>>>>   drivers/mmc/core/block.h |   7 ++
>>>>>   drivers/mmc/core/queue.c | 273
>>>>> ++++++++++++++++++++++++++++++++++++++++++++++-
>>>>>   drivers/mmc/core/queue.h |  42 +++++++-
>>>>>   4 files changed, 510 insertions(+), 7 deletions(-)
>>>>>
>>>>> diff --git a/drivers/mmc/core/block.c b/drivers/mmc/core/block.c
>>>>> index
>>>>> 915290c74363..2d25115637b7 100644
>>>>> --- a/drivers/mmc/core/block.c
>>>>> +++ b/drivers/mmc/core/block.c
>>>>> @@ -109,6 +109,7 @@ struct mmc_blk_data {
>>>>>   #define MMC_BLK_WRITE        BIT(1)
>>>>>   #define MMC_BLK_DISCARD        BIT(2)
>>>>>   #define MMC_BLK_SECDISCARD    BIT(3)
>>>>> +#define MMC_BLK_CQE_RECOVERY    BIT(4)
>>>>>
>>>>>       /*
>>>>>        * Only set in main mmc_blk_data associated @@ -1612,6
>>>>> +1613,198 @@ static void mmc_blk_data_prep(struct mmc_queue *mq,
>>>>> struct mmc_queue_req *mqrq,
>>>>>           *do_data_tag_p = do_data_tag;
>>>>>   }
>>>>>
>>>>> +#define MMC_CQE_RETRIES 2
>>>
>>>
>>>>> +        blk_queue_rq_timed_out(mq->queue, mmc_cqe_timed_out);
>>>>> +        blk_queue_rq_timeout(mq->queue, 60 * HZ);
>>>>
>>>
>>> ------8<-------
>>>
>>>> Hi Adrian,
>>>>
>>>> These days I'm doing CMDQ stress test, and find one issue.
>>>> On our i.MX8QXP-ARM2 board, the RAM is 3GB. eMMC is 32GB.
>>>> I use command 'free -m' get the total memory is 2800M, and the free
>>>> memory is 2500M.
>>>>
>>>> I use 'mkfs.ext4' to format ext4 file system on the eMMC under
>>>> HS400ES CMDQ mode, works fine.
>>>>
>>>> When I use the following command to stress test CMDQ, it works fine.
>>>> bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 2048 -r 1024
>>>>
>>>> But when I change to use a large file size to do the same stress
>>>> test, using
>>>> bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 4096 -r 2048
>>>> or
>>>> bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 5600
>>>>
>>>> I get the following dump message.  According to the log,
>>>> mmc_cqe_timed_out() was trigged.
>>>> Seems mmc was blocked in somewhere.
>>>> Then I try to debug this issue, and open MMC_DEBUG in config, do the
>>>> same test, print the detail Command sending information on the
>>>> console, but finally can't reproduce.
>>
>> mmc_cqe_timed_out() is a 60 second timeout provided by the block layer.
>> Refer "blk_queue_rq_timeout(mq->queue, 60 * HZ)" in mmc_init_queue().
>> 60s is quite a long time so I would first want to determine if the task was really
>> queued that long.  I would instrument some code into cqhci_request() to
>> record the start time on struct mmc_request, and then print the time taken
>> when there is a problem.
>>
> 
> Hi Adrian, 
> 
> According to your suggestion, I add the following code to print the time.
> When issue happens, seems the request really pending for over 60s!
> 
> done
> Writing intelligently...[  689.209548] mmc0: cqhci: timeout for tag 9
> [  689.213658] the mrq all use 62123742 us
> [  689.217487] mmc0: cqhci: ============ CQHCI REGISTER DUMP ===========
> [  689.223927] mmc0: cqhci: Caps:      0x0000310a | Version:  0x00000510
> [  689.230363] mmc0: cqhci: Config:    0x00001001 | Control:  0x00000000
> [  689.236800] mmc0: cqhci: Int stat:  0x00000000 | Int enab: 0x00000006
> [  689.243238] mmc0: cqhci: Int sig:   0x00000006 | Int Coal: 0x00000000
> [  689.249675] mmc0: cqhci: TDL base:  0x90079000 | TDL up32: 0x00000000
> [  689.256113] mmc0: cqhci: Doorbell:  0x1fffffff | TCN:      0x00000000
> [  689.262550] mmc0: cqhci: Dev queue: 0x1fffefff | Dev Pend: 0x1fff7fff
> [  689.268988] mmc0: cqhci: Task clr:  0x00000000 | SSC1:     0x00011000
> [  689.275425] mmc0: cqhci: SSC2:      0x00000001 | DCMD rsp: 0x00000800
> [  689.281862] mmc0: cqhci: RED mask:  0xfdf9a080 | TERRI:    0x00000000
> [  689.288300] mmc0: cqhci: Resp idx:  0x0000002f | Resp arg: 0x00000900
> [  689.294737] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
> [  689.301176] mmc0: sdhci: Sys addr:  0xb602f000 | Version:  0x00000002
> [  689.307612] mmc0: sdhci: Blk size:  0x00000200 | Blk cnt:  0x00000400
> [  689.314050] mmc0: sdhci: Argument:  0x000f0400 | Trn mode: 0x00000023
> [  689.320487] mmc0: sdhci: Present:   0x01fd858f | Host ctl: 0x00000030
> [  689.326925] mmc0: sdhci: Power:     0x00000002 | Blk gap:  0x00000080
> [  689.333362] mmc0: sdhci: Wake-up:   0x00000008 | Clock:    0x0000000f
> [  689.339800] mmc0: sdhci: Timeout:   0x0000008f | Int stat: 0x00000000
> [  689.346237] mmc0: sdhci: Int enab:  0x107f4000 | Sig enab: 0x107f4000
> [  689.352674] mmc0: sdhci: AC12 err:  0x00000000 | Slot int: 0x00000502
> [  689.359113] mmc0: sdhci: Caps:      0x07eb0000 | Caps_1:   0x8000b407
> [  689.365549] mmc0: sdhci: Cmd:       0x00002c1a | Max curr: 0x00ffffff
> [  689.371987] mmc0: sdhci: Resp[0]:   0x00000900 | Resp[1]:  0xffffffff
> [  689.378424] mmc0: sdhci: Resp[2]:   0x328f5903 | Resp[3]:  0x00d02700
> [  689.384861] mmc0: sdhci: Host ctl2: 0x00000008
> [  689.389302] mmc0: sdhci: ADMA Err:  0x00000009 | ADMA Ptr: 0x9009a400
> [  689.395737] mmc0: sdhci: ============================================
> [  689.402212] mmc0: running CQE recovery

Tag 9 has been queued (bit set in Dev Pend) which means it is up to the eMMC
to select it for execution.  You should dump the times for the other mrq's
to see how long they have been waiting and try to determine if anything is
being processed.

If the eMMC is just taking a really long time to process tasks we could
extend the timeout, but it is hard to see how that is acceptable to a final
product.  At this point it looks like the eMMC may have a flaw in the way it
selects tasks for execution.

Re: [PATCH V4 09/11] mmc: block: Add CQE support

[Adrian Hunter]

On 09/08/17 11:16, Adrian Hunter wrote:
> On 09/08/17 10:57, Bough Chen wrote:
>>> -----Original Message-----
>>> From: linux-mmc-owner@vger.kernel.org [mailto:linux-mmc-
>>> owner@vger.kernel.org] On Behalf Of Adrian Hunter
>>> Sent: Wednesday, August 09, 2017 1:58 PM
>>> To: Shawn Lin <shawn.lin@rock-chips.com>; Bough Chen
>>> <haibo.chen@nxp.com>
>>> Cc: Ulf Hansson <ulf.hansson@linaro.org>; linux-mmc <linux-
>>> mmc@vger.kernel.org>; Alex Lemberg <alex.lemberg@sandisk.com>; Mateusz
>>> Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
>>> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung <jh80.chung@samsung.com>;
>>> Dong Aisheng <dongas86@gmail.com>; Das Asutosh
>>> <asutoshd@codeaurora.org>; Zhangfei Gao <zhangfei.gao@gmail.com>;
>>> Dorfman Konstantin <kdorfman@codeaurora.org>; Sahitya Tummala
>>> <stummala@codeaurora.org>; Harjani Ritesh <riteshh@codeaurora.org>; Venu
>>> Byravarasu <vbyravarasu@nvidia.com>; Linus Walleij <linus.walleij@linaro.org>
>>> Subject: Re: [PATCH V4 09/11] mmc: block: Add CQE support
>>>
>>> On 09/08/17 03:55, Shawn Lin wrote:
>>>> Hi,
>>>>
>>>> On 2017/8/8 20:07, Bough Chen wrote:
>>>>>> -----Original Message-----
>>>>>> From: Adrian Hunter [mailto:adrian.hunter@intel.com]
>>>>>> Sent: Friday, July 21, 2017 5:50 PM
>>>>>> To: Ulf Hansson <ulf.hansson@linaro.org>
>>>>>> Cc: linux-mmc <linux-mmc@vger.kernel.org>; Bough Chen
>>>>>> <haibo.chen@nxp.com>; Alex Lemberg <alex.lemberg@sandisk.com>;
>>>>>> Mateusz Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
>>>>>> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung
>>>>>> <jh80.chung@samsung.com>; Dong Aisheng <dongas86@gmail.com>; Das
>>>>>> Asutosh <asutoshd@codeaurora.org>; Zhangfei Gao
>>>>>> <zhangfei.gao@gmail.com>; Dorfman Konstantin
>>>>>> <kdorfman@codeaurora.org>; David Griego <david.griego@linaro.org>;
>>>>>> Sahitya Tummala <stummala@codeaurora.org>; Harjani Ritesh
>>>>>> <riteshh@codeaurora.org>; Venu Byravarasu <vbyravarasu@nvidia.com>;
>>>>>> Linus Walleij <linus.walleij@linaro.org>; Shawn Lin
>>>>>> <shawn.lin@rock-chips.com>
>>>>>> Subject: [PATCH V4 09/11] mmc: block: Add CQE support
>>>>>>
>>>>>> Add CQE support to the block driver, including:
>>>>>>     - optionally using DCMD for flush requests
>>>>>>     - manually issuing discard requests
>>>>>>     - issuing read / write requests to the CQE
>>>>>>     - supporting block-layer timeouts
>>>>>>     - handling recovery
>>>>>>     - supporting re-tuning
>>>>>>
>>>>>> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
>>>>>> ---
>>>>>>   drivers/mmc/core/block.c | 195
>>> ++++++++++++++++++++++++++++++++-
>>>>>>   drivers/mmc/core/block.h |   7 ++
>>>>>>   drivers/mmc/core/queue.c | 273
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++-
>>>>>>   drivers/mmc/core/queue.h |  42 +++++++-
>>>>>>   4 files changed, 510 insertions(+), 7 deletions(-)
>>>>>>
>>>>>> diff --git a/drivers/mmc/core/block.c b/drivers/mmc/core/block.c
>>>>>> index
>>>>>> 915290c74363..2d25115637b7 100644
>>>>>> --- a/drivers/mmc/core/block.c
>>>>>> +++ b/drivers/mmc/core/block.c
>>>>>> @@ -109,6 +109,7 @@ struct mmc_blk_data {
>>>>>>   #define MMC_BLK_WRITE        BIT(1)
>>>>>>   #define MMC_BLK_DISCARD        BIT(2)
>>>>>>   #define MMC_BLK_SECDISCARD    BIT(3)
>>>>>> +#define MMC_BLK_CQE_RECOVERY    BIT(4)
>>>>>>
>>>>>>       /*
>>>>>>        * Only set in main mmc_blk_data associated @@ -1612,6
>>>>>> +1613,198 @@ static void mmc_blk_data_prep(struct mmc_queue *mq,
>>>>>> struct mmc_queue_req *mqrq,
>>>>>>           *do_data_tag_p = do_data_tag;
>>>>>>   }
>>>>>>
>>>>>> +#define MMC_CQE_RETRIES 2
>>>>
>>>>
>>>>>> +        blk_queue_rq_timed_out(mq->queue, mmc_cqe_timed_out);
>>>>>> +        blk_queue_rq_timeout(mq->queue, 60 * HZ);
>>>>>
>>>>
>>>> ------8<-------
>>>>
>>>>> Hi Adrian,
>>>>>
>>>>> These days I'm doing CMDQ stress test, and find one issue.
>>>>> On our i.MX8QXP-ARM2 board, the RAM is 3GB. eMMC is 32GB.
>>>>> I use command 'free -m' get the total memory is 2800M, and the free
>>>>> memory is 2500M.
>>>>>
>>>>> I use 'mkfs.ext4' to format ext4 file system on the eMMC under
>>>>> HS400ES CMDQ mode, works fine.
>>>>>
>>>>> When I use the following command to stress test CMDQ, it works fine.
>>>>> bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 2048 -r 1024
>>>>>
>>>>> But when I change to use a large file size to do the same stress
>>>>> test, using
>>>>> bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 4096 -r 2048
>>>>> or
>>>>> bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 5600
>>>>>
>>>>> I get the following dump message.  According to the log,
>>>>> mmc_cqe_timed_out() was trigged.
>>>>> Seems mmc was blocked in somewhere.
>>>>> Then I try to debug this issue, and open MMC_DEBUG in config, do the
>>>>> same test, print the detail Command sending information on the
>>>>> console, but finally can't reproduce.
>>>
>>> mmc_cqe_timed_out() is a 60 second timeout provided by the block layer.
>>> Refer "blk_queue_rq_timeout(mq->queue, 60 * HZ)" in mmc_init_queue().
>>> 60s is quite a long time so I would first want to determine if the task was really
>>> queued that long.  I would instrument some code into cqhci_request() to
>>> record the start time on struct mmc_request, and then print the time taken
>>> when there is a problem.
>>>
>>
>> Hi Adrian, 
>>
>> According to your suggestion, I add the following code to print the time.
>> When issue happens, seems the request really pending for over 60s!
>>
>> done
>> Writing intelligently...[  689.209548] mmc0: cqhci: timeout for tag 9
>> [  689.213658] the mrq all use 62123742 us
>> [  689.217487] mmc0: cqhci: ============ CQHCI REGISTER DUMP ===========
>> [  689.223927] mmc0: cqhci: Caps:      0x0000310a | Version:  0x00000510
>> [  689.230363] mmc0: cqhci: Config:    0x00001001 | Control:  0x00000000
>> [  689.236800] mmc0: cqhci: Int stat:  0x00000000 | Int enab: 0x00000006
>> [  689.243238] mmc0: cqhci: Int sig:   0x00000006 | Int Coal: 0x00000000
>> [  689.249675] mmc0: cqhci: TDL base:  0x90079000 | TDL up32: 0x00000000
>> [  689.256113] mmc0: cqhci: Doorbell:  0x1fffffff | TCN:      0x00000000
>> [  689.262550] mmc0: cqhci: Dev queue: 0x1fffefff | Dev Pend: 0x1fff7fff
>> [  689.268988] mmc0: cqhci: Task clr:  0x00000000 | SSC1:     0x00011000
>> [  689.275425] mmc0: cqhci: SSC2:      0x00000001 | DCMD rsp: 0x00000800
>> [  689.281862] mmc0: cqhci: RED mask:  0xfdf9a080 | TERRI:    0x00000000
>> [  689.288300] mmc0: cqhci: Resp idx:  0x0000002f | Resp arg: 0x00000900
>> [  689.294737] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
>> [  689.301176] mmc0: sdhci: Sys addr:  0xb602f000 | Version:  0x00000002
>> [  689.307612] mmc0: sdhci: Blk size:  0x00000200 | Blk cnt:  0x00000400
>> [  689.314050] mmc0: sdhci: Argument:  0x000f0400 | Trn mode: 0x00000023
>> [  689.320487] mmc0: sdhci: Present:   0x01fd858f | Host ctl: 0x00000030
>> [  689.326925] mmc0: sdhci: Power:     0x00000002 | Blk gap:  0x00000080
>> [  689.333362] mmc0: sdhci: Wake-up:   0x00000008 | Clock:    0x0000000f
>> [  689.339800] mmc0: sdhci: Timeout:   0x0000008f | Int stat: 0x00000000
>> [  689.346237] mmc0: sdhci: Int enab:  0x107f4000 | Sig enab: 0x107f4000
>> [  689.352674] mmc0: sdhci: AC12 err:  0x00000000 | Slot int: 0x00000502
>> [  689.359113] mmc0: sdhci: Caps:      0x07eb0000 | Caps_1:   0x8000b407
>> [  689.365549] mmc0: sdhci: Cmd:       0x00002c1a | Max curr: 0x00ffffff
>> [  689.371987] mmc0: sdhci: Resp[0]:   0x00000900 | Resp[1]:  0xffffffff
>> [  689.378424] mmc0: sdhci: Resp[2]:   0x328f5903 | Resp[3]:  0x00d02700
>> [  689.384861] mmc0: sdhci: Host ctl2: 0x00000008
>> [  689.389302] mmc0: sdhci: ADMA Err:  0x00000009 | ADMA Ptr: 0x9009a400
>> [  689.395737] mmc0: sdhci: ============================================
>> [  689.402212] mmc0: running CQE recovery
> 
> Tag 9 has been queued (bit set in Dev Pend) which means it is up to the eMMC
> to select it for execution.  You should dump the times for the other mrq's
> to see how long they have been waiting and try to determine if anything is
> being processed.
> 
> If the eMMC is just taking a really long time to process tasks we could
> extend the timeout, but it is hard to see how that is acceptable to a final
> product.  At this point it looks like the eMMC may have a flaw in the way it
> selects tasks for execution.

No, that is wrong sorry, the task is in the QSR (Dev queue) so it is the CQE
that has not selected it.

RE: [PATCH V4 09/11] mmc: block: Add CQE support

[Bough Chen]

> -----Original Message-----
> From: linux-mmc-owner@vger.kernel.org [mailto:linux-mmc-
> owner@vger.kernel.org] On Behalf Of Adrian Hunter
> Sent: Wednesday, August 09, 2017 4:31 PM
> To: Bough Chen <haibo.chen@nxp.com>; Shawn Lin <shawn.lin@rock-
> chips.com>
> Cc: Ulf Hansson <ulf.hansson@linaro.org>; linux-mmc <linux-
> mmc@vger.kernel.org>; Alex Lemberg <alex.lemberg@sandisk.com>; Mateusz
> Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung <jh80.chung@samsung.com>;
> Dong Aisheng <dongas86@gmail.com>; Das Asutosh
> <asutoshd@codeaurora.org>; Zhangfei Gao <zhangfei.gao@gmail.com>;
> Dorfman Konstantin <kdorfman@codeaurora.org>; Sahitya Tummala
> <stummala@codeaurora.org>; Harjani Ritesh <riteshh@codeaurora.org>; Venu
> Byravarasu <vbyravarasu@nvidia.com>; Linus Walleij <linus.walleij@linaro.org>
> Subject: Re: [PATCH V4 09/11] mmc: block: Add CQE support
> 
> On 09/08/17 11:16, Adrian Hunter wrote:
> > On 09/08/17 10:57, Bough Chen wrote:
> >>> -----Original Message-----
> >>> From: linux-mmc-owner@vger.kernel.org [mailto:linux-mmc-
> >>> owner@vger.kernel.org] On Behalf Of Adrian Hunter
> >>> Sent: Wednesday, August 09, 2017 1:58 PM
> >>> To: Shawn Lin <shawn.lin@rock-chips.com>; Bough Chen
> >>> <haibo.chen@nxp.com>
> >>> Cc: Ulf Hansson <ulf.hansson@linaro.org>; linux-mmc <linux-
> >>> mmc@vger.kernel.org>; Alex Lemberg <alex.lemberg@sandisk.com>;
> >>> Mateusz Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
> >>> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung
> >>> <jh80.chung@samsung.com>; Dong Aisheng <dongas86@gmail.com>; Das
> >>> Asutosh <asutoshd@codeaurora.org>; Zhangfei Gao
> >>> <zhangfei.gao@gmail.com>; Dorfman Konstantin
> >>> <kdorfman@codeaurora.org>; Sahitya Tummala
> >>> <stummala@codeaurora.org>; Harjani Ritesh <riteshh@codeaurora.org>;
> >>> Venu Byravarasu <vbyravarasu@nvidia.com>; Linus Walleij
> >>> <linus.walleij@linaro.org>
> >>> Subject: Re: [PATCH V4 09/11] mmc: block: Add CQE support
> >>>
> >>> On 09/08/17 03:55, Shawn Lin wrote:
> >>>> Hi,
> >>>>
> >>>> On 2017/8/8 20:07, Bough Chen wrote:
> >>>>>> -----Original Message-----
> >>>>>> From: Adrian Hunter [mailto:adrian.hunter@intel.com]
> >>>>>> Sent: Friday, July 21, 2017 5:50 PM
> >>>>>> To: Ulf Hansson <ulf.hansson@linaro.org>
> >>>>>> Cc: linux-mmc <linux-mmc@vger.kernel.org>; Bough Chen
> >>>>>> <haibo.chen@nxp.com>; Alex Lemberg <alex.lemberg@sandisk.com>;
> >>>>>> Mateusz Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
> >>>>>> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung
> >>>>>> <jh80.chung@samsung.com>; Dong Aisheng <dongas86@gmail.com>;
> Das
> >>>>>> Asutosh <asutoshd@codeaurora.org>; Zhangfei Gao
> >>>>>> <zhangfei.gao@gmail.com>; Dorfman Konstantin
> >>>>>> <kdorfman@codeaurora.org>; David Griego
> >>>>>> <david.griego@linaro.org>; Sahitya Tummala
> >>>>>> <stummala@codeaurora.org>; Harjani Ritesh
> >>>>>> <riteshh@codeaurora.org>; Venu Byravarasu
> >>>>>> <vbyravarasu@nvidia.com>; Linus Walleij
> >>>>>> <linus.walleij@linaro.org>; Shawn Lin <shawn.lin@rock-chips.com>
> >>>>>> Subject: [PATCH V4 09/11] mmc: block: Add CQE support
> >>>>>>
> >>>>>> Add CQE support to the block driver, including:
> >>>>>>     - optionally using DCMD for flush requests
> >>>>>>     - manually issuing discard requests
> >>>>>>     - issuing read / write requests to the CQE
> >>>>>>     - supporting block-layer timeouts
> >>>>>>     - handling recovery
> >>>>>>     - supporting re-tuning
> >>>>>>
> >>>>>> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
> >>>>>> ---
> >>>>>>   drivers/mmc/core/block.c | 195
> >>> ++++++++++++++++++++++++++++++++-
> >>>>>>   drivers/mmc/core/block.h |   7 ++
> >>>>>>   drivers/mmc/core/queue.c | 273
> >>>>>> ++++++++++++++++++++++++++++++++++++++++++++++-
> >>>>>>   drivers/mmc/core/queue.h |  42 +++++++-
> >>>>>>   4 files changed, 510 insertions(+), 7 deletions(-)
> >>>>>>
> >>>>>> diff --git a/drivers/mmc/core/block.c b/drivers/mmc/core/block.c
> >>>>>> index
> >>>>>> 915290c74363..2d25115637b7 100644
> >>>>>> --- a/drivers/mmc/core/block.c
> >>>>>> +++ b/drivers/mmc/core/block.c
> >>>>>> @@ -109,6 +109,7 @@ struct mmc_blk_data {
> >>>>>>   #define MMC_BLK_WRITE        BIT(1)
> >>>>>>   #define MMC_BLK_DISCARD        BIT(2)
> >>>>>>   #define MMC_BLK_SECDISCARD    BIT(3)
> >>>>>> +#define MMC_BLK_CQE_RECOVERY    BIT(4)
> >>>>>>
> >>>>>>       /*
> >>>>>>        * Only set in main mmc_blk_data associated @@ -1612,6
> >>>>>> +1613,198 @@ static void mmc_blk_data_prep(struct mmc_queue
> *mq,
> >>>>>> struct mmc_queue_req *mqrq,
> >>>>>>           *do_data_tag_p = do_data_tag;
> >>>>>>   }
> >>>>>>
> >>>>>> +#define MMC_CQE_RETRIES 2
> >>>>
> >>>>
> >>>>>> +        blk_queue_rq_timed_out(mq->queue, mmc_cqe_timed_out);
> >>>>>> +        blk_queue_rq_timeout(mq->queue, 60 * HZ);
> >>>>>
> >>>>
> >>>> ------8<-------
> >>>>
> >>>>> Hi Adrian,
> >>>>>
> >>>>> These days I'm doing CMDQ stress test, and find one issue.
> >>>>> On our i.MX8QXP-ARM2 board, the RAM is 3GB. eMMC is 32GB.
> >>>>> I use command 'free -m' get the total memory is 2800M, and the
> >>>>> free memory is 2500M.
> >>>>>
> >>>>> I use 'mkfs.ext4' to format ext4 file system on the eMMC under
> >>>>> HS400ES CMDQ mode, works fine.
> >>>>>
> >>>>> When I use the following command to stress test CMDQ, it works fine.
> >>>>> bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 2048 -r 1024
> >>>>>
> >>>>> But when I change to use a large file size to do the same stress
> >>>>> test, using
> >>>>> bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 4096 -r 2048
> >>>>> or
> >>>>> bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 5600
> >>>>>
> >>>>> I get the following dump message.  According to the log,
> >>>>> mmc_cqe_timed_out() was trigged.
> >>>>> Seems mmc was blocked in somewhere.
> >>>>> Then I try to debug this issue, and open MMC_DEBUG in config, do
> >>>>> the same test, print the detail Command sending information on the
> >>>>> console, but finally can't reproduce.
> >>>
> >>> mmc_cqe_timed_out() is a 60 second timeout provided by the block layer.
> >>> Refer "blk_queue_rq_timeout(mq->queue, 60 * HZ)" in
> mmc_init_queue().
> >>> 60s is quite a long time so I would first want to determine if the
> >>> task was really queued that long.  I would instrument some code into
> >>> cqhci_request() to record the start time on struct mmc_request, and
> >>> then print the time taken when there is a problem.
> >>>
> >>
> >> Hi Adrian,
> >>
> >> According to your suggestion, I add the following code to print the time.
> >> When issue happens, seems the request really pending for over 60s!
> >>
> >> done
> >> Writing intelligently...[  689.209548] mmc0: cqhci: timeout for tag 9
> >> [  689.213658] the mrq all use 62123742 us [  689.217487] mmc0:
> >> cqhci: ============ CQHCI REGISTER DUMP ===========
> >> [  689.223927] mmc0: cqhci: Caps:      0x0000310a | Version:  0x00000510
> >> [  689.230363] mmc0: cqhci: Config:    0x00001001 | Control:  0x00000000
> >> [  689.236800] mmc0: cqhci: Int stat:  0x00000000 | Int enab: 0x00000006
> >> [  689.243238] mmc0: cqhci: Int sig:   0x00000006 | Int Coal: 0x00000000
> >> [  689.249675] mmc0: cqhci: TDL base:  0x90079000 | TDL up32: 0x00000000
> >> [  689.256113] mmc0: cqhci: Doorbell:  0x1fffffff | TCN:      0x00000000
> >> [  689.262550] mmc0: cqhci: Dev queue: 0x1fffefff | Dev Pend: 0x1fff7fff
> >> [  689.268988] mmc0: cqhci: Task clr:  0x00000000 | SSC1:     0x00011000
> >> [  689.275425] mmc0: cqhci: SSC2:      0x00000001 | DCMD rsp: 0x00000800
> >> [  689.281862] mmc0: cqhci: RED mask:  0xfdf9a080 | TERRI:    0x00000000
> >> [  689.288300] mmc0: cqhci: Resp idx:  0x0000002f | Resp arg:
> >> 0x00000900 [  689.294737] mmc0: sdhci: ============ SDHCI REGISTER
> >> DUMP =========== [  689.301176] mmc0: sdhci: Sys addr:  0xb602f000 |
> >> Version:  0x00000002 [  689.307612] mmc0: sdhci: Blk size:
> >> 0x00000200 | Blk cnt:  0x00000400 [  689.314050] mmc0: sdhci: Argument:
> 0x000f0400 | Trn mode: 0x00000023
> >> [  689.320487] mmc0: sdhci: Present:   0x01fd858f | Host ctl: 0x00000030
> >> [  689.326925] mmc0: sdhci: Power:     0x00000002 | Blk gap:  0x00000080
> >> [  689.333362] mmc0: sdhci: Wake-up:   0x00000008 | Clock:    0x0000000f
> >> [  689.339800] mmc0: sdhci: Timeout:   0x0000008f | Int stat: 0x00000000
> >> [  689.346237] mmc0: sdhci: Int enab:  0x107f4000 | Sig enab:
> >> 0x107f4000 [  689.352674] mmc0: sdhci: AC12 err:  0x00000000 | Slot int:
> 0x00000502
> >> [  689.359113] mmc0: sdhci: Caps:      0x07eb0000 | Caps_1:   0x8000b407
> >> [  689.365549] mmc0: sdhci: Cmd:       0x00002c1a | Max curr: 0x00ffffff
> >> [  689.371987] mmc0: sdhci: Resp[0]:   0x00000900 | Resp[1]:  0xffffffff
> >> [  689.378424] mmc0: sdhci: Resp[2]:   0x328f5903 | Resp[3]:  0x00d02700
> >> [  689.384861] mmc0: sdhci: Host ctl2: 0x00000008 [  689.389302]
> >> mmc0: sdhci: ADMA Err:  0x00000009 | ADMA Ptr: 0x9009a400 [
> >> 689.395737] mmc0: sdhci:
> ============================================
> >> [  689.402212] mmc0: running CQE recovery
> >
> > Tag 9 has been queued (bit set in Dev Pend) which means it is up to
> > the eMMC to select it for execution.  You should dump the times for
> > the other mrq's to see how long they have been waiting and try to
> > determine if anything is being processed.
> >
> > If the eMMC is just taking a really long time to process tasks we
> > could extend the timeout, but it is hard to see how that is acceptable
> > to a final product.  At this point it looks like the eMMC may have a
> > flaw in the way it selects tasks for execution.
> 
> No, that is wrong sorry, the task is in the QSR (Dev queue) so it is the CQE that
> has not selected it.

The timeout tag is 9, for Dev queue: 0x1fffefff, bit 9 is 1, means task 9 already queue in eMMC. 
For Dev Pend: 0x1fff7fff, the bit 9 is also 1,  which means CQE already send CMD44 and CMD45, but still
not send CMD46/47.  Seems our CQE pending tag 9 for over 60s! I will check with our IC guys to confirm
the hardware mechanism.


> --
> To unsubscribe from this list: send the line "unsubscribe linux-mmc" in the
> body of a message to majordomo@vger.kernel.org More majordomo info at
> http://vger.kernel.org/majordomo-info.html

RE: [PATCH V4 09/11] mmc: block: Add CQE support

[Bough Chen]

> -----Original Message-----
> From: linux-mmc-owner@vger.kernel.org [mailto:linux-mmc-
> owner@vger.kernel.org] On Behalf Of Bough Chen
> Sent: Wednesday, August 09, 2017 5:42 PM
> To: Adrian Hunter <adrian.hunter@intel.com>; Shawn Lin <shawn.lin@rock-
> chips.com>
> Cc: Ulf Hansson <ulf.hansson@linaro.org>; linux-mmc <linux-
> mmc@vger.kernel.org>; Alex Lemberg <alex.lemberg@sandisk.com>; Mateusz
> Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung <jh80.chung@samsung.com>;
> Dong Aisheng <dongas86@gmail.com>; Das Asutosh
> <asutoshd@codeaurora.org>; Zhangfei Gao <zhangfei.gao@gmail.com>;
> Dorfman Konstantin <kdorfman@codeaurora.org>; Sahitya Tummala
> <stummala@codeaurora.org>; Harjani Ritesh <riteshh@codeaurora.org>; Venu
> Byravarasu <vbyravarasu@nvidia.com>; Linus Walleij <linus.walleij@linaro.org>
> Subject: RE: [PATCH V4 09/11] mmc: block: Add CQE support
> 
> > -----Original Message-----
> > From: linux-mmc-owner@vger.kernel.org [mailto:linux-mmc-
> > owner@vger.kernel.org] On Behalf Of Adrian Hunter
> > Sent: Wednesday, August 09, 2017 4:31 PM
> > To: Bough Chen <haibo.chen@nxp.com>; Shawn Lin <shawn.lin@rock-
> > chips.com>
> > Cc: Ulf Hansson <ulf.hansson@linaro.org>; linux-mmc <linux-
> > mmc@vger.kernel.org>; Alex Lemberg <alex.lemberg@sandisk.com>;
> Mateusz
> > Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
> > <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung <jh80.chung@samsung.com>;
> > Dong Aisheng <dongas86@gmail.com>; Das Asutosh
> > <asutoshd@codeaurora.org>; Zhangfei Gao <zhangfei.gao@gmail.com>;
> > Dorfman Konstantin <kdorfman@codeaurora.org>; Sahitya Tummala
> > <stummala@codeaurora.org>; Harjani Ritesh <riteshh@codeaurora.org>;
> > Venu Byravarasu <vbyravarasu@nvidia.com>; Linus Walleij
> > <linus.walleij@linaro.org>
> > Subject: Re: [PATCH V4 09/11] mmc: block: Add CQE support
> >
> > On 09/08/17 11:16, Adrian Hunter wrote:
> > > On 09/08/17 10:57, Bough Chen wrote:
> > >>> -----Original Message-----
> > >>> From: linux-mmc-owner@vger.kernel.org [mailto:linux-mmc-
> > >>> owner@vger.kernel.org] On Behalf Of Adrian Hunter
> > >>> Sent: Wednesday, August 09, 2017 1:58 PM
> > >>> To: Shawn Lin <shawn.lin@rock-chips.com>; Bough Chen
> > >>> <haibo.chen@nxp.com>
> > >>> Cc: Ulf Hansson <ulf.hansson@linaro.org>; linux-mmc <linux-
> > >>> mmc@vger.kernel.org>; Alex Lemberg <alex.lemberg@sandisk.com>;
> > >>> Mateusz Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
> > >>> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung
> > >>> <jh80.chung@samsung.com>; Dong Aisheng <dongas86@gmail.com>;
> Das
> > >>> Asutosh <asutoshd@codeaurora.org>; Zhangfei Gao
> > >>> <zhangfei.gao@gmail.com>; Dorfman Konstantin
> > >>> <kdorfman@codeaurora.org>; Sahitya Tummala
> > >>> <stummala@codeaurora.org>; Harjani Ritesh
> > >>> <riteshh@codeaurora.org>; Venu Byravarasu
> > >>> <vbyravarasu@nvidia.com>; Linus Walleij <linus.walleij@linaro.org>
> > >>> Subject: Re: [PATCH V4 09/11] mmc: block: Add CQE support
> > >>>
> > >>> On 09/08/17 03:55, Shawn Lin wrote:
> > >>>> Hi,
> > >>>>
> > >>>> On 2017/8/8 20:07, Bough Chen wrote:
> > >>>>>> -----Original Message-----
> > >>>>>> From: Adrian Hunter [mailto:adrian.hunter@intel.com]
> > >>>>>> Sent: Friday, July 21, 2017 5:50 PM
> > >>>>>> To: Ulf Hansson <ulf.hansson@linaro.org>
> > >>>>>> Cc: linux-mmc <linux-mmc@vger.kernel.org>; Bough Chen
> > >>>>>> <haibo.chen@nxp.com>; Alex Lemberg
> <alex.lemberg@sandisk.com>;
> > >>>>>> Mateusz Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
> > >>>>>> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung
> > >>>>>> <jh80.chung@samsung.com>; Dong Aisheng
> <dongas86@gmail.com>;
> > Das
> > >>>>>> Asutosh <asutoshd@codeaurora.org>; Zhangfei Gao
> > >>>>>> <zhangfei.gao@gmail.com>; Dorfman Konstantin
> > >>>>>> <kdorfman@codeaurora.org>; David Griego
> > >>>>>> <david.griego@linaro.org>; Sahitya Tummala
> > >>>>>> <stummala@codeaurora.org>; Harjani Ritesh
> > >>>>>> <riteshh@codeaurora.org>; Venu Byravarasu
> > >>>>>> <vbyravarasu@nvidia.com>; Linus Walleij
> > >>>>>> <linus.walleij@linaro.org>; Shawn Lin
> > >>>>>> <shawn.lin@rock-chips.com>
> > >>>>>> Subject: [PATCH V4 09/11] mmc: block: Add CQE support
> > >>>>>>
> > >>>>>> Add CQE support to the block driver, including:
> > >>>>>>     - optionally using DCMD for flush requests
> > >>>>>>     - manually issuing discard requests
> > >>>>>>     - issuing read / write requests to the CQE
> > >>>>>>     - supporting block-layer timeouts
> > >>>>>>     - handling recovery
> > >>>>>>     - supporting re-tuning
> > >>>>>>
> > >>>>>> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
> > >>>>>> ---
> > >>>>>>   drivers/mmc/core/block.c | 195
> > >>> ++++++++++++++++++++++++++++++++-
> > >>>>>>   drivers/mmc/core/block.h |   7 ++
> > >>>>>>   drivers/mmc/core/queue.c | 273
> > >>>>>> ++++++++++++++++++++++++++++++++++++++++++++++-
> > >>>>>>   drivers/mmc/core/queue.h |  42 +++++++-
> > >>>>>>   4 files changed, 510 insertions(+), 7 deletions(-)
> > >>>>>>
> > >>>>>> diff --git a/drivers/mmc/core/block.c
> > >>>>>> b/drivers/mmc/core/block.c index
> > >>>>>> 915290c74363..2d25115637b7 100644
> > >>>>>> --- a/drivers/mmc/core/block.c
> > >>>>>> +++ b/drivers/mmc/core/block.c
> > >>>>>> @@ -109,6 +109,7 @@ struct mmc_blk_data {
> > >>>>>>   #define MMC_BLK_WRITE        BIT(1)
> > >>>>>>   #define MMC_BLK_DISCARD        BIT(2)
> > >>>>>>   #define MMC_BLK_SECDISCARD    BIT(3)
> > >>>>>> +#define MMC_BLK_CQE_RECOVERY    BIT(4)
> > >>>>>>
> > >>>>>>       /*
> > >>>>>>        * Only set in main mmc_blk_data associated @@ -1612,6
> > >>>>>> +1613,198 @@ static void mmc_blk_data_prep(struct mmc_queue
> > *mq,
> > >>>>>> struct mmc_queue_req *mqrq,
> > >>>>>>           *do_data_tag_p = do_data_tag;
> > >>>>>>   }
> > >>>>>>
> > >>>>>> +#define MMC_CQE_RETRIES 2
> > >>>>
> > >>>>
> > >>>>>> +        blk_queue_rq_timed_out(mq->queue, mmc_cqe_timed_out);
> > >>>>>> +        blk_queue_rq_timeout(mq->queue, 60 * HZ);
> > >>>>>
> > >>>>
> > >>>> ------8<-------
> > >>>>
> > >>>>> Hi Adrian,
> > >>>>>
> > >>>>> These days I'm doing CMDQ stress test, and find one issue.
> > >>>>> On our i.MX8QXP-ARM2 board, the RAM is 3GB. eMMC is 32GB.
> > >>>>> I use command 'free -m' get the total memory is 2800M, and the
> > >>>>> free memory is 2500M.
> > >>>>>
> > >>>>> I use 'mkfs.ext4' to format ext4 file system on the eMMC under
> > >>>>> HS400ES CMDQ mode, works fine.
> > >>>>>
> > >>>>> When I use the following command to stress test CMDQ, it works fine.
> > >>>>> bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 2048 -r 1024
> > >>>>>
> > >>>>> But when I change to use a large file size to do the same stress
> > >>>>> test, using
> > >>>>> bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 4096 -r 2048
> > >>>>> or
> > >>>>> bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 5600
> > >>>>>
> > >>>>> I get the following dump message.  According to the log,
> > >>>>> mmc_cqe_timed_out() was trigged.
> > >>>>> Seems mmc was blocked in somewhere.
> > >>>>> Then I try to debug this issue, and open MMC_DEBUG in config, do
> > >>>>> the same test, print the detail Command sending information on
> > >>>>> the console, but finally can't reproduce.
> > >>>
> > >>> mmc_cqe_timed_out() is a 60 second timeout provided by the block
> layer.
> > >>> Refer "blk_queue_rq_timeout(mq->queue, 60 * HZ)" in
> > mmc_init_queue().
> > >>> 60s is quite a long time so I would first want to determine if the
> > >>> task was really queued that long.  I would instrument some code
> > >>> into
> > >>> cqhci_request() to record the start time on struct mmc_request,
> > >>> and then print the time taken when there is a problem.
> > >>>
> > >>
> > >> Hi Adrian,
> > >>
> > >> According to your suggestion, I add the following code to print the time.
> > >> When issue happens, seems the request really pending for over 60s!
> > >>
> > >> done
> > >> Writing intelligently...[  689.209548] mmc0: cqhci: timeout for tag
> > >> 9 [  689.213658] the mrq all use 62123742 us [  689.217487] mmc0:
> > >> cqhci: ============ CQHCI REGISTER DUMP ===========
> > >> [  689.223927] mmc0: cqhci: Caps:      0x0000310a | Version:  0x00000510
> > >> [  689.230363] mmc0: cqhci: Config:    0x00001001 | Control:  0x00000000
> > >> [  689.236800] mmc0: cqhci: Int stat:  0x00000000 | Int enab: 0x00000006
> > >> [  689.243238] mmc0: cqhci: Int sig:   0x00000006 | Int Coal: 0x00000000
> > >> [  689.249675] mmc0: cqhci: TDL base:  0x90079000 | TDL up32: 0x00000000
> > >> [  689.256113] mmc0: cqhci: Doorbell:  0x1fffffff | TCN:      0x00000000
> > >> [  689.262550] mmc0: cqhci: Dev queue: 0x1fffefff | Dev Pend: 0x1fff7fff
> > >> [  689.268988] mmc0: cqhci: Task clr:  0x00000000 | SSC1:     0x00011000
> > >> [  689.275425] mmc0: cqhci: SSC2:      0x00000001 | DCMD rsp: 0x00000800
> > >> [  689.281862] mmc0: cqhci: RED mask:  0xfdf9a080 | TERRI:    0x00000000
> > >> [  689.288300] mmc0: cqhci: Resp idx:  0x0000002f | Resp arg:
> > >> 0x00000900 [  689.294737] mmc0: sdhci: ============ SDHCI REGISTER
> > >> DUMP =========== [  689.301176] mmc0: sdhci: Sys addr:  0xb602f000
> > >> |
> > >> Version:  0x00000002 [  689.307612] mmc0: sdhci: Blk size:
> > >> 0x00000200 | Blk cnt:  0x00000400 [  689.314050] mmc0: sdhci: Argument:
> > 0x000f0400 | Trn mode: 0x00000023
> > >> [  689.320487] mmc0: sdhci: Present:   0x01fd858f | Host ctl: 0x00000030
> > >> [  689.326925] mmc0: sdhci: Power:     0x00000002 | Blk gap:  0x00000080
> > >> [  689.333362] mmc0: sdhci: Wake-up:   0x00000008 | Clock:    0x0000000f
> > >> [  689.339800] mmc0: sdhci: Timeout:   0x0000008f | Int stat: 0x00000000
> > >> [  689.346237] mmc0: sdhci: Int enab:  0x107f4000 | Sig enab:
> > >> 0x107f4000 [  689.352674] mmc0: sdhci: AC12 err:  0x00000000 | Slot int:
> > 0x00000502
> > >> [  689.359113] mmc0: sdhci: Caps:      0x07eb0000 | Caps_1:   0x8000b407
> > >> [  689.365549] mmc0: sdhci: Cmd:       0x00002c1a | Max curr: 0x00ffffff
> > >> [  689.371987] mmc0: sdhci: Resp[0]:   0x00000900 | Resp[1]:  0xffffffff
> > >> [  689.378424] mmc0: sdhci: Resp[2]:   0x328f5903 | Resp[3]:  0x00d02700
> > >> [  689.384861] mmc0: sdhci: Host ctl2: 0x00000008 [  689.389302]
> > >> mmc0: sdhci: ADMA Err:  0x00000009 | ADMA Ptr: 0x9009a400 [
> > >> 689.395737] mmc0: sdhci:
> > ============================================
> > >> [  689.402212] mmc0: running CQE recovery
> > >
> > > Tag 9 has been queued (bit set in Dev Pend) which means it is up to
> > > the eMMC to select it for execution.  You should dump the times for
> > > the other mrq's to see how long they have been waiting and try to
> > > determine if anything is being processed.
> > >
> > > If the eMMC is just taking a really long time to process tasks we
> > > could extend the timeout, but it is hard to see how that is
> > > acceptable to a final product.  At this point it looks like the eMMC
> > > may have a flaw in the way it selects tasks for execution.
> >
> > No, that is wrong sorry, the task is in the QSR (Dev queue) so it is
> > the CQE that has not selected it.
> 
> The timeout tag is 9, for Dev queue: 0x1fffefff, bit 9 is 1, means task 9 already
> queue in eMMC.
> For Dev Pend: 0x1fff7fff, the bit 9 is also 1,  which means CQE already send
> CMD44 and CMD45, but still not send CMD46/47.  Seems our CQE pending tag 9
> for over 60s! I will check with our IC guys to confirm the hardware mechanism.
> 

For the eMMC chip, the sequential wirte speed test by 'dd' is around 100MB/s.
If each tag try to write 1GB data, which meas each tag needs 10s to complete, once
The number of pending tags exceed 6, 60s timeout will be trigged.

I'm not sure how bonnie++ works, I will try to dump the CMDQ script list to verify whether this is the root cause.
   
> 
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-mmc"
> > in the body of a message to majordomo@vger.kernel.org More majordomo
> > info at http://vger.kernel.org/majordomo-info.html
> \x13  칻\x1c & ~ & \x18  +-  ݶ\x17  w  ˛   m b  f ȧ \x17  ܨ}   Ơz &j:+v        zZ+  +zf   h   ~    i   z \x1e w   ?
> & )ߢ^[f

Re: [PATCH V4 09/11] mmc: block: Add CQE support

[Adrian Hunter]

On 08/09/2017 01:35 PM, Bough Chen wrote:
>> -----Original Message-----
>> From: linux-mmc-owner@vger.kernel.org [mailto:linux-mmc-
>> owner@vger.kernel.org] On Behalf Of Bough Chen
>> Sent: Wednesday, August 09, 2017 5:42 PM
>> To: Adrian Hunter <adrian.hunter@intel.com>; Shawn Lin <shawn.lin@rock-
>> chips.com>
>> Cc: Ulf Hansson <ulf.hansson@linaro.org>; linux-mmc <linux-
>> mmc@vger.kernel.org>; Alex Lemberg <alex.lemberg@sandisk.com>; Mateusz
>> Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
>> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung <jh80.chung@samsung.com>;
>> Dong Aisheng <dongas86@gmail.com>; Das Asutosh
>> <asutoshd@codeaurora.org>; Zhangfei Gao <zhangfei.gao@gmail.com>;
>> Dorfman Konstantin <kdorfman@codeaurora.org>; Sahitya Tummala
>> <stummala@codeaurora.org>; Harjani Ritesh <riteshh@codeaurora.org>; Venu
>> Byravarasu <vbyravarasu@nvidia.com>; Linus Walleij <linus.walleij@linaro.org>
>> Subject: RE: [PATCH V4 09/11] mmc: block: Add CQE support
>>
>>> -----Original Message-----
>>> From: linux-mmc-owner@vger.kernel.org [mailto:linux-mmc-
>>> owner@vger.kernel.org] On Behalf Of Adrian Hunter
>>> Sent: Wednesday, August 09, 2017 4:31 PM
>>> To: Bough Chen <haibo.chen@nxp.com>; Shawn Lin <shawn.lin@rock-
>>> chips.com>
>>> Cc: Ulf Hansson <ulf.hansson@linaro.org>; linux-mmc <linux-
>>> mmc@vger.kernel.org>; Alex Lemberg <alex.lemberg@sandisk.com>;
>> Mateusz
>>> Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
>>> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung <jh80.chung@samsung.com>;
>>> Dong Aisheng <dongas86@gmail.com>; Das Asutosh
>>> <asutoshd@codeaurora.org>; Zhangfei Gao <zhangfei.gao@gmail.com>;
>>> Dorfman Konstantin <kdorfman@codeaurora.org>; Sahitya Tummala
>>> <stummala@codeaurora.org>; Harjani Ritesh <riteshh@codeaurora.org>;
>>> Venu Byravarasu <vbyravarasu@nvidia.com>; Linus Walleij
>>> <linus.walleij@linaro.org>
>>> Subject: Re: [PATCH V4 09/11] mmc: block: Add CQE support
>>>
>>> On 09/08/17 11:16, Adrian Hunter wrote:
>>>> On 09/08/17 10:57, Bough Chen wrote:
>>>>>> -----Original Message-----
>>>>>> From: linux-mmc-owner@vger.kernel.org [mailto:linux-mmc-
>>>>>> owner@vger.kernel.org] On Behalf Of Adrian Hunter
>>>>>> Sent: Wednesday, August 09, 2017 1:58 PM
>>>>>> To: Shawn Lin <shawn.lin@rock-chips.com>; Bough Chen
>>>>>> <haibo.chen@nxp.com>
>>>>>> Cc: Ulf Hansson <ulf.hansson@linaro.org>; linux-mmc <linux-
>>>>>> mmc@vger.kernel.org>; Alex Lemberg <alex.lemberg@sandisk.com>;
>>>>>> Mateusz Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
>>>>>> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung
>>>>>> <jh80.chung@samsung.com>; Dong Aisheng <dongas86@gmail.com>;
>> Das
>>>>>> Asutosh <asutoshd@codeaurora.org>; Zhangfei Gao
>>>>>> <zhangfei.gao@gmail.com>; Dorfman Konstantin
>>>>>> <kdorfman@codeaurora.org>; Sahitya Tummala
>>>>>> <stummala@codeaurora.org>; Harjani Ritesh
>>>>>> <riteshh@codeaurora.org>; Venu Byravarasu
>>>>>> <vbyravarasu@nvidia.com>; Linus Walleij <linus.walleij@linaro.org>
>>>>>> Subject: Re: [PATCH V4 09/11] mmc: block: Add CQE support
>>>>>>
>>>>>> On 09/08/17 03:55, Shawn Lin wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> On 2017/8/8 20:07, Bough Chen wrote:
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Adrian Hunter [mailto:adrian.hunter@intel.com]
>>>>>>>>> Sent: Friday, July 21, 2017 5:50 PM
>>>>>>>>> To: Ulf Hansson <ulf.hansson@linaro.org>
>>>>>>>>> Cc: linux-mmc <linux-mmc@vger.kernel.org>; Bough Chen
>>>>>>>>> <haibo.chen@nxp.com>; Alex Lemberg
>> <alex.lemberg@sandisk.com>;
>>>>>>>>> Mateusz Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
>>>>>>>>> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung
>>>>>>>>> <jh80.chung@samsung.com>; Dong Aisheng
>> <dongas86@gmail.com>;
>>> Das
>>>>>>>>> Asutosh <asutoshd@codeaurora.org>; Zhangfei Gao
>>>>>>>>> <zhangfei.gao@gmail.com>; Dorfman Konstantin
>>>>>>>>> <kdorfman@codeaurora.org>; David Griego
>>>>>>>>> <david.griego@linaro.org>; Sahitya Tummala
>>>>>>>>> <stummala@codeaurora.org>; Harjani Ritesh
>>>>>>>>> <riteshh@codeaurora.org>; Venu Byravarasu
>>>>>>>>> <vbyravarasu@nvidia.com>; Linus Walleij
>>>>>>>>> <linus.walleij@linaro.org>; Shawn Lin
>>>>>>>>> <shawn.lin@rock-chips.com>
>>>>>>>>> Subject: [PATCH V4 09/11] mmc: block: Add CQE support
>>>>>>>>>
>>>>>>>>> Add CQE support to the block driver, including:
>>>>>>>>>     - optionally using DCMD for flush requests
>>>>>>>>>     - manually issuing discard requests
>>>>>>>>>     - issuing read / write requests to the CQE
>>>>>>>>>     - supporting block-layer timeouts
>>>>>>>>>     - handling recovery
>>>>>>>>>     - supporting re-tuning
>>>>>>>>>
>>>>>>>>> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
>>>>>>>>> ---
>>>>>>>>>   drivers/mmc/core/block.c | 195
>>>>>> ++++++++++++++++++++++++++++++++-
>>>>>>>>>   drivers/mmc/core/block.h |   7 ++
>>>>>>>>>   drivers/mmc/core/queue.c | 273
>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>   drivers/mmc/core/queue.h |  42 +++++++-
>>>>>>>>>   4 files changed, 510 insertions(+), 7 deletions(-)
>>>>>>>>>
>>>>>>>>> diff --git a/drivers/mmc/core/block.c
>>>>>>>>> b/drivers/mmc/core/block.c index
>>>>>>>>> 915290c74363..2d25115637b7 100644
>>>>>>>>> --- a/drivers/mmc/core/block.c
>>>>>>>>> +++ b/drivers/mmc/core/block.c
>>>>>>>>> @@ -109,6 +109,7 @@ struct mmc_blk_data {
>>>>>>>>>   #define MMC_BLK_WRITE        BIT(1)
>>>>>>>>>   #define MMC_BLK_DISCARD        BIT(2)
>>>>>>>>>   #define MMC_BLK_SECDISCARD    BIT(3)
>>>>>>>>> +#define MMC_BLK_CQE_RECOVERY    BIT(4)
>>>>>>>>>
>>>>>>>>>       /*
>>>>>>>>>        * Only set in main mmc_blk_data associated @@ -1612,6
>>>>>>>>> +1613,198 @@ static void mmc_blk_data_prep(struct mmc_queue
>>> *mq,
>>>>>>>>> struct mmc_queue_req *mqrq,
>>>>>>>>>           *do_data_tag_p = do_data_tag;
>>>>>>>>>   }
>>>>>>>>>
>>>>>>>>> +#define MMC_CQE_RETRIES 2
>>>>>>>
>>>>>>>
>>>>>>>>> +        blk_queue_rq_timed_out(mq->queue, mmc_cqe_timed_out);
>>>>>>>>> +        blk_queue_rq_timeout(mq->queue, 60 * HZ);
>>>>>>>>
>>>>>>>
>>>>>>> ------8<-------
>>>>>>>
>>>>>>>> Hi Adrian,
>>>>>>>>
>>>>>>>> These days I'm doing CMDQ stress test, and find one issue.
>>>>>>>> On our i.MX8QXP-ARM2 board, the RAM is 3GB. eMMC is 32GB.
>>>>>>>> I use command 'free -m' get the total memory is 2800M, and the
>>>>>>>> free memory is 2500M.
>>>>>>>>
>>>>>>>> I use 'mkfs.ext4' to format ext4 file system on the eMMC under
>>>>>>>> HS400ES CMDQ mode, works fine.
>>>>>>>>
>>>>>>>> When I use the following command to stress test CMDQ, it works fine.
>>>>>>>> bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 2048 -r 1024
>>>>>>>>
>>>>>>>> But when I change to use a large file size to do the same stress
>>>>>>>> test, using
>>>>>>>> bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 4096 -r 2048
>>>>>>>> or
>>>>>>>> bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 5600
>>>>>>>>
>>>>>>>> I get the following dump message.  According to the log,
>>>>>>>> mmc_cqe_timed_out() was trigged.
>>>>>>>> Seems mmc was blocked in somewhere.
>>>>>>>> Then I try to debug this issue, and open MMC_DEBUG in config, do
>>>>>>>> the same test, print the detail Command sending information on
>>>>>>>> the console, but finally can't reproduce.
>>>>>>
>>>>>> mmc_cqe_timed_out() is a 60 second timeout provided by the block
>> layer.
>>>>>> Refer "blk_queue_rq_timeout(mq->queue, 60 * HZ)" in
>>> mmc_init_queue().
>>>>>> 60s is quite a long time so I would first want to determine if the
>>>>>> task was really queued that long.  I would instrument some code
>>>>>> into
>>>>>> cqhci_request() to record the start time on struct mmc_request,
>>>>>> and then print the time taken when there is a problem.
>>>>>>
>>>>>
>>>>> Hi Adrian,
>>>>>
>>>>> According to your suggestion, I add the following code to print the time.
>>>>> When issue happens, seems the request really pending for over 60s!
>>>>>
>>>>> done
>>>>> Writing intelligently...[  689.209548] mmc0: cqhci: timeout for tag
>>>>> 9 [  689.213658] the mrq all use 62123742 us [  689.217487] mmc0:
>>>>> cqhci: ============ CQHCI REGISTER DUMP ===========
>>>>> [  689.223927] mmc0: cqhci: Caps:      0x0000310a | Version:  0x00000510
>>>>> [  689.230363] mmc0: cqhci: Config:    0x00001001 | Control:  0x00000000
>>>>> [  689.236800] mmc0: cqhci: Int stat:  0x00000000 | Int enab: 0x00000006
>>>>> [  689.243238] mmc0: cqhci: Int sig:   0x00000006 | Int Coal: 0x00000000
>>>>> [  689.249675] mmc0: cqhci: TDL base:  0x90079000 | TDL up32: 0x00000000
>>>>> [  689.256113] mmc0: cqhci: Doorbell:  0x1fffffff | TCN:      0x00000000
>>>>> [  689.262550] mmc0: cqhci: Dev queue: 0x1fffefff | Dev Pend: 0x1fff7fff
>>>>> [  689.268988] mmc0: cqhci: Task clr:  0x00000000 | SSC1:     0x00011000
>>>>> [  689.275425] mmc0: cqhci: SSC2:      0x00000001 | DCMD rsp: 0x00000800
>>>>> [  689.281862] mmc0: cqhci: RED mask:  0xfdf9a080 | TERRI:    0x00000000
>>>>> [  689.288300] mmc0: cqhci: Resp idx:  0x0000002f | Resp arg:
>>>>> 0x00000900 [  689.294737] mmc0: sdhci: ============ SDHCI REGISTER
>>>>> DUMP =========== [  689.301176] mmc0: sdhci: Sys addr:  0xb602f000
>>>>> |
>>>>> Version:  0x00000002 [  689.307612] mmc0: sdhci: Blk size:
>>>>> 0x00000200 | Blk cnt:  0x00000400 [  689.314050] mmc0: sdhci: Argument:
>>> 0x000f0400 | Trn mode: 0x00000023
>>>>> [  689.320487] mmc0: sdhci: Present:   0x01fd858f | Host ctl: 0x00000030
>>>>> [  689.326925] mmc0: sdhci: Power:     0x00000002 | Blk gap:  0x00000080
>>>>> [  689.333362] mmc0: sdhci: Wake-up:   0x00000008 | Clock:    0x0000000f
>>>>> [  689.339800] mmc0: sdhci: Timeout:   0x0000008f | Int stat: 0x00000000
>>>>> [  689.346237] mmc0: sdhci: Int enab:  0x107f4000 | Sig enab:
>>>>> 0x107f4000 [  689.352674] mmc0: sdhci: AC12 err:  0x00000000 | Slot int:
>>> 0x00000502
>>>>> [  689.359113] mmc0: sdhci: Caps:      0x07eb0000 | Caps_1:   0x8000b407
>>>>> [  689.365549] mmc0: sdhci: Cmd:       0x00002c1a | Max curr: 0x00ffffff
>>>>> [  689.371987] mmc0: sdhci: Resp[0]:   0x00000900 | Resp[1]:  0xffffffff
>>>>> [  689.378424] mmc0: sdhci: Resp[2]:   0x328f5903 | Resp[3]:  0x00d02700
>>>>> [  689.384861] mmc0: sdhci: Host ctl2: 0x00000008 [  689.389302]
>>>>> mmc0: sdhci: ADMA Err:  0x00000009 | ADMA Ptr: 0x9009a400 [
>>>>> 689.395737] mmc0: sdhci:
>>> ============================================
>>>>> [  689.402212] mmc0: running CQE recovery
>>>>
>>>> Tag 9 has been queued (bit set in Dev Pend) which means it is up to
>>>> the eMMC to select it for execution.  You should dump the times for
>>>> the other mrq's to see how long they have been waiting and try to
>>>> determine if anything is being processed.
>>>>
>>>> If the eMMC is just taking a really long time to process tasks we
>>>> could extend the timeout, but it is hard to see how that is
>>>> acceptable to a final product.  At this point it looks like the eMMC
>>>> may have a flaw in the way it selects tasks for execution.
>>>
>>> No, that is wrong sorry, the task is in the QSR (Dev queue) so it is
>>> the CQE that has not selected it.
>>
>> The timeout tag is 9, for Dev queue: 0x1fffefff, bit 9 is 1, means task 9 already
>> queue in eMMC.
>> For Dev Pend: 0x1fff7fff, the bit 9 is also 1,  which means CQE already send
>> CMD44 and CMD45, but still not send CMD46/47.  Seems our CQE pending tag 9
>> for over 60s! I will check with our IC guys to confirm the hardware mechanism.
>>
> 
> For the eMMC chip, the sequential wirte speed test by 'dd' is around 100MB/s.
> If each tag try to write 1GB data, which meas each tag needs 10s to complete, once
> The number of pending tags exceed 6, 60s timeout will be trigged.

The request size is limited by the block layer due to host controller
parameters.  In the case of SDHCI to 512KiB.  So each tag is at most 512KiB.

Re: [PATCH V4 05/11] mmc: core: Add support for handling CQE requests

[Adrian Hunter]

On 07/08/17 17:21, Ulf Hansson wrote:
> On 21 July 2017 at 11:49, Adrian Hunter <adrian.hunter@intel.com> wrote:
>> Add core support for handling CQE requests, including starting, completing
>> and recovering.
>>
>> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
>> ---
>>  drivers/mmc/core/core.c  | 147 +++++++++++++++++++++++++++++++++++++++++++++--
>>  include/linux/mmc/core.h |   5 ++
>>  2 files changed, 147 insertions(+), 5 deletions(-)
>>
>> diff --git a/drivers/mmc/core/core.c b/drivers/mmc/core/core.c
>> index b0af9db18eef..5a9d837599a1 100644
>> --- a/drivers/mmc/core/core.c
>> +++ b/drivers/mmc/core/core.c
>> @@ -266,7 +266,8 @@ static void __mmc_start_request(struct mmc_host *host, struct mmc_request *mrq)
>>         host->ops->request(host, mrq);
>>  }
>>
>> -static void mmc_mrq_pr_debug(struct mmc_host *host, struct mmc_request *mrq)
>> +static void mmc_mrq_pr_debug(struct mmc_host *host, struct mmc_request *mrq,
>> +                            bool cqe)
>>  {
>>         if (mrq->sbc) {
>>                 pr_debug("<%s: starting CMD%u arg %08x flags %08x>\n",
>> @@ -275,9 +276,12 @@ static void mmc_mrq_pr_debug(struct mmc_host *host, struct mmc_request *mrq)
>>         }
>>
>>         if (mrq->cmd) {
>> -               pr_debug("%s: starting CMD%u arg %08x flags %08x\n",
>> -                        mmc_hostname(host), mrq->cmd->opcode, mrq->cmd->arg,
>> -                        mrq->cmd->flags);
>> +               pr_debug("%s: starting %sCMD%u arg %08x flags %08x\n",
>> +                        mmc_hostname(host), cqe ? "CQE direct " : "",
>> +                        mrq->cmd->opcode, mrq->cmd->arg, mrq->cmd->flags);
>> +       } else if (cqe) {
>> +               pr_debug("%s: starting CQE transfer for tag %d blkaddr %u\n",
>> +                        mmc_hostname(host), mrq->tag, mrq->data->blk_addr);
>>         }
>>
>>         if (mrq->data) {
>> @@ -345,7 +349,7 @@ static int mmc_start_request(struct mmc_host *host, struct mmc_request *mrq)
>>         if (mmc_card_removed(host->card))
>>                 return -ENOMEDIUM;
>>
>> -       mmc_mrq_pr_debug(host, mrq);
>> +       mmc_mrq_pr_debug(host, mrq, false);
>>
>>         WARN_ON(!host->claimed);
>>
>> @@ -485,6 +489,139 @@ void mmc_wait_for_req_done(struct mmc_host *host, struct mmc_request *mrq)
>>  }
>>  EXPORT_SYMBOL(mmc_wait_for_req_done);
>>
> 
> As this is an exported function, could you please add some function
> header information.

OK

> 
>> +int mmc_cqe_start_req(struct mmc_host *host, struct mmc_request *mrq)
>> +{
>> +       int err;
>> +
>> +       /* Caller must hold retuning while CQE is in use */
> 
> Would be nice to get some more clarification on *why* this is needed.
> Can you please extend the comment a bit on that?

OK

> 
>> +       err = mmc_retune(host);
>> +       if (err)
>> +               goto out_err;
>> +
>> +       mrq->host = host;
>> +
>> +       mmc_mrq_pr_debug(host, mrq, true);
>> +
>> +       err = mmc_mrq_prep(host, mrq);
>> +       if (err)
>> +               goto out_err;
>> +
>> +       err = host->cqe_ops->cqe_request(host, mrq);
>> +       if (err)
>> +               goto out_err;
>> +
>> +       trace_mmc_request_start(host, mrq);
>> +
>> +       return 0;
>> +
>> +out_err:
>> +       if (mrq->cmd) {
>> +               pr_debug("%s: failed to start CQE direct CMD%u, error %d\n",
>> +                        mmc_hostname(host), mrq->cmd->opcode, err);
>> +       } else {
>> +               pr_debug("%s: failed to start CQE transfer for tag %d, error %d\n",
>> +                        mmc_hostname(host), mrq->tag, err);
>> +       }
>> +       return err;
>> +}
>> +EXPORT_SYMBOL(mmc_cqe_start_req);
>> +
>> +static void __mmc_cqe_request_done(struct mmc_host *host,
>> +                                  struct mmc_request *mrq)
>> +{
>> +       mmc_should_fail_request(host, mrq);
>> +
>> +       /* Flag re-tuning needed on CRC errors */
>> +       if ((mrq->cmd && mrq->cmd->error == -EILSEQ) ||
>> +           (mrq->data && mrq->data->error == -EILSEQ))
>> +               mmc_retune_needed(host);
>> +
>> +       trace_mmc_request_done(host, mrq);
>> +
>> +       if (mrq->cmd) {
>> +               pr_debug("%s: CQE req done (direct CMD%u): %d\n",
>> +                        mmc_hostname(host), mrq->cmd->opcode, mrq->cmd->error);
>> +       } else {
>> +               pr_debug("%s: CQE transfer done tag %d\n",
>> +                        mmc_hostname(host), mrq->tag);
>> +       }
>> +
>> +       if (mrq->data) {
>> +               pr_debug("%s:     %d bytes transferred: %d\n",
>> +                        mmc_hostname(host),
>> +                        mrq->data->bytes_xfered, mrq->data->error);
>> +       }
>> +}
>> +
>> +/**
>> + *     mmc_cqe_request_done - CQE has finished processing an MMC request
>> + *     @host: MMC host which completed request
>> + *     @mrq: MMC request which completed
>> + *
>> + *     CQE drivers should call this function when they have completed
>> + *     their processing of a request.
>> + */
>> +void mmc_cqe_request_done(struct mmc_host *host, struct mmc_request *mrq)
>> +{
>> +       __mmc_cqe_request_done(host, mrq);
>> +
>> +       mrq->done(mrq);
>> +}
>> +EXPORT_SYMBOL(mmc_cqe_request_done);
>> +
>> +/**
>> + *     mmc_cqe_post_req - CQE post process of a completed MMC request
>> + *     @host: MMC host
>> + *     @mrq: MMC request to be processed
>> + */
>> +void mmc_cqe_post_req(struct mmc_host *host, struct mmc_request *mrq)
>> +{
>> +       if (host->cqe_ops->cqe_post_req)
>> +               host->cqe_ops->cqe_post_req(host, mrq);
>> +}
>> +EXPORT_SYMBOL(mmc_cqe_post_req);
>> +
>> +/* Arbitrary 1 second timeout */
>> +#define MMC_CQE_RECOVERY_TIMEOUT       1000
> 
> Why 1 second?

As it says it is arbitrary.  The CQHCI specification does not provide a
value.  It "feels" like a reasonable time to wait.

> 
> Nitpick:
> I am bit reluctant to use the terminology of "recovery". In then end
> it's about a request timeout and then act by doing a graceful abort.

No it is about any kind of error including data CRC's.  CQE requests have 2
retries and there is re-tuning if needed, so there is recovery from CRC errors.

> 
>> +
> 
> Please add function header.
> 
>> +int mmc_cqe_recovery(struct mmc_host *host)
>> +{
>> +       struct mmc_command cmd;
>> +       int err;
>> +
>> +       mmc_retune_hold_now(host);
>> +
>> +       /*
>> +        * Recovery is expected seldom, if at all, but it reduces performance,
>> +        * so make sure it is not completely silent.
>> +        */
>> +       pr_warn("%s: running CQE recovery\n", mmc_hostname(host));
>> +
>> +       host->cqe_ops->cqe_recovery_start(host);
> 
> What is expected to be done by the host from this callback?

We are going to send commands to the card, so CQE must prepare for that.

> 
> I have been thinking about deploying a common data request timeout
> behavior in the mmc core. In principle calling
> wait_for_completion_timeout() instead of just wait_for_completion().

We should increasingly be moving to asynchronous processing so we have to
support the path where there is no thread sitting waiting. i.e. the
wait_for...() functions should not be the focus.

Also the block layer has it's own timeout mechanism and we should use that
for the block driver - in the case of blk-mq I don't think it is optional.

> To deal with that, one would also need to invent a new host ops
> callback to allow the host driver to restore its HW to a know state.
> Very similar to what you also seems to need here for CQE.
> 
> I was just thinking that part of this code/callbacks could be re-used
> for other non-CQE data requests that timeouts. Do you think that could
> makes sense?

We need to keep CQE host ops separate because, design-wise, it is simpler
and more flexible to treat the CQE as a separate entity.  Then,
theoretically, any driver can use any CQE implementation.  Even if we never
have more than 1 CQE implementation, it still forces a cleaner API.

> 
>> +
>> +       memset(&cmd, 0, sizeof(cmd));
>> +       cmd.opcode       = MMC_STOP_TRANSMISSION,
>> +       cmd.flags        = MMC_RSP_R1B | MMC_CMD_AC,
>> +       cmd.flags       &= ~MMC_RSP_CRC; /* Ignore CRC */
>> +       cmd.busy_timeout = MMC_CQE_RECOVERY_TIMEOUT,
>> +       mmc_wait_for_cmd(host, &cmd, 0);
>> +
>> +       memset(&cmd, 0, sizeof(cmd));
>> +       cmd.opcode       = MMC_CMDQ_TASK_MGMT;
>> +       cmd.arg          = 1; /* Discard entire queue */
>> +       cmd.flags        = MMC_RSP_R1B | MMC_CMD_AC;
>> +       cmd.flags       &= ~MMC_RSP_CRC; /* Ignore CRC */
>> +       cmd.busy_timeout = MMC_CQE_RECOVERY_TIMEOUT,
>> +       err = mmc_wait_for_cmd(host, &cmd, 0);
>> +
>> +       host->cqe_ops->cqe_recovery_finish(host);
>> +
>> +       mmc_retune_release(host);
>> +
>> +       return err;
>> +}
>> +EXPORT_SYMBOL(mmc_cqe_recovery);
>> +
>>  /**
>>   *     mmc_is_req_done - Determine if a 'cap_cmd_during_tfr' request is done
>>   *     @host: MMC host
>> diff --git a/include/linux/mmc/core.h b/include/linux/mmc/core.h
>> index bf1788a224e6..1974fcfd4284 100644
>> --- a/include/linux/mmc/core.h
>> +++ b/include/linux/mmc/core.h
>> @@ -174,6 +174,11 @@ struct mmc_async_req *mmc_start_areq(struct mmc_host *host,
>>  int mmc_wait_for_cmd(struct mmc_host *host, struct mmc_command *cmd,
>>                 int retries);
>>
>> +int mmc_cqe_start_req(struct mmc_host *host, struct mmc_request *mrq);
>> +void mmc_cqe_request_done(struct mmc_host *host, struct mmc_request *mrq);
>> +void mmc_cqe_post_req(struct mmc_host *host, struct mmc_request *mrq);
>> +int mmc_cqe_recovery(struct mmc_host *host);
>> +
>>  int mmc_hw_reset(struct mmc_host *host);
>>  void mmc_set_data_timeout(struct mmc_data *data, const struct mmc_card *card);
>>
>> --
>> 1.9.1
>>
> 
> Kind regards
> Uffe
> 

Re: [PATCH V4 07/11] mmc: mmc: Enable CQE's

[Adrian Hunter]

On 07/08/17 17:51, Ulf Hansson wrote:
> On 21 July 2017 at 11:49, Adrian Hunter <adrian.hunter@intel.com> wrote:
>> Enable or disable CQE when a card is added or removed respectively.
> 
> As a standalone change, this is hard to understand.
> 
> Perhaps if you squash this with some the patche calling ->cqe_off()
> and the one actually setting ext_csd.cmdq_en, I can get a better
> picture.

No they are not related.

> 
>>
>> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
>> ---
>>  drivers/mmc/core/bus.c |  7 +++++++
>>  drivers/mmc/core/mmc.c | 13 +++++++++++++
>>  2 files changed, 20 insertions(+)
>>
>> diff --git a/drivers/mmc/core/bus.c b/drivers/mmc/core/bus.c
>> index 301246513a37..a4b49e25fe96 100644
>> --- a/drivers/mmc/core/bus.c
>> +++ b/drivers/mmc/core/bus.c
>> @@ -369,10 +369,17 @@ int mmc_add_card(struct mmc_card *card)
>>   */
>>  void mmc_remove_card(struct mmc_card *card)
>>  {
>> +       struct mmc_host *host = card->host;
>> +
>>  #ifdef CONFIG_DEBUG_FS
>>         mmc_remove_card_debugfs(card);
>>  #endif
>>
>> +       if (host->cqe_enabled) {
>> +               host->cqe_ops->cqe_disable(host);
>> +               host->cqe_enabled = false;
>> +       }
>> +
> 
> This doesn't feel like the correct place to disable cqe.
> 
> Primarily because I don't think you enable cqe in mmc_add_card(), so
> this isn't consistent.

mmc_add_card() and mmc_remove_card() are not paired functions. Instead
mmc_remove_card() is on the error path of the ..._init_card() functions and
consequently this is the best place for cqe->disable().

> 
>>         if (mmc_card_present(card)) {
>>                 if (mmc_host_is_spi(card->host)) {
>>                         pr_info("%s: SPI card removed\n",
>> diff --git a/drivers/mmc/core/mmc.c b/drivers/mmc/core/mmc.c
>> index 2ff0caf92bc8..92c6167d64e0 100644
>> --- a/drivers/mmc/core/mmc.c
>> +++ b/drivers/mmc/core/mmc.c
>> @@ -1804,6 +1804,19 @@ static int mmc_init_card(struct mmc_host *host, u32 ocr,
>>          */
>>         card->reenable_cmdq = card->ext_csd.cmdq_en;
>>
>> +       if (card->ext_csd.cmdq_en && (host->caps2 & MMC_CAP2_CQE) &&
>> +           !host->cqe_enabled) {
> 
> I assume card->ext_csd.cmdq_en can't be set, unless the MMC_CAP2_CQE
> bit also is set. So there should be no reason to check it here again?

Yes, it is a left-over from software command queue support.  I will remove it.

> 
> Also, can ever host->cqe_enabled be true in this path? If so, isn't
> that wrong by itself?

No, it can be set on the reset path.

> 
>> +               err = host->cqe_ops->cqe_enable(host, card);
>> +               if (err) {
>> +                       pr_err("%s: Failed to enable CQE, error %d\n",
>> +                               mmc_hostname(host), err);
>> +               } else {
>> +                       host->cqe_enabled = true;
>> +                       pr_info("%s: Command Queue Engine enabled\n",
>> +                               mmc_hostname(host));
>> +               }
>> +       }
>> +
>>         /*
>>          * The mandatory minimum values are defined for packed command.
>>          * read: 5, write: 3
>> --
>> 1.9.1
>>
> 
> Kind regards
> Uffe
> 

Re: [PATCH V4 09/11] mmc: block: Add CQE support

[Adrian Hunter]

On 09/08/17 15:45, Adrian Hunter wrote:
> On 08/09/2017 01:35 PM, Bough Chen wrote:
>>> -----Original Message-----
>>> From: linux-mmc-owner@vger.kernel.org [mailto:linux-mmc-
>>> owner@vger.kernel.org] On Behalf Of Bough Chen
>>> Sent: Wednesday, August 09, 2017 5:42 PM
>>> To: Adrian Hunter <adrian.hunter@intel.com>; Shawn Lin <shawn.lin@rock-
>>> chips.com>
>>> Cc: Ulf Hansson <ulf.hansson@linaro.org>; linux-mmc <linux-
>>> mmc@vger.kernel.org>; Alex Lemberg <alex.lemberg@sandisk.com>; Mateusz
>>> Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
>>> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung <jh80.chung@samsung.com>;
>>> Dong Aisheng <dongas86@gmail.com>; Das Asutosh
>>> <asutoshd@codeaurora.org>; Zhangfei Gao <zhangfei.gao@gmail.com>;
>>> Dorfman Konstantin <kdorfman@codeaurora.org>; Sahitya Tummala
>>> <stummala@codeaurora.org>; Harjani Ritesh <riteshh@codeaurora.org>; Venu
>>> Byravarasu <vbyravarasu@nvidia.com>; Linus Walleij <linus.walleij@linaro.org>
>>> Subject: RE: [PATCH V4 09/11] mmc: block: Add CQE support
>>>
>>>> -----Original Message-----
>>>> From: linux-mmc-owner@vger.kernel.org [mailto:linux-mmc-
>>>> owner@vger.kernel.org] On Behalf Of Adrian Hunter
>>>> Sent: Wednesday, August 09, 2017 4:31 PM
>>>> To: Bough Chen <haibo.chen@nxp.com>; Shawn Lin <shawn.lin@rock-
>>>> chips.com>
>>>> Cc: Ulf Hansson <ulf.hansson@linaro.org>; linux-mmc <linux-
>>>> mmc@vger.kernel.org>; Alex Lemberg <alex.lemberg@sandisk.com>;
>>> Mateusz
>>>> Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
>>>> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung <jh80.chung@samsung.com>;
>>>> Dong Aisheng <dongas86@gmail.com>; Das Asutosh
>>>> <asutoshd@codeaurora.org>; Zhangfei Gao <zhangfei.gao@gmail.com>;
>>>> Dorfman Konstantin <kdorfman@codeaurora.org>; Sahitya Tummala
>>>> <stummala@codeaurora.org>; Harjani Ritesh <riteshh@codeaurora.org>;
>>>> Venu Byravarasu <vbyravarasu@nvidia.com>; Linus Walleij
>>>> <linus.walleij@linaro.org>
>>>> Subject: Re: [PATCH V4 09/11] mmc: block: Add CQE support
>>>>
>>>> On 09/08/17 11:16, Adrian Hunter wrote:
>>>>> On 09/08/17 10:57, Bough Chen wrote:
>>>>>>> -----Original Message-----
>>>>>>> From: linux-mmc-owner@vger.kernel.org [mailto:linux-mmc-
>>>>>>> owner@vger.kernel.org] On Behalf Of Adrian Hunter
>>>>>>> Sent: Wednesday, August 09, 2017 1:58 PM
>>>>>>> To: Shawn Lin <shawn.lin@rock-chips.com>; Bough Chen
>>>>>>> <haibo.chen@nxp.com>
>>>>>>> Cc: Ulf Hansson <ulf.hansson@linaro.org>; linux-mmc <linux-
>>>>>>> mmc@vger.kernel.org>; Alex Lemberg <alex.lemberg@sandisk.com>;
>>>>>>> Mateusz Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
>>>>>>> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung
>>>>>>> <jh80.chung@samsung.com>; Dong Aisheng <dongas86@gmail.com>;
>>> Das
>>>>>>> Asutosh <asutoshd@codeaurora.org>; Zhangfei Gao
>>>>>>> <zhangfei.gao@gmail.com>; Dorfman Konstantin
>>>>>>> <kdorfman@codeaurora.org>; Sahitya Tummala
>>>>>>> <stummala@codeaurora.org>; Harjani Ritesh
>>>>>>> <riteshh@codeaurora.org>; Venu Byravarasu
>>>>>>> <vbyravarasu@nvidia.com>; Linus Walleij <linus.walleij@linaro.org>
>>>>>>> Subject: Re: [PATCH V4 09/11] mmc: block: Add CQE support
>>>>>>>
>>>>>>> On 09/08/17 03:55, Shawn Lin wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> On 2017/8/8 20:07, Bough Chen wrote:
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Adrian Hunter [mailto:adrian.hunter@intel.com]
>>>>>>>>>> Sent: Friday, July 21, 2017 5:50 PM
>>>>>>>>>> To: Ulf Hansson <ulf.hansson@linaro.org>
>>>>>>>>>> Cc: linux-mmc <linux-mmc@vger.kernel.org>; Bough Chen
>>>>>>>>>> <haibo.chen@nxp.com>; Alex Lemberg
>>> <alex.lemberg@sandisk.com>;
>>>>>>>>>> Mateusz Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
>>>>>>>>>> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung
>>>>>>>>>> <jh80.chung@samsung.com>; Dong Aisheng
>>> <dongas86@gmail.com>;
>>>> Das
>>>>>>>>>> Asutosh <asutoshd@codeaurora.org>; Zhangfei Gao
>>>>>>>>>> <zhangfei.gao@gmail.com>; Dorfman Konstantin
>>>>>>>>>> <kdorfman@codeaurora.org>; David Griego
>>>>>>>>>> <david.griego@linaro.org>; Sahitya Tummala
>>>>>>>>>> <stummala@codeaurora.org>; Harjani Ritesh
>>>>>>>>>> <riteshh@codeaurora.org>; Venu Byravarasu
>>>>>>>>>> <vbyravarasu@nvidia.com>; Linus Walleij
>>>>>>>>>> <linus.walleij@linaro.org>; Shawn Lin
>>>>>>>>>> <shawn.lin@rock-chips.com>
>>>>>>>>>> Subject: [PATCH V4 09/11] mmc: block: Add CQE support
>>>>>>>>>>
>>>>>>>>>> Add CQE support to the block driver, including:
>>>>>>>>>>     - optionally using DCMD for flush requests
>>>>>>>>>>     - manually issuing discard requests
>>>>>>>>>>     - issuing read / write requests to the CQE
>>>>>>>>>>     - supporting block-layer timeouts
>>>>>>>>>>     - handling recovery
>>>>>>>>>>     - supporting re-tuning
>>>>>>>>>>
>>>>>>>>>> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
>>>>>>>>>> ---
>>>>>>>>>>   drivers/mmc/core/block.c | 195
>>>>>>> ++++++++++++++++++++++++++++++++-
>>>>>>>>>>   drivers/mmc/core/block.h |   7 ++
>>>>>>>>>>   drivers/mmc/core/queue.c | 273
>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>>   drivers/mmc/core/queue.h |  42 +++++++-
>>>>>>>>>>   4 files changed, 510 insertions(+), 7 deletions(-)
>>>>>>>>>>
>>>>>>>>>> diff --git a/drivers/mmc/core/block.c
>>>>>>>>>> b/drivers/mmc/core/block.c index
>>>>>>>>>> 915290c74363..2d25115637b7 100644
>>>>>>>>>> --- a/drivers/mmc/core/block.c
>>>>>>>>>> +++ b/drivers/mmc/core/block.c
>>>>>>>>>> @@ -109,6 +109,7 @@ struct mmc_blk_data {
>>>>>>>>>>   #define MMC_BLK_WRITE        BIT(1)
>>>>>>>>>>   #define MMC_BLK_DISCARD        BIT(2)
>>>>>>>>>>   #define MMC_BLK_SECDISCARD    BIT(3)
>>>>>>>>>> +#define MMC_BLK_CQE_RECOVERY    BIT(4)
>>>>>>>>>>
>>>>>>>>>>       /*
>>>>>>>>>>        * Only set in main mmc_blk_data associated @@ -1612,6
>>>>>>>>>> +1613,198 @@ static void mmc_blk_data_prep(struct mmc_queue
>>>> *mq,
>>>>>>>>>> struct mmc_queue_req *mqrq,
>>>>>>>>>>           *do_data_tag_p = do_data_tag;
>>>>>>>>>>   }
>>>>>>>>>>
>>>>>>>>>> +#define MMC_CQE_RETRIES 2
>>>>>>>>
>>>>>>>>
>>>>>>>>>> +        blk_queue_rq_timed_out(mq->queue, mmc_cqe_timed_out);
>>>>>>>>>> +        blk_queue_rq_timeout(mq->queue, 60 * HZ);
>>>>>>>>>
>>>>>>>>
>>>>>>>> ------8<-------
>>>>>>>>
>>>>>>>>> Hi Adrian,
>>>>>>>>>
>>>>>>>>> These days I'm doing CMDQ stress test, and find one issue.
>>>>>>>>> On our i.MX8QXP-ARM2 board, the RAM is 3GB. eMMC is 32GB.
>>>>>>>>> I use command 'free -m' get the total memory is 2800M, and the
>>>>>>>>> free memory is 2500M.
>>>>>>>>>
>>>>>>>>> I use 'mkfs.ext4' to format ext4 file system on the eMMC under
>>>>>>>>> HS400ES CMDQ mode, works fine.
>>>>>>>>>
>>>>>>>>> When I use the following command to stress test CMDQ, it works fine.
>>>>>>>>> bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 2048 -r 1024
>>>>>>>>>
>>>>>>>>> But when I change to use a large file size to do the same stress
>>>>>>>>> test, using
>>>>>>>>> bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 4096 -r 2048
>>>>>>>>> or
>>>>>>>>> bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 5600
>>>>>>>>>
>>>>>>>>> I get the following dump message.  According to the log,
>>>>>>>>> mmc_cqe_timed_out() was trigged.
>>>>>>>>> Seems mmc was blocked in somewhere.
>>>>>>>>> Then I try to debug this issue, and open MMC_DEBUG in config, do
>>>>>>>>> the same test, print the detail Command sending information on
>>>>>>>>> the console, but finally can't reproduce.
>>>>>>>
>>>>>>> mmc_cqe_timed_out() is a 60 second timeout provided by the block
>>> layer.
>>>>>>> Refer "blk_queue_rq_timeout(mq->queue, 60 * HZ)" in
>>>> mmc_init_queue().
>>>>>>> 60s is quite a long time so I would first want to determine if the
>>>>>>> task was really queued that long.  I would instrument some code
>>>>>>> into
>>>>>>> cqhci_request() to record the start time on struct mmc_request,
>>>>>>> and then print the time taken when there is a problem.
>>>>>>>
>>>>>>
>>>>>> Hi Adrian,
>>>>>>
>>>>>> According to your suggestion, I add the following code to print the time.
>>>>>> When issue happens, seems the request really pending for over 60s!
>>>>>>
>>>>>> done
>>>>>> Writing intelligently...[  689.209548] mmc0: cqhci: timeout for tag
>>>>>> 9 [  689.213658] the mrq all use 62123742 us [  689.217487] mmc0:
>>>>>> cqhci: ============ CQHCI REGISTER DUMP ===========
>>>>>> [  689.223927] mmc0: cqhci: Caps:      0x0000310a | Version:  0x00000510
>>>>>> [  689.230363] mmc0: cqhci: Config:    0x00001001 | Control:  0x00000000
>>>>>> [  689.236800] mmc0: cqhci: Int stat:  0x00000000 | Int enab: 0x00000006
>>>>>> [  689.243238] mmc0: cqhci: Int sig:   0x00000006 | Int Coal: 0x00000000
>>>>>> [  689.249675] mmc0: cqhci: TDL base:  0x90079000 | TDL up32: 0x00000000
>>>>>> [  689.256113] mmc0: cqhci: Doorbell:  0x1fffffff | TCN:      0x00000000
>>>>>> [  689.262550] mmc0: cqhci: Dev queue: 0x1fffefff | Dev Pend: 0x1fff7fff
>>>>>> [  689.268988] mmc0: cqhci: Task clr:  0x00000000 | SSC1:     0x00011000
>>>>>> [  689.275425] mmc0: cqhci: SSC2:      0x00000001 | DCMD rsp: 0x00000800
>>>>>> [  689.281862] mmc0: cqhci: RED mask:  0xfdf9a080 | TERRI:    0x00000000
>>>>>> [  689.288300] mmc0: cqhci: Resp idx:  0x0000002f | Resp arg:
>>>>>> 0x00000900 [  689.294737] mmc0: sdhci: ============ SDHCI REGISTER
>>>>>> DUMP =========== [  689.301176] mmc0: sdhci: Sys addr:  0xb602f000
>>>>>> |
>>>>>> Version:  0x00000002 [  689.307612] mmc0: sdhci: Blk size:
>>>>>> 0x00000200 | Blk cnt:  0x00000400 [  689.314050] mmc0: sdhci: Argument:
>>>> 0x000f0400 | Trn mode: 0x00000023
>>>>>> [  689.320487] mmc0: sdhci: Present:   0x01fd858f | Host ctl: 0x00000030
>>>>>> [  689.326925] mmc0: sdhci: Power:     0x00000002 | Blk gap:  0x00000080
>>>>>> [  689.333362] mmc0: sdhci: Wake-up:   0x00000008 | Clock:    0x0000000f
>>>>>> [  689.339800] mmc0: sdhci: Timeout:   0x0000008f | Int stat: 0x00000000
>>>>>> [  689.346237] mmc0: sdhci: Int enab:  0x107f4000 | Sig enab:
>>>>>> 0x107f4000 [  689.352674] mmc0: sdhci: AC12 err:  0x00000000 | Slot int:
>>>> 0x00000502
>>>>>> [  689.359113] mmc0: sdhci: Caps:      0x07eb0000 | Caps_1:   0x8000b407
>>>>>> [  689.365549] mmc0: sdhci: Cmd:       0x00002c1a | Max curr: 0x00ffffff
>>>>>> [  689.371987] mmc0: sdhci: Resp[0]:   0x00000900 | Resp[1]:  0xffffffff
>>>>>> [  689.378424] mmc0: sdhci: Resp[2]:   0x328f5903 | Resp[3]:  0x00d02700
>>>>>> [  689.384861] mmc0: sdhci: Host ctl2: 0x00000008 [  689.389302]
>>>>>> mmc0: sdhci: ADMA Err:  0x00000009 | ADMA Ptr: 0x9009a400 [
>>>>>> 689.395737] mmc0: sdhci:
>>>> ============================================
>>>>>> [  689.402212] mmc0: running CQE recovery
>>>>>
>>>>> Tag 9 has been queued (bit set in Dev Pend) which means it is up to
>>>>> the eMMC to select it for execution.  You should dump the times for
>>>>> the other mrq's to see how long they have been waiting and try to
>>>>> determine if anything is being processed.
>>>>>
>>>>> If the eMMC is just taking a really long time to process tasks we
>>>>> could extend the timeout, but it is hard to see how that is
>>>>> acceptable to a final product.  At this point it looks like the eMMC
>>>>> may have a flaw in the way it selects tasks for execution.
>>>>
>>>> No, that is wrong sorry, the task is in the QSR (Dev queue) so it is
>>>> the CQE that has not selected it.
>>>
>>> The timeout tag is 9, for Dev queue: 0x1fffefff, bit 9 is 1, means task 9 already
>>> queue in eMMC.
>>> For Dev Pend: 0x1fff7fff, the bit 9 is also 1,  which means CQE already send
>>> CMD44 and CMD45, but still not send CMD46/47.  Seems our CQE pending tag 9
>>> for over 60s! I will check with our IC guys to confirm the hardware mechanism.
>>>
>>
>> For the eMMC chip, the sequential wirte speed test by 'dd' is around 100MB/s.
>> If each tag try to write 1GB data, which meas each tag needs 10s to complete, once
>> The number of pending tags exceed 6, 60s timeout will be trigged.
> 
> The request size is limited by the block layer due to host controller
> parameters.  In the case of SDHCI to 512KiB.  So each tag is at most 512KiB.
> 

I just found a bug in 32-bit DMA.  Are you using 32-bit DMA?  That could
also be causing your problem.  I will send a new version of the patches with
a fix, probably later today.

RE: [PATCH V4 09/11] mmc: block: Add CQE support

[Bough Chen]

> -----Original Message-----
> From: Adrian Hunter [mailto:adrian.hunter@intel.com]
> Sent: Thursday, August 10, 2017 6:19 PM
> To: Bough Chen <haibo.chen@nxp.com>; Shawn Lin <shawn.lin@rock-
> chips.com>
> Cc: Ulf Hansson <ulf.hansson@linaro.org>; linux-mmc <linux-
> mmc@vger.kernel.org>; Alex Lemberg <alex.lemberg@sandisk.com>; Mateusz
> Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung <jh80.chung@samsung.com>;
> Dong Aisheng <dongas86@gmail.com>; Das Asutosh
> <asutoshd@codeaurora.org>; Zhangfei Gao <zhangfei.gao@gmail.com>;
> Dorfman Konstantin <kdorfman@codeaurora.org>; Sahitya Tummala
> <stummala@codeaurora.org>; Harjani Ritesh <riteshh@codeaurora.org>; Venu
> Byravarasu <vbyravarasu@nvidia.com>; Linus Walleij <linus.walleij@linaro.org>
> Subject: Re: [PATCH V4 09/11] mmc: block: Add CQE support
> 
> On 09/08/17 15:45, Adrian Hunter wrote:
> > On 08/09/2017 01:35 PM, Bough Chen wrote:
> >>> -----Original Message-----
> >>> From: linux-mmc-owner@vger.kernel.org [mailto:linux-mmc-
> >>> owner@vger.kernel.org] On Behalf Of Bough Chen
> >>> Sent: Wednesday, August 09, 2017 5:42 PM
> >>> To: Adrian Hunter <adrian.hunter@intel.com>; Shawn Lin
> >>> <shawn.lin@rock- chips.com>
> >>> Cc: Ulf Hansson <ulf.hansson@linaro.org>; linux-mmc <linux-
> >>> mmc@vger.kernel.org>; Alex Lemberg <alex.lemberg@sandisk.com>;
> >>> Mateusz Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
> >>> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung
> >>> <jh80.chung@samsung.com>; Dong Aisheng <dongas86@gmail.com>; Das
> >>> Asutosh <asutoshd@codeaurora.org>; Zhangfei Gao
> >>> <zhangfei.gao@gmail.com>; Dorfman Konstantin
> >>> <kdorfman@codeaurora.org>; Sahitya Tummala
> >>> <stummala@codeaurora.org>; Harjani Ritesh <riteshh@codeaurora.org>;
> >>> Venu Byravarasu <vbyravarasu@nvidia.com>; Linus Walleij
> >>> <linus.walleij@linaro.org>
> >>> Subject: RE: [PATCH V4 09/11] mmc: block: Add CQE support
> >>>
> >>>> -----Original Message-----
> >>>> From: linux-mmc-owner@vger.kernel.org [mailto:linux-mmc-
> >>>> owner@vger.kernel.org] On Behalf Of Adrian Hunter
> >>>> Sent: Wednesday, August 09, 2017 4:31 PM
> >>>> To: Bough Chen <haibo.chen@nxp.com>; Shawn Lin <shawn.lin@rock-
> >>>> chips.com>
> >>>> Cc: Ulf Hansson <ulf.hansson@linaro.org>; linux-mmc <linux-
> >>>> mmc@vger.kernel.org>; Alex Lemberg <alex.lemberg@sandisk.com>;
> >>> Mateusz
> >>>> Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
> >>>> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung
> >>>> <jh80.chung@samsung.com>; Dong Aisheng <dongas86@gmail.com>;
> Das
> >>>> Asutosh <asutoshd@codeaurora.org>; Zhangfei Gao
> >>>> <zhangfei.gao@gmail.com>; Dorfman Konstantin
> >>>> <kdorfman@codeaurora.org>; Sahitya Tummala
> >>>> <stummala@codeaurora.org>; Harjani Ritesh <riteshh@codeaurora.org>;
> >>>> Venu Byravarasu <vbyravarasu@nvidia.com>; Linus Walleij
> >>>> <linus.walleij@linaro.org>
> >>>> Subject: Re: [PATCH V4 09/11] mmc: block: Add CQE support
> >>>>
> >>>> On 09/08/17 11:16, Adrian Hunter wrote:
> >>>>> On 09/08/17 10:57, Bough Chen wrote:
> >>>>>>> -----Original Message-----
> >>>>>>> From: linux-mmc-owner@vger.kernel.org [mailto:linux-mmc-
> >>>>>>> owner@vger.kernel.org] On Behalf Of Adrian Hunter
> >>>>>>> Sent: Wednesday, August 09, 2017 1:58 PM
> >>>>>>> To: Shawn Lin <shawn.lin@rock-chips.com>; Bough Chen
> >>>>>>> <haibo.chen@nxp.com>
> >>>>>>> Cc: Ulf Hansson <ulf.hansson@linaro.org>; linux-mmc <linux-
> >>>>>>> mmc@vger.kernel.org>; Alex Lemberg <alex.lemberg@sandisk.com>;
> >>>>>>> Mateusz Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
> >>>>>>> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung
> >>>>>>> <jh80.chung@samsung.com>; Dong Aisheng <dongas86@gmail.com>;
> >>> Das
> >>>>>>> Asutosh <asutoshd@codeaurora.org>; Zhangfei Gao
> >>>>>>> <zhangfei.gao@gmail.com>; Dorfman Konstantin
> >>>>>>> <kdorfman@codeaurora.org>; Sahitya Tummala
> >>>>>>> <stummala@codeaurora.org>; Harjani Ritesh
> >>>>>>> <riteshh@codeaurora.org>; Venu Byravarasu
> >>>>>>> <vbyravarasu@nvidia.com>; Linus Walleij
> >>>>>>> <linus.walleij@linaro.org>
> >>>>>>> Subject: Re: [PATCH V4 09/11] mmc: block: Add CQE support
> >>>>>>>
> >>>>>>> On 09/08/17 03:55, Shawn Lin wrote:
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>> On 2017/8/8 20:07, Bough Chen wrote:
> >>>>>>>>>> -----Original Message-----
> >>>>>>>>>> From: Adrian Hunter [mailto:adrian.hunter@intel.com]
> >>>>>>>>>> Sent: Friday, July 21, 2017 5:50 PM
> >>>>>>>>>> To: Ulf Hansson <ulf.hansson@linaro.org>
> >>>>>>>>>> Cc: linux-mmc <linux-mmc@vger.kernel.org>; Bough Chen
> >>>>>>>>>> <haibo.chen@nxp.com>; Alex Lemberg
> >>> <alex.lemberg@sandisk.com>;
> >>>>>>>>>> Mateusz Nowak <mateusz.nowak@intel.com>; Yuliy Izrailov
> >>>>>>>>>> <Yuliy.Izrailov@sandisk.com>; Jaehoon Chung
> >>>>>>>>>> <jh80.chung@samsung.com>; Dong Aisheng
> >>> <dongas86@gmail.com>;
> >>>> Das
> >>>>>>>>>> Asutosh <asutoshd@codeaurora.org>; Zhangfei Gao
> >>>>>>>>>> <zhangfei.gao@gmail.com>; Dorfman Konstantin
> >>>>>>>>>> <kdorfman@codeaurora.org>; David Griego
> >>>>>>>>>> <david.griego@linaro.org>; Sahitya Tummala
> >>>>>>>>>> <stummala@codeaurora.org>; Harjani Ritesh
> >>>>>>>>>> <riteshh@codeaurora.org>; Venu Byravarasu
> >>>>>>>>>> <vbyravarasu@nvidia.com>; Linus Walleij
> >>>>>>>>>> <linus.walleij@linaro.org>; Shawn Lin
> >>>>>>>>>> <shawn.lin@rock-chips.com>
> >>>>>>>>>> Subject: [PATCH V4 09/11] mmc: block: Add CQE support
> >>>>>>>>>>
> >>>>>>>>>> Add CQE support to the block driver, including:
> >>>>>>>>>>     - optionally using DCMD for flush requests
> >>>>>>>>>>     - manually issuing discard requests
> >>>>>>>>>>     - issuing read / write requests to the CQE
> >>>>>>>>>>     - supporting block-layer timeouts
> >>>>>>>>>>     - handling recovery
> >>>>>>>>>>     - supporting re-tuning
> >>>>>>>>>>
> >>>>>>>>>> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
> >>>>>>>>>> ---
> >>>>>>>>>>   drivers/mmc/core/block.c | 195
> >>>>>>> ++++++++++++++++++++++++++++++++-
> >>>>>>>>>>   drivers/mmc/core/block.h |   7 ++
> >>>>>>>>>>   drivers/mmc/core/queue.c | 273
> >>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++-
> >>>>>>>>>>   drivers/mmc/core/queue.h |  42 +++++++-
> >>>>>>>>>>   4 files changed, 510 insertions(+), 7 deletions(-)
> >>>>>>>>>>
> >>>>>>>>>> diff --git a/drivers/mmc/core/block.c
> >>>>>>>>>> b/drivers/mmc/core/block.c index
> >>>>>>>>>> 915290c74363..2d25115637b7 100644
> >>>>>>>>>> --- a/drivers/mmc/core/block.c
> >>>>>>>>>> +++ b/drivers/mmc/core/block.c
> >>>>>>>>>> @@ -109,6 +109,7 @@ struct mmc_blk_data {
> >>>>>>>>>>   #define MMC_BLK_WRITE        BIT(1)
> >>>>>>>>>>   #define MMC_BLK_DISCARD        BIT(2)
> >>>>>>>>>>   #define MMC_BLK_SECDISCARD    BIT(3)
> >>>>>>>>>> +#define MMC_BLK_CQE_RECOVERY    BIT(4)
> >>>>>>>>>>
> >>>>>>>>>>       /*
> >>>>>>>>>>        * Only set in main mmc_blk_data associated @@ -1612,6
> >>>>>>>>>> +1613,198 @@ static void mmc_blk_data_prep(struct
> mmc_queue
> >>>> *mq,
> >>>>>>>>>> struct mmc_queue_req *mqrq,
> >>>>>>>>>>           *do_data_tag_p = do_data_tag;
> >>>>>>>>>>   }
> >>>>>>>>>>
> >>>>>>>>>> +#define MMC_CQE_RETRIES 2
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>> +        blk_queue_rq_timed_out(mq->queue,
> mmc_cqe_timed_out);
> >>>>>>>>>> +        blk_queue_rq_timeout(mq->queue, 60 * HZ);
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> ------8<-------
> >>>>>>>>
> >>>>>>>>> Hi Adrian,
> >>>>>>>>>
> >>>>>>>>> These days I'm doing CMDQ stress test, and find one issue.
> >>>>>>>>> On our i.MX8QXP-ARM2 board, the RAM is 3GB. eMMC is 32GB.
> >>>>>>>>> I use command 'free -m' get the total memory is 2800M, and the
> >>>>>>>>> free memory is 2500M.
> >>>>>>>>>
> >>>>>>>>> I use 'mkfs.ext4' to format ext4 file system on the eMMC under
> >>>>>>>>> HS400ES CMDQ mode, works fine.
> >>>>>>>>>
> >>>>>>>>> When I use the following command to stress test CMDQ, it works
> fine.
> >>>>>>>>> bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 2048 -r 1024
> >>>>>>>>>
> >>>>>>>>> But when I change to use a large file size to do the same
> >>>>>>>>> stress test, using
> >>>>>>>>> bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 4096 -r 2048
> >>>>>>>>> or
> >>>>>>>>> bonnie++ -d /run/media/mmcblk0p1/ -u 0:0 -s 5600
> >>>>>>>>>
> >>>>>>>>> I get the following dump message.  According to the log,
> >>>>>>>>> mmc_cqe_timed_out() was trigged.
> >>>>>>>>> Seems mmc was blocked in somewhere.
> >>>>>>>>> Then I try to debug this issue, and open MMC_DEBUG in config,
> >>>>>>>>> do the same test, print the detail Command sending information
> >>>>>>>>> on the console, but finally can't reproduce.
> >>>>>>>
> >>>>>>> mmc_cqe_timed_out() is a 60 second timeout provided by the block
> >>> layer.
> >>>>>>> Refer "blk_queue_rq_timeout(mq->queue, 60 * HZ)" in
> >>>> mmc_init_queue().
> >>>>>>> 60s is quite a long time so I would first want to determine if
> >>>>>>> the task was really queued that long.  I would instrument some
> >>>>>>> code into
> >>>>>>> cqhci_request() to record the start time on struct mmc_request,
> >>>>>>> and then print the time taken when there is a problem.
> >>>>>>>
> >>>>>>
> >>>>>> Hi Adrian,
> >>>>>>
> >>>>>> According to your suggestion, I add the following code to print the time.
> >>>>>> When issue happens, seems the request really pending for over 60s!
> >>>>>>
> >>>>>> done
> >>>>>> Writing intelligently...[  689.209548] mmc0: cqhci: timeout for
> >>>>>> tag
> >>>>>> 9 [  689.213658] the mrq all use 62123742 us [  689.217487] mmc0:
> >>>>>> cqhci: ============ CQHCI REGISTER DUMP ===========
> >>>>>> [  689.223927] mmc0: cqhci: Caps:      0x0000310a | Version:  0x00000510
> >>>>>> [  689.230363] mmc0: cqhci: Config:    0x00001001 | Control:  0x00000000
> >>>>>> [  689.236800] mmc0: cqhci: Int stat:  0x00000000 | Int enab: 0x00000006
> >>>>>> [  689.243238] mmc0: cqhci: Int sig:   0x00000006 | Int Coal: 0x00000000
> >>>>>> [  689.249675] mmc0: cqhci: TDL base:  0x90079000 | TDL up32:
> 0x00000000
> >>>>>> [  689.256113] mmc0: cqhci: Doorbell:  0x1fffffff | TCN:      0x00000000
> >>>>>> [  689.262550] mmc0: cqhci: Dev queue: 0x1fffefff | Dev Pend:
> 0x1fff7fff
> >>>>>> [  689.268988] mmc0: cqhci: Task clr:  0x00000000 | SSC1:     0x00011000
> >>>>>> [  689.275425] mmc0: cqhci: SSC2:      0x00000001 | DCMD rsp:
> 0x00000800
> >>>>>> [  689.281862] mmc0: cqhci: RED mask:  0xfdf9a080 | TERRI:
> 0x00000000
> >>>>>> [  689.288300] mmc0: cqhci: Resp idx:  0x0000002f | Resp arg:
> >>>>>> 0x00000900 [  689.294737] mmc0: sdhci: ============ SDHCI
> >>>>>> REGISTER DUMP =========== [  689.301176] mmc0: sdhci: Sys addr:
> >>>>>> 0xb602f000
> >>>>>> |
> >>>>>> Version:  0x00000002 [  689.307612] mmc0: sdhci: Blk size:
> >>>>>> 0x00000200 | Blk cnt:  0x00000400 [  689.314050] mmc0: sdhci:
> Argument:
> >>>> 0x000f0400 | Trn mode: 0x00000023
> >>>>>> [  689.320487] mmc0: sdhci: Present:   0x01fd858f | Host ctl: 0x00000030
> >>>>>> [  689.326925] mmc0: sdhci: Power:     0x00000002 | Blk gap:
> 0x00000080
> >>>>>> [  689.333362] mmc0: sdhci: Wake-up:   0x00000008 | Clock:
> 0x0000000f
> >>>>>> [  689.339800] mmc0: sdhci: Timeout:   0x0000008f | Int stat:
> 0x00000000
> >>>>>> [  689.346237] mmc0: sdhci: Int enab:  0x107f4000 | Sig enab:
> >>>>>> 0x107f4000 [  689.352674] mmc0: sdhci: AC12 err:  0x00000000 | Slot int:
> >>>> 0x00000502
> >>>>>> [  689.359113] mmc0: sdhci: Caps:      0x07eb0000 | Caps_1:   0x8000b407
> >>>>>> [  689.365549] mmc0: sdhci: Cmd:       0x00002c1a | Max curr: 0x00ffffff
> >>>>>> [  689.371987] mmc0: sdhci: Resp[0]:   0x00000900 | Resp[1]:  0xffffffff
> >>>>>> [  689.378424] mmc0: sdhci: Resp[2]:   0x328f5903 | Resp[3]:
> 0x00d02700
> >>>>>> [  689.384861] mmc0: sdhci: Host ctl2: 0x00000008 [  689.389302]
> >>>>>> mmc0: sdhci: ADMA Err:  0x00000009 | ADMA Ptr: 0x9009a400 [
> >>>>>> 689.395737] mmc0: sdhci:
> >>>> ============================================
> >>>>>> [  689.402212] mmc0: running CQE recovery
> >>>>>
> >>>>> Tag 9 has been queued (bit set in Dev Pend) which means it is up
> >>>>> to the eMMC to select it for execution.  You should dump the times
> >>>>> for the other mrq's to see how long they have been waiting and try
> >>>>> to determine if anything is being processed.
> >>>>>
> >>>>> If the eMMC is just taking a really long time to process tasks we
> >>>>> could extend the timeout, but it is hard to see how that is
> >>>>> acceptable to a final product.  At this point it looks like the
> >>>>> eMMC may have a flaw in the way it selects tasks for execution.
> >>>>
> >>>> No, that is wrong sorry, the task is in the QSR (Dev queue) so it
> >>>> is the CQE that has not selected it.
> >>>
> >>> The timeout tag is 9, for Dev queue: 0x1fffefff, bit 9 is 1, means
> >>> task 9 already queue in eMMC.
> >>> For Dev Pend: 0x1fff7fff, the bit 9 is also 1,  which means CQE
> >>> already send
> >>> CMD44 and CMD45, but still not send CMD46/47.  Seems our CQE pending
> >>> tag 9 for over 60s! I will check with our IC guys to confirm the hardware
> mechanism.
> >>>
> >>
> >> For the eMMC chip, the sequential wirte speed test by 'dd' is around
> 100MB/s.
> >> If each tag try to write 1GB data, which meas each tag needs 10s to
> >> complete, once The number of pending tags exceed 6, 60s timeout will be
> trigged.
> >
> > The request size is limited by the block layer due to host controller
> > parameters.  In the case of SDHCI to 512KiB.  So each tag is at most 512KiB.
> >
> 
> I just found a bug in 32-bit DMA.  Are you using 32-bit DMA?  That could also be
> causing your problem.  I will send a new version of the patches with a fix,
> probably later today.

Yes, I'm using 32-bit ADMA.