All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v5 0/3] Handle update hardware queues and queue freeze more carefully
@ 2021-08-18 12:05 ` Daniel Wagner
  0 siblings, 0 replies; 22+ messages in thread
From: Daniel Wagner @ 2021-08-18 12:05 UTC (permalink / raw)
  To: linux-nvme
  Cc: linux-kernel, James Smart, Keith Busch, Ming Lei, Sagi Grimberg,
	Hannes Reinecke, Wen Xiong, Himanshu Madhani, Daniel Wagner

Hi,

I've dropped all non FC patches as they were bogus. I've retested this
version with all combinations and all looks good now. Also I gave
nvme-tcp a spin and again all is good.

Daniel

v1:
 - https://lore.kernel.org/linux-nvme/20210625101649.49296-1-dwagner@suse.de/
v2:
 - https://lore.kernel.org/linux-nvme/20210708092755.15660-1-dwagner@suse.de/
 - reviewed tags collected
 - added 'update hardware queues' for all transport
 - added fix for fc hanger in nvme_wait_freeze_timeout
v3:
 - https://lore.kernel.org/linux-nvme/20210720124353.127959-1-dwagner@suse.de/
 - dropped 'nvme-fc: Freeze queues before destroying them'
 - added James' two patches
v4:
 - https://lore.kernel.org/linux-nvme/20210802112658.75875-1-dwagner@suse.de/
 - added 'nvme-*: Unfreeze queues on reconnect'
 - added Hannes' reviewed tags
v5:
 - dropped non nvme-fc patches
 - updated 'nvme-fc: fix controller reset hang during traffic'

Daniel Wagner (1):
  nvme-fc: Wait with a timeout for queue to freeze

James Smart (2):
  nvme-fc: avoid race between time out and tear down
  nvme-fc: fix controller reset hang during traffic

 drivers/nvme/host/fc.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

-- 
2.29.2


^ permalink raw reply	[flat|nested] 22+ messages in thread

* [PATCH v5 0/3] Handle update hardware queues and queue freeze more carefully
@ 2021-08-18 12:05 ` Daniel Wagner
  0 siblings, 0 replies; 22+ messages in thread
From: Daniel Wagner @ 2021-08-18 12:05 UTC (permalink / raw)
  To: linux-nvme
  Cc: linux-kernel, James Smart, Keith Busch, Ming Lei, Sagi Grimberg,
	Hannes Reinecke, Wen Xiong, Himanshu Madhani, Daniel Wagner

Hi,

I've dropped all non FC patches as they were bogus. I've retested this
version with all combinations and all looks good now. Also I gave
nvme-tcp a spin and again all is good.

Daniel

v1:
 - https://lore.kernel.org/linux-nvme/20210625101649.49296-1-dwagner@suse.de/
v2:
 - https://lore.kernel.org/linux-nvme/20210708092755.15660-1-dwagner@suse.de/
 - reviewed tags collected
 - added 'update hardware queues' for all transport
 - added fix for fc hanger in nvme_wait_freeze_timeout
v3:
 - https://lore.kernel.org/linux-nvme/20210720124353.127959-1-dwagner@suse.de/
 - dropped 'nvme-fc: Freeze queues before destroying them'
 - added James' two patches
v4:
 - https://lore.kernel.org/linux-nvme/20210802112658.75875-1-dwagner@suse.de/
 - added 'nvme-*: Unfreeze queues on reconnect'
 - added Hannes' reviewed tags
v5:
 - dropped non nvme-fc patches
 - updated 'nvme-fc: fix controller reset hang during traffic'

Daniel Wagner (1):
  nvme-fc: Wait with a timeout for queue to freeze

James Smart (2):
  nvme-fc: avoid race between time out and tear down
  nvme-fc: fix controller reset hang during traffic

 drivers/nvme/host/fc.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply	[flat|nested] 22+ messages in thread

* [PATCH v5 1/3] nvme-fc: Wait with a timeout for queue to freeze
  2021-08-18 12:05 ` Daniel Wagner
@ 2021-08-18 12:05   ` Daniel Wagner
  -1 siblings, 0 replies; 22+ messages in thread
From: Daniel Wagner @ 2021-08-18 12:05 UTC (permalink / raw)
  To: linux-nvme
  Cc: linux-kernel, James Smart, Keith Busch, Ming Lei, Sagi Grimberg,
	Hannes Reinecke, Wen Xiong, Himanshu Madhani, Daniel Wagner,
	James Smart

Do not wait indifinitly for all queues to freeze. Instead use a
timeout and abort the operation if we get stuck.

Reviewed-by: James Smart <jsmart2021@gmail.com>
Reviewed-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: Daniel Wagner <dwagner@suse.de>
---
 drivers/nvme/host/fc.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index 7f462af1b02a..8e59e6d142c1 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -2963,7 +2963,14 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)
 		dev_info(ctrl->ctrl.device,
 			"reconnect: revising io queue count from %d to %d\n",
 			prior_ioq_cnt, nr_io_queues);
-		nvme_wait_freeze(&ctrl->ctrl);
+		if (!nvme_wait_freeze_timeout(&ctrl->ctrl, NVME_IO_TIMEOUT)) {
+			/*
+			 * If we timed out waiting for freeze we are likely to
+			 * be stuck.  Fail the controller initialization just
+			 * to be safe.
+			 */
+			return -ENODEV;
+		}
 		blk_mq_update_nr_hw_queues(&ctrl->tag_set, nr_io_queues);
 		nvme_unfreeze(&ctrl->ctrl);
 	}
-- 
2.29.2


^ permalink raw reply related	[flat|nested] 22+ messages in thread

* [PATCH v5 1/3] nvme-fc: Wait with a timeout for queue to freeze
@ 2021-08-18 12:05   ` Daniel Wagner
  0 siblings, 0 replies; 22+ messages in thread
From: Daniel Wagner @ 2021-08-18 12:05 UTC (permalink / raw)
  To: linux-nvme
  Cc: linux-kernel, James Smart, Keith Busch, Ming Lei, Sagi Grimberg,
	Hannes Reinecke, Wen Xiong, Himanshu Madhani, Daniel Wagner,
	James Smart

Do not wait indifinitly for all queues to freeze. Instead use a
timeout and abort the operation if we get stuck.

Reviewed-by: James Smart <jsmart2021@gmail.com>
Reviewed-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: Daniel Wagner <dwagner@suse.de>
---
 drivers/nvme/host/fc.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index 7f462af1b02a..8e59e6d142c1 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -2963,7 +2963,14 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)
 		dev_info(ctrl->ctrl.device,
 			"reconnect: revising io queue count from %d to %d\n",
 			prior_ioq_cnt, nr_io_queues);
-		nvme_wait_freeze(&ctrl->ctrl);
+		if (!nvme_wait_freeze_timeout(&ctrl->ctrl, NVME_IO_TIMEOUT)) {
+			/*
+			 * If we timed out waiting for freeze we are likely to
+			 * be stuck.  Fail the controller initialization just
+			 * to be safe.
+			 */
+			return -ENODEV;
+		}
 		blk_mq_update_nr_hw_queues(&ctrl->tag_set, nr_io_queues);
 		nvme_unfreeze(&ctrl->ctrl);
 	}
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply related	[flat|nested] 22+ messages in thread

* [PATCH v5 2/3] nvme-fc: avoid race between time out and tear down
  2021-08-18 12:05 ` Daniel Wagner
@ 2021-08-18 12:05   ` Daniel Wagner
  -1 siblings, 0 replies; 22+ messages in thread
From: Daniel Wagner @ 2021-08-18 12:05 UTC (permalink / raw)
  To: linux-nvme
  Cc: linux-kernel, James Smart, Keith Busch, Ming Lei, Sagi Grimberg,
	Hannes Reinecke, Wen Xiong, Himanshu Madhani, James Smart,
	Chao Leng, Daniel Wagner

From: James Smart <jsmart2021@gmail.com>

To avoid race between time out and tear down, in tear down process,
first we quiesce the queue, and then delete the timer and cancel
the time out work for the queue.

This patch merges the admin and io sync ops into the queue teardown logic
as shown in the RDMA patch 3017013dcc "nvme-rdma: avoid race between time
out and tear down". There is no teardown_lock in nvme-fc.

Signed-off-by: James Smart <jsmart2021@gmail.com>
CC: Chao Leng <lengchao@huawei.com>
Tested-by: Daniel Wagner <dwagner@suse.de>
[dwagner: updated commit id referenced in commit message]
Reviewed-by: Daniel Wagner <dwagner@suse.de>
Reviewed-by: Hannes Reinecke <hare@suse.de>
---
 drivers/nvme/host/fc.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index 8e59e6d142c1..3ff783a2e9f7 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -2487,6 +2487,7 @@ __nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl *ctrl, bool start_queues)
 	 */
 	if (ctrl->ctrl.queue_count > 1) {
 		nvme_stop_queues(&ctrl->ctrl);
+		nvme_sync_io_queues(&ctrl->ctrl);
 		blk_mq_tagset_busy_iter(&ctrl->tag_set,
 				nvme_fc_terminate_exchange, &ctrl->ctrl);
 		blk_mq_tagset_wait_completed_request(&ctrl->tag_set);
@@ -2510,6 +2511,7 @@ __nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl *ctrl, bool start_queues)
 	 * clean up the admin queue. Same thing as above.
 	 */
 	blk_mq_quiesce_queue(ctrl->ctrl.admin_q);
+	blk_sync_queue(ctrl->ctrl.admin_q);
 	blk_mq_tagset_busy_iter(&ctrl->admin_tag_set,
 				nvme_fc_terminate_exchange, &ctrl->ctrl);
 	blk_mq_tagset_wait_completed_request(&ctrl->admin_tag_set);
-- 
2.29.2


^ permalink raw reply related	[flat|nested] 22+ messages in thread

* [PATCH v5 2/3] nvme-fc: avoid race between time out and tear down
@ 2021-08-18 12:05   ` Daniel Wagner
  0 siblings, 0 replies; 22+ messages in thread
From: Daniel Wagner @ 2021-08-18 12:05 UTC (permalink / raw)
  To: linux-nvme
  Cc: linux-kernel, James Smart, Keith Busch, Ming Lei, Sagi Grimberg,
	Hannes Reinecke, Wen Xiong, Himanshu Madhani, James Smart,
	Chao Leng, Daniel Wagner

From: James Smart <jsmart2021@gmail.com>

To avoid race between time out and tear down, in tear down process,
first we quiesce the queue, and then delete the timer and cancel
the time out work for the queue.

This patch merges the admin and io sync ops into the queue teardown logic
as shown in the RDMA patch 3017013dcc "nvme-rdma: avoid race between time
out and tear down". There is no teardown_lock in nvme-fc.

Signed-off-by: James Smart <jsmart2021@gmail.com>
CC: Chao Leng <lengchao@huawei.com>
Tested-by: Daniel Wagner <dwagner@suse.de>
[dwagner: updated commit id referenced in commit message]
Reviewed-by: Daniel Wagner <dwagner@suse.de>
Reviewed-by: Hannes Reinecke <hare@suse.de>
---
 drivers/nvme/host/fc.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index 8e59e6d142c1..3ff783a2e9f7 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -2487,6 +2487,7 @@ __nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl *ctrl, bool start_queues)
 	 */
 	if (ctrl->ctrl.queue_count > 1) {
 		nvme_stop_queues(&ctrl->ctrl);
+		nvme_sync_io_queues(&ctrl->ctrl);
 		blk_mq_tagset_busy_iter(&ctrl->tag_set,
 				nvme_fc_terminate_exchange, &ctrl->ctrl);
 		blk_mq_tagset_wait_completed_request(&ctrl->tag_set);
@@ -2510,6 +2511,7 @@ __nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl *ctrl, bool start_queues)
 	 * clean up the admin queue. Same thing as above.
 	 */
 	blk_mq_quiesce_queue(ctrl->ctrl.admin_q);
+	blk_sync_queue(ctrl->ctrl.admin_q);
 	blk_mq_tagset_busy_iter(&ctrl->admin_tag_set,
 				nvme_fc_terminate_exchange, &ctrl->ctrl);
 	blk_mq_tagset_wait_completed_request(&ctrl->admin_tag_set);
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply related	[flat|nested] 22+ messages in thread

* [PATCH v5 3/3] nvme-fc: fix controller reset hang during traffic
  2021-08-18 12:05 ` Daniel Wagner
@ 2021-08-18 12:05   ` Daniel Wagner
  -1 siblings, 0 replies; 22+ messages in thread
From: Daniel Wagner @ 2021-08-18 12:05 UTC (permalink / raw)
  To: linux-nvme
  Cc: linux-kernel, James Smart, Keith Busch, Ming Lei, Sagi Grimberg,
	Hannes Reinecke, Wen Xiong, Himanshu Madhani, James Smart,
	Daniel Wagner

From: James Smart <jsmart2021@gmail.com>

commit fe35ec58f0d3 ("block: update hctx map when use multiple maps")
exposed an issue where we may hang trying to wait for queue freeze
during I/O. We call blk_mq_update_nr_hw_queues which may attempt to freeze
the queue. However we never started queue freeze when starting the
reset, which means that we have inflight pending requests that entered the
queue that we will not complete once the queue is quiesced.

So start a freeze before we quiesce the queue, and unfreeze the queue
after we successfully connected the I/O queues (the unfreeze is already
present in the code). blk_mq_update_nr_hw_queues will be called only
after we are sure that the queue was already frozen.

This follows to how the pci driver handles resets.

This patch added logic introduced in commit 9f98772ba307 "nvme-rdma: fix
controller reset hang during traffic".

Signed-off-by: James Smart <jsmart2021@gmail.com>
CC: Sagi Grimberg <sagi@grimberg.me>
Tested-by: Daniel Wagner <dwagner@suse.de>
Reviewed-by: Daniel Wagner <dwagner@suse.de>
---
 drivers/nvme/host/fc.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index 3ff783a2e9f7..99dadab2724c 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -2974,9 +2974,10 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)
 			return -ENODEV;
 		}
 		blk_mq_update_nr_hw_queues(&ctrl->tag_set, nr_io_queues);
-		nvme_unfreeze(&ctrl->ctrl);
 	}
 
+	nvme_unfreeze(&ctrl->ctrl);
+
 	return 0;
 
 out_delete_hw_queues:
@@ -3215,6 +3216,9 @@ nvme_fc_delete_association(struct nvme_fc_ctrl *ctrl)
 	ctrl->iocnt = 0;
 	spin_unlock_irqrestore(&ctrl->lock, flags);
 
+	if (ctrl->ctrl.queue_count > 1)
+		nvme_start_freeze(&ctrl->ctrl);
+
 	__nvme_fc_abort_outstanding_ios(ctrl, false);
 
 	/* kill the aens as they are a separate path */
-- 
2.29.2


^ permalink raw reply related	[flat|nested] 22+ messages in thread

* [PATCH v5 3/3] nvme-fc: fix controller reset hang during traffic
@ 2021-08-18 12:05   ` Daniel Wagner
  0 siblings, 0 replies; 22+ messages in thread
From: Daniel Wagner @ 2021-08-18 12:05 UTC (permalink / raw)
  To: linux-nvme
  Cc: linux-kernel, James Smart, Keith Busch, Ming Lei, Sagi Grimberg,
	Hannes Reinecke, Wen Xiong, Himanshu Madhani, James Smart,
	Daniel Wagner

From: James Smart <jsmart2021@gmail.com>

commit fe35ec58f0d3 ("block: update hctx map when use multiple maps")
exposed an issue where we may hang trying to wait for queue freeze
during I/O. We call blk_mq_update_nr_hw_queues which may attempt to freeze
the queue. However we never started queue freeze when starting the
reset, which means that we have inflight pending requests that entered the
queue that we will not complete once the queue is quiesced.

So start a freeze before we quiesce the queue, and unfreeze the queue
after we successfully connected the I/O queues (the unfreeze is already
present in the code). blk_mq_update_nr_hw_queues will be called only
after we are sure that the queue was already frozen.

This follows to how the pci driver handles resets.

This patch added logic introduced in commit 9f98772ba307 "nvme-rdma: fix
controller reset hang during traffic".

Signed-off-by: James Smart <jsmart2021@gmail.com>
CC: Sagi Grimberg <sagi@grimberg.me>
Tested-by: Daniel Wagner <dwagner@suse.de>
Reviewed-by: Daniel Wagner <dwagner@suse.de>
---
 drivers/nvme/host/fc.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index 3ff783a2e9f7..99dadab2724c 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -2974,9 +2974,10 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)
 			return -ENODEV;
 		}
 		blk_mq_update_nr_hw_queues(&ctrl->tag_set, nr_io_queues);
-		nvme_unfreeze(&ctrl->ctrl);
 	}
 
+	nvme_unfreeze(&ctrl->ctrl);
+
 	return 0;
 
 out_delete_hw_queues:
@@ -3215,6 +3216,9 @@ nvme_fc_delete_association(struct nvme_fc_ctrl *ctrl)
 	ctrl->iocnt = 0;
 	spin_unlock_irqrestore(&ctrl->lock, flags);
 
+	if (ctrl->ctrl.queue_count > 1)
+		nvme_start_freeze(&ctrl->ctrl);
+
 	__nvme_fc_abort_outstanding_ios(ctrl, false);
 
 	/* kill the aens as they are a separate path */
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply related	[flat|nested] 22+ messages in thread

* Re: [PATCH v5 3/3] nvme-fc: fix controller reset hang during traffic
  2021-08-18 12:05   ` Daniel Wagner
@ 2021-08-18 12:21     ` Hannes Reinecke
  -1 siblings, 0 replies; 22+ messages in thread
From: Hannes Reinecke @ 2021-08-18 12:21 UTC (permalink / raw)
  To: Daniel Wagner, linux-nvme
  Cc: linux-kernel, James Smart, Keith Busch, Ming Lei, Sagi Grimberg,
	Wen Xiong, Himanshu Madhani, James Smart

On 8/18/21 2:05 PM, Daniel Wagner wrote:
> From: James Smart <jsmart2021@gmail.com>
> 
> commit fe35ec58f0d3 ("block: update hctx map when use multiple maps")
> exposed an issue where we may hang trying to wait for queue freeze
> during I/O. We call blk_mq_update_nr_hw_queues which may attempt to freeze
> the queue. However we never started queue freeze when starting the
> reset, which means that we have inflight pending requests that entered the
> queue that we will not complete once the queue is quiesced.
> 
> So start a freeze before we quiesce the queue, and unfreeze the queue
> after we successfully connected the I/O queues (the unfreeze is already
> present in the code). blk_mq_update_nr_hw_queues will be called only
> after we are sure that the queue was already frozen.
> 
> This follows to how the pci driver handles resets.
> 
> This patch added logic introduced in commit 9f98772ba307 "nvme-rdma: fix
> controller reset hang during traffic".
> 
> Signed-off-by: James Smart <jsmart2021@gmail.com>
> CC: Sagi Grimberg <sagi@grimberg.me>
> Tested-by: Daniel Wagner <dwagner@suse.de>
> Reviewed-by: Daniel Wagner <dwagner@suse.de>
> ---
>  drivers/nvme/host/fc.c | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
> index 3ff783a2e9f7..99dadab2724c 100644
> --- a/drivers/nvme/host/fc.c
> +++ b/drivers/nvme/host/fc.c
> @@ -2974,9 +2974,10 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)
>  			return -ENODEV;
>  		}
>  		blk_mq_update_nr_hw_queues(&ctrl->tag_set, nr_io_queues);
> -		nvme_unfreeze(&ctrl->ctrl);
>  	}
>  
> +	nvme_unfreeze(&ctrl->ctrl);
> +
>  	return 0;
>  
>  out_delete_hw_queues:
> @@ -3215,6 +3216,9 @@ nvme_fc_delete_association(struct nvme_fc_ctrl *ctrl)
>  	ctrl->iocnt = 0;
>  	spin_unlock_irqrestore(&ctrl->lock, flags);
>  
> +	if (ctrl->ctrl.queue_count > 1)
> +		nvme_start_freeze(&ctrl->ctrl);
> +
>  	__nvme_fc_abort_outstanding_ios(ctrl, false);
>  
>  	/* kill the aens as they are a separate path */
> 
Reviewed-by: Hannes Reinecke <hare@suse.de>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare@suse.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH v5 3/3] nvme-fc: fix controller reset hang during traffic
@ 2021-08-18 12:21     ` Hannes Reinecke
  0 siblings, 0 replies; 22+ messages in thread
From: Hannes Reinecke @ 2021-08-18 12:21 UTC (permalink / raw)
  To: Daniel Wagner, linux-nvme
  Cc: linux-kernel, James Smart, Keith Busch, Ming Lei, Sagi Grimberg,
	Wen Xiong, Himanshu Madhani, James Smart

On 8/18/21 2:05 PM, Daniel Wagner wrote:
> From: James Smart <jsmart2021@gmail.com>
> 
> commit fe35ec58f0d3 ("block: update hctx map when use multiple maps")
> exposed an issue where we may hang trying to wait for queue freeze
> during I/O. We call blk_mq_update_nr_hw_queues which may attempt to freeze
> the queue. However we never started queue freeze when starting the
> reset, which means that we have inflight pending requests that entered the
> queue that we will not complete once the queue is quiesced.
> 
> So start a freeze before we quiesce the queue, and unfreeze the queue
> after we successfully connected the I/O queues (the unfreeze is already
> present in the code). blk_mq_update_nr_hw_queues will be called only
> after we are sure that the queue was already frozen.
> 
> This follows to how the pci driver handles resets.
> 
> This patch added logic introduced in commit 9f98772ba307 "nvme-rdma: fix
> controller reset hang during traffic".
> 
> Signed-off-by: James Smart <jsmart2021@gmail.com>
> CC: Sagi Grimberg <sagi@grimberg.me>
> Tested-by: Daniel Wagner <dwagner@suse.de>
> Reviewed-by: Daniel Wagner <dwagner@suse.de>
> ---
>  drivers/nvme/host/fc.c | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
> index 3ff783a2e9f7..99dadab2724c 100644
> --- a/drivers/nvme/host/fc.c
> +++ b/drivers/nvme/host/fc.c
> @@ -2974,9 +2974,10 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)
>  			return -ENODEV;
>  		}
>  		blk_mq_update_nr_hw_queues(&ctrl->tag_set, nr_io_queues);
> -		nvme_unfreeze(&ctrl->ctrl);
>  	}
>  
> +	nvme_unfreeze(&ctrl->ctrl);
> +
>  	return 0;
>  
>  out_delete_hw_queues:
> @@ -3215,6 +3216,9 @@ nvme_fc_delete_association(struct nvme_fc_ctrl *ctrl)
>  	ctrl->iocnt = 0;
>  	spin_unlock_irqrestore(&ctrl->lock, flags);
>  
> +	if (ctrl->ctrl.queue_count > 1)
> +		nvme_start_freeze(&ctrl->ctrl);
> +
>  	__nvme_fc_abort_outstanding_ios(ctrl, false);
>  
>  	/* kill the aens as they are a separate path */
> 
Reviewed-by: Hannes Reinecke <hare@suse.de>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare@suse.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH v5 0/3] Handle update hardware queues and queue freeze more carefully
  2021-08-18 12:05 ` Daniel Wagner
@ 2021-08-20  8:48   ` Daniel Wagner
  -1 siblings, 0 replies; 22+ messages in thread
From: Daniel Wagner @ 2021-08-20  8:48 UTC (permalink / raw)
  To: linux-nvme
  Cc: linux-kernel, James Smart, Keith Busch, Ming Lei, Sagi Grimberg,
	Hannes Reinecke, Wen Xiong, Himanshu Madhani

On Wed, Aug 18, 2021 at 02:05:27PM +0200, Daniel Wagner wrote:
> I've dropped all non FC patches as they were bogus. I've retested this
> version with all combinations and all looks good now. Also I gave
> nvme-tcp a spin and again all is good.

I forgot to mention I also dropped the first three patches from v4.
Which seems to break her testing again.

Wendy reported all her tests pass with Ming's V7 of 'blk-mq: fix
blk_mq_alloc_request_hctx' and this series *only* if 'nvme-fc: Update
hardware queues before using them' from previous version is also used.

After starring at it once more, I think I finally understood the
problem. So when we do

        ret = nvme_fc_create_hw_io_queues(ctrl, ctrl->ctrl.sqsize + 1);
        if (ret)
                goto out_free_io_queues;

        ret = nvme_fc_connect_io_queues(ctrl, ctrl->ctrl.sqsize + 1);
        if (ret)
                goto out_delete_hw_queues;

and the number of queues has changed, the connect call will fail:

 nvme2: NVME-FC{2}: create association : host wwpn 0x100000109b5a4dfa rport wwpn 0x50050768101935e5: NQN "nqn.1986-03.com.ibm:nvme:2145.0000020420006CEA"
 nvme2: Connect command failed, error wo/DNR bit: -16389

and we stop the current reconnect attempt and reschedule a new
reconnect attempt:

 nvme2: NVME-FC{2}: reset: Reconnect attempt failed (-5)
 nvme2: NVME-FC{2}: Reconnect attempt in 2 seconds

Then we try to do the same thing again which fails, thus we never
make progress.

So clearly we need to update number of queues at one point. What would
be the right thing to do here? As I understood we need to be careful
with frozen requests. Can we abort them (is this even possible in this
state?) and requeue them before we update the queue numbers?

Daniel

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH v5 0/3] Handle update hardware queues and queue freeze more carefully
@ 2021-08-20  8:48   ` Daniel Wagner
  0 siblings, 0 replies; 22+ messages in thread
From: Daniel Wagner @ 2021-08-20  8:48 UTC (permalink / raw)
  To: linux-nvme
  Cc: linux-kernel, James Smart, Keith Busch, Ming Lei, Sagi Grimberg,
	Hannes Reinecke, Wen Xiong, Himanshu Madhani

On Wed, Aug 18, 2021 at 02:05:27PM +0200, Daniel Wagner wrote:
> I've dropped all non FC patches as they were bogus. I've retested this
> version with all combinations and all looks good now. Also I gave
> nvme-tcp a spin and again all is good.

I forgot to mention I also dropped the first three patches from v4.
Which seems to break her testing again.

Wendy reported all her tests pass with Ming's V7 of 'blk-mq: fix
blk_mq_alloc_request_hctx' and this series *only* if 'nvme-fc: Update
hardware queues before using them' from previous version is also used.

After starring at it once more, I think I finally understood the
problem. So when we do

        ret = nvme_fc_create_hw_io_queues(ctrl, ctrl->ctrl.sqsize + 1);
        if (ret)
                goto out_free_io_queues;

        ret = nvme_fc_connect_io_queues(ctrl, ctrl->ctrl.sqsize + 1);
        if (ret)
                goto out_delete_hw_queues;

and the number of queues has changed, the connect call will fail:

 nvme2: NVME-FC{2}: create association : host wwpn 0x100000109b5a4dfa rport wwpn 0x50050768101935e5: NQN "nqn.1986-03.com.ibm:nvme:2145.0000020420006CEA"
 nvme2: Connect command failed, error wo/DNR bit: -16389

and we stop the current reconnect attempt and reschedule a new
reconnect attempt:

 nvme2: NVME-FC{2}: reset: Reconnect attempt failed (-5)
 nvme2: NVME-FC{2}: Reconnect attempt in 2 seconds

Then we try to do the same thing again which fails, thus we never
make progress.

So clearly we need to update number of queues at one point. What would
be the right thing to do here? As I understood we need to be careful
with frozen requests. Can we abort them (is this even possible in this
state?) and requeue them before we update the queue numbers?

Daniel

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH v5 0/3] Handle update hardware queues and queue freeze more carefully
  2021-08-20  8:48   ` Daniel Wagner
@ 2021-08-20 11:55     ` Daniel Wagner
  -1 siblings, 0 replies; 22+ messages in thread
From: Daniel Wagner @ 2021-08-20 11:55 UTC (permalink / raw)
  To: linux-nvme
  Cc: linux-kernel, James Smart, Keith Busch, Ming Lei, Sagi Grimberg,
	Hannes Reinecke, Wen Xiong, Himanshu Madhani

On Fri, Aug 20, 2021 at 10:48:32AM +0200, Daniel Wagner wrote:
> Then we try to do the same thing again which fails, thus we never
> make progress.
> 
> So clearly we need to update number of queues at one point. What would
> be the right thing to do here? As I understood we need to be careful
> with frozen requests. Can we abort them (is this even possible in this
> state?) and requeue them before we update the queue numbers?

After starring a bit longer at the reset path, I think there is no
pending request in any queue. nvme_fc_delete_association() calls
__nvme_fc_abort_outstanding_ios() which makes sure all queues are
drained (usage counter is 0). Also it clears the NVME_FC_Q_LIVE bit,
which prevents further request added to queues.

I start wonder why we have to do the nvme_start_freeze() in the first
place and why we want to wait for the freeze. 88e837ed0f1f ("nvme-fc:
wait for queues to freeze before calling update_hr_hw_queues") doesn't
really tell why we need wait for the freeze.

Given we know the usage counter of the queues is 0, I think we are
safe to move the blk_mq_update_nr_hw_queues() before the start queue
code. Also note nvme_fc_create_hw_io_queues() calls
blk_mq_freeze_queue() but it wont block as we are sure there is no
pending request.

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH v5 0/3] Handle update hardware queues and queue freeze more carefully
@ 2021-08-20 11:55     ` Daniel Wagner
  0 siblings, 0 replies; 22+ messages in thread
From: Daniel Wagner @ 2021-08-20 11:55 UTC (permalink / raw)
  To: linux-nvme
  Cc: linux-kernel, James Smart, Keith Busch, Ming Lei, Sagi Grimberg,
	Hannes Reinecke, Wen Xiong, Himanshu Madhani

On Fri, Aug 20, 2021 at 10:48:32AM +0200, Daniel Wagner wrote:
> Then we try to do the same thing again which fails, thus we never
> make progress.
> 
> So clearly we need to update number of queues at one point. What would
> be the right thing to do here? As I understood we need to be careful
> with frozen requests. Can we abort them (is this even possible in this
> state?) and requeue them before we update the queue numbers?

After starring a bit longer at the reset path, I think there is no
pending request in any queue. nvme_fc_delete_association() calls
__nvme_fc_abort_outstanding_ios() which makes sure all queues are
drained (usage counter is 0). Also it clears the NVME_FC_Q_LIVE bit,
which prevents further request added to queues.

I start wonder why we have to do the nvme_start_freeze() in the first
place and why we want to wait for the freeze. 88e837ed0f1f ("nvme-fc:
wait for queues to freeze before calling update_hr_hw_queues") doesn't
really tell why we need wait for the freeze.

Given we know the usage counter of the queues is 0, I think we are
safe to move the blk_mq_update_nr_hw_queues() before the start queue
code. Also note nvme_fc_create_hw_io_queues() calls
blk_mq_freeze_queue() but it wont block as we are sure there is no
pending request.

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH v5 0/3] Handle update hardware queues and queue freeze more carefully
  2021-08-20 11:55     ` Daniel Wagner
@ 2021-08-20 15:27       ` James Smart
  -1 siblings, 0 replies; 22+ messages in thread
From: James Smart @ 2021-08-20 15:27 UTC (permalink / raw)
  To: Daniel Wagner, linux-nvme
  Cc: linux-kernel, James Smart, Keith Busch, Ming Lei, Sagi Grimberg,
	Hannes Reinecke, Wen Xiong, Himanshu Madhani

On 8/20/2021 4:55 AM, Daniel Wagner wrote:
> On Fri, Aug 20, 2021 at 10:48:32AM +0200, Daniel Wagner wrote:
>> Then we try to do the same thing again which fails, thus we never
>> make progress.
>>
>> So clearly we need to update number of queues at one point. What would
>> be the right thing to do here? As I understood we need to be careful
>> with frozen requests. Can we abort them (is this even possible in this
>> state?) and requeue them before we update the queue numbers?
> 
> After starring a bit longer at the reset path, I think there is no
> pending request in any queue. nvme_fc_delete_association() calls
> __nvme_fc_abort_outstanding_ios() which makes sure all queues are
> drained (usage counter is 0). Also it clears the NVME_FC_Q_LIVE bit,
> which prevents further request added to queues.

yes, as long as we haven't attempted to create the io queues via 
nvme_fc_connect_io_queues(), nothing should be successful queueing and 
running down the hctx to start the io. nvme_fc_connect_io_queues() will 
use the queue for the Connect cmd, which is probably what generated the 
prior -16389 error.

Which says:"nvme-fc: Update hardware queues before using them" should be 
good to use.

> 
> I start wonder why we have to do the nvme_start_freeze() in the first
> place and why we want to wait for the freeze. 88e837ed0f1f ("nvme-fc:
> wait for queues to freeze before calling update_hr_hw_queues") doesn't
> really tell why we need wait for the freeze.

I think that is probably going to be true as well - no need to 
freeze/unfreeze around this path.  This was also a rather late add (last 
oct), so we had been running without the freezes for a long time, 
granted few devices change their queue counts.

I'll have to see if I can find what prompted the change. At first blush, 
I'm fine reverting it.

> 
> Given we know the usage counter of the queues is 0, I think we are
> safe to move the blk_mq_update_nr_hw_queues() before the start queue
> code. Also note nvme_fc_create_hw_io_queues() calls
> blk_mq_freeze_queue() but it wont block as we are sure there is no
> pending request.

Agree.

-- james

> 
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme
> 


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH v5 0/3] Handle update hardware queues and queue freeze more carefully
@ 2021-08-20 15:27       ` James Smart
  0 siblings, 0 replies; 22+ messages in thread
From: James Smart @ 2021-08-20 15:27 UTC (permalink / raw)
  To: Daniel Wagner, linux-nvme
  Cc: linux-kernel, James Smart, Keith Busch, Ming Lei, Sagi Grimberg,
	Hannes Reinecke, Wen Xiong, Himanshu Madhani

On 8/20/2021 4:55 AM, Daniel Wagner wrote:
> On Fri, Aug 20, 2021 at 10:48:32AM +0200, Daniel Wagner wrote:
>> Then we try to do the same thing again which fails, thus we never
>> make progress.
>>
>> So clearly we need to update number of queues at one point. What would
>> be the right thing to do here? As I understood we need to be careful
>> with frozen requests. Can we abort them (is this even possible in this
>> state?) and requeue them before we update the queue numbers?
> 
> After starring a bit longer at the reset path, I think there is no
> pending request in any queue. nvme_fc_delete_association() calls
> __nvme_fc_abort_outstanding_ios() which makes sure all queues are
> drained (usage counter is 0). Also it clears the NVME_FC_Q_LIVE bit,
> which prevents further request added to queues.

yes, as long as we haven't attempted to create the io queues via 
nvme_fc_connect_io_queues(), nothing should be successful queueing and 
running down the hctx to start the io. nvme_fc_connect_io_queues() will 
use the queue for the Connect cmd, which is probably what generated the 
prior -16389 error.

Which says:"nvme-fc: Update hardware queues before using them" should be 
good to use.

> 
> I start wonder why we have to do the nvme_start_freeze() in the first
> place and why we want to wait for the freeze. 88e837ed0f1f ("nvme-fc:
> wait for queues to freeze before calling update_hr_hw_queues") doesn't
> really tell why we need wait for the freeze.

I think that is probably going to be true as well - no need to 
freeze/unfreeze around this path.  This was also a rather late add (last 
oct), so we had been running without the freezes for a long time, 
granted few devices change their queue counts.

I'll have to see if I can find what prompted the change. At first blush, 
I'm fine reverting it.

> 
> Given we know the usage counter of the queues is 0, I think we are
> safe to move the blk_mq_update_nr_hw_queues() before the start queue
> code. Also note nvme_fc_create_hw_io_queues() calls
> blk_mq_freeze_queue() but it wont block as we are sure there is no
> pending request.

Agree.

-- james

> 
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme
> 


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH v5 0/3] Handle update hardware queues and queue freeze more carefully
  2021-08-18 12:05 ` Daniel Wagner
@ 2021-08-23  8:01   ` Christoph Hellwig
  -1 siblings, 0 replies; 22+ messages in thread
From: Christoph Hellwig @ 2021-08-23  8:01 UTC (permalink / raw)
  To: Daniel Wagner
  Cc: linux-nvme, linux-kernel, James Smart, Keith Busch, Ming Lei,
	Sagi Grimberg, Hannes Reinecke, Wen Xiong, Himanshu Madhani

Thanks,

applied to nvme-5.15.

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH v5 0/3] Handle update hardware queues and queue freeze more carefully
@ 2021-08-23  8:01   ` Christoph Hellwig
  0 siblings, 0 replies; 22+ messages in thread
From: Christoph Hellwig @ 2021-08-23  8:01 UTC (permalink / raw)
  To: Daniel Wagner
  Cc: linux-nvme, linux-kernel, James Smart, Keith Busch, Ming Lei,
	Sagi Grimberg, Hannes Reinecke, Wen Xiong, Himanshu Madhani

Thanks,

applied to nvme-5.15.

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH v5 0/3] Handle update hardware queues and queue freeze more carefully
  2021-08-23  8:01   ` Christoph Hellwig
@ 2021-08-23  9:14     ` Daniel Wagner
  -1 siblings, 0 replies; 22+ messages in thread
From: Daniel Wagner @ 2021-08-23  9:14 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: linux-nvme, linux-kernel, James Smart, Keith Busch, Ming Lei,
	Sagi Grimberg, Hannes Reinecke, Wen Xiong, Himanshu Madhani

Hi Christoph,

On Mon, Aug 23, 2021 at 09:01:46AM +0100, Christoph Hellwig wrote:
> applied to nvme-5.15.

Could you drop these patches again? James just sent me a new version of
the series which addresses the points we discussed in this thread. They
look good to me but I'd like to give them a spin and will post them
shortly. Also James is giving them a test run.

Daniel

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH v5 0/3] Handle update hardware queues and queue freeze more carefully
@ 2021-08-23  9:14     ` Daniel Wagner
  0 siblings, 0 replies; 22+ messages in thread
From: Daniel Wagner @ 2021-08-23  9:14 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: linux-nvme, linux-kernel, James Smart, Keith Busch, Ming Lei,
	Sagi Grimberg, Hannes Reinecke, Wen Xiong, Himanshu Madhani

Hi Christoph,

On Mon, Aug 23, 2021 at 09:01:46AM +0100, Christoph Hellwig wrote:
> applied to nvme-5.15.

Could you drop these patches again? James just sent me a new version of
the series which addresses the points we discussed in this thread. They
look good to me but I'd like to give them a spin and will post them
shortly. Also James is giving them a test run.

Daniel

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH v5 0/3] Handle update hardware queues and queue freeze more carefully
  2021-08-23  9:14     ` Daniel Wagner
@ 2021-08-23  9:19       ` Christoph Hellwig
  -1 siblings, 0 replies; 22+ messages in thread
From: Christoph Hellwig @ 2021-08-23  9:19 UTC (permalink / raw)
  To: Daniel Wagner
  Cc: Christoph Hellwig, linux-nvme, linux-kernel, James Smart,
	Keith Busch, Ming Lei, Sagi Grimberg, Hannes Reinecke, Wen Xiong,
	Himanshu Madhani

On Mon, Aug 23, 2021 at 11:14:05AM +0200, Daniel Wagner wrote:
> Hi Christoph,
> 
> On Mon, Aug 23, 2021 at 09:01:46AM +0100, Christoph Hellwig wrote:
> > applied to nvme-5.15.
> 
> Could you drop these patches again? James just sent me a new version of
> the series which addresses the points we discussed in this thread. They
> look good to me but I'd like to give them a spin and will post them
> shortly. Also James is giving them a test run.

Ok.

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH v5 0/3] Handle update hardware queues and queue freeze more carefully
@ 2021-08-23  9:19       ` Christoph Hellwig
  0 siblings, 0 replies; 22+ messages in thread
From: Christoph Hellwig @ 2021-08-23  9:19 UTC (permalink / raw)
  To: Daniel Wagner
  Cc: Christoph Hellwig, linux-nvme, linux-kernel, James Smart,
	Keith Busch, Ming Lei, Sagi Grimberg, Hannes Reinecke, Wen Xiong,
	Himanshu Madhani

On Mon, Aug 23, 2021 at 11:14:05AM +0200, Daniel Wagner wrote:
> Hi Christoph,
> 
> On Mon, Aug 23, 2021 at 09:01:46AM +0100, Christoph Hellwig wrote:
> > applied to nvme-5.15.
> 
> Could you drop these patches again? James just sent me a new version of
> the series which addresses the points we discussed in this thread. They
> look good to me but I'd like to give them a spin and will post them
> shortly. Also James is giving them a test run.

Ok.

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply	[flat|nested] 22+ messages in thread

end of thread, other threads:[~2021-08-23  9:21 UTC | newest]

Thread overview: 22+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-08-18 12:05 [PATCH v5 0/3] Handle update hardware queues and queue freeze more carefully Daniel Wagner
2021-08-18 12:05 ` Daniel Wagner
2021-08-18 12:05 ` [PATCH v5 1/3] nvme-fc: Wait with a timeout for queue to freeze Daniel Wagner
2021-08-18 12:05   ` Daniel Wagner
2021-08-18 12:05 ` [PATCH v5 2/3] nvme-fc: avoid race between time out and tear down Daniel Wagner
2021-08-18 12:05   ` Daniel Wagner
2021-08-18 12:05 ` [PATCH v5 3/3] nvme-fc: fix controller reset hang during traffic Daniel Wagner
2021-08-18 12:05   ` Daniel Wagner
2021-08-18 12:21   ` Hannes Reinecke
2021-08-18 12:21     ` Hannes Reinecke
2021-08-20  8:48 ` [PATCH v5 0/3] Handle update hardware queues and queue freeze more carefully Daniel Wagner
2021-08-20  8:48   ` Daniel Wagner
2021-08-20 11:55   ` Daniel Wagner
2021-08-20 11:55     ` Daniel Wagner
2021-08-20 15:27     ` James Smart
2021-08-20 15:27       ` James Smart
2021-08-23  8:01 ` Christoph Hellwig
2021-08-23  8:01   ` Christoph Hellwig
2021-08-23  9:14   ` Daniel Wagner
2021-08-23  9:14     ` Daniel Wagner
2021-08-23  9:19     ` Christoph Hellwig
2021-08-23  9:19       ` Christoph Hellwig

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.