[PATCH] IB: make INFINIBAND_ADDR_TRANS configurable

[PATCH] IB: make INFINIBAND_ADDR_TRANS configurable

[Greg Thelen]

Allow INFINIBAND without INFINIBAND_ADDR_TRANS.

Signed-off-by: Greg Thelen <gthelen@google.com>
Cc: Tarick Bedeir <tarick@google.com>
---
 drivers/infiniband/Kconfig | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/infiniband/Kconfig b/drivers/infiniband/Kconfig
index ee270e065ba9..f20a3977087c 100644
--- a/drivers/infiniband/Kconfig
+++ b/drivers/infiniband/Kconfig
@@ -61,7 +61,7 @@ config INFINIBAND_ON_DEMAND_PAGING
 	  pages on demand instead.
 
 config INFINIBAND_ADDR_TRANS
-	bool
+	bool "InfiniBand address translation"
 	depends on INFINIBAND
 	default y
 
-- 
2.17.0.484.g0c8726318c-goog

Re: [PATCH] IB: make INFINIBAND_ADDR_TRANS configurable

[Bart Van Assche]

On Fri, 2018-04-13 at 00:06 -0700, Greg Thelen wrote:
> Allow INFINIBAND without INFINIBAND_ADDR_TRANS.
> 
> Signed-off-by: Greg Thelen <gthelen@google.com>
> Cc: Tarick Bedeir <tarick@google.com>
> ---
>  drivers/infiniband/Kconfig | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/infiniband/Kconfig b/drivers/infiniband/Kconfig
> index ee270e065ba9..f20a3977087c 100644
> --- a/drivers/infiniband/Kconfig
> +++ b/drivers/infiniband/Kconfig
> @@ -61,7 +61,7 @@ config INFINIBAND_ON_DEMAND_PAGING
>  	  pages on demand instead.
>  
>  config INFINIBAND_ADDR_TRANS
> -	bool
> +	bool "InfiniBand address translation"
>  	depends on INFINIBAND
>  	default y

Hello Greg,

Please change the short description from "InfiniBand address translation" (which
is a misleading description) into "RDMA/CM". Please also add a help text that
explains to people who are not RDMA experts what the RDMA/CM is.

Thanks,

Bart.

[PATCH v2] IB: make INFINIBAND_ADDR_TRANS configurable

[Greg Thelen]

Allow INFINIBAND without INFINIBAND_ADDR_TRANS.

Signed-off-by: Greg Thelen <gthelen@google.com>
Cc: Tarick Bedeir <tarick@google.com>
Change-Id: I6fbbf8a432e467710fa65e4904b7d61880b914e5
---
 drivers/infiniband/Kconfig | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/drivers/infiniband/Kconfig b/drivers/infiniband/Kconfig
index ee270e065ba9..2a972ed6851b 100644
--- a/drivers/infiniband/Kconfig
+++ b/drivers/infiniband/Kconfig
@@ -61,9 +61,12 @@ config INFINIBAND_ON_DEMAND_PAGING
 	  pages on demand instead.
 
 config INFINIBAND_ADDR_TRANS
-	bool
+	bool "RDMA/CM"
 	depends on INFINIBAND
 	default y
+	---help---
+	  Support for RDMA communication manager (CM).
+	  This allows for a generic connection abstraction over RDMA.
 
 config INFINIBAND_ADDR_TRANS_CONFIGFS
 	bool
-- 
2.17.0.484.g0c8726318c-goog

Re: [PATCH v2] IB: make INFINIBAND_ADDR_TRANS configurable

[Dennis Dalessandro]

On 4/13/2018 1:27 PM, Greg Thelen wrote:
> Allow INFINIBAND without INFINIBAND_ADDR_TRANS.
> 
> Signed-off-by: Greg Thelen <gthelen@google.com>
> Cc: Tarick Bedeir <tarick@google.com>
> Change-Id: I6fbbf8a432e467710fa65e4904b7d61880b914e5

Forgot to remove the Gerrit thing.

-Denny

Re: [PATCH v2] IB: make INFINIBAND_ADDR_TRANS configurable

[Greg Thelen]

On Sat, Apr 14, 2018 at 8:13 AM Dennis Dalessandro <
dennis.dalessandro@intel.com> wrote:

> On 4/13/2018 1:27 PM, Greg Thelen wrote:
> > Allow INFINIBAND without INFINIBAND_ADDR_TRANS.
> >
> > Signed-off-by: Greg Thelen <gthelen@google.com>
> > Cc: Tarick Bedeir <tarick@google.com>
> > Change-Id: I6fbbf8a432e467710fa65e4904b7d61880b914e5

> Forgot to remove the Gerrit thing.

> -Denny

Ack.  My bad.  Will repost.  Unfortunately checkpatch didn't notice.

[PATCH v3] IB: make INFINIBAND_ADDR_TRANS configurable

[Greg Thelen]

Allow INFINIBAND without INFINIBAND_ADDR_TRANS.

Signed-off-by: Greg Thelen <gthelen@google.com>
Cc: Tarick Bedeir <tarick@google.com>
---
 drivers/infiniband/Kconfig | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/drivers/infiniband/Kconfig b/drivers/infiniband/Kconfig
index ee270e065ba9..2a972ed6851b 100644
--- a/drivers/infiniband/Kconfig
+++ b/drivers/infiniband/Kconfig
@@ -61,9 +61,12 @@ config INFINIBAND_ON_DEMAND_PAGING
 	  pages on demand instead.
 
 config INFINIBAND_ADDR_TRANS
-	bool
+	bool "RDMA/CM"
 	depends on INFINIBAND
 	default y
+	---help---
+	  Support for RDMA communication manager (CM).
+	  This allows for a generic connection abstraction over RDMA.
 
 config INFINIBAND_ADDR_TRANS_CONFIGFS
 	bool
-- 
2.17.0.484.g0c8726318c-goog

Re: [PATCH v3] IB: make INFINIBAND_ADDR_TRANS configurable

[kbuild test robot]

[-- Attachment #1: Type: text/plain, Size: 12785 bytes --]

Hi Greg,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on linus/master]
[also build test ERROR on v4.16 next-20180413]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Greg-Thelen/IB-make-INFINIBAND_ADDR_TRANS-configurable/20180414-234042
config: x86_64-randconfig-x011-201815 (attached as .config)
compiler: gcc-7 (Debian 7.3.0-1) 7.3.0
reproduce:
        # save the attached .config to linux build tree
        make ARCH=x86_64 

All errors (new ones prefixed by >>):

   drivers/nvme/host/rdma.o: In function `nvme_rdma_stop_queue':
>> drivers/nvme/host/rdma.c:554: undefined reference to `rdma_disconnect'
   drivers/nvme/host/rdma.o: In function `nvme_rdma_create_qp':
>> drivers/nvme/host/rdma.c:258: undefined reference to `rdma_create_qp'
   drivers/nvme/host/rdma.o: In function `nvme_rdma_free_queue':
>> drivers/nvme/host/rdma.c:570: undefined reference to `rdma_destroy_id'
   drivers/nvme/host/rdma.o: In function `nvme_rdma_alloc_queue':
>> drivers/nvme/host/rdma.c:511: undefined reference to `__rdma_create_id'
>> drivers/nvme/host/rdma.c:523: undefined reference to `rdma_resolve_addr'
   drivers/nvme/host/rdma.c:544: undefined reference to `rdma_destroy_id'
   drivers/nvme/host/rdma.o: In function `nvme_rdma_addr_resolved':
>> drivers/nvme/host/rdma.c:1461: undefined reference to `rdma_resolve_route'
   drivers/nvme/host/rdma.o: In function `nvme_rdma_create_queue_ib':
>> drivers/nvme/host/rdma.c:485: undefined reference to `rdma_destroy_qp'
   drivers/nvme/host/rdma.o: In function `nvme_rdma_route_resolved':
>> drivers/nvme/host/rdma.c:1512: undefined reference to `rdma_connect'
   drivers/nvme/host/rdma.o: In function `nvme_rdma_conn_rejected':
>> drivers/nvme/host/rdma.c:1436: undefined reference to `rdma_reject_msg'
>> drivers/nvme/host/rdma.c:1437: undefined reference to `rdma_consumer_reject_data'

vim +554 drivers/nvme/host/rdma.c

f41725bb Israel Rukshin    2017-11-26  423  
ca6e95bb Sagi Grimberg     2017-05-04  424  static int nvme_rdma_create_queue_ib(struct nvme_rdma_queue *queue)
71102307 Christoph Hellwig 2016-07-06  425  {
ca6e95bb Sagi Grimberg     2017-05-04  426  	struct ib_device *ibdev;
71102307 Christoph Hellwig 2016-07-06  427  	const int send_wr_factor = 3;			/* MR, SEND, INV */
71102307 Christoph Hellwig 2016-07-06  428  	const int cq_factor = send_wr_factor + 1;	/* + RECV */
71102307 Christoph Hellwig 2016-07-06  429  	int comp_vector, idx = nvme_rdma_queue_idx(queue);
71102307 Christoph Hellwig 2016-07-06  430  	int ret;
71102307 Christoph Hellwig 2016-07-06  431  
ca6e95bb Sagi Grimberg     2017-05-04  432  	queue->device = nvme_rdma_find_get_device(queue->cm_id);
ca6e95bb Sagi Grimberg     2017-05-04  433  	if (!queue->device) {
ca6e95bb Sagi Grimberg     2017-05-04  434  		dev_err(queue->cm_id->device->dev.parent,
ca6e95bb Sagi Grimberg     2017-05-04  435  			"no client data found!\n");
ca6e95bb Sagi Grimberg     2017-05-04  436  		return -ECONNREFUSED;
ca6e95bb Sagi Grimberg     2017-05-04  437  	}
ca6e95bb Sagi Grimberg     2017-05-04  438  	ibdev = queue->device->dev;
71102307 Christoph Hellwig 2016-07-06  439  
71102307 Christoph Hellwig 2016-07-06  440  	/*
0b36658c Sagi Grimberg     2017-07-13  441  	 * Spread I/O queues completion vectors according their queue index.
0b36658c Sagi Grimberg     2017-07-13  442  	 * Admin queues can always go on completion vector 0.
71102307 Christoph Hellwig 2016-07-06  443  	 */
0b36658c Sagi Grimberg     2017-07-13  444  	comp_vector = idx == 0 ? idx : idx - 1;
71102307 Christoph Hellwig 2016-07-06  445  
71102307 Christoph Hellwig 2016-07-06  446  	/* +1 for ib_stop_cq */
ca6e95bb Sagi Grimberg     2017-05-04  447  	queue->ib_cq = ib_alloc_cq(ibdev, queue,
ca6e95bb Sagi Grimberg     2017-05-04  448  				cq_factor * queue->queue_size + 1,
ca6e95bb Sagi Grimberg     2017-05-04  449  				comp_vector, IB_POLL_SOFTIRQ);
71102307 Christoph Hellwig 2016-07-06  450  	if (IS_ERR(queue->ib_cq)) {
71102307 Christoph Hellwig 2016-07-06  451  		ret = PTR_ERR(queue->ib_cq);
ca6e95bb Sagi Grimberg     2017-05-04  452  		goto out_put_dev;
71102307 Christoph Hellwig 2016-07-06  453  	}
71102307 Christoph Hellwig 2016-07-06  454  
71102307 Christoph Hellwig 2016-07-06  455  	ret = nvme_rdma_create_qp(queue, send_wr_factor);
71102307 Christoph Hellwig 2016-07-06  456  	if (ret)
71102307 Christoph Hellwig 2016-07-06  457  		goto out_destroy_ib_cq;
71102307 Christoph Hellwig 2016-07-06  458  
71102307 Christoph Hellwig 2016-07-06  459  	queue->rsp_ring = nvme_rdma_alloc_ring(ibdev, queue->queue_size,
71102307 Christoph Hellwig 2016-07-06  460  			sizeof(struct nvme_completion), DMA_FROM_DEVICE);
71102307 Christoph Hellwig 2016-07-06  461  	if (!queue->rsp_ring) {
71102307 Christoph Hellwig 2016-07-06  462  		ret = -ENOMEM;
71102307 Christoph Hellwig 2016-07-06  463  		goto out_destroy_qp;
71102307 Christoph Hellwig 2016-07-06  464  	}
71102307 Christoph Hellwig 2016-07-06  465  
f41725bb Israel Rukshin    2017-11-26  466  	ret = ib_mr_pool_init(queue->qp, &queue->qp->rdma_mrs,
f41725bb Israel Rukshin    2017-11-26  467  			      queue->queue_size,
f41725bb Israel Rukshin    2017-11-26  468  			      IB_MR_TYPE_MEM_REG,
f41725bb Israel Rukshin    2017-11-26  469  			      nvme_rdma_get_max_fr_pages(ibdev));
f41725bb Israel Rukshin    2017-11-26  470  	if (ret) {
f41725bb Israel Rukshin    2017-11-26  471  		dev_err(queue->ctrl->ctrl.device,
f41725bb Israel Rukshin    2017-11-26  472  			"failed to initialize MR pool sized %d for QID %d\n",
f41725bb Israel Rukshin    2017-11-26  473  			queue->queue_size, idx);
f41725bb Israel Rukshin    2017-11-26  474  		goto out_destroy_ring;
f41725bb Israel Rukshin    2017-11-26  475  	}
f41725bb Israel Rukshin    2017-11-26  476  
eb1bd249 Max Gurtovoy      2017-11-28  477  	set_bit(NVME_RDMA_Q_TR_READY, &queue->flags);
eb1bd249 Max Gurtovoy      2017-11-28  478  
71102307 Christoph Hellwig 2016-07-06  479  	return 0;
71102307 Christoph Hellwig 2016-07-06  480  
f41725bb Israel Rukshin    2017-11-26  481  out_destroy_ring:
f41725bb Israel Rukshin    2017-11-26  482  	nvme_rdma_free_ring(ibdev, queue->rsp_ring, queue->queue_size,
f41725bb Israel Rukshin    2017-11-26  483  			    sizeof(struct nvme_completion), DMA_FROM_DEVICE);
71102307 Christoph Hellwig 2016-07-06  484  out_destroy_qp:
1f61def9 Max Gurtovoy      2017-11-06 @485  	rdma_destroy_qp(queue->cm_id);
71102307 Christoph Hellwig 2016-07-06  486  out_destroy_ib_cq:
71102307 Christoph Hellwig 2016-07-06  487  	ib_free_cq(queue->ib_cq);
ca6e95bb Sagi Grimberg     2017-05-04  488  out_put_dev:
ca6e95bb Sagi Grimberg     2017-05-04  489  	nvme_rdma_dev_put(queue->device);
71102307 Christoph Hellwig 2016-07-06  490  	return ret;
71102307 Christoph Hellwig 2016-07-06  491  }
71102307 Christoph Hellwig 2016-07-06  492  
41e8cfa1 Sagi Grimberg     2017-07-10  493  static int nvme_rdma_alloc_queue(struct nvme_rdma_ctrl *ctrl,
71102307 Christoph Hellwig 2016-07-06  494  		int idx, size_t queue_size)
71102307 Christoph Hellwig 2016-07-06  495  {
71102307 Christoph Hellwig 2016-07-06  496  	struct nvme_rdma_queue *queue;
8f4e8dac Max Gurtovoy      2017-02-19  497  	struct sockaddr *src_addr = NULL;
71102307 Christoph Hellwig 2016-07-06  498  	int ret;
71102307 Christoph Hellwig 2016-07-06  499  
71102307 Christoph Hellwig 2016-07-06  500  	queue = &ctrl->queues[idx];
71102307 Christoph Hellwig 2016-07-06  501  	queue->ctrl = ctrl;
71102307 Christoph Hellwig 2016-07-06  502  	init_completion(&queue->cm_done);
71102307 Christoph Hellwig 2016-07-06  503  
71102307 Christoph Hellwig 2016-07-06  504  	if (idx > 0)
71102307 Christoph Hellwig 2016-07-06  505  		queue->cmnd_capsule_len = ctrl->ctrl.ioccsz * 16;
71102307 Christoph Hellwig 2016-07-06  506  	else
71102307 Christoph Hellwig 2016-07-06  507  		queue->cmnd_capsule_len = sizeof(struct nvme_command);
71102307 Christoph Hellwig 2016-07-06  508  
71102307 Christoph Hellwig 2016-07-06  509  	queue->queue_size = queue_size;
71102307 Christoph Hellwig 2016-07-06  510  
71102307 Christoph Hellwig 2016-07-06 @511  	queue->cm_id = rdma_create_id(&init_net, nvme_rdma_cm_handler, queue,
71102307 Christoph Hellwig 2016-07-06  512  			RDMA_PS_TCP, IB_QPT_RC);
71102307 Christoph Hellwig 2016-07-06  513  	if (IS_ERR(queue->cm_id)) {
71102307 Christoph Hellwig 2016-07-06  514  		dev_info(ctrl->ctrl.device,
71102307 Christoph Hellwig 2016-07-06  515  			"failed to create CM ID: %ld\n", PTR_ERR(queue->cm_id));
71102307 Christoph Hellwig 2016-07-06  516  		return PTR_ERR(queue->cm_id);
71102307 Christoph Hellwig 2016-07-06  517  	}
71102307 Christoph Hellwig 2016-07-06  518  
8f4e8dac Max Gurtovoy      2017-02-19  519  	if (ctrl->ctrl.opts->mask & NVMF_OPT_HOST_TRADDR)
0928f9b4 Sagi Grimberg     2017-02-05  520  		src_addr = (struct sockaddr *)&ctrl->src_addr;
8f4e8dac Max Gurtovoy      2017-02-19  521  
0928f9b4 Sagi Grimberg     2017-02-05  522  	queue->cm_error = -ETIMEDOUT;
0928f9b4 Sagi Grimberg     2017-02-05 @523  	ret = rdma_resolve_addr(queue->cm_id, src_addr,
0928f9b4 Sagi Grimberg     2017-02-05  524  			(struct sockaddr *)&ctrl->addr,
71102307 Christoph Hellwig 2016-07-06  525  			NVME_RDMA_CONNECT_TIMEOUT_MS);
71102307 Christoph Hellwig 2016-07-06  526  	if (ret) {
71102307 Christoph Hellwig 2016-07-06  527  		dev_info(ctrl->ctrl.device,
71102307 Christoph Hellwig 2016-07-06  528  			"rdma_resolve_addr failed (%d).\n", ret);
71102307 Christoph Hellwig 2016-07-06  529  		goto out_destroy_cm_id;
71102307 Christoph Hellwig 2016-07-06  530  	}
71102307 Christoph Hellwig 2016-07-06  531  
71102307 Christoph Hellwig 2016-07-06  532  	ret = nvme_rdma_wait_for_cm(queue);
71102307 Christoph Hellwig 2016-07-06  533  	if (ret) {
71102307 Christoph Hellwig 2016-07-06  534  		dev_info(ctrl->ctrl.device,
d8bfceeb Sagi Grimberg     2017-10-11  535  			"rdma connection establishment failed (%d)\n", ret);
71102307 Christoph Hellwig 2016-07-06  536  		goto out_destroy_cm_id;
71102307 Christoph Hellwig 2016-07-06  537  	}
71102307 Christoph Hellwig 2016-07-06  538  
5013e98b Sagi Grimberg     2017-10-11  539  	set_bit(NVME_RDMA_Q_ALLOCATED, &queue->flags);
71102307 Christoph Hellwig 2016-07-06  540  
71102307 Christoph Hellwig 2016-07-06  541  	return 0;
71102307 Christoph Hellwig 2016-07-06  542  
71102307 Christoph Hellwig 2016-07-06  543  out_destroy_cm_id:
71102307 Christoph Hellwig 2016-07-06 @544  	rdma_destroy_id(queue->cm_id);
eb1bd249 Max Gurtovoy      2017-11-28  545  	nvme_rdma_destroy_queue_ib(queue);
71102307 Christoph Hellwig 2016-07-06  546  	return ret;
71102307 Christoph Hellwig 2016-07-06  547  }
71102307 Christoph Hellwig 2016-07-06  548  
71102307 Christoph Hellwig 2016-07-06  549  static void nvme_rdma_stop_queue(struct nvme_rdma_queue *queue)
71102307 Christoph Hellwig 2016-07-06  550  {
a57bd541 Sagi Grimberg     2017-08-28  551  	if (!test_and_clear_bit(NVME_RDMA_Q_LIVE, &queue->flags))
a57bd541 Sagi Grimberg     2017-08-28  552  		return;
a57bd541 Sagi Grimberg     2017-08-28  553  
71102307 Christoph Hellwig 2016-07-06 @554  	rdma_disconnect(queue->cm_id);
71102307 Christoph Hellwig 2016-07-06  555  	ib_drain_qp(queue->qp);
71102307 Christoph Hellwig 2016-07-06  556  }
71102307 Christoph Hellwig 2016-07-06  557  
71102307 Christoph Hellwig 2016-07-06  558  static void nvme_rdma_free_queue(struct nvme_rdma_queue *queue)
71102307 Christoph Hellwig 2016-07-06  559  {
5013e98b Sagi Grimberg     2017-10-11  560  	if (!test_and_clear_bit(NVME_RDMA_Q_ALLOCATED, &queue->flags))
a57bd541 Sagi Grimberg     2017-08-28  561  		return;
a57bd541 Sagi Grimberg     2017-08-28  562  
bd9f0759 Sagi Grimberg     2017-10-19  563  	if (nvme_rdma_queue_idx(queue) == 0) {
bd9f0759 Sagi Grimberg     2017-10-19  564  		nvme_rdma_free_qe(queue->device->dev,
bd9f0759 Sagi Grimberg     2017-10-19  565  			&queue->ctrl->async_event_sqe,
bd9f0759 Sagi Grimberg     2017-10-19  566  			sizeof(struct nvme_command), DMA_TO_DEVICE);
bd9f0759 Sagi Grimberg     2017-10-19  567  	}
bd9f0759 Sagi Grimberg     2017-10-19  568  
71102307 Christoph Hellwig 2016-07-06  569  	nvme_rdma_destroy_queue_ib(queue);
71102307 Christoph Hellwig 2016-07-06 @570  	rdma_destroy_id(queue->cm_id);
71102307 Christoph Hellwig 2016-07-06  571  }
71102307 Christoph Hellwig 2016-07-06  572  

:::::: The code at line 554 was first introduced by commit
:::::: 7110230719602852481c2793d054f866b2bf4a2b nvme-rdma: add a NVMe over Fabrics RDMA host driver

:::::: TO: Christoph Hellwig <hch@lst.de>
:::::: CC: Jens Axboe <axboe@fb.com>

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 27526 bytes --]

Re: [PATCH v3] IB: make INFINIBAND_ADDR_TRANS configurable

[kbuild test robot]

[-- Attachment #1: Type: text/plain, Size: 11915 bytes --]

Hi Greg,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on linus/master]
[also build test ERROR on v4.16 next-20180413]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Greg-Thelen/IB-make-INFINIBAND_ADDR_TRANS-configurable/20180414-234042
config: i386-randconfig-x005-201815 (attached as .config)
compiler: gcc-7 (Debian 7.3.0-1) 7.3.0
reproduce:
        # save the attached .config to linux build tree
        make ARCH=i386 

All errors (new ones prefixed by >>):

   drivers/nvme/host/rdma.o: In function `nvme_rdma_stop_queue':
   drivers/nvme/host/rdma.c:554: undefined reference to `rdma_disconnect'
   drivers/nvme/host/rdma.o: In function `nvme_rdma_free_queue':
   drivers/nvme/host/rdma.c:570: undefined reference to `rdma_destroy_id'
   drivers/nvme/host/rdma.o: In function `nvme_rdma_alloc_queue':
   drivers/nvme/host/rdma.c:511: undefined reference to `__rdma_create_id'
   drivers/nvme/host/rdma.c:523: undefined reference to `rdma_resolve_addr'
   drivers/nvme/host/rdma.c:544: undefined reference to `rdma_destroy_id'
   drivers/nvme/host/rdma.o: In function `nvme_rdma_create_qp':
   drivers/nvme/host/rdma.c:258: undefined reference to `rdma_create_qp'
   drivers/nvme/host/rdma.o: In function `nvme_rdma_create_queue_ib':
   drivers/nvme/host/rdma.c:485: undefined reference to `rdma_destroy_qp'
   drivers/nvme/host/rdma.o: In function `nvme_rdma_addr_resolved':
   drivers/nvme/host/rdma.c:1461: undefined reference to `rdma_resolve_route'
   drivers/nvme/host/rdma.o: In function `nvme_rdma_route_resolved':
   drivers/nvme/host/rdma.c:1512: undefined reference to `rdma_connect'
   drivers/nvme/host/rdma.o: In function `nvme_rdma_conn_rejected':
   drivers/nvme/host/rdma.c:1436: undefined reference to `rdma_reject_msg'
   drivers/nvme/host/rdma.c:1437: undefined reference to `rdma_consumer_reject_data'
   drivers/infiniband/ulp/srp/ib_srp.o: In function `srp_create_ch_ib':
>> drivers/infiniband/ulp/srp/ib_srp.c:585: undefined reference to `rdma_create_qp'
>> drivers/infiniband/ulp/srp/ib_srp.c:647: undefined reference to `rdma_destroy_qp'
   drivers/infiniband/ulp/srp/ib_srp.o: In function `srp_disconnect_target':
>> drivers/infiniband/ulp/srp/ib_srp.c:977: undefined reference to `rdma_disconnect'
   drivers/infiniband/ulp/srp/ib_srp.o: In function `srp_new_rdma_cm_id':
>> drivers/infiniband/ulp/srp/ib_srp.c:336: undefined reference to `__rdma_create_id'
>> drivers/infiniband/ulp/srp/ib_srp.c:345: undefined reference to `rdma_resolve_addr'
>> drivers/infiniband/ulp/srp/ib_srp.c:369: undefined reference to `rdma_destroy_id'
   drivers/infiniband/ulp/srp/ib_srp.o: In function `srp_rdma_lookup_path':
>> drivers/infiniband/ulp/srp/ib_srp.c:790: undefined reference to `rdma_resolve_route'
   drivers/infiniband/ulp/srp/ib_srp.o: In function `srp_send_req':
>> drivers/infiniband/ulp/srp/ib_srp.c:938: undefined reference to `rdma_connect'
   drivers/infiniband/ulp/srp/ib_srp.o: In function `srp_free_ch_ib':
   drivers/infiniband/ulp/srp/ib_srp.c:677: undefined reference to `rdma_destroy_id'
   drivers/infiniband/ulp/srp/ib_srp.o: In function `srp_rdma_cm_handler':
   drivers/infiniband/ulp/srp/ib_srp.c:2808: undefined reference to `rdma_disconnect'

vim +585 drivers/infiniband/ulp/srp/ib_srp.c

7dad6b2e Bart Van Assche   2014-10-21  542  
509c07bc Bart Van Assche   2014-10-30  543  static int srp_create_ch_ib(struct srp_rdma_ch *ch)
aef9ec39 Roland Dreier     2005-11-02  544  {
509c07bc Bart Van Assche   2014-10-30  545  	struct srp_target_port *target = ch->target;
62154b2e Bart Van Assche   2014-05-20  546  	struct srp_device *dev = target->srp_host->srp_dev;
aef9ec39 Roland Dreier     2005-11-02  547  	struct ib_qp_init_attr *init_attr;
73aa89ed Ishai Rabinovitz  2012-11-26  548  	struct ib_cq *recv_cq, *send_cq;
73aa89ed Ishai Rabinovitz  2012-11-26  549  	struct ib_qp *qp;
d1b4289e Bart Van Assche   2014-05-20  550  	struct ib_fmr_pool *fmr_pool = NULL;
5cfb1782 Bart Van Assche   2014-05-20  551  	struct srp_fr_pool *fr_pool = NULL;
509c5f33 Bart Van Assche   2016-05-12  552  	const int m = 1 + dev->use_fast_reg * target->mr_per_cmd * 2;
aef9ec39 Roland Dreier     2005-11-02  553  	int ret;
aef9ec39 Roland Dreier     2005-11-02  554  
aef9ec39 Roland Dreier     2005-11-02  555  	init_attr = kzalloc(sizeof *init_attr, GFP_KERNEL);
aef9ec39 Roland Dreier     2005-11-02  556  	if (!init_attr)
aef9ec39 Roland Dreier     2005-11-02  557  		return -ENOMEM;
aef9ec39 Roland Dreier     2005-11-02  558  
561392d4 Steve Wise        2016-02-17  559  	/* queue_size + 1 for ib_drain_rq() */
1dc7b1f1 Christoph Hellwig 2015-11-13  560  	recv_cq = ib_alloc_cq(dev->dev, ch, target->queue_size + 1,
1dc7b1f1 Christoph Hellwig 2015-11-13  561  				ch->comp_vector, IB_POLL_SOFTIRQ);
73aa89ed Ishai Rabinovitz  2012-11-26  562  	if (IS_ERR(recv_cq)) {
73aa89ed Ishai Rabinovitz  2012-11-26  563  		ret = PTR_ERR(recv_cq);
da9d2f07 Roland Dreier     2010-02-24  564  		goto err;
aef9ec39 Roland Dreier     2005-11-02  565  	}
aef9ec39 Roland Dreier     2005-11-02  566  
1dc7b1f1 Christoph Hellwig 2015-11-13  567  	send_cq = ib_alloc_cq(dev->dev, ch, m * target->queue_size,
1dc7b1f1 Christoph Hellwig 2015-11-13  568  				ch->comp_vector, IB_POLL_DIRECT);
73aa89ed Ishai Rabinovitz  2012-11-26  569  	if (IS_ERR(send_cq)) {
73aa89ed Ishai Rabinovitz  2012-11-26  570  		ret = PTR_ERR(send_cq);
da9d2f07 Roland Dreier     2010-02-24  571  		goto err_recv_cq;
9c03dc9f Bart Van Assche   2010-02-02  572  	}
9c03dc9f Bart Van Assche   2010-02-02  573  
aef9ec39 Roland Dreier     2005-11-02  574  	init_attr->event_handler       = srp_qp_event;
5cfb1782 Bart Van Assche   2014-05-20  575  	init_attr->cap.max_send_wr     = m * target->queue_size;
7dad6b2e Bart Van Assche   2014-10-21  576  	init_attr->cap.max_recv_wr     = target->queue_size + 1;
aef9ec39 Roland Dreier     2005-11-02  577  	init_attr->cap.max_recv_sge    = 1;
aef9ec39 Roland Dreier     2005-11-02  578  	init_attr->cap.max_send_sge    = 1;
5cfb1782 Bart Van Assche   2014-05-20  579  	init_attr->sq_sig_type         = IB_SIGNAL_REQ_WR;
aef9ec39 Roland Dreier     2005-11-02  580  	init_attr->qp_type             = IB_QPT_RC;
73aa89ed Ishai Rabinovitz  2012-11-26  581  	init_attr->send_cq             = send_cq;
73aa89ed Ishai Rabinovitz  2012-11-26  582  	init_attr->recv_cq             = recv_cq;
aef9ec39 Roland Dreier     2005-11-02  583  
19f31343 Bart Van Assche   2018-01-22  584  	if (target->using_rdma_cm) {
19f31343 Bart Van Assche   2018-01-22 @585  		ret = rdma_create_qp(ch->rdma_cm.cm_id, dev->pd, init_attr);
19f31343 Bart Van Assche   2018-01-22  586  		qp = ch->rdma_cm.cm_id->qp;
19f31343 Bart Van Assche   2018-01-22  587  	} else {
62154b2e Bart Van Assche   2014-05-20  588  		qp = ib_create_qp(dev->pd, init_attr);
19f31343 Bart Van Assche   2018-01-22  589  		if (!IS_ERR(qp)) {
19f31343 Bart Van Assche   2018-01-22  590  			ret = srp_init_ib_qp(target, qp);
19f31343 Bart Van Assche   2018-01-22  591  			if (ret)
19f31343 Bart Van Assche   2018-01-22  592  				ib_destroy_qp(qp);
19f31343 Bart Van Assche   2018-01-22  593  		} else {
73aa89ed Ishai Rabinovitz  2012-11-26  594  			ret = PTR_ERR(qp);
19f31343 Bart Van Assche   2018-01-22  595  		}
19f31343 Bart Van Assche   2018-01-22  596  	}
19f31343 Bart Van Assche   2018-01-22  597  	if (ret) {
19f31343 Bart Van Assche   2018-01-22  598  		pr_err("QP creation failed for dev %s: %d\n",
19f31343 Bart Van Assche   2018-01-22  599  		       dev_name(&dev->dev->dev), ret);
da9d2f07 Roland Dreier     2010-02-24  600  		goto err_send_cq;
aef9ec39 Roland Dreier     2005-11-02  601  	}
aef9ec39 Roland Dreier     2005-11-02  602  
002f1567 Bart Van Assche   2015-08-10  603  	if (dev->use_fast_reg) {
5cfb1782 Bart Van Assche   2014-05-20  604  		fr_pool = srp_alloc_fr_pool(target);
5cfb1782 Bart Van Assche   2014-05-20  605  		if (IS_ERR(fr_pool)) {
5cfb1782 Bart Van Assche   2014-05-20  606  			ret = PTR_ERR(fr_pool);
5cfb1782 Bart Van Assche   2014-05-20  607  			shost_printk(KERN_WARNING, target->scsi_host, PFX
5cfb1782 Bart Van Assche   2014-05-20  608  				     "FR pool allocation failed (%d)\n", ret);
5cfb1782 Bart Van Assche   2014-05-20  609  			goto err_qp;
5cfb1782 Bart Van Assche   2014-05-20  610  		}
002f1567 Bart Van Assche   2015-08-10  611  	} else if (dev->use_fmr) {
d1b4289e Bart Van Assche   2014-05-20  612  		fmr_pool = srp_alloc_fmr_pool(target);
d1b4289e Bart Van Assche   2014-05-20  613  		if (IS_ERR(fmr_pool)) {
d1b4289e Bart Van Assche   2014-05-20  614  			ret = PTR_ERR(fmr_pool);
d1b4289e Bart Van Assche   2014-05-20  615  			shost_printk(KERN_WARNING, target->scsi_host, PFX
d1b4289e Bart Van Assche   2014-05-20  616  				     "FMR pool allocation failed (%d)\n", ret);
d1b4289e Bart Van Assche   2014-05-20  617  			goto err_qp;
d1b4289e Bart Van Assche   2014-05-20  618  		}
d1b4289e Bart Van Assche   2014-05-20  619  	}
d1b4289e Bart Van Assche   2014-05-20  620  
509c07bc Bart Van Assche   2014-10-30  621  	if (ch->qp)
9566b054 Bart Van Assche   2017-10-11  622  		srp_destroy_qp(ch);
509c07bc Bart Van Assche   2014-10-30  623  	if (ch->recv_cq)
1dc7b1f1 Christoph Hellwig 2015-11-13  624  		ib_free_cq(ch->recv_cq);
509c07bc Bart Van Assche   2014-10-30  625  	if (ch->send_cq)
1dc7b1f1 Christoph Hellwig 2015-11-13  626  		ib_free_cq(ch->send_cq);
73aa89ed Ishai Rabinovitz  2012-11-26  627  
509c07bc Bart Van Assche   2014-10-30  628  	ch->qp = qp;
509c07bc Bart Van Assche   2014-10-30  629  	ch->recv_cq = recv_cq;
509c07bc Bart Van Assche   2014-10-30  630  	ch->send_cq = send_cq;
73aa89ed Ishai Rabinovitz  2012-11-26  631  
7fbc67df Sagi Grimberg     2015-08-24  632  	if (dev->use_fast_reg) {
7fbc67df Sagi Grimberg     2015-08-24  633  		if (ch->fr_pool)
7fbc67df Sagi Grimberg     2015-08-24  634  			srp_destroy_fr_pool(ch->fr_pool);
7fbc67df Sagi Grimberg     2015-08-24  635  		ch->fr_pool = fr_pool;
7fbc67df Sagi Grimberg     2015-08-24  636  	} else if (dev->use_fmr) {
7fbc67df Sagi Grimberg     2015-08-24  637  		if (ch->fmr_pool)
7fbc67df Sagi Grimberg     2015-08-24  638  			ib_destroy_fmr_pool(ch->fmr_pool);
7fbc67df Sagi Grimberg     2015-08-24  639  		ch->fmr_pool = fmr_pool;
7fbc67df Sagi Grimberg     2015-08-24  640  	}
7fbc67df Sagi Grimberg     2015-08-24  641  
da9d2f07 Roland Dreier     2010-02-24  642  	kfree(init_attr);
da9d2f07 Roland Dreier     2010-02-24  643  	return 0;
da9d2f07 Roland Dreier     2010-02-24  644  
da9d2f07 Roland Dreier     2010-02-24  645  err_qp:
19f31343 Bart Van Assche   2018-01-22  646  	if (target->using_rdma_cm)
19f31343 Bart Van Assche   2018-01-22 @647  		rdma_destroy_qp(ch->rdma_cm.cm_id);
19f31343 Bart Van Assche   2018-01-22  648  	else
95c2ef50 Israel Rukshin    2017-05-11  649  		ib_destroy_qp(qp);
da9d2f07 Roland Dreier     2010-02-24  650  
da9d2f07 Roland Dreier     2010-02-24  651  err_send_cq:
1dc7b1f1 Christoph Hellwig 2015-11-13  652  	ib_free_cq(send_cq);
da9d2f07 Roland Dreier     2010-02-24  653  
da9d2f07 Roland Dreier     2010-02-24  654  err_recv_cq:
1dc7b1f1 Christoph Hellwig 2015-11-13  655  	ib_free_cq(recv_cq);
aef9ec39 Roland Dreier     2005-11-02  656  
da9d2f07 Roland Dreier     2010-02-24  657  err:
aef9ec39 Roland Dreier     2005-11-02  658  	kfree(init_attr);
aef9ec39 Roland Dreier     2005-11-02  659  	return ret;
aef9ec39 Roland Dreier     2005-11-02  660  }
aef9ec39 Roland Dreier     2005-11-02  661  

:::::: The code at line 585 was first introduced by commit
:::::: 19f313438c7754e6cc2bceddeebeaa5132e2e0a1 IB/srp: Add RDMA/CM support

:::::: TO: Bart Van Assche <bart.vanassche@wdc.com>
:::::: CC: Doug Ledford <dledford@redhat.com>

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 28716 bytes --]

Re: [PATCH v2] IB: make INFINIBAND_ADDR_TRANS configurable

[Joe Perches]

On Sat, 2018-04-14 at 15:34 +0000, Greg Thelen wrote:
> On Sat, Apr 14, 2018 at 8:13 AM Dennis Dalessandro <
> dennis.dalessandro@intel.com> wrote:
> 
> > On 4/13/2018 1:27 PM, Greg Thelen wrote:
> > > Allow INFINIBAND without INFINIBAND_ADDR_TRANS.
> > > 
> > > Signed-off-by: Greg Thelen <gthelen@google.com>
> > > Cc: Tarick Bedeir <tarick@google.com>
> > > Change-Id: I6fbbf8a432e467710fa65e4904b7d61880b914e5
> > Forgot to remove the Gerrit thing.
> > -Denny
> 
> Ack.  My bad.  Will repost.  Unfortunately checkpatch didn't notice.

Probably because Change-Id: is after a Signed-off-by: line

Re: [PATCH] IB: make INFINIBAND_ADDR_TRANS configurable

[oulijun]

在 2018/4/13 20:47, Bart Van Assche 写道:
> On Fri, 2018-04-13 at 00:06 -0700, Greg Thelen wrote:
>> Allow INFINIBAND without INFINIBAND_ADDR_TRANS.
>>
>> Signed-off-by: Greg Thelen <gthelen@google.com>
>> Cc: Tarick Bedeir <tarick@google.com>
>> ---
>>  drivers/infiniband/Kconfig | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/drivers/infiniband/Kconfig b/drivers/infiniband/Kconfig
>> index ee270e065ba9..f20a3977087c 100644
>> --- a/drivers/infiniband/Kconfig
>> +++ b/drivers/infiniband/Kconfig
>> @@ -61,7 +61,7 @@ config INFINIBAND_ON_DEMAND_PAGING
>>  	  pages on demand instead.
>>  
>>  config INFINIBAND_ADDR_TRANS
>> -	bool
>> +	bool "InfiniBand address translation"
>>  	depends on INFINIBAND
>>  	default y
> 
> Hello Greg,
> 
> Please change the short description from "InfiniBand address translation" (which
> is a misleading description) into "RDMA/CM". Please also add a help text that
> explains to people who are not RDMA experts what the RDMA/CM is.
> 
> Thanks,
> 
> Bart.
> 
Hi, Bart
  We need to select the INFINIBAND without INFINIBAND_ADDR_TRANS if use cm for others?

Thanks
Lijun Ou
> 
> 
> \x04�{.n�+�������+%��lzwm��b�맲��r��zX��\x1aݙ���\x17��ܨ}���Ơz�&j:+v���\a����zZ+��+zf���h���~����i���z�\x1e�w���?����&�)ߢ^[f
> 

Re: [PATCH] IB: make INFINIBAND_ADDR_TRANS configurable

[Christoph Hellwig]

On Fri, Apr 13, 2018 at 12:06:44AM -0700, Greg Thelen wrote:
> Allow INFINIBAND without INFINIBAND_ADDR_TRANS.

Why?  We are pushing everyone heavily to use RDMA/CM, so making it
optional seems rather counter-intuitive.

You'll also have to fix tons of ULPs to explicitly depend on
INFINIBAND_ADDR_TRANS, or make code conditional on it.

Re: [PATCH] IB: make INFINIBAND_ADDR_TRANS configurable

[Greg Thelen]

On Sun, Apr 15, 2018 at 5:06 AM Christoph Hellwig <hch@infradead.org> wrote:

> On Fri, Apr 13, 2018 at 12:06:44AM -0700, Greg Thelen wrote:
> > Allow INFINIBAND without INFINIBAND_ADDR_TRANS.

> Why?  We are pushing everyone heavily to use RDMA/CM, so making it
> optional seems rather counter-intuitive.

Bugs.  We don't currently use CM.  But fuzzy testing has been finding a
fair number CM bugs, so we were thinking of disabling it until we need it.

> You'll also have to fix tons of ULPs to explicitly depend on
> INFINIBAND_ADDR_TRANS, or make code conditional on it.

I think I've identified the set of options which use
INFINIBAND_ADDR_TRANS without a kconfig depends:
* CIFS_SMB_DIRECT
* INFINIBAND_SRPT
* NVME_RDMA
* NVME_TARGET_RDMA

I have patches for the above, but need to finish the commit logs.  Let me
know if they'll be nacked and I'll just patch my kernel and forget
upstreaming.

Re: [PATCH] IB: make INFINIBAND_ADDR_TRANS configurable

[Christoph Hellwig]

On Mon, Apr 16, 2018 at 04:02:12AM +0000, Greg Thelen wrote:
> I think I've identified the set of options which use
> INFINIBAND_ADDR_TRANS without a kconfig depends:
> * CIFS_SMB_DIRECT
> * INFINIBAND_SRPT
> * NVME_RDMA
> * NVME_TARGET_RDMA
> 
> I have patches for the above, but need to finish the commit logs.  Let me
> know if they'll be nacked and I'll just patch my kernel and forget
> upstreaming.

I think those are what is needed.  But better do a full cycle through
the build bot first..

Btw, should we rename the config symbol to something containing
RDMACM?  The current name seems rather confusing?

Re: [PATCH] IB: make INFINIBAND_ADDR_TRANS configurable

[Jason Gunthorpe]

On Mon, Apr 16, 2018 at 01:56:20AM -0700, Christoph Hellwig wrote:
> On Mon, Apr 16, 2018 at 04:02:12AM +0000, Greg Thelen wrote:
> > I think I've identified the set of options which use
> > INFINIBAND_ADDR_TRANS without a kconfig depends:
> > * CIFS_SMB_DIRECT
> > * INFINIBAND_SRPT
> > * NVME_RDMA
> > * NVME_TARGET_RDMA
> > 
> > I have patches for the above, but need to finish the commit logs.  Let me
> > know if they'll be nacked and I'll just patch my kernel and forget
> > upstreaming.
> 
> I think those are what is needed.  But better do a full cycle through
> the build bot first..

Yes, you have to post the full series and have the build bots say it
is clean, collect acks from the various maintainers and then we can
apply the whole thing through RDMA..

I n't want to apply kconfig changes piecemeal, we just get bug reports
from build bots that way.

Jason