RE: [PATCH v2 13/17] IB/Verbs: Reform cma/ucma with management helpers

From: "Hefty, Sean" <sean.hefty@intel.com>
To: Michael Wang <yun.wang@profitbricks.com>,
	Roland Dreier <roland@kernel.org>,
	"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>
Cc: Hal Rosenstock <hal.rosenstock@gmail.com>,
	Tom Tucker <tom@opengridcomputing.com>,
	Steve Wise <swise@opengridcomputing.com>,
	Hoang-Nam Nguyen <hnguyen@de.ibm.com>,
	Christoph Raisch <raisch@de.ibm.com>,
	infinipath <infinipath@intel.com>, Eli Cohen <eli@mellanox.com>,
	"Latif, Faisal" <faisal.latif@intel.com>,
	Upinder Malhi <umalhi@cisco.com>,
	Trond Myklebust <trond.myklebust@primarydata.com>,
	"J. Bruce Fields" <bfields@fieldses.org>,
	"David S. Miller" <davem@davemloft.net>,
	"Weiny, Ira" <ira.weiny@intel.com>,
	PJ Waskiewicz <pj.waskiewicz@solidfire.com>,
	"Nikolova, Tatyana E" <tatyana.e.nikolova@intel.com>,
	Or Gerlitz <ogerlitz@mellanox.com>,
	Jack Morgenstein <jackm@dev.mellanox.co.il>,
	Haggai Eran <haggaie@mellanox.com>,
	Ilya Nelkenbaum <ilyan@mellanox.com>,
	Yann Droneaud <ydroneaud@opteya.com>,
	Bart Van Assche <bvanassche@acm.org>,
	Shachar Raindel <raindel@me>
Subject: RE: [PATCH v2 13/17] IB/Verbs: Reform cma/ucma with management helpers
Date: Wed, 8 Apr 2015 17:02:11 +0000	[thread overview]
Message-ID: <1828884A29C6694DAF28B7E6B8A82373A8FBF037@ORSMSX109.amr.corp.intel.com> (raw)
In-Reply-To: <5524F6BD.30105@profitbricks.com>

> On 04/07/2015 11:36 PM, Hefty, Sean wrote:
> >> diff --git a/drivers/infiniband/core/cma.c
> b/drivers/infiniband/core/cma.c
> >> index d8a8ea7..c23f483 100644
> >> --- a/drivers/infiniband/core/cma.c
> >> +++ b/drivers/infiniband/core/cma.c
> >> @@ -435,10 +435,10 @@ static int cma_resolve_ib_dev(struct
> rdma_id_private
> >> *id_priv)
> >>  	pkey = ntohs(addr->sib_pkey);
> >>
> >>  	list_for_each_entry(cur_dev, &dev_list, list) {
> >> -		if (rdma_node_get_transport(cur_dev->device->node_type) !=
> >> RDMA_TRANSPORT_IB)
> >> -			continue;
> >> -
> >>  		for (p = 1; p <= cur_dev->device->phys_port_cnt; ++p) {
> >> +			if (!rdma_ib_mgmt(cur_dev->device, p))
> >> +				continue;
> >
> > This check wants to be something like is_af_ib_supported().  Checking
> for IB transport may actually be better than checking for IB management.
> I don't know if IBoE/RoCE devices support AF_IB.
> 
> The wrapper make sense, but do we have the guarantee that IBoE port won't
> be used for AF_IB address? I just can't locate the place we filtered it
> out...

I can't think of a reason why IBoE wouldn't work with AF_IB, but I'm not sure if anyone has tested it.  The original check would have let IBoE through.  When I suggested checking for IB transport, I meant the actual transport protocol, which would have included both IB and IBoE.

> >> @@ -700,8 +700,7 @@ static int cma_ib_init_qp_attr(struct
> rdma_id_private
> >> *id_priv,
> >>  	int ret;
> >>  	u16 pkey;
> >>
> >> -	if (rdma_port_get_link_layer(id_priv->id.device, id_priv-
> >>> id.port_num) ==
> >> -	    IB_LINK_LAYER_INFINIBAND)
> >> +	if (rdma_transport_ib(id_priv->id.device, id_priv->id.port_num))
> >>  		pkey = ib_addr_get_pkey(dev_addr);
> >>  	else
> >>  		pkey = 0xffff;
> >
> > Check here should be against the link layer, not transport.
> 
> I guess the name confusing us again... what if use rdma_tech_ib() here?
> it's the only tech using IB link layers, others are all ETH.

Yes, that would work.

> >>  	id_priv->id.route.addr.dev_addr.dev_type =
> >> -		(rdma_port_get_link_layer(cma_dev->device, p) ==
> >> IB_LINK_LAYER_INFINIBAND) ?
> >> +		(rdma_transport_ib(cma_dev->device, p)) ?
> >>  		ARPHRD_INFINIBAND : ARPHRD_ETHER;
> >
> > This wants the link layer, or maybe use cap_ipoib.
> 
> Is this related with ipoib only?

ARPHDR_INFINIBAND is related to ipoib.  In your next update, maybe go with tech_ib.  I don't know the status of ipoib over iboe.