TCM/LIO ib_srpt testing

TCM/LIO ib_srpt testing

[Vasilis Liaskovitis]

Hi,

I am trying to test the TCM/LIO ib_srpt target driver from master
branch of lio-core-2.6

In the past, using branch tcm_ib_srpt-38, I have successfully
initialized the srp target and initiator with the "manual steps" in:
http://linux-iscsi.org/wiki/SCSI_RDMA_Protocol#Manual_steps

However with master, the manual steps fail at:
root@server1:~# mkdir -p /sys/kernel/config/target/srpt/mlx4_0/mlx4_0/lun/lun_0
mkdir: cannot create directory
`/sys/kernel/config/target/srpt/mlx4_0': Invalid argument
Has the recommended ib_srpt initialization procedure changed recently?

The target and initiator are the same machine in this test.  This is
on Mellanox QDR (mlx4 backend driver)

When I tested the tcm_ib_srpt-38 branch of lio-core-2.6, the same
manual steps worked fine:

[ 3725.279283] <<<<<<<<<<<<<<<<<<<<<< BEGIN FABRIC API >>>>>>>>>>>>>>>>>>>>>>
[ 3725.279336] Initialized struct target_fabric_configfs:
ffff88061a3f5800 for srpt
[ 3725.279416] <<<<<<<<<<<<<<<<<<<<<< END FABRIC API >>>>>>>>>>>>>>>>>>>>>>
[ 3725.279539] TARGET_CORE[srpt]: Allocated Normal struct
se_portal_group for endpoint: 0x2c903000f5e7a, Portal Tag: 1
[ 3725.282984] Target_Core_ConfigFS: REGISTER -> group:
ffffffffa0273380 name: srpt
[ 3725.283071] Target_Core_ConfigFS: REGISTER -> Located fabric: srpt
[ 3725.284622] Target_Core_ConfigFS: REGISTER tfc_wwn_cit -> ffff88061a3f5b08
[ 3725.284677] Target_Core_ConfigFS: REGISTER -> Allocated Fabric: srpt
[ 3725.284727] Target_Core_ConfigFS: REGISTER -> Set tf->tf_fabric for srpt
[ 3725.286314] iblock/srpt: Adding to default ALUA Target Port Group:
alua/default_tg_pt_gp
[ 3725.286403] srpt_TPG[1]_LUN[0] - Activated srpt Logical Unit from CORE HBA: 1
[ 3725.377681] srpt_TPG[1] - Added ACL with TCQ Depth: 1 for srpt
Initiator Node: 0x0002c903000f5e7b
[ 3725.379182] srpt_TPG[1]_LUN[0->0] - Added RW ACL for
InitiatorNode: 0x0002c903000f5e7b

And the scsi device seemed to appear correctly on the initiator side:

[ 3758.027108] scsi host1: ib_srp: new target: id_ext 0002c903000f5e7a
ioc_guid 0002c903000f5e7a pkey ffff service_id 0002c903000f5e7a dgid
fe80:0000:0000:0000:0002:c903:000f:5e7b
[ 3758.027959] Received SRP_LOGIN_REQ with i_port_id
0x0:0x2c903000f5e7b, t_port_id 0x2c903000f5e7a:0x2c903000f5e7a and
it_iu_len 260 on port 1 (guid=0xfe80000000000000:0x2c903000f5e7b)
[ 3758.031561] Session : kernel thread ib_srpt_compl (PID 2396) started
[ 3758.031631] TARGET_CORE[srpt]: Registered fabric_sess_ptr: ffff88021aaad000
[ 3758.031898] scsi1 : SRP.T10:0002C903000F5E7A
[ 3758.032317] scsi 1:0:0:0: Direct-Access     LIO-ORG  IBLOCK
  4.0  PQ: 0 ANSI: 5
[ 3758.032751] sd 1:0:0:0: [sdc] 9762242560 512-byte logical blocks:
(4.99 TB/4.54 TiB)
[ 3758.033193] sd 1:0:0:0: [sdc] Write Protect is off
[ 3758.033245] sd 1:0:0:0: [sdc] Mode Sense: 2f 00 00 00
[ 3758.033389] sd 1:0:0:0: [sdc] Write cache: disabled, read cache:
enabled, doesn't support DPO or FUA
[ 3768.046220]  sdc: unknown partition table
[ 3768.046823] sd 1:0:0:0: [sdc] Attached SCSI disk

However, the tcm_ib_srpt-38 branch results in a  kernel panic during
I/O traffic (e.g. doing simple dd tests on exported disk)
I was hoping that the latest patches in master would fix this.

Does anyone have tips on initializing ib_srpt manually, with
lio-utils, or with rtsadmin?
.
Also, If there's another branch more suitable for the latest LIO
patches please lat me know.

thanks,

- Vasilis

Re: TCM/LIO ib_srpt testing

[Nicholas A. Bellinger]

On Tue, 2011-05-17 at 12:37 +0200, Vasilis Liaskovitis wrote:
> Hi,
> 
> I am trying to test the TCM/LIO ib_srpt target driver from master
> branch of lio-core-2.6
> 

Greetings Vasilis,

> In the past, using branch tcm_ib_srpt-38, I have successfully
> initialized the srp target and initiator with the "manual steps" in:
> http://linux-iscsi.org/wiki/SCSI_RDMA_Protocol#Manual_steps
> 
> However with master, the manual steps fail at:
> root@server1:~# mkdir -p /sys/kernel/config/target/srpt/mlx4_0/mlx4_0/lun/lun_0
> mkdir: cannot create directory
> `/sys/kernel/config/target/srpt/mlx4_0': Invalid argument
> Has the recommended ib_srpt initialization procedure changed recently?
> 

Yes.  In order for ib_srpt to properly work with rtslib+rtsadmin, the
control plane has been changed from Bart's original patch to reference 
HW IB Port GUID instead of the symbolic /sys/class/infiniband/ name

The example of this is available here:

http://www.linux-iscsi.org/wiki/SCSI_RDMA_Protocol/RTSadmin


> The target and initiator are the same machine in this test.  This is
> on Mellanox QDR (mlx4 backend driver)
> 
> When I tested the tcm_ib_srpt-38 branch of lio-core-2.6, the same
> manual steps worked fine:
> 
> [ 3725.279283] <<<<<<<<<<<<<<<<<<<<<< BEGIN FABRIC API >>>>>>>>>>>>>>>>>>>>>>
> [ 3725.279336] Initialized struct target_fabric_configfs:
> ffff88061a3f5800 for srpt
> [ 3725.279416] <<<<<<<<<<<<<<<<<<<<<< END FABRIC API >>>>>>>>>>>>>>>>>>>>>>
> [ 3725.279539] TARGET_CORE[srpt]: Allocated Normal struct
> se_portal_group for endpoint: 0x2c903000f5e7a, Portal Tag: 1
> [ 3725.282984] Target_Core_ConfigFS: REGISTER -> group:
> ffffffffa0273380 name: srpt
> [ 3725.283071] Target_Core_ConfigFS: REGISTER -> Located fabric: srpt
> [ 3725.284622] Target_Core_ConfigFS: REGISTER tfc_wwn_cit -> ffff88061a3f5b08
> [ 3725.284677] Target_Core_ConfigFS: REGISTER -> Allocated Fabric: srpt
> [ 3725.284727] Target_Core_ConfigFS: REGISTER -> Set tf->tf_fabric for srpt
> [ 3725.286314] iblock/srpt: Adding to default ALUA Target Port Group:
> alua/default_tg_pt_gp
> [ 3725.286403] srpt_TPG[1]_LUN[0] - Activated srpt Logical Unit from CORE HBA: 1
> [ 3725.377681] srpt_TPG[1] - Added ACL with TCQ Depth: 1 for srpt
> Initiator Node: 0x0002c903000f5e7b
> [ 3725.379182] srpt_TPG[1]_LUN[0->0] - Added RW ACL for
> InitiatorNode: 0x0002c903000f5e7b
> 
> And the scsi device seemed to appear correctly on the initiator side:
> 
> [ 3758.027108] scsi host1: ib_srp: new target: id_ext 0002c903000f5e7a
> ioc_guid 0002c903000f5e7a pkey ffff service_id 0002c903000f5e7a dgid
> fe80:0000:0000:0000:0002:c903:000f:5e7b
> [ 3758.027959] Received SRP_LOGIN_REQ with i_port_id
> 0x0:0x2c903000f5e7b, t_port_id 0x2c903000f5e7a:0x2c903000f5e7a and
> it_iu_len 260 on port 1 (guid=0xfe80000000000000:0x2c903000f5e7b)
> [ 3758.031561] Session : kernel thread ib_srpt_compl (PID 2396) started
> [ 3758.031631] TARGET_CORE[srpt]: Registered fabric_sess_ptr: ffff88021aaad000
> [ 3758.031898] scsi1 : SRP.T10:0002C903000F5E7A
> [ 3758.032317] scsi 1:0:0:0: Direct-Access     LIO-ORG  IBLOCK
>   4.0  PQ: 0 ANSI: 5
> [ 3758.032751] sd 1:0:0:0: [sdc] 9762242560 512-byte logical blocks:
> (4.99 TB/4.54 TiB)
> [ 3758.033193] sd 1:0:0:0: [sdc] Write Protect is off
> [ 3758.033245] sd 1:0:0:0: [sdc] Mode Sense: 2f 00 00 00
> [ 3758.033389] sd 1:0:0:0: [sdc] Write cache: disabled, read cache:
> enabled, doesn't support DPO or FUA
> [ 3768.046220]  sdc: unknown partition table
> [ 3768.046823] sd 1:0:0:0: [sdc] Attached SCSI disk
> 
> However, the tcm_ib_srpt-38 branch results in a  kernel panic during
> I/O traffic (e.g. doing simple dd tests on exported disk)
> I was hoping that the latest patches in master would fix this.
> 

Note that tcm_ib_srpt-38 is an out-of-date branch that is missing some
necessary target core fixes..

At this point all of the active fabric module development branches
(including the ones for ib_srpt) have been merged into branch lio-4.1
@ .39-rc7 code, and into branch lio-4.0 @ .38.3.  Please use one of
these for your testing with ib_srpt to get the latest drivers/target/
bugfixes currently in James'es queue for mainline.

> Does anyone have tips on initializing ib_srpt manually, with
> lio-utils, or with rtsadmin?

Btw, rtsadmin-v2 includes a default /var/target/fabric/ib_srpt.spec very
similar to what's in the above URL, and the IB HCA Port GUIDs will
automatically appear as creatable wwn= parameters under in the top level
fabric object /ib_srpt

--nab

Re: TCM/LIO ib_srpt testing

[Bart Van Assche]

On Tue, May 17, 2011 at 11:37 PM, Nicholas A. Bellinger
<nab@linux-iscsi.org> wrote:
> On Tue, 2011-05-17 at 12:37 +0200, Vasilis Liaskovitis wrote:
> > In the past, using branch tcm_ib_srpt-38, I have successfully
> > initialized the srp target and initiator with the "manual steps" in:
> > http://linux-iscsi.org/wiki/SCSI_RDMA_Protocol#Manual_steps
> >
> > However with master, the manual steps fail at:
> > root@server1:~# mkdir -p /sys/kernel/config/target/srpt/mlx4_0/mlx4_0/lun/lun_0
> > mkdir: cannot create directory
> > `/sys/kernel/config/target/srpt/mlx4_0': Invalid argument
> > Has the recommended ib_srpt initialization procedure changed recently?
>
> Yes.  In order for ib_srpt to properly work with rtslib+rtsadmin, the
> control plane has been changed from Bart's original patch to reference
> HW IB Port GUID instead of the symbolic /sys/class/infiniband/ name

Most users expect the original behavior. If you agree I can prepare a
new patch where the target name format can be selected via a kernel
module parameter and in which Christoph Hellwig's recent comments are
addressed too.

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: TCM/LIO ib_srpt testing

[Bart Van Assche]

On Tue, May 17, 2011 at 12:37 PM, Vasilis Liaskovitis
<vliaskov@gmail.com> wrote:
> I am trying to test the TCM/LIO ib_srpt target driver from master
> branch of lio-core-2.6
>
> However, the tcm_ib_srpt-38 branch results in a  kernel panic during
> I/O traffic (e.g. doing simple dd tests on exported disk)
> I was hoping that the latest patches in master would fix this.

I have never seen such behavior during any of the tests I ran. I'll
see whether I can reproduce that behavior with lio-4.1.

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: TCM/LIO ib_srpt testing

[Nicholas A. Bellinger]

On Wed, 2011-05-18 at 12:19 +0200, Bart Van Assche wrote:
> On Tue, May 17, 2011 at 11:37 PM, Nicholas A. Bellinger
> <nab@linux-iscsi.org> wrote:
> > On Tue, 2011-05-17 at 12:37 +0200, Vasilis Liaskovitis wrote:
> > > In the past, using branch tcm_ib_srpt-38, I have successfully
> > > initialized the srp target and initiator with the "manual steps" in:
> > > http://linux-iscsi.org/wiki/SCSI_RDMA_Protocol#Manual_steps
> > >
> > > However with master, the manual steps fail at:
> > > root@server1:~# mkdir -p /sys/kernel/config/target/srpt/mlx4_0/mlx4_0/lun/lun_0
> > > mkdir: cannot create directory
> > > `/sys/kernel/config/target/srpt/mlx4_0': Invalid argument
> > > Has the recommended ib_srpt initialization procedure changed recently?
> >
> > Yes.  In order for ib_srpt to properly work with rtslib+rtsadmin, the
> > control plane has been changed from Bart's original patch to reference
> > HW IB Port GUID instead of the symbolic /sys/class/infiniband/ name
> 
> Most users expect the original behavior. If you agree I can prepare a
> new patch where the target name format can be selected via a kernel
> module parameter and in which Christoph Hellwig's recent comments are
> addressed too.
> 

Since we don't have any real userspace to break with 
/sys/kernel/config/target/srpt/ at this point, we need to avoid multiple
configfs layouts in mainline without a really solid backwards compatible
reason for needing to do so.

I don't have a strong issue with this myself, and would even be happy to
support it in lio upstream and backports if it's really useful, but I
think for mainline we have to decide on a single configfs layout for
modern v2 userspace.

Thank you for your comments,

--nab