IB/iSER with Linux 3.0 and Debian: Lesson learned

IB/iSER with Linux 3.0 and Debian: Lesson learned

[Sebastian Riemer]

Hi list,

I've already sent this to the open-iscsi mailing list, but I guess
this is more relevant for linux-rdma.

Finally I've got IB/iSER running on Debian Squeeze with Linux kernel 3.0
smoothly.

The problem was that we did not have the suitable OFED for our kernel
and we did not use the open-iscsi from OFED. Kernel 3.0 is supported
since OFED-1.5.4 from 2011-12-05.

So, I've taken the 1.5.2-based stuff from Debian/Experimental and I've
updated it to 1.5.4 from OFA. Then, I've noticed that Debian doesn't
build "ib_iser" in the OFA kernel source and that they don't build the
open-iscsi kernel/user-space code - I made it do so.

The next problem was that open-iscsi kernel code in OFED-1.5.4 is for <=
2.6.32 based RedHat distributions. I had to port the source from 2.6.30
to 3.0 due to kernel API changes. OFA even forgot libiscsi_tcp.[ch] in
OFED-1.5.4. So, I had to import it from 2.6.30 mainline.
I did so, because we wanted to compare TCP and iSER speed over
InfiniBand. Our Solaris COMSTAR targets provide both.

After fixing the kernel, there was still a problem in the open-iscsi
2.0.869 user-space from OFED. Some sysfs magic has changed - so that the
iSCSI host number couldn't be found.

After fixing that, it worked for me.

Cheers,

Sebastian


--
Sebastian Riemer
Linux Kernel Developer

ProfitBricks GmbH
Greifswalder Str. 207
10405 Berlin, Germany

Tel.:  +49 - 30 - 51 64 09 20
Fax:   +49 - 30 - 51 64 09 22
Email: sebastian.riemer-EIkl63zCoXaH+58JC4qpiA@public.gmane.org
Web:   http://www.profitbricks.com/

Sitz der Gesellschaft: Berlin
Registergericht: Amtsgericht Charlottenburg, HRB 125506 B
Geschäftsführer: Andreas Gauger, Achim Weiss
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: IB/iSER with Linux 3.0 and Debian: Lesson learned

[Or Gerlitz]

On 12/19/2011 11:14 AM, Sebastian Riemer wrote:
> Finally I've got IB/iSER running on Debian Squeeze with Linux kernel 3.0 smoothly.
> The problem was that we did not have the suitable OFED for our kernel

Sebastian,

Beep, I'd like to better/understand the problem before looking on your 
struggle for solution...
I understand that your Debian system runs kernel 3.0 - however, you 
didn't say what version of the iscsi initiator utils is provided with 
that distro nor what were the problems to make it work/well with iser, 
could you elaborate on that?

Or.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: IB/iSER with Linux 3.0 and Debian: Lesson learned

[Sebastian Riemer]

2011/12/20 Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>:
>
> Beep, I'd like to better/understand the problem before looking on your
> struggle for solution...
> I understand that your Debian system runs kernel 3.0 - however, you didn't
> say what version of the iscsi initiator utils is provided with that distro
> nor what were the problems to make it work/well with iser, could you
> elaborate on that?
>
> Or.
>

Ah, O.K. - I wrote that on the open-iscsi list. Debian Squeeze (in
general 2.6.32 based) comes with open-iscsi 2.0.871.3-2squeeze1. We've
used that version together with the in-tree mainline kernel 3.0 OFA
kernel modules and Debian Squeeze OFED-1.4 user-space. But there were
lots of iSER connection aborts (and even log-outs) after only 5s
connection loss instead of 120s
node.session.timeo.replacement_timeout. The many connection losses
where also caused by the missing of a suitable OFED.

After installation of the OFA kernel modules (without open-iscsi
modules) from OFED-1.5.4 the kernel had oopses in ib_iser. Therefore,
the suitable open-iscsi code had to be found (in OFED). And due to the
fact that it didn't support 3.0 kernels it also had to be ported.
There where many ABI and API changes in mainline open-iscsi kernel
code between 2.6.30 and 3.0.

I've fixed the following kernel API changes in the open-iscsi code
from OFA kernel source from OFED-1.5.4:
- kfifo API >= 2.6.33
- scsi_host API >= 2.6.33
- scsi_host API >= 2.6.37

Before that I've added the code and compilation of libiscsi_tcp from 2.6.30.

After stress testing the storage on a test machine with that fixed
OFED + iSER all other machines on that IB switch had IB connection
losses. So, we decided to roll out OFED-1.5.4 with fixed open-iscsi
code to all machines in our data center. And this works very well,
now. General network performance also doubled up.

Btw.: We need such a new kernel because of some cool virtualization,
cgroups and performance features.

I wrote the mail in this mailing list in order to show that open-iscsi
in OFED-1.5.4 isn't suitable for 3.0 kernel, that libiscsi_tcp is
missing and that we at ProfitBricks have a good test case with our
IaaS Cloud Computing.

Btw.: I like the proposed approach for new OFED releases. Version
checks in the code between kernel and user-space (like DRBD does)
would be great.

Cheers,

Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: IB/iSER with Linux 3.0 and Debian: Lesson learned

[Or Gerlitz]

Sebastian Riemer wrote:
> Debian Squeeze (in general 2.6.32 based) comes with open-iscsi 
> 2.0.871.3-2squeeze1. We've used that version together with the in-tree 
> mainline kernel 3.0 OFA kernel modules and Debian Squeeze OFED-1.4 
> user-space. But there were lots of iSER connection aborts (and even 
> log-outs) after only 5s connection loss instead of 120s 
> node.session.timeo.replacement_timeout. 
> Btw.: We need such a new kernel because of some cool virtualization,
> cgroups and performance features.
>

Beep(2), so your system has distro which is based on kernel 2.6.32 and 
iscsi initiator tools version 2.0.871 and per your needs, you've booted 
it with kernel 3.0 .

At this point should you have stop and make sure that this combo works, 
iscsi wise (simpler to test iscsi/tcp... no need in rdma knowledge), did 
you do this validation?

If this combo isn't working, you have to update the iscsi tools to a 
version which supports your kernel.

If this combo is working iscsi/tcp wise but not ib_iser wise, its seems 
to be a bug, and I'd be happy to help with finding the root cause, etc.

But this way or another, OFA isn't an iscsi tools factory, nor have 
anyone that can/want to support iscsi tools, we (folks from the rdma 
vendors community that deal with iscsi) are working with the upstream 
iscsi maintainer to address iscsi issues. The fact that OFA ships iscsi 
code except for ib_iser/cxgb4i/etc modules is a bug, BTW, I'll act to 
change that.


Or.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: IB/iSER with Linux 3.0 and Debian: Lesson learned

[Sebastian Riemer]

[-- Attachment #1: Type: text/plain, Size: 2725 bytes --]

2011/12/20 Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>:
>
> Beep(2), so your system has distro which is based on kernel 2.6.32 and iscsi
> initiator tools version 2.0.871 and per your needs, you've booted it with
> kernel 3.0 .
>
> At this point should you have stop and make sure that this combo works,
> iscsi wise (simpler to test iscsi/tcp... no need in rdma knowledge), did you
> do this validation?

Yes, of cause I did - TCP worked very well. I've also updated from git
to latest 2.0.872 (latest change 2011-11-01) for testing. TCP always
worked and iSER was always unusable.

>
> If this combo isn't working, you have to update the iscsi tools to a version
> which supports your kernel.
>

Yes, I even took the max. 2.6.35 supported kernel stuff from
open-iscsi git and have changed the Makefile to make it compile
against my kernel. With that I ensured that open-iscsi kernel code and
user-space match. TCP O.K. - no iSER with that!

> If this combo is working iscsi/tcp wise but not ib_iser wise, its seems to
> be a bug, and I'd be happy to help with finding the root cause, etc.

Have you ever developed/tested the ib_iser module for/with > 2.6.30
kernels? I've seen that there were lots of changes in the whole
open-iscsi kernel stack between 2.6.30 and 2.6.32. The whole ABI has
changed in libiscsi. They added stuff e.g. at the first position in
enums. If ib_iser isn't aware of such changes lots of crap can happen.
...and happened to me while testing by the way.

>
> But this way or another, OFA isn't an iscsi tools factory, nor have anyone
> that can/want to support iscsi tools, we (folks from the rdma vendors
> community that deal with iscsi) are working with the upstream iscsi
> maintainer to address iscsi issues. The fact that OFA ships iscsi code
> except for ib_iser/cxgb4i/etc modules is a bug, BTW, I'll act to change
> that.
>

ib_iser has tight dependencies to open-iscsi code (see attached). In
my opinion an ib_iser developer should work that tight together with
the open-iscsi guys. They should inform you about all ABI and API
changes in libiscsi so that you can react on that. As an user of
IB/iSER it is really confusing that this isn't the case and that OFA
only provides 2.6.30 based open-iscsi stuff which only works for iSER
due to missing libiscsi_tcp in there. At least this could be fixed
easily.

Now I know why everybody has his own OFED - so do we now. E.g. the
QLogic stuff isn't even compilable without the QLogic OFED, because
they only put their patches in there. Luckily, we have only Mellanox
HCAs in our productive environment.

Would it help, if we provide our patches for open-iscsi and IB/iSER >
2.6.32 to bring that into mainline OFED?

Sebastian

[-- Attachment #2: open-iscsi.png --]
[-- Type: image/png, Size: 7835 bytes --]

Re: IB/iSER with Linux 3.0 and Debian: Lesson learned

[Joe Landman]

On 12/20/2011 03:33 PM, Sebastian Riemer wrote:

> Now I know why everybody has his own OFED - so do we now. E.g. the
> QLogic stuff isn't even compilable without the QLogic OFED, because
> they only put their patches in there. Luckily, we have only Mellanox
> HCAs in our productive environment.

Actually it is, but we've have to hack the install scripts pretty hard 
to ... er ... fix some things.  And even then, they don't install 
libipath rpms (for rpm based distros) which turns out to be ... quite 
important ... for RDMA.

We've generally started with the OFED drops and hacked the install 
scripts even for Ubuntu/Debian.  Yeah, would be cleaner to use the 
.debs, and I think Guy Coates has the infrastructure set up for this.

> Would it help, if we provide our patches for open-iscsi and IB/iSER>
> 2.6.32 to bring that into mainline OFED?

As Or notes, OFED is providing the kernel modules more than the iscsi 
code drop.  Would be better for all (cough cough) to push changes back 
to the iscsi initiator maintainer (Mike Christie I think).
>
> Sebastian


-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman-nyOC7EYE20mM0MU9lROt9DlRY1/6cnIP@public.gmane.org
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: IB/iSER with Linux 3.0 and Debian: Lesson learned

[Or Gerlitz]

Sebastian Riemer <sebastian.riemer-EIkl63zCoXaH+58JC4qpiA@public.gmane.org> wrote:
> 2011/12/20 Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>:

>> Beep(2), so your system has distro which is based on kernel 2.6.32 and iscsi
>> initiator tools version 2.0.871 and per your needs, you've booted it with kernel 3.0 .
>> At this point should you have stop and make sure that this combo works, iscsi wise
>> (simpler to test iscsi/tcp... no need in rdma knowledge), did you do this validation?

> I did - TCP worked very well. I've also updated from git to latest 2.0.872 (latest
> (change 2011-11-01) for testing. TCP always worked and iSER was always unusable.

news to me, see next

> Yes, I even took the max. 2.6.35 supported kernel stuff from
> open-iscsi git and have changed the Makefile to make it compile
> against my kernel. With that I ensured that open-iscsi kernel code and
> user-space match. TCP O.K. - no iSER with that!
>
>> If this combo is working iscsi/tcp wise but not ib_iser wise, its seems to
>> be a bug, and I'd be happy to help with finding the root cause, etc.
>
> Have you ever developed/tested the ib_iser module for/with > 2.6.30 kernels?

horses, please, stay at home, or at least run a little bit slower,
just for you - from 2 minutes
ago - iser works well with 3.2.0-rc5 (its say -dirty b/c its a
development system and the kernel has some patches, but not iser ones)
and iscsi-initiator-utils of 6.2.0.872-21.el6, I will try tomorrow
with upstream iscsi-initiator-utils and see if there's a problem
there.

[root@vsa1 ~]# uname -r
3.2.0-rc5-dirty

[root@vsa1 ~]# rpm -q iscsi-initiator-utils
iscsi-initiator-utils-6.2.0.872-21.el6.x86_64

[root@vsa1 ~]# iscsiadm -m node -T iqn.vsa1.ib -l
Logging in to [iface: default, target: iqn.vsa1.ib, portal: 192.168.20.19,3260]
Login to [iface: default, target: iqn.vsa1.ib, portal:
192.168.20.19,3260] successful.

[root@vsa1 ~]# iscsiadm -m session
iser: [4] 192.168.20.19:3260,1 iqn.vsa1.ib

[root@vsa1 ~]# iscsiadm -m session -P 3
iSCSI Transport Class version 2.0-870
version 2.0-872
Target: iqn.vsa1.ib
        Current Portal: 192.168.20.19:3260,1
        Persistent Portal: 192.168.20.19:3260,1
                **********
                Interface:
                **********
                Iface Name: default
                Iface Transport: iser
                Iface Initiatorname: iqn.1994-05.com.redhat:41e62a1ec85d
                Iface IPaddress: <empty>
                Iface HWaddress: <empty>
                Iface Netdev: <empty>
                SID: 4
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
                ************************
                Negotiated iSCSI params:
                ************************
                HeaderDigest: None
                DataDigest: None
                MaxRecvDataSegmentLength: 128
                MaxXmitDataSegmentLength: 8192
                FirstBurstLength: 65536
                MaxBurstLength: 262144
                ImmediateData: No
                InitialR2T: Yes
                MaxOutstandingR2T: 1
                ************************
                Attached SCSI devices:
                ************************
                Host Number: 7  State: running
                scsi7 Channel 00 Id 0 Lun: 0
                scsi7 Channel 00 Id 0 Lun: 1
                        Attached scsi disk sdb          State: running
[root@vsa1 ~]# iscsiadm -m node -T iqn.vsa1.ib
# BEGIN RECORD 2.0-872
node.name = iqn.vsa1.ib
node.tpgt = 1
node.startup = manual
iface.hwaddress = <empty>
iface.ipaddress = <empty>
iface.iscsi_ifacename = default
iface.net_ifacename = <empty>
iface.transport_name = iser
iface.initiatorname = <empty>
node.discovery_address = 192.168.20.19
node.discovery_port = 3260
node.discovery_type = send_targets
node.session.initial_cmdsn = 0
node.session.initial_login_retry_max = 8
node.session.xmit_thread_priority = -20
node.session.cmds_max = 128
node.session.queue_depth = 32
node.session.auth.authmethod = None
node.session.auth.username = <empty>
node.session.auth.password = <empty>
node.session.auth.username_in = <empty>
node.session.auth.password_in = <empty>
node.session.timeo.replacement_timeout = 120
node.session.err_timeo.abort_timeout = 15
node.session.err_timeo.lu_reset_timeout = 30
node.session.err_timeo.tgt_reset_timeout = 30
node.session.err_timeo.host_reset_timeout = 60
node.session.iscsi.FastAbort = Yes
node.session.iscsi.InitialR2T = No
node.session.iscsi.ImmediateData = Yes
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192
node.session.iscsi.DefaultTime2Retain = 0
node.session.iscsi.DefaultTime2Wait = 2
node.session.iscsi.MaxConnections = 1
node.session.iscsi.MaxOutstandingR2T = 1
node.session.iscsi.ERL = 0
node.conn[0].address = 192.168.20.19
node.conn[0].port = 3260
node.conn[0].startup = manual
node.conn[0].tcp.window_size = 524288
node.conn[0].tcp.type_of_service = 0
node.conn[0].timeo.logout_timeout = 15
node.conn[0].timeo.login_timeout = 15
node.conn[0].timeo.auth_timeout = 45
node.conn[0].timeo.noop_out_interval = 5
node.conn[0].timeo.noop_out_timeout = 5
node.conn[0].iscsi.MaxXmitDataSegmentLength = 0
node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144
node.conn[0].iscsi.HeaderDigest = None
node.conn[0].iscsi.IFMarker = No
node.conn[0].iscsi.OFMarker = No
# END RECORD

[root@vsa1 ~]# iscsiadm -m node -T iqn.vsa1.ib -u
Logging out of session [sid: 4, target: iqn.vsa1.ib, portal: 192.168.20.19,3260]
Logout of [sid: 4, target: iqn.vsa1.ib, portal: 192.168.20.19,3260] successful.

and this is the system log

Dec 20 23:36:47 vsa1 root: AAAAAAAAAA iSER TEST WITH 3.2-rc5
Dec 20 23:36:52 vsa1 kernel: iser: iser_connect:connecting to:
192.168.20.19, port 0xbc0c
Dec 20 23:36:52 vsa1 kernel: iser: iser_cma_handler:event 0 status 0
conn ffff88062429d2b8 id ffff880620193000
Dec 20 23:36:52 vsa1 kernel: iser: iser_cma_handler:event 2 status 0
conn ffff88062429d2b8 id ffff880620193000
Dec 20 23:36:52 vsa1 kernel: iser: iser_create_ib_conn_res:setting
conn ffff88062429d2b8 cma_id ffff880620193000: fmr_pool
ffff880613b61500 qp ffff8805fec70400
Dec 20 23:36:52 vsa1 kernel: iser: iser_cma_handler:event 9 status 0
conn ffff88062429d2b8 id ffff880620193000
Dec 20 23:36:52 vsa1 kernel: iser: iscsi_iser_ep_poll:ib conn
ffff88062429d2b8 rc = 1
Dec 20 23:36:52 vsa1 kernel: scsi5 : iSCSI Initiator over iSER, v.0.1
Dec 20 23:36:52 vsa1 kernel: iser: iscsi_iser_conn_bind:binding
iscsi/iser conn ffff8804f9af8358 ffff8804f9af85c8 to ib_conn
ffff88062429d2b8
Dec 20 23:36:52 vsa1 kernel: scsi 5:0:0:0: RAID              Mellanox
vsa              1    PQ: 0 ANSI: 5
Dec 20 23:36:52 vsa1 kernel: scsi 5:0:0:0: Attached scsi generic sg2 type 12
Dec 20 23:36:52 vsa1 kernel: scsi 5:0:0:1: Direct-Access     Mellanox
VIRTUAL-DISK     0001 PQ: 0 ANSI: 5
Dec 20 23:36:52 vsa1 kernel: sd 5:0:0:1: Attached scsi generic sg3 type 0
Dec 20 23:36:52 vsa1 kernel: sd 5:0:0:1: [sdb] 2147483648 512-byte
logical blocks: (1.09 TB/1.00 TiB)
Dec 20 23:36:52 vsa1 kernel: sd 5:0:0:1: [sdb] Write Protect is off
Dec 20 23:36:52 vsa1 kernel: sd 5:0:0:1: [sdb] Write cache: enabled,
read cache: enabled, doesn't support DPO or FUA
Dec 20 23:36:52 vsa1 kernel: sdb: unknown partition table
Dec 20 23:36:52 vsa1 kernel: sd 5:0:0:1: [sdb] Attached SCSI disk
Dec 20 23:36:52 vsa1 iscsid: Could not set session2 priority.
READ/WRITE throughout and latency could be affected.
Dec 20 23:36:52 vsa1 iscsid: Connection2:0 to [target: iqn.vsa1.ib,
portal: 192.168.20.19,3260] through [iface: default] is operational
now
Dec 20 23:36:54 vsa1 kernel: sd 5:0:0:1: [sdb] Synchronizing SCSI cache
Dec 20 23:36:55 vsa1 kernel: iser: iscsi_iser_ep_disconnect:ib conn
ffff88062429d2b8 state 2
Dec 20 23:36:55 vsa1 kernel: iser: iser_cma_handler:event 10 status 0
conn ffff88062429d2b8 id ffff880620193000
Dec 20 23:36:55 vsa1 kernel: iser: iser_free_ib_conn_res:freeing conn
ffff88062429d2b8 cma_id ffff880620193000 fmr pool ffff880613b61500 qp
ffff8805fec70400
Dec 20 23:36:55 vsa1 kernel: iser: iser_device_try_release:device
ffff880554ba6000 refcount 0
Dec 20 23:36:55 vsa1 iscsid: Connection2:0 to [target: iqn.vsa1.ib,
portal: 192.168.20.19,3260] through [iface: default] is shutdown.


> I've seen that there were lots of changes in the whole
> open-iscsi kernel stack between 2.6.30 and 2.6.32. The whole ABI has
> changed in libiscsi. They added stuff e.g. at the first position in
> enums. If ib_iser isn't aware of such changes lots of crap can happen.
> ...and happened to me while testing by the way.

I don't use ofed at all, work only with upstream or distro code.
AFAIK, the upstream kernel is functional with iser at all times,
again, I will do the validation with the iscsi  tools and if the
upstream iscsi tools aren't functional with the upstream iser code but
are functional with the upstream iscsi/tcp code, we will (the iscsi
maintainer and myself) fix that, and thanks for this possible heads
up.

>> But this way or another, OFA isn't an iscsi tools factory, nor have anyone
>> that can/want to support iscsi tools, we (folks from the rdma vendors
>> community that deal with iscsi) are working with the upstream iscsi
>> maintainer to address iscsi issues. The fact that OFA ships iscsi code
>> except for ib_iser/cxgb4i/etc modules is a bug, BTW, I'll act to change that.

> ib_iser has tight dependencies to open-iscsi code (see attached). In
> my opinion an ib_iser developer should work that tight together with
> the open-iscsi guys. They should inform you about all ABI and API
> changes in libiscsi so that you can react on that. As an user of
> IB/iSER it is really confusing that this isn't the case and that OFA
> only provides 2.6.30 based open-iscsi stuff which only works for iSER
> due to missing libiscsi_tcp in there. At least this could be fixed easily.

no way, OFA isn't iscsi factory, we can't support the iscsi kernel
modules except for iser, nor any of the iscsi user space tools - and
its a historic bug that someone with wrong ambitions added iscsi
modules/tools info the ofa stack. The OFA stack should be compatible
with the kernel/distro it is running on. As you can see in the
maintainers file, I act as the iser maintainer, and I do work closely
with the iscsi maintainer, maybe should work closer if indeed you
stepped on a problem with the upstream iscsi tools, as for the iscsi
tools provided with debian, I am not sure what was the problem, send
me tgz with the sources to my @mellanox address and I can try look on
that.


> Would it help, if we provide our patches for open-iscsi and IB/iSER >
> 2.6.32 to bring that into mainline OFED?

patches to ib/iser are always wellcome, send me and cc this list, for
open-iscsi, not to me.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: IB/iSER with Linux 3.0 and Debian: Lesson learned

[Sebastian Riemer]

>
>> Would it help, if we provide our patches for open-iscsi and IB/iSER>
>> 2.6.32 to bring that into mainline OFED?
>
> As Or notes, OFED is providing the kernel modules more than the iscsi code drop.  Would be better for all (cough cough) to push changes back to the iscsi initiator maintainer (Mike Christie I think).

No, I don't think so. Mike's open-iscsi works for many kernels and he
provides libiscsi. OFA decided to make open-iscsi and ib_iser 2.6.30
based in OFED-1.5.4, but they say that they also support 3.0 in OFED.
Now, they could provide two more open-iscsi and ib_iser versions
(>=2.6.33, >=2.6.37) or just patch the version they already have. API
changes are no magic and everyone can see what they do in the mainline
if the API changes.

Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: IB/iSER with Linux 3.0 and Debian: Lesson learned

[Sebastian Riemer]

2011/12/20 Or Gerlitz <or.gerlitz-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
>
> horses, please, stay at home, or at least run a little bit slower,
> just for you - from 2 minutes
> ago - iser works well with 3.2.0-rc5 (its say -dirty b/c its a
> development system and the kernel has some patches, but not iser ones)
> and iscsi-initiator-utils of 6.2.0.872-21.el6, I will try tomorrow
> with upstream iscsi-initiator-utils and see if there's a problem
> there.
>

O.K., sorry - I just wanted to know how this is developed. Even if the
kernel code matches 100%, how do I know which is the matching tested
user-space code for that?

>
> I don't use ofed at all, work only with upstream or distro code.
> AFAIK, the upstream kernel is functional with iser at all times,
> again, I will do the validation with the iscsi  tools and if the
> upstream iscsi tools aren't functional with the upstream iser code but
> are functional with the upstream iscsi/tcp code, we will (the iscsi
> maintainer and myself) fix that, and thanks for this possible heads
> up.
>

Thanks for that info. In OFED I thought I would have everything needed
and matching together. Due to the new kernel we have our own
distribution extends, packet repo, build server,... So, for OFED we're
the distribution.

>
> no way, OFA isn't iscsi factory, we can't support the iscsi kernel
> modules except for iser, nor any of the iscsi user space tools - and
> its a historic bug that someone with wrong ambitions added iscsi
> modules/tools info the ofa stack. The OFA stack should be compatible
> with the kernel/distro it is running on. As you can see in the
> maintainers file, I act as the iser maintainer, and I do work closely
> with the iscsi maintainer, maybe should work closer if indeed you
> stepped on a problem with the upstream iscsi tools, as for the iscsi
> tools provided with debian, I am not sure what was the problem, send
> me tgz with the sources to my @mellanox address and I can try look on
> that.
>

Perhaps, I really stepped on a rare case where this was broken. As
I've already asked: How do I find the matching, tested open-iscsi and
OFA user-space code for a mainline kernel?

Sorry again, I didn't want provoke. I just want to understand and make
things work.

Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: IB/iSER with Linux 3.0 and Debian: Lesson learned

[Or Gerlitz]

On 12/21/2011 10:01 AM, Sebastian Riemer wrote:
>> could you provide quick pointers / 1-2 examples  for such API/ABI changes
>> which aren't deployed in the upstream iser code?
>>

you wrote long emails, I'm asking for one concrete example for that enum 
crunching  of adding entries
not at the end, can you, please?

>
> Sorry, I've meant the OFED code and not the mainline. In the mainline
> kernel this always would be fixed at least to make it compile.

I want to double check I'm with you - so when you said that iser didn't 
work e.g "TCP worked very well. I've also updated from git to latest 
2.0.872 (latest change 2011-11-01) for testing. TCP always worked and 
iSER was always unusable." you actually wanted to say "iser from ofed" 
and not iser from this or that upstream kernel?



> If something is in the mainline It doesn't mean that it really works perfectly from there.
> Some drivers are better from out-of-tree. I thought that this is also the case with OFED.

as life, mainline isn't perfect, but it doesn't say that ofed is perfect 
nor that by any bit its better then mainline, you may know and if you 
don't here are the news: the ofa community has to decided to stop 
producing ofed in the way it was done over the years, namely from now 
(Jan 2012) and onward, ofed will be only backports provided from 
mainline, no additions, so this false betterness claim can't even be 
stated anymore. Now, even this "backporting only" new mode has to be 
defined - since for example, is the iscsi case... except for iser, ofed 
will not provide the iscsi modules nor tools, so its not clear/how 
trivial/for someone takes (say) iser from 3.2 and backport it to (say) 
3.6.35 in manner that it will be operable with 3.6.35 and unknown 
version of the tools.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: IB/iSER with Linux 3.0 and Debian: Lesson learned

[Sebastian Riemer]

> you wrote long emails, I'm asking for one concrete example for that enum
> crunching  of adding entries
> not at the end, can you, please?

I've meant e.g. the iscsi tasks in libiscsi.h between 2.6.30 and
2.6.32. But I've meant this for OFED and not the mainline kernel.

2.6.30:
enum {
         ISCSI_TASK_COMPLETED,
         ISCSI_TASK_PENDING,
         ISCSI_TASK_RUNNING,
};

2.6.32:
enum {
         ISCSI_TASK_FREE,
         ISCSI_TASK_COMPLETED,
         ISCSI_TASK_PENDING,
         ISCSI_TASK_RUNNING,
         ISCSI_TASK_ABRT_TMF,            /* aborted due to TMF */
         ISCSI_TASK_ABRT_SESS_RECOV,     /* aborted due to session recovery */
};

> I want to double check I'm with you - so when you said that iser didn't work
> e.g "TCP worked very well. I've also updated from git to latest 2.0.872
> (latest change 2011-11-01) for testing. TCP always worked and iSER was
> always unusable." you actually wanted to say "iser from ofed" and not iser
> from this or that upstream kernel?
>

I've tried both. The iser from OFED oopsed (because it is 2.6.30 based
- didn't match the 3.0 open-iscsi in-tree) and everything from
upstream kernel 3.0.4 was pretty unstable (mentioned connection aborts
after 5s). And I guess because of the OFED-1.4 user-space from Squeeze
the IB connection was that unstable. The OFED user-space must match
the kernel code of cause. Before I took over the kernel maintaining at
ProfitBricks only few knew about that problem in the company. So, I
thought making everything OFED-1.5.4 is the right approach of doing
that.

>
> as life, mainline isn't perfect, but it doesn't say that ofed is perfect nor
> that by any bit its better then mainline, you may know and if you don't here
> are the news: the ofa community has to decided to stop producing ofed in the
> way it was done over the years, namely from now (Jan 2012) and onward, ofed
> will be only backports provided from mainline, no additions, so this false
> betterness claim can't even be stated anymore. Now, even this "backporting
> only" new mode has to be defined - since for example, is the iscsi case...
> except for iser, ofed will not provide the iscsi modules nor tools, so its
> not clear/how trivial/for someone takes (say) iser from 3.2 and backport it
> to (say) 3.6.35 in manner that it will be operable with 3.6.35 and unknown
> version of the tools.
>

As I wrote, I like that new approach. If OFED-3.2 will match mainline
3.2 this would be great, but then you'll also have to provide the
open-iscsi user-space which you've used for testing in there.
Or/and can't you just provide a list of tools, OFA user-space etc.
which you've tested (e.g. like that BUILD_ID file in OFED)?
I really hope that this makes things better/easier for InfiniBand and
iSER users. I'm looking forward to test that.

My question was more like: "How was it tried to ensure to match kernel
and user-space code right now?" and I did not want to read the
developer's favorite "With the next release everything will be
better." ;-)

Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: IB/iSER with Linux 3.0 and Debian: Lesson learned

[Or Gerlitz]

I tested the upstream kernel iser against the upstream iscsi tools  from 
git://github.com/mikechristie/open-iscsi
(commit 4323e342d2c9fb8ed7233ce855001c189ec55b23), it works

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: IB/iSER with Linux 3.0 and Debian: Lesson learned

[Sebastian Riemer]

2011/12/21 Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>:
> I tested the upstream kernel iser against the upstream iscsi tools  from
> git://github.com/mikechristie/open-iscsi
> (commit 4323e342d2c9fb8ed7233ce855001c189ec55b23), it works
>

To bring this to an end: I believe you. Most likely I had that much
trouble because of the OFED-1.4 user-space which did not match the
kernel code.

As you don't answer my question how I can find out what's the matching
user-space for a given upstream kernel - I will just use what I have
now (should work like on RedHat) and will be looking forward to the
new OFED release approach.

Thanks for taking the time to make things clear to me.

Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

IB/iSER major problems with Linux 3.0 and Solaris targets

[Sebastian Riemer]

[-- Attachment #1: Type: text/plain, Size: 2768 bytes --]

Hi list, hi Or,

at the moment we use our kernel 3.0 ported OFA kernel modules from
OFED-1.5.4 to make iSER work together with our kernel 3.0 ported
open-iscsi 2.0.869 and Solaris 11 COMSTAR targets.

Now, we've got the problem that ib_iser from in there ignores its
"debug_level" parameter. /sys/module/ib_iser/parameters/debug_level
shows "0" but it spams the whole kernel log with debug messages. Seems
to be a code bug.

We've also tested a 3.0.15 mainline kernel with in-tree IB modules
together with the OFED-1.5.4 user-space and this has much better IPoIB
performance than the kernel stuff from OFED. So, we want to use them
instead, but there is the same problem with the iSER debug messages and
iSER doesn't work together with our Solaris 11 COMSTAR targets.

We've tested this with open-iscsi (2.0.872, commit
4323e342d2c9fb8ed7233ce855001c189ec55b23) user-space.
TCP is O.K. but iSER reports an error while login attempt:
"iscsiadm: initiator reported error (11 - iSCSI PDU timed out)"

The sent PDUs from iscsid debugging are the same, but there is an IO
page fault in the kernel log. I've attached the relevant part and the
iscsid log.

This looks interesting:
"iser: iser_drain_tx_cq:tx id ffff88402391f898 status 4 vend_err 57"

Or, could you please investigate/explain?

It is a pain that we need both: working iSER and IPoIB traffic with good
performance.

Cheers,

Sebastian


On 19/12/11 10:14, Sebastian Riemer wrote:
> Hi list,
>
> I've already sent this to the open-iscsi mailing list, but I guess
> this is more relevant for linux-rdma.
>
> Finally I've got IB/iSER running on Debian Squeeze with Linux kernel 3.0
> smoothly.
>
> The problem was that we did not have the suitable OFED for our kernel
> and we did not use the open-iscsi from OFED. Kernel 3.0 is supported
> since OFED-1.5.4 from 2011-12-05.
>
> So, I've taken the 1.5.2-based stuff from Debian/Experimental and I've
> updated it to 1.5.4 from OFA. Then, I've noticed that Debian doesn't
> build "ib_iser" in the OFA kernel source and that they don't build the
> open-iscsi kernel/user-space code - I made it do so.
>
> The next problem was that open-iscsi kernel code in OFED-1.5.4 is for <=
> 2.6.32 based RedHat distributions. I had to port the source from 2.6.30
> to 3.0 due to kernel API changes. OFA even forgot libiscsi_tcp.[ch] in
> OFED-1.5.4. So, I had to import it from 2.6.30 mainline.
> I did so, because we wanted to compare TCP and iSER speed over
> InfiniBand. Our Solaris COMSTAR targets provide both.
>
> After fixing the kernel, there was still a problem in the open-iscsi
> 2.0.869 user-space from OFED. Some sysfs magic has changed - so that the
> iSCSI host number couldn't be found.
>
> After fixing that, it worked for me.
>
> Cheers,
>
> Sebastian
>   


[-- Attachment #2: kernel.log --]
[-- Type: text/x-log, Size: 8655 bytes --]

Jan 11 12:53:25 pserver214 kernel: [  716.518372] SCSI subsystem initialized
Jan 11 12:53:25 pserver214 kernel: [  716.521146] Loading iSCSI transport class v2.0-870.
Jan 11 12:53:25 pserver214 kernel: [  716.528756] iscsi: registered transport (tcp)
Jan 11 12:53:30 pserver214 kernel: [  721.903544] iscsi: registered transport (iser)
Jan 11 12:54:46 pserver214 kernel: [  797.537439] iser: iser_connect:connecting to: 10.1.24.204, port 0xbc0c
Jan 11 12:54:46 pserver214 kernel: [  797.563158] iser: iser_cma_handler:event 0 status 0 conn ffff880807b17a80 id ffff880807594400
Jan 11 12:54:46 pserver214 kernel: [  797.566402] iser: iser_cma_handler:event 2 status 0 conn ffff880807b17a80 id ffff880807594400
Jan 11 12:54:46 pserver214 kernel: [  797.579704] iser: iser_create_ib_conn_res:setting conn ffff880807b17a80 cma_id ffff880807594400: fmr_pool ffff88082426b400 qp ffff8807ed22aa00
Jan 11 12:54:46 pserver214 kernel: [  797.586557] iser: iser_cma_handler:event 9 status 0 conn ffff880807b17a80 id ffff880807594400
Jan 11 12:54:46 pserver214 kernel: [  797.787932] iser: iscsi_iser_ep_poll:ib conn ffff880807b17a80 rc = 1
Jan 11 12:54:46 pserver214 kernel: [  797.788137] scsi0 : iSCSI Initiator over iSER, v.0.1
Jan 11 12:54:46 pserver214 kernel: [  797.794249] iser: iscsi_iser_conn_bind:binding iscsi/iser conn ffff8808058deab8 ffff8808058decc8 to ib_conn ffff880807b17a80
Jan 11 12:54:46 pserver214 kernel: [  797.794710] AMD-Vi: Event logged [IO_PAGE_FAULT device=03:00.0 domain=0x0013 address=0x0000000006488000 flags=0x0050]
Jan 11 12:54:46 pserver214 kernel: [  797.794919] AMD-Vi: Event logged [IO_PAGE_FAULT device=03:00.0 domain=0x0013 address=0x0000000006488200 flags=0x0050]
Jan 11 12:54:46 pserver214 kernel: [  797.794998] iser: iser_drain_tx_cq:tx id ffff88402391f898 status 4 vend_err 57
Jan 11 12:54:46 pserver214 kernel: [  797.795006]  connection1:0: detected conn error (1011)
Jan 11 12:54:46 pserver214 kernel: [  797.795338] AMD-Vi: Event logged [IO_PAGE_FAULT device=03:00.0 domain=0x0013 address=0x0000000006488100 flags=0x0050]
Jan 11 12:54:46 pserver214 kernel: [  797.795535] AMD-Vi: Event logged [IO_PAGE_FAULT device=03:00.0 domain=0x0013 address=0x0000000006488040 flags=0x0050]
Jan 11 12:54:46 pserver214 kernel: [  797.795730] AMD-Vi: Event logged [IO_PAGE_FAULT device=03:00.0 domain=0x0013 address=0x00000000064880c0 flags=0x0050]
Jan 11 12:54:46 pserver214 kernel: [  797.795925] AMD-Vi: Event logged [IO_PAGE_FAULT device=03:00.0 domain=0x0013 address=0x0000000006488080 flags=0x0050]
Jan 11 12:54:46 pserver214 kernel: [  797.796119] AMD-Vi: Event logged [IO_PAGE_FAULT device=03:00.0 domain=0x0013 address=0x0000000006488140 flags=0x0050]
Jan 11 12:54:46 pserver214 kernel: [  797.796314] AMD-Vi: Event logged [IO_PAGE_FAULT device=03:00.0 domain=0x0013 address=0x0000000006488180 flags=0x0050]
Jan 11 12:54:46 pserver214 kernel: [  797.796508] AMD-Vi: Event logged [IO_PAGE_FAULT device=03:00.0 domain=0x0013 address=0x00000000064881c0 flags=0x0050]
Jan 11 12:55:01 pserver214 kernel: [  812.811486] iser: iscsi_iser_ep_disconnect:ib conn ffff880807b17a80 state 4
Jan 11 12:55:01 pserver214 kernel: [  812.812093] iser: iser_cma_handler:event 10 status 0 conn ffff880807b17a80 id ffff880807594400
Jan 11 12:55:01 pserver214 kernel: [  812.812232] iser: iser_free_ib_conn_res:freeing conn ffff880807b17a80 cma_id ffff880807594400 fmr pool ffff88082426b400 qp ffff8807ed22aa00
Jan 11 12:55:01 pserver214 kernel: [  812.813821] iser: iser_device_try_release:device ffff8808059fc300 refcount 0
Jan 11 12:55:04 pserver214 kernel: [  815.064508] iser: iser_connect:connecting to: 10.1.24.204, port 0xbc0c
Jan 11 12:55:04 pserver214 kernel: [  815.082863] iser: iser_cma_handler:event 0 status 0 conn ffff8807fb8de280 id ffff880807595000
Jan 11 12:55:04 pserver214 kernel: [  815.086057] iser: iser_cma_handler:event 2 status 0 conn ffff8807fb8de280 id ffff880807595000
Jan 11 12:55:04 pserver214 kernel: [  815.099221] iser: iser_create_ib_conn_res:setting conn ffff8807fb8de280 cma_id ffff880807595000: fmr_pool ffff8807ed2f3f80 qp ffff8807ed228a00
Jan 11 12:55:04 pserver214 kernel: [  815.101435] iser: iser_cma_handler:event 9 status 0 conn ffff8807fb8de280 id ffff880807595000
Jan 11 12:55:04 pserver214 kernel: [  815.314980] iser: iscsi_iser_ep_poll:ib conn ffff8807fb8de280 rc = 1
Jan 11 12:55:04 pserver214 kernel: [  815.315113] iser: iscsi_iser_conn_bind:binding iscsi/iser conn ffff8808058deab8 ffff8808058decc8 to ib_conn ffff8807fb8de280
Jan 11 12:55:04 pserver214 kernel: [  815.315644] iser: iser_drain_tx_cq:tx id ffff88402391f898 status 4 vend_err 57
Jan 11 12:55:04 pserver214 kernel: [  815.315779]  connection1:0: detected conn error (1011)
Jan 11 12:55:19 pserver214 kernel: [  830.082349] iser: iscsi_iser_ep_disconnect:ib conn ffff8807fb8de280 state 4
Jan 11 12:55:19 pserver214 kernel: [  830.082928] iser: iser_cma_handler:event 10 status 0 conn ffff8807fb8de280 id ffff880807595000
Jan 11 12:55:19 pserver214 kernel: [  830.083079] iser: iser_free_ib_conn_res:freeing conn ffff8807fb8de280 cma_id ffff880807595000 fmr pool ffff8807ed2f3f80 qp ffff8807ed228a00
Jan 11 12:55:19 pserver214 kernel: [  830.084398] iser: iser_device_try_release:device ffff8807fc662000 refcount 0
Jan 11 12:55:21 pserver214 kernel: [  832.335256] iser: iser_connect:connecting to: 10.1.24.204, port 0xbc0c
Jan 11 12:55:21 pserver214 kernel: [  832.353035] iser: iser_cma_handler:event 0 status 0 conn ffff880805448280 id ffff880807596c00
Jan 11 12:55:21 pserver214 kernel: [  832.356099] iser: iser_cma_handler:event 2 status 0 conn ffff880805448280 id ffff880807596c00
Jan 11 12:55:21 pserver214 kernel: [  832.369360] iser: iser_create_ib_conn_res:setting conn ffff880805448280 cma_id ffff880807596c00: fmr_pool ffff8808062fe380 qp ffff8807ed229400
Jan 11 12:55:21 pserver214 kernel: [  832.372343] iser: iser_cma_handler:event 9 status 0 conn ffff880805448280 id ffff880807596c00
Jan 11 12:55:21 pserver214 kernel: [  832.585722] iser: iscsi_iser_ep_poll:ib conn ffff880805448280 rc = 1
Jan 11 12:55:21 pserver214 kernel: [  832.585856] iser: iscsi_iser_conn_bind:binding iscsi/iser conn ffff8808058deab8 ffff8808058decc8 to ib_conn ffff880805448280
Jan 11 12:55:21 pserver214 kernel: [  832.586375] iser: iser_drain_tx_cq:tx id ffff88402391f898 status 4 vend_err 57
Jan 11 12:55:21 pserver214 kernel: [  832.586510]  connection1:0: detected conn error (1011)
Jan 11 12:55:36 pserver214 kernel: [  847.353001] iser: iscsi_iser_ep_disconnect:ib conn ffff880805448280 state 4
Jan 11 12:55:36 pserver214 kernel: [  847.353665] iser: iser_cma_handler:event 10 status 0 conn ffff880805448280 id ffff880807596c00
Jan 11 12:55:36 pserver214 kernel: [  847.353807] iser: iser_free_ib_conn_res:freeing conn ffff880805448280 cma_id ffff880807596c00 fmr pool ffff8808062fe380 qp ffff8807ed229400
Jan 11 12:55:36 pserver214 kernel: [  847.355194] iser: iser_device_try_release:device ffff8807ed232240 refcount 0
Jan 11 12:55:38 pserver214 kernel: [  849.605959] iser: iser_connect:connecting to: 10.1.24.204, port 0xbc0c
Jan 11 12:55:38 pserver214 kernel: [  849.622975] iser: iser_cma_handler:event 0 status 0 conn ffff880806366280 id ffff880807591400
Jan 11 12:55:38 pserver214 kernel: [  849.625917] iser: iser_cma_handler:event 2 status 0 conn ffff880806366280 id ffff880807591400
Jan 11 12:55:38 pserver214 kernel: [  849.639061] iser: iser_create_ib_conn_res:setting conn ffff880806366280 cma_id ffff880807591400: fmr_pool ffff8807ed23f380 qp ffff8807ed22ae00
Jan 11 12:55:38 pserver214 kernel: [  849.641741] iser: iser_cma_handler:event 9 status 0 conn ffff880806366280 id ffff880807591400
Jan 11 12:55:38 pserver214 kernel: [  849.856434] iser: iscsi_iser_ep_poll:ib conn ffff880806366280 rc = 1
Jan 11 12:55:38 pserver214 kernel: [  849.856562] iser: iscsi_iser_conn_bind:binding iscsi/iser conn ffff8808058deab8 ffff8808058decc8 to ib_conn ffff880806366280
Jan 11 12:55:38 pserver214 kernel: [  849.857080] iser: iser_drain_tx_cq:tx id ffff88402391f898 status 4 vend_err 57
Jan 11 12:55:38 pserver214 kernel: [  849.857215]  connection1:0: detected conn error (1011)
Jan 11 12:55:53 pserver214 kernel: [  864.623753] iser: iscsi_iser_ep_disconnect:ib conn ffff880806366280 state 4
Jan 11 12:55:53 pserver214 kernel: [  864.624413] iser: iser_cma_handler:event 10 status 0 conn ffff880806366280 id ffff880807591400
Jan 11 12:55:53 pserver214 kernel: [  864.624558] iser: iser_free_ib_conn_res:freeing conn ffff880806366280 cma_id ffff880807591400 fmr pool ffff8807ed23f380 qp ffff8807ed22ae00
Jan 11 12:55:53 pserver214 kernel: [  864.625841] iser: iser_device_try_release:device ffff8807fc585a40 refcount 0

[-- Attachment #3: 2.0.872-iser.log --]
[-- Type: text/x-log, Size: 27580 bytes --]

iscsid: sysfs_init: sysfs_path='/sys'

iscsid: sysfs_attr_get_value: open '/module/scsi_transport_iscsi'/'version'

iscsid: sysfs_attr_get_value: new uncached attribute '/sys/module/scsi_transport_iscsi/version'

iscsid: sysfs_attr_get_value: add to cache '/sys/module/scsi_transport_iscsi/version'

iscsid: sysfs_attr_get_value: cache '/sys/module/scsi_transport_iscsi/version' with attribute value '2.0-870'

iscsid: transport class version 2.0-870. iscsid version 2.0-872
iscsid: in ctldev_open
iscsid: created NETLINK_ISCSI socket...
iscsid: InitiatorName=iqn.2010-03.com.profitbricks:cloud:pserver:pserver214
iscsid: InitiatorName=iqn.2010-03.com.profitbricks:cloud:pserver:pserver214
iscsid: InitiatorAlias=pserver214
iscsid: Max file limits 1024 1024

iscsid: found .svn

iscsid: found 10.1.23.200,3260

iscsid: Looking for config file /etc/iscsi/send_targets/10.1.23.200,3260

iscsid: Looking for config file /etc/iscsi/send_targets/10.1.23.200,3260 config st_config.
iscsid: Could not open /etc/iscsi/send_targets/10.1.23.200,3260/st_config: No such file or directory

iscsid: Could not read discovery record for 10.1.23.200:3260.
iscsid: found 10.1.23.206,3260

iscsid: Looking for config file /etc/iscsi/send_targets/10.1.23.206,3260

iscsid: Looking for config file /etc/iscsi/send_targets/10.1.23.206,3260 config st_config.
iscsid: Could not open /etc/iscsi/send_targets/10.1.23.206,3260/st_config: No such file or directory

iscsid: Could not read discovery record for 10.1.23.206:3260.
iscsid: found 10.1.23.202,3260

iscsid: Looking for config file /etc/iscsi/send_targets/10.1.23.202,3260

iscsid: Looking for config file /etc/iscsi/send_targets/10.1.23.202,3260 config st_config.
iscsid: Could not open /etc/iscsi/send_targets/10.1.23.202,3260/st_config: No such file or directory

iscsid: Could not read discovery record for 10.1.23.202:3260.
iscsid: Could not increase process priority: Success
iscsid: reaped pid 10598, reap_count now 0
iscsid: poll result 1
iscsid: in read_transports
iscsid: Adding new transport iser
iscsid: Matched transport iser

iscsid: sysfs_attr_get_value: open '/class/iscsi_transport/iser'/'handle'

iscsid: sysfs_attr_get_value: new uncached attribute '/sys/class/iscsi_transport/iser/handle'

iscsid: sysfs_attr_get_value: add to cache '/sys/class/iscsi_transport/iser/handle'

iscsid: sysfs_attr_get_value: cache '/sys/class/iscsi_transport/iser/handle' with attribute value '18446744072100522112'

iscsid: sysfs_attr_get_value: open '/class/iscsi_transport/iser'/'caps'

iscsid: sysfs_attr_get_value: new uncached attribute '/sys/class/iscsi_transport/iser/caps'

iscsid: sysfs_attr_get_value: add to cache '/sys/class/iscsi_transport/iser/caps'

iscsid: sysfs_attr_get_value: cache '/sys/class/iscsi_transport/iser/caps' with attribute value '0x9'

iscsid: Adding new transport tcp
iscsid: Matched transport tcp

iscsid: sysfs_attr_get_value: open '/class/iscsi_transport/tcp'/'handle'

iscsid: sysfs_attr_get_value: new uncached attribute '/sys/class/iscsi_transport/tcp/handle'

iscsid: sysfs_attr_get_value: add to cache '/sys/class/iscsi_transport/tcp/handle'

iscsid: sysfs_attr_get_value: cache '/sys/class/iscsi_transport/tcp/handle' with attribute value '18446744072100476864'

iscsid: sysfs_attr_get_value: open '/class/iscsi_transport/tcp'/'caps'

iscsid: sysfs_attr_get_value: new uncached attribute '/sys/class/iscsi_transport/tcp/caps'

iscsid: sysfs_attr_get_value: add to cache '/sys/class/iscsi_transport/tcp/caps'

iscsid: sysfs_attr_get_value: cache '/sys/class/iscsi_transport/tcp/caps' with attribute value '0x39'

iscsid: Allocted session 0x21ad380
iscsid: no authentication configured...
iscsid: resolved 10.1.24.204 to 10.1.24.204
iscsid: setting iface iser, dev ib1.bbbb, set ip , hw default, transport iser.

iscsid: get ev context 0x21b83d0
iscsid: in ktransport_ep_connect
iscsid: in __kipc_call
iscsid: in kwritev
iscsid: in nlpayload_read
iscsid: in nlpayload_read
iscsid: ktransport_ep_connect got handle 1
iscsid: sched conn context 0x21b83d0 event 2, tmo 0
iscsid: thread 0x21b83d0 schedule: delay 0 state 3
iscsid: Setting login timer 0x21b4578 timeout 15
iscsid: thread 0x21b4578 schedule: delay 60 state 3
iscsid: thread 021b4578 wait some more
iscsid: exec thread 021b83d0 callback
iscsid: put ev context 0x21b83d0
iscsid: in ktransport_ep_poll
iscsid: in __kipc_call
iscsid: in kwritev
iscsid: in nlpayload_read
iscsid: in nlpayload_read
iscsid: in kcreate_session
iscsid: in __kipc_call
iscsid: in kwritev
iscsid: in nlpayload_read
iscsid: expecting event 28, got 106, handling...
iscsid: in ctldev_handle
iscsid: in nl_read
iscsid: ctldev_handle got event type 106

iscsid: in nlpayload_read
iscsid: sysfs_attr_get_value: open '/class/scsi_host/host0'/'proc_name'

iscsid: sysfs_attr_get_value: new uncached attribute '/sys/class/scsi_host/host0/proc_name'

iscsid: sysfs_attr_get_value: add to cache '/sys/class/scsi_host/host0/proc_name'

iscsid: sysfs_attr_get_value: cache '/sys/class/scsi_host/host0/proc_name' with attribute value 'iscsi_iser'

iscsid: in read_transports
iscsid: Updating transport iser
iscsid: sysfs_attr_get_value: open '/class/iscsi_transport/iser'/'handle'

iscsid: sysfs_attr_get_value: new uncached attribute '/sys/class/iscsi_transport/iser/handle'

iscsid: sysfs_attr_get_value: add to cache '/sys/class/iscsi_transport/iser/handle'

iscsid: sysfs_attr_get_value: cache '/sys/class/iscsi_transport/iser/handle' with attribute value '18446744072100522112'

iscsid: sysfs_attr_get_value: open '/class/iscsi_transport/iser'/'caps'

iscsid: sysfs_attr_get_value: new uncached attribute '/sys/class/iscsi_transport/iser/caps'

iscsid: sysfs_attr_get_value: add to cache '/sys/class/iscsi_transport/iser/caps'

iscsid: sysfs_attr_get_value: cache '/sys/class/iscsi_transport/iser/caps' with attribute value '0x9'

iscsid: Updating transport tcp
iscsid: sysfs_attr_get_value: open '/class/iscsi_transport/tcp'/'handle'

iscsid: sysfs_attr_get_value: new uncached attribute '/sys/class/iscsi_transport/tcp/handle'

iscsid: sysfs_attr_get_value: add to cache '/sys/class/iscsi_transport/tcp/handle'

iscsid: sysfs_attr_get_value: cache '/sys/class/iscsi_transport/tcp/handle' with attribute value '18446744072100476864'

iscsid: sysfs_attr_get_value: open '/class/iscsi_transport/tcp'/'caps'

iscsid: sysfs_attr_get_value: new uncached attribute '/sys/class/iscsi_transport/tcp/caps'

iscsid: sysfs_attr_get_value: add to cache '/sys/class/iscsi_transport/tcp/caps'

iscsid: sysfs_attr_get_value: cache '/sys/class/iscsi_transport/tcp/caps' with attribute value '0x39'

iscsid: in nlpayload_read
iscsid: in nlpayload_read
iscsid: Could not set session1 priority. READ/WRITE throughout and latency could be affected.

iscsid: created new iSCSI session sid 1 host no 0
iscsid: in kcreate_conn
iscsid: in __kipc_call
iscsid: in kwritev
iscsid: in nlpayload_read
iscsid: in nlpayload_read
iscsid: created new iSCSI connection 1:0
iscsid: in kbind_conn
iscsid: in __kipc_call
iscsid: in kwritev
iscsid: in nlpayload_read
iscsid: in nlpayload_read
iscsid: bound iSCSI connection 1:0 to session 1
iscsid: sysfs_attr_get_value: open '/class/iscsi_connection/connection1:0'/'exp_statsn'

iscsid: sysfs_attr_get_value: new uncached attribute '/sys/class/iscsi_connection/connection1:0/exp_statsn'

iscsid: sysfs_attr_get_value: add to cache '/sys/class/iscsi_connection/connection1:0/exp_statsn'

iscsid: sysfs_attr_get_value: cache '/sys/class/iscsi_connection/connection1:0/exp_statsn' with attribute value '0'

iscsid: sending login PDU with current stage 1, next stage 3, transit 0x80, isid 0x00023d010000 exp_statsn 0
iscsid: >    InitiatorName=iqn.2010-03.com.profitbricks:cloud:pserver:pserver214
iscsid: >    InitiatorAlias=pserver214
iscsid: >    TargetName=iqn.2010-03.com.profitbricks:cloud:customers:storage200
iscsid: >    SessionType=Normal
iscsid: >    DefaultTime2Wait=2
iscsid: >    DefaultTime2Retain=0
iscsid: >    IFMarker=No
iscsid: >    OFMarker=No
iscsid: >    ErrorRecoveryLevel=0
iscsid: >    InitialR2T=No
iscsid: >    ImmediateData=Yes
iscsid: >    MaxBurstLength=16776192
iscsid: >    FirstBurstLength=262144
iscsid: >    MaxOutstandingR2T=1
iscsid: >    MaxConnections=1
iscsid: >    DataPDUInOrder=Yes
iscsid: >    DataSequenceInOrder=Yes
iscsid: >    InitiatorRecvDataSegmentLength=131072
iscsid: >    TargetRecvDataSegmentLength=8192
iscsid: >    RDMAExtensions=Yes
iscsid: in ksend_pdu_begin
iscsid: send PDU began for hdr 48 bytes and data 516 bytes
iscsid: in kwritev
iscsid: wrote 48 bytes of PDU header
iscsid: in kwritev
iscsid: wrote 516 bytes of PDU data
iscsid: in ksend_pdu_end
iscsid: in __kipc_call
iscsid: in kwritev
iscsid: in nlpayload_read
iscsid: in nlpayload_read
iscsid: send PDU finished for conn 1:0
iscsid: thread removed

iscsid: poll result 1
iscsid: in ctldev_handle
iscsid: in nl_read
iscsid: ctldev_handle got event type 102

iscsid: get ev context 0x21b83d0
iscsid: message real length is 72 bytes, recv_handle 0x21b8420
iscsid: in nlpayload_read
iscsid: sched conn context 0x21b83d0 event 3, tmo 0
iscsid: thread 0x21b83d0 schedule: delay 0 state 3
iscsid: exec thread 021b83d0 callback
iscsid: Kernel reported iSCSI connection 1:0 error (1011 - ISCSI_ERR_CONN_FAILED: iSCSI connection failed) state (2)
iscsid: put ev context 0x21b83d0
iscsid: ignoring conn error in login. let it timeout
iscsid: thread removed

iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 was scheduled at 50:60, curtime 110 q_forw 0x674190 &pend_list 0x674190
iscsid: thread 021b4578 now in actor_list
iscsid: exec thread 021b4578 callback
iscsid: iscsi_login_eh
iscsid: login failed ISCSI_CONN_STATE_IN_LOGIN/R_STAGE_NO_CHANGE 0
iscsid: re-opening session 1 (reopen_cnt 0)
iscsid: thread 021b4578 delete: state 3
iscsid: thread 021b45b0 delete: state 0
iscsid: in ktransport_ep_disconnect
iscsid: in __kipc_call
iscsid: in kwritev
iscsid: in nlpayload_read
iscsid: in nlpayload_read
iscsid: in kstop_conn
iscsid: in __kipc_call
iscsid: in kwritev
iscsid: in nlpayload_read
iscsid: in nlpayload_read
iscsid: connection 1:0 is stopped for recovery
iscsid: Waiting 2 seconds before trying to reconnect.

iscsid: Requeue reopen attempt in 2 secs

iscsid: thread 021b4578 delete: state 3
iscsid: thread 0x21b4578 schedule: delay 8 state 3
iscsid: thread removed

iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 was scheduled at 110:8, curtime 119 q_forw 0x674190 &pend_list 0x674190
iscsid: thread 021b4578 now in actor_list
iscsid: exec thread 021b4578 callback
iscsid: iscsi_login_eh
iscsid: login failed ISCSI_CONN_STATE_XPT_WAIT/R_STAGE_NO_CHANGE
iscsid: in ktransport_ep_disconnect
iscsid: get ev context 0x21b83d0
iscsid: in ktransport_ep_connect
iscsid: in __kipc_call
iscsid: in kwritev
iscsid: in nlpayload_read
iscsid: in nlpayload_read
iscsid: ktransport_ep_connect got handle 1
iscsid: sched conn context 0x21b83d0 event 2, tmo 0
iscsid: thread 0x21b83d0 schedule: delay 0 state 3
iscsid: Setting login timer 0x21b4578 timeout 15
iscsid: thread 0x21b4578 schedule: delay 60 state 3
iscsid: thread removed

iscsid: thread 021b83d0 removed from poll_list
iscsid: exec thread 021b83d0 callback
iscsid: put ev context 0x21b83d0
iscsid: in ktransport_ep_poll
iscsid: in __kipc_call
iscsid: in kwritev
iscsid: in nlpayload_read
iscsid: in nlpayload_read
iscsid: in kbind_conn
iscsid: in __kipc_call
iscsid: in kwritev
iscsid: in nlpayload_read
iscsid: in nlpayload_read
iscsid: bound iSCSI connection 1:0 to session 1
iscsid: sysfs_attr_get_value: open '/class/iscsi_connection/connection1:0'/'exp_statsn'

iscsid: sysfs_attr_get_value: new uncached attribute '/sys/class/iscsi_connection/connection1:0/exp_statsn'

iscsid: sysfs_attr_get_value: add to cache '/sys/class/iscsi_connection/connection1:0/exp_statsn'

iscsid: sysfs_attr_get_value: cache '/sys/class/iscsi_connection/connection1:0/exp_statsn' with attribute value '0'

iscsid: sending login PDU with current stage 1, next stage 3, transit 0x80, isid 0x00023d010000 exp_statsn 0
iscsid: >    InitiatorName=iqn.2010-03.com.profitbricks:cloud:pserver:pserver214
iscsid: >    InitiatorAlias=pserver214
iscsid: >    TargetName=iqn.2010-03.com.profitbricks:cloud:customers:storage200
iscsid: >    SessionType=Normal
iscsid: >    DefaultTime2Wait=2
iscsid: >    DefaultTime2Retain=0
iscsid: >    IFMarker=No
iscsid: >    OFMarker=No
iscsid: >    ErrorRecoveryLevel=0
iscsid: >    InitialR2T=No
iscsid: >    ImmediateData=Yes
iscsid: >    MaxBurstLength=16776192
iscsid: >    FirstBurstLength=262144
iscsid: >    MaxOutstandingR2T=1
iscsid: >    MaxConnections=1
iscsid: >    DataPDUInOrder=Yes
iscsid: >    DataSequenceInOrder=Yes
iscsid: >    InitiatorRecvDataSegmentLength=131072
iscsid: >    TargetRecvDataSegmentLength=8192
iscsid: >    RDMAExtensions=Yes
iscsid: in ksend_pdu_begin
iscsid: send PDU began for hdr 48 bytes and data 516 bytes
iscsid: in kwritev
iscsid: wrote 48 bytes of PDU header
iscsid: in kwritev
iscsid: wrote 516 bytes of PDU data
iscsid: in ksend_pdu_end
iscsid: in __kipc_call
iscsid: in kwritev
iscsid: in nlpayload_read
iscsid: in nlpayload_read
iscsid: send PDU finished for conn 1:0
iscsid: thread removed

iscsid: poll result 1
iscsid: in ctldev_handle
iscsid: in nl_read
iscsid: ctldev_handle got event type 102

iscsid: get ev context 0x21b83d0
iscsid: message real length is 72 bytes, recv_handle 0x21b8420
iscsid: in nlpayload_read
iscsid: sched conn context 0x21b83d0 event 3, tmo 0
iscsid: thread 0x21b83d0 schedule: delay 0 state 3
iscsid: exec thread 021b83d0 callback
iscsid: Kernel reported iSCSI connection 1:0 error (1011 - ISCSI_ERR_CONN_FAILED: iSCSI connection failed) state (2)
iscsid: put ev context 0x21b83d0
iscsid: ignoring conn error in login. let it timeout
iscsid: thread removed

iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 was scheduled at 119:60, curtime 179 q_forw 0x674190 &pend_list 0x674190
iscsid: thread 021b4578 now in actor_list
iscsid: exec thread 021b4578 callback
iscsid: iscsi_login_eh
iscsid: login failed ISCSI_CONN_STATE_IN_LOGIN/R_STAGE_NO_CHANGE 1
iscsid: re-opening session 1 (reopen_cnt 1)
iscsid: thread 021b4578 delete: state 3
iscsid: thread 021b45b0 delete: state 3
iscsid: in ktransport_ep_disconnect
iscsid: in __kipc_call
iscsid: in kwritev
iscsid: in nlpayload_read
iscsid: in nlpayload_read
iscsid: in kstop_conn
iscsid: in __kipc_call
iscsid: in kwritev
iscsid: in nlpayload_read
iscsid: in nlpayload_read
iscsid: connection 1:0 is stopped for recovery
iscsid: Waiting 2 seconds before trying to reconnect.

iscsid: Requeue reopen attempt in 2 secs

iscsid: thread 021b4578 delete: state 3
iscsid: thread 0x21b4578 schedule: delay 8 state 3
iscsid: thread removed

iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 was scheduled at 179:8, curtime 188 q_forw 0x674190 &pend_list 0x674190
iscsid: thread 021b4578 now in actor_list
iscsid: exec thread 021b4578 callback
iscsid: iscsi_login_eh
iscsid: login failed ISCSI_CONN_STATE_XPT_WAIT/R_STAGE_NO_CHANGE
iscsid: in ktransport_ep_disconnect
iscsid: get ev context 0x21b83d0
iscsid: in ktransport_ep_connect
iscsid: in __kipc_call
iscsid: in kwritev
iscsid: in nlpayload_read
iscsid: in nlpayload_read
iscsid: ktransport_ep_connect got handle 1
iscsid: sched conn context 0x21b83d0 event 2, tmo 0
iscsid: thread 0x21b83d0 schedule: delay 0 state 3
iscsid: Setting login timer 0x21b4578 timeout 15
iscsid: thread 0x21b4578 schedule: delay 60 state 3
iscsid: thread removed

iscsid: thread 021b83d0 removed from poll_list
iscsid: exec thread 021b83d0 callback
iscsid: put ev context 0x21b83d0
iscsid: in ktransport_ep_poll
iscsid: in __kipc_call
iscsid: in kwritev
iscsid: in nlpayload_read
iscsid: in nlpayload_read
iscsid: in kbind_conn
iscsid: in __kipc_call
iscsid: in kwritev
iscsid: in nlpayload_read
iscsid: in nlpayload_read
iscsid: bound iSCSI connection 1:0 to session 1
iscsid: sysfs_attr_get_value: open '/class/iscsi_connection/connection1:0'/'exp_statsn'

iscsid: sysfs_attr_get_value: new uncached attribute '/sys/class/iscsi_connection/connection1:0/exp_statsn'

iscsid: sysfs_attr_get_value: add to cache '/sys/class/iscsi_connection/connection1:0/exp_statsn'

iscsid: sysfs_attr_get_value: cache '/sys/class/iscsi_connection/connection1:0/exp_statsn' with attribute value '0'

iscsid: sending login PDU with current stage 1, next stage 3, transit 0x80, isid 0x00023d010000 exp_statsn 0
iscsid: >    InitiatorName=iqn.2010-03.com.profitbricks:cloud:pserver:pserver214
iscsid: >    InitiatorAlias=pserver214
iscsid: >    TargetName=iqn.2010-03.com.profitbricks:cloud:customers:storage200
iscsid: >    SessionType=Normal
iscsid: >    DefaultTime2Wait=2
iscsid: >    DefaultTime2Retain=0
iscsid: >    IFMarker=No
iscsid: >    OFMarker=No
iscsid: >    ErrorRecoveryLevel=0
iscsid: >    InitialR2T=No
iscsid: >    ImmediateData=Yes
iscsid: >    MaxBurstLength=16776192
iscsid: >    FirstBurstLength=262144
iscsid: >    MaxOutstandingR2T=1
iscsid: >    MaxConnections=1
iscsid: >    DataPDUInOrder=Yes
iscsid: >    DataSequenceInOrder=Yes
iscsid: >    InitiatorRecvDataSegmentLength=131072
iscsid: >    TargetRecvDataSegmentLength=8192
iscsid: >    RDMAExtensions=Yes
iscsid: in ksend_pdu_begin
iscsid: send PDU began for hdr 48 bytes and data 516 bytes
iscsid: in kwritev
iscsid: wrote 48 bytes of PDU header
iscsid: in kwritev
iscsid: wrote 516 bytes of PDU data
iscsid: in ksend_pdu_end
iscsid: in __kipc_call
iscsid: in kwritev
iscsid: in nlpayload_read
iscsid: in nlpayload_read
iscsid: send PDU finished for conn 1:0
iscsid: thread removed

iscsid: poll result 1
iscsid: in ctldev_handle
iscsid: in nl_read
iscsid: ctldev_handle got event type 102

iscsid: get ev context 0x21b83d0
iscsid: message real length is 72 bytes, recv_handle 0x21b8420
iscsid: in nlpayload_read
iscsid: sched conn context 0x21b83d0 event 3, tmo 0
iscsid: thread 0x21b83d0 schedule: delay 0 state 3
iscsid: exec thread 021b83d0 callback
iscsid: Kernel reported iSCSI connection 1:0 error (1011 - ISCSI_ERR_CONN_FAILED: iSCSI connection failed) state (2)
iscsid: put ev context 0x21b83d0
iscsid: ignoring conn error in login. let it timeout
iscsid: thread removed

iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 was scheduled at 188:60, curtime 248 q_forw 0x674190 &pend_list 0x674190
iscsid: thread 021b4578 now in actor_list
iscsid: exec thread 021b4578 callback
iscsid: iscsi_login_eh
iscsid: login failed ISCSI_CONN_STATE_IN_LOGIN/R_STAGE_NO_CHANGE 2
iscsid: re-opening session 1 (reopen_cnt 2)
iscsid: thread 021b4578 delete: state 3
iscsid: thread 021b45b0 delete: state 3
iscsid: in ktransport_ep_disconnect
iscsid: in __kipc_call
iscsid: in kwritev
iscsid: in nlpayload_read
iscsid: in nlpayload_read
iscsid: in kstop_conn
iscsid: in __kipc_call
iscsid: in kwritev
iscsid: in nlpayload_read
iscsid: in nlpayload_read
iscsid: connection 1:0 is stopped for recovery
iscsid: Waiting 2 seconds before trying to reconnect.

iscsid: Requeue reopen attempt in 2 secs

iscsid: thread 021b4578 delete: state 3
iscsid: thread 0x21b4578 schedule: delay 8 state 3
iscsid: thread removed

iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 was scheduled at 248:8, curtime 257 q_forw 0x674190 &pend_list 0x674190
iscsid: thread 021b4578 now in actor_list
iscsid: exec thread 021b4578 callback
iscsid: iscsi_login_eh
iscsid: login failed ISCSI_CONN_STATE_XPT_WAIT/R_STAGE_NO_CHANGE
iscsid: in ktransport_ep_disconnect
iscsid: get ev context 0x21b83d0
iscsid: in ktransport_ep_connect
iscsid: in __kipc_call
iscsid: in kwritev
iscsid: in nlpayload_read
iscsid: in nlpayload_read
iscsid: ktransport_ep_connect got handle 1
iscsid: sched conn context 0x21b83d0 event 2, tmo 0
iscsid: thread 0x21b83d0 schedule: delay 0 state 3
iscsid: Setting login timer 0x21b4578 timeout 15
iscsid: thread 0x21b4578 schedule: delay 60 state 3
iscsid: thread removed

iscsid: thread 021b83d0 removed from poll_list
iscsid: exec thread 021b83d0 callback
iscsid: put ev context 0x21b83d0
iscsid: in ktransport_ep_poll
iscsid: in __kipc_call
iscsid: in kwritev
iscsid: in nlpayload_read
iscsid: in nlpayload_read
iscsid: in kbind_conn
iscsid: in __kipc_call
iscsid: in kwritev
iscsid: in nlpayload_read
iscsid: in nlpayload_read
iscsid: bound iSCSI connection 1:0 to session 1
iscsid: sysfs_attr_get_value: open '/class/iscsi_connection/connection1:0'/'exp_statsn'

iscsid: sysfs_attr_get_value: new uncached attribute '/sys/class/iscsi_connection/connection1:0/exp_statsn'

iscsid: sysfs_attr_get_value: add to cache '/sys/class/iscsi_connection/connection1:0/exp_statsn'

iscsid: sysfs_attr_get_value: cache '/sys/class/iscsi_connection/connection1:0/exp_statsn' with attribute value '0'

iscsid: sending login PDU with current stage 1, next stage 3, transit 0x80, isid 0x00023d010000 exp_statsn 0
iscsid: >    InitiatorName=iqn.2010-03.com.profitbricks:cloud:pserver:pserver214
iscsid: >    InitiatorAlias=pserver214
iscsid: >    TargetName=iqn.2010-03.com.profitbricks:cloud:customers:storage200
iscsid: >    SessionType=Normal
iscsid: >    DefaultTime2Wait=2
iscsid: >    DefaultTime2Retain=0
iscsid: >    IFMarker=No
iscsid: >    OFMarker=No
iscsid: >    ErrorRecoveryLevel=0
iscsid: >    InitialR2T=No
iscsid: >    ImmediateData=Yes
iscsid: >    MaxBurstLength=16776192
iscsid: >    FirstBurstLength=262144
iscsid: >    MaxOutstandingR2T=1
iscsid: >    MaxConnections=1
iscsid: >    DataPDUInOrder=Yes
iscsid: >    DataSequenceInOrder=Yes
iscsid: >    InitiatorRecvDataSegmentLength=131072
iscsid: >    TargetRecvDataSegmentLength=8192
iscsid: >    RDMAExtensions=Yes
iscsid: in ksend_pdu_begin
iscsid: send PDU began for hdr 48 bytes and data 516 bytes
iscsid: in kwritev
iscsid: wrote 48 bytes of PDU header
iscsid: in kwritev
iscsid: wrote 516 bytes of PDU data
iscsid: in ksend_pdu_end
iscsid: in __kipc_call
iscsid: in kwritev
iscsid: in nlpayload_read
iscsid: in nlpayload_read
iscsid: send PDU finished for conn 1:0
iscsid: thread removed

iscsid: poll result 1
iscsid: in ctldev_handle
iscsid: in nl_read
iscsid: ctldev_handle got event type 102

iscsid: get ev context 0x21b83d0
iscsid: message real length is 72 bytes, recv_handle 0x21b8420
iscsid: in nlpayload_read
iscsid: sched conn context 0x21b83d0 event 3, tmo 0
iscsid: thread 0x21b83d0 schedule: delay 0 state 3
iscsid: exec thread 021b83d0 callback
iscsid: Kernel reported iSCSI connection 1:0 error (1011 - ISCSI_ERR_CONN_FAILED: iSCSI connection failed) state (2)
iscsid: put ev context 0x21b83d0
iscsid: ignoring conn error in login. let it timeout
iscsid: thread removed

iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 wait some more
iscsid: thread 021b4578 was scheduled at 257:60, curtime 317 q_forw 0x674190 &pend_list 0x674190
iscsid: thread 021b4578 now in actor_list
iscsid: exec thread 021b4578 callback
iscsid: iscsi_login_eh
iscsid: login failed ISCSI_CONN_STATE_IN_LOGIN/R_STAGE_NO_CHANGE 3
iscsid: Giving up on initial login attempt after 60 seconds.

iscsid: disconnect conn
iscsid: in ktransport_ep_disconnect
iscsid: in __kipc_call
iscsid: in kwritev
iscsid: in nlpayload_read
iscsid: in nlpayload_read
iscsid: stop conn (conn state 2)
iscsid: in kstop_conn
iscsid: in __kipc_call
iscsid: in kwritev
iscsid: in nlpayload_read
iscsid: in nlpayload_read
iscsid: kdestroy conn
iscsid: in kdestroy_conn
iscsid: in __kipc_call
iscsid: in kwritev
iscsid: in nlpayload_read
iscsid: in nlpayload_read
iscsid: kdestroy session 1
iscsid: in kdestroy_session
iscsid: in __kipc_call
iscsid: in kwritev
iscsid: in nlpayload_read
iscsid: expecting event 12, got 105, handling...
iscsid: in ctldev_handle
iscsid: in nl_read
iscsid: ctldev_handle got event type 105

iscsid: get ev context 0x21b83d0
iscsid: message real length is 72 bytes, recv_handle 0x21b8420
iscsid: in nlpayload_read
iscsid: sched conn context 0x21b83d0 event 5, tmo 0
iscsid: thread 0x21b83d0 schedule: delay 0 state 3
iscsid: in nlpayload_read
iscsid: expecting event 12, got 104, handling...
iscsid: in ctldev_handle
iscsid: in nl_read
iscsid: ctldev_handle got event type 104

iscsid: in nlpayload_read
iscsid: session destroyed sid 1 host no 0
iscsid: in nlpayload_read
iscsid: in nlpayload_read
iscsid: Connection1:0 to [target: iqn.2010-03.com.profitbricks:cloud:customers:storage200, portal: 10.1.24.204,3260] through [iface: iser] is shutdown.
iscsid: mgmt_ipc_write_rsp: rsp to fd 5
iscsid: thread 021b4578 delete: state 3
iscsid: thread 021b45b0 delete: state 3
iscsid: destroying session

iscsid: thread 021b83d0 delete: state 4
iscsid: deleting a scheduled/waiting thread!
iscsid: put ev context 0x21b83d0
iscsid: Releasing session 0x21ad380
iscsid: thread removed

iscsid: pid 10597 caught signal 15
iscsid: iscsid shutting down.

Re: IB/iSER major problems with Linux 3.0 and Solaris targets

[Or Gerlitz]

Sebastian Riemer <sebastian.riemer-EIkl63zCoXaH+58JC4qpiA@public.gmane.org> wrote:
> [...] we've also tested a 3.0.15 mainline kernel with in-tree IB modules
> together with the OFED-1.5.4 user-space and this has much better IPoIB
> performance than the kernel stuff from OFED. So, we want to use them
> instead, but there is the same problem with the iSER debug messages and
> iSER doesn't work together with our Solaris 11 COMSTAR targets.

On the full part of the glass, its nice to hear the upstream has good
IPoIB performance, on the other part, I'll give 3.0.15 a try tomorrow,
however, the error you're getting "iser_drain_tx_cq:tx id
ffff88402391f898 status 4 vend_err 57" means that iser got local
protection error (=4) on the first buffer we used with IB (the
connection handshake buffers belong to the IB CM, not to the ULP)
which is the login request. Sounds like something is broken maybe dma
mapping wise, for this reason I think its likely that the problem
might not hit me on my testbed (but I will try, sure).

What environment you're running in - is that Dom0 for some hypervisor?
if yes, is it KVM which means plain Linux or Xen? it would help if you
send me your (compressed, please) .config  - send to my @mellanox
email. Also, please run with iser debug logs open, set debug_level=2
(lets discuss the extra verbosity later, please) and also the session
debug param for the libiscsi module. Is there a chance you can build
the latest kernel 3.2 for your environment and run with it once to see
if the problem happens there as well?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: IB/iSER major problems with Linux 3.0 and Solaris targets

[Or Gerlitz]

On 1/12/2012 11:23 AM, Sebastian Riemer wrote:
> We are running iSER directly on the host. KVM is compiled in but there 
> aren't any VMs on our iSER test server. It is a diskless SuperMicro 
> server with NFS root. On productive servers we have a live-image and 
> KVM uses the iSER driven block devices for storage. This is the IB HCA 
> (mlx4): Mellanox MT26428 [ConnectX IB QDR, PCIe 2.0 5GT/s] We've 
> updated the firmware lately on all servers. How can I find out the 
> firmware version? With tvflash or mstflint? 

If you  have build the kernel IB user space support (uverbs) and the IB 
libs, do "ibv_devinfo" if not, just ossi "cat 
/sys/class/infiniband/mlx4_0/*" and send the output. To be clear, iser 
does work for you on the productive servers but not on this server?

> The storage has the same IB HCA, and they are connected via a switch. 
> I'll ask someone of the SysOps which one it is and if they have the 
> latest firmware on it. Perhaps this could be the problem.

As its local protection error on TX, I don't see how this could relate 
to the target node.


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: IB/iSER major problems with Linux 3.0 and Solaris targets

[Sebastian Riemer]

On 12/01/12 10:29, Or Gerlitz wrote:
> If you  have build the kernel IB user space support (uverbs) and the
> IB libs, do "ibv_devinfo" if not, just ossi "cat
> /sys/class/infiniband/mlx4_0/*" and send the output. To be clear, iser
> does work for you on the productive servers but not on this server?

Yes, we've got consistent OFED-1.5.4 user-space. "ibv_devinfo" reports a
"mismatch between the kernel and the userspace libraries" - "kernel does
not support XRC.". "ibverbs-driver-mlx4" is at version
1.0.1-1.20.g6771d22 and "libibverbs" is at version 1.1.4-1.24.gb89d4d7.

But O.K., the other method shows firmware version "2.9.1000".

iSER only works on productive servers, because we use the OFA kernel
modules from OFED for them at the moment (with 3.0 ported *iscsi*
drivers). But there the IPoIB traffic is too slow for us.
We connect customer VMs with IPv6 between different servers via IB.

And yes, we could also test kernel 3.2 on our iSER test server.

Regards,
Sebastian


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: IB/iSER major problems with Linux 3.0 and Solaris targets

[Sebastian Riemer]

On 12/01/12 11:16, Sebastian Riemer wrote:
> On 12/01/12 10:29, Or Gerlitz wrote:
>   
>> If you  have build the kernel IB user space support (uverbs) and the
>> IB libs, do "ibv_devinfo" if not, just ossi "cat
>> /sys/class/infiniband/mlx4_0/*" and send the output. To be clear, iser
>> does work for you on the productive servers but not on this server?
>>     
> Yes, we've got consistent OFED-1.5.4 user-space. "ibv_devinfo" reports a
> "mismatch between the kernel and the userspace libraries" - "kernel does
> not support XRC.". "ibverbs-driver-mlx4" is at version
> 1.0.1-1.20.g6771d22 and "libibverbs" is at version 1.1.4-1.24.gb89d4d7.
>
> But O.K., the other method shows firmware version "2.9.1000".
>
>   

I've found out that we have two single port MHQH19B-XTR InfiniBand HCAs.

lspci output:
03:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0
5GT/s - IB QDR / 10GigE] (rev b0)
04:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0
5GT/s - IB QDR / 10GigE] (rev b0)

The first one is ib1. And the second is ib0.
/sys/devices/pci0000:00/0000:00:0c.0/0000:03:00.0/net/ib1
/sys/devices/pci0000:00/0000:00:0b.0/0000:04:00.0/net/ib0

The iSER traffic is on ib1 (the HCA which reported the error) and ib0 is
for IPoIB traffic. I don't know if the mlx4 driver has a problem with
that hardware config.

Here is the requested data:
mlx4_0:
board_id       MT_0D90110009
fw_ver         2.9.1000
hca_type       MT26428
hw_rev         b0
node_desc      pserver214 HCA-1 (mlx4_0 - MT26428)
node_guid      0002:c903:000f:5f76
node_type      1: CA
sys_image_guid 0002:c903:000f:5f79
uevent         NAME=mlx4_0

mlx4_1:
board_id       MT_0D90110009
fw_ver         2.9.1000
hca_type       MT26428
hw_rev         b0
node_desc      pserver214 HCA-2 (mlx4_1 - MT26428)
node_guid      0002:c903:000f:5f26
node_type      1: CA
sys_image_guid 0002:c903:000f:5f29
uevent         NAME=mlx4_1

Both are connected to the storage but in different subnets and without
multipathing.

How do I find out if ib1 is on mlx4_1 or mlx4_0?

Cheers,
Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: IB/iSER major problems with Linux 3.0 and Solaris targets

[Or Gerlitz]

On 1/12/2012 5:18 PM, Sebastian Riemer wrote:
> How do I find out if ib1 is on mlx4_1 or mlx4_0

you do "ip addr show" and compare with 
/sys/class/infiniband/mlx4_*/ports/1/gid/0

you didn't send the kernel logs from the failure after opening the iser  
(debug_level=2) and libiscsi (debug_libiscsi_session=1 
debug_libiscsi_conn=1) debug prints

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: IB/iSER major problems with Linux 3.0 and Solaris targets

[Or Gerlitz]

On 1/11/2012 10:09 PM, Or Gerlitz wrote:
> [...] I'll give 3.0.15 a try tomorrow, however, the error you're 
> getting "iser_drain_tx_cq:tx id ffff88402391f898 status 4 vend_err 57" 
> means that iser got local protection error (=4) on the first buffer we 
> used with IB (the connection handshake buffers belong to the IB CM, 
> not to the ULP) which is the login request. Sounds like something is 
> broken maybe dma mapping wise, for this reason I think its likely that 
> the problem might not hit me on my testbed [...]

okay, I've tried 3.0.15 with your .config slightly changed for my local 
SATA disk, will send you copy of my .config , and, iser works for me... 
so you need to try a bit harder and send me your logs... I'm using 
iscsi-initiator-utils-6.2.0.872-21.el6.x86_64

Or.

My board ID is MT_0D81120009 which is a bit different but the HCA is 
ConnectX b0 as yours, I'm using non GA firmware, but I find it hard to 
believe this is the reason for your failure

> # ibstat
> CA 'mlx4_0'
>         CA type: MT26428
>         Number of ports: 2
>         Firmware version: 2.9.4270
>         Hardware version: b0
>         Node GUID: 0x0002c9030010c6e8
>         System image GUID: 0x0002c9030010c6eb
>         Port 1:
>                 State: Active
>                 Physical state: LinkUp
>                 Rate: 40
>                 Base lid: 10
>                 LMC: 0
>                 SM lid: 6
>                 Capability mask: 0x02510868
>                 Port GUID: 0x0002c9030010c6e9


> [  134.869036] iscsi: registered transport (tcp)
> [  134.987553] iscsi: registered transport (iser)
> [  136.075198] iser: iser_connect:connecting to: 192.168.20.19, port 
> 0xbc0c
> [  136.100162] iser: iser_cma_handler:event 0 status 0 conn 
> ffff88020eb7ba80 id ffff8802252aec00
> [  136.111158] iser: iser_cma_handler:event 2 status 0 conn 
> ffff88020eb7ba80 id ffff8802252aec00
> [  136.130923] iser: iser_create_ib_conn_res:setting conn 
> ffff88020eb7ba80 cma_id ffff8802252aec00: fmr_pool ffff880224c17880 qp 
> ffff8802154a4600
> [  136.150646] iser: iser_cma_handler:event 9 status 0 conn 
> ffff88020eb7ba80 id ffff8802252aec00
> [  136.332263] iser: iscsi_iser_ep_poll:ib conn ffff88020eb7ba80 rc = 1
> [  136.338710] scsi3 : iSCSI Initiator over iSER, v.0.1
> [  136.346240] iser: iscsi_iser_conn_bind:binding iscsi/iser conn 
> ffff880225294ab8 ffff880225294cc8 to ib_conn ffff88020eb7ba80
> [  136.609277] scsi 3:0:0:0: RAID              Mellanox 
> vsa              1    PQ: 0 ANSI: 5
> [  136.617604] scsi 3:0:0:0: Attached scsi generic sg3 type 12
> [  136.623454] scsi 3:0:0:1: Direct-Access     Mellanox 
> VIRTUAL-DISK     0001 PQ: 0 ANSI: 5
> [  136.631820] sd 3:0:0:1: Attached scsi generic sg4 type 0
> [  136.631848] sd 3:0:0:1: [sdc] 2147483648 512-byte logical blocks: 
> (1.09 TB/1.00 TiB)
> [  136.645040] sd 3:0:0:1: [sdc] Write Protect is off
> [  136.649880] sd 3:0:0:1: [sdc] Mode Sense: 49 00 00 08
> [  136.649975] sd 3:0:0:1: [sdc] Write cache: enabled, read cache: 
> enabled, doesn't support DPO or FUA
> [  136.659680]  sdc: unknown partition table
> [  136.664071] sd 3:0:0:1: [sdc] Attached SCSI disk
> [  211.020096] sd 3:0:0:1: [sdc] Synchronizing SCSI cache
> [  211.526075] iser: iscsi_iser_ep_disconnect:ib conn ffff88020eb7ba80 
> state 2
> [  211.534048] iser: iser_cma_handler:event 10 status 0 conn 
> ffff88020eb7ba80 id ffff8802252aec00
> [  211.542750] iser: iser_free_ib_conn_res:freeing conn 
> ffff88020eb7ba80 cma_id ffff8802252aec00 fmr pool ffff880224c17880 qp 
> ffff8802154a4600
> [  211.556053] iser: iser_device_try_release:device ffff880225b30480 
> refcount 0

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: IB/iSER major problems with Linux 3.0 and Solaris targets

[Sebastian Riemer]

[-- Attachment #1: Type: text/plain, Size: 610 bytes --]

On 12/01/12 17:14, Or Gerlitz wrote:
> 
> you didn't send the kernel logs from the failure after opening the iser 
> (debug_level=2) and libiscsi (debug_libiscsi_session=1
> debug_libiscsi_conn=1) debug prints

OK, I've also set mlx4_core debug_level=2 and have verified in
/sys/module that the parameters are really set.
Please find attached the relevant part of the kernel log while login
attempt.

I've also attached the log from the working stuff from OFED-1.5.4 for a
compare and the settings from discovery.

I'll try to backport the IB + iSCSI kernel code from 3.2.1 to 3.0.15 next.

Cheers,

Sebastian

[-- Attachment #2: iser_dbg_dmesg.log.gz --]
[-- Type: application/x-gzip, Size: 2673 bytes --]

[-- Attachment #3: iser_ofa_dbg_dmesg.log.gz --]
[-- Type: application/x-gzip, Size: 2284 bytes --]

[-- Attachment #4: iser1_ofa --]
[-- Type: text/plain, Size: 1605 bytes --]

node.name = iqn.2010-03.com.profitbricks:cloud:customers:storage200
node.tpgt = 2
node.startup = manual
iface.hwaddress = default
iface.iscsi_ifacename = iser1
iface.net_ifacename = ib1.bbbb
iface.transport_name = iser
node.discovery_address = 10.1.24.204
node.discovery_port = 3260
node.discovery_type = send_targets
node.session.initial_cmdsn = 0
node.session.initial_login_retry_max = 4
node.session.cmds_max = 128
node.session.queue_depth = 32
node.session.auth.authmethod = None
node.session.timeo.replacement_timeout = 30
node.session.err_timeo.abort_timeout = 15
node.session.err_timeo.lu_reset_timeout = 20
node.session.err_timeo.host_reset_timeout = 60
node.session.iscsi.FastAbort = Yes
node.session.iscsi.InitialR2T = No
node.session.iscsi.ImmediateData = Yes
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192
node.session.iscsi.DefaultTime2Retain = 0
node.session.iscsi.DefaultTime2Wait = 2
node.session.iscsi.MaxConnections = 1
node.session.iscsi.MaxOutstandingR2T = 1
node.session.iscsi.ERL = 0
node.conn[0].address = 10.1.24.204
node.conn[0].port = 3260
node.conn[0].startup = manual
node.conn[0].tcp.window_size = 524288
node.conn[0].tcp.type_of_service = 0
node.conn[0].timeo.logout_timeout = 15
node.conn[0].timeo.login_timeout = 15
node.conn[0].timeo.auth_timeout = 45
node.conn[0].timeo.noop_out_interval = 5
node.conn[0].timeo.noop_out_timeout = 5
node.conn[0].iscsi.MaxRecvDataSegmentLength = 131072
node.conn[0].iscsi.HeaderDigest = None
node.conn[0].iscsi.DataDigest = None
node.conn[0].iscsi.IFMarker = No
node.conn[0].iscsi.OFMarker = No