Re: iscsi_trx going into D state

[Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org>]

Laurance,

I'm really starting to think that the stars aligned with the phase of
the moon or something when I reproduced this in my lab before because
I've been unable to reproduce it on Infiniband the last two days. The
problem with this issue is that it is so hard to trigger, but causes a
lot of problems when it does happen. I really hate wasting people's
time when I can't reproduce it myself reliably. Please don't waste too
much time if you can't get it reproduced on Infiniband, I'll have to
wait until someone with the ConnectX-4-LX cards can replicate it.

Hmmm.... you do have ConnectX-4 cards which may have the same bug it
Ethernet mode. I don't see the RoCE bug on my ConnectX-3 cards, but
your ConnectX-4 cards may work. Try putting the cards into Ethernet
mode, set the speed and advertised speed to something lower than the
max speed and verify that the link speed is that (ethtool). On the
ConnectX-4-LX cards, I just had to set both interfaces down and then
back up at the same time, on the ConnectX-3 I had to pull the cable
(shutting down the client might have worked). Then set up target and
client with iSER, format and run the test and it should trigger
automatically.

Looking at release notes on the ConnectX-4-LX cards, the latest
firmware may fix the bug that so easily exposes the problem with that
card. My cards are SuperMicro branded cards and don't have the new
firmware available yet.

Good luck.
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


On Fri, Jan 13, 2017 at 8:10 AM, Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
>
>
> ----- Original Message -----
>> From: "Robert LeBlanc" <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org>
>> To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
>> Cc: "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Nicholas A. Bellinger" <nab-IzHhD5pYlfBP7FQvKIMDCQ@public.gmane.org>, "Zhu Lingshan"
>> <lszhu-IBi9RG/b67k@public.gmane.org>, "linux-rdma" <linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>, linux-scsi-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, "Sagi Grimberg"
>> <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, "Christoph Hellwig" <hch-jcswGhMUV9g@public.gmane.org>
>> Sent: Thursday, January 12, 2017 4:26:05 PM
>> Subject: Re: iscsi_trx going into D state
>>
>> Sorry sent prematurely...
>>
>> On Thu, Jan 12, 2017 at 2:22 PM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote:
>> > I'm having trouble replicating the D state issue on Infiniband (I was
>> > able to trigger it reliably a couple weeks back, I don't know if OFED
>> > to verify the same results happen there as well.
>>
>> I'm having trouble replicating the D state issue on Infiniband (I was
>> able to trigger it reliably a couple weeks back, I don't know if OFED
>> being installed is altering things but it only installed for 3.10. The
>> ConnectX-4-LX exposes the issue easily if you have those cards.) to
>> verify the same results happen there as well.
>>
>> ----------------
>> Robert LeBlanc
>> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
> I am only back in the office next Wednesday.
> I have this all setup using ConnectX-4 with IB/ISER but have no way of remotely creating the disconnect as I currently have it back-to-back.
> Have run multiple tests with IB and ISER hard resting the client to break the IB connection but have not been able to reproduce as yet.
> So it will have to wait until I can pull cables next week as that seemed to be the way you have been reproducing this.
>
> This is in a code area I also don't have a lot of knowledge of the flow but have started trying to understand it better.
>
> Thanks
> Laurence
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html