[nvme-fabrics]Fabrics over ROCEv2

[nvme-fabrics]Fabrics over ROCEv2

[Marta Rybczynska]

Hello,
I'm doing some interoperability tests with NVMEoF in different configurations.
I have problems setting up Mellanox ConnectX-3 Pro to perform NVMEoF in RoCE v2 mode.

I have limited the set up to 2 Mellanox cards:
* if they are connected by a cable directly it works fine, I can set up a NVMEoF
and access the disk just right. It uses ROCEv1

* if they are connected by a router (in fact a 3rd machine with different 40Gbit
cards) the discover/connect operations do not work. In the packet trace I can see
that 1st card has sent an UDP packet that looks right, but the second system
answers with ICMP port unreachable.

Has anyone tried a setup in ROCEv2 mode?

Thanks in advance,


-- 

Marta Rybczynska 

Phone : +33 6 71 09 68 03 
mrybczyn-FNhOzJFKnXGHXe+LvDLADg@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[nvme-fabrics]Fabrics over ROCEv2

[Marta Rybczynska]

Hello,
I'm doing some interoperability tests with NVMEoF in different configurations.
I have problems setting up Mellanox ConnectX-3 Pro to perform NVMEoF in RoCE v2 mode.

I have limited the set up to 2 Mellanox cards:
* if they are connected by a cable directly it works fine, I can set up a NVMEoF
and access the disk just right. It uses ROCEv1

* if they are connected by a router (in fact a 3rd machine with different 40Gbit
cards) the discover/connect operations do not work. In the packet trace I can see
that 1st card has sent an UDP packet that looks right, but the second system
answers with ICMP port unreachable.

Has anyone tried a setup in ROCEv2 mode?

Thanks in advance,


-- 

Marta Rybczynska 

Phone : +33 6 71 09 68 03 
mrybczyn at kalray.eu

Re: [nvme-fabrics]Fabrics over ROCEv2

[Christoph Hellwig]

Hi Marta,

we've tested the NVMe over Fabrics code with both Mellanox and Emulex
RoCE cards, although I don't remember if we were using RoCEv2.

But your problem doesn't sound NVMe specific.  Can your first check if a
simple test using ib_send_bw works?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[nvme-fabrics]Fabrics over ROCEv2

[Christoph Hellwig]

Hi Marta,

we've tested the NVMe over Fabrics code with both Mellanox and Emulex
RoCE cards, although I don't remember if we were using RoCEv2.

But your problem doesn't sound NVMe specific.  Can your first check if a
simple test using ib_send_bw works?

[nvme-fabrics]Fabrics over ROCEv2

[Marta Rybczynska]

Re-send to linux-rdma, it didn't make it to the list yesterday.

==
Hello,
I'm doing some interoperability tests with NVMEoF in different configurations.
I have problems setting up Mellanox ConnectX-3 Pro to perform NVMEoF in RoCE v2 mode.

I have limited the set up to 2 Mellanox cards:
* if they are connected by a cable directly it works fine, I can set up a NVMEoF
and access the disk just right. It uses ROCEv1

* if they are connected by a router (in fact a 3rd machine with different 40Gbit
cards) the discover/connect operations do not work. In the packet trace I can see
that 1st card has sent an UDP packet that looks right, but the second system
answers with ICMP port unreachable.

Has anyone tried a setup in ROCEv2 mode?

Thanks in advance,

-- 

Marta Rybczynska 

Phone : +33 6 71 09 68 03 
mrybczyn-FNhOzJFKnXGHXe+LvDLADg@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [nvme-fabrics]Fabrics over ROCEv2

[Marta Rybczynska]

Hello Christoph,
Thanks for the answer. I do indeed have more general problems with
ROCEv2: ib_send_bw works fine for ROCEv1 but not with ROCEv2. I'm
going to follow with the Mellanox persons as it seems to be mlx4 or
configuration-related.

Marta

----- Le 24 Juin 16, à 9:13, Christoph Hellwig hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org a écrit :

> Hi Marta,
> 
> we've tested the NVMe over Fabrics code with both Mellanox and Emulex
> RoCE cards, although I don't remember if we were using RoCEv2.
> 
> But your problem doesn't sound NVMe specific.  Can your first check if a
> simple test using ib_send_bw works?

-- 
-- 

Marta Rybczynska 

Phone : +33 6 71 09 68 03 
mrybczyn-FNhOzJFKnXGHXe+LvDLADg@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[nvme-fabrics]Fabrics over ROCEv2

[Marta Rybczynska]

Hello Christoph,
Thanks for the answer. I do indeed have more general problems with
ROCEv2: ib_send_bw works fine for ROCEv1 but not with ROCEv2. I'm
going to follow with the Mellanox persons as it seems to be mlx4 or
configuration-related.

Marta

----- Le 24 Juin 16, ? 9:13, Christoph Hellwig hch at infradead.org a ?crit :

> Hi Marta,
> 
> we've tested the NVMe over Fabrics code with both Mellanox and Emulex
> RoCE cards, although I don't remember if we were using RoCEv2.
> 
> But your problem doesn't sound NVMe specific.  Can your first check if a
> simple test using ib_send_bw works?

-- 
-- 

Marta Rybczynska 

Phone : +33 6 71 09 68 03 
mrybczyn at kalray.eu

Re: [nvme-fabrics]Fabrics over ROCEv2

[Majd Dibbiny]

> On Jun 24, 2016, at 2:02 PM, Marta Rybczynska <mrybczyn-FNhOzJFKnXGHXe+LvDLADg@public.gmane.org> wrote:
> 
> Hello Christoph,
> Thanks for the answer. I do indeed have more general problems with
> ROCEv2: ib_send_bw works fine for ROCEv1 but not with ROCEv2. I'm
> going to follow with the Mellanox persons as it seems to be mlx4 or
> configuration-related.
> 
Hi Marta,

Do you run with RDMA-CM?
Can you please provide the command lines you use and the GIDs table?

Thanks
> Marta
> 
> ----- Le 24 Juin 16, à 9:13, Christoph Hellwig hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org a écrit :
> 
>> Hi Marta,
>> 
>> we've tested the NVMe over Fabrics code with both Mellanox and Emulex
>> RoCE cards, although I don't remember if we were using RoCEv2.
>> 
>> But your problem doesn't sound NVMe specific.  Can your first check if a
>> simple test using ib_send_bw works?
> 
> -- 
> -- 
> 
> Marta Rybczynska 
> 
> Phone : +33 6 71 09 68 03 
> mrybczyn-FNhOzJFKnXGHXe+LvDLADg@public.gmane.org
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[nvme-fabrics]Fabrics over ROCEv2

[Majd Dibbiny]

> On Jun 24, 2016,@2:02 PM, Marta Rybczynska <mrybczyn@kalray.eu> wrote:
> 
> Hello Christoph,
> Thanks for the answer. I do indeed have more general problems with
> ROCEv2: ib_send_bw works fine for ROCEv1 but not with ROCEv2. I'm
> going to follow with the Mellanox persons as it seems to be mlx4 or
> configuration-related.
> 
Hi Marta,

Do you run with RDMA-CM?
Can you please provide the command lines you use and the GIDs table?

Thanks
> Marta
> 
> ----- Le 24 Juin 16, ? 9:13, Christoph Hellwig hch at infradead.org a ?crit :
> 
>> Hi Marta,
>> 
>> we've tested the NVMe over Fabrics code with both Mellanox and Emulex
>> RoCE cards, although I don't remember if we were using RoCEv2.
>> 
>> But your problem doesn't sound NVMe specific.  Can your first check if a
>> simple test using ib_send_bw works?
> 
> -- 
> -- 
> 
> Marta Rybczynska 
> 
> Phone : +33 6 71 09 68 03 
> mrybczyn at kalray.eu
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [nvme-fabrics]Fabrics over ROCEv2

[Marta Rybczynska]

----- Le 24 Juin 16, à 15:40, Majd Dibbiny majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org a écrit :

>> On Jun 24, 2016, at 2:02 PM, Marta Rybczynska <mrybczyn-FNhOzJFKnXGHXe+LvDLADg@public.gmane.org> wrote:
>> 
>> Hello Christoph,
>> Thanks for the answer. I do indeed have more general problems with
>> ROCEv2: ib_send_bw works fine for ROCEv1 but not with ROCEv2. I'm
>> going to follow with the Mellanox persons as it seems to be mlx4 or
>> configuration-related.
>> 
> Hi Marta,
> 
> Do you run with RDMA-CM?
> Can you please provide the command lines you use and the GIDs table?
> 

Hello Majd,
Yes, it's using RDMA-CM. I have re-connected the network and the machines have 
rebooted (unrelated panic) and now it's working fine. If I'm able to reproduce 
the problem, I will keep you updated.

Thanks for the help all,
Marta
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[nvme-fabrics]Fabrics over ROCEv2

[Marta Rybczynska]

----- Le 24 Juin 16, ? 15:40, Majd Dibbiny majd at mellanox.com a ?crit :

>> On Jun 24, 2016,@2:02 PM, Marta Rybczynska <mrybczyn@kalray.eu> wrote:
>> 
>> Hello Christoph,
>> Thanks for the answer. I do indeed have more general problems with
>> ROCEv2: ib_send_bw works fine for ROCEv1 but not with ROCEv2. I'm
>> going to follow with the Mellanox persons as it seems to be mlx4 or
>> configuration-related.
>> 
> Hi Marta,
> 
> Do you run with RDMA-CM?
> Can you please provide the command lines you use and the GIDs table?
> 

Hello Majd,
Yes, it's using RDMA-CM. I have re-connected the network and the machines have 
rebooted (unrelated panic) and now it's working fine. If I'm able to reproduce 
the problem, I will keep you updated.

Thanks for the help all,
Marta

Re: [nvme-fabrics]Fabrics over ROCEv2

[Majd Dibbiny]

> On Jun 24, 2016, at 5:32 PM, Marta Rybczynska <mrybczyn-FNhOzJFKnXGHXe+LvDLADg@public.gmane.org> wrote:
> 
> 
> 
> ----- Le 24 Juin 16, à 15:40, Majd Dibbiny majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org a écrit :
> 
>>> On Jun 24, 2016, at 2:02 PM, Marta Rybczynska <mrybczyn-FNhOzJFKnXGHXe+LvDLADg@public.gmane.org> wrote:
>>> 
>>> Hello Christoph,
>>> Thanks for the answer. I do indeed have more general problems with
>>> ROCEv2: ib_send_bw works fine for ROCEv1 but not with ROCEv2. I'm
>>> going to follow with the Mellanox persons as it seems to be mlx4 or
>>> configuration-related.
>> Hi Marta,
>> 
>> Do you run with RDMA-CM?
>> Can you please provide the command lines you use and the GIDs table?
> 
> Hello Majd,
> Yes, it's using RDMA-CM. I have re-connected the network and the
This might be related to the fact that you need to configure the RDMA CM mode using configfs. (By default it's V1) 
> machines have 
> rebooted (unrelated panic) and now it's working fine. If I'm able to reproduce 
> the problem, I will keep you updated.
> 
> Thanks for the help all,
> Marta
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[nvme-fabrics]Fabrics over ROCEv2

[Majd Dibbiny]

> On Jun 24, 2016,@5:32 PM, Marta Rybczynska <mrybczyn@kalray.eu> wrote:
> 
> 
> 
> ----- Le 24 Juin 16, ? 15:40, Majd Dibbiny majd at mellanox.com a ?crit :
> 
>>> On Jun 24, 2016,@2:02 PM, Marta Rybczynska <mrybczyn@kalray.eu> wrote:
>>> 
>>> Hello Christoph,
>>> Thanks for the answer. I do indeed have more general problems with
>>> ROCEv2: ib_send_bw works fine for ROCEv1 but not with ROCEv2. I'm
>>> going to follow with the Mellanox persons as it seems to be mlx4 or
>>> configuration-related.
>> Hi Marta,
>> 
>> Do you run with RDMA-CM?
>> Can you please provide the command lines you use and the GIDs table?
> 
> Hello Majd,
> Yes, it's using RDMA-CM. I have re-connected the network and the
This might be related to the fact that you need to configure the RDMA CM mode using configfs. (By default it's V1) 
> machines have 
> rebooted (unrelated panic) and now it's working fine. If I'm able to reproduce 
> the problem, I will keep you updated.
> 
> Thanks for the help all,
> Marta

Re: [nvme-fabrics]Fabrics over ROCEv2

[Marta Rybczynska]

----- Le 24 Juin 16, à 16:38, Majd Dibbiny majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org a écrit :

>> On Jun 24, 2016, at 5:32 PM, Marta Rybczynska <mrybczyn-FNhOzJFKnXGHXe+LvDLADg@public.gmane.org> wrote:
>> 
>> 
>> 
>> ----- Le 24 Juin 16, à 15:40, Majd Dibbiny majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org a écrit :
>> 
>>>> On Jun 24, 2016, at 2:02 PM, Marta Rybczynska <mrybczyn-FNhOzJFKnXGHXe+LvDLADg@public.gmane.org> wrote:
>>>> 
>>>> Hello Christoph,
>>>> Thanks for the answer. I do indeed have more general problems with
>>>> ROCEv2: ib_send_bw works fine for ROCEv1 but not with ROCEv2. I'm
>>>> going to follow with the Mellanox persons as it seems to be mlx4 or
>>>> configuration-related.
>>> Hi Marta,
>>> 
>>> Do you run with RDMA-CM?
>>> Can you please provide the command lines you use and the GIDs table?
>> 
>> Hello Majd,
>> Yes, it's using RDMA-CM. I have re-connected the network and the
> This might be related to the fact that you need to configure the RDMA CM mode
> using configfs. (By default it's V1)


This isn't in fact working for me in 4.7+ (does work fine in OFED kernel 3.10).
Write to default_roce_mode does give Permission denied even for root (haven't 
debugged this one, however).

-- 

Marta Rybczynska 

Phone : +33 6 71 09 68 03 
mrybczyn-FNhOzJFKnXGHXe+LvDLADg@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[nvme-fabrics]Fabrics over ROCEv2

[Marta Rybczynska]

----- Le 24 Juin 16, ? 16:38, Majd Dibbiny majd at mellanox.com a ?crit :

>> On Jun 24, 2016,@5:32 PM, Marta Rybczynska <mrybczyn@kalray.eu> wrote:
>> 
>> 
>> 
>> ----- Le 24 Juin 16, ? 15:40, Majd Dibbiny majd at mellanox.com a ?crit :
>> 
>>>> On Jun 24, 2016,@2:02 PM, Marta Rybczynska <mrybczyn@kalray.eu> wrote:
>>>> 
>>>> Hello Christoph,
>>>> Thanks for the answer. I do indeed have more general problems with
>>>> ROCEv2: ib_send_bw works fine for ROCEv1 but not with ROCEv2. I'm
>>>> going to follow with the Mellanox persons as it seems to be mlx4 or
>>>> configuration-related.
>>> Hi Marta,
>>> 
>>> Do you run with RDMA-CM?
>>> Can you please provide the command lines you use and the GIDs table?
>> 
>> Hello Majd,
>> Yes, it's using RDMA-CM. I have re-connected the network and the
> This might be related to the fact that you need to configure the RDMA CM mode
> using configfs. (By default it's V1)


This isn't in fact working for me in 4.7+ (does work fine in OFED kernel 3.10).
Write to default_roce_mode does give Permission denied even for root (haven't 
debugged this one, however).

-- 

Marta Rybczynska 

Phone : +33 6 71 09 68 03 
mrybczyn at kalray.eu