How do I generate and set a GUID on an Ethernet VF with ConnectX-4?

How do I generate and set a GUID on an Ethernet VF with ConnectX-4?

[Robert LeBlanc]

One problem we found with rping is that when the GUID is '0', it
fails. However, all of the VFs that are created are created with a '0'
GUID. How do I generate a GUID that will not conflict for the VFs?
Once I have a generated GUID, how do I set it on a VF using the inbox
drivers? I can't find a /sys entry that will allow me to write it and
the one mentioned in the Mellanox OFED doesn't exist.

I'll need to find a way to script this as part of my udev rules that
adds the VFs.

Thank you,
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How do I generate and set a GUID on an Ethernet VF with ConnectX-4?

[Majd Dibbiny]

> On May 30, 2017, at 8:55 PM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote:
> 
> One problem we found with rping is that when the GUID is '0', it
> fails. However, all of the VFs that are created are created with a '0'
> GUID. How do I generate a GUID that will not conflict for the VFs?
Hi Robert,

IIRC, once you set a MAC address for a VF, the node guid will be derived automatically for it. Also, it can be controlled using iptool.


> Once I have a generated GUID, how do I set it on a VF using the inbox
> drivers? I can't find a /sys entry that will allow me to write it and
> the one mentioned in the Mellanox OFED doesn't exist.
> 
> I'll need to find a way to script this as part of my udev rules that
> adds the VFs.
> 
> Thank you,
> ----------------
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How do I generate and set a GUID on an Ethernet VF with ConnectX-4?

[Robert LeBlanc]

I think there may be more than just changing the MAC. The Mellanox
documentation talks about unbinding and re-binding the VF. The odd
thing is that the MAC for the VF under the parent adapter is '0', but
the adapter itself has a MAC address. You had a good idea, but I think
we are still missing something. I do think that changing the MAC is
the right thing to get the GUID changed according to the Mellanox
documentation.

4: rs81p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state
UP mode DEFAULT group default qlen 1000
   link/ether 24:8a:07:1e:7c:60 brd ff:ff:ff:ff:ff:ff
   vf 0 MAC 12:34:56:78:9a:bc, spoof checking off, link-state auto
   vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
8: rs81p1v1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode
DEFAULT group default qlen 1000
   link/ether e6:17:5f:85:45:e6 brd ff:ff:ff:ff:ff:ff
17: rs81p1v0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq
state UP mode DEFAULT group default qlen 1000
   link/ether ee:af:9f:66:2b:0c brd ff:ff:ff:ff:ff:ff
# ibv_devices
   device                 node GUID
   ------              ----------------
   mlx5_0              248a0703001e7c60
   mlx5_2              0000000000000000
   mlx5_4              0000000000000000
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


On Wed, May 31, 2017 at 2:28 AM, Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote:
>
>> On May 30, 2017, at 8:55 PM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote:
>>
>> One problem we found with rping is that when the GUID is '0', it
>> fails. However, all of the VFs that are created are created with a '0'
>> GUID. How do I generate a GUID that will not conflict for the VFs?
> Hi Robert,
>
> IIRC, once you set a MAC address for a VF, the node guid will be derived automatically for it. Also, it can be controlled using iptool.
>
>
>> Once I have a generated GUID, how do I set it on a VF using the inbox
>> drivers? I can't find a /sys entry that will allow me to write it and
>> the one mentioned in the Mellanox OFED doesn't exist.
>>
>> I'll need to find a way to script this as part of my udev rules that
>> adds the VFs.
>>
>> Thank you,
>> ----------------
>> Robert LeBlanc
>> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How do I generate and set a GUID on an Ethernet VF with ConnectX-4?

[Majd Dibbiny]

> On May 31, 2017, at 6:16 PM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote:
> 
> I think there may be more than just changing the MAC. The Mellanox
> documentation talks about unbinding and re-binding the VF.
Correct.. I thought you've already done that.
So basically the flow is as follow:
1. Create VFs on the hypervisor
2. Configure MAC addresses to the VFs on the Hypervisor
3. Unbind the VFs drivers
4. Bind back

This way, the VFs will reload with the configured data..
> The odd
> thing is that the MAC for the VF under the parent adapter is '0', but
> the othadapter itself has a MAC address. You had a good idea, but I think
> we are still missing something. I do think that changing the MAC is
> the right thing to get the GUID changed according to the Mellanox
> documentation.
> 
> 4: rs81p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state
> UP mode DEFAULT group default qlen 1000
>   link/ether 24:8a:07:1e:7c:60 brd ff:ff:ff:ff:ff:ff
>   vf 0 MAC 12:34:56:78:9a:bc, spoof checking off, link-state auto
>   vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
> 8: rs81p1v1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode
> DEFAULT group default qlen 1000
>   link/ether e6:17:5f:85:45:e6 brd ff:ff:ff:ff:ff:ff
> 17: rs81p1v0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq
> state UP mode DEFAULT group default qlen 1000
>   link/ether ee:af:9f:66:2b:0c brd ff:ff:ff:ff:ff:ff
> # ibv_devices
>   device                 node GUID
>   ------              ----------------
>   mlx5_0              248a0703001e7c60
>   mlx5_2              0000000000000000
>   mlx5_4              0000000000000000
> ----------------
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
> 
> 
>> On Wed, May 31, 2017 at 2:28 AM, Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote:
>> 
>>> On May 30, 2017, at 8:55 PM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote:
>>> 
>>> One problem we found with rping is that when the GUID is '0', it
>>> fails. However, all of the VFs that are created are created with a '0'
>>> GUID. How do I generate a GUID that will not conflict for the VFs?
>> Hi Robert,
>> 
>> IIRC, once you set a MAC address for a VF, the node guid will be derived automatically for it. Also, it can be controlled using iptool.
>> 
>> 
>>> Once I have a generated GUID, how do I set it on a VF using the inbox
>>> drivers? I can't find a /sys entry that will allow me to write it and
>>> the one mentioned in the Mellanox OFED doesn't exist.
>>> 
>>> I'll need to find a way to script this as part of my udev rules that
>>> adds the VFs.
>>> 
>>> Thank you,
>>> ----------------
>>> Robert LeBlanc
>>> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How do I generate and set a GUID on an Ethernet VF with ConnectX-4?

[Robert LeBlanc]

On Wed, May 31, 2017 at 9:25 AM, Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote:
>
>> On May 31, 2017, at 6:16 PM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote:
>>
>> I think there may be more than just changing the MAC. The Mellanox
>> documentation talks about unbinding and re-binding the VF.
> Correct.. I thought you've already done that.
> So basically the flow is as follow:
> 1. Create VFs on the hypervisor
> 2. Configure MAC addresses to the VFs on the Hypervisor
> 3. Unbind the VFs drivers
> 4. Bind back

We aren't using the VFs for VMs, but I don't think that matters. I
think even reading the documentation several times, I was trying to
access the bind/unbind sysfs entries from the wrong part of the tree.
I did an unbind/bind and the MAC address on the rs81p1v0 device and
the GUID was set. All looks good now!

Thanks,

----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How do I generate and set a GUID on an Ethernet VF with ConnectX-4?

[Majd Dibbiny]

> On May 31, 2017, at 6:40 PM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote:
> 
>> On Wed, May 31, 2017 at 9:25 AM, Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote:
>> 
>>> On May 31, 2017, at 6:16 PM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote:
>>> 
>>> I think there may be more than just changing the MAC. The Mellanox
>>> documentation talks about unbinding and re-binding the VF.
>> Correct.. I thought you've already done that.
>> So basically the flow is as follow:
>> 1. Create VFs on the hypervisor
>> 2. Configure MAC addresses to the VFs on the Hypervisor
>> 3. Unbind the VFs drivers
>> 4. Bind back
> 
> We aren't using the VFs for VMs, but I don't think that matters. I
> think even reading the documentation several times, I was trying to
> access the bind/unbind sysfs entries from the wrong part of the tree.
> I did an unbind/bind and the MAC address on the rs81p1v0 device and
> the GUID was set. All looks good now!
Glad to hear:)
Is this the same issue you were facing with rdma-cm segfaults?
> 
> Thanks,
> 
> ----------------
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How do I generate and set a GUID on an Ethernet VF with ConnectX-4?

[Robert LeBlanc]

On Wed, May 31, 2017 at 9:50 AM, Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote:
> Is this the same issue you were facing with rdma-cm segfaults?

The '0' GUID was causing a segfault in rping, udaddy, ib_send_bw in
rdma_cm mode, etc. When both nodes have a GUID, it works fine. It
could be argued that it should be a bit more helpful and exit with a
nice message rather than a segfault. It also could be argued that a
random GUID/MAC should be assigned at VF creation time and if you want
something static to go through the hoops (it sounds like a good
default so it works out of the box, I even though I thought I read
that was the behavior but I can't find that document again). I can
work around it by adding the random generation to my udev rules, but
it is not convenient. I think I'll have to check if a MAC is already
assigned or I could get into a loop, I'll have to look at it more
later. I need to get source address binding working in iSER first.

Thanks again,

----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How do I generate and set a GUID on an Ethernet VF with ConnectX-4?

[Majd Dibbiny]

> On May 31, 2017, at 7:02 PM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote:
> 
>> On Wed, May 31, 2017 at 9:50 AM, Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote:
>> Is this the same issue you were facing with rdma-cm segfaults?
> 
> The '0' GUID was causing a segfault in rping, udaddy, ib_send_bw in
> rdma_cm mode, etc. When both nodes have a GUID, it works fine. It
> could be argued that it should be a bit more helpful and exit with a
> nice message rather than a segfault. It also could be argued that a
> random GUID/MAC should be assigned at VF creation time and if you want
> something static to go through the hoops (it sounds like a good
> default so it works out of the box, I even though I thought I read
> that was the behavior but I can't find that document again). I can
> work around it by adding the random generation to my udev rules, but
> it is not convenient. I think I'll have to check if a MAC is already
> assigned or I could get into a loop, I'll have to look at it more
> later. I need to get source address binding working in iSER first.

Thanks Robert,

I'll take this into consideration and we'll work on the OOB experience with SRIOV.
> 
> Thanks again,
> 
> ----------------
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html