Quick mlx4 IB SR-IOV howto?

Quick mlx4 IB SR-IOV howto?

[Roland Dreier]

So I have SR-IOV enabled on a ConnectX-3 adapter, and I loaded the
driver with num_vfs=1 probe_vf=1, so in the host I see:

# The master device
$ ibv_devinfo -d mlx4_1
hca_id: mlx4_1
        transport:                      InfiniBand (0)
        fw_ver:                         2.11.500
        node_guid:                      0002:c903:0036:6670
        sys_image_guid:                 0002:c903:0036:6673
        vendor_id:                      0x02c9
        vendor_part_id:                 4099
        hw_ver:                         0x0
        board_id:                       MT_1100120019
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                2048 (4)
                        active_mtu:             2048 (4)
                        sm_lid:                 1
                        port_lid:               2
                        port_lmc:               0x00
                        link_layer:             InfiniBand

# The slave VF
$ ibv_devinfo -d mlx4_3
hca_id: mlx4_3
        transport:                      InfiniBand (0)
        fw_ver:                         2.11.500
        node_guid:                      0014:0500:8c20:105c
        sys_image_guid:                 0002:c903:0036:6673
        vendor_id:                      0x02c9
        vendor_part_id:                 4100
        hw_ver:                         0x0
        board_id:                       MT_1100120019
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_DOWN (1)
                        max_mtu:                2048 (4)
                        active_mtu:             2048 (4)
                        sm_lid:                 1
                        port_lid:               2
                        port_lmc:               0x00
                        link_layer:             InfiniBand

What do I need for the slave VF's port to become active?
I'm running opensm 3.3.13 on a different box, is that new enough?
(does SR-IOV require any SM support?)

Thanks,
  Roland
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Quick mlx4 IB SR-IOV howto?

[Hal Rosenstock]

On 9/26/2012 1:14 PM, Roland Dreier wrote:
> I'm running opensm 3.3.13 on a different box, is that new enough?
> (does SR-IOV require any SM support?)

SRIOV support went into OpenSM 3.3.14. The features for SRIOV are alias
(or virtual) GUID support and some minor partition manager changes (to
allow both full and limited members of same partition on same port).

See
http://git.openfabrics.org/git?p=~alexnetes/opensm.git;a=blob;f=doc/opensm-sriov.txt;h=bbdff48a4cf62398449ea74037e70024208ebc4d;hb=8e6dfbd0b9284fe0083b3f378ec313c3acfb060c

-- Hal

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Quick mlx4 IB SR-IOV howto?

[Or Gerlitz]

On Wed, Sep 26, 2012 at 7:14 PM, Roland Dreier <roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org> wrote:
> So I have SR-IOV enabled on a ConnectX-3 adapter, and I loaded the
> driver with num_vfs=1 probe_vf=1, so in the host I see:
>
> # The master device
> $ ibv_devinfo -d mlx4_1
> hca_id: mlx4_1
>         transport:                      InfiniBand (0)
>         fw_ver:                         2.11.500
>         node_guid:                      0002:c903:0036:6670
>         sys_image_guid:                 0002:c903:0036:6673
>         vendor_id:                      0x02c9
>         vendor_part_id:                 4099
>         hw_ver:                         0x0
>         board_id:                       MT_1100120019
>         phys_port_cnt:                  1
>                 port:   1
>                         state:                  PORT_ACTIVE (4)
>                         max_mtu:                2048 (4)
>                         active_mtu:             2048 (4)
>                         sm_lid:                 1
>                         port_lid:               2
>                         port_lmc:               0x00
>                         link_layer:             InfiniBand
>
> # The slave VF
> $ ibv_devinfo -d mlx4_3
> hca_id: mlx4_3
>         transport:                      InfiniBand (0)
>         fw_ver:                         2.11.500
>         node_guid:                      0014:0500:8c20:105c
>         sys_image_guid:                 0002:c903:0036:6673
>         vendor_id:                      0x02c9
>         vendor_part_id:                 4100
>         hw_ver:                         0x0
>         board_id:                       MT_1100120019
>         phys_port_cnt:                  1
>                 port:   1
>                         state:                  PORT_DOWN (1)
>                         max_mtu:                2048 (4)
>                         active_mtu:             2048 (4)
>                         sm_lid:                 1
>                         port_lid:               2
>                         port_lmc:               0x00
>                         link_layer:             InfiniBand
>
> What do I need for the slave VF's port to become active?
> I'm running opensm 3.3.13 on a different box, is that new enough?
> (does SR-IOV require any SM support?)

Yes, as Hal noted, at minimum you need opensm 3.3.14
(http://marc.info/?l=linux-rdma&m=133819320432335&w=2) as it is the
1st version to support alias-guid et al stuff needed for SRIOV, 3.3.15
is also out now, so you want the 2nd version that supports this...
basically you need IB link for the PPF and the slave to get an alias
guid registered for it @ the SM. We (IL team) were off Tues/Wed as of
a holiday, will try to get you further details tonight and if not, by
tomorrow, sure.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Quick mlx4 IB SR-IOV howto?

[Or Gerlitz]

On Wed, Sep 26, 2012 at 10:22 PM, Or Gerlitz <or.gerlitz-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> On Wed, Sep 26, 2012 at 7:14 PM, Roland Dreier <roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org> wrote:
>> What do I need for the slave VF's port to become active?
>> I'm running opensm 3.3.13 on a different box, is that new enough?
>> (does SR-IOV require any SM support?)
>
> Yes, as Hal noted, at minimum you need opensm 3.3.14


Roland, did this help? do you have IB link for the VF? IPoIB working on it?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Quick mlx4 IB SR-IOV howto?

[Roland Dreier]

On Wed, Sep 26, 2012 at 2:30 PM, Or Gerlitz <or.gerlitz-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Roland, did this help? do you have IB link for the VF? IPoIB working on it?

Sorry, replied to Hal only by accident.  Yes, latest opensm makes things
work fine for me.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Quick mlx4 IB SR-IOV howto?

[Or Gerlitz]

On 27/09/2012 08:47, Roland Dreier wrote:
> On Wed, Sep 26, 2012 at 2:30 PM, Or Gerlitz <or.gerlitz-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> Roland, did this help? do you have IB link for the VF? IPoIB working on it?
> Sorry, replied to Hal only by accident.  Yes, latest opensm makes things
> work fine for me.

Good, so you have basic IB functionality working fine for the VF, e.g 
IPoIB, verbs tests, etc?

Or.


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Quick mlx4 IB SR-IOV howto?

[Roland Dreier]

> Good, so you have basic IB functionality working fine for the VF, e.g IPoIB,
> verbs tests, etc?

Haven't tried too much, but yeah, everything I tried worked with no problem.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Quick mlx4 IB SR-IOV howto?

[Yi-Man Ma]

Roland Dreier <roland@...> writes:

> 
> So I have SR-IOV enabled on a ConnectX-3 adapter, and I loaded the
> driver with num_vfs=1 probe_vf=1, so in the host I see:
>  
> Thanks,
>   Roland
> --
 

Hi,

You enabled SR-IOV on ConnectX-3 adapter successfully.
I also tried to do the same thing, but I failed.
What did you do to enable SR-IOV on ConnectX-3 adapter?
I tried following steps:
1. Enable Intel VT-d in BIOS and in the kernel.
2. rmmod mlx4_en
   rmmode_mlx4_core
   modprobe mlx4_core  sr_iov=1 probe_vf=1
But it failed.

Do I still have other things to do for enabling SR-IOV on ConnectX-3 adapter?

Thanks,
Yi-Man

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Quick mlx4 IB SR-IOV howto?

[Dotan Barak]

Hi Yi-Man.

>   
>
> Hi,
>
> You enabled SR-IOV on ConnectX-3 adapter successfully.
> I also tried to do the same thing, but I failed.
> What did you do to enable SR-IOV on ConnectX-3 adapter?
> I tried following steps:
> 1. Enable Intel VT-d in BIOS and in the kernel.
In some servers this isn't enough, there is a "virtualization 
technology" option and you need to enable it too.
> 2. rmmod mlx4_en
>     rmmode_mlx4_core
>     modprobe mlx4_core  sr_iov=1 probe_vf=1
> But it failed.
Do you have an SR-IOV capable firmware in your HCA?

> Do I still have other things to do for enabling SR-IOV on ConnectX-3 adapter?
>
>
The output of dmesg or /var/log/messages with the error message can be 
useful.

Thanks
Dotan
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html