mlx5 under FreeBSD

mlx5 under FreeBSD

[Mit Matelske]

Is anyone working to include support for the mlx5 PMD under FreeBSD? 

I've started down this road by building Linux compatibility layers for the required Netlink and Ethtool calls in the driver, but would gladly accept help and advice from those much more knowledgeable then myself! 

Though not important, why did Mellanox build a PMD that relied on the kernel driver being in place unlike most other PMDs? 

Thanks! 

Mit Matelske 

Re: mlx5 under FreeBSD

[Thomas Monjalon]

15/11/2018 17:48, Mit Matelske:
> Is anyone working to include support for the mlx5 PMD under FreeBSD?

I think Stephen (Cc) looked at it.

> I've started down this road by building Linux compatibility layers for the required Netlink and Ethtool calls in the driver, but would gladly accept help and advice from those much more knowledgeable then myself!

Cc also mlx5 maintainers.

> Though not important, why did Mellanox build a PMD that relied on the kernel driver being in place unlike most other PMDs? 

Because it allows to choose which flows go to DPDK and which ones
are processed by the kernel.
Actually, you should ask why other PMDs don't have this feature ;)

Re: mlx5 under FreeBSD

[Mit Matelske]

> 15/11/2018 17:48, Mit Matelske:
>> Is anyone working to include support for the mlx5 PMD under FreeBSD?
> 
> I think Stephen (Cc) looked at it.
> 
>> I've started down this road by building Linux compatibility layers for the
>> required Netlink and Ethtool calls in the driver, but would gladly accept help
>> and advice from those much more knowledgeable then myself!
> 
> Cc also mlx5 maintainers.

Thomas-

Thanks for looping the correct people in!

> 
>> Though not important, why did Mellanox build a PMD that relied on the kernel
>> driver being in place unlike most other PMDs?
> 
> Because it allows to choose which flows go to DPDK and which ones
> are processed by the kernel.
> Actually, you should ask why other PMDs don't have this feature ;)

Very fair response.  We actually wrote our own "shim" into the stack for all the
relevant drivers we use to both keep the existing ifnet interfaces around and to
pass packets to and from the stack from every interface.

Your comment matches one of my co-worker's when I explained to him how the mlx5
driver works.

Mit

Re: mlx5 under FreeBSD

[Shahaf Shuler]

Hi Mit,

Monday, November 19, 2018 1:23 AM, Mit Matelske:
> Subject: Re: [dpdk-dev] mlx5 under FreeBSD
> 
> > 15/11/2018 17:48, Mit Matelske:
> >> Is anyone working to include support for the mlx5 PMD under FreeBSD?
> >
> > I think Stephen (Cc) looked at it.
> >
> >> I've started down this road by building Linux compatibility layers
> >> for the required Netlink and Ethtool calls in the driver, but would
> >> gladly accept help and advice from those much more knowledgeable then
> myself!

Are you sure netlink and ioctl is all you need?
Looks like also sysfs stuff, and I am not sure what else is missing from the DPDK eal layer, the FreeBSD has many not supported functions. 

We haven't started to work on it because the majority of our use cases are for Linux. Nevertheless, we can help with code reviews and guidance. 

> >
> > Cc also mlx5 maintainers.
> 
> Thomas-
> 
> Thanks for looping the correct people in!
> 
> >
> >> Though not important, why did Mellanox build a PMD that relied on the
> >> kernel driver being in place unlike most other PMDs?
> >
> > Because it allows to choose which flows go to DPDK and which ones are
> > processed by the kernel.
> > Actually, you should ask why other PMDs don't have this feature ;)

+1. 

> 
> Very fair response.  We actually wrote our own "shim" into the stack for all
> the relevant drivers we use to both keep the existing ifnet interfaces around
> and to pass packets to and from the stack from every interface.
> 
> Your comment matches one of my co-worker's when I explained to him how
> the mlx5 driver works.
> 
> Mit
> 

Re: mlx5 under FreeBSD

[Mit Matelske]

> Monday, November 19, 2018 1:23 AM, Mit Matelske:
>> Subject: Re: [dpdk-dev] mlx5 under FreeBSD
>> 
>> > 15/11/2018 17:48, Mit Matelske:
>> >> Is anyone working to include support for the mlx5 PMD under FreeBSD?
>> >
>> > I think Stephen (Cc) looked at it.
>> >
>> >> I've started down this road by building Linux compatibility layers
>> >> for the required Netlink and Ethtool calls in the driver, but would
>> >> gladly accept help and advice from those much more knowledgeable then
>> myself!
> 
> Are you sure netlink and ioctl is all you need?
> Looks like also sysfs stuff, and I am not sure what else is missing from the
> DPDK eal layer, the FreeBSD has many not supported functions.

No, I'm not sure of anything.  I've just started focusing on this.  I do know to
get it to compile I had to include subsets the following includes:

libmnl/libmnl.h
linux/ethtool.h
linux/netlink.h
linux/neighbour.h
linux/types.h
linux/rtnetlink.h
linux/pkt_cls.h
linux/pkt_sched.h
linux/if_ether.h
linux/tc_act/tc_gact.h
linux/tc_act/tc_mirred.h

My initial goal is to change the DPDK code as little as possible, but that
might not be be possible.  You'd have a much better feel for that than me
at this point.

> 
> We haven't started to work on it because the majority of our use cases are for
> Linux. 

I figured as much.  

> Nevertheless, we can help with code reviews and guidance.

Once I get somewhere I will definitely take you up on that.

Thanks!

Mit

> 
>> >
>> > Cc also mlx5 maintainers.
>> 
>> Thomas-
>> 
>> Thanks for looping the correct people in!
>> 
>> >
>> >> Though not important, why did Mellanox build a PMD that relied on the
>> >> kernel driver being in place unlike most other PMDs?
>> >
>> > Because it allows to choose which flows go to DPDK and which ones are
>> > processed by the kernel.
>> > Actually, you should ask why other PMDs don't have this feature ;)
> 
> +1.
> 
>> 
>> Very fair response.  We actually wrote our own "shim" into the stack for all
>> the relevant drivers we use to both keep the existing ifnet interfaces around
>> and to pass packets to and from the stack from every interface.
>> 
>> Your comment matches one of my co-worker's when I explained to him how
>> the mlx5 driver works.
>> 
>> Mit
>>

Re: mlx5 under FreeBSD

[Stephen Hemminger]

On Mon, 19 Nov 2018 09:09:22 -0600 (CST)
Mit Matelske <mit@pt.net> wrote:

> > Monday, November 19, 2018 1:23 AM, Mit Matelske:  
> >> Subject: Re: [dpdk-dev] mlx5 under FreeBSD
> >>   
> >> > 15/11/2018 17:48, Mit Matelske:  
> >> >> Is anyone working to include support for the mlx5 PMD under FreeBSD?  
> >> >
> >> > I think Stephen (Cc) looked at it.
> >> >  
> >> >> I've started down this road by building Linux compatibility layers
> >> >> for the required Netlink and Ethtool calls in the driver, but would
> >> >> gladly accept help and advice from those much more knowledgeable then  
> >> myself!  
> > 
> > Are you sure netlink and ioctl is all you need?
> > Looks like also sysfs stuff, and I am not sure what else is missing from the
> > DPDK eal layer, the FreeBSD has many not supported functions.  
> 
> No, I'm not sure of anything.  I've just started focusing on this.  I do know to
> get it to compile I had to include subsets the following includes:
> 
> libmnl/libmnl.h
> linux/ethtool.h
> linux/netlink.h
> linux/neighbour.h
> linux/types.h
> linux/rtnetlink.h
> linux/pkt_cls.h
> linux/pkt_sched.h
> linux/if_ether.h
> linux/tc_act/tc_gact.h
> linux/tc_act/tc_mirred.h
> 
> My initial goal is to change the DPDK code as little as possible, but that
> might not be be possible.  You'd have a much better feel for that than me
> at this point.
> 
> > 
> > We haven't started to work on it because the majority of our use cases are for
> > Linux.   
> 
> I figured as much.  
> 
> > Nevertheless, we can help with code reviews and guidance.  
> 
> Once I get somewhere I will definitely take you up on that.
> 
> Thanks!
> 
> Mit
> 
> >   
> >> >
> >> > Cc also mlx5 maintainers.  
> >> 
> >> Thomas-
> >> 
> >> Thanks for looping the correct people in!
> >>   
> >> >  
> >> >> Though not important, why did Mellanox build a PMD that relied on the
> >> >> kernel driver being in place unlike most other PMDs?  
> >> >
> >> > Because it allows to choose which flows go to DPDK and which ones are
> >> > processed by the kernel.
> >> > Actually, you should ask why other PMDs don't have this feature ;)  
> > 
> > +1.
> >   
> >> 
> >> Very fair response.  We actually wrote our own "shim" into the stack for all
> >> the relevant drivers we use to both keep the existing ifnet interfaces around
> >> and to pass packets to and from the stack from every interface.
> >> 
> >> Your comment matches one of my co-worker's when I explained to him how
> >> the mlx5 driver works.
> >> 
> >> Mit
> >>  

Rather than building a complex shim, why not just have an OS dependent section
of the driver.  The bigger issue is that the kernel side functionality in BSD
is different or missing.  For example TAP API is different and not multi-queue.

Re: mlx5 under FreeBSD

[Mit Matelske]

> 
> Rather than building a complex shim, why not just have an OS dependent section
> of the driver.  The bigger issue is that the kernel side functionality in BSD
> is different or missing.  For example TAP API is different and not multi-queue.

Once I started working on it, I had to go this route.  I haven't been able to but much 
time into it yet, but it is coming along:

EAL: PCI device 0000:b3:00.0 on NUMA socket 0
EAL:   probe driver: 15b3:1013 net_mlx5
net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_0" uverbs0 /sys/class/infiniband_verbs/uverbs0 /sys/class/infiniband/mlx5_0
net_mlx5: mlx5.c:1325: mlx5_pci_probe(): PCI information matches for device "mlx5_0"
net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_1" uverbs1 /sys/class/infiniband_verbs/uverbs1 /sys/class/infiniband/mlx5_1
net_mlx5: mlx5.c:1382: mlx5_pci_probe(): no switch support detected
net_mlx5: mlx5.c:796: mlx5_dev_spawn(): MPW isn't supported
net_mlx5: mlx5.c:852: mlx5_dev_spawn(): tunnel offloading disabled due to old OFED/rdma-core version
net_mlx5: mlx5.c:864: mlx5_dev_spawn(): MPLS over GRE/UDP tunnel offloading disabled due to old OFED/rdma-core version or firmware configuration
net_mlx5: mlx5.c:877: mlx5_dev_spawn(): naming Ethernet device "0000:b3:00.0"
net_mlx5: mlx5.c:1007: mlx5_dev_spawn(): checksum offloading is supported
net_mlx5: mlx5.c:1024: mlx5_dev_spawn(): maximum Rx indirection table size is 512
net_mlx5: mlx5.c:1028: mlx5_dev_spawn(): VLAN stripping is not supported
net_mlx5: mlx5.c:1032: mlx5_dev_spawn(): FCS stripping configuration is not supported
net_mlx5: mlx5.c:1037: mlx5_dev_spawn(): hardware Rx end alignment padding is not supported
net_mlx5: mlx5.c:1052: mlx5_dev_spawn(): MPS is disabled
net_mlx5: mlx5.c:628: mlx5_uar_init_primary(): port 0 reserved UAR address space: 0x880000000
net_mlx5: mlx5.c:1107: mlx5_dev_spawn(): port 0 MAC address is 00:90:0b:78:2a:ea
net_mlx5: mlx5.c:1114: mlx5_dev_spawn(): port 0 ifname is "mce0"
net_mlx5: mlx5.c:1127: mlx5_dev_spawn(): port 0 MTU is 1500
net_mlx5: mlx5.c:1142: mlx5_dev_spawn(): flow rules relying on switch offloads will not be supported: cannot open libmnl socket: Address family not supported by protocol family
net_mlx5: mlx5.c:1176: mlx5_dev_spawn(): port 0 forcing Ethernet interface up
net_mlx5: mlx5_ethdev.c:735: mlx5_link_update_unlocked_gs(): port 0 ioctl(SIOCGIFFLAGS) failed: No such file or directory
net_mlx5: mlx5_ethdev.c:664: mlx5_link_update_unlocked_gset(): port 0 ioctl(SIOCGIFFLAGS) failed: No such file or directory
net_mlx5: mlx5_rxq.c:2150: mlx5_hrxq_drop_new(): port 0 cannot allocate QP for drop queue
Segmentation fault (core dumped)

Working through mlx5_ethdev now...

Mit

Re: mlx5 under FreeBSD

[Stephen Hemminger]

On Wed, 21 Nov 2018 11:11:51 -0600 (CST)
Mit Matelske <mit@pt.net> wrote:

> > 
> > Rather than building a complex shim, why not just have an OS dependent section
> > of the driver.  The bigger issue is that the kernel side functionality in BSD
> > is different or missing.  For example TAP API is different and not multi-queue.  
> 
> Once I started working on it, I had to go this route.  I haven't been able to but much 
> time into it yet, but it is coming along:
> 
> EAL: PCI device 0000:b3:00.0 on NUMA socket 0
> EAL:   probe driver: 15b3:1013 net_mlx5
> net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_0" uverbs0 /sys/class/infiniband_verbs/uverbs0 /sys/class/infiniband/mlx5_0
> net_mlx5: mlx5.c:1325: mlx5_pci_probe(): PCI information matches for device "mlx5_0"
> net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_1" uverbs1 /sys/class/infiniband_verbs/uverbs1 /sys/class/infiniband/mlx5_1
> net_mlx5: mlx5.c:1382: mlx5_pci_probe(): no switch support detected
> net_mlx5: mlx5.c:796: mlx5_dev_spawn(): MPW isn't supported
> net_mlx5: mlx5.c:852: mlx5_dev_spawn(): tunnel offloading disabled due to old OFED/rdma-core version
> net_mlx5: mlx5.c:864: mlx5_dev_spawn(): MPLS over GRE/UDP tunnel offloading disabled due to old OFED/rdma-core version or firmware configuration
> net_mlx5: mlx5.c:877: mlx5_dev_spawn(): naming Ethernet device "0000:b3:00.0"
> net_mlx5: mlx5.c:1007: mlx5_dev_spawn(): checksum offloading is supported
> net_mlx5: mlx5.c:1024: mlx5_dev_spawn(): maximum Rx indirection table size is 512
> net_mlx5: mlx5.c:1028: mlx5_dev_spawn(): VLAN stripping is not supported
> net_mlx5: mlx5.c:1032: mlx5_dev_spawn(): FCS stripping configuration is not supported
> net_mlx5: mlx5.c:1037: mlx5_dev_spawn(): hardware Rx end alignment padding is not supported
> net_mlx5: mlx5.c:1052: mlx5_dev_spawn(): MPS is disabled
> net_mlx5: mlx5.c:628: mlx5_uar_init_primary(): port 0 reserved UAR address space: 0x880000000
> net_mlx5: mlx5.c:1107: mlx5_dev_spawn(): port 0 MAC address is 00:90:0b:78:2a:ea
> net_mlx5: mlx5.c:1114: mlx5_dev_spawn(): port 0 ifname is "mce0"
> net_mlx5: mlx5.c:1127: mlx5_dev_spawn(): port 0 MTU is 1500
> net_mlx5: mlx5.c:1142: mlx5_dev_spawn(): flow rules relying on switch offloads will not be supported: cannot open libmnl socket: Address family not supported by protocol family
> net_mlx5: mlx5.c:1176: mlx5_dev_spawn(): port 0 forcing Ethernet interface up
> net_mlx5: mlx5_ethdev.c:735: mlx5_link_update_unlocked_gs(): port 0 ioctl(SIOCGIFFLAGS) failed: No such file or directory
> net_mlx5: mlx5_ethdev.c:664: mlx5_link_update_unlocked_gset(): port 0 ioctl(SIOCGIFFLAGS) failed: No such file or directory
> net_mlx5: mlx5_rxq.c:2150: mlx5_hrxq_drop_new(): port 0 cannot allocate QP for drop queue
> Segmentation fault (core dumped)
> 
> Working through mlx5_ethdev now...
> 
> Mit

Also if you are targeting Azure or Hyper-V, then don't bother porting TAP/VDEV_NETVSC/Failsafe model
because that is being replaced by netvsc native PMD.  It would be easier to just get that working.
It would mean writing a  uio equivalent driver for FreeBSD.

Re: mlx5 under FreeBSD

[Mit Matelske]

I was wondering if I could lean on the experts to help me out with an issue?  I'm finally testing this port but don't understand why the following error is happening:

net_mlx5: mlx5_rxq.c:2151: mlx5_hrxq_drop_new(): port 0 cannot allocate QP for drop queue

The driver is giving the following error corresponding with that DPDK message:

Feb 28 15:00:44 fb100g1 kernel: mlx5_core0: ERR: mlx5_cmd_check:701:(pid 6819): CREATE_TIR(0x900) op_mod(0x0) failed, status bad parameter(0x3), syndrome (0x569ee6)

The mlx5_core_create_tir function completes successfully for the en driver, just not the ib driver when trying to build the hash tables.  Any ideas?

I've instrumented libmlx5.so.1, mlx5.ko and mlx5ib.ko to debug this and everything looks fine according to my limited knowledge.  Not sure why the card is balking on this action.

Thanks!

Mit Matelske

----- Original Message -----
From: "stephen" <stephen@networkplumber.org>
To: "Mit Matelske" <mit@pt.net>
Cc: "shahafs" <shahafs@mellanox.com>, "Thomas Monjalon" <thomas@monjalon.net>, "dev" <dev@dpdk.org>, "Yongseok Koh" <yskoh@mellanox.com>
Sent: Wednesday, November 21, 2018 2:12:04 PM
Subject: Re: [dpdk-dev] mlx5 under FreeBSD

On Wed, 21 Nov 2018 11:11:51 -0600 (CST)
Mit Matelske <mit@pt.net> wrote:

> > 
> > Rather than building a complex shim, why not just have an OS dependent section
> > of the driver.  The bigger issue is that the kernel side functionality in BSD
> > is different or missing.  For example TAP API is different and not multi-queue.  
> 
> Once I started working on it, I had to go this route.  I haven't been able to but much 
> time into it yet, but it is coming along:
> 
> EAL: PCI device 0000:b3:00.0 on NUMA socket 0
> EAL:   probe driver: 15b3:1013 net_mlx5
> net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_0" uverbs0 /sys/class/infiniband_verbs/uverbs0 /sys/class/infiniband/mlx5_0
> net_mlx5: mlx5.c:1325: mlx5_pci_probe(): PCI information matches for device "mlx5_0"
> net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_1" uverbs1 /sys/class/infiniband_verbs/uverbs1 /sys/class/infiniband/mlx5_1
> net_mlx5: mlx5.c:1382: mlx5_pci_probe(): no switch support detected
> net_mlx5: mlx5.c:796: mlx5_dev_spawn(): MPW isn't supported
> net_mlx5: mlx5.c:852: mlx5_dev_spawn(): tunnel offloading disabled due to old OFED/rdma-core version
> net_mlx5: mlx5.c:864: mlx5_dev_spawn(): MPLS over GRE/UDP tunnel offloading disabled due to old OFED/rdma-core version or firmware configuration
> net_mlx5: mlx5.c:877: mlx5_dev_spawn(): naming Ethernet device "0000:b3:00.0"
> net_mlx5: mlx5.c:1007: mlx5_dev_spawn(): checksum offloading is supported
> net_mlx5: mlx5.c:1024: mlx5_dev_spawn(): maximum Rx indirection table size is 512
> net_mlx5: mlx5.c:1028: mlx5_dev_spawn(): VLAN stripping is not supported
> net_mlx5: mlx5.c:1032: mlx5_dev_spawn(): FCS stripping configuration is not supported
> net_mlx5: mlx5.c:1037: mlx5_dev_spawn(): hardware Rx end alignment padding is not supported
> net_mlx5: mlx5.c:1052: mlx5_dev_spawn(): MPS is disabled
> net_mlx5: mlx5.c:628: mlx5_uar_init_primary(): port 0 reserved UAR address space: 0x880000000
> net_mlx5: mlx5.c:1107: mlx5_dev_spawn(): port 0 MAC address is 00:90:0b:78:2a:ea
> net_mlx5: mlx5.c:1114: mlx5_dev_spawn(): port 0 ifname is "mce0"
> net_mlx5: mlx5.c:1127: mlx5_dev_spawn(): port 0 MTU is 1500
> net_mlx5: mlx5.c:1142: mlx5_dev_spawn(): flow rules relying on switch offloads will not be supported: cannot open libmnl socket: Address family not supported by protocol family
> net_mlx5: mlx5.c:1176: mlx5_dev_spawn(): port 0 forcing Ethernet interface up
> net_mlx5: mlx5_ethdev.c:735: mlx5_link_update_unlocked_gs(): port 0 ioctl(SIOCGIFFLAGS) failed: No such file or directory
> net_mlx5: mlx5_ethdev.c:664: mlx5_link_update_unlocked_gset(): port 0 ioctl(SIOCGIFFLAGS) failed: No such file or directory
> net_mlx5: mlx5_rxq.c:2150: mlx5_hrxq_drop_new(): port 0 cannot allocate QP for drop queue
> Segmentation fault (core dumped)
> 
> Working through mlx5_ethdev now...
> 
> Mit

Also if you are targeting Azure or Hyper-V, then don't bother porting TAP/VDEV_NETVSC/Failsafe model
because that is being replaced by netvsc native PMD.  It would be easier to just get that working.
It would mean writing a  uio equivalent driver for FreeBSD.

Re: mlx5 under FreeBSD

[Shahaf Shuler]

Friday, March 1, 2019 5:58 PM, Mit Matelske:
> Subject: Re: [dpdk-dev] mlx5 under FreeBSD
> 
> I was wondering if I could lean on the experts to help me out with an issue?
> I'm finally testing this port but don't understand why the following error is
> happening:
> 
> net_mlx5: mlx5_rxq.c:2151: mlx5_hrxq_drop_new(): port 0 cannot allocate
> QP for drop queue
> 
> The driver is giving the following error corresponding with that DPDK
> message:
> 
> Feb 28 15:00:44 fb100g1 kernel: mlx5_core0: ERR: mlx5_cmd_check:701:(pid
> 6819): CREATE_TIR(0x900) op_mod(0x0) failed, status bad parameter(0x3),
> syndrome (0x569ee6)

Hard to say why. I tried to look for the syndrome above on FW but couldn't find. Are you sure this is correct one? 
Which FW version you use? 

> 
> The mlx5_core_create_tir function completes successfully for the en driver,
> just not the ib driver when trying to build the hash tables.  Any ideas?
> 
> I've instrumented libmlx5.so.1, mlx5.ko and mlx5ib.ko to debug this and
> everything looks fine according to my limited knowledge.  Not sure why the
> card is balking on this action.
> 
> Thanks!
> 
> Mit Matelske
> 
> ----- Original Message -----
> From: "stephen" <stephen@networkplumber.org>
> To: "Mit Matelske" <mit@pt.net>
> Cc: "shahafs" <shahafs@mellanox.com>, "Thomas Monjalon"
> <thomas@monjalon.net>, "dev" <dev@dpdk.org>, "Yongseok Koh"
> <yskoh@mellanox.com>
> Sent: Wednesday, November 21, 2018 2:12:04 PM
> Subject: Re: [dpdk-dev] mlx5 under FreeBSD
> 
> On Wed, 21 Nov 2018 11:11:51 -0600 (CST) Mit Matelske <mit@pt.net>
> wrote:
> 
> > >
> > > Rather than building a complex shim, why not just have an OS
> > > dependent section of the driver.  The bigger issue is that the
> > > kernel side functionality in BSD is different or missing.  For example TAP
> API is different and not multi-queue.
> >
> > Once I started working on it, I had to go this route.  I haven't been
> > able to but much time into it yet, but it is coming along:
> >
> > EAL: PCI device 0000:b3:00.0 on NUMA socket 0
> > EAL:   probe driver: 15b3:1013 net_mlx5
> > net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_0"
> > uverbs0 /sys/class/infiniband_verbs/uverbs0
> > /sys/class/infiniband/mlx5_0
> > net_mlx5: mlx5.c:1325: mlx5_pci_probe(): PCI information matches for
> device "mlx5_0"
> > net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_1"
> > uverbs1 /sys/class/infiniband_verbs/uverbs1
> > /sys/class/infiniband/mlx5_1
> > net_mlx5: mlx5.c:1382: mlx5_pci_probe(): no switch support detected
> > net_mlx5: mlx5.c:796: mlx5_dev_spawn(): MPW isn't supported
> > net_mlx5: mlx5.c:852: mlx5_dev_spawn(): tunnel offloading disabled due
> > to old OFED/rdma-core version
> > net_mlx5: mlx5.c:864: mlx5_dev_spawn(): MPLS over GRE/UDP tunnel
> > offloading disabled due to old OFED/rdma-core version or firmware
> > configuration
> > net_mlx5: mlx5.c:877: mlx5_dev_spawn(): naming Ethernet device
> "0000:b3:00.0"
> > net_mlx5: mlx5.c:1007: mlx5_dev_spawn(): checksum offloading is
> > supported
> > net_mlx5: mlx5.c:1024: mlx5_dev_spawn(): maximum Rx indirection table
> > size is 512
> > net_mlx5: mlx5.c:1028: mlx5_dev_spawn(): VLAN stripping is not
> > supported
> > net_mlx5: mlx5.c:1032: mlx5_dev_spawn(): FCS stripping configuration
> > is not supported
> > net_mlx5: mlx5.c:1037: mlx5_dev_spawn(): hardware Rx end alignment
> > padding is not supported
> > net_mlx5: mlx5.c:1052: mlx5_dev_spawn(): MPS is disabled
> > net_mlx5: mlx5.c:628: mlx5_uar_init_primary(): port 0 reserved UAR
> > address space: 0x880000000
> > net_mlx5: mlx5.c:1107: mlx5_dev_spawn(): port 0 MAC address is
> > 00:90:0b:78:2a:ea
> > net_mlx5: mlx5.c:1114: mlx5_dev_spawn(): port 0 ifname is "mce0"
> > net_mlx5: mlx5.c:1127: mlx5_dev_spawn(): port 0 MTU is 1500
> > net_mlx5: mlx5.c:1142: mlx5_dev_spawn(): flow rules relying on switch
> > offloads will not be supported: cannot open libmnl socket: Address
> > family not supported by protocol family
> > net_mlx5: mlx5.c:1176: mlx5_dev_spawn(): port 0 forcing Ethernet
> > interface up
> > net_mlx5: mlx5_ethdev.c:735: mlx5_link_update_unlocked_gs(): port 0
> > ioctl(SIOCGIFFLAGS) failed: No such file or directory
> > net_mlx5: mlx5_ethdev.c:664: mlx5_link_update_unlocked_gset(): port 0
> > ioctl(SIOCGIFFLAGS) failed: No such file or directory
> > net_mlx5: mlx5_rxq.c:2150: mlx5_hrxq_drop_new(): port 0 cannot
> > allocate QP for drop queue Segmentation fault (core dumped)
> >
> > Working through mlx5_ethdev now...
> >
> > Mit
> 
> Also if you are targeting Azure or Hyper-V, then don't bother porting
> TAP/VDEV_NETVSC/Failsafe model because that is being replaced by netvsc
> native PMD.  It would be easier to just get that working.
> It would mean writing a  uio equivalent driver for FreeBSD.

Re: mlx5 under FreeBSD

[Mit Matelske]

dev.mlx5_core.1.hw.board_id: LNR3270110033
dev.mlx5_core.1.hw.fw_version: 12.17.2020

Thanks!

Mit Matelske

----- Original Message -----
From: "shahafs" <shahafs@mellanox.com>
To: "Mit Matelske" <mit@pt.net>, "stephen" <stephen@networkplumber.org>
Cc: "Thomas Monjalon" <thomas@monjalon.net>, "dev" <dev@dpdk.org>, "Yongseok Koh" <yskoh@mellanox.com>
Sent: Sunday, March 3, 2019 2:43:41 AM
Subject: RE: [dpdk-dev] mlx5 under FreeBSD

Friday, March 1, 2019 5:58 PM, Mit Matelske:
> Subject: Re: [dpdk-dev] mlx5 under FreeBSD
> 
> I was wondering if I could lean on the experts to help me out with an issue?
> I'm finally testing this port but don't understand why the following error is
> happening:
> 
> net_mlx5: mlx5_rxq.c:2151: mlx5_hrxq_drop_new(): port 0 cannot allocate
> QP for drop queue
> 
> The driver is giving the following error corresponding with that DPDK
> message:
> 
> Feb 28 15:00:44 fb100g1 kernel: mlx5_core0: ERR: mlx5_cmd_check:701:(pid
> 6819): CREATE_TIR(0x900) op_mod(0x0) failed, status bad parameter(0x3),
> syndrome (0x569ee6)

Hard to say why. I tried to look for the syndrome above on FW but couldn't find. Are you sure this is correct one? 
Which FW version you use? 

> 
> The mlx5_core_create_tir function completes successfully for the en driver,
> just not the ib driver when trying to build the hash tables.  Any ideas?
> 
> I've instrumented libmlx5.so.1, mlx5.ko and mlx5ib.ko to debug this and
> everything looks fine according to my limited knowledge.  Not sure why the
> card is balking on this action.
> 
> Thanks!
> 
> Mit Matelske
> 
> ----- Original Message -----
> From: "stephen" <stephen@networkplumber.org>
> To: "Mit Matelske" <mit@pt.net>
> Cc: "shahafs" <shahafs@mellanox.com>, "Thomas Monjalon"
> <thomas@monjalon.net>, "dev" <dev@dpdk.org>, "Yongseok Koh"
> <yskoh@mellanox.com>
> Sent: Wednesday, November 21, 2018 2:12:04 PM
> Subject: Re: [dpdk-dev] mlx5 under FreeBSD
> 
> On Wed, 21 Nov 2018 11:11:51 -0600 (CST) Mit Matelske <mit@pt.net>
> wrote:
> 
> > >
> > > Rather than building a complex shim, why not just have an OS
> > > dependent section of the driver.  The bigger issue is that the
> > > kernel side functionality in BSD is different or missing.  For example TAP
> API is different and not multi-queue.
> >
> > Once I started working on it, I had to go this route.  I haven't been
> > able to but much time into it yet, but it is coming along:
> >
> > EAL: PCI device 0000:b3:00.0 on NUMA socket 0
> > EAL:   probe driver: 15b3:1013 net_mlx5
> > net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_0"
> > uverbs0 /sys/class/infiniband_verbs/uverbs0
> > /sys/class/infiniband/mlx5_0
> > net_mlx5: mlx5.c:1325: mlx5_pci_probe(): PCI information matches for
> device "mlx5_0"
> > net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_1"
> > uverbs1 /sys/class/infiniband_verbs/uverbs1
> > /sys/class/infiniband/mlx5_1
> > net_mlx5: mlx5.c:1382: mlx5_pci_probe(): no switch support detected
> > net_mlx5: mlx5.c:796: mlx5_dev_spawn(): MPW isn't supported
> > net_mlx5: mlx5.c:852: mlx5_dev_spawn(): tunnel offloading disabled due
> > to old OFED/rdma-core version
> > net_mlx5: mlx5.c:864: mlx5_dev_spawn(): MPLS over GRE/UDP tunnel
> > offloading disabled due to old OFED/rdma-core version or firmware
> > configuration
> > net_mlx5: mlx5.c:877: mlx5_dev_spawn(): naming Ethernet device
> "0000:b3:00.0"
> > net_mlx5: mlx5.c:1007: mlx5_dev_spawn(): checksum offloading is
> > supported
> > net_mlx5: mlx5.c:1024: mlx5_dev_spawn(): maximum Rx indirection table
> > size is 512
> > net_mlx5: mlx5.c:1028: mlx5_dev_spawn(): VLAN stripping is not
> > supported
> > net_mlx5: mlx5.c:1032: mlx5_dev_spawn(): FCS stripping configuration
> > is not supported
> > net_mlx5: mlx5.c:1037: mlx5_dev_spawn(): hardware Rx end alignment
> > padding is not supported
> > net_mlx5: mlx5.c:1052: mlx5_dev_spawn(): MPS is disabled
> > net_mlx5: mlx5.c:628: mlx5_uar_init_primary(): port 0 reserved UAR
> > address space: 0x880000000
> > net_mlx5: mlx5.c:1107: mlx5_dev_spawn(): port 0 MAC address is
> > 00:90:0b:78:2a:ea
> > net_mlx5: mlx5.c:1114: mlx5_dev_spawn(): port 0 ifname is "mce0"
> > net_mlx5: mlx5.c:1127: mlx5_dev_spawn(): port 0 MTU is 1500
> > net_mlx5: mlx5.c:1142: mlx5_dev_spawn(): flow rules relying on switch
> > offloads will not be supported: cannot open libmnl socket: Address
> > family not supported by protocol family
> > net_mlx5: mlx5.c:1176: mlx5_dev_spawn(): port 0 forcing Ethernet
> > interface up
> > net_mlx5: mlx5_ethdev.c:735: mlx5_link_update_unlocked_gs(): port 0
> > ioctl(SIOCGIFFLAGS) failed: No such file or directory
> > net_mlx5: mlx5_ethdev.c:664: mlx5_link_update_unlocked_gset(): port 0
> > ioctl(SIOCGIFFLAGS) failed: No such file or directory
> > net_mlx5: mlx5_rxq.c:2150: mlx5_hrxq_drop_new(): port 0 cannot
> > allocate QP for drop queue Segmentation fault (core dumped)
> >
> > Working through mlx5_ethdev now...
> >
> > Mit
> 
> Also if you are targeting Azure or Hyper-V, then don't bother porting
> TAP/VDEV_NETVSC/Failsafe model because that is being replaced by netvsc
> native PMD.  It would be easier to just get that working.
> It would mean writing a  uio equivalent driver for FreeBSD.

Re: mlx5 under FreeBSD

[Shahaf Shuler]

Hi Mit,

Monday, March 4, 2019 6:45 PM, Mit Matelske:
> Subject: Re: [dpdk-dev] mlx5 under FreeBSD
> 
> dev.mlx5_core.1.hw.board_id: LNR3270110033
> dev.mlx5_core.1.hw.fw_version: 12.17.2020

Getting back to you on this one.

>From the FW it looks like you are trying to create a TIR (QP) which doesn't use Toeplitz hash function (IBV_RX_HASH_FUNC_TOEPLITZ) but the rss key is not reserved. 

Makes sense? 

> 
> Thanks!
> 
> Mit Matelske
> 
> ----- Original Message -----
> From: "shahafs" <shahafs@mellanox.com>
> To: "Mit Matelske" <mit@pt.net>, "stephen"
> <stephen@networkplumber.org>
> Cc: "Thomas Monjalon" <thomas@monjalon.net>, "dev" <dev@dpdk.org>,
> "Yongseok Koh" <yskoh@mellanox.com>
> Sent: Sunday, March 3, 2019 2:43:41 AM
> Subject: RE: [dpdk-dev] mlx5 under FreeBSD
> 
> Friday, March 1, 2019 5:58 PM, Mit Matelske:
> > Subject: Re: [dpdk-dev] mlx5 under FreeBSD
> >
> > I was wondering if I could lean on the experts to help me out with an issue?
> > I'm finally testing this port but don't understand why the following
> > error is
> > happening:
> >
> > net_mlx5: mlx5_rxq.c:2151: mlx5_hrxq_drop_new(): port 0 cannot
> > allocate QP for drop queue
> >
> > The driver is giving the following error corresponding with that DPDK
> > message:
> >
> > Feb 28 15:00:44 fb100g1 kernel: mlx5_core0: ERR:
> > mlx5_cmd_check:701:(pid
> > 6819): CREATE_TIR(0x900) op_mod(0x0) failed, status bad
> > parameter(0x3), syndrome (0x569ee6)
> 
> Hard to say why. I tried to look for the syndrome above on FW but couldn't
> find. Are you sure this is correct one?
> Which FW version you use?
> 
> >
> > The mlx5_core_create_tir function completes successfully for the en
> > driver, just not the ib driver when trying to build the hash tables.  Any
> ideas?
> >
> > I've instrumented libmlx5.so.1, mlx5.ko and mlx5ib.ko to debug this
> > and everything looks fine according to my limited knowledge.  Not sure
> > why the card is balking on this action.
> >
> > Thanks!
> >
> > Mit Matelske
> >
> > ----- Original Message -----
> > From: "stephen" <stephen@networkplumber.org>
> > To: "Mit Matelske" <mit@pt.net>
> > Cc: "shahafs" <shahafs@mellanox.com>, "Thomas Monjalon"
> > <thomas@monjalon.net>, "dev" <dev@dpdk.org>, "Yongseok Koh"
> > <yskoh@mellanox.com>
> > Sent: Wednesday, November 21, 2018 2:12:04 PM
> > Subject: Re: [dpdk-dev] mlx5 under FreeBSD
> >
> > On Wed, 21 Nov 2018 11:11:51 -0600 (CST) Mit Matelske <mit@pt.net>
> > wrote:
> >
> > > >
> > > > Rather than building a complex shim, why not just have an OS
> > > > dependent section of the driver.  The bigger issue is that the
> > > > kernel side functionality in BSD is different or missing.  For
> > > > example TAP
> > API is different and not multi-queue.
> > >
> > > Once I started working on it, I had to go this route.  I haven't
> > > been able to but much time into it yet, but it is coming along:
> > >
> > > EAL: PCI device 0000:b3:00.0 on NUMA socket 0
> > > EAL:   probe driver: 15b3:1013 net_mlx5
> > > net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_0"
> > > uverbs0 /sys/class/infiniband_verbs/uverbs0
> > > /sys/class/infiniband/mlx5_0
> > > net_mlx5: mlx5.c:1325: mlx5_pci_probe(): PCI information matches for
> > device "mlx5_0"
> > > net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_1"
> > > uverbs1 /sys/class/infiniband_verbs/uverbs1
> > > /sys/class/infiniband/mlx5_1
> > > net_mlx5: mlx5.c:1382: mlx5_pci_probe(): no switch support detected
> > > net_mlx5: mlx5.c:796: mlx5_dev_spawn(): MPW isn't supported
> > > net_mlx5: mlx5.c:852: mlx5_dev_spawn(): tunnel offloading disabled
> > > due to old OFED/rdma-core version
> > > net_mlx5: mlx5.c:864: mlx5_dev_spawn(): MPLS over GRE/UDP tunnel
> > > offloading disabled due to old OFED/rdma-core version or firmware
> > > configuration
> > > net_mlx5: mlx5.c:877: mlx5_dev_spawn(): naming Ethernet device
> > "0000:b3:00.0"
> > > net_mlx5: mlx5.c:1007: mlx5_dev_spawn(): checksum offloading is
> > > supported
> > > net_mlx5: mlx5.c:1024: mlx5_dev_spawn(): maximum Rx indirection
> > > table size is 512
> > > net_mlx5: mlx5.c:1028: mlx5_dev_spawn(): VLAN stripping is not
> > > supported
> > > net_mlx5: mlx5.c:1032: mlx5_dev_spawn(): FCS stripping configuration
> > > is not supported
> > > net_mlx5: mlx5.c:1037: mlx5_dev_spawn(): hardware Rx end alignment
> > > padding is not supported
> > > net_mlx5: mlx5.c:1052: mlx5_dev_spawn(): MPS is disabled
> > > net_mlx5: mlx5.c:628: mlx5_uar_init_primary(): port 0 reserved UAR
> > > address space: 0x880000000
> > > net_mlx5: mlx5.c:1107: mlx5_dev_spawn(): port 0 MAC address is
> > > 00:90:0b:78:2a:ea
> > > net_mlx5: mlx5.c:1114: mlx5_dev_spawn(): port 0 ifname is "mce0"
> > > net_mlx5: mlx5.c:1127: mlx5_dev_spawn(): port 0 MTU is 1500
> > > net_mlx5: mlx5.c:1142: mlx5_dev_spawn(): flow rules relying on
> > > switch offloads will not be supported: cannot open libmnl socket:
> > > Address family not supported by protocol family
> > > net_mlx5: mlx5.c:1176: mlx5_dev_spawn(): port 0 forcing Ethernet
> > > interface up
> > > net_mlx5: mlx5_ethdev.c:735: mlx5_link_update_unlocked_gs(): port 0
> > > ioctl(SIOCGIFFLAGS) failed: No such file or directory
> > > net_mlx5: mlx5_ethdev.c:664: mlx5_link_update_unlocked_gset(): port
> > > 0
> > > ioctl(SIOCGIFFLAGS) failed: No such file or directory
> > > net_mlx5: mlx5_rxq.c:2150: mlx5_hrxq_drop_new(): port 0 cannot
> > > allocate QP for drop queue Segmentation fault (core dumped)
> > >
> > > Working through mlx5_ethdev now...
> > >
> > > Mit
> >
> > Also if you are targeting Azure or Hyper-V, then don't bother porting
> > TAP/VDEV_NETVSC/Failsafe model because that is being replaced by
> > netvsc native PMD.  It would be easier to just get that working.
> > It would mean writing a  uio equivalent driver for FreeBSD.

Re: mlx5 under FreeBSD

[Mit Matelske]

> > 
> > dev.mlx5_core.1.hw.board_id: LNR3270110033
> > dev.mlx5_core.1.hw.fw_version: 12.17.2020
>
> Getting back to you on this one.
>
> From the FW it looks like you are trying to create a TIR (QP) which doesn't use Toeplitz hash function (IBV_RX_HASH_FUNC_TOEPLITZ) but the rss key is not reserved. 
> 
> Makes sense? 

Kinda, except I believe the DPDK code is trying to create a TIR with IBV_RX_HASH_FUNC_TOEPLITZ set and the RSS key defined:


   qp = mlx5_glue->create_qp_ex(priv->ctx,
       &(struct ibv_qp_init_attr_ex){
         .qp_type = IBV_QPT_RAW_PACKET,
         .comp_mask =
            IBV_QP_INIT_ATTR_PD |
            IBV_QP_INIT_ATTR_IND_TABLE |
            IBV_QP_INIT_ATTR_RX_HASH,
         .rx_hash_conf = (struct ibv_rx_hash_conf){
            .rx_hash_function =
               IBV_RX_HASH_FUNC_TOEPLITZ,
            .rx_hash_key_len = MLX5_RSS_HASH_KEY_LEN,
            .rx_hash_key = rss_hash_default_key,
            .rx_hash_fields_mask = 0,
            },
         .rwq_ind_tbl = ind_tbl->ind_table,
         .pd = priv->pd
       });


That call goes into create_qp in libmlx5 which eventually makes the following call:


  ret = ibv_cmd_create_qp_ex2(context, &qp->verbs_qp,
                   sizeof(qp->verbs_qp), attr,
                   &cmd_ex_rss.ibv_cmd, sizeof(cmd_ex_rss.ibv_cmd),
                   sizeof(cmd_ex_rss), &resp.ibv_resp,
                   sizeof(resp.ibv_resp), sizeof(resp));

Relevant variables from that call:

cmd_ex_rss.rx_hash_function: 1
cmd_ex_rss.rx_key_len: 40
cmd_ex_rss.rx_hash_key:
2C C6 81 D1 5B DB F4 F7 FC A2 
83 19 DB 1A 3E 94 6B 9E 38 D9 
2C 9C 03 D1 AD 99 44 A7 D9 56 
3D 59 06 3C 25 F3 FC 1F DC 2A 

All this is "stock" FB 12.0 and DPDK 18.08 code, FWIW.  

Any and all help is appreciated.  I'm a little out of my depths with this.

Thanks!

Mit Matelske

> 
> Thanks!
> 
> Mit Matelske
> 
> ----- Original Message -----
> From: "shahafs" <shahafs@mellanox.com>
> To: "Mit Matelske" <mit@pt.net>, "stephen"
> <stephen@networkplumber.org>
> Cc: "Thomas Monjalon" <thomas@monjalon.net>, "dev" <dev@dpdk.org>,
> "Yongseok Koh" <yskoh@mellanox.com>
> Sent: Sunday, March 3, 2019 2:43:41 AM
> Subject: RE: [dpdk-dev] mlx5 under FreeBSD
> 
> Friday, March 1, 2019 5:58 PM, Mit Matelske:
> > Subject: Re: [dpdk-dev] mlx5 under FreeBSD
> >
> > I was wondering if I could lean on the experts to help me out with an issue?
> > I'm finally testing this port but don't understand why the following
> > error is
> > happening:
> >
> > net_mlx5: mlx5_rxq.c:2151: mlx5_hrxq_drop_new(): port 0 cannot
> > allocate QP for drop queue
> >
> > The driver is giving the following error corresponding with that DPDK
> > message:
> >
> > Feb 28 15:00:44 fb100g1 kernel: mlx5_core0: ERR:
> > mlx5_cmd_check:701:(pid
> > 6819): CREATE_TIR(0x900) op_mod(0x0) failed, status bad
> > parameter(0x3), syndrome (0x569ee6)
> 
> Hard to say why. I tried to look for the syndrome above on FW but couldn't
> find. Are you sure this is correct one?
> Which FW version you use?
> 
> >
> > The mlx5_core_create_tir function completes successfully for the en
> > driver, just not the ib driver when trying to build the hash tables.  Any
> ideas?
> >
> > I've instrumented libmlx5.so.1, mlx5.ko and mlx5ib.ko to debug this
> > and everything looks fine according to my limited knowledge.  Not sure
> > why the card is balking on this action.
> >
> > Thanks!
> >
> > Mit Matelske
> >
> > ----- Original Message -----
> > From: "stephen" <stephen@networkplumber.org>
> > To: "Mit Matelske" <mit@pt.net>
> > Cc: "shahafs" <shahafs@mellanox.com>, "Thomas Monjalon"
> > <thomas@monjalon.net>, "dev" <dev@dpdk.org>, "Yongseok Koh"
> > <yskoh@mellanox.com>
> > Sent: Wednesday, November 21, 2018 2:12:04 PM
> > Subject: Re: [dpdk-dev] mlx5 under FreeBSD
> >
> > On Wed, 21 Nov 2018 11:11:51 -0600 (CST) Mit Matelske <mit@pt.net>
> > wrote:
> >
> > > >
> > > > Rather than building a complex shim, why not just have an OS
> > > > dependent section of the driver.  The bigger issue is that the
> > > > kernel side functionality in BSD is different or missing.  For
> > > > example TAP
> > API is different and not multi-queue.
> > >
> > > Once I started working on it, I had to go this route.  I haven't
> > > been able to but much time into it yet, but it is coming along:
> > >
> > > EAL: PCI device 0000:b3:00.0 on NUMA socket 0
> > > EAL:   probe driver: 15b3:1013 net_mlx5
> > > net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_0"
> > > uverbs0 /sys/class/infiniband_verbs/uverbs0
> > > /sys/class/infiniband/mlx5_0
> > > net_mlx5: mlx5.c:1325: mlx5_pci_probe(): PCI information matches for
> > device "mlx5_0"
> > > net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_1"
> > > uverbs1 /sys/class/infiniband_verbs/uverbs1
> > > /sys/class/infiniband/mlx5_1
> > > net_mlx5: mlx5.c:1382: mlx5_pci_probe(): no switch support detected
> > > net_mlx5: mlx5.c:796: mlx5_dev_spawn(): MPW isn't supported
> > > net_mlx5: mlx5.c:852: mlx5_dev_spawn(): tunnel offloading disabled
> > > due to old OFED/rdma-core version
> > > net_mlx5: mlx5.c:864: mlx5_dev_spawn(): MPLS over GRE/UDP tunnel
> > > offloading disabled due to old OFED/rdma-core version or firmware
> > > configuration
> > > net_mlx5: mlx5.c:877: mlx5_dev_spawn(): naming Ethernet device
> > "0000:b3:00.0"
> > > net_mlx5: mlx5.c:1007: mlx5_dev_spawn(): checksum offloading is
> > > supported
> > > net_mlx5: mlx5.c:1024: mlx5_dev_spawn(): maximum Rx indirection
> > > table size is 512
> > > net_mlx5: mlx5.c:1028: mlx5_dev_spawn(): VLAN stripping is not
> > > supported
> > > net_mlx5: mlx5.c:1032: mlx5_dev_spawn(): FCS stripping configuration
> > > is not supported
> > > net_mlx5: mlx5.c:1037: mlx5_dev_spawn(): hardware Rx end alignment
> > > padding is not supported
> > > net_mlx5: mlx5.c:1052: mlx5_dev_spawn(): MPS is disabled
> > > net_mlx5: mlx5.c:628: mlx5_uar_init_primary(): port 0 reserved UAR
> > > address space: 0x880000000
> > > net_mlx5: mlx5.c:1107: mlx5_dev_spawn(): port 0 MAC address is
> > > 00:90:0b:78:2a:ea
> > > net_mlx5: mlx5.c:1114: mlx5_dev_spawn(): port 0 ifname is "mce0"
> > > net_mlx5: mlx5.c:1127: mlx5_dev_spawn(): port 0 MTU is 1500
> > > net_mlx5: mlx5.c:1142: mlx5_dev_spawn(): flow rules relying on
> > > switch offloads will not be supported: cannot open libmnl socket:
> > > Address family not supported by protocol family
> > > net_mlx5: mlx5.c:1176: mlx5_dev_spawn(): port 0 forcing Ethernet
> > > interface up
> > > net_mlx5: mlx5_ethdev.c:735: mlx5_link_update_unlocked_gs(): port 0
> > > ioctl(SIOCGIFFLAGS) failed: No such file or directory
> > > net_mlx5: mlx5_ethdev.c:664: mlx5_link_update_unlocked_gset(): port
> > > 0
> > > ioctl(SIOCGIFFLAGS) failed: No such file or directory
> > > net_mlx5: mlx5_rxq.c:2150: mlx5_hrxq_drop_new(): port 0 cannot
> > > allocate QP for drop queue Segmentation fault (core dumped)
> > >
> > > Working through mlx5_ethdev now...
> > >
> > > Mit
> >
> > Also if you are targeting Azure or Hyper-V, then don't bother porting
> > TAP/VDEV_NETVSC/Failsafe model because that is being replaced by
> > netvsc native PMD.  It would be easier to just get that working.
> > It would mean writing a  uio equivalent driver for FreeBSD.

Re: mlx5 under FreeBSD

[Shahaf Shuler]

Wednesday, March 6, 2019 11:02 PM, Mit Matelske:
> Subject: Re: [dpdk-dev] mlx5 under FreeBSD
> 
> > >
> > > dev.mlx5_core.1.hw.board_id: LNR3270110033
> > > dev.mlx5_core.1.hw.fw_version: 12.17.2020
> >
> > Getting back to you on this one.
> >
> > From the FW it looks like you are trying to create a TIR (QP) which doesn't
> use Toeplitz hash function (IBV_RX_HASH_FUNC_TOEPLITZ) but the rss key
> is not reserved.
> >
> > Makes sense?
> 
> Kinda, except I believe the DPDK code is trying to create a TIR with
> IBV_RX_HASH_FUNC_TOEPLITZ set and the RSS key defined:
> 
> 
>    qp = mlx5_glue->create_qp_ex(priv->ctx,
>        &(struct ibv_qp_init_attr_ex){
>          .qp_type = IBV_QPT_RAW_PACKET,
>          .comp_mask =
>             IBV_QP_INIT_ATTR_PD |
>             IBV_QP_INIT_ATTR_IND_TABLE |
>             IBV_QP_INIT_ATTR_RX_HASH,
>          .rx_hash_conf = (struct ibv_rx_hash_conf){
>             .rx_hash_function =
>                IBV_RX_HASH_FUNC_TOEPLITZ,
>             .rx_hash_key_len = MLX5_RSS_HASH_KEY_LEN,
>             .rx_hash_key = rss_hash_default_key,
>             .rx_hash_fields_mask = 0,
>             },
>          .rwq_ind_tbl = ind_tbl->ind_table,
>          .pd = priv->pd
>        });
> 
> 
> That call goes into create_qp in libmlx5 which eventually makes the following
> call:
> 
> 
>   ret = ibv_cmd_create_qp_ex2(context, &qp->verbs_qp,
>                    sizeof(qp->verbs_qp), attr,
>                    &cmd_ex_rss.ibv_cmd, sizeof(cmd_ex_rss.ibv_cmd),
>                    sizeof(cmd_ex_rss), &resp.ibv_resp,
>                    sizeof(resp.ibv_resp), sizeof(resp));
> 
> Relevant variables from that call:
> 
> cmd_ex_rss.rx_hash_function: 1
> cmd_ex_rss.rx_key_len: 40
> cmd_ex_rss.rx_hash_key:
> 2C C6 81 D1 5B DB F4 F7 FC A2
> 83 19 DB 1A 3E 94 6B 9E 38 D9
> 2C 9C 03 D1 AD 99 44 A7 D9 56
> 3D 59 06 3C 25 F3 FC 1F DC 2A

What you print here is the verbs layer, and looks OK. I am more interested on the values before the command reach the device. 
Can you print the values of: rx_hash_fn and rx_hash_toeplitz_key from the CREATE_TIR input mailbox command?

Relevant function on mlx5_ib kernel module are:  mlx5_core_create_tir or create_rss_raw_qp_tir.

> 
> All this is "stock" FB 12.0 and DPDK 18.08 code, FWIW.
> 
> Any and all help is appreciated.  I'm a little out of my depths with this.
> 
> Thanks!
> 
> Mit Matelske
> 
> >
> > Thanks!
> >
> > Mit Matelske
> >
> > ----- Original Message -----
> > From: "shahafs" <shahafs@mellanox.com>
> > To: "Mit Matelske" <mit@pt.net>, "stephen"
> > <stephen@networkplumber.org>
> > Cc: "Thomas Monjalon" <thomas@monjalon.net>, "dev"
> <dev@dpdk.org>,
> > "Yongseok Koh" <yskoh@mellanox.com>
> > Sent: Sunday, March 3, 2019 2:43:41 AM
> > Subject: RE: [dpdk-dev] mlx5 under FreeBSD
> >
> > Friday, March 1, 2019 5:58 PM, Mit Matelske:
> > > Subject: Re: [dpdk-dev] mlx5 under FreeBSD
> > >
> > > I was wondering if I could lean on the experts to help me out with an
> issue?
> > > I'm finally testing this port but don't understand why the following
> > > error is
> > > happening:
> > >
> > > net_mlx5: mlx5_rxq.c:2151: mlx5_hrxq_drop_new(): port 0 cannot
> > > allocate QP for drop queue
> > >
> > > The driver is giving the following error corresponding with that
> > > DPDK
> > > message:
> > >
> > > Feb 28 15:00:44 fb100g1 kernel: mlx5_core0: ERR:
> > > mlx5_cmd_check:701:(pid
> > > 6819): CREATE_TIR(0x900) op_mod(0x0) failed, status bad
> > > parameter(0x3), syndrome (0x569ee6)
> >
> > Hard to say why. I tried to look for the syndrome above on FW but
> > couldn't find. Are you sure this is correct one?
> > Which FW version you use?
> >
> > >
> > > The mlx5_core_create_tir function completes successfully for the en
> > > driver, just not the ib driver when trying to build the hash tables.
> > > Any
> > ideas?
> > >
> > > I've instrumented libmlx5.so.1, mlx5.ko and mlx5ib.ko to debug this
> > > and everything looks fine according to my limited knowledge.  Not
> > > sure why the card is balking on this action.
> > >
> > > Thanks!
> > >
> > > Mit Matelske
> > >
> > > ----- Original Message -----
> > > From: "stephen" <stephen@networkplumber.org>
> > > To: "Mit Matelske" <mit@pt.net>
> > > Cc: "shahafs" <shahafs@mellanox.com>, "Thomas Monjalon"
> > > <thomas@monjalon.net>, "dev" <dev@dpdk.org>, "Yongseok Koh"
> > > <yskoh@mellanox.com>
> > > Sent: Wednesday, November 21, 2018 2:12:04 PM
> > > Subject: Re: [dpdk-dev] mlx5 under FreeBSD
> > >
> > > On Wed, 21 Nov 2018 11:11:51 -0600 (CST) Mit Matelske <mit@pt.net>
> > > wrote:
> > >
> > > > >
> > > > > Rather than building a complex shim, why not just have an OS
> > > > > dependent section of the driver.  The bigger issue is that the
> > > > > kernel side functionality in BSD is different or missing.  For
> > > > > example TAP
> > > API is different and not multi-queue.
> > > >
> > > > Once I started working on it, I had to go this route.  I haven't
> > > > been able to but much time into it yet, but it is coming along:
> > > >
> > > > EAL: PCI device 0000:b3:00.0 on NUMA socket 0
> > > > EAL:   probe driver: 15b3:1013 net_mlx5
> > > > net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_0"
> > > > uverbs0 /sys/class/infiniband_verbs/uverbs0
> > > > /sys/class/infiniband/mlx5_0
> > > > net_mlx5: mlx5.c:1325: mlx5_pci_probe(): PCI information matches
> > > > for
> > > device "mlx5_0"
> > > > net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_1"
> > > > uverbs1 /sys/class/infiniband_verbs/uverbs1
> > > > /sys/class/infiniband/mlx5_1
> > > > net_mlx5: mlx5.c:1382: mlx5_pci_probe(): no switch support
> > > > detected
> > > > net_mlx5: mlx5.c:796: mlx5_dev_spawn(): MPW isn't supported
> > > > net_mlx5: mlx5.c:852: mlx5_dev_spawn(): tunnel offloading disabled
> > > > due to old OFED/rdma-core version
> > > > net_mlx5: mlx5.c:864: mlx5_dev_spawn(): MPLS over GRE/UDP tunnel
> > > > offloading disabled due to old OFED/rdma-core version or firmware
> > > > configuration
> > > > net_mlx5: mlx5.c:877: mlx5_dev_spawn(): naming Ethernet device
> > > "0000:b3:00.0"
> > > > net_mlx5: mlx5.c:1007: mlx5_dev_spawn(): checksum offloading is
> > > > supported
> > > > net_mlx5: mlx5.c:1024: mlx5_dev_spawn(): maximum Rx indirection
> > > > table size is 512
> > > > net_mlx5: mlx5.c:1028: mlx5_dev_spawn(): VLAN stripping is not
> > > > supported
> > > > net_mlx5: mlx5.c:1032: mlx5_dev_spawn(): FCS stripping
> > > > configuration is not supported
> > > > net_mlx5: mlx5.c:1037: mlx5_dev_spawn(): hardware Rx end alignment
> > > > padding is not supported
> > > > net_mlx5: mlx5.c:1052: mlx5_dev_spawn(): MPS is disabled
> > > > net_mlx5: mlx5.c:628: mlx5_uar_init_primary(): port 0 reserved UAR
> > > > address space: 0x880000000
> > > > net_mlx5: mlx5.c:1107: mlx5_dev_spawn(): port 0 MAC address is
> > > > 00:90:0b:78:2a:ea
> > > > net_mlx5: mlx5.c:1114: mlx5_dev_spawn(): port 0 ifname is "mce0"
> > > > net_mlx5: mlx5.c:1127: mlx5_dev_spawn(): port 0 MTU is 1500
> > > > net_mlx5: mlx5.c:1142: mlx5_dev_spawn(): flow rules relying on
> > > > switch offloads will not be supported: cannot open libmnl socket:
> > > > Address family not supported by protocol family
> > > > net_mlx5: mlx5.c:1176: mlx5_dev_spawn(): port 0 forcing Ethernet
> > > > interface up
> > > > net_mlx5: mlx5_ethdev.c:735: mlx5_link_update_unlocked_gs(): port
> > > > 0
> > > > ioctl(SIOCGIFFLAGS) failed: No such file or directory
> > > > net_mlx5: mlx5_ethdev.c:664: mlx5_link_update_unlocked_gset():
> > > > port
> > > > 0
> > > > ioctl(SIOCGIFFLAGS) failed: No such file or directory
> > > > net_mlx5: mlx5_rxq.c:2150: mlx5_hrxq_drop_new(): port 0 cannot
> > > > allocate QP for drop queue Segmentation fault (core dumped)
> > > >
> > > > Working through mlx5_ethdev now...
> > > >
> > > > Mit
> > >
> > > Also if you are targeting Azure or Hyper-V, then don't bother
> > > porting TAP/VDEV_NETVSC/Failsafe model because that is being
> > > replaced by netvsc native PMD.  It would be easier to just get that
> working.
> > > It would mean writing a  uio equivalent driver for FreeBSD.

Re: mlx5 under FreeBSD

[Mit Matelske]

> 
> ----- Original Message -----
> From: "shahafs" <shahafs@mellanox.com>
> To: "Mit Matelske" <mit@pt.net>
> 
> What you print here is the verbs layer, and looks OK. I am more interested on the values before the command reach the device. 
> Can you print the values of: rx_hash_fn and rx_hash_toeplitz_key from the CREATE_TIR input mailbox command?
> 
> Relevant function on mlx5_ib kernel module are:  mlx5_core_create_tir or create_rss_raw_qp_tir.

In create_rss_raw_qp_tir, starting at line 1374 I added the following printfs:

   switch (ucmd.rx_hash_function) {
   case MLX5_RX_HASH_FUNC_TOEPLITZ:
   {
      void *rss_key = MLX5_ADDR_OF(tirc, tirc, rx_hash_toeplitz_key);
      size_t len = MLX5_FLD_SZ_BYTES(tirc, rx_hash_toeplitz_key);

      if (len != ucmd.rx_key_len) {
         err = -EINVAL;
         goto err;
      }

printf("setting MLX5_RX_HASH_FUNC_TOEPLITZ\n");
      MLX5_SET(tirc, tirc, rx_hash_fn, MLX5_RX_HASH_FUNC_TOEPLITZ);
      MLX5_SET(tirc, tirc, rx_hash_symmetric, 1);
      memcpy(rss_key, ucmd.rx_hash_key, len);
printf("rx_hash_toeplitz_key(%ld):\n", len);
for(int ix = 0; ix < len; ++ix) {
   printf("%02X ", ((u_char *)rss_key)[ix]);
   if (((ix + 1) % 10) == 0) printf("\n");
}
      break;
   }
   default:
      err = -EOPNOTSUPP;
      goto err;
   }


Which resulted in:

Mar  7 09:25:56 fb100g1 kernel: setting MLX5_RX_HASH_FUNC_TOEPLITZ
Mar  7 09:25:56 fb100g1 kernel: rx_hash_toeplitz_key(40):
Mar  7 09:25:56 fb100g1 kernel: 2C C6 81 D1 5B DB F4 F7 FC A2 
Mar  7 09:25:56 fb100g1 kernel: 83 19 DB 1A 3E 94 6B 9E 38 D9 
Mar  7 09:25:56 fb100g1 kernel: 2C 9C 03 D1 AD 99 44 A7 D9 56 
Mar  7 09:25:56 fb100g1 kernel: 3D 59 06 3C 25 F3 FC 1F DC 2A 

So, to me, everything looks good?

Thanks again!

Mit Matelske



> 
> All this is "stock" FB 12.0 and DPDK 18.08 code, FWIW.
> 
> Any and all help is appreciated.  I'm a little out of my depths with this.
> 
> Thanks!
> 
> Mit Matelske
> 
> >
> > Thanks!
> >
> > Mit Matelske
> >
> > ----- Original Message -----
> > From: "shahafs" <shahafs@mellanox.com>
> > To: "Mit Matelske" <mit@pt.net>, "stephen"
> > <stephen@networkplumber.org>
> > Cc: "Thomas Monjalon" <thomas@monjalon.net>, "dev"
> <dev@dpdk.org>,
> > "Yongseok Koh" <yskoh@mellanox.com>
> > Sent: Sunday, March 3, 2019 2:43:41 AM
> > Subject: RE: [dpdk-dev] mlx5 under FreeBSD
> >
> > Friday, March 1, 2019 5:58 PM, Mit Matelske:
> > > Subject: Re: [dpdk-dev] mlx5 under FreeBSD
> > >
> > > I was wondering if I could lean on the experts to help me out with an
> issue?
> > > I'm finally testing this port but don't understand why the following
> > > error is
> > > happening:
> > >
> > > net_mlx5: mlx5_rxq.c:2151: mlx5_hrxq_drop_new(): port 0 cannot
> > > allocate QP for drop queue
> > >
> > > The driver is giving the following error corresponding with that
> > > DPDK
> > > message:
> > >
> > > Feb 28 15:00:44 fb100g1 kernel: mlx5_core0: ERR:
> > > mlx5_cmd_check:701:(pid
> > > 6819): CREATE_TIR(0x900) op_mod(0x0) failed, status bad
> > > parameter(0x3), syndrome (0x569ee6)
> >
> > Hard to say why. I tried to look for the syndrome above on FW but
> > couldn't find. Are you sure this is correct one?
> > Which FW version you use?
> >
> > >
> > > The mlx5_core_create_tir function completes successfully for the en
> > > driver, just not the ib driver when trying to build the hash tables.
> > > Any
> > ideas?
> > >
> > > I've instrumented libmlx5.so.1, mlx5.ko and mlx5ib.ko to debug this
> > > and everything looks fine according to my limited knowledge.  Not
> > > sure why the card is balking on this action.
> > >
> > > Thanks!
> > >
> > > Mit Matelske
> > >
> > > ----- Original Message -----
> > > From: "stephen" <stephen@networkplumber.org>
> > > To: "Mit Matelske" <mit@pt.net>
> > > Cc: "shahafs" <shahafs@mellanox.com>, "Thomas Monjalon"
> > > <thomas@monjalon.net>, "dev" <dev@dpdk.org>, "Yongseok Koh"
> > > <yskoh@mellanox.com>
> > > Sent: Wednesday, November 21, 2018 2:12:04 PM
> > > Subject: Re: [dpdk-dev] mlx5 under FreeBSD
> > >
> > > On Wed, 21 Nov 2018 11:11:51 -0600 (CST) Mit Matelske <mit@pt.net>
> > > wrote:
> > >
> > > > >
> > > > > Rather than building a complex shim, why not just have an OS
> > > > > dependent section of the driver.  The bigger issue is that the
> > > > > kernel side functionality in BSD is different or missing.  For
> > > > > example TAP
> > > API is different and not multi-queue.
> > > >
> > > > Once I started working on it, I had to go this route.  I haven't
> > > > been able to but much time into it yet, but it is coming along:
> > > >
> > > > EAL: PCI device 0000:b3:00.0 on NUMA socket 0
> > > > EAL:   probe driver: 15b3:1013 net_mlx5
> > > > net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_0"
> > > > uverbs0 /sys/class/infiniband_verbs/uverbs0
> > > > /sys/class/infiniband/mlx5_0
> > > > net_mlx5: mlx5.c:1325: mlx5_pci_probe(): PCI information matches
> > > > for
> > > device "mlx5_0"
> > > > net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_1"
> > > > uverbs1 /sys/class/infiniband_verbs/uverbs1
> > > > /sys/class/infiniband/mlx5_1
> > > > net_mlx5: mlx5.c:1382: mlx5_pci_probe(): no switch support
> > > > detected
> > > > net_mlx5: mlx5.c:796: mlx5_dev_spawn(): MPW isn't supported
> > > > net_mlx5: mlx5.c:852: mlx5_dev_spawn(): tunnel offloading disabled
> > > > due to old OFED/rdma-core version
> > > > net_mlx5: mlx5.c:864: mlx5_dev_spawn(): MPLS over GRE/UDP tunnel
> > > > offloading disabled due to old OFED/rdma-core version or firmware
> > > > configuration
> > > > net_mlx5: mlx5.c:877: mlx5_dev_spawn(): naming Ethernet device
> > > "0000:b3:00.0"
> > > > net_mlx5: mlx5.c:1007: mlx5_dev_spawn(): checksum offloading is
> > > > supported
> > > > net_mlx5: mlx5.c:1024: mlx5_dev_spawn(): maximum Rx indirection
> > > > table size is 512
> > > > net_mlx5: mlx5.c:1028: mlx5_dev_spawn(): VLAN stripping is not
> > > > supported
> > > > net_mlx5: mlx5.c:1032: mlx5_dev_spawn(): FCS stripping
> > > > configuration is not supported
> > > > net_mlx5: mlx5.c:1037: mlx5_dev_spawn(): hardware Rx end alignment
> > > > padding is not supported
> > > > net_mlx5: mlx5.c:1052: mlx5_dev_spawn(): MPS is disabled
> > > > net_mlx5: mlx5.c:628: mlx5_uar_init_primary(): port 0 reserved UAR
> > > > address space: 0x880000000
> > > > net_mlx5: mlx5.c:1107: mlx5_dev_spawn(): port 0 MAC address is
> > > > 00:90:0b:78:2a:ea
> > > > net_mlx5: mlx5.c:1114: mlx5_dev_spawn(): port 0 ifname is "mce0"
> > > > net_mlx5: mlx5.c:1127: mlx5_dev_spawn(): port 0 MTU is 1500
> > > > net_mlx5: mlx5.c:1142: mlx5_dev_spawn(): flow rules relying on
> > > > switch offloads will not be supported: cannot open libmnl socket:
> > > > Address family not supported by protocol family
> > > > net_mlx5: mlx5.c:1176: mlx5_dev_spawn(): port 0 forcing Ethernet
> > > > interface up
> > > > net_mlx5: mlx5_ethdev.c:735: mlx5_link_update_unlocked_gs(): port
> > > > 0
> > > > ioctl(SIOCGIFFLAGS) failed: No such file or directory
> > > > net_mlx5: mlx5_ethdev.c:664: mlx5_link_update_unlocked_gset():
> > > > port
> > > > 0
> > > > ioctl(SIOCGIFFLAGS) failed: No such file or directory
> > > > net_mlx5: mlx5_rxq.c:2150: mlx5_hrxq_drop_new(): port 0 cannot
> > > > allocate QP for drop queue Segmentation fault (core dumped)
> > > >
> > > > Working through mlx5_ethdev now...
> > > >
> > > > Mit
> > >
> > > Also if you are targeting Azure or Hyper-V, then don't bother
> > > porting TAP/VDEV_NETVSC/Failsafe model because that is being
> > > replaced by netvsc native PMD.  It would be easier to just get that
> working.
> > > It would mean writing a  uio equivalent driver for FreeBSD.

Re: mlx5 under FreeBSD

[Shahaf Shuler]

Thursday, March 7, 2019 5:28 PM, Mit Matelske:
> Subject: Re: [dpdk-dev] mlx5 under FreeBSD
> 
> 
> >
> > ----- Original Message -----
> > From: "shahafs" <shahafs@mellanox.com>
> > To: "Mit Matelske" <mit@pt.net>
> >
> > What you print here is the verbs layer, and looks OK. I am more interested
> on the values before the command reach the device.
> > Can you print the values of: rx_hash_fn and rx_hash_toeplitz_key from the
> CREATE_TIR input mailbox command?
> >
> > Relevant function on mlx5_ib kernel module are:  mlx5_core_create_tir or
> create_rss_raw_qp_tir.
> 
> In create_rss_raw_qp_tir, starting at line 1374 I added the following printfs:
> 
>    switch (ucmd.rx_hash_function) {
>    case MLX5_RX_HASH_FUNC_TOEPLITZ:
>    {
>       void *rss_key = MLX5_ADDR_OF(tirc, tirc, rx_hash_toeplitz_key);
>       size_t len = MLX5_FLD_SZ_BYTES(tirc, rx_hash_toeplitz_key);
> 
>       if (len != ucmd.rx_key_len) {
>          err = -EINVAL;
>          goto err;
>       }
> 
> printf("setting MLX5_RX_HASH_FUNC_TOEPLITZ\n");
>       MLX5_SET(tirc, tirc, rx_hash_fn, MLX5_RX_HASH_FUNC_TOEPLITZ);
>       MLX5_SET(tirc, tirc, rx_hash_symmetric, 1);
>       memcpy(rss_key, ucmd.rx_hash_key, len);
> printf("rx_hash_toeplitz_key(%ld):\n", len); for(int ix = 0; ix < len; ++ix) {
>    printf("%02X ", ((u_char *)rss_key)[ix]);
>    if (((ix + 1) % 10) == 0) printf("\n"); }
>       break;
>    }
>    default:
>       err = -EOPNOTSUPP;
>       goto err;
>    }
> 
> 
> Which resulted in:
> 
> Mar  7 09:25:56 fb100g1 kernel: setting MLX5_RX_HASH_FUNC_TOEPLITZ
> Mar  7 09:25:56 fb100g1 kernel: rx_hash_toeplitz_key(40):
> Mar  7 09:25:56 fb100g1 kernel: 2C C6 81 D1 5B DB F4 F7 FC A2 Mar  7 09:25:56
> fb100g1 kernel: 83 19 DB 1A 3E 94 6B 9E 38 D9 Mar  7 09:25:56 fb100g1 kernel:
> 2C 9C 03 D1 AD 99 44 A7 D9 56 Mar  7 09:25:56 fb100g1 kernel: 3D 59 06 3C 25
> F3 FC 1F DC 2A
> 
> So, to me, everything looks good?

Yes it looks good. Wired, because it doesn't match the syndrome you got from the device.
I don't have any more good ideas here. 
Have you tried to compare the input parameters for this command between a working version of Mellanox PMD over Linux and this version? 

> 
> Thanks again!
> 
> Mit Matelske
> 
> 
> 
> >
> > All this is "stock" FB 12.0 and DPDK 18.08 code, FWIW.
> >
> > Any and all help is appreciated.  I'm a little out of my depths with this.
> >
> > Thanks!
> >
> > Mit Matelske
> >
> > >
> > > Thanks!
> > >
> > > Mit Matelske
> > >
> > > ----- Original Message -----
> > > From: "shahafs" <shahafs@mellanox.com>
> > > To: "Mit Matelske" <mit@pt.net>, "stephen"
> > > <stephen@networkplumber.org>
> > > Cc: "Thomas Monjalon" <thomas@monjalon.net>, "dev"
> > <dev@dpdk.org>,
> > > "Yongseok Koh" <yskoh@mellanox.com>
> > > Sent: Sunday, March 3, 2019 2:43:41 AM
> > > Subject: RE: [dpdk-dev] mlx5 under FreeBSD
> > >
> > > Friday, March 1, 2019 5:58 PM, Mit Matelske:
> > > > Subject: Re: [dpdk-dev] mlx5 under FreeBSD
> > > >
> > > > I was wondering if I could lean on the experts to help me out with
> > > > an
> > issue?
> > > > I'm finally testing this port but don't understand why the
> > > > following error is
> > > > happening:
> > > >
> > > > net_mlx5: mlx5_rxq.c:2151: mlx5_hrxq_drop_new(): port 0 cannot
> > > > allocate QP for drop queue
> > > >
> > > > The driver is giving the following error corresponding with that
> > > > DPDK
> > > > message:
> > > >
> > > > Feb 28 15:00:44 fb100g1 kernel: mlx5_core0: ERR:
> > > > mlx5_cmd_check:701:(pid
> > > > 6819): CREATE_TIR(0x900) op_mod(0x0) failed, status bad
> > > > parameter(0x3), syndrome (0x569ee6)
> > >
> > > Hard to say why. I tried to look for the syndrome above on FW but
> > > couldn't find. Are you sure this is correct one?
> > > Which FW version you use?
> > >
> > > >
> > > > The mlx5_core_create_tir function completes successfully for the
> > > > en driver, just not the ib driver when trying to build the hash tables.
> > > > Any
> > > ideas?
> > > >
> > > > I've instrumented libmlx5.so.1, mlx5.ko and mlx5ib.ko to debug
> > > > this and everything looks fine according to my limited knowledge.
> > > > Not sure why the card is balking on this action.
> > > >
> > > > Thanks!
> > > >
> > > > Mit Matelske
> > > >
> > > > ----- Original Message -----
> > > > From: "stephen" <stephen@networkplumber.org>
> > > > To: "Mit Matelske" <mit@pt.net>
> > > > Cc: "shahafs" <shahafs@mellanox.com>, "Thomas Monjalon"
> > > > <thomas@monjalon.net>, "dev" <dev@dpdk.org>, "Yongseok Koh"
> > > > <yskoh@mellanox.com>
> > > > Sent: Wednesday, November 21, 2018 2:12:04 PM
> > > > Subject: Re: [dpdk-dev] mlx5 under FreeBSD
> > > >
> > > > On Wed, 21 Nov 2018 11:11:51 -0600 (CST) Mit Matelske <mit@pt.net>
> > > > wrote:
> > > >
> > > > > >
> > > > > > Rather than building a complex shim, why not just have an OS
> > > > > > dependent section of the driver.  The bigger issue is that the
> > > > > > kernel side functionality in BSD is different or missing.  For
> > > > > > example TAP
> > > > API is different and not multi-queue.
> > > > >
> > > > > Once I started working on it, I had to go this route.  I haven't
> > > > > been able to but much time into it yet, but it is coming along:
> > > > >
> > > > > EAL: PCI device 0000:b3:00.0 on NUMA socket 0
> > > > > EAL:   probe driver: 15b3:1013 net_mlx5
> > > > > net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_0"
> > > > > uverbs0 /sys/class/infiniband_verbs/uverbs0
> > > > > /sys/class/infiniband/mlx5_0
> > > > > net_mlx5: mlx5.c:1325: mlx5_pci_probe(): PCI information matches
> > > > > for
> > > > device "mlx5_0"
> > > > > net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_1"
> > > > > uverbs1 /sys/class/infiniband_verbs/uverbs1
> > > > > /sys/class/infiniband/mlx5_1
> > > > > net_mlx5: mlx5.c:1382: mlx5_pci_probe(): no switch support
> > > > > detected
> > > > > net_mlx5: mlx5.c:796: mlx5_dev_spawn(): MPW isn't supported
> > > > > net_mlx5: mlx5.c:852: mlx5_dev_spawn(): tunnel offloading
> > > > > disabled due to old OFED/rdma-core version
> > > > > net_mlx5: mlx5.c:864: mlx5_dev_spawn(): MPLS over GRE/UDP
> tunnel
> > > > > offloading disabled due to old OFED/rdma-core version or
> > > > > firmware configuration
> > > > > net_mlx5: mlx5.c:877: mlx5_dev_spawn(): naming Ethernet device
> > > > "0000:b3:00.0"
> > > > > net_mlx5: mlx5.c:1007: mlx5_dev_spawn(): checksum offloading is
> > > > > supported
> > > > > net_mlx5: mlx5.c:1024: mlx5_dev_spawn(): maximum Rx indirection
> > > > > table size is 512
> > > > > net_mlx5: mlx5.c:1028: mlx5_dev_spawn(): VLAN stripping is not
> > > > > supported
> > > > > net_mlx5: mlx5.c:1032: mlx5_dev_spawn(): FCS stripping
> > > > > configuration is not supported
> > > > > net_mlx5: mlx5.c:1037: mlx5_dev_spawn(): hardware Rx end
> > > > > alignment padding is not supported
> > > > > net_mlx5: mlx5.c:1052: mlx5_dev_spawn(): MPS is disabled
> > > > > net_mlx5: mlx5.c:628: mlx5_uar_init_primary(): port 0 reserved
> > > > > UAR address space: 0x880000000
> > > > > net_mlx5: mlx5.c:1107: mlx5_dev_spawn(): port 0 MAC address is
> > > > > 00:90:0b:78:2a:ea
> > > > > net_mlx5: mlx5.c:1114: mlx5_dev_spawn(): port 0 ifname is "mce0"
> > > > > net_mlx5: mlx5.c:1127: mlx5_dev_spawn(): port 0 MTU is 1500
> > > > > net_mlx5: mlx5.c:1142: mlx5_dev_spawn(): flow rules relying on
> > > > > switch offloads will not be supported: cannot open libmnl socket:
> > > > > Address family not supported by protocol family
> > > > > net_mlx5: mlx5.c:1176: mlx5_dev_spawn(): port 0 forcing Ethernet
> > > > > interface up
> > > > > net_mlx5: mlx5_ethdev.c:735: mlx5_link_update_unlocked_gs():
> > > > > port
> > > > > 0
> > > > > ioctl(SIOCGIFFLAGS) failed: No such file or directory
> > > > > net_mlx5: mlx5_ethdev.c:664: mlx5_link_update_unlocked_gset():
> > > > > port
> > > > > 0
> > > > > ioctl(SIOCGIFFLAGS) failed: No such file or directory
> > > > > net_mlx5: mlx5_rxq.c:2150: mlx5_hrxq_drop_new(): port 0 cannot
> > > > > allocate QP for drop queue Segmentation fault (core dumped)
> > > > >
> > > > > Working through mlx5_ethdev now...
> > > > >
> > > > > Mit
> > > >
> > > > Also if you are targeting Azure or Hyper-V, then don't bother
> > > > porting TAP/VDEV_NETVSC/Failsafe model because that is being
> > > > replaced by netvsc native PMD.  It would be easier to just get
> > > > that
> > working.
> > > > It would mean writing a  uio equivalent driver for FreeBSD.

Re: mlx5 under FreeBSD

[Mit Matelske]

I started spinning up a Linux box to do that exact test on Friday.  I'll post the results later.

Thanks!

Mit Matelske

----- Original Message -----
From: "shahafs" <shahafs@mellanox.com>
To: "Mit Matelske" <mit@pt.net>
Cc: "stephen" <stephen@networkplumber.org>, "Thomas Monjalon" <thomas@monjalon.net>, "dev" <dev@dpdk.org>, "Yongseok Koh" <yskoh@mellanox.com>
Sent: Saturday, March 9, 2019 11:57:10 PM
Subject: RE: mlx5 under FreeBSD

Thursday, March 7, 2019 5:28 PM, Mit Matelske:
> Subject: Re: [dpdk-dev] mlx5 under FreeBSD
> 
> 
> >
> > ----- Original Message -----
> > From: "shahafs" <shahafs@mellanox.com>
> > To: "Mit Matelske" <mit@pt.net>
> >
> > What you print here is the verbs layer, and looks OK. I am more interested
> on the values before the command reach the device.
> > Can you print the values of: rx_hash_fn and rx_hash_toeplitz_key from the
> CREATE_TIR input mailbox command?
> >
> > Relevant function on mlx5_ib kernel module are:  mlx5_core_create_tir or
> create_rss_raw_qp_tir.
> 
> In create_rss_raw_qp_tir, starting at line 1374 I added the following printfs:
> 
>    switch (ucmd.rx_hash_function) {
>    case MLX5_RX_HASH_FUNC_TOEPLITZ:
>    {
>       void *rss_key = MLX5_ADDR_OF(tirc, tirc, rx_hash_toeplitz_key);
>       size_t len = MLX5_FLD_SZ_BYTES(tirc, rx_hash_toeplitz_key);
> 
>       if (len != ucmd.rx_key_len) {
>          err = -EINVAL;
>          goto err;
>       }
> 
> printf("setting MLX5_RX_HASH_FUNC_TOEPLITZ\n");
>       MLX5_SET(tirc, tirc, rx_hash_fn, MLX5_RX_HASH_FUNC_TOEPLITZ);
>       MLX5_SET(tirc, tirc, rx_hash_symmetric, 1);
>       memcpy(rss_key, ucmd.rx_hash_key, len);
> printf("rx_hash_toeplitz_key(%ld):\n", len); for(int ix = 0; ix < len; ++ix) {
>    printf("%02X ", ((u_char *)rss_key)[ix]);
>    if (((ix + 1) % 10) == 0) printf("\n"); }
>       break;
>    }
>    default:
>       err = -EOPNOTSUPP;
>       goto err;
>    }
> 
> 
> Which resulted in:
> 
> Mar  7 09:25:56 fb100g1 kernel: setting MLX5_RX_HASH_FUNC_TOEPLITZ
> Mar  7 09:25:56 fb100g1 kernel: rx_hash_toeplitz_key(40):
> Mar  7 09:25:56 fb100g1 kernel: 2C C6 81 D1 5B DB F4 F7 FC A2 Mar  7 09:25:56
> fb100g1 kernel: 83 19 DB 1A 3E 94 6B 9E 38 D9 Mar  7 09:25:56 fb100g1 kernel:
> 2C 9C 03 D1 AD 99 44 A7 D9 56 Mar  7 09:25:56 fb100g1 kernel: 3D 59 06 3C 25
> F3 FC 1F DC 2A
> 
> So, to me, everything looks good?

Yes it looks good. Wired, because it doesn't match the syndrome you got from the device.
I don't have any more good ideas here. 
Have you tried to compare the input parameters for this command between a working version of Mellanox PMD over Linux and this version? 

> 
> Thanks again!
> 
> Mit Matelske
> 
> 
> 
> >
> > All this is "stock" FB 12.0 and DPDK 18.08 code, FWIW.
> >
> > Any and all help is appreciated.  I'm a little out of my depths with this.
> >
> > Thanks!
> >
> > Mit Matelske
> >
> > >
> > > Thanks!
> > >
> > > Mit Matelske
> > >
> > > ----- Original Message -----
> > > From: "shahafs" <shahafs@mellanox.com>
> > > To: "Mit Matelske" <mit@pt.net>, "stephen"
> > > <stephen@networkplumber.org>
> > > Cc: "Thomas Monjalon" <thomas@monjalon.net>, "dev"
> > <dev@dpdk.org>,
> > > "Yongseok Koh" <yskoh@mellanox.com>
> > > Sent: Sunday, March 3, 2019 2:43:41 AM
> > > Subject: RE: [dpdk-dev] mlx5 under FreeBSD
> > >
> > > Friday, March 1, 2019 5:58 PM, Mit Matelske:
> > > > Subject: Re: [dpdk-dev] mlx5 under FreeBSD
> > > >
> > > > I was wondering if I could lean on the experts to help me out with
> > > > an
> > issue?
> > > > I'm finally testing this port but don't understand why the
> > > > following error is
> > > > happening:
> > > >
> > > > net_mlx5: mlx5_rxq.c:2151: mlx5_hrxq_drop_new(): port 0 cannot
> > > > allocate QP for drop queue
> > > >
> > > > The driver is giving the following error corresponding with that
> > > > DPDK
> > > > message:
> > > >
> > > > Feb 28 15:00:44 fb100g1 kernel: mlx5_core0: ERR:
> > > > mlx5_cmd_check:701:(pid
> > > > 6819): CREATE_TIR(0x900) op_mod(0x0) failed, status bad
> > > > parameter(0x3), syndrome (0x569ee6)
> > >
> > > Hard to say why. I tried to look for the syndrome above on FW but
> > > couldn't find. Are you sure this is correct one?
> > > Which FW version you use?
> > >
> > > >
> > > > The mlx5_core_create_tir function completes successfully for the
> > > > en driver, just not the ib driver when trying to build the hash tables.
> > > > Any
> > > ideas?
> > > >
> > > > I've instrumented libmlx5.so.1, mlx5.ko and mlx5ib.ko to debug
> > > > this and everything looks fine according to my limited knowledge.
> > > > Not sure why the card is balking on this action.
> > > >
> > > > Thanks!
> > > >
> > > > Mit Matelske
> > > >
> > > > ----- Original Message -----
> > > > From: "stephen" <stephen@networkplumber.org>
> > > > To: "Mit Matelske" <mit@pt.net>
> > > > Cc: "shahafs" <shahafs@mellanox.com>, "Thomas Monjalon"
> > > > <thomas@monjalon.net>, "dev" <dev@dpdk.org>, "Yongseok Koh"
> > > > <yskoh@mellanox.com>
> > > > Sent: Wednesday, November 21, 2018 2:12:04 PM
> > > > Subject: Re: [dpdk-dev] mlx5 under FreeBSD
> > > >
> > > > On Wed, 21 Nov 2018 11:11:51 -0600 (CST) Mit Matelske <mit@pt.net>
> > > > wrote:
> > > >
> > > > > >
> > > > > > Rather than building a complex shim, why not just have an OS
> > > > > > dependent section of the driver.  The bigger issue is that the
> > > > > > kernel side functionality in BSD is different or missing.  For
> > > > > > example TAP
> > > > API is different and not multi-queue.
> > > > >
> > > > > Once I started working on it, I had to go this route.  I haven't
> > > > > been able to but much time into it yet, but it is coming along:
> > > > >
> > > > > EAL: PCI device 0000:b3:00.0 on NUMA socket 0
> > > > > EAL:   probe driver: 15b3:1013 net_mlx5
> > > > > net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_0"
> > > > > uverbs0 /sys/class/infiniband_verbs/uverbs0
> > > > > /sys/class/infiniband/mlx5_0
> > > > > net_mlx5: mlx5.c:1325: mlx5_pci_probe(): PCI information matches
> > > > > for
> > > > device "mlx5_0"
> > > > > net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_1"
> > > > > uverbs1 /sys/class/infiniband_verbs/uverbs1
> > > > > /sys/class/infiniband/mlx5_1
> > > > > net_mlx5: mlx5.c:1382: mlx5_pci_probe(): no switch support
> > > > > detected
> > > > > net_mlx5: mlx5.c:796: mlx5_dev_spawn(): MPW isn't supported
> > > > > net_mlx5: mlx5.c:852: mlx5_dev_spawn(): tunnel offloading
> > > > > disabled due to old OFED/rdma-core version
> > > > > net_mlx5: mlx5.c:864: mlx5_dev_spawn(): MPLS over GRE/UDP
> tunnel
> > > > > offloading disabled due to old OFED/rdma-core version or
> > > > > firmware configuration
> > > > > net_mlx5: mlx5.c:877: mlx5_dev_spawn(): naming Ethernet device
> > > > "0000:b3:00.0"
> > > > > net_mlx5: mlx5.c:1007: mlx5_dev_spawn(): checksum offloading is
> > > > > supported
> > > > > net_mlx5: mlx5.c:1024: mlx5_dev_spawn(): maximum Rx indirection
> > > > > table size is 512
> > > > > net_mlx5: mlx5.c:1028: mlx5_dev_spawn(): VLAN stripping is not
> > > > > supported
> > > > > net_mlx5: mlx5.c:1032: mlx5_dev_spawn(): FCS stripping
> > > > > configuration is not supported
> > > > > net_mlx5: mlx5.c:1037: mlx5_dev_spawn(): hardware Rx end
> > > > > alignment padding is not supported
> > > > > net_mlx5: mlx5.c:1052: mlx5_dev_spawn(): MPS is disabled
> > > > > net_mlx5: mlx5.c:628: mlx5_uar_init_primary(): port 0 reserved
> > > > > UAR address space: 0x880000000
> > > > > net_mlx5: mlx5.c:1107: mlx5_dev_spawn(): port 0 MAC address is
> > > > > 00:90:0b:78:2a:ea
> > > > > net_mlx5: mlx5.c:1114: mlx5_dev_spawn(): port 0 ifname is "mce0"
> > > > > net_mlx5: mlx5.c:1127: mlx5_dev_spawn(): port 0 MTU is 1500
> > > > > net_mlx5: mlx5.c:1142: mlx5_dev_spawn(): flow rules relying on
> > > > > switch offloads will not be supported: cannot open libmnl socket:
> > > > > Address family not supported by protocol family
> > > > > net_mlx5: mlx5.c:1176: mlx5_dev_spawn(): port 0 forcing Ethernet
> > > > > interface up
> > > > > net_mlx5: mlx5_ethdev.c:735: mlx5_link_update_unlocked_gs():
> > > > > port
> > > > > 0
> > > > > ioctl(SIOCGIFFLAGS) failed: No such file or directory
> > > > > net_mlx5: mlx5_ethdev.c:664: mlx5_link_update_unlocked_gset():
> > > > > port
> > > > > 0
> > > > > ioctl(SIOCGIFFLAGS) failed: No such file or directory
> > > > > net_mlx5: mlx5_rxq.c:2150: mlx5_hrxq_drop_new(): port 0 cannot
> > > > > allocate QP for drop queue Segmentation fault (core dumped)
> > > > >
> > > > > Working through mlx5_ethdev now...
> > > > >
> > > > > Mit
> > > >
> > > > Also if you are targeting Azure or Hyper-V, then don't bother
> > > > porting TAP/VDEV_NETVSC/Failsafe model because that is being
> > > > replaced by netvsc native PMD.  It would be easier to just get
> > > > that
> > working.
> > > > It would mean writing a  uio equivalent driver for FreeBSD.