Re: SR-IOV + switchdev + vlan + Mellanox: Cannot ping

[Jiri Pirko <jiri@resnulli.us>]

Fri, Apr 26, 2024 at 10:35:28PM CEST, gshanemiller6@gmail.com wrote:
>Problem:
>-----------------------------------------------------------------
>root@machA $ ping 10.xx.xx.194
>PING 10.xx.xx.194 (10.xx.xx.194) 56(84) bytes of data
>From 10.xx.xx.191 icmp seq=10 Destination Host Unreachable
>Proximate Cause:
>-----------------------------------------------------------------
>This seems to be a side effect of "switchdev" mode. When the identical
>configuration is set up EXCEPT that the SR-IOV virtualized NIC is left
>"legacy", ping (and ncat) works just fine.
>
>As far as I can tell I need a bridge or bridge commands, but I have no
>idea where to start. This environment will not allow me to add modify
>commands when enabling switchdev mode. devlink seems to accept
>"switchdev" alone without modifiers.

You have to configure forwarding between appropriate representors. Use
ovs (probably easiest) or tc.

>
>Note: putting a NIC into switchdev mode makes the virtual functions
>show as "link-state disable" which is confusing. (See below.) Contrary
>to what it seems to suggest, the virtual NICs are up and running
>
>Running "arp -e" on machine A shows machine B's ieth3v0 MAC address as
>incomplete suggesting switchdev+ARP is broken.
>
>Problem Environment:
>-----------------------------------------------------------------
>OS: RHEL 8.6 4.18.0-372.46.1.el8 x64
>NICs: Mellanox ConnectX-6
>
>Machine A Links:
>70 tst@ieth3: <...LOWER_UP...> mtu 1500
>   link/ether xx.xx.xx.xx.xx.xx
>   vlan protocol 802.1Q id 133 <REORDER_HDR>
>   Inet 10.xx.xx.191
>
>Machine B Links With ieth3 in SR-IOV mode in switchdev mode:
># Physical Function and its virtual functions:
>                                                 2: ieth3:
><...PROMISC,UP,LOWER_UP> mtu 1500
>    link/ether xx.xx.xx.xx.xx.f6 portname p0 switchid xxxxe988
>    vf 0 link/ether xx.xx.xx.xx.xx.00 vlan 133 spoof off, link-state
>disable, trust off
>    . . .
># Port representers
>893: ieth3r0: <...UP,LOWER_UP> mtu 1500
>link/ether xx.xx.xx.xx.xx.e1 portname pf0vf0 switchid xxxxe988
>. . .
># Virtual Links
>897: ieth3v0: <...UP,LOWER_UP> mtu 1500
>  link/ether xx.xx.xx.xx.xx.00 promiscuity 0
>  inet 10.xx.xx.194/24 scope global ieth3v0
>  . . .
>
>SR-IOV Setup Summary
>-----------------------------------------------------------------
>This is done right since, in legacy mode, ping/ncat works fine:
>
>1. Enable IOMMU, Vtx in BIOS
>2. Boot Linux with iommu=on on command line
>3. Install Mellanox OFED
>4. Enable SR-IOV for max 8 devices in Mellanox firmware
>(reboot)
>5. Create 4 virtual NICs w/ SR-IOV
>6. Configure 4 virtual NICs mac, trust off, spoofchk off, state auto
>7. Unbind virtual NICs
>8. Put ieth3 into switchdev mode
>9. Rebind virtual NICs
>10. Bring all links up
>11. Assign IPV4 addresses to virtual links
>