Permanently binding NIC ports with DPDK drivers

Permanently binding NIC ports with DPDK drivers

[Montorsi, Francesco]

Hi,
Is there a way to permanently (i.e., have the configuration automatically applied after reboot) bind a NIC port to DPDK?

In case there's none, I'm thinking to save in my software a list of the NIC ports chosen by the user for use with DPDK and then, upon software startup to just do
    for (int i=0; i < ...; i++)
     system("dpdk_nic_bind.py --bind=igb_uio " + PCI_device_chosen[i]);
Do you see any problem with that? 

Thanks!
Francesco Montorsi

Re: Permanently binding NIC ports with DPDK drivers

[Bruce Richardson]

On Wed, Nov 11, 2015 at 04:13:01PM +0000, Montorsi, Francesco wrote:
> Hi,
> Is there a way to permanently (i.e., have the configuration automatically applied after reboot) bind a NIC port to DPDK?
> 
> In case there's none, I'm thinking to save in my software a list of the NIC ports chosen by the user for use with DPDK and then, upon software startup to just do
>     for (int i=0; i < ...; i++)
>      system("dpdk_nic_bind.py --bind=igb_uio " + PCI_device_chosen[i]);
> Do you see any problem with that? 
> 
> Thanks!
> Francesco Montorsi
> 

Hi Francesco,

I'm not aware of any way to make the bindings permanent across reboots. What you
have suggested will work, but there are probably better ways to do the same thing.
For example, a couple of lines in an rc.local script can reapply the bindings at
boot for you. I'm sure others can suggest other ways of having the same effect,
for example, there may be a way to automatically do this using udev or systemd
or some such package.

/Bruce

Re: Permanently binding NIC ports with DPDK drivers

[Montorsi, Francesco]

Hi Bruce,

> -----Original Message-----
> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> I'm not aware of any way to make the bindings permanent across reboots.
> What you have suggested will work, but there are probably better ways to
> do the same thing.

I agree... let's see if somebody else has suggestions :)

In any case my idea is to make my software as much independent as possible from troubles with future HW and future DPDK versions. A way to do that would be to leave all the bind steps and intelligence inside the dpdk_nic_bind.py script and just use that (since it will be probably always up to date and correct). My only concern is that (reading the python code) dpdk_nic_bind.py script does not return with an error code != 0 if something bad happens during binding... maybe it may be worth doing such a small change...

Just my 2 cents,
Francesco

Re: Permanently binding NIC ports with DPDK drivers

[Stephen Hemminger]

On Wed, 11 Nov 2015 16:59:14 +0000
"Montorsi, Francesco" <fmontorsi@empirix.com> wrote:

> Hi Bruce,
> 
> > -----Original Message-----
> > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > I'm not aware of any way to make the bindings permanent across reboots.
> > What you have suggested will work, but there are probably better ways to
> > do the same thing.
> 
> I agree... let's see if somebody else has suggestions :)
> 
> In any case my idea is to make my software as much independent as possible from troubles with future HW and future DPDK versions. A way to do that would be to leave all the bind steps and intelligence inside the dpdk_nic_bind.py script and just use that (since it will be probably always up to date and correct). My only concern is that (reading the python code) dpdk_nic_bind.py script does not return with an error code != 0 if something bad happens during binding... maybe it may be worth doing such a small change...
> 
> Just my 2 cents,
> Francesco
> 
> 
> 
> 

I would recommend using PCI id's and not depending in anyway on port.
If you want, I can submit a patch to add derive the systemd/udev compatiable
name from existing DPDK port.

Re: Permanently binding NIC ports with DPDK drivers

[Matthew Hall]

In my development environment I set up an at-boot provisioning script that 
does it. I recommend using scripts and not shelling out from C code. ;)

On Wed, Nov 11, 2015 at 04:13:01PM +0000, Montorsi, Francesco wrote:
> Hi,
> Is there a way to permanently (i.e., have the configuration automatically applied after reboot) bind a NIC port to DPDK?
> 
> In case there's none, I'm thinking to save in my software a list of the NIC ports chosen by the user for use with DPDK and then, upon software startup to just do
>     for (int i=0; i < ...; i++)
>      system("dpdk_nic_bind.py --bind=igb_uio " + PCI_device_chosen[i]);
> Do you see any problem with that? 
> 
> Thanks!
> Francesco Montorsi
> 

Re: Permanently binding NIC ports with DPDK drivers

[Mcnamara, John]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Montorsi, Francesco
> Sent: Wednesday, November 11, 2015 4:13 PM
> To: dev@dpdk.org
> Subject: [dpdk-dev] Permanently binding NIC ports with DPDK drivers
> 
> Hi,
> Is there a way to permanently (i.e., have the configuration automatically
> applied after reboot) bind a NIC port to DPDK?


Hi,

The Ubuntu dpdk package for 15.10 contains system scripts with functions for reserving hugepages and binding interfaces on bootup:


    /etc/dpdk/dpdk.conf
    /etc/dpdk/interfaces
    /etc/init.d/dpdk
    /lib/dpdk/dpdk-init
    /lib/systemd/system/dpdk.service
    /sbin/dpdk_nic_bind
    /usr/bin/testpmd
    /usr/share/doc/dpdk/README.Debian
    /usr/share/doc/dpdk/changelog.Debian.gz
    /usr/share/doc/dpdk/copyright
    /usr/share/dpdk/tools/cpu_layout.py
    /usr/share/dpdk/tools/dpdk_nic_bind.py
    /usr/share/dpdk/tools/setup.sh
    /usr/share/python/runtime.d/dpdk.rtupdate

    http://packages.ubuntu.com/wily/amd64/dpdk/filelist

If you have the latest version of Ubuntu you can check that out or else download and extract the files from the .deb to see how they do it.

John.
-- 

Re: Permanently binding NIC ports with DPDK drivers

[Panu Matilainen]

On 11/11/2015 06:28 PM, Bruce Richardson wrote:
> On Wed, Nov 11, 2015 at 04:13:01PM +0000, Montorsi, Francesco wrote:
>> Hi,
>> Is there a way to permanently (i.e., have the configuration automatically applied after reboot) bind a NIC port to DPDK?
>>
>> In case there's none, I'm thinking to save in my software a list of the NIC ports chosen by the user for use with DPDK and then, upon software startup to just do
>>      for (int i=0; i < ...; i++)
>>       system("dpdk_nic_bind.py --bind=igb_uio " + PCI_device_chosen[i]);
>> Do you see any problem with that?
>>
>> Thanks!
>> Francesco Montorsi
>>
>
> Hi Francesco,
>
> I'm not aware of any way to make the bindings permanent across reboots. What you
> have suggested will work, but there are probably better ways to do the same thing.
> For example, a couple of lines in an rc.local script can reapply the bindings at
> boot for you. I'm sure others can suggest other ways of having the same effect,
> for example, there may be a way to automatically do this using udev or systemd
> or some such package.

I've been looking into this recently, here's what I have so far:
http://laiskiainen.org/git/?p=driverctl.git

For the impatient, "make rpm" should produce something usable for recent 
Fedora/RHEL systems, usage looks somewhat like this:

Find devices currently driven by ixgbe driver:
# driverctl -v list-devices | grep ixgbe
0000:01:00.0 ixgbe (Ethernet 10G 4P X520/I350 rNDC)
0000:01:00.1 ixgbe (Ethernet 10G 4P X520/I350 rNDC)

Change them to use the vfio-pci driver permanently:
# driverctl set-override 0000:01:00.0 vfio-pci
# driverctl set-override 0000:01:00.1 vfio-pci

Find devices with driver overrides:
[root@wsfd-netdev32 ~]# driverctl -v list-devices|grep \*
0000:01:00.0 vfio-pci [*] (Ethernet 10G 4P X520/I350 rNDC)
0000:01:00.1 vfio-pci [*] (Ethernet 10G 4P X520/I350 rNDC)

Remove the permanent driver override for device 0000:01:00.1:
# driverctl unset-override 0000:01:00.1

In addition it has udev rules to export vfio and uio devices on systemd 
level, eg the above looks like this with normal drivers:

# systemctl |grep 0000:01:00
sys-devices-pci0000:00-0000:00:03.0-0000:01:00.0-net-em1.device 
                  loaded active plugged   Ethernet 10G 4P X520/I350 rNDC
sys-devices-pci0000:00-0000:00:03.0-0000:01:00.1-net-em2.device 
                  loaded active plugged   Ethernet 10G 4P X520/I350 rNDC

When changed to vfio, with upstream systemd/udev rules they would just 
disappear entirely, but with the driverctl rules they become:

# systemctl |grep 0000:01:00
sys-devices-pci0000:00-0000:00:03.0-0000:01:00.0-vfio.device 
                  loaded active plugged 
/sys/devices/pci0000:00/0000:00:03.0/0000:01:00.0/vfio
sys-devices-pci0000:00-0000:00:03.0-0000:01:00.1-vfio.device 
                  loaded active plugged 
/sys/devices/pci0000:00/0000:00:03.0/0000:01:00.1/vfio

	- Panu -

Re: Permanently binding NIC ports with DPDK drivers

[Montorsi, Francesco]

Hi Panu,

> -----Original Message-----
> From: Panu Matilainen [mailto:pmatilai@redhat.com]
> I've been looking into this recently, here's what I have so far:
> http://laiskiainen.org/git/?p=driverctl.git
> 
Thanks I tried the script (I just had to change /bin/sh into /bin/bash on first line) and it works fine on my Ubuntu 14.04.
Unfortunately I'm forced to run my sw on SLES 11.3 (which btw uses kernel 3.0.76 which I don't know if is compatible with VFIO) and there the output of "list-devices" is plain empty. I will try to look at what's happening...

Thanks,
Francesco

Re: Permanently binding NIC ports with DPDK drivers

[Montorsi, Francesco]

Hi John,

> -----Original Message-----
> From: Mcnamara, John [mailto:john.mcnamara@intel.com]
> 
> The Ubuntu dpdk package for 15.10 contains system scripts with functions for
> reserving hugepages and binding interfaces on bootup:
> 
> 
>     /etc/dpdk/dpdk.conf
>     /etc/dpdk/interfaces
>     /etc/init.d/dpdk
>     /lib/dpdk/dpdk-init
>     /lib/systemd/system/dpdk.service
>     /sbin/dpdk_nic_bind
>     /usr/bin/testpmd
>     /usr/share/doc/dpdk/README.Debian
>     /usr/share/doc/dpdk/changelog.Debian.gz
>     /usr/share/doc/dpdk/copyright
>     /usr/share/dpdk/tools/cpu_layout.py
>     /usr/share/dpdk/tools/dpdk_nic_bind.py
>     /usr/share/dpdk/tools/setup.sh
>     /usr/share/python/runtime.d/dpdk.rtupdate
> 
>     http://packages.ubuntu.com/wily/amd64/dpdk/filelist
> 
> If you have the latest version of Ubuntu you can check that out or else
> download and extract the files from the .deb to see how they do it.
> 

This certainly looks very useful. I inspected the package and the files you mentioned and indeed it looks like a good way to go, specially if Ubuntu distribution is moving in that direction (hopefully other distros will follow too).

Thanks a lot!

Francesco

Re: Permanently binding NIC ports with DPDK drivers

[Panu Matilainen]

On 11/13/2015 01:44 PM, Montorsi, Francesco wrote:
> Hi Panu,
>
>> -----Original Message-----
>> From: Panu Matilainen [mailto:pmatilai@redhat.com]
>> I've been looking into this recently, here's what I have so far:
>> http://laiskiainen.org/git/?p=driverctl.git
>>
> Thanks I tried the script (I just had to change /bin/sh into
> /bin/bash on first line) and it works fine on my Ubuntu 14.04.

Oops, fixed now. Thanks for pointing that out, the line between sh and 
bash is (too) blurry on Fedora & friends.

> Unfortunately I'm forced to run my sw on SLES 11.3 (which btw uses
> kernel 3.0.76 which I don't know if is compatible with VFIO) and there
> the output of "list-devices" is plain empty. I will try to look at
> what's happening...

list-devices currently only lists devices which support driver 
overriding[*], which is a relatively new kernel feature and would be 
missing from 3.0.76:
https://www.redhat.com/archives/libvir-list/2014-April/msg00222.html

It might well be possible to fall back to the "old style" bind system if 
the kernel doesn't support driver_override. I've no immediate plans to 
look into that but lets just say patches would at least be considered :)

	- Panu -

Re: Permanently binding NIC ports with DPDK drivers

[Panu Matilainen]

On 11/13/2015 01:48 PM, Montorsi, Francesco wrote:
> Hi John,
>
>> -----Original Message-----
>> From: Mcnamara, John [mailto:john.mcnamara@intel.com]
>>
>> The Ubuntu dpdk package for 15.10 contains system scripts with functions for
>> reserving hugepages and binding interfaces on bootup:
>>
>>
>>      /etc/dpdk/dpdk.conf
>>      /etc/dpdk/interfaces
>>      /etc/init.d/dpdk
>>      /lib/dpdk/dpdk-init
>>      /lib/systemd/system/dpdk.service
>>      /sbin/dpdk_nic_bind
>>      /usr/bin/testpmd
>>      /usr/share/doc/dpdk/README.Debian
>>      /usr/share/doc/dpdk/changelog.Debian.gz
>>      /usr/share/doc/dpdk/copyright
>>      /usr/share/dpdk/tools/cpu_layout.py
>>      /usr/share/dpdk/tools/dpdk_nic_bind.py
>>      /usr/share/dpdk/tools/setup.sh
>>      /usr/share/python/runtime.d/dpdk.rtupdate
>>
>>      http://packages.ubuntu.com/wily/amd64/dpdk/filelist
>>
>> If you have the latest version of Ubuntu you can check that out or else
>> download and extract the files from the .deb to see how they do it.
>>
>
> This certainly looks very useful. I inspected the package and the
> files you mentioned and indeed it looks like a good way to go, specially
> if Ubuntu distribution is moving in that direction (hopefully other
> distros will follow too).
>
> Thanks a lot!

The downside of the Ubuntu-approach is that the setup is fairly static 
and does not support automatically binding hotplugged devices, AFAICS.
Whether that matters to your use-case is of course en entirely different 
question.

	- Panu -

> Francesco
>
>

Re: Permanently binding NIC ports with DPDK drivers

[Aaron Conole]

Panu Matilainen <pmatilai@redhat.com> writes:
> On 11/11/2015 06:28 PM, Bruce Richardson wrote:
>> On Wed, Nov 11, 2015 at 04:13:01PM +0000, Montorsi, Francesco wrote:
>>> Hi,
>>> Is there a way to permanently (i.e., have the configuration automatically applied after reboot) bind a NIC port to DPDK?
>>>
>>> In case there's none, I'm thinking to save in my software a list of the NIC ports chosen by the user for use with DPDK and then, upon software startup to just do
>>>      for (int i=0; i < ...; i++)
>>>       system("dpdk_nic_bind.py --bind=igb_uio " + PCI_device_chosen[i]);
>>> Do you see any problem with that?
>>>
>>> Thanks!
>>> Francesco Montorsi
>>>
>>
>> Hi Francesco,
>>
>> I'm not aware of any way to make the bindings permanent across
>> reboots. What you
>> have suggested will work, but there are probably better ways to do
>> the same thing.
>> For example, a couple of lines in an rc.local script can reapply the
>> bindings at
>> boot for you. I'm sure others can suggest other ways of having the
>> same effect,
>> for example, there may be a way to automatically do this using udev or systemd
>> or some such package.
>
> I've been looking into this recently, here's what I have so far:
> http://laiskiainen.org/git/?p=driverctl.git
>

I've been using this recently, on my systems.

+1 for driverctl :-)

Re: Permanently binding NIC ports with DPDK drivers

[Panu Matilainen]

On 08/25/2016 02:01 PM, Ferruh Yigit wrote:
> On 8/25/2016 5:57 AM, Keren Hochman wrote:
>> Hi,
>> I there a way to permanently bind a nic port when using DPDK drier with
>> kernel < 3.6 ? (In this kernel VFIO driver is not supported)?
>> Thanks, Keren
>>
>
> There was a tool from Panu for this purpose:
> http://dpdk.org/ml/archives/dev/2015-December/029500.html

Yup, but driverctl uses the newer driver_override binding feature which 
is only in kernel >= 3.16 so it's of no use with kernel < 3.6.

	- Panu -

Re: Permanently binding NIC ports with DPDK drivers

[Ferruh Yigit]

On 8/25/2016 5:57 AM, Keren Hochman wrote:
> Hi,
> I there a way to permanently bind a nic port when using DPDK drier with
> kernel < 3.6 ? (In this kernel VFIO driver is not supported)?
> Thanks, Keren
> 

There was a tool from Panu for this purpose:
http://dpdk.org/ml/archives/dev/2015-December/029500.html

Regards,
ferruh

Permanently binding NIC ports with DPDK drivers

[Keren Hochman]

Hi,
I there a way to permanently bind a nic port when using DPDK drier with
kernel < 3.6 ? (In this kernel VFIO driver is not supported)?
Thanks, Keren