l2fwd not working on PF while working on VF

l2fwd not working on PF while working on VF

[Tomasz K]

Dear All

I have a setup with Intel 82576 DPDK NIC.
For one of the available DPDK ports i have created 8 VFs using igb_uio
based on steps presented in programmers guide. Some of them are attached to
VM, some maybe used on host

After some tests with l2fwd application i found out that it is not working
for PF, but works perfectly for any VF (both on host and in VM).

I'm still trying to find root cause of this issue but maybe someone
encountered such problem

Thanks in advance
Tomasz Kasowicz

Setup:
Intel(R) Xeon(R) CPU E31220 @ 3.10GHz running Ubuntu 12.04 LTS

pci_unbind.py --status

Network devices using IGB_UIO driver
====================================
0000:01:00.1 '82576NS Gigabit Network Connection' drv=igb_uio unused=igb
0000:02:11.1 '82576 Virtual Function' drv=igb_uio unused=igbvf

Network devices using kernel driver
===================================
0000:00:19.0 '82579LM Gigabit Network Connection' if=eth1 drv=e1000e
unused= *Active*
0000:01:00.0 '82576NS Gigabit Network Connection' if=eth0 drv=igb
unused=igb_uio *Active*
0000:02:10.1 '82576 Virtual Function' if= drv=pci-stub unused=igbvf,igb_uio
0000:02:10.3 '82576 Virtual Function' if= drv=pci-stub unused=igbvf,igb_uio
0000:02:10.5 '82576 Virtual Function' if= drv=pci-stub unused=igbvf,igb_uio
0000:02:10.7 '82576 Virtual Function' if= drv=pci-stub unused=igbvf,igb_uio
0000:02:11.3 '82576 Virtual Function' if= drv=pci-stub unused=igbvf,igb_uio
0000:02:11.5 '82576 Virtual Function' if= drv=pci-stub unused=igbvf,igb_uio
0000:02:11.7 '82576 Virtual Function' if= drv=pci-stub unused=igbvf,igb_uio
0000:04:00.0 '82574L Gigabit Network Connection' if=eth3 drv=e1000e
unused=igb_uio

Other network devices
=====================
<none>


cat /sys/bus/pci/devices/0000\:01\:00.1/max_vfs
8

ls -alrt /sys/bus/pci/devices/0000\:01\:00.1/virtfn*
lrwxrwxrwx 1 root root 0 Apr  9 12:09
/sys/bus/pci/devices/0000:01:00.1/virtfn7 -> ../0000:02:11.7
lrwxrwxrwx 1 root root 0 Apr  9 12:09
/sys/bus/pci/devices/0000:01:00.1/virtfn6 -> ../0000:02:11.5
lrwxrwxrwx 1 root root 0 Apr  9 12:09
/sys/bus/pci/devices/0000:01:00.1/virtfn5 -> ../0000:02:11.3
lrwxrwxrwx 1 root root 0 Apr  9 12:09
/sys/bus/pci/devices/0000:01:00.1/virtfn4 -> ../0000:02:11.1
lrwxrwxrwx 1 root root 0 Apr  9 12:09
/sys/bus/pci/devices/0000:01:00.1/virtfn3 -> ../0000:02:10.7
lrwxrwxrwx 1 root root 0 Apr  9 12:09
/sys/bus/pci/devices/0000:01:00.1/virtfn2 -> ../0000:02:10.5
lrwxrwxrwx 1 root root 0 Apr  9 12:09
/sys/bus/pci/devices/0000:01:00.1/virtfn1 -> ../0000:02:10.3
lrwxrwxrwx 1 root root 0 Apr  9 12:09
/sys/bus/pci/devices/0000:01:00.1/virtfn0 -> ../0000:02:10.1

Re: l2fwd not working on PF while working on VF

[Richardson, Bruce]

> -----Original Message-----
> From: dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] On Behalf Of Tomasz K
> Sent: Wednesday, April 09, 2014 3:04 PM
> To: dev
> Subject: [dpdk-dev] l2fwd not working on PF while working on VF
> 
> Dear All
> 
> I have a setup with Intel 82576 DPDK NIC.
> For one of the available DPDK ports i have created 8 VFs using igb_uio based on
> steps presented in programmers guide. Some of them are attached to VM, some
> maybe used on host
> 
> After some tests with l2fwd application i found out that it is not working for PF,
> but works perfectly for any VF (both on host and in VM).
> 
How many queues are being allocated to each VF? According to the datasheet, the 82576 NIC has only got 16 RX and TX queues, so if you have 8 VF's with 2 RX and TX queues allocated to each, you may have no remaining queues left for the PF to use on the host.

Re: l2fwd not working on PF while working on VF

[Tomasz K]

Hi Bruce

Thanks for the tip. I changed VFs to 6 and now everything works ok on PF.

Thank You very much
Tomasz Kasowicz


2014-04-09 17:10 GMT+02:00 Richardson, Bruce <bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>:

> > -----Original Message-----
> > From: dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] On Behalf Of Tomasz K
> > Sent: Wednesday, April 09, 2014 3:04 PM
> > To: dev
> > Subject: [dpdk-dev] l2fwd not working on PF while working on VF
> >
> > Dear All
> >
> > I have a setup with Intel 82576 DPDK NIC.
> > For one of the available DPDK ports i have created 8 VFs using igb_uio
> based on
> > steps presented in programmers guide. Some of them are attached to VM,
> some
> > maybe used on host
> >
> > After some tests with l2fwd application i found out that it is not
> working for PF,
> > but works perfectly for any VF (both on host and in VM).
> >
> How many queues are being allocated to each VF? According to the
> datasheet, the 82576 NIC has only got 16 RX and TX queues, so if you have 8
> VF's with 2 RX and TX queues allocated to each, you may have no remaining
> queues left for the PF to use on the host.
>

Re: l2fwd not working on PF while working on VF

[Chris Wright]

* Richardson, Bruce (bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org) wrote:
> > -----Original Message-----
> > From: dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] On Behalf Of Tomasz K
> > Sent: Wednesday, April 09, 2014 3:04 PM
> > To: dev
> > Subject: [dpdk-dev] l2fwd not working on PF while working on VF
> > 
> > Dear All
> > 
> > I have a setup with Intel 82576 DPDK NIC.
> > For one of the available DPDK ports i have created 8 VFs using igb_uio based on
> > steps presented in programmers guide. Some of them are attached to VM, some
> > maybe used on host
> > 
> > After some tests with l2fwd application i found out that it is not working for PF,
> > but works perfectly for any VF (both on host and in VM).
>
> How many queues are being allocated to each VF? According to the datasheet, the 82576 NIC has only got 16 RX and TX queues, so if you have 8 VF's with 2 RX and TX queues allocated to each, you may have no remaining queues left for the PF to use on the host.

BTW, Linux driver caps VFs at 7 for this reason (#include <do drivers in
kernel rant>).  Any reason not to do this w/ igb_uio...e.g.

thanks,
-chris
---
From: Chris Wright <chrisw-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Subject: [PATCH] igb_uio: cap max VFs at 7 to reserve one for PF

To keep from confusing users, cap max VFs at 7, despite PCI SR-IOV config
space showing a max of 8.  This reserves a queue pair for the PF.

Signed-off-by: Chris Wright <chrisw-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
---

 lib/librte_eal/linuxapp/igb_uio/igb_uio.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
index 6db8a4a..096fb01 100644
--- a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
+++ b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
@@ -130,6 +130,12 @@ store_max_vfs(struct device *dev, struct device_attribute *attr,
 	if (0 != strict_strtoul(buf, 0, &max_vfs))
 		return -EINVAL;
 
+	/* reserve a queue pair for PF */
+	if (max_vfs > 7) {
+		dev_warn(&pdev->dev, "Maxixum of 7 VFs per PF, using max\n");
+		max_vfs = 7;
+	}
+
 	if (0 == max_vfs)
 		pci_disable_sriov(pdev);
 	else if (0 == local_pci_num_vf(pdev))