[PATCH 0/2] Fix 2 bugs of i40e VF interrupt found in l3fwd-power

[PATCH 0/2] Fix 2 bugs of i40e VF interrupt found in l3fwd-power

[Wei Dai]

These 2 bugs can be observed from example/l3fwd-power run with i40e VF
bound to VFIO-PCI. The test steps are as follows:
1. Disable LSC interrupt by clearing port_conf.intr_conf.lsc=0, as i40e
   VF doesn't support LSC interrupt to avoid rte_eth_dev_configure() failure.
2. Create a VF from i40e host PF. Let PF run with kernel driver and bind
   its VF to VFIO-PCI
3. Run l3fwd-power like: l3fwd-power -l 18-19 -- -p 0x1 --config='(0,0,19)'

Then, the following error message appears like:
EAL: Error enabling MSI-X interrupts for fd 18
This error is from rte_intr_enable( )/vfio_enable_msix( ) when enabling
Rx queue interrupt.
Same as the patch 06938770186a ("net/ixgbe: fix VFIO interrupt mapping in VF"),
to change VFIO MSI-X interrupts mapping, previous mapping should be cleared
first to avoid above error.

After fixing above VFIO-PCI MSI-X interrupt mapping. There is still the following
2nd bug: l3fwd-power still can not be waked up by incoming packets.

Same as the patch ca9d6597184b ("net/ixgbe: fix Rx queue interrupt mapping in VF"),
the interrupt vector of Rx queues should be mapped to vector 1 to fix above 2nd bug.

Signed-off-by: Wei Dai <wei.dai@intel.com>

Wei Dai (2):
  net/i40e: fix VFIO interrupt mapping in VF
  net/i40e: fix Rx queue interrupt mapping in VF

 drivers/net/i40e/i40e_ethdev_vf.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

-- 
2.7.5

[PATCH 1/2] net/i40e: fix VFIO interrupt mapping in VF

[Wei Dai]

When a VF port is bound to VFIO-PIC, only miscellaneous interrupt
is mapped to VFIO vector 0 in i40evf_dev_init( ).
In ixgbevf_dev_interrupt_handler( ), if previous VFIO interrupt
mapping set in i40evf_dev_init( ) is not cleard, it will fail when
calling rte_intr_enable( ) tries to map Rx queue interrupt to other
VFIO vectors. This patch clears the VFIO interrupt mappings before
setting both miscellaneous and Rx queue interrupt mappings again
to avoid failure.

Fixes: 4b90a3ff26c5 ("i40evf: support Rx interrupt")
Cc: stable@dpdk.org

Signed-off-by: Wei Dai <wei.dai@intel.com>
---
 drivers/net/i40e/i40e_ethdev_vf.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/net/i40e/i40e_ethdev_vf.c b/drivers/net/i40e/i40e_ethdev_vf.c
index 3b76c9e..567b7d0 100644
--- a/drivers/net/i40e/i40e_ethdev_vf.c
+++ b/drivers/net/i40e/i40e_ethdev_vf.c
@@ -1432,8 +1432,9 @@ i40evf_dev_interrupt_handler(void *param)
 				   " do nothing");
 
 done:
-	i40evf_enable_irq0(hw);
+	rte_intr_disable(dev->intr_handle);
 	rte_intr_enable(dev->intr_handle);
+	i40evf_enable_irq0(hw);
 }
 
 static int
-- 
2.7.5

Re: [PATCH 1/2] net/i40e: fix VFIO interrupt mapping in VF

[Wu, Jingjing]

> -----Original Message-----
> From: Dai, Wei
> Sent: Thursday, November 2, 2017 9:11 PM
> To: Wu, Jingjing <jingjing.wu@intel.com>; Xing, Beilei <beilei.xing@intel.com>;
> Liang, Cunming <cunming.liang@intel.com>
> Cc: dev@dpdk.org; Dai, Wei <wei.dai@intel.com>; stable@dpdk.org
> Subject: [PATCH 1/2] net/i40e: fix VFIO interrupt mapping in VF
> 
> When a VF port is bound to VFIO-PIC, only miscellaneous interrupt is mapped to
> VFIO vector 0 in i40evf_dev_init( ).
> In ixgbevf_dev_interrupt_handler( ), if previous VFIO interrupt mapping set in

Ixgbevf? Should be i40evf?

> i40evf_dev_init( ) is not cleard, it will fail when calling rte_intr_enable( ) tries
> to map Rx queue interrupt to other VFIO vectors. This patch clears the VFIO
> interrupt mappings before setting both miscellaneous and Rx queue interrupt
> mappings again to avoid failure.
> 
> Fixes: 4b90a3ff26c5 ("i40evf: support Rx interrupt")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Wei Dai <wei.dai@intel.com>
> ---
>  drivers/net/i40e/i40e_ethdev_vf.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/net/i40e/i40e_ethdev_vf.c
> b/drivers/net/i40e/i40e_ethdev_vf.c
> index 3b76c9e..567b7d0 100644
> --- a/drivers/net/i40e/i40e_ethdev_vf.c
> +++ b/drivers/net/i40e/i40e_ethdev_vf.c
> @@ -1432,8 +1432,9 @@ i40evf_dev_interrupt_handler(void *param)
>  				   " do nothing");
> 
>  done:
> -	i40evf_enable_irq0(hw);
> +	rte_intr_disable(dev->intr_handle);
>  	rte_intr_enable(dev->intr_handle);
> +	i40evf_enable_irq0(hw);

If that reason, you can move the rte_intr_disable to dev_start when deal with the rxq setting in intr_conf.
>  }
> 
>  static int
> --
> 2.7.5

Re: [PATCH 1/2] net/i40e: fix VFIO interrupt mapping in VF

[Dai, Wei]

Hi, Jingjing
Thanks for your feedback.

> > Subject: [PATCH 1/2] net/i40e: fix VFIO interrupt mapping in VF
> >
> > When a VF port is bound to VFIO-PIC, only miscellaneous interrupt is
> > mapped to VFIO vector 0 in i40evf_dev_init( ).
> > In ixgbevf_dev_interrupt_handler( ), if previous VFIO interrupt
> > mapping set in
> 
> Ixgbevf? Should be i40evf?
Yes, it is my typo error, it should be i40evf. 
> 
> > i40evf_dev_init( ) is not cleard, it will fail when calling
> > rte_intr_enable( ) tries to map Rx queue interrupt to other VFIO
> > vectors. This patch clears the VFIO interrupt mappings before setting
> > both miscellaneous and Rx queue interrupt mappings again to avoid
> failure.
> >
> > Fixes: 4b90a3ff26c5 ("i40evf: support Rx interrupt")
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Wei Dai <wei.dai@intel.com>
> > ---
> >  drivers/net/i40e/i40e_ethdev_vf.c | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/net/i40e/i40e_ethdev_vf.c
> > b/drivers/net/i40e/i40e_ethdev_vf.c
> > index 3b76c9e..567b7d0 100644
> > --- a/drivers/net/i40e/i40e_ethdev_vf.c
> > +++ b/drivers/net/i40e/i40e_ethdev_vf.c
> > @@ -1432,8 +1432,9 @@ i40evf_dev_interrupt_handler(void *param)
> >  				   " do nothing");
> >
> >  done:
> > -	i40evf_enable_irq0(hw);
> > +	rte_intr_disable(dev->intr_handle);
> >  	rte_intr_enable(dev->intr_handle);
> > +	i40evf_enable_irq0(hw);
> 
> If that reason, you can move the rte_intr_disable to dev_start when deal
> with the rxq setting in intr_conf.

Yes, I have followed your suggestion and tested it with my following code change.
it works well.
The function rte_intr_disable( ) can be moved into i40evf_dev_start( ) and only
be called it if rx queue interrupt is enabled.
Meanwhile, the calling of rte_intr_enable( ) in i40evf_dev_interrupt_handle( )
and i40evf_dev_rx_queue_intr_enable( ) can also be removed.


> >  }
> >
> >  static int
> > --
> > 2.7.5

[PATCH 2/2] net/i40e: fix Rx queue interrupt mapping in VF

[Wei Dai]

When a VF port is bound to VFIO-PCI, miscellaneous interrupt is
mapped to MSI-X vector 0 and Rx queues interrupt are mapped to
other vectors in vfio_enable_msix( ). To simplify implementation,
all VFIO-PCI bound i40e VF Rx queue interrupts can be mapped in
vector 1. And as current igb_uio only support only one vector,
i40e VF PMD should use vector 0 for igb_uio and vector 1 for
VFIO-PCI. Without this patch, VF Rx queue interrupt is mapped
to vector 0 in register settings and mapped to VFIO vector 1
in vfio_enable_msix( ), and then all Rx queue interrupts will
be missed.

Fixes: 4b90a3ff26c5 ("i40evf: support Rx interrupt")
Fixes: 975ffea6f671 ("net/i40e: remove DPDK PF version specific code")
Cc: stable@dpdk.org

Signed-off-by: Wei Dai <wei.dai@intel.com>
---
 drivers/net/i40e/i40e_ethdev_vf.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/net/i40e/i40e_ethdev_vf.c b/drivers/net/i40e/i40e_ethdev_vf.c
index 567b7d0..94fb4b1 100644
--- a/drivers/net/i40e/i40e_ethdev_vf.c
+++ b/drivers/net/i40e/i40e_ethdev_vf.c
@@ -654,7 +654,7 @@ i40evf_config_irq_map(struct rte_eth_dev *dev)
 	int i, err;
 
 	if (rte_intr_allow_others(intr_handle))
-		vector_id = I40EVF_VSI_DEFAULT_MSIX_INTR_LNX;
+		vector_id = I40EVF_VSI_DEFAULT_MSIX_INTR;
 	else
 		vector_id = I40E_MISC_VEC_ID;
 
-- 
2.7.5

Re: [PATCH 2/2] net/i40e: fix Rx queue interrupt mapping in VF

[Wu, Jingjing]

> -----Original Message-----
> From: Dai, Wei
> Sent: Thursday, November 2, 2017 9:11 PM
> To: Wu, Jingjing <jingjing.wu@intel.com>; Xing, Beilei <beilei.xing@intel.com>;
> Liang, Cunming <cunming.liang@intel.com>
> Cc: dev@dpdk.org; Dai, Wei <wei.dai@intel.com>; stable@dpdk.org
> Subject: [PATCH 2/2] net/i40e: fix Rx queue interrupt mapping in VF
> 
> When a VF port is bound to VFIO-PCI, miscellaneous interrupt is mapped to
> MSI-X vector 0 and Rx queues interrupt are mapped to other vectors in
> vfio_enable_msix( ). To simplify implementation, all VFIO-PCI bound i40e VF Rx
> queue interrupts can be mapped in vector 1. And as current igb_uio only
> support only one vector, i40e VF PMD should use vector 0 for igb_uio and
> vector 1 for VFIO-PCI. Without this patch, VF Rx queue interrupt is mapped to
> vector 0 in register settings and mapped to VFIO vector 1 in
> vfio_enable_msix( ), and then all Rx queue interrupts will be missed.
> 
> Fixes: 4b90a3ff26c5 ("i40evf: support Rx interrupt")
> Fixes: 975ffea6f671 ("net/i40e: remove DPDK PF version specific code")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Wei Dai <wei.dai@intel.com>
> ---
>  drivers/net/i40e/i40e_ethdev_vf.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/net/i40e/i40e_ethdev_vf.c
> b/drivers/net/i40e/i40e_ethdev_vf.c
> index 567b7d0..94fb4b1 100644
> --- a/drivers/net/i40e/i40e_ethdev_vf.c
> +++ b/drivers/net/i40e/i40e_ethdev_vf.c
> @@ -654,7 +654,7 @@ i40evf_config_irq_map(struct rte_eth_dev *dev)
>  	int i, err;
> 
>  	if (rte_intr_allow_others(intr_handle))
> -		vector_id = I40EVF_VSI_DEFAULT_MSIX_INTR_LNX;
> +		vector_id = I40EVF_VSI_DEFAULT_MSIX_INTR;

Firstly, We can use I40E_MISC_VEC_ID and I40E_RX_VEC_START to replace I40EVF_VSI_DEFAULT_MSIX_INTR_LNX and I40EVF_VSI_DEFAULT_MSIX_INTR.

You need to check if rx interrupt is enabled or not, if not we should use I40E_MISC_VEC_ID.
If rx interrupt is enabled, and rte_intr_allow_others(intr_handle) is true, then use I40E_RX_VEC_START.

And I think in VFIO cases, we can use more than one interrupt for the IRQ mapping. You can refer to the function i40e_vsi_queues_bind_intr.

Re: [PATCH 2/2] net/i40e: fix Rx queue interrupt mapping in VF

[Dai, Wei]

Hi, Jingjing 
Thanks for your feedback.
 
> > Subject: [PATCH 2/2] net/i40e: fix Rx queue interrupt mapping in VF
> >
> > When a VF port is bound to VFIO-PCI, miscellaneous interrupt is mapped
> > to MSI-X vector 0 and Rx queues interrupt are mapped to other vectors
> > in vfio_enable_msix( ). To simplify implementation, all VFIO-PCI bound
> > i40e VF Rx queue interrupts can be mapped in vector 1. And as current
> > igb_uio only support only one vector, i40e VF PMD should use vector 0
> > for igb_uio and vector 1 for VFIO-PCI. Without this patch, VF Rx queue
> > interrupt is mapped to vector 0 in register settings and mapped to
> > VFIO vector 1 in vfio_enable_msix( ), and then all Rx queue interrupts will
> be missed.
> >
> > Fixes: 4b90a3ff26c5 ("i40evf: support Rx interrupt")
> > Fixes: 975ffea6f671 ("net/i40e: remove DPDK PF version specific code")
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Wei Dai <wei.dai@intel.com>
> > ---
> >  drivers/net/i40e/i40e_ethdev_vf.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/drivers/net/i40e/i40e_ethdev_vf.c
> > b/drivers/net/i40e/i40e_ethdev_vf.c
> > index 567b7d0..94fb4b1 100644
> > --- a/drivers/net/i40e/i40e_ethdev_vf.c
> > +++ b/drivers/net/i40e/i40e_ethdev_vf.c
> > @@ -654,7 +654,7 @@ i40evf_config_irq_map(struct rte_eth_dev *dev)
> >  	int i, err;
> >
> >  	if (rte_intr_allow_others(intr_handle))
> > -		vector_id = I40EVF_VSI_DEFAULT_MSIX_INTR_LNX;
> > +		vector_id = I40EVF_VSI_DEFAULT_MSIX_INTR;
> 
> Firstly, We can use I40E_MISC_VEC_ID and I40E_RX_VEC_START to replace
> I40EVF_VSI_DEFAULT_MSIX_INTR_LNX and
> I40EVF_VSI_DEFAULT_MSIX_INTR.
It looks good using I40E_RX_VEC_START instead of I40EVF_VSI_DEFAULT_MSIX_INTR.
> 
> You need to check if rx interrupt is enabled or not, if not we should use
> I40E_MISC_VEC_ID.
> If rx interrupt is enabled, and rte_intr_allow_others(intr_handle) is true, then
> use I40E_RX_VEC_START.
Yes I have tested it and confirm Rx interrupt is enabled with l3fwd-power.
> 
> And I think in VFIO cases, we can use more than one interrupt for the IRQ
> mapping. You can refer to the function i40e_vsi_queues_bind_intr.
> 

[PATCH v2 0/2] Fix 2 bugs of i40e VF interrupt found in l3fwd-power

[Wei Dai]

These 2 bugs can be observed from example/l3fwd-power run with i40e VF
bound to VFIO-PCI. The test steps are as follows:
1. Disable LSC interrupt by clearing port_conf.intr_conf.lsc=0, as i40e
   VF doesn't support LSC interrupt to avoid rte_eth_dev_configure() failure.
2. Create a VF from i40e host PF. Let PF run with kernel driver and bind
   its VF to VFIO-PCI
3. Run l3fwd-power like: l3fwd-power -l 18-19 -- -p 0x1 --config='(0,0,19)'

Then, the following error message appears like:
EAL: Error enabling MSI-X interrupts for fd 18
This error is from rte_intr_enable( )/vfio_enable_msix( ) when enabling
Rx queue interrupt.
Same as the patch 06938770186a ("net/ixgbe: fix VFIO interrupt mapping in VF"),
to change VFIO MSI-X interrupts mapping, previous mapping should be cleared
first to avoid above error.

After fixing above VFIO-PCI MSI-X interrupt mapping. There is still the following
2nd bug: l3fwd-power still can not be waked up by incoming packets.

Same as the patch ca9d6597184b ("net/ixgbe: fix Rx queue interrupt mapping in VF"),
the interrupt vector of Rx queues should be mapped to vector 1 to fix above 2nd bug.

These patches have passed test with l3fwd-power using i40e VF bound to VFIO-PCI.
They also passed the test with testpmd rxonly and txonly mode with igb_uio and VFIO-PCI.

Signed-off-by: Wei Dai <wei.dai@intel.com>

---
v2: only remap VFIO interrupt in i40evf_dev_start( )


Wei Dai (2):
  net/i40e: fix VFIO interrupt mapping in VF
  net/i40e: fix Rx queue interrupt mapping in VF

 drivers/net/i40e/i40e_ethdev_vf.c | 18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)

-- 
2.7.4

[PATCH v2 1/2] net/i40e: fix VFIO interrupt mapping in VF

[Wei Dai]

When a VF port is bound to VFIO-PIC, only miscellaneous interrupt
is mapped to VFIO vector 0 in i40evf_dev_init( ).
In i40evf_dev_interrupt_handle( ) and i40evf_dev_rx_queue_intr_enable( ),
if previous VFIO interrupt mapping set in i40evf_dev_init( ) is not
cleared, it will fail when PMD tries to map Rx queue interrupt to other
VFIO vectors by calling rte_intr_enable( ).
This patch clears the VFIO interrupt mappings before setting both
miscellaneous and Rx queue interrupt mappings again to avoid failure.
And remove the calling of rte_intr_enable( ) in
i40evf_dev_interrupt_handle( ) and i40evf_dev_rx_queue_intr_enable( )
as VFIO interrupt mapping can only be set in i40e_dev_start( ) and it
is not necessary to be set in above 2 functions repeatedly.

Fixes: 4b90a3ff26c5 ("i40evf: support Rx interrupt")
Cc: stable@dpdk.org

Signed-off-by: Wei Dai <wei.dai@intel.com>
---
 drivers/net/i40e/i40e_ethdev_vf.c | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/drivers/net/i40e/i40e_ethdev_vf.c b/drivers/net/i40e/i40e_ethdev_vf.c
index 3b76c9e..46dac86 100644
--- a/drivers/net/i40e/i40e_ethdev_vf.c
+++ b/drivers/net/i40e/i40e_ethdev_vf.c
@@ -1433,7 +1433,6 @@ i40evf_dev_interrupt_handler(void *param)
 
 done:
 	i40evf_enable_irq0(hw);
-	rte_intr_enable(dev->intr_handle);
 }
 
 static int
@@ -1888,8 +1887,6 @@ i40evf_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
 
 	I40EVF_WRITE_FLUSH(hw);
 
-	rte_intr_enable(&pci_dev->intr_handle);
-
 	return 0;
 }
 
@@ -2041,7 +2038,20 @@ i40evf_dev_start(struct rte_eth_dev *dev)
 		goto err_mac;
 	}
 
+	/* When a VF port is bound to VFIO-PCI, only miscellaneous interrupt
+	 * is mapped to VFIO vector 0 in i40evf_dev_init( ).
+	 * If previous VFIO interrupt mapping set in i40evf_dev_init( ) is
+	 * not cleared, it will fail when rte_intr_enable( ) tries to map Rx
+	 * queue interrupt to other VFIO vectors.
+	 * So clear uio/vfio intr/evevnfd first to avoid failure.
+	 */
+	if (dev->data->dev_conf.intr_conf.rxq != 0) {
+		rte_intr_disable(intr_handle);
+		rte_intr_enable(intr_handle);
+	}
+
 	i40evf_enable_queues_intr(dev);
+
 	return 0;
 
 err_mac:
-- 
2.7.4

[PATCH v2 2/2] net/i40e: fix Rx queue interrupt mapping in VF

[Wei Dai]

When a VF port is bound to VFIO-PCI, miscellaneous interrupt is
mapped to MSI-X vector 0 and Rx queues interrupt are mapped to
other vectors in vfio_enable_msix( ). To simplify implementation,
all VFIO-PCI bound i40e VF Rx queue interrupts can be mapped in
vector 1. And as current igb_uio only support only one vector,
i40e VF PMD should use vector 0 for igb_uio and vector 1 for
VFIO-PCI. Without this patch, VF Rx queue interrupt is mapped
to vector 0 in register settings and mapped to VFIO vector 1
in vfio_enable_msix( ), and then all Rx queue interrupts will
be missed.

Fixes: 4b90a3ff26c5 ("i40evf: support Rx interrupt")
Fixes: 975ffea6f671 ("net/i40e: remove DPDK PF version specific code")
Cc: stable@dpdk.org

Signed-off-by: Wei Dai <wei.dai@intel.com>
---
 drivers/net/i40e/i40e_ethdev_vf.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/net/i40e/i40e_ethdev_vf.c b/drivers/net/i40e/i40e_ethdev_vf.c
index 46dac86..70ff3fd 100644
--- a/drivers/net/i40e/i40e_ethdev_vf.c
+++ b/drivers/net/i40e/i40e_ethdev_vf.c
@@ -654,7 +654,7 @@ i40evf_config_irq_map(struct rte_eth_dev *dev)
 	int i, err;
 
 	if (rte_intr_allow_others(intr_handle))
-		vector_id = I40EVF_VSI_DEFAULT_MSIX_INTR_LNX;
+		vector_id = I40E_RX_VEC_START;
 	else
 		vector_id = I40E_MISC_VEC_ID;
 
-- 
2.7.4

Re: [PATCH v2 0/2] Fix 2 bugs of i40e VF interrupt found in l3fwd-power

[Yao, Lei A]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Wei Dai
> Sent: Friday, November 3, 2017 3:14 PM
> To: Wu, Jingjing <jingjing.wu@intel.com>; Xing, Beilei
> <beilei.xing@intel.com>; Liang, Cunming <cunming.liang@intel.com>
> Cc: dev@dpdk.org; Dai, Wei <wei.dai@intel.com>
> Subject: [dpdk-dev] [PATCH v2 0/2] Fix 2 bugs of i40e VF interrupt found in
> l3fwd-power
> 
> These 2 bugs can be observed from example/l3fwd-power run with i40e VF
> bound to VFIO-PCI. The test steps are as follows:
> 1. Disable LSC interrupt by clearing port_conf.intr_conf.lsc=0, as i40e
>    VF doesn't support LSC interrupt to avoid rte_eth_dev_configure() failure.
> 2. Create a VF from i40e host PF. Let PF run with kernel driver and bind
>    its VF to VFIO-PCI
> 3. Run l3fwd-power like: l3fwd-power -l 18-19 -- -p 0x1 --config='(0,0,19)'
> 
> Then, the following error message appears like:
> EAL: Error enabling MSI-X interrupts for fd 18
> This error is from rte_intr_enable( )/vfio_enable_msix( ) when enabling
> Rx queue interrupt.
> Same as the patch 06938770186a ("net/ixgbe: fix VFIO interrupt mapping in
> VF"),
> to change VFIO MSI-X interrupts mapping, previous mapping should be
> cleared
> first to avoid above error.
> 
> After fixing above VFIO-PCI MSI-X interrupt mapping. There is still the
> following
> 2nd bug: l3fwd-power still can not be waked up by incoming packets.
> 
> Same as the patch ca9d6597184b ("net/ixgbe: fix Rx queue interrupt mapping
> in VF"),
> the interrupt vector of Rx queues should be mapped to vector 1 to fix above
> 2nd bug.
> 
> These patches have passed test with l3fwd-power using i40e VF bound to
> VFIO-PCI.
> They also passed the test with testpmd rxonly and txonly mode with igb_uio
> and VFIO-PCI.
> 
> Signed-off-by: Wei Dai <wei.dai@intel.com>
Tested-by: Lei Yao<lei.a.yao@intel.com>
> 
> ---
> v2: only remap VFIO interrupt in i40evf_dev_start( )
> 
> 
> Wei Dai (2):
>   net/i40e: fix VFIO interrupt mapping in VF
>   net/i40e: fix Rx queue interrupt mapping in VF
> 
>  drivers/net/i40e/i40e_ethdev_vf.c | 18 ++++++++++++++----
>  1 file changed, 14 insertions(+), 4 deletions(-)
> 
> --
> 2.7.4
Following test case is tested and passed with this patch.
L3fwd-power with 1 i40e PF  (vfio-pci) can receive interrupt
L3fwd-power with 1 i40e VF  (vfio-pci) can receive interrupt
L3fwd-power with  2 i40e VF (vfio-pci) can receive interrupt

OS: Ubuntu 16.04
CPU: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
NIC: Ethernet Controller X710

[PATCH v3 0/2] Fix 2 bugs of i40e VF interrupt found in l3fwd-power

[Wei Dai]

These 2 bugs can be observed from example/l3fwd-power run with i40e VF
bound to VFIO-PCI. The test steps are as follows:
1. Disable LSC interrupt by clearing port_conf.intr_conf.lsc=0, as i40e
   VF doesn't support LSC interrupt to avoid rte_eth_dev_configure() failure.
2. Create a VF from i40e host PF. Let PF run with kernel driver and bind
   its VF to VFIO-PCI
3. Run l3fwd-power like: l3fwd-power -l 18-19 -- -p 0x1 --config='(0,0,19)'

Then, the following error message appears like:
EAL: Error enabling MSI-X interrupts for fd 18
This error is from rte_intr_enable( )/vfio_enable_msix( ) when enabling
Rx queue interrupt.
Same as the patch 06938770186a ("net/ixgbe: fix VFIO interrupt mapping in VF"),
to change VFIO MSI-X interrupts mapping, previous mapping should be cleared
first to avoid above error.

After fixing above VFIO-PCI MSI-X interrupt mapping. There is still the following
2nd bug: l3fwd-power still can not be waked up by incoming packets.

Same as the patch ca9d6597184b ("net/ixgbe: fix Rx queue interrupt mapping in VF"),
the interrupt vector of Rx queues should be mapped to vector 1 to fix above 2nd bug.

These patches have passed test with l3fwd-power using i40e VF bound to VFIO-PCI.
They also passed the test with testpmd rxonly and txonly mode with igb_uio and VFIO-PCI.

Signed-off-by: Wei Dai <wei.dai@intel.com>
Tested-by: Lei Yao <lei.a.yao@intel.com>

---
v3:
  keep rte_intr_enable( ) for safety and align the style with other PMD.
  remove 2 unsed macros.

v2:
  only remap VFIO interrupt in i40evf_dev_start( )

Wei Dai (2):
  net/i40e: fix VFIO interrupt mapping in VF
  net/i40e: fix Rx queue interrupt mapping in VF

 drivers/net/i40e/i40e_ethdev_vf.c | 18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)

-- 
2.7.4

[PATCH v3 1/2] net/i40e: fix VFIO interrupt mapping in VF

[Wei Dai]

When a VF port is bound to VFIO-PIC, only miscellaneous interrupt
is mapped to VFIO vector 0 in i40evf_dev_init( ).
In i40evf_dev_interrupt_handle( ) and i40evf_dev_rx_queue_intr_enable( ),
if previous VFIO interrupt mapping set in i40evf_dev_init( ) is not
cleared, it will fail when PMD tries to map Rx queue interrupt to other
VFIO vectors by calling rte_intr_enable( ).
This patch clears the VFIO interrupt mappings before setting both
miscellaneous and Rx queue interrupt mappings again to avoid failure.
And remove the calling of rte_intr_enable( ) in
i40evf_dev_interrupt_handler( ) as there is no need to map VFIO interrupt
in this function repeatedly.

Fixes: 4b90a3ff26c5 ("i40evf: support Rx interrupt")
Cc: stable@dpdk.org

Signed-off-by: Wei Dai <wei.dai@intel.com>
---
 drivers/net/i40e/i40e_ethdev_vf.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/drivers/net/i40e/i40e_ethdev_vf.c b/drivers/net/i40e/i40e_ethdev_vf.c
index 3b76c9e..f2af022 100644
--- a/drivers/net/i40e/i40e_ethdev_vf.c
+++ b/drivers/net/i40e/i40e_ethdev_vf.c
@@ -1433,7 +1433,6 @@ i40evf_dev_interrupt_handler(void *param)
 
 done:
 	i40evf_enable_irq0(hw);
-	rte_intr_enable(dev->intr_handle);
 }
 
 static int
@@ -2041,7 +2040,20 @@ i40evf_dev_start(struct rte_eth_dev *dev)
 		goto err_mac;
 	}
 
+	/* When a VF port is bound to VFIO-PCI, only miscellaneous interrupt
+	 * is mapped to VFIO vector 0 in i40evf_dev_init( ).
+	 * If previous VFIO interrupt mapping set in i40evf_dev_init( ) is
+	 * not cleared, it will fail when rte_intr_enable( ) tries to map Rx
+	 * queue interrupt to other VFIO vectors.
+	 * So clear uio/vfio intr/evevnfd first to avoid failure.
+	 */
+	if (dev->data->dev_conf.intr_conf.rxq != 0) {
+		rte_intr_disable(intr_handle);
+		rte_intr_enable(intr_handle);
+	}
+
 	i40evf_enable_queues_intr(dev);
+
 	return 0;
 
 err_mac:
-- 
2.7.4

Re: [PATCH v3 1/2] net/i40e: fix VFIO interrupt mapping in VF

[Wu, Jingjing]

> -----Original Message-----
> From: Dai, Wei
> Sent: Friday, November 3, 2017 4:47 PM
> To: Wu, Jingjing <jingjing.wu@intel.com>; Xing, Beilei <beilei.xing@intel.com>
> Cc: dev@dpdk.org; Dai, Wei <wei.dai@intel.com>; stable@dpdk.org
> Subject: [PATCH v3 1/2] net/i40e: fix VFIO interrupt mapping in VF
> 
> When a VF port is bound to VFIO-PIC, only miscellaneous interrupt
> is mapped to VFIO vector 0 in i40evf_dev_init( ).
> In i40evf_dev_interrupt_handle( ) and i40evf_dev_rx_queue_intr_enable( ),
> if previous VFIO interrupt mapping set in i40evf_dev_init( ) is not
> cleared, it will fail when PMD tries to map Rx queue interrupt to other
> VFIO vectors by calling rte_intr_enable( ).
> This patch clears the VFIO interrupt mappings before setting both
> miscellaneous and Rx queue interrupt mappings again to avoid failure.
> And remove the calling of rte_intr_enable( ) in
> i40evf_dev_interrupt_handler( ) as there is no need to map VFIO interrupt
> in this function repeatedly.
> 
> Fixes: 4b90a3ff26c5 ("i40evf: support Rx interrupt")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Wei Dai <wei.dai@intel.com>

Acked-by: Jingjing Wu <jingjing.wu@intel.com>

[PATCH v3 2/2] net/i40e: fix Rx queue interrupt mapping in VF

[Wei Dai]

When a VF port is bound to VFIO-PCI, miscellaneous interrupt is
mapped to MSI-X vector 0 and Rx queues interrupt are mapped to
other vectors in vfio_enable_msix( ). To simplify implementation,
all VFIO-PCI bound i40e VF Rx queue interrupts can be mapped in
vector 1. And as current igb_uio only support only one vector,
i40e VF PMD should use vector 0 for igb_uio and vector 1 for
VFIO-PCI. Without this patch, VF Rx queue interrupt is mapped
to vector 0 in register settings and mapped to VFIO vector 1
in vfio_enable_msix( ), and then all Rx queue interrupts will
be missed.
Also remove 2 unsed macro definitions.

Fixes: 4b90a3ff26c5 ("i40evf: support Rx interrupt")
Fixes: 975ffea6f671 ("net/i40e: remove DPDK PF version specific code")
Cc: stable@dpdk.org

Signed-off-by: Wei Dai <wei.dai@intel.com>
---
 drivers/net/i40e/i40e_ethdev_vf.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/drivers/net/i40e/i40e_ethdev_vf.c b/drivers/net/i40e/i40e_ethdev_vf.c
index f2af022..1118c74 100644
--- a/drivers/net/i40e/i40e_ethdev_vf.c
+++ b/drivers/net/i40e/i40e_ethdev_vf.c
@@ -68,8 +68,6 @@
 #include "i40e_rxtx.h"
 #include "i40e_ethdev.h"
 #include "i40e_pf.h"
-#define I40EVF_VSI_DEFAULT_MSIX_INTR     1
-#define I40EVF_VSI_DEFAULT_MSIX_INTR_LNX 0
 
 /* busy wait delay in msec */
 #define I40EVF_BUSY_WAIT_DELAY 10
@@ -654,7 +652,7 @@ i40evf_config_irq_map(struct rte_eth_dev *dev)
 	int i, err;
 
 	if (rte_intr_allow_others(intr_handle))
-		vector_id = I40EVF_VSI_DEFAULT_MSIX_INTR_LNX;
+		vector_id = I40E_RX_VEC_START;
 	else
 		vector_id = I40E_MISC_VEC_ID;
 
-- 
2.7.4

Re: [PATCH v3 2/2] net/i40e: fix Rx queue interrupt mapping in VF

[Wu, Jingjing]

> -----Original Message-----
> From: Dai, Wei
> Sent: Friday, November 3, 2017 4:48 PM
> To: Wu, Jingjing <jingjing.wu@intel.com>; Xing, Beilei <beilei.xing@intel.com>
> Cc: dev@dpdk.org; Dai, Wei <wei.dai@intel.com>; stable@dpdk.org
> Subject: [PATCH v3 2/2] net/i40e: fix Rx queue interrupt mapping in VF
> 
> When a VF port is bound to VFIO-PCI, miscellaneous interrupt is
> mapped to MSI-X vector 0 and Rx queues interrupt are mapped to
> other vectors in vfio_enable_msix( ). To simplify implementation,
> all VFIO-PCI bound i40e VF Rx queue interrupts can be mapped in
> vector 1. And as current igb_uio only support only one vector,
> i40e VF PMD should use vector 0 for igb_uio and vector 1 for
> VFIO-PCI. Without this patch, VF Rx queue interrupt is mapped
> to vector 0 in register settings and mapped to VFIO vector 1
> in vfio_enable_msix( ), and then all Rx queue interrupts will
> be missed.
> Also remove 2 unsed macro definitions.
> 
> Fixes: 4b90a3ff26c5 ("i40evf: support Rx interrupt")
> Fixes: 975ffea6f671 ("net/i40e: remove DPDK PF version specific code")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Wei Dai <wei.dai@intel.com>
Acked-by: Jingjing Wu <jingjing.wu@intel.com>

Re: [PATCH v3 0/2] Fix 2 bugs of i40e VF interrupt found in l3fwd-power

[Dai, Wei]

> -----Original Message-----
> From: Dai, Wei
> Sent: Friday, November 3, 2017 4:47 PM
> To: Wu, Jingjing <jingjing.wu@intel.com>; Xing, Beilei
> <beilei.xing@intel.com>
> Cc: dev@dpdk.org; Dai, Wei <wei.dai@intel.com>
> Subject: [PATCH v3 0/2] Fix 2 bugs of i40e VF interrupt found in l3fwd-power
> 
> These 2 bugs can be observed from example/l3fwd-power run with i40e VF
> bound to VFIO-PCI. The test steps are as follows:
> 1. Disable LSC interrupt by clearing port_conf.intr_conf.lsc=0, as i40e
>    VF doesn't support LSC interrupt to avoid rte_eth_dev_configure()
> failure.
> 2. Create a VF from i40e host PF. Let PF run with kernel driver and bind
>    its VF to VFIO-PCI
> 3. Run l3fwd-power like: l3fwd-power -l 18-19 -- -p 0x1 --config='(0,0,19)'
> 
> Then, the following error message appears like:
> EAL: Error enabling MSI-X interrupts for fd 18 This error is from
> rte_intr_enable( )/vfio_enable_msix( ) when enabling Rx queue interrupt.
> Same as the patch 06938770186a ("net/ixgbe: fix VFIO interrupt mapping in
> VF"), to change VFIO MSI-X interrupts mapping, previous mapping should be
> cleared first to avoid above error.
> 
> After fixing above VFIO-PCI MSI-X interrupt mapping. There is still the
> following 2nd bug: l3fwd-power still can not be waked up by incoming
> packets.
> 
> Same as the patch ca9d6597184b ("net/ixgbe: fix Rx queue interrupt
> mapping in VF"), the interrupt vector of Rx queues should be mapped to
> vector 1 to fix above 2nd bug.
> 
> These patches have passed test with l3fwd-power using i40e VF bound to
> VFIO-PCI.
> They also passed the test with testpmd rxonly and txonly mode with igb_uio
> and VFIO-PCI.
> 
> Signed-off-by: Wei Dai <wei.dai@intel.com>
> Tested-by: Lei Yao <lei.a.yao@intel.com>
> 
> ---
> v3:
>   keep rte_intr_enable( ) for safety and align the style with other PMD.
Keep rte_intr_enable( )  in i40evf_dev_rx_queue_intr_enable( ) for safety
And align the style with other PMD.
>   remove 2 unsed macros.
> 
> v2:
>   only remap VFIO interrupt in i40evf_dev_start( )
> 
> Wei Dai (2):
>   net/i40e: fix VFIO interrupt mapping in VF
>   net/i40e: fix Rx queue interrupt mapping in VF
> 
>  drivers/net/i40e/i40e_ethdev_vf.c | 18 ++++++++++++++----
>  1 file changed, 14 insertions(+), 4 deletions(-)
> 
> --
> 2.7.4

Re: [PATCH v3 0/2] Fix 2 bugs of i40e VF interrupt found in l3fwd-power

[Ferruh Yigit]

On 11/3/2017 1:47 AM, Wei Dai wrote:
> These 2 bugs can be observed from example/l3fwd-power run with i40e VF
> bound to VFIO-PCI. The test steps are as follows:
> 1. Disable LSC interrupt by clearing port_conf.intr_conf.lsc=0, as i40e
>    VF doesn't support LSC interrupt to avoid rte_eth_dev_configure() failure.
> 2. Create a VF from i40e host PF. Let PF run with kernel driver and bind
>    its VF to VFIO-PCI
> 3. Run l3fwd-power like: l3fwd-power -l 18-19 -- -p 0x1 --config='(0,0,19)'
> 
> Then, the following error message appears like:
> EAL: Error enabling MSI-X interrupts for fd 18
> This error is from rte_intr_enable( )/vfio_enable_msix( ) when enabling
> Rx queue interrupt.
> Same as the patch 06938770186a ("net/ixgbe: fix VFIO interrupt mapping in VF"),
> to change VFIO MSI-X interrupts mapping, previous mapping should be cleared
> first to avoid above error.
> 
> After fixing above VFIO-PCI MSI-X interrupt mapping. There is still the following
> 2nd bug: l3fwd-power still can not be waked up by incoming packets.
> 
> Same as the patch ca9d6597184b ("net/ixgbe: fix Rx queue interrupt mapping in VF"),
> the interrupt vector of Rx queues should be mapped to vector 1 to fix above 2nd bug.
> 
> These patches have passed test with l3fwd-power using i40e VF bound to VFIO-PCI.
> They also passed the test with testpmd rxonly and txonly mode with igb_uio and VFIO-PCI.
> 
> Signed-off-by: Wei Dai <wei.dai@intel.com>
> Tested-by: Lei Yao <lei.a.yao@intel.com>

Acked-by: Jingjing Wu <jingjing.wu@intel.com>

Series applied to dpdk-next-net/master, thanks.