[BUG] via-rhine: NOHZ: local_softirq_pending 08

[BUG] via-rhine: NOHZ: local_softirq_pending 08

[Jamie Gloudon]

Hey,

   While conducting some tests on a VT6105M card. I noticed this message:

kernel: [  160.311113] NOHZ: local_softirq_pending 08

I was able to reliably reproduce the error message by setting autoneg off and changing the interface speed from 100 to 10 with ethtool.

Regards,
Jamie Gloudon

Re: [BUG] via-rhine: NOHZ: local_softirq_pending 08

[Jamie Gloudon]

On Wed, Jan 23, 2013 at 08:08:29PM -0400, Jamie Gloudon wrote:
> Hey,
> 
>    While conducting some tests on a VT6105M card. I noticed this message:
> 
> kernel: [  160.311113] NOHZ: local_softirq_pending 08
> 
> I was able to reliably reproduce the error message by setting autoneg off and changing the interface speed from 100 to 10 with ethtool.
> 
> Regards,
> Jamie Gloudon

I forget to mention this error message occur on stable 3.7.

Re: [BUG] via-rhine: NOHZ: local_softirq_pending 08

[David Miller]

From: Jamie Gloudon <jamie.gloudon@gmail.com>
Date: Wed, 23 Jan 2013 20:08:29 -0400

>    While conducting some tests on a VT6105M card. I noticed this message:
> 
> kernel: [  160.311113] NOHZ: local_softirq_pending 08
> 
> I was able to reliably reproduce the error message by setting autoneg off and changing the interface speed from 100 to 10 with ethtool.

I wonder if this is due to some oversights during the NAPI changes
that went in about a year ago.

Francois, if you move rhine_tx() work into the driver's NAPI poll
routine, as you did in:

commit 7ab87ff4c770eed71e3777936299292739fcd0fe
Author: Francois Romieu <romieu@fr.zoreil.com>
Date:   Fri Jan 6 21:42:26 2012 +0100

    via-rhine: move work from irq handler to softirq and beyond.

I don't think you can't keep using dev_kfree_skb_irq() there.

Francois any objections to this patch?

====================
via-rhine: Don't use dev_kfree_skb_irq() in softirq context.

When the TX reclaim of this driver was moved into NAPI poll,
using dev_kfree_skb_irq() stopped being appropriate.

Use plain dev_kfree_skb() instead.

Reported-by: Jamie Gloudon <jamie.gloudon@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

diff --git a/drivers/net/ethernet/via/via-rhine.c b/drivers/net/ethernet/via/via-rhine.c
index 7992b3e..f08e970 100644
--- a/drivers/net/ethernet/via/via-rhine.c
+++ b/drivers/net/ethernet/via/via-rhine.c
@@ -1801,7 +1801,7 @@ static void rhine_tx(struct net_device *dev)
 					 rp->tx_skbuff[entry]->len,
 					 PCI_DMA_TODEVICE);
 		}
-		dev_kfree_skb_irq(rp->tx_skbuff[entry]);
+		dev_kfree_skb(rp->tx_skbuff[entry]);
 		rp->tx_skbuff[entry] = NULL;
 		entry = (++rp->dirty_tx) % TX_RING_SIZE;
 	}

Re: [BUG] via-rhine: NOHZ: local_softirq_pending 08

[Francois Romieu]

David Miller <davem@davemloft.net> :
[...]
> I don't think you can't keep using dev_kfree_skb_irq() there.
> 
> Francois any objections to this patch?

None.

I have almost surely broken via-rhine with NOHZ as I broke the r8169
driver before 7dbb491878a2c51d372a8890fa45a8ff80358af1. Something like
the patch below could be needed as well (quick shot before bed time
and ~24h delay if Jamie wants to try something now).

diff --git a/drivers/net/ethernet/via/via-rhine.c b/drivers/net/ethernet/via/via-rhine.c
index 7992b3e..5c7e900 100644
--- a/drivers/net/ethernet/via/via-rhine.c
+++ b/drivers/net/ethernet/via/via-rhine.c
@@ -2010,11 +2010,7 @@ static void rhine_slow_event_task(struct work_struct *work)
 	if (intr_status & IntrPCIErr)
 		netif_warn(rp, hw, dev, "PCI error\n");
 
-	napi_disable(&rp->napi);
-	rhine_irq_disable(rp);
-	/* Slow and safe. Consider __napi_schedule as a replacement ? */
-	napi_enable(&rp->napi);
-	napi_schedule(&rp->napi);
+	iowrite16(RHINE_EVENT & 0xffff, rp->base + IntrEnable);
 
 out_unlock:
 	mutex_unlock(&rp->task_lock);

Re: [BUG] via-rhine: NOHZ: local_softirq_pending 08

[David Miller]

From: Francois Romieu <romieu@fr.zoreil.com>
Date: Wed, 30 Jan 2013 01:04:32 +0100

> David Miller <davem@davemloft.net> :
> [...]
>> I don't think you can't keep using dev_kfree_skb_irq() there.
>> 
>> Francois any objections to this patch?
> 
> None.
> 
> I have almost surely broken via-rhine with NOHZ as I broke the r8169
> driver before 7dbb491878a2c51d372a8890fa45a8ff80358af1. Something like
> the patch below could be needed as well (quick shot before bed time
> and ~24h delay if Jamie wants to try something now).

Good catch, Jamie can you test this combined patch for us?

====================
via-rhine: Fix bugs in NAPI support.

1) rhine_tx() should use dev_kfree_skb() not dev_kfree_skb_irq()

2) rhine_slow_event_task's NAPI triggering logic is racey, it
   should just hit the interrupt mask register.  This is the
   same as commit 7dbb491878a2c51d372a8890fa45a8ff80358af1
   ("r8169: avoid NAPI scheduling delay.") made to fix the same
   problem in the r8169 driver.  From Francois Romieu.

Reported-by: Jamie Gloudon <jamie.gloudon@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

diff --git a/drivers/net/ethernet/via/via-rhine.c b/drivers/net/ethernet/via/via-rhine.c
index 7992b3e..78ace59 100644
--- a/drivers/net/ethernet/via/via-rhine.c
+++ b/drivers/net/ethernet/via/via-rhine.c
@@ -1801,7 +1801,7 @@ static void rhine_tx(struct net_device *dev)
 					 rp->tx_skbuff[entry]->len,
 					 PCI_DMA_TODEVICE);
 		}
-		dev_kfree_skb_irq(rp->tx_skbuff[entry]);
+		dev_kfree_skb(rp->tx_skbuff[entry]);
 		rp->tx_skbuff[entry] = NULL;
 		entry = (++rp->dirty_tx) % TX_RING_SIZE;
 	}
@@ -2010,11 +2010,7 @@ static void rhine_slow_event_task(struct work_struct *work)
 	if (intr_status & IntrPCIErr)
 		netif_warn(rp, hw, dev, "PCI error\n");
 
-	napi_disable(&rp->napi);
-	rhine_irq_disable(rp);
-	/* Slow and safe. Consider __napi_schedule as a replacement ? */
-	napi_enable(&rp->napi);
-	napi_schedule(&rp->napi);
+	iowrite16(RHINE_EVENT & 0xffff, rp->base + IntrEnable);
 
 out_unlock:
 	mutex_unlock(&rp->task_lock);

Re: [BUG] via-rhine: NOHZ: local_softirq_pending 08

[Jamie Gloudon]

On Wed, Jan 30, 2013 at 01:04:32AM +0100, Francois Romieu wrote:
> David Miller <davem@davemloft.net> :
> [...]
> > I don't think you can't keep using dev_kfree_skb_irq() there.
> > 
> > Francois any objections to this patch?
> 
> None.
> 
> I have almost surely broken via-rhine with NOHZ as I broke the r8169
> driver before 7dbb491878a2c51d372a8890fa45a8ff80358af1. Something like
> the patch below could be needed as well (quick shot before bed time
> and ~24h delay if Jamie wants to try something now).
> 
> diff --git a/drivers/net/ethernet/via/via-rhine.c b/drivers/net/ethernet/via/via-rhine.c
> index 7992b3e..5c7e900 100644
> --- a/drivers/net/ethernet/via/via-rhine.c
> +++ b/drivers/net/ethernet/via/via-rhine.c
> @@ -2010,11 +2010,7 @@ static void rhine_slow_event_task(struct work_struct *work)
>  	if (intr_status & IntrPCIErr)
>  		netif_warn(rp, hw, dev, "PCI error\n");
>  
> -	napi_disable(&rp->napi);
> -	rhine_irq_disable(rp);
> -	/* Slow and safe. Consider __napi_schedule as a replacement ? */
> -	napi_enable(&rp->napi);
> -	napi_schedule(&rp->napi);
> +	iowrite16(RHINE_EVENT & 0xffff, rp->base + IntrEnable);
>  
>  out_unlock:
>  	mutex_unlock(&rp->task_lock);

Francois's patch fixed the issue for me. thanks!

Regards,
Jamie Gloudon

Re: [BUG] via-rhine: NOHZ: local_softirq_pending 08

[Jamie Gloudon]

On Tue, Jan 29, 2013 at 10:53:28PM -0500, David Miller wrote:
> From: Francois Romieu <romieu@fr.zoreil.com>
> Date: Wed, 30 Jan 2013 01:04:32 +0100
> 
> > David Miller <davem@davemloft.net> :
> > [...]
> >> I don't think you can't keep using dev_kfree_skb_irq() there.
> >> 
> >> Francois any objections to this patch?
> > 
> > None.
> > 
> > I have almost surely broken via-rhine with NOHZ as I broke the r8169
> > driver before 7dbb491878a2c51d372a8890fa45a8ff80358af1. Something like
> > the patch below could be needed as well (quick shot before bed time
> > and ~24h delay if Jamie wants to try something now).
> 
> Good catch, Jamie can you test this combined patch for us?
> 
> ====================
> via-rhine: Fix bugs in NAPI support.
> 
> 1) rhine_tx() should use dev_kfree_skb() not dev_kfree_skb_irq()
> 
> 2) rhine_slow_event_task's NAPI triggering logic is racey, it
>    should just hit the interrupt mask register.  This is the
>    same as commit 7dbb491878a2c51d372a8890fa45a8ff80358af1
>    ("r8169: avoid NAPI scheduling delay.") made to fix the same
>    problem in the r8169 driver.  From Francois Romieu.
> 
> Reported-by: Jamie Gloudon <jamie.gloudon@gmail.com>
> Signed-off-by: David S. Miller <davem@davemloft.net>
> 
> diff --git a/drivers/net/ethernet/via/via-rhine.c b/drivers/net/ethernet/via/via-rhine.c
> index 7992b3e..78ace59 100644
> --- a/drivers/net/ethernet/via/via-rhine.c
> +++ b/drivers/net/ethernet/via/via-rhine.c
> @@ -1801,7 +1801,7 @@ static void rhine_tx(struct net_device *dev)
>  					 rp->tx_skbuff[entry]->len,
>  					 PCI_DMA_TODEVICE);
>  		}
> -		dev_kfree_skb_irq(rp->tx_skbuff[entry]);
> +		dev_kfree_skb(rp->tx_skbuff[entry]);
>  		rp->tx_skbuff[entry] = NULL;
>  		entry = (++rp->dirty_tx) % TX_RING_SIZE;
>  	}
> @@ -2010,11 +2010,7 @@ static void rhine_slow_event_task(struct work_struct *work)
>  	if (intr_status & IntrPCIErr)
>  		netif_warn(rp, hw, dev, "PCI error\n");
>  
> -	napi_disable(&rp->napi);
> -	rhine_irq_disable(rp);
> -	/* Slow and safe. Consider __napi_schedule as a replacement ? */
> -	napi_enable(&rp->napi);
> -	napi_schedule(&rp->napi);
> +	iowrite16(RHINE_EVENT & 0xffff, rp->base + IntrEnable);
>  
>  out_unlock:
>  	mutex_unlock(&rp->task_lock);

I did some quick netperf test with both patches applied, everything
appears to be fine (no kernel panic or tput performance degradation).