[bisected regression] e1000e: "Detected Hardware Unit Hang"

[bisected regression] e1000e: "Detected Hardware Unit Hang"

[Thomas Jarosch]

Hello,

after updating a good bunch of production level machines
from kernel 3.4.101 to kernel 3.14.25, a few of them started
to show serious trouble when there was a lot of network traffic.

---------------------------------------------------------------
Jan 14 11:14:57 intrartc kernel: e1000e 0000:00:19.0 eth0: Detected Hardware Unit Hang:
Jan 14 11:14:57 intrartc kernel:  TDH                  <3b>
Jan 14 11:14:57 intrartc kernel:  TDT                  <76>
Jan 14 11:14:57 intrartc kernel:  next_to_use          <76>
Jan 14 11:14:57 intrartc kernel:  next_to_clean        <31>
Jan 14 11:14:57 intrartc kernel: buffer_info[next_to_clean]:
Jan 14 11:14:57 intrartc kernel:  time_stamp           <ffff328c>
Jan 14 11:14:57 intrartc kernel:  next_to_watch        <3b>
Jan 14 11:14:57 intrartc kernel:  jiffies              <ffff33b9>
Jan 14 11:14:57 intrartc kernel:  next_to_watch.status <0>
Jan 14 11:14:57 intrartc kernel: MAC Status             <40080083>
Jan 14 11:14:57 intrartc kernel: PHY Status             <796d>
Jan 14 11:14:57 intrartc kernel: PHY 1000BASE-T Status  <3800>
Jan 14 11:14:57 intrartc kernel: PHY Extended Status    <3000>
Jan 14 11:14:57 intrartc kernel: PCI Status             <10>
Jan 14 11:14:59 intrartc kernel: e1000e 0000:00:19.0 eth0: Detected Hardware Unit Hang:
..
---------------------------------------------------------------

All of those troubled machines use an Intel DH61CR board and
are driven by the e1000e driver. Kernels 3.7.0 to 3.19-rc4 are affected.

The problem vanishes when you disable TSO. This is the
recommended "solution" on serverfault and others.
http://ehc.ac/p/e1000/bugs/378/
http://serverfault.com/questions/616485/e1000e-reset-adapter-unexpectedly-detected-hardware-unit-hang

I have a test setup that can trigger the problem within seconds
and bisected it down to this commit (hi Eric!):
---------------------------------------------------------------
commit 69b08f62e17439ee3d436faf0b9a7ca6fffb78db
Author: Eric Dumazet <edumazet@google.com>
Date:   Wed Sep 26 06:46:57 2012 +0000

    net: use bigger pages in __netdev_alloc_frag

    We currently use percpu order-0 pages in __netdev_alloc_frag
    to deliver fragments used by __netdev_alloc_skb()

    Depending on NIC driver and arch being 32 or 64 bit, it allows a page to
    be split in several fragments (between 1 and 8), assuming PAGE_SIZE=4096

    Switching to bigger pages (32768 bytes for PAGE_SIZE=4096 case) allows :

    - Better filling of space (the ending hole overhead is less an issue)

    - Less calls to page allocator or accesses to page->_count

    - Could allow struct skb_shared_info futures changes without major
    performance impact.

    This patch implements a transparent fallback to smaller
    pages in case of memory pressure.

    It also uses a standard "struct page_frag" instead of a custom one.

    Signed-off-by: Eric Dumazet <edumazet@google.com>
    Cc: Alexander Duyck <alexander.h.duyck@intel.com>
    Cc: Benjamin LaHaise <bcrl@kvack.org>
    Signed-off-by: David S. Miller <davem@davemloft.net>
---------------------------------------------------------------

Reverting the commit f.e. in kernel 3.7.0  solves the issue.
I've done some more tests:

    3.18.0 32bit + PAE: broken
    3.6.0 32bit + PAE: works
    3.7.0 32bit + PAE: broken
    3.7.0 32bit + PAE + revert 69b08f62e17439ee3d436faf0b9a7ca6fffb78db -> works

    3.7.0 32bit (without PAE) -> broken
    3.7.0 32bit + "GFP_COMP" flag removed in __netdev_alloc_frag(): broken
    3.7.0 32bit + "GFP_COMP" flag replaced with
                              "GFP_DMA" in __netdev_alloc_frag(): works!
    3.7.0 32bit + "GFP_COMP" flag + "GFP_DMA" flag: broken
    3.19-rc4 32bit: broken


The problem is triggered only when the traffic is forwarded to another client.
(this client is behind NAT). Generating traffic directly
on the system did not trigger the issue.

To me it looks like Eric's change uncovered a memory allocation
issue in the e1000e driver: It probably uses a memory address
unsuitable for DMA or so. This is just a guess though.

Funny fact: I have another Intel DH61CR board that does not show the problem.
I've borrowed (...) the mainboard from one affected box for my bisect test setup.

Please CC: comments. Thanks.

Best regards,
Thomas

Re: [bisected regression] e1000e: "Detected Hardware Unit Hang"

[Eric Dumazet]

On Wed, 2015-01-14 at 16:32 +0100, Thomas Jarosch wrote:
> Hello,
> 
> after updating a good bunch of production level machines
> from kernel 3.4.101 to kernel 3.14.25, a few of them started
> to show serious trouble when there was a lot of network traffic.
> 
> ---------------------------------------------------------------
> Jan 14 11:14:57 intrartc kernel: e1000e 0000:00:19.0 eth0: Detected Hardware Unit Hang:
> Jan 14 11:14:57 intrartc kernel:  TDH                  <3b>
> Jan 14 11:14:57 intrartc kernel:  TDT                  <76>
> Jan 14 11:14:57 intrartc kernel:  next_to_use          <76>
> Jan 14 11:14:57 intrartc kernel:  next_to_clean        <31>
> Jan 14 11:14:57 intrartc kernel: buffer_info[next_to_clean]:
> Jan 14 11:14:57 intrartc kernel:  time_stamp           <ffff328c>
> Jan 14 11:14:57 intrartc kernel:  next_to_watch        <3b>
> Jan 14 11:14:57 intrartc kernel:  jiffies              <ffff33b9>
> Jan 14 11:14:57 intrartc kernel:  next_to_watch.status <0>
> Jan 14 11:14:57 intrartc kernel: MAC Status             <40080083>
> Jan 14 11:14:57 intrartc kernel: PHY Status             <796d>
> Jan 14 11:14:57 intrartc kernel: PHY 1000BASE-T Status  <3800>
> Jan 14 11:14:57 intrartc kernel: PHY Extended Status    <3000>
> Jan 14 11:14:57 intrartc kernel: PCI Status             <10>
> Jan 14 11:14:59 intrartc kernel: e1000e 0000:00:19.0 eth0: Detected Hardware Unit Hang:
> ..
> ---------------------------------------------------------------
> 
> All of those troubled machines use an Intel DH61CR board and
> are driven by the e1000e driver. Kernels 3.7.0 to 3.19-rc4 are affected.
> 
> The problem vanishes when you disable TSO. This is the
> recommended "solution" on serverfault and others.
> http://ehc.ac/p/e1000/bugs/378/
> http://serverfault.com/questions/616485/e1000e-reset-adapter-unexpectedly-detected-hardware-unit-hang
> 
> I have a test setup that can trigger the problem within seconds
> and bisected it down to this commit (hi Eric!):
> ---------------------------------------------------------------
> commit 69b08f62e17439ee3d436faf0b9a7ca6fffb78db
> Author: Eric Dumazet <edumazet@google.com>
> Date:   Wed Sep 26 06:46:57 2012 +0000
> 
>     net: use bigger pages in __netdev_alloc_frag
> 
>     We currently use percpu order-0 pages in __netdev_alloc_frag
>     to deliver fragments used by __netdev_alloc_skb()
> 
>     Depending on NIC driver and arch being 32 or 64 bit, it allows a page to
>     be split in several fragments (between 1 and 8), assuming PAGE_SIZE=4096
> 
>     Switching to bigger pages (32768 bytes for PAGE_SIZE=4096 case) allows :
> 
>     - Better filling of space (the ending hole overhead is less an issue)
> 
>     - Less calls to page allocator or accesses to page->_count
> 
>     - Could allow struct skb_shared_info futures changes without major
>     performance impact.
> 
>     This patch implements a transparent fallback to smaller
>     pages in case of memory pressure.
> 
>     It also uses a standard "struct page_frag" instead of a custom one.
> 
>     Signed-off-by: Eric Dumazet <edumazet@google.com>
>     Cc: Alexander Duyck <alexander.h.duyck@intel.com>
>     Cc: Benjamin LaHaise <bcrl@kvack.org>
>     Signed-off-by: David S. Miller <davem@davemloft.net>
> ---------------------------------------------------------------
> 
> Reverting the commit f.e. in kernel 3.7.0  solves the issue.
> I've done some more tests:
> 
>     3.18.0 32bit + PAE: broken
>     3.6.0 32bit + PAE: works
>     3.7.0 32bit + PAE: broken
>     3.7.0 32bit + PAE + revert 69b08f62e17439ee3d436faf0b9a7ca6fffb78db -> works
> 
>     3.7.0 32bit (without PAE) -> broken
>     3.7.0 32bit + "GFP_COMP" flag removed in __netdev_alloc_frag(): broken
>     3.7.0 32bit + "GFP_COMP" flag replaced with
>                               "GFP_DMA" in __netdev_alloc_frag(): works!
>     3.7.0 32bit + "GFP_COMP" flag + "GFP_DMA" flag: broken
>     3.19-rc4 32bit: broken
> 
> 
> The problem is triggered only when the traffic is forwarded to another client.
> (this client is behind NAT). Generating traffic directly
> on the system did not trigger the issue.
> 
> To me it looks like Eric's change uncovered a memory allocation
> issue in the e1000e driver: It probably uses a memory address
> unsuitable for DMA or so. This is just a guess though.
> 
> Funny fact: I have another Intel DH61CR board that does not show the problem.
> I've borrowed (...) the mainboard from one affected box for my bisect test setup.
> 
> Please CC: comments. Thanks.

I would try to use lower data per txd. I am not sure 24KB is really
supported.

( check commit d821a4c4d11ad160925dab2bb009b8444beff484 for details)

diff --git a/drivers/net/ethernet/intel/e1000e/netdev.c b/drivers/net/ethernet/intel/e1000e/netdev.c
index e14fd85f64eb..8d973f7edfbd 100644
--- a/drivers/net/ethernet/intel/e1000e/netdev.c
+++ b/drivers/net/ethernet/intel/e1000e/netdev.c
@@ -3897,7 +3897,7 @@ void e1000e_reset(struct e1000_adapter *adapter)
 	 * limit of 24KB due to receive synchronization limitations.
 	 */
 	adapter->tx_fifo_limit = min_t(u32, ((er32(PBA) >> 16) << 10) - 96,
-				       24 << 10);
+				       8 << 10);
 
 	/* Disable Adaptive Interrupt Moderation if 2 full packets cannot
 	 * fit in receive buffer.

Re: [bisected regression] e1000e: "Detected Hardware Unit Hang"

[Thomas Jarosch]

On Wednesday, 14. January 2015 09:20:52 Eric Dumazet wrote:
> I would try to use lower data per txd. I am not sure 24KB is really
> supported.
> 
> ( check commit d821a4c4d11ad160925dab2bb009b8444beff484 for details)
> 
> diff --git a/drivers/net/ethernet/intel/e1000e/netdev.c
> b/drivers/net/ethernet/intel/e1000e/netdev.c index
> e14fd85f64eb..8d973f7edfbd 100644
> --- a/drivers/net/ethernet/intel/e1000e/netdev.c
> +++ b/drivers/net/ethernet/intel/e1000e/netdev.c
> @@ -3897,7 +3897,7 @@ void e1000e_reset(struct e1000_adapter *adapter)
>  	 * limit of 24KB due to receive synchronization limitations.
>  	 */
>  	adapter->tx_fifo_limit = min_t(u32, ((er32(PBA) >> 16) << 10) - 96,
> -				       24 << 10);
> +				       8 << 10);
> 
>  	/* Disable Adaptive Interrupt Moderation if 2 full packets cannot
>  	 * fit in receive buffer.

Thanks for checking!

I just tried that change on top of git f800c25 (git HEAD), same problem. 
Let's see what the Intel wizards come up with.

What "works" is to decrease the page size in git HEAD, too:

diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index 85ab7d7..9f0ef97 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -2108,7 +2108,7 @@ static inline void __skb_queue_purge(struct 
sk_buff_head *list)
                kfree_skb(skb);
 }
 
-#define NETDEV_FRAG_PAGE_MAX_ORDER get_order(32768)
+#define NETDEV_FRAG_PAGE_MAX_ORDER get_order(4096)
 #define NETDEV_FRAG_PAGE_MAX_SIZE  (PAGE_SIZE << NETDEV_FRAG_PAGE_MAX_ORDER)
 #define NETDEV_PAGECNT_MAX_BIAS           NETDEV_FRAG_PAGE_MAX_SIZE



When I try a page size of 8192, it starts failing again. I'll now run
a stress test with 4096 to see if the problem is really gone
or just happens more rarely.

Cheers,
Thomas

Re: [bisected regression] e1000e: "Detected Hardware Unit Hang"

[Eric Dumazet]

On Thu, 2015-01-15 at 11:11 +0100, Thomas Jarosch wrote:
> On Wednesday, 14. January 2015 09:20:52 Eric Dumazet wrote:
> > I would try to use lower data per txd. I am not sure 24KB is really
> > supported.
> > 
> > ( check commit d821a4c4d11ad160925dab2bb009b8444beff484 for details)
> > 
> > diff --git a/drivers/net/ethernet/intel/e1000e/netdev.c
> > b/drivers/net/ethernet/intel/e1000e/netdev.c index
> > e14fd85f64eb..8d973f7edfbd 100644
> > --- a/drivers/net/ethernet/intel/e1000e/netdev.c
> > +++ b/drivers/net/ethernet/intel/e1000e/netdev.c
> > @@ -3897,7 +3897,7 @@ void e1000e_reset(struct e1000_adapter *adapter)
> >  	 * limit of 24KB due to receive synchronization limitations.
> >  	 */
> >  	adapter->tx_fifo_limit = min_t(u32, ((er32(PBA) >> 16) << 10) - 96,
> > -				       24 << 10);
> > +				       8 << 10);
> > 
> >  	/* Disable Adaptive Interrupt Moderation if 2 full packets cannot
> >  	 * fit in receive buffer.
> 
> Thanks for checking!
> 
> I just tried that change on top of git f800c25 (git HEAD), same problem. 
> Let's see what the Intel wizards come up with.
> 
> What "works" is to decrease the page size in git HEAD, too:
> 
> diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
> index 85ab7d7..9f0ef97 100644
> --- a/include/linux/skbuff.h
> +++ b/include/linux/skbuff.h
> @@ -2108,7 +2108,7 @@ static inline void __skb_queue_purge(struct 
> sk_buff_head *list)
>                 kfree_skb(skb);
>  }
>  
> -#define NETDEV_FRAG_PAGE_MAX_ORDER get_order(32768)
> +#define NETDEV_FRAG_PAGE_MAX_ORDER get_order(4096)
>  #define NETDEV_FRAG_PAGE_MAX_SIZE  (PAGE_SIZE << NETDEV_FRAG_PAGE_MAX_ORDER)
>  #define NETDEV_PAGECNT_MAX_BIAS           NETDEV_FRAG_PAGE_MAX_SIZE
> 
> 
> 
> When I try a page size of 8192, it starts failing again. I'll now run
> a stress test with 4096 to see if the problem is really gone
> or just happens more rarely.

Sure, you basically reverted my patch.

You are not the first to report a problem caused by this patch.

This patch is known to have uncovered some driver bugs.

We are not going to revert it. We are going to fix the real bugs.

Thanks

Re: Re: [bisected regression] e1000e: "Detected Hardware Unit Hang"

[Thomas Jarosch]

On Thursday, 15. January 2015 06:43:29 Eric Dumazet wrote:
> > -#define NETDEV_FRAG_PAGE_MAX_ORDER get_order(32768)
> > +#define NETDEV_FRAG_PAGE_MAX_ORDER get_order(4096)
> > 
> >  #define NETDEV_FRAG_PAGE_MAX_SIZE  (PAGE_SIZE <<
> >  NETDEV_FRAG_PAGE_MAX_ORDER) #define NETDEV_PAGECNT_MAX_BIAS          
> >  NETDEV_FRAG_PAGE_MAX_SIZE> 
> > When I try a page size of 8192, it starts failing again. I'll now run
> > a stress test with 4096 to see if the problem is really gone
> > or just happens more rarely.
> 
> Sure, you basically reverted my patch.
> 
> You are not the first to report a problem caused by this patch.
> 
> This patch is known to have uncovered some driver bugs.
> 
> We are not going to revert it. We are going to fix the real bugs.
> 
> Thanks

A colleague mentioned to me he saw the "Hardware Unit Hang" message every 
few days even running on kernel 3.4 (without your patch). Basically I'm 
testing now if that's still the case with 3.19-rc4+ or not.

I'm all for fixing the root cause. I'm just interested if the e1000e
hang can even be triggered when using a max frag page size of 4096.
So far it transferred 751.6 GiB without a hiccup.

Cheers,
Thomas

Re: [bisected regression] e1000e: "Detected Hardware Unit Hang"

[Jeff Kirsher]

[-- Attachment #1: Type: text/plain, Size: 2404 bytes --]

On Thu, 2015-01-15 at 06:43 -0800, Eric Dumazet wrote:
> On Thu, 2015-01-15 at 11:11 +0100, Thomas Jarosch wrote:
> > On Wednesday, 14. January 2015 09:20:52 Eric Dumazet wrote:
> > > I would try to use lower data per txd. I am not sure 24KB is really
> > > supported.
> > > 
> > > ( check commit d821a4c4d11ad160925dab2bb009b8444beff484 for details)
> > > 
> > > diff --git a/drivers/net/ethernet/intel/e1000e/netdev.c
> > > b/drivers/net/ethernet/intel/e1000e/netdev.c index
> > > e14fd85f64eb..8d973f7edfbd 100644
> > > --- a/drivers/net/ethernet/intel/e1000e/netdev.c
> > > +++ b/drivers/net/ethernet/intel/e1000e/netdev.c
> > > @@ -3897,7 +3897,7 @@ void e1000e_reset(struct e1000_adapter *adapter)
> > >  	 * limit of 24KB due to receive synchronization limitations.
> > >  	 */
> > >  	adapter->tx_fifo_limit = min_t(u32, ((er32(PBA) >> 16) << 10) - 96,
> > > -				       24 << 10);
> > > +				       8 << 10);
> > > 
> > >  	/* Disable Adaptive Interrupt Moderation if 2 full packets cannot
> > >  	 * fit in receive buffer.
> > 
> > Thanks for checking!
> > 
> > I just tried that change on top of git f800c25 (git HEAD), same problem. 
> > Let's see what the Intel wizards come up with.
> > 
> > What "works" is to decrease the page size in git HEAD, too:
> > 
> > diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
> > index 85ab7d7..9f0ef97 100644
> > --- a/include/linux/skbuff.h
> > +++ b/include/linux/skbuff.h
> > @@ -2108,7 +2108,7 @@ static inline void __skb_queue_purge(struct 
> > sk_buff_head *list)
> >                 kfree_skb(skb);
> >  }
> >  
> > -#define NETDEV_FRAG_PAGE_MAX_ORDER get_order(32768)
> > +#define NETDEV_FRAG_PAGE_MAX_ORDER get_order(4096)
> >  #define NETDEV_FRAG_PAGE_MAX_SIZE  (PAGE_SIZE << NETDEV_FRAG_PAGE_MAX_ORDER)
> >  #define NETDEV_PAGECNT_MAX_BIAS           NETDEV_FRAG_PAGE_MAX_SIZE
> > 
> > 
> > 
> > When I try a page size of 8192, it starts failing again. I'll now run
> > a stress test with 4096 to see if the problem is really gone
> > or just happens more rarely.
> 
> Sure, you basically reverted my patch.
> 
> You are not the first to report a problem caused by this patch.
> 
> This patch is known to have uncovered some driver bugs.
> 
> We are not going to revert it. We are going to fix the real bugs.
> 
> Thanks
> 
> 

Agreed, we are looking into issue Thomas.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Re: [bisected regression] e1000e: "Detected Hardware Unit Hang"

[Eric Dumazet]

On Thu, 2015-01-15 at 15:58 +0100, Thomas Jarosch wrote:

> A colleague mentioned to me he saw the "Hardware Unit Hang" message every 
> few days even running on kernel 3.4 (without your patch). Basically I'm 
> testing now if that's still the case with 3.19-rc4+ or not.
> 
> I'm all for fixing the root cause. I'm just interested if the e1000e
> hang can even be triggered when using a max frag page size of 4096.
> So far it transferred 751.6 GiB without a hiccup.

You told it was forwarding setup.

1) What is the NIC receiving traffic.
2) What happens if you disable GRO on it ?

Re: Re: Re: [bisected regression] e1000e: "Detected Hardware Unit Hang"

[Thomas Jarosch]

On Thursday, 15. January 2015 07:25:32 Eric Dumazet wrote:
> On Thu, 2015-01-15 at 15:58 +0100, Thomas Jarosch wrote:
> > A colleague mentioned to me he saw the "Hardware Unit Hang" message
> > every
> > few days even running on kernel 3.4 (without your patch). Basically I'm
> > testing now if that's still the case with 3.19-rc4+ or not.
> > 
> > I'm all for fixing the root cause. I'm just interested if the e1000e
> > hang can even be triggered when using a max frag page size of 4096.
> > So far it transferred 751.6 GiB without a hiccup.
> 
> You told it was forwarding setup.
> 
> 1) What is the NIC receiving traffic.
> 2) What happens if you disable GRO on it ?

The setup is like this:

Win7 notebook (client)
    -> "private LAN" eth0 (e1000e)
        -> "external traffic" eth1 (r8169)

            -> local HTTP server in the intranet
               (2x e1000e using bonding)


Disabling gro on eth1 (r8169) seems to make eth0 (e1000e) stable.
As it usually hangs within seconds, it already transferred 28 GiB right now.

When I switch gro back on, it takes around three seconds until the hang.

Does that point into the right / any direction?

Thomas

Re: Re: Re: [bisected regression] e1000e: "Detected Hardware Unit Hang"

[Eric Dumazet]

On Thu, 2015-01-15 at 16:48 +0100, Thomas Jarosch wrote:
> On Thursday, 15. January 2015 07:25:32 Eric Dumazet wrote:
> > On Thu, 2015-01-15 at 15:58 +0100, Thomas Jarosch wrote:
> > > A colleague mentioned to me he saw the "Hardware Unit Hang" message
> > > every
> > > few days even running on kernel 3.4 (without your patch). Basically I'm
> > > testing now if that's still the case with 3.19-rc4+ or not.
> > > 
> > > I'm all for fixing the root cause. I'm just interested if the e1000e
> > > hang can even be triggered when using a max frag page size of 4096.
> > > So far it transferred 751.6 GiB without a hiccup.
> > 
> > You told it was forwarding setup.
> > 
> > 1) What is the NIC receiving traffic.
> > 2) What happens if you disable GRO on it ?
> 
> The setup is like this:
> 
> Win7 notebook (client)
>     -> "private LAN" eth0 (e1000e)
>         -> "external traffic" eth1 (r8169)
> 
>             -> local HTTP server in the intranet
>                (2x e1000e using bonding)
> 
> 
> Disabling gro on eth1 (r8169) seems to make eth0 (e1000e) stable.
> As it usually hangs within seconds, it already transferred 28 GiB right now.
> 
> When I switch gro back on, it takes around three seconds until the hang.
> 
> Does that point into the right / any direction?

Sure. 

Please apply this patch, and try to lower
/proc/sys/net/core/gro_max_frags and see if this makes a difference
(leaving GRO enabled)

(start with 7 and increase it, limit being 17)

diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 642d426a668f8ac94daf334c00117f96789f3990..817aee05a1b0623e5752beb0952a6fe6d66e583f 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -3400,6 +3400,7 @@ extern int		netdev_max_backlog;
 extern int		netdev_tstamp_prequeue;
 extern int		weight_p;
 extern int		bpf_jit_enable;
+extern int		sysctl_gro_max_frags;
 
 bool netdev_has_upper_dev(struct net_device *dev, struct net_device *upper_dev);
 struct net_device *netdev_upper_get_next_dev_rcu(struct net_device *dev,
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 56db472e9b864e805e0ab36dd73a0404d2fc66d5..c2c2e7e53014617c5da574f2eb8a2889ed743719 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -3197,6 +3197,8 @@ err:
 }
 EXPORT_SYMBOL_GPL(skb_segment);
 
+int sysctl_gro_max_frags = MAX_SKB_FRAGS;
+
 int skb_gro_receive(struct sk_buff **head, struct sk_buff *skb)
 {
 	struct skb_shared_info *pinfo, *skbinfo = skb_shinfo(skb);
@@ -3219,8 +3221,8 @@ int skb_gro_receive(struct sk_buff **head, struct sk_buff *skb)
 		int i = skbinfo->nr_frags;
 		int nr_frags = pinfo->nr_frags + i;
 
-		if (nr_frags > MAX_SKB_FRAGS)
-			goto merge;
+		if (nr_frags > sysctl_gro_max_frags)
+			return -E2BIG;
 
 		offset -= headlen;
 		pinfo->nr_frags = nr_frags;
@@ -3252,8 +3254,8 @@ int skb_gro_receive(struct sk_buff **head, struct sk_buff *skb)
 		unsigned int first_size = headlen - offset;
 		unsigned int first_offset;
 
-		if (nr_frags + 1 + skbinfo->nr_frags > MAX_SKB_FRAGS)
-			goto merge;
+		if (nr_frags + 1 + skbinfo->nr_frags > sysctl_gro_max_frags)
+			return -E2BIG;
 
 		first_offset = skb->data -
 			       (unsigned char *)page_address(page) +
diff --git a/net/core/sysctl_net_core.c b/net/core/sysctl_net_core.c
index 31baba2a71ce15e49450f69dae81e7d3be1ff3f2..de73d51381bf8acd0aedeb859ed961468441014a 100644
--- a/net/core/sysctl_net_core.c
+++ b/net/core/sysctl_net_core.c
@@ -278,6 +278,13 @@ static struct ctl_table net_core_table[] = {
 		.proc_handler	= proc_dointvec
 	},
 	{
+		.procname	= "gro_max_frags",
+		.data		= &sysctl_gro_max_frags,
+		.maxlen		= sizeof(int),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec
+	},
+	{
 		.procname	= "netdev_rss_key",
 		.data		= &netdev_rss_key,
 		.maxlen		= sizeof(int),

Re: Re: Re: Re: [bisected regression] e1000e: "Detected Hardware Unit Hang"

[Thomas Jarosch]

On Thursday, 15. January 2015 08:00:58 Eric Dumazet wrote:
> Please apply this patch, and try to lower
> /proc/sys/net/core/gro_max_frags and see if this makes a difference
> (leaving GRO enabled)
> 
> (start with 7 and increase it, limit being 17)

Patch applied to 3.19-rc4+.

Results:
 7: hang
 8: hang
 9: hang
10: hang
11: hang
12: hang
13: hang
14: hang
15: hang
16: hang
17: hang

for the sake of completeness:
1: hang
2: hang
3: hang
4: hang
5: hang
6: hang

Regarding the test procedure: I stopped the download script on the client,
changed gro_max_frags and started the download again. No cable unplugging / 
reboot of the box in between. Just mentioning it to make sure it somehow 
does not affect what we actually wanted to test.

Additional tests have been done with gro_max_frags 1, 7 and 17:
- stop networking + unload e1000e -> restart -> download: hang

One thing I can say from the testing: The more I increase gro_max_frags,
the longer it takes to trigger it. I tried each setting below three times.
A value of 17 is really noticeable.

1: 3-8 seconds till hang
7: 7-10 seconds till hang
17: 23-26 seconds till hang

Thomas

Re: Re: Re: Re: [bisected regression] e1000e: "Detected Hardware Unit Hang"

[Eric Dumazet]

On Thu, 2015-01-15 at 18:04 +0100, Thomas Jarosch wrote:
> On Thursday, 15. January 2015 08:00:58 Eric Dumazet wrote:
> > Please apply this patch, and try to lower
> > /proc/sys/net/core/gro_max_frags and see if this makes a difference
> > (leaving GRO enabled)
> > 
> > (start with 7 and increase it, limit being 17)
> 
> Patch applied to 3.19-rc4+.
> 
> Results:
>  7: hang
>  8: hang
>  9: hang
> 10: hang
> 11: hang
> 12: hang
> 13: hang
> 14: hang
> 15: hang
> 16: hang
> 17: hang
> 
> for the sake of completeness:
> 1: hang

This is weird : This should have same effect then GRO off (at most one
segment per packet)

> 2: hang
> 3: hang
> 4: hang
> 5: hang
> 6: hang
> 
> Regarding the test procedure: I stopped the download script on the client,
> changed gro_max_frags and started the download again. No cable unplugging / 
> reboot of the box in between. Just mentioning it to make sure it somehow 
> does not affect what we actually wanted to test.
> 
> Additional tests have been done with gro_max_frags 1, 7 and 17:
> - stop networking + unload e1000e -> restart -> download: hang
> 
> One thing I can say from the testing: The more I increase gro_max_frags,
> the longer it takes to trigger it. I tried each setting below three times.
> A value of 17 is really noticeable.
> 
> 1: 3-8 seconds till hang
> 7: 7-10 seconds till hang
> 17: 23-26 seconds till hang

Could you try the following ?

diff --git a/drivers/net/ethernet/intel/e1000e/netdev.c b/drivers/net/ethernet/intel/e1000e/netdev.c
index 38cb586b1bf42fa7a50e19f3e650e8c139788820..6d93facddab78f8db7000fddaa24322651a0eae9 100644
--- a/drivers/net/ethernet/intel/e1000e/netdev.c
+++ b/drivers/net/ethernet/intel/e1000e/netdev.c
@@ -1264,7 +1264,7 @@ static bool e1000_clean_tx_irq(struct e1000_ring *tx_ring)
 
 	netdev_completed_queue(netdev, pkts_compl, bytes_compl);
 
-#define TX_WAKE_THRESHOLD 32
+#define TX_WAKE_THRESHOLD (MAX_SKB_FRAGS * 3 + 2)
 	if (count && netif_carrier_ok(netdev) &&
 	    e1000_desc_unused(tx_ring) >= TX_WAKE_THRESHOLD) {
 		/* Make sure that anybody stopping the queue after this
@@ -5650,10 +5650,7 @@ static netdev_tx_t e1000_xmit_frame(struct sk_buff *skb,
 		netdev_sent_queue(netdev, skb->len);
 		e1000_tx_queue(tx_ring, tx_flags, count);
 		/* Make sure there is space in the ring for the next send. */
-		e1000_maybe_stop_tx(tx_ring,
-				    (MAX_SKB_FRAGS *
-				     DIV_ROUND_UP(PAGE_SIZE,
-						  adapter->tx_fifo_limit) + 2));
+		e1000_maybe_stop_tx(tx_ring, 3 * MAX_SKB_FRAGS + 2);
 	} else {
 		dev_kfree_skb_any(skb);
 		tx_ring->buffer_info[first].time_stamp = 0;

Re: [bisected regression] e1000e: "Detected Hardware Unit Hang"

[Thomas Jarosch]

On Thursday, 15. January 2015 09:20:37 Eric Dumazet wrote:
> > for the sake of completeness:
> > 1: hang
> 
> This is weird : This should have same effect then GRO off (at most one
> segment per packet)

I thought so, too. OTOH the code path was changed from
"goto merge" to "return -E2BIG". I didn't look at the code
what happens at the "merge" label.

> Could you try the following ?
> 
> diff --git a/drivers/net/ethernet/intel/e1000e/netdev.c
> b/drivers/net/ethernet/intel/e1000e/netdev.c index
> 38cb586b1bf42fa7a50e19f3e650e8c139788820..6d93facddab78f8db7000fddaa24322
> 651a0eae9 100644 --- a/drivers/net/ethernet/intel/e1000e/netdev.c
> +++ b/drivers/net/ethernet/intel/e1000e/netdev.c
> @@ -1264,7 +1264,7 @@ static bool e1000_clean_tx_irq(struct e1000_ring
> *tx_ring)
> 
>  	netdev_completed_queue(netdev, pkts_compl, bytes_compl);
> 
> -#define TX_WAKE_THRESHOLD 32
> +#define TX_WAKE_THRESHOLD (MAX_SKB_FRAGS * 3 + 2)
>  	if (count && netif_carrier_ok(netdev) &&
>  	    e1000_desc_unused(tx_ring) >= TX_WAKE_THRESHOLD) {
>  		/* Make sure that anybody stopping the queue after this
> @@ -5650,10 +5650,7 @@ static netdev_tx_t e1000_xmit_frame(struct sk_buff
> *skb, netdev_sent_queue(netdev, skb->len);
>  		e1000_tx_queue(tx_ring, tx_flags, count);
>  		/* Make sure there is space in the ring for the next send. */
> -		e1000_maybe_stop_tx(tx_ring,
> -				    (MAX_SKB_FRAGS *
> -				     DIV_ROUND_UP(PAGE_SIZE,
> -						  adapter->tx_fifo_limit) + 2));
> +		e1000_maybe_stop_tx(tx_ring, 3 * MAX_SKB_FRAGS + 2);
>  	} else {
>  		dev_kfree_skb_any(skb);
>  		tx_ring->buffer_info[first].time_stamp = 0;

I was just about to leave the office... and switched the test box back on.

Also reverted to vanilla 3.19-rc4+ and applied the above patch instead.
Still hangs. Grrr. More testing tomorrow.


------------------------------------------------------------------------------
New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
GigeNET is offering a free month of service with a new server in Ashburn.
Choose from 2 high performing configs, both with 100TB of bandwidth.
Higher redundancy.Lower latency.Increased capacity.Completely compliant.
http://p.sf.net/sfu/gigenet
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired

Re: Re: Re: Re: Re: [bisected regression] e1000e: "Detected Hardware Unit Hang"

[Eric Dumazet]

On Thu, 2015-01-15 at 18:37 +0100, Thomas Jarosch wrote:
> On Thursday, 15. January 2015 09:20:37 Eric Dumazet wrote:
> > > for the sake of completeness:
> > > 1: hang
> > 
> > This is weird : This should have same effect then GRO off (at most one
> > segment per packet)
> 
> I thought so, too. OTOH the code path was changed from
> "goto merge" to "return -E2BIG". I didn't look at the code
> what happens at the "merge" label.

If you leave the "goto merge", then GRO still builds big GRO packets,
using a linked list of packets.

Then because drivers do not generally support NETIF_F_FRAGLIST,
core networking code linearizes such GRO packets before reaching
ndo_start_xmit()

Check 8a29111c7ca68d928dfab58636f3f6acf0ac04f7 "net: gro: allow to build
full sized skb" for details about this.

Re: Re: Re: [bisected regression] e1000e: "Detected Hardware Unit Hang"

[Thomas Jarosch]

On Thursday, 15. January 2015 07:25:32 Eric Dumazet wrote:
> On Thu, 2015-01-15 at 15:58 +0100, Thomas Jarosch wrote:
> > A colleague mentioned to me he saw the "Hardware Unit Hang" message
> > every
> > few days even running on kernel 3.4 (without your patch). Basically I'm
> > testing now if that's still the case with 3.19-rc4+ or not.
> > 
> > I'm all for fixing the root cause. I'm just interested if the e1000e
> > hang can even be triggered when using a max frag page size of 4096.
> > So far it transferred 751.6 GiB without a hiccup.
> 
> You told it was forwarding setup.
> 
> 1) What is the NIC receiving traffic.
> 2) What happens if you disable GRO on it ?

one more interesting thing happened: On one production machine,
again an Intel DH61CR board, the issue was triggered even with TSO disabled.
My colleague tried to disable GRO + GSO on the e1000e adapter, too,
though not on the other interfaces.

It's strange the issue appears with TSO disabled,
that worked for three other production level machines.

We've emergency-installed the "4096" max frag page size workaround
for now as fifty people were a bit unhappy without network access... :D

Cheers,
Thomas

Re: [bisected regression] e1000e: "Detected Hardware Unit Hang"

[Thomas Jarosch]

Hi Jeff,

On Thursday, 15. January 2015 06:59:13 Jeff Kirsher wrote:
> > Sure, you basically reverted my patch.
> > 
> > You are not the first to report a problem caused by this patch.
> > 
> > This patch is known to have uncovered some driver bugs.
> > 
> > We are not going to revert it. We are going to fix the real bugs.
> > 
> > Thanks
> 
> Agreed, we are looking into issue Thomas.

any news from the Intel labs what might be going on here?

We started seeing those hangs on "MSI B85M ECO" boards, too,
though it's way more sporadic there.

Thanks,
Thomas

Re: [bisected regression] e1000e: "Detected Hardware Unit Hang"

[Jeff Kirsher]

[-- Attachment #1.1: Type: text/plain, Size: 765 bytes --]

On Wed, 2015-02-11 at 12:23 +0100, Thomas Jarosch wrote:
> Hi Jeff,
> 
> On Thursday, 15. January 2015 06:59:13 Jeff Kirsher wrote:
> > > Sure, you basically reverted my patch.
> > > 
> > > You are not the first to report a problem caused by this patch.
> > > 
> > > This patch is known to have uncovered some driver bugs.
> > > 
> > > We are not going to revert it. We are going to fix the real bugs.
> > > 
> > > Thanks
> > 
> > Agreed, we are looking into issue Thomas.
> 
> any news from the Intel labs what might be going on here?
> 
> We started seeing those hangs on "MSI B85M ECO" boards, too,
> though it's way more sporadic there.

I have not heard anything, so I have added Aaron Brown to see if he has
any additional information.

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 441 bytes --]

------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/

[-- Attachment #3: Type: text/plain, Size: 257 bytes --]

_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired

Re: [bisected regression] e1000e: "Detected Hardware Unit Hang"

[nick]

On 2015-02-11 06:23 AM, Thomas Jarosch wrote:
> Hi Jeff,
> 
> On Thursday, 15. January 2015 06:59:13 Jeff Kirsher wrote:
>>> Sure, you basically reverted my patch.
>>>
>>> You are not the first to report a problem caused by this patch.
>>>
>>> This patch is known to have uncovered some driver bugs.
>>>
>>> We are not going to revert it. We are going to fix the real bugs.
>>>
>>> Thanks
>>
>> Agreed, we are looking into issue Thomas.
> 
> any news from the Intel labs what might be going on here?
> 
> We started seeing those hangs on "MSI B85M ECO" boards, too,
> though it's way more sporadic there.
> 
> Thanks,
> Thomas
> 
> 
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming. The Go Parallel Website,
> sponsored by Intel and developed in partnership with Slashdot Media, is your
> hub for all things parallel software development, from weekly thought
> leadership blogs to news, videos, case studies, tutorials and more. Take a
> look and join the conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> E1000-devel mailing list
> E1000-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/e1000-devel
> To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
> 
Jeff and Thomas,
I am unable to find the commit id in my email box for the git bisected commit. Would someone send me the commit id so I can look into starting to solve the bugs 
the commit discussed here as uncovered in the driver supported by Intel.
Thanks A lot,
Nick 

------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired

RE: [bisected regression] e1000e: "Detected Hardware Unit Hang"

[Brown, Aaron F]

> From: Kirsher, Jeffrey T
> Sent: Wednesday, February 11, 2015 3:34 AM
> To: Thomas Jarosch; Brown, Aaron F
> Cc: 'Linux Netdev List'; Eric Dumazet; e1000-devel
> Subject: Re: [bisected regression] e1000e: "Detected Hardware Unit Hang"
> 
> On Wed, 2015-02-11 at 12:23 +0100, Thomas Jarosch wrote:
> > Hi Jeff,
> >
> > On Thursday, 15. January 2015 06:59:13 Jeff Kirsher wrote:
> > > > Sure, you basically reverted my patch.
> > > >
> > > > You are not the first to report a problem caused by this patch.
> > > >
> > > > This patch is known to have uncovered some driver bugs.
> > > >
> > > > We are not going to revert it. We are going to fix the real bugs.
> > > >
> > > > Thanks
> > >
> > > Agreed, we are looking into issue Thomas.
> >
> > any news from the Intel labs what might be going on here?
> >
> > We started seeing those hangs on "MSI B85M ECO" boards, too,
> > though it's way more sporadic there.
> 
> I have not heard anything, so I have added Aaron Brown to see if he has
> any additional information.

I do not have any real info.  I had been asked to try and reproduce some unit hangs (maybe for this) recently and did not succeed in producing them on the parts I have.  Reading through the thread I see this is showing up in a NAT environment.  The port that is getting the unit hang in the NAT system?

I will make some attempts at replicating this with the port in a NAT and or forwarding role.  Has a bug been opened for this?  Or has information for this specific unit hang been entered into one of the other unit hang bugs opened against e1000e?

Re: [bisected regression] e1000e: "Detected Hardware Unit Hang"

[Thomas Jarosch]

Hi Aaron,

On Thursday, 12. February 2015 23:28:27 Brown, Aaron F wrote:
> I do not have any real info.  I had been asked to try and reproduce some
> unit hangs (maybe for this) recently and did not succeed in producing
> them on the parts I have.  Reading through the thread I see this is
> showing up in a NAT environment.  The port that is getting the unit hang
> in the NAT system?

yes, the e1000e NIC is serving the NATed Windows client.

The setup was outlined here:

    http://marc.info/?l=linux-netdev&m=142133691713824&w=2

> I will make some attempts at replicating this with the port in a NAT and
> or forwarding role.  Has a bug been opened for this?  Or has information
> for this specific unit hang been entered into one of the other unit hang
> bugs opened against e1000e?

I didn't do anything(tm). This report sounds like the same issue:

    http://ehc.ac/p/e1000/bugs/378/

Oliver Wagner wrote the problem started to appear
after updating from kernel 3.5 to 3.8.0.35 (new frag size code).

I just noticed now he wrote he has two identical boxes:

---------------------------------------------------
- Box with symptoms: Router/Firewall, packet forwarding
  between different VLANs on eth0 and eth1
- Box without symptoms: Fileserver, eth0/eth1 bonded
  (VLANs used, but no forwarding)
---------------------------------------------------

So it looks like it's related to forwarding somehow,
I've made the same experience IIRC.

Cheers,
Thomas

RE: [bisected regression] e1000e: "Detected Hardware Unit Hang"

[Brown, Aaron F]

> -----Original Message-----
> From: Thomas Jarosch [mailto:thomas.jarosch@intra2net.com]
> Sent: Friday, February 13, 2015 8:15 AM
> To: Brown, Aaron F
> Cc: Kirsher, Jeffrey T; 'Linux Netdev List'; Eric Dumazet; e1000-devel
> Subject: Re: [bisected regression] e1000e: "Detected Hardware Unit Hang"
> 
> Hi Aaron,
> 
> On Thursday, 12. February 2015 23:28:27 Brown, Aaron F wrote:
> > I do not have any real info.  I had been asked to try and reproduce some
> > unit hangs (maybe for this) recently and did not succeed in producing
> > them on the parts I have.  Reading through the thread I see this is
> > showing up in a NAT environment.  The port that is getting the unit hang
> > in the NAT system?
> 
> yes, the e1000e NIC is serving the NATed Windows client.
> 
> The setup was outlined here:
> 
>     http://marc.info/?l=linux-netdev&m=142133691713824&w=2
> 
> > I will make some attempts at replicating this with the port in a NAT and
> > or forwarding role.  Has a bug been opened for this?  Or has information
> > for this specific unit hang been entered into one of the other unit hang
> > bugs opened against e1000e?
> 
> I didn't do anything(tm). This report sounds like the same issue:
> 
>     http://ehc.ac/p/e1000/bugs/378/
> 
> Oliver Wagner wrote the problem started to appear
> after updating from kernel 3.5 to 3.8.0.35 (new frag size code).
> 
> I just noticed now he wrote he has two identical boxes:
> 
> ---------------------------------------------------
> - Box with symptoms: Router/Firewall, packet forwarding
>   between different VLANs on eth0 and eth1
> - Box without symptoms: Fileserver, eth0/eth1 bonded
>   (VLANs used, but no forwarding)
> ---------------------------------------------------
> 
> So it looks like it's related to forwarding somehow,
> I've made the same experience IIRC.

Thanks, that (and the multiple bug write-ups on sourceforge) gave me more than enough to go on.  I was able to replicate it on a handful of systems in my lab.  On effected systems setting up a NAT and stressing the interfaces with even moderate traffic levels triggers it pretty quickly.  It appears that the NAT part is unnecessary, just setting the systems up as a software router and running some traffic across it also triggers it giving the same apparent behavior (tx hang, watchdog timeout trace, port reset.)

And with an internal reproduction of the issue I have created an internal bug report, described my set of reproductions, referenced the similar external ones and assigned it to our current e1000e developer.

Thanks again,
Aaron

> 
> Cheers,
> Thomas

Re: [bisected regression] e1000e: "Detected Hardware Unit Hang"

[Thomas Jarosch]

Hi Aaron,

On Saturday, 21. February 2015 01:59:35 Brown, Aaron F wrote:
> Thanks, that (and the multiple bug write-ups on sourceforge) gave me more
> than enough to go on.  I was able to replicate it on a handful of systems
> in my lab.  On effected systems setting up a NAT and stressing the
> interfaces with even moderate traffic levels triggers it pretty quickly. 
> It appears that the NAT part is unnecessary, just setting the systems up
> as a software router and running some traffic across it also triggers it
> giving the same apparent behavior (tx hang, watchdog timeout trace, port
> reset.)
> 
> And with an internal reproduction of the issue I have created an internal
> bug report, described my set of reproductions, referenced the similar
> external ones and assigned it to our current e1000e developer.

just wanted to quickly check if there has been any progress
since the internal bug report has been filed?

Cheers,
Thomas


------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired

RE: [bisected regression] e1000e: "Detected Hardware Unit Hang"

[Brown, Aaron F]

> -----Original Message-----
> From: Thomas Jarosch [mailto:thomas.jarosch@intra2net.com]
> Sent: Monday, March 23, 2015 6:58 AM
> To: Brown, Aaron F
> Cc: Kirsher, Jeffrey T; 'Linux Netdev List'; Eric Dumazet; e1000-devel
> Subject: Re: [bisected regression] e1000e: "Detected Hardware Unit Hang"
> 
> Hi Aaron,
> 
> On Saturday, 21. February 2015 01:59:35 Brown, Aaron F wrote:
> > Thanks, that (and the multiple bug write-ups on sourceforge) gave me
> more
> > than enough to go on.  I was able to replicate it on a handful of
> systems
> > in my lab.  On effected systems setting up a NAT and stressing the
> > interfaces with even moderate traffic levels triggers it pretty quickly.
> > It appears that the NAT part is unnecessary, just setting the systems up
> > as a software router and running some traffic across it also triggers it
> > giving the same apparent behavior (tx hang, watchdog timeout trace, port
> > reset.)
> >
> > And with an internal reproduction of the issue I have created an
> internal
> > bug report, described my set of reproductions, referenced the similar
> > external ones and assigned it to our current e1000e developer.
> 
> just wanted to quickly check if there has been any progress
> since the internal bug report has been filed?

No, no updates beyond a bit of investigation.
 
> 
> Cheers,
> Thomas

Re: RE: [bisected regression] e1000e: "Detected Hardware Unit Hang"

[Thomas Jarosch]

Hi Aaron,

On Monday, 23. March 2015 22:37:08 Brown, Aaron F wrote:
> > >
> > > And with an internal reproduction of the issue I have created an
> > 
> > internal
> > 
> > > bug report, described my set of reproductions, referenced the similar
> > > external ones and assigned it to our current e1000e developer.
> > 
> > 
> > just wanted to quickly check if there has been any progress
> > since the internal bug report has been filed?
> 
> 
> No, no updates beyond a bit of investigation.

any news on this from the Intel labs?

Another two months passed ;) It would be nice to get rid
of the workaround that limits the max fragment size to 4096.

Thanks,
Thomas

RE: RE: [bisected regression] e1000e: "Detected Hardware Unit Hang"

[Brown, Aaron F]

> From: Thomas Jarosch [mailto:thomas.jarosch@intra2net.com]
> Sent: Wednesday, May 27, 2015 9:01 AM
> To: Brown, Aaron F
> Cc: Kirsher, Jeffrey T; 'Linux Netdev List'; Eric Dumazet; e1000-devel
> Subject: Re: RE: [bisected regression] e1000e: "Detected Hardware Unit
> Hang"
> 
> Hi Aaron,
> 
> On Monday, 23. March 2015 22:37:08 Brown, Aaron F wrote:
> > > >
> > > > And with an internal reproduction of the issue I have created an
> > >
> > > internal
> > >
> > > > bug report, described my set of reproductions, referenced the
> similar
> > > > external ones and assigned it to our current e1000e developer.
> > >
> > >
> > > just wanted to quickly check if there has been any progress
> > > since the internal bug report has been filed?
> >
> >
> > No, no updates beyond a bit of investigation.
> 
> any news on this from the Intel labs?

Nothing significant.  Another one of our testers (whom works more closely with the current e1000e driver owner than I) has managed to replicate it on several systems and I know the developer spent some time poking around the setup, but I don't think he's found the root cause yet and has been busy chasing a number of other issues.

> 
> Another two months passed ;) It would be nice to get rid
> of the workaround that limits the max fragment size to 4096.
> 
> Thanks,
> Thomas

Re: [bisected regression] e1000e: "Detected Hardware Unit Hang"

[Thomas Jarosch]

Hi Jeff and Yanir,

On Saturday, 30. May 2015 01:18:44 Brown, Aaron F wrote:
> > any news on this from the Intel labs?
> 
> Nothing significant.  Another one of our testers (whom works more closely
> with the current e1000e driver owner than I) has managed to replicate it
> on several systems and I know the developer spent some time poking around
> the setup, but I don't think he's found the root cause yet and has been
> busy chasing a number of other issues.

so, any news from the Intel labs? I've seen some "hang fixes"
on 03.06.2015, but I'm not sure if they are related to this issue.

This problem is pretty annoying: We have a performance penalty for all 
network cards right now as the buffer size of the core network stack
had to be decreased to 4096 bytes again on our side.
(https://www.marc.info/?l=linux-netdev&m=142131668206333)
Better than no e1000e network connectivity though.

The initial report on this issue was on 14.01.2015:
https://www.marc.info/?l=linux-netdev&m=142124954120315

Best regards,
Thomas

Re: [bisected regression] e1000e: "Detected Hardware Unit Hang"

[Juliana Rodrigueiro]

Hi All.

While updating to kernel 4.19, we realised that a problem reported in 2015 for 
kernel 3.7 is still around. Please see this link for more details: https://
marc.info/?l=linux-netdev&m=142124954120315

Basically, when using the e1000e driver, each few minutes the following 
messages appear in dmesg or system log.

[12465.174759] e1000e 0000:00:19.0 eth0: Detected Hardware Unit Hang:
  TDH                  <c6>
  TDT                  <fb>
  next_to_use          <fb>
  next_to_clean        <c4>
buffer_info[next_to_clean]:
  time_stamp           <2e5e92>
  next_to_watch        <c6>
  jiffies              <2e67e8>
  next_to_watch.status <0>
MAC Status             <40080083>
PHY Status             <796d>
PHY 1000BASE-T Status  <7800>
PHY Extended Status    <3000>
PCI Status             <10>

Back in 2015, we applied a workaround that decreases the page size:

diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index 85ab7d7..9f0ef97 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -2108,7 +2108,7 @@ static inline void __skb_queue_purge(struct 
sk_buff_head *list)
                kfree_skb(skb);
 }
 
-#define NETDEV_FRAG_PAGE_MAX_ORDER get_order(32768)
+#define NETDEV_FRAG_PAGE_MAX_ORDER get_order(4096)
 #define NETDEV_FRAG_PAGE_MAX_SIZE  (PAGE_SIZE << NETDEV_FRAG_PAGE_MAX_ORDER)
 #define NETDEV_PAGECNT_MAX_BIAS           NETDEV_FRAG_PAGE_MAX_SIZE
 

Testing kernel 4.19 with the same hardware showed the same problems, so we 
tried to adapt the old workaround to the current code:

diff -u -r -p linux-4.19.i686/net/core/sock.c linux-4.19.i686.e1000e/net/core/
sock.c
--- linux-4.19.i686/net/core/sock.c     2019-03-22 13:55:24.198266383 +0100
+++ linux-4.19.i686.e1000e/net/core/sock.c      2019-03-22 13:56:43.165765856 
+0100
@@ -2183,7 +2183,8 @@ static void sk_leave_memory_pressure(str
 }
 
 /* On 32bit arches, an skb frag is limited to 2^15 */
-#define SKB_FRAG_PAGE_ORDER    get_order(32768)
+/* Limit to 4096 instead of 32768 */
+#define SKB_FRAG_PAGE_ORDER    get_order(4096)
 
 /**
  * skb_page_frag_refill - check that a page_frag contains enough room


Unfortunately, this patch does not help with the "Unit Hang" messages anymore, 
the problem occurs with any page size.


Some insight in how to deal with this problem would be very much appreciated.

Thank you!