tg3 bad performance, lots of hardware interrupts

tg3 bad performance, lots of hardware interrupts

[Harald Hannelius]

Hi there,

I experience a lot of hardware interrupts with a BCM5751 PCI-express NIC 
(tg3). modprobe tg3, ifconfig ethX up and friends makes the system 
unresponsive. Just having the interface up makes the system sluggish.

Onboard forcedeth works fine (with the same cable).

iperf gives me just 2Mbps on a 1Gbps ethernet. Load average near 1.0. top 
reports 40-50%hi (hardware interrupts) when generating traffic over that 
interface.

The system is a Supermicro H8SMI-2 motherboard, HP EA833AA BROADCOM 
NETXTREME PCI-express NIC, 2GB RAM, Dual-Core opteron 2.8GHz.

All of our other servers with broadcom NIC's work fine with tg3, but they 
aren't PCI-express.

I have tried booting with pci=nomsi and pci=routeirq (BIOS with or without 
"pnp OS" defined). Ubuntu 7.10 live-cd same result. Haven't tried other 
OS'es.

irq_balancing enabled.

Kernel conf: http://www.iki.fi/~harald/kernconf.gz (9kB).

Any hints on what to check for? Hardware or tg3 driverproblem?


I would be glad to report more info, if needed. I have profiling support 
but I have never profiled a kernel before. I haven't got any other PCI-e 
NIC's to test with, yet.


# uname -r
2.6.24.4

# dmesg|grep tg
tg3.c:v3.86 (November 9, 2007)

# lspci -vvv
07:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5751 Gigabit Ethernet PCI Express (rev 21)
 	Subsystem: Broadcom Corporation NetXtreme BCM5751 Gigabit Ethernet PCI Express
 	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
 	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
 	Latency: 0, Cache Line Size: 64 bytes
 	Interrupt: pin A routed to IRQ 216
 	Region 0: Memory at febf0000 (64-bit, non-prefetchable) [size=64K]
 	Expansion ROM at febe0000 [disabled] [size=64K]
 	Capabilities: [48] Power Management version 2
 		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
 		Status: D0 PME-Enable+ DSel=0 DScale=1 PME-
 	Capabilities: [50] Vital Product Data
 	Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3 Enable+
 		Address: 00000000fee0300c  Data: 4142
 	Capabilities: [d0] Express Endpoint IRQ 0
 		Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag+
 		Device: Latency L0s <4us, L1 unlimited
 		Device: AtnBtn- AtnInd- PwrInd-
 		Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
 		Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
 		Device: MaxPayload 128 bytes, MaxReadReq 4096 bytes
 		Link: Supported Speed 2.5Gb/s, Width x1, ASPM L0s L1, Port 0
 		Link: Latency L0s <4us, L1 <64us
 		Link: ASPM L0s L1 Enabled RCB 64 bytes CommClk+ ExtSynch-
 		Link: Speed 2.5Gb/s, Width x1


-- 
A: Top Posters!                                      |  s/y Charlotta |
Q: What is the most annoying thing on mailing lists? |    FIN-2674    |
   http://www.fe83.org/ Finn Express Purjehtijat ry   |  ============= |
Harald H Hannelius | harald (At) iki (dot) fi | GSM +358 50 594 1020

Re: tg3 bad performance, lots of hardware interrupts

[David Miller]

From: Harald Hannelius <harald@iki.fi>
Date: Thu, 27 Mar 2008 15:53:56 +0200 (EET)

You'll have better luck if you report this to the networking
developer list (netdev) and the tg3 driver maintainer (Michael
Chan), who are both now CC:'d.

> I experience a lot of hardware interrupts with a BCM5751 PCI-express NIC 
> (tg3). modprobe tg3, ifconfig ethX up and friends makes the system 
> unresponsive. Just having the interface up makes the system sluggish.
> 
> Onboard forcedeth works fine (with the same cable).
> 
> iperf gives me just 2Mbps on a 1Gbps ethernet. Load average near 1.0. top 
> reports 40-50%hi (hardware interrupts) when generating traffic over that 
> interface.
> 
> The system is a Supermicro H8SMI-2 motherboard, HP EA833AA BROADCOM 
> NETXTREME PCI-express NIC, 2GB RAM, Dual-Core opteron 2.8GHz.
> 
> All of our other servers with broadcom NIC's work fine with tg3, but they 
> aren't PCI-express.
> 
> I have tried booting with pci=nomsi and pci=routeirq (BIOS with or without 
> "pnp OS" defined). Ubuntu 7.10 live-cd same result. Haven't tried other 
> OS'es.
> 
> irq_balancing enabled.
> 
> Kernel conf: http://www.iki.fi/~harald/kernconf.gz (9kB).
> 
> Any hints on what to check for? Hardware or tg3 driverproblem?
> 
> 
> I would be glad to report more info, if needed. I have profiling support 
> but I have never profiled a kernel before. I haven't got any other PCI-e 
> NIC's to test with, yet.
> 
> 
> # uname -r
> 2.6.24.4
> 
> # dmesg|grep tg
> tg3.c:v3.86 (November 9, 2007)
> 
> # lspci -vvv
> 07:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5751 Gigabit Ethernet PCI Express (rev 21)
>  	Subsystem: Broadcom Corporation NetXtreme BCM5751 Gigabit Ethernet PCI Express
>  	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
>  	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
>  	Latency: 0, Cache Line Size: 64 bytes
>  	Interrupt: pin A routed to IRQ 216
>  	Region 0: Memory at febf0000 (64-bit, non-prefetchable) [size=64K]
>  	Expansion ROM at febe0000 [disabled] [size=64K]
>  	Capabilities: [48] Power Management version 2
>  		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
>  		Status: D0 PME-Enable+ DSel=0 DScale=1 PME-
>  	Capabilities: [50] Vital Product Data
>  	Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3 Enable+
>  		Address: 00000000fee0300c  Data: 4142
>  	Capabilities: [d0] Express Endpoint IRQ 0
>  		Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag+
>  		Device: Latency L0s <4us, L1 unlimited
>  		Device: AtnBtn- AtnInd- PwrInd-
>  		Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
>  		Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>  		Device: MaxPayload 128 bytes, MaxReadReq 4096 bytes
>  		Link: Supported Speed 2.5Gb/s, Width x1, ASPM L0s L1, Port 0
>  		Link: Latency L0s <4us, L1 <64us
>  		Link: ASPM L0s L1 Enabled RCB 64 bytes CommClk+ ExtSynch-
>  		Link: Speed 2.5Gb/s, Width x1
> 
> 
> -- 
> A: Top Posters!                                      |  s/y Charlotta |
> Q: What is the most annoying thing on mailing lists? |    FIN-2674    |
>    http://www.fe83.org/ Finn Express Purjehtijat ry   |  ============= |
> Harald H Hannelius | harald (At) iki (dot) fi | GSM +358 50 594 1020
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: tg3 bad performance, lots of hardware interrupts

[Michael Chan]

On Thu, 2008-03-27 at 14:49 -0700, David Miller wrote:
> From: Harald Hannelius <harald@iki.fi>
> Date: Thu, 27 Mar 2008 15:53:56 +0200 (EET)
> 
> > I experience a lot of hardware interrupts with a BCM5751 PCI-express NIC 
> > (tg3). modprobe tg3, ifconfig ethX up and friends makes the system 
> > unresponsive. Just having the interface up makes the system sluggish.
> > 

I just tested a similar NIC using the same kernel and driver, but I did
not notice anything unusual.  netperf gave me 941Mbps.

> > Onboard forcedeth works fine (with the same cable).
> > 
> > iperf gives me just 2Mbps on a 1Gbps ethernet. Load average near 1.0. top 
> > reports 40-50%hi (hardware interrupts) when generating traffic over that 
> > interface.

Can you look at /proc/interrupts to see roughly how many are reported
per second when link is down, link is up with no traffic, and with
traffic?

Finally, you can also try ethtool -t eth0 to see if it passes a simple
self test.

Re: tg3 bad performance, lots of hardware interrupts

[Harald Hannelius]

On Thu, 27 Mar 2008, Michael Chan wrote:

> On Thu, 2008-03-27 at 14:49 -0700, David Miller wrote:
>> From: Harald Hannelius <harald@iki.fi>
>> Date: Thu, 27 Mar 2008 15:53:56 +0200 (EET)
>>
>>> I experience a lot of hardware interrupts with a BCM5751 PCI-express NIC
>>> (tg3). modprobe tg3, ifconfig ethX up and friends makes the system
>>> unresponsive. Just having the interface up makes the system sluggish.
>>>
>
> I just tested a similar NIC using the same kernel and driver, but I did
> not notice anything unusual.  netperf gave me 941Mbps.
>
>>> Onboard forcedeth works fine (with the same cable).
>>>
>>> iperf gives me just 2Mbps on a 1Gbps ethernet. Load average near 1.0. top
>>> reports 40-50%hi (hardware interrupts) when generating traffic over that
>>> interface.
>
> Can you look at /proc/interrupts to see roughly how many are reported
> per second when link is down, link is up with no traffic, and with
> traffic?
>
> Finally, you can also try ethtool -t eth0 to see if it passes a simple
> self test.

Phew, I thought that running ethtool -t was like doing stop-A-sync on a 
Sun. It took almost half an hour to run that ethtool -t command;

'mpstat 2' output while running 'ethtool -t eth2':

Full log here (2160 lines): http://www.iki.fi/~harald/mpstat.log

A few lines from that log here;

Linux 2.6.24.4 (mauer)  03/28/2008

10:37:14 AM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
10:37:16 AM  all    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00    115.50
10:37:18 AM  all    0.00    0.00    1.00    0.00    0.00    0.00    0.00  99.00    123.50
10:37:20 AM  all    0.00    0.00    0.75    0.00    0.00    0.00    0.00   99.25    142.79
10:37:23 AM  all    0.00    0.00   25.66    0.00   23.77    2.26    0.00   48.30     76.38
10:37:25 AM  all    0.00    0.00   17.13    0.00   31.12   21.68    0.00   30.07     75.12
10:37:27 AM  all    0.00    0.00   12.17    0.00   25.48   28.90    0.00   33.46    103.00
10:37:29 AM  all    0.00    0.00    0.40    0.00   38.15   23.29    0.00   38.15     70.15
10:37:31 AM  all    0.00    0.00    0.40    0.00   41.90   23.32    0.00   34.39     71.43
10:37:33 AM  all    0.00    0.00    2.83    0.00   29.68   34.98    0.00   32.51    123.38
10:37:35 AM  all    0.00    0.00   11.07    0.00   33.21   24.81    0.00   30.92     78.74
... goes on like this for almost half-an-hour ...
11:04:03 AM  all    0.00    0.00    0.00    0.00   34.69   22.86    0.00   42.45     71.50
11:04:05 AM  all    0.00    0.00    1.53    0.00   33.72   26.82    0.00   37.93     84.50
11:04:07 AM  all    0.00    0.00    0.40    0.00   38.15   22.89    0.00   38.55     74.26
11:04:09 AM  all    0.00    0.00    0.00    0.00   38.80   21.60    0.00   39.60     72.64
11:04:11 AM  all    0.00    0.00    0.00    0.00   42.17   23.29    0.00   34.54     69.50
11:04:13 AM  all    0.00    0.00    3.60    1.44   31.29   33.45    0.00   30.22    111.39
11:04:15 AM  all    0.00    0.00    0.00    0.00    0.51    0.76    0.00   98.73    110.50
11:04:17 AM  all    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00    119.50
11:04:19 AM  all    0.00    0.00    0.25    1.24    0.00    0.00    0.00  98.51    126.87
11:04:21 AM  all    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00    106.50
... command finished ...

Sadly I didn't capture the output from ethtool -t, failed anyway. I'm 
running that test a second time right now, I'll post the output later 
(when finished). While the 'ethtool -t eth2' is running eth0 (forcedeth) 
doesn't respond to ping, when going to the console it didn't wake up with 
a keypress, nor did capslock/numlock react.

'mpstat 2' while running modprobe tg3:
Linux 2.6.24.4 (mauer)  03/28/2008

02:47:05 PM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
02:47:07 PM  all    0.00    0.00    0.75    0.00    0.00    0.00    0.00   99.25    138.00
02:47:09 PM  all    0.00    0.00    0.25    0.00    0.00    0.00    0.00   99.75    120.00
02:47:11 PM  all    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00    132.50
02:47:13 PM  all    0.00    0.00   14.19    0.00   17.16    1.65    0.00  67.00     92.08
02:47:15 PM  all    0.00    0.00   19.60    0.00   30.40    1.20    0.00  48.80     60.50
02:47:17 PM  all    0.56    0.00    4.20    0.00    5.88    0.28    0.00  89.08    127.14
02:47:19 PM  all    0.00    0.00    0.75    0.00    0.25    0.00    0.00  99.00    127.00

'mpstat 2' while running ifconfig eth2 up:
Linux 2.6.24.4 (mauer)  03/28/2008

02:47:50 PM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
02:47:52 PM  all    0.00    0.00    0.40    0.00   44.95   16.49    0.00   38.16    194.47
02:47:54 PM  all    0.00    0.00    0.76    0.00   46.91   16.77    0.00   35.56    205.47
02:47:56 PM  all    0.00    0.00    2.06    0.00   15.88   14.12    0.00   67.94    107.96

Hope my overlong lines don't get wrapped...


No hints on dmesg on what's going on.



-- 
A: Top Posters!                                      |  s/y Charlotta |
Q: What is the most annoying thing on mailing lists? |    FIN-2674    |
   http://www.fe83.org/ Finn Express Purjehtijat ry   |  ============= |
Harald H Hannelius | harald (At) iki (dot) fi | GSM +358 50 594 1020

Re: tg3 bad performance, lots of hardware interrupts

[Jiri Kosina]

On Fri, 28 Mar 2008, Michael Chan wrote:

> > Phew, I thought that running ethtool -t was like doing stop-A-sync on 
> > a Sun. It took almost half an hour to run that ethtool -t command;
> Something is very wrong.  ethtool -t should only take a few seconds to
> complete.  You can try ethtool -t eth0 online to reduce the number of
> tests to see if it makes a difference.
> How many of these NICs do you have?  If you have more than one, do they
> all behave the same way?  Have they ever worked well before?

Harald, is the IRQ of eth0 shared with any other device? (cat 
/proc/interrupts will show).

-- 
Jiri Kosina
SUSE Labs

Re: tg3 bad performance, lots of hardware interrupts

[Harald Hannelius]

On Fri, 28 Mar 2008, Michael Chan wrote:
> On Fri, 2008-03-28 at 15:04 +0200, Harald Hannelius wrote:

>> Phew, I thought that running ethtool -t was like doing stop-A-sync on
>> a
>> Sun. It took almost half an hour to run that ethtool -t command;
>
> Something is very wrong.  ethtool -t should only take a few seconds to
> complete.  You can try ethtool -t eth0 online to reduce the number of
> tests to see if it makes a difference.

Here's the output of ethtool -t eth2:

The test result is PASS
The test extra info:
nvram test     (online)          0
link test      (online)          0
register test  (offline)         0
memory test    (offline)         0
loopback test  (offline)         0
interrupt test (offline)         0

I just started a 'ethtool -t eth2 online' and that one took just some 10 
seconds or so.

# ethtool -t eth2 online 2>&1 | tee ethtool-output.log
The test result is PASS
The test extra info:
nvram test     (online) 	 0
link test      (online) 	 0
register test  (offline)	 0
memory test    (offline)	 0
loopback test  (offline)	 0
interrupt test (offline)	 0
# mpstat 2 2>&1 | tee ethtool2.log
Linux 2.6.24.4 (mauer) 	03/28/2008

05:22:42 PM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
05:22:44 PM  all    0.00    0.00    0.00    0.00   11.63    7.56    0.00   80.81    102.99
05:22:46 PM  all    0.00    0.00    0.27    0.00    8.33    4.84    0.00   86.56    123.00
05:22:48 PM  all    0.00    0.00    0.00    0.00   12.03    7.16    0.00   80.80    108.46
05:22:50 PM  all    0.00    0.00    0.00    0.00   10.47    5.23    0.00   84.30    113.93
05:22:52 PM  all    0.00    0.00    0.00    0.00   12.93   10.06    0.00   77.01    135.00
05:22:55 PM  all    0.00    0.00   18.11    0.00   24.25   12.62    0.00   45.02    158.28
05:22:57 PM  all    0.00    0.00    0.82    0.00   13.11   10.38    0.00   75.68    146.04
05:22:59 PM  all    0.00    0.00    0.29    0.00   18.77   12.61    0.00   68.33    136.32
05:23:01 PM  all    0.00    0.00    0.00    0.00   13.64    7.67    0.00   78.69    112.50
05:23:03 PM  all    0.00    0.00    0.00    0.00    9.19    4.86    0.00   85.95    110.50
05:23:05 PM  all    0.00    0.00    0.00    0.00   13.64    7.67    0.00   78.69    108.00

> How many of these NICs do you have?  If you have more than one, do they
> all behave the same way?  Have they ever worked well before?

It's a brand spanking new computer, equipped with three of these BCM5751 
NICs made by HP. I have ripped out all but one, and I have also tested the 
NIC in all three available PCIe slots. Same result.

When all three NIC's where plugged in they didn't work. All three behave 
the same one-by-one.

I don't know if they have worked before.

-- 
A: Top Posters!                                      |  s/y Charlotta |
Q: What is the most annoying thing on mailing lists? |    FIN-2674    |
   http://www.fe83.org/ Finn Express Purjehtijat ry   |  ============= |
Harald H Hannelius | harald (At) iki (dot) fi | GSM +358 50 594 1020

Re: tg3 bad performance, lots of hardware interrupts

[Harald Hannelius]

On Fri, 28 Mar 2008, Jiri Kosina wrote:
> On Fri, 28 Mar 2008, Michael Chan wrote:

>>> Phew, I thought that running ethtool -t was like doing stop-A-sync on
>>> a Sun. It took almost half an hour to run that ethtool -t command;
>> Something is very wrong.  ethtool -t should only take a few seconds to
>> complete.  You can try ethtool -t eth0 online to reduce the number of
>> tests to see if it makes a difference.
>> How many of these NICs do you have?  If you have more than one, do they
>> all behave the same way?  Have they ever worked well before?
>
> Harald, is the IRQ of eth0 shared with any other device? (cat
> /proc/interrupts will show).

# cat /proc/interrupts
            CPU0       CPU1
   0:        111          1   IO-APIC-edge      timer
   1:          0          2   IO-APIC-edge      i8042
   2:          0          0    XT-PIC-XT        cascade
   5:          0          0   IO-APIC-fasteoi   sata_nv
   7:        856         51   IO-APIC-fasteoi   ohci_hcd:usb2
  10:          0          3   IO-APIC-fasteoi   sata_nv, ehci_hcd:usb1
  11:       4305          7   IO-APIC-fasteoi   sata_nv
  12:          0          4   IO-APIC-edge      i8042
216:       4217     128932   PCI-MSI-edge      eth2
217:     161107     685351   PCI-MSI-edge      eth0
NMI:          0          0   Non-maskable interrupts
LOC:    2380762    2619917   Local timer interrupts
RES:       3000       3269   Rescheduling interrupts
CAL:         16         31   function call interrupts
TLB:         64        111   TLB shootdowns
TRM:          0          0   Thermal event interrupts
SPU:          0          0   Spurious interrupts
ERR:          1
MIS:          0

Well, shared or not, yes and no. I think that /proc/interrupts contains 
soft-interrupts. The problem child is interface eth2.

As rapported by ifconfig the interface is on IRQ 5:

# ifconfig eth2
eth2      Link encap:Ethernet  HWaddr 00:10:18:30:E6:D6
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:196898 errors:0 dropped:0 overruns:0 frame:0
           TX packets:19 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:69887991 (66.6 MiB)  TX bytes:1216 (1.1 KiB)
           Interrupt:5

That'd be the same as sata_nv.

# ifconfig eth2
eth2      Link encap:Ethernet  HWaddr 00:10:18:30:E6:D6
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:196898 errors:0 dropped:0 overruns:0 frame:0
           TX packets:19 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:69887991 (66.6 MiB)  TX bytes:1216 (1.1 KiB)
           Interrupt:5

I changed the settings "PnP OS" in the BIOS (acpi on/off?) and tried 
booting with both pci=routeirq (or smth like that, see original post) on 
and off to no avail.

I'm stumped. I have never experienced anything quite like this before. 
Usually an IRQ-conflict has crashed my computers, not just slowed them 
down (or maybe these dual-core opterons are just that incredibly fast 
nowadays that the do nothing incredibly fast :) ). Then again, I haven't 
had an IRQ-conflict on my boxen in years.

Buggy motherboard? Buggy NIC? The motherboard has the latest available 
BIOS as per supermicro's webpage.

I'm getting three PCIe e1000's next week, I'll try with these instead.


-- 
A: Top Posters!                                      |  s/y Charlotta |
Q: What is the most annoying thing on mailing lists? |    FIN-2674    |
   http://www.fe83.org/ Finn Express Purjehtijat ry   |  ============= |
Harald H Hannelius | harald (At) iki (dot) fi | GSM +358 50 594 1020

Re: tg3 bad performance, lots of hardware interrupts

[Michael Chan]

On Fri, 2008-03-28 at 15:04 +0200, Harald Hannelius wrote:
> Phew, I thought that running ethtool -t was like doing stop-A-sync on
> a 
> Sun. It took almost half an hour to run that ethtool -t command;

Something is very wrong.  ethtool -t should only take a few seconds to
complete.  You can try ethtool -t eth0 online to reduce the number of
tests to see if it makes a difference.

How many of these NICs do you have?  If you have more than one, do they
all behave the same way?  Have they ever worked well before?

Re: tg3 bad performance, lots of hardware interrupts

[Harald Hannelius]

On Fri, 28 Mar 2008, Michael Chan wrote:

> On Fri, 2008-03-28 at 19:37 +0200, Harald Hannelius wrote:
>> # cat /proc/interrupts
>>             CPU0       CPU1
>>    0:        111          1   IO-APIC-edge      timer
>>    1:          0          2   IO-APIC-edge      i8042
>>    2:          0          0    XT-PIC-XT        cascade
>>    5:          0          0   IO-APIC-fasteoi   sata_nv
>>    7:        856         51   IO-APIC-fasteoi   ohci_hcd:usb2
>>   10:          0          3   IO-APIC-fasteoi   sata_nv, ehci_hcd:usb1
>>   11:       4305          7   IO-APIC-fasteoi   sata_nv
>>   12:          0          4   IO-APIC-edge      i8042
>> 216:       4217     128932   PCI-MSI-edge      eth2
>> 217:     161107     685351   PCI-MSI-edge      eth0
>> NMI:          0          0   Non-maskable interrupts
>> LOC:    2380762    2619917   Local timer interrupts
>> RES:       3000       3269   Rescheduling interrupts
>> CAL:         16         31   function call interrupts
>> TLB:         64        111   TLB shootdowns
>> TRM:          0          0   Thermal event interrupts
>> SPU:          0          0   Spurious interrupts
>> ERR:          1
>> MIS:          0
>>
>> Well, shared or not, yes and no. I think that /proc/interrupts
>> contains
>> soft-interrupts. The problem child is interface eth2.
>>
>> As rapported by ifconfig the interface is on IRQ 5:
>
> eth2 is using MSI.  When using MSI, the IRQ reported by ifconfig is not
> accurate.  You said you have tried booting with nomsi, but have you
> confirmed that by checking /proc/interrupts?

I did, at least the interface wasn't on MSI-edge anymore. I cannot 
remember what IRQ the NIC took when booted with pci=nomsi (something like 
that), it could have been 10 or other IRQ below 20, if I recall correctly.


-- 
A: Top Posters!                                      |  s/y Charlotta |
Q: What is the most annoying thing on mailing lists? |    FIN-2674    |
   http://www.fe83.org/ Finn Express Purjehtijat ry   |  ============= |
Harald H Hannelius | harald (At) iki (dot) fi | GSM +358 50 594 1020

Re: tg3 bad performance, lots of hardware interrupts

[Michael Chan]

On Fri, 2008-03-28 at 19:37 +0200, Harald Hannelius wrote:
> # cat /proc/interrupts
>             CPU0       CPU1
>    0:        111          1   IO-APIC-edge      timer
>    1:          0          2   IO-APIC-edge      i8042
>    2:          0          0    XT-PIC-XT        cascade
>    5:          0          0   IO-APIC-fasteoi   sata_nv
>    7:        856         51   IO-APIC-fasteoi   ohci_hcd:usb2
>   10:          0          3   IO-APIC-fasteoi   sata_nv, ehci_hcd:usb1
>   11:       4305          7   IO-APIC-fasteoi   sata_nv
>   12:          0          4   IO-APIC-edge      i8042
> 216:       4217     128932   PCI-MSI-edge      eth2
> 217:     161107     685351   PCI-MSI-edge      eth0
> NMI:          0          0   Non-maskable interrupts
> LOC:    2380762    2619917   Local timer interrupts
> RES:       3000       3269   Rescheduling interrupts
> CAL:         16         31   function call interrupts
> TLB:         64        111   TLB shootdowns
> TRM:          0          0   Thermal event interrupts
> SPU:          0          0   Spurious interrupts
> ERR:          1
> MIS:          0
> 
> Well, shared or not, yes and no. I think that /proc/interrupts
> contains 
> soft-interrupts. The problem child is interface eth2.
> 
> As rapported by ifconfig the interface is on IRQ 5:

eth2 is using MSI.  When using MSI, the IRQ reported by ifconfig is not
accurate.  You said you have tried booting with nomsi, but have you
confirmed that by checking /proc/interrupts?

> 
> # ifconfig eth2
> eth2      Link encap:Ethernet  HWaddr 00:10:18:30:E6:D6
>            UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>            RX packets:196898 errors:0 dropped:0 overruns:0 frame:0
>            TX packets:19 errors:0 dropped:0 overruns:0 carrier:0
>            collisions:0 txqueuelen:1000
>            RX bytes:69887991 (66.6 MiB)  TX bytes:1216 (1.1 KiB)
>            Interrupt:5
> 

Re: tg3 bad performance, lots of hardware interrupts

[Harald Hannelius]

On Fri, 28 Mar 2008, Harald Hannelius wrote:
> On Fri, 28 Mar 2008, Jiri Kosina wrote:
>> On Fri, 28 Mar 2008, Michael Chan wrote:
>
>>> Something is very wrong.  ethtool -t should only take a few seconds to
>>> complete.  You can try ethtool -t eth0 online to reduce the number of
>>> tests to see if it makes a difference.
>>> How many of these NICs do you have?  If you have more than one, do they
>>> all behave the same way?  Have they ever worked well before?
>> 
>> Harald, is the IRQ of eth0 shared with any other device? (cat
>> /proc/interrupts will show).
>
> # cat /proc/interrupts
>           CPU0       CPU1
>  0:        111          1   IO-APIC-edge      timer
>  1:          0          2   IO-APIC-edge      i8042
>  2:          0          0    XT-PIC-XT        cascade
>  5:          0          0   IO-APIC-fasteoi   sata_nv
>  7:        856         51   IO-APIC-fasteoi   ohci_hcd:usb2
> 10:          0          3   IO-APIC-fasteoi   sata_nv, ehci_hcd:usb1
> 11:       4305          7   IO-APIC-fasteoi   sata_nv
> 12:          0          4   IO-APIC-edge      i8042
> 216:       4217     128932   PCI-MSI-edge      eth2
> 217:     161107     685351   PCI-MSI-edge      eth0
> NMI:          0          0   Non-maskable interrupts
> LOC:    2380762    2619917   Local timer interrupts
> RES:       3000       3269   Rescheduling interrupts
> CAL:         16         31   function call interrupts
> TLB:         64        111   TLB shootdowns
> TRM:          0          0   Thermal event interrupts
> SPU:          0          0   Spurious interrupts
> ERR:          1
> MIS:          0
>
> Well, shared or not, yes and no. I think that /proc/interrupts contains 
> soft-interrupts. The problem child is interface eth2.
>
> As rapported by ifconfig the interface is on IRQ 5:
>
> # ifconfig eth2
> eth2      Link encap:Ethernet  HWaddr 00:10:18:30:E6:D6
>          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>          RX packets:196898 errors:0 dropped:0 overruns:0 frame:0
>          TX packets:19 errors:0 dropped:0 overruns:0 carrier:0
>          collisions:0 txqueuelen:1000
>          RX bytes:69887991 (66.6 MiB)  TX bytes:1216 (1.1 KiB)
>          Interrupt:5
>
> That'd be the same as sata_nv.
>
> # ifconfig eth2
> eth2      Link encap:Ethernet  HWaddr 00:10:18:30:E6:D6
>          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>          RX packets:196898 errors:0 dropped:0 overruns:0 frame:0
>          TX packets:19 errors:0 dropped:0 overruns:0 carrier:0
>          collisions:0 txqueuelen:1000
>          RX bytes:69887991 (66.6 MiB)  TX bytes:1216 (1.1 KiB)
>          Interrupt:5
>
> I changed the settings "PnP OS" in the BIOS (acpi on/off?) and tried booting 
> with both pci=routeirq (or smth like that, see original post) on and off to 
> no avail.
>
> I'm stumped. I have never experienced anything quite like this before. 
> Usually an IRQ-conflict has crashed my computers, not just slowed them down 
> (or maybe these dual-core opterons are just that incredibly fast nowadays 
> that the do nothing incredibly fast :) ). Then again, I haven't had an 
> IRQ-conflict on my boxen in years.
>
> Buggy motherboard? Buggy NIC? The motherboard has the latest available BIOS 
> as per supermicro's webpage.
>
> I'm getting three PCIe e1000's next week, I'll try with these instead.

For the record, I popped in a couple of PCI-express e1000's and they work 
flawlessly. It's either the interaction between those HP-cards and the 
motherboard, or something with the tg3 driver, I suppose.

Funny though, that e1000e didn't detect the cards, but e1000 did.

-- 
A: Top Posters!                                      |  s/y Charlotta |
Q: What is the most annoying thing on mailing lists? |    FIN-2674    |
   http://www.fe83.org/ Finn Express Purjehtijat ry   |  ============= |
Harald H Hannelius | harald (At) iki (dot) fi | GSM +358 50 594 1020