* high latency on 82573L
@ 2010-09-02 3:39 Tony Jones
2010-09-02 18:49 ` Allan, Bruce W
0 siblings, 1 reply; 6+ messages in thread
From: Tony Jones @ 2010-09-02 3:39 UTC (permalink / raw)
To: jeffrey.t.kirsher, jesse.brandeburg, bruce.w.allan,
alexander.h.duyck, peter.p.waskiewicz.jr, john.ronciak
Cc: e1000-devel, linux-kernel, bphilips
Hi.
Since commit 6f461f6c7c (e1000e: enable/disable ASPM L0s and L1 and ERT
according to hardware errata) I'm seeing high latencies on my Thinkpad T60p.
02:00.0 Ethernet controller: Intel Corporation 82573L Gigabit Ethernet Controller
Subsystem: Lenovo ThinkPad T60
Flags: bus master, fast devsel, latency 0, IRQ 30
Memory at ee000000 (32-bit, non-prefetchable) [size=128K]
I/O ports at 3000 [size=32]
Capabilities: [c8] Power Management version 2
Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [e0] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number 00-1a-6b-ff-ff-6c-7e-a4
Kernel driver in use: e1000e
# uname -r
2.6.36-rc3-0-default
# ping -c 20 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=1.06 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=1007 ms
64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=698 ms
64 bytes from 192.168.1.1: icmp_seq=4 ttl=64 time=198 ms
64 bytes from 192.168.1.1: icmp_seq=5 ttl=64 time=697 ms
64 bytes from 192.168.1.1: icmp_seq=6 ttl=64 time=1007 ms
64 bytes from 192.168.1.1: icmp_seq=7 ttl=64 time=690 ms
64 bytes from 192.168.1.1: icmp_seq=8 ttl=64 time=1007 ms
64 bytes from 192.168.1.1: icmp_seq=9 ttl=64 time=682 ms
64 bytes from 192.168.1.1: icmp_seq=10 ttl=64 time=1008 ms
64 bytes from 192.168.1.1: icmp_seq=11 ttl=64 time=1.03 ms
64 bytes from 192.168.1.1: icmp_seq=12 ttl=64 time=1007 ms
64 bytes from 192.168.1.1: icmp_seq=13 ttl=64 time=0.874 ms
64 bytes from 192.168.1.1: icmp_seq=14 ttl=64 time=1009 ms
64 bytes from 192.168.1.1: icmp_seq=15 ttl=64 time=1.72 ms
64 bytes from 192.168.1.1: icmp_seq=16 ttl=64 time=1006 ms
64 bytes from 192.168.1.1: icmp_seq=17 ttl=64 time=650 ms
64 bytes from 192.168.1.1: icmp_seq=18 ttl=64 time=1008 ms
64 bytes from 192.168.1.1: icmp_seq=19 ttl=64 time=642 ms
64 bytes from 192.168.1.1: icmp_seq=20 ttl=64 time=1643 ms
--- 192.168.1.1 ping statistics ---
20 packets transmitted, 20 received, 0% packet loss, time 19063ms
rtt min/avg/max/mdev = 0.874/698.661/1643.392/439.376 ms, pipe 2
This is 2.6.36-rc3 yet commit 19833b5dff (e1000e: disable ASPM L1 on 82573)
isn't having any effect for me.
For our OpenSUSE 11.3 kernels (2.6.34 based), reverting just 6f461f6c7c
solves the issue. For .36-rc3 reverting 6f461f6c7c plus of course 19833b5dff
does the trick.
I'm happy to perform any testing to help narrow this down. LMK.
Tony
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: high latency on 82573L
2010-09-02 3:39 high latency on 82573L Tony Jones
@ 2010-09-02 18:49 ` Allan, Bruce W
2010-09-03 17:51 ` Tony Jones
0 siblings, 1 reply; 6+ messages in thread
From: Allan, Bruce W @ 2010-09-02 18:49 UTC (permalink / raw)
To: Tony Jones, Kirsher, Jeffrey T, Brandeburg, Jesse, Duyck,
Alexander H, Waskiewicz Jr, Peter P, Ronciak, John
Cc: e1000-devel, linux-kernel, bphilips
On Wednesday, September 01, 2010 8:39 PM, Tony Jones wrote:
> Hi.
>
> Since commit 6f461f6c7c (e1000e: enable/disable ASPM L0s and L1 and
> ERT according to hardware errata) I'm seeing high latencies on my
> Thinkpad T60p.
>
> 02:00.0 Ethernet controller: Intel Corporation 82573L Gigabit
> Ethernet Controller Subsystem: Lenovo ThinkPad T60
> Flags: bus master, fast devsel, latency 0, IRQ 30
> Memory at ee000000 (32-bit, non-prefetchable) [size=128K]
> I/O ports at 3000 [size=32]
> Capabilities: [c8] Power Management version 2
> Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
> Capabilities: [e0] Express Endpoint, MSI 00
> Capabilities: [100] Advanced Error Reporting
> Capabilities: [140] Device Serial Number 00-1a-6b-ff-ff-6c-7e-a4
> Kernel driver in use: e1000e
>
> # uname -r
> 2.6.36-rc3-0-default
> # ping -c 20 192.168.1.1
> PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
> 64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=1.06 ms
> 64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=1007 ms
> 64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=698 ms
> 64 bytes from 192.168.1.1: icmp_seq=4 ttl=64 time=198 ms
> 64 bytes from 192.168.1.1: icmp_seq=5 ttl=64 time=697 ms
> 64 bytes from 192.168.1.1: icmp_seq=6 ttl=64 time=1007 ms
> 64 bytes from 192.168.1.1: icmp_seq=7 ttl=64 time=690 ms
> 64 bytes from 192.168.1.1: icmp_seq=8 ttl=64 time=1007 ms
> 64 bytes from 192.168.1.1: icmp_seq=9 ttl=64 time=682 ms
> 64 bytes from 192.168.1.1: icmp_seq=10 ttl=64 time=1008 ms
> 64 bytes from 192.168.1.1: icmp_seq=11 ttl=64 time=1.03 ms
> 64 bytes from 192.168.1.1: icmp_seq=12 ttl=64 time=1007 ms
> 64 bytes from 192.168.1.1: icmp_seq=13 ttl=64 time=0.874 ms
> 64 bytes from 192.168.1.1: icmp_seq=14 ttl=64 time=1009 ms
> 64 bytes from 192.168.1.1: icmp_seq=15 ttl=64 time=1.72 ms
> 64 bytes from 192.168.1.1: icmp_seq=16 ttl=64 time=1006 ms
> 64 bytes from 192.168.1.1: icmp_seq=17 ttl=64 time=650 ms
> 64 bytes from 192.168.1.1: icmp_seq=18 ttl=64 time=1008 ms
> 64 bytes from 192.168.1.1: icmp_seq=19 ttl=64 time=642 ms
> 64 bytes from 192.168.1.1: icmp_seq=20 ttl=64 time=1643 ms
>
> --- 192.168.1.1 ping statistics ---
> 20 packets transmitted, 20 received, 0% packet loss, time 19063ms
> rtt min/avg/max/mdev = 0.874/698.661/1643.392/439.376 ms, pipe 2
>
> This is 2.6.36-rc3 yet commit 19833b5dff (e1000e: disable ASPM L1 on
> 82573) isn't having any effect for me.
>
> For our OpenSUSE 11.3 kernels (2.6.34 based), reverting just
> 6f461f6c7c
> solves the issue. For .36-rc3 reverting 6f461f6c7c plus of course
> 19833b5dff does the trick.
>
> I'm happy to perform any testing to help narrow this down. LMK.
>
> Tony
Please provide more verbose lspci output and include the PCI config
space, i.e. 'lspci -s 2:0.0 -vvv -xxx' after the driver is loaded,
your kernel .config and the full version number of the OpenSUSE
kernel. Are there any messages in the system log regarding disabling
ASPM L0s and/or L1 on that device?
I can understand the latency with the OpenSUSE 2.6.34-based kernels
assuming commit 19833b5dff is not present, but I do not understand
the latency with 2.6.36-rc3.
Thanks,
Bruce.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: high latency on 82573L
2010-09-02 18:49 ` Allan, Bruce W
@ 2010-09-03 17:51 ` Tony Jones
2010-09-03 18:59 ` Allan, Bruce W
0 siblings, 1 reply; 6+ messages in thread
From: Tony Jones @ 2010-09-03 17:51 UTC (permalink / raw)
To: Allan, Bruce W
Cc: Kirsher, Jeffrey T, Brandeburg, Jesse, Duyck, Alexander H,
Waskiewicz Jr, Peter P, Ronciak, John, e1000-devel, linux-kernel,
bphilips
On Thu, Sep 02, 2010 at 11:49:12AM -0700, Allan, Bruce W wrote:
> Please provide more verbose lspci output and include the PCI config
> space, i.e. 'lspci -s 2:0.0 -vvv -xxx' after the driver is loaded,
# lspci -s 2:0.0 -vvv -xxx
02:00.0 Ethernet controller: Intel Corporation 82573L Gigabit Ethernet Controller
Subsystem: Lenovo ThinkPad T60
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 46
Region 0: Memory at ee000000 (32-bit, non-prefetchable) [size=128K]
Region 2: I/O ports at 3000 [size=32]
Capabilities: [c8] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 00000000fee0100c Data: 41c9
Capabilities: [e0] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr+ TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <128ns, L1 <64us
ClockPM+ Surprise- LLActRep- BwNot-
LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr+ BadTLP+ BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
AERCap: First Error Pointer: 14, GenCap- CGenEn- ChkCap- ChkEn-
Capabilities: [140 v1] Device Serial Number 00-1a-6b-ff-ff-6c-7e-a4
Kernel driver in use: e1000e
00: 86 80 9a 10 07 05 10 00 00 00 00 02 10 00 00 00
10: 00 00 00 ee 00 00 00 00 01 30 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 aa 17 01 20
30: 00 00 00 00 c8 00 00 00 00 00 00 00 0b 01 00 00
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 01 d0 22 c8 00 20 00 0f
d0: 05 e0 81 00 0c 10 e0 fe 00 00 00 00 c9 41 00 00
e0: 10 00 01 00 c1 0c 00 00 1f 28 1a 00 11 1c 07 00
f0: 42 01 11 10 00 00 00 00 00 00 00 00 00 00 00 00
> kernel. Are there any messages in the system log regarding disabling
> ASPM L0s and/or L1 on that device?
It would appear it is being disabled:
[ 0.194271] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
[ 0.297112] pci 0000:01:00.0: disabling ASPM on pre-1.1 PCIe device. You can enable it with 'pcie_aspm=force'
[ 0.298003] pci 0000:02:00.0: disabling ASPM on pre-1.1 PCIe device. You can enable it with 'pcie_aspm=force'
[ 0.299123] pci 0000:03:00.0: disabling ASPM on pre-1.1 PCIe device. You can enable it with 'pcie_aspm=force'
[ 18.135907] e1000e 0000:02:00.0: Disabling ASPM L1
[ 18.137262] e1000e 0000:02:00.0: Disabling ASPM L0s
but I see the same high ping latencies.
> I can understand the latency with the OpenSUSE 2.6.34-based kernels
> assuming commit 19833b5dff is not present, but I do not understand
> the latency with 2.6.36-rc3.
The first thing I tried was OpenSUSE 2.6.34 plus 19833b5dff. This led me to
think it wasn't related to ASPM so I resorted to a bisect which ended up showing
it was 6f461f6c7c.
Anyways, all of the above is from vanilla 2.6.36-rc3 so lets ignore OpenSUSE
kernels.
http://ftp.suse.com/pub/people/tonyj/82573L/config is the config for .36-rc3
generated using localmodconfig, defaults chosen for all prompts.
http://ftp.suse.com/pub/people/tonyj/82573L/dmesg is the full dmesg
Tony
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: high latency on 82573L
2010-09-03 17:51 ` Tony Jones
@ 2010-09-03 18:59 ` Allan, Bruce W
2010-09-08 18:21 ` Jesse Barnes
0 siblings, 1 reply; 6+ messages in thread
From: Allan, Bruce W @ 2010-09-03 18:59 UTC (permalink / raw)
To: Tony Jones
Cc: Kirsher, Jeffrey T, Brandeburg, Jesse, Duyck, Alexander H,
Waskiewicz Jr, Peter P, Ronciak, John, e1000-devel, linux-kernel,
bphilips, linux-pci, jbarnes
On Friday, September 03, 2010 10:51 AM, Tony Jones wrote:
> On Thu, Sep 02, 2010 at 11:49:12AM -0700, Allan, Bruce W wrote:
>> Please provide more verbose lspci output and include the PCI config
>> space, i.e. 'lspci -s 2:0.0 -vvv -xxx' after the driver is loaded,
>
> # lspci -s 2:0.0 -vvv -xxx
> 02:00.0 Ethernet controller: Intel Corporation 82573L Gigabit
> Ethernet Controller Subsystem: Lenovo ThinkPad T60
> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B-
> ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ
> 46 Region 0: Memory at ee000000 (32-bit, non-prefetchable)
> [size=128K] Region 2: I/O ports at 3000 [size=32]
> Capabilities: [c8] Power Management version 2
> Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable-
> DSel=0 DScale=1 PME- Capabilities: [d0] MSI: Enable+ Count=1/1
> Maskable- 64bit+ Address: 00000000fee0100c Data: 41c9
> Capabilities: [e0] Express (v1) Endpoint, MSI 00
> DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1
> <64us ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
> DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
> RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
> MaxPayload 128 bytes, MaxReadReq 512 bytes
> DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr+ TransPend-
> LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0
> <128ns, L1 <64us ClockPM+ Surprise- LLActRep- BwNot-
> LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+
> ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
> LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive-
> BWMgmt- ABWMgmt- Capabilities: [100 v1] Advanced Error Reporting
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF-
> MalfTLP- ECRC- UnsupReq+ ACSViol- UEMsk: DLP- SDES- TLP- FCP-
> CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq-
> ACSViol- UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr+ BadTLP+
> BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP-
> BadDLLP- Rollover- Timeout- NonFatalErr- AERCap: First Error
> Pointer: 14, GenCap- CGenEn- ChkCap- ChkEn- Capabilities: [140 v1]
> Device Serial Number 00-1a-6b-ff-ff-6c-7e-a4 Kernel driver in use:
> e1000e 00: 86 80 9a 10 07 05 10 00 00 00 00 02 10 00 00 00 10: 00 00
> 00 ee 00 00 00 00 01 30 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00
> 00 00 00 00 aa 17 01 20 30: 00 00 00 00 c8 00 00 00 00 00 00 00 0b 01
> 00 00 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 01 d0 22 c8 00 20 00 0f
> d0: 05 e0 81 00 0c 10 e0 fe 00 00 00 00 c9 41 00 00
> e0: 10 00 01 00 c1 0c 00 00 1f 28 1a 00 11 1c 07 00
> f0: 42 01 11 10 00 00 00 00 00 00 00 00 00 00 00 00
>
>> kernel. Are there any messages in the system log regarding disabling
>> ASPM L0s and/or L1 on that device?
>
> It would appear it is being disabled:
>
> [ 0.194271] ACPI FADT declares the system doesn't support PCIe
> ASPM, so disable it [ 0.297112] pci 0000:01:00.0: disabling ASPM
> on pre-1.1 PCIe device. You can enable it with 'pcie_aspm=force' [
> 0.298003] pci 0000:02:00.0: disabling ASPM on pre-1.1 PCIe device.
> You can enable it with 'pcie_aspm=force' [ 0.299123] pci
> 0000:03:00.0: disabling ASPM on pre-1.1 PCIe device. You can enable
> it with 'pcie_aspm=force' [ 18.135907] e1000e 0000:02:00.0:
> Disabling ASPM L1 [ 18.137262] e1000e 0000:02:00.0: Disabling ASPM
> L0s
>
> but I see the same high ping latencies.
>
>> I can understand the latency with the OpenSUSE 2.6.34-based kernels
>> assuming commit 19833b5dff is not present, but I do not understand
>> the latency with 2.6.36-rc3.
>
> The first thing I tried was OpenSUSE 2.6.34 plus 19833b5dff. This
> led me to
> think it wasn't related to ASPM so I resorted to a bisect which ended
> up showing
> it was 6f461f6c7c.
>
> Anyways, all of the above is from vanilla 2.6.36-rc3 so lets ignore
> OpenSUSE
> kernels.
>
> http://ftp.suse.com/pub/people/tonyj/82573L/config is the config for
> .36-rc3
> generated using localmodconfig, defaults chosen for all prompts.
>
> http://ftp.suse.com/pub/people/tonyj/82573L/dmesg is the full dmesg
>
> Tony
ASPM L1 must be disabled on this device otherwise the latency described
above will happen. And even though there are log messages indicating
ASPM L1 is disabled, it really isn't according to the verbose lspci
output and PCI config space for the 2:0.0 device (see LnkCtl above).
Since CONFIG_PCIEASPM is enabled in your kernel config, the driver is
calling the kernel function pci_disable_link_state() to disable ASPM L1
which it fails to do because the variable aspm_disabled=1 (as indicated
by the "ACPI FADT declares the system doesn't support PCIe ASPM, so
disable it" message).
I'm unclear on whether the aspm_disabled variable is meant to indicate
ASPM L0s or both ASPM L0s _and_ L1 are disabled (added PCI maintainer
and linux-pci mail-list). To resolve this issue, we need to either a)
change e1000e to directly write the PCI config space to disable ASPM L1
as was done before 6f461f6c7c, or b) fix pci_disable_link_state() et. al.
to allow for ASPM L1 to be disabled properly. I would prefer the latter
option so that other drivers do not have to use the same kludge to write
to the PCI config space. Any input from the PCI guys?
Alternatively in the meantime, if you disable CONFIG_PCIEASPM the e100e
driver will act how it did before 6f461f6c7c, i.e. it will directly write
the PCI config space to disable ASPM L1.
Thanks,
Bruce.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: high latency on 82573L
2010-09-03 18:59 ` Allan, Bruce W
@ 2010-09-08 18:21 ` Jesse Barnes
2010-09-20 22:31 ` Allan, Bruce W
0 siblings, 1 reply; 6+ messages in thread
From: Jesse Barnes @ 2010-09-08 18:21 UTC (permalink / raw)
To: Allan, Bruce W
Cc: Tony Jones, Kirsher, Jeffrey T, Brandeburg, Jesse, Duyck,
Alexander H, Waskiewicz Jr, Peter P, Ronciak, John, e1000-devel,
linux-kernel, bphilips, linux-pci
On Fri, 3 Sep 2010 11:59:30 -0700
"Allan, Bruce W" <bruce.w.allan@intel.com> wrote:
> On Friday, September 03, 2010 10:51 AM, Tony Jones wrote:
> > On Thu, Sep 02, 2010 at 11:49:12AM -0700, Allan, Bruce W wrote:
> >> Please provide more verbose lspci output and include the PCI config
> >> space, i.e. 'lspci -s 2:0.0 -vvv -xxx' after the driver is loaded,
> >
> > # lspci -s 2:0.0 -vvv -xxx
> > 02:00.0 Ethernet controller: Intel Corporation 82573L Gigabit
> > Ethernet Controller Subsystem: Lenovo ThinkPad T60
> > Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> > Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B-
> > ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> > Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ
> > 46 Region 0: Memory at ee000000 (32-bit, non-prefetchable)
> > [size=128K] Region 2: I/O ports at 3000 [size=32]
> > Capabilities: [c8] Power Management version 2
> > Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
> > PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable-
> > DSel=0 DScale=1 PME- Capabilities: [d0] MSI: Enable+ Count=1/1
> > Maskable- 64bit+ Address: 00000000fee0100c Data: 41c9
> > Capabilities: [e0] Express (v1) Endpoint, MSI 00
> > DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1
> > <64us ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
> > DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
> > RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
> > MaxPayload 128 bytes, MaxReadReq 512 bytes
> > DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr+ TransPend-
> > LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0
> > <128ns, L1 <64us ClockPM+ Surprise- LLActRep- BwNot-
> > LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+
> > ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
> > LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive-
> > BWMgmt- ABWMgmt- Capabilities: [100 v1] Advanced Error Reporting
> > UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF-
> > MalfTLP- ECRC- UnsupReq+ ACSViol- UEMsk: DLP- SDES- TLP- FCP-
> > CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq-
> > ACSViol- UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> > RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr+ BadTLP+
> > BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP-
> > BadDLLP- Rollover- Timeout- NonFatalErr- AERCap: First Error
> > Pointer: 14, GenCap- CGenEn- ChkCap- ChkEn- Capabilities: [140 v1]
> > Device Serial Number 00-1a-6b-ff-ff-6c-7e-a4 Kernel driver in use:
> > e1000e 00: 86 80 9a 10 07 05 10 00 00 00 00 02 10 00 00 00 10: 00 00
> > 00 ee 00 00 00 00 01 30 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00
> > 00 00 00 00 aa 17 01 20 30: 00 00 00 00 c8 00 00 00 00 00 00 00 0b 01
> > 00 00 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > c0: 00 00 00 00 00 00 00 00 01 d0 22 c8 00 20 00 0f
> > d0: 05 e0 81 00 0c 10 e0 fe 00 00 00 00 c9 41 00 00
> > e0: 10 00 01 00 c1 0c 00 00 1f 28 1a 00 11 1c 07 00
> > f0: 42 01 11 10 00 00 00 00 00 00 00 00 00 00 00 00
> >
> >> kernel. Are there any messages in the system log regarding disabling
> >> ASPM L0s and/or L1 on that device?
> >
> > It would appear it is being disabled:
> >
> > [ 0.194271] ACPI FADT declares the system doesn't support PCIe
> > ASPM, so disable it [ 0.297112] pci 0000:01:00.0: disabling ASPM
> > on pre-1.1 PCIe device. You can enable it with 'pcie_aspm=force' [
> > 0.298003] pci 0000:02:00.0: disabling ASPM on pre-1.1 PCIe device.
> > You can enable it with 'pcie_aspm=force' [ 0.299123] pci
> > 0000:03:00.0: disabling ASPM on pre-1.1 PCIe device. You can enable
> > it with 'pcie_aspm=force' [ 18.135907] e1000e 0000:02:00.0:
> > Disabling ASPM L1 [ 18.137262] e1000e 0000:02:00.0: Disabling ASPM
> > L0s
> >
> > but I see the same high ping latencies.
> >
> >> I can understand the latency with the OpenSUSE 2.6.34-based kernels
> >> assuming commit 19833b5dff is not present, but I do not understand
> >> the latency with 2.6.36-rc3.
> >
> > The first thing I tried was OpenSUSE 2.6.34 plus 19833b5dff. This
> > led me to
> > think it wasn't related to ASPM so I resorted to a bisect which ended
> > up showing
> > it was 6f461f6c7c.
> >
> > Anyways, all of the above is from vanilla 2.6.36-rc3 so lets ignore
> > OpenSUSE
> > kernels.
> >
> > http://ftp.suse.com/pub/people/tonyj/82573L/config is the config for
> > .36-rc3
> > generated using localmodconfig, defaults chosen for all prompts.
> >
> > http://ftp.suse.com/pub/people/tonyj/82573L/dmesg is the full dmesg
> >
> > Tony
>
> ASPM L1 must be disabled on this device otherwise the latency described
> above will happen. And even though there are log messages indicating
> ASPM L1 is disabled, it really isn't according to the verbose lspci
> output and PCI config space for the 2:0.0 device (see LnkCtl above).
> Since CONFIG_PCIEASPM is enabled in your kernel config, the driver is
> calling the kernel function pci_disable_link_state() to disable ASPM L1
> which it fails to do because the variable aspm_disabled=1 (as indicated
> by the "ACPI FADT declares the system doesn't support PCIe ASPM, so
> disable it" message).
>
> I'm unclear on whether the aspm_disabled variable is meant to indicate
> ASPM L0s or both ASPM L0s _and_ L1 are disabled (added PCI maintainer
> and linux-pci mail-list). To resolve this issue, we need to either a)
> change e1000e to directly write the PCI config space to disable ASPM L1
> as was done before 6f461f6c7c, or b) fix pci_disable_link_state() et. al.
> to allow for ASPM L1 to be disabled properly. I would prefer the latter
> option so that other drivers do not have to use the same kludge to write
> to the PCI config space. Any input from the PCI guys?
Yeah, I'd prefer this code to be in the core. Are there any patches
available yet?
--
Jesse Barnes, Intel Open Source Technology Center
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: high latency on 82573L
2010-09-08 18:21 ` Jesse Barnes
@ 2010-09-20 22:31 ` Allan, Bruce W
0 siblings, 0 replies; 6+ messages in thread
From: Allan, Bruce W @ 2010-09-20 22:31 UTC (permalink / raw)
To: Jesse Barnes
Cc: Tony Jones, Kirsher, Jeffrey T, Brandeburg, Jesse, Duyck,
Alexander H, Waskiewicz Jr, Peter P, Ronciak, John, e1000-devel,
linux-kernel, bphilips, linux-pci
On Wednesday, September 08, 2010 11:21 AM, Jesse Barnes wrote:
> On Fri, 3 Sep 2010 11:59:30 -0700
> "Allan, Bruce W" <bruce.w.allan@intel.com> wrote:
>
>> ASPM L1 must be disabled on this device otherwise the latency
>> described above will happen. And even though there are log messages
>> indicating ASPM L1 is disabled, it really isn't according to the
>> verbose lspci output and PCI config space for the 2:0.0 device (see
>> LnkCtl above). Since CONFIG_PCIEASPM is enabled in your kernel
>> config, the driver is calling the kernel function
>> pci_disable_link_state() to disable ASPM L1 which it fails to do
>> because the variable aspm_disabled=1 (as indicated by the "ACPI FADT
>> declares the system doesn't support PCIe ASPM, so disable it"
>> message).
>>
>> I'm unclear on whether the aspm_disabled variable is meant to
>> indicate ASPM L0s or both ASPM L0s _and_ L1 are disabled (added PCI
>> maintainer and linux-pci mail-list). To resolve this issue, we need
>> to either a) change e1000e to directly write the PCI config space to
>> disable ASPM L1 as was done before 6f461f6c7c, or b) fix
>> pci_disable_link_state() et. al. to allow for ASPM L1 to be disabled
>> properly. I would prefer the latter option so that other drivers do
>> not have to use the same kludge to write to the PCI config space.
>> Any input from the PCI guys?
>
> Yeah, I'd prefer this code to be in the core. Are there any patches
> available yet?
Nothing from me at this point (been on vacation). I might be able to
look into this further but not for another week or so (need to catch
up on the backlog that piled up while away).
Bruce.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2010-09-20 22:31 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2010-09-02 3:39 high latency on 82573L Tony Jones
2010-09-02 18:49 ` Allan, Bruce W
2010-09-03 17:51 ` Tony Jones
2010-09-03 18:59 ` Allan, Bruce W
2010-09-08 18:21 ` Jesse Barnes
2010-09-20 22:31 ` Allan, Bruce W
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.