* 100%wa for long periods of time
@ 2010-03-29 3:44 Mark Knecht
2010-03-29 4:26 ` andrew hendry
0 siblings, 1 reply; 3+ messages in thread
From: Mark Knecht @ 2010-03-29 3:44 UTC (permalink / raw)
To: Linux Kernel List
[-- Attachment #1: Type: text/plain, Size: 5740 bytes --]
Hi all,
Long time lurker. Seldom post here. I'm not a programmer.
OK, so I'm bring up new hardware - Intel i7 920, Intel DX58SO
motherboard, 6GB Crucial triple channel DRAM. Using both the Gentoo
install CD (2.6.32 based) as well as 2.6.33-gentoo after booting the
install I am running into long periods of time - sometimes 30 seconds,
sometimes lasting minutes, where the machine appears to hang or slow
down very drastically. The only thing I've noticed so far is that
there are 100% waiting periods showing up in top. No messages in dmesg
or /var/log/messages. Just long periods where the the compiler acts
like the system has really slowed down and sometimes appears almost
completely dead.
When this clears up then the system screams along building
software. When it happens is see things like the following where CPU's
1 & 5 sit at 100% waiting which lasted 2 or 3 minutes before clearing
up. The CPU that goes to 100% does move around but it does seem to
land on #5 more than others.
keeper ~ # top
top - 02:27:17 up 28 min, 2 users, load average: 2.76, 1.95, 1.30
Tasks: 125 total, 1 running, 124 sleeping, 0 stopped, 0 zombie
Cpu0 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 0.0%us, 0.0%sy, 0.0%ni, 0.0%id,100.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu4 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu5 : 0.0%us, 0.3%sy, 0.0%ni, 0.0%id, 99.7%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7 : 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 6107024k total, 1448676k used, 4658348k free, 187492k buffers
Swap: 4200988k total, 0k used, 4200988k free, 915900k cached
keeper ~ # lspci
00:00.0 Host bridge: Intel Corporation X58 I/O Hub to ESI Port (rev 13)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI
Express Root Port 1 (rev 13)
00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI
Express Root Port 3 (rev 13)
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI
Express Root Port 7 (rev 13)
00:10.0 PIC: Intel Corporation 5520/5500/X58 Physical and Link Layer
Registers Port 0 (rev 13)
00:10.1 PIC: Intel Corporation 5520/5500/X58 Routing and Protocol
Layer Registers Port 0 (rev 13)
00:14.0 PIC: Intel Corporation 5520/5500/X58 I/O Hub System Management
Registers (rev 13)
00:14.1 PIC: Intel Corporation 5520/5500/X58 I/O Hub GPIO and Scratch
Pad Registers (rev 13)
00:14.2 PIC: Intel Corporation 5520/5500/X58 I/O Hub Control Status
and RAS Registers (rev 13)
00:14.3 PIC: Intel Corporation 5520/5500/X58 I/O Hub Throttle Registers (rev 13)
00:19.0 Ethernet controller: Intel Corporation 82567LM-2 Gigabit
Network Connection
00:1a.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB
UHCI Controller #4
00:1a.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB
UHCI Controller #5
00:1a.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB
UHCI Controller #6
00:1a.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2
EHCI Controller #2
00:1b.0 Audio device: Intel Corporation 82801JI (ICH10 Family) HD
Audio Controller
00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Port 1
00:1c.1 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Port 2
00:1c.4 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Port 5
00:1d.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB
UHCI Controller #1
00:1d.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB
UHCI Controller #2
00:1d.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB
UHCI Controller #3
00:1d.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2
EHCI Controller #1
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10 Family) SATA
AHCI Controller
00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller
02:00.0 VGA compatible controller: nVidia Corporation G96 [GeForce
9500 GT] (rev a1)
06:00.0 IDE interface: Marvell Technology Group Ltd. 88SE6121 SATA II
Controller (rev b2)
07:03.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22/A
IEEE-1394a-2000 Controller (PHY/Link)
keeper ~ #
keeper ~ # uname -a
Linux keeper 2.6.33-gentoo #1 SMP PREEMPT Mon Mar 29 01:12:08 PDT 2010
x86_64 Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz GenuineIntel GNU/Linux
keeper ~ #
keeper ~ # lsmod
Module Size Used by
ipv6 207757 24
usbhid 21529 0
snd_hda_codec_realtek 239530 1
uhci_hcd 18047 0
ehci_hcd 30854 0
usbcore 115023 4 usbhid,uhci_hcd,ehci_hcd
sg 23029 0
snd_hda_intel 17688 0
snd_hda_codec 45755 2 snd_hda_codec_realtek,snd_hda_intel
firewire_ohci 20022 0
e1000e 111701 0
snd_pcm 58104 2 snd_hda_intel,snd_hda_codec
processor 23121 0
rtc_cmos 7678 0
snd_timer 15030 1 snd_pcm
rtc_core 11093 1 rtc_cmos
snd 37476 5
snd_hda_codec_realtek,snd_hda_intel,snd_hda_codec,snd_pcm,snd_timer
soundcore 800 1 snd
snd_page_alloc 5809 2 snd_hda_intel,snd_pcm
firewire_core 36109 1 firewire_ohci
rtc_lib 1617 1 rtc_core
thermal 11650 0
keeper ~ #
Thanks,
Mark
[-- Attachment #2: config.gz --]
[-- Type: application/x-gzip, Size: 14854 bytes --]
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: 100%wa for long periods of time
2010-03-29 3:44 100%wa for long periods of time Mark Knecht
@ 2010-03-29 4:26 ` andrew hendry
2010-03-29 13:16 ` Mark Knecht
0 siblings, 1 reply; 3+ messages in thread
From: andrew hendry @ 2010-03-29 4:26 UTC (permalink / raw)
To: Mark Knecht; +Cc: Linux Kernel List
Hi,
Do you know what is causing the wait?
Probably not the best way, but I have used something like the
following for a similar case before:
echo "1" > /proc/sys/vm/block_dump
... wait a short while ...
echo "0" > /proc/sys/vm/block_dump
Then have a look at dmesg | egrep "READ|WRITE|dirtied"
Andrew.
On Mon, Mar 29, 2010 at 2:44 PM, Mark Knecht <markknecht@gmail.com> wrote:
> Hi all,
> Long time lurker. Seldom post here. I'm not a programmer.
>
> OK, so I'm bring up new hardware - Intel i7 920, Intel DX58SO
> motherboard, 6GB Crucial triple channel DRAM. Using both the Gentoo
> install CD (2.6.32 based) as well as 2.6.33-gentoo after booting the
> install I am running into long periods of time - sometimes 30 seconds,
> sometimes lasting minutes, where the machine appears to hang or slow
> down very drastically. The only thing I've noticed so far is that
> there are 100% waiting periods showing up in top. No messages in dmesg
> or /var/log/messages. Just long periods where the the compiler acts
> like the system has really slowed down and sometimes appears almost
> completely dead.
>
> When this clears up then the system screams along building
> software. When it happens is see things like the following where CPU's
> 1 & 5 sit at 100% waiting which lasted 2 or 3 minutes before clearing
> up. The CPU that goes to 100% does move around but it does seem to
> land on #5 more than others.
>
> keeper ~ # top
>
> top - 02:27:17 up 28 min, 2 users, load average: 2.76, 1.95, 1.30
> Tasks: 125 total, 1 running, 124 sleeping, 0 stopped, 0 zombie
> Cpu0 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu1 : 0.0%us, 0.0%sy, 0.0%ni, 0.0%id,100.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu2 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu4 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu5 : 0.0%us, 0.3%sy, 0.0%ni, 0.0%id, 99.7%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu6 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu7 : 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Mem: 6107024k total, 1448676k used, 4658348k free, 187492k buffers
> Swap: 4200988k total, 0k used, 4200988k free, 915900k cached
>
>
> keeper ~ # lspci
> 00:00.0 Host bridge: Intel Corporation X58 I/O Hub to ESI Port (rev 13)
> 00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI
> Express Root Port 1 (rev 13)
> 00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI
> Express Root Port 3 (rev 13)
> 00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI
> Express Root Port 7 (rev 13)
> 00:10.0 PIC: Intel Corporation 5520/5500/X58 Physical and Link Layer
> Registers Port 0 (rev 13)
> 00:10.1 PIC: Intel Corporation 5520/5500/X58 Routing and Protocol
> Layer Registers Port 0 (rev 13)
> 00:14.0 PIC: Intel Corporation 5520/5500/X58 I/O Hub System Management
> Registers (rev 13)
> 00:14.1 PIC: Intel Corporation 5520/5500/X58 I/O Hub GPIO and Scratch
> Pad Registers (rev 13)
> 00:14.2 PIC: Intel Corporation 5520/5500/X58 I/O Hub Control Status
> and RAS Registers (rev 13)
> 00:14.3 PIC: Intel Corporation 5520/5500/X58 I/O Hub Throttle Registers (rev 13)
> 00:19.0 Ethernet controller: Intel Corporation 82567LM-2 Gigabit
> Network Connection
> 00:1a.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB
> UHCI Controller #4
> 00:1a.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB
> UHCI Controller #5
> 00:1a.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB
> UHCI Controller #6
> 00:1a.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2
> EHCI Controller #2
> 00:1b.0 Audio device: Intel Corporation 82801JI (ICH10 Family) HD
> Audio Controller
> 00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Port 1
> 00:1c.1 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Port 2
> 00:1c.4 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Port 5
> 00:1d.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB
> UHCI Controller #1
> 00:1d.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB
> UHCI Controller #2
> 00:1d.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB
> UHCI Controller #3
> 00:1d.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2
> EHCI Controller #1
> 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
> 00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
> 00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10 Family) SATA
> AHCI Controller
> 00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller
> 02:00.0 VGA compatible controller: nVidia Corporation G96 [GeForce
> 9500 GT] (rev a1)
> 06:00.0 IDE interface: Marvell Technology Group Ltd. 88SE6121 SATA II
> Controller (rev b2)
> 07:03.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22/A
> IEEE-1394a-2000 Controller (PHY/Link)
> keeper ~ #
>
> keeper ~ # uname -a
> Linux keeper 2.6.33-gentoo #1 SMP PREEMPT Mon Mar 29 01:12:08 PDT 2010
> x86_64 Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz GenuineIntel GNU/Linux
> keeper ~ #
>
> keeper ~ # lsmod
> Module Size Used by
> ipv6 207757 24
> usbhid 21529 0
> snd_hda_codec_realtek 239530 1
> uhci_hcd 18047 0
> ehci_hcd 30854 0
> usbcore 115023 4 usbhid,uhci_hcd,ehci_hcd
> sg 23029 0
> snd_hda_intel 17688 0
> snd_hda_codec 45755 2 snd_hda_codec_realtek,snd_hda_intel
> firewire_ohci 20022 0
> e1000e 111701 0
> snd_pcm 58104 2 snd_hda_intel,snd_hda_codec
> processor 23121 0
> rtc_cmos 7678 0
> snd_timer 15030 1 snd_pcm
> rtc_core 11093 1 rtc_cmos
> snd 37476 5
> snd_hda_codec_realtek,snd_hda_intel,snd_hda_codec,snd_pcm,snd_timer
> soundcore 800 1 snd
> snd_page_alloc 5809 2 snd_hda_intel,snd_pcm
> firewire_core 36109 1 firewire_ohci
> rtc_lib 1617 1 rtc_core
> thermal 11650 0
> keeper ~ #
>
> Thanks,
> Mark
>
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: 100%wa for long periods of time
2010-03-29 4:26 ` andrew hendry
@ 2010-03-29 13:16 ` Mark Knecht
0 siblings, 0 replies; 3+ messages in thread
From: Mark Knecht @ 2010-03-29 13:16 UTC (permalink / raw)
To: andrew hendry; +Cc: Linux Kernel List
On Sun, Mar 28, 2010 at 9:26 PM, andrew hendry <andrew.hendry@gmail.com> wrote:
> Hi,
>
> Do you know what is causing the wait?
> Probably not the best way, but I have used something like the
> following for a similar case before:
> echo "1" > /proc/sys/vm/block_dump
> ... wait a short while ...
> echo "0" > /proc/sys/vm/block_dump
> Then have a look at dmesg | egrep "READ|WRITE|dirtied"
>
> Andrew.
>
> On Mon, Mar 29, 2010 at 2:44 PM, Mark Knecht <markknecht@gmail.com> wrote:
>> Hi all,
>> Long time lurker. Seldom post here. I'm not a programmer.
>>
>> OK, so I'm bring up new hardware - Intel i7 920, Intel DX58SO
>> motherboard, 6GB Crucial triple channel DRAM. Using both the Gentoo
>> install CD (2.6.32 based) as well as 2.6.33-gentoo after booting the
>> install I am running into long periods of time - sometimes 30 seconds,
>> sometimes lasting minutes, where the machine appears to hang or slow
>> down very drastically. The only thing I've noticed so far is that
>> there are 100% waiting periods showing up in top. No messages in dmesg
>> or /var/log/messages. Just long periods where the the compiler acts
>> like the system has really slowed down and sometimes appears almost
>> completely dead.
>>
>> When this clears up then the system screams along building
>> software. When it happens is see things like the following where CPU's
>> 1 & 5 sit at 100% waiting which lasted 2 or 3 minutes before clearing
>> up. The CPU that goes to 100% does move around but it does seem to
>> land on #5 more than others.
<SNIP>
OK, that catches some info. Here I was doing an emerge --sync and went
to 100% for a little while
flush-8:0(3365): WRITE block 33555792 on sda3
flush-8:0(3365): WRITE block 33555800 on sda3
flush-8:0(3365): WRITE block 33701984 on sda3
flush-8:0(3365): WRITE block 33720128 on sda3
flush-8:0(3365): WRITE block 33721496 on sda3
flush-8:0(3365): WRITE block 33816576 on sda3
flush-8:0(3365): WRITE block 33816584 on sda3
flush-8:0(3365): WRITE block 33816592 on sda3
flush-8:0(3365): WRITE block 33817664 on sda3
flush-8:0(3365): WRITE block 33817672 on sda3
flush-8:0(3365): WRITE block 33817680 on sda3
flush-8:0(3365): WRITE block 33817688 on sda3
flush-8:0(3365): WRITE block 33817696 on sda3
flush-8:0(3365): WRITE block 33817728 on sda3
keeper ~ #
When I first started building this machine 2 days ago I wanted to set
up RAID. I was using the Gentoo install CD at the time. In the process
of loading software I caught a couple error messages so I backed away
from RAID at the time. However since those messages report things
being 'blocked' for more than 120 second it's probably related. Again,
this hasn't happened since I disable RAID, but I do want to go back to
RAID as this is a low-end home file server.
File system type is ext3 if it matters.
INFO: task kjournald:17466 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kjournald D ffff8800280bbe00 0 17466 2 0x00000000
ffff8801adf9d890 0000000000000046 0000000000000000 0000000000000000
ffff8801adcbde44 0000000000004000 000000000000fe00 000000000000c878
0000000800000050 ffff88017a99aa40 ffff8801af90a150 ffff8801adf9db08
Call Trace:
[<ffffffff812dd063>] ? md_make_request+0xb6/0xf1
[<ffffffff8109c248>] ? sync_buffer+0x0/0x40
[<ffffffff8137a4fc>] ? io_schedule+0x2d/0x3a
[<ffffffff8109c283>] ? sync_buffer+0x3b/0x40
[<ffffffff8137a879>] ? __wait_on_bit+0x41/0x70
[<ffffffff8109c248>] ? sync_buffer+0x0/0x40
[<ffffffff8137a913>] ? out_of_line_wait_on_bit+0x6b/0x77
[<ffffffff810438b2>] ? wake_bit_function+0x0/0x23
[<ffffffff8109c637>] ? sync_dirty_buffer+0x72/0xaa
[<ffffffff81131b8e>] ? journal_commit_transaction+0xa74/0xde2
[<ffffffff8103abcc>] ? lock_timer_base+0x26/0x4b
[<ffffffff81043884>] ? autoremove_wake_function+0x0/0x2e
[<ffffffff81134804>] ? kjournald+0xe3/0x206
[<ffffffff81043884>] ? autoremove_wake_function+0x0/0x2e
[<ffffffff81134721>] ? kjournald+0x0/0x206
[<ffffffff81043591>] ? kthread+0x8b/0x93
[<ffffffff8100bd3a>] ? child_rip+0xa/0x20
[<ffffffff81043506>] ? kthread+0x0/0x93
[<ffffffff8100bd30>] ? child_rip+0x0/0x20
livecd ~ #
Thanks,
Mark
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2010-03-29 13:16 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2010-03-29 3:44 100%wa for long periods of time Mark Knecht
2010-03-29 4:26 ` andrew hendry
2010-03-29 13:16 ` Mark Knecht
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).