* crash/dmesg - RAID locking up?
@ 2010-03-28 22:31 Mark Knecht
[not found] ` <4BB0267E.8080605@stud.tu-ilmenau.de>
0 siblings, 1 reply; 2+ messages in thread
From: Mark Knecht @ 2010-03-28 22:31 UTC (permalink / raw)
To: Linux-RAID
So I'm giving up on mdadm/RAID1/WD10EARS Green Drive experiment for
the next day or two. I've tried 4 times in the last two days to see if
I can get this machine to boot using RAID. I keep running into crashes
that look like the one below. I will try a non-RAID install and see if
the hardware is stable and then come back to this later if
appropriate.
I was running a March, 2010 Gentoo install CD when this happened. One
terminal doing bash completion of all things was hung with 1 CPU stuck
in a 100% wait condition. New terminals continued to work if I didn't
try to access the hard drive. I can run things from the CD.
I have never seen a Gentoo install use nearly 2GB so something is
really going wrong here.
Cheers,
Mark
livecd ~ # cat /proc/mdstat
Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md3 : active raid1 sdc3[2] sdb3[1] sda3[0]
31463232 blocks [3/3] [UUU]
unused devices: <none>
livecd ~ #
livecd ~ # top
top - 06:24:39 up 1:28, 3 users, load average: 1.95, 2.52, 2.36
Tasks: 182 total, 1 running, 181 sleeping, 0 stopped, 0 zombie
Cpu0 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu4 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu5 : 0.0%us, 0.0%sy, 0.0%ni, 0.0%id,100.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 6105068k total, 1822336k used, 4282732k free, 172016k buffers
Swap: 12602964k total, 0k used, 12602964k free, 1326360k cached
livecd ~ # uname -a
Linux livecd 2.6.31-gentoo-r6 #1 SMP Thu Mar 4 13:09:55 UTC 2010
x86_64 Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz GenuineIntel GNU/Linux
livecd ~ #
Adding 4200988k swap on /dev/sda2. Priority:1 extents:1 across:4200988k
Adding 4200988k swap on /dev/sdb2. Priority:1 extents:1 across:4200988k
Adding 4200988k swap on /dev/sdc2. Priority:1 extents:1 across:4200988k
kjournald starting. Commit interval 5 seconds
EXT3 FS on md3, internal journal
EXT3-fs: mounted filesystem with writeback data mode.
INFO: task bash:17348 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
bash D ffff8800280d3e00 0 17348 17345 0x00000000
ffff8801afb1a670 0000000000000082 0000000000000000 00000000000000ff
ffff8801a788f038 0000000000004000 000000000000fe00 000000000000c878
ffff8801af1e39b8 ffff8801af1e39d0 ffff8801af91c730 ffff8801afb1a8e8
Call Trace:
[<ffffffff81209a92>] ? generic_make_request+0x220/0x26a
[<ffffffff8109c248>] ? sync_buffer+0x0/0x40
[<ffffffff8137a4fc>] ? io_schedule+0x2d/0x3a
[<ffffffff8109c283>] ? sync_buffer+0x3b/0x40
[<ffffffff8137a879>] ? __wait_on_bit+0x41/0x70
[<ffffffff8109c248>] ? sync_buffer+0x0/0x40
[<ffffffff8137a913>] ? out_of_line_wait_on_bit+0x6b/0x77
[<ffffffff810438b2>] ? wake_bit_function+0x0/0x23
[<ffffffff81132dff>] ? log_do_checkpoint+0x366/0x429
[<ffffffff81132f92>] ? __log_wait_for_space+0xd0/0x176
[<ffffffff81130e8e>] ? start_this_handle+0x2b2/0x2fb
[<ffffffff8108a698>] ? filldir+0x0/0xb7
[<ffffffff8108a698>] ? filldir+0x0/0xb7
[<ffffffff8113102d>] ? journal_start+0x9a/0xce
[<ffffffff810f3ae2>] ? ext3_dirty_inode+0x28/0x7b
[<ffffffff8109745d>] ? __mark_inode_dirty+0x26/0xf5
[<ffffffff8108ebca>] ? touch_atime+0x10a/0x113
[<ffffffff8108a85c>] ? vfs_readdir+0x7a/0x97
[<ffffffff8108a9b1>] ? sys_getdents+0x7a/0xc1
[<ffffffff8100adab>] ? system_call_fastpath+0x16/0x1b
INFO: task kjournald:17466 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kjournald D ffff8800280bbe00 0 17466 2 0x00000000
ffff8801adf9d890 0000000000000046 0000000000000000 0000000000000000
ffff8801adcbde44 0000000000004000 000000000000fe00 000000000000c878
0000000800000050 ffff88017a99aa40 ffff8801af90a150 ffff8801adf9db08
Call Trace:
[<ffffffff812dd063>] ? md_make_request+0xb6/0xf1
[<ffffffff8109c248>] ? sync_buffer+0x0/0x40
[<ffffffff8137a4fc>] ? io_schedule+0x2d/0x3a
[<ffffffff8109c283>] ? sync_buffer+0x3b/0x40
[<ffffffff8137a879>] ? __wait_on_bit+0x41/0x70
[<ffffffff8109c248>] ? sync_buffer+0x0/0x40
[<ffffffff8137a913>] ? out_of_line_wait_on_bit+0x6b/0x77
[<ffffffff810438b2>] ? wake_bit_function+0x0/0x23
[<ffffffff8109c637>] ? sync_dirty_buffer+0x72/0xaa
[<ffffffff81131b8e>] ? journal_commit_transaction+0xa74/0xde2
[<ffffffff8103abcc>] ? lock_timer_base+0x26/0x4b
[<ffffffff81043884>] ? autoremove_wake_function+0x0/0x2e
[<ffffffff81134804>] ? kjournald+0xe3/0x206
[<ffffffff81043884>] ? autoremove_wake_function+0x0/0x2e
[<ffffffff81134721>] ? kjournald+0x0/0x206
[<ffffffff81043591>] ? kthread+0x8b/0x93
[<ffffffff8100bd3a>] ? child_rip+0xa/0x20
[<ffffffff81043506>] ? kthread+0x0/0x93
[<ffffffff8100bd30>] ? child_rip+0x0/0x20
livecd ~ #
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: crash/dmesg - RAID locking up?
[not found] ` <4BB0267E.8080605@stud.tu-ilmenau.de>
@ 2010-03-29 12:44 ` Mark Knecht
0 siblings, 0 replies; 2+ messages in thread
From: Mark Knecht @ 2010-03-29 12:44 UTC (permalink / raw)
To: st0ff; +Cc: Linux-RAID
On Sun, Mar 28, 2010 at 9:03 PM, Stefan /*St0fF*/ Hübner
<stefan.huebner@stud.tu-ilmenau.de> wrote:
> Am 29.03.2010 00:31, schrieb Mark Knecht:
>> So I'm giving up on mdadm/RAID1/WD10EARS Green Drive experiment for
>> the next day or two. I've tried 4 times in the last two days to see if
>> I can get this machine to boot using RAID. I keep running into crashes
>> that look like the one below. I will try a non-RAID install and see if
>> the hardware is stable and then come back to this later if
>> appropriate.
>>
>> I was running a March, 2010 Gentoo install CD when this happened. One
>> terminal doing bash completion of all things was hung with 1 CPU stuck
>> in a 100% wait condition. New terminals continued to work if I didn't
>> try to access the hard drive. I can run things from the CD.
>
> Hi Mark, that's been the desktop-nature of the disks. Please run
> smartctl -A /dev/sdX on those disks and see if any have _raw_ values of
> "pending_sector_count" larger than zero.
> If so, let the disks get exchanged by your vendor, stating: "data loss
> already occurred" or "computer freezes with this disk connected".
> They'll (at least WD) exchange the disk without questions.
>
> To clarify: it seems to me that at least one disk has bad magnetization
> on at least one platter, causing it to enter error correction code all
> the time and by that not responding...
>
<SNIP>
None of the items that look like errors to me have counts greater than
0 so far. I do not see "pending_sector_Count" reported by these
drives.
- Mark
keeper ~ # smartctl -A /dev/sda
smartctl 5.39.1 2010-01-28 r3054 [x86_64-pc-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
Always - 0
3 Spin_Up_Time 0x0027 136 134 021 Pre-fail
Always - 6175
4 Start_Stop_Count 0x0032 100 100 000 Old_age
Always - 13
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age
Always - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age
Always - 21
10 Spin_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age
Always - 11
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
Always - 10
193 Load_Cycle_Count 0x0032 200 200 000 Old_age
Always - 345
194 Temperature_Celsius 0x0022 125 102 000 Old_age
Always - 22
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age
Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
Offline - 0
keeper ~ # smartctl -A /dev/sdb
smartctl 5.39.1 2010-01-28 r3054 [x86_64-pc-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
Always - 0
3 Spin_Up_Time 0x0027 134 130 021 Pre-fail
Always - 6258
4 Start_Stop_Count 0x0032 100 100 000 Old_age
Always - 14
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age
Always - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age
Always - 21
10 Spin_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age
Always - 12
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
Always - 11
193 Load_Cycle_Count 0x0032 200 200 000 Old_age
Always - 203
194 Temperature_Celsius 0x0022 125 098 000 Old_age
Always - 22
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age
Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
Offline - 0
keeper ~ # smartctl -A /dev/sdc
smartctl 5.39.1 2010-01-28 r3054 [x86_64-pc-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
Always - 0
3 Spin_Up_Time 0x0027 131 128 021 Pre-fail
Always - 6416
4 Start_Stop_Count 0x0032 100 100 000 Old_age
Always - 14
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age
Always - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age
Always - 21
10 Spin_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age
Always - 12
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
Always - 11
193 Load_Cycle_Count 0x0032 200 200 000 Old_age
Always - 199
194 Temperature_Celsius 0x0022 125 099 000 Old_age
Always - 22
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age
Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
Offline - 0
keeper ~ #
I ran the short selftest in the devices and there were no errors
reported. I've started the long selftest but that takes 4 hours to
complete so I'll have to report on that later.
keeper ~ # smartctl -l error /dev/sda
smartctl 5.39.1 2010-01-28 r3054 [x86_64-pc-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF READ SMART DATA SECTION ===
SMART Error Log Version: 1
No Errors Logged
keeper ~ # smartctl -l error /dev/sdb
smartctl 5.39.1 2010-01-28 r3054 [x86_64-pc-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF READ SMART DATA SECTION ===
SMART Error Log Version: 1
No Errors Logged
keeper ~ # smartctl -l error /dev/sdc
smartctl 5.39.1 2010-01-28 r3054 [x86_64-pc-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF READ SMART DATA SECTION ===
SMART Error Log Version: 1
No Errors Logged
keeper ~ #
Thanks,
Mark
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2010-03-29 12:44 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2010-03-28 22:31 crash/dmesg - RAID locking up? Mark Knecht
[not found] ` <4BB0267E.8080605@stud.tu-ilmenau.de>
2010-03-29 12:44 ` Mark Knecht
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.