All of lore.kernel.org
 help / color / mirror / Atom feed
* XFS appears to cause strange hang with md raid1 on reboot
@ 2013-01-28 23:28 Tom
  2013-01-29  0:05 ` Eric Sandeen
  2013-01-29 15:18 ` Ben Myers
  0 siblings, 2 replies; 22+ messages in thread
From: Tom @ 2013-01-28 23:28 UTC (permalink / raw)
  To: xfs


Dear XFS folks,

I have been using XFS for many years, starting on IRIX and then on RedHat
7.2, and now on CentOS/RHEL and Ubuntu.  Last time I posted to this
mailing list was 12 years ago.  :-)  I've been a happy customer!

I understand that RedHat does not formally support XFS as a root filesystem
on RHEL.  However, up until now, I've been using it very successfully for
years on both CentOS and Ubuntu.  On CentOS, I've successfully patched
Anaconda since CentOS 5.6 to allow XFS root file system support directly
from Anaconda (on both bare metal and Xen VMs).  Prior to that, I had code
in %post that would simply migrate an ext3 fs to XFS.  And I always run
md raid1 (except with Xen, since I use mirroring on the dom0).  I never use
hardware RAID since I want to keep my provisioning as generic as possible.

I've deployed many servers using XFS this way and it has always been
superior for my workloads....  and superior to ext3, and ext4.

....until CentOS 5.9 came out.  Now any systems that are running the stock
CentOS 5.9 kernel (including 5.X systems upgraded to this kernel) hang
on reboot.  If I downgrade to the 5.8 kernel, the problem is resolved.

I have taken an engineering approach to testing this problem in efforts
to help resolve it.  I filed a bug with CentOS, but it's probably not
going to go anywhere upstream since RedHat probably won't support XFS on
the root filesystem (why I still don't understand, since I fixed the
issues with Anaconda for myself and can Kickstart systems with XFS all
day long).

Therefore I hope anyone here can help.  In fact, I was specifically hoping
to catch Eric Sandeen's attention since this seems like a pretty serious
regression.  It's further aggravated by the fact that RedHat stays behind
with kernel version and backports modern fixes.  I scanned over the
2.6.18-348.el5 (stock 5.9 kernel) changelog, and I see a few suspicious
things, but I'm not sure.

Much more detail is available here (CentOS bug id 0006217) including steps
to reproduce the problem.  Also testing with and without md raid.
http://bugs.centos.org/view.php?id=6217

The one thing I haven't provided is a traceback.  I can provide that if it
would be helpful.

I am not in a big hurry for help, on the contrary I just want to open up
a dialog since perhaps others might be suffering from this.  And I want to
help resolve it if I can.

Any insight is appreciated.

-- Tom


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 22+ messages in thread
* Re: XFS appears to cause strange hang with md raid1 on reboot
  2013-01-28 23:28 XFS appears to cause strange hang with md raid1 on reboot Tom
@ 2013-01-29  0:05 ` Eric Sandeen
  2013-01-29 21:47   ` Tom
  2013-01-29 15:18 ` Ben Myers
  1 sibling, 1 reply; 22+ messages in thread
From: Eric Sandeen @ 2013-01-29  0:05 UTC (permalink / raw)
  To: Tom; +Cc: xfs

On 1/28/13 5:28 PM, Tom wrote:
> 
> Dear XFS folks,
> 
> I have been using XFS for many years, starting on IRIX and then on RedHat
> 7.2, and now on CentOS/RHEL and Ubuntu.  Last time I posted to this
> mailing list was 12 years ago.  :-)  I've been a happy customer!
> 
> I understand that RedHat does not formally support XFS as a root filesystem
> on RHEL.  

That's correct.  However, I have run xfs root on Centos5, and am
currently running xfs root on RHEL6.  On md raid1 in both instances. ;)

> However, up until now, I've been using it very successfully for
> years on both CentOS and Ubuntu.  On CentOS, I've successfully patched
> Anaconda since CentOS 5.6 to allow XFS root file system support directly
> from Anaconda (on both bare metal and Xen VMs).  Prior to that, I had code
> in %post that would simply migrate an ext3 fs to XFS.  And I always run
> md raid1 (except with Xen, since I use mirroring on the dom0).  I never use
> hardware RAID since I want to keep my provisioning as generic as possible.
> 
> I've deployed many servers using XFS this way and it has always been
> superior for my workloads....  and superior to ext3, and ext4.
> 
> ....until CentOS 5.9 came out.  Now any systems that are running the stock
> CentOS 5.9 kernel (including 5.X systems upgraded to this kernel) hang
> on reboot.  If I downgrade to the 5.8 kernel, the problem is resolved.

Just to be absolutely sure, do you have any xfs-kmod or kmod-xfs installed?
If so, remove it.

> I have taken an engineering approach to testing this problem in efforts
> to help resolve it.  I filed a bug with CentOS, but it's probably not
> going to go anywhere upstream since RedHat probably won't support XFS on
> the root filesystem (why I still don't understand, since I fixed the
> issues with Anaconda for myself and can Kickstart systems with XFS all
> day long).

It's for non-technical reasons.

> Therefore I hope anyone here can help.  In fact, I was specifically hoping
> to catch Eric Sandeen's attention since this seems like a pretty serious
> regression.  It's further aggravated by the fact that RedHat stays behind
> with kernel version and backports modern fixes.  I scanned over the
> 2.6.18-348.el5 (stock 5.9 kernel) changelog, and I see a few suspicious
> things, but I'm not sure.
> 
> Much more detail is available here (CentOS bug id 0006217) including steps
> to reproduce the problem.  Also testing with and without md raid.
> http://bugs.centos.org/view.php?id=6217

so it's hanging on the way down I guess?

I see:  "md: md1 switched to read-only mode"

Was that there before?

> The one thing I haven't provided is a traceback.  I can provide that if it
> would be helpful.

of course it would be . . . 

I don't see anything obvious between the two kernels you mention, and I
can't spend a ton of time digging into this, since most of my day is
taken up supporting the RHEL customers who pay my salary, nudge nudge. ;)

I'd look at the kernel changelogs for xfs & md, and see if anything
seems plausible.  Maybe diff the sources & see what changed, etc.

-Eric

> I am not in a big hurry for help, on the contrary I just want to open up
> a dialog since perhaps others might be suffering from this.  And I want to
> help resolve it if I can.
> 
> Any insight is appreciated.
> 
> -- Tom
> 
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 22+ messages in thread
* Re: XFS appears to cause strange hang with md raid1 on reboot
  2013-01-28 23:28 XFS appears to cause strange hang with md raid1 on reboot Tom
  2013-01-29  0:05 ` Eric Sandeen
@ 2013-01-29 15:18 ` Ben Myers
  2013-01-29 21:13   ` Tom
  2013-01-30  3:16   ` Tom
  1 sibling, 2 replies; 22+ messages in thread
From: Ben Myers @ 2013-01-29 15:18 UTC (permalink / raw)
  To: Tom; +Cc: xfs

Hey Tom,

On Mon, Jan 28, 2013 at 06:28:18PM -0500, Tom wrote:
> I have been using XFS for many years, starting on IRIX and then on RedHat
> 7.2, and now on CentOS/RHEL and Ubuntu.  Last time I posted to this
> mailing list was 12 years ago.  :-)  I've been a happy customer!

I'm glad to hear it!  ;)

> Much more detail is available here (CentOS bug id 0006217) including steps
> to reproduce the problem.  Also testing with and without md raid.
> http://bugs.centos.org/view.php?id=6217
> 
> The one thing I haven't provided is a traceback.  I can provide that if it
> would be helpful.

I took a brief look at your report.  I think a traceback would be helpful.  

echo t > /proc/sysrq-trigger should do the trick.  You're in the middle of
unmount so you might need to go serial console to capture the output.  Your
report metions freeze/thaw.  This is an area where we've had deadlocks in the
past, so I wouldn't be surprised if that is what you're hitting.  We'll have to
see.

Thanks,
	Ben

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 22+ messages in thread
* Re: XFS appears to cause strange hang with md raid1 on reboot
  2013-01-29 15:18 ` Ben Myers
@ 2013-01-29 21:13   ` Tom
  2013-01-30  3:16   ` Tom
  1 sibling, 0 replies; 22+ messages in thread
From: Tom @ 2013-01-29 21:13 UTC (permalink / raw)
  To: bpm; +Cc: storm9c1, xfs

In a previous message, Ben Myers wrote:
>
> I took a brief look at your report.  I think a traceback would be
> helpful.
>
> echo t > /proc/sysrq-trigger should do the trick.  You're in the middle
> of unmount so you might need to go serial console to capture the output.
>  Your report metions freeze/thaw.  This is an area where we've had
> deadlocks in the past, so I wouldn't be surprised if that is what you're
> hitting.  We'll have to see.
>

Hi Ben,

Sure.  If I do a reboot, and let it go through the motions, then I let it
hang for more than 120 seconds, sometimes it displays a traceback on it's
own.  If not, I can trigger it from the serial console (though probably
with break-t instead since this happens at the very end of the shutdown
sequence, so no access to the shell).

I'll have access to the test systems tonight and will get a traceback.
I'll also post what is displayed during the shutdown process right before
the hang.

Thanks again for your help.

-- Tom




_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 22+ messages in thread
* Re: XFS appears to cause strange hang with md raid1 on reboot
  2013-01-29  0:05 ` Eric Sandeen
@ 2013-01-29 21:47   ` Tom
  2013-01-29 21:55     ` Eric Sandeen
  0 siblings, 1 reply; 22+ messages in thread
From: Tom @ 2013-01-29 21:47 UTC (permalink / raw)
  To: sandeen; +Cc: xfs

In a previous message, Eric Sandeen wrote:
>
> That's correct.  However, I have run xfs root on Centos5, and am
> currently running xfs root on RHEL6.  On md raid1 in both instances. ;)
>

Eric,

Try it with 5.9 and md raid1.  I bet you won't be running it very
much longer.  ;-)

Unless of course you can't reproduce it.


>
> Just to be absolutely sure, do you have any xfs-kmod or kmod-xfs
> installed? If so, remove it.
>

No kmod.  I only use the kmod on 32-bit platforms.  As you know, it is
not needed on 64-bit.

This system is a freshly kickstarted system with only about 200 packages
installed (minimal base system just for stability testing):

[root@test9][/root]# rpm -qa | fgrep xfs
xfsprogs-2.9.4-1.el5.centos
xfsdump-2.2.46-1.el5.centos
[root@test9][/root]# rpm -qa | fgrep kmod
[root@test9][/root]# uname -a
Linux test9.xxxx.xxx 2.6.18-348.el5 #1 SMP Tue Jan 8 17:53:53 EST 2013 x86_6
4 x86_64 x86_64 GNU/Linux


>> (why I still don't understand,
>> since I fixed the issues with Anaconda for myself and can Kickstart
>> systems with XFS all day long).
>
> It's for non-technical reasons.
>

Hmmm, yeah, well I do understand.  ;-)  Just drives me nuts.


>
> so it's hanging on the way down I guess?
>
> I see:  "md: md1 switched to read-only mode"
>
> Was that there before?

Yes.  It's the very last thing printed when using any newer 5.X system.
Then it's supposed to physically reboot or shutdown.  But this is where it
hangs.  I will reply back to the xfs list (to a message from Ben) later
tonight with some more output from the serial console including a
traceback.

I'll also add these details in the open CentOS bug.


>
> I don't see anything obvious between the two kernels you mention, and I
> can't spend a ton of time digging into this, since most of my day is
> taken up supporting the RHEL customers who pay my salary, nudge nudge.
> ;)

Any help is appreciated.  Finding this bug may possibly help
ferret out another more possibly nefarious bug, which is why I don't
need instant gratification here.  I'm only looking to assist with the
solution, not cause more stress.

The "company" that I work for from 9-5 has an extensive RHEL 5.X and 6.X
deployment with Satellite, channels, full support, the whole nine yards.
They don't color outside the lines as I do -- they use ext3 for the root
filesystem instead.  Their loss.  :-)

So don't worry, said "company" "gives" plenty to the "cause".  ;-)

However, after 5pm, I do unrelated personal work and projects unrelated
to said "company", and one of those things is working with CentOS and
Ubuntu.  Using XFS quite extensively.


>
> I'd look at the kernel changelogs for xfs & md, and see if anything
> seems plausible.  Maybe diff the sources & see what changed, etc.
>

Yeah, I took a half-hearted look already.  But didn't diff any source
code yet.  I saw a freeze/thaw change and a few other md changes that
were suspect.  But haven't had a chance to dig deeper.

Thanks again.

-- Tom


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 22+ messages in thread
* Re: XFS appears to cause strange hang with md raid1 on reboot
  2013-01-29 21:47   ` Tom
@ 2013-01-29 21:55     ` Eric Sandeen
  2013-01-29 22:25       ` Tom
  0 siblings, 1 reply; 22+ messages in thread
From: Eric Sandeen @ 2013-01-29 21:55 UTC (permalink / raw)
  To: Tom; +Cc: xfs

On 1/29/13 3:47 PM, Tom wrote:
> In a previous message, Eric Sandeen wrote:

...

>> I don't see anything obvious between the two kernels you mention, and I
>> can't spend a ton of time digging into this, since most of my day is
>> taken up supporting the RHEL customers who pay my salary, nudge nudge.
>> ;)
> 
> Any help is appreciated.  Finding this bug may possibly help
> ferret out another more possibly nefarious bug, which is why I don't
> need instant gratification here.  I'm only looking to assist with the
> solution, not cause more stress.
> 
> The "company" that I work for from 9-5 has an extensive RHEL 5.X and 6.X
> deployment with Satellite, channels, full support, the whole nine yards.
> They don't color outside the lines as I do -- they use ext3 for the root
> filesystem instead.  Their loss.  :-)
> 
> So don't worry, said "company" "gives" plenty to the "cause".  ;-)
> 
> However, after 5pm, I do unrelated personal work and projects unrelated
> to said "company", and one of those things is working with CentOS and
> Ubuntu.  Using XFS quite extensively.

Sorry for hassling you ;)

>>
>> I'd look at the kernel changelogs for xfs & md, and see if anything
>> seems plausible.  Maybe diff the sources & see what changed, etc.
>>
> 
> Yeah, I took a half-hearted look already.  But didn't diff any source
> code yet.  I saw a freeze/thaw change and a few other md changes that
> were suspect.  But haven't had a chance to dig deeper.

I'd suspect the md changes were the trigger, though maybe it reveals
an xfs problem.  I don't see anything in xfs that changed which should
be causing this.

The backtrace will help, I think.

Thanks,
-Eric

> Thanks again.
> 
> -- Tom
> 
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 22+ messages in thread
* Re: XFS appears to cause strange hang with md raid1 on reboot
  2013-01-29 21:55     ` Eric Sandeen
@ 2013-01-29 22:25       ` Tom
  2013-01-29 22:39         ` Ben Myers
  2013-01-30  8:54         ` Stan Hoeppner
  0 siblings, 2 replies; 22+ messages in thread
From: Tom @ 2013-01-29 22:25 UTC (permalink / raw)
  To: sandeen; +Cc: xfs

In a previous message, Eric Sandeen wrote:
>
> Sorry for hassling you ;)
>

Nah, no problem, I think we all know what it's like in corporate
America........  Besides, when hassled, I hassle right on back -- just to
ensure that I'm taking a full participating role in the love.  :-D

Again, your help is appreciated and I'll be happy to give back as much
as I can.  I've been committed to XFS for 15 years, and RedHat for even
longer (way before CentOS existed).  And Linux longer yet!
For the SGI engineers on this list, I do miss IRIX though...  it will
always have a special place in my heart -- if only that OS and its tools
could have been open sourced...


-- Tom


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 22+ messages in thread
* Re: XFS appears to cause strange hang with md raid1 on reboot
  2013-01-29 22:25       ` Tom
@ 2013-01-29 22:39         ` Ben Myers
  2013-01-30  8:54         ` Stan Hoeppner
  1 sibling, 0 replies; 22+ messages in thread
From: Ben Myers @ 2013-01-29 22:39 UTC (permalink / raw)
  To: Tom; +Cc: sandeen, xfs

On Tue, Jan 29, 2013 at 05:25:26PM -0500, Tom wrote:
> For the SGI engineers on this list, I do miss IRIX though...  it will
> always have a special place in my heart -- if only that OS and its tools
> could have been open sourced...

I'm getting all choked up...

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 22+ messages in thread
* Re: XFS appears to cause strange hang with md raid1 on reboot
  2013-01-29 15:18 ` Ben Myers
  2013-01-29 21:13   ` Tom
@ 2013-01-30  3:16   ` Tom
  2013-01-30 22:51     ` Ben Myers
  2013-01-30 23:46     ` Dave Chinner
  1 sibling, 2 replies; 22+ messages in thread
From: Tom @ 2013-01-30  3:16 UTC (permalink / raw)
  To: xfs


Hi all,

I've update the CentOS bug (http://bugs.centos.org/view.php?id=6217) with
the following information:


-- More detail about the hang including traceback --

Using 5.9 kernel (348) without md raid:
Please stand by while rebooting the system...
md: stopping all md devices.
Synchronizing SCSI cache for disk sda:
Restarting system.
..
machine restart
(reboots normally)


With md raid1:
Unmounting pipe file systems:
Unmounting file systems:
Please stand by while rebooting the system...
md: stopping all md devices.
md: md2 switched to read-only mode.
md: md1 switched to read-only mode.
(hang)

Traceback:
INFO: task reboot:2063 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
reboot        D ffff810037df37e0     0  2063      1                  19
(NOTLB)
 ffff81005890ba08 0000000000000082 ffff81005890ba58 ffff81005beb1ea0
 0000000000000001 0000000000000007 ffff810058d67040 ffff810037df37e0
 000000596dd8a1e6 0000000000003df4 ffff810058d67228 000000008008d76f
Call Trace:
 [<ffffffff8002e4bc>] __wake_up+0x38/0x4f
 [<ffffffff80223bce>] md_write_start+0xf2/0x108
 [<ffffffff800a3bc2>] autoremove_wake_function+0x0/0x2e
 [<ffffffff8000ab62>] get_page_from_freelist+0x380/0x442
 [<ffffffff880b102c>] :raid1:make_request+0x38/0x5d8
 [<ffffffff8001c839>] generic_make_request+0x211/0x228
 [<ffffffff8002389f>] mempool_alloc+0x31/0xe7
 [<ffffffff8001a98f>] vsnprintf+0x5d7/0xb54
 [<ffffffff80033695>] submit_bio+0xe6/0xed
 [<ffffffff8807f801>] :xfs:_xfs_buf_ioapply+0x1f2/0x254
 [<ffffffff8807f89c>] :xfs:xfs_buf_iorequest+0x39/0x64
 [<ffffffff8808386c>] :xfs:xfs_bdstrat_cb+0x36/0x3a
 [<ffffffff8807c0a8>] :xfs:xfs_bwrite+0x5e/0xba
 [<ffffffff88077669>] :xfs:xfs_syncsub+0x119/0x226
 [<ffffffff88084ce2>] :xfs:xfs_fs_sync_super+0x33/0xdd
 [<ffffffff8010aa44>] quota_sync_sb+0x2e/0xf0
 [<ffffffff800e55bd>] __fsync_super+0x1b/0x9e
 [<ffffffff800e578a>] fsync_super+0x9/0x16
 [<ffffffff800e57c1>] fsync_bdev+0x2a/0x3b
 [<ffffffff8014ea59>] invalidate_partition+0x28/0x40
 [<ffffffff802225a8>] do_md_stop+0xa0/0x2ec
 [<ffffffff80224d41>] md_notify_reboot+0x5f/0x120
 [<ffffffff80067565>] notifier_call_chain+0x20/0x32
 [<ffffffff8009de98>] blocking_notifier_call_chain+0x22/0x36
 [<ffffffff8009e220>] kernel_restart_prepare+0x18/0x29
 [<ffffffff8009e280>] kernel_restart+0x9/0x46
 [<ffffffff8009e40a>] sys_reboot+0x146/0x1c7
 [<ffffffff8003b291>] hrtimer_try_to_cancel+0x4a/0x53
 [<ffffffff8005a753>] hrtimer_cancel+0xc/0x16
 [<ffffffff80063cf9>] do_nanosleep+0x47/0x70
 [<ffffffff8005a640>] hrtimer_nanosleep+0x58/0x118
 [<ffffffff800a5b84>] hrtimer_wakeup+0x0/0x22
 [<ffffffff8001e2f2>] sigprocmask+0xb7/0xdb
 [<ffffffff80054fe6>] sys_nanosleep+0x4c/0x62
 [<ffffffff8005d116>] system_call+0x7e/0x83


Filesystem            Size  Used Avail Use% Mounted on
/dev/md3              4.9G  784M  4.2G  16% /
/dev/md2              108M   11M   97M  11% /boot
tmpfs                 689M     0  689M   0% /dev/shm

[root@test9][/root]# swapon -s
Filename                                Type            Size    Used   
Priority
/dev/md1                                partition       2947832 0       -1


[root@test9][/root]# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sdb1[1] sda1[0]
      128384 blocks [2/2] [UU]

md1 : active raid1 sdb2[1] sda2[0]
      2947840 blocks [2/2] [UU]

md3 : active raid1 sdb3[1] sda3[0]
      5116608 blocks [2/2] [UU]

unused devices: <none>



_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 22+ messages in thread
* Re: XFS appears to cause strange hang with md raid1 on reboot
  2013-01-29 22:25       ` Tom
  2013-01-29 22:39         ` Ben Myers
@ 2013-01-30  8:54         ` Stan Hoeppner
  1 sibling, 0 replies; 22+ messages in thread
From: Stan Hoeppner @ 2013-01-30  8:54 UTC (permalink / raw)
  To: xfs

On 1/29/2013 4:25 PM, Tom wrote:

> For the SGI engineers on this list, I do miss IRIX though...  it will
> always have a special place in my heart -- if only that OS and its tools
> could have been open sourced...

Much of the differentiating/unique technology has been.  XFS and its
user space tools obviously.  Lesser known is that Paul Jackson and
others at SGI contributed the basic NUMA scalability code to Linux
during the Linux Scalability Effort in the early 2000s, based heavily on
IRIX concepts.  This included cpumemsets, directly taken from IRIX IIRC,
which evolved into Linux mainline cpusets.

This effort was necessary to get Linux to scale on Altix NUMALink
systems with up to 512 sockets, as well as IBM's Xeon based x440 NUMA
box with up to 16 sockets, HP's cell based Itanium systems w/32 sockets,
and clones of HP's cell design used by the likes of Bull, Unisys, NEC,
and others.  This work later turned out to be of great benefit to the
market at large, as the underpinnings were already in place when AMD
launched Opteron, whose 2/4/8 socket platforms use HT NUMA
interconnects.  Same for Intel when it finally went NUMA with QuickPath.
 This infrastructure also allowed for a relatively easy implementation
of multi-core support on single and multi-socket systems.

AIUI, from articles, not first hand experience,  is that what did not
make it into Linux was the tendency of some SGI engineers to write
bloated inefficient code with unnecessarily large data structures.  This
occurred because IRIX developers were working on 8-32 CPU systems (or
larger) with 8-32GB (or more) of RAM, and weren't concerned with single
processor performance or memory efficiency, as they had massive
resources available.  When the scalability concepts were brought over to
Linux, they were bolted onto the mantra of maximum single processor
performance and small memory footprint, and we got the best of both
worlds, without the previous IRIX overhead.

I recall reading an article from that period which described an early
Linux porting effort to Origin/MIPS.  This was a 16-way machine used as
a development mule to get Linux working on NUMALink, ready for
Altix/Itanium.  It was described that even before any NUMA scalability
work began, with a large number of operations the stock SMP Linux kernel
was much faster than IRIX on this 16 CPU origin box.  "Amazing" is a
description I recall.  It had been assumed Linux would be a dog since it
was optimized for small systems.  It turns out it worked just as well on
the big ones, due to said small system optimizations.

So in some respects it's probably better that IRIX simply was abandoned,
with some of the best parts making it into Linux.

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 22+ messages in thread
* Re: XFS appears to cause strange hang with md raid1 on reboot
  2013-01-30  3:16   ` Tom
@ 2013-01-30 22:51     ` Ben Myers
  2013-01-30 23:46     ` Dave Chinner
  1 sibling, 0 replies; 22+ messages in thread
From: Ben Myers @ 2013-01-30 22:51 UTC (permalink / raw)
  To: Tom; +Cc: xfs

Hi Tom,

On Tue, Jan 29, 2013 at 10:16:20PM -0500, Tom wrote:
> I've update the CentOS bug (http://bugs.centos.org/view.php?id=6217) with
> the following information:
> 
> 
> -- More detail about the hang including traceback --
> 
> Using 5.9 kernel (348) without md raid:
> Please stand by while rebooting the system...
> md: stopping all md devices.
> Synchronizing SCSI cache for disk sda:
> Restarting system.
> ..
> machine restart
> (reboots normally)
> 
> 
> With md raid1:
> Unmounting pipe file systems:
> Unmounting file systems:
> Please stand by while rebooting the system...
> md: stopping all md devices.
> md: md2 switched to read-only mode.
> md: md1 switched to read-only mode.
> (hang)
> 
> Traceback:
> INFO: task reboot:2063 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> reboot        D ffff810037df37e0     0  2063      1                  19
> (NOTLB)
>  ffff81005890ba08 0000000000000082 ffff81005890ba58 ffff81005beb1ea0
>  0000000000000001 0000000000000007 ffff810058d67040 ffff810037df37e0
>  000000596dd8a1e6 0000000000003df4 ffff810058d67228 000000008008d76f
> Call Trace:
>  [<ffffffff8002e4bc>] __wake_up+0x38/0x4f
>  [<ffffffff80223bce>] md_write_start+0xf2/0x108
>  [<ffffffff800a3bc2>] autoremove_wake_function+0x0/0x2e
>  [<ffffffff8000ab62>] get_page_from_freelist+0x380/0x442
>  [<ffffffff880b102c>] :raid1:make_request+0x38/0x5d8
>  [<ffffffff8001c839>] generic_make_request+0x211/0x228
>  [<ffffffff8002389f>] mempool_alloc+0x31/0xe7
>  [<ffffffff8001a98f>] vsnprintf+0x5d7/0xb54
>  [<ffffffff80033695>] submit_bio+0xe6/0xed

I think you should consider copying linux-raid@vger.kernel.org.

Did you have a chance to get the sysrq output?  It would be helpful to see the
other tasks on your machine.

Regards,
	Ben

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 22+ messages in thread
* Re: XFS appears to cause strange hang with md raid1 on reboot
  2013-01-30  3:16   ` Tom
  2013-01-30 22:51     ` Ben Myers
@ 2013-01-30 23:46     ` Dave Chinner
  2013-01-31  2:30       ` Tom
  2013-01-31  7:35       ` Stefan Ring
  1 sibling, 2 replies; 22+ messages in thread
From: Dave Chinner @ 2013-01-30 23:46 UTC (permalink / raw)
  To: Tom; +Cc: xfs

On Tue, Jan 29, 2013 at 10:16:20PM -0500, Tom wrote:
> With md raid1:
> Unmounting pipe file systems:
> Unmounting file systems:

So the filesystems are unmounted....

> Please stand by while rebooting the system...
> md: stopping all md devices.
> md: md2 switched to read-only mode.
> md: md1 switched to read-only mode.
> (hang)
> 
> Traceback:
> INFO: task reboot:2063 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> reboot        D ffff810037df37e0     0  2063      1                  19
> (NOTLB)
>  ffff81005890ba08 0000000000000082 ffff81005890ba58 ffff81005beb1ea0
>  0000000000000001 0000000000000007 ffff810058d67040 ffff810037df37e0
>  000000596dd8a1e6 0000000000003df4 ffff810058d67228 000000008008d76f
> Call Trace:
>  [<ffffffff8002e4bc>] __wake_up+0x38/0x4f
>  [<ffffffff80223bce>] md_write_start+0xf2/0x108
>  [<ffffffff800a3bc2>] autoremove_wake_function+0x0/0x2e
>  [<ffffffff8000ab62>] get_page_from_freelist+0x380/0x442
>  [<ffffffff880b102c>] :raid1:make_request+0x38/0x5d8
>  [<ffffffff8001c839>] generic_make_request+0x211/0x228
>  [<ffffffff8002389f>] mempool_alloc+0x31/0xe7
>  [<ffffffff8001a98f>] vsnprintf+0x5d7/0xb54
>  [<ffffffff80033695>] submit_bio+0xe6/0xed
>  [<ffffffff8807f801>] :xfs:_xfs_buf_ioapply+0x1f2/0x254
>  [<ffffffff8807f89c>] :xfs:xfs_buf_iorequest+0x39/0x64
>  [<ffffffff8808386c>] :xfs:xfs_bdstrat_cb+0x36/0x3a
>  [<ffffffff8807c0a8>] :xfs:xfs_bwrite+0x5e/0xba
>  [<ffffffff88077669>] :xfs:xfs_syncsub+0x119/0x226
>  [<ffffffff88084ce2>] :xfs:xfs_fs_sync_super+0x33/0xdd
>  [<ffffffff8010aa44>] quota_sync_sb+0x2e/0xf0
>  [<ffffffff800e55bd>] __fsync_super+0x1b/0x9e
>  [<ffffffff800e578a>] fsync_super+0x9/0x16
>  [<ffffffff800e57c1>] fsync_bdev+0x2a/0x3b
>  [<ffffffff8014ea59>] invalidate_partition+0x28/0x40
>  [<ffffffff802225a8>] do_md_stop+0xa0/0x2ec

And this says the filesystem is still mounted. Why?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 22+ messages in thread
* Re: XFS appears to cause strange hang with md raid1 on reboot
  2013-01-30 23:46     ` Dave Chinner
@ 2013-01-31  2:30       ` Tom
  2013-02-04 12:55         ` Dave Chinner
  2013-01-31  7:35       ` Stefan Ring
  1 sibling, 1 reply; 22+ messages in thread
From: Tom @ 2013-01-31  2:30 UTC (permalink / raw)
  To: xfs

In a previous message, Dave Chinner wrote:
>
> And this says the filesystem is still mounted. Why?
>

Good question.

As mentioned in the CentOS bug report, this is a freshly Kickstarted
CentOS 5.9 system, staged with minimal packages.

I'm not running any third party daemons or anything like that.  I
could wipe the system and re-kickstart it again over and over, and
the problem will exist with 5.9 and the 348 kernel.  I can also repeat
this on any x86 based platform with nothing special, no hardware RAID
controllers, just generic SATA or SCSI disks.  Doesn't happen with any
previous CentOS kernels.

Also mentioned, if I take a 5.8 system, which is running correctly,
and upgrade the kernel from 308 to 348, the problem shows up with the
new kernel.  If I downgrade 5.9 to 308, the problem is resolved as well.
Also if I use ext3 with md raid1, there is no problem either.  That's
why in my mind, all roads lead to some interaction between XFS and md
raid.  Of course I could be off with that assessment.

My next email will contain the "sysrq t" output that Ben requested.

Thanks!

-- Tom



_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 22+ messages in thread
* Re: XFS appears to cause strange hang with md raid1 on reboot
  2013-01-30 23:46     ` Dave Chinner
  2013-01-31  2:30       ` Tom
@ 2013-01-31  7:35       ` Stefan Ring
  1 sibling, 0 replies; 22+ messages in thread
From: Stefan Ring @ 2013-01-31  7:35 UTC (permalink / raw)
  To: Tom, xfs

> So the filesystems are unmounted....
>
> And this says the filesystem is still mounted. Why?

When I last had a problem with shutdown, it was because of mount
namespaces. You can unmount all day long, but as long as there is a
mount still open in a namespace, nothing will get unmounted. This was
on Fedora though; don't know if RHEL 5.x makes use of namespaces.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 22+ messages in thread
* Re: XFS appears to cause strange hang with md raid1 on reboot
  2013-01-31  2:30       ` Tom
@ 2013-02-04 12:55         ` Dave Chinner
  2013-02-05 18:22           ` Tom
  0 siblings, 1 reply; 22+ messages in thread
From: Dave Chinner @ 2013-02-04 12:55 UTC (permalink / raw)
  To: Tom; +Cc: xfs

On Wed, Jan 30, 2013 at 09:30:10PM -0500, Tom wrote:
> In a previous message, Dave Chinner wrote:
> >
> > And this says the filesystem is still mounted. Why?
> >
> 
> Good question.
> 
> As mentioned in the CentOS bug report, this is a freshly Kickstarted
> CentOS 5.9 system, staged with minimal packages.
....
> My next email will contain the "sysrq t" output that Ben requested.

Which doesn't tell us why the filesystem was not unmounted. We need
to know why/how the unmount failed to solve the problem, so you'll
need to do some debugging of the unmount to try to work that out...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 22+ messages in thread
* Re: XFS appears to cause strange hang with md raid1 on reboot
  2013-02-04 12:55         ` Dave Chinner
@ 2013-02-05 18:22           ` Tom
  2013-02-05 21:32             ` Dave Chinner
  0 siblings, 1 reply; 22+ messages in thread
From: Tom @ 2013-02-05 18:22 UTC (permalink / raw)
  To: david; +Cc: storm9c1, xfs

In a previous message, Dave Chinner wrote:
>
> Which doesn't tell us why the filesystem was not unmounted. We need to
> know why/how the unmount failed to solve the problem, so you'll need to
> do some debugging of the unmount to try to work that out...
>

Thanks Dave.

I can work on that tonight.

Oddly, since the problem is kernel related (308 doesn't do it, 348 does),
the system shuts down the same way no matter what kernel I'm using.  So
what would be the difference?  The umounts and running processes shouldn't
change.  Just seems to me that some resource isn't being released.  And
only with kernel 348 and XFS.  Quite puzzling..........

Any suggestions on how I would debug this?

-- Tom


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 22+ messages in thread
* Re: XFS appears to cause strange hang with md raid1 on reboot
  2013-02-05 18:22           ` Tom
@ 2013-02-05 21:32             ` Dave Chinner
  2013-02-05 23:05               ` Tom
  2013-02-06  4:08               ` Tom
  0 siblings, 2 replies; 22+ messages in thread
From: Dave Chinner @ 2013-02-05 21:32 UTC (permalink / raw)
  To: Tom; +Cc: xfs

On Tue, Feb 05, 2013 at 01:22:42PM -0500, Tom wrote:
> In a previous message, Dave Chinner wrote:
> >
> > Which doesn't tell us why the filesystem was not unmounted. We need to
> > know why/how the unmount failed to solve the problem, so you'll need to
> > do some debugging of the unmount to try to work that out...
> >
> 
> Thanks Dave.
> 
> I can work on that tonight.
> 
> Oddly, since the problem is kernel related (308 doesn't do it, 348 does),
> the system shuts down the same way no matter what kernel I'm using.  So
> what would be the difference?  The umounts and running processes shouldn't
> change.  Just seems to me that some resource isn't being released.  And
> only with kernel 348 and XFS.  Quite puzzling..........
> 
> Any suggestions on how I would debug this?

Find out if the unmount is returning an error first. If there is no
error, then you need to find what is doing bind mounts on your
system and make sure they are unmounted properly before the final
unmount is done. If lazy unmount is being done, make it a normal
unmount an see where the unmount is getting stcuk or taking time to
complete by using sysrq-w if it gets delayed for any length of time.

FWIW, because this is a old, old kernel, event tracing is not
available, so the single most useful tool for tracking this down is
not available...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 22+ messages in thread
* Re: XFS appears to cause strange hang with md raid1 on reboot
  2013-02-05 21:32             ` Dave Chinner
@ 2013-02-05 23:05               ` Tom
  2013-02-06  4:08               ` Tom
  1 sibling, 0 replies; 22+ messages in thread
From: Tom @ 2013-02-05 23:05 UTC (permalink / raw)
  To: david; +Cc: storm9c1, xfs

In a previous message, Dave Chinner wrote:
>>
>> Any suggestions on how I would debug this?
>
> Find out if the unmount is returning an error first. If there is no
> error, then you need to find what is doing bind mounts on your
> system and make sure they are unmounted properly before the final
> unmount is done. If lazy unmount is being done, make it a normal
> unmount an see where the unmount is getting stcuk or taking time to
> complete by using sysrq-w if it gets delayed for any length of time.
>

I agree.  However, I may be at the mercy of the upstream vendor.
This is a completely stock CentOS system with minimal packages
installed (save using XFS as the root fs).  Oracle Enterprise Linux
as well as Scientific Linux also suffers from the same problem.
It's curious because the upstream vendor must be doing something
at shutdown to trigger this.  I will get my hands dirty tonight and
see how the system performs these umounts at shutdown.

I am also prepared to blacklist the entire CentOS/RHEL 5.9 since
I've been struggling with this for a few weeks now.  I am getting
close to giving up and moving on.  I mean, I ran into this during
regression testing before moving my "latest" pointer from 5.8 to 5.9.
I can easily keep latest at 5.8 and not cause impact for now (which is
why I use the version "pointer" mentality).  But I'm also worried about
no upgrade path until this is resolved.  I am stubborn about using XFS
exclusively.  Otherwise I could just deploy on ext3 and never see this.


> FWIW, because this is a old, old kernel, event tracing is not
> available, so the single most useful tool for tracking this down is not
> available...

Yeah, I know.  Welcome to the RHEL/CentOS/OEL/SL world.  That's why I use
Ubuntu for the desktop (easy to play with, easy to break, easy to fix) and
RHEL for servers (hard to break, but when it does break, watch out).
It's been a long time since I saw RHEL break this bad...I'm usually pretty
good at working around problems like this.  But this one has me stumped.

Thanks again.  I'll keep you posted.

-- Tom


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 22+ messages in thread
* Re: XFS appears to cause strange hang with md raid1 on reboot
  2013-02-05 21:32             ` Dave Chinner
  2013-02-05 23:05               ` Tom
@ 2013-02-06  4:08               ` Tom
  2013-02-06 23:51                 ` Dave Chinner
  1 sibling, 1 reply; 22+ messages in thread
From: Tom @ 2013-02-06  4:08 UTC (permalink / raw)
  To: david; +Cc: storm9c1, xfs

In a previous message, Dave Chinner wrote:
>
> Find out if the unmount is returning an error first. If there is no
> error, then you need to find what is doing bind mounts on your
> system and make sure they are unmounted properly before the final
> unmount is done. If lazy unmount is being done, make it a normal
> unmount an see where the unmount is getting stcuk or taking time to
> complete by using sysrq-w if it gets delayed for any length of time.

OK, here is what I did tonight.  I added debug toward the end of
/etc/rc.d/rc6.d/S01reboot  ...where the umounts are normally handled.

Turns out that / and /proc cannot be unmounted (of course), so it gets
remounted as read-only.  See output below.

I also noticed that md3 (root fs) isn't showing up in this list
at the very end (I believe these messages are produced by the kernel
md driver):
md: md2 switched to read-only mode.
md: md1 switched to read-only mode.


So just for kicks, I added "mdadm --readonly --force /dev/md3" as well
after the umounts.  Of course /dev/md3 can't be forced into readonly
mode because the root file system is still mounted (albeit also read-only).
So no luck there.


Shutting down interface eth0:  [  OK  ]
Shutting down loopback interface:  [  OK  ]
Starting killall:  [  OK  ]
Sending all processes the TERM signal...
Sending all processes the KILL signal...
Saving random seed:
Syncing hardware clock to system time
Turning off swap:
Unmounting pipe file systems:
Unmounting file systems:

DEBUG: remounting '/' as read-only using 'mount -n -o ro,remount'
DEBUG: remounting '/proc' as read-only using 'mount -n -o ro,remount'
mdadm: failed to set readonly for /dev/md3: Device or resource busy

Please stand by while rebooting the system...
md: stopping all md devices.
md: md2 switched to read-only mode.
md: md1 switched to read-only mode.
(hang)

Just for kicks, I get the same output with the 308 kernel, with the
addition of this:

md: md3 still in use.

But the same system happily reboots just fine with the 308 kernel even
after producing that "still in use" message that 348 does not produce.


I did some more experiments with mdadm and I can't get any underlying
md device to go into read-only mode even if the fs is mounted read-only.
The only way I could get that to work is if the fs is completely unmounted.
Whether it is XFS or ext3.  Yet a system on ext3 reboots fine.

During reboot, I would expect /proc and / to be still mounted, albeit
read-only, and I would expect that md should be able to handle this.
But it can't.  What I didn't expect is the mdadm behavior to be consistent
between the 308 and 348 kernels.  But it is.  So something special happens
at the moment of reboot (that's different than what mdadm allows).

Now why this only happens with XFS and not ext3 is beyond me.

Is there more specific information that I can gather that may help?

-- Tom



_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 22+ messages in thread
* Re: XFS appears to cause strange hang with md raid1 on reboot
  2013-02-06  4:08               ` Tom
@ 2013-02-06 23:51                 ` Dave Chinner
  2013-02-07  4:18                   ` Tom
  0 siblings, 1 reply; 22+ messages in thread
From: Dave Chinner @ 2013-02-06 23:51 UTC (permalink / raw)
  To: Tom; +Cc: xfs

On Tue, Feb 05, 2013 at 11:08:52PM -0500, Tom wrote:
> In a previous message, Dave Chinner wrote:
> >
> > Find out if the unmount is returning an error first. If there is no
> > error, then you need to find what is doing bind mounts on your
> > system and make sure they are unmounted properly before the final
> > unmount is done. If lazy unmount is being done, make it a normal
> > unmount an see where the unmount is getting stcuk or taking time to
> > complete by using sysrq-w if it gets delayed for any length of time.
> 
> OK, here is what I did tonight.  I added debug toward the end of
> /etc/rc.d/rc6.d/S01reboot  ...where the umounts are normally handled.

> DEBUG: remounting '/' as read-only using 'mount -n -o ro,remount'
> DEBUG: remounting '/proc' as read-only using 'mount -n -o ro,remount'
> mdadm: failed to set readonly for /dev/md3: Device or resource busy

EBUSY means one of two possibilities:

	1. there's a file still open for write. => lsof
	2. there's an unlinked but still open file => lsof

But I don't think that's the problem at all.

> Please stand by while rebooting the system...
> md: stopping all md devices.
> md: md2 switched to read-only mode.
> md: md1 switched to read-only mode.
> (hang)
> 
> Just for kicks, I get the same output with the 308 kernel, with the
> addition of this:
> 
> md: md3 still in use.

Which implies that the problem is a change in behaviour in the md
layer or below. i.e. previously md just saw that it was busy and
did not try to tear down the device. Now it is trying to tear down
the device with a filesystem that is still active on it.

> But the same system happily reboots just fine with the 308 kernel even
> after producing that "still in use" message that 348 does not produce.

Right, because it correctly detects the filesystem is still in use
and doesn't try to tear down the device.

> I did some more experiments with mdadm and I can't get any underlying
> md device to go into read-only mode even if the fs is mounted read-only.
> The only way I could get that to work is if the fs is completely unmounted.
> Whether it is XFS or ext3.  Yet a system on ext3 reboots fine.

And that will be because ext3 won't be issuing any IO on the sync
that is triggered when tearing down the MD device. XFS is writing
the superblock, and that's where the MD device is hanging on itself.

> Is there more specific information that I can gather that may help?

No need - I can tell you the exact commit in the RHEL 5.9 tree that
caused this regression:

11ff4073: [md] Fix reboot stall with raid on megaraid_sas controller

The result is that the final shutdown of md devices now uses a
"force readonly" method, which means it ignores the fact that a
filesystem may still be active on top of it and rips the device out
from under the filesystem. This really only affects root devices,
and given that XFs is not supported as a root device on RHEL, it
isn't in the QE test matrix and so the problem was never noticed.

Feel free to report this all to the RH bugzilla - depending the
implications of the regression for supported configurations, it may
need to be fixed in RHEL anyway.

But now you know the problem, you can probably fix it yourself
rather than have to wait for RHEL/CentOS product cycle updates...

Cheers,

Dave.

PS: has the fact I quoted a RHEL5.9 commit id triggered a lightbulb
moment for you yet?  Hint: my other email address is
dchinner@redhat.com - this XFS community support effort was brought
to you by Red Hat.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 22+ messages in thread
* Re: XFS appears to cause strange hang with md raid1 on reboot
  2013-02-06 23:51                 ` Dave Chinner
@ 2013-02-07  4:18                   ` Tom
  0 siblings, 0 replies; 22+ messages in thread
From: Tom @ 2013-02-07  4:18 UTC (permalink / raw)
  To: david; +Cc: storm9c1, xfs

In a previous message, Dave Chinner wrote:
>
> No need - I can tell you the exact commit in the RHEL 5.9 tree that
> caused this regression:
>
> 11ff4073: [md] Fix reboot stall with raid on megaraid_sas controller
>

Dave,

This is excellent news!  At least we have some answers.


> This really only affects root devices,
> and given that XFs is not supported as a root device on RHEL, it
> isn't in the QE test matrix and so the problem was never noticed.

Well I noticed the problem for the sake of the community and reported
it.  :-)

I am happy that we have some Red Hat folks helping out here.

Thank goodness I do my own regression testing before I roll out a
new release.


>
> Feel free to report this all to the RH bugzilla - depending the
> implications of the regression for supported configurations, it may need
> to be fixed in RHEL anyway.

Yes, I could since I work full-time for a company that has an
extensive (7-figure) RHEL deployment, including XFS use licenses.
However, I'll have to go through an I.T. channel since they hold the
accounts.  I'll investigate whether they will let me do this since
they do not use XFS on root.  I would need them to back me up first.

>
> But now you know the problem, you can probably fix it yourself
> rather than have to wait for RHEL/CentOS product cycle updates...
>

Yes, thank you.


> PS: has the fact I quoted a RHEL5.9 commit id triggered a lightbulb
> moment for you yet?  Hint: my other email address is
> dchinner@redhat.com - this XFS community support effort was brought to
> you by Red Hat.

Indeed.  Again, I appreciate your help.  And from Red Hat, Inc. as well.

-- Tom


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 22+ messages in thread
* Re: XFS appears to cause strange hang with md raid1 on reboot
@ 2013-01-31  2:34 Tom
  0 siblings, 0 replies; 22+ messages in thread
From: Tom @ 2013-01-31  2:34 UTC (permalink / raw)
  To: xfs

In a previous message, Ben Myers wrote:
> Hi Tom,
>

Hi Ben,

>
> I think you should consider copying linux-raid@vger.kernel.org.
>

That will be next if you think it's appropriate.  Problem is that we
know how the RHEL/CentOS kernel is...


> Did you have a chance to get the sysrq output?  It would be helpful to
> see the other tasks on your machine.

Your wish is my command:

Script started on Wed 30 Jan 2013 09:12:25 PM EST
Stopping sshd: [  OK  ]
Shutting down postfix: [  OK  ]
Shutting down ntpd: [  OK  ]
Killing mdmonitor: [  OK  ]
Stopping mcstransd: [  OK  ]
Stopping portmap: [  OK  ]
Shutting down restorecond: [  OK  ]
Shutting down kernel logger: [  OK  ]
Shutting down system logger: [  OK  ]
Shutting down interface eth0:  [  OK  ]
Shutting down loopback interface:  [  OK  ]
Starting killall:  [  OK  ]
Sending all processes the TERM signal...
Sending all processes the KILL signal...
Saving random seed:
Syncing hardware clock to system time
Turning off swap:
Unmounting pipe file systems:
Unmounting file systems:
Please stand by while rebooting the system...
md: stopping all md devices.
md: md2 switched to read-only mode.
md: md1 switched to read-only mode.

telnet> send brk
t
SysRq : Show State

                                                       sibling
  task                 PC          pid father child younger older
init          S ffff810005e36420     0     1      0     2
(NOTLB)
 ffff81005be95a28 0000000000000086 00000000000002d0 ffff81005acfc3b4
0000000000000292 000000000000000a ffff81005be8c7a0 ffffffff8031cb60
000000444e0246a5 0000000000002014 ffff81005be8c988 0000000088061262
Call Trace:
 [<ffffffff8006389f>] schedule_timeout+0x8a/0xad
 [<ffffffff8009a92b>] process_timeout+0x0/0x5
 [<ffffffff80011bf0>] do_select+0x3dc/0x43c
 [<ffffffff8001f12b>] __pollwait+0x0/0xe2
 [<ffffffff8008f34a>] default_wake_function+0x0/0xe
 [<ffffffff80130625>] avc_alloc_node+0x3a/0x187
 [<ffffffff80130985>] avc_has_perm_noaudit+0x213/0x38a
 [<ffffffff80131590>] avc_has_perm+0x46/0x58
 [<ffffffff80131590>] avc_has_perm+0x46/0x58
 [<ffffffff80131e71>] inode_has_perm+0x56/0x63
 [<ffffffff8000a7b9>] __link_path_walk+0xf10/0xf39
 [<ffffffff800ee494>] core_sys_select+0x1bc/0x265
 [<ffffffff8002d0f1>] mntput_no_expire+0x19/0x89
 [<ffffffff8001b47a>] cp_new_stat+0xe5/0xfd
 [<ffffffff80016a1d>] sys_select+0x153/0x17c
 [<ffffffff8005d116>] system_call+0x7e/0x83

migration/0   S 0000000000000000     0     2      1             3
(L-TLB)
 ffff81005be99ea0 0000000000000046 ffff81005be8c040 ffff81005be8c078
ffffffff8031c800 0000000000000001 ffff81005be8c040 ffff81005be8e7e0
00000004ce553f1e 000000000000056a ffff81005be8c228 000000005be95cc0
Call Trace:
 [<ffffffff80045299>] migration_thread+0x1a1/0x239
 [<ffffffff800450f8>] migration_thread+0x0/0x239
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

ksoftirqd/0   S ffff810005e36420     0     3      1             4     2
(L-TLB)
 ffff81005be9ded0 0000000000000046 ffff81005be8e7e0 ffff810037fef820
0000003cd0ba4d94 000000000000000a ffff81005be8e7e0 ffffffff8031cb60
0000003fec6ce025 000000000000759f ffff81005be8e9c8 000000008008e8b8
Call Trace:
 [<ffffffff800973ed>] ksoftirqd+0x0/0xbf
 [<ffffffff80097428>] ksoftirqd+0x3b/0xbf
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

watchdog/0    S ffff810005e36420     0     4      1             5     3
(L-TLB)
 ffff81005be9fec0 0000000000000046 0000000000000000 ffff810037d104d0
ffffffffffffffff 0000000000000001 ffff81005be8e080 ffffffff8031cb60
000000444e9a79f3 0000000000000197 ffff81005be8e268 00000000800a8484
Call Trace:
 [<ffffffff800be377>] watchdog+0x0/0x5f
 [<ffffffff800be3c6>] watchdog+0x4f/0x5f
 [<ffffffff800be377>] watchdog+0x0/0x5f
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

events/0      S ffff810005e36420     0     5      1             6     4
(L-TLB)
 ffff81005b907e70 0000000000000046 ffff810005e38920 ffff810005e38950
0000000300000000 000000000000000a ffff810037fef820 ffffffff8031cb60
00000044547f155f 000000000000950c ffff810037fefa08 000000008002e4bc
Call Trace:
 [<ffffffff8004a0ea>] worker_thread+0x0/0x122
 [<ffffffff8004a1aa>] worker_thread+0xc0/0x122
 [<ffffffff8008f34a>] default_wake_function+0x0/0xe
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

khelper       S ffff810037d34780     0     6      1            19     5
(L-TLB)
 ffff810037f4be70 0000000000000046 ffff8100560bdcc8 ffff810037f4a000
0000000300000000 000000000000000a ffff810037fef0c0 ffff810059d80820
0000000ff1eecf8d 0000000000007485 ffff810037fef2a8 000000008002e4bc
Call Trace:
 [<ffffffff800a032f>] __call_usermodehelper+0x0/0x4f
 [<ffffffff8004d8d0>] run_workqueue+0xcd/0xfb
 [<ffffffff8004a0ea>] worker_thread+0x0/0x122
 [<ffffffff8004a1aa>] worker_thread+0xc0/0x122
 [<ffffffff8008f34a>] default_wake_function+0x0/0xe
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

kthread       S ffff810037d34740     0    19      1    23    2050     6
(L-TLB)
 ffff81005b911e70 0000000000000046 000000000000072e ffff810057275d68
0000000300000000 000000000000000a ffff810037fc87e0 ffff810059490860
0000000eb6ada09a 0000000000000eba ffff810037fc89c8 000000008002e4bc
Call Trace:
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff8004d8d0>] run_workqueue+0xcd/0xfb
 [<ffffffff8004a0ea>] worker_thread+0x0/0x122
 [<ffffffff8004a1aa>] worker_thread+0xc0/0x122
 [<ffffffff8008f34a>] default_wake_function+0x0/0xe
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

kblockd/0     S ffff810037dfb820     0    23     19            24
(L-TLB)
 ffff81005b973e70 0000000000000046 ffffffff801554bc ffffffff8014a6e7
0000000300000000 000000000000000a ffff81005b969860 ffff810037dfb820
0000003fec6c5835 0000000000000f02 ffff81005b969a48 000000008002e4bc
Call Trace:
 [<ffffffff801554bc>] cfq_kick_queue+0x0/0x89
 [<ffffffff8014a6e7>] elv_next_request+0x168/0x178
 [<ffffffff801554bc>] cfq_kick_queue+0x0/0x89
 [<ffffffff8004d8d0>] run_workqueue+0xcd/0xfb
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff8004a0ea>] worker_thread+0x0/0x122
 [<ffffffff8004a1aa>] worker_thread+0xc0/0x122
 [<ffffffff8008f34a>] default_wake_function+0x0/0xe
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

kacpid        S ffff810037fef0c0     0    24     19           127    23
(L-TLB)
 ffff81005b975e70 0000000000000046 ffff81005b96bc40 ffff81005b96bc48
0000000300000000 0000000000000001 ffff81005b969100 ffff810037fef0c0
00000004de51fab8 00000000000023cb ffff81005b9692e8 000000008002e4bc
Call Trace:
 [<ffffffff80186a34>] bind_to_cpu0+0x0/0x62
 [<ffffffff8004d8d0>] run_workqueue+0xcd/0xfb
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff8004a0ea>] worker_thread+0x0/0x122
 [<ffffffff8004a1aa>] worker_thread+0xc0/0x122
 [<ffffffff8008f34a>] default_wake_function+0x0/0xe
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

cqueue/0      S ffff810037fef0c0     0   127     19           130    24
(L-TLB)
 ffff810037df3e70 0000000000000046 00000004e0693e36 0000000000000000
ffff810037fc87e0 0000000000000001 ffff810037df47a0 ffff810037fef0c0
00000004e069e550 00000000000008a5 ffff810037df4988 000000008031c800
Call Trace:
 [<ffffffff80063002>] thread_return+0x62/0xfe
 [<ffffffff8009cb11>] recalc_sigpending_and_wake+0x9/0x1a
 [<ffffffff800294aa>] do_sigaction+0x14d/0x199
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff8004a0ea>] worker_thread+0x0/0x122
 [<ffffffff8004a1aa>] worker_thread+0xc0/0x122
 [<ffffffff8008f34a>] default_wake_function+0x0/0xe
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff800a05ce>] request_module+0x0/0x14d
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

khubd         S 0000000000000282     0   130     19           132   127
(L-TLB)
 ffff81005b9a3e10 0000000000000046 ffffffff8008f34a 8000018000000000
ffff81005b144800 000000000000000a ffff810037df8080 ffff81005bac70c0
00000005c414c5b0 0000000000003798 ffff810037df8268 000000005b1412a0
Call Trace:
 [<ffffffff8008f34a>] default_wake_function+0x0/0xe
 [<ffffffff801fc8d2>] hub_port_status+0x59/0xcf
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff801ff591>] hub_thread+0xa83/0xb01
 [<ffffffff800a3bc2>] autoremove_wake_function+0x0/0x2e
 [<ffffffff801feb0e>] hub_thread+0x0/0xb01
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff800a05ce>] request_module+0x0/0x14d
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff801d0013>] klist_drivers_get+0x0/0xc
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

kseriod       S 0000000000000282     0   132     19           202   130
(L-TLB)
 ffff810037dafe90 0000000000000046 0000000000008080 ffffffff8011481b
ffffffff80351640 000000000000000a ffff810037dfb0c0 ffff810058fe2040
0000000bf55568a9 000000000001e65e ffff810037dfb2a8 00000000881cb950
Call Trace:
 [<ffffffff8011481b>] sysfs_make_dirent+0x1b/0x85
 [<ffffffff801d1120>] driver_create_file+0x31/0x39
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80213a26>] serio_thread+0x29c/0x2e7
 [<ffffffff800a3bc2>] autoremove_wake_function+0x0/0x2e
 [<ffffffff8021378a>] serio_thread+0x0/0x2e7
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff800a05ce>] request_module+0x0/0x14d
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80133566>] selinux_d_instantiate+0x0/0x14
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

khungtaskd    S ffff810005e36420     0   202     19           203   132
(L-TLB)
 ffff81005baede60 0000000000000046 ffff81005baeddd0 ffffffff8008ed60
00000004e1c15ef4 000000000000000a ffff81005bacb860 ffffffff8031cb60
0000003ca8be31a6 0000000000002e7a ffff81005bacba48 000000005bacb898
Call Trace:
 [<ffffffff8008ed60>] __activate_task+0x56/0x6d
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff8006389f>] schedule_timeout+0x8a/0xad
 [<ffffffff8009a92b>] process_timeout+0x0/0x5
 [<ffffffff800be421>] watchdog+0x4b/0x1a3
 [<ffffffff800be3d6>] watchdog+0x0/0x1a3
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

pdflush       S ffff810037fc87e0     0   203     19           204   202
(L-TLB)
 ffff81005baefe70 0000000000000046 ffff81005bacb100 ffff81005be95d60
0000000000000282 0000000000000001 ffff81005bacb100 ffff810037fc87e0
00000004e1c1e550 000000000000084c ffff81005bacb2e8 000000008008e8b8
Call Trace:
 [<ffffffff8008eaf9>] task_rq_lock+0x3d/0x6f
 [<ffffffff800b2e06>] guarantee_online_cpus+0x59/0x87
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff8005689a>] pdflush+0x0/0x1fb
 [<ffffffff80056987>] pdflush+0xed/0x1fb
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

pdflush       S ffff810005e36420     0   204     19           205   203
(L-TLB)
 ffff81005baf3e70 0000000000000046 0000000000000000 ffffffff8032c100
0000000000000286 000000000000000a ffff81005bacc7a0 ffffffff8031cb60
000000438fd6cfea 0000000000000895 ffff81005bacc988 0000000000000282
Call Trace:
 [<ffffffff800cd832>] wb_kupdate+0x161/0x16a
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff8005689a>] pdflush+0x0/0x1fb
 [<ffffffff80056987>] pdflush+0xed/0x1fb
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

kswapd0       S ffff810037fc87e0     0   205     19           206   204
(L-TLB)
 ffff81005baf5dd0 0000000000000046 ffff81005bacc040 ffff810000014000
0000000000000282 0000000000000001 ffff81005bacc040 ffff810037fc87e0
00000004e1c367bd 0000000000000b35 ffff81005bacc228 0000000000000000
Call Trace:
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80057c42>] kswapd+0x12c/0x495
 [<ffffffff80063002>] thread_return+0x62/0xfe
 [<ffffffff800a3bc2>] autoremove_wake_function+0x0/0x2e
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80057b16>] kswapd+0x0/0x495
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

aio/0         S ffff810005e36420     0   206     19           348   205
(L-TLB)
 ffff81005baf7e70 0000000000000046 00000004e1c39d2a 0000000000000000
ffff810037fc87e0 0000000000000001 ffff81005bad67e0 ffffffff8031cb60
00000004e1c78cec 0000000000000917 ffff81005bad69c8 000000008031c800
Call Trace:
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff8004a0ea>] worker_thread+0x0/0x122
 [<ffffffff8004a1aa>] worker_thread+0xc0/0x122
 [<ffffffff8008f34a>] default_wake_function+0x0/0xe
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

kpsmoused     S 0000000000000282     0   348     19           365   206
(L-TLB)
 ffff81005b099e70 0000000000000046 000000059b4e55f1 0000000000000000
ffff810037fc87e0 0000000000000009 ffff81005b06c860 ffff81005b06c100
000000059b50f97d 00000000000009cd ffff81005b06ca48 000000008031c800
Call Trace:
 [<ffffffff80063002>] thread_return+0x62/0xfe
 [<ffffffff8009cb11>] recalc_sigpending_and_wake+0x9/0x1a
 [<ffffffff800294aa>] do_sigaction+0x14d/0x199
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff8004a0ea>] worker_thread+0x0/0x122
 [<ffffffff8004a1aa>] worker_thread+0xc0/0x122
 [<ffffffff8008f34a>] default_wake_function+0x0/0xe
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff800a05ce>] request_module+0x0/0x14d
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

xfs_mru_cache S 0000000000000282     0   365     19           366   348
(L-TLB)
 ffff81005b39be70 0000000000000046 00000005c5c42f44 0000000000000000
ffff810037fc87e0 0000000000000007 ffff81005b0657e0 ffff81005bac7820
00000005c5c443bf 0000000000000726 ffff81005b0659c8 0000000000000000
Call Trace:
 [<ffffffff80063002>] thread_return+0x62/0xfe
 [<ffffffff8009cb11>] recalc_sigpending_and_wake+0x9/0x1a
 [<ffffffff800294aa>] do_sigaction+0x14d/0x199
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff8004a0ea>] worker_thread+0x0/0x122
 [<ffffffff8004a1aa>] worker_thread+0xc0/0x122
 [<ffffffff8008f34a>] default_wake_function+0x0/0xe
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

xfslogd/0     S 0000000000000282     0   366     19           367   365
(L-TLB)
 ffff81005b39de70 0000000000000046 ffff81005b2da400 ffff81005b1b9a80
0000000300000000 000000000000000a ffff81005b065080 ffff810058b25080
0000003fec06893f 0000000000001235 ffff81005b065268 000000008002e4bc
Call Trace:
 [<ffffffff8807fa47>] :xfs:xfs_buf_iodone_work+0x0/0x6a
 [<ffffffff8004d8d0>] run_workqueue+0xcd/0xfb
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff8004a0ea>] worker_thread+0x0/0x122
 [<ffffffff8004a1aa>] worker_thread+0xc0/0x122
 [<ffffffff8008f34a>] default_wake_function+0x0/0xe
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

xfsdatad/0    S ffff810005e36420     0   367     19           376   366
(L-TLB)
 ffff81005b3a1e70 0000000000000046 0000000000000000 0000000000000001
0000000300000000 000000000000000a ffff81005b068820 ffffffff8031cb60
0000003fd568cbf7 0000000000001498 ffff81005b068a08 000000008002e4bc
Call Trace:
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff8004a0ea>] worker_thread+0x0/0x122
 [<ffffffff8004a1aa>] worker_thread+0xc0/0x122
 [<ffffffff8008f34a>] default_wake_function+0x0/0xe
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

ata/0         S 0000000000000282     0   376     19           377   367
(L-TLB)
 ffff81005ac03e70 0000000000000046 00000000fffb825b 00000001800638a7
0000000300000000 000000000000000a ffff81005bac7820 ffff81005b06c100
00000006122b8b9e 0000000000070d1d ffff81005bac7a08 000000008002e4bc
Call Trace:
 [<ffffffff88110856>] :libata:ata_pio_task+0x0/0xd7
 [<ffffffff8004d8d0>] run_workqueue+0xcd/0xfb
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff8004a0ea>] worker_thread+0x0/0x122
 [<ffffffff8004a1aa>] worker_thread+0xc0/0x122
 [<ffffffff8008f34a>] default_wake_function+0x0/0xe
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

ata_aux       S 0000000000000282     0   377     19           380   376
(L-TLB)
 ffff81005ac05e70 0000000000000046 00000005c8415ad8 0000000000000000
ffff810037fc87e0 0000000000000003 ffff81005b0740c0 ffff81005b06c100
00000005c8417824 0000000000000288 ffff81005b0742a8 0000000000000000
Call Trace:
 [<ffffffff80063002>] thread_return+0x62/0xfe
 [<ffffffff8009cb11>] recalc_sigpending_and_wake+0x9/0x1a
 [<ffffffff800294aa>] do_sigaction+0x14d/0x199
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff8004a0ea>] worker_thread+0x0/0x122
 [<ffffffff8004a1aa>] worker_thread+0xc0/0x122
 [<ffffffff8008f34a>] default_wake_function+0x0/0xe
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

scsi_eh_0     S 0000000000000282     0   380     19           381   377
(L-TLB)
 ffff81005b9d5e60 0000000000000046 ffff81005be16fc8 ffffffff8008d76f
0000000300000000 0000000000000005 ffff81005b0680c0 ffff81005b074820
00000005dabf7fba 0000000000008ca1 ffff81005b0682a8 000000008002e4bc
Call Trace:
 [<ffffffff8008d76f>] __wake_up_common+0x3e/0x68
 [<ffffffff880bf269>] :scsi_mod:__scsi_iterate_devices+0x56/0x6f
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff880c32fa>] :scsi_mod:scsi_error_handler+0x6a/0x4ce
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff880c3290>] :scsi_mod:scsi_error_handler+0x0/0x4ce
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff80015f17>] do_exit+0x925/0x931
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

scsi_eh_1     S 0000000000000282     0   381     19           382   380
(L-TLB)
 ffff81005b2cbe60 0000000000000046 ffff81005b28afc8 ffffffff8008d76f
0000000300000000 0000000000000008 ffff81005b071080 ffff81005b074820
00000005ece5556e 00000000000070e6 ffff81005b071268 000000008002e4bc
Call Trace:
 [<ffffffff8008d76f>] __wake_up_common+0x3e/0x68
 [<ffffffff880bf269>] :scsi_mod:__scsi_iterate_devices+0x56/0x6f
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff880c32fa>] :scsi_mod:scsi_error_handler+0x6a/0x4ce
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff880c3290>] :scsi_mod:scsi_error_handler+0x0/0x4ce
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

scsi_eh_2     S 0000000000000282     0   382     19           383   381
(L-TLB)
 ffff81005b2c7e60 0000000000000046 ffff81005b2cefc8 ffffffff8008d76f
0000000300000000 000000000000000a ffff81005bac70c0 ffff81005b074820
00000005ff8dbbf1 0000000000005494 ffff81005bac72a8 000000008002e4bc
Call Trace:
 [<ffffffff8008d76f>] __wake_up_common+0x3e/0x68
 [<ffffffff880bf269>] :scsi_mod:__scsi_iterate_devices+0x56/0x6f
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff880c32fa>] :scsi_mod:scsi_error_handler+0x6a/0x4ce
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff880c3290>] :scsi_mod:scsi_error_handler+0x0/0x4ce
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff80015f17>] do_exit+0x925/0x931
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

scsi_eh_3     S 0000000000000282     0   383     19           388   382
(L-TLB)
 ffff81005b3e7e60 0000000000000046 ffff81005b282fc8 ffffffff8008d76f
0000000300000000 000000000000000a ffff81005b06c100 ffff81005b074820
00000006122d75fa 0000000000004a1e ffff81005b06c2e8 000000008002e4bc
Call Trace:
 [<ffffffff8008d76f>] __wake_up_common+0x3e/0x68
 [<ffffffff880bf269>] :scsi_mod:__scsi_iterate_devices+0x56/0x6f
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff880c32fa>] :scsi_mod:scsi_error_handler+0x6a/0x4ce
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff880c3290>] :scsi_mod:scsi_error_handler+0x0/0x4ce
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

kstriped      S 0000000000000282     0   388     19           399   383
(L-TLB)
 ffff81005ac23e70 0000000000000046 00000006137fb3c8 0000000000000000
ffff810037fc87e0 0000000000000007 ffff81005bac07e0 ffff81005b06f040
00000006137fd135 00000000000006d0 ffff81005bac09c8 0000000000000000
Call Trace:
 [<ffffffff80063002>] thread_return+0x62/0xfe
 [<ffffffff8009cb11>] recalc_sigpending_and_wake+0x9/0x1a
 [<ffffffff800294aa>] do_sigaction+0x14d/0x199
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff8004a0ea>] worker_thread+0x0/0x122
 [<ffffffff8004a1aa>] worker_thread+0xc0/0x122
 [<ffffffff8008f34a>] default_wake_function+0x0/0xe
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

md3_raid1     S ffff810005e36420     0   399     19           402   388
(L-TLB)
 ffff81005beb1e40 0000000000000046 ffff81005be959c8 ffff81000617bd00
0000000000000246 000000000000000a ffff810037dfb820 ffffffff8031cb60
00000040674952a4 0000000000000480 ffff810037dfba08 0000000000007000
Call Trace:
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80063833>] schedule_timeout+0x1e/0xad
 [<ffffffff8003b128>] prepare_to_wait+0x34/0x61
 [<ffffffff8022501f>] md_thread+0xc4/0x10e
 [<ffffffff800a3bc2>] autoremove_wake_function+0x0/0x2e
 [<ffffffff80224f5b>] md_thread+0x0/0x10e
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

md1_raid1     S 0000000000000282     0   402     19           405   399
(L-TLB)
 ffff81005beb7e40 0000000000000046 ffff81005be959c8 ffff81005ba9ef00
0000000000000246 000000000000000a ffff81005b074820 ffff81005a07d7e0
0000004067491c2e 00000000000006e4 ffff81005b074a08 000000008008eb43
Call Trace:
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80063833>] schedule_timeout+0x1e/0xad
 [<ffffffff8003b128>] prepare_to_wait+0x34/0x61
 [<ffffffff8022501f>] md_thread+0xc4/0x10e
 [<ffffffff800a3bc2>] autoremove_wake_function+0x0/0x2e
 [<ffffffff80224f5b>] md_thread+0x0/0x10e
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

md2_raid1     S 0000000000000282     0   405     19           406   402
(L-TLB)
 ffff81005ac45e40 0000000000000046 ffff81005be959c8 ffff81005ba9ed00
0000000000000246 000000000000000a ffff81005b2a9100 ffff81005a07d7e0
0000004066228847 000000000000090b ffff81005b2a92e8 000000008008eb43
Call Trace:
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80063833>] schedule_timeout+0x1e/0xad
 [<ffffffff8003b128>] prepare_to_wait+0x34/0x61
 [<ffffffff8022501f>] md_thread+0xc4/0x10e
 [<ffffffff800a3bc2>] autoremove_wake_function+0x0/0x2e
 [<ffffffff80224f5b>] md_thread+0x0/0x10e
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

xfsbufd       S ffff810005e36420     0   406     19           407   405
(L-TLB)
 ffff81005acf5e60 0000000000000046 0000000000000008 0000000000000001
00000000589057c0 000000000000000a ffff81005b0717e0 ffffffff8031cb60
0000004438c41ed2 000000000000029e ffff81005b0719c8 0000000000000086
Call Trace:
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff8006389f>] schedule_timeout+0x8a/0xad
 [<ffffffff8009a92b>] process_timeout+0x0/0x5
 [<ffffffff8807fede>] :xfs:xfsbufd+0x4e/0xe6
 [<ffffffff8807fe90>] :xfs:xfsbufd+0x0/0xe6
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

xfsaild       S ffff810005e36420     0   407     19           408   406
(L-TLB)
 ffff81005ad6de60 0000000000000046 ffff81005ad6ddd0 ffffffff8008ed60
0000000acc805abe 000000000000000a ffff81005b2a9860 ffffffff8031cb60
0000004433f247e2 00000000000002b0 ffff81005b2a9a48 000000005b2a9898
Call Trace:
 [<ffffffff8008ed60>] __activate_task+0x56/0x6d
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff8006389f>] schedule_timeout+0x8a/0xad
 [<ffffffff8009a92b>] process_timeout+0x0/0x5
 [<ffffffff880864fe>] :xfs:xfsaild+0x33/0x77
 [<ffffffff880864cb>] :xfs:xfsaild+0x0/0x77
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

xfssyncd      S ffff810005e36420     0   408     19           434   407
(L-TLB)
 ffff81005adc1e50 0000000000000046 ffff81005b1b9600 ffff81005b2da400
ffff81005b2da4a8 000000000000000a ffff810037df87e0 ffffffff8031cb60
0000004299bccd8b 00000000000007ff ffff810037df89c8 000000008806c185
Call Trace:
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff8006389f>] schedule_timeout+0x8a/0xad
 [<ffffffff8009a92b>] process_timeout+0x0/0x5
 [<ffffffff88086059>] :xfs:xfssyncd+0x29/0x138
 [<ffffffff88086030>] :xfs:xfssyncd+0x0/0x138
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80133566>] selinux_d_instantiate+0x0/0x14
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

kauditd       S 0000000000000282     0   434     19           845   408
(L-TLB)
 ffff81005abafe90 0000000000000046 ffff81005ab8e7e0 ffff81005ab8e7e0
ffff81005ab8e7e0 0000000000000009 ffff81005ab8e7e0 ffff810037fc8080
0000000b6309ec99 000000000000058e ffff81005ab8e9c8 0000000080063002
Call Trace:
 [<ffffffff8002e4bc>] __wake_up+0x38/0x4f
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff800b6abf>] kauditd_thread+0x146/0x17d
 [<ffffffff8008f34a>] default_wake_function+0x0/0xe
 [<ffffffff800b6979>] kauditd_thread+0x0/0x17d
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

kedac         S ffff810005e36420     0   845     19          1459   434
(L-TLB)
 ffff81005a07fe70 0000000000000046 0000000000000016 ffffffff8022f6ef
0000000000000246 000000000000000a ffff810059d800c0 ffffffff8031cb60
000000442c1795b2 000000000000046c ffff810059d802a8 0000000000000000
Call Trace:
 [<ffffffff8022f6ef>] pci_conf1_read+0xcc/0xd7
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff8832ba13>] :edac_mc:edac_kernel_thread+0x0/0xfb
 [<ffffffff8006389f>] schedule_timeout+0x8a/0xad
 [<ffffffff8009a92b>] process_timeout+0x0/0x5
 [<ffffffff8832bae6>] :edac_mc:edac_kernel_thread+0xd3/0xfb
 [<ffffffff8832ba13>] :edac_mc:edac_kernel_thread+0x0/0xfb
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

kmpathd/0     S 0000000000000282     0  1459     19          1460   845
(L-TLB)
 ffff8100588bfe70 0000000000000046 0000000d1b7a2dfd 0000000000000000
ffff810037fc87e0 0000000000000003 ffff81005a5460c0 ffff81005a2d9080
0000000d1b7a449b 0000000000000461 ffff81005a5462a8 0000000000000000
Call Trace:
 [<ffffffff80063002>] thread_return+0x62/0xfe
 [<ffffffff8009cb11>] recalc_sigpending_and_wake+0x9/0x1a
 [<ffffffff800294aa>] do_sigaction+0x14d/0x199
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff8004a0ea>] worker_thread+0x0/0x122
 [<ffffffff8004a1aa>] worker_thread+0xc0/0x122
 [<ffffffff8008f34a>] default_wake_function+0x0/0xe
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

kmpath_handle S 0000000000000282     0  1460     19          1834  1459
(L-TLB)
 ffff81005a58de70 0000000000000046 0000000d1b7a75b6 0000000000000000
ffff810037fc87e0 0000000000000003 ffff81005bac0080 ffff81005a2d9080
0000000d1b7a966a 00000000000002e6 ffff81005bac0268 0000000000000000
Call Trace:
 [<ffffffff80063002>] thread_return+0x62/0xfe
 [<ffffffff8009cb11>] recalc_sigpending_and_wake+0x9/0x1a
 [<ffffffff800294aa>] do_sigaction+0x14d/0x199
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff8004a0ea>] worker_thread+0x0/0x122
 [<ffffffff8004a1aa>] worker_thread+0xc0/0x122
 [<ffffffff8008f34a>] default_wake_function+0x0/0xe
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

rpciod/0      S 0000000000000282     0  1834     19                1460
(L-TLB)
 ffff810056c21e70 0000000000000046 0000000eac2d4565 0000000000000000
ffff810037fc87e0 0000000000000008 ffff810059f15080 ffff810059f3d860
0000000eac2d600a 000000000000077d ffff810059f15268 0000000000000000
Call Trace:
 [<ffffffff80063002>] thread_return+0x62/0xfe
 [<ffffffff8009cb11>] recalc_sigpending_and_wake+0x9/0x1a
 [<ffffffff800294aa>] do_sigaction+0x14d/0x199
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff8004a0ea>] worker_thread+0x0/0x122
 [<ffffffff8004a1aa>] worker_thread+0xc0/0x122
 [<ffffffff8008f34a>] default_wake_function+0x0/0xe
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

reboot        D ffff810037dfb820     0  2050      1                  19
(NOTLB)
 ffff81005ab37a08 0000000000000082 ffff81005ab37a58 ffff81005beb1ea0
0000000000000001 0000000000000007 ffff81005a07d7e0 ffff810037dfb820
0000004067494e24 00000000000031f6 ffff81005a07d9c8 000000008008d76f
Call Trace:
 [<ffffffff8002e4bc>] __wake_up+0x38/0x4f
 [<ffffffff80223bce>] md_write_start+0xf2/0x108
 [<ffffffff800a3bc2>] autoremove_wake_function+0x0/0x2e
 [<ffffffff880b102c>] :raid1:make_request+0x38/0x5d8
 [<ffffffff8001c839>] generic_make_request+0x211/0x228
 [<ffffffff8002389f>] mempool_alloc+0x31/0xe7
 [<ffffffff8001a98f>] vsnprintf+0x5d7/0xb54
 [<ffffffff80033695>] submit_bio+0xe6/0xed
 [<ffffffff8807f801>] :xfs:_xfs_buf_ioapply+0x1f2/0x254
 [<ffffffff8807f89c>] :xfs:xfs_buf_iorequest+0x39/0x64
 [<ffffffff8808386c>] :xfs:xfs_bdstrat_cb+0x36/0x3a
 [<ffffffff8807c0a8>] :xfs:xfs_bwrite+0x5e/0xba
 [<ffffffff88077669>] :xfs:xfs_syncsub+0x119/0x226
 [<ffffffff88084ce2>] :xfs:xfs_fs_sync_super+0x33/0xdd
 [<ffffffff8010aa44>] quota_sync_sb+0x2e/0xf0
 [<ffffffff800e55bd>] __fsync_super+0x1b/0x9e
 [<ffffffff800e578a>] fsync_super+0x9/0x16
 [<ffffffff800e57c1>] fsync_bdev+0x2a/0x3b
 [<ffffffff8014ea59>] invalidate_partition+0x28/0x40
 [<ffffffff802225a8>] do_md_stop+0xa0/0x2ec
 [<ffffffff80224d41>] md_notify_reboot+0x5f/0x120
 [<ffffffff80067565>] notifier_call_chain+0x20/0x32
 [<ffffffff8009de98>] blocking_notifier_call_chain+0x22/0x36
 [<ffffffff8009e220>] kernel_restart_prepare+0x18/0x29
 [<ffffffff8009e280>] kernel_restart+0x9/0x46
 [<ffffffff8009e40a>] sys_reboot+0x146/0x1c7
 [<ffffffff8003b291>] hrtimer_try_to_cancel+0x4a/0x53
 [<ffffffff8005a753>] hrtimer_cancel+0xc/0x16
 [<ffffffff80063cf9>] do_nanosleep+0x47/0x70
 [<ffffffff8005a640>] hrtimer_nanosleep+0x58/0x118
 [<ffffffff800a5b84>] hrtimer_wakeup+0x0/0x22
 [<ffffffff8001e2f2>] sigprocmask+0xb7/0xdb
 [<ffffffff80054fe6>] sys_nanosleep+0x4c/0x62
 [<ffffffff8005d116>] system_call+0x7e/0x83


telnet> quit
Connection closed.
Script done on Wed 30 Jan 2013 09:13:47 PM EST



_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 22+ messages in thread
end of thread, other threads:[~2013-02-07  4:18 UTC | newest]

Thread overview: 22+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-01-28 23:28 XFS appears to cause strange hang with md raid1 on reboot Tom
2013-01-29  0:05 ` Eric Sandeen
2013-01-29 21:47   ` Tom
2013-01-29 21:55     ` Eric Sandeen
2013-01-29 22:25       ` Tom
2013-01-29 22:39         ` Ben Myers
2013-01-30  8:54         ` Stan Hoeppner
2013-01-29 15:18 ` Ben Myers
2013-01-29 21:13   ` Tom
2013-01-30  3:16   ` Tom
2013-01-30 22:51     ` Ben Myers
2013-01-30 23:46     ` Dave Chinner
2013-01-31  2:30       ` Tom
2013-02-04 12:55         ` Dave Chinner
2013-02-05 18:22           ` Tom
2013-02-05 21:32             ` Dave Chinner
2013-02-05 23:05               ` Tom
2013-02-06  4:08               ` Tom
2013-02-06 23:51                 ` Dave Chinner
2013-02-07  4:18                   ` Tom
2013-01-31  7:35       ` Stefan Ring
2013-01-31  2:34 Tom

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.