Crash with 3.8.3 and TuxOnIce

Crash with 3.8.3 and TuxOnIce

[Pedro Ribeiro]

[-- Attachment #1.1: Type: text/plain, Size: 741 bytes --]

Hi,

I'm using a TuxOnIce enabled kernel, and recently moved from 3.7.1 to
3.8.3. The former used to work nicely, but now I get a hard crash when I
try to hibernate.
This is solely related to TuxOnIce as it does not happen in the default
hibernation.

I know this is an unsupported out of tree patch, but can you please help me
debug it or point in the right direction?

Unfortunately as this was a kernel crash all I have are crappy photos, but
they do show a readable stack trace:
img18[dot]imageshack[dot]us/img18/395/imag0228hp[dot]jpg
img13[dot]imageshack[dot]us/img13/4375/imag0231p[dot]jpg

(I removed the HTTP part and the dots off the links else it would bounce
back from the list)

Thanks in advance for your help.

Regards,
Pedro

[-- Attachment #1.2: Type: text/html, Size: 1867 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Crash with 3.8.3 and TuxOnIce

[Dave Chinner]

On Tue, Mar 19, 2013 at 11:31:16PM +0000, Pedro Ribeiro wrote:
> Hi,
> 
> I'm using a TuxOnIce enabled kernel, and recently moved from 3.7.1 to
> 3.8.3. The former used to work nicely, but now I get a hard crash when I
> try to hibernate.
> This is solely related to TuxOnIce as it does not happen in the default
> hibernation.
> 
> I know this is an unsupported out of tree patch, but can you please help me
> debug it or point in the right direction?
> 
> Unfortunately as this was a kernel crash all I have are crappy photos, but
> they do show a readable stack trace:
> img18[dot]imageshack[dot]us/img18/395/imag0228hp[dot]jpg
> img13[dot]imageshack[dot]us/img13/4375/imag0231p[dot]jpg
> 
> (I removed the HTTP part and the dots off the links else it would bounce
> back from the list)

The list doesn't bounce URLs. Please make then clicky in future.

As it is, your kernel has oopsed in submit_bio() when writing a log
buffer writing a dummy log record.

The xfs_log_worker is trying to cover the log, and it seems like the
IO subsystem below it is doing something wrong. Perhaps Tux-on-Ice
is killing the IO subsystem without having first stopped all the
async filesystem processing first?

Looks like a Tux-on-ICE problem to me...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Crash with 3.8.3 and TuxOnIce

[Pedro Ribeiro]

[-- Attachment #1.1: Type: text/plain, Size: 2237 bytes --]

Thanks for the answer Dave.

Yes I would definitely say it's a ToI bug that perhaps has been dormant so
far. Unfortunately the ToI developer is very busy at the moment, so I will
have to debug and fix it myself.
This problem did not occur with 3.7 and the ToI code did not change.

Do you have any idea where I can start looking for the XFS change in 3.8
that triggered this behaviour in ToI? Or maybe it was a VFS change?

PS: the email definitely bounced back, most likely because imageshack is
blocked on the sgi server:

Technical details of permanent failure:
Google tried to deliver your message, but it was rejected by the server for
the recipient domain oss.sgi.com by cuda-allmx.sgi.com. [192.48.176.16].

The error that the other server returned was:
554 rejecting banned content

Regards,
Pedro

On 20 March 2013 00:44, Dave Chinner <david@fromorbit.com> wrote:

> On Tue, Mar 19, 2013 at 11:31:16PM +0000, Pedro Ribeiro wrote:
> > Hi,
> >
> > I'm using a TuxOnIce enabled kernel, and recently moved from 3.7.1 to
> > 3.8.3. The former used to work nicely, but now I get a hard crash when I
> > try to hibernate.
> > This is solely related to TuxOnIce as it does not happen in the default
> > hibernation.
> >
> > I know this is an unsupported out of tree patch, but can you please help
> me
> > debug it or point in the right direction?
> >
> > Unfortunately as this was a kernel crash all I have are crappy photos,
> but
> > they do show a readable stack trace:
> > img18[dot]imageshack[dot]us/img18/395/imag0228hp[dot]jpg
> > img13[dot]imageshack[dot]us/img13/4375/imag0231p[dot]jpg
> >
> > (I removed the HTTP part and the dots off the links else it would bounce
> > back from the list)
>
> The list doesn't bounce URLs. Please make then clicky in future.
>
> As it is, your kernel has oopsed in submit_bio() when writing a log
> buffer writing a dummy log record.
>
> The xfs_log_worker is trying to cover the log, and it seems like the
> IO subsystem below it is doing something wrong. Perhaps Tux-on-Ice
> is killing the IO subsystem without having first stopped all the
> async filesystem processing first?
>
> Looks like a Tux-on-ICE problem to me...
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
>

[-- Attachment #1.2: Type: text/html, Size: 4118 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Crash with 3.8.3 and TuxOnIce

[Dave Chinner]

On Wed, Mar 20, 2013 at 06:01:35PM +0000, Pedro Ribeiro wrote:
> Thanks for the answer Dave.
> 
> Yes I would definitely say it's a ToI bug that perhaps has been dormant so
> far. Unfortunately the ToI developer is very busy at the moment, so I will
> have to debug and fix it myself.
> This problem did not occur with 3.7 and the ToI code did not change.
> 
> Do you have any idea where I can start looking for the XFS change in 3.8
> that triggered this behaviour in ToI? Or maybe it was a VFS change?

It's almost certainly an XFS change that triggered it, but it
indicates (once again) that the hibernate code is simply not
quiescing filesystems properly (i.e. by freezing them). The work
that caused this problem is stopped by the filesystem when it
is frozen, and started again when it is thawed...

> PS: the email definitely bounced back, most likely because imageshack is
> blocked on the sgi server:
> 
> Technical details of permanent failure:
> Google tried to deliver your message, but it was rejected by the server for
> the recipient domain oss.sgi.com by cuda-allmx.sgi.com. [192.48.176.16].
> 
> The error that the other server returned was:
> 554 rejecting banned content

IOWs, a stupid spam filter.

I'll see if I can get this fixed.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Crash with 3.8.3 and TuxOnIce

[Pedro Ribeiro]

[-- Attachment #1.1: Type: text/plain, Size: 1735 bytes --]

On 21 March 2013 01:01, Dave Chinner <david@fromorbit.com> wrote:

> On Wed, Mar 20, 2013 at 06:01:35PM +0000, Pedro Ribeiro wrote:
> > Thanks for the answer Dave.
> >
> > Yes I would definitely say it's a ToI bug that perhaps has been dormant
> so
> > far. Unfortunately the ToI developer is very busy at the moment, so I
> will
> > have to debug and fix it myself.
> > This problem did not occur with 3.7 and the ToI code did not change.
> >
> > Do you have any idea where I can start looking for the XFS change in 3.8
> > that triggered this behaviour in ToI? Or maybe it was a VFS change?
>
> It's almost certainly an XFS change that triggered it, but it
> indicates (once again) that the hibernate code is simply not
> quiescing filesystems properly (i.e. by freezing them). The work
> that caused this problem is stopped by the filesystem when it
> is frozen, and started again when it is thawed...
>
> > PS: the email definitely bounced back, most likely because imageshack is
> > blocked on the sgi server:
> >
> > Technical details of permanent failure:
> > Google tried to deliver your message, but it was rejected by the server
> for
> > the recipient domain oss.sgi.com by cuda-allmx.sgi.com. [192.48.176.16].
> >
> > The error that the other server returned was:
> > 554 rejecting banned content
>
> IOWs, a stupid spam filter.
>
> I'll see if I can get this fixed.
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
>

Actually I've nailed it down to a commit between 3.7.1 and 3.7.10. I'll do
some git bisection and come back with the results.

Regarding ToI and filesystem freezing, I guess I need to start delving into
the code to see if I can fix it - long but fun journey ahead I guess.

Regards,
Pedro

[-- Attachment #1.2: Type: text/html, Size: 2600 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Crash with 3.8.3 and TuxOnIce

[Pedro Ribeiro]

[-- Attachment #1.1: Type: text/plain, Size: 2829 bytes --]

Hi Dave (and others),

I've pretty much established the responsible: commit
437a255aa23766666aec78af63be4c253faa8d57
(
http://git.kernel.org/cgit/linux/kernel/git/stable/stable-queue.git/tree/releases/3.7.2/xfs-fix-direct-io-nested-transaction-deadlock.patch?id=HEAD
).

Without this patch, the computer does not lock up in hibernate. So I
understand that this is most likely a bug in ToI, not in xfs. Does this
give you a better idea of how to solve the problem? The only xfs-specific
patch in ToI is below:

diff --git a/fs/xfs/xfs_trans_ail.c b/fs/xfs/xfs_trans_ail.c
index 0eda725..55de808 100644
--- a/fs/xfs/xfs_trans_ail.c
+++ b/fs/xfs/xfs_trans_ail.c
@@ -511,6 +511,7 @@ xfsaild(
  struct xfs_ail *ailp = data;
  long tout = 0; /* milliseconds */

+ set_freezable();
  current->flags |= PF_MEMALLOC;

  while (!kthread_should_stop()) {

Looking at the code blindly, it appears to be similar to what goes on in
other filesystems...

Regards,
Pedro


On 21 March 2013 17:45, Pedro Ribeiro <pedrib@gmail.com> wrote:

>
>
>
> On 21 March 2013 01:01, Dave Chinner <david@fromorbit.com> wrote:
>
>> On Wed, Mar 20, 2013 at 06:01:35PM +0000, Pedro Ribeiro wrote:
>> > Thanks for the answer Dave.
>> >
>> > Yes I would definitely say it's a ToI bug that perhaps has been dormant
>> so
>> > far. Unfortunately the ToI developer is very busy at the moment, so I
>> will
>> > have to debug and fix it myself.
>> > This problem did not occur with 3.7 and the ToI code did not change.
>> >
>> > Do you have any idea where I can start looking for the XFS change in 3.8
>> > that triggered this behaviour in ToI? Or maybe it was a VFS change?
>>
>> It's almost certainly an XFS change that triggered it, but it
>> indicates (once again) that the hibernate code is simply not
>> quiescing filesystems properly (i.e. by freezing them). The work
>> that caused this problem is stopped by the filesystem when it
>> is frozen, and started again when it is thawed...
>>
>> > PS: the email definitely bounced back, most likely because imageshack is
>> > blocked on the sgi server:
>> >
>> > Technical details of permanent failure:
>> > Google tried to deliver your message, but it was rejected by the server
>> for
>> > the recipient domain oss.sgi.com by cuda-allmx.sgi.com.
>> [192.48.176.16].
>> >
>> > The error that the other server returned was:
>> > 554 rejecting banned content
>>
>> IOWs, a stupid spam filter.
>>
>> I'll see if I can get this fixed.
>>
>> Cheers,
>>
>> Dave.
>> --
>> Dave Chinner
>> david@fromorbit.com
>>
>
> Actually I've nailed it down to a commit between 3.7.1 and 3.7.10. I'll do
> some git bisection and come back with the results.
>
> Regarding ToI and filesystem freezing, I guess I need to start delving
> into the code to see if I can fix it - long but fun journey ahead I guess.
>
> Regards,
> Pedro
>
>

[-- Attachment #1.2: Type: text/html, Size: 4841 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Crash with 3.8.3 and TuxOnIce

[Ben Myers]

On Tue, Mar 19, 2013 at 11:31:16PM +0000, Pedro Ribeiro wrote:
> I'm using a TuxOnIce enabled kernel, and recently moved from 3.7.1 to
> 3.8.3. The former used to work nicely, but now I get a hard crash when I
> try to hibernate.
> This is solely related to TuxOnIce as it does not happen in the default
> hibernation.
> 
> I know this is an unsupported out of tree patch, but can you please help me
> debug it or point in the right direction?
> 
> Unfortunately as this was a kernel crash all I have are crappy photos, but
> they do show a readable stack trace:
> img18[dot]imageshack[dot]us/img18/395/imag0228hp[dot]jpg
> img13[dot]imageshack[dot]us/img13/4375/imag0231p[dot]jpg

The admin made a change change in mailer settings.  Lets see if it works now:
http://img18.imageshack.us/img18/395/imag0228hp.jpg

-Ben

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Crash with 3.8.3 and TuxOnIce

[Dave Chinner]

On Wed, Mar 27, 2013 at 09:58:43PM +0000, Pedro Ribeiro wrote:
> Hi Dave (and others),
> 
> I've pretty much established the responsible: commit
> 437a255aa23766666aec78af63be4c253faa8d57
> (
> http://git.kernel.org/cgit/linux/kernel/git/stable/stable-queue.git/tree/releases/3.7.2/xfs-fix-direct-io-nested-transaction-deadlock.patch?id=HEAD
> ).

Seems completely unrelated to the problem you saw.

> Without this patch, the computer does not lock up in hibernate. So I
> understand that this is most likely a bug in ToI, not in xfs. Does this
> give you a better idea of how to solve the problem?

No.

> The only xfs-specific
> patch in ToI is below:
> 
> diff --git a/fs/xfs/xfs_trans_ail.c b/fs/xfs/xfs_trans_ail.c
> index 0eda725..55de808 100644
> --- a/fs/xfs/xfs_trans_ail.c
> +++ b/fs/xfs/xfs_trans_ail.c
> @@ -511,6 +511,7 @@ xfsaild(
>   struct xfs_ail *ailp = data;
>   long tout = 0; /* milliseconds */
> 
> + set_freezable();
>   current->flags |= PF_MEMALLOC;

We do not ever set the PF_NOFREEZE, so set_freezable() is a no-op.
If ToI has introduced new freeze API dependencies, then I'm not
going to try to understand or fix them.

>   while (!kthread_should_stop()) {
> 
> Looking at the code blindly, it appears to be similar to what goes on in
> other filesystems...

That loop has a call to try_to_freeze() in it, which is how such
kthreads are supposed to handle freezing. i.e. once they enter a
state in which they can freeze, they call try_to_freeze() and then
get moved to the refrigerator.

You need to talk to the ToI developers...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Crash with 3.8.3 and TuxOnIce

[Dave Chinner]

On Wed, Mar 27, 2013 at 06:39:12PM -0500, Ben Myers wrote:
> On Tue, Mar 19, 2013 at 11:31:16PM +0000, Pedro Ribeiro wrote:
> > I'm using a TuxOnIce enabled kernel, and recently moved from 3.7.1 to
> > 3.8.3. The former used to work nicely, but now I get a hard crash when I
> > try to hibernate.
> > This is solely related to TuxOnIce as it does not happen in the default
> > hibernation.
> > 
> > I know this is an unsupported out of tree patch, but can you please help me
> > debug it or point in the right direction?
> > 
> > Unfortunately as this was a kernel crash all I have are crappy photos, but
> > they do show a readable stack trace:
> > img18[dot]imageshack[dot]us/img18/395/imag0228hp[dot]jpg
> > img13[dot]imageshack[dot]us/img13/4375/imag0231p[dot]jpg
> 
> The admin made a change change in mailer settings.  Lets see if it works now:
> http://img18.imageshack.us/img18/395/imag0228hp.jpg

Thanks, Ben. it works just fine.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs