cmds segfault on pdj-fstest (ceph.git unstable)

cmds segfault on pdj-fstest (ceph.git unstable)

[Thomas Mueller]

hi 

since this morning cmds segfaults on the pdj-fstest. 
(git rev 12798de37169f6561f0dc7643ca626957f7ab37e)

What happend:
<start server with vstart.sh>
<mount>
08:39:08 [  176.460708] ceph: loaded (mon/mds/osd proto 15/32/24, osdmap 5/5 5/5)
08:39:08 [  176.463965] ceph: client4101 fsid a7b6294d-a074-8e0e-2cee-a48fe2f85c86
08:39:08 [  176.464240] ceph: mon1 127.0.0.1:6789 session established
<start pdj-fstest>
08:40:51 [  279.406980] cmds[6675]: segfault at 0 ip 00000000004bddbd sp 00007f6861ba34e0 error 4 in cmds[400000+331000]
08:40:51 [  279.409895] ceph: mds0 127.0.0.1:6802 socket closed
08:40:51 [  279.804386] ceph: mds0 127.0.0.1:6802 connection failed
08:40:52 [  280.804383] ceph: mds0 127.0.0.1:6802 connection failed
08:40:54 [  282.804381] ceph: mds0 127.0.0.1:6802 connection failed
08:40:58 [  286.804387] ceph: mds0 127.0.0.1:6802 connection failed
08:41:06 [  294.804793] ceph: mds0 127.0.0.1:6802 connection failed
<killed pdj-fstest>
08:41:22 [  310.804820] ceph: mds0 127.0.0.1:6802 connection failed
08:41:38 [  326.804288] ceph: mds0 caps stale
08:41:53 [  341.804327] ceph: mds0 caps stale
08:41:54 [  342.804245] ceph: mds0 127.0.0.1:6802 connection failed


the segfault is reproducable on my machine. yesterday 
(git rev  58fe4b8d5441982d0103817666a6d1bfdd8e04cd) this error was not present.


- Thomas

Re: cmds segfault on pdj-fstest (ceph.git unstable)

[Thomas Mueller]

Am Wed, 09 Jun 2010 06:48:58 +0000 schrieb Thomas Mueller:

> hi
> 
> since this morning cmds segfaults on the pdj-fstest. (git rev
> 12798de37169f6561f0dc7643ca626957f7ab37e)
...
> 
> the segfault is reproducable on my machine. yesterday (git rev 
> 58fe4b8d5441982d0103817666a6d1bfdd8e04cd) this error was not present.
> 
> 
> - Thomas

tried to track down where that segfault comes from - with "git bisect" 
(my first git bisect try)

the output after a while:

2a88e2e54ee0b9449e86cec02315e2809b75ca8b is the first bad commit
commit 2a88e2e54ee0b9449e86cec02315e2809b75ca8b
Author: Greg Farnum <gregf@hq.newdream.net>
Date:   Mon Jun 7 10:04:33 2010 -0700

    add checks for being a snapshot root to dir_is_nonempty

:040000 040000 6c45c68b84800b105299133374f0f91d82948659 2c1b290125ae435679ab29d6fbab7f5b8f81e4b9 M	src

Re: cmds segfault on pdj-fstest (ceph.git unstable)

[Henry C Chang]

>
>    add checks for being a snapshot root to dir_is_nonempty
>
> :040000 040000 6c45c68b84800b105299133374f0f91d82948659 2c1b290125ae435679ab29d6fbab7f5b8f81e4b9 M      src
>

I also got this segfault today. The segfault is caused by:
_dir_is_nonempty() has already done reply_request(), but the caller,
handle_client_unlink(), did it again.

I suggest to move out reply_request() from _dir_is_nonempty(), and
leave it to the caller.
Maybe as below:

Henry


===========================================
diff --git a/src/mds/Server.cc b/src/mds/Server.cc
index 29d0a32..4a90f4e 100644
--- a/src/mds/Server.cc
+++ b/src/mds/Server.cc
@@ -3976,7 +3976,6 @@ bool Server::_dir_is_nonempty(MDRequest *mdr, CInode *in)
               << dir->inode->get_projected_inode()->dirstat.size()
               << " on " << *dir->inode
               << dendl;
-      reply_request(mdr, -ENOTEMPTY);
       return true;
     }

@@ -4070,8 +4069,10 @@ void Server::handle_client_rename(MDRequest *mdr)
     }

     // non-empty dir?
-    if (oldin->is_dir() && _dir_is_nonempty(mdr, oldin))
+    if (oldin->is_dir() && _dir_is_nonempty(mdr, oldin)) {
+      reply_request(mdr, -ENOTEMPTY);
       return;
+    }
   }

   // -- some sanity checks --

Re: cmds segfault on pdj-fstest (ceph.git unstable)

[Gregory Farnum]

Thanks guys, fixed in 8be5b029d6dc4c5a1b6314eebf026907ccaf811a and
673fdc7aa29adb0826ad71b7c8faf32e1480267b.
(Would've just pushed yours, Harry, but I didn't see it in time -- sorry!)
-Greg

On Thu, Jun 10, 2010 at 4:38 AM, Henry C Chang
<henry_c_chang@tcloudcomputing.com> wrote:
>>
>>    add checks for being a snapshot root to dir_is_nonempty
>>
>> :040000 040000 6c45c68b84800b105299133374f0f91d82948659 2c1b290125ae435679ab29d6fbab7f5b8f81e4b9 M      src
>>
>
> I also got this segfault today. The segfault is caused by:
> _dir_is_nonempty() has already done reply_request(), but the caller,
> handle_client_unlink(), did it again.
>
> I suggest to move out reply_request() from _dir_is_nonempty(), and
> leave it to the caller.
> Maybe as below:
>
> Henry
>
>
> ===========================================
> diff --git a/src/mds/Server.cc b/src/mds/Server.cc
> index 29d0a32..4a90f4e 100644
> --- a/src/mds/Server.cc
> +++ b/src/mds/Server.cc
> @@ -3976,7 +3976,6 @@ bool Server::_dir_is_nonempty(MDRequest *mdr, CInode *in)
>               << dir->inode->get_projected_inode()->dirstat.size()
>               << " on " << *dir->inode
>               << dendl;
> -      reply_request(mdr, -ENOTEMPTY);
>       return true;
>     }
>
> @@ -4070,8 +4069,10 @@ void Server::handle_client_rename(MDRequest *mdr)
>     }
>
>     // non-empty dir?
> -    if (oldin->is_dir() && _dir_is_nonempty(mdr, oldin))
> +    if (oldin->is_dir() && _dir_is_nonempty(mdr, oldin)) {
> +      reply_request(mdr, -ENOTEMPTY);
>       return;
> +    }
>   }
>
>   // -- some sanity checks --
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: cmds segfault on pdj-fstest (ceph.git unstable)

[Gregory Farnum]

On Thu, Jun 10, 2010 at 9:11 AM, Gregory Farnum <gregf@hq.newdream.net> wrote:
> (Would've just pushed yours, Harry, but I didn't see it in time -- sorry!)
Apparently I can't do anything right this morning -- sorry twice, Henry!

Re: cmds segfault on pdj-fstest (ceph.git unstable)

[Henry C Chang]

On Fri, Jun 11, 2010 at 12:26 AM, Gregory Farnum <gregf@hq.newdream.net> wrote:
> On Thu, Jun 10, 2010 at 9:11 AM, Gregory Farnum <gregf@hq.newdream.net> wrote:
>> (Would've just pushed yours, Harry, but I didn't see it in time -- sorry!)
> Apparently I can't do anything right this morning -- sorry twice, Henry!
>

No problem! Never mind. :)

Henry