On Tue, Feb 23, 2010 at= 6:11 AM, Talyansky, Roman <roman.talyansky@sap.com> wrote:
Hi Sage,

As you advised us, we switched to the release 0.19 of ceph and ran into ano= ther bug in the ceph client. When writing to a file with the O_SYNC flag, = =A0"0" is always returned although the data is written to disk. This poses a problem in our benchmark which uses the return value as number= of bytes written. Also it seems that such behavior infringes the POSIX wri= te() contract.

Yeah, thanks. A fix = was pushed to the unstable branch. We will probably start maintaining a sta= ble version that will contain such fixes, but you can apply this in the mea= n time:

diff --git a/fs/ceph/file.c b/fs/ceph/file.c

index 2c4ae44..88932c9 100644

--- a/fs/ceph/file.c

+++ b/fs/ceph/file.c

@@ -807,7 +807,7 @@ static ssize_t ceph_aio= _write(struct kiocb *iocb, const struct iovec *iov,

=A0=A0 =A0 =A0 =A0struct ceph_osd_client *osdc =3D &ceph_client(in= ode->i_sb)->osdc;

=A0=A0 =A0 =A0 =A0loff_t endoff =3D pos += iov->iov_len;

=A0=A0 =A0 =A0 =A0int got =3D 0;

- = =A0 =A0 =A0 int ret;

+ =A0 =A0 =A0 int ret, err;

=A0=A0 =A0 =A0 =A0if (ceph_snap(inode) !=3D CEPH_NOSNAP= )

=A0=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0return -EROFS;

@@ -= 838,9 +838,12 @@ retry_snap:

=A0=A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0if ((ret >=3D 0 || ret =3D=3D -EIOCBQUEUED) &&

=A0=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0((file->f_flags & O_S= YNC) || IS_SYNC(file->f_mapping->host)

- =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0|| ceph_osdmap_flag(osdc->osdmap, CEPH_OSDMAP_NEA= RFULL)))

- =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 ret =3D vf= s_fsync_range(file, file->f_path.dentry,

+ =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0|| ceph_osdmap_flag(osdc->= osdmap, CEPH_OSDMAP_NEARFULL))) {

+ =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 err =3D vfs_fsync_range(file, file->f_path.dentry,

=A0=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0pos, pos + ret - 1, 1);

+ =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (err < 0)

+ =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 ret =3D err= ;

+ =A0 =A0 =A0 =A0 =A0 =A0 =A0 }

=A0=A0 =A0 =A0 =A0}

=A0=A0 =A0 =A0 =A0if (ret >=3D 0) {

=A0=A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0spin_lock(&inode->i_lock);

Yehuda

<= /div> --001636b14b85b6f4cb048048795f-- --===============1466363504195151363== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev --===============1466363504195151363== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ceph-devel mailing list Ceph-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ceph-devel --===============1466363504195151363==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Talyansky, Roman" Subject: Re: Write operation is stuck Date: Wed, 24 Feb 2010 14:34:23 +0100 Message-ID: References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1446135999712393956==" Return-path: In-Reply-To: Content-Language: en-US List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ceph-devel-bounces@lists.sourceforge.net To: Yehuda Sadeh Weinraub Cc: Sage Weil , "ceph-devel@lists.sourceforge.net" List-Id: ceph-devel.vger.kernel.org --===============1446135999712393956== Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_C6A64D82E3A5D24B949315CFBC1FA1AD072A3B5200DEWDFECCR01wd_" --_000_C6A64D82E3A5D24B949315CFBC1FA1AD072A3B5200DEWDFECCR01wd_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Yehuda, Thanks for the info on the fix. I'll incorporate it into the code and rerun= the experiments. It also seems that the code at that location became a bit more complex - ne= w #if occurred: #if LINUX_VERSION_CODE >=3D KERNEL_VERSION(2, 6, 32) And consequently the code under #else should be fixed as well. Thanks, Roman From: Yehuda Sadeh Weinraub [mailto:yehudasa@gmail.com] Sent: Tuesday, February 23, 2010 8:11 PM To: Talyansky, Roman Cc: Sage Weil; ceph-devel@lists.sourceforge.net Subject: Re: [ceph-devel] Write operation is stuck On Tue, Feb 23, 2010 at 6:11 AM, Talyansky, Roman > wrote: Hi Sage, As you advised us, we switched to the release 0.19 of ceph and ran into ano= ther bug in the ceph client. When writing to a file with the O_SYNC flag, = "0" is always returned although the data is written to disk. This poses a problem in our benchmark which uses the return value as number= of bytes written. Also it seems that such behavior infringes the POSIX wri= te() contract. Yeah, thanks. A fix was pushed to the unstable branch. We will probably sta= rt maintaining a stable version that will contain such fixes, but you can a= pply this in the mean time: diff --git a/fs/ceph/file.c b/fs/ceph/file.c index 2c4ae44..88932c9 100644 --- a/fs/ceph/file.c +++ b/fs/ceph/file.c @@ -807,7 +807,7 @@ static ssize_t ceph_aio_write(struct kiocb *iocb, const= struct iovec *iov, struct ceph_osd_client *osdc =3D &ceph_client(inode->i_sb)->osdc; loff_t endoff =3D pos + iov->iov_len; int got =3D 0; - int ret; + int ret, err; if (ceph_snap(inode) !=3D CEPH_NOSNAP) return -EROFS; @@ -838,9 +838,12 @@ retry_snap: if ((ret >=3D 0 || ret =3D=3D -EIOCBQUEUED) && ((file->f_flags & O_SYNC) || IS_SYNC(file->f_mapping->h= ost) - || ceph_osdmap_flag(osdc->osdmap, CEPH_OSDMAP_NEARFULL= ))) - ret =3D vfs_fsync_range(file, file->f_path.dentry, + || ceph_osdmap_flag(osdc->osdmap, CEPH_OSDMAP_NEARFULL= ))) { + err =3D vfs_fsync_range(file, file->f_path.dentry, pos, pos + ret - 1, 1); + if (err < 0) + ret =3D err; + } } if (ret >=3D 0) { spin_lock(&inode->i_lock); Yehuda --_000_C6A64D82E3A5D24B949315CFBC1FA1AD072A3B5200DEWDFECCR01wd_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hi Yehuda,

Thanks for the info on the fix. I’ll incorporate it in= to the code and rerun the experiments.

It also seems that the code at that location became a bit mo= re complex – new #if occurred:

#if LINUX_VERSION_CODE &g= t;=3D KERNEL_VERSION(2, 6, 32)

And consequently the code under #else should be fixed as wel= l.

Thanks,

Roman

From: Yehuda Sadeh Weinraub [mailto:yehudasa@gmail.com]
Sent: Tuesday, February 23, 2010 8:11 PM
To: Talyansky, Roman
Cc: Sage Weil; ceph-devel@lists.sourceforge.net
Subject: Re: [ceph-devel] Write operation is stuck=

On Tue, Feb 23, 2010 at 6:11 AM, Talyansky, Roman <= roman.talyansky@sap.com> wro= te:

Hi Sage,

As you advised us, we switched to the release 0.19 of ceph and ran into ano= ther bug in the ceph client. When writing to a file with the O_SYNC flag, "0" is always returned although the data is written to disk= .
This poses a problem in our benchmark which uses the return value as number= of bytes written. Also it seems that such behavior infringes the POSIX write() contract.

Yeah, thanks. A fix was pushed to the unstable branch.= We will probably start maintaining a stable version that will contain such fix= es, but you can apply this in the mean time:

diff --git a/fs/ceph/file.c b/fs/ceph/file.c

index 2c4ae44..88932c9 100644

--- a/fs/ceph/file.c

+++ b/fs/ceph/file.c

@@ -807,7 +807,7 @@ static ssize_t ceph_aio_write(stru= ct kiocb *iocb, const struct iovec *iov,

struct ceph_osd_clien= t *osdc =3D &ceph_client(inode->i_sb)->osdc;

loff_t endoff =3D pos= + iov->iov_len;

int got =3D 0;

- int ret;

+ int ret, err;

if (ceph_snap(inode) = !=3D CEPH_NOSNAP)

return -EROFS;

@@ -838,9 +838,12 @@ retry_snap:

if ((ret >=3D 0 || ret =3D=3D -EIOCBQUEUED) &&<= /p>

((file->f_flags & O_SYNC) || IS_SYNC(file->f_mapping->host)

- &nb= sp; || ceph_osdmap_flag(osdc->osdmap, CEPH_OSDMAP_NEARFULL)))

- &nb= sp; ret =3D vfs_fsync_range(file, file->f_path.dentry,<= o:p>

+ &nb= sp; || ceph_osdmap_flag(osdc->osdmap, CEPH_OSDMAP_NEARFULL))) {=

+ &nb= sp; err =3D vfs_fsync_range(file, file->f_path.dentry,<= o:p>

= ; pos, pos + ret - 1, 1);

+ &nb= sp; if (err < 0)

+ &nb= sp; ret =3D err;

+ }

}

if (ret >=3D 0) {<= o:p>

spin_lock(&inode->i_lock);

Yehuda

--_000_C6A64D82E3A5D24B949315CFBC1FA1AD072A3B5200DEWDFECCR01wd_-- --===============1446135999712393956== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev --===============1446135999712393956== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ceph-devel mailing list Ceph-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ceph-devel --===============1446135999712393956==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sage Weil Subject: Re: Write operation is stuck Date: Wed, 24 Feb 2010 06:56:13 -0800 (PST) Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ceph-devel-bounces@lists.sourceforge.net To: "Talyansky, Roman" Cc: "ceph-devel@lists.sourceforge.net" List-Id: ceph-devel.vger.kernel.org On Wed, 24 Feb 2010, Talyansky, Roman wrote: > Hi Yehuda, > > Thanks for the info on the fix. I'll incorporate it into the code and rerun the experiments. > It also seems that the code at that location became a bit more complex - new #if occurred: > > #if LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 32) > > And consequently the code under #else should be fixed as well. Yeah. I pushed this fix (and another fix that comes up when there's >1 mds) to the stable 'master' branch of ceph-client.git and ceph-client-standalone.git. The 'master-backport' branch of ceph-client-standalone.git has the backport #ifdefs (and builds back to 2.6.28 or so). Thanks! sage > > Thanks, > > Roman > > From: Yehuda Sadeh Weinraub [mailto:yehudasa@gmail.com] > Sent: Tuesday, February 23, 2010 8:11 PM > To: Talyansky, Roman > Cc: Sage Weil; ceph-devel@lists.sourceforge.net > Subject: Re: [ceph-devel] Write operation is stuck > > > On Tue, Feb 23, 2010 at 6:11 AM, Talyansky, Roman > wrote: > Hi Sage, > > As you advised us, we switched to the release 0.19 of ceph and ran into another bug in the ceph client. When writing to a file with the O_SYNC flag, "0" is always returned although the data is written to disk. > This poses a problem in our benchmark which uses the return value as number of bytes written. Also it seems that such behavior infringes the POSIX write() contract. > > Yeah, thanks. A fix was pushed to the unstable branch. We will probably start maintaining a stable version that will contain such fixes, but you can apply this in the mean time: > > diff --git a/fs/ceph/file.c b/fs/ceph/file.c > index 2c4ae44..88932c9 100644 > --- a/fs/ceph/file.c > +++ b/fs/ceph/file.c > @@ -807,7 +807,7 @@ static ssize_t ceph_aio_write(struct kiocb *iocb, const struct iovec *iov, > struct ceph_osd_client *osdc = &ceph_client(inode->i_sb)->osdc; > loff_t endoff = pos + iov->iov_len; > int got = 0; > - int ret; > + int ret, err; > > if (ceph_snap(inode) != CEPH_NOSNAP) > return -EROFS; > @@ -838,9 +838,12 @@ retry_snap: > > if ((ret >= 0 || ret == -EIOCBQUEUED) && > ((file->f_flags & O_SYNC) || IS_SYNC(file->f_mapping->host) > - || ceph_osdmap_flag(osdc->osdmap, CEPH_OSDMAP_NEARFULL))) > - ret = vfs_fsync_range(file, file->f_path.dentry, > + || ceph_osdmap_flag(osdc->osdmap, CEPH_OSDMAP_NEARFULL))) { > + err = vfs_fsync_range(file, file->f_path.dentry, > pos, pos + ret - 1, 1); > + if (err < 0) > + ret = err; > + } > } > if (ret >= 0) { > spin_lock(&inode->i_lock); > > > > Yehuda > ------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Talyansky, Roman" Subject: Re: Write operation is stuck Date: Wed, 24 Feb 2010 17:42:20 +0100 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: Content-Language: en-US List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ceph-devel-bounces@lists.sourceforge.net To: Sage Weil Cc: "ceph-devel@lists.sourceforge.net" List-Id: ceph-devel.vger.kernel.org Hi Sage, Besides the bug with the return value of write operation, the system hangs in a write operation. You can access the trace files of ceph servers at https://sapmats-de.sap-ag.de/download/download.cgi?id=DHTK24DYMIH8MJEYGMO5GT1GKDLBR9IJDKFAA3R0C4D1JGTYVW The traces were collected without "debug mds = 20" in the configuration file. I still have the hang system available. I also run the systems to regenerate the hang with "debug mds = 20" defined. It would be great if we could have a chat to resolve the hang in about 4 hours. Thanks, Roman -----Original Message----- From: Sage Weil [mailto:sage@newdream.net] Sent: Wednesday, February 24, 2010 4:56 PM To: Talyansky, Roman Cc: Yehuda Sadeh Weinraub; ceph-devel@lists.sourceforge.net Subject: Re: [ceph-devel] Write operation is stuck On Wed, 24 Feb 2010, Talyansky, Roman wrote: > Hi Yehuda, > > Thanks for the info on the fix. I'll incorporate it into the code and rerun the experiments. > It also seems that the code at that location became a bit more complex - new #if occurred: > > #if LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 32) > > And consequently the code under #else should be fixed as well. Yeah. I pushed this fix (and another fix that comes up when there's >1 mds) to the stable 'master' branch of ceph-client.git and ceph-client-standalone.git. The 'master-backport' branch of ceph-client-standalone.git has the backport #ifdefs (and builds back to 2.6.28 or so). Thanks! sage > > Thanks, > > Roman > > From: Yehuda Sadeh Weinraub [mailto:yehudasa@gmail.com] > Sent: Tuesday, February 23, 2010 8:11 PM > To: Talyansky, Roman > Cc: Sage Weil; ceph-devel@lists.sourceforge.net > Subject: Re: [ceph-devel] Write operation is stuck > > > On Tue, Feb 23, 2010 at 6:11 AM, Talyansky, Roman > wrote: > Hi Sage, > > As you advised us, we switched to the release 0.19 of ceph and ran into another bug in the ceph client. When writing to a file with the O_SYNC flag, "0" is always returned although the data is written to disk. > This poses a problem in our benchmark which uses the return value as number of bytes written. Also it seems that such behavior infringes the POSIX write() contract. > > Yeah, thanks. A fix was pushed to the unstable branch. We will probably start maintaining a stable version that will contain such fixes, but you can apply this in the mean time: > > diff --git a/fs/ceph/file.c b/fs/ceph/file.c > index 2c4ae44..88932c9 100644 > --- a/fs/ceph/file.c > +++ b/fs/ceph/file.c > @@ -807,7 +807,7 @@ static ssize_t ceph_aio_write(struct kiocb *iocb, const struct iovec *iov, > struct ceph_osd_client *osdc = &ceph_client(inode->i_sb)->osdc; > loff_t endoff = pos + iov->iov_len; > int got = 0; > - int ret; > + int ret, err; > > if (ceph_snap(inode) != CEPH_NOSNAP) > return -EROFS; > @@ -838,9 +838,12 @@ retry_snap: > > if ((ret >= 0 || ret == -EIOCBQUEUED) && > ((file->f_flags & O_SYNC) || IS_SYNC(file->f_mapping->host) > - || ceph_osdmap_flag(osdc->osdmap, CEPH_OSDMAP_NEARFULL))) > - ret = vfs_fsync_range(file, file->f_path.dentry, > + || ceph_osdmap_flag(osdc->osdmap, CEPH_OSDMAP_NEARFULL))) { > + err = vfs_fsync_range(file, file->f_path.dentry, > pos, pos + ret - 1, 1); > + if (err < 0) > + ret = err; > + } > } > if (ret >= 0) { > spin_lock(&inode->i_lock); > > > > Yehuda > ------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sage Weil Subject: Re: Write operation is stuck Date: Wed, 24 Feb 2010 10:43:34 -0800 (PST) Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ceph-devel-bounces@lists.sourceforge.net To: "Talyansky, Roman" Cc: "ceph-devel@lists.sourceforge.net" List-Id: ceph-devel.vger.kernel.org Hi Roman, On Wed, 24 Feb 2010, Talyansky, Roman wrote: > Hi Sage, > > Besides the bug with the return value of write operation, the system hangs in a write operation. > You can access the trace files of ceph servers at > https://sapmats-de.sap-ag.de/download/download.cgi?id=DHTK24DYMIH8MJEYGMO5GT1GKDLBR9IJDKFAA3R0C4D1JGTYVW The .tar.gz appears to be corrupt (gunzip complains)... :( > The traces were collected without "debug mds = 20" in the configuration > file. I still have the hang system available. I also run the systems to > regenerate the hang with "debug mds = 20" defined. > > It would be great if we could have a chat to resolve the hang in about 4 > hours. Sounds good. We should just be back from lunch, and will be in #ceph on irc.oftc.net, or on jabber! sage ------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Talyansky, Roman" Subject: Re: Write operation is stuck Date: Thu, 25 Feb 2010 00:21:53 +0100 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: Content-Language: en-US List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ceph-devel-bounces@lists.sourceforge.net To: Sage Weil Cc: "ceph-devel@lists.sourceforge.net" List-Id: ceph-devel.vger.kernel.org Hi Sage, That's right the file was corrupted. Currently I have extremely slow network. So probably I'll open access to the non-corrupted file tomorrow. Meanwhile the system with higher trace level also hangs and I'll be able to send you more informative traces. Thanks, Roman -----Original Message----- From: Sage Weil [mailto:sage@newdream.net] Sent: Wednesday, February 24, 2010 8:44 PM To: Talyansky, Roman Cc: ceph-devel@lists.sourceforge.net Subject: Re: [ceph-devel] Write operation is stuck Hi Roman, On Wed, 24 Feb 2010, Talyansky, Roman wrote: > Hi Sage, > > Besides the bug with the return value of write operation, the system hangs in a write operation. > You can access the trace files of ceph servers at > https://sapmats-de.sap-ag.de/download/download.cgi?id=DHTK24DYMIH8MJEYGMO5GT1GKDLBR9IJDKFAA3R0C4D1JGTYVW The .tar.gz appears to be corrupt (gunzip complains)... :( > The traces were collected without "debug mds = 20" in the configuration > file. I still have the hang system available. I also run the systems to > regenerate the hang with "debug mds = 20" defined. > > It would be great if we could have a chat to resolve the hang in about 4 > hours. Sounds good. We should just be back from lunch, and will be in #ceph on irc.oftc.net, or on jabber! sage ------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Talyansky, Roman" Subject: Re: Write operation is stuck Date: Thu, 25 Feb 2010 11:07:08 +0100 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: Content-Language: en-US List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ceph-devel-bounces@lists.sourceforge.net To: Sage Weil Cc: "ceph-devel@lists.sourceforge.net" List-Id: ceph-devel.vger.kernel.org Hi Sage, The file with the new traces can be found at https://sapmats-de.sap-ag.de/download/download.cgi?id=A90NAAZ5KZG8IXQP7Z9Z084IKX6HF47GE2OEEPT740RBRSGJNO Thanks, Roman -----Original Message----- From: Sage Weil [mailto:sage@newdream.net] Sent: Wednesday, February 24, 2010 8:44 PM To: Talyansky, Roman Cc: ceph-devel@lists.sourceforge.net Subject: Re: [ceph-devel] Write operation is stuck Hi Roman, On Wed, 24 Feb 2010, Talyansky, Roman wrote: > Hi Sage, > > Besides the bug with the return value of write operation, the system hangs in a write operation. > You can access the trace files of ceph servers at > https://sapmats-de.sap-ag.de/download/download.cgi?id=DHTK24DYMIH8MJEYGMO5GT1GKDLBR9IJDKFAA3R0C4D1JGTYVW The .tar.gz appears to be corrupt (gunzip complains)... :( > The traces were collected without "debug mds = 20" in the configuration > file. I still have the hang system available. I also run the systems to > regenerate the hang with "debug mds = 20" defined. > > It would be great if we could have a chat to resolve the hang in about 4 > hours. Sounds good. We should just be back from lunch, and will be in #ceph on irc.oftc.net, or on jabber! sage ------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev From mboxrd@z Thu Jan 1 00:00:00 1970 From: Bogdan Lobodzinski Subject: Write operation is stuck Date: Fri, 27 Aug 2010 12:18:29 +0000 (UTC) Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Return-path: Received: from lo.gmane.org ([80.91.229.12]:32912 "EHLO lo.gmane.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752524Ab0H0MZG (ORCPT ); Fri, 27 Aug 2010 08:25:06 -0400 Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1OoxzU-0008Km-Kh for ceph-devel@vger.kernel.org; Fri, 27 Aug 2010 14:25:04 +0200 Received: from h1bombeiros.desy.de ([131.169.60.116]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 27 Aug 2010 14:25:04 +0200 Received: from bogdan by h1bombeiros.desy.de with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 27 Aug 2010 14:25:04 +0200 Sender: ceph-devel-owner@vger.kernel.org List-ID: To: ceph-devel@vger.kernel.org Hello, working with ceph on my test configuration (3 nodes Ubuntu 10.04.1 LTS, Linux 2.6.32-24-generic-pae #41-Ubuntu SMP) After starting svn co https://root.cern.ch/svn/root/trunk root on the /ceph directory, the command become stuck, and also: root 5303 0.0 0.0 0 0 ? D Aug26 0:00 [kjournald] root 30181 0.0 0.0 6972 2056 pts/1 D+ 13:46 0:00 /usr//bin/cosd -i 2 -c /etc/ceph/ceph.conf any mount, unmount are going also to the state D. This is a permanennt behaviour of the ceph if the command is started. dmesg shows: ------------- [99048.567704] ------------[ cut here ]------------ [99048.568767] kernel BUG at /build/buildd/linux-2.6.32/fs/ext3/balloc.c:1384! [99048.568767] invalid opcode: 0000 [#1] SMP [99048.568767] last sysfs file: /sys/devices/pci0000:00/0000:00:00.0/device [99048.596652] Modules linked in: nfs lockd nfs_acl auth_rpcgss sunrpc ceph crc32c libcrc32c openafs(P) fbcon tileblit font bitblit softcursor vga vgastate mptctl radeon ttm drm_kms_helper drm bnx2 psmouse i5000_edac edac_core agpgart serio_raw i5k_amb i2c_algo_bit shpchp dell_wmi dcdbas usbhid mptsas mptscsih mptbase scsi_transport_sas [99048.596652] [99048.596652] Pid: 6258, comm: cosd Tainted: P (2.6.32-24-generic-pae #41-Ubuntu) PowerEdge 1950 [99048.596652] EIP: 0060:[] EFLAGS: 00210296 CPU: 3 [99048.596652] EIP is at ext3_try_to_allocate_with_rsv+0x1cd/0x2b0 [99048.596652] EAX: 00000027 EBX: f6dd5480 ECX: fffe48f7 EDX: 00000000 [99048.596652] ESI: 02147fff EDI: f625e200 EBP: f5ccbc54 ESP: f5ccbc14 [99048.596652] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [99048.596652] Process cosd (pid: 6258, ti=f5cca000 task=f6263300 task.ti=f5cca000) [99048.596652] Stack: [99048.596652] 00000428 f14f1bc0 c026cc88 00001000 00000007 f1a80e9c f6dd5494 02147fff [99048.596652] <0> f70d89c0 02140000 00000428 f625d800 00000001 00000428 f1058500 00000000 [99048.596652] <0> f5ccbcc8 c026e048 f14f1bc0 ffffffff f6dd5480 f5ccbcb0 f5ccbcb4 f5ccbc90 [99048.596652] Call Trace: [99048.596652] [] ? read_block_bitmap+0x48/0x160 [99048.596652] [] ? ext3_new_blocks+0x228/0x6c0 [99048.596652] [] ? mb_cache_entry_find_first+0x67/0x80 [99048.596652] [] ? ext3_new_block+0x25/0x30 [99048.596652] [] ? ext3_xattr_block_set+0x554/0x670 [99048.596652] [] ? ext3_xattr_set_entry+0x29/0x350 [99048.596652] [] ? ext3_xattr_set_handle+0x2cb/0x3e0 [99048.596652] [] ? ext3_xattr_set+0x75/0xc0 [99048.596652] [] ? ext3_xattr_user_set+0x76/0x80 [99048.596652] [] ? generic_setxattr+0x9c/0xb0 [99048.596652] [] ? generic_setxattr+0x0/0xb0 [99048.596652] [] ? __vfs_setxattr_noperm+0x44/0x160 [99048.596652] [] ? cap_inode_setxattr+0x2c/0x60 [99048.596652] [] ? vfs_setxattr+0x91/0xa0 [99048.596652] [] ? setxattr+0xb8/0x110 [99048.596652] [] ? __link_path_walk+0x632/0xca0 [99048.596652] [] ? enqueue_task_fair+0x39/0x80 [99048.596652] [] ? mntput_no_expire+0x1f/0xe0 [99048.596652] [] ? mntput_no_expire+0x1f/0xe0 [99048.596652] [] ? path_put+0x25/0x30 [99048.596652] [] ? putname+0x2b/0x40 [99048.596652] [] ? user_path_at+0x4a/0x80 [99048.596652] [] ? sys_futex+0x72/0x120 [99048.596652] [] ? sys_setxattr+0x83/0x90 [99048.596652] [] ? sysenter_do_call+0x12/0x28 [99048.596652] Code: 83 3a ff ff ff 90 39 45 dc c7 45 0c ff ff ff ff 0f 83 32 ff ff ff 8b 87 84 01 00 00 ba ba c6 5c c0 05 d0 00 00 00 e8 73 f1 ff<0f> 0b eb fe 8b 45 ec 89 55 d4 05 cc 00 00 00 89 45 ec e8 fc 53 [99048.596652] EIP: [] ext3_try_to_allocate_with_rsv+0x1cd/0x2b0 SS:ESP 0068:f5ccbc14 [99049.044090] ---[ end trace 35860103963ee444 ]--- h1farm184# -------------------- my ceph.conf is: ------- [global] pid file = /var/run/ceph/$name.pid debug ms = 1 keyring = /etc/ceph/keyring.bin ; monitors [mon] ;Directory for monitor files mon data = /x02/mon$id debug mon = 20 debug paxos = 20 mon lease wiggle room = 0.5 [mon0] host = h1farm182 mon addr = xxx.xxx.xx.116:6789 [mon1] host = h1farm183 mon addr = xxx.xxx.xx.117:6789 ; metadata servers [mds] debug mds = 20 mds log max segments = 2 keyring = /etc/ceph/keyring.$name [mds0] host = h1farm182 [mds1] host = h1farm183 [osd] sudo = true osd data = /x02/osd$id osd journal = /x02/osd$id/journal osd journal size = 100 keyring = /etc/ceph/keyring.$name debug osd = 20 debug journal = 20 debug filestore = 20 ;osd journal size = 100 [osd0] host = h1farm182 [osd1] host = h1farm183 [osd2] host = h1farm184 ------- Any idea how to improve the situation ? From mboxrd@z Thu Jan 1 00:00:00 1970 From: Wido den Hollander Subject: Re: Write operation is stuck Date: Fri, 27 Aug 2010 17:42:36 +0200 Message-ID: <1282923756.2548.137.camel@wido-laptop.pcextreme.nl> References: Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Return-path: Received: from smtp02.mail.pcextreme.nl ([109.72.87.138]:51330 "EHLO smtp02.mail.pcextreme.nl" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753016Ab0H0Pmi (ORCPT ); Fri, 27 Aug 2010 11:42:38 -0400 In-Reply-To: Sender: ceph-devel-owner@vger.kernel.org List-ID: To: Bogdan Lobodzinski Cc: ceph-devel@vger.kernel.org Hi Bogdan, Are you running your OSD data on ext3? It seems that you are hitting some ext3 bug. Could you try changing to btrfs? This since ext is not yet fully supported. Wido On Fri, 2010-08-27 at 12:18 +0000, Bogdan Lobodzinski wrote: > Hello, > > working with ceph on my test configuration > (3 nodes Ubuntu 10.04.1 LTS, Linux 2.6.32-24-generic-pae #41-Ubuntu SMP) > After starting > svn co https://root.cern.ch/svn/root/trunk root > > on the /ceph directory, the command become stuck, and also: > root 5303 0.0 0.0 0 0 ? D Aug26 0:00 [kjournald] > root 30181 0.0 0.0 6972 2056 pts/1 D+ 13:46 0:00 /usr//bin/cosd > -i 2 -c /etc/ceph/ceph.conf > > any mount, unmount are going also to the state D. > This is a permanennt behaviour of the ceph if the command is started. > > dmesg shows: > ------------- > [99048.567704] ------------[ cut here ]------------ > [99048.568767] kernel BUG at > /build/buildd/linux-2.6.32/fs/ext3/balloc.c:1384! > [99048.568767] invalid opcode: 0000 [#1] SMP > [99048.568767] last sysfs file: /sys/devices/pci0000:00/0000:00:00.0/device > [99048.596652] Modules linked in: nfs lockd nfs_acl auth_rpcgss sunrpc ceph > crc32c libcrc32c openafs(P) fbcon tileblit font bitblit softcursor vga > vgastate mptctl radeon ttm drm_kms_helper drm bnx2 psmouse i5000_edac > edac_core agpgart serio_raw i5k_amb i2c_algo_bit shpchp dell_wmi dcdbas > usbhid mptsas mptscsih mptbase scsi_transport_sas > [99048.596652] > [99048.596652] Pid: 6258, comm: cosd Tainted: P > (2.6.32-24-generic-pae #41-Ubuntu) PowerEdge 1950 > [99048.596652] EIP: 0060:[] EFLAGS: 00210296 CPU: 3 > [99048.596652] EIP is at ext3_try_to_allocate_with_rsv+0x1cd/0x2b0 > [99048.596652] EAX: 00000027 EBX: f6dd5480 ECX: fffe48f7 EDX: 00000000 > [99048.596652] ESI: 02147fff EDI: f625e200 EBP: f5ccbc54 ESP: f5ccbc14 > [99048.596652] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 > [99048.596652] Process cosd (pid: 6258, ti=f5cca000 task=f6263300 > task.ti=f5cca000) > [99048.596652] Stack: > [99048.596652] 00000428 f14f1bc0 c026cc88 00001000 00000007 f1a80e9c > f6dd5494 02147fff > [99048.596652] <0> f70d89c0 02140000 00000428 f625d800 00000001 00000428 > f1058500 00000000 > [99048.596652] <0> f5ccbcc8 c026e048 f14f1bc0 ffffffff f6dd5480 f5ccbcb0 > f5ccbcb4 f5ccbc90 > [99048.596652] Call Trace: > [99048.596652] [] ? read_block_bitmap+0x48/0x160 > [99048.596652] [] ? ext3_new_blocks+0x228/0x6c0 > [99048.596652] [] ? mb_cache_entry_find_first+0x67/0x80 > [99048.596652] [] ? ext3_new_block+0x25/0x30 > [99048.596652] [] ? ext3_xattr_block_set+0x554/0x670 > [99048.596652] [] ? ext3_xattr_set_entry+0x29/0x350 > [99048.596652] [] ? ext3_xattr_set_handle+0x2cb/0x3e0 > [99048.596652] [] ? ext3_xattr_set+0x75/0xc0 > [99048.596652] [] ? ext3_xattr_user_set+0x76/0x80 > [99048.596652] [] ? generic_setxattr+0x9c/0xb0 > [99048.596652] [] ? generic_setxattr+0x0/0xb0 > [99048.596652] [] ? __vfs_setxattr_noperm+0x44/0x160 > [99048.596652] [] ? cap_inode_setxattr+0x2c/0x60 > [99048.596652] [] ? vfs_setxattr+0x91/0xa0 > [99048.596652] [] ? setxattr+0xb8/0x110 > [99048.596652] [] ? __link_path_walk+0x632/0xca0 > [99048.596652] [] ? enqueue_task_fair+0x39/0x80 > [99048.596652] [] ? mntput_no_expire+0x1f/0xe0 > [99048.596652] [] ? mntput_no_expire+0x1f/0xe0 > [99048.596652] [] ? path_put+0x25/0x30 > [99048.596652] [] ? putname+0x2b/0x40 > [99048.596652] [] ? user_path_at+0x4a/0x80 > [99048.596652] [] ? sys_futex+0x72/0x120 > [99048.596652] [] ? sys_setxattr+0x83/0x90 > [99048.596652] [] ? sysenter_do_call+0x12/0x28 > [99048.596652] Code: 83 3a ff ff ff 90 39 45 dc c7 45 0c ff ff ff ff 0f 83 > 32 ff ff ff 8b 87 84 01 00 00 ba ba c6 5c c0 05 d0 00 00 00 e8 73 f1 ff<0f> > 0b eb fe 8b 45 ec 89 55 d4 05 cc 00 00 00 89 45 ec e8 fc 53 > [99048.596652] EIP: [] ext3_try_to_allocate_with_rsv+0x1cd/0x2b0 > SS:ESP 0068:f5ccbc14 > [99049.044090] ---[ end trace 35860103963ee444 ]--- > h1farm184# > -------------------- > > my ceph.conf is: > ------- > [global] > pid file = /var/run/ceph/$name.pid > debug ms = 1 > keyring = /etc/ceph/keyring.bin > ; monitors > [mon] > ;Directory for monitor files > mon data = /x02/mon$id > debug mon = 20 > debug paxos = 20 > mon lease wiggle room = 0.5 > > [mon0] > host = h1farm182 > mon addr = xxx.xxx.xx.116:6789 > [mon1] > host = h1farm183 > mon addr = xxx.xxx.xx.117:6789 > ; metadata servers > [mds] > debug mds = 20 > mds log max segments = 2 > keyring = /etc/ceph/keyring.$name > [mds0] > host = h1farm182 > [mds1] > host = h1farm183 > [osd] > sudo = true > osd data = /x02/osd$id > osd journal = /x02/osd$id/journal > osd journal size = 100 > keyring = /etc/ceph/keyring.$name > debug osd = 20 > debug journal = 20 > debug filestore = 20 > ;osd journal size = 100 > [osd0] > host = h1farm182 > [osd1] > host = h1farm183 > [osd2] > host = h1farm184 > > ------- > > Any idea how to improve the situation ? > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sage Weil Subject: Re: Write operation is stuck Date: Fri, 27 Aug 2010 09:09:03 -0700 (PDT) Message-ID: References: Mime-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Return-path: Received: from cobra.newdream.net ([66.33.216.30]:57368 "EHLO cobra.newdream.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752829Ab0H0QIQ (ORCPT ); Fri, 27 Aug 2010 12:08:16 -0400 In-Reply-To: Sender: ceph-devel-owner@vger.kernel.org List-ID: To: Bogdan Lobodzinski Cc: ceph-devel@vger.kernel.org Hi Bogdan, This is a bug in the ext3 xattr code. It seems to be gone in 2.6.34 and later. Or, you can switch to btrfs! sage On Fri, 27 Aug 2010, Bogdan Lobodzinski wrote: > Hello, > > working with ceph on my test configuration > (3 nodes Ubuntu 10.04.1 LTS, Linux 2.6.32-24-generic-pae #41-Ubuntu SMP) > After starting > svn co https://root.cern.ch/svn/root/trunk root > > on the /ceph directory, the command become stuck, and also: > root 5303 0.0 0.0 0 0 ? D Aug26 0:00 [kjournald] > root 30181 0.0 0.0 6972 2056 pts/1 D+ 13:46 0:00 /usr//bin/cosd > -i 2 -c /etc/ceph/ceph.conf > > any mount, unmount are going also to the state D. > This is a permanennt behaviour of the ceph if the command is started. > > dmesg shows: > ------------- > [99048.567704] ------------[ cut here ]------------ > [99048.568767] kernel BUG at > /build/buildd/linux-2.6.32/fs/ext3/balloc.c:1384! > [99048.568767] invalid opcode: 0000 [#1] SMP > [99048.568767] last sysfs file: /sys/devices/pci0000:00/0000:00:00.0/device > [99048.596652] Modules linked in: nfs lockd nfs_acl auth_rpcgss sunrpc ceph > crc32c libcrc32c openafs(P) fbcon tileblit font bitblit softcursor vga > vgastate mptctl radeon ttm drm_kms_helper drm bnx2 psmouse i5000_edac > edac_core agpgart serio_raw i5k_amb i2c_algo_bit shpchp dell_wmi dcdbas > usbhid mptsas mptscsih mptbase scsi_transport_sas > [99048.596652] > [99048.596652] Pid: 6258, comm: cosd Tainted: P > (2.6.32-24-generic-pae #41-Ubuntu) PowerEdge 1950 > [99048.596652] EIP: 0060:[] EFLAGS: 00210296 CPU: 3 > [99048.596652] EIP is at ext3_try_to_allocate_with_rsv+0x1cd/0x2b0 > [99048.596652] EAX: 00000027 EBX: f6dd5480 ECX: fffe48f7 EDX: 00000000 > [99048.596652] ESI: 02147fff EDI: f625e200 EBP: f5ccbc54 ESP: f5ccbc14 > [99048.596652] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 > [99048.596652] Process cosd (pid: 6258, ti=f5cca000 task=f6263300 > task.ti=f5cca000) > [99048.596652] Stack: > [99048.596652] 00000428 f14f1bc0 c026cc88 00001000 00000007 f1a80e9c > f6dd5494 02147fff > [99048.596652] <0> f70d89c0 02140000 00000428 f625d800 00000001 00000428 > f1058500 00000000 > [99048.596652] <0> f5ccbcc8 c026e048 f14f1bc0 ffffffff f6dd5480 f5ccbcb0 > f5ccbcb4 f5ccbc90 > [99048.596652] Call Trace: > [99048.596652] [] ? read_block_bitmap+0x48/0x160 > [99048.596652] [] ? ext3_new_blocks+0x228/0x6c0 > [99048.596652] [] ? mb_cache_entry_find_first+0x67/0x80 > [99048.596652] [] ? ext3_new_block+0x25/0x30 > [99048.596652] [] ? ext3_xattr_block_set+0x554/0x670 > [99048.596652] [] ? ext3_xattr_set_entry+0x29/0x350 > [99048.596652] [] ? ext3_xattr_set_handle+0x2cb/0x3e0 > [99048.596652] [] ? ext3_xattr_set+0x75/0xc0 > [99048.596652] [] ? ext3_xattr_user_set+0x76/0x80 > [99048.596652] [] ? generic_setxattr+0x9c/0xb0 > [99048.596652] [] ? generic_setxattr+0x0/0xb0 > [99048.596652] [] ? __vfs_setxattr_noperm+0x44/0x160 > [99048.596652] [] ? cap_inode_setxattr+0x2c/0x60 > [99048.596652] [] ? vfs_setxattr+0x91/0xa0 > [99048.596652] [] ? setxattr+0xb8/0x110 > [99048.596652] [] ? __link_path_walk+0x632/0xca0 > [99048.596652] [] ? enqueue_task_fair+0x39/0x80 > [99048.596652] [] ? mntput_no_expire+0x1f/0xe0 > [99048.596652] [] ? mntput_no_expire+0x1f/0xe0 > [99048.596652] [] ? path_put+0x25/0x30 > [99048.596652] [] ? putname+0x2b/0x40 > [99048.596652] [] ? user_path_at+0x4a/0x80 > [99048.596652] [] ? sys_futex+0x72/0x120 > [99048.596652] [] ? sys_setxattr+0x83/0x90 > [99048.596652] [] ? sysenter_do_call+0x12/0x28 > [99048.596652] Code: 83 3a ff ff ff 90 39 45 dc c7 45 0c ff ff ff ff 0f 83 > 32 ff ff ff 8b 87 84 01 00 00 ba ba c6 5c c0 05 d0 00 00 00 e8 73 f1 ff<0f> > 0b eb fe 8b 45 ec 89 55 d4 05 cc 00 00 00 89 45 ec e8 fc 53 > [99048.596652] EIP: [] ext3_try_to_allocate_with_rsv+0x1cd/0x2b0 > SS:ESP 0068:f5ccbc14 > [99049.044090] ---[ end trace 35860103963ee444 ]--- > h1farm184# > -------------------- > > my ceph.conf is: > ------- > [global] > pid file = /var/run/ceph/$name.pid > debug ms = 1 > keyring = /etc/ceph/keyring.bin > ; monitors > [mon] > ;Directory for monitor files > mon data = /x02/mon$id > debug mon = 20 > debug paxos = 20 > mon lease wiggle room = 0.5 > > [mon0] > host = h1farm182 > mon addr = xxx.xxx.xx.116:6789 > [mon1] > host = h1farm183 > mon addr = xxx.xxx.xx.117:6789 > ; metadata servers > [mds] > debug mds = 20 > mds log max segments = 2 > keyring = /etc/ceph/keyring.$name > [mds0] > host = h1farm182 > [mds1] > host = h1farm183 > [osd] > sudo = true > osd data = /x02/osd$id > osd journal = /x02/osd$id/journal > osd journal size = 100 > keyring = /etc/ceph/keyring.$name > debug osd = 20 > debug journal = 20 > debug filestore = 20 > ;osd journal size = 100 > [osd0] > host = h1farm182 > [osd1] > host = h1farm183 > [osd2] > host = h1farm184 > > ------- > > Any idea how to improve the situation ? > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Bogdan Lobodzinski Subject: Re: Write operation is stuck Date: Mon, 30 Aug 2010 17:32:24 +0200 (CEST) Message-ID: References: Mime-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Return-path: Received: from smtp-out-2.desy.de ([131.169.56.85]:43780 "EHLO smtp-out-2.desy.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755620Ab0H3P5u (ORCPT ); Mon, 30 Aug 2010 11:57:50 -0400 Received: from smtp-map-2.desy.de (smtp-map-2.desy.de [131.169.56.67]) by smtp-out-2.desy.de (DESY_OUT_1) with ESMTP id 6CB68C34 for ; Mon, 30 Aug 2010 17:32:24 +0200 (MEST) Received: from adserv71.win.desy.de (adserv71.win.desy.de [131.169.97.57]) by smtp-map-2.desy.de (DESY_MAP_2) with ESMTP id 6296EC2E for ; Mon, 30 Aug 2010 17:32:24 +0200 (MEST) In-Reply-To: Sender: ceph-devel-owner@vger.kernel.org List-ID: To: Sage Weil Cc: ceph-devel@vger.kernel.org Hello Sage, I moved to the kernel 2.6.35, keeping ext3 filesystem. After executing teh same command: svn co https://root.cern.ch/svn/root/trunk root System is again dead. The command and kjournald are stuck bogdan 8539 0.9 0.6 31168 22040 pts/0 DL+ 16:44 0:21 svn co https://root.cern.ch/svn/root/trunk root root 802 0.0 0.0 0 0 ? D 12:59 0:01 [kjournald] Looks like the bug is not fixed, dmesg shows: --------- [14325.304068] kernel BUG at /build/buildd/linux-maverick-2.6.35/fs/ext3/balloc.c:1385! [14325.304191] invalid opcode: 0000 [#1] SMP [14325.304263] last sysfs file: /sys/devices/pci0000:00/0000:00:00.0/device [14325.304266] Modules linked in: nfs lockd fscache nfs_acl auth_rpcgss sunrpc ceph crc32c libcrc32c radeon ttm drm_kms_helper drm mptctl psmouse agpgart i5000_edac usbhid hid edac_core i2c_algo_bit bnx2 i5k_amb dcdbas shpchp serio_raw mptsas mptscsih mptbase scsi_transport_sas [14325.304266] [14325.304266] Pid: 8391, comm: cosd Not tainted 2.6.35-14-generic #20~lucid2-Ubuntu 0DT097/PowerEdge 1950 [14325.304266] EIP: 0060:[] EFLAGS: 00210286 CPU: 1 [14325.304266] EIP is at ext3_try_to_allocate_with_rsv+0x1cd/0x2b0 [14325.304266] EAX: 00000027 EBX: c8641440 ECX: c07d7cfc EDX: 00000000 [14325.304266] ESI: 007b7fff EDI: f640fa00 EBP: f5823c50 ESP: f5823c10 [14325.304266] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [14325.304266] Process cosd (pid: 8391, ti=f5822000 task=f6b7bf70 task.ti=f5822000) [14325.304266] Stack: [14325.304266] 000000f6 c6c11930 c0273a58 00001000 f62e549c 00000007 c8641454 007b7fff [14325.304266] <0> f6f7e420 007b0000 000000f6 f640de00 00000001 000000f6 c4063ec0 00000000 [14325.304266] <0> f5823cc0 c0274daf c6c11930 ffffffff c8641440 f5823ca8 f5823cac c0256017 [14325.304266] Call Trace: [14325.304266] [] ? read_block_bitmap+0x48/0x160 [14325.304266] [] ? ext3_new_blocks+0x1ff/0x610 [14325.304266] [] ? mb_cache_entry_find_first+0x67/0x80 [14325.304266] [] ? ext3_new_block+0x25/0x30 [14325.304266] [] ? ext3_xattr_block_set+0x481/0x550 [14325.304266] [] ? ext3_xattr_set_entry+0x20/0x2f0 [14325.304266] [] ? ext3_xattr_set_handle+0x31b/0x400 [14325.304266] [] ? ext3_xattr_set+0x75/0xc0 [14325.304266] [] ? ext3_xattr_user_set+0x74/0x80 [14325.304266] [] ? generic_setxattr+0x9b/0xb0 [14325.304266] [] ? generic_setxattr+0x0/0xb0 [14325.304266] [] ? __vfs_setxattr_noperm+0x44/0x150 [14325.304266] [] ? cap_inode_setxattr+0x2c/0x60 [14325.304266] [] ? vfs_setxattr+0x91/0xa0 [14325.304266] [] ? setxattr+0xb8/0x110 [14325.304266] [] ? path_to_nameidata+0x1e/0x50 [14325.304266] [] ? link_path_walk+0x412/0x890 [14325.304266] [] ? enqueue_task_fair+0x39/0x80 [14325.304266] [] ? mntput_no_expire+0x1f/0xd0 [14325.304266] [] ? mntput_no_expire+0x1f/0xd0 [14325.304266] [] ? putname+0x2b/0x40 [14325.304266] [] ? user_path_at+0x4a/0x80 [14325.304266] [] ? sys_futex+0x72/0x120 [14325.304266] [] ? sys_setxattr+0x83/0x90 [14325.304266] [] ? syscall_call+0x7/0xb [14325.304266] [] ? cache_add_dev+0x73/0x195 [14325.304266] Code: 83 3a ff ff ff 90 39 45 dc c7 45 0c ff ff ff ff 0f 83 32 ff ff ff 8b 87 80 01 00 00 ba 5a 7e 5e c0 05 d0 00 00 00 e8 83 f1 ff ff <0f> 0b eb fe 8b 45 ec 89 55 d4 05 cc 00 00 00 89 45 ec e8 ec 4b [14325.304266] EIP: [] ext3_try_to_allocate_with_rsv+0x1cd/0x2b0 SS:ESP 0068:f5823c10 [14325.326777] ---[ end trace 53e0b3b55af7a83c ]--- [14384.001261] ceph: mds0 caps stale [14413.616132] ceph: tid 33594 timed out on osd2, will reset osd [14628.992279] ceph: mds0 hung --------- as a next step I wil try to use btrfs . Cheers, Bogdan On Fri, 27 Aug 2010, Sage Weil wrote: > Hi Bogdan, > > This is a bug in the ext3 xattr code. It seems to be gone in 2.6.34 and > later. Or, you can switch to btrfs! > > sage > > > On Fri, 27 Aug 2010, Bogdan Lobodzinski wrote: > >> Hello, >> >> working with ceph on my test configuration >> (3 nodes Ubuntu 10.04.1 LTS, Linux 2.6.32-24-generic-pae #41-Ubuntu SMP) >> After starting >> svn co https://root.cern.ch/svn/root/trunk root >> >> on the /ceph directory, the command become stuck, and also: >> root 5303 0.0 0.0 0 0 ? D Aug26 0:00 [kjournald] >> root 30181 0.0 0.0 6972 2056 pts/1 D+ 13:46 0:00 /usr//bin/cosd >> -i 2 -c /etc/ceph/ceph.conf >> >> any mount, unmount are going also to the state D. >> This is a permanennt behaviour of the ceph if the command is started. >> >> dmesg shows: >> ------------- >> [99048.567704] ------------[ cut here ]------------ >> [99048.568767] kernel BUG at >> /build/buildd/linux-2.6.32/fs/ext3/balloc.c:1384! >> [99048.568767] invalid opcode: 0000 [#1] SMP >> [99048.568767] last sysfs file: /sys/devices/pci0000:00/0000:00:00.0/device >> [99048.596652] Modules linked in: nfs lockd nfs_acl auth_rpcgss sunrpc ceph >> crc32c libcrc32c openafs(P) fbcon tileblit font bitblit softcursor vga >> vgastate mptctl radeon ttm drm_kms_helper drm bnx2 psmouse i5000_edac >> edac_core agpgart serio_raw i5k_amb i2c_algo_bit shpchp dell_wmi dcdbas >> usbhid mptsas mptscsih mptbase scsi_transport_sas >> [99048.596652] >> [99048.596652] Pid: 6258, comm: cosd Tainted: P >> (2.6.32-24-generic-pae #41-Ubuntu) PowerEdge 1950 >> [99048.596652] EIP: 0060:[] EFLAGS: 00210296 CPU: 3 >> [99048.596652] EIP is at ext3_try_to_allocate_with_rsv+0x1cd/0x2b0 >> [99048.596652] EAX: 00000027 EBX: f6dd5480 ECX: fffe48f7 EDX: 00000000 >> [99048.596652] ESI: 02147fff EDI: f625e200 EBP: f5ccbc54 ESP: f5ccbc14 >> [99048.596652] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 >> [99048.596652] Process cosd (pid: 6258, ti=f5cca000 task=f6263300 >> task.ti=f5cca000) >> [99048.596652] Stack: >> [99048.596652] 00000428 f14f1bc0 c026cc88 00001000 00000007 f1a80e9c >> f6dd5494 02147fff >> [99048.596652] <0> f70d89c0 02140000 00000428 f625d800 00000001 00000428 >> f1058500 00000000 >> [99048.596652] <0> f5ccbcc8 c026e048 f14f1bc0 ffffffff f6dd5480 f5ccbcb0 >> f5ccbcb4 f5ccbc90 >> [99048.596652] Call Trace: >> [99048.596652] [] ? read_block_bitmap+0x48/0x160 >> [99048.596652] [] ? ext3_new_blocks+0x228/0x6c0 >> [99048.596652] [] ? mb_cache_entry_find_first+0x67/0x80 >> [99048.596652] [] ? ext3_new_block+0x25/0x30 >> [99048.596652] [] ? ext3_xattr_block_set+0x554/0x670 >> [99048.596652] [] ? ext3_xattr_set_entry+0x29/0x350 >> [99048.596652] [] ? ext3_xattr_set_handle+0x2cb/0x3e0 >> [99048.596652] [] ? ext3_xattr_set+0x75/0xc0 >> [99048.596652] [] ? ext3_xattr_user_set+0x76/0x80 >> [99048.596652] [] ? generic_setxattr+0x9c/0xb0 >> [99048.596652] [] ? generic_setxattr+0x0/0xb0 >> [99048.596652] [] ? __vfs_setxattr_noperm+0x44/0x160 >> [99048.596652] [] ? cap_inode_setxattr+0x2c/0x60 >> [99048.596652] [] ? vfs_setxattr+0x91/0xa0 >> [99048.596652] [] ? setxattr+0xb8/0x110 >> [99048.596652] [] ? __link_path_walk+0x632/0xca0 >> [99048.596652] [] ? enqueue_task_fair+0x39/0x80 >> [99048.596652] [] ? mntput_no_expire+0x1f/0xe0 >> [99048.596652] [] ? mntput_no_expire+0x1f/0xe0 >> [99048.596652] [] ? path_put+0x25/0x30 >> [99048.596652] [] ? putname+0x2b/0x40 >> [99048.596652] [] ? user_path_at+0x4a/0x80 >> [99048.596652] [] ? sys_futex+0x72/0x120 >> [99048.596652] [] ? sys_setxattr+0x83/0x90 >> [99048.596652] [] ? sysenter_do_call+0x12/0x28 >> [99048.596652] Code: 83 3a ff ff ff 90 39 45 dc c7 45 0c ff ff ff ff 0f 83 >> 32 ff ff ff 8b 87 84 01 00 00 ba ba c6 5c c0 05 d0 00 00 00 e8 73 f1 ff<0f> >> 0b eb fe 8b 45 ec 89 55 d4 05 cc 00 00 00 89 45 ec e8 fc 53 >> [99048.596652] EIP: [] ext3_try_to_allocate_with_rsv+0x1cd/0x2b0 >> SS:ESP 0068:f5ccbc14 >> [99049.044090] ---[ end trace 35860103963ee444 ]--- >> h1farm184# >> -------------------- >> >> my ceph.conf is: >> ------- >> [global] >> pid file = /var/run/ceph/$name.pid >> debug ms = 1 >> keyring = /etc/ceph/keyring.bin >> ; monitors >> [mon] >> ;Directory for monitor files >> mon data = /x02/mon$id >> debug mon = 20 >> debug paxos = 20 >> mon lease wiggle room = 0.5 >> >> [mon0] >> host = h1farm182 >> mon addr = xxx.xxx.xx.116:6789 >> [mon1] >> host = h1farm183 >> mon addr = xxx.xxx.xx.117:6789 >> ; metadata servers >> [mds] >> debug mds = 20 >> mds log max segments = 2 >> keyring = /etc/ceph/keyring.$name >> [mds0] >> host = h1farm182 >> [mds1] >> host = h1farm183 >> [osd] >> sudo = true >> osd data = /x02/osd$id >> osd journal = /x02/osd$id/journal >> osd journal size = 100 >> keyring = /etc/ceph/keyring.$name >> debug osd = 20 >> debug journal = 20 >> debug filestore = 20 >> ;osd journal size = 100 >> [osd0] >> host = h1farm182 >> [osd1] >> host = h1farm183 >> [osd2] >> host = h1farm184 >> >> ------- >> >> Any idea how to improve the situation ? >> >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sage Weil Subject: Re: Write operation is stuck Date: Mon, 30 Aug 2010 12:39:11 -0700 (PDT) Message-ID: References: Mime-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Return-path: Received: from cobra.newdream.net ([66.33.216.30]:40336 "EHLO cobra.newdream.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751543Ab0H3TiQ (ORCPT ); Mon, 30 Aug 2010 15:38:16 -0400 In-Reply-To: Sender: ceph-devel-owner@vger.kernel.org List-ID: To: Bogdan Lobodzinski Cc: ceph-devel@vger.kernel.org On Mon, 30 Aug 2010, Bogdan Lobodzinski wrote: > > Hello Sage, > > I moved to the kernel 2.6.35, keeping ext3 filesystem. > After executing teh same command: > svn co https://root.cern.ch/svn/root/trunk root > > System is again dead. The command and kjournald are stuck > bogdan 8539 0.9 0.6 31168 22040 pts/0 DL+ 16:44 0:21 svn co > https://root.cern.ch/svn/root/trunk root > root 802 0.0 0.0 0 0 ? D 12:59 0:01 [kjournald] Hmm. Have you tried ext4? I stopped seeing this on my own machine with recent kernels, but it looks like it isn't in fact fixed. This should be reported to the ext4 list. Are you running ceph via vstart.sh or a custom ceph.conf? sage > > Looks like the bug is not fixed, dmesg shows: > --------- > [14325.304068] kernel BUG at > /build/buildd/linux-maverick-2.6.35/fs/ext3/balloc.c:1385! > [14325.304191] invalid opcode: 0000 [#1] SMP > [14325.304263] last sysfs file: /sys/devices/pci0000:00/0000:00:00.0/device > [14325.304266] Modules linked in: nfs lockd fscache nfs_acl auth_rpcgss sunrpc > ceph crc32c libcrc32c radeon ttm drm_kms_helper drm mptctl psmouse agpgart > i5000_edac usbhid hid edac_core i2c_algo_bit bnx2 i5k_amb dcdbas shpchp > serio_raw mptsas mptscsih mptbase scsi_transport_sas > [14325.304266] > [14325.304266] Pid: 8391, comm: cosd Not tainted 2.6.35-14-generic > #20~lucid2-Ubuntu 0DT097/PowerEdge 1950 > [14325.304266] EIP: 0060:[] EFLAGS: 00210286 CPU: 1 > [14325.304266] EIP is at ext3_try_to_allocate_with_rsv+0x1cd/0x2b0 > [14325.304266] EAX: 00000027 EBX: c8641440 ECX: c07d7cfc EDX: 00000000 > [14325.304266] ESI: 007b7fff EDI: f640fa00 EBP: f5823c50 ESP: f5823c10 > [14325.304266] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 > [14325.304266] Process cosd (pid: 8391, ti=f5822000 task=f6b7bf70 > task.ti=f5822000) > [14325.304266] Stack: > [14325.304266] 000000f6 c6c11930 c0273a58 00001000 f62e549c 00000007 c8641454 > 007b7fff > [14325.304266] <0> f6f7e420 007b0000 000000f6 f640de00 00000001 000000f6 > c4063ec0 00000000 > [14325.304266] <0> f5823cc0 c0274daf c6c11930 ffffffff c8641440 f5823ca8 > f5823cac c0256017 > [14325.304266] Call Trace: > [14325.304266] [] ? read_block_bitmap+0x48/0x160 > [14325.304266] [] ? ext3_new_blocks+0x1ff/0x610 > [14325.304266] [] ? mb_cache_entry_find_first+0x67/0x80 > [14325.304266] [] ? ext3_new_block+0x25/0x30 > [14325.304266] [] ? ext3_xattr_block_set+0x481/0x550 > [14325.304266] [] ? ext3_xattr_set_entry+0x20/0x2f0 > [14325.304266] [] ? ext3_xattr_set_handle+0x31b/0x400 > [14325.304266] [] ? ext3_xattr_set+0x75/0xc0 > [14325.304266] [] ? ext3_xattr_user_set+0x74/0x80 > [14325.304266] [] ? generic_setxattr+0x9b/0xb0 > [14325.304266] [] ? generic_setxattr+0x0/0xb0 > [14325.304266] [] ? __vfs_setxattr_noperm+0x44/0x150 > [14325.304266] [] ? cap_inode_setxattr+0x2c/0x60 > [14325.304266] [] ? vfs_setxattr+0x91/0xa0 > [14325.304266] [] ? setxattr+0xb8/0x110 > [14325.304266] [] ? path_to_nameidata+0x1e/0x50 > [14325.304266] [] ? link_path_walk+0x412/0x890 > [14325.304266] [] ? enqueue_task_fair+0x39/0x80 > [14325.304266] [] ? mntput_no_expire+0x1f/0xd0 > [14325.304266] [] ? mntput_no_expire+0x1f/0xd0 > [14325.304266] [] ? putname+0x2b/0x40 > [14325.304266] [] ? user_path_at+0x4a/0x80 > [14325.304266] [] ? sys_futex+0x72/0x120 > [14325.304266] [] ? sys_setxattr+0x83/0x90 > [14325.304266] [] ? syscall_call+0x7/0xb > [14325.304266] [] ? cache_add_dev+0x73/0x195 > [14325.304266] Code: 83 3a ff ff ff 90 39 45 dc c7 45 0c ff ff ff ff 0f 83 32 > ff ff ff 8b 87 80 01 00 00 ba 5a 7e 5e c0 05 d0 00 00 00 e8 83 f1 ff ff <0f> > 0b eb fe 8b 45 ec 89 55 d4 05 cc 00 00 00 89 45 ec e8 ec 4b > [14325.304266] EIP: [] ext3_try_to_allocate_with_rsv+0x1cd/0x2b0 > SS:ESP 0068:f5823c10 > [14325.326777] ---[ end trace 53e0b3b55af7a83c ]--- > [14384.001261] ceph: mds0 caps stale > [14413.616132] ceph: tid 33594 timed out on osd2, will reset osd > [14628.992279] ceph: mds0 hung > --------- > > as a next step I wil try to use btrfs . > > Cheers, > > Bogdan > > > On Fri, 27 Aug 2010, Sage Weil wrote: > > > Hi Bogdan, > > > > This is a bug in the ext3 xattr code. It seems to be gone in 2.6.34 and > > later. Or, you can switch to btrfs! > > > > sage > > > > > > On Fri, 27 Aug 2010, Bogdan Lobodzinski wrote: > > > > > Hello, > > > > > > working with ceph on my test configuration > > > (3 nodes Ubuntu 10.04.1 LTS, Linux 2.6.32-24-generic-pae #41-Ubuntu SMP) > > > After starting > > > svn co https://root.cern.ch/svn/root/trunk root > > > > > > on the /ceph directory, the command become stuck, and also: > > > root 5303 0.0 0.0 0 0 ? D Aug26 0:00 > > > [kjournald] > > > root 30181 0.0 0.0 6972 2056 pts/1 D+ 13:46 0:00 > > > /usr//bin/cosd > > > -i 2 -c /etc/ceph/ceph.conf > > > > > > any mount, unmount are going also to the state D. > > > This is a permanennt behaviour of the ceph if the command is started. > > > > > > dmesg shows: > > > ------------- > > > [99048.567704] ------------[ cut here ]------------ > > > [99048.568767] kernel BUG at > > > /build/buildd/linux-2.6.32/fs/ext3/balloc.c:1384! > > > [99048.568767] invalid opcode: 0000 [#1] SMP > > > [99048.568767] last sysfs file: > > > /sys/devices/pci0000:00/0000:00:00.0/device > > > [99048.596652] Modules linked in: nfs lockd nfs_acl auth_rpcgss sunrpc > > > ceph > > > crc32c libcrc32c openafs(P) fbcon tileblit font bitblit softcursor vga > > > vgastate mptctl radeon ttm drm_kms_helper drm bnx2 psmouse i5000_edac > > > edac_core agpgart serio_raw i5k_amb i2c_algo_bit shpchp dell_wmi dcdbas > > > usbhid mptsas mptscsih mptbase scsi_transport_sas > > > [99048.596652] > > > [99048.596652] Pid: 6258, comm: cosd Tainted: P > > > (2.6.32-24-generic-pae #41-Ubuntu) PowerEdge 1950 > > > [99048.596652] EIP: 0060:[] EFLAGS: 00210296 CPU: 3 > > > [99048.596652] EIP is at ext3_try_to_allocate_with_rsv+0x1cd/0x2b0 > > > [99048.596652] EAX: 00000027 EBX: f6dd5480 ECX: fffe48f7 EDX: 00000000 > > > [99048.596652] ESI: 02147fff EDI: f625e200 EBP: f5ccbc54 ESP: f5ccbc14 > > > [99048.596652] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 > > > [99048.596652] Process cosd (pid: 6258, ti=f5cca000 task=f6263300 > > > task.ti=f5cca000) > > > [99048.596652] Stack: > > > [99048.596652] 00000428 f14f1bc0 c026cc88 00001000 00000007 f1a80e9c > > > f6dd5494 02147fff > > > [99048.596652] <0> f70d89c0 02140000 00000428 f625d800 00000001 00000428 > > > f1058500 00000000 > > > [99048.596652] <0> f5ccbcc8 c026e048 f14f1bc0 ffffffff f6dd5480 f5ccbcb0 > > > f5ccbcb4 f5ccbc90 > > > [99048.596652] Call Trace: > > > [99048.596652] [] ? read_block_bitmap+0x48/0x160 > > > [99048.596652] [] ? ext3_new_blocks+0x228/0x6c0 > > > [99048.596652] [] ? mb_cache_entry_find_first+0x67/0x80 > > > [99048.596652] [] ? ext3_new_block+0x25/0x30 > > > [99048.596652] [] ? ext3_xattr_block_set+0x554/0x670 > > > [99048.596652] [] ? ext3_xattr_set_entry+0x29/0x350 > > > [99048.596652] [] ? ext3_xattr_set_handle+0x2cb/0x3e0 > > > [99048.596652] [] ? ext3_xattr_set+0x75/0xc0 > > > [99048.596652] [] ? ext3_xattr_user_set+0x76/0x80 > > > [99048.596652] [] ? generic_setxattr+0x9c/0xb0 > > > [99048.596652] [] ? generic_setxattr+0x0/0xb0 > > > [99048.596652] [] ? __vfs_setxattr_noperm+0x44/0x160 > > > [99048.596652] [] ? cap_inode_setxattr+0x2c/0x60 > > > [99048.596652] [] ? vfs_setxattr+0x91/0xa0 > > > [99048.596652] [] ? setxattr+0xb8/0x110 > > > [99048.596652] [] ? __link_path_walk+0x632/0xca0 > > > [99048.596652] [] ? enqueue_task_fair+0x39/0x80 > > > [99048.596652] [] ? mntput_no_expire+0x1f/0xe0 > > > [99048.596652] [] ? mntput_no_expire+0x1f/0xe0 > > > [99048.596652] [] ? path_put+0x25/0x30 > > > [99048.596652] [] ? putname+0x2b/0x40 > > > [99048.596652] [] ? user_path_at+0x4a/0x80 > > > [99048.596652] [] ? sys_futex+0x72/0x120 > > > [99048.596652] [] ? sys_setxattr+0x83/0x90 > > > [99048.596652] [] ? sysenter_do_call+0x12/0x28 > > > [99048.596652] Code: 83 3a ff ff ff 90 39 45 dc c7 45 0c ff ff ff ff 0f 83 > > > 32 ff ff ff 8b 87 84 01 00 00 ba ba c6 5c c0 05 d0 00 00 00 e8 73 f1 > > > ff<0f> > > > 0b eb fe 8b 45 ec 89 55 d4 05 cc 00 00 00 89 45 ec e8 fc 53 > > > [99048.596652] EIP: [] ext3_try_to_allocate_with_rsv+0x1cd/0x2b0 > > > SS:ESP 0068:f5ccbc14 > > > [99049.044090] ---[ end trace 35860103963ee444 ]--- > > > h1farm184# > > > -------------------- > > > > > > my ceph.conf is: > > > ------- > > > [global] > > > pid file = /var/run/ceph/$name.pid > > > debug ms = 1 > > > keyring = /etc/ceph/keyring.bin > > > ; monitors > > > [mon] > > > ;Directory for monitor files > > > mon data = /x02/mon$id > > > debug mon = 20 > > > debug paxos = 20 > > > mon lease wiggle room = 0.5 > > > > > > [mon0] > > > host = h1farm182 > > > mon addr = xxx.xxx.xx.116:6789 > > > [mon1] > > > host = h1farm183 > > > mon addr = xxx.xxx.xx.117:6789 > > > ; metadata servers > > > [mds] > > > debug mds = 20 > > > mds log max segments = 2 > > > keyring = /etc/ceph/keyring.$name > > > [mds0] > > > host = h1farm182 > > > [mds1] > > > host = h1farm183 > > > [osd] > > > sudo = true > > > osd data = /x02/osd$id > > > osd journal = /x02/osd$id/journal > > > osd journal size = 100 > > > keyring = /etc/ceph/keyring.$name > > > debug osd = 20 > > > debug journal = 20 > > > debug filestore = 20 > > > ;osd journal size = 100 > > > [osd0] > > > host = h1farm182 > > > [osd1] > > > host = h1farm183 > > > [osd2] > > > host = h1farm184 > > > > > > ------- > > > > > > Any idea how to improve the situation ? > > > > > > -- > > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > > > the body of a message to majordomo@vger.kernel.org > > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > > > > > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Bogdan Lobodzinski Subject: Re: Write operation is stuck Date: Tue, 31 Aug 2010 09:56:43 +0200 (CEST) Message-ID: References: Mime-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Return-path: Received: from smtp-out-1.desy.de ([131.169.56.84]:47742 "EHLO smtp-out-1.desy.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756897Ab0HaH4p (ORCPT ); Tue, 31 Aug 2010 03:56:45 -0400 Received: from smtp-map-1.desy.de (smtp-map-1.desy.de [131.169.56.66]) by smtp-out-1.desy.de (DESY_OUT_1) with ESMTP id F13A7173C for ; Tue, 31 Aug 2010 09:56:43 +0200 (MEST) Received: from adserv71.win.desy.de (adserv71.win.desy.de [131.169.97.57]) by smtp-map-1.desy.de (DESY_MAP_1) with ESMTP id E573D13E92 for ; Tue, 31 Aug 2010 09:56:43 +0200 (MEST) In-Reply-To: Sender: ceph-devel-owner@vger.kernel.org List-ID: To: Sage Weil Cc: ceph-devel@vger.kernel.org Hello Sage, On Mon, 30 Aug 2010, Sage Weil wrote: > On Mon, 30 Aug 2010, Bogdan Lobodzinski wrote: >> >> Hello Sage, >> >> I moved to the kernel 2.6.35, keeping ext3 filesystem. >> After executing teh same command: >> svn co https://root.cern.ch/svn/root/trunk root >> >> System is again dead. The command and kjournald are stuck >> bogdan 8539 0.9 0.6 31168 22040 pts/0 DL+ 16:44 0:21 svn co >> https://root.cern.ch/svn/root/trunk root >> root 802 0.0 0.0 0 0 ? D 12:59 0:01 [kjournald] > > Hmm. Have you tried ext4? > > I stopped seeing this on my own machine with recent kernels, but it looks > like it isn't in fact fixed. This should be reported to the ext4 list. > Are you running ceph via vstart.sh or a custom ceph.conf? I am using vstart.sh taken from compiled by me source tarball ceph-0.21.tar.gz (http://ceph.newdream.net/download/) and the client from git://git.kernel.org/pub/scm/linux/kernel/git/sage/ceph-client-standalone.git Cheers, Bogdan > > sage > >> >> Looks like the bug is not fixed, dmesg shows: >> --------- >> [14325.304068] kernel BUG at >> /build/buildd/linux-maverick-2.6.35/fs/ext3/balloc.c:1385! >> [14325.304191] invalid opcode: 0000 [#1] SMP >> [14325.304263] last sysfs file: /sys/devices/pci0000:00/0000:00:00.0/device >> [14325.304266] Modules linked in: nfs lockd fscache nfs_acl auth_rpcgss sunrpc >> ceph crc32c libcrc32c radeon ttm drm_kms_helper drm mptctl psmouse agpgart >> i5000_edac usbhid hid edac_core i2c_algo_bit bnx2 i5k_amb dcdbas shpchp >> serio_raw mptsas mptscsih mptbase scsi_transport_sas >> [14325.304266] >> [14325.304266] Pid: 8391, comm: cosd Not tainted 2.6.35-14-generic >> #20~lucid2-Ubuntu 0DT097/PowerEdge 1950 >> [14325.304266] EIP: 0060:[] EFLAGS: 00210286 CPU: 1 >> [14325.304266] EIP is at ext3_try_to_allocate_with_rsv+0x1cd/0x2b0 >> [14325.304266] EAX: 00000027 EBX: c8641440 ECX: c07d7cfc EDX: 00000000 >> [14325.304266] ESI: 007b7fff EDI: f640fa00 EBP: f5823c50 ESP: f5823c10 >> [14325.304266] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 >> [14325.304266] Process cosd (pid: 8391, ti=f5822000 task=f6b7bf70 >> task.ti=f5822000) >> [14325.304266] Stack: >> [14325.304266] 000000f6 c6c11930 c0273a58 00001000 f62e549c 00000007 c8641454 >> 007b7fff >> [14325.304266] <0> f6f7e420 007b0000 000000f6 f640de00 00000001 000000f6 >> c4063ec0 00000000 >> [14325.304266] <0> f5823cc0 c0274daf c6c11930 ffffffff c8641440 f5823ca8 >> f5823cac c0256017 >> [14325.304266] Call Trace: >> [14325.304266] [] ? read_block_bitmap+0x48/0x160 >> [14325.304266] [] ? ext3_new_blocks+0x1ff/0x610 >> [14325.304266] [] ? mb_cache_entry_find_first+0x67/0x80 >> [14325.304266] [] ? ext3_new_block+0x25/0x30 >> [14325.304266] [] ? ext3_xattr_block_set+0x481/0x550 >> [14325.304266] [] ? ext3_xattr_set_entry+0x20/0x2f0 >> [14325.304266] [] ? ext3_xattr_set_handle+0x31b/0x400 >> [14325.304266] [