Filesystems for ceph

Filesystems for ceph

[Christian Brunner]

We are having quite some problems with the underlying filesystem for
the cosd's and I would like to hear about other experiences. Here is
what we have gone through so far:

btrfs with 2.6.38:

- good performance
- frequently hitting of various BUG_ON conditions

btrfs with 2.6.39:

- big performance problems after a few days uptime
- occasionally hitting BUG_ON conditions

btrfs with 3.0:

- big performance problems after a few days uptime
- occasionally hitting a deadlock in the btrfs filesystem (cosd is in D-state)

ext4 with a RHEL6.0 kernel (don't remember exactly):

- almost immediate blowup of the kernel (OOPS)

From what I read in Fyodors emails ext4 in 2.6.39 isn't much better.

So, what filesystem would you recommend?

Thanks,
Christian

Re: Filesystems for ceph

[Fyodor Ustinov]

Christian Brunner <chb <at> muc.de> writes:

> 
> From what I read in Fyodors emails ext4 in 2.6.39 isn't much better.
> 
> So, what filesystem would you recommend?
> 

Christian, IMHO ext3/ext4/btrfs unusable for ceph. I tested some time xfs, and 
think to use it again. But I think that the ceph developers do not like xfs:)

WBR,
    Fyodor.

Re: Filesystems for ceph

[Sage Weil]

On Wed, 27 Jul 2011, Fyodor Ustinov wrote:
> Christian Brunner <chb <at> muc.de> writes:
> 
> > 
> > From what I read in Fyodors emails ext4 in 2.6.39 isn't much better.
> > 
> > So, what filesystem would you recommend?
> > 
> 
> Christian, IMHO ext3/ext4/btrfs unusable for ceph. I tested some time xfs, and 
> think to use it again. But I think that the ceph developers do not like xfs:)

What problems have you seen with ext4?

sage

Re: Filesystems for ceph

[Gregory Farnum]

Our largest installation hasn't seen any problems with btrfs on
2.6.38.6, although it may not be as busy as the clusters you're
running on.

I would recommend trying to use a newish stable release though, since
the major releases I think are still introducing new features into
btrfs, and code churn always creates bugs.
-Greg

On Wed, Jul 27, 2011 at 10:52 AM, Christian Brunner <chb@muc.de> wrote:
> We are having quite some problems with the underlying filesystem for
> the cosd's and I would like to hear about other experiences. Here is
> what we have gone through so far:
>
> btrfs with 2.6.38:
>
> - good performance
> - frequently hitting of various BUG_ON conditions
>
> btrfs with 2.6.39:
>
> - big performance problems after a few days uptime
> - occasionally hitting BUG_ON conditions
>
> btrfs with 3.0:
>
> - big performance problems after a few days uptime
> - occasionally hitting a deadlock in the btrfs filesystem (cosd is in D-state)
>
> ext4 with a RHEL6.0 kernel (don't remember exactly):
>
> - almost immediate blowup of the kernel (OOPS)
>
> From what I read in Fyodors emails ext4 in 2.6.39 isn't much better.
>
> So, what filesystem would you recommend?
>
> Thanks,
> Christian
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Filesystems for ceph

[Fyodor Ustinov]

On 07/28/2011 02:33 AM, Sage Weil wrote:
> On Wed, 27 Jul 2011, Fyodor Ustinov wrote:
>> Christian Brunner<chb<at>  muc.de>  writes:
>>
>>>  From what I read in Fyodors emails ext4 in 2.6.39 isn't much better.
>>>
>>> So, what filesystem would you recommend?
>>>
>> Christian, IMHO ext3/ext4/btrfs unusable for ceph. I tested some time xfs, and
>> think to use it again. But I think that the ceph developers do not like xfs:)
> What problems have you seen with ext4?
Look to my message to Greg. In a nutshell - I again have broken ext4 fs 
on OSD servers with 2.6.39 kernel.

WBR,
     Fyodor.

Re: Filesystems for ceph

[Sage Weil]

On Thu, 28 Jul 2011, Fyodor Ustinov wrote:
> On 07/28/2011 02:33 AM, Sage Weil wrote:
> > On Wed, 27 Jul 2011, Fyodor Ustinov wrote:
> > > Christian Brunner<chb<at>  muc.de>  writes:
> > > 
> > > >  From what I read in Fyodors emails ext4 in 2.6.39 isn't much better.
> > > > 
> > > > So, what filesystem would you recommend?
> > > > 
> > > Christian, IMHO ext3/ext4/btrfs unusable for ceph. I tested some time xfs,
> > > and
> > > think to use it again. But I think that the ceph developers do not like
> > > xfs:)
> > What problems have you seen with ext4?
>
> Look to my message to Greg. In a nutshell - I again have broken ext4 fs 
> on OSD servers with 2.6.39 kernel.

Oh, you mean the fsck errors?  Got it.  If you see it again (on another 
disk/machine, with a reasonably fresh fs) the ext4 guys will be interested 
in hearing about it.  It's probably in the xattr code, which ceph uses 
heavily but most people only set once per file (for selinux labels), if 
at all.

sage

Re: Filesystems for ceph

[Sage Weil]

On Wed, 27 Jul 2011, Christian Brunner wrote:
> We are having quite some problems with the underlying filesystem for
> the cosd's and I would like to hear about other experiences. Here is
> what we have gone through so far:
> 
> btrfs with 2.6.38:
> 
> - good performance
> - frequently hitting of various BUG_ON conditions
> 
> btrfs with 2.6.39:
> 
> - big performance problems after a few days uptime
> - occasionally hitting BUG_ON conditions
> 
> btrfs with 3.0:
> 
> - big performance problems after a few days uptime
> - occasionally hitting a deadlock in the btrfs filesystem (cosd is in D-state)

You might try 3.0 + the latest stuff Chris just sent to Linus (at least 
for the D-state problem).  I'm eager to see whether the latencytop info 
you sent is helpful for the rest.

FWIW we're running 2.6.38+ at the moment.  We get some warnings with the 
orphan code (fixed in .39), but it's otherwise been reasonably stable.  

sage

Re: Filesystems for ceph

[srimugunthan dhandapani]

On Thu, Jul 28, 2011 at 10:09 AM, Sage Weil <sage@newdream.net> wrote:
> On Wed, 27 Jul 2011, Christian Brunner wrote:
>> We are having quite some problems with the underlying filesystem for
>> the cosd's and I would like to hear about other experiences. Here is
>> what we have gone through so far:
>>
>> btrfs with 2.6.38:
>>
>> - good performance
>> - frequently hitting of various BUG_ON conditions
>>
>> btrfs with 2.6.39:
>>
>> - big performance problems after a few days uptime
>> - occasionally hitting BUG_ON conditions
>>
>> btrfs with 3.0:
>>
>> - big performance problems after a few days uptime
>> - occasionally hitting a deadlock in the btrfs filesystem (cosd is in D-state)
>
> You might try 3.0 + the latest stuff Chris just sent to Linus (at least
> for the D-state problem).  I'm eager to see whether the latencytop info
> you sent is helpful for the rest.
>
> FWIW we're running 2.6.38+ at the moment.  We get some warnings with the
> orphan code (fixed in .39), but it's otherwise been reasonably stable.

I thought that ext3/4 and btrfs are the only recommended underlying
filesystem for ceph. I didnt know we can use xfs.
With Glusterfs, theoretically, we can use any filesystem with extended
attributes support as backend filesystem. Is that the same case with
ceph too?
Though it may not be optimal, theoretically can i use a "filesystem X"
with extended attributes support for ceph?
thanks,
Mugunthan
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Filesystems for ceph

[Gregory Farnum]

On Thu, Jul 28, 2011 at 12:35 AM, srimugunthan dhandapani
<srimugunthan.dhandapani@gmail.com> wrote:
> I thought that ext3/4 and btrfs are the only recommended underlying
> filesystem for ceph. I didnt know we can use xfs.
> With Glusterfs, theoretically, we can use any filesystem with extended
> attributes support as backend filesystem. Is that the same case with
> ceph too?
> Though it may not be optimal, theoretically can i use a "filesystem X"
> with extended attributes support for ceph?
> thanks,
> Mugunthan

These days you can use any filesystem with xattr support as the
backing store for Ceph, yes. btrfs is our primary target for a few
reasons (unlimited xattrs, built-in snapshots make snapshots and
consistency easier, etc) but you can stick whatever you like back
there and Ceph will handle it with writeahead journaling, manual
copying for snapshots, etc etc. :)
-Greg

Re: Filesystems for ceph

[Fyodor Ustinov]

On 07/28/2011 07:35 AM, Sage Weil wrote:
> Oh, you mean the fsck errors?  Got it.  If you see it again (on another
> disk/machine, with a reasonably fresh fs) the ext4 guys will be interested
> in hearing about it.  It's probably in the xattr code, which ceph uses
> heavily but most people only set once per file (for selinux labels), if
> at all.
I now see it on _all_ of 12 osd (8 physical servers, each osd have own fs).

I plan to upgrade the kernel on the osd servers to 3.0 and if  do not 
get better - then time to shake the guys from ext4 team (with your help)

WBR,
     Fyodor.