Unmountable btrfs filesystems

Unmountable btrfs filesystems

[Wido den Hollander]

Hi,

On my dev cluster (10 nodes, 40 OSD's) I'm still trying to run Ceph on 
btrfs, but over the last couple of months I've lost multiple OSD's due 
to btrfs.

On my nodes I've set kernel.panic=60 so that whenever a kernel panic 
occurs I get the node back within two minutes.

Now, over the last time I've seen multiple nodes reboot (didn't see the 
strace), but afterwards the btrfs filesystems on that node were unmountable.

"btrfs: open_ctree failed"

I tried various kernels, the most recent 3.3.0 from kernel.ubuntu.com, 
but I'm still seeing this.

Is anyone seeing the same or did everybody migrate away to ext4 or XFS?

I still prefer btrfs due to the snapshotting, but loosing all these 
OSD's all the time is getting kind of frustrating.

Any thoughts or comments?

Wido

Re: Unmountable btrfs filesystems

[Mark Nelson]

On 6/16/12 1:46 PM, Wido den Hollander wrote:
> Hi,
>
> On my dev cluster (10 nodes, 40 OSD's) I'm still trying to run Ceph on
> btrfs, but over the last couple of months I've lost multiple OSD's due
> to btrfs.
>
> On my nodes I've set kernel.panic=60 so that whenever a kernel panic
> occurs I get the node back within two minutes.
>
> Now, over the last time I've seen multiple nodes reboot (didn't see the
> strace), but afterwards the btrfs filesystems on that node were
> unmountable.
>
> "btrfs: open_ctree failed"
>
> I tried various kernels, the most recent 3.3.0 from kernel.ubuntu.com,
> but I'm still seeing this.
>
> Is anyone seeing the same or did everybody migrate away to ext4 or XFS?
>
> I still prefer btrfs due to the snapshotting, but loosing all these
> OSD's all the time is getting kind of frustrating.
>
> Any thoughts or comments?
>
> Wido
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi Wido,

btrfsck might tell you what's wrong.  Sounds like there is a 
btrfs-restore command in the dangerdonteveruse branch you could try. 
Beyond that, I guess it just really comes down to tradeoffs.

Good luck! ;)

Mark

Re: Unmountable btrfs filesystems

[Martin Mailand]

Hi Wido,
until recently there were still a few bugs in btrfs which could be hit 
quite easily with ceph. The last big one was fixed here 
http://www.spinics.net/lists/ceph-devel/msg06270.html

I am running a ceph cluster with btrfs on a 3.5-rc2 without a problem, 
even under heavy test load.

Hope that's helped.

-martin


Am 16.06.2012 20:46, schrieb Wido den Hollander:
> I tried various kernels, the most recent 3.3.0 from kernel.ubuntu.com,
> but I'm still seeing this.
>
> Is anyone seeing the same or did everybody migrate away to ext4 or XFS?
>
> I still prefer btrfs due to the snapshotting, but loosing all these
> OSD's all the time is getting kind of frustrating.
>
> Any thoughts or comments?

Re: Unmountable btrfs filesystems

[Wido den Hollander]

Hi,

On 06/17/2012 03:55 PM, Martin Mailand wrote:
> Hi Wido,
> until recently there were still a few bugs in btrfs which could be hit
> quite easily with ceph. The last big one was fixed here
> http://www.spinics.net/lists/ceph-devel/msg06270.html
>
> I am running a ceph cluster with btrfs on a 3.5-rc2 without a problem,
> even under heavy test load.
>

I'll give that one a try, thanks!

I've also let the btrfs tools like fsck loose on those filesystems, but 
they were beyond any repair. Al kinds of errors on the filesystems.

Nothing wrong with the disk below them, but the filesystems were just 
completely broken, no way of repairing.

Wido

> Hope that's helped.
>
> -martin
>
>
> Am 16.06.2012 20:46, schrieb Wido den Hollander:
>> I tried various kernels, the most recent 3.3.0 from kernel.ubuntu.com,
>> but I'm still seeing this.
>>
>> Is anyone seeing the same or did everybody migrate away to ext4 or XFS?
>>
>> I still prefer btrfs due to the snapshotting, but loosing all these
>> OSD's all the time is getting kind of frustrating.
>>
>> Any thoughts or comments?

Re: Unmountable btrfs filesystems

[Guido Winkelmann]

Am Samstag, 16. Juni 2012, 14:12:03 schrieb Mark Nelson:
> btrfsck might tell you what's wrong.  Sounds like there is a
> btrfs-restore command in the dangerdonteveruse branch you could try.
> Beyond that, I guess it just really comes down to tradeoffs.

I've had similar problems in the recent past. Turns out Ceph makes heavy use 
of btrfs snapshots when running on btrfs, and btrfs-restore will not restore 
those, so it cannot be used to restore a broken osd.

	Guido

Re: Unmountable btrfs filesystems

[Guido Winkelmann]

Am Sonntag, 17. Juni 2012, 15:55:42 schrieb Martin Mailand:
> Hi Wido,
> until recently there were still a few bugs in btrfs which could be hit
> quite easily with ceph. The last big one was fixed here
> http://www.spinics.net/lists/ceph-devel/msg06270.html

I keep hearing things along the lines of "yes, btrfs is really really close to 
ready, we just had some really nasty bug in the last release, so you 
absolutely have to run the very latest Linux kernel" since at least Linux 3.1.

I think I will probably wait until there have been at least three major Linux 
releases with no serious btrfs issues before I start using it in production.

	Guido