Re: test osd on zfs

From: Jeff Mitchell <jeffrey.mitchell@gmail.com>
To: Mark Nelson <mark.nelson@inktank.com>
Cc: Stefan Priebe <s.priebe@profihost.ag>,
	Yehuda Sadeh <yehuda@inktank.com>,
	Brian Behlendorf <behlendorf1@llnl.gov>,
	Sage Weil <sage@inktank.com>,
	Henry C Chang <henry.cy.chang@gmail.com>,
	Aleksey Leonov <aleonov@nazarianin.com>,
	ceph-devel <ceph-devel@vger.kernel.org>
Subject: Re: test osd on zfs
Date: Wed, 17 Apr 2013 16:49:53 -0400	[thread overview]
Message-ID: <CAOx6V3aLJkcwsK=B8UKVCZMjuYWE_W+U_LbCgMLWVK4BmzCt8w@mail.gmail.com> (raw)
In-Reply-To: <516F0321.2@inktank.com>

> On 04/17/2013 02:09 PM, Stefan Priebe wrote:
>>
>> Sorry to disturb, but what is the raeson / advantage of using zfs for
>> ceph?

A few things off the top of my head:

1) Very mature filesystem with full xattr support (this bug
notwithstanding) and copy-on-write snapshots. While the port to Linux
sometimes has some rough edges (but in my experience over the past few
years is generally very good), the main code from Solaris (and now the
Illumos project) is well-tested and very well regarded. Btrfs has many
of the same features, but in my real-world experience I've had
multiple btrfs filesystems go corrupt with very innocuous usage
patterns and across a variety of kernel versions. The zfsonlinux bugs
don't tend to be data-destructive, once data is written to it.
2) Very intelligent caching; also supports external devices (like
SSDs) for a level 2 cache. This speeds up reads dramatically.
3) Very robust error-checking. There are lots of stories of ZFS
finding bad memory, bad controllers, and bad hard drives because of
its checksumming (which you can optionally turn off for speed). If you
set up the OSDs such that each OSD is based off of a ZFS mirror, you
get these benefits locally. For some people, especially when heavy on
reads (due to the intelligent caching), a solution that knocks the
remote replication level down by one but uses local mirrors for OSDs
may provide good functionality and safety compromises.

--Jeff