Re: Comparison to ZFS and BTRFS

[Kent Overstreet <kent.overstreet@gmail.com>]

On Wed, Apr 06, 2022 at 02:55:04AM -0400, Demi Marie Obenour wrote:
> How does bcachefs manage to outperform ZFS and BTRFS?  Obviously being
> licensed under GPL-compatible terms is an advantage for inclusion in
> Linux, but I am more interested in the technical aspects.
> 
> - How does bcachefs avoid the nasty performance pitfalls that plague
>   BTRFS?  Are VM disks and databases on bcachefs fast?

Clean modular design (the result of years of slow incremental work), and a
_blazingly_ fast B+ tree implementation.

We're not fast in every situation yet. We don't have a nocow (non copy-on-write)
mode, and slow random reads can be slow due to checksum granularity being at the
extent level (which is a good tradeoff in most situations, but we need an option
for smaller checksum granularity at some point).

> - How does bcachefs avoid the dreaded RAID write hole? 

We're copy on write - and this extends to our erasure coding implementation, we
don't update existing stripes in place - we create new stripes as needed,
reusing buckets from existing stripes that still have data.

> - How does an O_DIRECT loop device on bcachefs compare to a zvol on ZFS?

I'd have to benchmark/profile it. It appears there's some bugs in the way the
loop driver in O_DIRECT mode interacts with bcachefs according to xfstests, and
the loopback driver is implemented in a more heavyweight way that it needs to be
- there's room for improvement.

> - Is there a good description of the bcachefs on-disk format anywhere?

Try this: https://bcachefs.org/Architecture/

> - What are the internal abstraction layers used in bcachefs?  Is it a
>   key-value store with a filesystem on top of it, the way ZFS is?

It's just a key value store with a filesystem on top, moreso than the way ZFS
is, from what I understand of ZFS.

> - Is it possible to shrink a bcachefs filesystem?

Not yet, but it won't take much work to add

> Does bcachefs have
>   any restrictions regarding the size of disks in a pool, or can I just
>   throw a bunch of varying-size disks at bcachefs and have it spread the
>   data around automatically to provide the level of redundancy I want?

No restrictions, the allocator stripes across available devices but biases in
favor of devices with more free space.

> - Can bcachefs use faster storage as a cache for slower storage, or
>   otherwise move data around based on usage patterns?

Yes.

> - Can bcachefs saturate your typical NVMe drive on realistic workloads?
>   Can it do so with encryption enabled?

This sounds like a question for someone interested in benchmarking :)

> - Is support for swap files on bcachefs planned?  That would require
>   being able to perform O_DIRECT asynchronous writes without any memory
>   allocations.

Yes it's planned, the IO path already has the necessary support

> - Is bcachefs being used in production anywhere?

Yes