Re: Questions related to BCacheFS

[Kent Overstreet <kent.overstreet@linux.dev>]

On Sun, Nov 19, 2023 at 12:13:29PM +0100, Martin Steigerwald wrote:
> Take your time and have a great Sunday :)
> 
> Kent Overstreet - 19.11.23, 00:42:05 CET:
> > On Sun, Nov 19, 2023 at 12:15:19AM +0100, Martin Steigerwald wrote:
> > > Kent Overstreet - 18.11.23, 22:07:27 CET:
> > > > > As far as I understand one specific performance related aspect of
> > > > > BCacheFS would be low latencies due to the frontend / backend
> > > > > architecture which in principle is based on what has been there in
> > > > > BCache already. I am intending to explore a bit into that concept
> > > > > in
> > > > > my article.
> > > > 
> > > > The low latency stuff actually wasn't in bcache - that work came
> > > > later.
> > > 
> > > So the frontend / backend architecture is not that much of what makes
> > > BCacheFS unique? Important to know as it seems I may have
> > > misunderstood
> > > something here.
> > 
> > The "filesystem on top of a database" is the main thing that makes
> > bcachefs unique - you have that right.
> 
> Phew! Seems I did not get it completely wrong then :)
> 
> > bcache had much of the core btree design - log structured btree nodes
> > with eytzinger search trees; that's how we got a high enough performance
> > btree to make the "filesystem on top of a database" thing practical.
> > 
> > But the btree in bcache was, from a performance POV, prototype quality -
> > stable, but a lot of performance corner cases unfinished.
> > 
> > The latency work, real iterators, and the whole transaction layer came
> > later :)
> 
> So it is fair to say that being based on BCache enabled that low latency 
> work?
> 
> I am trying to find a balance here. The audience of the article are 
> experienced sys admins working in small and large organizations, but not 
> (primarily) kernel hackers. So I like to explain possible reasons to 
> consider BCacheFS while also having BTRFS and ZFS and XFS, but without 
> going into so much detail that one needs to be a kernel developer to 
> understand it. For XFS it is easy as while there was a proof of concept 
> with subvolumes and snapshots based on XFS in files on XFS, AFAIR from 
> Dave Chinner, some years back, it does not have those advanced features 
> (yet). For ZFS one can always argue it is not in the mainline kernel. But 
> regarding BTRFS it becomes important to really explain something. I 
> started to do some BCacheFS / BTRFS feature comparison chart, but I also 
> like to explain on the benefits BCacheFS can have in the text and a bit 
> about the background of those benefits. Of course also mentioning that 
> BCacheFS is in development for more than 10 years, even without taking all 
> the work for BCache itself into account.
> 
> So my idea currently is to explain that the BCache BTree and/or frontend/
> backend architecture, not sure how to best word it, enabled a database 
> approach to a filesystem to be feasible. And that in a sense it also 
> enabled the latency work. And I can mention sixlocks and all the nice 
> other stuff you mentioned. However… I may not explain exactly what those 
> nice things are and how they work for example. For two reasons: 1. I need 
> to understand those nice features myself, 2. limit of pages I may use and 
> scope of the article. Currently I understood that sixlocks for certain use 
> cases are the next best thing since the invention of a wheel or something 
> like that. But not much more :)
> 
> Would something like that be accurate enough in your opinion?

Yeah, that all sounds reasonable :)

It would be a great project to get all this stuff documented better...
when I have more free time... :)

> I will review the user manual once again, I read about the database 
> approach, and aim to find a good balance. Cause for certain I won't be 
> getting 24 pages for the article :)
> 
> 
> I got two setbacks regarding trying BCacheFS on my laptop yesterday and 
> today:
> 
> 1) Linux 6.7-rc1, as mentioned almost rc2, did not hibernate on ThinkPad 
> T14 AMD Gen 1. It just blanked screen and nothing happened. So I went back 
> to 6.6.1 temporarily. I do not really intend to do a git bisect between 
> rc1 and 6.6 on a production laptop. It would be very time-consuming and 
> possibly be dangerous. I may go back to 6.7 even without hibernation, just 
> using standby over night for a while. I bet BCacheFS is compatible with 
> hibernation (as in writing memory contents to encrypted swap and resuming 
> from it)?

I believe so - I have not tested hibernation specifically.

> 
> 2) But after I removed 6.7-rc1 yesterday night after having booted into 
> 6.6.1 this morning I was greeted by GRUB command line. As I had a meeting 
> scheduled my primary aim was to get things running again. I certainly did 
> not really get the humorous aspect about it. I thought I'd had copied etc 
> in the broken state to a another place, but do not find it anymore, maybe 
> I copied it to /root or the GRML live distro accidently and so it is gone. 
> Also I missed to copy the broken /boot to another place. So I am not 
> really able to do any forensic analysis of what might have happened. I do 
> not recall having seen any error messages either, but I may have missed 
> something. I reviewed hook and script for initial RAM disk in bcachefs-
> tools repo, but did not find anything in there that could have caused grub 
> 2 not finding its config file and modules anymore. Also it booted into 6.7 
> and even 6.6.1 then after having executed "make install" from bcachefs-
> tools directory. Also I see nothing being done with grub itself within 
> bcachefs-tools. So this really is quite the mystery for me. I am on Devuan 
> Ceres (based on Debian Sid) so maybe something else got messed up. Will 
> review their bug trackers. I really have no idea what went wrong here, 
> luckily I was able to recover with GRML. I use LVM on LUKS for BTRFS and 
> test BCacheFS filesystem.
> 
> Maybe I need to continue testing BCacheFS on a virtual machine, but I'd 
> really love to have a BCacheFS filesystem on my laptop and actually really 
> using it for something. Well I am going to leave it at that and probably 
> research on this after the weekend. Now it is time to have a Sunday off.

The Nixos install process with bcachefs as root is pretty smooth, just
make sure to set up a separate filesystem for /boot!