Something like ZFS Channel Programs for BTRFS & probably XFS or even VFS?

Something like ZFS Channel Programs for BTRFS & probably XFS or even VFS?

[Martin Steigerwald]

[repost. I didn´t notice autocompletion gave me wrong address for fsdevel, 
blacklisted now]

Hello.

What do you think of

http://open-zfs.org/wiki/Projects/ZFS_Channel_Programs

?

There are quite some BTRFS maintenance programs like the deduplication stuff. 
Also regular scrubs… and in certain circumstances probably balances can make 
sense.

In addition to this XFS got scrub functionality as well.

Now putting the foundation for such a functionality in the kernel I think 
would only be reasonable if it cannot be done purely within user space, so I 
wonder about the safety from other concurrent ZFS modification and atomicity 
that are mentioned on the wiki page. The second set of slides, those the 
OpenZFS Developer Commit 2014, which are linked to on the wiki page explain 
this more. (I didn´t look the first ones, as I am no fan of slideshare.net and 
prefer a simple PDF to download and view locally anytime, not for privacy 
reasons alone, but also to avoid a using a crappy webpage over a wonderfully 
functional PDF viewer fat client like Okular)

Also I wonder about putting a lua interpreter into the kernel, but it seems at 
least NetBSD developers added one to their kernel with version 7.0¹.

I also ask this cause I wondered about a kind of fsmaintd or volmaintd for 
quite a while, and thought… it would be nice to do this in a generic way, as 
BTRFS is not the only filesystem which supports maintenance operations. However 
if it can all just nicely be done in userspace, I am all for it.

[1] http://www.netbsd.org/releases/formal-7/NetBSD-7.0.html
(tons of presentation PDFs on their site as well)

Thanks,
-- 
Martin

Re: Something like ZFS Channel Programs for BTRFS & probably XFS or even VFS?

[Dave Chinner]

On Tue, Oct 03, 2017 at 03:19:18PM +0200, Martin Steigerwald wrote:
> [repost. I didn´t notice autocompletion gave me wrong address for fsdevel, 
> blacklisted now]
> 
> Hello.
> 
> What do you think of
> 
> http://open-zfs.org/wiki/Projects/ZFS_Channel_Programs

Domain not found.

-Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Something like ZFS Channel Programs for BTRFS & probably XFS or even VFS?

[Dave Chinner]

On Tue, Oct 03, 2017 at 03:19:18PM +0200, Martin Steigerwald wrote:
> [repost. I didn�t notice autocompletion gave me wrong address for fsdevel, 
> blacklisted now]
> 
> Hello.
> 
> What do you think of
> 
> http://open-zfs.org/wiki/Projects/ZFS_Channel_Programs

Domain not found.

-Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Something like ZFS Channel Programs for BTRFS & probably XFS or even VFS?

[Randy Dunlap]

On 10/03/17 13:10, Dave Chinner wrote:
> On Tue, Oct 03, 2017 at 03:19:18PM +0200, Martin Steigerwald wrote:
>> [repost. I didn´t notice autocompletion gave me wrong address for fsdevel, 
>> blacklisted now]
>>
>> Hello.
>>
>> What do you think of
>>
>> http://open-zfs.org/wiki/Projects/ZFS_Channel_Programs
> 
> Domain not found.

It works for me.


-- 
~Randy

Re: Something like ZFS Channel Programs for BTRFS & probably XFS or even VFS?

[Matthew Wilcox]

On Wed, Oct 04, 2017 at 07:10:35AM +1100, Dave Chinner wrote:
> On Tue, Oct 03, 2017 at 03:19:18PM +0200, Martin Steigerwald wrote:
> > [repost. I didn´t notice autocompletion gave me wrong address for fsdevel, 
> > blacklisted now]
> > 
> > Hello.
> > 
> > What do you think of
> > 
> > http://open-zfs.org/wiki/Projects/ZFS_Channel_Programs
> 
> Domain not found.

Must be an Australian problem ...

A ZFS channel program (ZCP) is a small script written in a domain specific
language that manipulate ZFS internals in a single, atomically-visible
operation. For instance, to delete all snapshots of a filesystem a ZCP
could be written which 1) generates the list of snapshots, 2) traverses
that list, and 3) destroys each snapshot unconditionally. Because
each of these statements would be evaluated from within the kernel,
ZCPs can guarantee safety from interference with other concurrent ZFS
modifications. Executing from inside the kernel allows us to guarantee
atomic visibility of these operations (correctness) and allows them to
be performed in a single transaction group (performance).

A successful implementation of ZCP will:

1. Support equivalent functionality for all of the current ZFS commands
with improved performance and correctness from the point of view of the
user of ZFS.

2. Facilitate the quick addition of new and useful commands as
ZCP enables the implementation of more powerful operations which
previously would have been unsafe to implement in user programs, or
would require modifications to the kernel for correctness. Since the
ZCP layer guarantees the atomicity of each ZCP, we only need to write
new sync_tasks for individual simple operations, then can use ZCPs to
chain those simple operations together into more complicated operations.

3. Allow ZFS users to safely implement their own ZFS operations without
performing operations they don’t have the privileges for.

4. Improve the performance and correctness of existing applications
built on ZFS operations.

Re: Something like ZFS Channel Programs for BTRFS & probably XFS or even VFS?

[Dave Chinner]

On Tue, Oct 03, 2017 at 01:40:51PM -0700, Matthew Wilcox wrote:
> On Wed, Oct 04, 2017 at 07:10:35AM +1100, Dave Chinner wrote:
> > On Tue, Oct 03, 2017 at 03:19:18PM +0200, Martin Steigerwald wrote:
> > > [repost. I didn´t notice autocompletion gave me wrong address for fsdevel, 
> > > blacklisted now]
> > > 
> > > Hello.
> > > 
> > > What do you think of
> > > 
> > > http://open-zfs.org/wiki/Projects/ZFS_Channel_Programs
> > 
> > Domain not found.
> 
> Must be an Australian problem ...

Probably, I forgot to stand on my head so everything must have been
sent to the server upside down....

Though it is a curious failure - it failed until I went to
"openzfs.org" and that redirected to "open-zfs.org" and now it all
works. Somewhat bizarre.

> A ZFS channel program (ZCP) is a small script written in a domain specific
> language that manipulate ZFS internals in a single, atomically-visible
> operation.  For instance, to delete all snapshots of a filesystem a ZCP
> could be written which 1) generates the list of snapshots, 2) traverses
> that list, and 3) destroys each snapshot unconditionally. Because
> each of these statements would be evaluated from within the kernel,
> ZCPs can guarantee safety from interference with other concurrent ZFS
> modifications. Executing from inside the kernel allows us to guarantee
> atomic visibility of these operations (correctness) and allows them to
> be performed in a single transaction group (performance).
>
> A successful implementation of ZCP will:
> 
> 1. Support equivalent functionality for all of the current ZFS commands
> with improved performance and correctness from the point of view of the
> user of ZFS.
> 
> 2. Facilitate the quick addition of new and useful commands as
> ZCP enables the implementation of more powerful operations which
> previously would have been unsafe to implement in user programs, or
> would require modifications to the kernel for correctness. Since the
> ZCP layer guarantees the atomicity of each ZCP, we only need to write
> new sync_tasks for individual simple operations, then can use ZCPs to
> chain those simple operations together into more complicated operations.
> 
> 3. Allow ZFS users to safely implement their own ZFS operations without
> performing operations they don’t have the privileges for.
> 
> 4. Improve the performance and correctness of existing applications
> built on ZFS operations.

/me goes and looks at the slides....

Seems like they are trying to solve a problem of their own making,
in that admin operations are run by the kernel from a separate task
that is really, really slow. So this scripting is a method of aggregating
multiple "sync tasks" into a single operation so there isn't delays
between tasks.

/me chokes on slide 8/8

"Add a Lua interpreter to the kernel, implement ZFS intrinsics (...)
as extensions to the Lua language...."

Somehow, I don't see that happening in Linux.

Yes, I can see us potentially adding some custom functionality in
filesystems with eBPF (e.g. custom allocation policies), but I think
admin operations need to be done from userspace through a clear,
stable interface that supports all the necessary primitives to
customise admin operations for different needs.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Something like ZFS Channel Programs for BTRFS & probably XFS or even VFS?

[Martin Steigerwald]

Hello.

What do you think of

http://open-zfs.org/wiki/Projects/ZFS_Channel_Programs

?

There are quite some BTRFS maintenance programs like the deduplication stuff. 
Also regular scrubs… and in certain circumstances probably balances can make 
sense.

In addition to this XFS got scrub functionality as well.

Now putting the foundation for such a functionality in the kernel I think 
would only be reasonable if it cannot be done purely within user space, so I 
wonder about the safety from other concurrent ZFS modification and atomicity 
that are mentioned on the wiki page. The second set of slides, those the 
OpenZFS Developer Commit 2014, which are linked to on the wiki page explain 
this more. (I didn´t look the first ones, as I am no fan of slideshare.net and 
prefer a simple PDF to download and view locally anytime, not for privacy 
reasons alone, but also to avoid a using a crappy webpage over a wonderfully 
functional PDF viewer fat client like Okular)

Also I wonder about putting a lua interpreter into the kernel, but it seems at 
least NetBSD developers added one to their kernel with version 7.0¹.

I also ask this cause I wondered about a kind of fsmaintd or volmaintd for 
quite a while, and thought… it would be nice to do this in a generic way, as 
BTRFS is not the only filesystem which supports maintenance operations. However 
if it can all just nicely be done in userspace, I am all for it.

[1] http://www.netbsd.org/releases/formal-7/NetBSD-7.0.html
(tons of presentation PDFs on their site as well)

Thanks,
-- 
Martin