[LSF/MM/BPF TOPIC] configfd as a replacement for both ioctls and fsconfig

[LSF/MM/BPF TOPIC] configfd as a replacement for both ioctls and fsconfig

[James Bottomley]

A shortened version of this topic was originally sent for LSF/MM 2020
which didn't happen due to the pandemic:

https://lore.kernel.org/all/1581781497.3847.5.camel@HansenPartnership.com/

However, now replacing ioctls is on the table:

https://lore.kernel.org/all/20220201013329.ofxhm4qingvddqhu@garbanzo/

as I've already stated in that thread, I think, used sparingly, ioctls
are fit for purpose and shouldn't be replaced and I'd definitely like
to argue for that position.

However, assuming that people would like to consider alternatives, I'd
like to propose configfd.  It was originally proposed as a
configuration mechanism for bind mounts that was a more general
replacement for fsconfig (which can only configure filesystems with
superblocks) and was going to be used by shiftfs.  However, since
shiftfs functionality was done a different way, configfd has
languished, although the patches are here:

https://lore.kernel.org/all/20200215153609.23797-1-James.Bottomley@HansenPartnership.com/

The point, though, is that configfd can configure pretty much anything;
it wouldn't just be limited to filesystem objects.  It takes the
fsconfig idea of using a file descriptor to carry configuration
information, which could be built up over many config calls and makes
it general enough to apply to anything.  One of the ideas of configfd
is that the data could be made fully introspectable ... as in not just
per item description, but the ability to get from the receiver what it
is expecting in terms of configuration options (this part was an idea
not present in the above patch series).

If the ioctl debate goes against ioctls, I think configfd would present
a more palatable alternative to netlink everywhere.

James

Re: [LSF/MM/BPF TOPIC] configfd as a replacement for both ioctls and fsconfig

[David Howells]

James Bottomley <James.Bottomley@HansenPartnership.com> wrote:

> 
> If the ioctl debate goes against ioctls, I think configfd would present
> a more palatable alternative to netlink everywhere.

It'd be nice to be able to set up a 'configuration transaction' and then do a
commit to apply it all in one go.

David

Re: [LSF/MM/BPF TOPIC] configfd as a replacement for both ioctls and fsconfig

[Steve French]

This is a good discussion to have

On Sat, Feb 5, 2022 at 6:02 AM James Bottomley
<James.Bottomley@hansenpartnership.com> wrote:
>
> A shortened version of this topic was originally sent for LSF/MM 2020
> which didn't happen due to the pandemic:
>
> https://lore.kernel.org/all/1581781497.3847.5.camel@HansenPartnership.com/
>
> However, now replacing ioctls is on the table:
>
> https://lore.kernel.org/all/20220201013329.ofxhm4qingvddqhu@garbanzo/
>
> as I've already stated in that thread, I think, used sparingly, ioctls
> are fit for purpose and shouldn't be replaced and I'd definitely like
> to argue for that position.
>
> However, assuming that people would like to consider alternatives, I'd
> like to propose configfd.  It was originally proposed as a
> configuration mechanism for bind mounts that was a more general
> replacement for fsconfig (which can only configure filesystems with
> superblocks) and was going to be used by shiftfs.  However, since
> shiftfs functionality was done a different way, configfd has
> languished, although the patches are here:
>
> https://lore.kernel.org/all/20200215153609.23797-1-James.Bottomley@HansenPartnership.com/
>
> The point, though, is that configfd can configure pretty much anything;
> it wouldn't just be limited to filesystem objects.  It takes the
> fsconfig idea of using a file descriptor to carry configuration
> information, which could be built up over many config calls and makes
> it general enough to apply to anything.  One of the ideas of configfd
> is that the data could be made fully introspectable ... as in not just
> per item description, but the ability to get from the receiver what it
> is expecting in terms of configuration options (this part was an idea
> not present in the above patch series).
>
> If the ioctl debate goes against ioctls, I think configfd would present
> a more palatable alternative to netlink everywhere.
>
> James
>
>


-- 
Thanks,

Steve

Re: [LSF/MM/BPF TOPIC] configfd as a replacement for both ioctls and fsconfig

[Luis Chamberlain]

On Tue, Feb 01, 2022 at 02:13:13PM +0000, David Howells wrote:
> James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
> 
> > 
> > If the ioctl debate goes against ioctls, I think configfd would present
> > a more palatable alternative to netlink everywhere.
> 
> It'd be nice to be able to set up a 'configuration transaction' and then do a
> commit to apply it all in one go.

Can't io-uring cmd effort help here?

  Luis

Re: [LSF/MM/BPF TOPIC] configfd as a replacement for both ioctls and fsconfig

[Luis Chamberlain]

On Tue, Feb 01, 2022 at 08:43:47AM -0500, James Bottomley wrote:
> https://lore.kernel.org/all/20200215153609.23797-1-James.Bottomley@HansenPartnership.com/
> 
> The point, though, is that configfd can configure pretty much anything;
> it wouldn't just be limited to filesystem objects.  It takes the
> fsconfig idea of using a file descriptor to carry configuration
> information, which could be built up over many config calls and makes
> it general enough to apply to anything.  One of the ideas of configfd
> is that the data could be made fully introspectable ... as in not just
> per item description, but the ability to get from the receiver what it
> is expecting in terms of configuration options (this part was an idea
> not present in the above patch series).
> 
> If the ioctl debate goes against ioctls, I think configfd would present
> a more palatable alternative to netlink everywhere.

FWIW it seems more grounded on the fs world for sure. And I didn't want to
barf when doing a cursory review.

  Luis

Re: [LSF/MM/BPF TOPIC] configfd as a replacement for both ioctls and fsconfig

[James Bottomley]

On Mon, 2022-02-28 at 16:45 -0800, Luis Chamberlain wrote:
> On Tue, Feb 01, 2022 at 02:13:13PM +0000, David Howells wrote:
> > James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
> > 
> > > If the ioctl debate goes against ioctls, I think configfd would
> > > present
> > > a more palatable alternative to netlink everywhere.
> > 
> > It'd be nice to be able to set up a 'configuration transaction' and
> > then do a
> > commit to apply it all in one go.
> 
> Can't io-uring cmd effort help here?

What io-uring cmd effort?  The one to add nvme completions?  If it's
the completions one, then the configfs interface currently doesn't have
an event notifier, which is what the completions patch set seems to
require.  On the other hand configfd is key/value for get/set with an
atomic activate using an fd, so it stands to reason epoll support could
be added for events on the fd ... we'd just have to define a retrieval
key for an indicator to say which events are ready.

James

Re: [LSF/MM/BPF TOPIC] configfd as a replacement for both ioctls and fsconfig

[Luis Chamberlain]

On Tue, Mar 01, 2022 at 11:47:56AM -0500, James Bottomley wrote:
> On Mon, 2022-02-28 at 16:45 -0800, Luis Chamberlain wrote:
> > On Tue, Feb 01, 2022 at 02:13:13PM +0000, David Howells wrote:
> > > James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
> > > 
> > > > If the ioctl debate goes against ioctls, I think configfd would
> > > > present
> > > > a more palatable alternative to netlink everywhere.
> > > 
> > > It'd be nice to be able to set up a 'configuration transaction' and
> > > then do a
> > > commit to apply it all in one go.
> > 
> > Can't io-uring cmd effort help here?
> 
> What io-uring cmd effort? 

The file operations version is the latest posted effort:

https://lore.kernel.org/linux-nvme/20210317221027.366780-1-axboe@kernel.dk/

> The one to add nvme completions?

Um, I would not call it that at all, but rather nvme passthrough. But
yes that is possible. But so are many other things, not just ioctls,
which is why I've been suggesting I think it does a disservice to the
efforto just say its useful for ioctl over io-uring. Anything with
a file_operations can tackle on cmd suport using io-uring as a
train.

> If it's
> the completions one, then the configfs interface currently doesn't have
> an event notifier, which is what the completions patch set seems to
> require.  On the other hand configfd is key/value for get/set with an
> atomic activate using an fd, so it stands to reason epoll support could
> be added for events on the fd ... we'd just have to define a retrieval
> key for an indicator to say which events are ready.

It sounds like it could use it.

  Luis

Re: [LSF/MM/BPF TOPIC] configfd as a replacement for both ioctls and fsconfig

[James Bottomley]

On Tue, 2022-03-01 at 08:53 -0800, Luis Chamberlain wrote:
> On Tue, Mar 01, 2022 at 11:47:56AM -0500, James Bottomley wrote:
> > On Mon, 2022-02-28 at 16:45 -0800, Luis Chamberlain wrote:
> > > On Tue, Feb 01, 2022 at 02:13:13PM +0000, David Howells wrote:
> > > > James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
> > > > 
> > > > > If the ioctl debate goes against ioctls, I think configfd
> > > > > would
> > > > > present
> > > > > a more palatable alternative to netlink everywhere.
> > > > 
> > > > It'd be nice to be able to set up a 'configuration transaction'
> > > > and
> > > > then do a
> > > > commit to apply it all in one go.
> > > 
> > > Can't io-uring cmd effort help here?
> > 
> > What io-uring cmd effort? 
> 
> The file operations version is the latest posted effort:
> 
> https://lore.kernel.org/linux-nvme/20210317221027.366780-1-axboe@kernel.dk/

Oh, right ... I'm afraid for storage if it hasn't been across linux-
block or linux-scsi, I likely won't have seen it.  However, reading the
thread, it really does look like the added file operation

int (*uring_cmd)(struct io_uring_cmd *, enum io_uring_cmd_flags);

Is pretty similar to an async ioctl as Christoph said.

> > The one to add nvme completions?
> 
> Um, I would not call it that at all, but rather nvme passthrough. But
> yes that is possible. But so are many other things, not just ioctls,
> which is why I've been suggesting I think it does a disservice to the
> efforto just say its useful for ioctl over io-uring. Anything with
> a file_operations can tackle on cmd suport using io-uring as a
> train.

It looks fairly similar given the iouring syscalls are on an fd except
that the structure above hash to be defined and becomes an ABI.  Since
they configfd uses typed key value pairs, i'd argue it's more generic
and introspectable.

James

Re: [LSF/MM/BPF TOPIC] configfd as a replacement for both ioctls and fsconfig

[Luis Chamberlain]

On Tue, Mar 01, 2022 at 12:14:49PM -0500, James Bottomley wrote:
> It looks fairly similar given the iouring syscalls are on an fd except
> that the structure above hash to be defined and becomes an ABI.  Since
> they configfd uses typed key value pairs, i'd argue it's more generic
> and introspectable.

I'm not suggesting using io-uring cmd as a configuration alternative
to configfd, I'm suggesting it can be an async *vehicle* for a bunch
of configurations one might need to make in the kernel. If we want
to reduce system call uses, we may want to allow something like configfd
to accept a batch of configuration options as well, as a batcha, and a
final commit to process them.

  Luis

Re: [LSF/MM/BPF TOPIC] configfd as a replacement for both ioctls and fsconfig

[James Bottomley]

On Tue, 2022-03-01 at 09:40 -0800, Luis Chamberlain wrote:
> On Tue, Mar 01, 2022 at 12:14:49PM -0500, James Bottomley wrote:
> > It looks fairly similar given the iouring syscalls are on an fd
> > except
> > that the structure above hash to be defined and becomes an
> > ABI.  Since
> > they configfd uses typed key value pairs, i'd argue it's more
> > generic
> > and introspectable.
> 
> I'm not suggesting using io-uring cmd as a configuration alternative
> to configfd, I'm suggesting it can be an async *vehicle* for a bunch
> of configurations one might need to make in the kernel. If we want
> to reduce system call uses, we may want to allow something like
> configfd to accept a batch of configuration options as well, as a
> batcha, and a final commit to process them.

Well, that's effectively how it does operate.  Configfd like configfs
is a dedicated fd you open exclusively for the purpose of
configuration.  You send it the key/value pairs via the action system
call.  Although the patch sent used "basic" types as values, nothing
prevents them being composite types that are aggregated, which would be
an easy mechanism for batching.

However, I'd like to add a note of caution: just because we *can* do
batching with the interface doesn't mean we should.  One of the
benefits of using simple basic types is easy interpretation by things
like seccomp; the more complex you make the type, the more internal
knowledge the seccomp/ebpf script needs.

So can I ask just how important batching for configuration changes is?
I get that there's some overhead for doing effectively one syscall per
k/v pair, but configuration operations aren't usually that time
critical.  If you're sending passthrough, then I can see you don't want
a load of syscalls per op, but equally a passthrough is just an opaque
packet that's likely not introspectable anyway, so it's a single k/v
pair.

James

Re: [LSF/MM/BPF TOPIC] configfd as a replacement for both ioctls and fsconfig

[Luis Chamberlain]

On Tue, Mar 01, 2022 at 01:08:34PM -0500, James Bottomley wrote:
> On Tue, 2022-03-01 at 09:40 -0800, Luis Chamberlain wrote:
> > On Tue, Mar 01, 2022 at 12:14:49PM -0500, James Bottomley wrote:
> > > It looks fairly similar given the iouring syscalls are on an fd
> > > except
> > > that the structure above hash to be defined and becomes an
> > > ABI.  Since
> > > they configfd uses typed key value pairs, i'd argue it's more
> > > generic
> > > and introspectable.
> > 
> > I'm not suggesting using io-uring cmd as a configuration alternative
> > to configfd, I'm suggesting it can be an async *vehicle* for a bunch
> > of configurations one might need to make in the kernel. If we want
> > to reduce system call uses, we may want to allow something like
> > configfd to accept a batch of configuration options as well, as a
> > batcha, and a final commit to process them.
> 
> Well, that's effectively how it does operate.  Configfd like configfs
> is a dedicated fd you open exclusively for the purpose of
> configuration.  You send it the key/value pairs via the action system
> call.  Although the patch sent used "basic" types as values, nothing
> prevents them being composite types that are aggregated, which would be
> an easy mechanism for batching.
> 
> However, I'd like to add a note of caution: just because we *can* do
> batching with the interface doesn't mean we should.  One of the
> benefits of using simple basic types is easy interpretation by things
> like seccomp; the more complex you make the type, the more internal
> knowledge the seccomp/ebpf script needs.

This alone makes a good argument against ioctls.

> So can I ask just how important batching for configuration changes is?

A great question.

I can't personally think of use cases unless you want batch configuration with
IO operations. io-uring stuff does have some link where in this regard,
given some IO ops also may expose new fds for instance, which you may
later want to use for further IO processing... but that seems very
different than just the question of batching for configuration.

> I get that there's some overhead for doing effectively one syscall per
> k/v pair, but configuration operations aren't usually that time
> critical. 

Agreed and I'd like to hear arguments which would suggest otherwise.

> If you're sending passthrough, then I can see you don't want
> a load of syscalls per op, but equally a passthrough is just an opaque
> packet that's likely not introspectable anyway, so it's a single k/v
> pair.

I think it might be fair to detach config / IO calls. Joshi however
might have more to say about if we really would need to mesh the two.

  Luis

Re: [LSF/MM/BPF TOPIC] configfd as a replacement for both ioctls and fsconfig

[David Howells]

Luis Chamberlain <mcgrof@kernel.org> wrote:

> > It'd be nice to be able to set up a 'configuration transaction' and then
> > do a commit to apply it all in one go.
> 
> Can't io-uring cmd effort help here?

I don't know.  Wouldn't that want to apply each element as a separate thing?

But you might want to do something more akin to a db transaction, where you
start a transaction, read stuff, consider your changes, propose your changes
and then commit - which would mean io_uring wouldn't help.

David

Re: [LSF/MM/BPF TOPIC] configfd as a replacement for both ioctls and fsconfig

[Luis Chamberlain]

On Wed, Mar 02, 2022 at 03:19:32PM +0000, David Howells wrote:
> Luis Chamberlain <mcgrof@kernel.org> wrote:
> 
> > > It'd be nice to be able to set up a 'configuration transaction' and then
> > > do a commit to apply it all in one go.
> > 
> > Can't io-uring cmd effort help here?
> 
> I don't know.  Wouldn't that want to apply each element as a separate thing?

There is nothing to stop us to design an API which starts a transaction / ends.
And io-uring cmd supports links so this is all possible in theory, I just don't
think anyone has done it before.

I mean... io-uring cmd file operation stuff is not even upstream yet...

> But you might want to do something more akin to a db transaction, where you
> start a transaction, read stuff, consider your changes, propose your changes
> and then commit - which would mean io_uring wouldn't help.

I think Pavel had some advanced use cases to support that with io-uring
cmd work. For instance open a file descriptor and then work on it all
in the same chain of commands sent.

  Luis

Re: [LSF/MM/BPF TOPIC] configfd as a replacement for both ioctls and fsconfig

[Jens Axboe]

On 3/2/22 2:54 PM, Luis Chamberlain wrote:
> On Wed, Mar 02, 2022 at 03:19:32PM +0000, David Howells wrote:
>> Luis Chamberlain <mcgrof@kernel.org> wrote:
>>
>>>> It'd be nice to be able to set up a 'configuration transaction' and then
>>>> do a commit to apply it all in one go.
>>>
>>> Can't io-uring cmd effort help here?
>>
>> I don't know.  Wouldn't that want to apply each element as a separate thing?
> 
> There is nothing to stop us to design an API which starts a
> transaction / ends. And io-uring cmd supports links so this is all
> possible in theory, I just don't think anyone has done it before.

Only thing you're missing there is unroll on failure. It's quite
possible to have linked operations, but the only recurse is "abort rest
of chain if one fails".

> I mean... io-uring cmd file operation stuff is not even upstream
> yet...

Mostly because it hasn't been pushed. Was waiting for a good idea on how
to handle extended commands, and actually had the epiphany and
implemented it next week. So a more palatable version should be posted
soon.

>> But you might want to do something more akin to a db transaction, where you
>> start a transaction, read stuff, consider your changes, propose your changes
>> and then commit - which would mean io_uring wouldn't help.
> 
> I think Pavel had some advanced use cases to support that with io-uring
> cmd work. For instance open a file descriptor and then work on it all
> in the same chain of commands sent.

That already exists in the kernel, it's direct descriptors. See:

https://lwn.net/Articles/863071/

With that, you can have a chain that does "open file, do X to file, do Y
to file, etc, close file" in one operation. Links like mentioned above,
but can pre-make requests that operate on a descriptor that has been
opened.

-- 
Jens Axboe

Re: [LSF/MM/BPF TOPIC] configfd as a replacement for both ioctls and fsconfig

[Luis Chamberlain]

On Wed, Mar 02, 2022 at 03:05:33PM -0700, Jens Axboe wrote:
> On 3/2/22 2:54 PM, Luis Chamberlain wrote:
> > On Wed, Mar 02, 2022 at 03:19:32PM +0000, David Howells wrote:
> >> Luis Chamberlain <mcgrof@kernel.org> wrote:
> >>
> >>>> It'd be nice to be able to set up a 'configuration transaction' and then
> >>>> do a commit to apply it all in one go.
> >>>
> >>> Can't io-uring cmd effort help here?
> >>
> >> I don't know.  Wouldn't that want to apply each element as a separate thing?
> > 
> > There is nothing to stop us to design an API which starts a
> > transaction / ends. And io-uring cmd supports links so this is all
> > possible in theory, I just don't think anyone has done it before.
> 
> Only thing you're missing there is unroll on failure. It's quite
> possible to have linked operations, but the only recurse is "abort rest
> of chain if one fails".

Yikes. I can see this getting complicated fast. But wouldn't dbs have
already dealt with this somehow?

But yes, my point still stands. io-uring cmd stuff really can be a game
changer on how we deal with new APIs. While unifying arch stuff to make
system calls easier to add, I'm actually thinking, if an async API
is needed, then perhaps we can *curtail* adding new system calls and just
handle these through the io-uring cmd stuff. Of course, so long as your
subsystem deals with file_operations.

> > I mean... io-uring cmd file operation stuff is not even upstream
> > yet...
> 
> Mostly because it hasn't been pushed. Was waiting for a good idea on how
> to handle extended commands, and actually had the epiphany and
> implemented it next week. So a more palatable version should be posted
> soon.

I am more than ecstatic to hear this! I cannot wait!
 
> >> But you might want to do something more akin to a db transaction, where you
> >> start a transaction, read stuff, consider your changes, propose your changes
> >> and then commit - which would mean io_uring wouldn't help.
> > 
> > I think Pavel had some advanced use cases to support that with io-uring
> > cmd work. For instance open a file descriptor and then work on it all
> > in the same chain of commands sent.
> 
> That already exists in the kernel, it's direct descriptors. See:
> 
> https://lwn.net/Articles/863071/
> 
> With that, you can have a chain that does "open file, do X to file, do Y
> to file, etc, close file" in one operation. Links like mentioned above,
> but can pre-make requests that operate on a descriptor that has been
> opened.

Groovy, thanks for the pointer!

  Luis