[Ksummit-discuss] [TECH TOPIC] Linux kernel SMB server (CIFSD)

[Ksummit-discuss] [TECH TOPIC] Linux kernel SMB server (CIFSD)

[Theodore Ts'o]

From: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>

[ Note: The following abstract was submitted via the Linsux Plumbers
  Conference website.  Per the instructions that were posted for the
  Maintainer's / Kernel Summit Call for Proposals[1], the proposal
  should also be posted on the ksummit-discuss list, so that people
  can comment on the proposal, and perhaps start a discussion before
  the summit.

  [1] https://lwn.net/Articles/788378/

  Please note that topic proposals for both the Kernel Summit and the
  Maintainer's Summit are still welcome, and the deadline has been
  extended to June 3rd. -- Ted ]

CIFSD is a new SMB server implementation for the Linux kernel
(out of tree, as of this moment), intended to be minimalistic,
yet providing better performance than user-space analogues.
This talk will briefly introduce CIFSD, its architecture (high
level) and its main design goals: outline components, their
responsibilities and communication protocols.

The work is still in progress, we are in pretty early stages,
so in the second part of the talk we will focus on current state
of affairs and our future development plans - we are looking at
possibility to merge CIFSD with the upstream Linux kernel,
sometime in the future.

Re: [Ksummit-discuss] [TECH TOPIC] Linux kernel SMB server (CIFSD)

[Sergey Senozhatsky]

Cc-ing Jan, Amir, Jeff

Hi!

On (05/30/19 01:59), Theodore Ts'o wrote:
> From: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
[..]
> CIFSD is a new SMB server implementation for the Linux kernel
> (out of tree, as of this moment), intended to be minimalistic,
> yet providing better performance than user-space analogues.
> This talk will briefly introduce CIFSD, its architecture (high
> level) and its main design goals: outline components, their
> responsibilities and communication protocols.
> 
> The work is still in progress, we are in pretty early stages,
> so in the second part of the talk we will focus on current state
> of affairs and our future development plans - we are looking at
> possibility to merge CIFSD with the upstream Linux kernel,
> sometime in the future.

To extend the topic with some discussion points:

- We are facing a rather familiar problem. Basically, SMB2 CHANGE_NOTIFY
  [1] is something what fsnotify normally does, except that we can't use
  it in cifsd kernel module. I see that NFS guys had [2] same issues some
  time ago.

  So the question is - how to do fs notify style monitoring in a kernel
  module (nfsd, cifsd)? Any chance the kernel can start exporting fsnotify
  symbols?

[1] https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/598f395a-e7a2-4cc8-afb3-ccb30dd2df7c
[2] https://lore.kernel.org/linux-fsdevel/1441968882-7851-5-git-send-email-jeff.layton@primarydata.com/

	-ss

Re: [Ksummit-discuss] [TECH TOPIC] Linux kernel SMB server (CIFSD)

[Sergey Senozhatsky]

On (07/24/19 14:52), Sergey Senozhatsky wrote:
> On (05/30/19 01:59), Theodore Ts'o wrote:
> > From: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
[..]
> To extend the topic with some discussion points:
> 
> - We are facing a rather familiar problem. Basically, SMB2 CHANGE_NOTIFY
>   [1] is something what fsnotify normally does, except that we can't use
>   it in cifsd kernel module. I see that NFS guys had [2] same issues some
>   time ago.
> 
>   So the question is - how to do fs notify style monitoring in a kernel
>   module (nfsd, cifsd)? Any chance the kernel can start exporting fsnotify
>   symbols?

I even looked at LSM hooks, because they do what we need - invoke
creat, open, truncate, chmod, chown, unlink, callbacks. But, first,
I do realize that LSM was not meant to be used as some sort of fsnotify
replacement, and, second, LSM symbols are not exported anyway :)

	-ss

Re: [Ksummit-discuss] [TECH TOPIC] Linux kernel SMB server (CIFSD)

[Amir Goldstein]

On Wed, Jul 24, 2019 at 11:03 AM Sergey Senozhatsky
<sergey.senozhatsky.work@gmail.com> wrote:
>
> On (07/24/19 14:52), Sergey Senozhatsky wrote:
> > On (05/30/19 01:59), Theodore Ts'o wrote:
> > > From: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
> [..]
> > To extend the topic with some discussion points:
> >
> > - We are facing a rather familiar problem. Basically, SMB2 CHANGE_NOTIFY
> >   [1] is something what fsnotify normally does, except that we can't use
> >   it in cifsd kernel module. I see that NFS guys had [2] same issues some
> >   time ago.
> >
> >   So the question is - how to do fs notify style monitoring in a kernel
> >   module (nfsd, cifsd)? Any chance the kernel can start exporting fsnotify
> >   symbols?

I don't see a problem with exporting those symbols.
As a matter of fact, I have a plan to use them also from overlayfs.
At the time that Jeff posted his patches, there was no active fsnotify
maintainer.
Jeff, did you abandon this effort for nfsd?

>
> I even looked at LSM hooks, because they do what we need - invoke
> creat, open, truncate, chmod, chown, unlink, callbacks. But, first,
> I do realize that LSM was not meant to be used as some sort of fsnotify
> replacement, and, second, LSM symbols are not exported anyway :)
>

fsnotify could actually be called from LSM hooks, see:
https://github.com/amir73il/linux/commits/fsnotify_dirent_perm

It may or may not be clear to you, but what you get extra from fsnotify
compared to using bare LSM hooks is:
- Minimal performance overhead for a non-marked object
- Ability to subscribe/unsubscribe to certain events dynamically
- Manage multiple subscriber instances (groups), like one per SMB session
- Queuing infrastructure for async events if you need it

Thanks,
Amir.

Re: [Ksummit-discuss] [TECH TOPIC] Linux kernel SMB server (CIFSD)

[Jeff Layton]

On Wed, 2019-07-24 at 13:50 +0300, Amir Goldstein wrote:
> On Wed, Jul 24, 2019 at 11:03 AM Sergey Senozhatsky
> <sergey.senozhatsky.work@gmail.com> wrote:
> > On (07/24/19 14:52), Sergey Senozhatsky wrote:
> > > On (05/30/19 01:59), Theodore Ts'o wrote:
> > > > From: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
> > [..]
> > > To extend the topic with some discussion points:
> > > 
> > > - We are facing a rather familiar problem. Basically, SMB2 CHANGE_NOTIFY
> > >   [1] is something what fsnotify normally does, except that we can't use
> > >   it in cifsd kernel module. I see that NFS guys had [2] same issues some
> > >   time ago.
> > > 
> > >   So the question is - how to do fs notify style monitoring in a kernel
> > >   module (nfsd, cifsd)? Any chance the kernel can start exporting fsnotify
> > >   symbols?
> 
> I don't see a problem with exporting those symbols.
> As a matter of fact, I have a plan to use them also from overlayfs.
> At the time that Jeff posted his patches, there was no active fsnotify
> maintainer.
> Jeff, did you abandon this effort for nfsd?
> 

(cc'ing Trond)

Trond has been driving those patches more recently. He posted a revised
set of them recently, and they still use fsnotify to detect unlink
activity:

https://www.spinics.net/lists/linux-nfs/msg73692.html

> > I even looked at LSM hooks, because they do what we need - invoke
> > creat, open, truncate, chmod, chown, unlink, callbacks. But, first,
> > I do realize that LSM was not meant to be used as some sort of fsnotify
> > replacement, and, second, LSM symbols are not exported anyway :)
> > 
> 
> fsnotify could actually be called from LSM hooks, see:
> https://github.com/amir73il/linux/commits/fsnotify_dirent_perm
> 
> It may or may not be clear to you, but what you get extra from fsnotify
> compared to using bare LSM hooks is:
> - Minimal performance overhead for a non-marked object
> - Ability to subscribe/unsubscribe to certain events dynamically
> - Manage multiple subscriber instances (groups), like one per SMB session
> - Queuing infrastructure for async events if you need it
-- 
Jeff Layton <jlayton@kernel.org>

Re: [Ksummit-discuss] [TECH TOPIC] Linux kernel SMB server (CIFSD)

[Sergey Senozhatsky]

On (07/24/19 13:50), Amir Goldstein wrote:
[..]
> I don't see a problem with exporting those symbols.
> As a matter of fact, I have a plan to use them also from overlayfs.
> At the time that Jeff posted his patches, there was no active fsnotify
> maintainer.

Cool.

> Jeff, did you abandon this effort for nfsd?
> 
> >
> > I even looked at LSM hooks, because they do what we need - invoke
> > creat, open, truncate, chmod, chown, unlink, callbacks. But, first,
> > I do realize that LSM was not meant to be used as some sort of fsnotify
> > replacement, and, second, LSM symbols are not exported anyway :)
> >
> 
> fsnotify could actually be called from LSM hooks, see:
> https://github.com/amir73il/linux/commits/fsnotify_dirent_perm
>
> It may or may not be clear to you, but what you get extra from fsnotify
> compared to using bare LSM hooks is:

Oh yeah, LSM does not really fit. I looked at it just out of curiosity.

> - Minimal performance overhead for a non-marked object
> - Ability to subscribe/unsubscribe to certain events dynamically

Absolutely, we need to "subscribe/unsubscribe dynamically".

> - Manage multiple subscriber instances (groups), like one per SMB session

We need this as well, you're right.

> - Queuing infrastructure for async events if you need it

	-ss

Re: [Ksummit-discuss] [TECH TOPIC] Linux kernel SMB server (CIFSD)

[Al Viro]

On Wed, Jul 24, 2019 at 01:50:57PM +0300, Amir Goldstein wrote:
> On Wed, Jul 24, 2019 at 11:03 AM Sergey Senozhatsky
> <sergey.senozhatsky.work@gmail.com> wrote:
> >
> > On (07/24/19 14:52), Sergey Senozhatsky wrote:
> > > On (05/30/19 01:59), Theodore Ts'o wrote:
> > > > From: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
> > [..]
> > > To extend the topic with some discussion points:
> > >
> > > - We are facing a rather familiar problem. Basically, SMB2 CHANGE_NOTIFY
> > >   [1] is something what fsnotify normally does, except that we can't use
> > >   it in cifsd kernel module. I see that NFS guys had [2] same issues some
> > >   time ago.
> > >
> > >   So the question is - how to do fs notify style monitoring in a kernel
> > >   module (nfsd, cifsd)? Any chance the kernel can start exporting fsnotify
> > >   symbols?
> 
> I don't see a problem with exporting those symbols.
> As a matter of fact, I have a plan to use them also from overlayfs.
> At the time that Jeff posted his patches, there was no active fsnotify
> maintainer.
> Jeff, did you abandon this effort for nfsd?
> 
> >
> > I even looked at LSM hooks, because they do what we need - invoke
> > creat, open, truncate, chmod, chown, unlink, callbacks. But, first,
> > I do realize that LSM was not meant to be used as some sort of fsnotify
> > replacement, and, second, LSM symbols are not exported anyway :)
> >
> 
> fsnotify could actually be called from LSM hooks, see:
> https://github.com/amir73il/linux/commits/fsnotify_dirent_perm
> 
> It may or may not be clear to you, but what you get extra from fsnotify
> compared to using bare LSM hooks is:
> - Minimal performance overhead for a non-marked object
> - Ability to subscribe/unsubscribe to certain events dynamically
> - Manage multiple subscriber instances (groups), like one per SMB session
> - Queuing infrastructure for async events if you need it

.... and then there's the sad fact that LSM hooks have no well-defined
semantics and their coverage is... suboptimal, to put it very mildly.

Re: [Ksummit-discuss] [TECH TOPIC] Linux kernel SMB server (CIFSD)

[Sergey Senozhatsky]

On (07/24/19 06:55), Jeff Layton wrote:
> (cc'ing Trond)
> 
> Trond has been driving those patches more recently. He posted a revised
> set of them recently, and they still use fsnotify to detect unlink
> activity:
> 
> https://www.spinics.net/lists/linux-nfs/msg73692.html

Jan, do you have objections?
Would be great to have fsnotify export patch in linux-next, maybe.

	-ss

Re: [Ksummit-discuss] [TECH TOPIC] Linux kernel SMB server (CIFSD)

[Jan Kara]

On Wed 24-07-19 06:55:53, Jeff Layton wrote:
> On Wed, 2019-07-24 at 13:50 +0300, Amir Goldstein wrote:
> > On Wed, Jul 24, 2019 at 11:03 AM Sergey Senozhatsky
> > <sergey.senozhatsky.work@gmail.com> wrote:
> > > On (07/24/19 14:52), Sergey Senozhatsky wrote:
> > > > On (05/30/19 01:59), Theodore Ts'o wrote:
> > > > > From: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
> > > [..]
> > > > To extend the topic with some discussion points:
> > > > 
> > > > - We are facing a rather familiar problem. Basically, SMB2 CHANGE_NOTIFY
> > > >   [1] is something what fsnotify normally does, except that we can't use
> > > >   it in cifsd kernel module. I see that NFS guys had [2] same issues some
> > > >   time ago.
> > > > 
> > > >   So the question is - how to do fs notify style monitoring in a kernel
> > > >   module (nfsd, cifsd)? Any chance the kernel can start exporting fsnotify
> > > >   symbols?
> > 
> > I don't see a problem with exporting those symbols.
> > As a matter of fact, I have a plan to use them also from overlayfs.
> > At the time that Jeff posted his patches, there was no active fsnotify
> > maintainer.
> > Jeff, did you abandon this effort for nfsd?
> > 
> 
> (cc'ing Trond)
> 
> Trond has been driving those patches more recently. He posted a revised
> set of them recently, and they still use fsnotify to detect unlink
> activity:
> 
> https://www.spinics.net/lists/linux-nfs/msg73692.html

The fsnotify export bits look sane to me except perhaps
fsnotify_put_group() and fsnotify_wait_marks_destroyed(). Their usage in
"[PATCH 05/16] nfsd: add a new struct file caching facility to nfsd" looks
buggy AFAICT. One is supposed to destroy a group with
fsnotify_destroy_group() instead of just dropping the group reference (that
does just the final cleanup but expects fsnotify_destroy_group() has been
already called).

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: [Ksummit-discuss] [TECH TOPIC] Linux kernel SMB server (CIFSD)

[Jan Kara]

On Thu 25-07-19 14:00:59, Sergey Senozhatsky wrote:
> On (07/24/19 06:55), Jeff Layton wrote:
> > (cc'ing Trond)
> > 
> > Trond has been driving those patches more recently. He posted a revised
> > set of them recently, and they still use fsnotify to detect unlink
> > activity:
> > 
> > https://www.spinics.net/lists/linux-nfs/msg73692.html
> 
> Jan, do you have objections?
> Would be great to have fsnotify export patch in linux-next, maybe.

I have no objections to export bits of fsnotify to modules. After all
kernel/audit_tree.c is another external user of the fsnotify bits, just it
happens not to be modularized so there was never need for exports.

Once NFS guys (or someone else) have something that is close to inclusion,
I can push the export patch to my tree (just CC me, Amir, and fsdevel on the
series). I prefer not to export symbols for something that is out of tree
and will not be getting upstream in a near future. After all the export
patch is easy to carry in your tree if you need it.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: [Ksummit-discuss] [TECH TOPIC] Linux kernel SMB server (CIFSD)

[Sergey Senozhatsky]

On (07/30/19 19:01), Jan Kara wrote:
> On Thu 25-07-19 14:00:59, Sergey Senozhatsky wrote:
> > On (07/24/19 06:55), Jeff Layton wrote:
> > > (cc'ing Trond)
> > > 
> > > Trond has been driving those patches more recently. He posted a revised
> > > set of them recently, and they still use fsnotify to detect unlink
> > > activity:
> > > 
> > > https://www.spinics.net/lists/linux-nfs/msg73692.html
> > 
> > Jan, do you have objections?
> > Would be great to have fsnotify export patch in linux-next, maybe.
> 
> I have no objections to export bits of fsnotify to modules. After all
> kernel/audit_tree.c is another external user of the fsnotify bits, just it
> happens not to be modularized so there was never need for exports.

That's great. At least we can expect that fsnotify will be exported
at some point and can start playing with fsnotify-based smb2_notify
implementation.

> Once NFS guys (or someone else) have something that is close to inclusion,
> I can push the export patch to my tree (just CC me, Amir, and fsdevel on the
> series).

OK, both OverlayFS and NFS have some plans for fsnotify, so let's see.

> I prefer not to export symbols for something that is out of tree
> and will not be getting upstream in a near future. After all the export
> patch is easy to carry in your tree if you need it.

Makes perfect sense.

	-ss