Re: [PATCH 0/6] vfs: provide automatic kernel freeze / resume

Re: [PATCH 0/6] vfs: provide automatic kernel freeze / resume

[Askar Safin]

Will this patch fix a long-standing fuse vs suspend bug? (
https://bugzilla.kernel.org/show_bug.cgi?id=34932 )

Please, CC when answering

-- 
Askar Safin

Re: [PATCH 0/6] vfs: provide automatic kernel freeze / resume

[Miklos Szeredi]

[-- Attachment #1: Type: text/plain, Size: 1666 bytes --]

On Sun, 14 May 2023 at 00:04, Askar Safin <safinaskar@gmail.com> wrote:
>
> Will this patch fix a long-standing fuse vs suspend bug? (
> https://bugzilla.kernel.org/show_bug.cgi?id=34932 )

No.

The solution to the fuse issue is to freeze processes that initiate
fuse requests *before* freezing processes that serve fuse requests.

The problem is finding out which is which.  This can be complicated by
the fact that a process could be both serving requests *and*
initiating them (even without knowing).

The best idea so far is to let fuse servers set a process flag
(PF_FREEZE_LATE) that is inherited across fork/clone.  For example the
sshfs server would do the following before starting request processing
or starting ssh:

  echo 1 > /proc/self/freeze_late

This would make the sshfs and ssh processes be frozen after processes
that call into the sshfs mount.

After normal (non-server) processes are frozen, server processes
should not be getting new requests and can be frozen.

Issues remaining:

 - if requests are stuck (e.g. network is down) then the requester
process can't be frozen and suspend will still fail.

 - if server process is generating filesystem activity (new fuse
requests) spontaneously, then there's nothing to differentiate between
server processes and we are back to the original problem

Solution to both these are probably non-kernel: impacted servers need
to receive notification from systemd when suspend is starting and act
accordingly.

Attaching work-in-progress patch.  This needs to be improved to freeze
server processes in a separate phase from kernel threads, but it
should be able to demonstrate the idea.

Thanks,
Miklos

[-- Attachment #2: 0001_freezer_configure_user_space_process_frozen_along_with_kernel_threads.patch --]
[-- Type: application/x-patch, Size: 7114 bytes --]

Re: [PATCH 0/6] vfs: provide automatic kernel freeze / resume

[Askar Safin]

Thanks a lot for the answer

On Tue, Jun 6, 2023 at 5:38 PM Miklos Szeredi <miklos@szeredi.hu> wrote:
>  - if requests are stuck (e.g. network is down) then the requester
> process can't be frozen and suspend will still fail

Unfortunately, this is exactly the problem I sometimes face. If
network is up, then suspend works normally. But if network is down
(and I'm trying to access sshfs filesystem in that moment), then
suspend doesn't work.

So, it seems your solution is not for me

> Solution to both these are probably non-kernel: impacted servers need
> to receive notification from systemd when suspend is starting and act
> accordingly.
Okay, I will probably forward this to sshfs devs

-- 
Askar Safin

Re: [fuse-devel] [PATCH 0/6] vfs: provide automatic kernel freeze / resume

[Nikolaus Rath]

On Jun 06 2023, Miklos Szeredi via fuse-devel <fuse-devel@lists.sourceforge.net> wrote:
> On Sun, 14 May 2023 at 00:04, Askar Safin <safinaskar@gmail.com> wrote:
>>
>> Will this patch fix a long-standing fuse vs suspend bug? (
>> https://bugzilla.kernel.org/show_bug.cgi?id=34932 )
>
> No.
>
> The solution to the fuse issue is to freeze processes that initiate
> fuse requests *before* freezing processes that serve fuse requests.
>
> The problem is finding out which is which.  This can be complicated by
> the fact that a process could be both serving requests *and*
> initiating them (even without knowing).
>
> The best idea so far is to let fuse servers set a process flag
> (PF_FREEZE_LATE) that is inherited across fork/clone.

Is that the same as what userspace calls PR_SET_IO_FLUSHER? From
prctl(2):

   PR_SET_IO_FLUSHER (since Linux 5.6)
          If a user process is involved in the block layer or filesystem I/O  path,  and
          can allocate memory while processing I/O requests it must set arg2 to 1.  This
          will put the process in the IO_FLUSHER state, which allows it  special  treat‐
          ment  to make progress when allocating memory. [..]

          The calling process must have the CAP_SYS_RESOURCE capability.[...]

          Examples  of  IO_FLUSHER  applications are FUSE daemons, SCSI device emulation
          daemons, and daemons that perform error handling like multipath path  recovery
          applications.
          

To me this sounds like it captures the relevant information (process is
involved in filesystem I/O) rather than just a preferred behavior (flush
late) and may thus be a better choice...

Best,
-Nikolaus

Re: [PATCH 0/6] vfs: provide automatic kernel freeze / resume

[Bernd Schubert]

On 6/6/23 16:37, Miklos Szeredi wrote:
> On Sun, 14 May 2023 at 00:04, Askar Safin <safinaskar@gmail.com> wrote:
>>
>> Will this patch fix a long-standing fuse vs suspend bug? (
>> https://bugzilla.kernel.org/show_bug.cgi?id=34932 )
> 
> No.
> 
> The solution to the fuse issue is to freeze processes that initiate
> fuse requests *before* freezing processes that serve fuse requests.
> 
> The problem is finding out which is which.  This can be complicated by
> the fact that a process could be both serving requests *and*
> initiating them (even without knowing).
> 
> The best idea so far is to let fuse servers set a process flag
> (PF_FREEZE_LATE) that is inherited across fork/clone.  For example the
> sshfs server would do the following before starting request processing
> or starting ssh:
> 
>    echo 1 > /proc/self/freeze_late
> 
> This would make the sshfs and ssh processes be frozen after processes
> that call into the sshfs mount.

Hmm, why would this need to be done manually on the server (daemon) 
side? It could be automated on the fuse kernel side, for example in 
process_init_reply() using current task context?

A slightly better version would give scores, the later the daemon/server 
is created the higher its freezing score - would help a bit with stacked 
fuse file systems, although not perfectly. For that struct task would 
need to be extended, though.

> 
> After normal (non-server) processes are frozen, server processes
> should not be getting new requests and can be frozen.
> 
> Issues remaining:
> 
>   - if requests are stuck (e.g. network is down) then the requester
> process can't be frozen and suspend will still fail.
> 
>   - if server process is generating filesystem activity (new fuse
> requests) spontaneously, then there's nothing to differentiate between
> server processes and we are back to the original problem
> 
> Solution to both these are probably non-kernel: impacted servers need
> to receive notification from systemd when suspend is starting and act
> accordingly.
> 
> Attaching work-in-progress patch.  This needs to be improved to freeze
> server processes in a separate phase from kernel threads, but it
> should be able to demonstrate the idea.


Thanks,
Bernd

Re: [fuse-devel] [PATCH 0/6] vfs: provide automatic kernel freeze / resume

[Miklos Szeredi]

On Tue, 6 Jun 2023 at 21:37, Nikolaus Rath <Nikolaus@rath.org> wrote:
>
> On Jun 06 2023, Miklos Szeredi via fuse-devel <fuse-devel@lists.sourceforge.net> wrote:
> > On Sun, 14 May 2023 at 00:04, Askar Safin <safinaskar@gmail.com> wrote:
> >>
> >> Will this patch fix a long-standing fuse vs suspend bug? (
> >> https://bugzilla.kernel.org/show_bug.cgi?id=34932 )
> >
> > No.
> >
> > The solution to the fuse issue is to freeze processes that initiate
> > fuse requests *before* freezing processes that serve fuse requests.
> >
> > The problem is finding out which is which.  This can be complicated by
> > the fact that a process could be both serving requests *and*
> > initiating them (even without knowing).
> >
> > The best idea so far is to let fuse servers set a process flag
> > (PF_FREEZE_LATE) that is inherited across fork/clone.
>
> Is that the same as what userspace calls PR_SET_IO_FLUSHER? From
> prctl(2):
>
>    PR_SET_IO_FLUSHER (since Linux 5.6)
>           If a user process is involved in the block layer or filesystem I/O  path,  and
>           can allocate memory while processing I/O requests it must set arg2 to 1.  This
>           will put the process in the IO_FLUSHER state, which allows it  special  treat‐
>           ment  to make progress when allocating memory. [..]
>
>           The calling process must have the CAP_SYS_RESOURCE capability.[...]

This is the issue.   We want suspend to work without needing privileges.

>
>           Examples  of  IO_FLUSHER  applications are FUSE daemons, SCSI device emulation
>           daemons, and daemons that perform error handling like multipath path  recovery
>           applications.

This looks incorrect.  FUSE shouldn't need this because it manages
writeback in a way not to require such special treatment.

It might make sense to use the prctl(2) API for this, but honestly I
prefer pseudo fs interfaces for such knobs.

Thanks,
Miklos

Re: [PATCH 0/6] vfs: provide automatic kernel freeze / resume

[Miklos Szeredi]

On Tue, 6 Jun 2023 at 22:18, Bernd Schubert <bernd.schubert@fastmail.fm> wrote:
>
>
>
> On 6/6/23 16:37, Miklos Szeredi wrote:
> > On Sun, 14 May 2023 at 00:04, Askar Safin <safinaskar@gmail.com> wrote:
> >>
> >> Will this patch fix a long-standing fuse vs suspend bug? (
> >> https://bugzilla.kernel.org/show_bug.cgi?id=34932 )
> >
> > No.
> >
> > The solution to the fuse issue is to freeze processes that initiate
> > fuse requests *before* freezing processes that serve fuse requests.
> >
> > The problem is finding out which is which.  This can be complicated by
> > the fact that a process could be both serving requests *and*
> > initiating them (even without knowing).
> >
> > The best idea so far is to let fuse servers set a process flag
> > (PF_FREEZE_LATE) that is inherited across fork/clone.  For example the
> > sshfs server would do the following before starting request processing
> > or starting ssh:
> >
> >    echo 1 > /proc/self/freeze_late
> >
> > This would make the sshfs and ssh processes be frozen after processes
> > that call into the sshfs mount.
>
> Hmm, why would this need to be done manually on the server (daemon)
> side? It could be automated on the fuse kernel side, for example in
> process_init_reply() using current task context?

Setting the flag for the current task wouldn't be sufficient, it would
need to set it for all threads of a process.  Even that wouldn't work
for e.g. sshfs, which forks off ssh before starting request
processing.

So I'd prefer setting this explicitly.   This could be done from
libfuse, before starting threads.  Or, as in the case of sshfs, it
could be done by the filesystem itself.

>
> A slightly better version would give scores, the later the daemon/server
> is created the higher its freezing score - would help a bit with stacked
> fuse file systems, although not perfectly. For that struct task would
> need to be extended, though.

If we can quiesce the top of the stack, then hopefully all the lower
ones will also have no activity.   There could be special cases, but
that would need to be dealt with in the fuse server itself.

Thanks,
Miklos

Re: [PATCH 0/6] vfs: provide automatic kernel freeze / resume

[Bernd Schubert]

On 6/7/23 09:21, Miklos Szeredi wrote:
> On Tue, 6 Jun 2023 at 22:18, Bernd Schubert <bernd.schubert@fastmail.fm> wrote:
>>
>>
>>
>> On 6/6/23 16:37, Miklos Szeredi wrote:
>>> On Sun, 14 May 2023 at 00:04, Askar Safin <safinaskar@gmail.com> wrote:
>>>>
>>>> Will this patch fix a long-standing fuse vs suspend bug? (
>>>> https://bugzilla.kernel.org/show_bug.cgi?id=34932 )
>>>
>>> No.
>>>
>>> The solution to the fuse issue is to freeze processes that initiate
>>> fuse requests *before* freezing processes that serve fuse requests.
>>>
>>> The problem is finding out which is which.  This can be complicated by
>>> the fact that a process could be both serving requests *and*
>>> initiating them (even without knowing).
>>>
>>> The best idea so far is to let fuse servers set a process flag
>>> (PF_FREEZE_LATE) that is inherited across fork/clone.  For example the
>>> sshfs server would do the following before starting request processing
>>> or starting ssh:
>>>
>>>     echo 1 > /proc/self/freeze_late
>>>
>>> This would make the sshfs and ssh processes be frozen after processes
>>> that call into the sshfs mount.
>>
>> Hmm, why would this need to be done manually on the server (daemon)
>> side? It could be automated on the fuse kernel side, for example in
>> process_init_reply() using current task context?
> 
> Setting the flag for the current task wouldn't be sufficient, it would
> need to set it for all threads of a process.  Even that wouldn't work
> for e.g. sshfs, which forks off ssh before starting request
> processing.

Assuming a fuse server process is not handing over requests to other 
threads/forked-processes, isn't the main issue that all fuse server 
tasks are frozen and none is left to take requests? A single non-frozen 
thread should be sufficient for that?


> 
> So I'd prefer setting this explicitly.   This could be done from
> libfuse, before starting threads.  Or, as in the case of sshfs, it
> could be done by the filesystem itself.

With a flag that should work, with my score proposal it would be difficult.

> 
>>
>> A slightly better version would give scores, the later the daemon/server
>> is created the higher its freezing score - would help a bit with stacked
>> fuse file systems, although not perfectly. For that struct task would
>> need to be extended, though.
> 
> If we can quiesce the top of the stack, then hopefully all the lower
> ones will also have no activity.   There could be special cases, but
> that would need to be dealt with in the fuse server itself.


Ah, when all non flagged processes are frozen first no IO should come 
in. Yeah, it mostly works, but I wonder if init/systemd is not going to 
set that flag as well. And then you have an issue when fuse is on a file 
system used by systemd. My long time ago initial interest on fuse is to 
use fuse as root file system and I still do that for some cases - not 
sure if a flag would be sufficient here. I think a freezing score would 
solve more issues.
Although probably better to do step by step - flag first and score can 
be added later.



Thanks,
Bernd

Re: [PATCH 0/6] vfs: provide automatic kernel freeze / resume

[Miklos Szeredi]

On Wed, 7 Jun 2023 at 10:13, Bernd Schubert <bernd.schubert@fastmail.fm> wrote:

> Assuming a fuse server process is not handing over requests to other
> threads/forked-processes, isn't the main issue that all fuse server
> tasks are frozen and none is left to take requests? A single non-frozen
> thread should be sufficient for that?

This *might* work.  But there could be auxiliary threads, or the
initial thread could be killed, etc.  It would not be reliable.

> Ah, when all non flagged processes are frozen first no IO should come
> in. Yeah, it mostly works, but I wonder if init/systemd is not going to
> set that flag as well. And then you have an issue when fuse is on a file
> system used by systemd. My long time ago initial interest on fuse is to
> use fuse as root file system and I still do that for some cases - not
> sure if a flag would be sufficient here. I think a freezing score would
> solve more issues.
> Although probably better to do step by step - flag first and score can
> be added later.

I'm not sure how systemd interacts with the freezing process.  If it
does, then the only sane thing to do is to make sure it doesn't have
any filesystem interaction during that time.

Hibernation is a different matter, because it needs the filesystem to
be in a working state while all userspace is frozen.  Hibernate to
fuse would bring up a lot of interesting design questions.

Thanks,
Miklos

Re: [PATCH 0/6] vfs: provide automatic kernel freeze / resume

[Askar Safin]

I found a workaround for sshfs+suspend problem!

On Tue, Jun 6, 2023 at 5:38 PM Miklos Szeredi <miklos@szeredi.hu> wrote:
> Issues remaining:
>
>  - if requests are stuck (e.g. network is down) then the requester
> process can't be frozen and suspend will still fail.

> Solution to both these are probably non-kernel: impacted servers need
> to receive notification from systemd when suspend is starting and act
> accordingly.

Okay, so you said that the only way to solve the problem "network is
down" is to fix the problem at the sshfs side. Unfortunately, sshfs
project was closed ( https://github.com/libfuse/sshfs ). So the only
remaining option is to use some hack. And I found such a hack!

I simply added "-o ServerAliveInterval=10" to sshfs command. This will
cause ssh process exit if remote side is unreachable. Thus the bug is
prevented. I tested the fix and it works.

But this will mean that ssh process will exit in such situation, and
thus sshfs process will exit, too. If this is not what you want, then
also add "-o reconnect" option, this will restart connection if ssh
dies. So the final command will look like this:

sshfs -o reconnect,ServerAliveInterval=10 ...

This finally solved the problem for me.

But one issue still remains: if the network goes down and then you
immediately try to access sshfs filesystem and then you will try to
suspend (and ssh doesn't yet noticed that the network gone down), then
suspend will still fail. (I tested this situation, the suspend
actually fails.) But I don't think I even will reach such situation.
The lesser ServerAliveInterval you will set, the lower is probability
that you will reach such situation

-- 
Askar Safin

Re: [PATCH 0/6] vfs: provide automatic kernel freeze / resume

[Miklos Szeredi]

On Wed, 7 Jun 2023 at 13:13, Askar Safin <safinaskar@gmail.com> wrote:
>
> I found a workaround for sshfs+suspend problem!
>
> On Tue, Jun 6, 2023 at 5:38 PM Miklos Szeredi <miklos@szeredi.hu> wrote:
> > Issues remaining:
> >
> >  - if requests are stuck (e.g. network is down) then the requester
> > process can't be frozen and suspend will still fail.
>
> > Solution to both these are probably non-kernel: impacted servers need
> > to receive notification from systemd when suspend is starting and act
> > accordingly.
>
> Okay, so you said that the only way to solve the problem "network is
> down" is to fix the problem at the sshfs side. Unfortunately, sshfs
> project was closed ( https://github.com/libfuse/sshfs ). So the only
> remaining option is to use some hack. And I found such a hack!
>
> I simply added "-o ServerAliveInterval=10" to sshfs command. This will
> cause ssh process exit if remote side is unreachable. Thus the bug is
> prevented. I tested the fix and it works.
>
> But this will mean that ssh process will exit in such situation, and
> thus sshfs process will exit, too. If this is not what you want, then
> also add "-o reconnect" option, this will restart connection if ssh
> dies. So the final command will look like this:
>
> sshfs -o reconnect,ServerAliveInterval=10 ...
>
> This finally solved the problem for me.
>
> But one issue still remains: if the network goes down and then you
> immediately try to access sshfs filesystem and then you will try to
> suspend (and ssh doesn't yet noticed that the network gone down), then
> suspend will still fail.

If the ServerAliveInterval is set to less than the freeze timeout (20s
by default) and you apply the patch and start sshfs like below, then
suspend should be reliable.

  (echo 1 >  /proc/self/freeze_late; sshfs ...)

Thanks,
Miklos

Re: [PATCH 0/6] vfs: provide automatic kernel freeze / resume

[Askar Safin]

> If the ServerAliveInterval is set to less than the freeze timeout (20s
> by default) and you apply the patch and start sshfs like below, then
> suspend should be reliable.
>
>   (echo 1 >  /proc/self/freeze_late; sshfs ...)

Thanks a lot. I don't want to apply patches. I'm happy with my solution.

Note for other people reading this thread: ssh docs say that actual
timeout is "ServerAliveInterval * ServerAliveCountMax", so if someone
will apply this patch, he should ensure that ServerAliveInterval *
ServerAliveCountMax is less than 20 seconds.


-- 
Askar Safin

Re: [PATCH 0/6] vfs: provide automatic kernel freeze / resume

[Luis Chamberlain]

I forgot to mention if you wanna test on a git tree:

https://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux-next.git/log/?h=20230507-fs-freeze

  Luis

[PATCH 0/6] vfs: provide automatic kernel freeze / resume

[Luis Chamberlain]

It's been about 5 months since the last v3 RFC for fs freeze. Now
that some of us will have time at LSFMM to discuss stuff I figured
it would be good to try to get the last pieces, if any discussed,
and put out a new patch set based on the latests feedback. This
time I boot tested, and stress tested the patches. From what I can
tell I confirm nothing regresses, we just end up now with a new world
where if your platform does support s3  / s4 we'll kick into gear the
automatic kernel freeze.

To help with testing, as this is a rather tiny bit obscure area with
virtualization, I've gone ahead and extended kdevops with support for
always enabling s3 / s4, so it should be easy to test guest bring up
there.

I've picked out using stress-ng now to have fun stress testing suspend,
the insane will try:

./stress-ng --touch 8192 --touch-method creat

Resume works but eventually suspend will trickle tons of OOMs and so
we gotta find a sweet spot to automate this somehow in fstests somehow.
I am not sure how we're gonna test this moving forward on fstests but
perhaps worth talking about at LSFMM for ideas.

Anyway, your filesystem will not participate in the auto kernel
freeze / thaw unless your filesystem gets the kthread freezer junk
removed and sets a flag. I'll post 3 patches for the 3 main filesystems
after this. I've carried and advanced the SmPL patch for a few years
now, and magically it all still works.

[0] https://lkml.kernel.org/r/20230114003409.1168311-1-mcgrof@kernel.org

Luis Chamberlain (6):
  fs: unify locking semantics for fs freeze / thaw
  fs: add frozen sb state helpers
  fs: distinguish between user initiated freeze and kernel initiated
    freeze
  fs: move !SB_BORN check early on freeze and add for thaw
  fs: add iterate_supers_excl() and iterate_supers_reverse_excl()
  fs: add automatic kernel fs freeze / thaw and remove kthread freezing

 block/bdev.c           |   9 +-
 fs/ext4/ext4_jbd2.c    |   2 +-
 fs/f2fs/gc.c           |   9 +-
 fs/gfs2/glops.c        |   2 +-
 fs/gfs2/super.c        |  11 +-
 fs/gfs2/sys.c          |  12 ++-
 fs/gfs2/util.c         |   7 +-
 fs/ioctl.c             |  14 ++-
 fs/quota/quota.c       |   4 +-
 fs/super.c             | 239 +++++++++++++++++++++++++++++++++--------
 fs/xfs/xfs_trans.c     |   3 +-
 include/linux/fs.h     |  54 +++++++++-
 kernel/power/process.c |  15 ++-
 13 files changed, 313 insertions(+), 68 deletions(-)

-- 
2.39.2