Re: [patch 4/5] sched: Delay task stack freeing on RT

[Andy Lutomirski <luto@kernel.org>]

On Fri, Oct 1, 2021 at 10:24 AM Thomas Gleixner <tglx@linutronix.de> wrote:
>
> On Fri, Oct 01 2021 at 09:12, Andy Lutomirski wrote:
> > On Wed, Sep 29, 2021 at 4:54 AM Peter Zijlstra <peterz@infradead.org> wrote:
> >> Having this logic split across two files seems unfortunate and prone to
> >> 'accidents'. Is there a real down-side to unconditionally doing it in
> >> delayed_put_task_struct() ?
> >>
> >> /me goes out for lunch... meanwhile tglx points at: 68f24b08ee89.
> >>
> >> Bah.. Andy?
> >
> > Could we make whatever we do here unconditional?
>
> Sure. I just was unsure about your reasoning in 68f24b08ee89.

Mmm, right.  The reasoning is that there are a lot of workloads that
frequently wait for a task to exit and immediately start a new task --
most shell scripts, for example.  I think I tested this with the
following amazing workload:

while true; do true; done

and we want to reuse the same stack each time from the cached stack
lookaside list instead of vfreeing and vmallocing a stack each time.
Deferring the release to the lookaside list breaks it.  Although I
suppose the fact that it works well right now is a bit fragile --
we're waking the parent (sh, etc) before releasing the stack, but
nothing gets to run until the stack is released.

>
> > And what actually causes the latency?  If it's vfree, shouldn't the
> > existing use of vfree_atomic() in free_thread_stack() handle it?  Or
> > is it the accounting?
>
> The accounting muck because it can go into the allocator and sleep in
> the worst case, which is nasty even on !RT kernels.

Wait, unaccounting memory can go into the allocator?  That seems quite nasty.

>
> But thinking some more, there is actually a way nastier issue on RT in
> the following case:
>
> CPU 0                           CPU 1
>   T1
>   spin_lock(L1)
>   rt_mutex_lock()
>       schedule()
>
>   T2
>      do_exit()
>      do_task_dead()             spin_unlock(L1)
>                                    wake(T1)
>      __schedule()
>        switch_to(T1)
>        finish_task_switch()
>          put_task_stack()
>            account()
>              ....
>              spin_lock(L2)
>
> So if L1 == L2 or L1 and L2 have a reverse dependency then this can just
> deadlock.
>
> We've never observed that, but the above case is obviously hard to
> hit. Nevertheless it's there.

Hmm.

ISTM it would be conceptually for do_exit() to handle its own freeing
in its own preemptible context.  Obviously that can't really work,
since we can't free a task_struct or a task stack while we're running
on it.  But I wonder if we could approximate it by putting this work
in a workqueue so that it all runs in a normal schedulable context.
To make the shell script case work nicely, we want to release the task
stack before notifying anyone waiting for the dying task to exit, but
maybe that's doable.  It could involve some nasty exit_signal hackery,
though.