Re: INFO: task hung in ext4_da_get_block_prep

Re: INFO: task hung in ext4_da_get_block_prep

[Tetsuo Handa]

On 2018/09/05 16:22, Dmitry Vyukov wrote:
> On Wed, Sep 5, 2018 at 5:41 AM, syzbot
> <syzbot+f0fc7f62e88b1de99af3@syzkaller.appspotmail.com> wrote:
>> Hello,
>>
>> syzbot found the following crash on:
>>
>> HEAD commit:    f2b6e66e9885 Add linux-next specific files for 20180904
>> git tree:       linux-next
>> console output: https://syzkaller.appspot.com/x/log.txt?x=1735dc92400000
>> kernel config:  https://syzkaller.appspot.com/x/.config?x=15ad48400e39c1b3
>> dashboard link: https://syzkaller.appspot.com/bug?extid=f0fc7f62e88b1de99af3
>> compiler:       gcc (GCC) 8.0.1 20180413 (experimental)
>> CC:             [adilger.kernel@dilger.ca linux-ext4@vger.kernel.org
>> linux-kernel@vger.kernel.org tytso@mit.edu]
>>
>> Unfortunately, I don't have any reproducer for this crash yet.
>>
>> IMPORTANT: if you fix the bug, please add the following tag to the commit:
>> Reported-by: syzbot+f0fc7f62e88b1de99af3@syzkaller.appspotmail.com
>>
>> [   7961]     0  7961    17585     8737   131072        0             0
>> syz-executor3
> 
> Hi Tetsuo,
> 
> Maybe you know what are these repeated lines with numbers?
> We started getting them on linux-next recently, also:
> https://syzkaller.appspot.com/bug?extid=f8fa79b458bcae4d913d
> They seem to cause various hangs/stalls.

Yes, these lines are from the OOM killer. (Thus, if we can, I want to
remove ext4 people before upstreaming this report.)

  dump_tasks mm/oom_kill.c:420 [inline]
  dump_header+0xf0d/0xf70 mm/oom_kill.c:450
  oom_kill_process.cold.28+0x10/0x95a mm/oom_kill.c:953
  out_of_memory+0xa88/0x1430 mm/oom_kill.c:1120

What is annoying is that one for_each_process() traversal with printk() is
taking 52 seconds which is too long to do under RCU section. Under such
situation, invoking the OOM killer for three times will exceed khungtaskd
threshold 140 seconds. Was syzbot trying to test fork bomb situation?

Anyway, we might need to introduce rcu_lock_break() like
check_hung_uninterruptible_tasks() does...

[  999.629589] [  16497]     0 16497    17585     8739   126976        0             0 syz-executor5
[ 1026.435955] [  32764]     0 32764    17585     8739   126976        0             0 syz-executor5
[ 1026.445027] [    311]     0   311    17585     8737   131072        0             0 syz-executor3
[ 1047.914324] [  10315]     0 10315    17585     8271   126976        0             0 syz-executor0
[ 1047.923384] Out of memory: Kill process 4670 (syz-fuzzer) score 53 or sacrifice child
[ 1047.931934] Killed process 5032 (syz-executor1) total-vm:70212kB, anon-rss:60kB, file-rss:0kB, shmem-rss:0kB
[ 1047.988138] syz-executor2 invoked oom-killer: gfp_mask=0x6040c0(GFP_KERNEL|__GFP_COMP), nodemask=(null), order=1, oom_score_adj=0
[ 1048.000015] syz-executor2 cpuset=syz2 mems_allowed=0
[ 1048.005199] CPU: 0 PID: 4700 Comm: syz-executor2 Not tainted 4.19.0-rc2-next-20180904+ #55
[ 1048.740679] [   2347]     0  2347      278      186    32768        0             0 none
[ 1051.319928] [  16497]     0 16497    17585     8739   126976        0             0 syz-executor5
[ 1096.740878] [   8841]     0  8841    17585     8232   126976        0             0 syz-executor5
[ 1078.140677] [  32764]     0 32764    17585     8739   126976        0             0 syz-executor5
[ 1078.149807] [    311]     0   311    17585     8737   131072        0             0 syz-executor3
[ 1096.740878] [   8841]     0  8841    17585     8232   126976        0             0 syz-executor5

Also, another notable thing is that the backtrace for some reason includes

[ 1048.211540]  ? oom_killer_disable+0x3a0/0x3a0

line. Was syzbot testing process freezing functionality?

Re: INFO: task hung in ext4_da_get_block_prep

[Dmitry Vyukov]

On Wed, Sep 5, 2018 at 12:53 PM, Tetsuo Handa
<penguin-kernel@i-love.sakura.ne.jp> wrote:
> On 2018/09/05 16:22, Dmitry Vyukov wrote:
>> On Wed, Sep 5, 2018 at 5:41 AM, syzbot
>> <syzbot+f0fc7f62e88b1de99af3@syzkaller.appspotmail.com> wrote:
>>> Hello,
>>>
>>> syzbot found the following crash on:
>>>
>>> HEAD commit:    f2b6e66e9885 Add linux-next specific files for 20180904
>>> git tree:       linux-next
>>> console output: https://syzkaller.appspot.com/x/log.txt?x=1735dc92400000
>>> kernel config:  https://syzkaller.appspot.com/x/.config?x=15ad48400e39c1b3
>>> dashboard link: https://syzkaller.appspot.com/bug?extid=f0fc7f62e88b1de99af3
>>> compiler:       gcc (GCC) 8.0.1 20180413 (experimental)
>>> CC:             [adilger.kernel@dilger.ca linux-ext4@vger.kernel.org
>>> linux-kernel@vger.kernel.org tytso@mit.edu]
>>>
>>> Unfortunately, I don't have any reproducer for this crash yet.
>>>
>>> IMPORTANT: if you fix the bug, please add the following tag to the commit:
>>> Reported-by: syzbot+f0fc7f62e88b1de99af3@syzkaller.appspotmail.com
>>>
>>> [   7961]     0  7961    17585     8737   131072        0             0
>>> syz-executor3
>>
>> Hi Tetsuo,
>>
>> Maybe you know what are these repeated lines with numbers?
>> We started getting them on linux-next recently, also:
>> https://syzkaller.appspot.com/bug?extid=f8fa79b458bcae4d913d
>> They seem to cause various hangs/stalls.
>
> Yes, these lines are from the OOM killer. (Thus, if we can, I want to
> remove ext4 people before upstreaming this report.)

This is not possible at the moment.

>   dump_tasks mm/oom_kill.c:420 [inline]
>   dump_header+0xf0d/0xf70 mm/oom_kill.c:450
>   oom_kill_process.cold.28+0x10/0x95a mm/oom_kill.c:953
>   out_of_memory+0xa88/0x1430 mm/oom_kill.c:1120
>
> What is annoying is that one for_each_process() traversal with printk() is
> taking 52 seconds which is too long to do under RCU section. Under such
> situation, invoking the OOM killer for three times will exceed khungtaskd
> threshold 140 seconds. Was syzbot trying to test fork bomb situation?

Hard to tell. I only know what's captured in the console output.

> Anyway, we might need to introduce rcu_lock_break() like
> check_hung_uninterruptible_tasks() does...
>
> [  999.629589] [  16497]     0 16497    17585     8739   126976        0             0 syz-executor5
> [ 1026.435955] [  32764]     0 32764    17585     8739   126976        0             0 syz-executor5
> [ 1026.445027] [    311]     0   311    17585     8737   131072        0             0 syz-executor3
> [ 1047.914324] [  10315]     0 10315    17585     8271   126976        0             0 syz-executor0
> [ 1047.923384] Out of memory: Kill process 4670 (syz-fuzzer) score 53 or sacrifice child
> [ 1047.931934] Killed process 5032 (syz-executor1) total-vm:70212kB, anon-rss:60kB, file-rss:0kB, shmem-rss:0kB
> [ 1047.988138] syz-executor2 invoked oom-killer: gfp_mask=0x6040c0(GFP_KERNEL|__GFP_COMP), nodemask=(null), order=1, oom_score_adj=0
> [ 1048.000015] syz-executor2 cpuset=syz2 mems_allowed=0
> [ 1048.005199] CPU: 0 PID: 4700 Comm: syz-executor2 Not tainted 4.19.0-rc2-next-20180904+ #55
> [ 1048.740679] [   2347]     0  2347      278      186    32768        0             0 none
> [ 1051.319928] [  16497]     0 16497    17585     8739   126976        0             0 syz-executor5
> [ 1096.740878] [   8841]     0  8841    17585     8232   126976        0             0 syz-executor5
> [ 1078.140677] [  32764]     0 32764    17585     8739   126976        0             0 syz-executor5
> [ 1078.149807] [    311]     0   311    17585     8737   131072        0             0 syz-executor3
> [ 1096.740878] [   8841]     0  8841    17585     8232   126976        0             0 syz-executor5
>
> Also, another notable thing is that the backtrace for some reason includes
>
> [ 1048.211540]  ? oom_killer_disable+0x3a0/0x3a0
>
> line. Was syzbot testing process freezing functionality?

What's the API for this?

Re: INFO: task hung in ext4_da_get_block_prep

[Tetsuo Handa]

Dmitry Vyukov wrote:
> > Also, another notable thing is that the backtrace for some reason includes
> >
> > [ 1048.211540]  ? oom_killer_disable+0x3a0/0x3a0
> >
> > line. Was syzbot testing process freezing functionality?
> 
> What's the API for this?
> 

I'm not a user of suspend/hibernation. But it seems that usage of the API
is to write one of words listed in /sys/power/state into /sys/power/state .

# echo suspend > /sys/power/state

Re: INFO: task hung in ext4_da_get_block_prep

[Dmitry Vyukov]

On Thu, Sep 6, 2018 at 7:53 AM, Tetsuo Handa
<penguin-kernel@i-love.sakura.ne.jp> wrote:
> Dmitry Vyukov wrote:
>> > Also, another notable thing is that the backtrace for some reason includes
>> >
>> > [ 1048.211540]  ? oom_killer_disable+0x3a0/0x3a0
>> >
>> > line. Was syzbot testing process freezing functionality?
>>
>> What's the API for this?
>>
>
> I'm not a user of suspend/hibernation. But it seems that usage of the API
> is to write one of words listed in /sys/power/state into /sys/power/state .
>
> # echo suspend > /sys/power/state

syzkaller should not write to /sys/power/state. The only mention of
"power" is in some selinux contexts.

[PATCH] mm, oom: Introduce time limit for dump_tasks duration.

[Tetsuo Handa]

On 2018/09/06 18:54, Dmitry Vyukov wrote:
> On Thu, Sep 6, 2018 at 7:53 AM, Tetsuo Handa
> <penguin-kernel@i-love.sakura.ne.jp> wrote:
>> Dmitry Vyukov wrote:
>>>> Also, another notable thing is that the backtrace for some reason includes
>>>>
>>>> [ 1048.211540]  ? oom_killer_disable+0x3a0/0x3a0
>>>>
>>>> line. Was syzbot testing process freezing functionality?
>>>
>>> What's the API for this?
>>>
>>
>> I'm not a user of suspend/hibernation. But it seems that usage of the API
>> is to write one of words listed in /sys/power/state into /sys/power/state .
>>
>> # echo suspend > /sys/power/state
> 
> syzkaller should not write to /sys/power/state. The only mention of
> "power" is in some selinux contexts.
> 

OK. Then, I have no idea.
Anyway, I think we can apply this patch.

Re: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.

[Dmitry Vyukov]

On Thu, Sep 6, 2018 at 12:58 PM, Tetsuo Handa
<penguin-kernel@i-love.sakura.ne.jp> wrote:
> On 2018/09/06 18:54, Dmitry Vyukov wrote:
>> On Thu, Sep 6, 2018 at 7:53 AM, Tetsuo Handa
>> <penguin-kernel@i-love.sakura.ne.jp> wrote:
>>> Dmitry Vyukov wrote:
>>>>> Also, another notable thing is that the backtrace for some reason includes
>>>>>
>>>>> [ 1048.211540]  ? oom_killer_disable+0x3a0/0x3a0
>>>>>
>>>>> line. Was syzbot testing process freezing functionality?
>>>>
>>>> What's the API for this?
>>>>
>>>
>>> I'm not a user of suspend/hibernation. But it seems that usage of the API
>>> is to write one of words listed in /sys/power/state into /sys/power/state .
>>>
>>> # echo suspend > /sys/power/state
>>
>> syzkaller should not write to /sys/power/state. The only mention of
>> "power" is in some selinux contexts.
>>
>
> OK. Then, I have no idea.
> Anyway, I think we can apply this patch.
>
> From 18876f287dd69a7c33f65c91cfcda3564233f55e Mon Sep 17 00:00:00 2001
> From: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
> Date: Thu, 6 Sep 2018 19:53:18 +0900
> Subject: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.
>
> Since printk() is slow, printing one line takes nearly 0.01 second.
> As a result, syzbot is stalling for 52 seconds trying to dump 5600

I wonder why there are so many of them?
We have at most 8 test processes (each having no more than 16 threads
if that matters).
No more than 1 instance of syz-executor1 at a time. But we see output
like the one below. It has lots of instances of syz-executor1 with
different pid's. So does it print all tasks that ever existed (kernel
does not store that info, right)? Or it livelocks picking up new and
new tasks as they are created slower than they are created? Or we have
tons of zombies?

...
[   8037]     0  8037    17618     8738   131072        0
0 syz-executor1
[   8039]     0  8039    17585     8737   131072        0
0 syz-executor3
[   8040]     0  8040    17618     8738   131072        0
0 syz-executor1
 schedule+0xfb/0x450 kernel/sched/core.c:3517
[   8056]     0  8056    17585     8738   126976        0
0 syz-executor4
[   8055]     0  8055    17618     8741   126976        0
0 syz-executor5
[   8060]     0  8060    17585     8740   126976        0
0 syz-executor0
[   8062]     0  8062    17585     8739   126976        0
0 syz-executor7
[   8063]     0  8063    17618     8741   126976        0
0 syz-executor5
[   8066]     0  8066    17585     8740   126976        0
0 syz-executor0
[   8067]     0  8067    17585     8737   126976        0
0 syz-executor6
[   8070]     0  8070    17618     8739   131072        0
0 syz-executor3
[   8073]     0  8073    17618     8738   131072        0
0 syz-executor1
[   8074]     0  8074    17585     8737   126976        0
0 syz-executor6
 __rwsem_down_read_failed_common kernel/locking/rwsem-xadd.c:269 [inline]
 rwsem_down_read_failed+0x362/0x610 kernel/locking/rwsem-xadd.c:286
[   8075]     0  8075    17618     8739   131072        0
0 syz-executor3
[   8077]     0  8077    17618     8738   131072        0
0 syz-executor1
[   8079]     0  8079    17585     8739   126976        0
0 syz-executor7
[   8092]     0  8092    17618     8738   131072        0
0 syz-executor1
...


> tasks at for_each_process() under RCU. Since such situation is almost
> inflight fork bomb attack (the OOM killer will print similar tasks for
> so many times), it makes little sense to print all candidate tasks.
> Thus, this patch introduces 3 seconds limit for printing.
>
> Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
> Cc: Dmitry Vyukov <dvyukov@google.com>
> ---
>  mm/oom_kill.c | 10 ++++++++++
>  1 file changed, 10 insertions(+)
>
> diff --git a/mm/oom_kill.c b/mm/oom_kill.c
> index f10aa53..48e5bf6 100644
> --- a/mm/oom_kill.c
> +++ b/mm/oom_kill.c
> @@ -399,14 +399,22 @@ static void dump_tasks(struct mem_cgroup *memcg, const nodemask_t *nodemask)
>  {
>         struct task_struct *p;
>         struct task_struct *task;
> +       unsigned long start;
> +       unsigned int skipped = 0;
>
>         pr_info("Tasks state (memory values in pages):\n");
>         pr_info("[  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name\n");
>         rcu_read_lock();
> +       start = jiffies;
>         for_each_process(p) {
>                 if (oom_unkillable_task(p, memcg, nodemask))
>                         continue;
>
> +               if (time_after(jiffies, start + 3 * HZ)) {
> +                       skipped++;
> +                       continue;
> +               }
> +
>                 task = find_lock_task_mm(p);
>                 if (!task) {
>                         /*
> @@ -426,6 +434,8 @@ static void dump_tasks(struct mem_cgroup *memcg, const nodemask_t *nodemask)
>                 task_unlock(task);
>         }
>         rcu_read_unlock();
> +       if (skipped)
> +               pr_info("Printing %u tasks omitted.\n", skipped);
>  }
>
>  static void dump_header(struct oom_control *oc, struct task_struct *p)
> --
> 1.8.3.1
>

Re: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.

[Tetsuo Handa]

On 2018/09/06 20:07, Dmitry Vyukov wrote:
>> Since printk() is slow, printing one line takes nearly 0.01 second.
>> As a result, syzbot is stalling for 52 seconds trying to dump 5600
> 
> I wonder why there are so many of them?
> We have at most 8 test processes (each having no more than 16 threads
> if that matters).
> No more than 1 instance of syz-executor1 at a time. But we see output
> like the one below. It has lots of instances of syz-executor1 with
> different pid's. So does it print all tasks that ever existed (kernel
> does not store that info, right)? Or it livelocks picking up new and
> new tasks as they are created slower than they are created? Or we have
> tons of zombies?
> 
> ...

I don't think they are zombies. Since tasks which already released ->mm
are not printed, these tasks are still alive.

  [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
> [   8037]     0  8037    17618     8738   131072        0             0 syz-executor1

Maybe something signal / fork() / exit() / wait() related regression?

Re: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.

[Michal Hocko]

On Thu 06-09-18 19:58:25, Tetsuo Handa wrote:
[...]
> >From 18876f287dd69a7c33f65c91cfcda3564233f55e Mon Sep 17 00:00:00 2001
> From: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
> Date: Thu, 6 Sep 2018 19:53:18 +0900
> Subject: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.
> 
> Since printk() is slow, printing one line takes nearly 0.01 second.
> As a result, syzbot is stalling for 52 seconds trying to dump 5600
> tasks at for_each_process() under RCU. Since such situation is almost
> inflight fork bomb attack (the OOM killer will print similar tasks for
> so many times), it makes little sense to print all candidate tasks.
> Thus, this patch introduces 3 seconds limit for printing.
> 
> Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
> Cc: Dmitry Vyukov <dvyukov@google.com>

You really love timeout based solutions with randomly chosen timeouts,
don't you. This is just ugly as hell. We already have means to disable
tasks dumping (see /proc/sys/vm/oom_dump_tasks).
-- 
Michal Hocko
SUSE Labs

Re: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.

[Tetsuo Handa]

On 2018/09/06 20:23, Michal Hocko wrote:
> On Thu 06-09-18 19:58:25, Tetsuo Handa wrote:
> [...]
>> >From 18876f287dd69a7c33f65c91cfcda3564233f55e Mon Sep 17 00:00:00 2001
>> From: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
>> Date: Thu, 6 Sep 2018 19:53:18 +0900
>> Subject: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.
>>
>> Since printk() is slow, printing one line takes nearly 0.01 second.
>> As a result, syzbot is stalling for 52 seconds trying to dump 5600
>> tasks at for_each_process() under RCU. Since such situation is almost
>> inflight fork bomb attack (the OOM killer will print similar tasks for
>> so many times), it makes little sense to print all candidate tasks.
>> Thus, this patch introduces 3 seconds limit for printing.
>>
>> Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
>> Cc: Dmitry Vyukov <dvyukov@google.com>
> 
> You really love timeout based solutions with randomly chosen timeouts,
> don't you. This is just ugly as hell. We already have means to disable
> tasks dumping (see /proc/sys/vm/oom_dump_tasks).

I know /proc/sys/vm/oom_dump_tasks . Showing some entries while not always
printing all entries might be helpful. For example, allow
/proc/sys/vm/oom_dump_tasks > 1 and use it as timeout in seconds.

Re: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.

[Michal Hocko]

On Thu 06-09-18 20:40:34, Tetsuo Handa wrote:
> On 2018/09/06 20:23, Michal Hocko wrote:
> > On Thu 06-09-18 19:58:25, Tetsuo Handa wrote:
> > [...]
> >> >From 18876f287dd69a7c33f65c91cfcda3564233f55e Mon Sep 17 00:00:00 2001
> >> From: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
> >> Date: Thu, 6 Sep 2018 19:53:18 +0900
> >> Subject: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.
> >>
> >> Since printk() is slow, printing one line takes nearly 0.01 second.
> >> As a result, syzbot is stalling for 52 seconds trying to dump 5600
> >> tasks at for_each_process() under RCU. Since such situation is almost
> >> inflight fork bomb attack (the OOM killer will print similar tasks for
> >> so many times), it makes little sense to print all candidate tasks.
> >> Thus, this patch introduces 3 seconds limit for printing.
> >>
> >> Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
> >> Cc: Dmitry Vyukov <dvyukov@google.com>
> > 
> > You really love timeout based solutions with randomly chosen timeouts,
> > don't you. This is just ugly as hell. We already have means to disable
> > tasks dumping (see /proc/sys/vm/oom_dump_tasks).
> 
> I know /proc/sys/vm/oom_dump_tasks . Showing some entries while not always
> printing all entries might be helpful.

Not really. It could be more confusing than helpful. The main purpose of
the listing is to double check the list to understand the oom victim
selection. If you have a partial list you simply cannot do that.

If the iteration takes too long and I can imagine it does with zillions
of tasks then the proper way around it is either release the lock
periodically after N tasks is processed or outright skip the whole thing
if there are too many tasks. The first option is obviously tricky to
prevent from duplicate entries or other artifacts.
-- 
Michal Hocko
SUSE Labs

Re: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.

[Dmitry Vyukov]

On Thu, Sep 6, 2018 at 1:53 PM, Michal Hocko <mhocko@kernel.org> wrote:
> On Thu 06-09-18 20:40:34, Tetsuo Handa wrote:
>> On 2018/09/06 20:23, Michal Hocko wrote:
>> > On Thu 06-09-18 19:58:25, Tetsuo Handa wrote:
>> > [...]
>> >> >From 18876f287dd69a7c33f65c91cfcda3564233f55e Mon Sep 17 00:00:00 2001
>> >> From: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
>> >> Date: Thu, 6 Sep 2018 19:53:18 +0900
>> >> Subject: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.
>> >>
>> >> Since printk() is slow, printing one line takes nearly 0.01 second.
>> >> As a result, syzbot is stalling for 52 seconds trying to dump 5600
>> >> tasks at for_each_process() under RCU. Since such situation is almost
>> >> inflight fork bomb attack (the OOM killer will print similar tasks for
>> >> so many times), it makes little sense to print all candidate tasks.
>> >> Thus, this patch introduces 3 seconds limit for printing.
>> >>
>> >> Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
>> >> Cc: Dmitry Vyukov <dvyukov@google.com>
>> >
>> > You really love timeout based solutions with randomly chosen timeouts,
>> > don't you. This is just ugly as hell. We already have means to disable
>> > tasks dumping (see /proc/sys/vm/oom_dump_tasks).
>>
>> I know /proc/sys/vm/oom_dump_tasks . Showing some entries while not always
>> printing all entries might be helpful.
>
> Not really. It could be more confusing than helpful. The main purpose of
> the listing is to double check the list to understand the oom victim
> selection. If you have a partial list you simply cannot do that.
>
> If the iteration takes too long and I can imagine it does with zillions
> of tasks then the proper way around it is either release the lock
> periodically after N tasks is processed or outright skip the whole thing
> if there are too many tasks. The first option is obviously tricky to
> prevent from duplicate entries or other artifacts.


So does anybody know if it can live lock picking up new tasks all the
time? That's what it looks like at first glance. I also don't remember
seeing anything similar in the past.
If it's a live lock and we resolve it, then we don't need to solve the
problem of too many tasks here.

Re: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.

[Michal Hocko]

Ccing Oleg.

On Thu 06-09-18 14:08:43, Dmitry Vyukov wrote:
[...]
> So does anybody know if it can live lock picking up new tasks all the
> time? That's what it looks like at first glance. I also don't remember
> seeing anything similar in the past.

That is an interesting question. I find it unlikely here because it is
quite hard to get new tasks spawned while you are genuinely OOM. But we
do have these for_each_process loops at other places as well. Some of
them even controlled from the userspace. Some of them like exit path
(zap_threads) sound even more interesting even when that is a rare path.

So a question for Oleg I guess. Is it possible that for_each_process
live locks (or stalls for way too long/unbounded amount of time) under
heavy fork/exit loads? Is there any protection from that?

> If it's a live lock and we resolve it, then we don't need to solve the
> problem of too many tasks here.

-- 
Michal Hocko
SUSE Labs

Re: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.

[Oleg Nesterov]

On 09/06, Michal Hocko wrote:
>
> Ccing Oleg.

Thanks, but somehow I can't find this patch on marc.info ...

> So a question for Oleg I guess. Is it possible that for_each_process
> live locks (or stalls for way too long/unbounded amount of time) under
> heavy fork/exit loads?

Oh yes, it can... plus other problems.

I even sent the initial patches which introduce for_each_process_break/continue
a long ago... I'll try to find them tommorrow and resend.

Oleg.

Re: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.

[Oleg Nesterov]

On 09/11, Oleg Nesterov wrote:
>
> On 09/06, Michal Hocko wrote:
> >
> > So a question for Oleg I guess. Is it possible that for_each_process
> > live locks (or stalls for way too long/unbounded amount of time) under
> > heavy fork/exit loads?
>
> Oh yes, it can... plus other problems.
>
> I even sent the initial patches which introduce for_each_process_break/continue
> a long ago... I'll try to find them tommorrow and resend.

Two years ago ;) I don't understand why there were ignored, please see
"[PATCH 0/2] introduce for_each_process_thread_break() and for_each_process_thread_continue()"
I sent a minute ago.

However, I didn't notice that the subject mentions oom/dump_tasks... As for
dump_tasks() it probably doesn't need the new helpers, I'll write another email
tomorrow, but perhaps the time limit is all we need.

Oleg.

Re: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.

[Tetsuo Handa]

On 2018/09/06 20:53, Michal Hocko wrote:
> On Thu 06-09-18 20:40:34, Tetsuo Handa wrote:
>> On 2018/09/06 20:23, Michal Hocko wrote:
>>> On Thu 06-09-18 19:58:25, Tetsuo Handa wrote:
>>> [...]
>>>> >From 18876f287dd69a7c33f65c91cfcda3564233f55e Mon Sep 17 00:00:00 2001
>>>> From: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
>>>> Date: Thu, 6 Sep 2018 19:53:18 +0900
>>>> Subject: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.
>>>>
>>>> Since printk() is slow, printing one line takes nearly 0.01 second.
>>>> As a result, syzbot is stalling for 52 seconds trying to dump 5600
>>>> tasks at for_each_process() under RCU. Since such situation is almost
>>>> inflight fork bomb attack (the OOM killer will print similar tasks for
>>>> so many times), it makes little sense to print all candidate tasks.
>>>> Thus, this patch introduces 3 seconds limit for printing.
>>>>
>>>> Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
>>>> Cc: Dmitry Vyukov <dvyukov@google.com>
>>>
>>> You really love timeout based solutions with randomly chosen timeouts,
>>> don't you. This is just ugly as hell. We already have means to disable
>>> tasks dumping (see /proc/sys/vm/oom_dump_tasks).
>>
>> I know /proc/sys/vm/oom_dump_tasks . Showing some entries while not always
>> printing all entries might be helpful.
> 
> Not really. It could be more confusing than helpful. The main purpose of
> the listing is to double check the list to understand the oom victim
> selection. If you have a partial list you simply cannot do that.

It serves as a safeguard for avoiding RCU stall warnings.

> 
> If the iteration takes too long and I can imagine it does with zillions
> of tasks then the proper way around it is either release the lock
> periodically after N tasks is processed or outright skip the whole thing
> if there are too many tasks. The first option is obviously tricky to
> prevent from duplicate entries or other artifacts.
> 

Can we add rcu_lock_break() like check_hung_uninterruptible_tasks() does?

Re: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.

[Michal Hocko]

On Thu 06-09-18 22:45:26, Tetsuo Handa wrote:
> On 2018/09/06 20:53, Michal Hocko wrote:
> > On Thu 06-09-18 20:40:34, Tetsuo Handa wrote:
> >> On 2018/09/06 20:23, Michal Hocko wrote:
> >>> On Thu 06-09-18 19:58:25, Tetsuo Handa wrote:
> >>> [...]
> >>>> >From 18876f287dd69a7c33f65c91cfcda3564233f55e Mon Sep 17 00:00:00 2001
> >>>> From: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
> >>>> Date: Thu, 6 Sep 2018 19:53:18 +0900
> >>>> Subject: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.
> >>>>
> >>>> Since printk() is slow, printing one line takes nearly 0.01 second.
> >>>> As a result, syzbot is stalling for 52 seconds trying to dump 5600
> >>>> tasks at for_each_process() under RCU. Since such situation is almost
> >>>> inflight fork bomb attack (the OOM killer will print similar tasks for
> >>>> so many times), it makes little sense to print all candidate tasks.
> >>>> Thus, this patch introduces 3 seconds limit for printing.
> >>>>
> >>>> Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
> >>>> Cc: Dmitry Vyukov <dvyukov@google.com>
> >>>
> >>> You really love timeout based solutions with randomly chosen timeouts,
> >>> don't you. This is just ugly as hell. We already have means to disable
> >>> tasks dumping (see /proc/sys/vm/oom_dump_tasks).
> >>
> >> I know /proc/sys/vm/oom_dump_tasks . Showing some entries while not always
> >> printing all entries might be helpful.
> > 
> > Not really. It could be more confusing than helpful. The main purpose of
> > the listing is to double check the list to understand the oom victim
> > selection. If you have a partial list you simply cannot do that.
> 
> It serves as a safeguard for avoiding RCU stall warnings.
> 
> > 
> > If the iteration takes too long and I can imagine it does with zillions
> > of tasks then the proper way around it is either release the lock
> > periodically after N tasks is processed or outright skip the whole thing
> > if there are too many tasks. The first option is obviously tricky to
> > prevent from duplicate entries or other artifacts.
> > 
> 
> Can we add rcu_lock_break() like check_hung_uninterruptible_tasks() does?

This would be a better variant of your timeout based approach. But it
can still produce an incomplete task list so it still consumes a lot of
resources to print a long list of tasks potentially while that list is not
useful for any evaluation. Maybe that is good enough. I don't know. I
would generally recommend to disable the whole thing with workloads with
many tasks though.

-- 
Michal Hocko
SUSE Labs

Re: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.

[Tetsuo Handa]

On 2018/09/06 23:39, Michal Hocko wrote:
>>>> I know /proc/sys/vm/oom_dump_tasks . Showing some entries while not always
>>>> printing all entries might be helpful.
>>>
>>> Not really. It could be more confusing than helpful. The main purpose of
>>> the listing is to double check the list to understand the oom victim
>>> selection. If you have a partial list you simply cannot do that.
>>
>> It serves as a safeguard for avoiding RCU stall warnings.
>>
>>>
>>> If the iteration takes too long and I can imagine it does with zillions
>>> of tasks then the proper way around it is either release the lock
>>> periodically after N tasks is processed or outright skip the whole thing
>>> if there are too many tasks. The first option is obviously tricky to
>>> prevent from duplicate entries or other artifacts.
>>>
>>
>> Can we add rcu_lock_break() like check_hung_uninterruptible_tasks() does?
> 
> This would be a better variant of your timeout based approach. But it
> can still produce an incomplete task list so it still consumes a lot of
> resources to print a long list of tasks potentially while that list is not
> useful for any evaluation. Maybe that is good enough. I don't know. I
> would generally recommend to disable the whole thing with workloads with
> many tasks though.
> 

The "safeguard" is useful when there are _unexpectedly_ many tasks (like
syzbot in this case). Why not to allow those who want to avoid lockup to
avoid lockup rather than forcing them to disable the whole thing?

Re: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.

[Michal Hocko]

On Fri 07-09-18 05:58:06, Tetsuo Handa wrote:
> On 2018/09/06 23:39, Michal Hocko wrote:
> >>>> I know /proc/sys/vm/oom_dump_tasks . Showing some entries while not always
> >>>> printing all entries might be helpful.
> >>>
> >>> Not really. It could be more confusing than helpful. The main purpose of
> >>> the listing is to double check the list to understand the oom victim
> >>> selection. If you have a partial list you simply cannot do that.
> >>
> >> It serves as a safeguard for avoiding RCU stall warnings.
> >>
> >>>
> >>> If the iteration takes too long and I can imagine it does with zillions
> >>> of tasks then the proper way around it is either release the lock
> >>> periodically after N tasks is processed or outright skip the whole thing
> >>> if there are too many tasks. The first option is obviously tricky to
> >>> prevent from duplicate entries or other artifacts.
> >>>
> >>
> >> Can we add rcu_lock_break() like check_hung_uninterruptible_tasks() does?
> > 
> > This would be a better variant of your timeout based approach. But it
> > can still produce an incomplete task list so it still consumes a lot of
> > resources to print a long list of tasks potentially while that list is not
> > useful for any evaluation. Maybe that is good enough. I don't know. I
> > would generally recommend to disable the whole thing with workloads with
> > many tasks though.
> > 
> 
> The "safeguard" is useful when there are _unexpectedly_ many tasks (like
> syzbot in this case). Why not to allow those who want to avoid lockup to
> avoid lockup rather than forcing them to disable the whole thing?

So you get an rcu lockup splat and what? Unless you have panic_on_rcu_stall
then this should be recoverable thing (assuming we cannot really
livelock as described by Dmitry).

-- 
Michal Hocko
SUSE Labs

Re: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.

[Dmitry Vyukov]

On Fri, Sep 7, 2018 at 10:27 AM, Michal Hocko <mhocko@kernel.org> wrote:
> On Fri 07-09-18 05:58:06, Tetsuo Handa wrote:
>> On 2018/09/06 23:39, Michal Hocko wrote:
>> >>>> I know /proc/sys/vm/oom_dump_tasks . Showing some entries while not always
>> >>>> printing all entries might be helpful.
>> >>>
>> >>> Not really. It could be more confusing than helpful. The main purpose of
>> >>> the listing is to double check the list to understand the oom victim
>> >>> selection. If you have a partial list you simply cannot do that.
>> >>
>> >> It serves as a safeguard for avoiding RCU stall warnings.
>> >>
>> >>>
>> >>> If the iteration takes too long and I can imagine it does with zillions
>> >>> of tasks then the proper way around it is either release the lock
>> >>> periodically after N tasks is processed or outright skip the whole thing
>> >>> if there are too many tasks. The first option is obviously tricky to
>> >>> prevent from duplicate entries or other artifacts.
>> >>>
>> >>
>> >> Can we add rcu_lock_break() like check_hung_uninterruptible_tasks() does?
>> >
>> > This would be a better variant of your timeout based approach. But it
>> > can still produce an incomplete task list so it still consumes a lot of
>> > resources to print a long list of tasks potentially while that list is not
>> > useful for any evaluation. Maybe that is good enough. I don't know. I
>> > would generally recommend to disable the whole thing with workloads with
>> > many tasks though.
>> >
>>
>> The "safeguard" is useful when there are _unexpectedly_ many tasks (like
>> syzbot in this case). Why not to allow those who want to avoid lockup to
>> avoid lockup rather than forcing them to disable the whole thing?
>
> So you get an rcu lockup splat and what? Unless you have panic_on_rcu_stall
> then this should be recoverable thing (assuming we cannot really
> livelock as described by Dmitry).


Should I add "vm.oom_dump_tasks = 0" to /etc/sysctl.conf on syzbot?
It looks like it will make things faster, not pollute console output,
prevent these stalls and that output does not seem to be too useful
for debugging.

But I am still concerned as to what has changed recently. Potentially
this happens only on linux-next, at least that's where I saw all
existing reports.
New tasks seem to be added to the tail of the tasks list, but this
part does not seem to be changed recently in linux-next..

Re: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.

[Tetsuo Handa]

On 2018/09/07 18:36, Dmitry Vyukov wrote:
> But I am still concerned as to what has changed recently. Potentially
> this happens only on linux-next, at least that's where I saw all
> existing reports.
> New tasks seem to be added to the tail of the tasks list, but this
> part does not seem to be changed recently in linux-next..
> 

As far as dump_tasks() is saying, these tasks are alive. Thus, I want to know
what these tasks are doing (i.e. SysRq-t output). Since this is occurring in
linux-next, we can try CONFIG_DEBUG_AID_FOR_SYZBOT=y case like
https://lkml.org/lkml/2018/9/3/353 does. 

Re: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.

[Michal Hocko]

On Fri 07-09-18 11:36:55, Dmitry Vyukov wrote:
> On Fri, Sep 7, 2018 at 10:27 AM, Michal Hocko <mhocko@kernel.org> wrote:
> > On Fri 07-09-18 05:58:06, Tetsuo Handa wrote:
> >> On 2018/09/06 23:39, Michal Hocko wrote:
> >> >>>> I know /proc/sys/vm/oom_dump_tasks . Showing some entries while not always
> >> >>>> printing all entries might be helpful.
> >> >>>
> >> >>> Not really. It could be more confusing than helpful. The main purpose of
> >> >>> the listing is to double check the list to understand the oom victim
> >> >>> selection. If you have a partial list you simply cannot do that.
> >> >>
> >> >> It serves as a safeguard for avoiding RCU stall warnings.
> >> >>
> >> >>>
> >> >>> If the iteration takes too long and I can imagine it does with zillions
> >> >>> of tasks then the proper way around it is either release the lock
> >> >>> periodically after N tasks is processed or outright skip the whole thing
> >> >>> if there are too many tasks. The first option is obviously tricky to
> >> >>> prevent from duplicate entries or other artifacts.
> >> >>>
> >> >>
> >> >> Can we add rcu_lock_break() like check_hung_uninterruptible_tasks() does?
> >> >
> >> > This would be a better variant of your timeout based approach. But it
> >> > can still produce an incomplete task list so it still consumes a lot of
> >> > resources to print a long list of tasks potentially while that list is not
> >> > useful for any evaluation. Maybe that is good enough. I don't know. I
> >> > would generally recommend to disable the whole thing with workloads with
> >> > many tasks though.
> >> >
> >>
> >> The "safeguard" is useful when there are _unexpectedly_ many tasks (like
> >> syzbot in this case). Why not to allow those who want to avoid lockup to
> >> avoid lockup rather than forcing them to disable the whole thing?
> >
> > So you get an rcu lockup splat and what? Unless you have panic_on_rcu_stall
> > then this should be recoverable thing (assuming we cannot really
> > livelock as described by Dmitry).
> 
> 
> Should I add "vm.oom_dump_tasks = 0" to /etc/sysctl.conf on syzbot?
> It looks like it will make things faster, not pollute console output,
> prevent these stalls and that output does not seem to be too useful
> for debugging.

I think that oom_dump_tasks has only very limited usefulness for your
testing.

> But I am still concerned as to what has changed recently. Potentially
> this happens only on linux-next, at least that's where I saw all
> existing reports.
> New tasks seem to be added to the tail of the tasks list, but this
> part does not seem to be changed recently in linux-next..

Yes, that would be interesting to find out.

-- 
Michal Hocko
SUSE Labs

Re: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.

[Dmitry Vyukov]

On Fri, Sep 7, 2018 at 1:08 PM, Michal Hocko <mhocko@kernel.org> wrote:
>> >> >>>> I know /proc/sys/vm/oom_dump_tasks . Showing some entries while not always
>> >> >>>> printing all entries might be helpful.
>> >> >>>
>> >> >>> Not really. It could be more confusing than helpful. The main purpose of
>> >> >>> the listing is to double check the list to understand the oom victim
>> >> >>> selection. If you have a partial list you simply cannot do that.
>> >> >>
>> >> >> It serves as a safeguard for avoiding RCU stall warnings.
>> >> >>
>> >> >>>
>> >> >>> If the iteration takes too long and I can imagine it does with zillions
>> >> >>> of tasks then the proper way around it is either release the lock
>> >> >>> periodically after N tasks is processed or outright skip the whole thing
>> >> >>> if there are too many tasks. The first option is obviously tricky to
>> >> >>> prevent from duplicate entries or other artifacts.
>> >> >>>
>> >> >>
>> >> >> Can we add rcu_lock_break() like check_hung_uninterruptible_tasks() does?
>> >> >
>> >> > This would be a better variant of your timeout based approach. But it
>> >> > can still produce an incomplete task list so it still consumes a lot of
>> >> > resources to print a long list of tasks potentially while that list is not
>> >> > useful for any evaluation. Maybe that is good enough. I don't know. I
>> >> > would generally recommend to disable the whole thing with workloads with
>> >> > many tasks though.
>> >> >
>> >>
>> >> The "safeguard" is useful when there are _unexpectedly_ many tasks (like
>> >> syzbot in this case). Why not to allow those who want to avoid lockup to
>> >> avoid lockup rather than forcing them to disable the whole thing?
>> >
>> > So you get an rcu lockup splat and what? Unless you have panic_on_rcu_stall
>> > then this should be recoverable thing (assuming we cannot really
>> > livelock as described by Dmitry).
>>
>>
>> Should I add "vm.oom_dump_tasks = 0" to /etc/sysctl.conf on syzbot?
>> It looks like it will make things faster, not pollute console output,
>> prevent these stalls and that output does not seem to be too useful
>> for debugging.
>
> I think that oom_dump_tasks has only very limited usefulness for your
> testing.
>
>> But I am still concerned as to what has changed recently. Potentially
>> this happens only on linux-next, at least that's where I saw all
>> existing reports.
>> New tasks seem to be added to the tail of the tasks list, but this
>> part does not seem to be changed recently in linux-next..
>
> Yes, that would be interesting to find out.


Looking at another similar report:
https://syzkaller.appspot.com/bug?extid=0d867757fdc016c0157e
It looks like it can be just syzkaller learning how to do fork bombs
after all (same binary multiplied infinite amount of times). Probably
required some creativity because test programs do not contain loops
per se and clone syscall does not accept start function pc.
I will set vm.oom_dump_tasks = 0 and try to additionally restrict it
with cgroups.

Re: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.

[Dmitry Vyukov]

On Sat, Sep 8, 2018 at 4:00 PM, Dmitry Vyukov <dvyukov@google.com> wrote:
> On Fri, Sep 7, 2018 at 1:08 PM, Michal Hocko <mhocko@kernel.org> wrote:
>>> >> >>>> I know /proc/sys/vm/oom_dump_tasks . Showing some entries while not always
>>> >> >>>> printing all entries might be helpful.
>>> >> >>>
>>> >> >>> Not really. It could be more confusing than helpful. The main purpose of
>>> >> >>> the listing is to double check the list to understand the oom victim
>>> >> >>> selection. If you have a partial list you simply cannot do that.
>>> >> >>
>>> >> >> It serves as a safeguard for avoiding RCU stall warnings.
>>> >> >>
>>> >> >>>
>>> >> >>> If the iteration takes too long and I can imagine it does with zillions
>>> >> >>> of tasks then the proper way around it is either release the lock
>>> >> >>> periodically after N tasks is processed or outright skip the whole thing
>>> >> >>> if there are too many tasks. The first option is obviously tricky to
>>> >> >>> prevent from duplicate entries or other artifacts.
>>> >> >>>
>>> >> >>
>>> >> >> Can we add rcu_lock_break() like check_hung_uninterruptible_tasks() does?
>>> >> >
>>> >> > This would be a better variant of your timeout based approach. But it
>>> >> > can still produce an incomplete task list so it still consumes a lot of
>>> >> > resources to print a long list of tasks potentially while that list is not
>>> >> > useful for any evaluation. Maybe that is good enough. I don't know. I
>>> >> > would generally recommend to disable the whole thing with workloads with
>>> >> > many tasks though.
>>> >> >
>>> >>
>>> >> The "safeguard" is useful when there are _unexpectedly_ many tasks (like
>>> >> syzbot in this case). Why not to allow those who want to avoid lockup to
>>> >> avoid lockup rather than forcing them to disable the whole thing?
>>> >
>>> > So you get an rcu lockup splat and what? Unless you have panic_on_rcu_stall
>>> > then this should be recoverable thing (assuming we cannot really
>>> > livelock as described by Dmitry).
>>>
>>>
>>> Should I add "vm.oom_dump_tasks = 0" to /etc/sysctl.conf on syzbot?
>>> It looks like it will make things faster, not pollute console output,
>>> prevent these stalls and that output does not seem to be too useful
>>> for debugging.
>>
>> I think that oom_dump_tasks has only very limited usefulness for your
>> testing.
>>
>>> But I am still concerned as to what has changed recently. Potentially
>>> this happens only on linux-next, at least that's where I saw all
>>> existing reports.
>>> New tasks seem to be added to the tail of the tasks list, but this
>>> part does not seem to be changed recently in linux-next..
>>
>> Yes, that would be interesting to find out.
>
>
> Looking at another similar report:
> https://syzkaller.appspot.com/bug?extid=0d867757fdc016c0157e
> It looks like it can be just syzkaller learning how to do fork bombs
> after all (same binary multiplied infinite amount of times). Probably
> required some creativity because test programs do not contain loops
> per se and clone syscall does not accept start function pc.
> I will set vm.oom_dump_tasks = 0 and try to additionally restrict it
> with cgroups.


FTR, syzkaller now restricts test processes with pids.max=32. This
should prevent any fork bombs.
https://github.com/google/syzkaller/commit/f167cb6b0957d34f95b1067525aa87083f264035

Re: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.

[Tetsuo Handa]

On 2018/09/07 17:27, Michal Hocko wrote:
> On Fri 07-09-18 05:58:06, Tetsuo Handa wrote:
>> On 2018/09/06 23:39, Michal Hocko wrote:
>>>>>> I know /proc/sys/vm/oom_dump_tasks . Showing some entries while not always
>>>>>> printing all entries might be helpful.
>>>>>
>>>>> Not really. It could be more confusing than helpful. The main purpose of
>>>>> the listing is to double check the list to understand the oom victim
>>>>> selection. If you have a partial list you simply cannot do that.
>>>>
>>>> It serves as a safeguard for avoiding RCU stall warnings.
>>>>
>>>>>
>>>>> If the iteration takes too long and I can imagine it does with zillions
>>>>> of tasks then the proper way around it is either release the lock
>>>>> periodically after N tasks is processed or outright skip the whole thing
>>>>> if there are too many tasks. The first option is obviously tricky to
>>>>> prevent from duplicate entries or other artifacts.
>>>>>
>>>>
>>>> Can we add rcu_lock_break() like check_hung_uninterruptible_tasks() does?
>>>
>>> This would be a better variant of your timeout based approach. But it
>>> can still produce an incomplete task list so it still consumes a lot of
>>> resources to print a long list of tasks potentially while that list is not
>>> useful for any evaluation. Maybe that is good enough. I don't know. I
>>> would generally recommend to disable the whole thing with workloads with
>>> many tasks though.
>>>
>>
>> The "safeguard" is useful when there are _unexpectedly_ many tasks (like
>> syzbot in this case). Why not to allow those who want to avoid lockup to
>> avoid lockup rather than forcing them to disable the whole thing?
> 
> So you get an rcu lockup splat and what? Unless you have panic_on_rcu_stall
> then this should be recoverable thing (assuming we cannot really
> livelock as described by Dmitry).
> 

syzbot is getting hung task panic (140 seconds) because one dump_tasks() from
out_of_memory() consumes 52 seconds on a 2 CPU machine because we have only 
cond_resched() which can yield CPU resource to tasks which need CPU resource.
This is similar to a bug shown below.

  [upstream] INFO: task hung in fsnotify_mark_destroy_workfn
  https://syzkaller.appspot.com/bug?id=0e75779a6f0faac461510c6330514e8f0e893038

  [upstream] INFO: task hung in fsnotify_connector_destroy_workfn
  https://syzkaller.appspot.com/bug?id=aa11d2d767f3750ef9a40d156a149e9cfa735b73

Continuing printk() until khungtaskd fires is a stupid behavior.

Re: INFO: task hung in ext4_da_get_block_prep

[syzbot]

Auto-closing this bug as obsolete.
Crashes did not happen for a while, no reproducer and no activity.