fs/proc: Crash observed in next_tgid (fs/proc/base.c)

fs/proc: Crash observed in next_tgid (fs/proc/base.c)

[Jitendra Sharma]

Hi Kees Cook/Luis,

We are observing one kernel crash in next_tgid function through 
getdents64 path. Call stack is as shown below:

-000|has_group_leader_pid(inline)
-000|next_tgid(
| [X20] ns = 0xFFFFFF87CABB1AC0,
| [locdesc] iter = (
| [locdesc] tgid = 424,
| [locdesc] task = ?))
| [X21] p = 0xFFFFFFD0FFFFF948
| [X21] task = 0xFFFFFFD0FFFFF948
-001|proc_pid_readdir(
| [X20] file = 0xFFFFFFD1AC60FC40,
| [X19] ctx = 0xFFFFFF8027363E40)
| [X21] ns = 0xFFFFFF87CABB1AC0
-002|proc_root_readdir(
| [X20] file = 0xFFFFFFD1AC60FC40,
| [X19] ctx = 0xFFFFFF8027363E40)
-003|iterate_dir(
| [X19] file = 0xFFFFFFD1AC60FC40,
| [X22] ctx = 0xFFFFFF8027363E40)
| [X23] inode = 0xFFFFFFD1F20246D0
-004|SYSC_getdents64(inline)
-004|sys_getdents64(
| ?,
| ?,
| [X19] count = 4200)
| [X19] count = 4200
| [X20] f = ([X20] file = 0xAC60FC43AC60FC40, [X20] flags = 1207898624)
| [X0] error = -1720
-005|el0_svc_naked(asm)
-->|exception
-006|NUX:0x78C5AD7D38(asm)
---|end of frame


 From this call stack,task: 0xFFFFFFD0FFFFF948, seems to be invalid. 
As(from ramdumps) it doesn't have any valid fields. And while trying to 
access the fields of this task struct in has_group_leader_pid, abort is 
happening.

 From the dumps, its not clear why the task struct is coming to be some 
invalid (Possibly task has already exited).  This issue is observed 
during normal monkey testing for long hours.

Could you please provide some pointers which could help in debugging 
this issue further.


Thanks,

Jitendra

-- 

QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation

Re: fs/proc: Crash observed in next_tgid (fs/proc/base.c)

[Kees Cook]

On Mon, Apr 15, 2019 at 7:58 AM Jitendra Sharma <shajit@codeaurora.org> wrote:
>
> Hi Kees Cook/Luis,
>
> We are observing one kernel crash in next_tgid function through
> getdents64 path. Call stack is as shown below:
>
> -000|has_group_leader_pid(inline)
> -000|next_tgid(
> | [X20] ns = 0xFFFFFF87CABB1AC0,
> | [locdesc] iter = (
> | [locdesc] tgid = 424,
> | [locdesc] task = ?))
> | [X21] p = 0xFFFFFFD0FFFFF948
> | [X21] task = 0xFFFFFFD0FFFFF948
> -001|proc_pid_readdir(
> | [X20] file = 0xFFFFFFD1AC60FC40,
> | [X19] ctx = 0xFFFFFF8027363E40)
> | [X21] ns = 0xFFFFFF87CABB1AC0
> -002|proc_root_readdir(
> | [X20] file = 0xFFFFFFD1AC60FC40,
> | [X19] ctx = 0xFFFFFF8027363E40)
> -003|iterate_dir(
> | [X19] file = 0xFFFFFFD1AC60FC40,
> | [X22] ctx = 0xFFFFFF8027363E40)
> | [X23] inode = 0xFFFFFFD1F20246D0
> -004|SYSC_getdents64(inline)
> -004|sys_getdents64(
> | ?,
> | ?,
> | [X19] count = 4200)
> | [X19] count = 4200
> | [X20] f = ([X20] file = 0xAC60FC43AC60FC40, [X20] flags = 1207898624)
> | [X0] error = -1720
> -005|el0_svc_naked(asm)
> -->|exception
> -006|NUX:0x78C5AD7D38(asm)
> ---|end of frame
>
>
>  From this call stack,task: 0xFFFFFFD0FFFFF948, seems to be invalid.
> As(from ramdumps) it doesn't have any valid fields. And while trying to
> access the fields of this task struct in has_group_leader_pid, abort is
> happening.
>
>  From the dumps, its not clear why the task struct is coming to be some
> invalid (Possibly task has already exited).  This issue is observed
> during normal monkey testing for long hours.
>
> Could you please provide some pointers which could help in debugging
> this issue further.

Do you have any hints on how to reproduce this? I assume something is
missing proper locking or RCU handling, but I don't see anything
obvious in the surrounding code yet...

-- 
Kees Cook

Re: fs/proc: Crash observed in next_tgid (fs/proc/base.c)

[Oleg Nesterov]

On 04/16, Kees Cook wrote:
>
> Do you have any hints on how to reproduce this? I assume something is
> missing proper locking or RCU handling,

or we simply have an unbalanced put_task_struct() anywhere else ...

> but I don't see anything
> obvious in the surrounding code yet...

I too do not see anything wrong in proc_pid_readdir() paths

Oleg.

Re: fs/proc: Crash observed in next_tgid (fs/proc/base.c)

[Marc Gonzalez]

On 15/04/2019 14:58, Jitendra Sharma wrote:

> We are observing one kernel crash in next_tgid function through 
> getdents64 path. Call stack is as shown below:

It might help if you specify the exact kernel version you are discussing,
as in which tag or commit hash you are running.

Also, what are you doing to trigger the issue?

Regards.

Re: fs/proc: Crash observed in next_tgid (fs/proc/base.c)

[Jitendra Sharma]

Thanks Marc and Kees for replying.
Answer to your queries:
Kernel version: 4.14.83
Test case: Not some specific test case. Issue reproduced while doing 
monkey testing for very long hours.

Thanks,
Jitendra

On 4/17/2019 5:41 PM, Marc Gonzalez wrote:
> On 15/04/2019 14:58, Jitendra Sharma wrote:
> 
>> We are observing one kernel crash in next_tgid function through
>> getdents64 path. Call stack is as shown below:
> 
> It might help if you specify the exact kernel version you are discussing,
> as in which tag or commit hash you are running.
> 
> Also, what are you doing to trigger the issue?
> 
> Regards.
> 

-- 

QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation

Re: fs/proc: Crash observed in next_tgid (fs/proc/base.c)

[Marc Gonzalez]

NB: it is preferable to avoid top-posting here.

On 17/04/2019 18:58, Jitendra Sharma wrote:

> Kernel version: 4.14.83

NB2: 4.14.83 is obsolete, as it stands 2255 patches behind the tip of
linux-4.14.y (though only 4 patches in fs/proc).

NB3: 4.14 is 108499(!!) patches behind v5.1-rc5 (latest release). It makes
sense to test on a recent kernel, in case someone has solved the issue along
the way (then the bisecting fun can begin).

> Test case: Not some specific test case. Issue reproduced while doing 
> monkey testing for very long hours.

I'm not sure what "monkey testing" entails...

Any relation to the infinite monkey theorem?
https://en.wikipedia.org/wiki/Infinite_monkey_theorem

Regards.

Re: fs/proc: Crash observed in next_tgid (fs/proc/base.c)

[Alexey Dobriyan]

> Test case: Not some specific test case.

Write 2 programs: one forks, clones, and execs randomly (but not going
out of control), another does open("/proc"), getdents in a loop.
Don't use shell scripts or readdir as they only bring overhead.

Run getdents in a loop and several forkers, see what happens.