thousands of kworker processes with 4.7.x and 4.8-rc*

thousands of kworker processes with 4.7.x and 4.8-rc*

[Tomasz Chmielewski]

On several servers running 4.7.x and 4.8-rc6/7 kernels I'm seeing 
thousands of kworker processes.

# ps auxf|grep -c kworker
2104


Load average goes into hundreds on a pretty much idle server (biggest 
CPU and RAM consumers are probably SSHD with one user logged in and 
rsyslog writing ~1 line per minute):

# uptime
  06:58:56 up 26 min,  1 user,  load average: 146.11, 215.46, 105.70


# uptime
  06:59:48 up 26 min,  1 user,  load average: 305.20, 240.84, 120.25


Sometimes seeing lots of them in "D" state:

root     19474  0.0  0.0      0     0 ?        D    06:54   0:00  \_ 
[kworker/0:208]
root     19475  0.0  0.0      0     0 ?        D    06:54   0:00  \_ 
[kworker/0:209]
root     19477  0.0  0.0      0     0 ?        D    06:54   0:00  \_ 
[kworker/0:211]
root     19480  0.0  0.0      0     0 ?        D    06:54   0:00  \_ 
[kworker/0:214]
root     19483  0.0  0.0      0     0 ?        D    06:54   0:00  \_ 
[kworker/0:217]
root     19485  0.0  0.0      0     0 ?        D    06:54   0:00  \_ 
[kworker/0:219]
root     19486  0.0  0.0      0     0 ?        D    06:54   0:00  \_ 
[kworker/0:220]
root     19487  0.0  0.0      0     0 ?        D    06:54   0:00  \_ 
[kworker/0:221]
root     19492  0.0  0.0      0     0 ?        D    06:54   0:00  \_ 
[kworker/0:226]
root     19533  0.0  0.0      0     0 ?        D    06:54   0:00  \_ 
[kworker/4:257]


Is it a known issue?

The server has 8 CPUs and 32 GB RAM.


Tomasz Chmielewski
https://lxadm.com

Re: thousands of kworker processes with 4.7.x and 4.8-rc*

[Tomasz Chmielewski]

On 2016-09-19 16:08, Tomasz Chmielewski wrote:
> On several servers running 4.7.x and 4.8-rc6/7 kernels I'm seeing
> thousands of kworker processes.
> # ps auxf|grep -c kworker
> 2104
> Load average goes into hundreds on a pretty much idle server (biggest
> CPU and RAM consumers are probably SSHD with one user logged in and
> rsyslog writing ~1 line per minute):
> # uptime
>  06:58:56 up 26 min,  1 user,  load average: 146.11, 215.46, 105.70
> # uptime
>  06:59:48 up 26 min,  1 user,  load average: 305.20, 240.84, 120.25
> Sometimes seeing lots of them in "D" state:
> root     19474  0.0  0.0      0     0 ?        D    06:54   0:00  \_
> [kworker/0:208]
> root     19475  0.0  0.0      0     0 ?        D    06:54   0:00  \_
> [kworker/0:209]


I did some experiments to see when the problem first appeared. Thousands 
of kworker processes start to show up in 4.7.0-rc5.

kernel version | kworker count after boot
-------------------------------------------
4.6.3 	        37
4.6.4 	        47
4.6.5 	        46
4.6.6 	        49
4.6.7 	        49
4.7.0-rc1 	46
4.7.0-rc2 	49
4.7.0-rc3	45
4.7.0-rc4	47
4.7.0-rc5	1592
4.7.0-rc6	1714
4.7.0-rc7 	1955
4.7.0 	        2088
4.7.1 	        1222
4.7.2 	        1699
4.7.3           1446
4.7.4 	        1781
4.8-rc1 	(not tested)
4.8-rc2 	2012
4.8-rc3 	1696
4.8-rc4 	1210
4.8-rc5 	1890
4.8-rc6 	1657
4.8-rc7 	1647


Tomasz Chmielewski
https://lxadm.com

Re: thousands of kworker processes with 4.7.x and 4.8-rc*

[Mike Galbraith]

On Fri, 2016-09-23 at 22:23 +0900, Tomasz Chmielewski wrote:
> On 2016-09-19 16:08, Tomasz Chmielewski wrote:
> > On several servers running 4.7.x and 4.8-rc6/7 kernels I'm seeing
> > thousands of kworker processes.
> > # ps auxf|grep -c kworker
> > 2104
> > Load average goes into hundreds on a pretty much idle server (biggest
> > CPU and RAM consumers are probably SSHD with one user logged in and
> > rsyslog writing ~1 line per minute):
> > # uptime
> >  06:58:56 up 26 min,  1 user,  load average: 146.11, 215.46, 105.70
> > # uptime
> >  06:59:48 up 26 min,  1 user,  load average: 305.20, 240.84, 120.25
> > Sometimes seeing lots of them in "D" state:
> > root     19474  0.0  0.0      0     0 ?        D    06:54   0:00  \_
> > [kworker/0:208]
> > root     19475  0.0  0.0      0     0 ?        D    06:54   0:00  \_
> > [kworker/0:209]
> 
> 
> I did some experiments to see when the problem first appeared. Thousands 
> of kworker processes start to show up in 4.7.0-rc5.
> 
> kernel version | kworker count after boot
> -------------------------------------------
> 4.6.3 > 	>         37
> 4.6.4 > 	>         47
> 4.6.5 > 	>         46
> 4.6.6 > 	>         49
> 4.6.7 > 	>         49
> 4.7.0-rc1 > 	> 46
> 4.7.0-rc2 > 	> 49
> 4.7.0-rc3> 	> 45
> 4.7.0-rc4> 	> 47
> 4.7.0-rc5> 	> 1592

Best bet would be to use 'git bisect' to locate the exact commit that
caused this, and post the bisection result along with your config.

AFAIK, nobody else is seeing this, is the kernel virgin source?

	-Mike

Re: thousands of kworker processes with 4.7.x and 4.8-rc*

[Tomasz Chmielewski]

On 2016-09-23 23:10, Mike Galbraith wrote:

>> I did some experiments to see when the problem first appeared. 
>> Thousands
>> of kworker processes start to show up in 4.7.0-rc5.
>> 
>> kernel version | kworker count after boot
>> -------------------------------------------
>> 4.6.3 > 	>         37
>> 4.6.4 > 	>         47
>> 4.6.5 > 	>         46
>> 4.6.6 > 	>         49
>> 4.6.7 > 	>         49
>> 4.7.0-rc1 > 	> 46
>> 4.7.0-rc2 > 	> 49
>> 4.7.0-rc3> 	> 45
>> 4.7.0-rc4> 	> 47
>> 4.7.0-rc5> 	> 1592
> 
> Best bet would be to use 'git bisect' to locate the exact commit that
> caused this, and post the bisection result along with your config.
> 
> AFAIK, nobody else is seeing this, is the kernel virgin source?

Yes, it's a kernel.org kernel.

I found some similar reports, though without much more info:

https://github.com/zfsonlinux/zfs/issues/5036 - kernel 4.7.2, initially 
attributed to ZFS on Linux, but then reproduced without ZFS

https://github.com/systemd/systemd/issues/4069 - kernel 4.7.2


I'll try to bisect.



Tomasz Chmielewski
https://lxadm.com

Re: thousands of kworker processes with 4.7.x and 4.8-rc*

[Tomasz Chmielewski]

On 2016-09-25 18:29, Tomasz Chmielewski wrote:

>> I'll try to bisect.
> 
> OK, not a kernel regression, but some config change caused it.
> However, I'm not able to locate which change exactly.
> 
> I'm attaching two configs which I've tried with 4.7.3 - one results in
> thousands of kworkers, and the other doesn't. Also included a diff
> between them.
> 
> Any obvious changes I should try?

The problem is the allocator.

-CONFIG_SLUB=y
+CONFIG_SLAB=y


With SLUB, I'm getting a handful of kworker processes, as expected.

With SLAB, I'm getting thousands of kworker processes.


Not sure if that's expected behaviour or not.


Tomasz Chmielewski
https://lxadm.com

Re: thousands of kworker processes with 4.7.x and 4.8-rc*

[Mike Galbraith]

On Sun, 2016-09-25 at 21:40 +0900, Tomasz Chmielewski wrote:

> The problem is the allocator.
> 
> -CONFIG_SLUB=y
> +CONFIG_SLAB=y
> 
> 
> With SLUB, I'm getting a handful of kworker processes, as expected.
> 
> With SLAB, I'm getting thousands of kworker processes.
> 
> 
> Not sure if that's expected behaviour or not.

I seriously doubt 1500+ kworkers piling up is expected.

4.7.0-rc4       47
4.7.0-rc5       1592

Presuming you didn't switch to SLAB and/or change userspace all around
while testing, that would still indicate a kernel regression lurking
between rc4 and rc5.

	-Mike

Re: thousands of kworker processes with 4.7.x and 4.8-rc*

[Mike Galbraith]

On Sun, 2016-09-25 at 19:21 +0200, Mike Galbraith wrote:
> On Sun, 2016-09-25 at 21:40 +0900, Tomasz Chmielewski wrote:
> 
> > The problem is the allocator.
> > 
> > -CONFIG_SLUB=y
> > +CONFIG_SLAB=y
> > 
> > 
> > With SLUB, I'm getting a handful of kworker processes, as expected.
> > 
> > With SLAB, I'm getting thousands of kworker processes.
> > 
> > 
> > Not sure if that's expected behaviour or not.
> 
> I seriously doubt 1500+ kworkers piling up is expected.
> 
> 4.7.0-rc4       47
> 4.7.0-rc5       1592
> 
> Presuming you didn't switch to SLAB and/or change userspace all around
> while testing, that would still indicate a kernel regression lurking
> between rc4 and rc5.

Nevermind, seems config change was how you met the thing.

FWIW, I turned on SLAB with a distro config here, and see no such
behavior.  If I could reproduce, I'd take the busted config back to
older kernels.

	-Mike

Re: thousands of kworker processes with 4.7.x and 4.8-rc*

[Nikolay Borisov]

On 25.09.2016 15:40, Tomasz Chmielewski wrote:
> On 2016-09-25 18:29, Tomasz Chmielewski wrote:
> 
>>> I'll try to bisect.
>>
>> OK, not a kernel regression, but some config change caused it.
>> However, I'm not able to locate which change exactly.
>>
>> I'm attaching two configs which I've tried with 4.7.3 - one results in
>> thousands of kworkers, and the other doesn't. Also included a diff
>> between them.
>>
>> Any obvious changes I should try?
> 
> The problem is the allocator.
> 
> -CONFIG_SLUB=y
> +CONFIG_SLAB=y
> 
> 
> With SLUB, I'm getting a handful of kworker processes, as expected.
> 
> With SLAB, I'm getting thousands of kworker processes.
> 
> 
> Not sure if that's expected behaviour or not.


Why don't you sample the stacks of some of those kworker processes to
see if they are all executing a parituclar piece of work. That might
help you narrow down where they originate from. Cat multiple
/proc/$kworker-pid/stack files and see if a pattern emerges.

Regards,
Nikolay

> 
> 
> Tomasz Chmielewski
> https://lxadm.com
> 
> 

Re: thousands of kworker processes with 4.7.x and 4.8-rc*

[Tomasz Chmielewski]

On 2016-09-26 04:07, Nikolay Borisov wrote:

>> Not sure if that's expected behaviour or not.
> 
> 
> Why don't you sample the stacks of some of those kworker processes to
> see if they are all executing a parituclar piece of work. That might
> help you narrow down where they originate from. Cat multiple
> /proc/$kworker-pid/stack files and see if a pattern emerges.

FYI, it was reproduced and bisected here (scroll to the bottom):

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1626564


Tomasz Chmielewski
https://lxadm.com