From: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
To: Eric Wheeler <linux-mm@lists.ewheeler.net>,
Michal Hocko <mhocko@kernel.org>
Cc: linux-mm@kvack.org, "Kirill A. Shutemov" <kirill@shutemov.name>,
Minchan Kim <minchan@kernel.org>, Tejun Heo <tj@kernel.org>,
Alasdair Kergon <agk@redhat.com>,
Mike Snitzer <snitzer@redhat.com>,
Kent Overstreet <kent.overstreet@gmail.com>
Subject: Re: Possible deadlock in v4.14.15 contention on shrinker_rwsem in shrink_slab()
Date: Sat, 27 Jan 2018 15:34:03 +0900 [thread overview]
Message-ID: <4e9300f9-14c4-84a9-2258-b7e52bb6f753@I-love.SAKURA.ne.jp> (raw)
In-Reply-To: <alpine.LRH.2.11.1801261846520.7450@mail.ewheeler.net>
On 2018/01/27 4:32, Eric Wheeler wrote:
> On Thu, 25 Jan 2018, Michal Hocko wrote:
>
>> [CC Kirill, Minchan]
>> On Wed 24-01-18 23:57:42, Eric Wheeler wrote:
>>> Hello all,
>>>
>>> We are getting processes stuck with /proc/pid/stack listing the following:
>>>
>>> [<ffffffffac0cd0d2>] io_schedule+0x12/0x40
>>> [<ffffffffac1b4695>] __lock_page+0x105/0x150
>>> [<ffffffffac1b4dc1>] pagecache_get_page+0x161/0x210
>>> [<ffffffffac1d4ab4>] shmem_unused_huge_shrink+0x334/0x3f0
>>> [<ffffffffac251546>] super_cache_scan+0x176/0x180
>>> [<ffffffffac1cb6c5>] shrink_slab+0x275/0x460
>>> [<ffffffffac1d0b8e>] shrink_node+0x10e/0x320
>>> [<ffffffffac1d0f3d>] node_reclaim+0x19d/0x250
>>> [<ffffffffac1be0aa>] get_page_from_freelist+0x16a/0xac0
>>> [<ffffffffac1bed87>] __alloc_pages_nodemask+0x107/0x290
>>> [<ffffffffac06dbc3>] pte_alloc_one+0x13/0x40
>>> [<ffffffffac1ef329>] __pte_alloc+0x19/0x100
>>> [<ffffffffac1f17b8>] alloc_set_pte+0x468/0x4c0
>>> [<ffffffffac1f184a>] finish_fault+0x3a/0x70
>>> [<ffffffffac1f369a>] __handle_mm_fault+0x94a/0x1190
>>> [<ffffffffac1f3fa4>] handle_mm_fault+0xc4/0x1d0
>>> [<ffffffffac0682a3>] __do_page_fault+0x253/0x4d0
>>> [<ffffffffac068553>] do_page_fault+0x33/0x120
>>> [<ffffffffac8019dc>] page_fault+0x4c/0x60
>>>
>>>
>>> For some reason io_schedule is not coming back,
>>
>> Is this a permanent state or does the holder eventually releases the
>> lock? It smells like somebody hasn't unlocked the shmem page. Tracking
>> those is a major PITA... :/
>
> Perpetual. It's been locked for a couple days now on two different
> servers, both running the same 4.14.15 build.
>
>
>> Do you remember the last good kernel?
>
> We were stable on 4.1.y for a long time. The only reason we are updating
> is because of the Spectre/Meltdown issues.
>
> I can probably test with 4.9 and let you know if we have the same problem.
> If you have any ideas on creating an easy way to reproduce the problem,
> then I can bisect---but bisecting one day at a time will take a long time,
> and could be prone to bugs which I would like to avoid on this production
> system.
>
> Note that we have cherry-picked neither of f80207727aaca3aa nor
> 0bcac06f27d75285 in our 4.14.15 build.
>
> Questions:
>
> 1. Is there a safe way to break this lock so I can migrate the VMs off of
> the server?
I don't know there is.
>
> 2. Would it be useful if I run the `stap` script attached in Tetsuo's
> patch?
Thinking from SysRq-t output, I feel that some disk read is stuck.
>
>
>
> == This is our current memory summary on the server, and /proc/meminfo is
> == at the very bottom of this email:
> ~]# free -m
> total used free shared buff/cache available
> Mem: 32140 7760 8452 103 15927 22964
> Swap: 9642 764 8877
> ~]# swapon -s
> Filename Type Size Used Priority
> /dev/zram0 partition 9873680 782848 100
> =====================================================
>
>
>
> Below is the output of sysrq-t *without* Tetsuo's patch. I will apply the
> patch, rebuild, and report back when it happens again.
Since rsyslogd failed to catch portion of SysRq-t output, I can't confirm
whether register_shrinker() was in progress (nor all kernel worker threads
were reported).
----------
tcp_recvmsg+0x586/0x9a0
journal: Missed 244 kernel messages
ret_from_fork+0x35/0x40
journal: Missed 2304 kernel messages
? kmem_cache_alloc+0xd2/0x1a0
journal: Missed 319 kernel messages
drbd7916_a_rt.e S 0 4989 2 0x80000080
journal: Missed 198 kernel messages
kernel_recvmsg+0x52/0x70
journal: Missed 1301 kernel messages
? handle_mm_fault+0xc4/0x1d0
journal: Missed 255 kernel messages
? vhost_dev_ioctl+0x3f0/0x3f0
journal: Missed 293 kernel messages
wait_woken+0x64/0x80
journal: Missed 197 kernel messages
ret_from_fork+0x35/0x40
journal: Missed 26 kernel messages
? drbd_destroy_connection+0x160/0x160 [drbd]
journal: Missed 17 kernel messages
? __schedule+0x1dc/0x770
journal: Missed 18 kernel messages
ret_from_fork+0x35/0x40
journal: Missed 27 kernel messages
kthread+0xfc/0x130
----------
But what I was surprised is number of kernel worker threads.
"grep ^kworker/ | sort" matched 314 threads and "grep ^kworker/0:"
matched 244.
----------
kworker/0:0 I 0 21214 2 0x80000080
kworker/0:0H I 0 4 2 0x80000000
kworker/0:1 I 0 21215 2 0x80000080
kworker/0:10 I 0 10340 2 0x80000080
kworker/0:100 I 0 21282 2 0x80000080
kworker/0:101 I 0 21283 2 0x80000080
kworker/0:102 I 0 23765 2 0x80000080
kworker/0:103 I 0 18598 2 0x80000080
kworker/0:104 I 0 21284 2 0x80000080
kworker/0:105 I 0 21285 2 0x80000080
kworker/0:106 I 0 21286 2 0x80000080
kworker/0:107 I 0 21287 2 0x80000080
kworker/0:108 I 0 21288 2 0x80000080
kworker/0:109 I 0 10382 2 0x80000080
kworker/0:11 I 0 10341 2 0x80000080
kworker/0:110 I 0 21289 2 0x80000080
kworker/0:111 I 0 21290 2 0x80000080
kworker/0:112 I 0 10383 2 0x80000080
kworker/0:113 I 0 21292 2 0x80000080
kworker/0:114 I 0 16684 2 0x80000080
kworker/0:115 I 0 21293 2 0x80000080
kworker/0:116 I 0 10384 2 0x80000080
kworker/0:117 I 0 10385 2 0x80000080
kworker/0:118 I 0 31163 2 0x80000080
kworker/0:119 I 0 21295 2 0x80000080
kworker/0:12 I 0 16573 2 0x80000080
kworker/0:120 I 0 18612 2 0x80000080
kworker/0:121 I 0 3835 2 0x80000080
kworker/0:122 I 0 3836 2 0x80000080
kworker/0:123 I 0 10386 2 0x80000080
kworker/0:124 I 0 21297 2 0x80000080
kworker/0:125 I 0 10387 2 0x80000080
kworker/0:126 I 0 10388 2 0x80000080
kworker/0:127 I 0 23691 2 0x80000080
kworker/0:128 I 0 3839 2 0x80000080
kworker/0:129 I 0 10390 2 0x80000080
kworker/0:13 I 0 21221 2 0x80000080
kworker/0:130 I 0 3841 2 0x80000080
kworker/0:131 I 0 10391 2 0x80000080
kworker/0:132 I 0 21301 2 0x80000080
kworker/0:133 I 0 3843 2 0x80000080
kworker/0:134 I 0 21302 2 0x80000080
kworker/0:135 I 0 3844 2 0x80000080
kworker/0:136 I 0 10392 2 0x80000080
kworker/0:137 I 0 10393 2 0x80000080
kworker/0:138 I 0 21305 2 0x80000080
kworker/0:139 I 0 10394 2 0x80000080
kworker/0:14 I 0 10342 2 0x80000080
kworker/0:140 I 0 21307 2 0x80000080
kworker/0:141 I 0 10395 2 0x80000080
kworker/0:142 I 0 10396 2 0x80000080
kworker/0:143 I 0 10397 2 0x80000080
kworker/0:144 I 0 10398 2 0x80000080
kworker/0:145 I 0 3850 2 0x80000080
kworker/0:146 I 0 23798 2 0x80000080
kworker/0:147 I 0 21311 2 0x80000080
kworker/0:148 I 0 26926 2 0x80000080
kworker/0:149 I 0 10399 2 0x80000080
kworker/0:15 I 0 10343 2 0x80000080
kworker/0:150 I 0 10400 2 0x80000080
kworker/0:151 I 0 10401 2 0x80000080
kworker/0:152 I 0 10403 2 0x80000080
kworker/0:153 I 0 3854 2 0x80000080
kworker/0:154 I 0 26931 2 0x80000080
kworker/0:155 I 0 21315 2 0x80000080
kworker/0:156 I 0 10404 2 0x80000080
kworker/0:157 I 0 21317 2 0x80000080
kworker/0:158 I 0 10405 2 0x80000080
kworker/0:159 I 0 21319 2 0x80000080
kworker/0:16 I 0 18530 2 0x80000080
kworker/0:160 I 0 21320 2 0x80000080
kworker/0:161 I 0 21321 2 0x80000080
kworker/0:162 I 0 21322 2 0x80000080
kworker/0:163 I 0 10406 2 0x80000080
kworker/0:164 I 0 21323 2 0x80000080
kworker/0:165 I 0 10407 2 0x80000080
kworker/0:166 I 0 10408 2 0x80000080
kworker/0:167 I 0 10409 2 0x80000080
kworker/0:168 I 0 22590 2 0x80000080
kworker/0:169 I 0 10410 2 0x80000080
kworker/0:17 I 0 10838 2 0x80000080
kworker/0:170 I 0 10411 2 0x80000080
kworker/0:171 I 0 10412 2 0x80000080
kworker/0:172 I 0 3866 2 0x80000080
kworker/0:173 I 0 10413 2 0x80000080
kworker/0:174 I 0 23709 2 0x80000080
kworker/0:175 I 0 21329 2 0x80000080
kworker/0:176 I 0 21330 2 0x80000080
kworker/0:177 I 0 21331 2 0x80000080
kworker/0:178 I 0 3869 2 0x80000080
kworker/0:179 I 0 10414 2 0x80000080
kworker/0:18 I 0 18531 2 0x80000080
kworker/0:180 I 0 10415 2 0x80000080
kworker/0:181 I 0 21333 2 0x80000080
kworker/0:182 I 0 23715 2 0x80000080
kworker/0:183 I 0 10416 2 0x80000080
kworker/0:184 I 0 10417 2 0x80000080
kworker/0:185 I 0 3872 2 0x80000080
kworker/0:186 I 0 3873 2 0x80000080
kworker/0:187 I 0 10418 2 0x80000080
kworker/0:188 I 0 21337 2 0x80000080
kworker/0:189 I 0 21338 2 0x80000080
kworker/0:19 I 0 21223 2 0x80000080
kworker/0:190 I 0 10419 2 0x80000080
kworker/0:191 I 0 10420 2 0x80000080
kworker/0:192 I 0 10421 2 0x80000080
kworker/0:193 I 0 21340 2 0x80000080
kworker/0:194 I 0 10422 2 0x80000080
kworker/0:195 I 0 3877 2 0x80000080
kworker/0:196 I 0 10423 2 0x80000080
kworker/0:197 I 0 21342 2 0x80000080
kworker/0:198 I 0 10424 2 0x80000080
kworker/0:199 I 0 3881 2 0x80000080
kworker/0:1H I 0 457 2 0x80000000
kworker/0:2 I 0 7149 2 0x80000080
kworker/0:20 I 0 10344 2 0x80000080
kworker/0:200 I 0 3882 2 0x80000080
kworker/0:201 I 0 21344 2 0x80000080
kworker/0:202 I 0 21345 2 0x80000080
kworker/0:203 I 0 10425 2 0x80000080
kworker/0:204 I 0 10426 2 0x80000080
kworker/0:205 I 0 10428 2 0x80000080
kworker/0:206 I 0 3887 2 0x80000080
kworker/0:207 I 0 10429 2 0x80000080
kworker/0:208 I 0 10430 2 0x80000080
kworker/0:209 I 0 10431 2 0x80000080
kworker/0:21 I 0 18533 2 0x80000080
kworker/0:210 I 0 10432 2 0x80000080
kworker/0:211 I 0 3890 2 0x80000080
kworker/0:212 I 0 10433 2 0x80000080
kworker/0:213 I 0 10434 2 0x80000080
kworker/0:214 I 0 10435 2 0x80000080
kworker/0:215 I 0 21352 2 0x80000080
kworker/0:216 I 0 10436 2 0x80000080
kworker/0:217 I 0 10437 2 0x80000080
kworker/0:218 I 0 21354 2 0x80000080
kworker/0:219 I 0 10439 2 0x80000080
kworker/0:22 I 0 10346 2 0x80000080
kworker/0:220 I 0 10440 2 0x80000080
kworker/0:221 I 0 21356 2 0x80000080
kworker/0:222 I 0 10441 2 0x80000080
kworker/0:223 I 0 21358 2 0x80000080
kworker/0:224 I 0 2432 2 0x80000080
kworker/0:225 I 0 2433 2 0x80000080
kworker/0:226 I 0 2434 2 0x80000080
kworker/0:227 I 0 2435 2 0x80000080
kworker/0:229 I 0 2437 2 0x80000080
kworker/0:23 I 0 21225 2 0x80000080
kworker/0:231 I 0 2439 2 0x80000080
kworker/0:232 I 0 2440 2 0x80000080
kworker/0:234 I 0 22654 2 0x80000080
kworker/0:236 I 0 2444 2 0x80000080
kworker/0:237 I 0 3909 2 0x80000080
kworker/0:24 I 0 21226 2 0x80000080
kworker/0:241 I 0 3913 2 0x80000080
kworker/0:244 I 0 3916 2 0x80000080
kworker/0:245 I 0 2449 2 0x80000080
kworker/0:246 I 0 2450 2 0x80000080
kworker/0:247 I 0 18632 2 0x80000080
kworker/0:25 I 0 21227 2 0x80000080
kworker/0:250 I 0 2453 2 0x80000080
kworker/0:253 I 0 2455 2 0x80000080
kworker/0:26 I 0 21228 2 0x80000080
kworker/0:262 I 0 31402 2 0x80000080
kworker/0:27 I 0 10347 2 0x80000080
kworker/0:28 I 0 21230 2 0x80000080
kworker/0:29 I 0 18538 2 0x80000080
kworker/0:3 I 0 21216 2 0x80000080
kworker/0:30 I 0 10348 2 0x80000080
kworker/0:31 I 0 18540 2 0x80000080
kworker/0:32 R running task 0 18541 2 0x80000080
kworker/0:33 I 0 10349 2 0x80000080
kworker/0:34 I 0 10350 2 0x80000080
kworker/0:35 I 0 10351 2 0x80000080
kworker/0:36 I 0 21234 2 0x80000080
kworker/0:37 I 0 18544 2 0x80000080
kworker/0:38 I 0 10352 2 0x80000080
kworker/0:39 I 0 10353 2 0x80000080
kworker/0:4 I 0 18521 2 0x80000080
kworker/0:40 I 0 21236 2 0x80000080
kworker/0:41 I 0 10354 2 0x80000080
kworker/0:42 I 0 18549 2 0x80000080
kworker/0:43 I 0 21237 2 0x80000080
kworker/0:44 I 0 10355 2 0x80000080
kworker/0:45 I 0 10356 2 0x80000080
kworker/0:46 I 0 10357 2 0x80000080
kworker/0:47 I 0 21241 2 0x80000080
kworker/0:48 I 0 21242 2 0x80000080
kworker/0:49 I 0 18554 2 0x80000080
kworker/0:5 I 0 18522 2 0x80000080
kworker/0:50 I 0 23141 2 0x80000080
kworker/0:51 I 0 10358 2 0x80000080
kworker/0:52 I 0 16589 2 0x80000080
kworker/0:53 I 0 18556 2 0x80000080
kworker/0:54 I 0 18557 2 0x80000080
kworker/0:55 I 0 21244 2 0x80000080
kworker/0:56 I 0 18558 2 0x80000080
kworker/0:57 I 0 23146 2 0x80000080
kworker/0:58 I 0 18559 2 0x80000080
kworker/0:59 I 0 21245 2 0x80000080
kworker/0:6 I 0 21217 2 0x80000080
kworker/0:60 I 0 21250 2 0x80000080
kworker/0:61 I 0 18561 2 0x80000080
kworker/0:62 I 0 10359 2 0x80000080
kworker/0:63 I 0 10360 2 0x80000080
kworker/0:64 I 0 21253 2 0x80000080
kworker/0:65 I 0 21254 2 0x80000080
kworker/0:66 I 0 21255 2 0x80000080
kworker/0:67 I 0 10361 2 0x80000080
kworker/0:68 I 0 21257 2 0x80000080
kworker/0:69 I 0 21258 2 0x80000080
kworker/0:7 I 0 10339 2 0x80000080
kworker/0:70 I 0 18570 2 0x80000080
kworker/0:71 I 0 10362 2 0x80000080
kworker/0:72 I 0 10363 2 0x80000080
kworker/0:73 I 0 21261 2 0x80000080
kworker/0:74 I 0 10365 2 0x80000080
kworker/0:75 I 0 10366 2 0x80000080
kworker/0:76 I 0 21264 2 0x80000080
kworker/0:77 I 0 21265 2 0x80000080
kworker/0:78 I 0 10367 2 0x80000080
kworker/0:79 I 0 10368 2 0x80000080
kworker/0:8 I 0 18525 2 0x80000080
kworker/0:80 I 0 10369 2 0x80000080
kworker/0:81 I 0 10370 2 0x80000080
kworker/0:82 I 0 10371 2 0x80000080
kworker/0:83 I 0 21270 2 0x80000080
kworker/0:84 I 0 10372 2 0x80000080
kworker/0:85 I 0 10373 2 0x80000080
kworker/0:86 I 0 18584 2 0x80000080
kworker/0:87 I 0 10374 2 0x80000080
kworker/0:88 I 0 10375 2 0x80000080
kworker/0:89 I 0 21274 2 0x80000080
kworker/0:9 I 0 18526 2 0x80000080
kworker/0:90 I 0 10376 2 0x80000080
kworker/0:91 I 0 10377 2 0x80000080
kworker/0:92 I 0 10378 2 0x80000080
kworker/0:93 I 0 10379 2 0x80000080
kworker/0:94 I 0 10380 2 0x80000080
kworker/0:95 I 0 10381 2 0x80000080
kworker/0:96 I 0 18593 2 0x80000080
kworker/0:97 I 0 21280 2 0x80000080
kworker/0:98 I 0 23179 2 0x80000080
kworker/0:99 I 0 21281 2 0x80000080
kworker/1:0 I 0 24108 2 0x80000080
kworker/1:0H I 0 18 2 0x80000000
kworker/1:1 I 0 24109 2 0x80000080
kworker/1:10 I 0 24118 2 0x80000080
kworker/1:11 I 0 24119 2 0x80000080
kworker/1:12 I 0 24120 2 0x80000080
kworker/1:13 I 0 24121 2 0x80000080
kworker/1:14 I 0 24122 2 0x80000080
kworker/1:15 I 0 24123 2 0x80000080
kworker/1:16 I 0 24124 2 0x80000080
kworker/1:17 I 0 24125 2 0x80000080
kworker/1:18 I 0 24126 2 0x80000080
kworker/1:19 I 0 24127 2 0x80000080
kworker/1:1H I 0 550 2 0x80000000
kworker/1:2 I 0 24110 2 0x80000080
kworker/1:20 I 0 24128 2 0x80000080
kworker/1:21 I 0 24129 2 0x80000080
kworker/1:22 I 0 24130 2 0x80000080
kworker/1:23 I 0 24131 2 0x80000080
kworker/1:24 I 0 24132 2 0x80000080
kworker/1:25 I 0 24133 2 0x80000080
kworker/1:26 I 0 5787 2 0x80000080
kworker/1:27 I 0 5788 2 0x80000080
kworker/1:28 I 0 24134 2 0x80000080
kworker/1:29 I 0 24135 2 0x80000080
kworker/1:3 I 0 24111 2 0x80000080
kworker/1:4 I 0 24112 2 0x80000080
kworker/1:5 I 0 24113 2 0x80000080
kworker/1:6 I 0 24114 2 0x80000080
kworker/1:7 I 0 24115 2 0x80000080
kworker/1:8 I 0 24116 2 0x80000080
kworker/1:9 I 0 24117 2 0x80000080
kworker/2:0 I 0 9749 2 0x80000080
kworker/2:0H I 0 24 2 0x80000000
kworker/2:1 I 0 9750 2 0x80000080
kworker/2:1H I 0 591 2 0x80000000
kworker/2:2 I 0 9751 2 0x80000080
kworker/2:68 I 0 30993 2 0x80000080
kworker/2:69 I 0 30995 2 0x80000080
kworker/3:0 I 0 31071 2 0x80000080
kworker/3:0H I 0 30 2 0x80000000
kworker/3:1H I 0 455 2 0x80000000
kworker/3:2 I 0 1299 2 0x80000080
kworker/3:3 I 0 4367 2 0x80000080
kworker/3:69 I 0 13343 2 0x80000080
kworker/3:70 R running task 0 13344 2 0x80000080
kworker/4:0 I 0 7454 2 0x80000080
kworker/4:0H I 0 36 2 0x80000000
kworker/4:1H I 0 456 2 0x80000000
kworker/4:8 I 0 23606 2 0x80000080
kworker/4:9 I 0 23607 2 0x80000080
kworker/5:0 I 0 7434 2 0x80000080
kworker/5:0H I 0 42 2 0x80000000
kworker/5:1 I 0 21046 2 0x80000080
kworker/5:1H I 0 454 2 0x80000000
kworker/5:3 I 0 22704 2 0x80000080
kworker/5:37 I 0 31097 2 0x80000080
kworker/6:0 I 0 4395 2 0x80000080
kworker/6:0H I 0 48 2 0x80000000
kworker/6:10 I 0 6159 2 0x80000080
kworker/6:1H I 0 553 2 0x80000000
kworker/7:0H I 0 54 2 0x80000000
kworker/7:1H I 0 549 2 0x80000000
kworker/7:2 I 0 22741 2 0x80000080
kworker/7:3 I 0 22742 2 0x80000080
kworker/7:5 I 0 22744 2 0x80000080
kworker/u16:0 I 0 14713 2 0x80000080
kworker/u16:1 D 0 9752 2 0x80000080
kworker/u16:2 I 0 3153 2 0x80000080
kworker/u16:3 I 0 28108 2 0x80000080
----------
Although most of them were idle, and the system had enough free memory
for creating workqueues, is there possibility that waiting for a work
item to complete get stuck due to workqueue availability?
( Was there no "Showing busy workqueues and worker pools:" line?
http://lkml.kernel.org/r/20170502041235.zqmywvj5tiiom3jk@merlins.org had it. )
----------
kworker/0:262 I 0 31402 2 0x80000080
Call Trace:
? __schedule+0x1dc/0x770
? cached_dev_write_complete+0x2c/0x60 [bcache]
schedule+0x32/0x80
worker_thread+0xc3/0x3e0
kthread+0xfc/0x130
? rescuer_thread+0x380/0x380
? kthread_park+0x60/0x60
ret_from_fork+0x35/0x40
----------
> ~]# cat /proc/meminfo
> MemTotal: 32912276 kB
> MemFree: 8646212 kB
> MemAvailable: 23506448 kB
> Buffers: 230592 kB
> Cached: 15443124 kB
> SwapCached: 6112 kB
> Active: 14235496 kB
> Inactive: 7679336 kB
> Active(anon): 3723980 kB
> Inactive(anon): 2634188 kB
> Active(file): 10511516 kB
> Inactive(file): 5045148 kB
> Unevictable: 233704 kB
> Mlocked: 233704 kB
> SwapTotal: 9873680 kB
> SwapFree: 9090832 kB
> Dirty: 40 kB
> Writeback: 0 kB
> AnonPages: 6435292 kB
> Mapped: 162024 kB
> Shmem: 105880 kB
> Slab: 635280 kB
> SReclaimable: 311468 kB
> SUnreclaim: 323812 kB
> KernelStack: 25296 kB
> PageTables: 31376 kB
> NFS_Unstable: 0 kB
> Bounce: 0 kB
> WritebackTmp: 0 kB
> CommitLimit: 26329816 kB
> Committed_AS: 16595004 kB
> VmallocTotal: 34359738367 kB
> VmallocUsed: 0 kB
> VmallocChunk: 0 kB
> HardwareCorrupted: 0 kB
> AnonHugePages: 6090752 kB
> ShmemHugePages: 0 kB
> ShmemPmdMapped: 0 kB
> CmaTotal: 0 kB
> CmaFree: 0 kB
> HugePages_Total: 0
> HugePages_Free: 0
> HugePages_Rsvd: 0
> HugePages_Surp: 0
> Hugepagesize: 2048 kB
> DirectMap4k: 1012624 kB
> DirectMap2M: 32514048 kB
One of workqueue threads was waiting at
----------
static void *new_read(struct dm_bufio_client *c, sector_t block,
enum new_flag nf, struct dm_buffer **bp)
{
int need_submit;
struct dm_buffer *b;
LIST_HEAD(write_list);
dm_bufio_lock(c);
b = __bufio_new(c, block, nf, &need_submit, &write_list);
#ifdef CONFIG_DM_DEBUG_BLOCK_STACK_TRACING
if (b && b->hold_count == 1)
buffer_record_stack(b);
#endif
dm_bufio_unlock(c);
__flush_write_list(&write_list);
if (!b)
return NULL;
if (need_submit)
submit_io(b, READ, read_endio);
wait_on_bit_io(&b->state, B_READING, TASK_UNINTERRUPTIBLE); // <= here
if (b->read_error) {
int error = blk_status_to_errno(b->read_error);
dm_bufio_release(b);
return ERR_PTR(error);
}
*bp = b;
return b->data;
}
----------
but what are possible reasons? Does this request depend on workqueue availability?
----------
kworker/0:32 R running task 0 18541 2 0x80000080
Call Trace:
? __schedule+0x1dc/0x770
schedule+0x32/0x80
worker_thread+0xc3/0x3e0
kthread+0xfc/0x130
? rescuer_thread+0x380/0x380
? kthread_park+0x60/0x60
ret_from_fork+0x35/0x40
kworker/3:70 R running task 0 13344 2 0x80000080
Workqueue: events_power_efficient fb_flashcursor
Call Trace:
? fb_flashcursor+0x131/0x140
? bit_clear+0x110/0x110
? process_one_work+0x141/0x340
? worker_thread+0x47/0x3e0
? kthread+0xfc/0x130
? rescuer_thread+0x380/0x380
? kthread_park+0x60/0x60
? ret_from_fork+0x35/0x40
kworker/u16:1 D 0 9752 2 0x80000080
Workqueue: dm-thin do_worker [dm_thin_pool]
Call Trace:
? __schedule+0x1dc/0x770
? out_of_line_wait_on_atomic_t+0x110/0x110
schedule+0x32/0x80
io_schedule+0x12/0x40
bit_wait_io+0xd/0x50
__wait_on_bit+0x5a/0x90
out_of_line_wait_on_bit+0x8e/0xb0
? bit_waitqueue+0x30/0x30
new_read+0x9f/0x100 [dm_bufio]
dm_bm_read_lock+0x21/0x70 [dm_persistent_data]
ro_step+0x31/0x60 [dm_persistent_data]
btree_lookup_raw.constprop.7+0x3a/0x100 [dm_persistent_data]
dm_btree_lookup+0x71/0x100 [dm_persistent_data]
__find_block+0x55/0xa0 [dm_thin_pool]
dm_thin_find_block+0x48/0x70 [dm_thin_pool]
process_cell+0x67/0x510 [dm_thin_pool]
? dm_bio_detain+0x4c/0x60 [dm_bio_prison]
process_bio+0xaa/0xc0 [dm_thin_pool]
do_worker+0x632/0x8b0 [dm_thin_pool]
? __switch_to+0xa8/0x480
process_one_work+0x141/0x340
worker_thread+0x47/0x3e0
kthread+0xfc/0x130
? rescuer_thread+0x380/0x380
? kthread_park+0x60/0x60
? SyS_exit_group+0x10/0x10
ret_from_fork+0x35/0x40
----------
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2018-01-27 6:34 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-01-24 23:57 Possible deadlock in v4.14.15 contention on shrinker_rwsem in shrink_slab() Eric Wheeler
2018-01-25 8:35 ` Michal Hocko
2018-01-25 10:51 ` Tetsuo Handa
2018-01-26 19:32 ` Eric Wheeler
2018-01-27 6:34 ` Tetsuo Handa [this message]
2018-01-27 14:13 ` Tetsuo Handa
2018-01-29 0:04 ` Eric Wheeler
2018-01-29 5:27 ` Tetsuo Handa
2018-02-03 7:48 ` Tetsuo Handa
2018-02-12 12:43 ` Michal Hocko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4e9300f9-14c4-84a9-2258-b7e52bb6f753@I-love.SAKURA.ne.jp \
--to=penguin-kernel@i-love.sakura.ne.jp \
--cc=agk@redhat.com \
--cc=kent.overstreet@gmail.com \
--cc=kirill@shutemov.name \
--cc=linux-mm@kvack.org \
--cc=linux-mm@lists.ewheeler.net \
--cc=mhocko@kernel.org \
--cc=minchan@kernel.org \
--cc=snitzer@redhat.com \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.