From: Marc Gonzalez <marc.w.gonzalez@free.fr>
To: Jianchao Wang <jianchao.w.wang@oracle.com>,
Christoph Hellwig <hch@infradead.org>,
Jens Axboe <axboe@kernel.dk>
Cc: fsdevel <linux-fsdevel@vger.kernel.org>,
linux-block <linux-block@vger.kernel.org>,
SCSI <linux-scsi@vger.kernel.org>,
Joao Pinto <jpinto@synopsys.com>,
Jeffrey Hugo <jhugo@codeaurora.org>,
Evan Green <evgreen@chromium.org>,
Matthias Kaehlcke <mka@chromium.org>,
Douglas Anderson <dianders@chromium.org>,
Stephen Boyd <swboyd@chromium.org>,
Tomas Winkler <tomas.winkler@intel.com>,
Adrian Hunter <adrian.hunter@intel.com>,
Alim Akhtar <alim.akhtar@samsung.com>,
Avri Altman <avri.altman@wdc.com>,
Bart Van Assche <bart.vanassche@wdc.com>,
Martin Petersen <martin.petersen@oracle.com>,
Bjorn Andersson <bjorn.andersson@linaro.org>
Subject: Re: dd hangs when reading large partitions
Date: Wed, 6 Feb 2019 17:16:26 +0100 [thread overview]
Message-ID: <7bd8b010-bf0c-ad64-f927-2d2187a18d0b@free.fr> (raw)
In-Reply-To: <5132e41b-cb1a-5b81-4a72-37d0f9ea4bb9@oracle.com>
On 23/01/2019 04:10, jianchao.wang wrote:
> Looking through the log
> https://pastebin.ubuntu.com/p/YSm82GxhNW/
>
> rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
> rcu: 6-...0: (13995 ticks this GP) idle=e16/1/0x4000000000000000 softirq=155/155 fqs=655
> rcu: (detected by 4, t=576151 jiffies, g=-391, q=18)
> Task dump for CPU 6:
> dd R running task 0 677 671 0x00000002
> Call trace:
> __switch_to+0x174/0x1e0
> ufshcd_queuecommand+0x84c/0x9a8
>
> The task is in RUNNING state when it was scheduled out.
> So it should be a preempt (the path is under preemptible rcu).
>
> And I wonder why it is not scheduled back for so long time that rcu stall was triggered.
> And who was occupying the cpu all the time ?
>
> Would you please try to show all running tasks on all cpu ?
>
> echo l > /proc/sysrq-trigger
>
> In addition, since the rcu grace period didn't pass, a lot of things could not be moved
> forward.
# echo w > /proc/sysrq-trigger
[ 517.619818] sysrq: Show Blocked State
[ 517.620542] task PC stack pid father
# echo l > /proc/sysrq-trigger
[ 530.692149] sysrq: Show backtrace of all active CPUs
# ps -o pid,user,stat,args
PID USER STAT COMMAND
1 root S init
2 root SW [kthreadd]
3 root IW< [rcu_gp]
4 root IW< [rcu_par_gp]
5 root IW [kworker/0:0-eve]
6 root IW< [kworker/0:0H]
7 root IW [kworker/u16:0-u]
8 root IW< [mm_percpu_wq]
9 root SW [ksoftirqd/0]
10 root IW [rcu_preempt]
11 root SW [migration/0]
12 root IW [kworker/0:1-rcu]
13 root SW [cpuhp/0]
14 root SW [cpuhp/1]
15 root SW [migration/1]
16 root SW [ksoftirqd/1]
17 root IW [kworker/1:0-eve]
18 root IW< [kworker/1:0H]
19 root SW [cpuhp/2]
20 root SW [migration/2]
21 root SW [ksoftirqd/2]
22 root IW [kworker/2:0-eve]
23 root IW< [kworker/2:0H]
24 root SW [cpuhp/3]
25 root SW [migration/3]
26 root SW [ksoftirqd/3]
27 root IW [kworker/3:0-eve]
28 root IW< [kworker/3:0H]
29 root SW [cpuhp/4]
30 root SW [migration/4]
31 root SW [ksoftirqd/4]
32 root IW [kworker/4:0-eve]
33 root IW< [kworker/4:0H-kb]
34 root SW [cpuhp/5]
35 root SW [migration/5]
36 root SW [ksoftirqd/5]
37 root IW [kworker/5:0-eve]
38 root IW< [kworker/5:0H]
39 root SW [cpuhp/6]
40 root SW [migration/6]
41 root SW [ksoftirqd/6]
42 root IW [kworker/6:0-eve]
43 root IW< [kworker/6:0H]
44 root SW [cpuhp/7]
45 root SW [migration/7]
46 root SW [ksoftirqd/7]
47 root IW [kworker/7:0-eve]
48 root IW< [kworker/7:0H]
49 root SW [kdevtmpfs]
50 root IW [kworker/u16:1-e]
52 root SW [rcu_tasks_kthre]
61 root IW [kworker/3:1-mm_]
68 root IW [kworker/5:1-eve]
76 root IW [kworker/6:1-eve]
77 root IW [kworker/7:1-mm_]
85 root IW [kworker/1:1-eve]
93 root IW [kworker/2:1-eve]
102 root IW [kworker/4:1-mm_]
202 root SW [khungtaskd]
203 root SW [oom_reaper]
204 root IW< [writeback]
206 root SW [kcompactd0]
208 root IW< [kblockd]
209 root IW< [devfreq_wq]
211 root IW [kworker/u16:2-e]
214 root IW [kworker/u16:3-e]
224 root IW [kworker/u16:4-e]
235 root IW [kworker/u16:5-e]
259 root IW [kworker/u16:6-e]
380 root SW [kswapd0]
509 root SW [scsi_eh_0]
510 root IW< [scsi_tmf_0]
511 root IW< [ufs_clk_gating_]
512 root IW< [ufs_clkscaling_]
551 root IW [kworker/u16:7-e]
635 root IW [kworker/u16:8-u]
656 root S /sbin/klogd -n
674 root S /bin/sh
675 root IW< [kworker/4:1H]
677 root R ps -o pid,user,stat,args
# dd if=/dev/sde of=/dev/null bs=1M & while true; do echo l > /proc/sysrq-trigger; sleep 1; done
https://pastebin.ubuntu.com/p/Sx4hXRbHby/
What information can I provide to help debug this issue?
Regards.
next prev parent reply other threads:[~2019-02-06 16:17 UTC|newest]
Thread overview: 34+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-01-18 12:10 dd hangs when reading large partitions Marc Gonzalez
2019-01-18 13:39 ` Ming Lei
2019-01-18 14:54 ` Marc Gonzalez
2019-01-18 15:18 ` jianchao.wang
2019-01-18 17:38 ` Marc Gonzalez
2019-01-18 17:48 ` Jens Axboe
2019-01-18 17:51 ` Bart Van Assche
2019-01-18 19:00 ` Jens Axboe
2019-01-19 9:56 ` Christoph Hellwig
2019-01-19 14:37 ` Jens Axboe
2019-01-19 16:09 ` Bart Van Assche
2019-01-21 8:33 ` Christoph Hellwig
2019-01-19 19:47 ` Marc Gonzalez
2019-01-19 20:45 ` Marc Gonzalez
2019-01-21 8:33 ` Christoph Hellwig
2019-01-21 15:22 ` Marc Gonzalez
2019-01-22 3:12 ` jianchao.wang
2019-01-22 10:59 ` Marc Gonzalez
2019-01-22 12:49 ` Marc Gonzalez
2019-01-22 16:17 ` Marc Gonzalez
2019-01-22 16:22 ` Greg Kroah-Hartman
2019-01-22 19:07 ` Evan Green
2019-01-23 3:10 ` jianchao.wang
2019-02-06 16:16 ` Marc Gonzalez [this message]
2019-02-06 17:05 ` Marc Gonzalez
2019-02-07 10:44 ` Marc Gonzalez
2019-02-07 16:56 ` Marc Gonzalez
2019-02-08 15:33 ` Marc Gonzalez
2019-02-08 15:49 ` Bart Van Assche
2019-02-09 11:57 ` Marc Gonzalez
2019-02-11 16:36 ` Marc Gonzalez
2019-02-11 17:27 ` Marc Gonzalez
2019-02-12 15:26 ` [SOLVED] " Marc Gonzalez
2019-01-18 19:27 ` Douglas Gilbert
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=7bd8b010-bf0c-ad64-f927-2d2187a18d0b@free.fr \
--to=marc.w.gonzalez@free.fr \
--cc=adrian.hunter@intel.com \
--cc=alim.akhtar@samsung.com \
--cc=avri.altman@wdc.com \
--cc=axboe@kernel.dk \
--cc=bart.vanassche@wdc.com \
--cc=bjorn.andersson@linaro.org \
--cc=dianders@chromium.org \
--cc=evgreen@chromium.org \
--cc=hch@infradead.org \
--cc=jhugo@codeaurora.org \
--cc=jianchao.w.wang@oracle.com \
--cc=jpinto@synopsys.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-scsi@vger.kernel.org \
--cc=martin.petersen@oracle.com \
--cc=mka@chromium.org \
--cc=swboyd@chromium.org \
--cc=tomas.winkler@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).