Re: dd hangs when reading large partitions

From: Marc Gonzalez <marc.w.gonzalez@free.fr>
To: Jianchao Wang <jianchao.w.wang@oracle.com>,
	Christoph Hellwig <hch@infradead.org>,
	Jens Axboe <axboe@kernel.dk>
Cc: fsdevel <linux-fsdevel@vger.kernel.org>,
	linux-block <linux-block@vger.kernel.org>,
	SCSI <linux-scsi@vger.kernel.org>,
	Joao Pinto <jpinto@synopsys.com>,
	Jeffrey Hugo <jhugo@codeaurora.org>,
	Evan Green <evgreen@chromium.org>,
	Matthias Kaehlcke <mka@chromium.org>,
	Douglas Anderson <dianders@chromium.org>,
	Stephen Boyd <swboyd@chromium.org>,
	Tomas Winkler <tomas.winkler@intel.com>,
	Adrian Hunter <adrian.hunter@intel.com>,
	Alim Akhtar <alim.akhtar@samsung.com>,
	Avri Altman <avri.altman@wdc.com>,
	Bart Van Assche <bart.vanassche@wdc.com>,
	Martin Petersen <martin.petersen@oracle.com>,
	Bjorn Andersson <bjorn.andersson@linaro.org>
Subject: Re: dd hangs when reading large partitions
Date: Wed, 6 Feb 2019 17:16:26 +0100	[thread overview]
Message-ID: <7bd8b010-bf0c-ad64-f927-2d2187a18d0b@free.fr> (raw)
In-Reply-To: <5132e41b-cb1a-5b81-4a72-37d0f9ea4bb9@oracle.com>

On 23/01/2019 04:10, jianchao.wang wrote:

> Looking through the log
> https://pastebin.ubuntu.com/p/YSm82GxhNW/
> 
> rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
> rcu:    6-...0: (13995 ticks this GP) idle=e16/1/0x4000000000000000 softirq=155/155 fqs=655 
> rcu:    (detected by 4, t=576151 jiffies, g=-391, q=18)
> Task dump for CPU 6:
> dd              R  running task        0   677    671 0x00000002
> Call trace:
>  __switch_to+0x174/0x1e0
>  ufshcd_queuecommand+0x84c/0x9a8
> 
> The task is in RUNNING state when it was scheduled out.
> So it should be a preempt (the path is under preemptible rcu).
> 
> And I wonder why it is not scheduled back for so long time that rcu stall was triggered.
> And who was occupying the cpu all the time ?
> 
> Would you please try to show all running tasks on all cpu ?
> 
> echo l > /proc/sysrq-trigger
> 
> In addition, since the rcu grace period didn't pass, a lot of things could not be moved
> forward.

# echo w > /proc/sysrq-trigger
[  517.619818] sysrq: Show Blocked State
[  517.620542]   task                        PC stack   pid father

# echo l > /proc/sysrq-trigger
[  530.692149] sysrq: Show backtrace of all active CPUs

# ps -o pid,user,stat,args
PID   USER     STAT COMMAND
    1 root     S    init
    2 root     SW   [kthreadd]
    3 root     IW<  [rcu_gp]
    4 root     IW<  [rcu_par_gp]
    5 root     IW   [kworker/0:0-eve]
    6 root     IW<  [kworker/0:0H]
    7 root     IW   [kworker/u16:0-u]
    8 root     IW<  [mm_percpu_wq]
    9 root     SW   [ksoftirqd/0]
   10 root     IW   [rcu_preempt]
   11 root     SW   [migration/0]
   12 root     IW   [kworker/0:1-rcu]
   13 root     SW   [cpuhp/0]
   14 root     SW   [cpuhp/1]
   15 root     SW   [migration/1]
   16 root     SW   [ksoftirqd/1]
   17 root     IW   [kworker/1:0-eve]
   18 root     IW<  [kworker/1:0H]
   19 root     SW   [cpuhp/2]
   20 root     SW   [migration/2]
   21 root     SW   [ksoftirqd/2]
   22 root     IW   [kworker/2:0-eve]
   23 root     IW<  [kworker/2:0H]
   24 root     SW   [cpuhp/3]
   25 root     SW   [migration/3]
   26 root     SW   [ksoftirqd/3]
   27 root     IW   [kworker/3:0-eve]
   28 root     IW<  [kworker/3:0H]
   29 root     SW   [cpuhp/4]
   30 root     SW   [migration/4]
   31 root     SW   [ksoftirqd/4]
   32 root     IW   [kworker/4:0-eve]
   33 root     IW<  [kworker/4:0H-kb]
   34 root     SW   [cpuhp/5]
   35 root     SW   [migration/5]
   36 root     SW   [ksoftirqd/5]
   37 root     IW   [kworker/5:0-eve]
   38 root     IW<  [kworker/5:0H]
   39 root     SW   [cpuhp/6]
   40 root     SW   [migration/6]
   41 root     SW   [ksoftirqd/6]
   42 root     IW   [kworker/6:0-eve]
   43 root     IW<  [kworker/6:0H]
   44 root     SW   [cpuhp/7]
   45 root     SW   [migration/7]
   46 root     SW   [ksoftirqd/7]
   47 root     IW   [kworker/7:0-eve]
   48 root     IW<  [kworker/7:0H]
   49 root     SW   [kdevtmpfs]
   50 root     IW   [kworker/u16:1-e]
   52 root     SW   [rcu_tasks_kthre]
   61 root     IW   [kworker/3:1-mm_]
   68 root     IW   [kworker/5:1-eve]
   76 root     IW   [kworker/6:1-eve]
   77 root     IW   [kworker/7:1-mm_]
   85 root     IW   [kworker/1:1-eve]
   93 root     IW   [kworker/2:1-eve]
  102 root     IW   [kworker/4:1-mm_]
  202 root     SW   [khungtaskd]
  203 root     SW   [oom_reaper]
  204 root     IW<  [writeback]
  206 root     SW   [kcompactd0]
  208 root     IW<  [kblockd]
  209 root     IW<  [devfreq_wq]
  211 root     IW   [kworker/u16:2-e]
  214 root     IW   [kworker/u16:3-e]
  224 root     IW   [kworker/u16:4-e]
  235 root     IW   [kworker/u16:5-e]
  259 root     IW   [kworker/u16:6-e]
  380 root     SW   [kswapd0]
  509 root     SW   [scsi_eh_0]
  510 root     IW<  [scsi_tmf_0]
  511 root     IW<  [ufs_clk_gating_]
  512 root     IW<  [ufs_clkscaling_]
  551 root     IW   [kworker/u16:7-e]
  635 root     IW   [kworker/u16:8-u]
  656 root     S    /sbin/klogd -n
  674 root     S    /bin/sh
  675 root     IW<  [kworker/4:1H]
  677 root     R    ps -o pid,user,stat,args

# dd if=/dev/sde of=/dev/null bs=1M & while true; do echo l > /proc/sysrq-trigger; sleep 1; done
https://pastebin.ubuntu.com/p/Sx4hXRbHby/

What information can I provide to help debug this issue?

Regards.