Re: [regression/bisected] 4.19 cycle boot time IO stalls

[Jens Axboe <axboe@kernel.dk>]

On 9/5/18 2:55 AM, Mike Galbraith wrote:
> Greetings,
> 
> I've been seeing $subject, decided to take the time to try to bisect
> the little bugger.  The hangs are not 100% repeatable, and while
> bisection with a 5 boot go/nogo threshold seemed to go smoothly, it
> ended up fingering a merge commit (sigh).
> 
> Box has an SSD (unused only by windows 10 box came with) and 3 spinning
> rust buckets that I normally use with BFQ via a udev rule, but CFQ does
> the same, so scheduler is seemingly irrelevant.  However, in 7 crash
> dumps, all of which look about like the data below the bisect log,
> there is something relevant, namely the hung 'tlp' task (powersaving
> script of some sort for laptops according to the package description). 
> That knob twiddling script is present/hung in all, making me a tad
> suspicious, and indeed, testing with the final (bad) kernel, all I have
> to do to eliminate hangs is to remove the 'tlp' package.  Verified via
> remove, 5 boots work fine, reinstall, 2 of 5 hang, remove again, 10 of
> 10 work fine, reinstall, 2 in a row hang.
> 
> Seems pretty certain that tlp script is what inspires bug to raise its
> ugly head.  WRT bisection result itself, munged merge seems far less
> likely than a false negative having knocked bisection off course.
> 
> 72f02ba66bd83b54054da20eae550123de84da6f is the first bad commit

I bet it's the host busy change from Ming, which I already
reported as being the culprit for another test failure I had. For
some reason it's not merged yet, nudge nudge Martin. You can test
by reverting:

commit 328728630d9f2bf14b82ca30b5e47489beefe361
Author: Ming Lei <ming.lei@redhat.com>
Date:   Sun Jun 24 22:03:27 2018 +0800

    scsi: core: avoid host-wide host_busy counter for scsi_mq

which was slightly modified by 265d59aacbce, so you'll want to
yank that one first.

BTW, that suse email for me hasn't worked in 12 years :-)

-- 
Jens Axboe