From: Dmitrii Tcvetkov <demfloro@demfloro.ru>
To: Paolo Valente <paolo.valente@linaro.org>
Cc: Jens Axboe <axboe@kernel.dk>,
linux-block <linux-block@vger.kernel.org>,
linux-kernel@vger.kernel.org
Subject: Re: Bisected GFP in bfq_bfqq_expire on v5.1-rc1
Date: Thu, 4 Apr 2019 22:22:57 +0300 [thread overview]
Message-ID: <20190404222257.0cfb1130@fire.localdomain> (raw)
In-Reply-To: <EBC090F1-72C0-4794-91CC-37F9A2A1DEFB@linaro.org>
[-- Attachment #1: Type: text/plain, Size: 1703 bytes --]
On Mon, 1 Apr 2019 12:35:11 +0200
Paolo Valente <paolo.valente@linaro.org> wrote:
>
>
> > Il giorno 1 apr 2019, alle ore 11:22, Dmitrii Tcvetkov
> > <demfloro@demfloro.ru> ha scritto:
> >
> > On Mon, 1 Apr 2019 11:01:27 +0200
> > Paolo Valente <paolo.valente@linaro.org> wrote:
> >> Ok, thank you. Could you please do a
> >>
> >> list *(bfq_bfqq_expire+0x1f3)
> >>
> >> for me?
> >>
> >> Thanks,
> >> Paolo
> >>
> >>>
> >>> <gpf.txt><gpf-w-bfq-group-iosched.txt><config.txt>
> >
> > Reading symbols from vmlinux...done.
> > (gdb) list *(bfq_bfqq_expire+0x1f3)
> > 0xffffffff813d02c3 is in bfq_bfqq_expire (block/bfq-iosched.c:3390).
> > 3385 * even in case bfqq and thus parent entities go on
> > receiving 3386 * service with the same budget.
> > 3387 */
> > 3388 entity = entity->parent;
> > 3389 for_each_entity(entity)
> > 3390 entity->service = 0;
> > 3391 }
> > 3392
> > 3393 /*
> > 3394 * Budget timeout is not implemented through a dedicated
> > timer, but
>
> Thank you very much. Unfortunately this doesn't ring any bell. I'm
> trying to reproduce the failure. It will probably take a little
> time. If I don't make it, I'll ask you to kindly retry after applying
> some instrumentation patch.
>
I looked at what git is doing just before panic and it's doing a lot of
lstat() syscalls on working tree.
I've attached a python script which reproduces the crash in about
10 seconds after it prepares testdir, git checkout origin/linux-5.0.y
reproduces it in about 2 seconds. I have to use multiprocessing Pool as
I couldn't reproduce the crash using ThreadPool, probably due to Python
GIL.
[-- Attachment #2: crash.py --]
[-- Type: text/plain, Size: 1555 bytes --]
#!/usr/bin/env python3
from glob import glob
from os import lstat,mkdir
from random import randint
from os.path import isdir,exists
from pathlib import Path
from time import sleep
from subprocess import run
from multiprocessing import Pool
def drop_caches():
with open('/proc/sys/vm/drop_caches','w') as f:
f.write('3')
def enable_bfq():
with open('/sys/block/sda/queue/scheduler','w') as f:
f.write('bfq')
def sync():
run(('sync'))
def prepare_tree(name):
def populate(dir, depth=6):
if not depth:
return
for fname in range(1,20):
if randint(0,100) > 80:
dirname = "{}{}/".format(dir,fname)
mkdir(dirname)
populate(dirname, depth - 1)
continue
fname = "{}{}".format(dir, fname)
Path(fname).touch(exist_ok=True)
if not isdir(name):
mkdir(name)
if not name.endswith('/'):
name = '{}/'.format(name)
populate(name)
def traverse(dir):
drop_caches()
for inode in glob("{}/*".format(dir)):
if isdir(inode):
traverse(inode)
else:
lstat(inode)
if randint(0,10) > 6:
sleep(0)
def main():
nproc = 16
dirname = 'testdir'
if not exists(dirname):
prepare_tree(dirname)
sync()
drop_caches()
enable_bfq()
drop_caches()
with Pool(nproc) as pool:
dirs = (dirname,) * nproc
pool.map(traverse,dirs)
main()
next prev parent reply other threads:[~2019-04-04 19:23 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-03-29 13:02 Bisected GFP in bfq_bfqq_expire on v5.1-rc1 Dmitrii Tcvetkov
2019-03-29 14:10 ` Jens Axboe
2019-04-01 7:29 ` Paolo Valente
2019-04-01 8:55 ` Dmitrii Tcvetkov
2019-04-01 9:01 ` Paolo Valente
2019-04-01 9:22 ` Dmitrii Tcvetkov
2019-04-01 10:35 ` Paolo Valente
2019-04-04 19:22 ` Dmitrii Tcvetkov [this message]
2019-04-09 9:55 ` Paolo Valente
2019-04-09 16:14 ` Dmitrii Tcvetkov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190404222257.0cfb1130@fire.localdomain \
--to=demfloro@demfloro.ru \
--cc=axboe@kernel.dk \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=paolo.valente@linaro.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.