All of lore.kernel.org
 help / color / mirror / Atom feed
From: Youfu Zhang <zhangyoufu@gmail.com>
To: tj@kernel.org, axboe@kernel.dk
Cc: cgroups@vger.kernel.org, linux-block@vger.kernel.org
Subject: [BUG] blk-throttle panic on 32bit machine after startup
Date: Mon, 18 Oct 2021 15:08:53 +0800	[thread overview]
Message-ID: <CAEKhA2x1Qi3Ywaj9fzdsaChabqDSMe2m2441wReg_V=39_Cuhg@mail.gmail.com> (raw)

Hi,

I ran into a kernel bug related to blk-throttle on CentOS 7 AltArch for i386.
Userspace programs may panic the kernel if they hit the I/O limit
within 5 minutes after startup.

Root cause:
1. jiffies was initialized to -300HZ during boot on 32bit machines
2. enable blkio cgroup hierarchy
   __DEVEL__sane_behavior for cgroup v1 or default hierarchy for cgroup v2
   EL7 kernel modified throtl_pd_init and always enable hierarchical throttling
3. enable & trigger blkio throttling within 5 minutes after startup
   bio propagated from child tg to parent
4. enter throtl_start_new_slice_with_credit
   if(time_after_eq(start, tg->slice_start[rw]))
   aka. time_after_eq(0xFFFxxxxx, 0) does not hold
   parent tg->slice_start[rw] was zero-initialized and not updated
5. enter throtl_trim_slice
   BUG_ON(time_before(tg->slice_end[rw], tg->slice_start[rw]))
   aka. time_before(0xFFFxxxxx, 0) triggers a panic

Reproducer: (tested on Alpine Linux x86 kernel 5.10.X)
#!/bin/sh
CGROUP_PATH="$(mktemp -d)"
mount -t cgroup2 none "$CGROUP_PATH"
echo +io >"$CGROUP_PATH/cgroup.subtree_control"
mkdir "$CGROUP_PATH/child"
echo "7:0 riops=2" >"$CGROUP_PATH/child/io.max"
echo 0 >"$CGROUP_PATH/child/cgroup.procs"
echo 3 >/proc/sys/vm/drop_caches
dd if=/dev/loop0 of=/dev/null count=3

WARNING: multiple messages have this Message-ID (diff)
From: Youfu Zhang <zhangyoufu-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
To: tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org,
	axboe-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org
Cc: cgroups-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
	linux-block-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Subject: [BUG] blk-throttle panic on 32bit machine after startup
Date: Mon, 18 Oct 2021 15:08:53 +0800	[thread overview]
Message-ID: <CAEKhA2x1Qi3Ywaj9fzdsaChabqDSMe2m2441wReg_V=39_Cuhg@mail.gmail.com> (raw)

Hi,

I ran into a kernel bug related to blk-throttle on CentOS 7 AltArch for i386.
Userspace programs may panic the kernel if they hit the I/O limit
within 5 minutes after startup.

Root cause:
1. jiffies was initialized to -300HZ during boot on 32bit machines
2. enable blkio cgroup hierarchy
   __DEVEL__sane_behavior for cgroup v1 or default hierarchy for cgroup v2
   EL7 kernel modified throtl_pd_init and always enable hierarchical throttling
3. enable & trigger blkio throttling within 5 minutes after startup
   bio propagated from child tg to parent
4. enter throtl_start_new_slice_with_credit
   if(time_after_eq(start, tg->slice_start[rw]))
   aka. time_after_eq(0xFFFxxxxx, 0) does not hold
   parent tg->slice_start[rw] was zero-initialized and not updated
5. enter throtl_trim_slice
   BUG_ON(time_before(tg->slice_end[rw], tg->slice_start[rw]))
   aka. time_before(0xFFFxxxxx, 0) triggers a panic

Reproducer: (tested on Alpine Linux x86 kernel 5.10.X)
#!/bin/sh
CGROUP_PATH="$(mktemp -d)"
mount -t cgroup2 none "$CGROUP_PATH"
echo +io >"$CGROUP_PATH/cgroup.subtree_control"
mkdir "$CGROUP_PATH/child"
echo "7:0 riops=2" >"$CGROUP_PATH/child/io.max"
echo 0 >"$CGROUP_PATH/child/cgroup.procs"
echo 3 >/proc/sys/vm/drop_caches
dd if=/dev/loop0 of=/dev/null count=3

             reply	other threads:[~2021-10-18  7:09 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-10-18  7:08 Youfu Zhang [this message]
2021-10-18  7:08 ` [BUG] blk-throttle panic on 32bit machine after startup Youfu Zhang
2021-10-18 15:22 ` Liqueur Librazy
2021-10-18 15:22   ` Liqueur Librazy
2021-10-19 17:45 ` Tejun Heo
2021-10-19 17:45   ` Tejun Heo
2021-10-21  4:26   ` Youfu Zhang
2021-10-18  8:00 Haoran Luo
2021-10-18  8:00 ` Haoran Luo
2021-10-18  9:25 Haoran Luo
2021-10-18  9:25 ` Haoran Luo

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAEKhA2x1Qi3Ywaj9fzdsaChabqDSMe2m2441wReg_V=39_Cuhg@mail.gmail.com' \
    --to=zhangyoufu@gmail.com \
    --cc=axboe@kernel.dk \
    --cc=cgroups@vger.kernel.org \
    --cc=linux-block@vger.kernel.org \
    --cc=tj@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.