Linux-Block Archive on lore.kernel.org
 help / color / Atom feed
From: Weiping Zhang <zwp10758@gmail.com>
To: Tejun Heo <tj@kernel.org>
Cc: Jens Axboe <axboe@kernel.dk>,
	linux-block@vger.kernel.org, cgroups@vger.kernel.org
Subject: Re: [RFC 0/3] blkcg: add blk-iotrack
Date: Thu, 26 Mar 2020 23:08:45 +0800
Message-ID: <CAA70yB4e65nbV=ZA8OT-SUkq+ZQOGGB9e-3QKJ_PqXjVaXGvFA@mail.gmail.com> (raw)
In-Reply-To: <CAA70yB5yH9H6-gaKfRSTmgd6vvzP4T9N7v-NAD0MsRL+YTexHw@mail.gmail.com>

Weiping Zhang <zwp10758@gmail.com> 于2020年3月26日周四 上午12:45写道:
>
> Tejun Heo <tj@kernel.org> 于2020年3月25日周三 下午10:12写道:
> >
> > On Wed, Mar 25, 2020 at 08:49:24PM +0800, Weiping Zhang wrote:
> > > For this patchset, iotrack can work well, I'm using it to monitor
> > > block cgroup for
> > > selecting a proper io isolation policy.
> >
> > Yeah, I get that but monitoring needs tend to diverge quite a bit
> > depending on the use cases making detailed monitoring often need fair
> > bit of flexibility, so I'm a bit skeptical about adding a fixed
> > controller for that. I think a better approach may be implementing
> > features which can make dynamic monitoring whether that's through bpf,
> > drgn or plain tracepoints easier and more efficient.
> >
> I agree with you, there are lots of io metrics needed in the real
> production system.
> The more flexible way is export all bio structure members of bio’s whole life to
> the userspace without re-compiling kernel, like what bpf did.
>
> Now the main block cgroup isolation policy:
>  blk-iocost and bfq are weght based, blk-iolatency is latency based.
> The blk-iotrack can track the real percentage for IOs,kB,on disk time(d2c),
> and total time, it’s a good indicator to the real weight. For blk-iolatency, the
> blk-iotrack has 8 lantency thresholds to show latency distribution, so if we
> change these thresholds around to blk-iolateny.target.latency, we can tune
> the target latency to a more proper value.
>
> blk-iotrack extends the basic io.stat. It just export the important
> basic io statistics
> for cgroup,like what /prof/diskstats for block device. And it’s easy
> programming,
> iotrack just working like iostat, but focus on cgroup.
>
> blk-iotrack is friendly with these block cgroup isolation policies, a
> indicator for
> cgroup weight and lantency.
>

Hi Tejun,

I do a testing at cgroup v2 and monitoring them by iotrack,
then I compare the fio's output and iotrack's, they can match well.

cgroup weight test:
/sys/fs/cgroup/test1
/sys/fs/cgroup/test2
test1.weight : test2.weight = 8 : 1

Now I just run randread-4K fio test by these three policy:
iocost, bfq, and nvme-wrr
For blk-iocost I run "iocost_coef_gen.py" and set result to the "io.cost.model".
259:0 ctrl=user model=linear rbps=3286476297 rseqiops=547837
rrandiops=793881 wbps=2001272356 wseqiops=482243 wrandiops=483037

But iocost_test1 cannot get 8/(8+1) iops,  and the total disk iops
is 737559 < 79388, even I change rrandiops=637000.

test case     bw         iops       rd_avg_lat  rd_p99_lat
==========================================================
iocost_test1  1550478    387619     659.76      1662.00
iocost_test2  1399761    349940     730.83      1712.00
wrr_test1     2618185    654546     390.59      1187.00
wrr_test2     362613     90653      2822.62     4358.00
bfq_test1     714127     178531     1432.43     489.00
bfq_test2     178821     44705      5721.76     552.00


The detail test report can be found at:
https://github.com/dublio/iotrack/wiki/cgroup-io-weight-test


> Thanks

  reply index

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-03-21  1:20 Weiping Zhang
2020-03-21  1:21 ` [RFC 1/3] update the real issue size when bio_split Weiping Zhang
2020-03-21  1:21 ` [RFC 2/3] bio: track timestamp of submitting bio the disk driver Weiping Zhang
2020-03-21  1:21 ` [RFC 3/3] blkcg: add blk-iotrack Weiping Zhang
2020-03-24 18:27 ` [RFC 0/3] " Tejun Heo
2020-03-25 12:49   ` Weiping Zhang
2020-03-25 14:12     ` Tejun Heo
2020-03-25 16:45       ` Weiping Zhang
2020-03-26 15:08         ` Weiping Zhang [this message]
2020-03-26 16:14           ` Tejun Heo
2020-03-26 16:27             ` Weiping Zhang
2020-03-31 14:19               ` Tejun Heo
2020-03-27  6:27 ` [RFC PATCH v2 " Weiping Zhang

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAA70yB4e65nbV=ZA8OT-SUkq+ZQOGGB9e-3QKJ_PqXjVaXGvFA@mail.gmail.com' \
    --to=zwp10758@gmail.com \
    --cc=axboe@kernel.dk \
    --cc=cgroups@vger.kernel.org \
    --cc=linux-block@vger.kernel.org \
    --cc=tj@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-Block Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-block/0 linux-block/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-block linux-block/ https://lore.kernel.org/linux-block \
		linux-block@vger.kernel.org
	public-inbox-index linux-block

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-block


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git