Re: testing io.low limit for blk-throttle

[Paolo Valente <paolo.valente@linaro.org>]

> Il giorno 23 apr 2018, alle ore 04:19, jianchao.wang =
<jianchao.w.wang@oracle.com> ha scritto:
>=20
> Hi Paolo
>=20
> As I said, I used to meet similar scenario.
> After dome debug, I found out 3 issues.
>=20
> Here is my setup command:
> mkdir test0 test1
> echo "259:0 riops=3D150000" > test0/io.max=20
> echo "259:0 riops=3D150000" > test1/io.max=20
> echo "259:0 riops=3D150000" > test2/io.max=20
> echo "259:0 riops=3D50000 wiops=3D50000 rbps=3D209715200 =
wbps=3D209715200 idle=3D200 latency=3D10" > test0/io.low=20
> echo "259:0 riops=3D50000 wiops=3D50000 rbps=3D209715200 =
wbps=3D209715200 idle=3D200 latency=3D10" > test1/io.low=20
>=20
> My NVMe card's max bps is ~600M, and max iops is ~160k.
> Two cgroups' io.low is bps 200M and 50k. io.max is iops 150k
>=20
> 1. I setup 2 cgroups test0 and test1, one process per cgroup.
> Even if only the process in test0 does IO, its iops is just 50k.
> This is fixed by following patch.
> https://marc.info/?l=3Dlinux-block&m=3D152325457607425&w=3D2=20
>=20
> 2. Let the process in test0 and test1 both do IO.
> Sometimes, the iops of both cgroup are 50k, look at the log, =
blk-throl's upgrade always fails.
> This is fixed by following patch:
> https://marc.info/?l=3Dlinux-block&m=3D152325456307423&w=3D2
>=20
> 3. After applied patch 1 and 2, still see that one of cgroup's iops =
will fall down to 30k ~ 40k but
> blk-throl doesn't downgrade. It is due to even if the iops has been =
lower than the io.low limit for some time,
> but the cgroup is idle, so downgrade fails. More detailed, it is due =
to the code segment in throtl_tg_is_idle
>=20
>            (tg->latency_target && tg->bio_cnt &&
> 		tg->bad_bio_cnt * 5 < tg->bio_cnt)
>=20
> I fixed it with following patch.
> But I'm not sure about this patch, so I didn't submit it.
> Please also try it. :)
>=20

Thanks for sharing this fix.  I tried it too, but nothing changes in
my test :(

At this point, my doubt is still: am I getting io.low limit right?  I
understand that an I/O-bound group should be guaranteed a rbps at
least equal to the rbps set with io.low for that group (of course,
provided that the sum of io.low limits is lower than the rate at which
the device serves all the I/O generated by the groups).  Is this
really what io.low shall guarantee?

Thanks,
Paolo


> diff --git a/block/blk-throttle.c b/block/blk-throttle.c
> index b5ba845..c9a43a4 100644
> --- a/block/blk-throttle.c
> +++ b/block/blk-throttle.c
> @@ -1819,7 +1819,7 @@ static unsigned long =
tg_last_low_overflow_time(struct throtl_grp *tg)
>        return ret;
> }
>=20
> -static bool throtl_tg_is_idle(struct throtl_grp *tg)
> +static bool throtl_tg_is_idle(struct throtl_grp *tg, bool latency)
> {
>        /*
>         * cgroup is idle if:
> @@ -1836,7 +1836,7 @@ static bool throtl_tg_is_idle(struct throtl_grp =
*tg)
>              tg->idletime_threshold =3D=3D DFL_IDLE_THRESHOLD ||
>              (ktime_get_ns() >> 10) - tg->last_finish_time > time ||
>              tg->avg_idletime > tg->idletime_threshold ||
> -             (tg->latency_target && tg->bio_cnt &&
> +             (tg->latency_target && tg->bio_cnt && latency &&
>                tg->bad_bio_cnt * 5 < tg->bio_cnt);
>        throtl_log(&tg->service_queue,
>                "avg_idle=3D%ld, idle_threshold=3D%ld, bad_bio=3D%d, =
total_bio=3D%d, is_idle=3D%d, scale=3D%d",
> @@ -1867,7 +1867,7 @@ static bool throtl_tg_can_upgrade(struct =
throtl_grp *tg)
>=20
>        if (time_after_eq(jiffies,
>                tg_last_low_overflow_time(tg) + tg->td->throtl_slice) =
&&
> -           throtl_tg_is_idle(tg))
> +           throtl_tg_is_idle(tg, true))
>                return true;
>        return false;
> }
> @@ -1983,7 +1983,7 @@ static bool throtl_tg_can_downgrade(struct =
throtl_grp *tg)
>        if (time_after_eq(now, td->low_upgrade_time + td->throtl_slice) =
&&
>            time_after_eq(now, tg_last_low_overflow_time(tg) +
>                                        td->throtl_slice) &&
> -           (!throtl_tg_is_idle(tg) ||
> +           (!throtl_tg_is_idle(tg, false) ||
>             !list_empty(&tg_to_blkg(tg)->blkcg->css.children)))
>                return true;
>        return false;
>=20
>=20
>=20
> On 04/22/2018 11:53 PM, Paolo Valente wrote:
>>=20
>>=20
>>> Il giorno 22 apr 2018, alle ore 15:29, jianchao.wang =
<jianchao.w.wang@oracle.com> ha scritto:
>>>=20
>>> Hi Paolo
>>>=20
>>> I used to meet similar issue on io.low.
>>> Can you try the following patch to see whether the issue could be =
fixed.
>>> =
https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__marc.info_-3Fl-3Dli=
nux-2Dblock-26m-3D152325456307423-26w-3D2&d=3DDwIFAg&c=3DRoP1YumCXCgaWHvlZ=
YR8PZh8Bv7qIrMUB65eapI_JnE&r=3D7WdAxUBeiTUTCy8v-7zXyr4qk7sx26ATvfo6QSTvZyQ=
&m=3DasJMDy9zIe2AqRVpoLbe9RMjsdZOJZ0HrRWTM3CPZeA&s=3DAZ4kllxCfaXspjeSylBpK=
8K7ai6IPjSiffrGmzt4VEM&e=3D
>>> =
https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__marc.info_-3Fl-3Dli=
nux-2Dblock-26m-3D152325457607425-26w-3D2&d=3DDwIFAg&c=3DRoP1YumCXCgaWHvlZ=
YR8PZh8Bv7qIrMUB65eapI_JnE&r=3D7WdAxUBeiTUTCy8v-7zXyr4qk7sx26ATvfo6QSTvZyQ=
&m=3DasJMDy9zIe2AqRVpoLbe9RMjsdZOJZ0HrRWTM3CPZeA&s=3D1EhsoSMte3kIxuVSYBFSE=
9W2jRKrIWI5z7-stlZ80H4&e=3D
>>>=20
>>=20
>> Just tried. Unfortunately, nothing seems to change :(
>>=20
>> Thanks,
>> Paolo
>>=20
>>> Thanks
>>> Jianchao
>>>=20
>>> On 04/22/2018 05:23 PM, Paolo Valente wrote:
>>>> Hi Shaohua, all,
>>>> at last, I started testing your io.low limit for blk-throttle.  One =
of
>>>> the things I'm interested in is how good throttling is in achieving =
a
>>>> high throughput in the presence of realistic, variable workloads.
>>>>=20
>>>> However, I seem to have bumped into a totally different problem.  =
The
>>>> io.low parameter doesn't seem to guarantee what I understand it is =
meant
>>>> to guarantee: minimum per-group bandwidths.  For example, with
>>>> - one group, the interfered, containing one process that does =
sequential
>>>> reads with fio
>>>> - io.low set to 100MB/s for the interfered
>>>> - six other groups, the interferers, with each interferer =
containing one
>>>> process doing sequential read with fio
>>>> - io.low set to 10MB/s for each interferer
>>>> - the workload executed on an SSD, with a 500MB/s of overall =
throughput
>>>> the interfered gets only 75MB/s.
>>>>=20
>>>> In particular, the throughput of the interfered becomes lower and
>>>> lower as the number of interferers is increased.  So you can make =
it
>>>> become even much lower than the 75MB/s in the example above.  There
>>>> seems to be no control on bandwidth.
>>>>=20
>>>> Am I doing something wrong?  Or did I simply misunderstand the goal =
of
>>>> io.low, and the only parameter for guaranteeing the desired =
bandwidth to
>>>> a group is io.max (to be used indirectly, by limiting the bandwidth =
of
>>>> the interferers)?
>>>>=20
>>>> If useful for you, you can reproduce the above test very quickly, =
by
>>>> using the S suite [1] and typing:
>>>>=20
>>>> cd thr-lat-with-interference
>>>> sudo ./thr-lat-with-interference.sh -b t -w 100000000 -W "10000000 =
10000000 10000000 10000000 10000000 10000000" -n 6 -T "read read read =
read read read" -R "0 0 0 0 0 0"
>>>>=20
>>>> Looking forward to your feedback,
>>>> Paolo
>>>>=20
>>>> [1]=20
>>>>=20
>>=20
>>=20