Re: How best to integrate dmClock QoS library into ceph codebase

["J. Eric Ivancich" <ivancich@redhat.com>]

Hi Sheng,

I’ll interleave responses below.

> On Jul 11, 2017, at 2:14 PM, sheng qiu <herbert1984106@gmail.com> wrote:
> We are trying to evaluate dmclock's effect on controlling the recovery
> traffic in order to reducing impact on client io.
> However, we are experiencing some problem and didn't get our expected results.
> 
> we setup a small cluster, with several OSD machines. In our
> configurations, we set recovery limit = 0.001 or even smaller, and
> res=0.0, wgt=1.0.
> we set client res = 20k or even higher, limit=0.0, wgt=500.

As presently implemented limits are not enforced. There is a PR that makes modification to enforce them (https://github.com/ceph/ceph/pull/16242), which I’m still evaluating. You can see some discussion of the issue (see: http://marc.info/?l=ceph-devel&m=149867479701646&w=2).

> Then we killed osd while doing fio on client side and bring it back to
> trigger recovery. We saw fio iops still reduced a lot comparable to
> not using dmclock queue. We did some debugging and saw that when
> recovery is active, fio requests enqueued much less frequent than
> before.

Are you saying that fio requests from the client slow down? I assume you’re using the fio tool. If so, what is max-jobs set to?

Also, are you saying that that fio iops had lower values with mclock compared with the weighted priority queue (“wpq”)?

> overall, seems dmclock's configuration on recovery part does not show
> any differences. Since the enqueue rate of fio requests are reduced,
> when dmclock try to dequeue a request, there's less chance to pull a
> fio request.

Theoretically, at least, with a higher reservation value, the request tags should have smaller reservation tags, which should bias mclock to dequeueing them. So I’d like to know more about your experiment.

Thank you,

Eric