Get IO request execution profile

Get IO request execution profile

[Konstantin Danilov]

Hi all,

is there a way to get something like 'dump historic ops' for particular request?
Probably from client side. Ideally I'd like to send a normal IO
request with special
flag and get execution profile in response metadata.
If it's not possible right now - is this an interesting feature to
implement, or I miss
something and it's useless?

Thanks

-- 
Kostiantyn Danilov aka koder.ua
Principal software engineer, Mirantis

skype:koder.ua
http://koder-ua.blogspot.com/
http://mirantis.com

Re: Get IO request execution profile

[John Spray]

On Mon, Jan 23, 2017 at 12:36 PM, Konstantin Danilov
<kdanilov@mirantis.com> wrote:
> Hi all,
>
> is there a way to get something like 'dump historic ops' for particular request?
> Probably from client side. Ideally I'd like to send a normal IO
> request with special
> flag and get execution profile in response metadata.
> If it's not possible right now - is this an interesting feature to
> implement, or I miss
> something and it's useless?

Potentially interesting idea, although it's not obvious to me why it's
useful to have this dumped on the client rather than the server?
Would be it equally useful to be able to tag a specific client on the
server side to have its requests dumped like this?

John

>
> Thanks
>
> --
> Kostiantyn Danilov aka koder.ua
> Principal software engineer, Mirantis
>
> skype:koder.ua
> http://koder-ua.blogspot.com/
> http://mirantis.com
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Get IO request execution profile

[Jason Dillaman]

Have you looked at the Zipkin tracking PR [1]?

[1] https://github.com/ceph/ceph/pull/11053

On Mon, Jan 23, 2017 at 7:54 AM, John Spray <jspray@redhat.com> wrote:
> On Mon, Jan 23, 2017 at 12:36 PM, Konstantin Danilov
> <kdanilov@mirantis.com> wrote:
>> Hi all,
>>
>> is there a way to get something like 'dump historic ops' for particular request?
>> Probably from client side. Ideally I'd like to send a normal IO
>> request with special
>> flag and get execution profile in response metadata.
>> If it's not possible right now - is this an interesting feature to
>> implement, or I miss
>> something and it's useless?
>
> Potentially interesting idea, although it's not obvious to me why it's
> useful to have this dumped on the client rather than the server?
> Would be it equally useful to be able to tag a specific client on the
> server side to have its requests dumped like this?
>
> John
>
>>
>> Thanks
>>
>> --
>> Kostiantyn Danilov aka koder.ua
>> Principal software engineer, Mirantis
>>
>> skype:koder.ua
>> http://koder-ua.blogspot.com/
>> http://mirantis.com
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Jason

Re: Get IO request execution profile

[Konstantin Danilov]

> Potentially interesting idea, although it's not obvious to me why it's
useful to have this dumped on the client rather than the server?

The idea is to monitor requests execution time per stage.
dump_historic_ops covers large part of this idea, meanwhile in case of
large cluster
it hard follow particular request. You have to calculate primary OSD
for it. Historic
store only slowest requests. To get a timing profile historic ops buffer have to
increased to fit all requests.

So if you like to get a requests execution profile(and potentially
monitor how it changes over time)
you can very easily get it for slowest requests, but for all kind of
requests it requires tricks and
put unnecessary additional load on OSD nodes.

> Have you looked at the Zipkin tracking PR [1]?
Nope. Is there any plans to merge it?

Thanks

On Mon, Jan 23, 2017 at 3:59 PM, Jason Dillaman <jdillama@redhat.com> wrote:
> Have you looked at the Zipkin tracking PR [1]?
>
> [1] https://github.com/ceph/ceph/pull/11053
>
> On Mon, Jan 23, 2017 at 7:54 AM, John Spray <jspray@redhat.com> wrote:
>> On Mon, Jan 23, 2017 at 12:36 PM, Konstantin Danilov
>> <kdanilov@mirantis.com> wrote:
>>> Hi all,
>>>
>>> is there a way to get something like 'dump historic ops' for particular request?
>>> Probably from client side. Ideally I'd like to send a normal IO
>>> request with special
>>> flag and get execution profile in response metadata.
>>> If it's not possible right now - is this an interesting feature to
>>> implement, or I miss
>>> something and it's useless?
>>
>> Potentially interesting idea, although it's not obvious to me why it's
>> useful to have this dumped on the client rather than the server?
>> Would be it equally useful to be able to tag a specific client on the
>> server side to have its requests dumped like this?
>>
>> John
>>
>>>
>>> Thanks
>>>
>>> --
>>> Kostiantyn Danilov aka koder.ua
>>> Principal software engineer, Mirantis
>>>
>>> skype:koder.ua
>>> http://koder-ua.blogspot.com/
>>> http://mirantis.com
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
> --
> Jason



-- 
Kostiantyn Danilov aka koder.ua
Principal software engineer, Mirantis

skype:koder.ua
http://koder-ua.blogspot.com/
http://mirantis.com

Re: Get IO request execution profile

[Jason Dillaman]

On Mon, Jan 23, 2017 at 10:59 AM, Konstantin Danilov
<kdanilov@mirantis.com> wrote:
>> Have you looked at the Zipkin tracking PR [1]?
> Nope. Is there any plans to merge it?

Definitely! It was delayed due to Kraken feature freeze, so now that
Kraken is out I'd expect it to pick up steam again and hopefully land
in Luminous.


-- 
Jason

Re: Get IO request execution profile

[Gregory Farnum]

On Mon, Jan 23, 2017 at 4:54 AM, John Spray <jspray@redhat.com> wrote:
> On Mon, Jan 23, 2017 at 12:36 PM, Konstantin Danilov
> <kdanilov@mirantis.com> wrote:
>> Hi all,
>>
>> is there a way to get something like 'dump historic ops' for particular request?
>> Probably from client side. Ideally I'd like to send a normal IO
>> request with special
>> flag and get execution profile in response metadata.
>> If it's not possible right now - is this an interesting feature to
>> implement, or I miss
>> something and it's useless?
>
> Potentially interesting idea, although it's not obvious to me why it's
> useful to have this dumped on the client rather than the server?
> Would be it equally useful to be able to tag a specific client on the
> server side to have its requests dumped like this?

We don't have very good client-side op tracking yet (at least, not the
same way we have the OpTracker server-side), but if we did that would
let us identify from a client if server issues were making it very
slow. Could be quite useful!
-Greg

>
> John
>
>>
>> Thanks
>>
>> --
>> Kostiantyn Danilov aka koder.ua
>> Principal software engineer, Mirantis
>>
>> skype:koder.ua
>> http://koder-ua.blogspot.com/
>> http://mirantis.com
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html