Merge ftrace data of host and guest

Merge ftrace data of host and guest

[Divya Vyas]

Hi,

I am adding trace data of host and guest using trace-merge script .

https://15351300108708232970.googlegroups.com/attach/66533d7b41eed0b/trace-merge.pl?part=0.1&view=1&vt=ANaJVrHKDEwmNeLGDINH0B5Ud4OIUrfQ2_2w2jo08nJujH7Oc9facKD3ckT2RbCiHkRAfnYxmZnBF7qi0SeMZr2IMsPBDNjgrYkblCRkmgSk2PryLVsGe5g

guest data

qemu-kvm-2687  [003] d...50550079203669: kvm_exit: [detail]
qemu-kvm-2687  [003] d...50550079206816: kvm_entry: [detail]
qemu-kvm-2687  [003] d...50550079240656: kvm_exit: [detail]
qemu-kvm-2687  [003] d...50550079243467: kvm_entry: [detail]
qemu-kvm-2687  [003] d...50550079256103: kvm_exit: [detail]
qemu-kvm-2687  [003] d...50550079268391: kvm_entry: [detail]
qemu-kvm-2687  [003] d...50550079280829: kvm_exit: [detail]
qemu-kvm-2687  [003] d...50550079286028: kvm_entry: [detail]

host data

comm-3826  [000] d...49836825726903: sched_wakeup: [detail]
comm-3826  [000] d...49836832225344: sched_switch: [detail]
~


Data after merging :

/opt/trace-merge/trace-merge 18446739773557710924 host.data guest.data

hcomm-3826  [000] d...49836825726903: sched_wakeup: [detail]
hcomm-3826  [000] d...49836832225344: sched_switch: [detail]
gqemu-kvm-2687  [003] d...54850231044361: kvm_exit: [detail]
gqemu-kvm-2687  [003] d...54850231047508: kvm_entry: [detail]
gqemu-kvm-2687  [003] d...54850231081348: kvm_exit: [detail]
gqemu-kvm-2687  [003] d...54850231084159: kvm_entry: [detail]
gqemu-kvm-2687  [003] d...54850231096795: kvm_exit: [detail]
gqemu-kvm-2687  [003] d...54850231109083: kvm_entry: [detail]
gqemu-kvm-2687  [003] d...54850231121521: kvm_exit: [detail]
gqemu-kvm-2687  [003] d...54850231126720: kvm_entry: [detail]

I think it is not desired output . Ths script is written for diffrent
format .ftrace format is ( trace-cmd report command generates) :

version = 6
CPU 0 is empty
cpus=2
          <idle>-0     [001]  4401.096833: netif_receive_skb:
dev=eth0 skbaddr=0xffff88001ed8fc00 len=84
          <idle>-0     [001]  4401.096899: net_dev_queue:
dev=eth0 skbaddr=0xffff88001a9fe000 len=98
          <idle>-0     [001]  4401.096935: net_dev_xmit:
dev=eth0 skbaddr=0xffff88001a9fe000 len=98 rc=0
          <idle>-0     [001]  4401.097277: netif_receive_skb:
dev=eth0 skbaddr=0xffff88001a9fe800 len=84
          <idle>-0     [001]  4401.097309: net_dev_queue:
dev=eth0 skbaddr=0xffff88001a9fea00 len=98
          <idle>-0     [001]  4401.097325: net_dev_xmit:
dev=eth0 skbaddr=0xffff88001a9fea00 len=98 rc=0
          <idle>-0     [001]  4401.097570: netif_receive_skb:
dev=eth0 skbaddr=0xffff88001a9fe800 len=84
          <idle>-0     [001]  4401.097592: net_dev_queue:
dev=eth0 skbaddr=0xffff88001a9fe000 len=98
          <idle>-0     [001]  4401.097607: net_dev_xmit:
dev=eth0 skbaddr=0xffff88001a9fe000 len=98 rc=0
          <idle>-0     [001]  4401.097830: netif_receive_skb:
dev=eth0 skbaddr=0xffff88001a9fe800 len=84
          <idle>-0     [001]  4401.097843: net_dev_queue:
dev=eth0 skbaddr=0xffff88001a9fea00 len=98


Can you let me know I there is any other method to merge the traces or
any further changes in the script

Re: Merge ftrace data of host and guest

[Steven Rostedt]

On Fri, Dec 19, 2014 at 05:52:52PM +0530, Divya Vyas wrote:
> Hi,

Hi, it would be  helpful if you Cc'd me. I would have responded quicker. Luckily,
someone noticed this email and told me about it.

> 
> I think it is not desired output . Ths script is written for diffrent
> format .ftrace format is ( trace-cmd report command generates) :

I'm actually working on trace-cmd this week and hopefully I'll be able to release
2.5 by x-mas.

I believe that Masami is going to start continuing work on the virt guest
tracer for trace-cmd.

Also, for now, for a crud way of merging, if both the gettimeofday clocks
of the host and guest are synced, you could do:

guest> trace-cmd record -o guest.dat --date  ...

host> trace-cmd record -o host.dat --date ...

and then run:

trace-cmd report -i guest.dat -i host.dat

and trace-cmd will merge the two. The --date will add a way to sync the trace
timestamps to a wall clock and that will be used to try to interleave the
events of both the data files.

-- Steve

Re: Merge ftrace data of host and guest

[Divya Vyas]

Hi Steven,

I am sorry but I am very new to trace-cmd , that's why I didn't knew
what you are working on it.

I will explain you whole scenario , Actually I have multiple guests
and a host and I want to see all traces on a single file  with
trace-cmd report command and may be kernelshark also .

trace-cmd report -i guest.dat -i host.dat command is giving me all
traces is single file but the timestamp is different for guest . So
the challenge for me is to sync the timestamp of guest and host .  Can
you advise me something in this ? How can I sync gettimeofday clocks
of guest and host ?

and kernelshark doesnt take trace-cmd report input , I think it takes
guest.dat and host.dat .
Is there any other viewer available for ftrace ?






On Sat, Dec 20, 2014 at 10:40 AM, Steven Rostedt <rostedt@goodmis.org> wrote:
> On Fri, Dec 19, 2014 at 05:52:52PM +0530, Divya Vyas wrote:
>> Hi,
>
> Hi, it would be  helpful if you Cc'd me. I would have responded quicker. Luckily,
> someone noticed this email and told me about it.
>
>>
>> I think it is not desired output . Ths script is written for diffrent
>> format .ftrace format is ( trace-cmd report command generates) :
>
> I'm actually working on trace-cmd this week and hopefully I'll be able to release
> 2.5 by x-mas.
>
> I believe that Masami is going to start continuing work on the virt guest
> tracer for trace-cmd.
>
> Also, for now, for a crud way of merging, if both the gettimeofday clocks
> of the host and guest are synced, you could do:
>
> guest> trace-cmd record -o guest.dat --date  ...
>
> host> trace-cmd record -o host.dat --date ...
>
> and then run:
>
> trace-cmd report -i guest.dat -i host.dat
>
> and trace-cmd will merge the two. The --date will add a way to sync the trace
> timestamps to a wall clock and that will be used to try to interleave the
> events of both the data files.
>
> -- Steve
>

Re: Merge ftrace data of host and guest

[Divya Vyas]

Hi ,

One more thing I want to add that Yoshihiro YUNOMAE's  trace-merge
script does this time synchronization. We have to calculate guest
offset and then subtract with host time .

But it is not applying on current trace-cmd report output format and
not giving desired output .

Thanks,
Divya Vyas

On Mon, Dec 22, 2014 at 12:34 PM, Divya Vyas <edivya.vyas@gmail.com> wrote:
> Hi Steven,
>
> I am sorry but I am very new to trace-cmd , that's why I didn't knew
> what you are working on it.
>
> I will explain you whole scenario , Actually I have multiple guests
> and a host and I want to see all traces on a single file  with
> trace-cmd report command and may be kernelshark also .
>
> trace-cmd report -i guest.dat -i host.dat command is giving me all
> traces is single file but the timestamp is different for guest . So
> the challenge for me is to sync the timestamp of guest and host .  Can
> you advise me something in this ? How can I sync gettimeofday clocks
> of guest and host ?
>
> and kernelshark doesnt take trace-cmd report input , I think it takes
> guest.dat and host.dat .
> Is there any other viewer available for ftrace ?
>
>
>
>
>
>
> On Sat, Dec 20, 2014 at 10:40 AM, Steven Rostedt <rostedt@goodmis.org> wrote:
>> On Fri, Dec 19, 2014 at 05:52:52PM +0530, Divya Vyas wrote:
>>> Hi,
>>
>> Hi, it would be  helpful if you Cc'd me. I would have responded quicker. Luckily,
>> someone noticed this email and told me about it.
>>
>>>
>>> I think it is not desired output . Ths script is written for diffrent
>>> format .ftrace format is ( trace-cmd report command generates) :
>>
>> I'm actually working on trace-cmd this week and hopefully I'll be able to release
>> 2.5 by x-mas.
>>
>> I believe that Masami is going to start continuing work on the virt guest
>> tracer for trace-cmd.
>>
>> Also, for now, for a crud way of merging, if both the gettimeofday clocks
>> of the host and guest are synced, you could do:
>>
>> guest> trace-cmd record -o guest.dat --date  ...
>>
>> host> trace-cmd record -o host.dat --date ...
>>
>> and then run:
>>
>> trace-cmd report -i guest.dat -i host.dat
>>
>> and trace-cmd will merge the two. The --date will add a way to sync the trace
>> timestamps to a wall clock and that will be used to try to interleave the
>> events of both the data files.
>>
>> -- Steve
>>

Re: Merge ftrace data of host and guest

[Masami Hiramatsu]

Hi,

I'm working on trace-cmd virt-server feature. This is just a basic feature
to allow trace-cmd using virtio-trace (no time synchronizing).
I'll send the updated version of the series soon.

(2014/12/22 16:07), Divya Vyas wrote:
> Hi ,
> 
> One more thing I want to add that Yoshihiro YUNOMAE's  trace-merge
> script does this time synchronization. We have to calculate guest
> offset and then subtract with host time .

Right, to complete this work we need trace-merge like merging feature.
I assume that we can do that by passing the offset of the timestamp
information with guest trace data (that is the simplest implementation).

However, currently trace-merge depends on x86 TSC, but some intel
processors TSC is not stable. That is current barrier of trace-merge.

Thank you,


> 
> But it is not applying on current trace-cmd report output format and
> not giving desired output .
> 
> Thanks,
> Divya Vyas
> 
> On Mon, Dec 22, 2014 at 12:34 PM, Divya Vyas <edivya.vyas@gmail.com> wrote:
>> Hi Steven,
>>
>> I am sorry but I am very new to trace-cmd , that's why I didn't knew
>> what you are working on it.
>>
>> I will explain you whole scenario , Actually I have multiple guests
>> and a host and I want to see all traces on a single file  with
>> trace-cmd report command and may be kernelshark also .
>>
>> trace-cmd report -i guest.dat -i host.dat command is giving me all
>> traces is single file but the timestamp is different for guest . So
>> the challenge for me is to sync the timestamp of guest and host .  Can
>> you advise me something in this ? How can I sync gettimeofday clocks
>> of guest and host ?
>>
>> and kernelshark doesnt take trace-cmd report input , I think it takes
>> guest.dat and host.dat .
>> Is there any other viewer available for ftrace ?
>>
>>
>>
>>
>>
>>
>> On Sat, Dec 20, 2014 at 10:40 AM, Steven Rostedt <rostedt@goodmis.org> wrote:
>>> On Fri, Dec 19, 2014 at 05:52:52PM +0530, Divya Vyas wrote:
>>>> Hi,
>>>
>>> Hi, it would be  helpful if you Cc'd me. I would have responded quicker. Luckily,
>>> someone noticed this email and told me about it.
>>>
>>>>
>>>> I think it is not desired output . Ths script is written for diffrent
>>>> format .ftrace format is ( trace-cmd report command generates) :
>>>
>>> I'm actually working on trace-cmd this week and hopefully I'll be able to release
>>> 2.5 by x-mas.
>>>
>>> I believe that Masami is going to start continuing work on the virt guest
>>> tracer for trace-cmd.
>>>
>>> Also, for now, for a crud way of merging, if both the gettimeofday clocks
>>> of the host and guest are synced, you could do:
>>>
>>> guest> trace-cmd record -o guest.dat --date  ...
>>>
>>> host> trace-cmd record -o host.dat --date ...
>>>
>>> and then run:
>>>
>>> trace-cmd report -i guest.dat -i host.dat
>>>
>>> and trace-cmd will merge the two. The --date will add a way to sync the trace
>>> timestamps to a wall clock and that will be used to try to interleave the
>>> events of both the data files.
>>>
>>> -- Steve
>>>
> 


-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: Merge ftrace data of host and guest

[Divya Vyas]

Hi,

I am working on merge of ftrace data of guest and host .

tsc_guest = tsc_host +offset

I got offset from /sys/kernel/debug/tracing/instances/tsc_offset/trace
 18446739224709377060

Where as my guest and host data is in below format :

 sshd-2332  [000]  4813.363939: net_dev_queue:        dev=eth0
skbaddr=0xffff88003b78f6e8 len=134
 sshd-2332  [000]  4813.363951: net_dev_xmit:         dev=eth0
skbaddr=0xffff88003b78f6e8 len=134 rc=0
 sshd-2332  [000]  4813.364010: netif_receive_skb:    dev=eth0
skbaddr=0xffff880015cbbc00 len=52
 sshd-2332  [000]  4813.364042: net_dev_queue:        dev=eth0
skbaddr=0xffff88002c086ce8 len=102
 sshd-2332  [000]  4813.364053: net_dev_xmit:         dev=eth0
skbaddr=0xffff88002c086ce8 len=102 rc=0
 trace-cmd-2496  [000]  4813.364098: netif_receive_skb:    dev=eth0
skbaddr=0xffff880015cbbc00 len=52
 <idle>-0     [000]  4814.384570: netif_receive_skb:    dev=eth0
skbaddr=0xffff88003a490a00 len=38
 <idle>-0     [000]  4816.384541: netif_receive_skb:    dev=eth0
skbaddr=0xffff88003a490900 len=38
 <idle>-0     [000]  4818.384681: netif_receive_skb:    dev=eth0
skbaddr=0xffff88003a490900 len=38

How can I add the timestamp and offset .Do I need to add directly the numbers ?



On Tue, Jan 13, 2015 at 11:03 AM, Divya Vyas <edivya.vyas@gmail.com> wrote:
> Hi,
>
> Is there any feature in ftrace like synchronizing the timestamp
>
> http://help.eclipse.org/luna/index.jsp?topic=%2Forg.eclipse.linuxtools.lttng.help%2Fdoc%2FTrace-synchronization.html
>
>
>
> On Wed, Jan 7, 2015 at 2:57 PM, Divya Vyas <edivya.vyas@gmail.com> wrote:
>> Hi Masami,
>>
>> Any updates on the merge feature?
>>
>>
>>
>> On Mon, Dec 22, 2014 at 4:36 PM, Divya Vyas <edivya.vyas@gmail.com> wrote:
>>> Hi Masami,
>>>
>>> Thanks for the patches .
>>>
>>> Are you working on this merging feature also
>>> https://lkml.org/lkml/2013/12/16/688 ? the d option ?
>>>
>>> trace-cmd report -i will not give expected output if the time is not
>>> synchronized . From kernel side we can get TSC offset  using
>>> tracepoint when guest boots . I think this tracepoint was added from
>>> 3.11  but from user side I got only the trace-merge script which takes
>>> offset and trace and host data .  But the format of trace data this
>>> script accepts is little different .
>>>
>>> My end goal is to to merge the traces in chronological order and see
>>> them in kernelshark .
>>>
>>>
>>> Thanks,
>>> Divya
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Dec 22, 2014 at 2:03 PM, Masami Hiramatsu
>>> <masami.hiramatsu.pt@hitachi.com> wrote:
>>>> Hi,
>>>>
>>>> I'm working on trace-cmd virt-server feature. This is just a basic feature
>>>> to allow trace-cmd using virtio-trace (no time synchronizing).
>>>> I'll send the updated version of the series soon.
>>>>
>>>> (2014/12/22 16:07), Divya Vyas wrote:
>>>>> Hi ,
>>>>>
>>>>> One more thing I want to add that Yoshihiro YUNOMAE's  trace-merge
>>>>> script does this time synchronization. We have to calculate guest
>>>>> offset and then subtract with host time .
>>>>
>>>> Right, to complete this work we need trace-merge like merging feature.
>>>> I assume that we can do that by passing the offset of the timestamp
>>>> information with guest trace data (that is the simplest implementation).
>>>>
>>>> However, currently trace-merge depends on x86 TSC, but some intel
>>>> processors TSC is not stable. That is current barrier of trace-merge.
>>>>
>>>> Thank you,
>>>>
>>>>
>>>>>
>>>>> But it is not applying on current trace-cmd report output format and
>>>>> not giving desired output .
>>>>>
>>>>> Thanks,
>>>>> Divya Vyas
>>>>>
>>>>> On Mon, Dec 22, 2014 at 12:34 PM, Divya Vyas <edivya.vyas@gmail.com> wrote:
>>>>>> Hi Steven,
>>>>>>
>>>>>> I am sorry but I am very new to trace-cmd , that's why I didn't knew
>>>>>> what you are working on it.
>>>>>>
>>>>>> I will explain you whole scenario , Actually I have multiple guests
>>>>>> and a host and I want to see all traces on a single file  with
>>>>>> trace-cmd report command and may be kernelshark also .
>>>>>>
>>>>>> trace-cmd report -i guest.dat -i host.dat command is giving me all
>>>>>> traces is single file but the timestamp is different for guest . So
>>>>>> the challenge for me is to sync the timestamp of guest and host .  Can
>>>>>> you advise me something in this ? How can I sync gettimeofday clocks
>>>>>> of guest and host ?
>>>>>>
>>>>>> and kernelshark doesnt take trace-cmd report input , I think it takes
>>>>>> guest.dat and host.dat .
>>>>>> Is there any other viewer available for ftrace ?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sat, Dec 20, 2014 at 10:40 AM, Steven Rostedt <rostedt@goodmis.org> wrote:
>>>>>>> On Fri, Dec 19, 2014 at 05:52:52PM +0530, Divya Vyas wrote:
>>>>>>>> Hi,
>>>>>>>
>>>>>>> Hi, it would be  helpful if you Cc'd me. I would have responded quicker. Luckily,
>>>>>>> someone noticed this email and told me about it.
>>>>>>>
>>>>>>>>
>>>>>>>> I think it is not desired output . Ths script is written for diffrent
>>>>>>>> format .ftrace format is ( trace-cmd report command generates) :
>>>>>>>
>>>>>>> I'm actually working on trace-cmd this week and hopefully I'll be able to release
>>>>>>> 2.5 by x-mas.
>>>>>>>
>>>>>>> I believe that Masami is going to start continuing work on the virt guest
>>>>>>> tracer for trace-cmd.
>>>>>>>
>>>>>>> Also, for now, for a crud way of merging, if both the gettimeofday clocks
>>>>>>> of the host and guest are synced, you could do:
>>>>>>>
>>>>>>> guest> trace-cmd record -o guest.dat --date  ...
>>>>>>>
>>>>>>> host> trace-cmd record -o host.dat --date ...
>>>>>>>
>>>>>>> and then run:
>>>>>>>
>>>>>>> trace-cmd report -i guest.dat -i host.dat
>>>>>>>
>>>>>>> and trace-cmd will merge the two. The --date will add a way to sync the trace
>>>>>>> timestamps to a wall clock and that will be used to try to interleave the
>>>>>>> events of both the data files.
>>>>>>>
>>>>>>> -- Steve
>>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Masami HIRAMATSU
>>>> Software Platform Research Dept. Linux Technology Research Center
>>>> Hitachi, Ltd., Yokohama Research Laboratory
>>>> E-mail: masami.hiramatsu.pt@hitachi.com
>>>>
>>>>