* Merge ftrace data of host and guest
@ 2014-12-19 12:22 Divya Vyas
2014-12-20 5:10 ` Steven Rostedt
0 siblings, 1 reply; 6+ messages in thread
From: Divya Vyas @ 2014-12-19 12:22 UTC (permalink / raw)
To: LKML
Hi,
I am adding trace data of host and guest using trace-merge script .
https://15351300108708232970.googlegroups.com/attach/66533d7b41eed0b/trace-merge.pl?part=0.1&view=1&vt=ANaJVrHKDEwmNeLGDINH0B5Ud4OIUrfQ2_2w2jo08nJujH7Oc9facKD3ckT2RbCiHkRAfnYxmZnBF7qi0SeMZr2IMsPBDNjgrYkblCRkmgSk2PryLVsGe5g
guest data
qemu-kvm-2687 [003] d...50550079203669: kvm_exit: [detail]
qemu-kvm-2687 [003] d...50550079206816: kvm_entry: [detail]
qemu-kvm-2687 [003] d...50550079240656: kvm_exit: [detail]
qemu-kvm-2687 [003] d...50550079243467: kvm_entry: [detail]
qemu-kvm-2687 [003] d...50550079256103: kvm_exit: [detail]
qemu-kvm-2687 [003] d...50550079268391: kvm_entry: [detail]
qemu-kvm-2687 [003] d...50550079280829: kvm_exit: [detail]
qemu-kvm-2687 [003] d...50550079286028: kvm_entry: [detail]
host data
comm-3826 [000] d...49836825726903: sched_wakeup: [detail]
comm-3826 [000] d...49836832225344: sched_switch: [detail]
~
Data after merging :
/opt/trace-merge/trace-merge 18446739773557710924 host.data guest.data
hcomm-3826 [000] d...49836825726903: sched_wakeup: [detail]
hcomm-3826 [000] d...49836832225344: sched_switch: [detail]
gqemu-kvm-2687 [003] d...54850231044361: kvm_exit: [detail]
gqemu-kvm-2687 [003] d...54850231047508: kvm_entry: [detail]
gqemu-kvm-2687 [003] d...54850231081348: kvm_exit: [detail]
gqemu-kvm-2687 [003] d...54850231084159: kvm_entry: [detail]
gqemu-kvm-2687 [003] d...54850231096795: kvm_exit: [detail]
gqemu-kvm-2687 [003] d...54850231109083: kvm_entry: [detail]
gqemu-kvm-2687 [003] d...54850231121521: kvm_exit: [detail]
gqemu-kvm-2687 [003] d...54850231126720: kvm_entry: [detail]
I think it is not desired output . Ths script is written for diffrent
format .ftrace format is ( trace-cmd report command generates) :
version = 6
CPU 0 is empty
cpus=2
<idle>-0 [001] 4401.096833: netif_receive_skb:
dev=eth0 skbaddr=0xffff88001ed8fc00 len=84
<idle>-0 [001] 4401.096899: net_dev_queue:
dev=eth0 skbaddr=0xffff88001a9fe000 len=98
<idle>-0 [001] 4401.096935: net_dev_xmit:
dev=eth0 skbaddr=0xffff88001a9fe000 len=98 rc=0
<idle>-0 [001] 4401.097277: netif_receive_skb:
dev=eth0 skbaddr=0xffff88001a9fe800 len=84
<idle>-0 [001] 4401.097309: net_dev_queue:
dev=eth0 skbaddr=0xffff88001a9fea00 len=98
<idle>-0 [001] 4401.097325: net_dev_xmit:
dev=eth0 skbaddr=0xffff88001a9fea00 len=98 rc=0
<idle>-0 [001] 4401.097570: netif_receive_skb:
dev=eth0 skbaddr=0xffff88001a9fe800 len=84
<idle>-0 [001] 4401.097592: net_dev_queue:
dev=eth0 skbaddr=0xffff88001a9fe000 len=98
<idle>-0 [001] 4401.097607: net_dev_xmit:
dev=eth0 skbaddr=0xffff88001a9fe000 len=98 rc=0
<idle>-0 [001] 4401.097830: netif_receive_skb:
dev=eth0 skbaddr=0xffff88001a9fe800 len=84
<idle>-0 [001] 4401.097843: net_dev_queue:
dev=eth0 skbaddr=0xffff88001a9fea00 len=98
Can you let me know I there is any other method to merge the traces or
any further changes in the script
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: Merge ftrace data of host and guest 2014-12-19 12:22 Merge ftrace data of host and guest Divya Vyas @ 2014-12-20 5:10 ` Steven Rostedt 2014-12-22 7:04 ` Divya Vyas 0 siblings, 1 reply; 6+ messages in thread From: Steven Rostedt @ 2014-12-20 5:10 UTC (permalink / raw) To: Divya Vyas; +Cc: LKML, Masami Hiramatsu On Fri, Dec 19, 2014 at 05:52:52PM +0530, Divya Vyas wrote: > Hi, Hi, it would be helpful if you Cc'd me. I would have responded quicker. Luckily, someone noticed this email and told me about it. > > I think it is not desired output . Ths script is written for diffrent > format .ftrace format is ( trace-cmd report command generates) : I'm actually working on trace-cmd this week and hopefully I'll be able to release 2.5 by x-mas. I believe that Masami is going to start continuing work on the virt guest tracer for trace-cmd. Also, for now, for a crud way of merging, if both the gettimeofday clocks of the host and guest are synced, you could do: guest> trace-cmd record -o guest.dat --date ... host> trace-cmd record -o host.dat --date ... and then run: trace-cmd report -i guest.dat -i host.dat and trace-cmd will merge the two. The --date will add a way to sync the trace timestamps to a wall clock and that will be used to try to interleave the events of both the data files. -- Steve ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Merge ftrace data of host and guest 2014-12-20 5:10 ` Steven Rostedt @ 2014-12-22 7:04 ` Divya Vyas 2014-12-22 7:07 ` Divya Vyas 0 siblings, 1 reply; 6+ messages in thread From: Divya Vyas @ 2014-12-22 7:04 UTC (permalink / raw) To: Steven Rostedt; +Cc: LKML, Masami Hiramatsu Hi Steven, I am sorry but I am very new to trace-cmd , that's why I didn't knew what you are working on it. I will explain you whole scenario , Actually I have multiple guests and a host and I want to see all traces on a single file with trace-cmd report command and may be kernelshark also . trace-cmd report -i guest.dat -i host.dat command is giving me all traces is single file but the timestamp is different for guest . So the challenge for me is to sync the timestamp of guest and host . Can you advise me something in this ? How can I sync gettimeofday clocks of guest and host ? and kernelshark doesnt take trace-cmd report input , I think it takes guest.dat and host.dat . Is there any other viewer available for ftrace ? On Sat, Dec 20, 2014 at 10:40 AM, Steven Rostedt <rostedt@goodmis.org> wrote: > On Fri, Dec 19, 2014 at 05:52:52PM +0530, Divya Vyas wrote: >> Hi, > > Hi, it would be helpful if you Cc'd me. I would have responded quicker. Luckily, > someone noticed this email and told me about it. > >> >> I think it is not desired output . Ths script is written for diffrent >> format .ftrace format is ( trace-cmd report command generates) : > > I'm actually working on trace-cmd this week and hopefully I'll be able to release > 2.5 by x-mas. > > I believe that Masami is going to start continuing work on the virt guest > tracer for trace-cmd. > > Also, for now, for a crud way of merging, if both the gettimeofday clocks > of the host and guest are synced, you could do: > > guest> trace-cmd record -o guest.dat --date ... > > host> trace-cmd record -o host.dat --date ... > > and then run: > > trace-cmd report -i guest.dat -i host.dat > > and trace-cmd will merge the two. The --date will add a way to sync the trace > timestamps to a wall clock and that will be used to try to interleave the > events of both the data files. > > -- Steve > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Merge ftrace data of host and guest 2014-12-22 7:04 ` Divya Vyas @ 2014-12-22 7:07 ` Divya Vyas 2014-12-22 8:33 ` Masami Hiramatsu 0 siblings, 1 reply; 6+ messages in thread From: Divya Vyas @ 2014-12-22 7:07 UTC (permalink / raw) To: Steven Rostedt; +Cc: LKML, Masami Hiramatsu Hi , One more thing I want to add that Yoshihiro YUNOMAE's trace-merge script does this time synchronization. We have to calculate guest offset and then subtract with host time . But it is not applying on current trace-cmd report output format and not giving desired output . Thanks, Divya Vyas On Mon, Dec 22, 2014 at 12:34 PM, Divya Vyas <edivya.vyas@gmail.com> wrote: > Hi Steven, > > I am sorry but I am very new to trace-cmd , that's why I didn't knew > what you are working on it. > > I will explain you whole scenario , Actually I have multiple guests > and a host and I want to see all traces on a single file with > trace-cmd report command and may be kernelshark also . > > trace-cmd report -i guest.dat -i host.dat command is giving me all > traces is single file but the timestamp is different for guest . So > the challenge for me is to sync the timestamp of guest and host . Can > you advise me something in this ? How can I sync gettimeofday clocks > of guest and host ? > > and kernelshark doesnt take trace-cmd report input , I think it takes > guest.dat and host.dat . > Is there any other viewer available for ftrace ? > > > > > > > On Sat, Dec 20, 2014 at 10:40 AM, Steven Rostedt <rostedt@goodmis.org> wrote: >> On Fri, Dec 19, 2014 at 05:52:52PM +0530, Divya Vyas wrote: >>> Hi, >> >> Hi, it would be helpful if you Cc'd me. I would have responded quicker. Luckily, >> someone noticed this email and told me about it. >> >>> >>> I think it is not desired output . Ths script is written for diffrent >>> format .ftrace format is ( trace-cmd report command generates) : >> >> I'm actually working on trace-cmd this week and hopefully I'll be able to release >> 2.5 by x-mas. >> >> I believe that Masami is going to start continuing work on the virt guest >> tracer for trace-cmd. >> >> Also, for now, for a crud way of merging, if both the gettimeofday clocks >> of the host and guest are synced, you could do: >> >> guest> trace-cmd record -o guest.dat --date ... >> >> host> trace-cmd record -o host.dat --date ... >> >> and then run: >> >> trace-cmd report -i guest.dat -i host.dat >> >> and trace-cmd will merge the two. The --date will add a way to sync the trace >> timestamps to a wall clock and that will be used to try to interleave the >> events of both the data files. >> >> -- Steve >> ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Merge ftrace data of host and guest 2014-12-22 7:07 ` Divya Vyas @ 2014-12-22 8:33 ` Masami Hiramatsu [not found] ` <CAGfiMvAALjFU+5FdmXa1VTQM6ZeEAZG5F1VDO60BRx4EK9zMZQ@mail.gmail.com> 0 siblings, 1 reply; 6+ messages in thread From: Masami Hiramatsu @ 2014-12-22 8:33 UTC (permalink / raw) To: Divya Vyas; +Cc: Steven Rostedt, LKML, yrl.pp-manager.tt Hi, I'm working on trace-cmd virt-server feature. This is just a basic feature to allow trace-cmd using virtio-trace (no time synchronizing). I'll send the updated version of the series soon. (2014/12/22 16:07), Divya Vyas wrote: > Hi , > > One more thing I want to add that Yoshihiro YUNOMAE's trace-merge > script does this time synchronization. We have to calculate guest > offset and then subtract with host time . Right, to complete this work we need trace-merge like merging feature. I assume that we can do that by passing the offset of the timestamp information with guest trace data (that is the simplest implementation). However, currently trace-merge depends on x86 TSC, but some intel processors TSC is not stable. That is current barrier of trace-merge. Thank you, > > But it is not applying on current trace-cmd report output format and > not giving desired output . > > Thanks, > Divya Vyas > > On Mon, Dec 22, 2014 at 12:34 PM, Divya Vyas <edivya.vyas@gmail.com> wrote: >> Hi Steven, >> >> I am sorry but I am very new to trace-cmd , that's why I didn't knew >> what you are working on it. >> >> I will explain you whole scenario , Actually I have multiple guests >> and a host and I want to see all traces on a single file with >> trace-cmd report command and may be kernelshark also . >> >> trace-cmd report -i guest.dat -i host.dat command is giving me all >> traces is single file but the timestamp is different for guest . So >> the challenge for me is to sync the timestamp of guest and host . Can >> you advise me something in this ? How can I sync gettimeofday clocks >> of guest and host ? >> >> and kernelshark doesnt take trace-cmd report input , I think it takes >> guest.dat and host.dat . >> Is there any other viewer available for ftrace ? >> >> >> >> >> >> >> On Sat, Dec 20, 2014 at 10:40 AM, Steven Rostedt <rostedt@goodmis.org> wrote: >>> On Fri, Dec 19, 2014 at 05:52:52PM +0530, Divya Vyas wrote: >>>> Hi, >>> >>> Hi, it would be helpful if you Cc'd me. I would have responded quicker. Luckily, >>> someone noticed this email and told me about it. >>> >>>> >>>> I think it is not desired output . Ths script is written for diffrent >>>> format .ftrace format is ( trace-cmd report command generates) : >>> >>> I'm actually working on trace-cmd this week and hopefully I'll be able to release >>> 2.5 by x-mas. >>> >>> I believe that Masami is going to start continuing work on the virt guest >>> tracer for trace-cmd. >>> >>> Also, for now, for a crud way of merging, if both the gettimeofday clocks >>> of the host and guest are synced, you could do: >>> >>> guest> trace-cmd record -o guest.dat --date ... >>> >>> host> trace-cmd record -o host.dat --date ... >>> >>> and then run: >>> >>> trace-cmd report -i guest.dat -i host.dat >>> >>> and trace-cmd will merge the two. The --date will add a way to sync the trace >>> timestamps to a wall clock and that will be used to try to interleave the >>> events of both the data files. >>> >>> -- Steve >>> > -- Masami HIRAMATSU Software Platform Research Dept. Linux Technology Research Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu.pt@hitachi.com ^ permalink raw reply [flat|nested] 6+ messages in thread
[parent not found: <CAGfiMvAALjFU+5FdmXa1VTQM6ZeEAZG5F1VDO60BRx4EK9zMZQ@mail.gmail.com>]
[parent not found: <CAGfiMvCQrcwsdDxM5XJtX=BZOF+LbpRJJgLCYEGTRNKkUioK9g@mail.gmail.com>]
[parent not found: <CAGfiMvA_K+r=p_G+PJ0SJdTqwZw3NBDtNO0=+we8x6-a6+ugPg@mail.gmail.com>]
* Re: Merge ftrace data of host and guest [not found] ` <CAGfiMvA_K+r=p_G+PJ0SJdTqwZw3NBDtNO0=+we8x6-a6+ugPg@mail.gmail.com> @ 2015-01-22 10:09 ` Divya Vyas 0 siblings, 0 replies; 6+ messages in thread From: Divya Vyas @ 2015-01-22 10:09 UTC (permalink / raw) To: Masami Hiramatsu, LKML; +Cc: Steven Rostedt, yrl.pp-manager.tt Hi, I am working on merge of ftrace data of guest and host . tsc_guest = tsc_host +offset I got offset from /sys/kernel/debug/tracing/instances/tsc_offset/trace 18446739224709377060 Where as my guest and host data is in below format : sshd-2332 [000] 4813.363939: net_dev_queue: dev=eth0 skbaddr=0xffff88003b78f6e8 len=134 sshd-2332 [000] 4813.363951: net_dev_xmit: dev=eth0 skbaddr=0xffff88003b78f6e8 len=134 rc=0 sshd-2332 [000] 4813.364010: netif_receive_skb: dev=eth0 skbaddr=0xffff880015cbbc00 len=52 sshd-2332 [000] 4813.364042: net_dev_queue: dev=eth0 skbaddr=0xffff88002c086ce8 len=102 sshd-2332 [000] 4813.364053: net_dev_xmit: dev=eth0 skbaddr=0xffff88002c086ce8 len=102 rc=0 trace-cmd-2496 [000] 4813.364098: netif_receive_skb: dev=eth0 skbaddr=0xffff880015cbbc00 len=52 <idle>-0 [000] 4814.384570: netif_receive_skb: dev=eth0 skbaddr=0xffff88003a490a00 len=38 <idle>-0 [000] 4816.384541: netif_receive_skb: dev=eth0 skbaddr=0xffff88003a490900 len=38 <idle>-0 [000] 4818.384681: netif_receive_skb: dev=eth0 skbaddr=0xffff88003a490900 len=38 How can I add the timestamp and offset .Do I need to add directly the numbers ? On Tue, Jan 13, 2015 at 11:03 AM, Divya Vyas <edivya.vyas@gmail.com> wrote: > Hi, > > Is there any feature in ftrace like synchronizing the timestamp > > http://help.eclipse.org/luna/index.jsp?topic=%2Forg.eclipse.linuxtools.lttng.help%2Fdoc%2FTrace-synchronization.html > > > > On Wed, Jan 7, 2015 at 2:57 PM, Divya Vyas <edivya.vyas@gmail.com> wrote: >> Hi Masami, >> >> Any updates on the merge feature? >> >> >> >> On Mon, Dec 22, 2014 at 4:36 PM, Divya Vyas <edivya.vyas@gmail.com> wrote: >>> Hi Masami, >>> >>> Thanks for the patches . >>> >>> Are you working on this merging feature also >>> https://lkml.org/lkml/2013/12/16/688 ? the d option ? >>> >>> trace-cmd report -i will not give expected output if the time is not >>> synchronized . From kernel side we can get TSC offset using >>> tracepoint when guest boots . I think this tracepoint was added from >>> 3.11 but from user side I got only the trace-merge script which takes >>> offset and trace and host data . But the format of trace data this >>> script accepts is little different . >>> >>> My end goal is to to merge the traces in chronological order and see >>> them in kernelshark . >>> >>> >>> Thanks, >>> Divya >>> >>> >>> >>> >>> >>> On Mon, Dec 22, 2014 at 2:03 PM, Masami Hiramatsu >>> <masami.hiramatsu.pt@hitachi.com> wrote: >>>> Hi, >>>> >>>> I'm working on trace-cmd virt-server feature. This is just a basic feature >>>> to allow trace-cmd using virtio-trace (no time synchronizing). >>>> I'll send the updated version of the series soon. >>>> >>>> (2014/12/22 16:07), Divya Vyas wrote: >>>>> Hi , >>>>> >>>>> One more thing I want to add that Yoshihiro YUNOMAE's trace-merge >>>>> script does this time synchronization. We have to calculate guest >>>>> offset and then subtract with host time . >>>> >>>> Right, to complete this work we need trace-merge like merging feature. >>>> I assume that we can do that by passing the offset of the timestamp >>>> information with guest trace data (that is the simplest implementation). >>>> >>>> However, currently trace-merge depends on x86 TSC, but some intel >>>> processors TSC is not stable. That is current barrier of trace-merge. >>>> >>>> Thank you, >>>> >>>> >>>>> >>>>> But it is not applying on current trace-cmd report output format and >>>>> not giving desired output . >>>>> >>>>> Thanks, >>>>> Divya Vyas >>>>> >>>>> On Mon, Dec 22, 2014 at 12:34 PM, Divya Vyas <edivya.vyas@gmail.com> wrote: >>>>>> Hi Steven, >>>>>> >>>>>> I am sorry but I am very new to trace-cmd , that's why I didn't knew >>>>>> what you are working on it. >>>>>> >>>>>> I will explain you whole scenario , Actually I have multiple guests >>>>>> and a host and I want to see all traces on a single file with >>>>>> trace-cmd report command and may be kernelshark also . >>>>>> >>>>>> trace-cmd report -i guest.dat -i host.dat command is giving me all >>>>>> traces is single file but the timestamp is different for guest . So >>>>>> the challenge for me is to sync the timestamp of guest and host . Can >>>>>> you advise me something in this ? How can I sync gettimeofday clocks >>>>>> of guest and host ? >>>>>> >>>>>> and kernelshark doesnt take trace-cmd report input , I think it takes >>>>>> guest.dat and host.dat . >>>>>> Is there any other viewer available for ftrace ? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Sat, Dec 20, 2014 at 10:40 AM, Steven Rostedt <rostedt@goodmis.org> wrote: >>>>>>> On Fri, Dec 19, 2014 at 05:52:52PM +0530, Divya Vyas wrote: >>>>>>>> Hi, >>>>>>> >>>>>>> Hi, it would be helpful if you Cc'd me. I would have responded quicker. Luckily, >>>>>>> someone noticed this email and told me about it. >>>>>>> >>>>>>>> >>>>>>>> I think it is not desired output . Ths script is written for diffrent >>>>>>>> format .ftrace format is ( trace-cmd report command generates) : >>>>>>> >>>>>>> I'm actually working on trace-cmd this week and hopefully I'll be able to release >>>>>>> 2.5 by x-mas. >>>>>>> >>>>>>> I believe that Masami is going to start continuing work on the virt guest >>>>>>> tracer for trace-cmd. >>>>>>> >>>>>>> Also, for now, for a crud way of merging, if both the gettimeofday clocks >>>>>>> of the host and guest are synced, you could do: >>>>>>> >>>>>>> guest> trace-cmd record -o guest.dat --date ... >>>>>>> >>>>>>> host> trace-cmd record -o host.dat --date ... >>>>>>> >>>>>>> and then run: >>>>>>> >>>>>>> trace-cmd report -i guest.dat -i host.dat >>>>>>> >>>>>>> and trace-cmd will merge the two. The --date will add a way to sync the trace >>>>>>> timestamps to a wall clock and that will be used to try to interleave the >>>>>>> events of both the data files. >>>>>>> >>>>>>> -- Steve >>>>>>> >>>>> >>>> >>>> >>>> -- >>>> Masami HIRAMATSU >>>> Software Platform Research Dept. Linux Technology Research Center >>>> Hitachi, Ltd., Yokohama Research Laboratory >>>> E-mail: masami.hiramatsu.pt@hitachi.com >>>> >>>> ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2015-01-22 10:10 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-12-19 12:22 Merge ftrace data of host and guest Divya Vyas
2014-12-20 5:10 ` Steven Rostedt
2014-12-22 7:04 ` Divya Vyas
2014-12-22 7:07 ` Divya Vyas
2014-12-22 8:33 ` Masami Hiramatsu
[not found] ` <CAGfiMvAALjFU+5FdmXa1VTQM6ZeEAZG5F1VDO60BRx4EK9zMZQ@mail.gmail.com>
[not found] ` <CAGfiMvCQrcwsdDxM5XJtX=BZOF+LbpRJJgLCYEGTRNKkUioK9g@mail.gmail.com>
[not found] ` <CAGfiMvA_K+r=p_G+PJ0SJdTqwZw3NBDtNO0=+we8x6-a6+ugPg@mail.gmail.com>
2015-01-22 10:09 ` Divya Vyas
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).