mailing list for trace users

mailing list for trace users

[Steven Rostedt]

What are people's thoughts about creating a linux-trace-users mailing
list on vger.kernel.org?

This would be a place for users of tracers and even perf. For asking
questions about how to use the debugfs/tracing directory or the perf
tool.

This will not be a place for kernel development. Patches for the tracing
infrastructure should still go through LKML. This will be a place to ask
how-to questions, or any other kind of help questions.

LKML can be quite intimidating, and it is not a place to ask help
questions anyway. I think adding a linux-trace-users mailing list would
be a good idea.

Thoughts?

-- Steve

Re: mailing list for trace users

[Frederic Weisbecker]

On Wed, Sep 16, 2009 at 04:16:22PM -0400, Steven Rostedt wrote:
> What are people's thoughts about creating a linux-trace-users mailing
> list on vger.kernel.org?
> 
> This would be a place for users of tracers and even perf. For asking
> questions about how to use the debugfs/tracing directory or the perf
> tool.
> 
> This will not be a place for kernel development. Patches for the tracing
> infrastructure should still go through LKML. This will be a place to ask
> how-to questions, or any other kind of help questions.
> 
> LKML can be quite intimidating, and it is not a place to ask help
> questions anyway. I think adding a linux-trace-users mailing list would
> be a good idea.
> 
> Thoughts?
> 
> -- Steve


Why not. That could also gather informations from LKML threads by
Cc'ing it while explaining how to use the tools in random discussions.

Re: mailing list for trace users

[Li Zefan]

Frederic Weisbecker wrote:
> On Wed, Sep 16, 2009 at 04:16:22PM -0400, Steven Rostedt wrote:
>> What are people's thoughts about creating a linux-trace-users mailing
>> list on vger.kernel.org?
>>
>> This would be a place for users of tracers and even perf. For asking
>> questions about how to use the debugfs/tracing directory or the perf
>> tool.
>>
>> This will not be a place for kernel development. Patches for the tracing
>> infrastructure should still go through LKML. This will be a place to ask
>> how-to questions, or any other kind of help questions.
>>
>> LKML can be quite intimidating, and it is not a place to ask help
>> questions anyway. I think adding a linux-trace-users mailing list would
>> be a good idea.
>>
>> Thoughts?
>>
>> -- Steve
> 
> 
> Why not. That could also gather informations from LKML threads by
> Cc'ing it while explaining how to use the tools in random discussions.
> 

+1

And then the mailing list needs to be renamed to linux-trace. And
we need to advertise it, in Documentation, Kconfig?

Re: mailing list for trace users

[Ingo Molnar]

* Li Zefan <lizf@cn.fujitsu.com> wrote:

> Frederic Weisbecker wrote:
> > On Wed, Sep 16, 2009 at 04:16:22PM -0400, Steven Rostedt wrote:
> >> What are people's thoughts about creating a linux-trace-users mailing
> >> list on vger.kernel.org?
> >>
> >> This would be a place for users of tracers and even perf. For asking
> >> questions about how to use the debugfs/tracing directory or the perf
> >> tool.
> >>
> >> This will not be a place for kernel development. Patches for the tracing
> >> infrastructure should still go through LKML. This will be a place to ask
> >> how-to questions, or any other kind of help questions.
> >>
> >> LKML can be quite intimidating, and it is not a place to ask help
> >> questions anyway. I think adding a linux-trace-users mailing list would
> >> be a good idea.
> >>
> >> Thoughts?
> >>
> >> -- Steve
> > 
> > 
> > Why not. That could also gather informations from LKML threads by 
> > Cc'ing it while explaining how to use the tools in random 
> > discussions.
> > 
> 
> +1
> 
> And then the mailing list needs to be renamed to linux-trace. And we 
> need to advertise it, in Documentation, Kconfig?

The list i can support is what was suggested here: 
tracing-users@vger.kernel.org, for user questions and solutions.

Development of tracing bits should still be very much done on lkml.

We dont want to go down the failed path of some other projects that went 
off to their private lists, became self-breeding, ego-boosting 
pat-each-other-on-the-shoulders-for-being-cool-guys forums, became more 
and more detached from mainline and the complexities there, and in some 
cases also became increasingly hostile towards mainline (hey everyone on 
that list agrees that we are cool guys writing cool code, why doesnt 
upstream love us unconditionally?) and sometimes started growing clique 
or clan social structures - which are outright harmful.

The topical filtering offered by such lists is an advantage (note that 
those can be had on lkml as well by using mail filtering) - but the 
social disadvantages are unsolvable, lets not go there.

Staying on lkml and staying connected is particularly important for a 
subsystem that by their very nature deal with many other subsystems, 
such as tracing / instrumentation.

	Ingo

Re: mailing list for trace users

[John Kacur]

On Wed, Sep 16, 2009 at 10:16 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
> What are people's thoughts about creating a linux-trace-users mailing
> list on vger.kernel.org?
>
> This would be a place for users of tracers and even perf. For asking
> questions about how to use the debugfs/tracing directory or the perf
> tool.
>
> This will not be a place for kernel development. Patches for the tracing
> infrastructure should still go through LKML. This will be a place to ask
> how-to questions, or any other kind of help questions.
>
> LKML can be quite intimidating, and it is not a place to ask help
> questions anyway. I think adding a linux-trace-users mailing list would
> be a good idea.
>

Sign me up!

Re: mailing list for trace users

[Avi Kivity]

On 09/16/2009 11:16 PM, Steven Rostedt wrote:
> What are people's thoughts about creating a linux-trace-users mailing
> list on vger.kernel.org?
>
> This would be a place for users of tracers and even perf. For asking
> questions about how to use the debugfs/tracing directory or the perf
> tool.
>
> This will not be a place for kernel development. Patches for the tracing
> infrastructure should still go through LKML. This will be a place to ask
> how-to questions, or any other kind of help questions.
>
> LKML can be quite intimidating, and it is not a place to ask help
> questions anyway. I think adding a linux-trace-users mailing list would
> be a good idea.
>    

Yes please.  Here's a question to start it off - how to I 'perf 
annotate' a symbol in a module?

$ perf report
# Samples: 68202
#
# Overhead          Command  Shared Object  Symbol
# ........  ...............  .............  ......
#
     84.17%  qemu-system-x86  [kernel]       [k] vmx_vcpu_run    [kvm_intel]
      4.28%  qemu-system-x86  [kernel]       [k] kvm_arch_vcpu_ioctl_run 
[kvm]
      0.88%  qemu-system-x86  [kernel]       [k] add_preempt_count
      0.75%  qemu-system-x86  [kernel]       [k] _spin_unlock_irqrestore
      0.68%  qemu-system-x86  [kernel]       [k] _spin_lock_irq

$ perf annotate -k ~avi/kvm/linux-2.6/vmlinux -m vmx_vcpu_run
  Error: symbol 'vmx_vcpu_run' not present amongst the samples.

builtin symbols work.

-- 
error compiling committee.c: too many arguments to function

Re: mailing list for trace users

[Peter Zijlstra]

On Tue, 2009-09-22 at 12:13 +0300, Avi Kivity wrote:
> 
> Yes please.  Here's a question to start it off - how to I 'perf 
> annotate' a symbol in a module?
> 
> $ perf report
> # Samples: 68202
> #
> # Overhead          Command  Shared Object  Symbol
> # ........  ...............  .............  ......
> #
>      84.17%  qemu-system-x86  [kernel]       [k] vmx_vcpu_run
> [kvm_intel]
>       4.28%  qemu-system-x86  [kernel]       [k]
> kvm_arch_vcpu_ioctl_run 
> [kvm]
>       0.88%  qemu-system-x86  [kernel]       [k] add_preempt_count
>       0.75%  qemu-system-x86  [kernel]       [k]
> _spin_unlock_irqrestore
>       0.68%  qemu-system-x86  [kernel]       [k] _spin_lock_irq
> 
> $ perf annotate -k ~avi/kvm/linux-2.6/vmlinux -m vmx_vcpu_run
>   Error: symbol 'vmx_vcpu_run' not present amongst the samples.
> 
> builtin symbols work.

We need a /proc/modules parser that then also locates and loads .ko
files which aren't stripped.

I think Mike played around with that a bit.

The typical problem is that people use 'make INSTALL_MOD_STRIP=1
modules_install' to install their modules, because otherwise the initrd
thingies explode, the side effect is that the !initrd modules are
stripped too.

Ideally initrd tools should strip whatever they stick in. Then again,
ideally we'd not have any initrd and modules crap to begin with ;-)

Re: mailing list for trace users

[Mike Galbraith]

On Tue, 2009-09-22 at 12:59 +0200, Peter Zijlstra wrote:

> We need a /proc/modules parser that then also locates and loads .ko
> files which aren't stripped.
> 
> I think Mike played around with that a bit.

Nope.

> The typical problem is that people use 'make INSTALL_MOD_STRIP=1
> modules_install' to install their modules, because otherwise the initrd
> thingies explode, the side effect is that the !initrd modules are
> stripped too.

Oh, yeah, that would explain it.  Stripped module is a nogo.

	-Mike

Re: mailing list for trace users

[Mike Galbraith]

On Tue, 2009-09-22 at 12:13 +0300, Avi Kivity wrote:
> Yes please.  Here's a question to start it off - how to I 'perf 
> annotate' a symbol in a module?
> 
> $ perf report
> # Samples: 68202
> #
> # Overhead          Command  Shared Object  Symbol
> # ........  ...............  .............  ......
> #
>      84.17%  qemu-system-x86  [kernel]       [k] vmx_vcpu_run    [kvm_intel]
>       4.28%  qemu-system-x86  [kernel]       [k] kvm_arch_vcpu_ioctl_run 
> [kvm]
>       0.88%  qemu-system-x86  [kernel]       [k] add_preempt_count
>       0.75%  qemu-system-x86  [kernel]       [k] _spin_unlock_irqrestore
>       0.68%  qemu-system-x86  [kernel]       [k] _spin_lock_irq
> 
> $ perf annotate -k ~avi/kvm/linux-2.6/vmlinux -m vmx_vcpu_run
>   Error: symbol 'vmx_vcpu_run' not present amongst the samples.
> 
> builtin symbols work.
> 

Hm, dunno.  Using tip v2.6.31-7960-gf3da2f6, module annotation seems to
work fine here.

marge:/root/tmp # perf report -k vmlinux -m -d [kernel]|grep ahci_interrupt
     1.21%          swapper  [k] ahci_interrupt [ahci]
     0.11%            :7222  [k] ahci_interrupt [ahci]
     0.04%            :4876  [k] ahci_interrupt [ahci]
     0.02%            :6628  [k] ahci_interrupt [ahci]
     0.01%            :7452  [k] ahci_interrupt [ahci]
     0.01%            :7407  [k] ahci_interrupt [ahci]
marge:/root/tmp # perf annotate -k vmlinux -m ahci_interrupt|grep -C 20 Disassembly


------------------------------------------------
 Percent |      Source code & Disassembly of ahci.ko
------------------------------------------------
         :
         :
         :
         :      Disassembly of section .text:
         :
         :      0000000000002a49 <ahci_interrupt>:
         :                      ata_port_freeze(ap);
         :              }
         :      }
         :
         :      static irqreturn_t ahci_interrupt(int irq, void *dev_instance)
         :      {
    0.00 :          2a49:       55                      push   %rbp
    0.69 :          2a4a:       48 89 e5                mov    %rsp,%rbp
    0.00 :          2a4d:       41 57                   push   %r15
    0.00 :          2a4f:       49 89 f7                mov    %rsi,%r15
    0.00 :          2a52:       41 56                   push   %r14
    0.00 :          2a54:       41 55                   push   %r13
    0.00 :          2a56:       41 54                   push   %r12
    0.00 :          2a58:       53                      push   %rbx
    0.00 :          2a59:       48 83 ec 58             sub    $0x58,%rsp
         :      { asm volatile("mov" size " %0,%1": :reg (val), \
         :      "m" (*(volatile type __force *)addr) barrier); }
         :
marge:/root/tmp #

Re: mailing list for trace users

[Mike Galbraith]

On Tue, 2009-09-22 at 12:13 +0300, Avi Kivity wrote:

> $ perf annotate -k ~avi/kvm/linux-2.6/vmlinux -m vmx_vcpu_run
>   Error: symbol 'vmx_vcpu_run' not present amongst the samples.
> 
> builtin symbols work.

Ah.  Did you reboot or unload/load the module between record and
annotate time?  There is no support for that at the moment.

There has been talk of eventually putting all pertinent data into
perf.data to support remote analysis etc, but as things stand, module
information is only available for the running kernel.

	-Mike

Re: mailing list for trace users

[Avi Kivity]

On 09/22/2009 02:28 PM, Mike Galbraith wrote:
> On Tue, 2009-09-22 at 12:13 +0300, Avi Kivity wrote:
>
>    
>> $ perf annotate -k ~avi/kvm/linux-2.6/vmlinux -m vmx_vcpu_run
>>    Error: symbol 'vmx_vcpu_run' not present amongst the samples.
>>
>> builtin symbols work.
>>      
> Ah.  Did you reboot or unload/load the module between record and
> annotate time?  There is no support for that at the moment.
>    

No.  Still loaded and running.

-- 
error compiling committee.c: too many arguments to function

Re: mailing list for trace users

[Mike Galbraith]

On Tue, 2009-09-22 at 14:34 +0300, Avi Kivity wrote:
> On 09/22/2009 02:28 PM, Mike Galbraith wrote:
> > On Tue, 2009-09-22 at 12:13 +0300, Avi Kivity wrote:
> >
> >    
> >> $ perf annotate -k ~avi/kvm/linux-2.6/vmlinux -m vmx_vcpu_run
> >>    Error: symbol 'vmx_vcpu_run' not present amongst the samples.
> >>
> >> builtin symbols work.
> >>      
> > Ah.  Did you reboot or unload/load the module between record and
> > annotate time?  There is no support for that at the moment.
> >    
> 
> No.  Still loaded and running.

Hm, must me a problem with parsing then.  If you add -v -v to the
command line it'll spit out debug data.  For vmx_vcpu_run you should see
a line like so if the module was parsed.

new symbol: ffffffffa0065a49 [00000466]: ahci_interrupt, hist: (nil), obj_start: 0x2a49

	-Mike

Re: mailing list for trace users

[Avi Kivity]

On 09/22/2009 02:47 PM, Mike Galbraith wrote:
>
> Hm, must me a problem with parsing then.  If you add -v -v to the
> command line it'll spit out debug data.  For vmx_vcpu_run you should see
> a line like so if the module was parsed.
>
> new symbol: ffffffffa0065a49 [00000466]: ahci_interrupt, hist: (nil), obj_start: 0x2a49
>
> 	-Mike
>
>    

$ perf annotate -v -v -k ~avi/kvm/linux-2.6/vmlinux -m vmx_vcpu_run  | 
grep vmx_vcpu_run
new symbol: ffffffffa006f596 [0000dead]: vmx_vcpu_run    [kvm_intel], 
hist: (nil), obj_start: (nil)
  ffffffffa006f596-ffffffffa006fb73 vmx_vcpu_run    [kvm_intel]
  Error: symbol 'vmx_vcpu_run' not present amongst the samples.

Looks like internal confusion.

-- 
error compiling committee.c: too many arguments to function

Re: mailing list for trace users

[Mike Galbraith]

On Tue, 2009-09-22 at 14:51 +0300, Avi Kivity wrote:
> On 09/22/2009 02:47 PM, Mike Galbraith wrote:
> >
> > Hm, must me a problem with parsing then.  If you add -v -v to the
> > command line it'll spit out debug data.  For vmx_vcpu_run you should see
> > a line like so if the module was parsed.
> >
> > new symbol: ffffffffa0065a49 [00000466]: ahci_interrupt, hist: (nil), obj_start: 0x2a49
> >
> > 	-Mike
> >
> >    
> 
> $ perf annotate -v -v -k ~avi/kvm/linux-2.6/vmlinux -m vmx_vcpu_run  | 
> grep vmx_vcpu_run
> new symbol: ffffffffa006f596 [0000dead]: vmx_vcpu_run    [kvm_intel], 
> hist: (nil), obj_start: (nil)
>   ffffffffa006f596-ffffffffa006fb73 vmx_vcpu_run    [kvm_intel]
>   Error: symbol 'vmx_vcpu_run' not present amongst the samples.
> 
> Looks like internal confusion.

Guess I need to build a kvm_intel gizmo.

	-Mike

Re: mailing list for trace users

[Mike Galbraith]

On Tue, 2009-09-22 at 13:54 +0200, Mike Galbraith wrote:
> On Tue, 2009-09-22 at 14:51 +0300, Avi Kivity wrote:
> > On 09/22/2009 02:47 PM, Mike Galbraith wrote:
> > >
> > > Hm, must me a problem with parsing then.  If you add -v -v to the
> > > command line it'll spit out debug data.  For vmx_vcpu_run you should see
> > > a line like so if the module was parsed.
> > >
> > > new symbol: ffffffffa0065a49 [00000466]: ahci_interrupt, hist: (nil), obj_start: 0x2a49
> > >
> > > 	-Mike
> > >
> > >    
> > 
> > $ perf annotate -v -v -k ~avi/kvm/linux-2.6/vmlinux -m vmx_vcpu_run  | 
> > grep vmx_vcpu_run
> > new symbol: ffffffffa006f596 [0000dead]: vmx_vcpu_run    [kvm_intel], 
> > hist: (nil), obj_start: (nil)
> >   ffffffffa006f596-ffffffffa006fb73 vmx_vcpu_run    [kvm_intel]
> >   Error: symbol 'vmx_vcpu_run' not present amongst the samples.
> > 
> > Looks like internal confusion.
> 
> Guess I need to build a kvm_intel gizmo.

Parses fine here, and INSTALL_MOD_STRIP=1 made zero difference.  can you
send me that module and .config gzipped up?

	-Mike

Re: mailing list for trace users

[Avi Kivity]

On 09/22/2009 04:53 PM, Mike Galbraith wrote:
>
> Parses fine here, and INSTALL_MOD_STRIP=1 made zero difference.  can you
> send me that module and .config gzipped up?
>
>    

Sent privately.

-- 
error compiling committee.c: too many arguments to function

Re: mailing list for trace users

[Mike Galbraith]

On Tue, 2009-09-22 at 17:03 +0300, Avi Kivity wrote:
> On 09/22/2009 04:53 PM, Mike Galbraith wrote:
> >
> > Parses fine here, and INSTALL_MOD_STRIP=1 made zero difference.  can you
> > send me that module and .config gzipped up?
> >
> >    
> 
> Sent privately.

I can reproduce with your config, so do don't (thankfully) need to rip
binary apart looking for tools things.  Will take a look.

	Mike

Re: mailing list for trace users

[Avi Kivity]

On 09/22/2009 10:09 PM, Mike Galbraith wrote:
> On Tue, 2009-09-22 at 17:03 +0300, Avi Kivity wrote:
>    
>> On 09/22/2009 04:53 PM, Mike Galbraith wrote:
>>      
>>> Parses fine here, and INSTALL_MOD_STRIP=1 made zero difference.  can you
>>> send me that module and .config gzipped up?
>>>
>>>
>>>        
>> Sent privately.
>>      
> I can reproduce with your config, so do don't (thankfully) need to rip
> binary apart looking for tools things.  Will take a look.
>    

Thanks!  While trace/perf still have some rough edges, they've already 
proven incredibly valuable for debugging and performance analysis.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: mailing list for trace users

[Mike Galbraith]

On Tue, 2009-09-22 at 22:14 +0300, Avi Kivity wrote:
> On 09/22/2009 10:09 PM, Mike Galbraith wrote:
> > On Tue, 2009-09-22 at 17:03 +0300, Avi Kivity wrote:
> >    
> >> On 09/22/2009 04:53 PM, Mike Galbraith wrote:
> >>      
> >>> Parses fine here, and INSTALL_MOD_STRIP=1 made zero difference.  can you
> >>> send me that module and .config gzipped up?
> >>>
> >>>
> >>>        
> >> Sent privately.
> >>      
> > I can reproduce with your config, so do don't (thankfully) need to rip
> > binary apart looking for tools things.  Will take a look.
> >    
> 
> Thanks!  While trace/perf still have some rough edges, they've already 
> proven incredibly valuable for debugging and performance analysis.

Heh, fixed that... pass the baggies please ;-)

With your config, symbols were being loaded twice.  Parsing of vmlinux
and modules goes fine.  Unless there are no modules currently loaded, or
_the last module scanned isn't loaded_ that is.  dso__load_modules()
then returns 0, and we fall back to kallsyms.. making a mess.

Anyway, there's some other bustage with your config, so I'm going to
have to do more symbol table debugging/validation.

	-Mike

[perf] Finding uninstalled modules Was Re: mailing list for trace users

[Arnaldo Carvalho de Melo]

Em Tue, Sep 22, 2009 at 02:51:19PM +0300, Avi Kivity escreveu:
> On 09/22/2009 02:47 PM, Mike Galbraith wrote:
>>
>> Hm, must me a problem with parsing then.  If you add -v -v to the
>> command line it'll spit out debug data.  For vmx_vcpu_run you should see
>> a line like so if the module was parsed.
>>
>> new symbol: ffffffffa0065a49 [00000466]: ahci_interrupt, hist: (nil), obj_start: 0x2a49
>>
>> 	-Mike
>
> $ perf annotate -v -v -k ~avi/kvm/linux-2.6/vmlinux -m vmx_vcpu_run  |  

Here is the problem, he is passing a vmlinux, that way we don't parse
/proc/kallsyms, so no module symbols, he uses -m to load the modules
symbols but mod_dso__load_module_paths only looks at /lib/modules/, i.e.
installed modules.

I guess Avi hasn't installed modules, right? So the right fix for this
case is to figure out where modules are from the path given to -k, i.e.
we first use ~avi/kvm/linux-2.6/ as the modules path prefix and then
fallback to /lib/modules if we can't find modules there, right?

> grep vmx_vcpu_run
> new symbol: ffffffffa006f596 [0000dead]: vmx_vcpu_run    [kvm_intel],  
> hist: (nil), obj_start: (nil)
>  ffffffffa006f596-ffffffffa006fb73 vmx_vcpu_run    [kvm_intel]
>  Error: symbol 'vmx_vcpu_run' not present amongst the samples.
>
> Looks like internal confusion.
>
> -- 
> error compiling committee.c: too many arguments to function

Re: [perf] Finding uninstalled modules Was Re: mailing list for trace users

[Avi Kivity]

On 09/22/2009 11:17 PM, Arnaldo Carvalho de Melo wrote:
>
>> $ perf annotate -v -v -k ~avi/kvm/linux-2.6/vmlinux -m vmx_vcpu_run  |
>>      
> Here is the problem, he is passing a vmlinux, that way we don't parse
> /proc/kallsyms, so no module symbols, he uses -m to load the modules
> symbols but mod_dso__load_module_paths only looks at /lib/modules/, i.e.
> installed modules.
>
> I guess Avi hasn't installed modules, right? So the right fix for this
> case is to figure out where modules are from the path given to -k, i.e.
> we first use ~avi/kvm/linux-2.6/ as the modules path prefix and then
> fallback to /lib/modules if we can't find modules there, right?
>    

Modules were installed (I always load them with modprobe).  It's 
possible that the installed modules were a later version than the loaded 
modules, but Mike's reply leads me to believe there was a real bug there.

-- 
error compiling committee.c: too many arguments to function

Re: [perf] Finding uninstalled modules Was Re: mailing list for trace users

[Arnaldo Carvalho de Melo]

Em Wed, Sep 23, 2009 at 11:31:04AM +0300, Avi Kivity escreveu:
> On 09/22/2009 11:17 PM, Arnaldo Carvalho de Melo wrote:
>>
>>> $ perf annotate -v -v -k ~avi/kvm/linux-2.6/vmlinux -m vmx_vcpu_run  |
>>>      
>> Here is the problem, he is passing a vmlinux, that way we don't parse
>> /proc/kallsyms, so no module symbols, he uses -m to load the modules
>> symbols but mod_dso__load_module_paths only looks at /lib/modules/, i.e.
>> installed modules.
>>
>> I guess Avi hasn't installed modules, right? So the right fix for this
>> case is to figure out where modules are from the path given to -k, i.e.
>> we first use ~avi/kvm/linux-2.6/ as the modules path prefix and then
>> fallback to /lib/modules if we can't find modules there, right?
>>    
>
> Modules were installed (I always load them with modprobe).  It's  
> possible that the installed modules were a later version than the loaded  
> modules, but Mike's reply leads me to believe there was a real bug there.

Yeah, I'm rewriting the module symbol lookup code in perf, they should
really be maps backed by DSOs, just like shared libraries, only that
shared among all threads.

Will continue working on this tomorrow. Now need to get some sleep, else
I'll miss the first day of plumbers 8)

- Arnaldo

Re: [perf] Finding uninstalled modules Was Re: mailing list for trace users

[Ingo Molnar]

* Avi Kivity <avi@redhat.com> wrote:

> On 09/22/2009 11:17 PM, Arnaldo Carvalho de Melo wrote:
>>
>>> $ perf annotate -v -v -k ~avi/kvm/linux-2.6/vmlinux -m vmx_vcpu_run  |
>>>      
>> Here is the problem, he is passing a vmlinux, that way we don't parse
>> /proc/kallsyms, so no module symbols, he uses -m to load the modules
>> symbols but mod_dso__load_module_paths only looks at /lib/modules/, i.e.
>> installed modules.
>>
>> I guess Avi hasn't installed modules, right? So the right fix for 
>> this case is to figure out where modules are from the path given to 
>> -k, i.e. we first use ~avi/kvm/linux-2.6/ as the modules path prefix 
>> and then fallback to /lib/modules if we can't find modules there, 
>> right?
>
> Modules were installed (I always load them with modprobe).  It's 
> possible that the installed modules were a later version than the 
> loaded modules, but Mike's reply leads me to believe there was a real 
> bug there.

Yes, definitely - 'perf annotate' not giving you what you expected is a 
bug by definition - regardless of how you build your kernel, how you 
loaded your modules and how the symbols and tables are distributed 
across the system.

	Ingo

[patch] Re: [perf] Finding uninstalled modules Was Re: mailing list for trace users

[Mike Galbraith]

On Wed, 2009-09-23 at 11:31 +0300, Avi Kivity wrote:
> On 09/22/2009 11:17 PM, Arnaldo Carvalho de Melo wrote:
> >
> >> $ perf annotate -v -v -k ~avi/kvm/linux-2.6/vmlinux -m vmx_vcpu_run  |
> >>      
> > Here is the problem, he is passing a vmlinux, that way we don't parse
> > /proc/kallsyms, so no module symbols, he uses -m to load the modules
> > symbols but mod_dso__load_module_paths only looks at /lib/modules/, i.e.
> > installed modules.
> >
> > I guess Avi hasn't installed modules, right? So the right fix for this
> > case is to figure out where modules are from the path given to -k, i.e.
> > we first use ~avi/kvm/linux-2.6/ as the modules path prefix and then
> > fallback to /lib/modules if we can't find modules there, right?
> >    
> 
> Modules were installed (I always load them with modprobe).  It's 
> possible that the installed modules were a later version than the loaded 
> modules, but Mike's reply leads me to believe there was a real bug there.

Yup, brown baggie variety.  Oh darn.

perf_counter tools: fix brown baggie module symbol loading bug.

If there are no modules currently loaded, or the last module scanned is not
loaded, dso__load_modules() steps on the value from dso__load_vmlinux(), so
we happily load the kallsyms symbols on top of what we've already loaded.

Fix that such that the total count of symbols loaded is returned.  Should
module symbol load fail after parsing of vmlinux, is's a hard failure, so
do not silently fall-back to kallsyms.

Signed-off-by: Mike Galbraith <efault@gmx.de>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
LKML-Reference: <new-submission>

 tools/perf/util/symbol.c |   17 +++++++++++++----
 1 files changed, 13 insertions(+), 4 deletions(-)

diff --git a/tools/perf/util/symbol.c b/tools/perf/util/symbol.c
index fd3d9c8..559fb06 100644
--- a/tools/perf/util/symbol.c
+++ b/tools/perf/util/symbol.c
@@ -833,7 +833,7 @@ int dso__load_modules(struct dso *self, symbol_filter_t filter, int v)
 	struct mod_dso *mods = mod_dso__new_dso("modules");
 	struct module *pos;
 	struct rb_node *next;
-	int err;
+	int err, count = 0;
 
 	err = mod_dso__load_modules(mods);
 
@@ -852,14 +852,16 @@ int dso__load_modules(struct dso *self, symbol_filter_t filter, int v)
 			break;
 
 		next = rb_next(&pos->rb_node);
+		count += err;
 	}
 
 	if (err < 0) {
 		mod_dso__delete_modules(mods);
 		mod_dso__delete_self(mods);
+		return err;
 	}
 
-	return err;
+	return count;
 }
 
 static inline void dso__fill_symbol_holes(struct dso *self)
@@ -913,8 +915,15 @@ int dso__load_kernel(struct dso *self, const char *vmlinux,
 
 	if (vmlinux) {
 		err = dso__load_vmlinux(self, vmlinux, filter, v);
-		if (err > 0 && use_modules)
-			err = dso__load_modules(self, filter, v);
+		if (err > 0 && use_modules) {
+			int syms = dso__load_modules(self, filter, v);
+
+			if (syms < 0) {
+				fprintf(stderr, "dso__load_modules failed!\n");
+				return syms;
+			}
+			err += syms;
+		}
 	}
 
 	if (err <= 0)

Re: [patch] Re: [perf] Finding uninstalled modules Was Re: mailing list for trace users

[Avi Kivity]

On 09/23/2009 12:20 PM, Mike Galbraith wrote:
>
> Yup, brown baggie variety.  Oh darn.
>
> perf_counter tools: fix brown baggie module symbol loading bug.
>
> If there are no modules currently loaded, or the last module scanned is not
> loaded, dso__load_modules() steps on the value from dso__load_vmlinux(), so
> we happily load the kallsyms symbols on top of what we've already loaded.
>
> Fix that such that the total count of symbols loaded is returned.  Should
> module symbol load fail after parsing of vmlinux, is's a hard failure, so
> do not silently fall-back to kallsyms.
>
>    

Still fails, but differently.  Now 'annotate -k ... -m -v -v' doesn't 
list vmx_vcpu_run at all, even though it's prominent in 'perf top'.

In addition to applying your patch I've merged current linus, so that 
may have introduced the problem.

If I don't supply -k -m, I get

$ perf annotate  -v -v vmx_vcpu_run | grep vmx_vcpu
new symbol: ffffffffa006f596 [0000dead]: vmx_vcpu_run    [kvm_intel], 
hist: (nil), obj_start: (nil)
new symbol: ffffffffa007025f [0000dead]: vmx_vcpu_put    [kvm_intel], 
hist: (nil), obj_start: (nil)
new symbol: ffffffffa0070bf6 [0000dead]: vmx_vcpu_load    [kvm_intel], 
hist: (nil), obj_start: (nil)
new symbol: ffffffffa0070d99 [0000dead]: vmx_vcpu_reset    [kvm_intel], 
hist: (nil), obj_start: (nil)
ffffffffa006f596-ffffffffa006fb73 vmx_vcpu_run    [kvm_intel]
  ffffffffa007025f-ffffffffa007026e vmx_vcpu_put    [kvm_intel]
  ffffffffa0070bf6-ffffffffa0070d98 vmx_vcpu_load    [kvm_intel]
  ffffffffa0070d99-ffffffffa0071191 vmx_vcpu_reset    [kvm_intel]
  Error: symbol 'vmx_vcpu_run' not present amongst the samples.


-- 
error compiling committee.c: too many arguments to function

Re: [patch] Re: [perf] Finding uninstalled modules Was Re: mailing list for trace users

[Mike Galbraith]

On Wed, 2009-09-23 at 12:55 +0300, Avi Kivity wrote:
> On 09/23/2009 12:20 PM, Mike Galbraith wrote:
> >
> > Yup, brown baggie variety.  Oh darn.
> >
> > perf_counter tools: fix brown baggie module symbol loading bug.
> >
> > If there are no modules currently loaded, or the last module scanned is not
> > loaded, dso__load_modules() steps on the value from dso__load_vmlinux(), so
> > we happily load the kallsyms symbols on top of what we've already loaded.
> >
> > Fix that such that the total count of symbols loaded is returned.  Should
> > module symbol load fail after parsing of vmlinux, is's a hard failure, so
> > do not silently fall-back to kallsyms.
> >
> >    
> 
> Still fails, but differently.  Now 'annotate -k ... -m -v -v' doesn't 
> list vmx_vcpu_run at all, even though it's prominent in 'perf top'.
> 
> In addition to applying your patch I've merged current linus, so that 
> may have introduced the problem.
> 
> If I don't supply -k -m, I get
> 
> $ perf annotate  -v -v vmx_vcpu_run | grep vmx_vcpu
> new symbol: ffffffffa006f596 [0000dead]: vmx_vcpu_run    [kvm_intel], 
> hist: (nil), obj_start: (nil)
> new symbol: ffffffffa007025f [0000dead]: vmx_vcpu_put    [kvm_intel], 
> hist: (nil), obj_start: (nil)
> new symbol: ffffffffa0070bf6 [0000dead]: vmx_vcpu_load    [kvm_intel], 
> hist: (nil), obj_start: (nil)
> new symbol: ffffffffa0070d99 [0000dead]: vmx_vcpu_reset    [kvm_intel], 
> hist: (nil), obj_start: (nil)
> ffffffffa006f596-ffffffffa006fb73 vmx_vcpu_run    [kvm_intel]
>   ffffffffa007025f-ffffffffa007026e vmx_vcpu_put    [kvm_intel]
>   ffffffffa0070bf6-ffffffffa0070d98 vmx_vcpu_load    [kvm_intel]
>   ffffffffa0070d99-ffffffffa0071191 vmx_vcpu_reset    [kvm_intel]
>   Error: symbol 'vmx_vcpu_run' not present amongst the samples.

Yeah, I saw oddities with your config.  I'm rebuilding your config now
with more modules to do more testing.

	-Mike

Re: [patch] Re: [perf] Finding uninstalled modules Was Re: mailing list for trace users

[Mike Galbraith]

On Wed, 2009-09-23 at 12:55 +0300, Avi Kivity wrote:
> On 09/23/2009 12:20 PM, Mike Galbraith wrote:
> >
> > Yup, brown baggie variety.  Oh darn.
> >
> > perf_counter tools: fix brown baggie module symbol loading bug.
> >
> > If there are no modules currently loaded, or the last module scanned is not
> > loaded, dso__load_modules() steps on the value from dso__load_vmlinux(), so
> > we happily load the kallsyms symbols on top of what we've already loaded.
> >
> > Fix that such that the total count of symbols loaded is returned.  Should
> > module symbol load fail after parsing of vmlinux, is's a hard failure, so
> > do not silently fall-back to kallsyms.
> >
> >    
> 
> Still fails, but differently.  Now 'annotate -k ... -m -v -v' doesn't 
> list vmx_vcpu_run at all, even though it's prominent in 'perf top'.

Hm.  I just did a record, then report with and without -k -m, and now
get identical reports with your config (plus some more modules) here.

> In addition to applying your patch I've merged current linus, so that 
> may have introduced the problem.

I'm testing with freshly pulled and built tip+patch
.
> If I don't supply -k -m, I get
> 
> $ perf annotate  -v -v vmx_vcpu_run | grep vmx_vcpu
> new symbol: ffffffffa006f596 [0000dead]: vmx_vcpu_run    [kvm_intel], 
> hist: (nil), obj_start: (nil)
> new symbol: ffffffffa007025f [0000dead]: vmx_vcpu_put    [kvm_intel], 
> hist: (nil), obj_start: (nil)
> new symbol: ffffffffa0070bf6 [0000dead]: vmx_vcpu_load    [kvm_intel], 
> hist: (nil), obj_start: (nil)
> new symbol: ffffffffa0070d99 [0000dead]: vmx_vcpu_reset    [kvm_intel], 
> hist: (nil), obj_start: (nil)
> ffffffffa006f596-ffffffffa006fb73 vmx_vcpu_run    [kvm_intel]
>   ffffffffa007025f-ffffffffa007026e vmx_vcpu_put    [kvm_intel]
>   ffffffffa0070bf6-ffffffffa0070d98 vmx_vcpu_load    [kvm_intel]
>   ffffffffa0070d99-ffffffffa0071191 vmx_vcpu_reset    [kvm_intel]
>   Error: symbol 'vmx_vcpu_run' not present amongst the samples.

You can't annotate module symbols without -m.  If you don't provide -k,
annotate will look for vmlinux in cwd.  If there's one there, it'll
silently parse it.  For a module symbol, no -m means no symbol.

marge:/root/tmp # perf annotate -v -v -m ext3_mark_iloc_dirty 2>&1| grep ext3_mark_iloc_dirty
new symbol: ffffffffa005f8cf [0000dead]: ext3_mark_iloc_dirty   [ext3], hist: (nil), obj_start: (nil)
 ffffffffa005f8cf-ffffffffa005fc20 ext3_mark_iloc_dirty [ext3]
 Error: symbol 'ext3_mark_iloc_dirty' not present amongst the samples.

With -k -m it works here.

marge:/root/tmp # perf annotate -v -v -k vmlinux.avi -m ext3_mark_iloc_dirty 2>&1| grep ext3_mark_iloc_dirty
new symbol: ffffffffa005f8cf [00000352]: ext3_mark_iloc_dirty, hist: (nil), obj_start: 0x38cf
 ffffffffa005f8cf-ffffffffa005fc20 ext3_mark_iloc_dirty         [ext3]
annotating [0x22c10f0]                       [kernel] : [0x25821c0]           ext3_mark_iloc_dirty
         :      00000000000038cf <ext3_mark_iloc_dirty>:
         :      int ext3_mark_iloc_dirty(handle_t *handle,
    0.00 :          38e0:       e8 00 00 00 00          callq  38e5 <ext3_mark_iloc_dirty+0x16>
    5.93 :          392a:       74 1d                   je     3949 <ext3_mark_iloc_dirty+0x7a>
    0.02 :          394c:       e8 00 00 00 00          callq  3951 <ext3_mark_iloc_dirty+0x82>
    1.39 :          3975:       75 2a                   jne    39a1 <ext3_mark_iloc_dirty+0xd2>
    0.88 :          3989:       75 41                   jne    39cc <ext3_mark_iloc_dirty+0xfd>
    0.32 :          399f:       eb 37                   jmp    39d8 <ext3_mark_iloc_dirty+0x109>
    0.00 :          39a8:       74 06                   je     39b0 <ext3_mark_iloc_dirty+0xe1>
    0.00 :          39aa:       8b 15 00 00 00 00       mov    0x0(%rip),%edx        # 39b0 <ext3_mark_iloc_dirty+0xe1>
    0.00 :          39c0:       74 06                   je     39c8 <ext3_mark_iloc_dirty+0xf9>
    0.00 :          39c2:       8b 15 00 00 00 00       mov    0x0(%rip),%edx        # 39c8 <ext3_mark_iloc_dirty+0xf9>
    0.00 :          3a3c:       74 0d                   je     3a4b <ext3_mark_iloc_dirty+0x17c>
    0.00 :          3a46:       e9 ab 00 00 00          jmpq   3af6 <ext3_mark_iloc_dirty+0x227>
    0.24 :          3a60:       0f 86 90 00 00 00       jbe    3af6 <ext3_mark_iloc_dirty+0x227>
    0.04 :          3a7d:       74 06                   je     3a85 <ext3_mark_iloc_dirty+0x1b6>
    0.11 :          3a83:       75 71                   jne    3af6 <ext3_mark_iloc_dirty+0x227>
    0.00 :          3a89:       e8 00 00 00 00          callq  3a8e <ext3_mark_iloc_dirty+0x1bf>
    0.00 :          3aa4:       e8 00 00 00 00          callq  3aa9 <ext3_mark_iloc_dirty+0x1da>
    0.00 :          3aae:       0f 85 f5 00 00 00       jne    3ba9 <ext3_mark_iloc_dirty+0x2da>
    0.00 :          3ab7:       e8 00 00 00 00          callq  3abc <ext3_mark_iloc_dirty+0x1ed>
    0.00 :          3ae9:       e8 00 00 00 00          callq  3aee <ext3_mark_iloc_dirty+0x21f>
    0.00 :          3af1:       e9 26 fe ff ff          jmpq   391c <ext3_mark_iloc_dirty+0x4d>
    0.02 :          3b14:       74 09                   je     3b1f <ext3_mark_iloc_dirty+0x250>
    0.38 :          3b1d:       75 3e                   jne    3b5d <ext3_mark_iloc_dirty+0x28e>
    0.00 :          3b2f:       0f 87 ba 00 00 00       ja     3bef <ext3_mark_iloc_dirty+0x320>
    0.00 :          3b41:       0f 87 a8 00 00 00       ja     3bef <ext3_mark_iloc_dirty+0x320>
    0.00 :          3b5b:       eb 12                   jmp    3b6f <ext3_mark_iloc_dirty+0x2a0>
    6.64 :          3b6d:       75 ee                   jne    3b5d <ext3_mark_iloc_dirty+0x28e>
    0.04 :          3b77:       74 07                   je     3b80 <ext3_mark_iloc_dirty+0x2b1>
    0.09 :          3b84:       e8 00 00 00 00          callq  3b89 <ext3_mark_iloc_dirty+0x2ba>
    0.00 :          3b98:       e8 00 00 00 00          callq  3b9d <ext3_mark_iloc_dirty+0x2ce>
    0.00 :          3bb5:       74 17                   je     3bce <ext3_mark_iloc_dirty+0x2ff>
    0.00 :          3bc9:       e8 00 00 00 00          callq  3bce <ext3_mark_iloc_dirty+0x2ff>
    0.00 :          3c1c:       e9 4e ff ff ff          jmpq   3b6f <ext3_mark_iloc_dirty+0x2a0>

Re: [patch] Re: [perf] Finding uninstalled modules Was Re: mailing list for trace users

[Mike Galbraith]

On Wed, 2009-09-23 at 13:32 +0200, Mike Galbraith wrote:
> On Wed, 2009-09-23 at 12:55 +0300, Avi Kivity wrote:
> > On 09/23/2009 12:20 PM, Mike Galbraith wrote:
> > >
> > > Yup, brown baggie variety.  Oh darn.
> > >
> > > perf_counter tools: fix brown baggie module symbol loading bug.
> > >
> > > If there are no modules currently loaded, or the last module scanned is not
> > > loaded, dso__load_modules() steps on the value from dso__load_vmlinux(), so
> > > we happily load the kallsyms symbols on top of what we've already loaded.
> > >
> > > Fix that such that the total count of symbols loaded is returned.  Should
> > > module symbol load fail after parsing of vmlinux, is's a hard failure, so
> > > do not silently fall-back to kallsyms.
> > >
> > >    
> > 
> > Still fails, but differently.  Now 'annotate -k ... -m -v -v' doesn't 
> > list vmx_vcpu_run at all, even though it's prominent in 'perf top'.
> 
> Hm.  I just did a record, then report with and without -k -m, and now
> get identical reports with your config (plus some more modules) here.

With your config, I can't even get my e1000 flood pinging friztbox to
show up in perf top without a 70 line monitor.   Mondo overhead.

------------------------------------------------------------------------------
   PerfTop:    3860 irqs/sec  kernel:94.0% [10000Hz instructions],  (all, cpu: 0)
------------------------------------------------------------------------------

             samples    pcnt   kernel function
             _______   _____   _______________

             1702.00 -  5.5% : _spin_lock_irqsave
             1659.00 -  5.3% : add_preempt_count
             1516.00 -  4.9% : sub_preempt_count
             1330.00 -  4.3% : _spin_unlock_irqrestore
             1132.00 -  3.6% : debug_smp_processor_id
              844.00 -  2.7% : find_busiest_group
              639.00 -  2.1% : schedule
              629.00 -  2.0% : get_parent_ip
              586.00 -  1.9% : test_ti_thread_flag
.....

              310.00 -  0.3% : sys_recvmsg
              303.00 -  0.3% : e1000_clean_rx_irq       [e1000e]
              302.00 -  0.3% : raw_sendmsg
              300.00 -  0.3% : e1000_xmit_frame [e1000e]

Re: [patch] Re: [perf] Finding uninstalled modules Was Re: mailing list for trace users

[Avi Kivity]

On 09/23/2009 02:31 PM, Mike Galbraith wrote:
>
> You can't annotate module symbols without -m.  If you don't provide -k,
> annotate will look for vmlinux in cwd.  If there's one there, it'll
> silently parse it.  For a module symbol, no -m means no symbol.
>
> marge:/root/tmp # perf annotate -v -v -m ext3_mark_iloc_dirty 2>&1| grep ext3_mark_iloc_dirty
> new symbol: ffffffffa005f8cf [0000dead]: ext3_mark_iloc_dirty   [ext3], hist: (nil), obj_start: (nil)
>   ffffffffa005f8cf-ffffffffa005fc20 ext3_mark_iloc_dirty [ext3]
>   Error: symbol 'ext3_mark_iloc_dirty' not present amongst the samples.
>
> With -k -m it works here.
>
>    


Not for me.  'perf report', for example, shows

     63.08%  qemu-system-x86  
[kernel]                                              [k] packet_exit
      4.71%  qemu-system-x86  
[kernel]                                              [k] hpet_next_event
      4.38%             init  
[kernel]                                              [k] 
mwait_idle_with_hints

While 'perf top' still shows vmx_vcpu_run.

-- 
error compiling committee.c: too many arguments to function

Re: [patch] Re: [perf] Finding uninstalled modules Was Re: mailing list for trace users

[Mike Galbraith]

On Wed, 2009-09-23 at 15:58 +0300, Avi Kivity wrote:

> Not for me.  'perf report', for example, shows
> 
>      63.08%  qemu-system-x86  
> [kernel]                                              [k] packet_exit
>       4.71%  qemu-system-x86  
> [kernel]                                              [k] hpet_next_event
>       4.38%             init  
> [kernel]                                              [k] 
> mwait_idle_with_hints
> 
> While 'perf top' still shows vmx_vcpu_run.

Ok, I'll find a nice repeatable module hitting load I can profile with
top/report and compare results.  Maybe something will fall out.

	-Mike

Re: [patch] Re: [perf] Finding uninstalled modules Was Re: mailing list for trace users

[Avi Kivity]

On 09/23/2009 03:58 PM, Avi Kivity wrote:On 09/23/2009 03:58 PM, Avi 
Kivity wrote:
> Not for me.  'perf report', for example, shows
>
>     63.08%  qemu-system-x86  
> [kernel]                                              [k] packet_exit
>      4.71%  qemu-system-x86  
> [kernel]                                              [k] hpet_next_event
>      4.38%             init  
> [kernel]                                              [k] 
> mwait_idle_with_hints
>
> While 'perf top' still shows vmx_vcpu_run.
>

strace says:

    getcwd("/home/avi/kvm/linux-2.6"..., 4096) = 24
    ...
    [no chdir]
    ...
    open("kernel/arch/x86/kvm/kvm-intel.ko", O_RDONLY) = -1 ENOENT (No 
such file or directory)

That "kernel/" looks like it was meant for /lib/modules, not a kernel 
tree.  If I run 'perf report' from /lib/modules/2.6.31 I see vmx_vcpu_run.

-- 
error compiling committee.c: too many arguments to function

Re: [patch] Re: [perf] Finding uninstalled modules Was Re: mailing list for trace users

[Mike Galbraith]

On Wed, 2009-09-23 at 16:10 +0300, Avi Kivity wrote:
> On 09/23/2009 03:58 PM, Avi Kivity wrote:On 09/23/2009 03:58 PM, Avi 
> Kivity wrote:
> > Not for me.  'perf report', for example, shows
> >
> >     63.08%  qemu-system-x86  
> > [kernel]                                              [k] packet_exit
> >      4.71%  qemu-system-x86  
> > [kernel]                                              [k] hpet_next_event
> >      4.38%             init  
> > [kernel]                                              [k] 
> > mwait_idle_with_hints
> >
> > While 'perf top' still shows vmx_vcpu_run.
> >
> 
> strace says:
> 
>     getcwd("/home/avi/kvm/linux-2.6"..., 4096) = 24
>     ...
>     [no chdir]
>     ...
>     open("kernel/arch/x86/kvm/kvm-intel.ko", O_RDONLY) = -1 ENOENT (No 
> such file or directory)

*blink*

> That "kernel/" looks like it was meant for /lib/modules, not a kernel 
> tree.  If I run 'perf report' from /lib/modules/2.6.31 I see vmx_vcpu_run.

So I need what's in your modules.dep to figure out where the rest of the
path went.

/lib/modules/2.6.32-tip-smp/kernel/arch/x86/kvm/kvm-intel.ko

static int mod_dso__load_module_paths(struct mod_dso *self)
{
        struct utsname uts;
        int count = 0, len;
        char *line = NULL;
        FILE *file;
        char *path;
        size_t n;

        if (uname(&uts) < 0)
                goto out_failure;

        len = strlen("/lib/modules/");
        len += strlen(uts.release);
        len += strlen("/modules.dep");

        path = calloc(1, len);
        if (path == NULL)
                goto out_failure;

        strcat(path, "/lib/modules/");
        strcat(path, uts.release);
        strcat(path, "/modules.dep");

        file = fopen(path, "r");

Re: [patch] Re: [perf] Finding uninstalled modules Was Re: mailing list for trace users

[Avi Kivity]

On 09/23/2009 04:50 PM, Mike Galbraith wrote:
> On Wed, 2009-09-23 at 16:10 +0300, Avi Kivity wrote:
>    
>> On 09/23/2009 03:58 PM, Avi Kivity wrote:On 09/23/2009 03:58 PM, Avi
>> Kivity wrote:
>>      
>>> Not for me.  'perf report', for example, shows
>>>
>>>      63.08%  qemu-system-x86
>>> [kernel]                                              [k] packet_exit
>>>       4.71%  qemu-system-x86
>>> [kernel]                                              [k] hpet_next_event
>>>       4.38%             init
>>> [kernel]                                              [k]
>>> mwait_idle_with_hints
>>>
>>> While 'perf top' still shows vmx_vcpu_run.
>>>
>>>        
>> strace says:
>>
>>      getcwd("/home/avi/kvm/linux-2.6"..., 4096) = 24
>>      ...
>>      [no chdir]
>>      ...
>>      open("kernel/arch/x86/kvm/kvm-intel.ko", O_RDONLY) = -1 ENOENT (No
>> such file or directory)
>>      
> *blink*
>
>    
>> That "kernel/" looks like it was meant for /lib/modules, not a kernel
>> tree.  If I run 'perf report' from /lib/modules/2.6.31 I see vmx_vcpu_run.
>>      
> So I need what's in your modules.dep to figure out where the rest of the
> path went.
>
>    

Mine says:

   kernel/arch/x86/kvm/kvm.ko:
   kernel/arch/x86/kvm/kvm-intel.ko: kernel/arch/x86/kvm/kvm.ko

Which is reasonable for /lib/modules/2.6.31, not for a source directory.

> /lib/modules/2.6.32-tip-smp/kernel/arch/x86/kvm/kvm-intel.ko
>
> static int mod_dso__load_module_paths(struct mod_dso *self)
> {
>          struct utsname uts;
>          int count = 0, len;
>          char *line = NULL;
>          FILE *file;
>          char *path;
>          size_t n;
>
>          if (uname(&uts)<  0)
>                  goto out_failure;
>
>          len = strlen("/lib/modules/");
>          len += strlen(uts.release);
>          len += strlen("/modules.dep");
>
>          path = calloc(1, len);
>    

len + 1

>          if (path == NULL)
>                  goto out_failure;
>
>          strcat(path, "/lib/modules/");
>          strcat(path, uts.release);
>          strcat(path, "/modules.dep");
>
>          file = fopen(path, "r");
>
>    

-- 
error compiling committee.c: too many arguments to function

Re: [patch] Re: [perf] Finding uninstalled modules Was Re: mailing list for trace users

[Mike Galbraith]

On Wed, 2009-09-23 at 17:00 +0300, Avi Kivity wrote:
> On 09/23/2009 04:50 PM, Mike Galbraith wrote:
>     
> > So I need what's in your modules.dep to figure out where the rest of the
> > path went.
> >
> >    
> 
> Mine says:
> 
>    kernel/arch/x86/kvm/kvm.ko:
>    kernel/arch/x86/kvm/kvm-intel.ko: kernel/arch/x86/kvm/kvm.ko
> 
> Which is reasonable for /lib/modules/2.6.31, not for a source directory.

Yup, there's the problem.  The code assumes that the path to modules.dep
and the path stored in modules.dep will agree.

Wants some robustification.

	-Mike

Re: [patch] Re: [perf] Finding uninstalled modules Was Re: mailing list for trace users

[Avi Kivity]

On 09/23/2009 05:09 PM, Mike Galbraith wrote:
>
>> Mine says:
>>
>>     kernel/arch/x86/kvm/kvm.ko:
>>     kernel/arch/x86/kvm/kvm-intel.ko: kernel/arch/x86/kvm/kvm.ko
>>
>> Which is reasonable for /lib/modules/2.6.31, not for a source directory.
>>      
> Yup, there's the problem.  The code assumes that the path to modules.dep
> and the path stored in modules.dep will agree.
>
> Wants some robustification.
>    

The trace shows modules.dep was picked up from /lib/modules/2.6.31, so 
if it continued that and picked up the modules from the same place, 
everything would work.  Why does it try to pick up the modules from cwd?

-- 
error compiling committee.c: too many arguments to function

Re: [patch] Re: [perf] Finding uninstalled modules Was Re: mailing list for trace users

[Mike Galbraith]

On Wed, 2009-09-23 at 17:39 +0300, Avi Kivity wrote:
> On 09/23/2009 05:09 PM, Mike Galbraith wrote:
> >
> >> Mine says:
> >>
> >>     kernel/arch/x86/kvm/kvm.ko:
> >>     kernel/arch/x86/kvm/kvm-intel.ko: kernel/arch/x86/kvm/kvm.ko
> >>
> >> Which is reasonable for /lib/modules/2.6.31, not for a source directory.
> >>      
> > Yup, there's the problem.  The code assumes that the path to modules.dep
> > and the path stored in modules.dep will agree.
> >
> > Wants some robustification.
> >    
> 
> The trace shows modules.dep was picked up from /lib/modules/2.6.31, so 
> if it continued that and picked up the modules from the same place, 
> everything would work.  Why does it try to pick up the modules from cwd?

I don't know where that cwd is coming from off the top of my head.

The symbol loading code reads the path stored in modules.dep, and chops
at ':'.

> >>      open("kernel/arch/x86/kvm/kvm-intel.ko", O_RDONLY) = -1 ENOENT (No
...
> Mine says:
> 
>    kernel/arch/x86/kvm/kvm.ko:
>    kernel/arch/x86/kvm/kvm-intel.ko: kernel/arch/x86/kvm/kvm.ko

	-Mike

Re: [patch] Re: [perf] Finding uninstalled modules Was Re: mailing list for trace users

[Avi Kivity]

On 09/23/2009 05:52 PM, Mike Galbraith wrote:
>
>> The trace shows modules.dep was picked up from /lib/modules/2.6.31, so
>> if it continued that and picked up the modules from the same place,
>> everything would work.  Why does it try to pick up the modules from cwd?
>>      
> I don't know where that cwd is coming from off the top of my head.
>
> The symbol loading code reads the path stored in modules.dep, and chops
> at ':'.
>    

Well, you don't need to do anything to open a file from cwd: that's the 
default.  You need to actively prepend /lib/modules/blah to get it to 
load from the correct location.  What I don't understand is why it is 
only hitting me (esp. as it used to work).

-- 
error compiling committee.c: too many arguments to function

Re: [patch] Re: [perf] Finding uninstalled modules Was Re: mailing list for trace users

[Mike Galbraith]

On Wed, 2009-09-23 at 17:56 +0300, Avi Kivity wrote:
> On 09/23/2009 05:52 PM, Mike Galbraith wrote:
> >
> >> The trace shows modules.dep was picked up from /lib/modules/2.6.31, so
> >> if it continued that and picked up the modules from the same place,
> >> everything would work.  Why does it try to pick up the modules from cwd?
> >>      
> > I don't know where that cwd is coming from off the top of my head.
> >
> > The symbol loading code reads the path stored in modules.dep, and chops
> > at ':'.
> >    
> 
> Well, you don't need to do anything to open a file from cwd: that's the 
> default.  You need to actively prepend /lib/modules/blah to get it to 
> load from the correct location.  What I don't understand is why it is 
> only hitting me (esp. as it used to work).

If your modules.dep, always was missing the /lib/modules/blah bit, it
never worked for you.  I wrote the thing with the rash assumption that
it always contained full path like mine does :)

So, I just need to check whether it's full path or not, prepend or take
the path as is, and hope there aren't several other ways to get screwed
up by modules.dep content.

Too bad the kernel doesn't store the path in /sys/modules/bla/path.
That would be nicer than rummaging around in a mutable file.

	-Mike

Re: [patch] Re: [perf] Finding uninstalled modules Was Re: mailing list for trace users

[Avi Kivity]

On 09/23/2009 06:05 PM, Mike Galbraith wrote:
>
>> Well, you don't need to do anything to open a file from cwd: that's the
>> default.  You need to actively prepend /lib/modules/blah to get it to
>> load from the correct location.  What I don't understand is why it is
>> only hitting me (esp. as it used to work).
>>      
> If your modules.dep, always was missing the /lib/modules/blah bit, it
> never worked for you.  I wrote the thing with the rash assumption that
> it always contained full path like mine does :)
>
>    

Ah, changed there, I missed that part earlier, sorry.  So it's probably 
a change in modules.dep generation instead of tools/perf.

> So, I just need to check whether it's full path or not, prepend or take
> the path as is, and hope there aren't several other ways to get screwed
> up by modules.dep content.
>    

Maybe we should fix that then (though I prefer relative paths myself).

> Too bad the kernel doesn't store the path in /sys/modules/bla/path.
> That would be nicer than rummaging around in a mutable file.
>    

But then you rely on a running kernel.  Be nice to be able to ship 
perf.data.

-- 

error compiling committee.c: too many arguments to function

Re: [patch] Re: [perf] Finding uninstalled modules Was Re: mailing list for trace users

[Mike Galbraith]

On Wed, 2009-09-23 at 18:09 +0300, Avi Kivity wrote:
> On 09/23/2009 06:05 PM, Mike Galbraith wrote:
> >
> >> Well, you don't need to do anything to open a file from cwd: that's the
> >> default.  You need to actively prepend /lib/modules/blah to get it to
> >> load from the correct location.  What I don't understand is why it is
> >> only hitting me (esp. as it used to work).
> >>      
> > If your modules.dep, always was missing the /lib/modules/blah bit, it
> > never worked for you.  I wrote the thing with the rash assumption that
> > it always contained full path like mine does :)
> >
> >    
> 
> Ah, changed there, I missed that part earlier, sorry.  So it's probably 
> a change in modules.dep generation instead of tools/perf.
> 
> > So, I just need to check whether it's full path or not, prepend or take
> > the path as is, and hope there aren't several other ways to get screwed
> > up by modules.dep content.
> >    
> 
> Maybe we should fix that then (though I prefer relative paths myself).

I'll do the check for relative vs full thing, and a bail noisily if it
finds something else.  (later i guess, wife is giving me the evil eye;)

> > Too bad the kernel doesn't store the path in /sys/modules/bla/path.
> > That would be nicer than rummaging around in a mutable file.
> >    
> 
> But then you rely on a running kernel.  Be nice to be able to ship 
> perf.data.

Long term, all pertinent data should go into a shippable perf.data.

	-Mike

Re: [patch] Re: [perf] Finding uninstalled modules Was Re: mailing list for trace users

[Mike Galbraith]

On Wed, 2009-09-23 at 18:09 +0300, Avi Kivity wrote:
> On 09/23/2009 06:05 PM, Mike Galbraith wrote:
> >
> > So, I just need to check whether it's full path or not, prepend or take
> > the path as is, and hope there aren't several other ways to get screwed
> > up by modules.dep content.
> >    
> 
> Maybe we should fix that then (though I prefer relative paths myself).

Ok, the below should (knock wood) handle your relative paths.  May look
a little overly paranoid, but it's not... they _are_ out to get me ;-)

perf_counter tools: handle relative paths while loading module symbols.

Inform util/module.c::mod_dso__load_module_paths() that relative paths do exist
in some modules.dep, and make it fail noisily should it encounter a path that it
doesn't understand, or a module it cannot open.

Signed-off-by: Mike Galbraith <efault@gmx.de>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Reported-by: Avi Kivity <avi@redhat.com>
LKML-Reference: <new-submission>

---
 tools/perf/util/module.c |  100 +++++++++++++++++++++++++++++++----------------
 1 file changed, 67 insertions(+), 33 deletions(-)

Index: linux-2.6/tools/perf/util/module.c
===================================================================
--- linux-2.6.orig/tools/perf/util/module.c
+++ linux-2.6/tools/perf/util/module.c
@@ -4,6 +4,7 @@
 #include "module.h"
 
 #include <libelf.h>
+#include <libgen.h>
 #include <gelf.h>
 #include <elf.h>
 #include <dirent.h>
@@ -409,35 +410,40 @@ out_failure:
 static int mod_dso__load_module_paths(struct mod_dso *self)
 {
 	struct utsname uts;
-	int count = 0, len;
+	int count = 0, len, err = -1;
 	char *line = NULL;
 	FILE *file;
-	char *path;
+	char *dpath, *dir;
 	size_t n;
 
 	if (uname(&uts) < 0)
-		goto out_failure;
+		return err;
 
 	len = strlen("/lib/modules/");
 	len += strlen(uts.release);
 	len += strlen("/modules.dep");
 
-	path = calloc(1, len);
-	if (path == NULL)
-		goto out_failure;
-
-	strcat(path, "/lib/modules/");
-	strcat(path, uts.release);
-	strcat(path, "/modules.dep");
+	dpath = calloc(1, len);
+	if (dpath == NULL)
+		return err;
+
+	strcat(dpath, "/lib/modules/");
+	strcat(dpath, uts.release);
+	strcat(dpath, "/modules.dep");
 
-	file = fopen(path, "r");
-	free(path);
+	file = fopen(dpath, "r");
 	if (file == NULL)
 		goto out_failure;
 
+	dir = dirname(dpath);
+	if (!dir)
+		goto out_failure;
+	strcat(dir, "/");
+
 	while (!feof(file)) {
-		char *name, *tmp;
 		struct module *module;
+		char *name, *path, *tmp;
+		FILE *modfile;
 		int line_len;
 
 		line_len = getline(&line, &n, file);
@@ -445,44 +451,65 @@ static int mod_dso__load_module_paths(st
 			break;
 
 		if (!line)
-			goto out_failure;
+			break;
 
 		line[--line_len] = '\0'; /* \n */
 
-		path = strtok(line, ":");
+		path = strchr(line, ':');
 		if (!path)
-			goto out_failure;
+			break;
+		*path = '\0';
 
-		name = strdup(path);
-		name = strtok(name, "/");
+		path = strdup(line);
+		if (!path)
+			break;
 
-		tmp = name;
+		if (!strstr(path, dir)) {
+			if (strncmp(path, "kernel/", 7))
+				break;
+
+			free(path);
+			path = calloc(1, strlen(dir) + strlen(line) + 1);
+			if (!path)
+				break;
+			strcat(path, dir);
+			strcat(path, line);
+		}
+
+		modfile = fopen(path, "r");
+		if (modfile == NULL)
+			break;
+		fclose(modfile);
+
+		name = strdup(path);
+		if (!name)
+			break;
+		tmp = name = strtok(name, "/");
 
 		while (tmp) {
 			tmp = strtok(NULL, "/");
 			if (tmp)
 				name = tmp;
 		}
+
 		name = strsep(&name, ".");
+		if (!name)
+			break;
 
-		/* Quirk: replace '-' with '_' in sound modules */
+		/* Quirk: replace '-' with '_' in all modules */
 		for (len = strlen(name); len; len--) {
 			if (*(name+len) == '-')
 				*(name+len) = '_';
 		}
 
 		module = module__new(name, path);
-		if (!module) {
-			fprintf(stderr, "load_module_paths: allocation error\n");
-			goto out_failure;
-		}
+		if (!module)
+			break;
 		mod_dso__insert_module(self, module);
 
 		module->sections = sec_dso__new_dso("sections");
-		if (!module->sections) {
-			fprintf(stderr, "load_module_paths: allocation error\n");
-			goto out_failure;
-		}
+		if (!module->sections)
+			break;
 
 		module->active = mod_dso__load_sections(module);
 
@@ -490,13 +517,20 @@ static int mod_dso__load_module_paths(st
 			count++;
 	}
 
-	free(line);
-	fclose(file);
-
-	return count;
+	if(feof(file))
+		err = count;
+	else
+		fprintf(stderr, "load_module_paths: modules.dep parse failure!\n");
 
 out_failure:
-	return -1;
+	if (dpath)
+		free(dpath);
+	if (file)
+		fclose(file);
+	if (line)
+		free(line);
+
+	return err;
 }
 
 int mod_dso__load_modules(struct mod_dso *dso)

[tip:perf/urgent] perf tools: Handle relative paths while loading module symbols

[tip-bot for Mike Galbraith]

Commit-ID:  dd906a0fe8d78b925702cd3916a65f34dfdfc011
Gitweb:     http://git.kernel.org/tip/dd906a0fe8d78b925702cd3916a65f34dfdfc011
Author:     Mike Galbraith <efault@gmx.de>
AuthorDate: Thu, 24 Sep 2009 10:07:08 +0200
Committer:  Ingo Molnar <mingo@elte.hu>
CommitDate: Thu, 24 Sep 2009 11:40:35 +0200

perf tools: Handle relative paths while loading module symbols

Inform util/module.c::mod_dso__load_module_paths() that relative
paths do exist in some modules.dep, and make it fail noisily should
it encounter a path that it doesn't understand, or a module it
cannot open.

Reported-by: Avi Kivity <avi@redhat.com>
Signed-off-by: Mike Galbraith <efault@gmx.de>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: rostedt@goodmis.org
Cc: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Masami Hiramatsu <mhiramat@redhat.com>
LKML-Reference: <1253779628.10513.8.camel@marge.simson.net>
Signed-off-by: Ingo Molnar <mingo@elte.hu>


---
 tools/perf/util/module.c |   96 +++++++++++++++++++++++++++++++--------------
 1 files changed, 66 insertions(+), 30 deletions(-)

diff --git a/tools/perf/util/module.c b/tools/perf/util/module.c
index 3d567fe..8f81622 100644
--- a/tools/perf/util/module.c
+++ b/tools/perf/util/module.c
@@ -4,6 +4,7 @@
 #include "module.h"
 
 #include <libelf.h>
+#include <libgen.h>
 #include <gelf.h>
 #include <elf.h>
 #include <dirent.h>
@@ -409,35 +410,40 @@ out_failure:
 static int mod_dso__load_module_paths(struct mod_dso *self)
 {
 	struct utsname uts;
-	int count = 0, len;
+	int count = 0, len, err = -1;
 	char *line = NULL;
 	FILE *file;
-	char *path;
+	char *dpath, *dir;
 	size_t n;
 
 	if (uname(&uts) < 0)
-		goto out_failure;
+		return err;
 
 	len = strlen("/lib/modules/");
 	len += strlen(uts.release);
 	len += strlen("/modules.dep");
 
-	path = calloc(1, len);
-	if (path == NULL)
-		goto out_failure;
+	dpath = calloc(1, len);
+	if (dpath == NULL)
+		return err;
 
-	strcat(path, "/lib/modules/");
-	strcat(path, uts.release);
-	strcat(path, "/modules.dep");
+	strcat(dpath, "/lib/modules/");
+	strcat(dpath, uts.release);
+	strcat(dpath, "/modules.dep");
 
-	file = fopen(path, "r");
-	free(path);
+	file = fopen(dpath, "r");
 	if (file == NULL)
 		goto out_failure;
 
+	dir = dirname(dpath);
+	if (!dir)
+		goto out_failure;
+	strcat(dir, "/");
+
 	while (!feof(file)) {
-		char *name, *tmp;
 		struct module *module;
+		char *name, *path, *tmp;
+		FILE *modfile;
 		int line_len;
 
 		line_len = getline(&line, &n, file);
@@ -445,17 +451,41 @@ static int mod_dso__load_module_paths(struct mod_dso *self)
 			break;
 
 		if (!line)
-			goto out_failure;
+			break;
 
 		line[--line_len] = '\0'; /* \n */
 
-		path = strtok(line, ":");
+		path = strchr(line, ':');
+		if (!path)
+			break;
+		*path = '\0';
+
+		path = strdup(line);
 		if (!path)
-			goto out_failure;
+			break;
+
+		if (!strstr(path, dir)) {
+			if (strncmp(path, "kernel/", 7))
+				break;
+
+			free(path);
+			path = calloc(1, strlen(dir) + strlen(line) + 1);
+			if (!path)
+				break;
+			strcat(path, dir);
+			strcat(path, line);
+		}
+
+		modfile = fopen(path, "r");
+		if (modfile == NULL)
+			break;
+		fclose(modfile);
 
 		name = strdup(path);
-		name = strtok(name, "/");
+		if (!name)
+			break;
 
+		name = strtok(name, "/");
 		tmp = name;
 
 		while (tmp) {
@@ -463,26 +493,25 @@ static int mod_dso__load_module_paths(struct mod_dso *self)
 			if (tmp)
 				name = tmp;
 		}
+
 		name = strsep(&name, ".");
+		if (!name)
+			break;
 
-		/* Quirk: replace '-' with '_' in sound modules */
+		/* Quirk: replace '-' with '_' in all modules */
 		for (len = strlen(name); len; len--) {
 			if (*(name+len) == '-')
 				*(name+len) = '_';
 		}
 
 		module = module__new(name, path);
-		if (!module) {
-			fprintf(stderr, "load_module_paths: allocation error\n");
-			goto out_failure;
-		}
+		if (!module)
+			break;
 		mod_dso__insert_module(self, module);
 
 		module->sections = sec_dso__new_dso("sections");
-		if (!module->sections) {
-			fprintf(stderr, "load_module_paths: allocation error\n");
-			goto out_failure;
-		}
+		if (!module->sections)
+			break;
 
 		module->active = mod_dso__load_sections(module);
 
@@ -490,13 +519,20 @@ static int mod_dso__load_module_paths(struct mod_dso *self)
 			count++;
 	}
 
-	free(line);
-	fclose(file);
-
-	return count;
+	if (feof(file))
+		err = count;
+	else
+		fprintf(stderr, "load_module_paths: modules.dep parsing failure!\n");
 
 out_failure:
-	return -1;
+	if (dpath)
+		free(dpath);
+	if (file)
+		fclose(file);
+	if (line)
+		free(line);
+
+	return err;
 }
 
 int mod_dso__load_modules(struct mod_dso *dso)

[tip:perf/urgent] perf tools: Fix module symbol loading bug

[tip-bot for Mike Galbraith]

Commit-ID:  508c4d0874acf8584787bbab7e4a3798e2834c1a
Gitweb:     http://git.kernel.org/tip/508c4d0874acf8584787bbab7e4a3798e2834c1a
Author:     Mike Galbraith <efault@gmx.de>
AuthorDate: Wed, 23 Sep 2009 11:20:58 +0200
Committer:  Ingo Molnar <mingo@elte.hu>
CommitDate: Wed, 23 Sep 2009 13:45:48 +0200

perf tools: Fix module symbol loading bug

Avi Kivity reported 'perf annotate' failures with modules, the
requested function was not annotated.

If there are no modules currently loaded, or the last module
scanned is not loaded, dso__load_modules() steps on the value from
dso__load_vmlinux(), so we happily load the kallsyms symbols on top
of what we've already loaded.

Fix that such that the total count of symbols loaded is returned.
Should module symbol load fail after parsing of vmlinux, is's a
hard failure, so do not silently fall-back to kallsyms.

Reported-by: Avi Kivity <avi@redhat.com>
Signed-off-by: Mike Galbraith <efault@gmx.de>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: rostedt@goodmis.org
Cc: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Masami Hiramatsu <mhiramat@redhat.com>
LKML-Reference: <1253697658.11461.36.camel@marge.simson.net>
Signed-off-by: Ingo Molnar <mingo@elte.hu>


---
 tools/perf/util/symbol.c |   17 +++++++++++++----
 1 files changed, 13 insertions(+), 4 deletions(-)

diff --git a/tools/perf/util/symbol.c b/tools/perf/util/symbol.c
index fd3d9c8..559fb06 100644
--- a/tools/perf/util/symbol.c
+++ b/tools/perf/util/symbol.c
@@ -833,7 +833,7 @@ int dso__load_modules(struct dso *self, symbol_filter_t filter, int v)
 	struct mod_dso *mods = mod_dso__new_dso("modules");
 	struct module *pos;
 	struct rb_node *next;
-	int err;
+	int err, count = 0;
 
 	err = mod_dso__load_modules(mods);
 
@@ -852,14 +852,16 @@ int dso__load_modules(struct dso *self, symbol_filter_t filter, int v)
 			break;
 
 		next = rb_next(&pos->rb_node);
+		count += err;
 	}
 
 	if (err < 0) {
 		mod_dso__delete_modules(mods);
 		mod_dso__delete_self(mods);
+		return err;
 	}
 
-	return err;
+	return count;
 }
 
 static inline void dso__fill_symbol_holes(struct dso *self)
@@ -913,8 +915,15 @@ int dso__load_kernel(struct dso *self, const char *vmlinux,
 
 	if (vmlinux) {
 		err = dso__load_vmlinux(self, vmlinux, filter, v);
-		if (err > 0 && use_modules)
-			err = dso__load_modules(self, filter, v);
+		if (err > 0 && use_modules) {
+			int syms = dso__load_modules(self, filter, v);
+
+			if (syms < 0) {
+				fprintf(stderr, "dso__load_modules failed!\n");
+				return syms;
+			}
+			err += syms;
+		}
 	}
 
 	if (err <= 0)

Re: mailing list for trace users

[David Miller]

From: Steven Rostedt <rostedt@goodmis.org>
Date: Wed, 16 Sep 2009 16:16:22 -0400

> What are people's thoughts about creating a linux-trace-users mailing
> list on vger.kernel.org?

I've created this list.

Re: mailing list for trace users

[Ingo Molnar]

* David Miller <davem@davemloft.net> wrote:

> From: Steven Rostedt <rostedt@goodmis.org>
> Date: Wed, 16 Sep 2009 16:16:22 -0400
> 
> > What are people's thoughts about creating a linux-trace-users mailing
> > list on vger.kernel.org?
> 
> I've created this list.

Could you please also create the linux-perf-users list, for perf events 
and the perf tool related user questions? (We have asked for this before 
but must have gotten lost somewhere)

Thanks a lot in advance,

	Ingo

Re: mailing list for trace users

[Masami Hiramatsu]

Ingo Molnar wrote:
> 
> * David Miller <davem@davemloft.net> wrote:
> 
>> From: Steven Rostedt <rostedt@goodmis.org>
>> Date: Wed, 16 Sep 2009 16:16:22 -0400
>>
>>> What are people's thoughts about creating a linux-trace-users mailing
>>> list on vger.kernel.org?
>>
>> I've created this list.
> 
> Could you please also create the linux-perf-users list, for perf events 
> and the perf tool related user questions? (We have asked for this before 
> but must have gotten lost somewhere)

I thought perf users also talk on the linux tracing list.
What will linux-tracing-users cover, only ftrace?

Thank you,

-- 
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America), Inc.
Software Solutions Division

e-mail: mhiramat@redhat.com

Re: mailing list for trace users

[Ingo Molnar]

* Masami Hiramatsu <mhiramat@redhat.com> wrote:

> Ingo Molnar wrote:
> > 
> > * David Miller <davem@davemloft.net> wrote:
> > 
> >> From: Steven Rostedt <rostedt@goodmis.org>
> >> Date: Wed, 16 Sep 2009 16:16:22 -0400
> >>
> >>> What are people's thoughts about creating a linux-trace-users mailing
> >>> list on vger.kernel.org?
> >>
> >> I've created this list.
> > 
> > Could you please also create the linux-perf-users list, for perf events 
> > and the perf tool related user questions? (We have asked for this before 
> > but must have gotten lost somewhere)
> 
> I thought perf users also talk on the linux tracing list.

No, there's a lot of non-tracing aspects of performance events. It does 
profiling, counting, latency analysis, etc.

> What will linux-tracing-users cover, only ftrace?

It would cover whenever users want to talk about tracing.

	Ingo

Re: mailing list for trace users

[Masami Hiramatsu]

Ingo Molnar wrote:
>>>>> What are people's thoughts about creating a linux-trace-users mailing
>>>>> list on vger.kernel.org?
>>>>
>>>> I've created this list.
>>>
>>> Could you please also create the linux-perf-users list, for perf events 
>>> and the perf tool related user questions? (We have asked for this before 
>>> but must have gotten lost somewhere)
>>
>> I thought perf users also talk on the linux tracing list.
> 
> No, there's a lot of non-tracing aspects of performance events. It does 
> profiling, counting, latency analysis, etc.

So, will it include topics about oprofile,readprof too?

>> What will linux-tracing-users cover, only ftrace?
> 
> It would cover whenever users want to talk about tracing.

Hmm, in that case, on which list should I ask 'how can I
use perf-trace?' and 'how can I use perf-record'? :-)
Maybe I'm just a bit confusing ...

Thank you,

-- 
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America), Inc.
Software Solutions Division

e-mail: mhiramat@redhat.com

Re: mailing list for trace users

[Ingo Molnar]

* Masami Hiramatsu <mhiramat@redhat.com> wrote:

> Ingo Molnar wrote:
> >>>>> What are people's thoughts about creating a linux-trace-users mailing
> >>>>> list on vger.kernel.org?
> >>>>
> >>>> I've created this list.
> >>>
> >>> Could you please also create the linux-perf-users list, for perf events 
> >>> and the perf tool related user questions? (We have asked for this before 
> >>> but must have gotten lost somewhere)
> >>
> >> I thought perf users also talk on the linux tracing list.
> > 
> > No, there's a lot of non-tracing aspects of performance events. It does 
> > profiling, counting, latency analysis, etc.
> 
> So, will it include topics about oprofile,readprof too?

Both oprofile and readprof are obsolete.

> >> What will linux-tracing-users cover, only ftrace?
> > 
> > It would cover whenever users want to talk about tracing.
> 
> Hmm, in that case, on which list should I ask 'how can I
> use perf-trace?' and 'how can I use perf-record'? :-)
> Maybe I'm just a bit confusing ...

Which list do you use if you want to ask about why a networked app 
schedules too much? lkml or netdev? Both if you want to reach everyone 
who is interested.

Note, perf trace is one out of 9 'perf' sub-commands. I dont think it's 
appropriate to name the list after that one subcommand ...

For your question i'd suggest to Cc: both. It might be relevant to the 
tracing engine, or it might be a perf events or perf tool detail.

	Ingo

Re: mailing list for trace users

[David Miller]

From: Ingo Molnar <mingo@elte.hu>
Date: Wed, 23 Sep 2009 13:47:25 +0200

> Could you please also create the linux-perf-users list, for perf events 
> and the perf tool related user questions? (We have asked for this before 
> but must have gotten lost somewhere)

I think your communities are similar enough and small enough that
you could share this list.

Thanks.

Re: mailing list for trace users

[Ingo Molnar]

* David Miller <davem@davemloft.net> wrote:

> From: Ingo Molnar <mingo@elte.hu>
> Date: Wed, 23 Sep 2009 13:47:25 +0200
> 
> > Could you please also create the linux-perf-users list, for perf 
> > events and the perf tool related user questions? (We have asked for 
> > this before but must have gotten lost somewhere)
> 
> I think your communities are similar enough and small enough that you 
> could share this list.

It's two separate subsystems. There's a lot of non-tracing aspects of 
performance events: it does profiling, counting, latency analysis, etc. 
The tool is named 'perf', the subsystem is named 'performance events', 
and the most typical workflows dont do any tracing.

And i beg to differ about the size of the communities. It's just been 
added upstream...

Also, what is the policy for adding new lists to vger.kernel.org? If a 
subsystem maintainer asks for a list named after a core kernel 
subsystem, how frequently is it rejected, and on what basis?

I find it sad that such an arbitrary looking negative decision from you 
forces a user list away from vger. I wouldnt mind it to be closed if it 
has no significant traffic after a year or lifetime or so - many vger 
lists have almost no traffic to begin with.

Also, i cannot help but to observe the fact that you've fought the 
original perfcounters project in a very ugly and public way less than a 
year ago. Dont you think that you 'deciding' this matter in such a 
negative fashion is a conflict of interest?

	Ingo

Re: mailing list for trace users

[John Kacur]

On Wed, Sep 23, 2009 at 9:30 PM, Ingo Molnar <mingo@elte.hu> wrote:
>
> * David Miller <davem@davemloft.net> wrote:
>
>> From: Ingo Molnar <mingo@elte.hu>
>> Date: Wed, 23 Sep 2009 13:47:25 +0200
>>
>> > Could you please also create the linux-perf-users list, for perf
>> > events and the perf tool related user questions? (We have asked for
>> > this before but must have gotten lost somewhere)
>>
>> I think your communities are similar enough and small enough that you
>> could share this list.
>
> It's two separate subsystems. There's a lot of non-tracing aspects of
> performance events: it does profiling, counting, latency analysis, etc.
> The tool is named 'perf', the subsystem is named 'performance events',
> and the most typical workflows dont do any tracing.
>
> And i beg to differ about the size of the communities. It's just been
> added upstream...
>
> Also, what is the policy for adding new lists to vger.kernel.org? If a
> subsystem maintainer asks for a list named after a core kernel
> subsystem, how frequently is it rejected, and on what basis?
>
> I find it sad that such an arbitrary looking negative decision from you
> forces a user list away from vger. I wouldnt mind it to be closed if it
> has no significant traffic after a year or lifetime or so - many vger
> lists have almost no traffic to begin with.
>
> Also, i cannot help but to observe the fact that you've fought the
> original perfcounters project in a very ugly and public way less than a
> year ago. Dont you think that you 'deciding' this matter in such a
> negative fashion is a conflict of interest?
>
>        Ingo

Yikes - Ingo, I don't want to spawn a long thread here about what to
call the list, but
as a native English speaker, I think that "linux-trace-users" sounds
much nicer than
"linux-tracing-users".

Perhaps, trace is an adjective describing the kind of users, in any
case, it doesn't
sound like "one trace" - so, could we go this one?
(if you insist on linux-tracing then you have to drop the word "users")

John

Re: mailing list for trace users

[John Kacur]

On Wed, Sep 23, 2009 at 9:40 PM, John Kacur <jkacur@gmail.com> wrote:
> On Wed, Sep 23, 2009 at 9:30 PM, Ingo Molnar <mingo@elte.hu> wrote:
>>
>> * David Miller <davem@davemloft.net> wrote:
>>
>>> From: Ingo Molnar <mingo@elte.hu>
>>> Date: Wed, 23 Sep 2009 13:47:25 +0200
>>>
>>> > Could you please also create the linux-perf-users list, for perf
>>> > events and the perf tool related user questions? (We have asked for
>>> > this before but must have gotten lost somewhere)
>>>
>>> I think your communities are similar enough and small enough that you
>>> could share this list.
>>
>> It's two separate subsystems. There's a lot of non-tracing aspects of
>> performance events: it does profiling, counting, latency analysis, etc.
>> The tool is named 'perf', the subsystem is named 'performance events',
>> and the most typical workflows dont do any tracing.
>>
>> And i beg to differ about the size of the communities. It's just been
>> added upstream...
>>
>> Also, what is the policy for adding new lists to vger.kernel.org? If a
>> subsystem maintainer asks for a list named after a core kernel
>> subsystem, how frequently is it rejected, and on what basis?
>>
>> I find it sad that such an arbitrary looking negative decision from you
>> forces a user list away from vger. I wouldnt mind it to be closed if it
>> has no significant traffic after a year or lifetime or so - many vger
>> lists have almost no traffic to begin with.
>>
>> Also, i cannot help but to observe the fact that you've fought the
>> original perfcounters project in a very ugly and public way less than a
>> year ago. Dont you think that you 'deciding' this matter in such a
>> negative fashion is a conflict of interest?
>>
>>        Ingo
>
> Yikes - Ingo, I don't want to spawn a long thread here about what to
> call the list, but
> as a native English speaker, I think that "linux-trace-users" sounds
> much nicer than
> "linux-tracing-users".
>
> Perhaps, trace is an adjective describing the kind of users, in any
> case, it doesn't
> sound like "one trace" - so, could we go this one?
> (if you insist on linux-tracing then you have to drop the word "users")
>
> John
>

That's funny, boast about my credentials as a native English speaker, and then
make a mistake.

s/go this one/go with this one

Re: mailing list for trace users

[Ingo Molnar]

* John Kacur <jkacur@gmail.com> wrote:

> > Yikes - Ingo, I don't want to spawn a long thread here about what to 
> > call the list, but as a native English speaker, I think that 
> > "linux-trace-users" sounds much nicer than "linux-tracing-users".
> >
> > Perhaps, trace is an adjective describing the kind of users, in any 
> > case, it doesn't sound like "one trace" - so, could we go this one? 
> > (if you insist on linux-tracing then you have to drop the word 
> > "users")
> >
> > John
> >
> 
> That's funny, boast about my credentials as a native English speaker, 
> and then make a mistake.
> 
> s/go this one/go with this one

I hope you are right about linux-trace-users sounding better to native 
speakers than linux-tracing-users - because it sure sounds awful to me
:-)

Any other English native speakers with an opinion one way or another? 

	Ingo

Re: mailing list for trace users

[Steven Rostedt]

On Wed, 2009-09-23 at 21:59 +0200, Ingo Molnar wrote:
> * John Kacur <jkacur@gmail.com> wrote:
> 
> > > Yikes - Ingo, I don't want to spawn a long thread here about what to 
> > > call the list, but as a native English speaker, I think that 
> > > "linux-trace-users" sounds much nicer than "linux-tracing-users".
> > >
> > > Perhaps, trace is an adjective describing the kind of users, in any 
> > > case, it doesn't sound like "one trace" - so, could we go this one? 
> > > (if you insist on linux-tracing then you have to drop the word 
> > > "users")
> > >
> > > John
> > >
> > 
> > That's funny, boast about my credentials as a native English speaker, 
> > and then make a mistake.
> > 
> > s/go this one/go with this one
> 
> I hope you are right about linux-trace-users sounding better to native 
> speakers than linux-tracing-users - because it sure sounds awful to me
> :-)
> 
> Any other English native speakers with an opinion one way or another? 

Yes "linux-trace-users" sounds better, because it doesn't sound like
proper English. It sort of breaks it up "linux-trace ... users" gives a
ring of "linux tracing for users".

But if you go with the more proper sounding 'linux-tracing-users" it (to
a native speaker) sounds more like a sentence, and my response is "Oh
God, Linux is now tracing its users, but I don't want to be traced!".

;-)

-- Steve

Re: mailing list for trace users

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> On Wed, 2009-09-23 at 21:59 +0200, Ingo Molnar wrote:
> > * John Kacur <jkacur@gmail.com> wrote:
> > 
> > > > Yikes - Ingo, I don't want to spawn a long thread here about what to 
> > > > call the list, but as a native English speaker, I think that 
> > > > "linux-trace-users" sounds much nicer than "linux-tracing-users".
> > > >
> > > > Perhaps, trace is an adjective describing the kind of users, in any 
> > > > case, it doesn't sound like "one trace" - so, could we go this one? 
> > > > (if you insist on linux-tracing then you have to drop the word 
> > > > "users")
> > > >
> > > > John
> > > >
> > > 
> > > That's funny, boast about my credentials as a native English speaker, 
> > > and then make a mistake.
> > > 
> > > s/go this one/go with this one
> > 
> > I hope you are right about linux-trace-users sounding better to native 
> > speakers than linux-tracing-users - because it sure sounds awful to me
> > :-)
> > 
> > Any other English native speakers with an opinion one way or another? 
> 
> Yes "linux-trace-users" sounds better, because it doesn't sound like 
> proper English. It sort of breaks it up "linux-trace ... users" gives 
> a ring of "linux tracing for users".
> 
> But if you go with the more proper sounding 'linux-tracing-users" it 
> (to a native speaker) sounds more like a sentence, and my response is 
> "Oh God, Linux is now tracing its users, but I don't want to be 
> traced!".
> 
> ;-)

Ok, i concur then :-)

	Ingo

Re: mailing list for trace users

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> On Wed, 2009-09-23 at 21:59 +0200, Ingo Molnar wrote:
> > * John Kacur <jkacur@gmail.com> wrote:
> > 
> > > > Yikes - Ingo, I don't want to spawn a long thread here about what to 
> > > > call the list, but as a native English speaker, I think that 
> > > > "linux-trace-users" sounds much nicer than "linux-tracing-users".
> > > >
> > > > Perhaps, trace is an adjective describing the kind of users, in any 
> > > > case, it doesn't sound like "one trace" - so, could we go this one? 
> > > > (if you insist on linux-tracing then you have to drop the word 
> > > > "users")
> > > >
> > > > John
> > > >
> > > 
> > > That's funny, boast about my credentials as a native English speaker, 
> > > and then make a mistake.
> > > 
> > > s/go this one/go with this one
> > 
> > I hope you are right about linux-trace-users sounding better to native 
> > speakers than linux-tracing-users - because it sure sounds awful to me
> > :-)
> > 
> > Any other English native speakers with an opinion one way or another? 
> 
> Yes "linux-trace-users" sounds better, because it doesn't sound like 
> proper English. It sort of breaks it up "linux-trace ... users" gives 
> a ring of "linux tracing for users".
> 
> But if you go with the more proper sounding 'linux-tracing-users" it 
> (to a native speaker) sounds more like a sentence, and my response is 
> "Oh God, Linux is now tracing its users, but I don't want to be 
> traced!".
> 
> ;-)

btw., i suggested tracing-users@vger.kernel.org in my original mail. The 
linux- prefix is pretty superfluous.

	Ingo

Re: mailing list for trace users

[Ingo Molnar]

* John Kacur <jkacur@gmail.com> wrote:

> On Wed, Sep 23, 2009 at 9:30 PM, Ingo Molnar <mingo@elte.hu> wrote:
> >
> > * David Miller <davem@davemloft.net> wrote:
> >
> >> From: Ingo Molnar <mingo@elte.hu>
> >> Date: Wed, 23 Sep 2009 13:47:25 +0200
> >>
> >> > Could you please also create the linux-perf-users list, for perf
> >> > events and the perf tool related user questions? (We have asked for
> >> > this before but must have gotten lost somewhere)
> >>
> >> I think your communities are similar enough and small enough that you
> >> could share this list.
> >
> > It's two separate subsystems. There's a lot of non-tracing aspects of
> > performance events: it does profiling, counting, latency analysis, etc.
> > The tool is named 'perf', the subsystem is named 'performance events',
> > and the most typical workflows dont do any tracing.
> >
> > And i beg to differ about the size of the communities. It's just been
> > added upstream...
> >
> > Also, what is the policy for adding new lists to vger.kernel.org? If a
> > subsystem maintainer asks for a list named after a core kernel
> > subsystem, how frequently is it rejected, and on what basis?
> >
> > I find it sad that such an arbitrary looking negative decision from you
> > forces a user list away from vger. I wouldnt mind it to be closed if it
> > has no significant traffic after a year or lifetime or so - many vger
> > lists have almost no traffic to begin with.
> >
> > Also, i cannot help but to observe the fact that you've fought the 
> > original perfcounters project in a very ugly and public way less 
> > than a year ago. Dont you think that you 'deciding' this matter in 
> > such a negative fashion is a conflict of interest?
> >
> > ? ? ? ?Ingo
> 
> Yikes - Ingo, I don't want to spawn a long thread here about what to 
> call the list, but as a native English speaker, I think that 
> "linux-trace-users" sounds much nicer than "linux-tracing-users".

That argument i can and will accept of course ...

The "sorry, i ignored you twice and now it's unfortunately too late" 
excuse given by David i will not ;-)

Note that this mail you replied to was about 
linux-perf-users@vger.kernel.org, which David refused to create, 
suggesting that perf users should mail to 
linux-trace-users@vger.kernel.org instead. (which is a curious argument 
- if you use a tool named 'perf', or if you are using PAPI to count 
events, would it occur to you to mail to that list?)

	Ingo

Re: mailing list for trace users

[John Kacur]

On Wed, Sep 23, 2009 at 9:49 PM, Ingo Molnar <mingo@elte.hu> wrote:
>
> * John Kacur <jkacur@gmail.com> wrote:
>
>> On Wed, Sep 23, 2009 at 9:30 PM, Ingo Molnar <mingo@elte.hu> wrote:
>> >
>> > * David Miller <davem@davemloft.net> wrote:
>> >
>> >> From: Ingo Molnar <mingo@elte.hu>
>> >> Date: Wed, 23 Sep 2009 13:47:25 +0200
>> >>
>> >> > Could you please also create the linux-perf-users list, for perf
>> >> > events and the perf tool related user questions? (We have asked for
>> >> > this before but must have gotten lost somewhere)
>> >>
>> >> I think your communities are similar enough and small enough that you
>> >> could share this list.
>> >
>> > It's two separate subsystems. There's a lot of non-tracing aspects of
>> > performance events: it does profiling, counting, latency analysis, etc.
>> > The tool is named 'perf', the subsystem is named 'performance events',
>> > and the most typical workflows dont do any tracing.
>> >
>> > And i beg to differ about the size of the communities. It's just been
>> > added upstream...
>> >
>> > Also, what is the policy for adding new lists to vger.kernel.org? If a
>> > subsystem maintainer asks for a list named after a core kernel
>> > subsystem, how frequently is it rejected, and on what basis?
>> >
>> > I find it sad that such an arbitrary looking negative decision from you
>> > forces a user list away from vger. I wouldnt mind it to be closed if it
>> > has no significant traffic after a year or lifetime or so - many vger
>> > lists have almost no traffic to begin with.
>> >
>> > Also, i cannot help but to observe the fact that you've fought the
>> > original perfcounters project in a very ugly and public way less
>> > than a year ago. Dont you think that you 'deciding' this matter in
>> > such a negative fashion is a conflict of interest?
>> >
>> > ? ? ? ?Ingo
>>
>> Yikes - Ingo, I don't want to spawn a long thread here about what to
>> call the list, but as a native English speaker, I think that
>> "linux-trace-users" sounds much nicer than "linux-tracing-users".
>
> That argument i can and will accept of course ...
>
> The "sorry, i ignored you twice and now it's unfortunately too late"
> excuse given by David i will not ;-)
>
> Note that this mail you replied to was about
> linux-perf-users@vger.kernel.org, which David refused to create,
> suggesting that perf users should mail to
> linux-trace-users@vger.kernel.org instead. (which is a curious argument
> - if you use a tool named 'perf', or if you are using PAPI to count
> events, would it occur to you to mail to that list?)
>
>        Ingo
>

Oh, for that matter, as a user of both tracing and the perf tool, I think it IS
appropriate to have separate lists.

John

Re: mailing list for trace users

[David Miller]

From: Ingo Molnar <mingo@elte.hu>
Date: Wed, 23 Sep 2009 21:30:20 +0200

> Also, what is the policy for adding new lists to vger.kernel.org? If a 
> subsystem maintainer asks for a list named after a core kernel 
> subsystem, how frequently is it rejected, and on what basis?

Ingo, relax, I'll create your list.

What's really sad is that you don't attend enough conferences so that
you can meet face to face with people and you (and others) would as a
result avoid many unnecessarily heated discussions as a result.

Re: mailing list for trace users

[Ingo Molnar]

* David Miller <davem@davemloft.net> wrote:

> From: Ingo Molnar <mingo@elte.hu>
> Date: Wed, 23 Sep 2009 21:30:20 +0200
> 
> > Also, what is the policy for adding new lists to vger.kernel.org? If 
> > a subsystem maintainer asks for a list named after a core kernel 
> > subsystem, how frequently is it rejected, and on what basis?
> 
> Ingo, relax, I'll create your list.

Thanks!

> What's really sad is that you don't attend enough conferences so that 
> you can meet face to face with people and you (and others) would as a 
> result avoid many unnecessarily heated discussions as a result.

I calmed down already ;-) Sorry about the tone - my remarks were clearly 
over the top - it felt really frustrating to get such a curt reply after 
months of failed attempts to create this list.

Thanks again for doing it. Development will happen on lkml as usual, but 
users/newbies generally prefer some topical-sounding email address to 
mail to, and for that one the naming does matter.

	Ingo

Re: mailing list for trace users

[David Miller]

I have created linux-perf-users@vger.kernel.org

Re: mailing list for trace users

[Ingo Molnar]

* David Miller <davem@davemloft.net> wrote:

> I have created linux-perf-users@vger.kernel.org

Thanks a lot David!

	Ingo

Re: mailing list for trace users

[Ingo Molnar]

* David Miller <davem@davemloft.net> wrote:

> From: Steven Rostedt <rostedt@goodmis.org>
> Date: Wed, 16 Sep 2009 16:16:22 -0400
> 
> > What are people's thoughts about creating a linux-trace-users 
> > mailing list on vger.kernel.org?
> 
> I've created this list.

btw., could you please change the name to linux-tracing-users? That's 
the name of the subsystem and the name of the activity as well. A 
'trace' is just (one) end result of that activity.

Thanks,

	Ingo

Re: mailing list for trace users

[David Miller]

From: Ingo Molnar <mingo@elte.hu>
Date: Wed, 23 Sep 2009 13:48:23 +0200

> 
> * David Miller <davem@davemloft.net> wrote:
> 
>> From: Steven Rostedt <rostedt@goodmis.org>
>> Date: Wed, 16 Sep 2009 16:16:22 -0400
>> 
>> > What are people's thoughts about creating a linux-trace-users 
>> > mailing list on vger.kernel.org?
>> 
>> I've created this list.
> 
> btw., could you please change the name to linux-tracing-users? That's 
> the name of the subsystem and the name of the activity as well. A 
> 'trace' is just (one) end result of that activity.

No, the name is arbitrary and people are already propagating
the existience of the name that has been chosen already.

SOrry.

Re: mailing list for trace users

[Ingo Molnar]

* David Miller <davem@davemloft.net> wrote:

> From: Ingo Molnar <mingo@elte.hu>
> Date: Wed, 23 Sep 2009 13:48:23 +0200
> 
> > 
> > * David Miller <davem@davemloft.net> wrote:
> > 
> >> From: Steven Rostedt <rostedt@goodmis.org>
> >> Date: Wed, 16 Sep 2009 16:16:22 -0400
> >> 
> >> > What are people's thoughts about creating a linux-trace-users 
> >> > mailing list on vger.kernel.org?
> >> 
> >> I've created this list.
> > 
> > btw., could you please change the name to linux-tracing-users? 
> > That's the name of the subsystem and the name of the activity as 
> > well. A 'trace' is just (one) end result of that activity.
> 
> No, the name is arbitrary and people are already propagating the 
> existience of the name that has been chosen already.
> 
> SOrry.

David, i have asked for the proper name _BEFORE_ you have created it, 
yesterday. I also asked you again today to change the name _minutes_ 
after you mailed about the sucky 'linux-trace-users@vger.kernel.org' 
name. You ignored all that.

I dont know about you, but the activity is 'tracing' not 'trace'. People 
mailing to that list want to talk about 'tracing' - not about a 'trace'. 
What is a 'trace user'? It makes no sense and is a misnomer.

The subsystem entry in MAINTAINERS clearly says:

TRACING
M:      Steven Rostedt <rostedt@goodmis.org>
M:      Frederic Weisbecker <fweisbec@gmail.com>
M:      Ingo Molnar <mingo@redhat.com>
T:      git git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip.git tracing/core
S:      Maintained

It's no secret that you and me dont get along too well and have been 
involved in a number of very public spats in the past year or so.

I do think your actions here are designed to hurt the tracing (and also 
the perf events) subsystems just because i'm co-maintaining them, and i 
think that you are in a heavy conflict of interest over it.

You should have recused yourself from this decision - i suspect there's 
other, less biased people outside of you who can create new mailing 
lists on vger.kernel.org. They might refuse my request with some real 
reason.

	Ingo

Re: mailing list for trace users

[David Miller]

From: Ingo Molnar <mingo@elte.hu>
Date: Wed, 23 Sep 2009 21:41:22 +0200

> David, i have asked for the proper name _BEFORE_ you have created it, 

Ingo would you relax already? I'll change the name if you want
me to, no problem.

You're lack of face-to-face interaction with people is why you
constantly blow up like this.  Please make an effort to change
this in the future.

KTHX

Re: mailing list for trace users

[Ingo Molnar]

* David Miller <davem@davemloft.net> wrote:

> From: Ingo Molnar <mingo@elte.hu>
> Date: Wed, 23 Sep 2009 21:41:22 +0200
> 
> > David, i have asked for the proper name _BEFORE_ you have created 
> > it,
> 
> Ingo would you relax already? I'll change the name if you want me to, 
> no problem.

Seems like the native-speaker concensus is overwhelmingly against my 
proposal and in favor of the name you chose, so i stand corrected on 
that matter and please leave the linux-trace-users naming.

> You're lack of face-to-face interaction with people is why you 
> constantly blow up like this.  Please make an effort to change this in 
> the future.

I have no trouble with hundreds of people i work with, except you - and 
you are in a pretty similar position - and that's sad. Such 
incompatibilities happen (to no real fault of any of the sides) and 
we'll have to learn to deal with it.

Thanks,

	Ingo

Re: mailing list for trace users

[David Miller]

From: Ingo Molnar <mingo@elte.hu>
Date: Thu, 24 Sep 2009 00:10:59 +0200

> 
> * David Miller <davem@davemloft.net> wrote:
>
>> You're lack of face-to-face interaction with people is why you 
>> constantly blow up like this.  Please make an effort to change this in 
>> the future.
> 
> I have no trouble with hundreds of people i work with, except you - and 
> you are in a pretty similar position - and that's sad. Such 
> incompatibilities happen (to no real fault of any of the sides) and 
> we'll have to learn to deal with it.

This is not a me vs. you thing.  Everyone I have spoken to
here at LinuxCon and LinuxPlumbers has expressed extreme
dismay that you do not at least attend the kernel summit on
a regular basis.

Re: mailing list for trace users

[Ingo Molnar]

* David Miller <davem@davemloft.net> wrote:

> From: Ingo Molnar <mingo@elte.hu>
> Date: Thu, 24 Sep 2009 00:10:59 +0200
> 
> > 
> > * David Miller <davem@davemloft.net> wrote:
> >
> >> You're lack of face-to-face interaction with people is why you 
> >> constantly blow up like this.  Please make an effort to change this in 
> >> the future.
> > 
> > I have no trouble with hundreds of people i work with, except you - and 
> > you are in a pretty similar position - and that's sad. Such 
> > incompatibilities happen (to no real fault of any of the sides) and 
> > we'll have to learn to deal with it.
> 
> This is not a me vs. you thing.  Everyone I have spoken to
> here at LinuxCon and LinuxPlumbers has expressed extreme
> dismay that you do not at least attend the kernel summit on
> a regular basis.

Well, i'm going to attend a Linux kernel conference in two days, so no, 
i dont ignore conferences. (Wont attend the KS this year though - cannot 
attend too many conferences unfortunately - they are certainly fun :)

But that's beside the point even - i absolutely reject your apparent 
notion that i have to meet you 'face to face' to be treated in a 
friendly and honest way, such as in a routine matter of creating a 
mailing list on vger. "You have to visit the same pubs as us to be on 
the same frequency" is clan/tribe/clique thinking that i find 
fundamentally harmful.

( That also happens to be one of the main disagreements (and sources of
  conflicts) between you and me - the "us vs them" mentality on netdev 
  that i have a problem with. You know that. )

Anyway, i'm glad it got all resolved, and yes, after months of trying to 
achieve this new mailing list amicably i saw no other way but to raise 
attention to this matter in more forceful terms.

Btw., regardless of your flames against me in the past, i dont think you 
dislike perf events all that much anymore. The recent Niagara code you 
wrote in arch/sparc/kernel/perf_event.c certainly shows great care and 
interest, and i think/hope that you enjoyed dealing with that subsystem. 
You did contribute a number of fixes and i think you'd have told us 
frankly if you still disliked aspects of its design. So as far as my 
"you did this to hurt perf" accusation goes, that was unfair and over 
the top, and i apologize for it.

	Ingo

Re: mailing list for trace users

[David Miller]

From: Ingo Molnar <mingo@elte.hu>
Date: Thu, 24 Sep 2009 13:16:45 +0200

> But that's beside the point even - i absolutely reject your apparent 
> notion that i have to meet you 'face to face' to be treated in a 
> friendly and honest way, such as in a routine matter of creating a 
> mailing list on vger.

People who meet each other face to face treat each other better than
if they interact only on mailing lists, and that's just a fact of
life.

And until there is a way to transfer facial expressions, vocal
intonations, and all other human visual indications of emotions and
intent over email, it's going to stay that way.

Re: mailing list for trace users

[David Miller]

From: Ingo Molnar <mingo@elte.hu>
Date: Thu, 24 Sep 2009 13:16:45 +0200

> Btw., regardless of your flames against me in the past, i dont think
> you dislike perf events all that much anymore.

Indeed, I love 'perf' and if you were here you would be hearing
me recommending it and singing my praises about it on a daily
basis to everyone :-)

Re: mailing list for trace users

[Ingo Molnar]

* David Miller <davem@davemloft.net> wrote:

> From: Ingo Molnar <mingo@elte.hu>
> Date: Thu, 24 Sep 2009 13:16:45 +0200
> 
> > Btw., regardless of your flames against me in the past, i dont think
> > you dislike perf events all that much anymore.
> 
> Indeed, I love 'perf' and if you were here you would be hearing me 
> recommending it and singing my praises about it on a daily basis to 
> everyone :-)

*Blush* :-)

	Ingo