extracting data access from functions

extracting data access from functions

[Bart Trojanowski]

I am trying to do some data mining on a large code base.  I would like
to generate a list of structure members accessed by certain functions.

After studying the examples in the pahole.git tree, I am able to walk
the dwarves structures down to the right tags.  However, I am not sure
how to make the next step.

For example, given

void foo(struct s *s)
{
        ... s->var ...
}

I would like to have a tool that reports foo accessing s->var.

Is that possible with the DWARF debug tags?  Maybe using elftools?  I am
trying to avoid the temptation of parsing the C code in perl :)

Thanks in advance.

Cheers,
-Bart

-- 
				WebSig: http://www.jukie.net/~bart/sig/
--
To unsubscribe from this list: send the line "unsubscribe dwarves" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: extracting data access from functions

[Arnaldo Carvalho de Melo]

Em Tue, Jan 27, 2009 at 10:11:25AM -0500, Bart Trojanowski escreveu:
> I am trying to do some data mining on a large code base.  I would like
> to generate a list of structure members accessed by certain functions.
> 
> After studying the examples in the pahole.git tree, I am able to walk
> the dwarves structures down to the right tags.  However, I am not sure
> how to make the next step.
> 
> For example, given
> 
> void foo(struct s *s)
> {
>         ... s->var ...
> }
> 
> I would like to have a tool that reports foo accessing s->var.
> 
> Is that possible with the DWARF debug tags?  Maybe using elftools?  I am
> trying to avoid the temptation of parsing the C code in perl :)
> 
> Thanks in advance.

With DWARF you can:

1. find out struct layout
2. find which functions have parameters of the type you want
3. find which functions have variables of the type you want
4. Know where such variables and parameters are on memory
   using DWARF location expressions (google for loc2c)

And that is it.

Then... you would need to use libdisasm, that is part of elfutils but is
still a bit rough, being designed mostly for what you get from the
binutils utilities disassembly options.

Combinining the above you will be able to do what you want.

And I really, really want to have that, but have no time at the moment.

I want to look at usage patterns to figure out what are the member
working sets to pass that to my struct member reorganizing routines, run
with performance counters before/after, all automated, and reap the
profits 8)

- Arnaldo
--
To unsubscribe from this list: send the line "unsubscribe dwarves" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: extracting data access from functions

[Bart Trojanowski]

* Arnaldo Carvalho de Melo <acme-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> [090127 10:27]:
> With DWARF you can:
> 
> 1. find out struct layout
> 2. find which functions have parameters of the type you want
> 3. find which functions have variables of the type you want
> 4. Know where such variables and parameters are on memory
>    using DWARF location expressions (google for loc2c)
> 
> And that is it.

Thanks for the quick reply and clarification.

> Then... you would need to use libdisasm, that is part of elfutils but is
> still a bit rough, being designed mostly for what you get from the
> binutils utilities disassembly options.
>
> Combinining the above you will be able to do what you want.

That sounds interesting, but unfortunately the man pages notes that...
>> The x86-64 architecture is not supported..LP
... being ia32-only makes it a no go for me.

Is there another route you could think of that is not platform-specific?

I guess binutils/objdump doesn't actually understand what's happening it
just links it back to the code.  Hmm... and things like gdb are aware of
things because they are platform specific.

> I want to look at usage patterns to figure out what are the member
> working sets to pass that to my struct member reorganizing routines, run
> with performance counters before/after, all automated, and reap the
> profits 8)

I hear ya. :)

Thanks again.

-Bart

-- 
				WebSig: http://www.jukie.net/~bart/sig/
--
To unsubscribe from this list: send the line "unsubscribe dwarves" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: extracting data access from functions

[Arnaldo Carvalho de Melo]

Em Tue, Jan 27, 2009 at 10:52:22AM -0500, Bart Trojanowski escreveu:
> * Arnaldo Carvalho de Melo <acme-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> [090127 10:27]:
> > With DWARF you can:
> > 
> > 1. find out struct layout
> > 2. find which functions have parameters of the type you want
> > 3. find which functions have variables of the type you want
> > 4. Know where such variables and parameters are on memory
> >    using DWARF location expressions (google for loc2c)
> > 
> > And that is it.
> 
> Thanks for the quick reply and clarification.
> 
> > Then... you would need to use libdisasm, that is part of elfutils but is
> > still a bit rough, being designed mostly for what you get from the
> > binutils utilities disassembly options.
> >
> > Combinining the above you will be able to do what you want.
> 
> That sounds interesting, but unfortunately the man pages notes that...
> >> The x86-64 architecture is not supported..LP
> ... being ia32-only makes it a no go for me.

Forget about the man page, this is very fluid, look at the latest
released version, 0.139, on fedorahosted, lemme dig the URL

https://fedorahosted.org/elfutils/

There is now a git tree, check it and, again, forget about the man page
:)
 
> Is there another route you could think of that is not platform-specific?
> 
> I guess binutils/objdump doesn't actually understand what's happening it
> just links it back to the code.  Hmm... and things like gdb are aware of
> things because they are platform specific.
> 
> > I want to look at usage patterns to figure out what are the member
> > working sets to pass that to my struct member reorganizing routines, run
> > with performance counters before/after, all automated, and reap the
> > profits 8)
> 
> I hear ya. :)
> 
> Thanks again.

Gimme 5 minutes and I'll get that done! ;-)

- Arnaldo
--
To unsubscribe from this list: send the line "unsubscribe dwarves" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: extracting data access from functions

[Bart Trojanowski]

* Arnaldo Carvalho de Melo <acme-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> [090127 11:01]:
> > > Then... you would need to use libdisasm, that is part of elfutils but is
> > > still a bit rough, being designed mostly for what you get from the
> > > binutils utilities disassembly options.
> > >
> > > Combinining the above you will be able to do what you want.
> > 
> > That sounds interesting, but unfortunately the man pages notes that...
> > >> The x86-64 architecture is not supported..LP
> > ... being ia32-only makes it a no go for me.
> 
> Forget about the man page, this is very fluid, look at the latest
> released version, 0.139, on fedorahosted, lemme dig the URL
> 
> https://fedorahosted.org/elfutils/
> 
> There is now a git tree, check it and, again, forget about the man page
> :)

Man page has been forgotten :)

Now, on the topic of elfutils.  I had already upgraded to elfutils 0.138
from git.  I was unable to get 0.139 (or HEAD) to build on debian/lenny.

automake emits the following warnings:

libdwfl/Makefile.am:78: ZLIB does not appear in AM_CONDITIONAL
libdwfl/Makefile.am:81: BZLIB does not appear in AM_CONDITIONAL

configure fails for me with the following error:

checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
./configure: line 4538: syntax error near unexpected token `z,Z,z,gzdirect,gzip'
./configure: line 4538: `eu_ZIPLIB(z,Z,z,gzdirect,gzip)'

I don't grok what autoconf is trying to tell me here.  Do I need a more
recent gzip libs or just new autotools?

-Bart

-- 
				WebSig: http://www.jukie.net/~bart/sig/
--
To unsubscribe from this list: send the line "unsubscribe dwarves" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: extracting data access from functions

[Roland McGrath]

elfutils-devel-TuqUDEhatI7GMZAyRF5v151Ccm5ICvs9@public.gmane.org is the place to post about your
build issues.  You're just in luck that I happen to be on this list
too to have noticed.  You probably need a newer automake or something
and run "autoreconf -f -i"; I'm using 1.10.1 and autoconf-2.63.  (Or
you can just use the rpm/tarball instead of git, and not run auto*
yourself.)

For what you want I suspect that none of our stuff is the way to go
about it, really.  As acme pointed out, lots of that information is
lost in compilation and is not really accessible any more.  What we
can do (and not at all easily any time soon) is rediscover some of it.
But it's all there in the source, and I'm sure there are
source-parsing tools that do what you want around to be found (cscope
maybe? some IDE's features?).

On the (unwritten) list for the future is doing something quite like
what you asked about.  But it's a far ways off, and, again, it's
really almost certainly not the right way to skin your particular cat.

What it is I have in mind for the future is a combination of a "smart
disassembler" that generates DWARF location expressions describing
instruction operands, with lots of general smarts for hacking location
expressions.  Then you could disassemble, analyze the DWARF describing
the containing scope, and match up each instruction operand's location
expression with the locations of variables in scope.  One target use
for this is to identify the "hot variables" (or hot struct fields) by
identifying "hot spots" in the code via profiling etc., and then analyzing
the locations are used in that code to yield the source constructs that
correspond.

I think this will be a fabulous thing.  But it's a long ways off.
And, it's not at all the most sensible or straightforward approach for
addressing your case where plain source analysis would be much easier
and more informative.


Thanks,
Roland
--
To unsubscribe from this list: send the line "unsubscribe dwarves" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html