[Cocci] Coccinelle Compilation Problem: configure file

[Cocci] Coccinelle Compilation Problem: configure file

[Aftab Hussain]

Dear Members of the Coccinelle Team,

Hope you are doing well. Me and my team mates have been recently working on
program analysis, and are very keen on working with your tool.
Unfortunately, we faced a compilation problem.

I was trying to compile Coccinelle using the source (
https://github.com/coccinelle/coccinelle). I followed all instructions,
mentioned in the "*install.txt*" file, but
when I enter the following command in the "*coccinelle-master*" directory,

*./configure --enable-release*

I get the following error message,

*bash: ./configure: No such file or directory*

I see the "configure" file is missing from your source. I would be most
grateful if you could please advise on how I may fix this.

Thanks,
Aftab

Aftab Hussain
PhD Student,
Department of Computer Science,
University of California, Irvine
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://systeme.lip6.fr/pipermail/cocci/attachments/20160303/0fd4bdcd/attachment.html>

[Cocci] Coccinelle Compilation Problem: configure file

[Julia Lawall]

On Thu, 3 Mar 2016, Aftab Hussain wrote:

> Dear Members of the Coccinelle Team,
> 
> Hope you are doing well. Me and my team mates have been recently working on
> program analysis, and are very keen on working with your tool.
> Unfortunately, we faced a compilation problem.
> 
> I was trying to compile Coccinelle using the source
> (https://github.com/coccinelle/coccinelle). I followed all instructions,
> mentioned in the "install.txt" file, but
> when I enter the following command in the "coccinelle-master" directory,
> 
> ./configure --enable-release
> 
> I get the following error message,
> 
> bash: ./configure: No such file or directory
> 
> I see the "configure" file is missing from your source. I would be most
> grateful if you could please advise on how I may fix this.

Did you run ./autogen first?  Probably the install.txt file needs to be 
updated.  Thanks for the report.

julia

> 
> Thanks,
> Aftab
> 
> Aftab Hussain
> PhD Student,
> Department of Computer Science,
> University of California, Irvine
> 
> 

[Cocci] Regeneration of the script "configure" for Coccinelle

[SF Markus Elfring]

> Me and my team mates have been recently working on program analysis,
> and are very keen on working with your tool.

I am curious in which directions you would like to experiment with this
software further.


> Unfortunately, we faced a compilation problem.

I find that this issue does not affect "compilation" so far.
How much do you care for the configuration of the software build process?


> I see the "configure" file is missing from your source.

Julia Lawall pointed the possibility out to regenerate it by the
execution of the script "autogen".


> I would be most grateful if you could please advise on how I may fix this.

Are you interested to get also another approach working by using the
command "autoreconf"?

Regards,
Markus

[Cocci] Coccinelle Compilation Problem: configure file

[Aftab Hussain]

Hi Julia Lawall and the Coccinelle Team,

I really appreciate your response. I was able to generate the configure
file using *>./autogen*,
and all the installation steps. Thank you so much. Unfortunately, now I'm
unable to execute spatch.

On executing this:

*spatch -sp_file demos/simple.cocci demos/simple.c -o /tmp/new_simple.c*

I get "spatch is not installed".

Also when I execute the following to test spatch,


*COCCINELLE_HOME=/home/aftab/workspace/coccinelle-master ./scripts/spatch
-sp_file demos/simple.cocci demos/simple.c -o /tmp/new_simple.c*
I get,

*bash: ./scripts/spatch: No such file or directory*

I would like to let you know that I performed all the installation steps (make,
make install), and I noticed no errors.
However, I am sceptical about why "make install" is carrying out the
following operations,



*rm -f /usr/local/lib/coccinelle/spatchrm -f
/usr/local/lib/coccinelle/spatch.optrm -f /usr/local/bin/pycocci*

I wonder whether this is causing the problem. I would be most glad if you
could please help me install spatch.

Just in case, I installed the latest versions of OCaml and Menhir using
OPAM.

Best Regards,
Aftab

On Thu, Mar 3, 2016 at 11:09 PM, Julia Lawall <julia.lawall@lip6.fr> wrote:

> On Thu, 3 Mar 2016, Aftab Hussain wrote:
>
> > Dear Members of the Coccinelle Team,
> >
> > Hope you are doing well. Me and my team mates have been recently working
> on
> > program analysis, and are very keen on working with your tool.
> > Unfortunately, we faced a compilation problem.
> >
> > I was trying to compile Coccinelle using the source
> > (https://github.com/coccinelle/coccinelle). I followed all instructions,
> > mentioned in the "install.txt" file, but
> > when I enter the following command in the "coccinelle-master" directory,
> >
> > ./configure --enable-release
> >
> > I get the following error message,
> >
> > bash: ./configure: No such file or directory
> >
> > I see the "configure" file is missing from your source. I would be most
> > grateful if you could please advise on how I may fix this.
>
> Did you run ./autogen first?  Probably the install.txt file needs to be
> updated.  Thanks for the report.
>
> julia
>
> >
> > Thanks,
> > Aftab
> >
> > Aftab Hussain
> > PhD Student,
> > Department of Computer Science,
> > University of California, Irvine
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://systeme.lip6.fr/pipermail/cocci/attachments/20160307/c282673f/attachment.html>

[Cocci] Parsing C source files by Coccinelle

[SF Markus Elfring]

> I am interested in exploring how Coccinelle is parsing C programs,

The software library "http://cristal.inria.fr/~fpottier/menhir/"
is involved to some degree.


> in order to detect the bugs indicated by SmPL patches.

Would you like to develop your own source code search patterns?


> This is the reason why I was trying to examine the source
> and perform the compilation myself.

Will it be a bit easier to ask about relevant implementation details here?


> However, I am facing issues executing spatch.

Which ones?


> The installation probably didn't work.

How did you get this impression?

Regards,
Markus

[Cocci] Parsing C source files by Coccinelle

[SF Markus Elfring]

> In particular, we would like to see whether it is possible to improve Coccinelle results
> by supplying it with aliasing information about variables.

Does such a goal indicate that you are going to experiment also with data flow analysis?


> On executing,
> *
> spatch -sp_file demos/simple.cocci demos/simple.c -o /tmp/new_simple.c*
> 
> I get "spatch is not installed".

Did you call the command "make install" with appropriate permissions
and configuration parameters?


> Also when I execute the following to test spatch,
> 
> *COCCINELLE_HOME=/home/aftab/workspace/coccinelle-master ./scripts/spatch -sp_file demos/simple.cocci demos/simple.c -o /tmp/new_simple.c
> *
> I get,
> 
> *bash: ./scripts/spatch: No such file or directory*
> 
> I would like to let you know that I performed all the installation steps (make, make install),
> and I noticed no errors.

Interesting ?


> However, I am sceptical about why "make install" is carrying out the following operations,
> 
> *rm -f /usr/local/lib/coccinelle/spatch
> rm -f /usr/local/lib/coccinelle/spatch.opt
> rm -f /usr/local/bin/pycocci*
> 
> I wonder whether this is causing the problem.

I am also unsure about your software situation at the moment.
Would you like to share any more information about your test environment?


> Would be great if you could advise me on this.

I hope that we can help you a bit more.

Regards,
Markus

[Cocci] Parsing C source files by Coccinelle

[Julia Lawall]

On Mon, 7 Mar 2016, SF Markus Elfring wrote:

> > I am interested in exploring how Coccinelle is parsing C programs,
>
> The software library "http://cristal.inria.fr/~fpottier/menhir/"
> is involved to some degree.

None whatsoever, actually.  A parser is implemented using ocamlyacc,
combined with some hauristics that reorganize the input tokens a little
bit.  The approach is described in the following paper:

Yoann Padioleau:
Parsing C/C++ Code without Pre-processing. CC 2009: 109-125

julia

>
>
> > in order to detect the bugs indicated by SmPL patches.
>
> Would you like to develop your own source code search patterns?
>
>
> > This is the reason why I was trying to examine the source
> > and perform the compilation myself.
>
> Will it be a bit easier to ask about relevant implementation details here?
>
>
> > However, I am facing issues executing spatch.
>
> Which ones?
>
>
> > The installation probably didn't work.
>
> How did you get this impression?
>
> Regards,
> Markus
> _______________________________________________
> Cocci mailing list
> Cocci at systeme.lip6.fr
> https://systeme.lip6.fr/mailman/listinfo/cocci
>

[Cocci] Parsing C source files by Coccinelle

[SF Markus Elfring]

>> The software library "http://cristal.inria.fr/~fpottier/menhir/"
>> is involved to some degree.
> 
> None whatsoever, actually.

An option like "--enable-menhirLib" is provided under the configuration category
"Optional Features", isn't it?



> A parser is implemented using ocamlyacc, combined with some hauristics
> that reorganize the input tokens a little bit.  The approach is described in the following paper:
> 
> Yoann Padioleau:
> Parsing C/C++ Code without Pre-processing. CC 2009: 109-125

How much did the software design and corresponding implementation evolve further
since the description of this approach?

Regards,
Markus

[Cocci] Parsing C source files by Coccinelle

[Julia Lawall]

On Mon, 7 Mar 2016, SF Markus Elfring wrote:

> >> The software library "http://cristal.inria.fr/~fpottier/menhir/"
> >> is involved to some degree.
> >
> > None whatsoever, actually.
>
> An option like "--enable-menhirLib" is provided under the configuration category
> "Optional Features", isn't it?

This is for the parsing of the semantic patch.  Not the parsing of the C
code.

> > A parser is implemented using ocamlyacc, combined with some hauristics
> > that reorganize the input tokens a little bit.  The approach is described in the following paper:
> >
> > Yoann Padioleau:
> > Parsing C/C++ Code without Pre-processing. CC 2009: 109-125
>
> How much did the software design and corresponding implementation evolve further
> since the description of this approach?

Not at all.

julia

[Cocci] Coccinelle Compilation Problem: configure file

[Sébastien Hinderer]

Dear Aftab,

Many thanks for your interest in Coccinelle.

> Hi Julia Lawall and the Coccinelle Team,
> 
> I really appreciate your response. I was able to generate the configure
> file using *>./autogen*,
> and all the installation steps. Thank you so much. Unfortunately, now I'm
> unable to execute spatch.
> 
> On executing this:
> 
> *spatch -sp_file demos/simple.cocci demos/simple.c -o /tmp/new_simple.c*
> 
> I get "spatch is not installed".

Odd. What does
which spatch
say? Can you look at the file it points to?
This file should be a shell script that tries to find the real spatch
executable. If you are a bit familiar with shell scripts, perhaps you
can go through this file and see whether the executables it tries to
look for are there?

> Also when I execute the following to test spatch,
> 
> 
> *COCCINELLE_HOME=/home/aftab/workspace/coccinelle-master ./scripts/spatch
> -sp_file demos/simple.cocci demos/simple.c -o /tmp/new_simple.c*
> I get,
> 
> *bash: ./scripts/spatch: No such file or directory*

You can try to replace ./scripts/spatch by ./spatch in this command, I
think it should work better.

> I would like to let you know that I performed all the installation steps (make,
> make install), and I noticed no errors.
> However, I am sceptical about why "make install" is carrying out the
> following operations,
> 
> 
> 
> *rm -f /usr/local/lib/coccinelle/spatchrm -f
> /usr/local/lib/coccinelle/spatch.optrm -f /usr/local/bin/pycocci*
> 
> I wonder whether this is causing the problem. I would be most glad if you
> could please help me install spatch.

I don't think these commands can cause a problem: they are supposed to
make sure there is nothing left from previous installs.

Feel free to send us the output of make install to make sure there is no
problem there.


> 
> Just in case, I installed the latest versions of OCaml and Menhir using
> OPAM.

Okay. FYI, Coccinelle is also packaged for opam. It is not the latest
version but depending on what you intend to do it might be good enough,
at least for a start.

S?bastien.

[Cocci] Parsing C source files by Coccinelle

[Sébastien Hinderer]

Hi,

SF Markus Elfring (2016/03/07 19:39 +0100):
> > I am interested in exploring how Coccinelle is parsing C programs,
> 
> The software library "http://cristal.inria.fr/~fpottier/menhir/"
> is involved to some degree.

No, not for parsing C. Currently, Menhir (an advanced alternative to
ocamlyacc) is used only to generate the SmPL parser, not the C parser.

S?bastien.

[Cocci] Parsing C source files by Coccinelle

[SF Markus Elfring]

>> An option like "--enable-menhirLib" is provided under the configuration category
>> "Optional Features", isn't it?
> 
> This is for the parsing of the semantic patch.

Does this information mean that the software library "Menhir" is not so optional at the moment?


> Not the parsing of the C code.

Is the tool "ocamlyacc" sufficient for this data processing so far?


>>> Yoann Padioleau:
>>> Parsing C/C++ Code without Pre-processing. CC 2009: 109-125
>>
>> How much did the software design and corresponding implementation evolve further
>> since the description of this approach?
> 
> Not at all.

I have got an other impression through the years.

Your software supports source code transformations also without including header files
and other pre-processing stuff.

Example:
The option "--include-headers-for-types" was added by the release "Coccinelle 1.0.0-rc22".

A developer should decide whether header information is needed for a script in the semantic
patch language or not.


Would you like to increase the usage of "Menhir" as a LR(1) parser generator for the OCaml
programming language.

Regards,
Markus

[Cocci] Parsing C source files by Coccinelle

[SF Markus Elfring]

>> The software library "http://cristal.inria.fr/~fpottier/menhir/"
>> is involved to some degree.
> 
> No, not for parsing C. Currently, Menhir (an advanced alternative to
> ocamlyacc) is used only to generate the SmPL parser, not the C parser.

Thanks also for your clarification.

Would the Coccinelle software benefit from the tool "Menhir" if it will
take care for the parsing of C source files, too?

Regards,
Markus

[Cocci] Parsing C source files by Coccinelle

[Sébastien Hinderer]

SF Markus Elfring (2016/03/08 10:20 +0100):
> Would the Coccinelle software benefit from the tool "Menhir" if it will
> take care for the parsing of C source files, too?

>From a performance point of view it is not sure there would be any gain.
But obviously it would make more sense to use only one parser generator
rather than two. However a bug in Menhir currently prevents its usage to
generate the C parser. Once that bug will be fixed then using Menhir to
generate the C parser will become an option.

S?bastien.

[Cocci] Parsing C source files by Coccinelle

[SF Markus Elfring]

> But obviously it would make more sense to use only one parser generator
> rather than two.

I am curious on how this aspect will evolve.


> However a bug in Menhir currently prevents its usage to generate the C parser.

Would you like to refer to any specific report for this issue?

Regards,
Markus

[Cocci] Parsing C source files by Coccinelle

[Iago Abal]

On Mon, Mar 7, 2016 at 8:55 PM, SF Markus Elfring <
elfring@users.sourceforge.net> wrote:

> > In particular, we would like to see whether it is possible to improve
> Coccinelle results
> > by supplying it with aliasing information about variables.
>
> Does such a goal indicate that you are going to experiment also with data
> flow analysis?
>

If you are talking about using Coccinelle as a bug finder, and empower it
with alias analysis, then I have spent about a year working on a similar
idea. I am available to discuss my experience with you, if you want. I
don't know exactly in what direction you want to "improve" Coccinelle
results, but let me warn you that in principle alias analysis may just give
you a lot more false positives. It's a double-edged sword... it will help
you with some issues, but you'll get new issues to deal with. Linked data
structures are a common source of confusion, since you will have trouble to
distinguish between individual elements of, for instance, a linked list. If
your alias analysis is very precise it is going to be hard to scale up; and
if your alias analysis is not precise enough, it may not be very useful.
Sometimes, in functions with complex manipulation of pointers, I find that
alias analysis hurts more than it helps. That said, I think advanced
Coccinelle users could make good use of aliasing information.

Iago
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://systeme.lip6.fr/pipermail/cocci/attachments/20160308/3bd96470/attachment-0001.html>

[Cocci] Parsing C source files by Coccinelle

[SF Markus Elfring]

> If you are talking about using Coccinelle as a bug finder,
> and empower it with alias analysis,

I am also curious on how such technologies will fit together.


> then I have spent about a year working on a similar idea.

How similar was this approach?


> I am available to discuss my experience with you,

Nice ?


> if you want.

Yes, please.


> I don't know exactly in what direction you want to "improve" Coccinelle results,
> but let me warn you that in principle alias analysis may just give you
> a lot more false positives.

Do you need also a special software infrastructure to reduce the probability
for the undesired reporting of false positives?


> Sometimes, in functions with complex manipulation of pointers,
> I find that alias analysis hurts more than it helps.

Interesting view ?


> That said, I think advanced Coccinelle users could make good use
> of aliasing information.

How would these data be integrated for the use in SmPL scripts?

Regards,
Markus

[Cocci] Parsing C source files by Coccinelle

[Julia Lawall]

On Tue, 8 Mar 2016, SF Markus Elfring wrote:

> >> An option like "--enable-menhirLib" is provided under the configuration category
> >> "Optional Features", isn't it?
> >
> > This is for the parsing of the semantic patch.
>
> Does this information mean that the software library "Menhir" is not so optional at the moment?

Menhirlib is needed from the Coccinelle sources if you do not already have
menhir installed on your machine.  Thus there are two options.

> >>> Yoann Padioleau:
> >>> Parsing C/C++ Code without Pre-processing. CC 2009: 109-125
> >>
> >> How much did the software design and corresponding implementation evolve further
> >> since the description of this approach?
> >
> > Not at all.
>
> I have got an other impression through the years.
>
> Your software supports source code transformations also without
> including header files and other pre-processing stuff.
>
> Example:
> The option "--include-headers-for-types" was added by the release "Coccinelle 1.0.0-rc22".

It's true, but that has nothing to do with the parsing algorithm.  It has
to do with what files are selected for parsing.  I don't think there has
been any substantial change in the actual parsing algorithm.

julia

[Cocci] Checking build parameters for Menhir

[SF Markus Elfring]

> Menhirlib is needed from the Coccinelle sources if you do not already have
> menhir installed on your machine.  Thus there are two options.

Can the build configuration parameters become clearer for this software requirement?

Regards,
Markus

[Cocci] Parsing C source files by Coccinelle

[Iago Abal]

> How similar was this approach?
>
The query language is not as expressive as Coccinelle SmPL, but you can
write queries like

  ... when P
  Q

Here P and Q, apart from matching syntax, can make use of aliasing and
other lightweight semantic information.


> Do you need also a special software infrastructure to reduce the
> probability
> for the undesired reporting of false positives?
>
In my case, alias analysis allowed me to find bugs that I could not find
easily with Coccinelle, using similar or relatively simpler queries
(simpler here is arguable of course). But it has introduced a lot more
false positives. I basically had to fine tune the alias analysis, and made
a lot of unsound and sometimes project-specific assumptions. Good support
of structs is a must and is probably the trickiest part.

How would these data be integrated for the use in SmPL scripts?
>

In its simplest form, using OCaml or Python scripts within SmPL patches to
access external aliasing data. You could ran a separate tool to build an
aliasing database, and then once you have matched two variables `x' and `y'
with Coccinelle, you may query such database to know whether those two
variables may be aliased. Some features would need a proper integration
with Coccinelle though, and that's another nontrivial task.

If you want to do this you should also note that Coccinelle runs on code on
which neither macros nor #if have been resolved. Your alias analysis will
likely work on preprocessed code, therefore your aliasing information may
be incomplete or even misleading. All in all, truly integrating alias
analysis into a fully functional Coccinelle is a project that will take you
a few years, IMHO.

Iago
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://systeme.lip6.fr/pipermail/cocci/attachments/20160308/850bb3a2/attachment.html>

[Cocci] Parsing C source files by Coccinelle

[SF Markus Elfring]

> Some features would need a proper integration with Coccinelle though,

Have you got any further details in mind already?


> and that's another nontrivial task.

Do you know any more software developers who would like to help here?

Regards,
Markus

[Cocci] Parsing C source files by Coccinelle

[Iago Abal]

> > Some features would need a proper integration with Coccinelle though,
>
> Have you got any further details in mind already?
>
Not very far. :) One of the problems I see are when clauses.

Imagine that you are interested in finding uses after free. You may write a
pattern more or less like

   kfree(x);
   ... when != y = E
   z

where `y' and `z' are supposed to be aliases of `x'. If you say "when != x
= E" you can miss cases in which `x' is assigned through an alias, but with
"when != y = E" that would match any variable `y' regardless of being
aliased with `x' or not. You would like to say, "when != y = E such that
aliased(x,y)", and for that Coccinelle's CTL engine should know about
aliasing, or have support for user-supplied predicates.


> > and that's another nontrivial task.
>
> Do you know any more software developers who would like to help here?

Not really. You may prefer to think what is cool about using Coccinelle as
a bug finder, and put a bit of that into an existing bug finder instead.

Iago
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://systeme.lip6.fr/pipermail/cocci/attachments/20160309/4914571e/attachment.html>

[Cocci] Parsing C source files by Coccinelle

[SF Markus Elfring]

> Imagine that you are interested in finding uses after free. You may write a pattern more or less like
> 
>    kfree(x);
>    ... when != y = E
>    z
> 
> If you say "when != x = E" you can miss cases in which `x' is assigned through an alias,
> but with "when != y = E" that would match any variable `y' regardless of being aliased with `x' or not.

Would you like to restrict the source code search on data accesses over pointers
(or C++ references) in this use case?


> You would like to say, "when != y = E such that aliased(x,y)",

Would such a specification be a higher-level search operation?


> and for that Coccinelle's CTL engine should know about aliasing,

Did you fiddle with extensions for the technology "computation tree logic" eventually?


> or have support for user-supplied predicates.

Would you like to submit a feature request?

Regards,
Markus

[Cocci] Parsing C source files by Coccinelle

[Julia Lawall]

On Wed, 9 Mar 2016, Iago Abal wrote:

>
>       > Some features would need a proper integration with Coccinelle
>       though,
>
>       Have you got any further details in mind already?
>
> Not very far. :) One of the problems I see are when clauses.
>
> Imagine that you are interested in finding uses after free. You may write a
> pattern more or less like
>
> ?? kfree(x);
> ?? ... when != y = E
> ?? z
>
> where `y' and `z' are supposed to be aliases of `x'. If you say "when != x =
> E" you can miss cases in which `x' is assigned through an alias, but with
> "when != y = E" that would match any variable `y' regardless of being
> aliased with `x' or not. You would like to say, "when != y = E such that
> aliased(x,y)", and for that Coccinelle's CTL engine should know about
> aliasing, or have support for user-supplied predicates.

Predicates are coming.  There are already predicates on position
variables.  However, I would be a bit nervous about predicates on the
relationship between two unbound metavariables.  In the past, eg the
subexpression relationship:

expression e1 <= r.e2;

the expression of which a subexpression is to be found has been restricted
to be an inherited metavariable, and thus to have a fixed value.  I don't
know what would be the performance impact of removing this restriction,
though.  Maybe it would be fine in practice.

What is planned for the predicates is that the metavariable being declared
is an argument of the predicate, and then any number of inherited
metavariables as well.

julia