[Cocci] 1.0.2

[Cocci] 1.0.2

[Julia Lawall]

Some of the main changes:

* OCaml 3.10 is no longer supported.

* --use-gitgrep falls back to --use-coccigrep for a non-git repository.
Thus, it is generally safe to use --use-gitgrep, except when processing an
unmanaged subdirectory of a git repository.

* It is no longer possible to create an identifier in OCaml code by saying
x := "identifier_name".  Instead some constructors have been added for
constructing various kinds of code:

val make_ident : string -> Ast_c.metavar_binding_kind
   (* string is the identifier name *)

val make_expr : string -> Ast_c.metavar_binding_kind
   (* string is the expression *)

val make_expr_with_env : string -> string -> Ast_c.metavar_binding_kind
   (* The first argument is a series of variable declarations, designed to
      provide type information for the second argument, which is an
      expression.  For example, make_expr_with_env "int x;" "x+3". *)

val make_stmt : string -> Ast_c.metavar_binding_kind
   (* The string argument is a statement *)

val make_stmt_with_env : string -> string -> Ast_c.metavar_binding_kind
   (* The first argument is a series of declarations and the second
      argument is the statement to create *)

val make_type : string -> Ast_c.metavar_binding_kind
   (* The string is the type to create *)

val make_listlen : int -> Ast_c.metavar_binding_kind
   (* The integer is the list length to create, eg for a parameter list *)

val make_position :
  Common.filename ->
  string -> int -> int -> int -> int -> Ast_c.metavar_binding_kind
    (* The arguments are: file name (string), function name, start line,
       start column, end line, end column.  Lines start with 1, columns
       start with 0. *)

* exp at S will bind statement metavariable to the smallest statement
enclosing the expression exp.  For example, for the C code, return x;, x at S
would match S to return x;

* If an identifier metavariable is constrained to match a set of possible
names, eg identifier fn = {kmalloc, kfree};, then these names will be
taken into account when selecting files.  On the other hand, constraints
expressed as regular expressions are still not taken into account.

* Autogen creates a version number that corresponds to the most recent
commit.

* -j should now work correctly.

julia

[Cocci] release: 1.0.2: Better file selection

[SF Markus Elfring]

> * If an identifier metavariable is constrained to match a set of
>   possible names, eg identifier fn = {kmalloc, kfree};, then these names
>   will be taken into account when selecting files.

Will any more information about this file selection algorithm appear
also in the manual?

Regards,
Markus

[Cocci] release: 1.0.2: Software evolution around parallelisation support

[SF Markus Elfring]

> * -j should now work correctly.

An incompatibility for the SmPL rules "initialize" and "finalize"
is mentioned in the manual.
https://github.com/coccinelle/coccinelle/blob/a46bef70162d17cec6b0fc6101d737989f735ee4/docs/manual/spatch_options.tex#L673

Can this software limitation be explained better there?


Does the parallelisation support need any more fine-tuning?
* How often is task preparation required before further data processing
  can be performed in parallel?

* Which programming interfaces do you prefer for the storage
  of collected data in the fork-join work flow?

Regards,
Markus

[Cocci] release: 1.0.2: Software evolution around parallelisation support

[Julia Lawall]

On Sun, 23 Aug 2015, SF Markus Elfring wrote:

> > * -j should now work correctly.
>
> An incompatibility for the SmPL rules "initialize" and "finalize"
> is mentioned in the manual.
> https://github.com/coccinelle/coccinelle/blob/a46bef70162d17cec6b0fc6101d737989f735ee4/docs/manual/spatch_options.tex#L673
>
> Can this software limitation be explained better there?

There is no shared memory in ocaml.  initialize and finalize naturally
suggest that locations created in initialize can all be collected in a
single execution of finalize. Since this is not possible, initialize and
finalize are not allowed.  If you are OK with disjoint memory spaces per
thread, just use the existing script for parallelism.

>
>
> Does the parallelisation support need any more fine-tuning?
> * How often is task preparation required before further data processing
>   can be performed in parallel?
>
> * Which programming interfaces do you prefer for the storage
>   of collected data in the fork-join work flow?

No idea what either of these questions mean, but the whole point is that
there is no join.

julia

[Cocci] release: 1.0.2: Software evolution around parallelisation support

[SF Markus Elfring]

> There is no shared memory in ocaml.

Specific services (like databases) can provide some shared data storage.


> initialize and finalize naturally suggest that locations created in initialize
> can all be collected in a single execution of finalize. Since this is not possible,

I imagine that these SmPL rules could still be supported for their
original purpose.


> initialize and finalize are not allowed.

It might be easier for a moment to exclude these components under the
constraints of limited software development resources.
Such limitations will result in some software development challenges,
won't it?


> If you are OK with disjoint memory spaces per thread,
> just use the existing script for parallelism.

How do you think about the potential for extensions around task-specific
preparation and clean-up?


>> Does the parallelisation support need any more fine-tuning?
>> * How often is task preparation required before further data processing
>>   can be performed in parallel?
>>
>> * Which programming interfaces do you prefer for the storage
>>   of collected data in the fork-join work flow?
> No idea what either of these questions mean, but the whole point is that
> there is no join.

This technical restriction can lead to data processing difficulties,
can't it?

Are there any situations where SmPL scripts will need more control about
parallel work?

Regards,
Markus

[Cocci] release: 1.0.2: Software evolution around parallelisation support

[SF Markus Elfring]

>> Does the parallelisation support need any more fine-tuning?
>> * How often is task preparation required before further data processing
>>   can be performed in parallel?
>>
>> * Which programming interfaces do you prefer for the storage
>>   of collected data in the fork-join work flow?
> 
> No idea what either of these questions mean, but the whole point is that
> there is no join.

This view needs an adjustment.

A fork-join work flow is implemented by the Coccinelle software for
its main programs with the help of the application programming interface "parmap",
isn't it?
https://rdicosmo.github.io/parmap/

The manual contains the following information for this approach.
* Preparation for spawning subprocesses:
  "This option furthermore creates a temporary directory in the directory
  from which spatch is executed that has the name of the semantic patch
  (without its extension) and that contains stdout and stderr files
  generated by the various processes."

* Actions for joining:
  "When the semantic patch completes, the contents of these files are printed
  to standard output and standard error, respectively, and the directory
  is removed."


I propose to reactivate the support for the SmPL rules "initialize" and "finalize"
for all uses cases without unwanted software limitations.
I imagine that such rules can still be evaluated and executed at the appropriate
data processing phases after a bit more software development even if
the multi-programming technique is reused.

Regards,
Markus

[Cocci] release: 1.0.2: Software evolution around parallelisation support

[Julia Lawall]

On Tue, 25 Aug 2015, SF Markus Elfring wrote:

> >> Does the parallelisation support need any more fine-tuning?
> >> * How often is task preparation required before further data processing
> >>   can be performed in parallel?
> >>
> >> * Which programming interfaces do you prefer for the storage
> >>   of collected data in the fork-join work flow?
> > 
> > No idea what either of these questions mean, but the whole point is that
> > there is no join.
> 
> This view needs an adjustment.
> 
> A fork-join work flow is implemented by the Coccinelle software for
> its main programs with the help of the application programming interface "parmap",
> isn't it?
> https://rdicosmo.github.io/parmap/

Parmap only forks.  It can join what is returned explicitly.  The problem 
is that the scripting language environment is separate from the 
environment of Coccinelle itself.  So there is no easy way to get the 
right values into the right variables.  Coccinelle doesn't know anything 
about the variables of the ocaml or python scripts.

> The manual contains the following information for this approach.
> * Preparation for spawning subprocesses:
>   "This option furthermore creates a temporary directory in the directory
>   from which spatch is executed that has the name of the semantic patch
>   (without its extension) and that contains stdout and stderr files
>   generated by the various processes."
> 
> * Actions for joining:
>   "When the semantic patch completes, the contents of these files are printed
>   to standard output and standard error, respectively, and the directory
>   is removed."

Yes but stndard output is a global thing.

> 
> I propose to reactivate the support for the SmPL rules "initialize" and "finalize"
> for all uses cases without unwanted software limitations.

There is no way that Coccinelle can tell what is a "use case with an 
unwanted software limitation.  Coccinelle doesn't know what the scripting 
code does.  I would prefer not to promise a behavior that is not 
implemented.  As I said before, if an unshared initialize and finalize are 
acceptable to you, then just use the preexisting parallelism solution, 
which is still supported.

julia

[Cocci] release: 1.0.2: Software evolution around parallelisation support

[SF Markus Elfring]

> The problem is that the scripting language environment is separate from the 
> environment of Coccinelle itself.  So there is no easy way to get the 
> right values into the right variables.

I can also understand corresponding software development challennges to some degree.


> Coccinelle doesn't know anything about the variables of the ocaml or python scripts.

How do you think about possibilities to extend its knowledge?
Can an improved software design help to achieve the needed data exchange?


>> I propose to reactivate the support for the SmPL rules "initialize" and "finalize"
>> for all uses cases without unwanted software limitations.
> 
> There is no way that Coccinelle can tell what is a "use case with an 
> unwanted software limitation.

Can the exclusion of these rules be seen as a temporary software restriction
when the parameter "--jobs" to the program "spatch"?


> Coccinelle doesn't know what the scripting code does.

But you have still got control on the data which are transferred between
the involved subprocesses.


> I would prefer not to promise a behavior that is not implemented.

I hope that corresponding technical concerns can be improved.


> As I said before, if an unshared initialize and finalize are acceptable to you,
> then just use the preexisting parallelism solution, which is still supported.

I am using this approach with a make script for a while.

How do you think about variants for these actions at the "unshared" stage
and the task/subprocess level?

Regards,
Markus

[Cocci] release: 1.0.2: Software evolution around parallelisation support

[SF Markus Elfring]

> There is no shared memory in ocaml.

How do you think about to reuse any other application programming interfaces
for inter-process communication?
https://ocaml.github.io/ocamlunix/sockets.html


Can it be determined from the content of a semantic patch language script
if the spawned subprocesses need any more data exchange (with the parent process)
besides the usual files like standard output and standard error?

Regards,
Markus

[Cocci] release: 1.0.2: Software evolution around parallelisation support

[Julia Lawall]

On Wed, 26 Aug 2015, SF Markus Elfring wrote:

> > There is no shared memory in ocaml.
>
> How do you think about to reuse any other application programming interfaces
> for inter-process communication?
> https://ocaml.github.io/ocamlunix/sockets.html
>
>
> Can it be determined from the content of a semantic patch language script
> if the spawned subprocesses need any more data exchange (with the parent process)
> besides the usual files like standard output and standard error?

In the short term, there is no improvement planned in this direction.  It
is important that Coccinelle does not know anything about the script code
- we don't want to include parsers for other languages, and it would
further raise the barrier for adding other scripting languages.  If you
need initialize and finalize, just use the current parallelism solution.
You will get n results instead of 1 result, but it is better than getting
15 000 results, as would happen with parmap.

julia

[Cocci] release: 1.0.2: Software evolution around parallelisation support

[SF Markus Elfring]

>> Can it be determined from the content of a semantic patch language script
>> if the spawned subprocesses need any more data exchange (with the parent process)
>> besides the usual files like standard output and standard error?
> 
> In the short term, there is no improvement planned in this direction.
> It is important that Coccinelle does not know anything about the script code

Thanks for your feedback.


> - we don't want to include parsers for other languages,

I assume that might mean that no such extensions will be added in the near future.
I guess that the scripting interfaces are generally extensible so that
other programming languages can be integrated in principle if someone would be
keen enough to invest further software development efforts.


> and it would further raise the barrier for adding other scripting languages.

The inherent system complexity was already increased just by the added
parallelisation support with the parmap interface.


> If you need initialize and finalize, just use the current parallelism solution.

I am using this approach around the parameters "--index" and "--max" for a while.
I came similar challenges along during the development of my make scripts.


I imagine that these SmPL rules can still be executed by the main process
while the execution of the other rules will be delegated to subprocesses
(or even threads). The scripting environments in the subprocesses would additionally
check then which variables accesses should be transformed into data processing
requests for the main process.
When would you like to pick corresponding software development challenges up
around inter-process communication?


> You will get n results instead of 1 result, but it is better than getting
> 15 000 results, as would happen with parmap.

There are some trade-offs involved. I find it nicer when the Coccinelle software
can be directly reused for more use cases.

Regards,
Markus