[cocci] Replace custom function multiple times

[cocci] Replace custom function multiple times

[Sebastiano Miano]

Hello,

I am trying to use Coccinelle to replace a function that is called
multiple times in the code with a different version whose name is
taken from the string variable that is passed as the second parameter.

To explain better, my C code is the following:

#include <string.h>

static int process(__u64 off) {
  MERGE_FUNCTION_OPENED("file.c", "ping_pong", arg1, arg2, arg3, arg4, NULL);
  MERGE_FUNCTION_OPENED("file2.c", "ping_pong2", arg1, arg2, arg3, NULL);
  return 0;
}

I want my code to be transformed into the following version:

#include <string.h>
#include "file.h"

static int process(__u64 off) {
  ping_pong(arg1, arg2, arg3, arg4, NULL);
  ping_pong2(arg1, arg2, arg3, NULL);
  return 0;
}

where the string that is passed as the second parameter is used as the
function name, and all the other arguments are copied as they are.

I managed to write a working cocci script (listed below) that is doing the job.

@merge_annotation@
expression path, fn;
@@

MERGE_FUNCTION_OPENED(path, fn, ...);

@script:python merge_p@
path << merge_annotation.path;
fn << merge_annotation.fn;
new_fn;
include_name;
@@
print "Obtained path: %s and file name: %s" % (path,fn)
coccinelle.new_fn = cocci.make_expr(fn.strip('\"'))

import os
pre, ext = os.path.splitext(path.strip('\"'))
p = "%s.h" % (pre)

coccinelle.include_name = cocci.make_ident("\n#include \"%s\"" % (p))

@c@
expression merge_p.new_fn;
expression E1, E2;
expression list E3;
@@
- MERGE_FUNCTION_OPENED(E1, E2, E3);
+ new_fn(E3);

@add_include depends on c@
identifier merge_p.include_name;
@@

#include <...>
+ include_name

Unfortunately, this script works only when a single instance of the
MERGE_FUNCTION_OPENED(...) function is used. When I use both of them I
get a "c: already tagged token:" error.
Using "++" instead of "+" removes the error, but produces a code that
contains 2 calls to the ping_pong function, and 2 calls to the
ping_pong2 function.
Any ideas on how to solve this issue?

Thanks in advance,
Sebastiano

Re: [cocci] Replace custom function multiple times

[Julia Lawall]

On Wed, 2 Nov 2022, Sebastiano Miano wrote:

> Hello,
>
> I am trying to use Coccinelle to replace a function that is called
> multiple times in the code with a different version whose name is
> taken from the string variable that is passed as the second parameter.
>
> To explain better, my C code is the following:
>
> #include <string.h>
>
> static int process(__u64 off) {
>   MERGE_FUNCTION_OPENED("file.c", "ping_pong", arg1, arg2, arg3, arg4, NULL);
>   MERGE_FUNCTION_OPENED("file2.c", "ping_pong2", arg1, arg2, arg3, NULL);
>   return 0;
> }
>
> I want my code to be transformed into the following version:
>
> #include <string.h>
> #include "file.h"
>
> static int process(__u64 off) {
>   ping_pong(arg1, arg2, arg3, arg4, NULL);
>   ping_pong2(arg1, arg2, arg3, NULL);
>   return 0;
> }
>
> where the string that is passed as the second parameter is used as the
> function name, and all the other arguments are copied as they are.
>
> I managed to write a working cocci script (listed below) that is doing the job.
>
> @merge_annotation@
> expression path, fn;
> @@
>
> MERGE_FUNCTION_OPENED(path, fn, ...);
>
> @script:python merge_p@
> path << merge_annotation.path;
> fn << merge_annotation.fn;
> new_fn;
> include_name;
> @@
> print "Obtained path: %s and file name: %s" % (path,fn)
> coccinelle.new_fn = cocci.make_expr(fn.strip('\"'))
>
> import os
> pre, ext = os.path.splitext(path.strip('\"'))
> p = "%s.h" % (pre)
>
> coccinelle.include_name = cocci.make_ident("\n#include \"%s\"" % (p))
>
> @c@
> expression merge_p.new_fn;
> expression E1, E2;
> expression list E3;

The problem is here.  You don't put any constraints on the arguments of
MERGE_FUNCTION_OPENED, so both new variables (ping_pong and ping_pong2)
affect both calls.  You should use merge_annotation.path and
merge_annotation.fn to be sure to get only the calls that are related to
the new names you have made.

julia

> @@
> - MERGE_FUNCTION_OPENED(E1, E2, E3);
> + new_fn(E3);
>
> @add_include depends on c@
> identifier merge_p.include_name;
> @@
>
> #include <...>
> + include_name
>
> Unfortunately, this script works only when a single instance of the
> MERGE_FUNCTION_OPENED(...) function is used. When I use both of them I
> get a "c: already tagged token:" error.
> Using "++" instead of "+" removes the error, but produces a code that
> contains 2 calls to the ping_pong function, and 2 calls to the
> ping_pong2 function.
> Any ideas on how to solve this issue?
>
> Thanks in advance,
> Sebastiano
>

Re: [cocci] Replace custom function multiple times

[Sebastiano Miano]

Hi Julia,

On Wed, 2 Nov 2022 at 18:30, Julia Lawall <julia.lawall@inria.fr> wrote:
>
>
>
> On Wed, 2 Nov 2022, Sebastiano Miano wrote:
>
> > Hello,
> >
> > I am trying to use Coccinelle to replace a function that is called
> > multiple times in the code with a different version whose name is
> > taken from the string variable that is passed as the second parameter.
> >
> > To explain better, my C code is the following:
> >
> > #include <string.h>
> >
> > static int process(__u64 off) {
> >   MERGE_FUNCTION_OPENED("file.c", "ping_pong", arg1, arg2, arg3, arg4, NULL);
> >   MERGE_FUNCTION_OPENED("file2.c", "ping_pong2", arg1, arg2, arg3, NULL);
> >   return 0;
> > }
> >
> > I want my code to be transformed into the following version:
> >
> > #include <string.h>
> > #include "file.h"
> >
> > static int process(__u64 off) {
> >   ping_pong(arg1, arg2, arg3, arg4, NULL);
> >   ping_pong2(arg1, arg2, arg3, NULL);
> >   return 0;
> > }
> >
> > where the string that is passed as the second parameter is used as the
> > function name, and all the other arguments are copied as they are.
> >
> > I managed to write a working cocci script (listed below) that is doing the job.
> >
> > @merge_annotation@
> > expression path, fn;
> > @@
> >
> > MERGE_FUNCTION_OPENED(path, fn, ...);
> >
> > @script:python merge_p@
> > path << merge_annotation.path;
> > fn << merge_annotation.fn;
> > new_fn;
> > include_name;
> > @@
> > print "Obtained path: %s and file name: %s" % (path,fn)
> > coccinelle.new_fn = cocci.make_expr(fn.strip('\"'))
> >
> > import os
> > pre, ext = os.path.splitext(path.strip('\"'))
> > p = "%s.h" % (pre)
> >
> > coccinelle.include_name = cocci.make_ident("\n#include \"%s\"" % (p))
> >
> > @c@
> > expression merge_p.new_fn;
> > expression E1, E2;
> > expression list E3;
>
> The problem is here.  You don't put any constraints on the arguments of
> MERGE_FUNCTION_OPENED, so both new variables (ping_pong and ping_pong2)
> affect both calls.  You should use merge_annotation.path and
> merge_annotation.fn to be sure to get only the calls that are related to
> the new names you have made.

That's amazing, it works now.
Thanks a lot for the clarification.

>
> julia
>
> > @@
> > - MERGE_FUNCTION_OPENED(E1, E2, E3);
> > + new_fn(E3);
> >
> > @add_include depends on c@
> > identifier merge_p.include_name;
> > @@
> >
> > #include <...>
> > + include_name
> >
> > Unfortunately, this script works only when a single instance of the
> > MERGE_FUNCTION_OPENED(...) function is used. When I use both of them I
> > get a "c: already tagged token:" error.
> > Using "++" instead of "+" removes the error, but produces a code that
> > contains 2 calls to the ping_pong function, and 2 calls to the
> > ping_pong2 function.
> > Any ideas on how to solve this issue?
> >
> > Thanks in advance,
> > Sebastiano
> >

Re: [cocci] Replace custom function multiple times

[Markus Elfring]

> Any ideas on how to solve this issue?


How do you think about to use the following SmPL rule variants instead?

@initialize:python@
@@
import os

@searching@
expression path, fn;
@@
 MERGE_FUNCTION_OPENED(path, fn, ...);

@script:python generation@
path << searching.path;
fn << searching.fn;
new_fn;
include_name;
@@
coccinelle.new_fn = cocci.make_expr(fn.strip('\"'))
pre, ext = os.path.splitext(path.strip('\"'))
coccinelle.include_name = cocci.make_ident("\n#include \"%s.h\"" % (pre))

@replacement@
expression searching.path, searching.fn, generation.new_fn;
@@
-MERGE_FUNCTION_OPENED
+new_fn
 (
- path, fn,
  ...
 );

@addition depends on replacement@
identifier generation.include_name;
@@
 #include <...>
+include_name


Would you like to reconsider also the implementation detail that an “identifier”
is used for the construction of a preprocessor directive so far?

Regards,
Markus

Re: [cocci] Replace custom function multiple times

[Sebastiano Miano]

Thanks Markus.
It definitely looks cleaner now.

On Wed, 2 Nov 2022 at 20:15, Markus Elfring <Markus.Elfring@web.de> wrote:
>
>
> > Any ideas on how to solve this issue?
>
>
> How do you think about to use the following SmPL rule variants instead?
>
> @initialize:python@
> @@
> import os
>
> @searching@
> expression path, fn;
> @@
>  MERGE_FUNCTION_OPENED(path, fn, ...);
>
> @script:python generation@
> path << searching.path;
> fn << searching.fn;
> new_fn;
> include_name;
> @@
> coccinelle.new_fn = cocci.make_expr(fn.strip('\"'))
> pre, ext = os.path.splitext(path.strip('\"'))
> coccinelle.include_name = cocci.make_ident("\n#include \"%s.h\"" % (pre))
>
> @replacement@
> expression searching.path, searching.fn, generation.new_fn;
> @@
> -MERGE_FUNCTION_OPENED
> +new_fn
>  (
> - path, fn,
>   ...
>  );
>
> @addition depends on replacement@
> identifier generation.include_name;
> @@
>  #include <...>
> +include_name
>
>
> Would you like to reconsider also the implementation detail that an “identifier”
> is used for the construction of a preprocessor directive so far?

I would love to reconsider it, but I didn't find an alternate solution.
Do you have any suggestions?

Thanks,
Sebastiano

Re: [cocci] Replace custom function multiple times

[Julia Lawall]

[-- Attachment #1: Type: text/plain, Size: 1453 bytes --]



On Thu, 3 Nov 2022, Sebastiano Miano wrote:

> Thanks Markus.
> It definitely looks cleaner now.
>
> On Wed, 2 Nov 2022 at 20:15, Markus Elfring <Markus.Elfring@web.de> wrote:
> >
> >
> > > Any ideas on how to solve this issue?
> >
> >
> > How do you think about to use the following SmPL rule variants instead?
> >
> > @initialize:python@
> > @@
> > import os
> >
> > @searching@
> > expression path, fn;
> > @@
> >  MERGE_FUNCTION_OPENED(path, fn, ...);
> >
> > @script:python generation@
> > path << searching.path;
> > fn << searching.fn;
> > new_fn;
> > include_name;
> > @@
> > coccinelle.new_fn = cocci.make_expr(fn.strip('\"'))
> > pre, ext = os.path.splitext(path.strip('\"'))
> > coccinelle.include_name = cocci.make_ident("\n#include \"%s.h\"" % (pre))
> >
> > @replacement@
> > expression searching.path, searching.fn, generation.new_fn;
> > @@
> > -MERGE_FUNCTION_OPENED
> > +new_fn
> >  (
> > - path, fn,
> >   ...
> >  );
> >
> > @addition depends on replacement@
> > identifier generation.include_name;
> > @@
> >  #include <...>
> > +include_name
> >
> >
> > Would you like to reconsider also the implementation detail that an “identifier”
> > is used for the construction of a preprocessor directive so far?
>
> I would love to reconsider it, but I didn't find an alternate solution.
> Do you have any suggestions?

I don't think any other solution is available.  There is not much
functionality available around includes.

julia

Re: [cocci] Replace custom function multiple times

[Markus Elfring]

>> Any ideas on how to solve this issue?
> How do you think about to use the following SmPL rule variants instead?

I got the impression that the usage of position variables can have desirable
data processing effects for the safer evaluation of some SmPL rules.

See also previous clarification approaches:
2016-10-26
Propagating values back from Python script to SmPL rule with other metavariable
type than “identifier”
https://github.com/coccinelle/coccinelle/issues/86#issuecomment-256453084


Thus I tried another SmPL script variant out.

@initialize:python@
@@
import os

@searching@
position pos;
expression path, fn;
@@
 MERGE_FUNCTION_OPENED@pos(path, fn, ...);

@script:python generation@
pos << searching.pos;
path << searching.path;
fn << searching.fn;
new_fn;
include_name;
@@
coccinelle.new_fn = cocci.make_expr(fn.strip('\"'))
pre, ext = os.path.splitext(path.strip('\"'))
coccinelle.include_name = cocci.make_ident("\n#include \"%s.h\"" % (pre))

@replacement@
position searching.pos;
expression searching.path, searching.fn, generation.new_fn;
@@
-MERGE_FUNCTION_OPENED@pos
+new_fn
 (
- path, fn,
  ...
 );

@addition depends on replacement@
identifier generation.include_name;
@@
 #include <...>
+include_name


Will interests grow for remaining development concerns (also according to
the construction of preprocessor directives)?

Regards,
Markus

Re: [cocci] Replace custom function multiple times

[Markus Elfring]

> It definitely looks cleaner now.


Thanks for your positive feedback.



>> @addition depends on replacement@
>> identifier generation.include_name;
>> @@
>>  #include <...>
>> +include_name
>>
>>
>> Would you like to reconsider also the implementation detail that an “identifier”
>> is used for the construction of a preprocessor directive so far?
> I would love to reconsider it, but I didn't find an alternate solution.


Corresponding clarification approaches are in waiting queues, aren't they?


> Do you have any suggestions?

Yes, of course.

I hope that several ideas can become more attractive and they can acquire
sufficient development resources accordingly.

Regards,
Markus

Re: [cocci] Replace custom function multiple times

[Julia Lawall]

[-- Attachment #1: Type: text/plain, Size: 1777 bytes --]



On Thu, 3 Nov 2022, Markus Elfring wrote:

> >> Any ideas on how to solve this issue?
> > How do you think about to use the following SmPL rule variants instead?
>
> I got the impression that the usage of position variables can have desirable
> data processing effects for the safer evaluation of some SmPL rules.


It's better without position variables.  If the same path and fn are used
in multiple cases, adding the position variables will cause the python
code to be run for each call, whereas now it is only run for each unique
pair of path and fn.

julia

>
> See also previous clarification approaches:
> 2016-10-26
> Propagating values back from Python script to SmPL rule with other metavariable
> type than “identifier”
> https://github.com/coccinelle/coccinelle/issues/86#issuecomment-256453084
>
>
> Thus I tried another SmPL script variant out.
>
> @initialize:python@
> @@
> import os
>
> @searching@
> position pos;
> expression path, fn;
> @@
>  MERGE_FUNCTION_OPENED@pos(path, fn, ...);
>
> @script:python generation@
> pos << searching.pos;
> path << searching.path;
> fn << searching.fn;
> new_fn;
> include_name;
> @@
> coccinelle.new_fn = cocci.make_expr(fn.strip('\"'))
> pre, ext = os.path.splitext(path.strip('\"'))
> coccinelle.include_name = cocci.make_ident("\n#include \"%s.h\"" % (pre))
>
> @replacement@
> position searching.pos;
> expression searching.path, searching.fn, generation.new_fn;
> @@
> -MERGE_FUNCTION_OPENED@pos
> +new_fn
>  (
> - path, fn,
>   ...
>  );
>
> @addition depends on replacement@
> identifier generation.include_name;
> @@
>  #include <...>
> +include_name
>
>
> Will interests grow for remaining development concerns (also according to
> the construction of preprocessor directives)?
>
> Regards,
> Markus
>

Re: [cocci] Replace custom function multiple times

[Markus Elfring]

>> I got the impression that the usage of position variables can have desirable
>> data processing effects for the safer evaluation of some SmPL rules.
>
> It's better without position variables.  If the same path and fn are used
> in multiple cases, adding the position variables will cause the python
> code to be run for each call, whereas now it is only run for each unique
> pair of path and fn.

Can such information be transformed into enhancements for the software documentation?

Regards,
Markus

Re: [cocci] Replace custom function multiple times

[Markus Elfring]

>> @searching@
>> expression path, fn;
>> @@
>>  MERGE_FUNCTION_OPENED(path, fn, ...);

Do you find the metavariable type “constant char[]” more appropriate here?



>> @script:python generation@
>> path << searching.path;
>> fn << searching.fn;
>> new_fn;
>> include_name;
>> @@
>> coccinelle.new_fn = cocci.make_expr(fn.strip('\"'))


Are ordinary function names passed by string literals instead of detailed expressions?
How do you think about to use the method “make_ident” here?

Regards,
Markus

Re: [cocci] Replace custom function multiple times

[Julia Lawall]

[-- Attachment #1: Type: text/plain, Size: 692 bytes --]



On Fri, 4 Nov 2022, Markus Elfring wrote:

> >> @searching@
> >> expression path, fn;
> >> @@
> >>  MERGE_FUNCTION_OPENED(path, fn, ...);
>
> Do you find the metavariable type “constant char[]” more appropriate here?

This is a reasonable suggestion.  But if the rule works as is, then it's
already fine.

julia

>
>
>
> >> @script:python generation@
> >> path << searching.path;
> >> fn << searching.fn;
> >> new_fn;
> >> include_name;
> >> @@
> >> coccinelle.new_fn = cocci.make_expr(fn.strip('\"'))
>
>
> Are ordinary function names passed by string literals instead of detailed expressions?
> How do you think about to use the method “make_ident” here?
>
> Regards,
> Markus
>
>

Re: [cocci] Replace custom function multiple times

[Markus Elfring]

>> Do you find the metavariable type “constant char[]” more appropriate here?
> This is a reasonable suggestion.  But if the rule works as is, then it's
> already fine.


* Will developers become occasionally interested in different software run time characteristics?

* Can interests grow for improved handling of string literals (also with help
  of the semantic patch language)?

Regards,
Markus