[Cocci] SmPL script construction with included files

[Cocci] SmPL script construction with included files

[SF Markus Elfring]

Hello,

The semantic patch language supports also the inclusion of external files.
http://coccinelle.lip6.fr/docs/main_grammar001.html

* Does syntax matter for this SmPL statement?

* Can SmPL rules be constructed from smaller parts by it?

Regards,
Markus

[Cocci] SmPL script construction with included files

[Julia Lawall]

On Tue, 15 Nov 2016, SF Markus Elfring wrote:

> Hello,
>
> The semantic patch language supports also the inclusion of external files.
> http://coccinelle.lip6.fr/docs/main_grammar001.html
>
> * Does syntax matter for this SmPL statement?
>
> * Can SmPL rules be constructed from smaller parts by it?

I don't understand what it would mean for syntax to or not to matter.  The
main issue seems to be that the includes can only appear before the normal
SmPL rules.

julia

[Cocci] SmPL script construction with included files

[SF Markus Elfring]

>> * Can SmPL rules be constructed from smaller parts by it?
> 
> I don't understand what it would mean for syntax to or not to matter.

I am curious if the included data need to be usable also alone in the
SmPL sense.


> The main issue seems to be that the includes can only appear before the normal
> SmPL rules.

I would appreciate if the SmPL include statement could be used at more
script places.

Regards,
Markus

[Cocci] SmPL script construction with included files

[Julia Lawall]

On Tue, 15 Nov 2016, SF Markus Elfring wrote:

> >> * Can SmPL rules be constructed from smaller parts by it?
> >
> > I don't understand what it would mean for syntax to or not to matter.
>
> I am curious if the included data need to be usable also alone in the
> SmPL sense.

I think that if you have multiple includes, later includes can refer to
rules in earlier includes.

julia

[Cocci] SmPL script construction with included files

[SF Markus Elfring]

> I don't understand what it would mean for syntax to or not to matter.

I am checking if the mentioned SmPL statement could be used just for the
inclusion of a bit of code for a metavariable constraint within a script
template.


> The main issue seems to be that the includes can only appear before the normal
> SmPL rules.

I guess that this information means that I need to concatenate pieces by
tools which are external to the semantic patch language in my use case.

Regards,
Markus

[Cocci] SmPL script construction with included files

[Julia Lawall]

On Wed, 16 Nov 2016, SF Markus Elfring wrote:

> > I don't understand what it would mean for syntax to or not to matter.
>
> I am checking if the mentioned SmPL statement could be used just for the
> inclusion of a bit of code for a metavariable constraint within a script
> template.

In a script rule, the metavariable declarations are interpreted by
Coccinelle, not the scripting language, so an include would not be allowed
there.

Script rules also don't have constraints on metavariables anyway, so I'm
not sure what you are asking about.

julia

[Cocci] SmPL script construction with included files

[SF Markus Elfring]

> In a script rule, the metavariable declarations are interpreted by Coccinelle,
> not the scripting language, so an include would not be allowed there.

This detail was clear for me in principle.


> Script rules also don't have constraints on metavariables anyway,

Can other SmPL rule types specify constraints in the code form of a
regular expression or a programming language script?


> so I'm not sure what you are asking about.

I am looking for ways to avoid source code duplication by inclusion of
other files.
Would it make sense to allow the use of an include statement within the
SmPL variable definition block of a rule?

Regards,
Markus

[Cocci] SmPL script construction with included files

[Julia Lawall]

On Wed, 16 Nov 2016, SF Markus Elfring wrote:

> > In a script rule, the metavariable declarations are interpreted by Coccinelle,
> > not the scripting language, so an include would not be allowed there.
>
> This detail was clear for me in principle.
>
>
> > Script rules also don't have constraints on metavariables anyway,
>
> Can other SmPL rule types specify constraints in the code form of a
> regular expression or a programming language script?

At the moment, the specification of constraints is not very uniform across
the different metavariable types.  We are working on improving this.

> > so I'm not sure what you are asking about.
>
> I am looking for ways to avoid source code duplication by inclusion of
> other files.
> Would it make sense to allow the use of an include statement within the
> SmPL variable definition block of a rule?

I have the impression that what you want would be adequately implemented
by a virtual identifier that could be tested in the code or by a virtual
rule.  For example, the semantic patches in the Linux kernel provide
alternate implementations depending on whether one uses the arguments -D
patch, -D context, etc.

julia

[Cocci] Further developments around SmPL constraints?

[SF Markus Elfring]

>> Can other SmPL rule types specify constraints in the code form of a
>> regular expression or a programming language script?
> At the moment, the specification of constraints is not very uniform across
> the different metavariable types.  We are working on improving this.

* Have you got any time frame in mind for this software evolution?
  How many people are involved already?

* Is this the reason that the functionality ?Script constraints on
meta-variables? is not explained in the SmPL manual so far?
 
https://github.com/coccinelle/coccinelle/blob/cbc751b30d9e02390d60ebed643c8e4a3fa0bb2b/tests/idcon_python.cocci#L1

Regards,
Markus

[Cocci] SmPL script construction with included files

[SF Markus Elfring]

>> I am looking for ways to avoid source code duplication by inclusion of
>> other files.
>> Would it make sense to allow the use of an include statement within the
>> SmPL variable definition block of a rule?
> I have the impression that what you want would be adequately implemented
> by a virtual identifier that could be tested in the code or by a virtual rule.

These syntax elements will fit also to some use cases.


> For example, the semantic patches in the Linux kernel provide alternate
> implementations depending on whether one uses the arguments -D patch, -D context, etc.

These parameters have got an influence on rule dependencies.

I am thinking about an other use case a bit more where I have got a few
SmPL script templates which are working to some degree. If I would like
to use also the functionality ?Script constraints on meta-variables?
(instead of a big regular expression) for a function name selection on
demand there, I am going to recombine existing pieces by a corresponding
make script as a kind of preprocessing step.

Regards,
Markus

[Cocci] SmPL script construction with included files

[Julia Lawall]

> I am thinking about an other use case a bit more where I have got a few
> SmPL script templates which are working to some degree. If I would like
> to use also the functionality ?Script constraints on meta-variables?
> (instead of a big regular expression) for a function name selection on
> demand there, I am going to recombine existing pieces by a corresponding
> make script as a kind of preprocessing step.

The idea with the script constraints on metavariables feature is that the
script code there should only contain a function call.  The actual
function definition should be elsewhere; probably in an initialize rule.

julia

[Cocci] SmPL script construction with included files

[SF Markus Elfring]

>
> The idea with the script constraints on metavariables feature is that the
> script code there should only contain a function call.  The actual
> function definition should be elsewhere; probably in an initialize rule.

This information can be reasonable to some degree.

How will the software situation evolve if each script constraint will
need another bit of initialisation code?
Can it become an interesting software development challenge then if the
constraint specification approach should be switchable for a few of them
by a corresponding make script?

Regards,
Markus

[Cocci] Further developments around function callbacks for SmPL metavariables?

[SF Markus Elfring]

> The idea with the script constraints on metavariables feature is that the
> script code there should only contain a function call.

I would like to add another view to this software evolution for
further considerations.
The purpose of such a function call can be interpreted in different ways,
can't it?

1. I get the impression from the word ?constraint? that something should
   be restricted. Does this aspect mean that the functionality is finally
   mapped to a predicate?

2. Can it be also used as a data generator or content provider for any other
   non-boolean value?

Regards,
Markus

[Cocci] Further developments around function callbacks for SmPL metavariables?

[Julia Lawall]

On Thu, 17 Nov 2016, SF Markus Elfring wrote:

> > The idea with the script constraints on metavariables feature is that the
> > script code there should only contain a function call.
>
> I would like to add another view to this software evolution for
> further considerations.
> The purpose of such a function call can be interpreted in different ways,
> can't it?
>
> 1. I get the impression from the word ?constraint? that something should
>    be restricted. Does this aspect mean that the functionality is finally
>    mapped to a predicate?
>
> 2. Can it be also used as a data generator or content provider for any other
>    non-boolean value?

No.  The return value of the function call must be a boolean.
The metavariable matches if and only if the function call returns true.

julia

[Cocci] Inclusion of script constraints in SmPL

[SF Markus Elfring]

> The idea with the script constraints on metavariables feature is that the
> script code there should only contain a function call.  The actual
> function definition should be elsewhere; probably in an initialize rule.

* Did you consider the use case already when such a constraint specification
  would be included from an external file as a component or together
  with a rule selection for the semantic patch language?

* Would such an inclusion approach trigger special requirements on the
  position for corresponding initialisation code?

Regards,
Markus

[Cocci] Inclusion of script constraints in SmPL

[Julia Lawall]

On Fri, 18 Nov 2016, SF Markus Elfring wrote:

> > The idea with the script constraints on metavariables feature is that the
> > script code there should only contain a function call.  The actual
> > function definition should be elsewhere; probably in an initialize rule.
>
> * Did you consider the use case already when such a constraint specification
>   would be included from an external file as a component or together
>   with a rule selection for the semantic patch language?
>
> * Would such an inclusion approach trigger special requirements on the
>   position for corresponding initialisation code?

The constraint specification should be a single function call.  You can
set up your @initialize:python@ code to import whatever you want.

julia

[Cocci] Inclusion of script constraints in SmPL

[SF Markus Elfring]

>> * Would such an inclusion approach trigger special requirements on the
>>   position for corresponding initialisation code?
> 
> The constraint specification should be a single function call.

Does the script constraint syntax allow more programming language code there?


> You can set up your @initialize:python@ code to import whatever you want.

I would prefer to keep initialisation code for the implementation of the
predicate function closely together with the call of the included metavariable.

Regards,
Markus

[Cocci] Inclusion of script constraints in SmPL

[Julia Lawall]

On Fri, 18 Nov 2016, SF Markus Elfring wrote:

> >> * Would such an inclusion approach trigger special requirements on the
> >>   position for corresponding initialisation code?
> >
> > The constraint specification should be a single function call.
>
> Does the script constraint syntax allow more programming language code there?

For all practical purposes, no.  Concretely, the script constraint code is
initially parsed using a C parser, not a parser for the script language.
Function calls, of the form f(a,b,c) are in the intersection of C and the
supported scripting languages.

> > You can set up your @initialize:python@ code to import whatever you want.
>
> I would prefer to keep initialisation code for the implementation of the
> predicate function closely together with the call of the included
> metavariable.

You are most welcome to implement in ocaml the lexer and parser for the
scripting language of your choice.

julia

[Cocci] Inclusion of script constraints in SmPL

[SF Markus Elfring]

>> Does the script constraint syntax allow more programming language code there?
> 
> For all practical purposes, no.

If you would like to be so strict, I wonder then about the curly brackets
in the published test example.
https://github.com/coccinelle/coccinelle/blob/cbc751b30d9e02390d60ebed643c8e4a3fa0bb2b/tests/idcon_python.cocci#L1


> Concretely, the script constraint code is initially parsed using a C parser,

I find such an information interesting to some degree.

Why do you pass data to a software component for ?C? at this place?


> not a parser for the script language.

I do not see that would be needed in the restricted use case.


> Function calls, of the form f(a,b,c) are in the intersection of C and the
> supported scripting languages.

Have you got more control about the construction of a single function call?


>>> You can set up your @initialize:python@ code to import whatever you want.
>>
>> I would prefer to keep initialisation code for the implementation of the
>> predicate function closely together with the call of the included
>> metavariable.
> 
> You are most welcome to implement in ocaml the lexer and parser for the
> scripting language of your choice.

It seems that I stumble once again on special cases which become relevant
for code composability in bigger usage scenarios.
I am trying to clarify the existing programming interfaces for the semantic
patch language a bit more as usual.

Regards,
Markus

[Cocci] Inclusion of script constraints in SmPL

[Julia Lawall]

On Fri, 18 Nov 2016, SF Markus Elfring wrote:

> >> Does the script constraint syntax allow more programming language code there?
> >
> > For all practical purposes, no.
>
> If you would like to be so strict, I wonder then about the curly brackets
> in the published test example.
> https://github.com/coccinelle/coccinelle/blob/cbc751b30d9e02390d60ebed643c8e4a3fa0bb2b/tests/idcon_python.cocci#L1

The syntax is curly brackets around a C expression.

> > Concretely, the script constraint code is initially parsed using a C parser,
>
> I find such an information interesting to some degree.
>
> Why do you pass data to a software component for ?C? at this place?

Seemed convenient.  Perhaps it could be possible to parse to a closing },
but your goal seems to be to put more complex terms in the script code
that might then include }, making things more complicated.

> > not a parser for the script language.
>
> I do not see that would be needed in the restricted use case.
>
>
> > Function calls, of the form f(a,b,c) are in the intersection of C and the
> > supported scripting languages.
>
> Have you got more control about the construction of a single function call?

No.

julia

[Cocci] Inclusion of script constraints in SmPL

[SF Markus Elfring]

>> https://github.com/coccinelle/coccinelle/blob/cbc751b30d9e02390d60ebed643c8e4a3fa0bb2b/tests/idcon_python.cocci#L1
> 
> The syntax is curly brackets around a C expression.

I do not understand this information completely at the moment.


>>> Concretely, the script constraint code is initially parsed using a C parser,
>>
>> I find such an information interesting to some degree.
>>
>> Why do you pass data to a software component for ?C? at this place?
> 
> Seemed convenient.

I can not follow your views for software convenience with this detail so far.


> Perhaps it could be possible to parse to a closing },

I have got another idea in the meantime.

I propose to distinguish these delimiters better for different use cases.


> but your goal seems to be to put more complex terms in the script code
> that might then include }, making things more complicated.

I hope that it will be possible to pass growing programming scripts there.


>> Have you got more control about the construction of a single function call?
> 
> No.

I find this answer strange.


How do you think about to support cases like named and unnamed (ad hoc)
predicates there?

Examples:


@ad_hoc_replacement@
identifier ID: script:python() { return ID in ['a', 'c'] };
@@
-ID
+36


@initialize:python@
@@
def is_watched(id):
    return id in ['a', 'c']

@structured_replacement@
identifier ID: script:python() ? is_watched;
@@
-ID
+18



Regards,
Markus

[Cocci] Inclusion of script constraints in SmPL

[Julia Lawall]

On Fri, 18 Nov 2016, SF Markus Elfring wrote:

> >> https://github.com/coccinelle/coccinelle/blob/cbc751b30d9e02390d60ebed643c8e4a3fa0bb2b/tests/idcon_python.cocci#L1
> >
> > The syntax is curly brackets around a C expression.
>
> I do not understand this information completely at the moment.
>
>
> >>> Concretely, the script constraint code is initially parsed using a C parser,
> >>
> >> I find such an information interesting to some degree.
> >>
> >> Why do you pass data to a software component for ?C? at this place?
> >
> > Seemed convenient.
>
> I can not follow your views for software convenience with this detail so far.
>
>
> > Perhaps it could be possible to parse to a closing },
>
> I have got another idea in the meantime.
>
> I propose to distinguish these delimiters better for different use cases.
>
>
> > but your goal seems to be to put more complex terms in the script code
> > that might then include }, making things more complicated.
>
> I hope that it will be possible to pass growing programming scripts there.
>
>
> >> Have you got more control about the construction of a single function call?
> >
> > No.
>
> I find this answer strange.
>
>
> How do you think about to support cases like named and unnamed (ad hoc)
> predicates there?
>
> Examples:
>
>
> @ad_hoc_replacement@
> identifier ID: script:python() { return ID in ['a', 'c'] };
> @@
> -ID
> +36
>
>
> @initialize:python@
> @@
> def is_watched(id):
>     return id in ['a', 'c']
>
> @structured_replacement@
> identifier ID: script:python() ? is_watched;
> @@
> -ID
> +18

No interest in supporting either.  How are these so vastly better than:

@initialize:python@
@@
def is_watched(id):
    return id in ['a', 'c']

@structured_replacement@
identifier ID: script:python() { is_watched(ID) };
@@
-ID
+18

julia

[Cocci] Better syntax for SmPL script constraints

[SF Markus Elfring]

> No interest in supporting either.

This is a pity at the moment.

Are there any other software development concerns involved which result
into such a rejection?


> How are these so vastly better than:
> 
> @initialize:python@
> @@
> def is_watched(id):
>     return id in ['a', 'c']
> 
> @structured_replacement@
> identifier ID: script:python()


> { is_watched(ID) };

I find this notation improvable. Can a few characters be omitted
for the data processing of a named predicate to make it a bit more succinct?

I find that curly brackets would only be useful there if a function body
would be directly passed instead of a function name.
Is the suggested usage of ad hoc (unnamed) predicates occasionally
more convenient?

Regards,
Markus