[cocci] Rechecking Python mapping for position variables

[cocci] Rechecking Python mapping for position variables

[Markus Elfring]

Hello,

I tried another script variant out for the semantic patch language
(once more according to the software combination “Coccinelle 1.1.1-00571-g162a9149”).


@initialize:python@
@@
import pprint

@find@
position pos;
@@
 return@pos ...;

@script:python display@
places << find.pos;
@@
pprint.pprint(places)



Corresponding source file example:
int main(void)
{
 return 0;
}



Test result:
Markus_Elfring@Sonne:…/Projekte/Coccinelle/Probe> spatch display_position_information_for_one_return.cocci only_return_from_main.c
…
(<coccilib.elems.Location instance at 0x7f8fc4fcefa0>,)



Why was a tuple constructed for a single source code place?
https://docs.python.org/3/reference/datamodel.html#immutable-sequences

Can a “Location” object be provided directly according to a metavariable type
like “position”?
https://gitlab.inria.fr/coccinelle/coccinelle/-/blob/8d66511d00b27de44e5c21b60d41938d32321b1a/python/coccilib/elems.py

Regards,
Markus

Re: [cocci] Rechecking Python mapping for position variables

[Julia Lawall]

[-- Attachment #1: Type: text/plain, Size: 1532 bytes --]



On Fri, 24 Nov 2023, Markus Elfring wrote:

> Hello,
>
> I tried another script variant out for the semantic patch language
> (once more according to the software combination “Coccinelle 1.1.1-00571-g162a9149”).
>
>
> @initialize:python@
> @@
> import pprint
>
> @find@
> position pos;
> @@
>  return@pos ...;
>
> @script:python display@
> places << find.pos;
> @@
> pprint.pprint(places)
>
>
>
> Corresponding source file example:
> int main(void)
> {
>  return 0;
> }
>
>
>
> Test result:
> Markus_Elfring@Sonne:…/Projekte/Coccinelle/Probe> spatch display_position_information_for_one_return.cocci only_return_from_main.c
> …
> (<coccilib.elems.Location instance at 0x7f8fc4fcefa0>,)

The type of a position variable in python is an array.  In OCaml it is a
list.  This is completely independent of the number of element involved.

> Why was a tuple constructed for a single source code place?
> https://docs.python.org/3/reference/datamodel.html#immutable-sequences
>
> Can a “Location” object be provided directly according to a metavariable type
> like “position”?
> https://gitlab.inria.fr/coccinelle/coccinelle/-/blob/8d66511d00b27de44e5c21b60d41938d32321b1a/python/coccilib/elems.py

I don't understand the question.  I guess that a position variable is
mapped to an array of locations.  So if you want a location, you would
take the first element of the array.  But again, I don't know python, so
perhaps something else is going on.  I guess you do know python, so you
could figure it out yourself.

julia

Re: [cocci] Rechecking Python mapping for position variables

[Markus Elfring]

>> Test result:
>> Markus_Elfring@Sonne:…/Projekte/Coccinelle/Probe> spatch display_position_information_for_one_return.cocci only_return_from_main.c
>> …
>> (<coccilib.elems.Location instance at 0x7f8fc4fcefa0>,)
>
> The type of a position variable in python is an array.

It seems that the output from a call of the function “pprint” did not indicate this.
https://docs.python.org/3/library/pprint.html#pprint.pprint


> In OCaml it is a list.

Will any more clarifications become helpful for the involved data structures?


> This is completely independent of the number of element involved.

Will further case distinctions be taken into account?


>> Why was a tuple constructed for a single source code place?
>> https://docs.python.org/3/reference/datamodel.html#immutable-sequences
>>
>> Can a “Location” object be provided directly according to a metavariable type
>> like “position”?
>> https://gitlab.inria.fr/coccinelle/coccinelle/-/blob/8d66511d00b27de44e5c21b60d41938d32321b1a/python/coccilib/elems.py
>
> I don't understand the question.  I guess that a position variable is
> mapped to an array of locations.

This view can be reasonable.


> So if you want a location, you would take the first element of the array.

How do you think about the possibility to avoid such data processing?

May it be expected that only a single source code place will be passed to
the shown script rule?

Regards,
Markus

Re: [cocci] Rechecking Python mapping for position variables

[Julia Lawall]

[-- Attachment #1: Type: text/plain, Size: 1889 bytes --]



On Fri, 24 Nov 2023, Markus Elfring wrote:

> >> Test result:
> >> Markus_Elfring@Sonne:…/Projekte/Coccinelle/Probe> spatch display_position_information_for_one_return.cocci only_return_from_main.c
> >> …
> >> (<coccilib.elems.Location instance at 0x7f8fc4fcefa0>,)
> >
> > The type of a position variable in python is an array.
>
> It seems that the output from a call of the function “pprint” did not indicate this.
> https://docs.python.org/3/library/pprint.html#pprint.pprint

I have no idea.  You access the thing as p[0].line, p[1].line, etc.  I
don't know what type is involved.

>
> > In OCaml it is a list.
>
> Will any more clarifications become helpful for the involved data structures?
>
>
> > This is completely independent of the number of element involved.
>
> Will further case distinctions be taken into account?
>
>
> >> Why was a tuple constructed for a single source code place?
> >> https://docs.python.org/3/reference/datamodel.html#immutable-sequences
> >>
> >> Can a “Location” object be provided directly according to a metavariable type
> >> like “position”?
> >> https://gitlab.inria.fr/coccinelle/coccinelle/-/blob/8d66511d00b27de44e5c21b60d41938d32321b1a/python/coccilib/elems.py
> >
> > I don't understand the question.  I guess that a position variable is
> > mapped to an array of locations.
>
> This view can be reasonable.
>
>
> > So if you want a location, you would take the first element of the array.
>
> How do you think about the possibility to avoid such data processing?
>
> May it be expected that only a single source code place will be passed to
> the shown script rule?

No.  When you use ... by default it considers all paths.  Therefore in

a();
...
b@p();

multiple positions can be matched.  It doesn't make any sense to have a
special case for the case where there happens to be only one.

julia

>
> Regards,
> Markus
>

Re: [cocci] Rechecking Python mapping for position variables

[Markus Elfring]

>>>> Test result:
>>>> Markus_Elfring@Sonne:…/Projekte/Coccinelle/Probe> spatch display_position_information_for_one_return.cocci only_return_from_main.c
>>>> …
>>>> (<coccilib.elems.Location instance at 0x7f8fc4fcefa0>,)
>>>
>>> The type of a position variable in python is an array.
>>
>> It seems that the output from a call of the function “pprint” did not indicate this.
>> https://docs.python.org/3/library/pprint.html#pprint.pprint
>
> I have no idea.

Such feedback should be reconsidered.


> You access the thing as p[0].line, p[1].line, etc.

The specification of “array” indices would be needed if several items
would actually be available for a script rule.


> I don't know what type is involved.

I guess that the knowledge can be extended accordingly.


>>> So if you want a location, you would take the first element of the array.
>>
>> How do you think about the possibility to avoid such data processing?
>>
>> May it be expected that only a single source code place will be passed to
>> the shown script rule?
>
> No.  When you use ... by default it considers all paths.  Therefore in
>
> a();
> ...
> b@p();
>
> multiple positions can be matched.

This search functionality is generally fine.


> It doesn't make any sense to have a special case for the case
> where there happens to be only one.

I got an other development view.

* Can it actually happen that more position items are passed to a script rule
  by a local variable?

* Would it be possible to restrict the transferred data amount by a related
  metavariable type?


Regards,
Markus

Re: [cocci] Rechecking Python mapping for position variables

[Julia Lawall]

> * Can it actually happen that more position items are passed to a script rule
>   by a local variable?

Sure.  Try

@@
expression E;
@@

a();
...
b@p(E);

and then

int main() {
  a();
  if (x)
    b(1);
  else
    b(2);
  return 0;
}

>
> * Would it be possible to restrict the transferred data amount by a related
>   metavariable type?

I see no point in providing new functionality to do this.  I guess you
could do:

position p : script:python() { p.length == 1 };

Or something like that.

julia

Re: [cocci] Rechecking Python mapping for position variables

[Markus Elfring]

>> * Can it actually happen that more position items are passed to a script rule
>>   by a local variable?
>
> Sure.  Try

@initialize:python@
@@
import pprint

@find@
expression E;
position P;
@@
a();
...
b@P(E);

@script:python display@
places << find.P;
@@
pprint.pprint(places)



> int main() {
>   a();
>   if (x)
>     b(1);
>   else
>     b(2);
>   return 0;
> }

Test result:
Markus_Elfring@Sonne:…/Projekte/Coccinelle/Probe> spatch test-example-20231124.cocci example-Julia_Lawall-20231124.c
…
(<coccilib.elems.Location instance at 0x7ffbc3d2b910>,
 <coccilib.elems.Location instance at 0x7ffbc3d2e500>)


>> * Would it be possible to restrict the transferred data amount by a related
>>   metavariable type?
>
> I see no point in providing new functionality to do this.

Can any other SmPL script developers convince you better (than me)
about corresponding possibilities for data processing adjustments?

I would occasionally appreciate if fields like “line_end” and “column_end”
can be omitted from the data exchange.
https://gitlab.inria.fr/coccinelle/coccinelle/-/blob/8d66511d00b27de44e5c21b60d41938d32321b1a/python/python_documentation.md#coccimake_positionfl-fn-startl-startc-endl-endc

Regards,
Markus

Re: [cocci] Rechecking Python mapping for position variables

[Markus Elfring]

> Test result:
> Markus_Elfring@Sonne:…/Projekte/Coccinelle/Probe> spatch test-example-20231124.cocci example-Julia_Lawall-20231124.c
> …
> (<coccilib.elems.Location instance at 0x7ffbc3d2b910>,
>  <coccilib.elems.Location instance at 0x7ffbc3d2e500>)

This test case shows that the script rule “display” was executed only once.
The found source code places were accordingly passed in a single data structure.


Please compare the software behaviour once more also according to following
test files.


@initialize:python@
@@
import pprint

@find@
identifier target;
position pos;
@@
 goto target@pos;

@script:python display@
places << find.pos;
@@
pprint.pprint(places)



void my_test(void)
{
 if (1)
    goto x;

 if (2)
    goto y;

 if (3)
    goto z;

 x:
 y:
 z:
 test = 0;
}



Test result:
Markus_Elfring@Sonne:…/Projekte/Coccinelle/Probe> spatch ../janitor/use_common_code17.cocci three_questionable_gotos.c
…
(<coccilib.elems.Location instance at 0x7f8bd82e6050>,)
(<coccilib.elems.Location instance at 0x7f8bd82e6730>,)
(<coccilib.elems.Location instance at 0x7f8bd82e8640>,)


Would you ever similarly like to receive position information in one sequence
instead of separate record sets?

Can a more consistent data grouping be achieved for such use cases?

Regards,
Markus

Re: [cocci] Rechecking Python mapping for position variables

[Julia Lawall]

[-- Attachment #1: Type: text/plain, Size: 1773 bytes --]



On Fri, 24 Nov 2023, Markus Elfring wrote:

> > Test result:
> > Markus_Elfring@Sonne:…/Projekte/Coccinelle/Probe> spatch test-example-20231124.cocci example-Julia_Lawall-20231124.c
> > …
> > (<coccilib.elems.Location instance at 0x7ffbc3d2b910>,
> >  <coccilib.elems.Location instance at 0x7ffbc3d2e500>)
>
> This test case shows that the script rule “display” was executed only once.
> The found source code places were accordingly passed in a single data structure.

This is normal. There is one match.  So all of the positions that are part
of that match are collected together.

>
>
> Please compare the software behaviour once more also according to following
> test files.
>
>
> @initialize:python@
> @@
> import pprint
>
> @find@
> identifier target;
> position pos;
> @@
>  goto target@pos;
>
> @script:python display@
> places << find.pos;
> @@
> pprint.pprint(places)
>
>
>
> void my_test(void)
> {
>  if (1)
>     goto x;
>
>  if (2)
>     goto y;
>
>  if (3)
>     goto z;
>
>  x:
>  y:
>  z:
>  test = 0;
> }
>
>
>
> Test result:
> Markus_Elfring@Sonne:…/Projekte/Coccinelle/Probe> spatch ../janitor/use_common_code17.cocci three_questionable_gotos.c
> …
> (<coccilib.elems.Location instance at 0x7f8bd82e6050>,)
> (<coccilib.elems.Location instance at 0x7f8bd82e6730>,)
> (<coccilib.elems.Location instance at 0x7f8bd82e8640>,)
>
>
> Would you ever similarly like to receive position information in one sequence
> instead of separate record sets?

No.  One can import many kinds of metavariables into a python script.
Each execution of a python script is with respect to a unique snapshot of
metavariable values resulting from a single match.

julia

> Can a more consistent data grouping be achieved for such use cases?
>
> Regards,
> Markus
>

Re: [cocci] Rechecking Python mapping for position variables

[Markus Elfring]

>> Test result:
>> Markus_Elfring@Sonne:…/Projekte/Coccinelle/Probe> spatch ../janitor/use_common_code17.cocci three_questionable_gotos.c
>> …
>> (<coccilib.elems.Location instance at 0x7f8bd82e6050>,)
>> (<coccilib.elems.Location instance at 0x7f8bd82e6730>,)
>> (<coccilib.elems.Location instance at 0x7f8bd82e8640>,)
>>
>>
>> Would you ever similarly like to receive position information in one sequence
>> instead of separate record sets?
>
> No.  One can import many kinds of metavariables into a python script.
> Each execution of a python script is with respect to a unique snapshot of
> metavariable values resulting from a single match.

I hope that case distinctions can be reconsidered further also for
the handling of “multiple matches”.


>> Can a more consistent data grouping be achieved for such use cases?

I assume that circumstances can evolve in ways which might make corresponding
use cases more attractive and desirable.

Regards,
Markus

Re: [cocci] Improving support for processing with data collections

[Markus Elfring]

>>> Test result:
>>> Markus_Elfring@Sonne:…/Projekte/Coccinelle/Probe> spatch test-example-20231124.cocci example-Julia_Lawall-20231124.c
>>> …
>>> (<coccilib.elems.Location instance at 0x7ffbc3d2b910>,
>>>  <coccilib.elems.Location instance at 0x7ffbc3d2e500>)
>>
>> This test case shows that the script rule “display” was executed only once.
>> The found source code places were accordingly passed in a single data structure.
>
> This is normal. There is one match.  So all of the positions that are part
> of that match are collected together.

The Coccinelle software is capable to put some data into a sequence.


>> Test result:
>> Markus_Elfring@Sonne:…/Projekte/Coccinelle/Probe> spatch ../janitor/use_common_code17.cocci three_questionable_gotos.c
>> …
>> (<coccilib.elems.Location instance at 0x7f8bd82e6050>,)
>> (<coccilib.elems.Location instance at 0x7f8bd82e6730>,)
>> (<coccilib.elems.Location instance at 0x7f8bd82e8640>,)
>>
>>
>> Would you ever similarly like to receive position information in one sequence
>> instead of separate record sets?
>
> No.  One can import many kinds of metavariables into a python script.
> Each execution of a python script is with respect to a unique snapshot of
> metavariable values resulting from a single match.

I suggest to take another look at corresponding algorithmic properties.
Some data processing can occasionally be performed immediately.
But some actions depend on decisions which can only be performed
after sufficient data were collected.
One well-known use case is the determination if source code elements
were repeated (or not).

The current handling of multiple matches has got the consequence
that provided data need to be stored in further data structures.
These collected data can be used then in another SmPL rule of
the kind “finalize”. Such a finalisation rule is executed just before
the program “spatch” will be finished.
https://gitlab.inria.fr/coccinelle/coccinelle/-/blob/f0cc79b51b1f836b34ea3ba584589851e65f073c/docs/manual/cocci_syntax.tex#L688

If further data processing would be attempted, data need to prepared in ways
for crossing the involved process boundaries so that they can be reused
again later.
I imagine that we would occasionally like to benefit more from the desire
to perform selected data processing within one program run as much as possible.
Thus I propose to add a key word to the script rule specifications
so that the rule execution would be intended for data collection.
Decisions can be taken then in the same way as in the single match case.
Metavariables can accordingly be constructed for their application in
subsequent SmPL rules.

Do you find a word like “union” or “merge” helpful for a software extension?

Regards,
Markus

Re: [cocci] Rechecking Python mapping for position variables

[Markus Elfring]

>> Test result:
>> Markus_Elfring@Sonne:…/Projekte/Coccinelle/Probe> spatch display_position_information_for_one_return.cocci only_return_from_main.c
>> …
>> (<coccilib.elems.Location instance at 0x7f8fc4fcefa0>,)
>
> The type of a position variable in python is an array.  In OCaml it is
> a list.  This is completely independent of the number of element involved.

The terms differ for the applied data structures with the involved
programming languages.

Another SmPL script example:
@initialize:python@
@@
import inspect, pprint, sys

def display_members(item):
   sys.stdout.write("\n===\n\n")

   for member in inspect.getmembers(item):
      pprint.pprint(member)

@find@
position pos;
@@
 return@pos ...;

@script:python display@
places << find.pos;
@@
pprint.pprint(places)

for place in places:
   display_members(place)



Test result:
Markus_Elfring@Sonne:…/Projekte/Coccinelle/Probe> spatch display_position_information_for_one_return2.cocci only_return_from_main.c
…
(<coccilib.elems.Location instance at 0x7f6c9a2fefa0>,)

===

('__doc__', None)
('__init__',
 <bound method Location.__init__ of <coccilib.elems.Location instance at 0x7f6c9a2fefa0>>)
('__module__', 'coccilib.elems')
('column', '1')
('column_end', '7')
('current_element', 'main')
('current_element_column', '0')
('current_element_column_end', '1')
('current_element_line', '1')
('current_element_line_end', '4')
('file', 'only_return_from_main.c')
('line', '3')
('line_end', '3')



Would you like to reconsider the representation of shown number attributes
as text strings (instead of integers)?

How do you think about to enable the usage of an integral data type here?

Regards,
Markus

Re: [cocci] Rechecking Python mapping for position variables

[Julia Lawall]

[-- Attachment #1: Type: text/plain, Size: 1900 bytes --]



On Sun, 26 Nov 2023, Markus Elfring wrote:

> >> Test result:
> >> Markus_Elfring@Sonne:…/Projekte/Coccinelle/Probe> spatch display_position_information_for_one_return.cocci only_return_from_main.c
> >> …
> >> (<coccilib.elems.Location instance at 0x7f8fc4fcefa0>,)
> >
> > The type of a position variable in python is an array.  In OCaml it is
> > a list.  This is completely independent of the number of element involved.
>
> The terms differ for the applied data structures with the involved
> programming languages.
>
> Another SmPL script example:
> @initialize:python@
> @@
> import inspect, pprint, sys
>
> def display_members(item):
>    sys.stdout.write("\n===\n\n")
>
>    for member in inspect.getmembers(item):
>       pprint.pprint(member)
>
> @find@
> position pos;
> @@
>  return@pos ...;
>
> @script:python display@
> places << find.pos;
> @@
> pprint.pprint(places)
>
> for place in places:
>    display_members(place)
>
>
>
> Test result:
> Markus_Elfring@Sonne:…/Projekte/Coccinelle/Probe> spatch display_position_information_for_one_return2.cocci only_return_from_main.c
> …
> (<coccilib.elems.Location instance at 0x7f6c9a2fefa0>,)
>
> ===
>
> ('__doc__', None)
> ('__init__',
>  <bound method Location.__init__ of <coccilib.elems.Location instance at 0x7f6c9a2fefa0>>)
> ('__module__', 'coccilib.elems')
> ('column', '1')
> ('column_end', '7')
> ('current_element', 'main')
> ('current_element_column', '0')
> ('current_element_column_end', '1')
> ('current_element_line', '1')
> ('current_element_line_end', '4')
> ('file', 'only_return_from_main.c')
> ('line', '3')
> ('line_end', '3')
>
>
>
> Would you like to reconsider the representation of shown number attributes
> as text strings (instead of integers)?
>
> How do you think about to enable the usage of an integral data type here?

I odn't want to change the interface.  If you want numbers, use OCaml.

julia

Re: [cocci] Rechecking Python mapping for position variables

[Markus Elfring]

>> Would you like to reconsider the representation of shown number attributes
>> as text strings (instead of integers)?
>>
>> How do you think about to enable the usage of an integral data type here?
>
> I odn't want to change the interface.

Can any other developers convince you better to consider further
software adjustment possibilities?


> If you want numbers, use OCaml.

The properties of the supported programming languages influence
their attractivity for more desirable developments.

Regards,
Markus

Re: [cocci] Rechecking Python mapping for position variables

[Markus Elfring]

>> Would you like to reconsider the representation of shown number attributes
>> as text strings (instead of integers)?
>>
>> How do you think about to enable the usage of an integral data type here?
>
> I odn't want to change the interface.

I hope that circumstances can evolve into more desirable directions.
https://gitlab.inria.fr/coccinelle/coccinelle/-/blob/07e1ee1eca39bf52828826d1ac773facd47491b6/docs/manual/cocci_syntax.tex#L684


> If you want numbers, use OCaml.

At which place are available data transferred to the evaluation environment
of the other supported scripting language?

Can it be determined anyhow where questionable data type selections (like “string”)
are performed so far?

Regards,
Markus