[cocci] Does SmPL support regex groups?

[cocci] Does SmPL support regex groups?

[Eric Wheeler]

Hi all,

I would like to replace this:

	void on_struct_save_as_gnuplot_activate(
	    GtkMenuItem     *menuitem,
	    gpointer         user_data)
	{
	  file_chooser = Open_Filechooser( GTK_FILE_CHOOSER_ACTION_SAVE,
	      "*.gplot", NULL, "untitled.gplot", rc_config.working_dir );
	}

And extract the extension ".gplot" as an identifier and pass it to the 
function get_nec_filename_stem, like so:

	void on_struct_save_as_gnuplot_activate(
	    GtkMenuItem     *menuitem,
	    gpointer         user_data)
	{
	  file_chooser = Open_Filechooser( GTK_FILE_CHOOSER_ACTION_SAVE,
	      "*.gplot", NULL, get_nec_filename_stem(newfn, ".gplot", PATH_MAX), rc_config.working_dir );
	}

The SmPL patch below works for what I want statically, but notice the 
commented `expression E`:

Is there a way to regex-group `expression E =~` with a parenthesis and then specify
something like E[0] to pick up the grouped regex match?

	@ replace_untitled @
	identifier F;
	parameter list PL;
	expression list EL1, EL2;
	//expression E =~ "\"untitled\.(.*)\"";
	@@

	F(PL)
	{
	+   char newfn[PATH_MAX];
	    ...
	    file_chooser = Open_Filechooser(EL1,
	-       "untitled.gplot",
	+       get_nec_filename_stem(newfn, ".gplot", PATH_MAX), 
		EL2);
	    ...
	}


This doesn't work, but it explains what I'm trying to do:

	@ replace_untitled @
	identifier F;
	parameter list PL;
	expression list EL1, EL2;
	expression E =~ "untitled.(.*)";
	@@

	F(PL)
	{   
	+   char newfn[PATH_MAX];
	    ...
	    file_chooser = Open_Filechooser(EL1,
	-       E,
	+       get_nec_filename_stem(newfn, E[0], PATH_MAX),
		EL2);
	    ...
	}

It replaces as follows and does match "untitled.(.*)", but of course [0] is incorrect SmPL grammar:

	@@ -250,10 +253,13 @@ on_struct_save_as_gnuplot_activate(
	     GtkMenuItem     *menuitem,
	     gpointer         user_data)
	 {
	+  char newfn[PATH_MAX];
	   /* Open file chooser to save structure image */
	   SetFlag( STRUCT_GNUPLOT_SAVE );
	   file_chooser = Open_Filechooser( GTK_FILE_CHOOSER_ACTION_SAVE,
	-      "*.gplot", NULL, "untitled.gplot", rc_config.working_dir );
	+      "*.gplot", NULL,
	+      get_nec_filename_stem(newfn, "untitled.gplot"[0], PATH_MAX),
	+      rc_config.working_dir );
	 }

I tried something like this:
	expression F =~ "\..*";
	expression E =~ "untitled" ##F;
but that is invalid too.

Ideas?


--
Eric Wheeler

Re: [cocci] Does SmPL support regex groups?

[Julia Lawall]

Regex groups are not supported.  You can use python to do whatever you
want with the matched terms.  See demos/pythontococci.cocci

julia

On Tue, 8 Mar 2022, Eric Wheeler wrote:

> Hi all,
>
> I would like to replace this:
>
> 	void on_struct_save_as_gnuplot_activate(
> 	    GtkMenuItem     *menuitem,
> 	    gpointer         user_data)
> 	{
> 	  file_chooser = Open_Filechooser( GTK_FILE_CHOOSER_ACTION_SAVE,
> 	      "*.gplot", NULL, "untitled.gplot", rc_config.working_dir );
> 	}
>
> And extract the extension ".gplot" as an identifier and pass it to the
> function get_nec_filename_stem, like so:
>
> 	void on_struct_save_as_gnuplot_activate(
> 	    GtkMenuItem     *menuitem,
> 	    gpointer         user_data)
> 	{
> 	  file_chooser = Open_Filechooser( GTK_FILE_CHOOSER_ACTION_SAVE,
> 	      "*.gplot", NULL, get_nec_filename_stem(newfn, ".gplot", PATH_MAX), rc_config.working_dir );
> 	}
>
> The SmPL patch below works for what I want statically, but notice the
> commented `expression E`:
>
> Is there a way to regex-group `expression E =~` with a parenthesis and then specify
> something like E[0] to pick up the grouped regex match?
>
> 	@ replace_untitled @
> 	identifier F;
> 	parameter list PL;
> 	expression list EL1, EL2;
> 	//expression E =~ "\"untitled\.(.*)\"";
> 	@@
>
> 	F(PL)
> 	{
> 	+   char newfn[PATH_MAX];
> 	    ...
> 	    file_chooser = Open_Filechooser(EL1,
> 	-       "untitled.gplot",
> 	+       get_nec_filename_stem(newfn, ".gplot", PATH_MAX),
> 		EL2);
> 	    ...
> 	}
>
>
> This doesn't work, but it explains what I'm trying to do:
>
> 	@ replace_untitled @
> 	identifier F;
> 	parameter list PL;
> 	expression list EL1, EL2;
> 	expression E =~ "untitled.(.*)";
> 	@@
>
> 	F(PL)
> 	{
> 	+   char newfn[PATH_MAX];
> 	    ...
> 	    file_chooser = Open_Filechooser(EL1,
> 	-       E,
> 	+       get_nec_filename_stem(newfn, E[0], PATH_MAX),
> 		EL2);
> 	    ...
> 	}
>
> It replaces as follows and does match "untitled.(.*)", but of course [0] is incorrect SmPL grammar:
>
> 	@@ -250,10 +253,13 @@ on_struct_save_as_gnuplot_activate(
> 	     GtkMenuItem     *menuitem,
> 	     gpointer         user_data)
> 	 {
> 	+  char newfn[PATH_MAX];
> 	   /* Open file chooser to save structure image */
> 	   SetFlag( STRUCT_GNUPLOT_SAVE );
> 	   file_chooser = Open_Filechooser( GTK_FILE_CHOOSER_ACTION_SAVE,
> 	-      "*.gplot", NULL, "untitled.gplot", rc_config.working_dir );
> 	+      "*.gplot", NULL,
> 	+      get_nec_filename_stem(newfn, "untitled.gplot"[0], PATH_MAX),
> 	+      rc_config.working_dir );
> 	 }
>
> I tried something like this:
> 	expression F =~ "\..*";
> 	expression E =~ "untitled" ##F;
> but that is invalid too.
>
> Ideas?
>
>
> --
> Eric Wheeler
>

Re: [cocci] Does SmPL support regex groups?

[Markus Elfring]

> And extract the extension ".gplot" as an identifier


It seems that you are looking for a string literal
(which would contain a file name).



> and pass it to the function get_nec_filename_stem, like so:


Literals can be found with the means of the semantic patch language
by the metavariable type “constant”.

https://gitlab.inria.fr/coccinelle/coccinelle/-/blob/5479fc964d209209e84ddd5caf3f395667721e7c/docs/manual/cocci_syntax.tex#L216
https://github.com/coccinelle/coccinelle/blob/ae337fce1512ff15aabc3ad5b6d2e537f97ab62a/docs/manual/cocci_syntax.tex#L216


> The SmPL patch below works for what I want statically, but notice the
> commented `expression E`:
>
> Is there a way to regex-group `expression E =~` with a parenthesis and then specify
> something like E[0] to pick up the grouped regex match?


Would you like to achieve any further software evolution also according to
my feature request “Reuse of content from capturing groups in regular expressions”
from 2015-08-27?
https://github.com/coccinelle/coccinelle/issues/46



> 	@ replace_untitled @
> 	identifier F;
> 	parameter list PL;
> 	expression list EL1, EL2;
> 	//expression E =~ "\"untitled\.(.*)\"";
> 	@@
>
> 	F(PL)
> 	{
> 	+   char newfn[PATH_MAX];
> 	    ...
> 	    file_chooser = Open_Filechooser(EL1,
> 	-       "untitled.gplot",
> 	+       get_nec_filename_stem(newfn, ".gplot", PATH_MAX),
> 		EL2);
> 	    ...
> 	}


I would like to point the possibility out that SmPL ellipses
can be used instead of the metavariables “PL”, “EL1” and “EL2”.



> Ideas?


Would you get any further ideas from the clarification approach for the topic
“Development challenges around using the function “make_expr” for expression
construction by SmPL” (from 2016-10-27)?
https://github.com/coccinelle/coccinelle/issues/88

Excerpt from my script:

@initialize:python@
@@
import re

@find_update_candidate@
constant input_string;
expression input_context;
position pos;
@@
 seq_printf@pos(input_context, input_string);

@script:python selection@
call;
param << find_update_candidate.input_string;
pos << find_update_candidate.pos;
text;
@@
match = re.match('"([^"]+)"', param)
if match:
…


Regards,
Markus

Re: [cocci] Does SmPL support regex groups?

[Eric Wheeler]

[-- Attachment #1: Type: text/plain, Size: 2523 bytes --]

On Wed, 9 Mar 2022, Markus Elfring wrote:
> > And extract the extension ".gplot" as an identifier
> 
> 
> It seems that you are looking for a string literal
> (which would contain a file name).
> 
> 
> 
> > and pass it to the function get_nec_filename_stem, like so:
> 
> Literals can be found with the means of the semantic patch language
> by the metavariable type “constant”.
> 
> https://gitlab.inria.fr/coccinelle/coccinelle/-/blob/5479fc964d209209e84ddd5caf3f395667721e7c/docs/manual/cocci_syntax.tex#L216
> https://github.com/coccinelle/coccinelle/blob/ae337fce1512ff15aabc3ad5b6d2e537f97ab62a/docs/manual/cocci_syntax.tex#L216

Can literals match substrings like ".gplot" for cut-and-splice?

> > The SmPL patch below works for what I want statically, but notice the
> > commented `expression E`:
> >
> > Is there a way to regex-group `expression E =~` with a parenthesis and then specify
> > something like E[0] to pick up the grouped regex match?
> 
> 
> Would you like to achieve any further software evolution also according to
> my feature request “Reuse of content from capturing groups in regular expressions”
> from 2015-08-27?
> https://github.com/coccinelle/coccinelle/issues/46

Github says 404 not found.

-Eric

> 
> 
> 
> > 	@ replace_untitled @
> > 	identifier F;
> > 	parameter list PL;
> > 	expression list EL1, EL2;
> > 	//expression E =~ "\"untitled\.(.*)\"";
> > 	@@
> >
> > 	F(PL)
> > 	{
> > 	+   char newfn[PATH_MAX];
> > 	    ...
> > 	    file_chooser = Open_Filechooser(EL1,
> > 	-       "untitled.gplot",
> > 	+       get_nec_filename_stem(newfn, ".gplot", PATH_MAX),
> > 		EL2);
> > 	    ...
> > 	}
> 
> 
> I would like to point the possibility out that SmPL ellipses
> can be used instead of the metavariables “PL”, “EL1” and “EL2”.
> 
> 
> 
> > Ideas?
> 
> 
> Would you get any further ideas from the clarification approach for the topic
> “Development challenges around using the function “make_expr” for expression
> construction by SmPL” (from 2016-10-27)?
> https://github.com/coccinelle/coccinelle/issues/88
> 
> Excerpt from my script:
> 
> @initialize:python@
> @@
> import re
> 
> @find_update_candidate@
> constant input_string;
> expression input_context;
> position pos;
> @@
>  seq_printf@pos(input_context, input_string);
> 
> @script:python selection@
> call;
> param << find_update_candidate.input_string;
> pos << find_update_candidate.pos;
> text;
> @@
> match = re.match('"([^"]+)"', param)
> if match:
> …
> 
> 
> Regards,
> Markus
> 
> 

Re: [cocci] Does SmPL support regex groups?

[Eric Wheeler]

> Regex groups are not supported.  You can use python to do whatever you
> want with the matched terms.  See demos/pythontococci.cocci

Thanks Julia.  It seems the Coccinelle =~ matches do not support escaping.  
I was getting errors trying to match like so:

expression E =~ "\.gplot";

It works without the \ but errors with the . 
	File "cocci/untitled-fix.cocci", line 5, column 35, charpos = 144
	  around = '\.',
	  whole content = expression EXT, arg, E =~ "untitled\.(.*)";

It failed scaping \" as well but now I'm unable to reproduce the problem.  
Sounds like python is the way to go, built-in regex's are useful and might 
need a bit of cleanup.

+1 for group and escaping support if it might be a trivial patch.


--
Eric Wheeler

On Wed, 9 Mar 2022, Julia Lawall wrote:


> 
> julia
> 
> On Tue, 8 Mar 2022, Eric Wheeler wrote:
> 
> > Hi all,
> >
> > I would like to replace this:
> >
> > 	void on_struct_save_as_gnuplot_activate(
> > 	    GtkMenuItem     *menuitem,
> > 	    gpointer         user_data)
> > 	{
> > 	  file_chooser = Open_Filechooser( GTK_FILE_CHOOSER_ACTION_SAVE,
> > 	      "*.gplot", NULL, "untitled.gplot", rc_config.working_dir );
> > 	}
> >
> > And extract the extension ".gplot" as an identifier and pass it to the
> > function get_nec_filename_stem, like so:
> >
> > 	void on_struct_save_as_gnuplot_activate(
> > 	    GtkMenuItem     *menuitem,
> > 	    gpointer         user_data)
> > 	{
> > 	  file_chooser = Open_Filechooser( GTK_FILE_CHOOSER_ACTION_SAVE,
> > 	      "*.gplot", NULL, get_nec_filename_stem(newfn, ".gplot", PATH_MAX), rc_config.working_dir );
> > 	}
> >
> > The SmPL patch below works for what I want statically, but notice the
> > commented `expression E`:
> >
> > Is there a way to regex-group `expression E =~` with a parenthesis and then specify
> > something like E[0] to pick up the grouped regex match?
> >
> > 	@ replace_untitled @
> > 	identifier F;
> > 	parameter list PL;
> > 	expression list EL1, EL2;
> > 	//expression E =~ "\"untitled\.(.*)\"";
> > 	@@
> >
> > 	F(PL)
> > 	{
> > 	+   char newfn[PATH_MAX];
> > 	    ...
> > 	    file_chooser = Open_Filechooser(EL1,
> > 	-       "untitled.gplot",
> > 	+       get_nec_filename_stem(newfn, ".gplot", PATH_MAX),
> > 		EL2);
> > 	    ...
> > 	}
> >
> >
> > This doesn't work, but it explains what I'm trying to do:
> >
> > 	@ replace_untitled @
> > 	identifier F;
> > 	parameter list PL;
> > 	expression list EL1, EL2;
> > 	expression E =~ "untitled.(.*)";
> > 	@@
> >
> > 	F(PL)
> > 	{
> > 	+   char newfn[PATH_MAX];
> > 	    ...
> > 	    file_chooser = Open_Filechooser(EL1,
> > 	-       E,
> > 	+       get_nec_filename_stem(newfn, E[0], PATH_MAX),
> > 		EL2);
> > 	    ...
> > 	}
> >
> > It replaces as follows and does match "untitled.(.*)", but of course [0] is incorrect SmPL grammar:
> >
> > 	@@ -250,10 +253,13 @@ on_struct_save_as_gnuplot_activate(
> > 	     GtkMenuItem     *menuitem,
> > 	     gpointer         user_data)
> > 	 {
> > 	+  char newfn[PATH_MAX];
> > 	   /* Open file chooser to save structure image */
> > 	   SetFlag( STRUCT_GNUPLOT_SAVE );
> > 	   file_chooser = Open_Filechooser( GTK_FILE_CHOOSER_ACTION_SAVE,
> > 	-      "*.gplot", NULL, "untitled.gplot", rc_config.working_dir );
> > 	+      "*.gplot", NULL,
> > 	+      get_nec_filename_stem(newfn, "untitled.gplot"[0], PATH_MAX),
> > 	+      rc_config.working_dir );
> > 	 }
> >
> > I tried something like this:
> > 	expression F =~ "\..*";
> > 	expression E =~ "untitled" ##F;
> > but that is invalid too.
> >
> > Ideas?
> >
> >
> > --
> > Eric Wheeler
> >
> 

Re: [cocci] Does SmPL support regex groups?

[Markus Elfring]

>> Literals can be found with the means of the semantic patch language
>> by the metavariable type “constant”.
>>
>> https://gitlab.inria.fr/coccinelle/coccinelle/-/blob/5479fc964d209209e84ddd5caf3f395667721e7c/docs/manual/cocci_syntax.tex#L216
>> https://github.com/coccinelle/coccinelle/blob/ae337fce1512ff15aabc3ad5b6d2e537f97ab62a/docs/manual/cocci_syntax.tex#L216
> Can literals match substrings like ".gplot" for cut-and-splice?


The Coccinelle software contains some known limitations.

Some limits can be adjusted if you can manage to activate extra possibilities
by switching parts of the corresponding data processing to one of the supported
scripting languages (like Python and OCaml).
The application of (advanced) regular expressions provides various chances,
doesn't it?

Regards,
Markus

Re: [cocci] Does SmPL support regex groups?

[Julia Lawall]

On Wed, 9 Mar 2022, Eric Wheeler wrote:

> > Regex groups are not supported.  You can use python to do whatever you
> > want with the matched terms.  See demos/pythontococci.cocci
>
> Thanks Julia.  It seems the Coccinelle =~ matches do not support escaping.
> I was getting errors trying to match like so:
>
> expression E =~ "\.gplot";
>
> It works without the \ but errors with the .
> 	File "cocci/untitled-fix.cocci", line 5, column 35, charpos = 144
> 	  around = '\.',
> 	  whole content = expression EXT, arg, E =~ "untitled\.(.*)";
>
> It failed scaping \" as well but now I'm unable to reproduce the problem.
> Sounds like python is the way to go, built-in regex's are useful and might
> need a bit of cleanup.

I would imagine that you need to put "\\.gplot"

I would imagine that escaping is trivial to addres.  I have no idea what
one would even want from grouping.  That is, what notation do you expect
to use to refer to the groups.  And which group, if there are multiple
metavariables that use regular expressions?

julia

>
> +1 for group and escaping support if it might be a trivial patch.
>
>
> --
> Eric Wheeler
>
> On Wed, 9 Mar 2022, Julia Lawall wrote:
>
>
> >
> > julia
> >
> > On Tue, 8 Mar 2022, Eric Wheeler wrote:
> >
> > > Hi all,
> > >
> > > I would like to replace this:
> > >
> > > 	void on_struct_save_as_gnuplot_activate(
> > > 	    GtkMenuItem     *menuitem,
> > > 	    gpointer         user_data)
> > > 	{
> > > 	  file_chooser = Open_Filechooser( GTK_FILE_CHOOSER_ACTION_SAVE,
> > > 	      "*.gplot", NULL, "untitled.gplot", rc_config.working_dir );
> > > 	}
> > >
> > > And extract the extension ".gplot" as an identifier and pass it to the
> > > function get_nec_filename_stem, like so:
> > >
> > > 	void on_struct_save_as_gnuplot_activate(
> > > 	    GtkMenuItem     *menuitem,
> > > 	    gpointer         user_data)
> > > 	{
> > > 	  file_chooser = Open_Filechooser( GTK_FILE_CHOOSER_ACTION_SAVE,
> > > 	      "*.gplot", NULL, get_nec_filename_stem(newfn, ".gplot", PATH_MAX), rc_config.working_dir );
> > > 	}
> > >
> > > The SmPL patch below works for what I want statically, but notice the
> > > commented `expression E`:
> > >
> > > Is there a way to regex-group `expression E =~` with a parenthesis and then specify
> > > something like E[0] to pick up the grouped regex match?
> > >
> > > 	@ replace_untitled @
> > > 	identifier F;
> > > 	parameter list PL;
> > > 	expression list EL1, EL2;
> > > 	//expression E =~ "\"untitled\.(.*)\"";
> > > 	@@
> > >
> > > 	F(PL)
> > > 	{
> > > 	+   char newfn[PATH_MAX];
> > > 	    ...
> > > 	    file_chooser = Open_Filechooser(EL1,
> > > 	-       "untitled.gplot",
> > > 	+       get_nec_filename_stem(newfn, ".gplot", PATH_MAX),
> > > 		EL2);
> > > 	    ...
> > > 	}
> > >
> > >
> > > This doesn't work, but it explains what I'm trying to do:
> > >
> > > 	@ replace_untitled @
> > > 	identifier F;
> > > 	parameter list PL;
> > > 	expression list EL1, EL2;
> > > 	expression E =~ "untitled.(.*)";
> > > 	@@
> > >
> > > 	F(PL)
> > > 	{
> > > 	+   char newfn[PATH_MAX];
> > > 	    ...
> > > 	    file_chooser = Open_Filechooser(EL1,
> > > 	-       E,
> > > 	+       get_nec_filename_stem(newfn, E[0], PATH_MAX),
> > > 		EL2);
> > > 	    ...
> > > 	}
> > >
> > > It replaces as follows and does match "untitled.(.*)", but of course [0] is incorrect SmPL grammar:
> > >
> > > 	@@ -250,10 +253,13 @@ on_struct_save_as_gnuplot_activate(
> > > 	     GtkMenuItem     *menuitem,
> > > 	     gpointer         user_data)
> > > 	 {
> > > 	+  char newfn[PATH_MAX];
> > > 	   /* Open file chooser to save structure image */
> > > 	   SetFlag( STRUCT_GNUPLOT_SAVE );
> > > 	   file_chooser = Open_Filechooser( GTK_FILE_CHOOSER_ACTION_SAVE,
> > > 	-      "*.gplot", NULL, "untitled.gplot", rc_config.working_dir );
> > > 	+      "*.gplot", NULL,
> > > 	+      get_nec_filename_stem(newfn, "untitled.gplot"[0], PATH_MAX),
> > > 	+      rc_config.working_dir );
> > > 	 }
> > >
> > > I tried something like this:
> > > 	expression F =~ "\..*";
> > > 	expression E =~ "untitled" ##F;
> > > but that is invalid too.
> > >
> > > Ideas?
> > >
> > >
> > > --
> > > Eric Wheeler
> > >
> >
>

Re: [cocci] Does SmPL support regex groups?

[Eric Wheeler]

On Wed, 9 Mar 2022, Julia Lawall wrote:
> On Wed, 9 Mar 2022, Eric Wheeler wrote:
> > > Regex groups are not supported.  You can use python to do whatever you
> > > want with the matched terms.  See demos/pythontococci.cocci
> >
> > Thanks Julia.  It seems the Coccinelle =~ matches do not support escaping.
> > I was getting errors trying to match like so:
> >
> > expression E =~ "\.gplot";
> >
> > It works without the \ but errors with the .
> > 	File "cocci/untitled-fix.cocci", line 5, column 35, charpos = 144
> > 	  around = '\.',
> > 	  whole content = expression EXT, arg, E =~ "untitled\.(.*)";
> >
> > It failed scaping \" as well but now I'm unable to reproduce the problem.
> > Sounds like python is the way to go, built-in regex's are useful and might
> > need a bit of cleanup.
> 
> I would imagine that you need to put "\\.gplot"
> I would imagine that escaping is trivial to addres.  

Running this against a few examples:

	@ r @
	constant C =~ "(a)\\.([a-z])";
	@@
	-C
	+"foo"

No match:
	main()
	{   
	    char *a = "a.b";
	} 

No match:
	main()
	{   
	    char *a = "aXb";
	} 

No match:
	main()
	{   
	    char *a = "a\\.b";
	}   

Does match:
	main()
	{   
	    char *a = "a\\b";
	}
... so I'm not sure what escapes are doing with the "." in the regex, it 
seems to have disappeared!

It occurs to me that to match an actual "." I could have used "[.]" in the 
regex instead of trying to escape with "\."

However, for quotes, this spatch:
	@ r @
	constant C =~ "\\\"ab\\\"";
	@@
	-C
	+"foo"

Does indeed match this:
	main()
	{   
	    char *a = "\"ab\"";
	}  
and it makes sense that \ needs escaped and the " needs escaped, thus \\\" 
matches the C string of "\"".  It might be more intuative to the SmPL 
author if the regex matched without all those extra escapes.
 
> I have no idea what one would even want from grouping.

See my example below for a real-world case, but first, a simpler contrived 
example...

> That is, what notation do you expect to use to refer to the groups.  
> And which group, if there are multiple metavariables that use regular 
> expressions?

I would think that the group would be bound to the metavariable since 
the metavariable is tied 1:1 to the regex.

I'm not sure what syntax fits best with the Coccinelle paradigm, so 
whatever works best for you in terms of group reference.  In the examples 
below I use @n where n is the group number.

Here is a possible example to replace the string "az" with "z" and "run 
quick" with "quick run" where @n could be the group number:

	@ r @
	constant C =~ "(a)([a-z])";
	constant D =~ "(run|walk) (quick)";
	@@
	-C
	+C@2
	...
	-D
	+D@2 D@1 /* "quick run", or possibly, "quick walk" */


In my example from the original thread post (below), I was trying to 
replace all constant strings "untitled(\..*$)" with f(group1) because the 
refactor changes things like "untitled.png" to only pass the extension 
".png" (get_nec_filename_stem() will use lookup the original filename so 
"untitled" needs to be dropped and the extension placed in the function 
call.).

There were lots of "untitled.png", "untitled.nec", "untitled.s1p", etc., 
so a quick regex could pull out the filename extension in this case and 
put it in the function call in the right position:

	@ replace_untitled @
	identifier F;
	parameter list PL;
	expression list EL1, EL2;
	constant C =~ "untitled(\..*)";
	@@

	F(PL)
	{   
	+   char newfn[PATH_MAX];
	    ...
	    file_chooser = Open_Filechooser(EL1,
	-       C,
	+       get_nec_filename_stem(newfn, C@1, PATH_MAX),
		EL2);
	    ...
	}

I'm not sure how common of a use-case this was, but if OCaml already 
supports regex groups then perhaps it is trivial to create a "group" 
notation in SmPL for metavariables using regex's.  (Or perhaps not, I 
really haven't used OCaml so I can't comment on how easy that would be!)

Anyway, my needs for this are already met with a combination of SmPL 
cleverness and a bit of manual intervention, so no need to rush about 
trying to implement this on my behalf unless it would benefit others.

-Eric

Re: [cocci] Does SmPL support regex groups?

[Julia Lawall]

> I'm not sure how common of a use-case this was, but if OCaml already
> supports regex groups then perhaps it is trivial to create a "group"
> notation in SmPL for metavariables using regex's.  (Or perhaps not, I
> really haven't used OCaml so I can't comment on how easy that would be!)

The problem is how to refer to the groups in the semantic patch.

I guess that the @n notation is possible, but I'm not enthusiastic.  But
on the ihter hand I never use regular expressions either.  There is still
the option of script code, as in demos/pythontococci.

julia

Re: [cocci] Does SmPL support regex groups?

[Markus Elfring]

> I have no idea what one would even want from grouping.

I find this feedback surprising.


> That is, what notation do you expect to use to refer to the groups.
> And which group, if there are multiple metavariables that use regular expressions?

I see possibilities like the following.

A)
The specification of these regular expressions is connected with a constraint
for specific metavariables.
Such metavariables could provide additional data by special access operators
and corresponding programming interfaces.

B)
If direct access to special information would not be offered, I imagine that
additional options could be specified for the evaluation of capture groups
(and the selection of regex engines).
Additional variables could store captured contents.


Regards,
Markus

Re: [cocci] Does SmPL support regex groups?

[Markus Elfring]

> Anyway, my needs for this are already met with a combination of SmPL
> cleverness and a bit of manual intervention,

Would you like to share a bit more information about a (temporary) solution
which you find good enough at the moment?


> so no need to rush about trying to implement this on my behalf

Thanks for your nice feedback.


> unless it would benefit others.

I guess that further demonstrations for the application of (advanced)
regular expressions in combination with the semantic patch language can help
to clarify corresponding use cases.

Regards,
Markus

Re: [cocci] Does SmPL support regex groups?

[Eric Wheeler]

On Thu, 10 Mar 2022, Markus Elfring wrote:
> > Anyway, my needs for this are already met with a combination of SmPL
> > cleverness and a bit of manual intervention,
> 
> Would you like to share a bit more information about a (temporary) solution
> which you find good enough at the moment?

This was the resulting diff from git after doing the complete refactor to 
remove "untitled" from default filenames:

	@@ -991,6 +1019,7 @@ on_freqplots_save_as_activate(
	     GtkMenuItem     *menuitem,
	     gpointer         user_data)
	 {
	+  char newfn[PATH_MAX];
	   saveas_drawingarea = freqplots_drawingarea;
	   saveas_width  = freqplots_width;
	   saveas_height = freqplots_height;
	@@ -998,7 +1027,8 @@ on_freqplots_save_as_activate(
	   /* Open file chooser to save frequency plots */
	   SetFlag( IMAGE_SAVE );
	   file_chooser = Open_Filechooser( GTK_FILE_CHOOSER_ACTION_SAVE,
	-      "*.png", NULL, _("untitled.png"), rc_config.working_dir );
	+      "*.png", NULL, get_nec_filename_stem(newfn, ".png", PATH_MAX),
	+      rc_config.working_dir );
	 }

This was the Coccinelle SmPL:

	@ replace_untitled @
	identifier F != {on_nec2_save_as_clicked};
	parameter list PL;
	expression list EL1, EL2;
	expression EXT, arg, E =~ "untitled\.(.*)";
	@@

	F(PL)
	{
	+	char newfn[PATH_MAX];
		...
		file_chooser = Open_Filechooser(GTK_FILE_CHOOSER_ACTION_SAVE, EXT, arg,
	-		_(E),
	+		get_nec_filename_stem(newfn, EXT, PATH_MAX),
			EL2);
		...
	}

but that just moved the existing "*.png" in the metavariable EXT into EXT 
within get_nec_filename_stem(), which wasn't quite enough, because 
get_nec_filename_stem() wants ".png" not "*.png": so I manually deleted 
the *'s.

It seems that I use Coccinelle daily: there is always something that is 
easier solved with SmPL than manual editing, even if there is some minor 
intervention after SmPL does the heavy lifting.


--
Eric Wheeler



> 
> 
> > so no need to rush about trying to implement this on my behalf
> 
> Thanks for your nice feedback.
> 
> 
> > unless it would benefit others.
> 
> I guess that further demonstrations for the application of (advanced)
> regular expressions in combination with the semantic patch language can help
> to clarify corresponding use cases.
> 
> Regards,
> Markus
> 

Re: [cocci] Does SmPL support regex groups?

[Nicolas Palix]

Hi,

If there is not too much extensions, maybe you can enumerate them in the 
SmPL, or generate a SmPL script per-extension after a collecting phase.


The Regexp are the ones from the Ocaml Str module, or PCRE according to 
the configuration.
https://github.com/coccinelle/coccinelle/blob/master/globals/regexp_pcre.ml

So, the group definition are legal for the matching, but maybe depends 
of the underlying Regexp used.

Moreover, but because we have Python and OCaml escapes with more power 
to generate thing, using group seems hard to use from a SmPL syntax 
perspective.

I like the C@1 notation to get the group, but how would you say
to concatenate it with a prefix/suffix either static or variable (from 
another metavariable) ?

As your example suggests, maybe keeping only the first group if any is 
defined in the + part would be great, and all the bind string otherwise.

I retrieve the matching phase here
https://github.com/coccinelle/coccinelle/blob/master/engine/cocci_vs_c.ml

Julia, where can I find the + part that where the bind string
is used ? Can we still have the Regexp at that point ?


On 10/03/2022 23:03, Eric Wheeler wrote:
> On Thu, 10 Mar 2022, Markus Elfring wrote:
>>> Anyway, my needs for this are already met with a combination of SmPL
>>> cleverness and a bit of manual intervention,
>>
>> Would you like to share a bit more information about a (temporary) solution
>> which you find good enough at the moment?
> 
> This was the resulting diff from git after doing the complete refactor to
> remove "untitled" from default filenames:
> 
> 	@@ -991,6 +1019,7 @@ on_freqplots_save_as_activate(
> 	     GtkMenuItem     *menuitem,
> 	     gpointer         user_data)
> 	 {
> 	+  char newfn[PATH_MAX];
> 	   saveas_drawingarea = freqplots_drawingarea;
> 	   saveas_width  = freqplots_width;
> 	   saveas_height = freqplots_height;
> 	@@ -998,7 +1027,8 @@ on_freqplots_save_as_activate(
> 	   /* Open file chooser to save frequency plots */
> 	   SetFlag( IMAGE_SAVE );
> 	   file_chooser = Open_Filechooser( GTK_FILE_CHOOSER_ACTION_SAVE,
> 	-      "*.png", NULL, _("untitled.png"), rc_config.working_dir );
> 	+      "*.png", NULL, get_nec_filename_stem(newfn, ".png", PATH_MAX),
> 	+      rc_config.working_dir );
> 	 }
> 
> This was the Coccinelle SmPL:
> 
> 	@ replace_untitled @
> 	identifier F != {on_nec2_save_as_clicked};
> 	parameter list PL;
> 	expression list EL1, EL2;
> 	expression EXT, arg, E =~ "untitled\.(.*)";
> 	@@
> 
> 	F(PL)
> 	{
> 	+	char newfn[PATH_MAX];
> 		...
> 		file_chooser = Open_Filechooser(GTK_FILE_CHOOSER_ACTION_SAVE, EXT, arg,
> 	-		_(E),
> 	+		get_nec_filename_stem(newfn, EXT, PATH_MAX),
> 			EL2);
> 		...
> 	}
> 
> but that just moved the existing "*.png" in the metavariable EXT into EXT
> within get_nec_filename_stem(), which wasn't quite enough, because
> get_nec_filename_stem() wants ".png" not "*.png": so I manually deleted
> the *'s.
> 
> It seems that I use Coccinelle daily: there is always something that is
> easier solved with SmPL than manual editing, even if there is some minor
> intervention after SmPL does the heavy lifting.
> 
> 
> --
> Eric Wheeler
> 
> 
> 
>>
>>
>>> so no need to rush about trying to implement this on my behalf
>>
>> Thanks for your nice feedback.
>>
>>
>>> unless it would benefit others.
>>
>> I guess that further demonstrations for the application of (advanced)
>> regular expressions in combination with the semantic patch language can help
>> to clarify corresponding use cases.
>>
>> Regards,
>> Markus
>>


-- 
Nicolas Palix

Re: [cocci] Does SmPL support regex groups?

[Julia Lawall]

On Fri, 11 Mar 2022, Nicolas Palix wrote:

> Hi,
>
> If there is not too much extensions, maybe you can enumerate them in the SmPL,
> or generate a SmPL script per-extension after a collecting phase.
>
>
> The Regexp are the ones from the Ocaml Str module, or PCRE according to the
> configuration.
> https://github.com/coccinelle/coccinelle/blob/master/globals/regexp_pcre.ml
>
> So, the group definition are legal for the matching, but maybe depends of the
> underlying Regexp used.
>
> Moreover, but because we have Python and OCaml escapes with more power to
> generate thing, using group seems hard to use from a SmPL syntax perspective.
>
> I like the C@1 notation to get the group, but how would you say
> to concatenate it with a prefix/suffix either static or variable (from another
> metavariable) ?
>
> As your example suggests, maybe keeping only the first group if any is defined
> in the + part would be great, and all the bind string otherwise.
>
> I retrieve the matching phase here
> https://github.com/coccinelle/coccinelle/blob/master/engine/cocci_vs_c.ml
>
> Julia, where can I find the + part that where the bind string
> is used ? Can we still have the Regexp at that point ?

The actual use of the + code is in parsing_c/unparse_cocci.ml.  For the
regexp to be available, it would have to be stored in the matavars_binding
type that is defined in parsing_c/ast_c.ml

References to grouping only makes sense in + code.

References to grouping would be ok for metavariables of type constant.
But they could lead to problems for other kinds of metavariables.  For
example, a expression metavariable could end up bound to something that is
not an expression.  So Coccinelle would have to reparse the extracted
string like one does in python or ocaml scripts (pythontococci.ml or
ocamltococci.ml).

julia

>
>
> On 10/03/2022 23:03, Eric Wheeler wrote:
> > On Thu, 10 Mar 2022, Markus Elfring wrote:
> > > > Anyway, my needs for this are already met with a combination of SmPL
> > > > cleverness and a bit of manual intervention,
> > >
> > > Would you like to share a bit more information about a (temporary)
> > > solution
> > > which you find good enough at the moment?
> >
> > This was the resulting diff from git after doing the complete refactor to
> > remove "untitled" from default filenames:
> >
> > 	@@ -991,6 +1019,7 @@ on_freqplots_save_as_activate(
> > 	     GtkMenuItem     *menuitem,
> > 	     gpointer         user_data)
> > 	 {
> > 	+  char newfn[PATH_MAX];
> > 	   saveas_drawingarea = freqplots_drawingarea;
> > 	   saveas_width  = freqplots_width;
> > 	   saveas_height = freqplots_height;
> > 	@@ -998,7 +1027,8 @@ on_freqplots_save_as_activate(
> > 	   /* Open file chooser to save frequency plots */
> > 	   SetFlag( IMAGE_SAVE );
> > 	   file_chooser = Open_Filechooser( GTK_FILE_CHOOSER_ACTION_SAVE,
> > 	-      "*.png", NULL, _("untitled.png"), rc_config.working_dir );
> > 	+      "*.png", NULL, get_nec_filename_stem(newfn, ".png", PATH_MAX),
> > 	+      rc_config.working_dir );
> > 	 }
> >
> > This was the Coccinelle SmPL:
> >
> > 	@ replace_untitled @
> > 	identifier F != {on_nec2_save_as_clicked};
> > 	parameter list PL;
> > 	expression list EL1, EL2;
> > 	expression EXT, arg, E =~ "untitled\.(.*)";
> > 	@@
> >
> > 	F(PL)
> > 	{
> > 	+	char newfn[PATH_MAX];
> > 		...
> > 		file_chooser = Open_Filechooser(GTK_FILE_CHOOSER_ACTION_SAVE,
> > EXT, arg,
> > 	-		_(E),
> > 	+		get_nec_filename_stem(newfn, EXT, PATH_MAX),
> > 			EL2);
> > 		...
> > 	}
> >
> > but that just moved the existing "*.png" in the metavariable EXT into EXT
> > within get_nec_filename_stem(), which wasn't quite enough, because
> > get_nec_filename_stem() wants ".png" not "*.png": so I manually deleted
> > the *'s.
> >
> > It seems that I use Coccinelle daily: there is always something that is
> > easier solved with SmPL than manual editing, even if there is some minor
> > intervention after SmPL does the heavy lifting.
> >
> >
> > --
> > Eric Wheeler
> >
> >
> >
> > >
> > >
> > > > so no need to rush about trying to implement this on my behalf
> > >
> > > Thanks for your nice feedback.
> > >
> > >
> > > > unless it would benefit others.
> > >
> > > I guess that further demonstrations for the application of (advanced)
> > > regular expressions in combination with the semantic patch language can
> > > help
> > > to clarify corresponding use cases.
> > >
> > > Regards,
> > > Markus
> > >
>
>
> --
> Nicolas Palix
>

Re: [cocci] Does SmPL support regex groups?

[Markus Elfring]

> I like the C@1 notation to get the group,


I suggest to reconsider this specification approach further.



> but how would you say to concatenate it with a prefix/suffix either static or variable
> (from another metavariable) ?


I would like to point other software design options out.

The syntax for the @ operator of the semantic patch language is described in the way
that it should be connected with variable identifiers.

https://gitlab.inria.fr/coccinelle/coccinelle/-/blob/5479fc964d209209e84ddd5caf3f395667721e7c/docs/manual/cocci_syntax.tex#L410

https://github.com/coccinelle/coccinelle/blob/ae337fce1512ff15aabc3ad5b6d2e537f97ab62a/docs/manual/cocci_syntax.tex#L410


One of the supported metavariables has got the type “position”.
This metavariable kind has got the special property that it refers to
a concrete source code place (and not to a syntax element).

Thus I propose the addition of support for the metavariable type “capture group”.

See also:
2015-08-27: Feature request “Reuse of content from capturing groups in regular expressions”
https://github.com/coccinelle/coccinelle/issues/46


I propose also to take the support for named capture groups into account.
https://www.regular-expressions.info/named.html

Thus I imagine that there is a need to enable the specification of an index
for special metavariable types.
The corresponding index can be specified as a number or an identifier.

I guess that path specifications can become relevant for references to captured contents
from nested groups.

Regards,
Markus

Re: [cocci] Does SmPL support regex groups?

[Nicolas Palix]

For the record, is it the expected SmPL ?

@ fu @
identifier F,file_chooser;
parameter list PL;
expression list EL1, EL2;
constant U =~ "untitled";
@@

F(PL)
{
     ...
     file_chooser = Open_Filechooser(EL1, U, EL2);
     ...
}

@script:python py@
U << fu.U;
C;
@@

print U # Rewrite U to extract the extension
coccinelle.C = cocci.make_expr("\".gplot\"")

@ replace_untitled @
identifier F,file_chooser;
parameter list PL;
expression list EL1, EL2;
constant fu.U;
expression py.C;
@@

F(PL)
{
+   char newfn[PATH_MAX];
     ...
     file_chooser = Open_Filechooser(EL1,
-       U,
+       get_nec_filename_stem(newfn, C, PATH_MAX),
	EL2);
     ...
}



-- 
Nicolas Palix

Re: [cocci] Does SmPL support regex groups?

[Markus Elfring]

> For the record, is it the expected SmPL ?


Your SmPL script variant points specific development ideas out.



> @script:python py@
> U << fu.U;
> C;
> @@
>
> print U # Rewrite U to extract the extension
> coccinelle.C = cocci.make_expr("\".gplot\"")


I find that this code part would need another case distinction
(after the use of a capturing group within a regular expression)
so that the file extension will be extracted as expected for the
desired source code transformation.

Regards,
Markus

Re: [cocci] Does SmPL support regex groups?

[Eric Wheeler]

On Tue, 22 Mar 2022, Nicolas Palix wrote:
> For the record, is it the expected SmPL ?
> 
> @ fu @
> identifier F,file_chooser;
> parameter list PL;
> expression list EL1, EL2;
> constant U =~ "untitled";
> @@
> 
> F(PL)
> {
>     ...
>     file_chooser = Open_Filechooser(EL1, U, EL2);
>     ...
> }
> 
> @script:python py@
> U << fu.U;
> C;
> @@
> 
> print U # Rewrite U to extract the extension
> coccinelle.C = cocci.make_expr("\".gplot\"")

The extension varies, there were half a dozen or so: I had to replace 
several Open_Filechooser() calls that had different extensions.  Here is 
the diff for that commit:
    https://github.com/KJ7LNW/xnec2c/commit/9fb0381c24365311e3bb1fc8158f18112864ebfb#diff-1d0efcfbefd013aedf0477bd522a1e4506339c5ef239a6ef770f39270747c14cL222

Here is one such diff for .png:
  file_chooser = Open_Filechooser( GTK_FILE_CHOOSER_ACTION_SAVE,
-      "*.png", NULL, "untitled.png", rc_config.working_dir );
+      "*.png", NULL, get_nec_filename_stem(newfn, ".png", PATH_MAX),
      rc_config.working_dir );

I'm not so familiar with Python, so forgive my likely incorrect syntax, 
but this might work:

	coccinelle.C = re.match("(\..*)$", U).group(1)

-Eric

> @ replace_untitled @
> identifier F,file_chooser;
> parameter list PL;
> expression list EL1, EL2;
> constant fu.U;
> expression py.C;
> @@
> 
> F(PL)
> {
> +   char newfn[PATH_MAX];
>     ...
>     file_chooser = Open_Filechooser(EL1,
> -       U,
> +       get_nec_filename_stem(newfn, C, PATH_MAX),
> 	EL2);
>     ...
> }
> 
> 
> 
> -- 
> Nicolas Palix
>