[Cocci] struct replacement

[Cocci] struct replacement

[Max]

Hi.

I have been trying to use spatch to update c struct but haven't found a 
way to do so yet.

For example, we have

struct old_thing {

uint8_t a;

uint16_t b[5];

};

which I'd like to replace with

struct new_thing {

uint8_t x1;

uint32_t x2;

uint8_t x3[11];

};

So pretty much replace every function parameter:

f1(... struct old_thing *t...) => f1(... struct new_thing *t...)

Replace the field access and variable definitions:

...

struct old_thing t1; => struct new_thing t1;

struct old_thing *t2; => struct new_thing *t2;

...

t1.a => t1.x1

t2->b => t2->x3

...

The thing which escapes me is how to make coccinelle treat it like a type?

Do I need to use @@typedef struct old_thing@@? Do I use it as 
@@identifier old_stuff@@ or @@identifier struct old_stuff@@?

Doesn't seem to work for me.

Any links to .spatch examples playing around with structs would be 
greatly appreciated.

-- 
- Max Suraev <msuraev@sysmocom.de>       http://www.sysmocom.de/
=======================================================================
* sysmocom - systems for mobile communications GmbH
* Alt-Moabit 93
* 10559 Berlin, Germany
* Sitz / Registered office: Berlin, HRB 134158 B
* Geschaeftsfuehrer / Managing Directors: Harald Welte

_______________________________________________
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] struct replacement

[Julia Lawall]

On Wed, 12 Dec 2018, Max wrote:

> Hi.
>
> I have been trying to use spatch to update c struct but haven't found a way to
> do so yet.
>
> For example, we have
>
> struct old_thing {
>
> uint8_t a;
>
> uint16_t b[5];
>
> };
>
> which I'd like to replace with
>
> struct new_thing {
>
> uint8_t x1;
>
> uint32_t x2;
>
> uint8_t x3[11];
>
> };
>
> So pretty much replace every function parameter:
>
> f1(... struct old_thing *t...) => f1(... struct new_thing *t...)
>
> Replace the field access and variable definitions:
>
> ...
>
> struct old_thing t1; => struct new_thing t1;
>
> struct old_thing *t2; => struct new_thing *t2;
>
> ...
>
> t1.a => t1.x1
>
> t2->b => t2->x3
>
> ...
>
> The thing which escapes me is how to make coccinelle treat it like a type?
>
> Do I need to use @@typedef struct old_thing@@? Do I use it as @@identifier
> old_stuff@@ or @@identifier struct old_stuff@@?
>
> Doesn't seem to work for me.
>
> Any links to .spatch examples playing around with structs would be greatly
> appreciated.

Again, please send your attempt.  What you want to do actually looks
straightforward, so it's hard for me to guess where the problem could be.

thanks,
julia

>
> --
> - Max Suraev <msuraev@sysmocom.de>       http://www.sysmocom.de/
> =======================================================================
> * sysmocom - systems for mobile communications GmbH
> * Alt-Moabit 93
> * 10559 Berlin, Germany
> * Sitz / Registered office: Berlin, HRB 134158 B
> * Geschaeftsfuehrer / Managing Directors: Harald Welte
>
> _______________________________________________
> Cocci mailing list
> Cocci@systeme.lip6.fr
> https://systeme.lip6.fr/mailman/listinfo/cocci
>
_______________________________________________
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] struct replacement

[Max]

Here's the .spatch I've used:

_________________
@@
identifier a;
@@
- struct old_thing a;
+ struct new_thing a;

@@
identifier b;
@@
- struct old_thing *b;
+ struct new_thing *b;

@@
function f;
identifier c;
@@
- f(... struct old_thing *c ...);
+ f(... struct new_thing *c ...);

_________________

I've run it as "spatch --in-place --sp-file test.spatch -I include --dir 
src" to apply to all source and include files at the same time.

The first 2 patchers work fine replacing all the variable declarations 
inside functions and structs as expected. But they do not affect 
function type signature which I'm trying to correct with the 3rd patcher 
but it gives me exception Lexer_cocci.Lexical("invalid in a + context: 
...") error.

I've tried using expression instead of function but it seems like I'm 
applying "..." operator wrong. I've seen example doing somewhat similar 
with elementary types but I'm at loss as to how to apply this to struct 
pointers.

-- 
- Max Suraev <msuraev@sysmocom.de>       http://www.sysmocom.de/
=======================================================================
* sysmocom - systems for mobile communications GmbH
* Alt-Moabit 93
* 10559 Berlin, Germany
* Sitz / Registered office: Berlin, HRB 134158 B
* Geschaeftsfuehrer / Managing Directors: Harald Welte

_______________________________________________
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] struct replacement

[Julia Lawall]

> @@
> function f;
> identifier c;
> @@
> - f(... struct old_thing *c ...);
> + f(... struct new_thing *c ...);

You can't put ... in + code.  Coccinelle doesn't know what you want to
add.

This has the form of a function call.  If you want to make a function
prototype, it needs a type in front of it.  This can be a type
metavariable, declared with type t;

If this is to be a function prototype, the parameter should be surrounded
with commas.  If there is no comma before or after the parameter
declaration, Coccinelle will take care of doing the right thing.

But actually your rule should be:

@@
@@

struct
-   old_thing
+   new_thing

Then it will just change the type whereever it occurs.

julia
_______________________________________________
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] struct replacement

[Julia Lawall]

On Sat, 15 Dec 2018, SF Markus Elfring wrote:

> > If you want to make a function prototype, it needs a type in front of it.
>
> This information is appropriate according to the syntax of the C programming language.
>
>
> > This can be a type metavariable, declared with type t;
>
> Can this detail be treated as optional by the means of the semantic patch language
> (without using extra SmPL disjunctions)?

No, it is not optinal in the case of a function prototype.  It is optional
in the case of a function definition.  A prototype without a specified
return type looks too much like a function call, and causes parsing
issues.

>
>
> > But actually your rule should be:
>
> @replacement@
> @@
>  struct
> - old_thing
> + new_thing
>
> Such a succinct change specification seems to be nice.
> Will further collateral evolution matter for this transformation approach?
>
> * Will any related adjustments be needed for data type definitions?
>   https://en.cppreference.com/w/c/language/typedef
>
> * Should source code be updated also in header files then?

To include header files, ie to cause them to be treated as independent
files, like .c files, give the command line argument --include-headers.

julia
_______________________________________________
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] struct replacement

[Max]

That almost works: both function parameters and variable declarations 
are updated in .c files but the function prototypes in .h are ignored.

I call it as follows:

spatch --in-place --sp-file struct.spatch -I include --dir src

The source code from 'src' is updated but header files in 'include' are 
not affected.

Specifying multiple dir doesn't work: "ignoring all but the last 
specified directory".

Adding --include-headers doesn't do what I hoped it would.

Using top-level directory doesn't help as well: .c in multiple sub-dirs 
are processed but .h are not.

Reading docs so far revealed no way to update function definition in 
both .c and .h at the same time.

Does spatch have to be called in some special way for this to work? Or 
.spatch file should have some particular syntax so it would affect both 
.c and .h simultaneously?

13.12.18 21:01, Julia Lawall пишет:
>
> If this is to be a function prototype, the parameter should be surrounded
> with commas.  If there is no comma before or after the parameter
> declaration, Coccinelle will take care of doing the right thing.
>
> But actually your rule should be:
>
> @@
> @@
>
> struct
> -   old_thing
> +   new_thing
>
> Then it will just change the type whereever it occurs.
>
-- 
- Max Suraev <msuraev@sysmocom.de>       http://www.sysmocom.de/
=======================================================================
* sysmocom - systems for mobile communications GmbH
* Alt-Moabit 93
* 10559 Berlin, Germany
* Sitz / Registered office: Berlin, HRB 134158 B
* Geschaeftsfuehrer / Managing Directors: Harald Welte

_______________________________________________
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] struct replacement

[Julia Lawall]

[-- Attachment #1: Type: text/plain, Size: 2274 bytes --]



On Tue, 18 Dec 2018, Max wrote:

> That almost works: both function parameters and variable declarations
> are updated in .c files but the function prototypes in .h are ignored.
>
> I call it as follows:
>
> spatch --in-place --sp-file struct.spatch -I include --dir src
>
> The source code from 'src' is updated but header files in 'include' are
> not affected.
>
> Specifying multiple dir doesn't work: "ignoring all but the last
> specified directory".
>
> Adding --include-headers doesn't do what I hoped it would.
>
> Using top-level directory doesn't help as well: .c in multiple sub-dirs
> are processed but .h are not.
>
> Reading docs so far revealed no way to update function definition in
> both .c and .h at the same time.
>
> Does spatch have to be called in some special way for this to work? Or
> .spatch file should have some particular syntax so it would affect both
> .c and .h simultaneously?

By default, Coccinelle only includes the headers that have the same name
as the .c file.  Ie for foo.c, it considers that foo.h is prbably
important, and thus worth the often large extra runtime cost to take into
account.

If you want more aggressive inclusion of headers, you need --all-includes
(the headers explicitly mentioned in the .c file) or --recursive-includes
(headers included by other headers).

julia


>
> 13.12.18 21:01, Julia Lawall пишет:
> >
> > If this is to be a function prototype, the parameter should be surrounded
> > with commas.  If there is no comma before or after the parameter
> > declaration, Coccinelle will take care of doing the right thing.
> >
> > But actually your rule should be:
> >
> > @@
> > @@
> >
> > struct
> > -   old_thing
> > +   new_thing
> >
> > Then it will just change the type whereever it occurs.
> >
> --
> - Max Suraev <msuraev@sysmocom.de>       http://www.sysmocom.de/
> =======================================================================
> * sysmocom - systems for mobile communications GmbH
> * Alt-Moabit 93
> * 10559 Berlin, Germany
> * Sitz / Registered office: Berlin, HRB 134158 B
> * Geschaeftsfuehrer / Managing Directors: Harald Welte
>
> _______________________________________________
> Cocci mailing list
> Cocci@systeme.lip6.fr
> https://systeme.lip6.fr/mailman/listinfo/cocci
>

[-- Attachment #2: Type: text/plain, Size: 136 bytes --]

_______________________________________________
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] struct replacement

[Max]

I see. Yes,  --all-includes did the trick for me, thank you.

Although I'm still have to update struct fields in addition to struct 
name and it seems like I'm using typedef in a wrong way.

For example:

_______________________

@@
@@
struct
- old_addr
+ new_address

@@
identifier a;
typedef new_address;
@@
new_address a;
- a.len;
+ a.length;

_______________________

This does not cause field updates although first part (names) works.

Using 'typedef struct new_address' instead gives "meta: parse error".

Using 'old_addr' instead of 'new_address' in 2nd part doesn't make any 
difference.

The len/length are pretty generic names used in multiple structs. How do 
I instruct Coccinelle to make replacement only for identifiers which 
belong to a particular struct type (or pointer to it)?

Is there some specific metaver syntax related to structs which I haven't 
found yet?

18.12.18 17:19, Julia Lawall пишет:
>
> By default, Coccinelle only includes the headers that have the same name
> as the .c file.  Ie for foo.c, it considers that foo.h is prbably
> important, and thus worth the often large extra runtime cost to take into
> account.
>
> If you want more aggressive inclusion of headers, you need --all-includes
> (the headers explicitly mentioned in the .c file) or --recursive-includes
> (headers included by other headers).
>
> julia

-- 
- Max Suraev <msuraev@sysmocom.de>       http://www.sysmocom.de/
=======================================================================
* sysmocom - systems for mobile communications GmbH
* Alt-Moabit 93
* 10559 Berlin, Germany
* Sitz / Registered office: Berlin, HRB 134158 B
* Geschaeftsfuehrer / Managing Directors: Harald Welte

_______________________________________________
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] struct replacement

[Julia Lawall]

[-- Attachment #1: Type: text/plain, Size: 2750 bytes --]



On Tue, 18 Dec 2018, Max wrote:

> I see. Yes,  --all-includes did the trick for me, thank you.
>
> Although I'm still have to update struct fields in addition to struct
> name and it seems like I'm using typedef in a wrong way.
>
> For example:
>
> _______________________
>
> @@
> @@
> struct
> - old_addr
> + new_address
>
> @@
> identifier a;
> typedef new_address;
> @@
> new_address a;
> - a.len;
> + a.length;

I don't understand what you are trying to do here.  typedef new_address;
is find for indicating that new_address is a typedef.  Ie you have
somewhere said:

typedef ... new_address;

new_address a; will then match a declaration where the type of the
declared variable is new_address.  But I don't understand - a.len;.
First, this is only going to match a statement that comes right after the
variable declaration.  Second, a.len; is a very strange statement, because
it doesn't do anything.

Maybe you want:

@@
typedef new_address;
new_address a;
@@
- a.len
+ a.length

That is, a should be a metavariable representing an expression of the
given type, and then you want to replace a len field by length.

You could also write the change as:

a.
-len
+length

julia

>
> _______________________
>
> This does not cause field updates although first part (names) works.
>
> Using 'typedef struct new_address' instead gives "meta: parse error".
>
> Using 'old_addr' instead of 'new_address' in 2nd part doesn't make any
> difference.
>
> The len/length are pretty generic names used in multiple structs. How do
> I instruct Coccinelle to make replacement only for identifiers which
> belong to a particular struct type (or pointer to it)?
>
> Is there some specific metaver syntax related to structs which I haven't
> found yet?
>
> 18.12.18 17:19, Julia Lawall пишет:
> >
> > By default, Coccinelle only includes the headers that have the same name
> > as the .c file.  Ie for foo.c, it considers that foo.h is prbably
> > important, and thus worth the often large extra runtime cost to take into
> > account.
> >
> > If you want more aggressive inclusion of headers, you need --all-includes
> > (the headers explicitly mentioned in the .c file) or --recursive-includes
> > (headers included by other headers).
> >
> > julia
>
> --
> - Max Suraev <msuraev@sysmocom.de>       http://www.sysmocom.de/
> =======================================================================
> * sysmocom - systems for mobile communications GmbH
> * Alt-Moabit 93
> * 10559 Berlin, Germany
> * Sitz / Registered office: Berlin, HRB 134158 B
> * Geschaeftsfuehrer / Managing Directors: Harald Welte
>
> _______________________________________________
> Cocci mailing list
> Cocci@systeme.lip6.fr
> https://systeme.lip6.fr/mailman/listinfo/cocci
>

[-- Attachment #2: Type: text/plain, Size: 136 bytes --]

_______________________________________________
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] struct replacement

[Max]

So, if I don't have any typedef in the code and I just want to replace 
the name of the field of a struct?

for example to modify this function:

f() {

struct my_struct *m;

struct my_struct x;

...

m->len = 10; // this should become m->length = 10;

...

fill(x.buf); // this should become fill(x.addr);

}

I'm trying to use

@@
identifier a;
@@
struct my_struct a.
- len
+ length

Which gives syntax error. I need somehow to tell spatch that "identifier 
a" has type "struct my_struct" but "type struct osmo_gsn_address;" seems 
to be invalid syntax.

18.12.18 17:54, Julia Lawall пишет:
>
> I don't understand what you are trying to do here.  typedef new_address;
> is find for indicating that new_address is a typedef.  Ie you have
> somewhere said:
>
> typedef ... new_address;
>
> new_address a; will then match a declaration where the type of the
> declared variable is new_address.  But I don't understand - a.len;.
> First, this is only going to match a statement that comes right after the
> variable declaration.  Second, a.len; is a very strange statement, because
> it doesn't do anything.
>
> Maybe you want:
>
> @@
> typedef new_address;
> new_address a;
> @@
> - a.len
> + a.length
>
> That is, a should be a metavariable representing an expression of the
> given type, and then you want to replace a len field by length.
>
> You could also write the change as:
>
> a.
> -len
> +length
>
> julia

-- 
- Max Suraev <msuraev@sysmocom.de>       http://www.sysmocom.de/
=======================================================================
* sysmocom - systems for mobile communications GmbH
* Alt-Moabit 93
* 10559 Berlin, Germany
* Sitz / Registered office: Berlin, HRB 134158 B
* Geschaeftsfuehrer / Managing Directors: Harald Welte

_______________________________________________
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] struct replacement

[Julia Lawall]

[-- Attachment #1: Type: text/plain, Size: 2384 bytes --]



On Tue, 18 Dec 2018, Max wrote:

> So, if I don't have any typedef in the code and I just want to replace
> the name of the field of a struct?
>
> for example to modify this function:
>
> f() {
>
> struct my_struct *m;
>
> struct my_struct x;
>
> ...
>
> m->len = 10; // this should become m->length = 10;
>
> ...
>
> fill(x.buf); // this should become fill(x.addr);
>
> }
>
> I'm trying to use
>
> @@
> identifier a;
> @@
> struct my_struct a.
> - len
> + length
>
> Which gives syntax error. I need somehow to tell spatch that "identifier
> a" has type "struct my_struct" but "type struct osmo_gsn_address;" seems
> to be invalid syntax.

You give information about metavariables in betwen the first two @@.  Not
down in the code matching part.

Here you want struct my_struct a; in the metavariable list, and then the
same pattern as before after the second @@.

You can also do

@@
{new_address,struct my_struct} a;
@@

a.
- len
+ length

julia

>
> 18.12.18 17:54, Julia Lawall пишет:
> >
> > I don't understand what you are trying to do here.  typedef new_address;
> > is find for indicating that new_address is a typedef.  Ie you have
> > somewhere said:
> >
> > typedef ... new_address;
> >
> > new_address a; will then match a declaration where the type of the
> > declared variable is new_address.  But I don't understand - a.len;.
> > First, this is only going to match a statement that comes right after the
> > variable declaration.  Second, a.len; is a very strange statement, because
> > it doesn't do anything.
> >
> > Maybe you want:
> >
> > @@
> > typedef new_address;
> > new_address a;
> > @@
> > - a.len
> > + a.length
> >
> > That is, a should be a metavariable representing an expression of the
> > given type, and then you want to replace a len field by length.
> >
> > You could also write the change as:
> >
> > a.
> > -len
> > +length
> >
> > julia
>
> --
> - Max Suraev <msuraev@sysmocom.de>       http://www.sysmocom.de/
> =======================================================================
> * sysmocom - systems for mobile communications GmbH
> * Alt-Moabit 93
> * 10559 Berlin, Germany
> * Sitz / Registered office: Berlin, HRB 134158 B
> * Geschaeftsfuehrer / Managing Directors: Harald Welte
>
> _______________________________________________
> Cocci mailing list
> Cocci@systeme.lip6.fr
> https://systeme.lip6.fr/mailman/listinfo/cocci
>

[-- Attachment #2: Type: text/plain, Size: 136 bytes --]

_______________________________________________
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] struct replacement

[Max]

Right!

Spatch is smart enough to recognize that "struct smth" is a type definition.

Thank you, works even better than I've expected now :)

18.12.18 18:18, Julia Lawall пишет:
>
> You give information about metavariables in betwen the first two @@.  Not
> down in the code matching part.
>
> Here you want struct my_struct a; in the metavariable list, and then the
> same pattern as before after the second @@.
>
> You can also do
>
> @@
> {new_address,struct my_struct} a;
> @@
>
> a.
> - len
> + length
>
> julia
>
-- 
- Max Suraev <msuraev@sysmocom.de>       http://www.sysmocom.de/
=======================================================================
* sysmocom - systems for mobile communications GmbH
* Alt-Moabit 93
* 10559 Berlin, Germany
* Sitz / Registered office: Berlin, HRB 134158 B
* Geschaeftsfuehrer / Managing Directors: Harald Welte

_______________________________________________
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci