[Cocci] Help with include coccinelle & include paths?

[Cocci] Help with include coccinelle & include paths?

[Kumar Gala]

Hi,

I?m looking at using coccinelle with the Zephyr project:

https://www.zephyrproject.org/
https://gerrit.zephyrproject.org/r/gitweb?p=zephyr.git;a=summary

We utilize Kconfig & Kbuild from linux kernel.  However, I?m trying to figure out how to deal with our includes.  I?ve played around with the various options to spatch, but either get a StackOverflow --recursive-includes or not getting all cases figured out since typedef & #defines aren?t getting parsed fully.

Wondering what are best practices or things to try here.  I?m also not clear on what the messages one gets when enable some of the debug options like --verbose-parsing --type-error-msg.

- k

[Cocci] Help with include coccinelle & include paths?

[Julia Lawall]

On Thu, 26 Jan 2017, Kumar Gala wrote:

> Hi,
>
> I?m looking at using coccinelle with the Zephyr project:
>
> https://www.zephyrproject.org/
> https://gerrit.zephyrproject.org/r/gitweb?p=zephyr.git;a=summary
>
> We utilize Kconfig & Kbuild from linux kernel.  However, I?m trying to figure out how to deal with our includes.  I?ve played around with the various options to spatch, but either get a StackOverflow --recursive-includes or not getting all cases figured out since typedef & #defines aren?t getting parsed fully.
>
> Wondering what are best practices or things to try here.  I?m also not clear on what the messages one gets when enable some of the debug options like --verbose-parsing --type-error-msg.

If you just want the includes for parsing and type inference, but not for
transformation, it may help to use --include-headers-for-types.  I'm a bit
surprised about the stack overflow, though.  Maybe there is just some
header file that is badly structured in some way.

Includes won't help you much with #defines.  Coccinelle doesn't normally
take them into account even when the definition is available.  If there
are some non-Linux like macros in your code that reak parsing, you will
need to update standard.h.  What you can do is run:

spatch --parse-c mysoftware

and then it will parse all the files in mysoftware and make a list of the
tokens that most commonly cause errors.  It is usully sufficient to add a
few definitions to standard.h to resolve most of the problems.

Coccinelle parses one top-evel unit (function etc) at a time.  When you
use --verbose-parsing, it will print out the units that have parsing
problems.  It puts BAD on the line with the error, and bad on the lines
that are skipped due to the error.  When there are errors, Coccinelle does
try up to 4 times to parse the code, with each time being a bit more
relaxed.  So something can fail and then ultimately succeed.

I'll run parse-c on the above git link and see if any major issues emerge.

julia

[Cocci] Help with include coccinelle & include paths?

[Kumar Gala]

> On Jan 26, 2017, at 9:19 AM, Julia Lawall <julia.lawall@lip6.fr> wrote:
> 
> 
> 
> On Thu, 26 Jan 2017, Kumar Gala wrote:
> 
>> Hi,
>> 
>> I?m looking at using coccinelle with the Zephyr project:
>> 
>> https://www.zephyrproject.org/
>> https://gerrit.zephyrproject.org/r/gitweb?p=zephyr.git;a=summary
>> 
>> We utilize Kconfig & Kbuild from linux kernel.  However, I?m trying to figure out how to deal with our includes.  I?ve played around with the various options to spatch, but either get a StackOverflow --recursive-includes or not getting all cases figured out since typedef & #defines aren?t getting parsed fully.
>> 
>> Wondering what are best practices or things to try here.  I?m also not clear on what the messages one gets when enable some of the debug options like --verbose-parsing --type-error-msg.
> 
> If you just want the includes for parsing and type inference, but not for
> transformation, it may help to use --include-headers-for-types.  I'm a bit
> surprised about the stack overflow, though.  Maybe there is just some
> header file that is badly structured in some way.
> 
> Includes won't help you much with #defines.  Coccinelle doesn't normally
> take them into account even when the definition is available.  If there
> are some non-Linux like macros in your code that reak parsing, you will
> need to update standard.h.  What you can do is run:
> 
> spatch --parse-c mysoftware
> 
> and then it will parse all the files in mysoftware and make a list of the
> tokens that most commonly cause errors.  It is usully sufficient to add a
> few definitions to standard.h to resolve most of the problems.
> 
> Coccinelle parses one top-evel unit (function etc) at a time.  When you
> use --verbose-parsing, it will print out the units that have parsing
> problems.  It puts BAD on the line with the error, and bad on the lines
> that are skipped due to the error.  When there are errors, Coccinelle does
> try up to 4 times to parse the code, with each time being a bit more
> relaxed.  So something can fail and then ultimately succeed.
> 
> I'll run parse-c on the above git link and see if any major issues emerge.
> 
> julia

When I tried parse-c I get the stack overflow (tired ulimit -s 50000 as well):

[galak at atx zephyr]$ spatch --macro-file-builtins $ZEPHYR_BASE/zephyr-std.h --parse-c $ZEPHYR_BASE
init_defs_builtins: /mnt/nvme/zephyr/zephyr-std.h
checking stack size (do ulimit -s 50000 if problem)
Fatal error: exception Stack overflow

- k

[Cocci] Help with include coccinelle & include paths?

[Julia Lawall]

On Thu, 26 Jan 2017, Kumar Gala wrote:

>
> > On Jan 26, 2017, at 9:19 AM, Julia Lawall <julia.lawall@lip6.fr> wrote:
> >
> >
> >
> > On Thu, 26 Jan 2017, Kumar Gala wrote:
> >
> >> Hi,
> >>
> >> I?m looking at using coccinelle with the Zephyr project:
> >>
> >> https://www.zephyrproject.org/
> >> https://gerrit.zephyrproject.org/r/gitweb?p=zephyr.git;a=summary
> >>
> >> We utilize Kconfig & Kbuild from linux kernel.  However, I?m trying to figure out how to deal with our includes.  I?ve played around with the various options to spatch, but either get a StackOverflow --recursive-includes or not getting all cases figured out since typedef & #defines aren?t getting parsed fully.
> >>
> >> Wondering what are best practices or things to try here.  I?m also not clear on what the messages one gets when enable some of the debug options like --verbose-parsing --type-error-msg.
> >
> > If you just want the includes for parsing and type inference, but not for
> > transformation, it may help to use --include-headers-for-types.  I'm a bit
> > surprised about the stack overflow, though.  Maybe there is just some
> > header file that is badly structured in some way.
> >
> > Includes won't help you much with #defines.  Coccinelle doesn't normally
> > take them into account even when the definition is available.  If there
> > are some non-Linux like macros in your code that reak parsing, you will
> > need to update standard.h.  What you can do is run:
> >
> > spatch --parse-c mysoftware
> >
> > and then it will parse all the files in mysoftware and make a list of the
> > tokens that most commonly cause errors.  It is usully sufficient to add a
> > few definitions to standard.h to resolve most of the problems.
> >
> > Coccinelle parses one top-evel unit (function etc) at a time.  When you
> > use --verbose-parsing, it will print out the units that have parsing
> > problems.  It puts BAD on the line with the error, and bad on the lines
> > that are skipped due to the error.  When there are errors, Coccinelle does
> > try up to 4 times to parse the code, with each time being a bit more
> > relaxed.  So something can fail and then ultimately succeed.
> >
> > I'll run parse-c on the above git link and see if any major issues emerge.
> >
> > julia
>
> When I tried parse-c I get the stack overflow (tired ulimit -s 50000 as well):
>
> [galak at atx zephyr]$ spatch --macro-file-builtins $ZEPHYR_BASE/zephyr-std.h --parse-c $ZEPHYR_BASE
> init_defs_builtins: /mnt/nvme/zephyr/zephyr-std.h
> checking stack size (do ulimit -s 50000 if problem)
> Fatal error: exception Stack overflow

ulimit -s unlimited should make it work

julia

[Cocci] Help with include coccinelle & include paths?

[Kumar Gala]

> On Jan 26, 2017, at 9:42 AM, Julia Lawall <julia.lawall@lip6.fr> wrote:
> 
> 
> 
> On Thu, 26 Jan 2017, Kumar Gala wrote:
> 
>> 
>>> On Jan 26, 2017, at 9:19 AM, Julia Lawall <julia.lawall@lip6.fr> wrote:
>>> 
>>> 
>>> 
>>> On Thu, 26 Jan 2017, Kumar Gala wrote:
>>> 
>>>> Hi,
>>>> 
>>>> I?m looking at using coccinelle with the Zephyr project:
>>>> 
>>>> https://www.zephyrproject.org/
>>>> https://gerrit.zephyrproject.org/r/gitweb?p=zephyr.git;a=summary
>>>> 
>>>> We utilize Kconfig & Kbuild from linux kernel.  However, I?m trying to figure out how to deal with our includes.  I?ve played around with the various options to spatch, but either get a StackOverflow --recursive-includes or not getting all cases figured out since typedef & #defines aren?t getting parsed fully.
>>>> 
>>>> Wondering what are best practices or things to try here.  I?m also not clear on what the messages one gets when enable some of the debug options like --verbose-parsing --type-error-msg.
>>> 
>>> If you just want the includes for parsing and type inference, but not for
>>> transformation, it may help to use --include-headers-for-types.  I'm a bit
>>> surprised about the stack overflow, though.  Maybe there is just some
>>> header file that is badly structured in some way.
>>> 
>>> Includes won't help you much with #defines.  Coccinelle doesn't normally
>>> take them into account even when the definition is available.  If there
>>> are some non-Linux like macros in your code that reak parsing, you will
>>> need to update standard.h.  What you can do is run:
>>> 
>>> spatch --parse-c mysoftware
>>> 
>>> and then it will parse all the files in mysoftware and make a list of the
>>> tokens that most commonly cause errors.  It is usully sufficient to add a
>>> few definitions to standard.h to resolve most of the problems.
>>> 
>>> Coccinelle parses one top-evel unit (function etc) at a time.  When you
>>> use --verbose-parsing, it will print out the units that have parsing
>>> problems.  It puts BAD on the line with the error, and bad on the lines
>>> that are skipped due to the error.  When there are errors, Coccinelle does
>>> try up to 4 times to parse the code, with each time being a bit more
>>> relaxed.  So something can fail and then ultimately succeed.
>>> 
>>> I'll run parse-c on the above git link and see if any major issues emerge.
>>> 
>>> julia
>> 
>> When I tried parse-c I get the stack overflow (tired ulimit -s 50000 as well):
>> 
>> [galak at atx zephyr]$ spatch --macro-file-builtins $ZEPHYR_BASE/zephyr-std.h --parse-c $ZEPHYR_BASE
>> init_defs_builtins: /mnt/nvme/zephyr/zephyr-std.h
>> checking stack size (do ulimit -s 50000 if problem)
>> Fatal error: exception Stack overflow
> 
> ulimit -s unlimited should make it work
> 
> julia

thanks

- k

[Cocci] Help with include coccinelle & include paths?

[Kumar Gala]

> On Jan 26, 2017, at 9:19 AM, Julia Lawall <julia.lawall@lip6.fr> wrote:
> 
> 
> 
> On Thu, 26 Jan 2017, Kumar Gala wrote:
> 
>> Hi,
>> 
>> I?m looking at using coccinelle with the Zephyr project:
>> 
>> https://www.zephyrproject.org/
>> https://gerrit.zephyrproject.org/r/gitweb?p=zephyr.git;a=summary
>> 
>> We utilize Kconfig & Kbuild from linux kernel.  However, I?m trying to figure out how to deal with our includes.  I?ve played around with the various options to spatch, but either get a StackOverflow --recursive-includes or not getting all cases figured out since typedef & #defines aren?t getting parsed fully.
>> 
>> Wondering what are best practices or things to try here.  I?m also not clear on what the messages one gets when enable some of the debug options like --verbose-parsing --type-error-msg.
> 
> If you just want the includes for parsing and type inference, but not for
> transformation, it may help to use --include-headers-for-types.  I'm a bit
> surprised about the stack overflow, though.  Maybe there is just some
> header file that is badly structured in some way.
> 
> Includes won't help you much with #defines.  Coccinelle doesn't normally
> take them into account even when the definition is available.  If there
> are some non-Linux like macros in your code that reak parsing, you will
> need to update standard.h.  What you can do is run:
> 
> spatch --parse-c mysoftware
> 
> and then it will parse all the files in mysoftware and make a list of the
> tokens that most commonly cause errors.  It is usully sufficient to add a
> few definitions to standard.h to resolve most of the problems.
> 
> Coccinelle parses one top-evel unit (function etc) at a time.  When you
> use --verbose-parsing, it will print out the units that have parsing
> problems.  It puts BAD on the line with the error, and bad on the lines
> that are skipped due to the error.  When there are errors, Coccinelle does
> try up to 4 times to parse the code, with each time being a bit more
> relaxed.  So something can fail and then ultimately succeed.
> 
> I'll run parse-c on the above git link and see if any major issues emerge.
> 
> julia

So this I?m trying to use a modified version of Johann?s script to convert printf formatters that are passed a {u}int32_t to use ?%l{d,x,u,?}?.  My version of the script is attached.  I?m trying to figure out why some typedef / structs are being processed/figured out.

I do:
~/git/coccinelle/spatch.opt --sp-file ./pri.sp --macro-file-builtins $ZEPHYR_BASE/zephyr-std.h tests/legacy/kernel/test_mail/src/

I?d expect to match line 206/207 of tests/legacy/kernel/test_mail/src/mail.c, where MSTmsg is of type ?struct k_msg? and k_msg is defined in include/legacy.h.  Where the size field is a uint32_t.

So trying to figure out why the struct is getting resolved and match.

Thanks for all the help!!!

- k

-------------- next part --------------
A non-text attachment was scrubbed...
Name: pri.sp
Type: application/octet-stream
Size: 4317 bytes
Desc: not available
URL: <https://systeme.lip6.fr/pipermail/cocci/attachments/20170126/71ccbae5/attachment.obj>

[Cocci] Help with include coccinelle & include paths?

[Julia Lawall]

> So this I?m trying to use a modified version of Johann?s script to convert printf formatters that are passed a {u}int32_t to use ?%l{d,x,u,?}?.  My version of the script is attached.  I?m trying to figure out why some typedef / structs are being processed/figured out.
>
> I do:
> ~/git/coccinelle/spatch.opt --sp-file ./pri.sp --macro-file-builtins $ZEPHYR_BASE/zephyr-std.h tests/legacy/kernel/test_mail/src/
>
> I?d expect to match line 206/207 of tests/legacy/kernel/test_mail/src/mail.c, where MSTmsg is of type ?struct k_msg? and k_msg is defined in include/legacy.h.  Where the size field is a uint32_t.
>
> So trying to figure out why the struct is getting resolved and match.

I'm not completely sure from your message whether it is working or not.
If you give the options --recursive-includes -I $ZEPHYR_BASE/include
then I think it should work.

If you give the name of a directory, Coccinelle will add dir/include to
the include path.  But since you are working on a subdirectory, you don't
benefit from that.

julia

[Cocci] Help with include coccinelle & include paths?

[Kumar Gala]

> On Jan 26, 2017, at 1:17 PM, Julia Lawall <julia.lawall@lip6.fr> wrote:
> 
>> So this I?m trying to use a modified version of Johann?s script to convert printf formatters that are passed a {u}int32_t to use ?%l{d,x,u,?}?.  My version of the script is attached.  I?m trying to figure out why some typedef / structs are being processed/figured out.
>> 
>> I do:
>> ~/git/coccinelle/spatch.opt --sp-file ./pri.sp --macro-file-builtins $ZEPHYR_BASE/zephyr-std.h tests/legacy/kernel/test_mail/src/
>> 
>> I?d expect to match line 206/207 of tests/legacy/kernel/test_mail/src/mail.c, where MSTmsg is of type ?struct k_msg? and k_msg is defined in include/legacy.h.  Where the size field is a uint32_t.
>> 
>> So trying to figure out why the struct is getting resolved and match.
> 
> I'm not completely sure from your message whether it is working or not.
> If you give the options --recursive-includes -I $ZEPHYR_BASE/include
> then I think it should work.
> 
> If you give the name of a directory, Coccinelle will add dir/include to
> the include path.  But since you are working on a subdirectory, you don't
> benefit from that.
> 
> julia

Sorry if I wasn?t clear.  Its partially working.  Some issue with file vs dir.  If I do:

This FAILS:

~/git/coccinelle/spatch.opt --sp-file ./pri.sp --recursive-includes  -I $ZEPHYR_BASE/include  tests/legacy/kernel/test_mail/src/

This PASSES:

~/git/coccinelle/spatch.opt --sp-file ./pri.sp --recursive-includes  -I $ZEPHYR_BASE/include  tests/legacy/kernel/test_mail/src/mail.c

This also PASSES:
~/git/coccinelle/spatch.opt --sp-file ./pri.sp --recursive-includes  -I $ZEPHYR_BASE/include  tests/legacy/kernel/test_mail/src/*.c

So not sure what exactly happens differently if a dir vs file.

- k

[Cocci] Help with include coccinelle & include paths?

[Julia Lawall]

On Thu, 26 Jan 2017, Kumar Gala wrote:

>
> > On Jan 26, 2017, at 1:17 PM, Julia Lawall <julia.lawall@lip6.fr> wrote:
> >
> >> So this I?m trying to use a modified version of Johann?s script to convert printf formatters that are passed a {u}int32_t to use ?%l{d,x,u,?}?.  My version of the script is attached.  I?m trying to figure out why some typedef / structs are being processed/figured out.
> >>
> >> I do:
> >> ~/git/coccinelle/spatch.opt --sp-file ./pri.sp --macro-file-builtins $ZEPHYR_BASE/zephyr-std.h tests/legacy/kernel/test_mail/src/
> >>
> >> I?d expect to match line 206/207 of tests/legacy/kernel/test_mail/src/mail.c, where MSTmsg is of type ?struct k_msg? and k_msg is defined in include/legacy.h.  Where the size field is a uint32_t.
> >>
> >> So trying to figure out why the struct is getting resolved and match.
> >
> > I'm not completely sure from your message whether it is working or not.
> > If you give the options --recursive-includes -I $ZEPHYR_BASE/include
> > then I think it should work.
> >
> > If you give the name of a directory, Coccinelle will add dir/include to
> > the include path.  But since you are working on a subdirectory, you don't
> > benefit from that.
> >
> > julia
>
> Sorry if I wasn?t clear.  Its partially working.  Some issue with file vs dir.  If I do:
>
> This FAILS:
>
> ~/git/coccinelle/spatch.opt --sp-file ./pri.sp --recursive-includes  -I $ZEPHYR_BASE/include  tests/legacy/kernel/test_mail/src/

I get a change on lines 206/207 with this command.  Actually, I have no
difference, except in the diff line, between the above and the mail.c
case.

julia

> This PASSES:
>
> ~/git/coccinelle/spatch.opt --sp-file ./pri.sp --recursive-includes  -I $ZEPHYR_BASE/include  tests/legacy/kernel/test_mail/src/mail.c
>
> This also PASSES:
> ~/git/coccinelle/spatch.opt --sp-file ./pri.sp --recursive-includes  -I $ZEPHYR_BASE/include  tests/legacy/kernel/test_mail/src/*.c
>
> So not sure what exactly happens differently if a dir vs file.
>
> - k
>
>
>

[Cocci] Help with include coccinelle & include paths?

[Kumar Gala]

> On Jan 26, 2017, at 1:57 PM, Julia Lawall <julia.lawall@lip6.fr> wrote:
> 
> 
> 
> On Thu, 26 Jan 2017, Kumar Gala wrote:
> 
>> 
>>> On Jan 26, 2017, at 1:17 PM, Julia Lawall <julia.lawall@lip6.fr> wrote:
>>> 
>>>> So this I?m trying to use a modified version of Johann?s script to convert printf formatters that are passed a {u}int32_t to use ?%l{d,x,u,?}?.  My version of the script is attached.  I?m trying to figure out why some typedef / structs are being processed/figured out.
>>>> 
>>>> I do:
>>>> ~/git/coccinelle/spatch.opt --sp-file ./pri.sp --macro-file-builtins $ZEPHYR_BASE/zephyr-std.h tests/legacy/kernel/test_mail/src/
>>>> 
>>>> I?d expect to match line 206/207 of tests/legacy/kernel/test_mail/src/mail.c, where MSTmsg is of type ?struct k_msg? and k_msg is defined in include/legacy.h.  Where the size field is a uint32_t.
>>>> 
>>>> So trying to figure out why the struct is getting resolved and match.
>>> 
>>> I'm not completely sure from your message whether it is working or not.
>>> If you give the options --recursive-includes -I $ZEPHYR_BASE/include
>>> then I think it should work.
>>> 
>>> If you give the name of a directory, Coccinelle will add dir/include to
>>> the include path.  But since you are working on a subdirectory, you don't
>>> benefit from that.
>>> 
>>> julia
>> 
>> Sorry if I wasn?t clear.  Its partially working.  Some issue with file vs dir.  If I do:
>> 
>> This FAILS:
>> 
>> ~/git/coccinelle/spatch.opt --sp-file ./pri.sp --recursive-includes  -I $ZEPHYR_BASE/include  tests/legacy/kernel/test_mail/src/
> 
> I get a change on lines 206/207 with this command.  Actually, I have no
> difference, except in the diff line, between the above and the mail.c
> case.

Interesting, I?m seeing different behavior, if I --cache-limit 0 that seems to help some in the dir case.

> 
> julia
> 
>> This PASSES:
>> 
>> ~/git/coccinelle/spatch.opt --sp-file ./pri.sp --recursive-includes  -I $ZEPHYR_BASE/include  tests/legacy/kernel/test_mail/src/mail.c
>> 
>> This also PASSES:
>> ~/git/coccinelle/spatch.opt --sp-file ./pri.sp --recursive-includes  -I $ZEPHYR_BASE/include  tests/legacy/kernel/test_mail/src/*.c
>> 
>> So not sure what exactly happens differently if a dir vs file.
>> 
>> - k
>> 
>> 

[Cocci] Help with include coccinelle & include paths?

[Kumar Gala]

> On Jan 26, 2017, at 2:01 PM, Kumar Gala <kumar.gala@linaro.org> wrote:
> 
>> 
>> On Jan 26, 2017, at 1:57 PM, Julia Lawall <julia.lawall@lip6.fr> wrote:
>> 
>> 
>> 
>> On Thu, 26 Jan 2017, Kumar Gala wrote:
>> 
>>> 
>>>> On Jan 26, 2017, at 1:17 PM, Julia Lawall <julia.lawall@lip6.fr> wrote:
>>>> 
>>>>> So this I?m trying to use a modified version of Johann?s script to convert printf formatters that are passed a {u}int32_t to use ?%l{d,x,u,?}?.  My version of the script is attached.  I?m trying to figure out why some typedef / structs are being processed/figured out.
>>>>> 
>>>>> I do:
>>>>> ~/git/coccinelle/spatch.opt --sp-file ./pri.sp --macro-file-builtins $ZEPHYR_BASE/zephyr-std.h tests/legacy/kernel/test_mail/src/
>>>>> 
>>>>> I?d expect to match line 206/207 of tests/legacy/kernel/test_mail/src/mail.c, where MSTmsg is of type ?struct k_msg? and k_msg is defined in include/legacy.h.  Where the size field is a uint32_t.
>>>>> 
>>>>> So trying to figure out why the struct is getting resolved and match.
>>>> 
>>>> I'm not completely sure from your message whether it is working or not.
>>>> If you give the options --recursive-includes -I $ZEPHYR_BASE/include
>>>> then I think it should work.
>>>> 
>>>> If you give the name of a directory, Coccinelle will add dir/include to
>>>> the include path.  But since you are working on a subdirectory, you don't
>>>> benefit from that.
>>>> 
>>>> julia
>>> 
>>> Sorry if I wasn?t clear.  Its partially working.  Some issue with file vs dir.  If I do:
>>> 
>>> This FAILS:
>>> 
>>> ~/git/coccinelle/spatch.opt --sp-file ./pri.sp --recursive-includes  -I $ZEPHYR_BASE/include tests/legacy/kernel/test_mail/src/
>> 
>> I get a change on lines 206/207 with this command.  Actually, I have no
>> difference, except in the diff line, between the above and the mail.c
>> case.
> 
> Interesting, I?m seeing different behavior, if I --cache-limit 0 that seems to help some in the dir case.

With ?cache-limit 0, I only get a subset of the fixes I expect to mail.c

Not sure if there?s some debug info I can dump that would be helpful.

- k

[Cocci] Help with include coccinelle & include paths?

[Julia Lawall]

> Not sure if there?s some debug info I can dump that would be helpful.

--debug --show-trying

thanks,
julia

[Cocci] Help with include coccinelle & include paths?

[Kumar Gala]

> On Jan 26, 2017, at 2:19 PM, Julia Lawall <julia.lawall@lip6.fr> wrote:
> 
>> Not sure if there?s some debug info I can dump that would be helpful.
> 
> --debug --show-trying
> 
> thanks,
> julia

For the FILE case:
~/git/coccinelle/spatch.opt --sp-file ./pri.sp --macro-file-builtins $ZEPHYR_BASE/zephyr-std.h --recursive-includes -I $ZEPHYR_BASE/include tests/legacy/kernel/test_mail/src/mail.^C--debug --show-trying 2> file.good.log 1> file.good.out

For the DIR case:

~/git/coccinelle/spatch.opt --sp-file ./pri.sp --macro-file-builtins $ZEPHYR_BASE/zephyr-std.h --recursive-includes -I $ZEPHYR_BASE/include tests/legacy/kernel/test_mail/src/ --debug --show-trying 2> dir.bad.log 1>dir.bad.out

Not sure if these logs show something useful.

- k

-------------- next part --------------
A non-text attachment was scrubbed...
Name: dir.bad.log
Type: application/octet-stream
Size: 172102 bytes
Desc: not available
URL: <https://systeme.lip6.fr/pipermail/cocci/attachments/20170126/bd825ed5/attachment-0004.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dir.bad.out
Type: application/octet-stream
Size: 6622 bytes
Desc: not available
URL: <https://systeme.lip6.fr/pipermail/cocci/attachments/20170126/bd825ed5/attachment-0005.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: file.good.log
Type: application/octet-stream
Size: 1376314 bytes
Desc: not available
URL: <https://systeme.lip6.fr/pipermail/cocci/attachments/20170126/bd825ed5/attachment-0006.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: file.good.out
Type: application/octet-stream
Size: 39623 bytes
Desc: not available
URL: <https://systeme.lip6.fr/pipermail/cocci/attachments/20170126/bd825ed5/attachment-0007.obj>

[Cocci] Help with include coccinelle & include paths?

[Julia Lawall]

On Thu, 26 Jan 2017, Kumar Gala wrote:

>
> > On Jan 26, 2017, at 2:19 PM, Julia Lawall <julia.lawall@lip6.fr> wrote:
> >
> >> Not sure if there?s some debug info I can dump that would be helpful.
> >
> > --debug --show-trying
> >
> > thanks,
> > julia
>
> For the FILE case:
> ~/git/coccinelle/spatch.opt --sp-file ./pri.sp --macro-file-builtins $ZEPHYR_BASE/zephyr-std.h --recursive-includes -I $ZEPHYR_BASE/include tests/legacy/kernel/test_mail/src/mail.^C--debug --show-trying 2> file.good.log 1> file.good.out
>
> For the DIR case:
>
> ~/git/coccinelle/spatch.opt --sp-file ./pri.sp --macro-file-builtins $ZEPHYR_BASE/zephyr-std.h --recursive-includes -I $ZEPHYR_BASE/include tests/legacy/kernel/test_mail/src/ --debug --show-trying 2> dir.bad.log 1>dir.bad.out
>
> Not sure if these logs show something useful.

We aren't processing the files in the same order.  Your .log file in the
dir case shows main.c and the mail.c, while mine has mail.c and then
main.c.  I would guess that it is learning something from the parsing of
main.c that is throwing it off in the parsing of mail.c.

Could you add --verbose-parsing somewhere after --debug in the above
command lines and merge standard output and standard error into a single
file?

thanks,
julia

[Cocci] Help with include coccinelle & include paths?

[Kumar Gala]

> On Jan 26, 2017, at 2:54 PM, Julia Lawall <julia.lawall@lip6.fr> wrote:
> 
> 
> 
> On Thu, 26 Jan 2017, Kumar Gala wrote:
> 
>> 
>>> On Jan 26, 2017, at 2:19 PM, Julia Lawall <julia.lawall@lip6.fr> wrote:
>>> 
>>>> Not sure if there?s some debug info I can dump that would be helpful.
>>> 
>>> --debug --show-trying
>>> 
>>> thanks,
>>> julia
>> 
>> For the FILE case:
>> ~/git/coccinelle/spatch.opt --sp-file ./pri.sp --macro-file-builtins $ZEPHYR_BASE/zephyr-std.h --recursive-includes -I $ZEPHYR_BASE/include tests/legacy/kernel/test_mail/src/mail.^C--debug --show-trying 2> file.good.log 1> file.good.out
>> 
>> For the DIR case:
>> 
>> ~/git/coccinelle/spatch.opt --sp-file ./pri.sp --macro-file-builtins $ZEPHYR_BASE/zephyr-std.h --recursive-includes -I $ZEPHYR_BASE/include tests/legacy/kernel/test_mail/src/ --debug --show-trying 2> dir.bad.log 1>dir.bad.out
>> 
>> Not sure if these logs show something useful.
> 
> We aren't processing the files in the same order.  Your .log file in the
> dir case shows main.c and the mail.c, while mine has mail.c and then
> main.c.  I would guess that it is learning something from the parsing of
> main.c that is throwing it off in the parsing of mail.c.
> 
> Could you add --verbose-parsing somewhere after --debug in the above
> command lines and merge standard output and standard error into a single
> file?
> 
> thanks,
> julia

 ~/git/coccinelle/spatch.opt --sp-file ./pri.sp --macro-file-builtins $ZEPHYR_BASE/zephyr-std.h --recursive-includes -I $ZEPHYR_BASE/include tests/legacy/kernel/test_mail/src/ --debug --show-trying --verbose-parsing &> dir.log

[Cocci] Help with include coccinelle & include paths?

[Kumar Gala]

> On Jan 26, 2017, at 2:54 PM, Julia Lawall <julia.lawall@lip6.fr> wrote:
> 
> 
> 
> On Thu, 26 Jan 2017, Kumar Gala wrote:
> 
>> 
>>> On Jan 26, 2017, at 2:19 PM, Julia Lawall <julia.lawall@lip6.fr> wrote:
>>> 
>>>> Not sure if there?s some debug info I can dump that would be helpful.
>>> 
>>> --debug --show-trying
>>> 
>>> thanks,
>>> julia
>> 
>> For the FILE case:
>> ~/git/coccinelle/spatch.opt --sp-file ./pri.sp --macro-file-builtins $ZEPHYR_BASE/zephyr-std.h --recursive-includes -I $ZEPHYR_BASE/include tests/legacy/kernel/test_mail/src/mail.^C--debug --show-trying 2> file.good.log 1> file.good.out
>> 
>> For the DIR case:
>> 
>> ~/git/coccinelle/spatch.opt --sp-file ./pri.sp --macro-file-builtins $ZEPHYR_BASE/zephyr-std.h --recursive-includes -I $ZEPHYR_BASE/include tests/legacy/kernel/test_mail/src/ --debug --show-trying 2> dir.bad.log 1>dir.bad.out
>> 
>> Not sure if these logs show something useful.
> 
> We aren't processing the files in the same order.  Your .log file in the
> dir case shows main.c and the mail.c, while mine has mail.c and then
> main.c.  I would guess that it is learning something from the parsing of
> main.c that is throwing it off in the parsing of mail.c.
> 
> Could you add --verbose-parsing somewhere after --debug in the above
> command lines and merge standard output and standard error into a single
> file?
> 
> thanks,
> julia

This time with log:

 ~/git/coccinelle/spatch.opt --sp-file ./pri.sp --macro-file-builtins $ZEPHYR_BASE/zephyr-std.h --recursive-includes -I $ZEPHYR_BASE/include tests/legacy/kernel/test_mail/src/ --debug --show-trying --verbose-parsing &> dir.log

- k
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dir.log
Type: application/octet-stream
Size: 213930 bytes
Desc: not available
URL: <https://systeme.lip6.fr/pipermail/cocci/attachments/20170126/4b4ed79e/attachment-0001.obj>
-------------- next part --------------

[Cocci] Help with include coccinelle & include paths?

[Julia Lawall]

On Thu, 26 Jan 2017, Kumar Gala wrote:

>
> > On Jan 26, 2017, at 2:54 PM, Julia Lawall <julia.lawall@lip6.fr> wrote:
> >
> >
> >
> > On Thu, 26 Jan 2017, Kumar Gala wrote:
> >
> >>
> >>> On Jan 26, 2017, at 2:19 PM, Julia Lawall <julia.lawall@lip6.fr> wrote:
> >>>
> >>>> Not sure if there?s some debug info I can dump that would be helpful.
> >>>
> >>> --debug --show-trying
> >>>
> >>> thanks,
> >>> julia
> >>
> >> For the FILE case:
> >> ~/git/coccinelle/spatch.opt --sp-file ./pri.sp --macro-file-builtins $ZEPHYR_BASE/zephyr-std.h --recursive-includes -I $ZEPHYR_BASE/include tests/legacy/kernel/test_mail/src/mail.^C--debug --show-trying 2> file.good.log 1> file.good.out
> >>
> >> For the DIR case:
> >>
> >> ~/git/coccinelle/spatch.opt --sp-file ./pri.sp --macro-file-builtins $ZEPHYR_BASE/zephyr-std.h --recursive-includes -I $ZEPHYR_BASE/include tests/legacy/kernel/test_mail/src/ --debug --show-trying 2> dir.bad.log 1>dir.bad.out
> >>
> >> Not sure if these logs show something useful.
> >
> > We aren't processing the files in the same order.  Your .log file in the
> > dir case shows main.c and the mail.c, while mine has mail.c and then
> > main.c.  I would guess that it is learning something from the parsing of
> > main.c that is throwing it off in the parsing of mail.c.
> >
> > Could you add --verbose-parsing somewhere after --debug in the above
> > command lines and merge standard output and standard error into a single
> > file?
> >
> > thanks,
> > julia
>
> This time with log:
>
>  ~/git/coccinelle/spatch.opt --sp-file ./pri.sp --macro-file-builtins $ZEPHYR_BASE/zephyr-std.h --recursive-includes -I $ZEPHYR_BASE/include tests/legacy/kernel/test_mail/src/ --debug --show-trying --verbose-parsing &> dir.log

The inclusion of header files that have already been included in another
.c file does not seem to be working properly.  I will have to look more
into it tomorrow.

julia

[Cocci] Help with include coccinelle & include paths?

[Kumar Gala]

> On Jan 26, 2017, at 3:37 PM, Julia Lawall <julia.lawall@lip6.fr> wrote:
> 
> 
> 
> On Thu, 26 Jan 2017, Kumar Gala wrote:
> 
>> 
>>> On Jan 26, 2017, at 2:54 PM, Julia Lawall <julia.lawall@lip6.fr> wrote:
>>> 
>>> 
>>> 
>>> On Thu, 26 Jan 2017, Kumar Gala wrote:
>>> 
>>>> 
>>>>> On Jan 26, 2017, at 2:19 PM, Julia Lawall <julia.lawall@lip6.fr> wrote:
>>>>> 
>>>>>> Not sure if there?s some debug info I can dump that would be helpful.
>>>>> 
>>>>> --debug --show-trying
>>>>> 
>>>>> thanks,
>>>>> julia
>>>> 
>>>> For the FILE case:
>>>> ~/git/coccinelle/spatch.opt --sp-file ./pri.sp --macro-file-builtins $ZEPHYR_BASE/zephyr-std.h --recursive-includes -I $ZEPHYR_BASE/include tests/legacy/kernel/test_mail/src/mail.^C--debug --show-trying 2> file.good.log 1> file.good.out
>>>> 
>>>> For the DIR case:
>>>> 
>>>> ~/git/coccinelle/spatch.opt --sp-file ./pri.sp --macro-file-builtins $ZEPHYR_BASE/zephyr-std.h --recursive-includes -I $ZEPHYR_BASE/include tests/legacy/kernel/test_mail/src/ --debug --show-trying 2> dir.bad.log 1>dir.bad.out
>>>> 
>>>> Not sure if these logs show something useful.
>>> 
>>> We aren't processing the files in the same order.  Your .log file in the
>>> dir case shows main.c and the mail.c, while mine has mail.c and then
>>> main.c.  I would guess that it is learning something from the parsing of
>>> main.c that is throwing it off in the parsing of mail.c.
>>> 
>>> Could you add --verbose-parsing somewhere after --debug in the above
>>> command lines and merge standard output and standard error into a single
>>> file?
>>> 
>>> thanks,
>>> julia
>> 
>> This time with log:
>> 
>> ~/git/coccinelle/spatch.opt --sp-file ./pri.sp --macro-file-builtins $ZEPHYR_BASE/zephyr-std.h --recursive-includes -I $ZEPHYR_BASE/include tests/legacy/kernel/test_mail/src/ --debug --show-trying --verbose-parsing &> dir.log
> 
> The inclusion of header files that have already been included in another
> .c file does not seem to be working properly.  I will have to look more
> into it tomorrow.
> 
> julia

Gotcha, thanks for help debugging this.

- k

[Cocci] Help with include coccinelle & include paths?

[Julia Lawall]

I have undone a rather nice optimization to get the header files to be
reprocessed each time they are included by a .c file.  The problem is that
parsing a header file has some side effects, eg finding about typedefs,
etc.  Unfortunately, we were caching only the AST of the header file, but
not either the ASTs of the header files it includes or the effects that
parsing the header files could have.

Now I have undone the optimization, and hopefully it will work better,
although more slowly.

Available on github.

julia

[Cocci] Help with include coccinelle & include paths?

[Kumar Gala]

> On Jan 27, 2017, at 11:30 AM, Julia Lawall <julia.lawall@lip6.fr> wrote:
> 
> I have undone a rather nice optimization to get the header files to be
> reprocessed each time they are included by a .c file.  The problem is that
> parsing a header file has some side effects, eg finding about typedefs,
> etc.  Unfortunately, we were caching only the AST of the header file, but
> not either the ASTs of the header files it includes or the effects that
> parsing the header files could have.
> 
> Now I have undone the optimization, and hopefully it will work better,
> although more slowly.
> 
> Available on github.
> 
> julia

Thanks, that seems to help a lot.

I was wondering what the following message means, the bit about the header file:

HANDLING: tests/bluetooth/test_bluetooth/src/bluetooth.c
a header file was modified: bluetooth.h

- k