[Cocci] Checking build dependencies for "pyml"

[Cocci] Checking build dependencies for "pyml"

[SF Markus Elfring]

Hello,

I have tried to generate the current software also on my openSUSE Tumbleweed system.


elfring at Sonne:~/Projekte/Coccinelle/20160205> ./configure --enable-release --enable-ocaml && LANG=C make
?
make[2]: *** @MAKE_pyml@: No such file or directory.  Stop.
?


Unfortunately, I stumble on an error message.

How is the development status after the commit "Fixed: Bundles compilation
and pyml update" on 2016-06-26?
https://github.com/coccinelle/coccinelle/commit/62f2df8c7d35967a1eca4dcd487d59a9918dd5aa

Regards,
Markus

[Cocci] Checking build dependencies for "pyml"

[Thierry Martinez]

Hello, Markus.

Markus:
> make[2]: *** @MAKE_pyml@: No such file or directory.  Stop.

Thank you for the report. Does this error continue to happen if you
delete the subdirectory autom4te.cache/ and then execute ./autogen,
./configure, etc.?

-- 
Thierry.

SF Markus Elfring (2016/07/07, 23:40 MSK)?:
> Hello,
>
> I have tried to generate the current software also on my openSUSE Tumbleweed system.
>
>
> elfring at Sonne:~/Projekte/Coccinelle/20160205> ./configure --enable-release --enable-ocaml && LANG=C make
> ?
> make[2]: *** @MAKE_pyml@: No such file or directory.  Stop.
> ?
>
>
> Unfortunately, I stumble on an error message.
>
> How is the development status after the commit "Fixed: Bundles compilation
> and pyml update" on 2016-06-26?
> https://github.com/coccinelle/coccinelle/commit/62f2df8c7d35967a1eca4dcd487d59a9918dd5aa
>
> Regards,
> Markus
> _______________________________________________
> Cocci mailing list
> Cocci at systeme.lip6.fr
> https://systeme.lip6.fr/mailman/listinfo/cocci

[Cocci] Checking build dependencies for "pyml" (and Menhir)

[SF Markus Elfring]

>> make[2]: *** @MAKE_pyml@: No such file or directory.  Stop.
> 
> Thank you for the report. Does this error continue to happen if you
> delete the subdirectory autom4te.cache/ and then execute ./autogen,
> ./configure, etc.?

Did a configuration script become outdated?
-rwxr-xr-x 1 elfring users 524178 Mar 15 15:30 configure


It seems that the desired software build can be continued after another update.

elfring at Sonne:~/Projekte/Coccinelle/20160205> ./autogen && ./configure && LANG=C make
?
/usr/bin/ocamlc.opt -unsafe -I ../commons -I ../commons/ocamlextra -I ../globals -I /home/elfring/Projekte/Coccinelle/20160205/bundles/menhirLib/ -c parser_cocci_menhir.mli
File "parser_cocci_menhir.mli", line 262, characters 10-56:
Error: Unbound module MenhirLib.IncrementalEngine
make[6]: *** [Makefile:89: parser_cocci_menhir.cmi] Error 2
?


How should such a build dependency be resolved?

Regards,
Markus

[Cocci] Checking build dependencies for "pyml" (and Menhir)

[Thierry Martinez]

Markus:
> Did a configuration script become outdated?
> -rwxr-xr-x 1 elfring users 524178 Mar 15 15:30 configure

Yes, the configure.ac script has been changed to handle the change from
Pycaml to Py.ml.

> /usr/bin/ocamlc.opt -unsafe -I ../commons -I ../commons/ocamlextra -I ../globals -I /home/elfring/Projekte/Coccinelle/20160205/bundles/menhirLib/ -c parser_cocci_menhir.mli
> File "parser_cocci_menhir.mli", line 262, characters 10-56:
> Error: Unbound module MenhirLib.IncrementalEngine

Your parser_cocci_menhir.{ml,mli} have been generated with a version of
Menhir more recent than the one that is bundled by Coccinelle.

You can just safely remove parsing_cocci/parser_cocci_menhir.{ml,mli}
files (or just run make distclean) and try to run make again.

-- 
Thierry.

SF Markus Elfring (2016/07/08, 08:21 MSK)?:
>>> make[2]: *** @MAKE_pyml@: No such file or directory.  Stop.
>> 
>> Thank you for the report. Does this error continue to happen if you
>> delete the subdirectory autom4te.cache/ and then execute ./autogen,
>> ./configure, etc.?
>
> Did a configuration script become outdated?
> -rwxr-xr-x 1 elfring users 524178 Mar 15 15:30 configure
>
>
> It seems that the desired software build can be continued after another update.
>
> elfring at Sonne:~/Projekte/Coccinelle/20160205> ./autogen && ./configure && LANG=C make
> ?
> /usr/bin/ocamlc.opt -unsafe -I ../commons -I ../commons/ocamlextra -I ../globals -I /home/elfring/Projekte/Coccinelle/20160205/bundles/menhirLib/ -c parser_cocci_menhir.mli
> File "parser_cocci_menhir.mli", line 262, characters 10-56:
> Error: Unbound module MenhirLib.IncrementalEngine
> make[6]: *** [Makefile:89: parser_cocci_menhir.cmi] Error 2
> ?
>
>
> How should such a build dependency be resolved?
>
> Regards,
> Markus

[Cocci] Checking build dependencies for "pyml" (and Menhir)

[Julia Lawall]

On Fri, 8 Jul 2016, Thierry Martinez wrote:

> Markus:
> > Did a configuration script become outdated?
> > -rwxr-xr-x 1 elfring users 524178 Mar 15 15:30 configure
>
> Yes, the configure.ac script has been changed to handle the change from
> Pycaml to Py.ml.
>
> > /usr/bin/ocamlc.opt -unsafe -I ../commons -I ../commons/ocamlextra -I ../globals -I /home/elfring/Projekte/Coccinelle/20160205/bundles/menhirLib/ -c parser_cocci_menhir.mli
> > File "parser_cocci_menhir.mli", line 262, characters 10-56:
> > Error: Unbound module MenhirLib.IncrementalEngine
>
> Your parser_cocci_menhir.{ml,mli} have been generated with a version of
> Menhir more recent than the one that is bundled by Coccinelle.
>
> You can just safely remove parsing_cocci/parser_cocci_menhir.{ml,mli}
> files (or just run make distclean) and try to run make again.

Probably it will be necessary to run autogen and configure again as well.

julia

>
> --
> Thierry.
>
> SF Markus Elfring (2016/07/08, 08:21 MSK)?:
> >>> make[2]: *** @MAKE_pyml@: No such file or directory.  Stop.
> >>
> >> Thank you for the report. Does this error continue to happen if you
> >> delete the subdirectory autom4te.cache/ and then execute ./autogen,
> >> ./configure, etc.?
> >
> > Did a configuration script become outdated?
> > -rwxr-xr-x 1 elfring users 524178 Mar 15 15:30 configure
> >
> >
> > It seems that the desired software build can be continued after another update.
> >
> > elfring at Sonne:~/Projekte/Coccinelle/20160205> ./autogen && ./configure && LANG=C make
> > ?
> > /usr/bin/ocamlc.opt -unsafe -I ../commons -I ../commons/ocamlextra -I ../globals -I /home/elfring/Projekte/Coccinelle/20160205/bundles/menhirLib/ -c parser_cocci_menhir.mli
> > File "parser_cocci_menhir.mli", line 262, characters 10-56:
> > Error: Unbound module MenhirLib.IncrementalEngine
> > make[6]: *** [Makefile:89: parser_cocci_menhir.cmi] Error 2
> > ?
> >
> >
> > How should such a build dependency be resolved?
> >
> > Regards,
> > Markus
>
> _______________________________________________
> Cocci mailing list
> Cocci at systeme.lip6.fr
> https://systeme.lip6.fr/mailman/listinfo/cocci
>

[Cocci] Checking build dependencies for "pyml" (and Menhir)

[Thierry Martinez]

Julia:
> Probably it will be necessary to run autogen and configure again as well.

In the case of make distclean, yes.
If the files parsing_cocci/parser_cocci_menhir.{ml,mli} are just removed, it
should be fine to run make with the current Makefile.

-- 
Thierry.

Julia Lawall (2016/07/08, 13:12 MSK)?:
> On Fri, 8 Jul 2016, Thierry Martinez wrote:
>
>> Markus:
>> > Did a configuration script become outdated?
>> > -rwxr-xr-x 1 elfring users 524178 Mar 15 15:30 configure
>>
>> Yes, the configure.ac script has been changed to handle the change from
>> Pycaml to Py.ml.
>>
>> > /usr/bin/ocamlc.opt -unsafe -I ../commons -I ../commons/ocamlextra -I ../globals -I /home/elfring/Projekte/Coccinelle/20160205/bundles/menhirLib/ -c parser_cocci_menhir.mli
>> > File "parser_cocci_menhir.mli", line 262, characters 10-56:
>> > Error: Unbound module MenhirLib.IncrementalEngine
>>
>> Your parser_cocci_menhir.{ml,mli} have been generated with a version of
>> Menhir more recent than the one that is bundled by Coccinelle.
>>
>> You can just safely remove parsing_cocci/parser_cocci_menhir.{ml,mli}
>> files (or just run make distclean) and try to run make again.
>
> Probably it will be necessary to run autogen and configure again as well.
>
> julia
>
>>
>> --
>> Thierry.
>>
>> SF Markus Elfring (2016/07/08, 08:21 MSK):
>> >>> make[2]: *** @MAKE_pyml@: No such file or directory.  Stop.
>> >>
>> >> Thank you for the report. Does this error continue to happen if you
>> >> delete the subdirectory autom4te.cache/ and then execute ./autogen,
>> >> ./configure, etc.?
>> >
>> > Did a configuration script become outdated?
>> > -rwxr-xr-x 1 elfring users 524178 Mar 15 15:30 configure
>> >
>> >
>> > It seems that the desired software build can be continued after another update.
>> >
>> > elfring at Sonne:~/Projekte/Coccinelle/20160205> ./autogen && ./configure && LANG=C make
>> > ?
>> > /usr/bin/ocamlc.opt -unsafe -I ../commons -I ../commons/ocamlextra -I ../globals -I /home/elfring/Projekte/Coccinelle/20160205/bundles/menhirLib/ -c parser_cocci_menhir.mli
>> > File "parser_cocci_menhir.mli", line 262, characters 10-56:
>> > Error: Unbound module MenhirLib.IncrementalEngine
>> > make[6]: *** [Makefile:89: parser_cocci_menhir.cmi] Error 2
>> > ?
>> >
>> >
>> > How should such a build dependency be resolved?
>> >
>> > Regards,
>> > Markus
>>
>> _______________________________________________
>> Cocci mailing list
>> Cocci at systeme.lip6.fr
>> https://systeme.lip6.fr/mailman/listinfo/cocci
>>

[Cocci] Checking build dependencies for "pyml"

[SF Markus Elfring]

> Yes, the configure.ac script has been changed to handle the change from
> Pycaml to Py.ml.

Has such a software evolution got any influence on the Python application
programming interface by the semantic patch language?

Will the mapping (and corresponding data type conversion) will need
another look for integer values in this use case?

Regards,
Markus

[Cocci] Checking build dependencies for "Menhir"

[SF Markus Elfring]

> > /usr/bin/ocamlc.opt -unsafe -I ../commons -I ../commons/ocamlextra -I ../globals -I /home/elfring/Projekte/Coccinelle/20160205/bundles/menhirLib/ -c parser_cocci_menhir.mli
> > File "parser_cocci_menhir.mli", line 262, characters 10-56:
> > Error: Unbound module MenhirLib.IncrementalEngine
> 
> Your parser_cocci_menhir.{ml,mli} have been generated with a version of
> Menhir more recent than the one that is bundled by Coccinelle.

Thanks for your explanation.

Is the bundled Menhir version currently required for a working Coccinelle software?


> You can just safely remove parsing_cocci/parser_cocci_menhir.{ml,mli}
> files (or just run make distclean) and try to run make again.

I imagine that there are further challenges to consider for the safe
dependency resolution of involved libraries.
* Which releases can be picked up and finally used from special directories?
* Will any more fine-tuning be needed for corresponding directory selection?

Regards,
Markus

[Cocci] Checking build dependencies for "pyml"

[Thierry Martinez]

Markus:
> Has such a software evolution got any influence on the Python application
> programming interface by the semantic patch language?

The same ./spatch executable can now be run in a Python 2 and Python 3
environment (before, with Pycaml, Coccinelle had to be compiled in a
Python 2 environment to work with Python 2, and to be compiled in a
Python 3 environment to work with Python 3).

Python error handling is better as well, and the Python interface is now
only initialized when needed.

> Will the mapping (and corresponding data type conversion) will need
> another look for integer values in this use case?

The mapping is unchanged. But the better error handling should help to
debug scripts.

-- 
Thierry.

SF Markus Elfring (2016/07/08, 14:14 MSK)?:
>> Yes, the configure.ac script has been changed to handle the change from
>> Pycaml to Py.ml.
>
> Has such a software evolution got any influence on the Python application
> programming interface by the semantic patch language?
>
> Will the mapping (and corresponding data type conversion) will need
> another look for integer values in this use case?
>
> Regards,
> Markus

[Cocci] Checking build dependencies for "Menhir"

[Thierry Martinez]

Markus:
> Is the bundled Menhir version currently required for a working
> Coccinelle software?

No, you can use a more recent version of Menhir if you want.

The reason of having an old version of Menhir in the bundles is that the
newer versions of Menhir require OCaml 4.02 (for "match exception"), and
we would like to keep the compatibility with OCaml 4.01 (or even 3.12).

> I imagine that there are further challenges to consider for the safe
> dependency resolution of involved libraries.
> * Which releases can be picked up and finally used from special directories?
> * Will any more fine-tuning be needed for corresponding directory
> selection?

I don?t know if dependencies can be made more precise between the
Coccinelle code and bundled libraries.

In the current settings, you may have to run "make clean" or even "make
distclean" after a "git pull" to get a successful build (or else, there
is effectively a bug).  I think it is not too demanding, but I agree
that we should write more clearly in the commit message when cleaning is
required.

-- 
Thierry.

SF Markus Elfring (2016/07/08, 14:32 MSK)?:
>> > /usr/bin/ocamlc.opt -unsafe -I ../commons -I ../commons/ocamlextra -I ../globals -I /home/elfring/Projekte/Coccinelle/20160205/bundles/menhirLib/ -c parser_cocci_menhir.mli
>> > File "parser_cocci_menhir.mli", line 262, characters 10-56:
>> > Error: Unbound module MenhirLib.IncrementalEngine
>> 
>> Your parser_cocci_menhir.{ml,mli} have been generated with a version of
>> Menhir more recent than the one that is bundled by Coccinelle.
>
> Thanks for your explanation.
>
> Is the bundled Menhir version currently required for a working Coccinelle software?
>
>
>> You can just safely remove parsing_cocci/parser_cocci_menhir.{ml,mli}
>> files (or just run make distclean) and try to run make again.
>
> I imagine that there are further challenges to consider for the safe
> dependency resolution of involved libraries.
> * Which releases can be picked up and finally used from special directories?
> * Will any more fine-tuning be needed for corresponding directory selection?
>
> Regards,
> Markus

[Cocci] Checking build dependencies for "pyml"

[SF Markus Elfring]

> > Will the mapping (and corresponding data type conversion) will need
> > another look for integer values in this use case?
> 
> The mapping is unchanged.

How do you think about to achieve that integers will be preserved
(without conversion to strings) here?

Regards,
Markus

[Cocci] Checking build dependencies for "pyml"

[Julia Lawall]

On Fri, 8 Jul 2016, SF Markus Elfring wrote:

> > > Will the mapping (and corresponding data type conversion) will need
> > > another look for integer values in this use case?
> >
> > The mapping is unchanged.
>
> How do you think about to achieve that integers will be preserved
> (without conversion to strings) here?

Could you explain what exactly you are talking about, perhaps with a
small example?  There are things like line numbers that are converted to
strings.  There is no technical reason for that, but the person who first
implemented the python interface did that, and then it didn't seem like
big enough of a deal to break backward compatability.  But maybe you are
referring to somrething else.

julia

[Cocci] Checking build dependencies for "Menhir"

[SF Markus Elfring]

> > Is the bundled Menhir version currently required for a working
> > Coccinelle software?
> 
> No, you can use a more recent version of Menhir if you want.

I would appreciate that.

I am struggling to complete the needed dependencies for current releases
of various OCaml software.


> The reason of having an old version of Menhir in the bundles is that the
> newer versions of Menhir require OCaml 4.02 (for "match exception"), and
> we would like to keep the compatibility with OCaml 4.01 (or even 3.12).

Do such constraints need any more configuration checks in the build scripts?


> I don?t know if dependencies can be made more precise

I guess that it is generally possible to improve the build system safety.


> between the Coccinelle code and bundled libraries.

I am unsure on the efforts that we would like to invest here.


> I think it is not too demanding, but I agree that we should write
> more clearly in the commit message when cleaning is required.

I hope that such an extra build script clean-up should not be needed usually.

Regards,
Markus

[Cocci] Checking build dependencies for "pyml"

[SF Markus Elfring]

> There are things like line numbers that are converted to strings.

I am affected by such an implementation detail in some SmPL scripts.


> There is no technical reason for that, but the person who first
> implemented the python interface did that, and then it didn't seem like
> big enough of a deal to break backward compatability.

Do any more software developers care for this aspect?


> But maybe you are referring to somrething else.

I guess not. - I am just unsure if this will matter at more source code places.

Regards,
Markus

[Cocci] Checking build dependencies for "Menhir"

[SF Markus Elfring]

>> Is the bundled Menhir version currently required for a working
>> Coccinelle software?
> 
> No, you can use a more recent version of Menhir if you want.

?
checking for menhir... /usr/local/bin/menhir
configure: system menhir will be used
?
File "parser_cocci_menhir.mli", line 262, characters 10-56:
Error: Unbound module MenhirLib.IncrementalEngine
make[6]: *** [Makefile:89: parser_cocci_menhir.cmi] Error 2
?


A command like "make distclean" does not help so far.

Will there any more checks and settings be needed to get the desired
build working together with the software "OCaml 4.03.0-1.1"?

Regards,
Markus

[Cocci] Checking build dependencies for "Menhir"

[SF Markus Elfring]

> No, you can use a more recent version of Menhir if you want.

The following command succeeds.

elfring at Sonne:~/Projekte/Coccinelle/20160205/parsing_cocci> /usr/bin/ocamlc.opt -unsafe -I ../commons -I ../commons/ocamlextra -I ../globals -I /usr/local/share/menhir -c parser_cocci_menhir.mli


Now I wonder why the directory "bundles" was passed for the corresponding
include parameter after the "system tool" was selected by the
configuration script.
How should this detail be fixed?
https://github.com/coccinelle/coccinelle/blob/25c0b3278dff5e1ec4928724ff29ce977acc256b/configure.ac#L177

Regards,
Markus

[Cocci] Checking build dependencies for "Menhir"

[SF Markus Elfring]

I have gathered a bit more background information.

AC_REQ_COCCI_EXTPKG([menhirLib])
? AC_COCCI_OCAMLPKG()
? AC_CHECK_OCAML_PKG()
? $OCAMLFIND query

?
configure: configuring package menhirLib
checking for OCaml findlib package menhirLib... not found
?


The mentioned steps from m4 scripts generated such a display
after the command "make PREFIX=/usr/local install" succeeded
for the software "Menhir 20160526". It showed also the message
"Installing MenhirLib manually." which indicates a detail
that I find strange at the moment.
I would appreciate a display of the message "Installing MenhirLib
via ocamlfind." instead. Would you like to help to get
a command "src/_stage2/menhir.native --suggest-ocamlfind" working
in this direction?

Regards,
Markus

[Cocci] Checking build dependencies for "Menhir"

[Thierry Martinez]

Markus:
> Would you like to help to get
> a command "src/_stage2/menhir.native --suggest-ocamlfind" working
> in this direction?

Thank you for your report. The last GitHub update fixes the problem you
noticed and follows your suggestions.

menhir and menhirLib were detected independently, and the detection of
menhirLib required ocamlfind: so, if menhir was installed on your system
but if you didn't have ocamlfind, Coccinelle used the system menhir but
the bundled menhirLib, hence the failure your noticed in case of version
mismatch.

Now, the menhirLib is detected via menhir --suggest-menhirLib if
available (this option is only available in recent versions of menhir),
or via ocamlfind if available. Otherwise, the bundled versions of menhir
and menhirLib are used, regardless of what is installed.

-- 
Thierry.

SF Markus Elfring (2016/07/09, 07:45 CEST)?:
> I have gathered a bit more background information.
>
> AC_REQ_COCCI_EXTPKG([menhirLib])
> ? AC_COCCI_OCAMLPKG()
> ? AC_CHECK_OCAML_PKG()
> ? $OCAMLFIND query
>
> ?
> configure: configuring package menhirLib
> checking for OCaml findlib package menhirLib... not found
> ?
>
>
> The mentioned steps from m4 scripts generated such a display
> after the command "make PREFIX=/usr/local install" succeeded
> for the software "Menhir 20160526". It showed also the message
> "Installing MenhirLib manually." which indicates a detail
> that I find strange at the moment.
> I would appreciate a display of the message "Installing MenhirLib
> via ocamlfind." instead. Would you like to help to get
> a command "src/_stage2/menhir.native --suggest-ocamlfind" working
> in this direction?
>
> Regards,
> Markus

[Cocci] Checking build dependencies for "Menhir"

[SF Markus Elfring]

> menhir and menhirLib were detected independently, and the detection of
> menhirLib required ocamlfind: so, if menhir was installed on your system
> but if you didn't have ocamlfind,

This software tool was available in my use case.


> Coccinelle used the system menhir but the bundled menhirLib,
> hence the failure your noticed in case of version mismatch.

Did you consider also another implementation detail already?

* How do you think about the relevance of a make command parameter
  or setting in the environment like "USE_OCAMLFIND=1"?

* Fran?ois Pottier is thinking about improvements for the document "INSTALLATION".

Regards,
Markus