[Buildroot] [PATCH v2 0/3] CFFI backend

[Buildroot] [PATCH v2 0/3] CFFI backend

[yegorslists at googlemail.com]

This patch series introduces CFFI backend for Python packages. CFFI
can cross-compile C library wrappers and create *.py and *.so files
for the target rootfs. In order to do so CFFI uses pycparser to 
parse C heasers/sources. The target variant of CFFI is working with
already prepared *.py and *.so files and doesn't need pycparser any
longer.

The libffi headers patch is needed, because only headers for target
variant will be copied to the proper destination and host packages,
that don't use pkg-config cannot find libffi headers at its default
location.

Yegor

[Buildroot] [PATCH v2 1/3] libffi: fix headers location for host-libffi

[yegorslists at googlemail.com]

From: Yegor Yefremov <yegorslists@googlemail.com>

There is a special hook for target-libffi, that moves headers
from /usr/lib/libffi-version/include to /usr/include. This patch
adds the same procedure for host-libffi.

Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
---
 package/libffi/libffi.mk | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/package/libffi/libffi.mk b/package/libffi/libffi.mk
index 8f0f72d..47de8cd 100644
--- a/package/libffi/libffi.mk
+++ b/package/libffi/libffi.mk
@@ -22,6 +22,17 @@ endef
 
 LIBFFI_POST_INSTALL_STAGING_HOOKS += LIBFFI_MOVE_STAGING_HEADERS
 
+# Move the headers to the usual location, and adjust the .pc file
+# accordingly.
+define HOST_LIBFFI_MOVE_HOST_HEADERS
+	mv $(HOST_DIR)/usr/lib/libffi-$(LIBFFI_VERSION)/include/*.h $(HOST_DIR)/usr/include/
+	$(SED) '/^includedir.*/d' -e '/^Cflags:.*/d' \
+		$(HOST_DIR)/usr/lib/pkgconfig/libffi.pc
+	rm -rf $(HOST_DIR)/usr/lib/libffi-*
+endef
+
+HOST_LIBFFI_POST_INSTALL_HOOKS += HOST_LIBFFI_MOVE_HOST_HEADERS
+
 # Remove headers that are not at the usual location from the target
 define LIBFFI_REMOVE_TARGET_HEADERS
 	$(RM) -rf $(TARGET_DIR)/usr/lib/libffi-$(LIBFFI_VERSION)
-- 
2.1.4

[Buildroot] [PATCH v2 2/3] python-pycparser: new package

[yegorslists at googlemail.com]

From: Yegor Yefremov <yegorslists@googlemail.com>

pycparser requires C preprocessor in order to parse C related files,
hence available only as host package.

Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
---
Changes:
	v2: remove target package variant

 package/python-pycparser/python-pycparser.hash |  3 +++
 package/python-pycparser/python-pycparser.mk   | 14 ++++++++++++++
 2 files changed, 17 insertions(+)
 create mode 100644 package/python-pycparser/python-pycparser.hash
 create mode 100644 package/python-pycparser/python-pycparser.mk

diff --git a/package/python-pycparser/python-pycparser.hash b/package/python-pycparser/python-pycparser.hash
new file mode 100644
index 0000000..a87bfd4
--- /dev/null
+++ b/package/python-pycparser/python-pycparser.hash
@@ -0,0 +1,3 @@
+# md5 from https://pypi.python.org/pypi?:action=show_md5&digest=a2bc8d28c923b4fe2b2c3b4b51a4f935, sha256 locally computed
+md5     a2bc8d28c923b4fe2b2c3b4b51a4f935  pycparser-2.14.tar.gz
+sha256  7959b4a74abdc27b312fed1c21e6caf9309ce0b29ea86b591fd2e99ecdf27f73  pycparser-2.14.tar.gz
diff --git a/package/python-pycparser/python-pycparser.mk b/package/python-pycparser/python-pycparser.mk
new file mode 100644
index 0000000..328bc8a
--- /dev/null
+++ b/package/python-pycparser/python-pycparser.mk
@@ -0,0 +1,14 @@
+################################################################################
+#
+# python-pycparser
+#
+################################################################################
+
+PYTHON_PYCPARSER_VERSION = 2.14
+PYTHON_PYCPARSER_SOURCE = pycparser-$(PYTHON_PYCPARSER_VERSION).tar.gz
+PYTHON_PYCPARSER_SITE = https://pypi.python.org/packages/source/p/pycparser
+PYTHON_PYCPARSER_SETUP_TYPE = setuptools
+PYTHON_PYCPARSER_LICENSE = BSD-3c
+PYTHON_PYCPARSER_LICENSE_FILES = LICENSE
+
+$(eval $(host-python-package))
-- 
2.1.4

[Buildroot] [PATCH v2 3/3] python-cffi: add host variant

[yegorslists at googlemail.com]

From: Yegor Yefremov <yegorslists@googlemail.com>

Host variant is needed to cross-compile CFFI based C library wrappers.

Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
---
Changes:
	v2: add host-libffi dependency

 package/python-cffi/python-cffi.mk | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/package/python-cffi/python-cffi.mk b/package/python-cffi/python-cffi.mk
index 70fbe02..02c969b 100644
--- a/package/python-cffi/python-cffi.mk
+++ b/package/python-cffi/python-cffi.mk
@@ -11,5 +11,7 @@ PYTHON_CFFI_SETUP_TYPE = setuptools
 PYTHON_CFFI_DEPENDENCIES = host-pkgconf libffi
 PYTHON_CFFI_LICENSE = MIT
 PYTHON_CFFI_LICENSE_FILES = LICENSE
+HOST_PYTHON_CFFI_DEPENDENCIES = host-python-pycparser host-libffi
 
 $(eval $(python-package))
+$(eval $(host-python-package))
-- 
2.1.4

[Buildroot] [PATCH v2 3/3] python-cffi: add host variant

[Yegor Yefremov]

Hello Thomas, Yann,

On Tue, Jan 5, 2016 at 8:17 AM,  <yegorslists@googlemail.com> wrote:
> From: Yegor Yefremov <yegorslists@googlemail.com>
>
> Host variant is needed to cross-compile CFFI based C library wrappers.
>
> Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
> ---
> Changes:
>         v2: add host-libffi dependency
>
>  package/python-cffi/python-cffi.mk | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/package/python-cffi/python-cffi.mk b/package/python-cffi/python-cffi.mk
> index 70fbe02..02c969b 100644
> --- a/package/python-cffi/python-cffi.mk
> +++ b/package/python-cffi/python-cffi.mk
> @@ -11,5 +11,7 @@ PYTHON_CFFI_SETUP_TYPE = setuptools
>  PYTHON_CFFI_DEPENDENCIES = host-pkgconf libffi
>  PYTHON_CFFI_LICENSE = MIT
>  PYTHON_CFFI_LICENSE_FILES = LICENSE
> +HOST_PYTHON_CFFI_DEPENDENCIES = host-python-pycparser host-libffi
>
>  $(eval $(python-package))
> +$(eval $(host-python-package))
> --

I have now following theory regarding checking dependencies in target
and host variant.

If you look at python-tornado's setup.py file [1], you see, that they
say, install_requires will be checked only for setuptools. This is
also the case for host-python-cffi. The target variant will be
installed with --single-version-externally-managed option, that
performs "old-style" install [2] [3], i.e. without checking
install_requires.

That's my understanding for now.

[1] https://github.com/tornadoweb/tornado/blob/master/setup.py#L126
[2] https://pythonhosted.org/setuptools/setuptools.html
[3] https://bitbucket.org/pypa/setuptools/src/e517fced669d54bcaf3da2d8ca468d29120fedfc/setuptools/command/install.py?at=default&fileviewer=file-view-default#install.py-58

Yegor

[Buildroot] [PATCH v2 3/3] python-cffi: add host variant

[Yegor Yefremov]

On Tue, Jan 5, 2016 at 4:40 PM, Yegor Yefremov
<yegorslists@googlemail.com> wrote:
> Hello Thomas, Yann,
>
> On Tue, Jan 5, 2016 at 8:17 AM,  <yegorslists@googlemail.com> wrote:
>> From: Yegor Yefremov <yegorslists@googlemail.com>
>>
>> Host variant is needed to cross-compile CFFI based C library wrappers.
>>
>> Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
>> ---
>> Changes:
>>         v2: add host-libffi dependency
>>
>>  package/python-cffi/python-cffi.mk | 2 ++
>>  1 file changed, 2 insertions(+)
>>
>> diff --git a/package/python-cffi/python-cffi.mk b/package/python-cffi/python-cffi.mk
>> index 70fbe02..02c969b 100644
>> --- a/package/python-cffi/python-cffi.mk
>> +++ b/package/python-cffi/python-cffi.mk
>> @@ -11,5 +11,7 @@ PYTHON_CFFI_SETUP_TYPE = setuptools
>>  PYTHON_CFFI_DEPENDENCIES = host-pkgconf libffi
>>  PYTHON_CFFI_LICENSE = MIT
>>  PYTHON_CFFI_LICENSE_FILES = LICENSE
>> +HOST_PYTHON_CFFI_DEPENDENCIES = host-python-pycparser host-libffi
>>
>>  $(eval $(python-package))
>> +$(eval $(host-python-package))
>> --
>
> I have now following theory regarding checking dependencies in target
> and host variant.
>
> If you look at python-tornado's setup.py file [1], you see, that they
> say, install_requires will be checked only for setuptools. This is
> also the case for host-python-cffi. The target variant will be
> installed with --single-version-externally-managed option, that
> performs "old-style" install [2] [3], i.e. without checking
> install_requires.
>
> That's my understanding for now.
>
> [1] https://github.com/tornadoweb/tornado/blob/master/setup.py#L126
> [2] https://pythonhosted.org/setuptools/setuptools.html
> [3] https://bitbucket.org/pypa/setuptools/src/e517fced669d54bcaf3da2d8ca468d29120fedfc/setuptools/command/install.py?at=default&fileviewer=file-view-default#install.py-58

This SO answer also points to this conclusion:
http://stackoverflow.com/a/22179371/1113139

Yegor

[Buildroot] [PATCH v2 1/3] libffi: fix headers location for host-libffi

[Thomas Petazzoni]

Hello,

On Tue,  5 Jan 2016 08:17:22 +0100, yegorslists at googlemail.com wrote:
> From: Yegor Yefremov <yegorslists@googlemail.com>
> 
> There is a special hook for target-libffi, that moves headers
> from /usr/lib/libffi-version/include to /usr/include. This patch
> adds the same procedure for host-libffi.
> 
> Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
> ---
>  package/libffi/libffi.mk | 11 +++++++++++
>  1 file changed, 11 insertions(+)
> 
> diff --git a/package/libffi/libffi.mk b/package/libffi/libffi.mk
> index 8f0f72d..47de8cd 100644
> --- a/package/libffi/libffi.mk
> +++ b/package/libffi/libffi.mk
> @@ -22,6 +22,17 @@ endef
>  
>  LIBFFI_POST_INSTALL_STAGING_HOOKS += LIBFFI_MOVE_STAGING_HEADERS
>  
> +# Move the headers to the usual location, and adjust the .pc file
> +# accordingly.
> +define HOST_LIBFFI_MOVE_HOST_HEADERS
> +	mv $(HOST_DIR)/usr/lib/libffi-$(LIBFFI_VERSION)/include/*.h $(HOST_DIR)/usr/include/
> +	$(SED) '/^includedir.*/d' -e '/^Cflags:.*/d' \
> +		$(HOST_DIR)/usr/lib/pkgconfig/libffi.pc
> +	rm -rf $(HOST_DIR)/usr/lib/libffi-*
> +endef
> +
> +HOST_LIBFFI_POST_INSTALL_HOOKS += HOST_LIBFFI_MOVE_HOST_HEADERS

I've applied, but after doing some changes. First, the existing staging
fixup had a mistake, it was removing the directory from $(TARGET_DIR),
while it should have been removed from $(STAGING_DIR). I've fixed that
in a preparation commit:

  https://git.busybox.net/buildroot/commit/?id=2b73538920107b6c020ce5c4c8d9f7f672549bb1

Once this was fixed, your new code to take care of headers in
$(HOST_DIR) was exactly the same as the one taking care of the headers
in $(STAGING_DIR). So I've factorized the code in a common function,
which takes as argument the base directory. Which gives:

  https://git.busybox.net/buildroot/commit/?id=f41091176bb3e8da6cf78760b69f188e9868a259

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v2 2/3] python-pycparser: new package

[Thomas Petazzoni]

Dear yegorslists at googlemail.com,

On Tue,  5 Jan 2016 08:17:23 +0100, yegorslists at googlemail.com wrote:
> From: Yegor Yefremov <yegorslists@googlemail.com>
> 
> pycparser requires C preprocessor in order to parse C related files,
> hence available only as host package.
> 
> Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
> ---
> Changes:
> 	v2: remove target package variant

Applied, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v2 3/3] python-cffi: add host variant

[Thomas Petazzoni]

Hello,

On Tue,  5 Jan 2016 08:17:24 +0100, yegorslists at googlemail.com wrote:
> From: Yegor Yefremov <yegorslists@googlemail.com>
> 
> Host variant is needed to cross-compile CFFI based C library wrappers.
> 
> Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
> ---
> Changes:
> 	v2: add host-libffi dependency
> 
>  package/python-cffi/python-cffi.mk | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/package/python-cffi/python-cffi.mk b/package/python-cffi/python-cffi.mk
> index 70fbe02..02c969b 100644
> --- a/package/python-cffi/python-cffi.mk
> +++ b/package/python-cffi/python-cffi.mk
> @@ -11,5 +11,7 @@ PYTHON_CFFI_SETUP_TYPE = setuptools
>  PYTHON_CFFI_DEPENDENCIES = host-pkgconf libffi
>  PYTHON_CFFI_LICENSE = MIT
>  PYTHON_CFFI_LICENSE_FILES = LICENSE
> +HOST_PYTHON_CFFI_DEPENDENCIES = host-python-pycparser host-libffi
>  
>  $(eval $(python-package))
> +$(eval $(host-python-package))

This wasn't building properly here, because python-cffi uses pkg-config
to get the flags to link with libffi, and by default our pkg-config
returns results that are valid to build stuff for the target, not for
the host.

So, I've fixed that by passing a bunch of pkg-config environment
variables when building the host package.

Also, you were missing the host-pkgconf dependency for the dependencies
of host variant.

Committed with those issues fixed.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com