[Buildroot] [PATCH v2 0/3] python-scipy package

[Buildroot] [PATCH v2 0/3] python-scipy package

[Esben Haabendal]

Add new python-scipy package and fixup python-numpy so python-scipy can
be cross-compiled.

Changed in v2:

* Added mandatory and lovely comment header in python-scipy.mk file

Esben Haabendal (3):
  package/python-numpy: fix setup type
  package/python-numpy: fixup numpy distutils for cross compilation
  package/python-scipy: new package

 DEVELOPERS                                    |  1 +
 package/Config.in                             |  1 +
 .../0001-NPY_PKG_CONFIG_PATH.patch            | 84 +++++++++++++++++++
 package/python-numpy/python-numpy.mk          | 25 +++++-
 package/python-scipy/Config.in                | 21 +++++
 package/python-scipy/python-scipy.hash        |  9 ++
 package/python-scipy/python-scipy.mk          | 37 ++++++++
 7 files changed, 177 insertions(+), 1 deletion(-)
 create mode 100644 package/python-numpy/0001-NPY_PKG_CONFIG_PATH.patch
 create mode 100644 package/python-scipy/Config.in
 create mode 100644 package/python-scipy/python-scipy.hash
 create mode 100644 package/python-scipy/python-scipy.mk

-- 
2.23.0

[Buildroot] [PATCH v2 1/3] package/python-numpy: fix setup type

[Esben Haabendal]

The numpy package is using distutils, not setuptools.

Signed-off-by: Esben Haabendal <esben@geanix.com>
---
 package/python-numpy/python-numpy.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/python-numpy/python-numpy.mk b/package/python-numpy/python-numpy.mk
index 3b474efa6e6c..f1f6f8cba407 100644
--- a/package/python-numpy/python-numpy.mk
+++ b/package/python-numpy/python-numpy.mk
@@ -13,7 +13,7 @@ PYTHON_NUMPY_LICENSE_FILES = LICENSE.txt doc/sphinxext/LICENSE.txt \
 			numpy/linalg/lapack_lite/LICENSE.txt \
 			tools/npy_tempita/license.txt \
 			numpy/core/src/multiarray/dragon4.c
-PYTHON_NUMPY_SETUP_TYPE = setuptools
+PYTHON_NUMPY_SETUP_TYPE = distutils
 
 ifeq ($(BR2_PACKAGE_CLAPACK),y)
 PYTHON_NUMPY_DEPENDENCIES += clapack
-- 
2.23.0

[Buildroot] [PATCH v2 2/3] package/python-numpy: fixup numpy distutils for cross compilation

[Esben Haabendal]

Fix problems using the numpy distutils extension for cross compilation,
so that linking with npymath library will use target library when building
target packages.

Signed-off-by: Esben Haabendal <esben@geanix.com>
---
 .../0001-NPY_PKG_CONFIG_PATH.patch            | 84 +++++++++++++++++++
 package/python-numpy/python-numpy.mk          | 23 +++++
 2 files changed, 107 insertions(+)
 create mode 100644 package/python-numpy/0001-NPY_PKG_CONFIG_PATH.patch

diff --git a/package/python-numpy/0001-NPY_PKG_CONFIG_PATH.patch b/package/python-numpy/0001-NPY_PKG_CONFIG_PATH.patch
new file mode 100644
index 000000000000..cedf5ecc90e4
--- /dev/null
+++ b/package/python-numpy/0001-NPY_PKG_CONFIG_PATH.patch
@@ -0,0 +1,84 @@
+commit 153fc148eec60e5cbec0e80617f75a3a5dd2a3f8
+Author: Esben Haabendal <esben@geanix.com>
+Date:   Thu Sep 12 21:59:58 2019 +0200
+
+    ENH: Allow NPY_PKG_CONFIG_PATH environment variable override
+    
+    Allow overriding npy-pkg-config directory using the NPY_PKG_CONFIG_PATH
+    environment variable, making it easier to use numpy in cross-compilation
+    setups.
+
+Upstream-Status: Accepted (scheduled for 1.18.0)
+
+diff --git a/numpy/distutils/misc_util.py b/numpy/distutils/misc_util.py
+index 1e10e92fdadf..0eaaeb7364d4 100644
+--- a/numpy/distutils/misc_util.py
++++ b/numpy/distutils/misc_util.py
+@@ -1687,6 +1687,43 @@ class Configuration(object):
+ 
+         and will be installed as foo.ini in the 'lib' subpath.
+ 
++        Cross-compilation
++        -----------------
++        When cross-compiling with numpy distutils, it might be necessary to
++        use modified npy-pkg-config files.  Using the default/generated files
++        will link with the host libraries (i.e. libnpymath.a).  For
++        cross-compilation you of-course need to link with target libraries,
++        while using the host Python installation.
++
++        You can copy out the numpy/core/lib/npy-pkg-config directory, add a
++        pkgdir value to the .ini files and set NPY_PKG_CONFIG_PATH environment
++        variable to point to the directory with the modified npy-pkg-config
++        files.
++
++        Example npymath.ini modified for cross-compilation::
++
++            [meta]
++            Name=npymath
++            Description=Portable, core math library implementing C99 standard
++            Version=0.1
++
++            [variables]
++            pkgname=numpy.core
++            pkgdir=/build/arm-linux-gnueabi/sysroot/usr/lib/python3.7/site-packages/numpy/core
++            prefix=${pkgdir}
++            libdir=${prefix}/lib
++            includedir=${prefix}/include
++
++            [default]
++            Libs=-L${libdir} -lnpymath
++            Cflags=-I${includedir}
++            Requires=mlib
++
++            [msvc]
++            Libs=/LIBPATH:${libdir} npymath.lib
++            Cflags=/INCLUDE:${includedir}
++            Requires=mlib
++
+         """
+         if subst_dict is None:
+             subst_dict = {}
+@@ -2092,9 +2129,22 @@ def get_numpy_include_dirs():
+     return include_dirs
+ 
+ def get_npy_pkg_dir():
+-    """Return the path where to find the npy-pkg-config directory."""
++    """Return the path where to find the npy-pkg-config directory.
++
++    If the NPY_PKG_CONFIG_PATH environment variable is set, the value of that
++    is returned.  Otherwise, a path inside the location of the numpy module is
++    returned.
++
++    The NPY_PKG_CONFIG_PATH can be useful when cross-compiling, maintaining
++    customized npy-pkg-config .ini files for the cross-compilation
++    environment, and using them when cross-compiling.
++
++    """
+     # XXX: import here for bootstrapping reasons
+     import numpy
++    d = os.environ.get('NPY_PKG_CONFIG_PATH')
++    if d is not None:
++        return d
+     d = os.path.join(os.path.dirname(numpy.__file__),
+             'core', 'lib', 'npy-pkg-config')
+     return d
diff --git a/package/python-numpy/python-numpy.mk b/package/python-numpy/python-numpy.mk
index f1f6f8cba407..3edb34a79ff7 100644
--- a/package/python-numpy/python-numpy.mk
+++ b/package/python-numpy/python-numpy.mk
@@ -30,6 +30,29 @@ define PYTHON_NUMPY_CONFIGURE_CMDS
 	echo "include_dirs = $(STAGING_DIR)/usr/include" >> $(@D)/site.cfg
 endef
 
+# The numpy distutils extensions are not very cross friendly.  It comes with
+# it's own pkg-config look-alike, which we are patching to allow overriding
+# where it locates the configuration files.  This allows us to use fixed up
+# target configuration files, which we make sure includes full path to the
+# target staging area, so that when building for target, we actually use the
+# target libraries.  Without this, target builds using numpy distutils
+# extensions (such as fx. python-scipy) will use the host libraries, which
+# obviously will not work.
+ifeq ($(BR2_PACKAGE_PYTHON3),y)
+PYTHON_NUMPY_STAGING_DIR = $(STAGING_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages/numpy
+else
+PYTHON_NUMPY_STAGING_DIR = $(STAGING_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages/numpy
+endif
+PYTHON_NUMPY_NPY_PKG_CONFIG_PATH=$(PYTHON_NUMPY_STAGING_DIR)/core/lib/npy-pkg-config
+define PYTHON_NUMPY_FIXUP_NPY_PKG_CONFIG_FILES
+	sed -i '/pkgdir=/d' \
+		$(PYTHON_NUMPY_NPY_PKG_CONFIG_PATH)/npymath.ini
+	awk -i inplace \
+		'/prefix=/ {print "pkgdir='$(PYTHON_NUMPY_STAGING_DIR)/core'"}1' \
+		$(PYTHON_NUMPY_NPY_PKG_CONFIG_PATH)/npymath.ini
+endef
+PYTHON_NUMPY_POST_INSTALL_STAGING_HOOKS += PYTHON_NUMPY_FIXUP_NPY_PKG_CONFIG_FILES
+
 # Some package may include few headers from NumPy, so let's install it
 # in the staging area.
 PYTHON_NUMPY_INSTALL_STAGING = YES
-- 
2.23.0

[Buildroot] [PATCH v2 3/3] package/python-scipy: new package

[Esben Haabendal]

---
 DEVELOPERS                             |  1 +
 package/Config.in                      |  1 +
 package/python-scipy/Config.in         | 21 +++++++++++++++
 package/python-scipy/python-scipy.hash |  9 +++++++
 package/python-scipy/python-scipy.mk   | 37 ++++++++++++++++++++++++++
 5 files changed, 69 insertions(+)
 create mode 100644 package/python-scipy/Config.in
 create mode 100644 package/python-scipy/python-scipy.hash
 create mode 100644 package/python-scipy/python-scipy.mk

diff --git a/DEVELOPERS b/DEVELOPERS
index 67a0fef0886d..975a197404bb 100644
--- a/DEVELOPERS
+++ b/DEVELOPERS
@@ -723,6 +723,7 @@ F:	package/szip/
 N:	Esben Haabendal <esben@haabendal.dk>
 F:	boot/gummiboot/
 F:	package/python-kiwisolver/
+F:	package/python-scipy/
 
 N:	Etienne Carriere <etienne.carriere@linaro.org>
 F:	boot/optee-os/
diff --git a/package/Config.in b/package/Config.in
index dbf297f4df39..818a2abca591 100644
--- a/package/Config.in
+++ b/package/Config.in
@@ -1056,6 +1056,7 @@ menu "External python modules"
 	source "package/python-scapy/Config.in"
 	source "package/python-scapy3k/Config.in"
 	source "package/python-schedule/Config.in"
+	source "package/python-scipy/Config.in"
 	source "package/python-sdnotify/Config.in"
 	source "package/python-secretstorage/Config.in"
 	source "package/python-see/Config.in"
diff --git a/package/python-scipy/Config.in b/package/python-scipy/Config.in
new file mode 100644
index 000000000000..5b8192266843
--- /dev/null
+++ b/package/python-scipy/Config.in
@@ -0,0 +1,21 @@
+config BR2_PACKAGE_PYTHON_SCIPY
+	bool "python-scipy"
+	depends on BR2_PACKAGE_PYTHON3
+	depends on BR2_TOOLCHAIN_HAS_FORTRAN
+	depends on BR2_PACKAGE_PYTHON_NUMPY_ARCH_SUPPORTS
+	depends on BR2_PACKAGE_OPENBLAS_ARCH_SUPPORTS
+	select BR2_PACKAGE_HOST_PYTHON_CYTHON
+	select BR2_PACKAGE_HOST_PYTHON_NUMPY
+	select BR2_PACKAGE_PYTHON_NUMPY
+	select BR2_PACKAGE_OPENBLAS
+	select BR2_PACKAGE_CLAPACK
+	help
+	  The SciPy library is one of the core packages that make up the SciPy
+	  stack. It provides many user-friendly and efficient numerical
+	  routines such as routines for numerical integration, interpolation,
+	  optimization, linear algebra and statistics.
+
+	  https://www.scipy.org/scipylib/
+
+comment "python-scipy needs a toolchain w/ fortran"
+	depends on !BR2_TOOLCHAIN_HAS_FORTRAN
diff --git a/package/python-scipy/python-scipy.hash b/package/python-scipy/python-scipy.hash
new file mode 100644
index 000000000000..6b0c3605a3b8
--- /dev/null
+++ b/package/python-scipy/python-scipy.hash
@@ -0,0 +1,9 @@
+# Locally generated
+sha256 4ea68de2840cf7d35c58464412d21f6f154cab7fba610c11002603ee0b9e2372  python-scipy-1.3.1.tar.gz
+sha256 904942f7df1dc0f4a48af8d72405854fd96497bd1fe9ddb2c69d1797c22dfcd7  LICENSE.txt
+sha256 37e64a498894ac7c3b070023e3689e954a8ecf8a23b90968d09a455f1b4f7b35  scipy/linalg/src/lapack_deprecations/LICENSE
+sha256 606209a000716c5f66e33e180ce08434b96ed17db4975ab9723c6b5fbcc89609  scipy/ndimage/LICENSE.txt
+sha256 3df9207af2fdb861af0ae3b22026f163b9bcfab4e525dc4943afe2ffa3f77624  scipy/optimize/tnc/LICENSE
+sha256 f0cedf52503b2d42b83411a0a16e6fefac346dfad8fddc66f50050150123470c  scipy/sparse/linalg/dsolve/SuperLU/License.txt
+sha256 0926566f9f406178d1214f8cc796e166b1213dd7c05e0c5b461a8b8ac9e50bbe  scipy/sparse/linalg/eigen/arpack/ARPACK/COPYING
+sha256 51a5a08d537b34febb851c761496ead461f90b6c3c46a5248780d63870124e9c  scipy/spatial/qhull_src/COPYING.txt
diff --git a/package/python-scipy/python-scipy.mk b/package/python-scipy/python-scipy.mk
new file mode 100644
index 000000000000..74db98ed0ba0
--- /dev/null
+++ b/package/python-scipy/python-scipy.mk
@@ -0,0 +1,37 @@
+################################################################################
+#
+# python-scipy
+#
+################################################################################
+
+PYTHON_SCIPY_VERSION = 1.3.1
+PYTHON_SCIPY_SITE = $(call github,scipy,scipy,v$(PYTHON_SCIPY_VERSION))
+PYTHON_SCIPY_LICENSE = BSD-3-Clause, BSD-2-Clause, BSD, BSD-Style, PSF, \
+	Apache-2.0, MIT
+PYTHON_SCIPY_LICENSE_FILES = LICENSE.txt \
+	scipy/linalg/src/lapack_deprecations/LICENSE \
+	scipy/ndimage/LICENSE.txt \
+	scipy/optimize/tnc/LICENSE \
+	scipy/sparse/linalg/dsolve/SuperLU/License.txt \
+	scipy/sparse/linalg/eigen/arpack/ARPACK/COPYING \
+	scipy/spatial/qhull_src/COPYING.txt
+PYTHON_SCIPY_SETUP_TYPE = setuptools
+PYTHON_SCIPY_DEPENDENCIES += host-python-cython host-python-numpy python-numpy \
+	openblas clapack
+
+PYTHON_SCIPY_BUILD_OPTS = --fcompiler=gnu95
+PYTHON_SCIPY_ENV += F90=$(TARGET_CROSS)gfortran F77=$(TARGET_CROSS)gfortran
+
+define PYTHON_SCIPY_CONFIGURE_CMDS
+	-rm -f $(@D)/site.cfg
+	echo "[DEFAULT]" >> $(@D)/site.cfg
+	echo "library_dirs = $(STAGING_DIR)/usr/lib" >> $(@D)/site.cfg
+	echo "include_dirs = $(STAGING_DIR)/usr/include" >> $(@D)/site.cfg
+endef
+
+# Use the target numpy pkg-config configuration files modified for target
+# cross-compilation.  Without this, numpy distutils will cause the linker to
+# link with host libnpymath.a.
+PYTHON_SCIPY_ENV += NPY_PKG_CONFIG_PATH=$(PYTHON_NUMPY_NPY_PKG_CONFIG_PATH)
+
+$(eval $(python-package))
-- 
2.23.0

[Buildroot] [PATCH v2 3/3] package/python-scipy: new package

[Romain Naour]

Hi Esben,

There is a previous contribution to add python-scipy from Alexandre Payen:
http://patchwork.ozlabs.org/patch/1141144/

It would be great if you can work with Alexandre on python-scipy package.

Le 20/09/2019 ? 18:36, Esben Haabendal a ?crit?:
> ---
>  DEVELOPERS                             |  1 +
>  package/Config.in                      |  1 +
>  package/python-scipy/Config.in         | 21 +++++++++++++++
>  package/python-scipy/python-scipy.hash |  9 +++++++
>  package/python-scipy/python-scipy.mk   | 37 ++++++++++++++++++++++++++
>  5 files changed, 69 insertions(+)
>  create mode 100644 package/python-scipy/Config.in
>  create mode 100644 package/python-scipy/python-scipy.hash
>  create mode 100644 package/python-scipy/python-scipy.mk
> 
> diff --git a/DEVELOPERS b/DEVELOPERS
> index 67a0fef0886d..975a197404bb 100644
> --- a/DEVELOPERS
> +++ b/DEVELOPERS
> @@ -723,6 +723,7 @@ F:	package/szip/
>  N:	Esben Haabendal <esben@haabendal.dk>
>  F:	boot/gummiboot/
>  F:	package/python-kiwisolver/
> +F:	package/python-scipy/
>  
>  N:	Etienne Carriere <etienne.carriere@linaro.org>
>  F:	boot/optee-os/
> diff --git a/package/Config.in b/package/Config.in
> index dbf297f4df39..818a2abca591 100644
> --- a/package/Config.in
> +++ b/package/Config.in
> @@ -1056,6 +1056,7 @@ menu "External python modules"
>  	source "package/python-scapy/Config.in"
>  	source "package/python-scapy3k/Config.in"
>  	source "package/python-schedule/Config.in"
> +	source "package/python-scipy/Config.in"
>  	source "package/python-sdnotify/Config.in"
>  	source "package/python-secretstorage/Config.in"
>  	source "package/python-see/Config.in"
> diff --git a/package/python-scipy/Config.in b/package/python-scipy/Config.in
> new file mode 100644
> index 000000000000..5b8192266843
> --- /dev/null
> +++ b/package/python-scipy/Config.in
> @@ -0,0 +1,21 @@
> +config BR2_PACKAGE_PYTHON_SCIPY
> +	bool "python-scipy"
> +	depends on BR2_PACKAGE_PYTHON3
> +	depends on BR2_TOOLCHAIN_HAS_FORTRAN
> +	depends on BR2_PACKAGE_PYTHON_NUMPY_ARCH_SUPPORTS
> +	depends on BR2_PACKAGE_OPENBLAS_ARCH_SUPPORTS
> +	select BR2_PACKAGE_HOST_PYTHON_CYTHON
> +	select BR2_PACKAGE_HOST_PYTHON_NUMPY

There is no such option, the dependency is trigged by adding host-python-numpy
in DEPENDENCIES;

> +	select BR2_PACKAGE_PYTHON_NUMPY
> +	select BR2_PACKAGE_OPENBLAS
> +	select BR2_PACKAGE_CLAPACK

Python-scipy can be used with lapack instead of Clapack.
Clapack is not maintained any more and really out of date.

OpenBlas is using a bundled version of Lapack when gFortran compiler is
available. Clapack is an old f2c'ed version of lapack, so it can be used without
a Fortran Compiler. Since Python-scipy depends on Fortran, Clapack can be
removed from python-scipy.

Actually there is an existing issue (discussed during the Buildroot hackathon)
on *blas providers. We need to fix this before adding new packages using *blas
libraries :-/

Best regards,
Romain

> +	help
> +	  The SciPy library is one of the core packages that make up the SciPy
> +	  stack. It provides many user-friendly and efficient numerical
> +	  routines such as routines for numerical integration, interpolation,
> +	  optimization, linear algebra and statistics.
> +
> +	  https://www.scipy.org/scipylib/
> +
> +comment "python-scipy needs a toolchain w/ fortran"
> +	depends on !BR2_TOOLCHAIN_HAS_FORTRAN
> diff --git a/package/python-scipy/python-scipy.hash b/package/python-scipy/python-scipy.hash
> new file mode 100644
> index 000000000000..6b0c3605a3b8
> --- /dev/null
> +++ b/package/python-scipy/python-scipy.hash
> @@ -0,0 +1,9 @@
> +# Locally generated
> +sha256 4ea68de2840cf7d35c58464412d21f6f154cab7fba610c11002603ee0b9e2372  python-scipy-1.3.1.tar.gz
> +sha256 904942f7df1dc0f4a48af8d72405854fd96497bd1fe9ddb2c69d1797c22dfcd7  LICENSE.txt
> +sha256 37e64a498894ac7c3b070023e3689e954a8ecf8a23b90968d09a455f1b4f7b35  scipy/linalg/src/lapack_deprecations/LICENSE
> +sha256 606209a000716c5f66e33e180ce08434b96ed17db4975ab9723c6b5fbcc89609  scipy/ndimage/LICENSE.txt
> +sha256 3df9207af2fdb861af0ae3b22026f163b9bcfab4e525dc4943afe2ffa3f77624  scipy/optimize/tnc/LICENSE
> +sha256 f0cedf52503b2d42b83411a0a16e6fefac346dfad8fddc66f50050150123470c  scipy/sparse/linalg/dsolve/SuperLU/License.txt
> +sha256 0926566f9f406178d1214f8cc796e166b1213dd7c05e0c5b461a8b8ac9e50bbe  scipy/sparse/linalg/eigen/arpack/ARPACK/COPYING
> +sha256 51a5a08d537b34febb851c761496ead461f90b6c3c46a5248780d63870124e9c  scipy/spatial/qhull_src/COPYING.txt
> diff --git a/package/python-scipy/python-scipy.mk b/package/python-scipy/python-scipy.mk
> new file mode 100644
> index 000000000000..74db98ed0ba0
> --- /dev/null
> +++ b/package/python-scipy/python-scipy.mk
> @@ -0,0 +1,37 @@
> +################################################################################
> +#
> +# python-scipy
> +#
> +################################################################################
> +
> +PYTHON_SCIPY_VERSION = 1.3.1
> +PYTHON_SCIPY_SITE = $(call github,scipy,scipy,v$(PYTHON_SCIPY_VERSION))
> +PYTHON_SCIPY_LICENSE = BSD-3-Clause, BSD-2-Clause, BSD, BSD-Style, PSF, \
> +	Apache-2.0, MIT
> +PYTHON_SCIPY_LICENSE_FILES = LICENSE.txt \
> +	scipy/linalg/src/lapack_deprecations/LICENSE \
> +	scipy/ndimage/LICENSE.txt \
> +	scipy/optimize/tnc/LICENSE \
> +	scipy/sparse/linalg/dsolve/SuperLU/License.txt \
> +	scipy/sparse/linalg/eigen/arpack/ARPACK/COPYING \
> +	scipy/spatial/qhull_src/COPYING.txt
> +PYTHON_SCIPY_SETUP_TYPE = setuptools
> +PYTHON_SCIPY_DEPENDENCIES += host-python-cython host-python-numpy python-numpy \
> +	openblas clapack
> +
> +PYTHON_SCIPY_BUILD_OPTS = --fcompiler=gnu95
> +PYTHON_SCIPY_ENV += F90=$(TARGET_CROSS)gfortran F77=$(TARGET_CROSS)gfortran
> +
> +define PYTHON_SCIPY_CONFIGURE_CMDS
> +	-rm -f $(@D)/site.cfg
> +	echo "[DEFAULT]" >> $(@D)/site.cfg
> +	echo "library_dirs = $(STAGING_DIR)/usr/lib" >> $(@D)/site.cfg
> +	echo "include_dirs = $(STAGING_DIR)/usr/include" >> $(@D)/site.cfg
> +endef
> +
> +# Use the target numpy pkg-config configuration files modified for target
> +# cross-compilation.  Without this, numpy distutils will cause the linker to
> +# link with host libnpymath.a.
> +PYTHON_SCIPY_ENV += NPY_PKG_CONFIG_PATH=$(PYTHON_NUMPY_NPY_PKG_CONFIG_PATH)
> +
> +$(eval $(python-package))
> 

[Buildroot] [PATCH v2 1/3] package/python-numpy: fix setup type

[Yegor Yefremov]

Hi Esben,

On Fri, Sep 20, 2019 at 6:37 PM Esben Haabendal <esben@geanix.com> wrote:
>
> The numpy package is using distutils, not setuptools.

Are you sure about this? See
https://github.com/numpy/numpy/blob/master/numpy/distutils/core.py

Yegor

> Signed-off-by: Esben Haabendal <esben@geanix.com>
> ---
>  package/python-numpy/python-numpy.mk | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/package/python-numpy/python-numpy.mk b/package/python-numpy/python-numpy.mk
> index 3b474efa6e6c..f1f6f8cba407 100644
> --- a/package/python-numpy/python-numpy.mk
> +++ b/package/python-numpy/python-numpy.mk
> @@ -13,7 +13,7 @@ PYTHON_NUMPY_LICENSE_FILES = LICENSE.txt doc/sphinxext/LICENSE.txt \
>                         numpy/linalg/lapack_lite/LICENSE.txt \
>                         tools/npy_tempita/license.txt \
>                         numpy/core/src/multiarray/dragon4.c
> -PYTHON_NUMPY_SETUP_TYPE = setuptools
> +PYTHON_NUMPY_SETUP_TYPE = distutils
>
>  ifeq ($(BR2_PACKAGE_CLAPACK),y)
>  PYTHON_NUMPY_DEPENDENCIES += clapack
> --
> 2.23.0
>

[Buildroot] [PATCH v2 1/3] package/python-numpy: fix setup type

[Esben Haabendal]

Hi Yegor

Yegor Yefremov <yegorslists@googlemail.com> writes:

> On Fri, Sep 20, 2019 at 6:37 PM Esben Haabendal <esben@geanix.com> wrote:
>>
>> The numpy package is using distutils, not setuptools.
>
> Are you sure about this? See
> https://github.com/numpy/numpy/blob/master/numpy/distutils/core.py

https://github.com/numpy/numpy/blob/v1.16.4/setup.py#L392

Looks like a setuptools package to me.

/Esben

[Buildroot] [PATCH v2 3/3] package/python-scipy: new package

[Esben Haabendal]

Hi Romain

Romain Naour <romain.naour@smile.fr> writes:

> There is a previous contribution to add python-scipy from Alexandre Payen:
> http://patchwork.ozlabs.org/patch/1141144/
>
> It would be great if you can work with Alexandre on python-scipy package.

Sounds good.

Alexandre, do you have time to work on this?

> Le 20/09/2019 ? 18:36, Esben Haabendal a ?crit?:
>> ---
>>  DEVELOPERS                             |  1 +
>>  package/Config.in                      |  1 +
>>  package/python-scipy/Config.in         | 21 +++++++++++++++
>>  package/python-scipy/python-scipy.hash |  9 +++++++
>>  package/python-scipy/python-scipy.mk   | 37 ++++++++++++++++++++++++++
>>  5 files changed, 69 insertions(+)
>>  create mode 100644 package/python-scipy/Config.in
>>  create mode 100644 package/python-scipy/python-scipy.hash
>>  create mode 100644 package/python-scipy/python-scipy.mk
>> 
>> diff --git a/DEVELOPERS b/DEVELOPERS
>> index 67a0fef0886d..975a197404bb 100644
>> --- a/DEVELOPERS
>> +++ b/DEVELOPERS
>> @@ -723,6 +723,7 @@ F:	package/szip/
>>  N:	Esben Haabendal <esben@haabendal.dk>
>>  F:	boot/gummiboot/
>>  F:	package/python-kiwisolver/
>> +F:	package/python-scipy/
>>  
>>  N:	Etienne Carriere <etienne.carriere@linaro.org>
>>  F:	boot/optee-os/
>> diff --git a/package/Config.in b/package/Config.in
>> index dbf297f4df39..818a2abca591 100644
>> --- a/package/Config.in
>> +++ b/package/Config.in
>> @@ -1056,6 +1056,7 @@ menu "External python modules"
>>  	source "package/python-scapy/Config.in"
>>  	source "package/python-scapy3k/Config.in"
>>  	source "package/python-schedule/Config.in"
>> +	source "package/python-scipy/Config.in"
>>  	source "package/python-sdnotify/Config.in"
>>  	source "package/python-secretstorage/Config.in"
>>  	source "package/python-see/Config.in"
>> diff --git a/package/python-scipy/Config.in b/package/python-scipy/Config.in
>> new file mode 100644
>> index 000000000000..5b8192266843
>> --- /dev/null
>> +++ b/package/python-scipy/Config.in
>> @@ -0,0 +1,21 @@
>> +config BR2_PACKAGE_PYTHON_SCIPY
>> +	bool "python-scipy"
>> +	depends on BR2_PACKAGE_PYTHON3
>> +	depends on BR2_TOOLCHAIN_HAS_FORTRAN
>> +	depends on BR2_PACKAGE_PYTHON_NUMPY_ARCH_SUPPORTS
>> +	depends on BR2_PACKAGE_OPENBLAS_ARCH_SUPPORTS
>> +	select BR2_PACKAGE_HOST_PYTHON_CYTHON
>> +	select BR2_PACKAGE_HOST_PYTHON_NUMPY
>
> There is no such option, the dependency is trigged by adding host-python-numpy
> in DEPENDENCIES;

Ah, I wasn't aware of how host-foo packages in DEPENDENCIES works.
I will remove the bogus select.

>> +	select BR2_PACKAGE_PYTHON_NUMPY
>> +	select BR2_PACKAGE_OPENBLAS
>> +	select BR2_PACKAGE_CLAPACK
>
> Python-scipy can be used with lapack instead of Clapack.
> Clapack is not maintained any more and really out of date.

Yes, but as we currently only supports clapack for python-numpy builds,
It seems a bit odd to do it different for python-scipy.

What about adding python-scipy first, and then extending both
python-numpy and python-scipy to support lapack as (an up-to-date)
alternative to clapack afterwards?

> OpenBlas is using a bundled version of Lapack when gFortran compiler
> is available. Clapack is an old f2c'ed version of lapack, so it can be
> used without a Fortran Compiler. Since Python-scipy depends on
> Fortran, Clapack can be removed from python-scipy.
>
> Actually there is an existing issue (discussed during the Buildroot
> hackathon) on *blas providers. We need to fix this before adding new
> packages using *blas libraries :-/

Ok, so where does this place this work on python-scipy for now?
Is it a 100% no-go until *blas providers are reworked?

I think it would be a shame to block python-scipy, as it is pretty
useful package, and the effort to take it in (looking at this patch
series) is pretty minimal.

/Esben

[Buildroot] [PATCH v2 3/3] package/python-scipy: new package

[Alexandre PAYEN]

Hi Esben.

Le lun. 23 sept. 2019 ? 09:05, Esben Haabendal <esben@geanix.com> a ?crit :

> Hi Romain
>
> Romain Naour <romain.naour@smile.fr> writes:
>
> > There is a previous contribution to add python-scipy from Alexandre
> Payen:
> > http://patchwork.ozlabs.org/patch/1141144/
> >
> > It would be great if you can work with Alexandre on python-scipy package.
>
> Sounds good.
>
> Alexandre, do you have time to work on this?
>
I have some time if you need yes.

>
> > Le 20/09/2019 ? 18:36, Esben Haabendal a ?crit :
> >> ---
> >>  DEVELOPERS                             |  1 +
> >>  package/Config.in                      |  1 +
> >>  package/python-scipy/Config.in         | 21 +++++++++++++++
> >>  package/python-scipy/python-scipy.hash |  9 +++++++
> >>  package/python-scipy/python-scipy.mk   | 37 ++++++++++++++++++++++++++
> >>  5 files changed, 69 insertions(+)
> >>  create mode 100644 package/python-scipy/Config.in
> >>  create mode 100644 package/python-scipy/python-scipy.hash
> >>  create mode 100644 package/python-scipy/python-scipy.mk
> >>
> >> diff --git a/DEVELOPERS b/DEVELOPERS
> >> index 67a0fef0886d..975a197404bb 100644
> >> --- a/DEVELOPERS
> >> +++ b/DEVELOPERS
> >> @@ -723,6 +723,7 @@ F:       package/szip/
> >>  N:  Esben Haabendal <esben@haabendal.dk>
> >>  F:  boot/gummiboot/
> >>  F:  package/python-kiwisolver/
> >> +F:  package/python-scipy/
> >>
> >>  N:  Etienne Carriere <etienne.carriere@linaro.org>
> >>  F:  boot/optee-os/
> >> diff --git a/package/Config.in b/package/Config.in
> >> index dbf297f4df39..818a2abca591 100644
> >> --- a/package/Config.in
> >> +++ b/package/Config.in
> >> @@ -1056,6 +1056,7 @@ menu "External python modules"
> >>      source "package/python-scapy/Config.in"
> >>      source "package/python-scapy3k/Config.in"
> >>      source "package/python-schedule/Config.in"
> >> +    source "package/python-scipy/Config.in"
> >>      source "package/python-sdnotify/Config.in"
> >>      source "package/python-secretstorage/Config.in"
> >>      source "package/python-see/Config.in"
> >> diff --git a/package/python-scipy/Config.in
> b/package/python-scipy/Config.in
> >> new file mode 100644
> >> index 000000000000..5b8192266843
> >> --- /dev/null
> >> +++ b/package/python-scipy/Config.in
> >> @@ -0,0 +1,21 @@
> >> +config BR2_PACKAGE_PYTHON_SCIPY
> >> +    bool "python-scipy"
> >> +    depends on BR2_PACKAGE_PYTHON3
> >> +    depends on BR2_TOOLCHAIN_HAS_FORTRAN
> >> +    depends on BR2_PACKAGE_PYTHON_NUMPY_ARCH_SUPPORTS
> >> +    depends on BR2_PACKAGE_OPENBLAS_ARCH_SUPPORTS
> >> +    select BR2_PACKAGE_HOST_PYTHON_CYTHON
> >> +    select BR2_PACKAGE_HOST_PYTHON_NUMPY
> >
> > There is no such option, the dependency is trigged by adding
> host-python-numpy
> > in DEPENDENCIES;
>
> Ah, I wasn't aware of how host-foo packages in DEPENDENCIES works.
> I will remove the bogus select.
>
> >> +    select BR2_PACKAGE_PYTHON_NUMPY
> >> +    select BR2_PACKAGE_OPENBLAS
> >> +    select BR2_PACKAGE_CLAPACK
> >
> > Python-scipy can be used with lapack instead of Clapack.
> > Clapack is not maintained any more and really out of date.
>
> Yes, but as we currently only supports clapack for python-numpy builds,
> It seems a bit odd to do it different for python-scipy.
>
> What about adding python-scipy first, and then extending both
> python-numpy and python-scipy to support lapack as (an up-to-date)
> alternative to clapack afterwards?
>
This is exatcly what I did. I propose a virtual package for lapack/clapack.
But after digging into lapack/clapack, we realise this is much harder than
just adding a virtual package.

>
> > OpenBlas is using a bundled version of Lapack when gFortran compiler
> > is available. Clapack is an old f2c'ed version of lapack, so it can be
> > used without a Fortran Compiler. Since Python-scipy depends on
> > Fortran, Clapack can be removed from python-scipy.
> >
> > Actually there is an existing issue (discussed during the Buildroot
> > hackathon) on *blas providers. We need to fix this before adding new
> > packages using *blas libraries :-/
>
> Ok, so where does this place this work on python-scipy for now?
> Is it a 100% no-go until *blas providers are reworked?
>
For me, yes, the first setp is to fix those blas providers.
There is libflame[1], a lapack implementation and Blis[2], a blas
implementation.
`libflame includes a compatibility layer, lapack2flame, which includes a
complete LAPACK implementation.`
`While BLIS exports a new BLAS-like API, it also includes a BLAS
compatibility layer which gives application developers access to BLIS
implementations via traditional BLAS routine calls.`

This have to be test but I guess it could replace lapack/clapack and blas
implementation.
I suggest you to read the previous work on lapack/clapack and blas[3].

>
> I think it would be a shame to block python-scipy, as it is pretty
> useful package, and the effort to take it in (looking at this patch
> series) is pretty minimal.
>
> /Esben
>

Cheers,
Alexandre.

[1] : https://github.com/flame/libflame
[2] : https://github.com/flame/blis
[3] :
http://buildroot-busybox.2317881.n4.nabble.com/PATCH-1-5-package-python-numpy-fix-occasional-build-and-run-time-failure-with-lapack-td228787.html#a228788
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20190923/c7880a74/attachment.html>

[Buildroot] [PATCH v2 3/3] package/python-scipy: new package

[Esben Haabendal]

Hi Alexandre

Alexandre PAYEN <alexandre.payen@smile.fr> writes:

>> Alexandre, do you have time to work on this?
>>
> I have some time if you need yes.

Great :)

>> >> +    select BR2_PACKAGE_PYTHON_NUMPY
>> >> +    select BR2_PACKAGE_OPENBLAS
>> >> +    select BR2_PACKAGE_CLAPACK
>> >
>> > Python-scipy can be used with lapack instead of Clapack.
>> > Clapack is not maintained any more and really out of date.
>>
>> Yes, but as we currently only supports clapack for python-numpy builds,
>> It seems a bit odd to do it different for python-scipy.
>>
>> What about adding python-scipy first, and then extending both
>> python-numpy and python-scipy to support lapack as (an up-to-date)
>> alternative to clapack afterwards?
>>
> This is exatcly what I did. I propose a virtual package for lapack/clapack.
> But after digging into lapack/clapack, we realise this is much harder than
> just adding a virtual package.

What is harder, adding python-scipy first?

If so, I think you might been hindered by a cross-compilation unfriendly
mechanism in numpy distutils, which I have got a fix into numpy for.
Please take a look at 
https://github.com/numpy/numpy/pull/14495/commits/153fc148eec60e5cbec0e80617f75a3a5dd2a3f8

and how I use that in my python-scipy recipe sent to this list.

With that, I think the python-scipy recipe looks pretty simple and
innocent, and I don't think that merging it should be a problem for
reworking lapack/clpack/*blas later on.

>> > OpenBlas is using a bundled version of Lapack when gFortran compiler
>> > is available. Clapack is an old f2c'ed version of lapack, so it can be
>> > used without a Fortran Compiler. Since Python-scipy depends on
>> > Fortran, Clapack can be removed from python-scipy.
>> >
>> > Actually there is an existing issue (discussed during the Buildroot
>> > hackathon) on *blas providers. We need to fix this before adding new
>> > packages using *blas libraries :-/
>>
>> Ok, so where does this place this work on python-scipy for now?
>> Is it a 100% no-go until *blas providers are reworked?
>>
> For me, yes, the first setp is to fix those blas providers.

Ok, but wouldn't it help having the python-scipy in here to have a good
use-case for working with all this?

> There is libflame[1], a lapack implementation and Blis[2], a blas
> implementation.
> `libflame includes a compatibility layer, lapack2flame, which includes a
> complete LAPACK implementation.`
> `While BLIS exports a new BLAS-like API, it also includes a BLAS
> compatibility layer which gives application developers access to BLIS
> implementations via traditional BLAS routine calls.`
>
> This have to be test but I guess it could replace lapack/clapack and blas
> implementation.
> I suggest you to read the previous work on lapack/clapack and blas[3].

I am not at all a lapack user/developer, just trying to build something
with it, so please bear with me.

You write that the python-scipy requires lapack (and thus implicitl that
it does not work with clapack).  Why is that?  It seems to WORKSFORME,
so what am I missing?

>> I think it would be a shame to block python-scipy, as it is pretty
>> useful package, and the effort to take it in (looking at this patch
>> series) is pretty minimal.

> [3] :
> http://buildroot-busybox.2317881.n4.nabble.com/PATCH-1-5-package-python-numpy-fix-occasional-build-and-run-time-failure-with-lapack-td228787.html#a228788


/Esben

[Buildroot] [PATCH v2 3/3] package/python-scipy: new package

[Alexandre PAYEN]

Le lun. 23 sept. 2019 ? 10:51, Esben Haabendal <esben@geanix.com> a ?crit :

> Hi Alexandre
>
> Alexandre PAYEN <alexandre.payen@smile.fr> writes:
>
> >> Alexandre, do you have time to work on this?
> >>
> > I have some time if you need yes.
>
> Great :)
>
> >> >> +    select BR2_PACKAGE_PYTHON_NUMPY
> >> >> +    select BR2_PACKAGE_OPENBLAS
> >> >> +    select BR2_PACKAGE_CLAPACK
> >> >
> >> > Python-scipy can be used with lapack instead of Clapack.
> >> > Clapack is not maintained any more and really out of date.
> >>
> >> Yes, but as we currently only supports clapack for python-numpy builds,
> >> It seems a bit odd to do it different for python-scipy.
> >>
> >> What about adding python-scipy first, and then extending both
> >> python-numpy and python-scipy to support lapack as (an up-to-date)
> >> alternative to clapack afterwards?
> >>
> > This is exatcly what I did. I propose a virtual package for
> lapack/clapack.
> > But after digging into lapack/clapack, we realise this is much harder
> than
> > just adding a virtual package.
>
> What is harder, adding python-scipy first?
>
No, lapack/clapack and blas.

>
> If so, I think you might been hindered by a cross-compilation unfriendly
> mechanism in numpy distutils, which I have got a fix into numpy for.
> Please take a look at
>
> https://github.com/numpy/numpy/pull/14495/commits/153fc148eec60e5cbec0e80617f75a3a5dd2a3f8
>
> and how I use that in my python-scipy recipe sent to this list.
>
> With that, I think the python-scipy recipe looks pretty simple and
> innocent, and I don't think that merging it should be a problem for
> reworking lapack/clpack/*blas later on.

Yes, there was a cross-compilation problem. But we fix it poorly, I assume
your commit do the job.
I think adding python-scipy is not relevant because we don't understand
lapack/clapack and blas properly.
We need to fix blas/lapack before adding packages which uses it.

>
> >> > OpenBlas is using a bundled version of Lapack when gFortran compiler
> >> > is available. Clapack is an old f2c'ed version of lapack, so it can be
> >> > used without a Fortran Compiler. Since Python-scipy depends on
> >> > Fortran, Clapack can be removed from python-scipy.
> >> >
> >> > Actually there is an existing issue (discussed during the Buildroot
> >> > hackathon) on *blas providers. We need to fix this before adding new
> >> > packages using *blas libraries :-/
> >>
> >> Ok, so where does this place this work on python-scipy for now?
> >> Is it a 100% no-go until *blas providers are reworked?
> >>
> > For me, yes, the first setp is to fix those blas providers.
>
> Ok, but wouldn't it help having the python-scipy in here to have a good
> use-case for working with all this?
>
Yes, why not.
But python-numpy is already present. It's help testing lapack with or
without complex support.

>
> > There is libflame[1], a lapack implementation and Blis[2], a blas
> > implementation.
> > `libflame includes a compatibility layer, lapack2flame, which includes a
> > complete LAPACK implementation.`
> > `While BLIS exports a new BLAS-like API, it also includes a BLAS
> > compatibility layer which gives application developers access to BLIS
> > implementations via traditional BLAS routine calls.`
> >
> > This have to be test but I guess it could replace lapack/clapack and blas
> > implementation.
> > I suggest you to read the previous work on lapack/clapack and blas[3].
>
> I am not at all a lapack user/developer, just trying to build something
> with it, so please bear with me.
>
No one really use lapack, python-scipy use it, by building python-scipy,
you kind of become a lapack user.

>
> You write that the python-scipy requires lapack (and thus implicitl that
> it does not work with clapack).  Why is that?  It seems to WORKSFORME,
> so what am I missing?
>
No, clapack is just a C implementation of lapack. It had been generated in
C from fortran code with f2c.
Clapack is not maintained any more as lapack but there are still old. Also,
lapack need a fortran compiler as python-numpy/python-scipy then you should
use lapack instead of clapack.

>
> >> I think it would be a shame to block python-scipy, as it is pretty
> >> useful package, and the effort to take it in (looking at this patch
> >> series) is pretty minimal.
>
> > [3] :
> >
> http://buildroot-busybox.2317881.n4.nabble.com/PATCH-1-5-package-python-numpy-fix-occasional-build-and-run-time-failure-with-lapack-td228787.html#a228788
>
>
> /Esben
>

Alexandre.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20190923/58ecd5bd/attachment.html>

[Buildroot] [PATCH v2 3/3] package/python-scipy: new package

[Esben Haabendal]

Hi Alexandre

Alexandre PAYEN <alexandre.payen@smile.fr> writes:

> Le lun. 23 sept. 2019 ? 10:51, Esben Haabendal <esben@geanix.com> a ?crit :
>
>> Alexandre PAYEN <alexandre.payen@smile.fr> writes:
>>
>>> This is exatcly what I did. I propose a virtual package for
>>> lapack/clapack.  But after digging into lapack/clapack, we realise
>>> this is much harder than just adding a virtual package.
>>
>> What is harder, adding python-scipy first?
>>
> No, lapack/clapack and blas.

Ok.  So while delaying addition of python-scipy with a relatively simple
virtual package change, we now have a much harder task at hand.

>> If so, I think you might been hindered by a cross-compilation unfriendly
>> mechanism in numpy distutils, which I have got a fix into numpy for.
>> Please take a look at
>>
>> https://github.com/numpy/numpy/pull/14495/commits/153fc148eec60e5cbec0e80617f75a3a5dd2a3f8
>>
>> and how I use that in my python-scipy recipe sent to this list.
>>
>> With that, I think the python-scipy recipe looks pretty simple and
>> innocent, and I don't think that merging it should be a problem for
>> reworking lapack/clpack/*blas later on.
>
> Yes, there was a cross-compilation problem. But we fix it poorly, I assume
> your commit do the job.

I believe it does.

> I think adding python-scipy is not relevant because we don't understand
> lapack/clapack and blas properly.

Well, addition of python-scipy is relevant for me, as I need to have
support for it.  It is a firm product requirement, so the lack of it
requires me to carry it in an overlay or to maintain patches for it.

> We need to fix blas/lapack before adding packages which uses it.

It might be old, but clapack seems to do the job for python-scipy.  So I
don't see that we *need* to do something to blas/lapack before adding
python-scipy, but it definitely sounds like it would be good to do
something about it, with or without python-scipy.

>> >> > OpenBlas is using a bundled version of Lapack when gFortran compiler
>> >> > is available. Clapack is an old f2c'ed version of lapack, so it can be
>> >> > used without a Fortran Compiler. Since Python-scipy depends on
>> >> > Fortran, Clapack can be removed from python-scipy.
>> >> >
>> >> > Actually there is an existing issue (discussed during the Buildroot
>> >> > hackathon) on *blas providers. We need to fix this before adding new
>> >> > packages using *blas libraries :-/
>> >>
>> >> Ok, so where does this place this work on python-scipy for now?
>> >> Is it a 100% no-go until *blas providers are reworked?
>> >>
>> > For me, yes, the first setp is to fix those blas providers.
>>
>> Ok, but wouldn't it help having the python-scipy in here to have a good
>> use-case for working with all this?
>
> Yes, why not.

Ok, let's try to get a simple and clean python-scipy recipe in.

>> > There is libflame[1], a lapack implementation and Blis[2], a blas
>> > implementation.
>> > `libflame includes a compatibility layer, lapack2flame, which includes a
>> > complete LAPACK implementation.`
>> > `While BLIS exports a new BLAS-like API, it also includes a BLAS
>> > compatibility layer which gives application developers access to BLIS
>> > implementations via traditional BLAS routine calls.`
>> >
>> > This have to be test but I guess it could replace lapack/clapack and blas
>> > implementation.
>> > I suggest you to read the previous work on lapack/clapack and blas[3].
>>
>> I am not at all a lapack user/developer, just trying to build something
>> with it, so please bear with me.
>>
> No one really use lapack, python-scipy use it, by building python-scipy,
> you kind of become a lapack user.

True.  But, but it does not make me any more knowledgeable of the API or
internals.

>> You write that the python-scipy requires lapack (and thus implicitl that
>> it does not work with clapack).  Why is that?  It seems to WORKSFORME,
>> so what am I missing?
>>
> No, clapack is just a C implementation of lapack. It had been generated in
> C from fortran code with f2c.
> Clapack is not maintained any more as lapack but there are still old.

I am with you so far.

> Also, lapack need a fortran compiler as python-numpy/python-scipy then
> you should use lapack instead of clapack.

Well, I don't *have* to use lapack instead of clapack.  I could, and you
could argue I should.

But with this patch series, buildroot can build python-numpy and
python-scipy using clapack, for a working numpy/scipy enabled rootfs.
That sounds quite useful to me.

If I send a slightly modified version of this patch series, will you
give it a review and see if we can agree on a form that can get merged
now?

/Esben

[Buildroot] [PATCH v2 3/3] package/python-scipy: new package

[Alexandre PAYEN]

Hi Esben,

Le lun. 23 sept. 2019 ? 14:46, Esben Haabendal <esben@geanix.com> a ?crit :

> Hi Alexandre
>
> Alexandre PAYEN <alexandre.payen@smile.fr> writes:
>
> > Le lun. 23 sept. 2019 ? 10:51, Esben Haabendal <esben@geanix.com> a
> ?crit :
> >
> >> Alexandre PAYEN <alexandre.payen@smile.fr> writes:
> >>
> >>> This is exatcly what I did. I propose a virtual package for
> >>> lapack/clapack.  But after digging into lapack/clapack, we realise
> >>> this is much harder than just adding a virtual package.
> >>
> >> What is harder, adding python-scipy first?
> >>
> > No, lapack/clapack and blas.
>
> Ok.  So while delaying addition of python-scipy with a relatively simple
> virtual package change, we now have a much harder task at hand.
>
> >> If so, I think you might been hindered by a cross-compilation unfriendly
> >> mechanism in numpy distutils, which I have got a fix into numpy for.
> >> Please take a look at
> >>
> >>
> https://github.com/numpy/numpy/pull/14495/commits/153fc148eec60e5cbec0e80617f75a3a5dd2a3f8
> >>
> >> and how I use that in my python-scipy recipe sent to this list.
> >>
> >> With that, I think the python-scipy recipe looks pretty simple and
> >> innocent, and I don't think that merging it should be a problem for
> >> reworking lapack/clpack/*blas later on.
> >
> > Yes, there was a cross-compilation problem. But we fix it poorly, I
> assume
> > your commit do the job.
>
> I believe it does.
>
> > I think adding python-scipy is not relevant because we don't understand
> > lapack/clapack and blas properly.
>
> Well, addition of python-scipy is relevant for me, as I need to have
> support for it.  It is a firm product requirement, so the lack of it
> requires me to carry it in an overlay or to maintain patches for it.
>
> > We need to fix blas/lapack before adding packages which uses it.
>
> It might be old, but clapack seems to do the job for python-scipy.  So I
> don't see that we *need* to do something to blas/lapack before adding
> python-scipy, but it definitely sounds like it would be good to do
> something about it, with or without python-scipy.
>
> >> >> > OpenBlas is using a bundled version of Lapack when gFortran
> compiler
> >> >> > is available. Clapack is an old f2c'ed version of lapack, so it
> can be
> >> >> > used without a Fortran Compiler. Since Python-scipy depends on
> >> >> > Fortran, Clapack can be removed from python-scipy.
> >> >> >
> >> >> > Actually there is an existing issue (discussed during the Buildroot
> >> >> > hackathon) on *blas providers. We need to fix this before adding
> new
> >> >> > packages using *blas libraries :-/
> >> >>
> >> >> Ok, so where does this place this work on python-scipy for now?
> >> >> Is it a 100% no-go until *blas providers are reworked?
> >> >>
> >> > For me, yes, the first setp is to fix those blas providers.
> >>
> >> Ok, but wouldn't it help having the python-scipy in here to have a good
> >> use-case for working with all this?
> >
> > Yes, why not.
>
> Ok, let's try to get a simple and clean python-scipy recipe in.
>
> >> > There is libflame[1], a lapack implementation and Blis[2], a blas
> >> > implementation.
> >> > `libflame includes a compatibility layer, lapack2flame, which
> includes a
> >> > complete LAPACK implementation.`
> >> > `While BLIS exports a new BLAS-like API, it also includes a BLAS
> >> > compatibility layer which gives application developers access to BLIS
> >> > implementations via traditional BLAS routine calls.`
> >> >
> >> > This have to be test but I guess it could replace lapack/clapack and
> blas
> >> > implementation.
> >> > I suggest you to read the previous work on lapack/clapack and blas[3].
> >>
> >> I am not at all a lapack user/developer, just trying to build something
> >> with it, so please bear with me.
> >>
> > No one really use lapack, python-scipy use it, by building python-scipy,
> > you kind of become a lapack user.
>
> True.  But, but it does not make me any more knowledgeable of the API or
> internals.
>
> >> You write that the python-scipy requires lapack (and thus implicitl that
> >> it does not work with clapack).  Why is that?  It seems to WORKSFORME,
> >> so what am I missing?
> >>
> > No, clapack is just a C implementation of lapack. It had been generated
> in
> > C from fortran code with f2c.
> > Clapack is not maintained any more as lapack but there are still old.
>
> I am with you so far.
>
> > Also, lapack need a fortran compiler as python-numpy/python-scipy then
> > you should use lapack instead of clapack.
>
> Well, I don't *have* to use lapack instead of clapack.  I could, and you
> could argue I should.
>
> But with this patch series, buildroot can build python-numpy and
> python-scipy using clapack, for a working numpy/scipy enabled rootfs.
> That sounds quite useful to me.
>
> If I send a slightly modified version of this patch series, will you
> give it a review and see if we can agree on a form that can get merged
> now?
>
Please, proceed, I will review it.

>
> /Esben
>

Alexandre
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20190923/0586da5a/attachment.html>

[Buildroot] [PATCH v2 1/3] package/python-numpy: fix setup type

[Arnout Vandecappelle]

On 23/09/2019 08:45, Esben Haabendal wrote:
> Hi Yegor
> 
> Yegor Yefremov <yegorslists@googlemail.com> writes:
> 
>> On Fri, Sep 20, 2019 at 6:37 PM Esben Haabendal <esben@geanix.com> wrote:
>>>
>>> The numpy package is using distutils, not setuptools.
>>
>> Are you sure about this? See
>> https://github.com/numpy/numpy/blob/master/numpy/distutils/core.py
> 
> https://github.com/numpy/numpy/blob/v1.16.4/setup.py#L392
> 
> Looks like a setuptools package to me.

 Exactly, and that's how it is defined in Buildroot now. Changing it to
distutils would be wrong, no?

 Regards,
 Arnout

[Buildroot] [PATCH v2 1/3] package/python-numpy: fix setup type

[Esben Haabendal]

Arnout Vandecappelle <arnout@mind.be> writes:

> On 23/09/2019 08:45, Esben Haabendal wrote:
>> Hi Yegor
>> 
>> Yegor Yefremov <yegorslists@googlemail.com> writes:
>> 
>>> On Fri, Sep 20, 2019 at 6:37 PM Esben Haabendal <esben@geanix.com> wrote:
>>>>
>>>> The numpy package is using distutils, not setuptools.
>>>
>>> Are you sure about this? See
>>> https://github.com/numpy/numpy/blob/master/numpy/distutils/core.py
>> 
>> https://github.com/numpy/numpy/blob/v1.16.4/setup.py#L392
>> 
>> Looks like a setuptools package to me.
>
>  Exactly, and that's how it is defined in Buildroot now. Changing it to
> distutils would be wrong, no?

You are absolutely right.  EOUTOFCOFFEE.

/Esben