[Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1

[Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1

[guillaume.bressaix]

From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com>

The python 'setup.py' script needs header files
in $(@D)/pybind11 to work since v2.6.1, and these
files are generated by an internal minimalist cmake build.

Fixes
   http://autobuild.buildroot.net/results/b89f1de64b308dffa73675f1f31ccb0b7be5a10d
   http://autobuild.buildroot.net/results/d0287b7f64f206b0f074908c5780a3632e0cb799
   http://autobuild.buildroot.net/results/27efb545a5a719a5581c8f746d3a3555ff4216ce
   http://autobuild.buildroot.net/results/d2f0a0ad8f6c7178517df109e7d885dac9134c3a
   http://autobuild.buildroot.net/results/b57e9a3279260dae4a590f9421238fcabb2f7cab
   http://autobuild.buildroot.net/results/515e6f2fc6b5780260d98d6bb52b541ce4bf1afe
   http://autobuild.buildroot.net/results/d89c4ecc81222d4f80c951da2232d2e393fa1c69

Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com>

---
This remains a python-package.

'python setup.py' actually hardcodes a system call to cmake now,
which seems dirty at first. But if we have a cmake env when they
do that, the install variables are passed and installation
is smooth. They probably introduced this call to make setup.py
self sufficient, as most people install this package from 'pip'.

---
 package/python-pybind/python-pybind.mk | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/package/python-pybind/python-pybind.mk b/package/python-pybind/python-pybind.mk
index a6a1bdb976..bfd7f6f59a 100644
--- a/package/python-pybind/python-pybind.mk
+++ b/package/python-pybind/python-pybind.mk
@@ -9,5 +9,12 @@ PYTHON_PYBIND_SITE = $(call github,pybind,pybind11,v$(PYTHON_PYBIND_VERSION))
 PYTHON_PYBIND_LICENSE = BSD-3-Clause
 PYTHON_PYBIND_LICENSE_FILES = LICENSE
 PYTHON_PYBIND_SETUP_TYPE = setuptools
+PYTHON_PYBIND_INSTALL_STAGING = YES
+
+# every single 'python setup.py' call actually calls cmake
+# internally, to populate $(@D)/pybind11 with build requirements.
+# If we have a host-cmake env at that moment, 
+# then the final installation paths are properly defined
+PYTHON_PYBIND_DEPENDENCIES = host-cmake
 
 $(eval $(python-package))
-- 
2.20.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH v2 2/2] package/python-pybind: bump to version 2.8.1

[guillaume.bressaix]

From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com>

Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com>
---
 package/python-pybind/python-pybind.hash | 2 +-
 package/python-pybind/python-pybind.mk   | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/package/python-pybind/python-pybind.hash b/package/python-pybind/python-pybind.hash
index a68ac846e2..e1d86a4494 100644
--- a/package/python-pybind/python-pybind.hash
+++ b/package/python-pybind/python-pybind.hash
@@ -1,4 +1,4 @@
 # Locally calculated
-sha256  cdbe326d357f18b83d10322ba202d69f11b2f49e2d87ade0dc2be0c5c34f8e2a  python-pybind-2.6.1.tar.gz
+sha256  f1bcc07caa568eb312411dde5308b1e250bd0e1bc020fae855bf9f43209940cc  python-pybind-2.8.1.tar.gz
 # License files, locally calculated
 sha256  83965b843b98f670d3a85bd041ed4b372c8ec50d7b4a5995a83ac697ba675dcb  LICENSE
diff --git a/package/python-pybind/python-pybind.mk b/package/python-pybind/python-pybind.mk
index bfd7f6f59a..3aee9cf1e6 100644
--- a/package/python-pybind/python-pybind.mk
+++ b/package/python-pybind/python-pybind.mk
@@ -4,7 +4,7 @@
 #
 ################################################################################
 
-PYTHON_PYBIND_VERSION = 2.6.1
+PYTHON_PYBIND_VERSION = 2.8.1
 PYTHON_PYBIND_SITE = $(call github,pybind,pybind11,v$(PYTHON_PYBIND_VERSION))
 PYTHON_PYBIND_LICENSE = BSD-3-Clause
 PYTHON_PYBIND_LICENSE_FILES = LICENSE
-- 
2.20.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1

[Yann E. MORIN]

Guillaume, All,

On 2021-12-06 20:31 +0100, guillaume.bressaix@gmail.com spake thusly:
> From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com>
> 
> The python 'setup.py' script needs header files
> in $(@D)/pybind11 to work since v2.6.1, and these
> files are generated by an internal minimalist cmake build.
> 
> Fixes
>    http://autobuild.buildroot.net/results/b89f1de64b308dffa73675f1f31ccb0b7be5a10d
>    http://autobuild.buildroot.net/results/d0287b7f64f206b0f074908c5780a3632e0cb799
>    http://autobuild.buildroot.net/results/27efb545a5a719a5581c8f746d3a3555ff4216ce
>    http://autobuild.buildroot.net/results/d2f0a0ad8f6c7178517df109e7d885dac9134c3a
>    http://autobuild.buildroot.net/results/b57e9a3279260dae4a590f9421238fcabb2f7cab
>    http://autobuild.buildroot.net/results/515e6f2fc6b5780260d98d6bb52b541ce4bf1afe
>    http://autobuild.buildroot.net/results/d89c4ecc81222d4f80c951da2232d2e393fa1c69
> 
> Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com>
> 
> ---
> This remains a python-package.
> 
> 'python setup.py' actually hardcodes a system call to cmake now,
> which seems dirty at first. But if we have a cmake env when they
> do that, the install variables are passed and installation
> is smooth. They probably introduced this call to make setup.py
> self sufficient, as most people install this package from 'pip'.
> 
> ---
>  package/python-pybind/python-pybind.mk | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/package/python-pybind/python-pybind.mk b/package/python-pybind/python-pybind.mk
> index a6a1bdb976..bfd7f6f59a 100644
> --- a/package/python-pybind/python-pybind.mk
> +++ b/package/python-pybind/python-pybind.mk
> @@ -9,5 +9,12 @@ PYTHON_PYBIND_SITE = $(call github,pybind,pybind11,v$(PYTHON_PYBIND_VERSION))
>  PYTHON_PYBIND_LICENSE = BSD-3-Clause
>  PYTHON_PYBIND_LICENSE_FILES = LICENSE
>  PYTHON_PYBIND_SETUP_TYPE = setuptools
> +PYTHON_PYBIND_INSTALL_STAGING = YES
> +
> +# every single 'python setup.py' call actually calls cmake
> +# internally, to populate $(@D)/pybind11 with build requirements.
> +# If we have a host-cmake env at that moment, 
> +# then the final installation paths are properly defined
> +PYTHON_PYBIND_DEPENDENCIES = host-cmake

We do not hard-code a dependency to host-cmake, in case the system cmake
is sufficient. Instead, we use $(BR2_CMAKE_HOST_DEPENDENCY) which is set
appropriately.

Regards,
Yann E. MORIN.

>  $(eval $(python-package))
> -- 
> 2.20.1
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1

[Yann E. MORIN]

Guillaume, All,

On 2021-12-06 20:31 +0100, guillaume.bressaix@gmail.com spake thusly:
> From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com>
> 
> The python 'setup.py' script needs header files
> in $(@D)/pybind11 to work since v2.6.1, and these
> files are generated by an internal minimalist cmake build.
> 
> Fixes
>    http://autobuild.buildroot.net/results/b89f1de64b308dffa73675f1f31ccb0b7be5a10d
>    http://autobuild.buildroot.net/results/d0287b7f64f206b0f074908c5780a3632e0cb799
>    http://autobuild.buildroot.net/results/27efb545a5a719a5581c8f746d3a3555ff4216ce
>    http://autobuild.buildroot.net/results/d2f0a0ad8f6c7178517df109e7d885dac9134c3a
>    http://autobuild.buildroot.net/results/b57e9a3279260dae4a590f9421238fcabb2f7cab
>    http://autobuild.buildroot.net/results/515e6f2fc6b5780260d98d6bb52b541ce4bf1afe
>    http://autobuild.buildroot.net/results/d89c4ecc81222d4f80c951da2232d2e393fa1c69
> 
> Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com>
> 
> ---
> This remains a python-package.
> 
> 'python setup.py' actually hardcodes a system call to cmake now,
> which seems dirty at first. But if we have a cmake env when they
> do that, the install variables are passed and installation
> is smooth. They probably introduced this call to make setup.py
> self sufficient, as most people install this package from 'pip'.
> 
> ---
>  package/python-pybind/python-pybind.mk | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/package/python-pybind/python-pybind.mk b/package/python-pybind/python-pybind.mk
> index a6a1bdb976..bfd7f6f59a 100644
> --- a/package/python-pybind/python-pybind.mk
> +++ b/package/python-pybind/python-pybind.mk
> @@ -9,5 +9,12 @@ PYTHON_PYBIND_SITE = $(call github,pybind,pybind11,v$(PYTHON_PYBIND_VERSION))
>  PYTHON_PYBIND_LICENSE = BSD-3-Clause
>  PYTHON_PYBIND_LICENSE_FILES = LICENSE
>  PYTHON_PYBIND_SETUP_TYPE = setuptools
> +PYTHON_PYBIND_INSTALL_STAGING = YES
> +
> +# every single 'python setup.py' call actually calls cmake
> +# internally, to populate $(@D)/pybind11 with build requirements.
> +# If we have a host-cmake env at that moment, 
> +# then the final installation paths are properly defined
> +PYTHON_PYBIND_DEPENDENCIES = host-cmake

I am sorry, but I fail to see how this actually fixes things.

As I explained in my previous review, headers are supposed to be in
$(STAGING_DIR)/usr/include/pybind11/. That's at least where they are in
Debian-based distributions;

    $ apt-file search 'pybind11/pybind11.h'
    pybind11-dev: /usr/include/pybind11/pybind11.h

However, with the way pybind is currently packaged, the ehaders end up
in:

    $ sort output/build/python-pybind-2.8.1/.files-list.txt
    python-pybind,./usr/bin/pybind11-config
    [...]
    python-pybind,./usr/lib/python3.9/site-packages/pybind11/include/pybind11/pybind11.h
    [...]

So, those headers will most probably not be found by the compiler.

Also, since pybind is supposed to be a headers'-only library, there is
nothing supposed to go in target, so it should be;

    PYTHON_PYBIND_INSTALL_TARGET = NO

Furthermore, /usr/bin/pybind11-config is a python script which hardcodes
its shabang to #!/usr/bin/python, and this is incorrect:

  1. /usr/bin/python might be python2 or python3, we can't know
  2. anyway, it would not use our host python, so it would miss the
    proper modules search paths.

So, if another package want to call that /usr/bin/pybind11-config to
find pybind, they're gonna have incorrect results, if at all...

So I think we really, really need a package (like scipy) that actually
makes use of pybind, so we can see how it works, and to fix it...

Regards,
Yann E. MORIN.

>  $(eval $(python-package))
> -- 
> 2.20.1
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1

[Arnout Vandecappelle]

On 06/12/2021 22:57, Yann E. MORIN wrote:
> Guillaume, All,
> 
> On 2021-12-06 20:31 +0100, guillaume.bressaix@gmail.com spake thusly:
>> From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com>
>>
>> The python 'setup.py' script needs header files
>> in $(@D)/pybind11 to work since v2.6.1, and these
>> files are generated by an internal minimalist cmake build.
>>
>> Fixes
>>     http://autobuild.buildroot.net/results/b89f1de64b308dffa73675f1f31ccb0b7be5a10d
>>     http://autobuild.buildroot.net/results/d0287b7f64f206b0f074908c5780a3632e0cb799
>>     http://autobuild.buildroot.net/results/27efb545a5a719a5581c8f746d3a3555ff4216ce
>>     http://autobuild.buildroot.net/results/d2f0a0ad8f6c7178517df109e7d885dac9134c3a
>>     http://autobuild.buildroot.net/results/b57e9a3279260dae4a590f9421238fcabb2f7cab
>>     http://autobuild.buildroot.net/results/515e6f2fc6b5780260d98d6bb52b541ce4bf1afe
>>     http://autobuild.buildroot.net/results/d89c4ecc81222d4f80c951da2232d2e393fa1c69
>>
>> Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com>
>>
>> ---
>> This remains a python-package.
>>
>> 'python setup.py' actually hardcodes a system call to cmake now,
>> which seems dirty at first. But if we have a cmake env when they
>> do that, the install variables are passed and installation
>> is smooth. They probably introduced this call to make setup.py
>> self sufficient, as most people install this package from 'pip'.
>>
>> ---
>>   package/python-pybind/python-pybind.mk | 7 +++++++
>>   1 file changed, 7 insertions(+)
>>
>> diff --git a/package/python-pybind/python-pybind.mk b/package/python-pybind/python-pybind.mk
>> index a6a1bdb976..bfd7f6f59a 100644
>> --- a/package/python-pybind/python-pybind.mk
>> +++ b/package/python-pybind/python-pybind.mk
>> @@ -9,5 +9,12 @@ PYTHON_PYBIND_SITE = $(call github,pybind,pybind11,v$(PYTHON_PYBIND_VERSION))
>>   PYTHON_PYBIND_LICENSE = BSD-3-Clause
>>   PYTHON_PYBIND_LICENSE_FILES = LICENSE
>>   PYTHON_PYBIND_SETUP_TYPE = setuptools
>> +PYTHON_PYBIND_INSTALL_STAGING = YES
>> +
>> +# every single 'python setup.py' call actually calls cmake
>> +# internally, to populate $(@D)/pybind11 with build requirements.
>> +# If we have a host-cmake env at that moment,
>> +# then the final installation paths are properly defined
>> +PYTHON_PYBIND_DEPENDENCIES = host-cmake
> 
> I am sorry, but I fail to see how this actually fixes things.
> 
> As I explained in my previous review, headers are supposed to be in
> $(STAGING_DIR)/usr/include/pybind11/. That's at least where they are in
> Debian-based distributions;
> 
>      $ apt-file search 'pybind11/pybind11.h'
>      pybind11-dev: /usr/include/pybind11/pybind11.h
> 
> However, with the way pybind is currently packaged, the ehaders end up
> in:
> 
>      $ sort output/build/python-pybind-2.8.1/.files-list.txt
>      python-pybind,./usr/bin/pybind11-config
>      [...]
>      python-pybind,./usr/lib/python3.9/site-packages/pybind11/include/pybind11/pybind11.h
>      [...]
> 
> So, those headers will most probably not be found by the compiler.
> 
> Also, since pybind is supposed to be a headers'-only library, there is
> nothing supposed to go in target, so it should be;
> 
>      PYTHON_PYBIND_INSTALL_TARGET = NO

  Note that this is only relevant to make sure that pybind11-config doesn't 
appear on target. The headers themselves are removed automatically by Buildroot.

> 
> Furthermore, /usr/bin/pybind11-config is a python script which hardcodes
> its shabang to #!/usr/bin/python, and this is incorrect:
> 
>    1. /usr/bin/python might be python2 or python3, we can't know
>    2. anyway, it would not use our host python, so it would miss the
>      proper modules search paths.

  Yeah, this is something that needs to be fixed - in particular, there may not 
be a /usr/bin/python at all.

  However, I think that is kind of orthogonal to this patch. This patch fixes a 
build failure of pybind itself. Your proposed change would be to fix the build 
of a dependency of pybind.


> So, if another package want to call that /usr/bin/pybind11-config to
> find pybind, they're gonna have incorrect results, if at all...
> 
> So I think we really, really need a package (like scipy) that actually
> makes use of pybind, so we can see how it works, and to fix it...

  Note that there are already two [1][2] submissions of scipy in patchwork, so 
feel free to review those.

  Note also that scipy depends on host-pybind, not pybind itself. I'm not sure 
how *that* is supposed to work - although, being header-only, it probably 
doesn't really matter if you link with the host or the target version of the 
library, and the host-installed pybind11-config makes sure that you pick up the 
host headers (which are normally not in include path). So conceptually wrong but 
works in practice.


  In conclusion: I think this patch is going in the right direction (modulo 
BR2_CMAKE_HOST_DEPENDENCY). It's still "conceptually wrong but works" because it 
doesn't set the toolchain file and other cmake options, but that happens to work 
because it's header-only so it doesn't actually need any of the cmake options. 
So it's not entirely the correct thing to do, but it does guarantee to fix 
things. I'm not sure if that means we should merge it or not...

  The proper fix, I think, would require patching setup.py so we can pass in 
additional cmake options to it.


  Regards,
  Arnout

[1] https://patchwork.ozlabs.org/project/buildroot/list/?series=168992
[2] https://patchwork.ozlabs.org/project/buildroot/list/?series=186146

> 
> Regards,
> Yann E. MORIN.
> 
>>   $(eval $(python-package))
>> -- 
>> 2.20.1
>>
> 
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1

[Arnout Vandecappelle]

On 06/12/2021 22:45, Yann E. MORIN wrote:
> Guillaume, All,
> 
> On 2021-12-06 20:31 +0100, guillaume.bressaix@gmail.com spake thusly:
>> From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com>
>>
>> The python 'setup.py' script needs header files
>> in $(@D)/pybind11 to work since v2.6.1, and these
>> files are generated by an internal minimalist cmake build.
>>
>> Fixes
>>     http://autobuild.buildroot.net/results/b89f1de64b308dffa73675f1f31ccb0b7be5a10d
>>     http://autobuild.buildroot.net/results/d0287b7f64f206b0f074908c5780a3632e0cb799
>>     http://autobuild.buildroot.net/results/27efb545a5a719a5581c8f746d3a3555ff4216ce
>>     http://autobuild.buildroot.net/results/d2f0a0ad8f6c7178517df109e7d885dac9134c3a
>>     http://autobuild.buildroot.net/results/b57e9a3279260dae4a590f9421238fcabb2f7cab
>>     http://autobuild.buildroot.net/results/515e6f2fc6b5780260d98d6bb52b541ce4bf1afe
>>     http://autobuild.buildroot.net/results/d89c4ecc81222d4f80c951da2232d2e393fa1c69
>>
>> Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com>
>>
>> ---
>> This remains a python-package.
>>
>> 'python setup.py' actually hardcodes a system call to cmake now,
>> which seems dirty at first. But if we have a cmake env when they
>> do that, the install variables are passed and installation
>> is smooth. They probably introduced this call to make setup.py
>> self sufficient, as most people install this package from 'pip'.
>>
>> ---
>>   package/python-pybind/python-pybind.mk | 7 +++++++
>>   1 file changed, 7 insertions(+)
>>
>> diff --git a/package/python-pybind/python-pybind.mk b/package/python-pybind/python-pybind.mk
>> index a6a1bdb976..bfd7f6f59a 100644
>> --- a/package/python-pybind/python-pybind.mk
>> +++ b/package/python-pybind/python-pybind.mk
>> @@ -9,5 +9,12 @@ PYTHON_PYBIND_SITE = $(call github,pybind,pybind11,v$(PYTHON_PYBIND_VERSION))
>>   PYTHON_PYBIND_LICENSE = BSD-3-Clause
>>   PYTHON_PYBIND_LICENSE_FILES = LICENSE
>>   PYTHON_PYBIND_SETUP_TYPE = setuptools
>> +PYTHON_PYBIND_INSTALL_STAGING = YES
>> +
>> +# every single 'python setup.py' call actually calls cmake
>> +# internally, to populate $(@D)/pybind11 with build requirements.
>> +# If we have a host-cmake env at that moment,
>> +# then the final installation paths are properly defined
>> +PYTHON_PYBIND_DEPENDENCIES = host-cmake
> 
> We do not hard-code a dependency to host-cmake, in case the system cmake
> is sufficient. Instead, we use $(BR2_CMAKE_HOST_DEPENDENCY) which is set
> appropriately.

  This turns out not to work in Guillaume's test setup.

  It turns out that he has a preinstalled cmake which is very old, but also 
cmake3 which is recent enough. Therefore, BR2_CMAKE is set to /usr/bin/cmake3 
and BR2_CMAKE_HOST_DEPENDENCY is empty. However, setup.py hardcodes "cmake".

  So, possible solutions:

- Forget about setup.py and instead hack it into a cmake-package. Not ideal from 
a maintenance perspective, because later updates may have changes to setup.py 
that we need to replicate.

- Patch setup.py so we can tell it to use BR2_CMAKE instead of cmake-in-path. 
This is nice because we can use the occasion to also pass in the rest of the 
cmake options for proper cross-compilation.

- In host-skeleton, symlink BR2_CMAKE to host/bin/cmake, so that will be the 
'cmake' that is found in $PATH. This has the advantage of also covering other 
potential cases of the same problem, but I feel it's a bit meh.

  So I have a slight preference for the second option, but Yann you may have 
other ideas.


  Regards,
  Arnout
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1

[Andreas Naumann]

Hi all,

On 07.12.21 09:04, Arnout Vandecappelle wrote:
> 
> 
> On 06/12/2021 22:57, Yann E. MORIN wrote:
>> Guillaume, All,
>>
>> On 2021-12-06 20:31 +0100, guillaume.bressaix@gmail.com spake thusly:
>>> From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com>
>>>
>>> The python 'setup.py' script needs header files
>>> in $(@D)/pybind11 to work since v2.6.1, and these
>>> files are generated by an internal minimalist cmake build.
>>>
>>> Fixes
>>>     
>>> http://autobuild.buildroot.net/results/b89f1de64b308dffa73675f1f31ccb0b7be5a10d 
>>>
>>>     
>>> http://autobuild.buildroot.net/results/d0287b7f64f206b0f074908c5780a3632e0cb799 
>>>
>>>     
>>> http://autobuild.buildroot.net/results/27efb545a5a719a5581c8f746d3a3555ff4216ce 
>>>
>>>     
>>> http://autobuild.buildroot.net/results/d2f0a0ad8f6c7178517df109e7d885dac9134c3a 
>>>
>>>     
>>> http://autobuild.buildroot.net/results/b57e9a3279260dae4a590f9421238fcabb2f7cab 
>>>
>>>     
>>> http://autobuild.buildroot.net/results/515e6f2fc6b5780260d98d6bb52b541ce4bf1afe 
>>>
>>>     
>>> http://autobuild.buildroot.net/results/d89c4ecc81222d4f80c951da2232d2e393fa1c69 
>>>
>>>
>>> Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com>
>>>
>>> ---
>>> This remains a python-package.
>>>
>>> 'python setup.py' actually hardcodes a system call to cmake now,
>>> which seems dirty at first. But if we have a cmake env when they
>>> do that, the install variables are passed and installation
>>> is smooth. They probably introduced this call to make setup.py
>>> self sufficient, as most people install this package from 'pip'.
>>>
>>> ---
>>>   package/python-pybind/python-pybind.mk | 7 +++++++
>>>   1 file changed, 7 insertions(+)
>>>
>>> diff --git a/package/python-pybind/python-pybind.mk 
>>> b/package/python-pybind/python-pybind.mk
>>> index a6a1bdb976..bfd7f6f59a 100644
>>> --- a/package/python-pybind/python-pybind.mk
>>> +++ b/package/python-pybind/python-pybind.mk
>>> @@ -9,5 +9,12 @@ PYTHON_PYBIND_SITE = $(call 
>>> github,pybind,pybind11,v$(PYTHON_PYBIND_VERSION))
>>>   PYTHON_PYBIND_LICENSE = BSD-3-Clause
>>>   PYTHON_PYBIND_LICENSE_FILES = LICENSE
>>>   PYTHON_PYBIND_SETUP_TYPE = setuptools
>>> +PYTHON_PYBIND_INSTALL_STAGING = YES
>>> +
>>> +# every single 'python setup.py' call actually calls cmake
>>> +# internally, to populate $(@D)/pybind11 with build requirements.
>>> +# If we have a host-cmake env at that moment,
>>> +# then the final installation paths are properly defined
>>> +PYTHON_PYBIND_DEPENDENCIES = host-cmake
>>
>> I am sorry, but I fail to see how this actually fixes things.
>>
>> As I explained in my previous review, headers are supposed to be in
>> $(STAGING_DIR)/usr/include/pybind11/. That's at least where they are in
>> Debian-based distributions;
>>
>>      $ apt-file search 'pybind11/pybind11.h'
>>      pybind11-dev: /usr/include/pybind11/pybind11.h
>>
>> However, with the way pybind is currently packaged, the ehaders end up
>> in:
>>
>>      $ sort output/build/python-pybind-2.8.1/.files-list.txt
>>      python-pybind,./usr/bin/pybind11-config
>>      [...]
>>      
>> python-pybind,./usr/lib/python3.9/site-packages/pybind11/include/pybind11/pybind11.h 
>>
>>      [...]
>>
>> So, those headers will most probably not be found by the compiler.
>>
>> Also, since pybind is supposed to be a headers'-only library, there is
>> nothing supposed to go in target, so it should be;
>>
>>      PYTHON_PYBIND_INSTALL_TARGET = NO
> 
>   Note that this is only relevant to make sure that pybind11-config 
> doesn't appear on target. The headers themselves are removed 
> automatically by Buildroot.
> 
>>
>> Furthermore, /usr/bin/pybind11-config is a python script which hardcodes
>> its shabang to #!/usr/bin/python, and this is incorrect:
>>
>>    1. /usr/bin/python might be python2 or python3, we can't know
>>    2. anyway, it would not use our host python, so it would miss the
>>      proper modules search paths.
> 
>   Yeah, this is something that needs to be fixed - in particular, there 
> may not be a /usr/bin/python at all.
> 
>   However, I think that is kind of orthogonal to this patch. This patch 
> fixes a build failure of pybind itself. Your proposed change would be to 
> fix the build of a dependency of pybind.
> 
> 
>> So, if another package want to call that /usr/bin/pybind11-config to
>> find pybind, they're gonna have incorrect results, if at all...
>>
>> So I think we really, really need a package (like scipy) that actually
>> makes use of pybind, so we can see how it works, and to fix it...

By chance I was playing with such a package just yesterday: zxing-cpp. 
Not the version buildroot uses (from github glassechidna), but the 
nu-book one which claims to be faster. (And is the one where the python 
bindings in the PyPi package zxing-cpp originate from).

Until recently they shipped their own pybind11 version, which throws an 
error when included in a cmake project in a crosscompile environment. 
This is a known problem, https://github.com/pybind/pybind11/issues/2139 
but non of the proposed fixes has been merged yet.

However, the unreleased master version of zxing-cpp hast been changed so 
it can be configured to use the a system pybind11. So I tried to include 
the br packaged one (2.6.1 as well as 2.8.1). They both built on my 
system (no problem with the hardcoded host cmake here) but I needed to 
change three things:
1. Add PYTHON_PYBIND_INSTALL_STAGING = YES
2. Set ZXING_CPP_CONF_OPTS += 
-Dpybind11_DIR=$(STAGING_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages/pybind11/share/cmake/pybind11
3. Apply one of the proposed fixes for the pybind11 crosscompile problem 
(to tools/pybind11Tools.cmake)

So this patch here seems to help my (and possibly others) because 1. is 
already included. If interested, I could send a patch for 3.


> 
>   Note that there are already two [1][2] submissions of scipy in 
> patchwork, so feel free to review those.
> 
>   Note also that scipy depends on host-pybind, not pybind itself. I'm 
> not sure how *that* is supposed to work - although, being header-only, 


One assumption why scipy includes the host version of pybind is that it 
nicely avoids the problems where I need fix 1. and 3. for.

> it probably doesn't really matter if you link with the host or the 
> target version of the library, and the host-installed pybind11-config 
> makes sure that you pick up the host headers (which are normally not in 
> include path). So conceptually wrong but works in practice.

For getting conceptually right with my zxing-cpp I would appreciate some 
advice: When compiling it with above modificiations and 
BUILD_PYTHON_MODULE=ON, cmake happily compiles the zxing pybind wrapper. 
But of course wrappers/python/setup.py would need to be run as well. 
However mixing $(eval $(python-package)) and $(eval $(cmake-package)) is 
not possible. So do I still need to create a second package 
python-zxingcpp for the bindings only (which might be a pain because the 
wrappers CMakeLists is not written to use an existant installation of 
zxing-cpp but rather downloads it again) ?


regards,
Andreas



> 
> 
>   In conclusion: I think this patch is going in the right direction 
> (modulo BR2_CMAKE_HOST_DEPENDENCY). It's still "conceptually wrong but 
> works" because it doesn't set the toolchain file and other cmake 
> options, but that happens to work because it's header-only so it doesn't 
> actually need any of the cmake options. So it's not entirely the correct 
> thing to do, but it does guarantee to fix things. I'm not sure if that 
> means we should merge it or not...
> 
>   The proper fix, I think, would require patching setup.py so we can 
> pass in additional cmake options to it.
> 
> 
>   Regards,
>   Arnout
> 
> [1] https://patchwork.ozlabs.org/project/buildroot/list/?series=168992
> [2] https://patchwork.ozlabs.org/project/buildroot/list/?series=186146

So

> 
>>
>> Regards,
>> Yann E. MORIN.
>>
>>>   $(eval $(python-package))
>>> -- 
>>> 2.20.1
>>>
>>
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
> 
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1

[Guillaume Bres]

[-- Attachment #1.1: Type: text/plain, Size: 10062 bytes --]

Arnout,
I also prefer your second option.
I need to run some more tests & inquire what Yann proposed on IRC.
This might take a couple of days, I am not submitting until I reach a
solution, but will keep in touch on IRC.
At least I understand what is going on in all scenarios now

Andreas,

>They both built on my
>but I needed to change three things:
>1. Add PYTHON_PYBIND_INSTALL_STAGING = YES
>2. Set ZXING_CPP_CONF_OPTS +=
>-Dpybind11_DIR=$(STAGING_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages/pybind11/share/cmake/pybind11
>3. Apply one of the proposed fixes for the pybind11 crosscompile problem
>(to tools/pybind11Tools.cmake)

Once I come to a solution, my next contribution will allow to
install pybind to staging.
At that time we should have a solution good enough so other packages will
no longer have to internally ship pybind.

>Apply one of the proposed fixes for the pybind11 crosscompile problem (to
tools/pybind11Tools.cmake)
We should keep an eye on the pybind devs topics you pointed out. We might
have a near future solution,
but it's probably going to need to involve (hopefuly sanitized/simplified)
in also a near future

Guillaume W. Bres
Software engineer
<guillaume.bressaix@gmail.com>


Le mer. 8 déc. 2021 à 12:19, Andreas Naumann <dev@andin.de> a écrit :

> Hi all,
>
> On 07.12.21 09:04, Arnout Vandecappelle wrote:
> >
> >
> > On 06/12/2021 22:57, Yann E. MORIN wrote:
> >> Guillaume, All,
> >>
> >> On 2021-12-06 20:31 +0100, guillaume.bressaix@gmail.com spake thusly:
> >>> From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com>
> >>>
> >>> The python 'setup.py' script needs header files
> >>> in $(@D)/pybind11 to work since v2.6.1, and these
> >>> files are generated by an internal minimalist cmake build.
> >>>
> >>> Fixes
> >>>
> >>>
> http://autobuild.buildroot.net/results/b89f1de64b308dffa73675f1f31ccb0b7be5a10d
> >>>
> >>>
> >>>
> http://autobuild.buildroot.net/results/d0287b7f64f206b0f074908c5780a3632e0cb799
> >>>
> >>>
> >>>
> http://autobuild.buildroot.net/results/27efb545a5a719a5581c8f746d3a3555ff4216ce
> >>>
> >>>
> >>>
> http://autobuild.buildroot.net/results/d2f0a0ad8f6c7178517df109e7d885dac9134c3a
> >>>
> >>>
> >>>
> http://autobuild.buildroot.net/results/b57e9a3279260dae4a590f9421238fcabb2f7cab
> >>>
> >>>
> >>>
> http://autobuild.buildroot.net/results/515e6f2fc6b5780260d98d6bb52b541ce4bf1afe
> >>>
> >>>
> >>>
> http://autobuild.buildroot.net/results/d89c4ecc81222d4f80c951da2232d2e393fa1c69
> >>>
> >>>
> >>> Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com>
> >>>
> >>> ---
> >>> This remains a python-package.
> >>>
> >>> 'python setup.py' actually hardcodes a system call to cmake now,
> >>> which seems dirty at first. But if we have a cmake env when they
> >>> do that, the install variables are passed and installation
> >>> is smooth. They probably introduced this call to make setup.py
> >>> self sufficient, as most people install this package from 'pip'.
> >>>
> >>> ---
> >>>   package/python-pybind/python-pybind.mk | 7 +++++++
> >>>   1 file changed, 7 insertions(+)
> >>>
> >>> diff --git a/package/python-pybind/python-pybind.mk
> >>> b/package/python-pybind/python-pybind.mk
> >>> index a6a1bdb976..bfd7f6f59a 100644
> >>> --- a/package/python-pybind/python-pybind.mk
> >>> +++ b/package/python-pybind/python-pybind.mk
> >>> @@ -9,5 +9,12 @@ PYTHON_PYBIND_SITE = $(call
> >>> github,pybind,pybind11,v$(PYTHON_PYBIND_VERSION))
> >>>   PYTHON_PYBIND_LICENSE = BSD-3-Clause
> >>>   PYTHON_PYBIND_LICENSE_FILES = LICENSE
> >>>   PYTHON_PYBIND_SETUP_TYPE = setuptools
> >>> +PYTHON_PYBIND_INSTALL_STAGING = YES
> >>> +
> >>> +# every single 'python setup.py' call actually calls cmake
> >>> +# internally, to populate $(@D)/pybind11 with build requirements.
> >>> +# If we have a host-cmake env at that moment,
> >>> +# then the final installation paths are properly defined
> >>> +PYTHON_PYBIND_DEPENDENCIES = host-cmake
> >>
> >> I am sorry, but I fail to see how this actually fixes things.
> >>
> >> As I explained in my previous review, headers are supposed to be in
> >> $(STAGING_DIR)/usr/include/pybind11/. That's at least where they are in
> >> Debian-based distributions;
> >>
> >>      $ apt-file search 'pybind11/pybind11.h'
> >>      pybind11-dev: /usr/include/pybind11/pybind11.h
> >>
> >> However, with the way pybind is currently packaged, the ehaders end up
> >> in:
> >>
> >>      $ sort output/build/python-pybind-2.8.1/.files-list.txt
> >>      python-pybind,./usr/bin/pybind11-config
> >>      [...]
> >>
> >>
> python-pybind,./usr/lib/python3.9/site-packages/pybind11/include/pybind11/pybind11.h
>
> >>
> >>      [...]
> >>
> >> So, those headers will most probably not be found by the compiler.
> >>
> >> Also, since pybind is supposed to be a headers'-only library, there is
> >> nothing supposed to go in target, so it should be;
> >>
> >>      PYTHON_PYBIND_INSTALL_TARGET = NO
> >
> >   Note that this is only relevant to make sure that pybind11-config
> > doesn't appear on target. The headers themselves are removed
> > automatically by Buildroot.
> >
> >>
> >> Furthermore, /usr/bin/pybind11-config is a python script which hardcodes
> >> its shabang to #!/usr/bin/python, and this is incorrect:
> >>
> >>    1. /usr/bin/python might be python2 or python3, we can't know
> >>    2. anyway, it would not use our host python, so it would miss the
> >>      proper modules search paths.
> >
> >   Yeah, this is something that needs to be fixed - in particular, there
> > may not be a /usr/bin/python at all.
> >
> >   However, I think that is kind of orthogonal to this patch. This patch
> > fixes a build failure of pybind itself. Your proposed change would be to
> > fix the build of a dependency of pybind.
> >
> >
> >> So, if another package want to call that /usr/bin/pybind11-config to
> >> find pybind, they're gonna have incorrect results, if at all...
> >>
> >> So I think we really, really need a package (like scipy) that actually
> >> makes use of pybind, so we can see how it works, and to fix it...
>
> By chance I was playing with such a package just yesterday: zxing-cpp.
> Not the version buildroot uses (from github glassechidna), but the
> nu-book one which claims to be faster. (And is the one where the python
> bindings in the PyPi package zxing-cpp originate from).
>
> Until recently they shipped their own pybind11 version, which throws an
> error when included in a cmake project in a crosscompile environment.
> This is a known problem, https://github.com/pybind/pybind11/issues/2139
> but non of the proposed fixes has been merged yet.
>
> However, the unreleased master version of zxing-cpp hast been changed so
> it can be configured to use the a system pybind11. So I tried to include
> the br packaged one (2.6.1 as well as 2.8.1). They both built on my
> system (no problem with the hardcoded host cmake here) but I needed to
> change three things:
> 1. Add PYTHON_PYBIND_INSTALL_STAGING = YES
> 2. Set ZXING_CPP_CONF_OPTS +=
>
> -Dpybind11_DIR=$(STAGING_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages/pybind11/share/cmake/pybind11
> 3. Apply one of the proposed fixes for the pybind11 crosscompile problem
> (to tools/pybind11Tools.cmake)
>
> So this patch here seems to help my (and possibly others) because 1. is
> already included. If interested, I could send a patch for 3.
>
>
> >
> >   Note that there are already two [1][2] submissions of scipy in
> > patchwork, so feel free to review those.
> >
> >   Note also that scipy depends on host-pybind, not pybind itself. I'm
> > not sure how *that* is supposed to work - although, being header-only,
>
>
> One assumption why scipy includes the host version of pybind is that it
> nicely avoids the problems where I need fix 1. and 3. for.
>
> > it probably doesn't really matter if you link with the host or the
> > target version of the library, and the host-installed pybind11-config
> > makes sure that you pick up the host headers (which are normally not in
> > include path). So conceptually wrong but works in practice.
>
> For getting conceptually right with my zxing-cpp I would appreciate some
> advice: When compiling it with above modificiations and
> BUILD_PYTHON_MODULE=ON, cmake happily compiles the zxing pybind wrapper.
> But of course wrappers/python/setup.py would need to be run as well.
> However mixing $(eval $(python-package)) and $(eval $(cmake-package)) is
> not possible. So do I still need to create a second package
> python-zxingcpp for the bindings only (which might be a pain because the
> wrappers CMakeLists is not written to use an existant installation of
> zxing-cpp but rather downloads it again) ?
>
>
> regards,
> Andreas
>
>
>
> >
> >
> >   In conclusion: I think this patch is going in the right direction
> > (modulo BR2_CMAKE_HOST_DEPENDENCY). It's still "conceptually wrong but
> > works" because it doesn't set the toolchain file and other cmake
> > options, but that happens to work because it's header-only so it doesn't
> > actually need any of the cmake options. So it's not entirely the correct
> > thing to do, but it does guarantee to fix things. I'm not sure if that
> > means we should merge it or not...
> >
> >   The proper fix, I think, would require patching setup.py so we can
> > pass in additional cmake options to it.
> >
> >
> >   Regards,
> >   Arnout
> >
> > [1] https://patchwork.ozlabs.org/project/buildroot/list/?series=168992
> > [2] https://patchwork.ozlabs.org/project/buildroot/list/?series=186146
>
> So
>
> >
> >>
> >> Regards,
> >> Yann E. MORIN.
> >>
> >>>   $(eval $(python-package))
> >>> --
> >>> 2.20.1
> >>>
> >>
> > _______________________________________________
> > buildroot mailing list
> > buildroot@buildroot.org
> > https://lists.buildroot.org/mailman/listinfo/buildroot
> >
>

[-- Attachment #1.2: Type: text/html, Size: 14609 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1

[Yann E. MORIN]

Arnout, All,

On 2021-12-07 09:04 +0100, Arnout Vandecappelle spake thusly:
> On 06/12/2021 22:57, Yann E. MORIN wrote:
> >Guillaume, All,
> >
> >On 2021-12-06 20:31 +0100, guillaume.bressaix@gmail.com spake thusly:
> >>From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com>
> >>
> >>The python 'setup.py' script needs header files
> >>in $(@D)/pybind11 to work since v2.6.1, and these
> >>files are generated by an internal minimalist cmake build.
> >>
> >>Fixes
> >>    http://autobuild.buildroot.net/results/b89f1de64b308dffa73675f1f31ccb0b7be5a10d
> >>    http://autobuild.buildroot.net/results/d0287b7f64f206b0f074908c5780a3632e0cb799
> >>    http://autobuild.buildroot.net/results/27efb545a5a719a5581c8f746d3a3555ff4216ce
> >>    http://autobuild.buildroot.net/results/d2f0a0ad8f6c7178517df109e7d885dac9134c3a
> >>    http://autobuild.buildroot.net/results/b57e9a3279260dae4a590f9421238fcabb2f7cab
> >>    http://autobuild.buildroot.net/results/515e6f2fc6b5780260d98d6bb52b541ce4bf1afe
> >>    http://autobuild.buildroot.net/results/d89c4ecc81222d4f80c951da2232d2e393fa1c69
> >>
> >>Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com>
> >I am sorry, but I fail to see how this actually fixes things.
[--SNIP--]
> >Also, since pybind is supposed to be a headers'-only library, there is
> >nothing supposed to go in target, so it should be;
> >     PYTHON_PYBIND_INSTALL_TARGET = NO
>  Note that this is only relevant to make sure that pybind11-config doesn't
> appear on target. The headers themselves are removed automatically by
> Buildroot.

It also installs a buncha stuff in /usr/lib/python3.9/site-packages/pybind11/
which are totally useless because that needed only for the packaging of
pybind11, not at runtime (as far as I could understand).

> >Furthermore, /usr/bin/pybind11-config is a python script which hardcodes
> >its shabang to #!/usr/bin/python, and this is incorrect:
> >
> >   1. /usr/bin/python might be python2 or python3, we can't know
> >   2. anyway, it would not use our host python, so it would miss the
> >     proper modules search paths.
> 
>  Yeah, this is something that needs to be fixed - in particular, there may
> not be a /usr/bin/python at all.
> 
>  However, I think that is kind of orthogonal to this patch. This patch fixes
> a build failure of pybind itself. Your proposed change would be to fix the
> build of a dependency of pybind.

Well, we can also easily fix it with just this:

    PYTHON_PYBIND_CONFIGURE_CMDS = true
    PYTHON_PYBIND_BUILD_CMDS = true
    PYTHON_PYBIND_INSTALL_TARGET_CMDS = true
    PYTHON_PYBIND_INSTALL_STAGING_CMDS = true
    HOST_PYTHON_PYBIND_CONFIGURE_CMDS = true
    HOST_PYTHON_PYBIND_BUILD_CMDS = true
    HOST_PYTHON_PYBIND_INSTALL_CMDS = true

Of course, this won't be very useful... Fixing the build of pybind in a
way that does not make it usable by its dependents is useless, imho. If
we fix it, we must fix it so that 1. it indeed builds and installs, and
2. that it is usable by its dependents, which means it should install
properly.

>  Note also that scipy depends on host-pybind, not pybind itself. I'm not
> sure how *that* is supposed to work - although, being header-only, it
> probably doesn't really matter if you link with the host or the target
> version of the library, and the host-installed pybind11-config makes sure
> that you pick up the host headers (which are normally not in include path).
> So conceptually wrong but works in practice.

And if we keep it like that without proper explanations why that is, or
if we do not fix it so that it behaves properly, it is going to become
a maintenance minefield down the road... Already is, anyway... :-]

>  In conclusion: I think this patch is going in the right direction (modulo
> BR2_CMAKE_HOST_DEPENDENCY). It's still "conceptually wrong but works"
> because it doesn't set the toolchain file and other cmake options, but that
> happens to work because it's header-only so it doesn't actually need any of
> the cmake options. So it's not entirely the correct thing to do, but it does
> guarantee to fix things. I'm not sure if that means we should merge it or
> not...

For me, no. But since we can't continue having build failures, and
becasue we have no package that depends on pybind, if we eventualyl do
not get a solution, then we can still remove it (it's broken!).

>  The proper fix, I think, would require patching setup.py so we can pass in
> additional cmake options to it.

Or as we've been discussing in another mail and IRC, by providing a
cmake wrapper in $(@D)/bin/cmake (not unlike some packages for which we
do a python symlink in $(@D)/bin/pyhon).

But now that the cmake issue has been identified, I think that we are
narrowing down on to a proper solution (Famous Last Words™).

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1

[Tomi Valkeinen]

Hi,

On 06/12/2021 23:57, Yann E. MORIN wrote:

> So I think we really, really need a package (like scipy) that actually
> makes use of pybind, so we can see how it works, and to fix it...

Jfyi, there are two relatively simple packages in buildroot that can use 
pybind: kmsxx and rwmem. Both use meson for building.

I can test the next version of this series with those packages.

  Tomi
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH v2 1/3] package/python-pybind: fix upgrade to 2.6.1

[guillaume.bressaix]

From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com>

Fixes
   http://autobuild.buildroot.net/results/b89f1de64b308dffa73675f1f31ccb0b7be5a10d
   http://autobuild.buildroot.net/results/d0287b7f64f206b0f074908c5780a3632e0cb799
   http://autobuild.buildroot.net/results/27efb545a5a719a5581c8f746d3a3555ff4216ce
   http://autobuild.buildroot.net/results/d2f0a0ad8f6c7178517df109e7d885dac9134c3a
   http://autobuild.buildroot.net/results/b57e9a3279260dae4a590f9421238fcabb2f7cab
   http://autobuild.buildroot.net/results/515e6f2fc6b5780260d98d6bb52b541ce4bf1afe
   http://autobuild.buildroot.net/results/d89c4ecc81222d4f80c951da2232d2e393fa1c69

Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com>
---
setup.py now calls 'cmake', as is, internally, with DESTDIR=$(@D)
to provide some setup.py build requirements.

The autobuild failures were actually of different kinds:
  (1) 'cmake' too old causes an early crash as it is unable to parse their CMakeList.txt
   (host machine dependent)
  (2) $(@D)/pybind missing because cmake call was faulty (happened all the time)

The provided patch allows us to fully control the hidden cmake context:
  * which 'cmake' is actually called is important, as most of the time we
  end up using /usr/bin/cmake.
  By adding BR2_CMAKE_DEPENDENCY we make sure (1) never happens and
  (2) is always solved with a complete cmake context control.

We could convert python-pybind to a cmake package but it involves
so much patching and tweaking (mostly for the python install),
that it is now clear that keeping the package as is, is the best option.

I also fixed the previous package declaration / handling:
  * this is a header only lib, so installing to target does not make sense,
  and installing to staging is mandatory

  * most of the time, this package will be used at compile time,
  on host, so having the host-package option is mandatory.
  This package is then used to customize the host/usr/python
  capacity, with C++ macro bindings.
  This patch serie provides an example of use,
  that does exactly that, and works for both python2 and python3
  (refer to following patch)

---
 ...-py-improve-cmake-context-definition.patch | 48 +++++++++++++++++++
 package/python-pybind/python-pybind.mk        | 47 ++++++++++++++++++
 2 files changed, 95 insertions(+)
 create mode 100644 package/python-pybind/0001-setup-py-improve-cmake-context-definition.patch

diff --git a/package/python-pybind/0001-setup-py-improve-cmake-context-definition.patch b/package/python-pybind/0001-setup-py-improve-cmake-context-definition.patch
new file mode 100644
index 0000000000..0952d44464
--- /dev/null
+++ b/package/python-pybind/0001-setup-py-improve-cmake-context-definition.patch
@@ -0,0 +1,48 @@
+From aeee170bc90d89973268f286c8bfa12afd8f330c Mon Sep 17 00:00:00 2001
+From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com>
+Date: Thu, 9 Dec 2021 20:30:08 +0100
+Subject: [PATCH] setup.py: improve cmake context definition
+
+Use two env. variables to define which cmake
+is internally called and pass a toolchain file
+for complex cross-compilation contexts.
+
+Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com>
+---
+ setup.py | 7 +++++--
+ 1 file changed, 5 insertions(+), 2 deletions(-)
+
+diff --git a/setup.py b/setup.py
+index 3a03279..2dff721 100644
+--- a/setup.py
++++ b/setup.py
+@@ -23,6 +23,8 @@ VERSION_REGEX = re.compile(
+ # files, and the sys.prefix files (CMake and headers).
+
+ global_sdist = os.environ.get("PYBIND11_GLOBAL_SDIST", False)
++global_cmake = os.environ.get("PYBIND11_CMAKE", "unknown")
++global_cmake_toolchain_file = os.environ.get("PYBIND11_CMAKE_TOOLCHAIN_FILE", "unknown")
+
+ setup_py = "tools/setup_global.py.in" if global_sdist else "tools/setup_main.py.in"
+ extra_cmd = 'cmdclass["sdist"] = SDist\n'
+@@ -101,14 +103,15 @@ def remove_output(*sources):
+ with remove_output("pybind11/include", "pybind11/share"):
+     # Generate the files if they are not present.
+     with TemporaryDirectory() as tmpdir:
+-        cmd = ["cmake", "-S", ".", "-B", tmpdir] + [
++        cmd = [global_cmake, "-S", ".", "-B", tmpdir] + [
+             "-DCMAKE_INSTALL_PREFIX=pybind11",
++            "-DCMAKE_TOOLCHAIN_FILE={:s}".format(global_cmake_toolchain_file),
+             "-DBUILD_TESTING=OFF",
+             "-DPYBIND11_NOPYTHON=ON",
+         ]
+         cmake_opts = dict(cwd=DIR, stdout=sys.stdout, stderr=sys.stderr)
+         subprocess.check_call(cmd, **cmake_opts)
+-        subprocess.check_call(["cmake", "--install", tmpdir], **cmake_opts)
++        subprocess.check_call([global_cmake, "--install", tmpdir], **cmake_opts)
+
+     txt = get_and_replace(setup_py, version=version, extra_cmd=extra_cmd)
+     code = compile(txt, setup_py, "exec")
+--
+1.8.3.1
+
diff --git a/package/python-pybind/python-pybind.mk b/package/python-pybind/python-pybind.mk
index a6a1bdb976..442ed3328f 100644
--- a/package/python-pybind/python-pybind.mk
+++ b/package/python-pybind/python-pybind.mk
@@ -10,4 +10,51 @@ PYTHON_PYBIND_LICENSE = BSD-3-Clause
 PYTHON_PYBIND_LICENSE_FILES = LICENSE
 PYTHON_PYBIND_SETUP_TYPE = setuptools
 
+# Only installs header files
+PYTHON_PYBIND_INSTALL_TARGET = NO
+PYTHON_PYBIND_INSTALL_STAGING = YES
+
+# their cmakelist.txt file needs a recent cmake
+PYTHON_PYBIND_DEPENDENCIES += $(BR2_CMAKE_HOST_DEPENDENCY)
+HOST_PYTHON_PYBIND_DEPENDENCIES += $(BR2_CMAKE_HOST_DEPENDENCY)
+
+# every single setup.py interaction calls 'cmake' (as is) intrinsicly.
+# we deliver a patch to customize setup.py to define the cmake context
+# using the following variables
+PYTHON_PYBIND_MAKE_ENV = $(MAKE_ENV)
+PYTHON_PYBIND_SETUPTOOLS_ENV = $(PKG_PYTHON_SETUPTOOLS_ENV)
+PYTHON_PYBIND_SETUPTOOLS_ENV += PYBIND11_CMAKE=$(BR2_CMAKE)
+PYTHON_PYBIND_SETUPTOOLS_ENV += PYBIND11_CMAKE_TOOLCHAIN_FILE=$(HOST_DIR)/share/buildroot/toolchainfile.cmake
+
+HOST_PYTHON_PYBIND_MAKE_ENV = $(MAKE_ENV)
+HOST_PYTHON_PYBIND_SETUPTOOLS_ENV = $(PKG_PYTHON_SETUPTOOLS_ENV)
+HOST_PYTHON_PYBIND_SETUPTOOLS_ENV += PYBIND11_CMAKE=$(BR2_CMAKE)
+HOST_PYTHON_PYBIND_SETUPTOOLS_ENV += PYBIND11_CMAKE_TOOLCHAIN_FILE=$(HOST_DIR)/share/buildroot/toolchainfile.cmake
+
+define PYTHON_PYBIND_BUILD_CMDS
+	cd $(@D); \
+		$(PYTHON_PYBIND_SETUPTOOLS_ENV) $(HOST_DIR)/usr/bin/python setup.py build
+endef
+
+define HOST_PYTHON_PYBIND_BUILD_CMDS
+	cd $(@D); \
+		$(PYTHON_PYBIND_SETUPTOOLS_ENV) $(HOST_DIR)/usr/bin/python setup.py build
+endef
+
+# define installation destinations properly
+define PYTHON_PYBIND_INSTALL_STAGING_CMDS
+	cd $(@D); \
+		$(PYTHON_PYBIND_SETUPTOOLS_ENV) \
+			PREFIX=$(STAGING_DIR)/usr \
+				$(HOST_DIR)/bin/python setup.py install
+endef
+
+define HOST_PYTHON_PYBIND_INSTALL_CMDS
+	cd $(@D); \
+		$(PYTHON_PYBIND_SETUPTOOLS_ENV) \
+			PREFIX=$(HOST_DIR)/usr \
+				$(HOST_DIR)/bin/python setup.py install
+endef
+
 $(eval $(python-package))
+$(eval $(host-python-package))
-- 
2.30.2

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH v2 2/3] package/python-pybind: bump to version 2.8.1

[guillaume.bressaix]

From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com>

Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com>
---
 package/python-pybind/python-pybind.hash | 2 +-
 package/python-pybind/python-pybind.mk   | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/package/python-pybind/python-pybind.hash b/package/python-pybind/python-pybind.hash
index a68ac846e2..e1d86a4494 100644
--- a/package/python-pybind/python-pybind.hash
+++ b/package/python-pybind/python-pybind.hash
@@ -1,4 +1,4 @@
 # Locally calculated
-sha256  cdbe326d357f18b83d10322ba202d69f11b2f49e2d87ade0dc2be0c5c34f8e2a  python-pybind-2.6.1.tar.gz
+sha256  f1bcc07caa568eb312411dde5308b1e250bd0e1bc020fae855bf9f43209940cc  python-pybind-2.8.1.tar.gz
 # License files, locally calculated
 sha256  83965b843b98f670d3a85bd041ed4b372c8ec50d7b4a5995a83ac697ba675dcb  LICENSE
diff --git a/package/python-pybind/python-pybind.mk b/package/python-pybind/python-pybind.mk
index 442ed3328f..cf6a642922 100644
--- a/package/python-pybind/python-pybind.mk
+++ b/package/python-pybind/python-pybind.mk
@@ -4,7 +4,7 @@
 #
 ################################################################################
 
-PYTHON_PYBIND_VERSION = 2.6.1
+PYTHON_PYBIND_VERSION = 2.8.1
 PYTHON_PYBIND_SITE = $(call github,pybind,pybind11,v$(PYTHON_PYBIND_VERSION))
 PYTHON_PYBIND_LICENSE = BSD-3-Clause
 PYTHON_PYBIND_LICENSE_FILES = LICENSE
-- 
2.30.2

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH v2 3/3] support/testing: add test for python-pybind

[guillaume.bressaix]

From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com>

python-pybind is a header only library, most of the time used as a host-package,
to customize the host/usr/python interpreter capacity (at build time) with C++ based features.

Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com>
---
This test case enhances the /host/usr/python interpreter with a C++ macro.

The test delivers a C++ macro contained in its br-external/example.cpp.
We compile it against the host context.
We install it to $(HOST_DIR)/usr/lib/python3.9 as it is natively contained
in the host-python system path (import is easy).
The macro is an integer adder, written in C++ that we can now call in host-python.

The test infrastructure should actually be declared "host-generic-package"
instead of "generic-package", to be consistent with Buildroot infrastructure.
But I am not able to get this to actually do anything, if I declare this package
as a host-generic and "Config.in.host" instead of "Config.in". To me, this is
not too important

* TestPythonPybind allows testing for both Py2 and Py3 like it is done
in many other packages

* TestPythonPybind does not inherit from TestPythonPackageBase as usually done
  (1) because I need the br2-external feature to ship the example.cpp containing the custom macro
  (2) because TestPythonPackageBase always involves the emulator, which we do not want in our context

* Although TestPythonPybind does have a TestPythonBase.config base at the same time,
as it is most convenient to have a valid python context

* TestPythonPybind uses subprocess, as similarly done in 'core' on 'infra' test cases,
because we do not involve the emulator

* TestPythonPybind tests (1+2)=3 written in C++, used in Python 2/3
* TestPythonPybind tests a custom module attribute, written in C++, used in Python 2/3

---
 DEVELOPERS                                    |  3 ++
 .../br2-external/python-pybind/Config.in      |  1 +
 .../br2-external/python-pybind/external.desc  |  1 +
 .../br2-external/python-pybind/external.mk    |  1 +
 .../package/python-pybind-example/Config.in   |  5 ++
 .../package/python-pybind-example/example.cpp | 16 +++++++
 .../python-pybind-example.mk                  | 47 ++++++++++++++++++
 .../tests/package/sample_python_pybind.py     |  4 ++
 .../tests/package/test_python_pybind.py       | 48 +++++++++++++++++++
 9 files changed, 126 insertions(+)
 create mode 100644 support/testing/tests/package/br2-external/python-pybind/Config.in
 create mode 100644 support/testing/tests/package/br2-external/python-pybind/external.desc
 create mode 100644 support/testing/tests/package/br2-external/python-pybind/external.mk
 create mode 100644 support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/Config.in
 create mode 100644 support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/example.cpp
 create mode 100644 support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/python-pybind-example.mk
 create mode 100644 support/testing/tests/package/sample_python_pybind.py
 create mode 100644 support/testing/tests/package/test_python_pybind.py

diff --git a/DEVELOPERS b/DEVELOPERS
index 21e4f2a84c..130ac3db73 100644
--- a/DEVELOPERS
+++ b/DEVELOPERS
@@ -1120,6 +1120,9 @@ F:	package/liquid-dsp/
 F:	package/pixiewps/
 F:	package/python-pybind/
 F:	package/reaver/
+F:	support/testing/tests/package/br2-external/python-pybind
+F:	support/testing/tests/package/sample_python_pybind.py
+F:	support/testing/tests/package/test_python_pybind.py
 
 N:	Guo Ren <ren_guo@c-sky.com>
 F:	arch/Config.in.csky
diff --git a/support/testing/tests/package/br2-external/python-pybind/Config.in b/support/testing/tests/package/br2-external/python-pybind/Config.in
new file mode 100644
index 0000000000..70c77157b3
--- /dev/null
+++ b/support/testing/tests/package/br2-external/python-pybind/Config.in
@@ -0,0 +1 @@
+source "$BR2_EXTERNAL_PYTHON_PYBIND_PATH/package/python-pybind-example/Config.in"
diff --git a/support/testing/tests/package/br2-external/python-pybind/external.desc b/support/testing/tests/package/br2-external/python-pybind/external.desc
new file mode 100644
index 0000000000..eef5e0f5a0
--- /dev/null
+++ b/support/testing/tests/package/br2-external/python-pybind/external.desc
@@ -0,0 +1 @@
+name: PYTHON_PYBIND
diff --git a/support/testing/tests/package/br2-external/python-pybind/external.mk b/support/testing/tests/package/br2-external/python-pybind/external.mk
new file mode 100644
index 0000000000..3501f3135e
--- /dev/null
+++ b/support/testing/tests/package/br2-external/python-pybind/external.mk
@@ -0,0 +1 @@
+include $(sort $(wildcard $(BR2_EXTERNAL_PYTHON_PYBIND_PATH)/package/*/*.mk))
diff --git a/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/Config.in b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/Config.in
new file mode 100644
index 0000000000..773d021a53
--- /dev/null
+++ b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/Config.in
@@ -0,0 +1,5 @@
+config BR2_PACKAGE_PYTHON_PYBIND_EXAMPLE
+	bool "python-pybind-example"
+	depends on BR2_PACKAGE_PYTHON_PYBIND
+	help
+	  This test creates a cpp macro later used on target in python
diff --git a/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/example.cpp b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/example.cpp
new file mode 100644
index 0000000000..f2eea8e48d
--- /dev/null
+++ b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/example.cpp
@@ -0,0 +1,16 @@
+#include <pybind11/pybind11.h>
+namespace py = pybind11;
+
+int add (int i, int j) {
+    return i + j;
+}
+
+PYBIND11_MODULE (example, m) {
+    // optional module description
+    m.doc() = "pybind11 example plugin";
+    // test a module method
+    m.def("add", &add, "example::add adds two integer numbers");
+    // test a module attribute
+    py::object hello = py::cast("Hello World");
+    m.attr("says") = hello;
+}
diff --git a/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/python-pybind-example.mk b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/python-pybind-example.mk
new file mode 100644
index 0000000000..85dd09d93f
--- /dev/null
+++ b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/python-pybind-example.mk
@@ -0,0 +1,47 @@
+################################################################################
+#
+# python-pybind-example
+#
+################################################################################
+
+# this builds a C++ macro "add(a,b)"
+# that we expose to host-python with a custom install
+# and that the python test script will later use
+PYTHON_PYBIND_EXAMPLE_DEPENDENCIES = host-python-pybind
+
+PYTHON_PYBIND_EXAMPLE_CXX_FLAGS = $(HOST_CXXFLAGS)
+PYTHON_PYBIND_EXAMPLE_CXX_FLAGS += -Wall -shared
+PYTHON_PYBIND_EXAMPLE_CXX_FLAGS += -std=c++11 -fPIC
+
+ifeq ($(BR2_PACKAGE_PYTHON3),y)
+PYTHON_PYBIND_PYVER = python$(PYTHON3_VERSION_MAJOR)
+PYTHON_PYBIND_PYTHON = $(HOST_DIR)/usr/bin/python3
+else
+PYTHON_PYBIND_PYVER = python$(PYTHON_VERSION_MAJOR)
+PYTHON_PYBIND_PYTHON = $(HOST_DIR)/usr/bin/python2
+endif
+
+# retrieve pybind11 include paths
+PYTHON_PYBIND_EXAMPLE_CXX_FLAGS += \
+	$(shell $(PYTHON_PYBIND_PYTHON) -m pybind11 --includes)
+
+# .so to be installed must have exact suffix
+# otherwise import() in python will not work
+HOST_LIB_BINARY_SUFFIX = \
+	$(shell $(PYTHON_PYBIND_PYTHON)-config --extension-suffix)
+
+define PYTHON_PYBIND_EXAMPLE_BUILD_CMDS
+	$(INSTALL) $(PYTHON_PYBIND_EXAMPLE_PKGDIR)/example.cpp $(@D)
+	cd $(@D); \
+			$(HOSTCXX) $(PYTHON_PYBIND_EXAMPLE_CXX_FLAGS) \
+				example.cpp -o example$(HOST_LIB_BINARY_SUFFIX)
+endef
+
+# installs into host/python sys.path
+# so the new macro is naturally known
+define PYTHON_PYBIND_EXAMPLE_INSTALL_TARGET_CMDS
+	$(INSTALL) -D -m 755 $(@D)/example$(HOST_LIB_BINARY_SUFFIX) \
+		$(HOST_DIR)/usr/lib/$(PYTHON_PYBIND_PYVER)/example$(HOST_LIB_BINARY_SUFFIX)
+endef
+
+$(eval $(generic-package))
diff --git a/support/testing/tests/package/sample_python_pybind.py b/support/testing/tests/package/sample_python_pybind.py
new file mode 100644
index 0000000000..605c0bab15
--- /dev/null
+++ b/support/testing/tests/package/sample_python_pybind.py
@@ -0,0 +1,4 @@
+#!/usr/bin/env python
+import example
+print(example.add(1, 2))
+print(example.says)
diff --git a/support/testing/tests/package/test_python_pybind.py b/support/testing/tests/package/test_python_pybind.py
new file mode 100644
index 0000000000..7a68d6512a
--- /dev/null
+++ b/support/testing/tests/package/test_python_pybind.py
@@ -0,0 +1,48 @@
+import os
+import infra
+import subprocess
+from tests.package.test_python import TestPythonPackageBase
+
+class TestPythonPybind (infra.basetest.BRTest):
+	config = TestPythonPackageBase.config # python minimal requirements
+	sample_script = "tests/package/sample_python_pybind.py"
+	# ship examples macro & installs it to host
+	br2_external = [infra.filepath("tests/package/br2-external/python-pybind")]
+
+class TestPythonPy2Pybind (TestPythonPybind):
+	__test__ = True
+	config = TestPythonPybind.config + \
+		"""
+		BR2_PACKAGE_PYTHON=y
+		BR2_PACKAGE_PYTHON_PYBIND=y
+		BR2_PACKAGE_PYTHON_PYBIND_EXAMPLE=y
+		"""
+
+	def test_run(self):
+		host_interpreter = os.path.join(self.builddir,"host","usr","bin","python")
+		# test new c++->py macro exposed to host interpreter
+		cmd = [host_interpreter, infra.filepath(self.sample_script)]
+		output = subprocess.check_output(cmd)
+		result = int(output.splitlines()[0])
+		attribute = output.splitlines()[1].decode("utf-8")
+		self.assertEqual(result,3) # 2+1 using '+' macro from cpp
+		self.assertEqual(attribute,"Hello World") # macro special attribute
+
+class TestPythonPy3Pybind (TestPythonPybind):
+	__test__ = True
+	config = TestPythonPybind.config + \
+		"""
+		BR2_PACKAGE_PYTHON3=y
+		BR2_PACKAGE_PYTHON_PYBIND=y
+		BR2_PACKAGE_PYTHON_PYBIND_EXAMPLE=y
+		"""
+
+	def test_run(self):
+		host_interpreter = os.path.join(self.builddir,"host","usr","bin","python")
+		# test new c++->py macro exposed to host interpreter
+		cmd = [host_interpreter, infra.filepath(self.sample_script)]
+		output = subprocess.check_output(cmd)
+		result = int(output.splitlines()[0])
+		attribute = output.splitlines()[1].decode("utf-8")
+		self.assertEqual(result,3) # 2+1 using '+' macro from cpp
+		self.assertEqual(attribute,"Hello World") # macro special attribute
-- 
2.30.2

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1

[Arnout Vandecappelle]

On 03/01/2022 09:24, Tomi Valkeinen wrote:
> Hi,
> 
> On 06/12/2021 23:57, Yann E. MORIN wrote:
> 
>> So I think we really, really need a package (like scipy) that actually
>> makes use of pybind, so we can see how it works, and to fix it...
> 
> Jfyi, there are two relatively simple packages in buildroot that can use pybind: 
> kmsxx and rwmem. Both use meson for building.
> 
> I can test the next version of this series with those packages.

  I tried to do that. Unfortunately, it turns out that both of those packages 
use the CMake way of finding the pybind11 dependency, and apparently our 
cross-compilation.conf doesn't support cmake yet... So it's not that trivial.

  I merged the pybind stuff but wasn't able to test the cmake handling, so it's 
even possible that that is broken as well...

  Regards,
  Arnout

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1

[Arnout Vandecappelle]

On 08/12/2021 12:19, Andreas Naumann wrote:
[snip]
> By chance I was playing with such a package just yesterday: zxing-cpp. Not the 
> version buildroot uses (from github glassechidna), but the nu-book one which 
> claims to be faster. (And is the one where the python bindings in the PyPi 
> package zxing-cpp originate from).
> 
> Until recently they shipped their own pybind11 version, which throws an error 
> when included in a cmake project in a crosscompile environment. This is a known 
> problem, https://github.com/pybind/pybind11/issues/2139 but non of the proposed 
> fixes has been merged yet.
> 
> However, the unreleased master version of zxing-cpp hast been changed so it can 
> be configured to use the a system pybind11. So I tried to include the br 
> packaged one (2.6.1 as well as 2.8.1). They both built on my system (no problem 
> with the hardcoded host cmake here) but I needed to change three things:
> 1. Add PYTHON_PYBIND_INSTALL_STAGING = YES
> 2. Set ZXING_CPP_CONF_OPTS += 
> -Dpybind11_DIR=$(STAGING_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages/pybind11/share/cmake/pybind11 
> 
> 3. Apply one of the proposed fixes for the pybind11 crosscompile problem (to 
> tools/pybind11Tools.cmake)
> 
> So this patch here seems to help my (and possibly others) because 1. is already 
> included. If interested, I could send a patch for 3.

  I applied a slightly different approach to master [1]. If you'd like to give 
that one a go, we'd welcome an updated zxing-cpp package!

  Regards,
  Arnout

[1] 
https://git.buildroot.org/buildroot/commit/?id=a19b822239c36cb40a632899bb5c3e848a5fc81e

[snip]
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 1/3] package/python-pybind: fix upgrade to 2.6.1

[Arnout Vandecappelle]

On 04/01/2022 13:38, guillaume.bressaix@gmail.com wrote:
> From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com>
> 
> Fixes
>     http://autobuild.buildroot.net/results/b89f1de64b308dffa73675f1f31ccb0b7be5a10d
>     http://autobuild.buildroot.net/results/d0287b7f64f206b0f074908c5780a3632e0cb799
>     http://autobuild.buildroot.net/results/27efb545a5a719a5581c8f746d3a3555ff4216ce
>     http://autobuild.buildroot.net/results/d2f0a0ad8f6c7178517df109e7d885dac9134c3a
>     http://autobuild.buildroot.net/results/b57e9a3279260dae4a590f9421238fcabb2f7cab
>     http://autobuild.buildroot.net/results/515e6f2fc6b5780260d98d6bb52b541ce4bf1afe
>     http://autobuild.buildroot.net/results/d89c4ecc81222d4f80c951da2232d2e393fa1c69
> 
> Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com>

  I applied a completely reworked version to master, thanks.

  Regards,
  Arnout

[snip]
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 3/3] support/testing: add test for python-pybind

[Arnout Vandecappelle]

On 04/01/2022 13:39, guillaume.bressaix@gmail.com wrote:
> From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com>
> 
> python-pybind is a header only library, most of the time used as a host-package,
> to customize the host/usr/python interpreter capacity (at build time) with C++ based features.
> 
> Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com>

  I applied to master with many changes:

      - Retain python3 only.
      - python-pybind is a target package, not host.
      - Select python-pybind instead of depend.
      - Simplify python-pybind-example package.
      - Check in python-pybind-example build if pybind11.get_include()
        produces output.
      - Don't use python3 -m pybind11 --includes: it includes the main python
        includes, which are for the host, not for the target.
      - Use TestPythonPackageBase instead of open-coding something imported
        with host python.

  However, I hadn't noticed that the original author of this test was Adam. In 
the future, please make sure that you don't claim authorship on a patch that is 
actually from someone else!

  Regards,
  Arnout

> ---
> This test case enhances the /host/usr/python interpreter with a C++ macro.
> 
> The test delivers a C++ macro contained in its br-external/example.cpp.
> We compile it against the host context.
> We install it to $(HOST_DIR)/usr/lib/python3.9 as it is natively contained
> in the host-python system path (import is easy).
> The macro is an integer adder, written in C++ that we can now call in host-python.
> 
> The test infrastructure should actually be declared "host-generic-package"
> instead of "generic-package", to be consistent with Buildroot infrastructure.
> But I am not able to get this to actually do anything, if I declare this package
> as a host-generic and "Config.in.host" instead of "Config.in". To me, this is
> not too important
> 
> * TestPythonPybind allows testing for both Py2 and Py3 like it is done
> in many other packages
> 
> * TestPythonPybind does not inherit from TestPythonPackageBase as usually done
>    (1) because I need the br2-external feature to ship the example.cpp containing the custom macro
>    (2) because TestPythonPackageBase always involves the emulator, which we do not want in our context
> 
> * Although TestPythonPybind does have a TestPythonBase.config base at the same time,
> as it is most convenient to have a valid python context
> 
> * TestPythonPybind uses subprocess, as similarly done in 'core' on 'infra' test cases,
> because we do not involve the emulator
> 
> * TestPythonPybind tests (1+2)=3 written in C++, used in Python 2/3
> * TestPythonPybind tests a custom module attribute, written in C++, used in Python 2/3
> 
> ---
>   DEVELOPERS                                    |  3 ++
>   .../br2-external/python-pybind/Config.in      |  1 +
>   .../br2-external/python-pybind/external.desc  |  1 +
>   .../br2-external/python-pybind/external.mk    |  1 +
>   .../package/python-pybind-example/Config.in   |  5 ++
>   .../package/python-pybind-example/example.cpp | 16 +++++++
>   .../python-pybind-example.mk                  | 47 ++++++++++++++++++
>   .../tests/package/sample_python_pybind.py     |  4 ++
>   .../tests/package/test_python_pybind.py       | 48 +++++++++++++++++++
>   9 files changed, 126 insertions(+)
>   create mode 100644 support/testing/tests/package/br2-external/python-pybind/Config.in
>   create mode 100644 support/testing/tests/package/br2-external/python-pybind/external.desc
>   create mode 100644 support/testing/tests/package/br2-external/python-pybind/external.mk
>   create mode 100644 support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/Config.in
>   create mode 100644 support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/example.cpp
>   create mode 100644 support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/python-pybind-example.mk
>   create mode 100644 support/testing/tests/package/sample_python_pybind.py
>   create mode 100644 support/testing/tests/package/test_python_pybind.py
> 
> diff --git a/DEVELOPERS b/DEVELOPERS
> index 21e4f2a84c..130ac3db73 100644
> --- a/DEVELOPERS
> +++ b/DEVELOPERS
> @@ -1120,6 +1120,9 @@ F:	package/liquid-dsp/
>   F:	package/pixiewps/
>   F:	package/python-pybind/
>   F:	package/reaver/
> +F:	support/testing/tests/package/br2-external/python-pybind
> +F:	support/testing/tests/package/sample_python_pybind.py
> +F:	support/testing/tests/package/test_python_pybind.py
>   
>   N:	Guo Ren <ren_guo@c-sky.com>
>   F:	arch/Config.in.csky
> diff --git a/support/testing/tests/package/br2-external/python-pybind/Config.in b/support/testing/tests/package/br2-external/python-pybind/Config.in
> new file mode 100644
> index 0000000000..70c77157b3
> --- /dev/null
> +++ b/support/testing/tests/package/br2-external/python-pybind/Config.in
> @@ -0,0 +1 @@
> +source "$BR2_EXTERNAL_PYTHON_PYBIND_PATH/package/python-pybind-example/Config.in"
> diff --git a/support/testing/tests/package/br2-external/python-pybind/external.desc b/support/testing/tests/package/br2-external/python-pybind/external.desc
> new file mode 100644
> index 0000000000..eef5e0f5a0
> --- /dev/null
> +++ b/support/testing/tests/package/br2-external/python-pybind/external.desc
> @@ -0,0 +1 @@
> +name: PYTHON_PYBIND
> diff --git a/support/testing/tests/package/br2-external/python-pybind/external.mk b/support/testing/tests/package/br2-external/python-pybind/external.mk
> new file mode 100644
> index 0000000000..3501f3135e
> --- /dev/null
> +++ b/support/testing/tests/package/br2-external/python-pybind/external.mk
> @@ -0,0 +1 @@
> +include $(sort $(wildcard $(BR2_EXTERNAL_PYTHON_PYBIND_PATH)/package/*/*.mk))
> diff --git a/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/Config.in b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/Config.in
> new file mode 100644
> index 0000000000..773d021a53
> --- /dev/null
> +++ b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/Config.in
> @@ -0,0 +1,5 @@
> +config BR2_PACKAGE_PYTHON_PYBIND_EXAMPLE
> +	bool "python-pybind-example"
> +	depends on BR2_PACKAGE_PYTHON_PYBIND
> +	help
> +	  This test creates a cpp macro later used on target in python
> diff --git a/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/example.cpp b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/example.cpp
> new file mode 100644
> index 0000000000..f2eea8e48d
> --- /dev/null
> +++ b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/example.cpp
> @@ -0,0 +1,16 @@
> +#include <pybind11/pybind11.h>
> +namespace py = pybind11;
> +
> +int add (int i, int j) {
> +    return i + j;
> +}
> +
> +PYBIND11_MODULE (example, m) {
> +    // optional module description
> +    m.doc() = "pybind11 example plugin";
> +    // test a module method
> +    m.def("add", &add, "example::add adds two integer numbers");
> +    // test a module attribute
> +    py::object hello = py::cast("Hello World");
> +    m.attr("says") = hello;
> +}
> diff --git a/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/python-pybind-example.mk b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/python-pybind-example.mk
> new file mode 100644
> index 0000000000..85dd09d93f
> --- /dev/null
> +++ b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/python-pybind-example.mk
> @@ -0,0 +1,47 @@
> +################################################################################
> +#
> +# python-pybind-example
> +#
> +################################################################################
> +
> +# this builds a C++ macro "add(a,b)"
> +# that we expose to host-python with a custom install
> +# and that the python test script will later use
> +PYTHON_PYBIND_EXAMPLE_DEPENDENCIES = host-python-pybind
> +
> +PYTHON_PYBIND_EXAMPLE_CXX_FLAGS = $(HOST_CXXFLAGS)
> +PYTHON_PYBIND_EXAMPLE_CXX_FLAGS += -Wall -shared
> +PYTHON_PYBIND_EXAMPLE_CXX_FLAGS += -std=c++11 -fPIC
> +
> +ifeq ($(BR2_PACKAGE_PYTHON3),y)
> +PYTHON_PYBIND_PYVER = python$(PYTHON3_VERSION_MAJOR)
> +PYTHON_PYBIND_PYTHON = $(HOST_DIR)/usr/bin/python3
> +else
> +PYTHON_PYBIND_PYVER = python$(PYTHON_VERSION_MAJOR)
> +PYTHON_PYBIND_PYTHON = $(HOST_DIR)/usr/bin/python2
> +endif
> +
> +# retrieve pybind11 include paths
> +PYTHON_PYBIND_EXAMPLE_CXX_FLAGS += \
> +	$(shell $(PYTHON_PYBIND_PYTHON) -m pybind11 --includes)
> +
> +# .so to be installed must have exact suffix
> +# otherwise import() in python will not work
> +HOST_LIB_BINARY_SUFFIX = \
> +	$(shell $(PYTHON_PYBIND_PYTHON)-config --extension-suffix)
> +
> +define PYTHON_PYBIND_EXAMPLE_BUILD_CMDS
> +	$(INSTALL) $(PYTHON_PYBIND_EXAMPLE_PKGDIR)/example.cpp $(@D)
> +	cd $(@D); \
> +			$(HOSTCXX) $(PYTHON_PYBIND_EXAMPLE_CXX_FLAGS) \
> +				example.cpp -o example$(HOST_LIB_BINARY_SUFFIX)
> +endef
> +
> +# installs into host/python sys.path
> +# so the new macro is naturally known
> +define PYTHON_PYBIND_EXAMPLE_INSTALL_TARGET_CMDS
> +	$(INSTALL) -D -m 755 $(@D)/example$(HOST_LIB_BINARY_SUFFIX) \
> +		$(HOST_DIR)/usr/lib/$(PYTHON_PYBIND_PYVER)/example$(HOST_LIB_BINARY_SUFFIX)
> +endef
> +
> +$(eval $(generic-package))
> diff --git a/support/testing/tests/package/sample_python_pybind.py b/support/testing/tests/package/sample_python_pybind.py
> new file mode 100644
> index 0000000000..605c0bab15
> --- /dev/null
> +++ b/support/testing/tests/package/sample_python_pybind.py
> @@ -0,0 +1,4 @@
> +#!/usr/bin/env python
> +import example
> +print(example.add(1, 2))
> +print(example.says)
> diff --git a/support/testing/tests/package/test_python_pybind.py b/support/testing/tests/package/test_python_pybind.py
> new file mode 100644
> index 0000000000..7a68d6512a
> --- /dev/null
> +++ b/support/testing/tests/package/test_python_pybind.py
> @@ -0,0 +1,48 @@
> +import os
> +import infra
> +import subprocess
> +from tests.package.test_python import TestPythonPackageBase
> +
> +class TestPythonPybind (infra.basetest.BRTest):
> +	config = TestPythonPackageBase.config # python minimal requirements
> +	sample_script = "tests/package/sample_python_pybind.py"
> +	# ship examples macro & installs it to host
> +	br2_external = [infra.filepath("tests/package/br2-external/python-pybind")]
> +
> +class TestPythonPy2Pybind (TestPythonPybind):
> +	__test__ = True
> +	config = TestPythonPybind.config + \
> +		"""
> +		BR2_PACKAGE_PYTHON=y
> +		BR2_PACKAGE_PYTHON_PYBIND=y
> +		BR2_PACKAGE_PYTHON_PYBIND_EXAMPLE=y
> +		"""
> +
> +	def test_run(self):
> +		host_interpreter = os.path.join(self.builddir,"host","usr","bin","python")
> +		# test new c++->py macro exposed to host interpreter
> +		cmd = [host_interpreter, infra.filepath(self.sample_script)]
> +		output = subprocess.check_output(cmd)
> +		result = int(output.splitlines()[0])
> +		attribute = output.splitlines()[1].decode("utf-8")
> +		self.assertEqual(result,3) # 2+1 using '+' macro from cpp
> +		self.assertEqual(attribute,"Hello World") # macro special attribute
> +
> +class TestPythonPy3Pybind (TestPythonPybind):
> +	__test__ = True
> +	config = TestPythonPybind.config + \
> +		"""
> +		BR2_PACKAGE_PYTHON3=y
> +		BR2_PACKAGE_PYTHON_PYBIND=y
> +		BR2_PACKAGE_PYTHON_PYBIND_EXAMPLE=y
> +		"""
> +
> +	def test_run(self):
> +		host_interpreter = os.path.join(self.builddir,"host","usr","bin","python")
> +		# test new c++->py macro exposed to host interpreter
> +		cmd = [host_interpreter, infra.filepath(self.sample_script)]
> +		output = subprocess.check_output(cmd)
> +		result = int(output.splitlines()[0])
> +		attribute = output.splitlines()[1].decode("utf-8")
> +		self.assertEqual(result,3) # 2+1 using '+' macro from cpp
> +		self.assertEqual(attribute,"Hello World") # macro special attribute
> 
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 2/3] package/python-pybind: bump to version 2.8.1

[Arnout Vandecappelle]

On 04/01/2022 13:39, guillaume.bressaix@gmail.com wrote:
> From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com>
> 
> Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com>

  I bumped to 2.9.0 instead and applied to master, thanks.

  Regards,
  Arnout

> ---
>   package/python-pybind/python-pybind.hash | 2 +-
>   package/python-pybind/python-pybind.mk   | 2 +-
>   2 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/package/python-pybind/python-pybind.hash b/package/python-pybind/python-pybind.hash
> index a68ac846e2..e1d86a4494 100644
> --- a/package/python-pybind/python-pybind.hash
> +++ b/package/python-pybind/python-pybind.hash
> @@ -1,4 +1,4 @@
>   # Locally calculated
> -sha256  cdbe326d357f18b83d10322ba202d69f11b2f49e2d87ade0dc2be0c5c34f8e2a  python-pybind-2.6.1.tar.gz
> +sha256  f1bcc07caa568eb312411dde5308b1e250bd0e1bc020fae855bf9f43209940cc  python-pybind-2.8.1.tar.gz
>   # License files, locally calculated
>   sha256  83965b843b98f670d3a85bd041ed4b372c8ec50d7b4a5995a83ac697ba675dcb  LICENSE
> diff --git a/package/python-pybind/python-pybind.mk b/package/python-pybind/python-pybind.mk
> index 442ed3328f..cf6a642922 100644
> --- a/package/python-pybind/python-pybind.mk
> +++ b/package/python-pybind/python-pybind.mk
> @@ -4,7 +4,7 @@
>   #
>   ################################################################################
>   
> -PYTHON_PYBIND_VERSION = 2.6.1
> +PYTHON_PYBIND_VERSION = 2.8.1
>   PYTHON_PYBIND_SITE = $(call github,pybind,pybind11,v$(PYTHON_PYBIND_VERSION))
>   PYTHON_PYBIND_LICENSE = BSD-3-Clause
>   PYTHON_PYBIND_LICENSE_FILES = LICENSE
> 
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1

[Tomi Valkeinen]

On 07/01/2022 23:49, Arnout Vandecappelle wrote:
> 
> 
> On 03/01/2022 09:24, Tomi Valkeinen wrote:
>> Hi,
>>
>> On 06/12/2021 23:57, Yann E. MORIN wrote:
>>
>>> So I think we really, really need a package (like scipy) that actually
>>> makes use of pybind, so we can see how it works, and to fix it...
>>
>> Jfyi, there are two relatively simple packages in buildroot that can 
>> use pybind: kmsxx and rwmem. Both use meson for building.
>>
>> I can test the next version of this series with those packages.
> 
>   I tried to do that. Unfortunately, it turns out that both of those 
> packages use the CMake way of finding the pybind11 dependency, and 
> apparently our cross-compilation.conf doesn't support cmake yet... So 
> it's not that trivial.

Yes, I had a look at this also. kmsxx and rwmem use the plain meson 
dependency() to find the package. Works on Ubuntu, as Ubuntu's pybind11 
installs the cmake files.

I started testing with a more manual approach to finding pybind11, but 
didn't get it working yet.

  Tomi
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 3/3] support/testing: add test for python-pybind

[Guillaume Bres]

[-- Attachment #1.1: Type: text/plain, Size: 14422 bytes --]

Hello Arnout,

>In the future, please make sure that you don't claim authorship on a patch
that is actually from someone else!

well I don't understand how the first submitted test/example relates to a
real use case or demonstrates something viable.
For me pybind is a compile time feature to enhance the compiler capacity.
So I was really confused as to what was submitted 2 years ago could
actually demonstrate anything.
Hence the different and kind of tweaked approach that I proposed.

I think we will refer to these notes as the holy bible as of now
https://git.busybox.net/buildroot/commit/?id=a19b822239c36cb40a632899bb5c3e848a5fc81e

> Generated with scanpypi and converted to host.
that seems a useful tool to use

> I bumped [python-pybind] to 2.9.0 instead and applied to master, thanks.
thanks I did not know it was out there

Guillaume W. Bres
Software engineer
<guillaume.bressaix@gmail.com>


Le ven. 7 janv. 2022 à 22:57, Arnout Vandecappelle <arnout@mind.be> a
écrit :

>
>
> On 04/01/2022 13:39, guillaume.bressaix@gmail.com wrote:
> > From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com>
> >
> > python-pybind is a header only library, most of the time used as a
> host-package,
> > to customize the host/usr/python interpreter capacity (at build time)
> with C++ based features.
> >
> > Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com>
>
>   I applied to master with many changes:
>
>       - Retain python3 only.
>       - python-pybind is a target package, not host.
>       - Select python-pybind instead of depend.
>       - Simplify python-pybind-example package.
>       - Check in python-pybind-example build if pybind11.get_include()
>         produces output.
>       - Don't use python3 -m pybind11 --includes: it includes the main
> python
>         includes, which are for the host, not for the target.
>       - Use TestPythonPackageBase instead of open-coding something imported
>         with host python.
>
>   However, I hadn't noticed that the original author of this test was
> Adam. In
> the future, please make sure that you don't claim authorship on a patch
> that is
> actually from someone else!
>
>   Regards,
>   Arnout
>
> > ---
> > This test case enhances the /host/usr/python interpreter with a C++
> macro.
> >
> > The test delivers a C++ macro contained in its br-external/example.cpp.
> > We compile it against the host context.
> > We install it to $(HOST_DIR)/usr/lib/python3.9 as it is natively
> contained
> > in the host-python system path (import is easy).
> > The macro is an integer adder, written in C++ that we can now call in
> host-python.
> >
> > The test infrastructure should actually be declared
> "host-generic-package"
> > instead of "generic-package", to be consistent with Buildroot
> infrastructure.
> > But I am not able to get this to actually do anything, if I declare this
> package
> > as a host-generic and "Config.in.host" instead of "Config.in". To me,
> this is
> > not too important
> >
> > * TestPythonPybind allows testing for both Py2 and Py3 like it is done
> > in many other packages
> >
> > * TestPythonPybind does not inherit from TestPythonPackageBase as
> usually done
> >    (1) because I need the br2-external feature to ship the example.cpp
> containing the custom macro
> >    (2) because TestPythonPackageBase always involves the emulator, which
> we do not want in our context
> >
> > * Although TestPythonPybind does have a TestPythonBase.config base at
> the same time,
> > as it is most convenient to have a valid python context
> >
> > * TestPythonPybind uses subprocess, as similarly done in 'core' on
> 'infra' test cases,
> > because we do not involve the emulator
> >
> > * TestPythonPybind tests (1+2)=3 written in C++, used in Python 2/3
> > * TestPythonPybind tests a custom module attribute, written in C++, used
> in Python 2/3
> >
> > ---
> >   DEVELOPERS                                    |  3 ++
> >   .../br2-external/python-pybind/Config.in      |  1 +
> >   .../br2-external/python-pybind/external.desc  |  1 +
> >   .../br2-external/python-pybind/external.mk    |  1 +
> >   .../package/python-pybind-example/Config.in   |  5 ++
> >   .../package/python-pybind-example/example.cpp | 16 +++++++
> >   .../python-pybind-example.mk                  | 47 ++++++++++++++++++
> >   .../tests/package/sample_python_pybind.py     |  4 ++
> >   .../tests/package/test_python_pybind.py       | 48 +++++++++++++++++++
> >   9 files changed, 126 insertions(+)
> >   create mode 100644
> support/testing/tests/package/br2-external/python-pybind/Config.in
> >   create mode 100644
> support/testing/tests/package/br2-external/python-pybind/external.desc
> >   create mode 100644
> support/testing/tests/package/br2-external/python-pybind/external.mk
> >   create mode 100644
> support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/Config.in
> >   create mode 100644
> support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/example.cpp
> >   create mode 100644
> support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/
> python-pybind-example.mk
> >   create mode 100644
> support/testing/tests/package/sample_python_pybind.py
> >   create mode 100644 support/testing/tests/package/test_python_pybind.py
> >
> > diff --git a/DEVELOPERS b/DEVELOPERS
> > index 21e4f2a84c..130ac3db73 100644
> > --- a/DEVELOPERS
> > +++ b/DEVELOPERS
> > @@ -1120,6 +1120,9 @@ F:      package/liquid-dsp/
> >   F:  package/pixiewps/
> >   F:  package/python-pybind/
> >   F:  package/reaver/
> > +F:   support/testing/tests/package/br2-external/python-pybind
> > +F:   support/testing/tests/package/sample_python_pybind.py
> > +F:   support/testing/tests/package/test_python_pybind.py
> >
> >   N:  Guo Ren <ren_guo@c-sky.com>
> >   F:  arch/Config.in.csky
> > diff --git
> a/support/testing/tests/package/br2-external/python-pybind/Config.in
> b/support/testing/tests/package/br2-external/python-pybind/Config.in
> > new file mode 100644
> > index 0000000000..70c77157b3
> > --- /dev/null
> > +++ b/support/testing/tests/package/br2-external/python-pybind/Config.in
> > @@ -0,0 +1 @@
> > +source
> "$BR2_EXTERNAL_PYTHON_PYBIND_PATH/package/python-pybind-example/Config.in"
> > diff --git
> a/support/testing/tests/package/br2-external/python-pybind/external.desc
> b/support/testing/tests/package/br2-external/python-pybind/external.desc
> > new file mode 100644
> > index 0000000000..eef5e0f5a0
> > --- /dev/null
> > +++
> b/support/testing/tests/package/br2-external/python-pybind/external.desc
> > @@ -0,0 +1 @@
> > +name: PYTHON_PYBIND
> > diff --git a/support/testing/tests/package/br2-external/python-pybind/
> external.mk b/support/testing/tests/package/br2-external/python-pybind/
> external.mk
> > new file mode 100644
> > index 0000000000..3501f3135e
> > --- /dev/null
> > +++ b/support/testing/tests/package/br2-external/python-pybind/
> external.mk
> > @@ -0,0 +1 @@
> > +include $(sort $(wildcard
> $(BR2_EXTERNAL_PYTHON_PYBIND_PATH)/package/*/*.mk))
> > diff --git
> a/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/Config.in
> b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/Config.in
> > new file mode 100644
> > index 0000000000..773d021a53
> > --- /dev/null
> > +++
> b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/Config.in
> > @@ -0,0 +1,5 @@
> > +config BR2_PACKAGE_PYTHON_PYBIND_EXAMPLE
> > +     bool "python-pybind-example"
> > +     depends on BR2_PACKAGE_PYTHON_PYBIND
> > +     help
> > +       This test creates a cpp macro later used on target in python
> > diff --git
> a/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/example.cpp
> b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/example.cpp
> > new file mode 100644
> > index 0000000000..f2eea8e48d
> > --- /dev/null
> > +++
> b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/example.cpp
> > @@ -0,0 +1,16 @@
> > +#include <pybind11/pybind11.h>
> > +namespace py = pybind11;
> > +
> > +int add (int i, int j) {
> > +    return i + j;
> > +}
> > +
> > +PYBIND11_MODULE (example, m) {
> > +    // optional module description
> > +    m.doc() = "pybind11 example plugin";
> > +    // test a module method
> > +    m.def("add", &add, "example::add adds two integer numbers");
> > +    // test a module attribute
> > +    py::object hello = py::cast("Hello World");
> > +    m.attr("says") = hello;
> > +}
> > diff --git
> a/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/
> python-pybind-example.mk
> b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/
> python-pybind-example.mk
> > new file mode 100644
> > index 0000000000..85dd09d93f
> > --- /dev/null
> > +++
> b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/
> python-pybind-example.mk
> > @@ -0,0 +1,47 @@
> >
> +################################################################################
> > +#
> > +# python-pybind-example
> > +#
> >
> +################################################################################
> > +
> > +# this builds a C++ macro "add(a,b)"
> > +# that we expose to host-python with a custom install
> > +# and that the python test script will later use
> > +PYTHON_PYBIND_EXAMPLE_DEPENDENCIES = host-python-pybind
> > +
> > +PYTHON_PYBIND_EXAMPLE_CXX_FLAGS = $(HOST_CXXFLAGS)
> > +PYTHON_PYBIND_EXAMPLE_CXX_FLAGS += -Wall -shared
> > +PYTHON_PYBIND_EXAMPLE_CXX_FLAGS += -std=c++11 -fPIC
> > +
> > +ifeq ($(BR2_PACKAGE_PYTHON3),y)
> > +PYTHON_PYBIND_PYVER = python$(PYTHON3_VERSION_MAJOR)
> > +PYTHON_PYBIND_PYTHON = $(HOST_DIR)/usr/bin/python3
> > +else
> > +PYTHON_PYBIND_PYVER = python$(PYTHON_VERSION_MAJOR)
> > +PYTHON_PYBIND_PYTHON = $(HOST_DIR)/usr/bin/python2
> > +endif
> > +
> > +# retrieve pybind11 include paths
> > +PYTHON_PYBIND_EXAMPLE_CXX_FLAGS += \
> > +     $(shell $(PYTHON_PYBIND_PYTHON) -m pybind11 --includes)
> > +
> > +# .so to be installed must have exact suffix
> > +# otherwise import() in python will not work
> > +HOST_LIB_BINARY_SUFFIX = \
> > +     $(shell $(PYTHON_PYBIND_PYTHON)-config --extension-suffix)
> > +
> > +define PYTHON_PYBIND_EXAMPLE_BUILD_CMDS
> > +     $(INSTALL) $(PYTHON_PYBIND_EXAMPLE_PKGDIR)/example.cpp $(@D)
> > +     cd $(@D); \
> > +                     $(HOSTCXX) $(PYTHON_PYBIND_EXAMPLE_CXX_FLAGS) \
> > +                             example.cpp -o
> example$(HOST_LIB_BINARY_SUFFIX)
> > +endef
> > +
> > +# installs into host/python sys.path
> > +# so the new macro is naturally known
> > +define PYTHON_PYBIND_EXAMPLE_INSTALL_TARGET_CMDS
> > +     $(INSTALL) -D -m 755 $(@D)/example$(HOST_LIB_BINARY_SUFFIX) \
> > +
>  $(HOST_DIR)/usr/lib/$(PYTHON_PYBIND_PYVER)/example$(HOST_LIB_BINARY_SUFFIX)
> > +endef
> > +
> > +$(eval $(generic-package))
> > diff --git a/support/testing/tests/package/sample_python_pybind.py
> b/support/testing/tests/package/sample_python_pybind.py
> > new file mode 100644
> > index 0000000000..605c0bab15
> > --- /dev/null
> > +++ b/support/testing/tests/package/sample_python_pybind.py
> > @@ -0,0 +1,4 @@
> > +#!/usr/bin/env python
> > +import example
> > +print(example.add(1, 2))
> > +print(example.says)
> > diff --git a/support/testing/tests/package/test_python_pybind.py
> b/support/testing/tests/package/test_python_pybind.py
> > new file mode 100644
> > index 0000000000..7a68d6512a
> > --- /dev/null
> > +++ b/support/testing/tests/package/test_python_pybind.py
> > @@ -0,0 +1,48 @@
> > +import os
> > +import infra
> > +import subprocess
> > +from tests.package.test_python import TestPythonPackageBase
> > +
> > +class TestPythonPybind (infra.basetest.BRTest):
> > +     config = TestPythonPackageBase.config # python minimal requirements
> > +     sample_script = "tests/package/sample_python_pybind.py"
> > +     # ship examples macro & installs it to host
> > +     br2_external =
> [infra.filepath("tests/package/br2-external/python-pybind")]
> > +
> > +class TestPythonPy2Pybind (TestPythonPybind):
> > +     __test__ = True
> > +     config = TestPythonPybind.config + \
> > +             """
> > +             BR2_PACKAGE_PYTHON=y
> > +             BR2_PACKAGE_PYTHON_PYBIND=y
> > +             BR2_PACKAGE_PYTHON_PYBIND_EXAMPLE=y
> > +             """
> > +
> > +     def test_run(self):
> > +             host_interpreter =
> os.path.join(self.builddir,"host","usr","bin","python")
> > +             # test new c++->py macro exposed to host interpreter
> > +             cmd = [host_interpreter,
> infra.filepath(self.sample_script)]
> > +             output = subprocess.check_output(cmd)
> > +             result = int(output.splitlines()[0])
> > +             attribute = output.splitlines()[1].decode("utf-8")
> > +             self.assertEqual(result,3) # 2+1 using '+' macro from cpp
> > +             self.assertEqual(attribute,"Hello World") # macro special
> attribute
> > +
> > +class TestPythonPy3Pybind (TestPythonPybind):
> > +     __test__ = True
> > +     config = TestPythonPybind.config + \
> > +             """
> > +             BR2_PACKAGE_PYTHON3=y
> > +             BR2_PACKAGE_PYTHON_PYBIND=y
> > +             BR2_PACKAGE_PYTHON_PYBIND_EXAMPLE=y
> > +             """
> > +
> > +     def test_run(self):
> > +             host_interpreter =
> os.path.join(self.builddir,"host","usr","bin","python")
> > +             # test new c++->py macro exposed to host interpreter
> > +             cmd = [host_interpreter,
> infra.filepath(self.sample_script)]
> > +             output = subprocess.check_output(cmd)
> > +             result = int(output.splitlines()[0])
> > +             attribute = output.splitlines()[1].decode("utf-8")
> > +             self.assertEqual(result,3) # 2+1 using '+' macro from cpp
> > +             self.assertEqual(attribute,"Hello World") # macro special
> attribute
> >
>

[-- Attachment #1.2: Type: text/html, Size: 18282 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1

[Arnout Vandecappelle]

On 08/01/2022 11:09, Tomi Valkeinen wrote:
> On 07/01/2022 23:49, Arnout Vandecappelle wrote:
>>
>>
>> On 03/01/2022 09:24, Tomi Valkeinen wrote:
>>> Hi,
>>>
>>> On 06/12/2021 23:57, Yann E. MORIN wrote:
>>>
>>>> So I think we really, really need a package (like scipy) that actually
>>>> makes use of pybind, so we can see how it works, and to fix it...
>>>
>>> Jfyi, there are two relatively simple packages in buildroot that can use 
>>> pybind: kmsxx and rwmem. Both use meson for building.
>>>
>>> I can test the next version of this series with those packages.
>>
>>   I tried to do that. Unfortunately, it turns out that both of those packages 
>> use the CMake way of finding the pybind11 dependency, and apparently our 
>> cross-compilation.conf doesn't support cmake yet... So it's not that trivial.
> 
> Yes, I had a look at this also. kmsxx and rwmem use the plain meson dependency() 
> to find the package. Works on Ubuntu, as Ubuntu's pybind11 installs the cmake 
> files.

  I also made sure that python-pybind installs the cmake files. I just haven't 
tested if they actually work. I already noticed that there's one of the files 
that is not cross-compile friendly because it found /usr/bin/python. But maybe 
that one isn't actually used in practice.

> I started testing with a more manual approach to finding pybind11, but didn't 
> get it working yet.

  No, the cmake approach should be fine. We just have to update the meson 
cross-compilation.conf so it points to STAGING_DIR/usr/share/cmake.

  Regards,
  Arnout

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot