Upstream scipy recipe to meta-openembedded

Upstream scipy recipe to meta-openembedded

[Gregory Anders]

Hello all,

I wrote and maintain a recipe to provide scipy in the OpenEmbedded build 
system [1]. I would like to upstream this recipe into meta-python so 
that it is more broadly available to those who need it as well as to 
reduce my own maintenance burden.

Before sending a patch, however, I'd like to solicit feedback from the 
meta-oe community on how best to integrate this recipe into the 
meta-openembedded project. If you look at the source for the recipe at 
[1] you will see that there are a few steps/hacks that need to be done 
in order to get scipy to correctly cross-compile:

1. We have to add a CMake parameter to lapack ("-DCBLAS=ON"). This 
    causes LAPACK to build the C BLAS library instead of the Fortran BLAS 
    library. This is necessary as scipy tries to link against `cblas_*` 
    functions.
2. We have to patch numpy to force it not to include rpaths in the 
    compiled shared object file. Otherwise the bitbake QA fails as it 
    detects rpaths from the build system.
3. Two variables must be set in either the local.conf or distro.conf 
    file to enable Fortran support. These can't be added in the scipy 
    recipe directly (at least, last time I tried it they couldn't be) so 
    users have to manually add these themselves.

I am by no means an expert in bitbake or in the broader OE build 
environment. If you know of a better solution to any of the problems 
outlined above, please share them. I am also more than happy to provide 
more detail if anyone has any questions.

Ultimately, the ideal goal is to have a single 'python3_scipy.bb' recipe 
file that I can send as a patch for inclusion in the meta-python layer 
rather than maintaining an entirely separate meta-scipy layer.

Thanks!

Greg

[1]: https://github.com/gpanders/meta-scipy

Re: [oe] Upstream scipy recipe to meta-openembedded

[Khem Raj]

On 7/12/21 8:19 AM, Gregory Anders wrote:
> Hello all,
> 
> I wrote and maintain a recipe to provide scipy in the OpenEmbedded build 
> system [1]. I would like to upstream this recipe into meta-python so 
> that it is more broadly available to those who need it as well as to 
> reduce my own maintenance burden.
> 
> Before sending a patch, however, I'd like to solicit feedback from the 
> meta-oe community on how best to integrate this recipe into the 
> meta-openembedded project. If you look at the source for the recipe at 
> [1] you will see that there are a few steps/hacks that need to be done 
> in order to get scipy to correctly cross-compile:
> 
> 1. We have to add a CMake parameter to lapack ("-DCBLAS=ON"). This    
> causes LAPACK to build the C BLAS library instead of the Fortran BLAS    
> library. This is necessary as scipy tries to link against `cblas_*`    
> functions.
> 2. We have to patch numpy to force it not to include rpaths in the    
> compiled shared object file. Otherwise the bitbake QA fails as it    
> detects rpaths from the build system.
> 3. Two variables must be set in either the local.conf or distro.conf    
> file to enable Fortran support. These can't be added in the scipy    
> recipe directly (at least, last time I tried it they couldn't be) so    
> users have to manually add these themselves.
> 
> I am by no means an expert in bitbake or in the broader OE build 
> environment. If you know of a better solution to any of the problems 
> outlined above, please share them. I am also more than happy to provide 
> more detail if anyone has any questions.
> 
> Ultimately, the ideal goal is to have a single 'python3_scipy.bb' recipe 
> file that I can send as a patch for inclusion in the meta-python layer 
> rather than maintaining an entirely separate meta-scipy layer.
> 

I think meta-python is right layer for this.

> Thanks!
> 
> Greg
> 
> [1]: https://github.com/gpanders/meta-scipy
> 
> 
> 
> 

Re: [oe] Upstream scipy recipe to meta-openembedded

[Zoltan Boszormenyi]

2021. 07. 12. 18:01 keltezéssel, Khem Raj írta:
> 
> 
> On 7/12/21 8:19 AM, Gregory Anders wrote:
>> Hello all,
>>
>> I wrote and maintain a recipe to provide scipy in the OpenEmbedded build system [1]. I 
>> would like to upstream this recipe into meta-python so that it is more broadly available 
>> to those who need it as well as to reduce my own maintenance burden.
>>
>> Before sending a patch, however, I'd like to solicit feedback from the meta-oe community 
>> on how best to integrate this recipe into the meta-openembedded project. If you look at 
>> the source for the recipe at [1] you will see that there are a few steps/hacks that need 
>> to be done in order to get scipy to correctly cross-compile:
>>
>> 1. We have to add a CMake parameter to lapack ("-DCBLAS=ON"). This causes LAPACK to 
>> build the C BLAS library instead of the Fortran BLAS library. This is necessary as scipy 
>> tries to link against `cblas_*` functions.

FYI, there's another layer (https://github.com/tuxable-ltd/meta-scikit-learn that
I had to fork and make hardknott-compatible) that uses python3-scipy as a build
and runtime dependency for python3-scikit-learn so import sklearn.linear_model"
is possible. That layer also extends python3-scipy with a native build variant
which doesn't exist in your layer AND patches lapack further to use openblas.

>> 2. We have to patch numpy to force it not to include rpaths in the compiled shared 
>> object file. Otherwise the bitbake QA fails as it detects rpaths from the build system.
>> 3. Two variables must be set in either the local.conf or distro.conf file to enable 
>> Fortran support. These can't be added in the scipy recipe directly (at least, last time 
>> I tried it they couldn't be) so users have to manually add these themselves.
>>
>> I am by no means an expert in bitbake or in the broader OE build environment. If you 
>> know of a better solution to any of the problems outlined above, please share them. I am 
>> also more than happy to provide more detail if anyone has any questions.
>>
>> Ultimately, the ideal goal is to have a single 'python3_scipy.bb' recipe file that I can 
>> send as a patch for inclusion in the meta-python layer rather than maintaining an 
>> entirely separate meta-scipy layer.
>>
> 
> I think meta-python is right layer for this.
> 
>> Thanks!
>>
>> Greg
>>
>> [1]: https://github.com/gpanders/meta-scipy
>>
>>
>>
>>
> 
> 
> 
> 

Re: [oe] Upstream scipy recipe to meta-openembedded

[Jack Mitchell]

On 13/07/2021 11:20, Zoltan Boszormenyi via lists.openembedded.org wrote:
> 2021. 07. 12. 18:01 keltezéssel, Khem Raj írta:
>>
>>
>> On 7/12/21 8:19 AM, Gregory Anders wrote:
>>> Hello all,
>>>
>>> I wrote and maintain a recipe to provide scipy in the OpenEmbedded
>>> build system [1]. I would like to upstream this recipe into
>>> meta-python so that it is more broadly available to those who need it
>>> as well as to reduce my own maintenance burden.
>>>
>>> Before sending a patch, however, I'd like to solicit feedback from
>>> the meta-oe community on how best to integrate this recipe into the
>>> meta-openembedded project. If you look at the source for the recipe
>>> at [1] you will see that there are a few steps/hacks that need to be
>>> done in order to get scipy to correctly cross-compile:
>>>
>>> 1. We have to add a CMake parameter to lapack ("-DCBLAS=ON"). This
>>> causes LAPACK to build the C BLAS library instead of the Fortran BLAS
>>> library. This is necessary as scipy tries to link against `cblas_*`
>>> functions.
> 
> FYI, there's another layer
> (https://github.com/tuxable-ltd/meta-scikit-learn that
> I had to fork and make hardknott-compatible) that uses python3-scipy as
> a build
> and runtime dependency for python3-scikit-learn so import
> sklearn.linear_model"
> is possible. That layer also extends python3-scipy with a native build
> variant
> which doesn't exist in your layer AND patches lapack further to use
> openblas.

Yes, that would be my layer. I had never intended to upstream the
scikit-learn portions as I don't believe it is maintainable in a generic
layer due to the niche use-cases and awkwardness of the build system
itself and it's dependencies, in addition to the very large build time
which would put strain on auto-builders.

What I had intended to do though, which I haven't got round to yet is
try to provide a virtual provider for blas/cblas in meta-oe which would
allow openblas to be properly upstreamed and then packageconfigs and
preferred versions used for selection as distro policy.

-- 
Jack Mitchell, Consultant
https://www.tuxable.co.uk

> 
>>> 2. We have to patch numpy to force it not to include rpaths in the
>>> compiled shared object file. Otherwise the bitbake QA fails as it
>>> detects rpaths from the build system.
>>> 3. Two variables must be set in either the local.conf or distro.conf
>>> file to enable Fortran support. These can't be added in the scipy
>>> recipe directly (at least, last time I tried it they couldn't be) so
>>> users have to manually add these themselves.
>>>
>>> I am by no means an expert in bitbake or in the broader OE build
>>> environment. If you know of a better solution to any of the problems
>>> outlined above, please share them. I am also more than happy to
>>> provide more detail if anyone has any questions.
>>>
>>> Ultimately, the ideal goal is to have a single 'python3_scipy.bb'
>>> recipe file that I can send as a patch for inclusion in the
>>> meta-python layer rather than maintaining an entirely separate
>>> meta-scipy layer.
>>>
>>
>> I think meta-python is right layer for this.
>>
>>> Thanks!
>>>
>>> Greg
>>>
>>> [1]: https://github.com/gpanders/meta-scipy
>>>
>>>
>>>
>>>
>>
>>
>>
>>
> 
> 
> 
> 
> 

Re: [oe] Upstream scipy recipe to meta-openembedded

[Gregory Anders]

On Tue, 13 Jul 2021 11:46 +0100, Jack Mitchell wrote:
>On 13/07/2021 11:20, Zoltan Boszormenyi via lists.openembedded.org wrote:
>>
>> FYI, there's another layer
>> (https://github.com/tuxable-ltd/meta-scikit-learn that
>> I had to fork and make hardknott-compatible) that uses python3-scipy as
>> a build
>> and runtime dependency for python3-scikit-learn so import
>> sklearn.linear_model"
>> is possible. That layer also extends python3-scipy with a native build
>> variant
>> which doesn't exist in your layer AND patches lapack further to use
>> openblas.
>
>Yes, that would be my layer. I had never intended to upstream the
>scikit-learn portions as I don't believe it is maintainable in a generic
>layer due to the niche use-cases and awkwardness of the build system
>itself and it's dependencies, in addition to the very large build time
>which would put strain on auto-builders.
>
>What I had intended to do though, which I haven't got round to yet is
>try to provide a virtual provider for blas/cblas in meta-oe which would
>allow openblas to be properly upstreamed and then packageconfigs and
>preferred versions used for selection as distro policy.
>

Thanks, I wasn't aware of this project. And it looks like you actually 
sent a PR to my meta-scipy layer at one point, so thank you!

Jack, would you mind if I simply directed people from my meta-scipy 
repository to your meta-scikit-learn layer? I am no longer using scipy 
in OE so maintaining this layer (e.g. testing contributions and updating 
for new versions) is falling behind.

If not then that is totally fine, I will just add a call for a new 
maintainer.

Thanks,

Greg

Re: [oe] Upstream scipy recipe to meta-openembedded

[Jack Mitchell]

On 13/07/2021 14:09, Gregory Anders wrote:
> On Tue, 13 Jul 2021 11:46 +0100, Jack Mitchell wrote:
>> On 13/07/2021 11:20, Zoltan Boszormenyi via lists.openembedded.org wrote:
>>>
>>> FYI, there's another layer
>>> (https://github.com/tuxable-ltd/meta-scikit-learn that
>>> I had to fork and make hardknott-compatible) that uses python3-scipy as
>>> a build
>>> and runtime dependency for python3-scikit-learn so import
>>> sklearn.linear_model"
>>> is possible. That layer also extends python3-scipy with a native build
>>> variant
>>> which doesn't exist in your layer AND patches lapack further to use
>>> openblas.
>>
>> Yes, that would be my layer. I had never intended to upstream the
>> scikit-learn portions as I don't believe it is maintainable in a generic
>> layer due to the niche use-cases and awkwardness of the build system
>> itself and it's dependencies, in addition to the very large build time
>> which would put strain on auto-builders.
>>
>> What I had intended to do though, which I haven't got round to yet is
>> try to provide a virtual provider for blas/cblas in meta-oe which would
>> allow openblas to be properly upstreamed and then packageconfigs and
>> preferred versions used for selection as distro policy.
>>
> 
> Thanks, I wasn't aware of this project. And it looks like you actually
> sent a PR to my meta-scipy layer at one point, so thank you!
> 
> Jack, would you mind if I simply directed people from my meta-scipy
> repository to your meta-scikit-learn layer? I am no longer using scipy
> in OE so maintaining this layer (e.g. testing contributions and updating
> for new versions) is falling behind.
> 
> If not then that is totally fine, I will just add a call for a new
> maintainer.
> 
> Thanks,
> 
> Greg

Hi Greg,

I should be able to take it over as I'll continue to be using it in the
future. How do you want to transfer ownership, shall I fork it and you
point to the new repo on the master branch, or can you get github to
re-direct to the new home?

Regards,
Jack.

-- 
Jack Mitchell, Consultant
https://www.tuxable.co.uk

Re: [oe] Upstream scipy recipe to meta-openembedded

[Gregory Anders]

On Thu, 15 Jul 2021 09:44 +0100, Jack Mitchell wrote:
>Hi Greg,
>
>I should be able to take it over as I'll continue to be using it in the
>future. How do you want to transfer ownership, shall I fork it and you
>point to the new repo on the master branch, or can you get github to
>re-direct to the new home?

Thanks Jack. I believe I can simply transfer the repo to your user on 
GitHub. You should receive an email asking for your confirmation.