* Upstream scipy recipe to meta-openembedded
@ 2021-07-12 15:19 Gregory Anders
2021-07-12 16:01 ` [oe] " Khem Raj
0 siblings, 1 reply; 7+ messages in thread
From: Gregory Anders @ 2021-07-12 15:19 UTC (permalink / raw)
To: openembedded-devel
Hello all,
I wrote and maintain a recipe to provide scipy in the OpenEmbedded build
system [1]. I would like to upstream this recipe into meta-python so
that it is more broadly available to those who need it as well as to
reduce my own maintenance burden.
Before sending a patch, however, I'd like to solicit feedback from the
meta-oe community on how best to integrate this recipe into the
meta-openembedded project. If you look at the source for the recipe at
[1] you will see that there are a few steps/hacks that need to be done
in order to get scipy to correctly cross-compile:
1. We have to add a CMake parameter to lapack ("-DCBLAS=ON"). This
causes LAPACK to build the C BLAS library instead of the Fortran BLAS
library. This is necessary as scipy tries to link against `cblas_*`
functions.
2. We have to patch numpy to force it not to include rpaths in the
compiled shared object file. Otherwise the bitbake QA fails as it
detects rpaths from the build system.
3. Two variables must be set in either the local.conf or distro.conf
file to enable Fortran support. These can't be added in the scipy
recipe directly (at least, last time I tried it they couldn't be) so
users have to manually add these themselves.
I am by no means an expert in bitbake or in the broader OE build
environment. If you know of a better solution to any of the problems
outlined above, please share them. I am also more than happy to provide
more detail if anyone has any questions.
Ultimately, the ideal goal is to have a single 'python3_scipy.bb' recipe
file that I can send as a patch for inclusion in the meta-python layer
rather than maintaining an entirely separate meta-scipy layer.
Thanks!
Greg
[1]: https://github.com/gpanders/meta-scipy
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [oe] Upstream scipy recipe to meta-openembedded
2021-07-12 15:19 Upstream scipy recipe to meta-openembedded Gregory Anders
@ 2021-07-12 16:01 ` Khem Raj
2021-07-13 10:20 ` Zoltan Boszormenyi
0 siblings, 1 reply; 7+ messages in thread
From: Khem Raj @ 2021-07-12 16:01 UTC (permalink / raw)
To: Gregory Anders, openembedded-devel
On 7/12/21 8:19 AM, Gregory Anders wrote:
> Hello all,
>
> I wrote and maintain a recipe to provide scipy in the OpenEmbedded build
> system [1]. I would like to upstream this recipe into meta-python so
> that it is more broadly available to those who need it as well as to
> reduce my own maintenance burden.
>
> Before sending a patch, however, I'd like to solicit feedback from the
> meta-oe community on how best to integrate this recipe into the
> meta-openembedded project. If you look at the source for the recipe at
> [1] you will see that there are a few steps/hacks that need to be done
> in order to get scipy to correctly cross-compile:
>
> 1. We have to add a CMake parameter to lapack ("-DCBLAS=ON"). This
> causes LAPACK to build the C BLAS library instead of the Fortran BLAS
> library. This is necessary as scipy tries to link against `cblas_*`
> functions.
> 2. We have to patch numpy to force it not to include rpaths in the
> compiled shared object file. Otherwise the bitbake QA fails as it
> detects rpaths from the build system.
> 3. Two variables must be set in either the local.conf or distro.conf
> file to enable Fortran support. These can't be added in the scipy
> recipe directly (at least, last time I tried it they couldn't be) so
> users have to manually add these themselves.
>
> I am by no means an expert in bitbake or in the broader OE build
> environment. If you know of a better solution to any of the problems
> outlined above, please share them. I am also more than happy to provide
> more detail if anyone has any questions.
>
> Ultimately, the ideal goal is to have a single 'python3_scipy.bb' recipe
> file that I can send as a patch for inclusion in the meta-python layer
> rather than maintaining an entirely separate meta-scipy layer.
>
I think meta-python is right layer for this.
> Thanks!
>
> Greg
>
> [1]: https://github.com/gpanders/meta-scipy
>
>
>
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [oe] Upstream scipy recipe to meta-openembedded
2021-07-12 16:01 ` [oe] " Khem Raj
@ 2021-07-13 10:20 ` Zoltan Boszormenyi
2021-07-13 10:46 ` Jack Mitchell
0 siblings, 1 reply; 7+ messages in thread
From: Zoltan Boszormenyi @ 2021-07-13 10:20 UTC (permalink / raw)
To: Khem Raj, Gregory Anders, openembedded-devel
2021. 07. 12. 18:01 keltezéssel, Khem Raj írta:
>
>
> On 7/12/21 8:19 AM, Gregory Anders wrote:
>> Hello all,
>>
>> I wrote and maintain a recipe to provide scipy in the OpenEmbedded build system [1]. I
>> would like to upstream this recipe into meta-python so that it is more broadly available
>> to those who need it as well as to reduce my own maintenance burden.
>>
>> Before sending a patch, however, I'd like to solicit feedback from the meta-oe community
>> on how best to integrate this recipe into the meta-openembedded project. If you look at
>> the source for the recipe at [1] you will see that there are a few steps/hacks that need
>> to be done in order to get scipy to correctly cross-compile:
>>
>> 1. We have to add a CMake parameter to lapack ("-DCBLAS=ON"). This causes LAPACK to
>> build the C BLAS library instead of the Fortran BLAS library. This is necessary as scipy
>> tries to link against `cblas_*` functions.
FYI, there's another layer (https://github.com/tuxable-ltd/meta-scikit-learn that
I had to fork and make hardknott-compatible) that uses python3-scipy as a build
and runtime dependency for python3-scikit-learn so import sklearn.linear_model"
is possible. That layer also extends python3-scipy with a native build variant
which doesn't exist in your layer AND patches lapack further to use openblas.
>> 2. We have to patch numpy to force it not to include rpaths in the compiled shared
>> object file. Otherwise the bitbake QA fails as it detects rpaths from the build system.
>> 3. Two variables must be set in either the local.conf or distro.conf file to enable
>> Fortran support. These can't be added in the scipy recipe directly (at least, last time
>> I tried it they couldn't be) so users have to manually add these themselves.
>>
>> I am by no means an expert in bitbake or in the broader OE build environment. If you
>> know of a better solution to any of the problems outlined above, please share them. I am
>> also more than happy to provide more detail if anyone has any questions.
>>
>> Ultimately, the ideal goal is to have a single 'python3_scipy.bb' recipe file that I can
>> send as a patch for inclusion in the meta-python layer rather than maintaining an
>> entirely separate meta-scipy layer.
>>
>
> I think meta-python is right layer for this.
>
>> Thanks!
>>
>> Greg
>>
>> [1]: https://github.com/gpanders/meta-scipy
>>
>>
>>
>>
>
>
>
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [oe] Upstream scipy recipe to meta-openembedded
2021-07-13 10:20 ` Zoltan Boszormenyi
@ 2021-07-13 10:46 ` Jack Mitchell
2021-07-13 13:09 ` Gregory Anders
0 siblings, 1 reply; 7+ messages in thread
From: Jack Mitchell @ 2021-07-13 10:46 UTC (permalink / raw)
To: zboszor, Khem Raj, Gregory Anders, openembedded-devel
On 13/07/2021 11:20, Zoltan Boszormenyi via lists.openembedded.org wrote:
> 2021. 07. 12. 18:01 keltezéssel, Khem Raj írta:
>>
>>
>> On 7/12/21 8:19 AM, Gregory Anders wrote:
>>> Hello all,
>>>
>>> I wrote and maintain a recipe to provide scipy in the OpenEmbedded
>>> build system [1]. I would like to upstream this recipe into
>>> meta-python so that it is more broadly available to those who need it
>>> as well as to reduce my own maintenance burden.
>>>
>>> Before sending a patch, however, I'd like to solicit feedback from
>>> the meta-oe community on how best to integrate this recipe into the
>>> meta-openembedded project. If you look at the source for the recipe
>>> at [1] you will see that there are a few steps/hacks that need to be
>>> done in order to get scipy to correctly cross-compile:
>>>
>>> 1. We have to add a CMake parameter to lapack ("-DCBLAS=ON"). This
>>> causes LAPACK to build the C BLAS library instead of the Fortran BLAS
>>> library. This is necessary as scipy tries to link against `cblas_*`
>>> functions.
>
> FYI, there's another layer
> (https://github.com/tuxable-ltd/meta-scikit-learn that
> I had to fork and make hardknott-compatible) that uses python3-scipy as
> a build
> and runtime dependency for python3-scikit-learn so import
> sklearn.linear_model"
> is possible. That layer also extends python3-scipy with a native build
> variant
> which doesn't exist in your layer AND patches lapack further to use
> openblas.
Yes, that would be my layer. I had never intended to upstream the
scikit-learn portions as I don't believe it is maintainable in a generic
layer due to the niche use-cases and awkwardness of the build system
itself and it's dependencies, in addition to the very large build time
which would put strain on auto-builders.
What I had intended to do though, which I haven't got round to yet is
try to provide a virtual provider for blas/cblas in meta-oe which would
allow openblas to be properly upstreamed and then packageconfigs and
preferred versions used for selection as distro policy.
--
Jack Mitchell, Consultant
https://www.tuxable.co.uk
>
>>> 2. We have to patch numpy to force it not to include rpaths in the
>>> compiled shared object file. Otherwise the bitbake QA fails as it
>>> detects rpaths from the build system.
>>> 3. Two variables must be set in either the local.conf or distro.conf
>>> file to enable Fortran support. These can't be added in the scipy
>>> recipe directly (at least, last time I tried it they couldn't be) so
>>> users have to manually add these themselves.
>>>
>>> I am by no means an expert in bitbake or in the broader OE build
>>> environment. If you know of a better solution to any of the problems
>>> outlined above, please share them. I am also more than happy to
>>> provide more detail if anyone has any questions.
>>>
>>> Ultimately, the ideal goal is to have a single 'python3_scipy.bb'
>>> recipe file that I can send as a patch for inclusion in the
>>> meta-python layer rather than maintaining an entirely separate
>>> meta-scipy layer.
>>>
>>
>> I think meta-python is right layer for this.
>>
>>> Thanks!
>>>
>>> Greg
>>>
>>> [1]: https://github.com/gpanders/meta-scipy
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>
>
>
>
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [oe] Upstream scipy recipe to meta-openembedded
2021-07-13 10:46 ` Jack Mitchell
@ 2021-07-13 13:09 ` Gregory Anders
2021-07-15 8:44 ` Jack Mitchell
0 siblings, 1 reply; 7+ messages in thread
From: Gregory Anders @ 2021-07-13 13:09 UTC (permalink / raw)
To: Jack Mitchell; +Cc: zboszor, Khem Raj, openembedded-devel
On Tue, 13 Jul 2021 11:46 +0100, Jack Mitchell wrote:
>On 13/07/2021 11:20, Zoltan Boszormenyi via lists.openembedded.org wrote:
>>
>> FYI, there's another layer
>> (https://github.com/tuxable-ltd/meta-scikit-learn that
>> I had to fork and make hardknott-compatible) that uses python3-scipy as
>> a build
>> and runtime dependency for python3-scikit-learn so import
>> sklearn.linear_model"
>> is possible. That layer also extends python3-scipy with a native build
>> variant
>> which doesn't exist in your layer AND patches lapack further to use
>> openblas.
>
>Yes, that would be my layer. I had never intended to upstream the
>scikit-learn portions as I don't believe it is maintainable in a generic
>layer due to the niche use-cases and awkwardness of the build system
>itself and it's dependencies, in addition to the very large build time
>which would put strain on auto-builders.
>
>What I had intended to do though, which I haven't got round to yet is
>try to provide a virtual provider for blas/cblas in meta-oe which would
>allow openblas to be properly upstreamed and then packageconfigs and
>preferred versions used for selection as distro policy.
>
Thanks, I wasn't aware of this project. And it looks like you actually
sent a PR to my meta-scipy layer at one point, so thank you!
Jack, would you mind if I simply directed people from my meta-scipy
repository to your meta-scikit-learn layer? I am no longer using scipy
in OE so maintaining this layer (e.g. testing contributions and updating
for new versions) is falling behind.
If not then that is totally fine, I will just add a call for a new
maintainer.
Thanks,
Greg
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [oe] Upstream scipy recipe to meta-openembedded
2021-07-13 13:09 ` Gregory Anders
@ 2021-07-15 8:44 ` Jack Mitchell
2021-07-15 13:08 ` Gregory Anders
0 siblings, 1 reply; 7+ messages in thread
From: Jack Mitchell @ 2021-07-15 8:44 UTC (permalink / raw)
To: Gregory Anders; +Cc: zboszor, Khem Raj, openembedded-devel
On 13/07/2021 14:09, Gregory Anders wrote:
> On Tue, 13 Jul 2021 11:46 +0100, Jack Mitchell wrote:
>> On 13/07/2021 11:20, Zoltan Boszormenyi via lists.openembedded.org wrote:
>>>
>>> FYI, there's another layer
>>> (https://github.com/tuxable-ltd/meta-scikit-learn that
>>> I had to fork and make hardknott-compatible) that uses python3-scipy as
>>> a build
>>> and runtime dependency for python3-scikit-learn so import
>>> sklearn.linear_model"
>>> is possible. That layer also extends python3-scipy with a native build
>>> variant
>>> which doesn't exist in your layer AND patches lapack further to use
>>> openblas.
>>
>> Yes, that would be my layer. I had never intended to upstream the
>> scikit-learn portions as I don't believe it is maintainable in a generic
>> layer due to the niche use-cases and awkwardness of the build system
>> itself and it's dependencies, in addition to the very large build time
>> which would put strain on auto-builders.
>>
>> What I had intended to do though, which I haven't got round to yet is
>> try to provide a virtual provider for blas/cblas in meta-oe which would
>> allow openblas to be properly upstreamed and then packageconfigs and
>> preferred versions used for selection as distro policy.
>>
>
> Thanks, I wasn't aware of this project. And it looks like you actually
> sent a PR to my meta-scipy layer at one point, so thank you!
>
> Jack, would you mind if I simply directed people from my meta-scipy
> repository to your meta-scikit-learn layer? I am no longer using scipy
> in OE so maintaining this layer (e.g. testing contributions and updating
> for new versions) is falling behind.
>
> If not then that is totally fine, I will just add a call for a new
> maintainer.
>
> Thanks,
>
> Greg
Hi Greg,
I should be able to take it over as I'll continue to be using it in the
future. How do you want to transfer ownership, shall I fork it and you
point to the new repo on the master branch, or can you get github to
re-direct to the new home?
Regards,
Jack.
--
Jack Mitchell, Consultant
https://www.tuxable.co.uk
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [oe] Upstream scipy recipe to meta-openembedded
2021-07-15 8:44 ` Jack Mitchell
@ 2021-07-15 13:08 ` Gregory Anders
0 siblings, 0 replies; 7+ messages in thread
From: Gregory Anders @ 2021-07-15 13:08 UTC (permalink / raw)
To: Jack Mitchell; +Cc: zboszor, Khem Raj, openembedded-devel
On Thu, 15 Jul 2021 09:44 +0100, Jack Mitchell wrote:
>Hi Greg,
>
>I should be able to take it over as I'll continue to be using it in the
>future. How do you want to transfer ownership, shall I fork it and you
>point to the new repo on the master branch, or can you get github to
>re-direct to the new home?
Thanks Jack. I believe I can simply transfer the repo to your user on
GitHub. You should receive an email asking for your confirmation.
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2021-07-15 13:08 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-07-12 15:19 Upstream scipy recipe to meta-openembedded Gregory Anders
2021-07-12 16:01 ` [oe] " Khem Raj
2021-07-13 10:20 ` Zoltan Boszormenyi
2021-07-13 10:46 ` Jack Mitchell
2021-07-13 13:09 ` Gregory Anders
2021-07-15 8:44 ` Jack Mitchell
2021-07-15 13:08 ` Gregory Anders
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.