[PATCH 1/2] nativesdk-packagegroup-python: buildtools-tarball: Introduce NativeSDK package group

[PATCH 1/2] nativesdk-packagegroup-python: buildtools-tarball: Introduce NativeSDK package group

[Marek Vasut]

Introduce and use nativesdk-packagegroup-python , which contains all
necessary python components for the native SDK. Make use of this
package group in buildtools-tarball.bb immediatelly.

Signed-off-by: Marek Vasut <marex@denx.de>
---
 meta/recipes-core/meta/buildtools-tarball.bb       | 28 +----------------
 .../python/nativesdk-packagegroup-python.bb        | 36 ++++++++++++++++++++++
 2 files changed, 37 insertions(+), 27 deletions(-)
 create mode 100644 meta/recipes-devtools/python/nativesdk-packagegroup-python.bb

diff --git a/meta/recipes-core/meta/buildtools-tarball.bb b/meta/recipes-core/meta/buildtools-tarball.bb
index 62e1e0b..7d20238 100644
--- a/meta/recipes-core/meta/buildtools-tarball.bb
+++ b/meta/recipes-core/meta/buildtools-tarball.bb
@@ -8,33 +8,7 @@ LIC_FILES_CHKSUM = "file://${COREBASE}/LICENSE;md5=4d92cd373abda3937c2bc47fbc49d
 TOOLCHAIN_TARGET_TASK ?= ""
 
 TOOLCHAIN_HOST_TASK ?= "\
-    nativesdk-python-core \
-    nativesdk-python-textutils \
-    nativesdk-python-sqlite3 \
-    nativesdk-python-pickle \
-    nativesdk-python-logging \
-    nativesdk-python-elementtree \
-    nativesdk-python-curses \
-    nativesdk-python-compile \
-    nativesdk-python-compiler \
-    nativesdk-python-fcntl \
-    nativesdk-python-shell \
-    nativesdk-python-misc \
-    nativesdk-python-multiprocessing \
-    nativesdk-python-subprocess \
-    nativesdk-python-xmlrpc \
-    nativesdk-python-netclient \
-    nativesdk-python-netserver \
-    nativesdk-python-distutils \
-    nativesdk-python-unixadmin \
-    nativesdk-python-compression \
-    nativesdk-python-json \
-    nativesdk-python-unittest \
-    nativesdk-python-mmap \
-    nativesdk-python-difflib \
-    nativesdk-python-pprint \
-    nativesdk-python-git \
-    nativesdk-python-pkgutil \
+    nativesdk-packagegroup-python \
     nativesdk-ncurses-terminfo-base \
     nativesdk-chrpath \
     nativesdk-tar \
diff --git a/meta/recipes-devtools/python/nativesdk-packagegroup-python.bb b/meta/recipes-devtools/python/nativesdk-packagegroup-python.bb
new file mode 100644
index 0000000..fdc1006
--- /dev/null
+++ b/meta/recipes-devtools/python/nativesdk-packagegroup-python.bb
@@ -0,0 +1,36 @@
+SUMMARY = "NativeSDK Python package group"
+DESCRIPTION = "NativeSDK Python package group"
+PR = "r0"
+LICENSE = "MIT"
+
+inherit packagegroup
+
+RDEPENDS_${PN} = "				\
+	nativesdk-python-core			\
+	nativesdk-python-textutils		\
+	nativesdk-python-sqlite3		\
+	nativesdk-python-pickle			\
+	nativesdk-python-logging		\
+	nativesdk-python-elementtree		\
+	nativesdk-python-curses			\
+	nativesdk-python-compile		\
+	nativesdk-python-compiler		\
+	nativesdk-python-fcntl			\
+	nativesdk-python-shell			\
+	nativesdk-python-misc			\
+	nativesdk-python-multiprocessing	\
+	nativesdk-python-subprocess		\
+	nativesdk-python-xmlrpc			\
+	nativesdk-python-netclient		\
+	nativesdk-python-netserver		\
+	nativesdk-python-distutils		\
+	nativesdk-python-unixadmin		\
+	nativesdk-python-compression		\
+	nativesdk-python-json			\
+	nativesdk-python-unittest		\
+	nativesdk-python-mmap			\
+	nativesdk-python-difflib		\
+	nativesdk-python-pprint			\
+	nativesdk-python-git			\
+	nativesdk-python-pkgutil		\
+    "
-- 
2.0.0

[PATCH 2/2] meta-toolchain-qt: Install full set of python modules in Qt SDK toolchain

[Marek Vasut]

The Qt SDK toolchain pulls in python via packagegroup-cross-canadian-${MACHINE}
and ships it. But the python is missing many modules and is rather incomplete.

The environment-setup-* script configures the PATH variable to point into
it's own sysroot first, it means the python from the SDK is used as well
when the environment is configured. It actually does make sense to use the
python from the SDK, since the SDK should provide the tools needed to build
additional software.

The problem is, the python in the SDK image is stripped to a bare minimum
of modules, which prevents some software from being built with the SDK as
is. Add the same python modules as the buildtools-tarball.bb adds to make
the python shipped with the SDK complete via nativesdk-packagegroup-python.

Signed-off-by: Marek Vasut <marex@denx.de>
---
 meta/recipes-qt/meta/meta-toolchain-qt.inc | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-qt/meta/meta-toolchain-qt.inc b/meta/recipes-qt/meta/meta-toolchain-qt.inc
index 6b162bd..b8b9c78 100644
--- a/meta/recipes-qt/meta/meta-toolchain-qt.inc
+++ b/meta/recipes-qt/meta/meta-toolchain-qt.inc
@@ -1,4 +1,8 @@
-TOOLCHAIN_HOST_TASK = "nativesdk-packagegroup-${QTNAME}-toolchain-host packagegroup-cross-canadian-${MACHINE}"
+TOOLCHAIN_HOST_TASK = " \
+	nativesdk-packagegroup-${QTNAME}-toolchain-host \
+	packagegroup-cross-canadian-${MACHINE} \
+	nativesdk-packagegroup-python \
+	"
 TOOLCHAIN_TARGET_TASK = "packagegroup-${QTNAME}-toolchain-target"
 TOOLCHAIN_OUTPUTNAME = "${SDK_NAME}-toolchain-${QTNAME}-${DISTRO_VERSION}"
 
-- 
2.0.0

Re: [PATCH 1/2] nativesdk-packagegroup-python: buildtools-tarball: Introduce NativeSDK package group

[Richard Purdie]

On Thu, 2014-09-18 at 11:15 +0200, Marek Vasut wrote:
> Introduce and use nativesdk-packagegroup-python , which contains all
> necessary python components for the native SDK. Make use of this
> package group in buildtools-tarball.bb immediatelly.

Specifically, isn't this the list of packages needed to run bitbake and
its associated utils? "python" isn't therefore a particularly good
description of this package?

Cheers,

Richard

> Signed-off-by: Marek Vasut <marex@denx.de>
> ---
>  meta/recipes-core/meta/buildtools-tarball.bb       | 28 +----------------
>  .../python/nativesdk-packagegroup-python.bb        | 36 ++++++++++++++++++++++
>  2 files changed, 37 insertions(+), 27 deletions(-)
>  create mode 100644 meta/recipes-devtools/python/nativesdk-packagegroup-python.bb
> 
> diff --git a/meta/recipes-core/meta/buildtools-tarball.bb b/meta/recipes-core/meta/buildtools-tarball.bb
> index 62e1e0b..7d20238 100644
> --- a/meta/recipes-core/meta/buildtools-tarball.bb
> +++ b/meta/recipes-core/meta/buildtools-tarball.bb
> @@ -8,33 +8,7 @@ LIC_FILES_CHKSUM = "file://${COREBASE}/LICENSE;md5=4d92cd373abda3937c2bc47fbc49d
>  TOOLCHAIN_TARGET_TASK ?= ""
>  
>  TOOLCHAIN_HOST_TASK ?= "\
> -    nativesdk-python-core \
> -    nativesdk-python-textutils \
> -    nativesdk-python-sqlite3 \
> -    nativesdk-python-pickle \
> -    nativesdk-python-logging \
> -    nativesdk-python-elementtree \
> -    nativesdk-python-curses \
> -    nativesdk-python-compile \
> -    nativesdk-python-compiler \
> -    nativesdk-python-fcntl \
> -    nativesdk-python-shell \
> -    nativesdk-python-misc \
> -    nativesdk-python-multiprocessing \
> -    nativesdk-python-subprocess \
> -    nativesdk-python-xmlrpc \
> -    nativesdk-python-netclient \
> -    nativesdk-python-netserver \
> -    nativesdk-python-distutils \
> -    nativesdk-python-unixadmin \
> -    nativesdk-python-compression \
> -    nativesdk-python-json \
> -    nativesdk-python-unittest \
> -    nativesdk-python-mmap \
> -    nativesdk-python-difflib \
> -    nativesdk-python-pprint \
> -    nativesdk-python-git \
> -    nativesdk-python-pkgutil \
> +    nativesdk-packagegroup-python \
>      nativesdk-ncurses-terminfo-base \
>      nativesdk-chrpath \
>      nativesdk-tar \
> diff --git a/meta/recipes-devtools/python/nativesdk-packagegroup-python.bb b/meta/recipes-devtools/python/nativesdk-packagegroup-python.bb
> new file mode 100644
> index 0000000..fdc1006
> --- /dev/null
> +++ b/meta/recipes-devtools/python/nativesdk-packagegroup-python.bb
> @@ -0,0 +1,36 @@
> +SUMMARY = "NativeSDK Python package group"
> +DESCRIPTION = "NativeSDK Python package group"
> +PR = "r0"
> +LICENSE = "MIT"
> +
> +inherit packagegroup
> +
> +RDEPENDS_${PN} = "				\
> +	nativesdk-python-core			\
> +	nativesdk-python-textutils		\
> +	nativesdk-python-sqlite3		\
> +	nativesdk-python-pickle			\
> +	nativesdk-python-logging		\
> +	nativesdk-python-elementtree		\
> +	nativesdk-python-curses			\
> +	nativesdk-python-compile		\
> +	nativesdk-python-compiler		\
> +	nativesdk-python-fcntl			\
> +	nativesdk-python-shell			\
> +	nativesdk-python-misc			\
> +	nativesdk-python-multiprocessing	\
> +	nativesdk-python-subprocess		\
> +	nativesdk-python-xmlrpc			\
> +	nativesdk-python-netclient		\
> +	nativesdk-python-netserver		\
> +	nativesdk-python-distutils		\
> +	nativesdk-python-unixadmin		\
> +	nativesdk-python-compression		\
> +	nativesdk-python-json			\
> +	nativesdk-python-unittest		\
> +	nativesdk-python-mmap			\
> +	nativesdk-python-difflib		\
> +	nativesdk-python-pprint			\
> +	nativesdk-python-git			\
> +	nativesdk-python-pkgutil		\
> +    "

Re: [PATCH 1/2] nativesdk-packagegroup-python: buildtools-tarball: Introduce NativeSDK package group

[Marek Vasut]

On Thursday, September 18, 2014 at 03:30:11 PM, Richard Purdie wrote:
> On Thu, 2014-09-18 at 11:15 +0200, Marek Vasut wrote:
> > Introduce and use nativesdk-packagegroup-python , which contains all
> > necessary python components for the native SDK. Make use of this
> > package group in buildtools-tarball.bb immediatelly.
> 
> Specifically, isn't this the list of packages needed to run bitbake and
> its associated utils? "python" isn't therefore a particularly good
> description of this package?

This is really just and only python packages, nothing else.

Best regards,
Marek Vasut

Re: [PATCH 1/2] nativesdk-packagegroup-python: buildtools-tarball: Introduce NativeSDK package group

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1008 bytes --]

On Fri, Sep 19, 2014 at 12:06 AM, Marek Vasut <marex@denx.de> wrote:

> On Thursday, September 18, 2014 at 03:30:11 PM, Richard Purdie wrote:
> > On Thu, 2014-09-18 at 11:15 +0200, Marek Vasut wrote:
> > > Introduce and use nativesdk-packagegroup-python , which contains all
> > > necessary python components for the native SDK. Make use of this
> > > package group in buildtools-tarball.bb immediatelly.
> >
> > Specifically, isn't this the list of packages needed to run bitbake and
> > its associated utils? "python" isn't therefore a particularly good
> > description of this package?
>
> This is really just and only python packages, nothing else.


The fact that it only contains python packages doesn't mean it contains all
python packages, or is somehow generic. It's a specific subset of python
modules used by bitbake.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1483 bytes --]

Re: [PATCH 1/2] nativesdk-packagegroup-python: buildtools-tarball: Introduce NativeSDK package group

[Marek Vasut]

On Friday, September 19, 2014 at 06:13:46 PM, Christopher Larson wrote:
> On Fri, Sep 19, 2014 at 12:06 AM, Marek Vasut <marex@denx.de> wrote:
> > On Thursday, September 18, 2014 at 03:30:11 PM, Richard Purdie wrote:
> > > On Thu, 2014-09-18 at 11:15 +0200, Marek Vasut wrote:
> > > > Introduce and use nativesdk-packagegroup-python , which contains all
> > > > necessary python components for the native SDK. Make use of this
> > > > package group in buildtools-tarball.bb immediatelly.
> > > 
> > > Specifically, isn't this the list of packages needed to run bitbake and
> > > its associated utils? "python" isn't therefore a particularly good
> > > description of this package?
> > 
> > This is really just and only python packages, nothing else.
> 
> The fact that it only contains python packages doesn't mean it contains all
> python packages, or is somehow generic. It's a specific subset of python
> modules used by bitbake.

OK, so uh, what would be the right solution here to make the toolchain usable? 
As I am no expert, I really don't know what to do now. Can you help please ?

Best regards,
Marek Vasut

Re: [PATCH 1/2] nativesdk-packagegroup-python: buildtools-tarball: Introduce NativeSDK package group

[Paul Eggleton]

Hi Marek,

On Friday 19 September 2014 18:27:30 Marek Vasut wrote:
> On Friday, September 19, 2014 at 06:13:46 PM, Christopher Larson wrote:
> > On Fri, Sep 19, 2014 at 12:06 AM, Marek Vasut <marex@denx.de> wrote:
> > > On Thursday, September 18, 2014 at 03:30:11 PM, Richard Purdie wrote:
> > > > On Thu, 2014-09-18 at 11:15 +0200, Marek Vasut wrote:
> > > > > Introduce and use nativesdk-packagegroup-python , which contains all
> > > > > necessary python components for the native SDK. Make use of this
> > > > > package group in buildtools-tarball.bb immediatelly.
> > > > 
> > > > Specifically, isn't this the list of packages needed to run bitbake
> > > > and
> > > > its associated utils? "python" isn't therefore a particularly good
> > > > description of this package?
> > > 
> > > This is really just and only python packages, nothing else.
> > 
> > The fact that it only contains python packages doesn't mean it contains
> > all
> > python packages, or is somehow generic. It's a specific subset of python
> > modules used by bitbake.
> 
> OK, so uh, what would be the right solution here to make the toolchain
> usable? As I am no expert, I really don't know what to do now. Can you help
> please ?

There's clearly a bug here, adding a Python-using component does not mean you 
should have python in the SDK but that's what currently happens. There is a 
bug open now covering this, we will do our best to address it in a timely 
manner. (FWIW, I would have thought this issue has been present for some time 
rather than being a recent regression.)

In the mean time however you can work around it by adding 
nativesdk-python-modules (an existing meta-package) to your SDK via 
TOOLCHAIN_HOST_TASK.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 1/2] nativesdk-packagegroup-python: buildtools-tarball: Introduce NativeSDK package group

[Marek Vasut]

On Friday, September 19, 2014 at 06:42:14 PM, Paul Eggleton wrote:
> Hi Marek,

Hi Paul,

> On Friday 19 September 2014 18:27:30 Marek Vasut wrote:
> > On Friday, September 19, 2014 at 06:13:46 PM, Christopher Larson wrote:
> > > On Fri, Sep 19, 2014 at 12:06 AM, Marek Vasut <marex@denx.de> wrote:
> > > > On Thursday, September 18, 2014 at 03:30:11 PM, Richard Purdie wrote:
> > > > > On Thu, 2014-09-18 at 11:15 +0200, Marek Vasut wrote:
> > > > > > Introduce and use nativesdk-packagegroup-python , which contains
> > > > > > all necessary python components for the native SDK. Make use of
> > > > > > this package group in buildtools-tarball.bb immediatelly.
> > > > > 
> > > > > Specifically, isn't this the list of packages needed to run bitbake
> > > > > and
> > > > > its associated utils? "python" isn't therefore a particularly good
> > > > > description of this package?
> > > > 
> > > > This is really just and only python packages, nothing else.
> > > 
> > > The fact that it only contains python packages doesn't mean it contains
> > > all
> > > python packages, or is somehow generic. It's a specific subset of
> > > python modules used by bitbake.
> > 
> > OK, so uh, what would be the right solution here to make the toolchain
> > usable? As I am no expert, I really don't know what to do now. Can you
> > help please ?
> 
> There's clearly a bug here, adding a Python-using component does not mean
> you should have python in the SDK but that's what currently happens. There
> is a bug open now covering this, we will do our best to address it in a
> timely manner.

Thank you, I really appreciate it !

> (FWIW, I would have thought this issue has been present for
> some time rather than being a recent regression.)

Yes, it was. It only hit me recently when U-Boot switched to Kconfig, which 
needs a python module, which is missing from the SDK toolchain though.

> In the mean time however you can work around it by adding
> nativesdk-python-modules (an existing meta-package) to your SDK via
> TOOLCHAIN_HOST_TASK.

Ah, thank you for the hint!

Best regards,
Marek Vasut

Re: [PATCH 1/2] nativesdk-packagegroup-python: buildtools-tarball: Introduce NativeSDK package group

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 966 bytes --]

On Fri, Sep 19, 2014 at 9:42 AM, Paul Eggleton <
paul.eggleton@linux.intel.com> wrote:

> There's clearly a bug here, adding a Python-using component does not mean
> you
> should have python in the SDK but that's what currently happens. There is a
> bug open now covering this, we will do our best to address it in a timely
> manner. (FWIW, I would have thought this issue has been present for some
> time
> rather than being a recent regression.)
>

Ah, we've hit something similar. We had a need to add the git perl tools to
the buildtools-tarball to support the gitsm:// fetcher (git-submodule is
perl, iirc), which sucked in nativesdk-perl. At the moment we're doing a
horrible hack and just rm'ing perl itself after it gets installed, do you
have the yocto bug # for the python one?
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1341 bytes --]

Re: [PATCH 1/2] nativesdk-packagegroup-python: buildtools-tarball: Introduce NativeSDK package group

[Paul Eggleton]

On Friday 19 September 2014 10:02:29 Christopher Larson wrote:
> On Fri, Sep 19, 2014 at 9:42 AM, Paul Eggleton <
> 
> paul.eggleton@linux.intel.com> wrote:
> > There's clearly a bug here, adding a Python-using component does not mean
> > you
> > should have python in the SDK but that's what currently happens. There is
> > a
> > bug open now covering this, we will do our best to address it in a timely
> > manner. (FWIW, I would have thought this issue has been present for some
> > time
> > rather than being a recent regression.)
> 
> Ah, we've hit something similar. We had a need to add the git perl tools to
> the buildtools-tarball to support the gitsm:// fetcher (git-submodule is
> perl, iirc), which sucked in nativesdk-perl. At the moment we're doing a
> horrible hack and just rm'ing perl itself after it gets installed, do you
> have the yocto bug # for the python one?

https://bugzilla.yoctoproject.org/show_bug.cgi?id=6735

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre