populate_sdk_ext: Unable to locate package nativesdk-buildtools-perl-dummy

populate_sdk_ext: Unable to locate package nativesdk-buildtools-perl-dummy

[Michel D'HOOGE]

[-- Attachment #1: Type: text/plain, Size: 1621 bytes --]

Hi, 


I try to produce an extensible SDK for core-image-sato with the following configuration: 


BB_VERSION = "1.32.0" 
BUILD_SYS = "x86_64-linux" 
NATIVELSBSTRING = "universal" 
TARGET_SYS = "x86_64-poky-linux" 
MACHINE = "vtc7200" 
DISTRO = "poky-systemd" 
DISTRO_VERSION = "2.2" 
TUNE_FEATURES = "m64 corei7" 


The machine is based on an intel-corei7, with some features removed. And the distro is poky-based, with only systemd (as detailed in the ref manual). And I get the following error: 


ERROR : buildtools-tarball-1.0-r0 do_populate_sdk: Unable to install packages. Command '/mnt/Yocto/vedecom-x86/build/tmp/sysroots/x86_ 
64-linux/usr/bin/apt-get install --force-yes --allow-unauthenticated nativesdk-locale-base-en-us nativesdk-git nativesdk-make native 
sdk-ca-certificates nativesdk-tar nativesdk-python3-modules nativesdk-texinfo nativesdk-python3-git nativesdk-ncurses-terminfo-base n 
ativesdk-chrpath nativesdk-python3-misc nativesdk-git-perltools nativesdk-buildtools-perl-dummy nativesdk-pigz nativesdk-wget natives 
dk-python3-core' returned 100: 
Reading package lists... 
Building dependency tree... 
Reading state information... 
E: Unable to locate package nativesdk-buildtools-perl-dummy 




The package exists and is in: 
tmp/deploy/deb/buildtools-dummy-nativesdk/nativesdk-buildtools-perl-dummy_1.0-r2_allarch.deb 
while the other listed packages are in tmp/deploy/deb/x86_64-nativesdk. 


It seems this package is empty, so maybe I should simply remove it from poky/meta/recipes-core/meta/buildtools-tarball.bb. 


Thanks for your help 
Michel 

[-- Attachment #2: Type: text/html, Size: 7447 bytes --]

Re: populate_sdk_ext: Unable to locate package nativesdk-buildtools-perl-dummy

[Paul Eggleton]

Hi Michel,

On Wed, 16 Nov 2016 11:48:21 Michel D'HOOGE wrote:
> I try to produce an extensible SDK for core-image-sato with the following
> configuration:
> 
> BB_VERSION = "1.32.0"
> BUILD_SYS = "x86_64-linux"
> NATIVELSBSTRING = "universal"
> TARGET_SYS = "x86_64-poky-linux"
> MACHINE = "vtc7200"
> DISTRO = "poky-systemd"
> DISTRO_VERSION = "2.2"
> TUNE_FEATURES = "m64 corei7"
> 
> The machine is based on an intel-corei7, with some features removed. And the
> distro is poky-based, with only systemd (as detailed in the ref manual).
> And I get the following error:
> 
> 
> ERROR : buildtools-tarball-1.0-r0 do_populate_sdk: Unable to install
> packages. Command '/mnt/Yocto/vedecom-x86/build/tmp/sysroots/x86_
> 64-linux/usr/bin/apt-get install --force-yes --allow-unauthenticated
> nativesdk-locale-base-en-us nativesdk-git nativesdk-make native
> sdk-ca-certificates nativesdk-tar nativesdk-python3-modules
> nativesdk-texinfo nativesdk-python3-git nativesdk-ncurses-terminfo-base n
> ativesdk-chrpath nativesdk-python3-misc nativesdk-git-perltools
> nativesdk-buildtools-perl-dummy nativesdk-pigz nativesdk-wget natives
> dk-python3-core' returned 100:
> Reading package lists...
> Building dependency tree...
> Reading state information...
> E: Unable to locate package nativesdk-buildtools-perl-dummy
> 
> The package exists and is in:
> tmp/deploy/deb/buildtools-dummy-nativesdk/nativesdk-buildtools-perl-dummy_1.
> 0-r2_allarch.deb while the other listed packages are in
> tmp/deploy/deb/x86_64-nativesdk.
> 
> It seems this package is empty, so maybe I should simply remove it from
> poky/meta/recipes-core/meta/buildtools-tarball.bb.

FYI the purpose of this package is to take precedence over the real nativesdk-
perl so that it doesn't get into the buildtools, since we prefer to avoid the 
issues associated with overriding the host perl there. If you remove this 
package then perl will enter buildtools and you will experience other issues.

I think the issue is this hasn't really been tested with deb packaging. You 
have hit a genuine bug though - would you mind filing a bug at 
http://bugzilla.yoctoproject.org?

Thanks,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: populate_sdk_ext: Unable to locate package nativesdk-buildtools-perl-dummy

[Michel D'HOOGE]

Paul,

> From: "Paul Eggleton" <paul.eggleton@linux.intel.com>
> Sent: Friday, 18 November, 2016 2:46:07 AM

> I think the issue is this hasn't really been tested with deb
> packaging. You
> have hit a genuine bug though - would you mind filing a bug at
> http://bugzilla.yoctoproject.org?


Thanks for your help. I'll file a bug report as soon as I better understand the whole problem :-D

Michel

Re: populate_sdk_ext: Unable to locate package nativesdk-buildtools-perl-dummy

[Michel D'HOOGE]

> From: "Paul Eggleton" <paul.eggleton@linux.intel.com>
> Sent: Friday, 18 November, 2016 2:46:07 AM

> I think the issue is this hasn't really been tested with deb
> packaging. You
> have hit a genuine bug though - would you mind filing a bug at
> http://bugzilla.yoctoproject.org?


I still don't fully understand the problem, so I don't know where to file the bug!
But I think apt-native and buildtools-tarball are OK.

Right now I believe this is related to the ARCH value of the dummy debian packages.
It is "allarch" (in the filename and the Packages corresponding summary file)
whereas similar packages have the "all" arch.
I'm trying to understand why there is such a difference...

If someone has some knowledge on the subject to help me, it'd be great :-)

Cheers
Michel

Re: populate_sdk_ext: Unable to locate package nativesdk-buildtools-perl-dummy

[Paul Eggleton]

On Mon, 21 Nov 2016 18:22:10 Michel D'HOOGE wrote:
> > From: "Paul Eggleton" <paul.eggleton@linux.intel.com>
> > Sent: Friday, 18 November, 2016 2:46:07 AM
> > 
> > I think the issue is this hasn't really been tested with deb
> > packaging. You
> > have hit a genuine bug though - would you mind filing a bug at
> > http://bugzilla.yoctoproject.org?
> 
> I still don't fully understand the problem, so I don't know where to file
> the bug! But I think apt-native and buildtools-tarball are OK.
> 
> Right now I believe this is related to the ARCH value of the dummy debian
> packages. It is "allarch" (in the filename and the Packages corresponding
> summary file) whereas similar packages have the "all" arch.
> I'm trying to understand why there is such a difference...

I suspect it has to do with the arch mapping that goes on in 
meta/classes/package_deb.bbclass; it probably doesn't understand the dummy 
architecture we're giving it. The thing is even if it were to be "corrected" 
to fall back to "all" instead of "allarch" that probably still wouldn't work 
properly because the architecture of the perl packages would be a higher 
priority than "all" and thus the dummy package wouldn't take precedence.

In any case feel free to file a bug under "OE-Core -> Core" in bugzilla, that 
would be a reasonable category for this issue.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: populate_sdk_ext: Unable to locate package nativesdk-buildtools-perl-dummy

[Michel D'HOOGE]

> From: "Paul Eggleton" <paul.eggleton@linux.intel.com>
> Sent: Monday, 21 November, 2016 10:15:16 PM

> I suspect it has to do with the arch mapping that goes on in
> meta/classes/package_deb.bbclass; it probably doesn't understand the
> dummy
> architecture we're giving it. The thing is even if it were to be
> "corrected"
> to fall back to "all" instead of "allarch" that probably still
> wouldn't work
> properly because the architecture of the perl packages would be a
> higher
> priority than "all" and thus the dummy package wouldn't take
> precedence.

It seems to be OK.
The only other package with 'perl' in its name is 'nativesdk-git-perltools'.

But I have (another) problem with the manifest files:
- all "target" manifest files are empty
- the 'poky-systemd-blah-blah-toolchain-ext-2.2.host.manifest' has a single entry:
    meta-environment-extsdk-vtc7200
So maybe I don't see everything...


> In any case feel free to file a bug under "OE-Core -> Core" in
> bugzilla, that
> would be a reasonable category for this issue.

Done: https://bugzilla.yoctoproject.org/show_bug.cgi?id=10700

Michel

Re: populate_sdk_ext: Unable to locate package nativesdk-buildtools-perl-dummy

[Paul Eggleton]

On Tue, 22 Nov 2016 10:44:48 Michel D'HOOGE wrote:
> > From: "Paul Eggleton" <paul.eggleton@linux.intel.com>
> > Sent: Monday, 21 November, 2016 10:15:16 PM
> > 
> > I suspect it has to do with the arch mapping that goes on in
> > meta/classes/package_deb.bbclass; it probably doesn't understand the
> > dummy architecture we're giving it. The thing is even if it were to be
> > "corrected" to fall back to "all" instead of "allarch" that probably still
> > wouldn't work properly because the architecture of the perl packages would
> > be a higher priority than "all" and thus the dummy package wouldn't take
> > precedence.
> 
> It seems to be OK.
> The only other package with 'perl' in its name is 'nativesdk-git-perltools'.

OK, so it may only be working by chance then - if you did have the nativesdk-
perl package built then I suspect you'd find it would get installed. I'm 
puzzled as to why it hasn't been built though, since it is a dependency of 
some of the other items we do include in buildtools (notably git) and does get 
built when you use rpm as the backend, last I checked anyway.
 
> But I have (another) problem with the manifest files:
> - all "target" manifest files are empty
> - the 'poky-systemd-blah-blah-toolchain-ext-2.2.host.manifest' has a single
> entry: meta-environment-extsdk-vtc7200
> So maybe I don't see everything...

That's expected for the extensible SDK - it isn't built with the same kinds of 
packages that the standard SDK is built with.

> > In any case feel free to file a bug under "OE-Core -> Core" in
> > bugzilla, that> would be a reasonable category for this issue.
> 
> Done: https://bugzilla.yoctoproject.org/show_bug.cgi?id=10700

Thanks.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: populate_sdk_ext: Unable to locate package nativesdk-buildtools-perl-dummy

[Michel D'HOOGE]

Hi Paul,

I'm still investigating the whole thing because nothing works so far!
The SDK I managed to produce didn't install, and now I can't even
produce an SDK anymore :'-(

I chose Debian PM because this is what I already know and thought
it would be easier than to learn about the RPM world.
Not sure it was my best move ;-)


During the install, the error was:
ERROR: e2fsprogs-native-1.43-r1 do_populate_sysroot_setscene:
 Error executing a python function in exec_python_func() autogenerated:
[...]
File: '/mnt/Yocto/sdk/greenfeed/layers/poky/meta/lib/oe/qa.py', lineno: 102, function: getWord
     0101:    def getWord(self, offset):
 *** 0102:        return struct.unpack_from(self.sex+"i", self.data, offset)[0]
Exception: struct.error: unpack_from requires a buffer of at least 4 bytes


And now, the build fails with:
Exception: FileNotFoundError: [Errno 2] No such file or directory:
'/mnt/Yocto/build/tmp/sysroots/vtc7110/locked-sigs/locked-sigs-extsdk-toolchain.inc'
ERROR: image-greenfeed-0.1-r0 do_populate_sdk_ext: Function failed: copy_buildsystem


This is just FYI, I'm still working on it...
I'll try the devpyshell for the 1st time!


> OK, so it may only be working by chance then - if you did have the
> nativesdk-perl package built then I suspect you'd find it would get installed.
> I'm puzzled as to why it hasn't been built though, since it is a
> dependency of
> some of the other items we do include in buildtools (notably git) and
> does get built when you use rpm as the backend, last I checked anyway.

Well, a lot of nativesdk-perl packages exist, but I didn't see them 
in the manifest.


Be back soon
Michel

Re: populate_sdk_ext: Unable to locate package nativesdk-buildtools-perl-dummy

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 1092 bytes --]

On 24 November 2016 at 07:50, Michel D'HOOGE <michel.dhooge@free.fr> wrote:

> During the install, the error was:
> ERROR: e2fsprogs-native-1.43-r1 do_populate_sysroot_setscene:
>  Error executing a python function in exec_python_func() autogenerated:
> [...]
> File: '/mnt/Yocto/sdk/greenfeed/layers/poky/meta/lib/oe/qa.py', lineno:
> 102, function: getWord
>      0101:    def getWord(self, offset):
>  *** 0102:        return struct.unpack_from(self.sex+"i", self.data,
> offset)[0]
> Exception: struct.error: unpack_from requires a buffer of at least 4 bytes


I fixed this in oe-core master a few weeks ago.  oe-core
a66660aa5bb709547ce0b65a4563e4217c3c3d9f.

I'll submit a backport to morty request now.

I chose Debian PM because this is what I already know and thought
> it would be easier than to learn about the RPM world.
> Not sure it was my best move ;-)
>

If you can in the meantime I do recommend switching to opkg or rpm.
Changing packaging format will just re-package everything assuming you
still have the tmp/ so won't take long at all.

Ross

[-- Attachment #2: Type: text/html, Size: 2607 bytes --]

populate_sdk_ext: Huge(?) differences between RPM & DEB

[Michel D'HOOGE]

[-- Attachment #1: Type: text/plain, Size: 892 bytes --]

Ross, 

> From: "Ross Burton" <ross.burton@intel.com>
> Sent: Friday, 25 November, 2016 6:34:01 PM

> I fixed this in oe-core master a few weeks ago. oe-core
> a66660aa5bb709547ce0b65a4563e4217c3c3d9f.
Thank you for your answer. I applied your commit. And after struggling a little with some side-effects that led to erasing my build/tmp, everything is now fine, both with rpm & deb. 

Well… When I say "fine", I mean no errors during the baking. But the results are a bit different: 

RPM-based 
Installer is 1.7 GiB. 
SDK is 10.3 GiB, 168361 files & 35049 folders. 

Debian-based: 
Installer is 1.3GiB. 
8.9GiB, 172366 files & 34795 folders. 

What is the same in both runs is the emptiness of the manifest files! 
Both target.manifests are empty. And the host manifests only have a single line (meta-environment-extsdk) 

Does that make sense? 

Thanks 
Michel

[-- Attachment #2: Type: text/html, Size: 1969 bytes --]