Question about shipping files to package.

Question about shipping files to package.

[Wayne Li]

[-- Attachment #1: Type: text/plain, Size: 1482 bytes --]

Dear Yocto Developers,

So I created a bitbake recipe to integrate kvm into my image as an
out-of-tree kernel module.  Here's my recipe right now:

LICENSE = "GPLv2"
LIC_FILES_CHKSUM = "file://COPYING;md5=c616d0e7924e9e78ee192d99a3b26fbd"

inherit module

SRC_URI =
"file:///homead/QorIQ-SDK-V2.0-20160527-yocto/sources/meta-virtualization/recipes-kernel/kvm-kmodule/kvm-kmod-3.10.21.tar.bz2"

S = "${WORKDIR}/kvm-kmod-3.10.21"

do_configure() {
    ./configure --arch=ppc64
--kerneldir=/homead/QorIQ-SDK-V2.0-20160527-yocto/build_t4240rdb-64b/tmp/work/t4240rdb_64b-fsl-linux/kernel-devsrc/1.0-r0/image/usr/src/kernel
}

FILES_${PN} += "/lib/modules/4.1.8-rt8+gbd51baf"

Bitbaking this recipe completes with no problems but no kernel module is
created (compiling the source code should create a kernel module file
kvm.ko).  I was wondering if the problem might be because of what I set
FILES_${PN} to be.  Before I set the FILES variable I was getting an error
saying something along the lines of, "Files/directories were installed but
not shipped."  Then I more or less just guessed a directory and set my
FILES variable to it and then the recipe finished bitbaking with no errors.

But now that the kvm.ko file isn't even being created, I am wondering if it
might be because I set the FILES variable wrong?  What does the word "ship"
mean?  And along those lines what exactly is a "package" in the setting of
Yocto project?

-Thanks!, Wayne Li

[-- Attachment #2: Type: text/html, Size: 1788 bytes --]

Re: [yocto] Question about shipping files to package.

[Sean McKay]

[-- Attachment #1: Type: text/plain, Size: 4009 bytes --]

I’m not an expert on compiling kernel modules, but I can at least answer some of your questions to get started. I’m also not an actual expert, but just another user, so it’s entirely possible that someone more knowledgeable will come along and tell us that I’m wrong.

With that said, first, I have to ask the simple stuff: when you say that no .ko is being created, where are you looking? If you’re not sure where you should be looking, I’d recommend by doing a find inside of your ${WORKDIR} to see whether or not you’ve got any .ko files. They won’t quite be out in plain sight.

In general (though again, I can’t speak for the module class), compilations take place in a build directory inside of the recipe’s workdir (${B} in bitbake parlance). The install process then copies the created files to a staging directory (${D} in bitbake parlance) that’s usually ${WORKDIR}/image.

Packaging is the process of taking those output files from ${D} and packaging them into one of a few well defined packaging formats. If you’re using the defaults, this would be RPM. A single recipe can (and often does) produce multiple packages. When bitbake moves onto the packaging step (I’m simplifying a bit), it creates directories for each package that’s going to be generated inside of ${WORKDIR}/packages-split. It then looks at the FILES variable for the particular package it’s working on and copies any files that match the patterns in that variable into that package’s directory in packages-split. When it’s done copying files, each of those directories are turned into the appropriate package (probably .rpm) file.
So, for a somewhat more concrete example, when you said ‘FILES_${PN} += "/lib/modules/4.1.8-rt8+gbd51baf"’, you told bitbake that anything that matched /lib/modules/4.1.8-rt8+gbd51baf should get put in the ${PN} (main) package.

In the context of the error message you were previously receiving, “installed” is the do_install step, which is referring to that staging process (analogous to running ‘make install’ on something you just downloaded from the internet). Something is ‘shipped’ if it is packaged into one of the final rpm files.

Cheers!
-Sean McKay

From: yocto@lists.yoctoproject.org <yocto@lists.yoctoproject.org> On Behalf Of Wayne Li
Sent: Thursday, December 5, 2019 2:26 PM
To: bitbake-devel <bitbake-devel@lists.openembedded.org>; Yocto Project Discussion <yocto@yoctoproject.org>
Subject: [yocto] Question about shipping files to package.

Dear Yocto Developers,

So I created a bitbake recipe to integrate kvm into my image as an out-of-tree kernel module.  Here's my recipe right now:

LICENSE = "GPLv2"
LIC_FILES_CHKSUM = "file://COPYING;md5=c616d0e7924e9e78ee192d99a3b26fbd"

inherit module

SRC_URI = "file:///homead/QorIQ-SDK-V2.0-20160527-yocto/sources/meta-virtualization/recipes-kernel/kvm-kmodule/kvm-kmod-3.10.21.tar.bz2"

S = "${WORKDIR}/kvm-kmod-3.10.21"

do_configure() {
    ./configure --arch=ppc64 --kerneldir=/homead/QorIQ-SDK-V2.0-20160527-yocto/build_t4240rdb-64b/tmp/work/t4240rdb_64b-fsl-linux/kernel-devsrc/1.0-r0/image/usr/src/kernel
}

FILES_${PN} += "/lib/modules/4.1.8-rt8+gbd51baf"

Bitbaking this recipe completes with no problems but no kernel module is created (compiling the source code should create a kernel module file kvm.ko).  I was wondering if the problem might be because of what I set FILES_${PN} to be.  Before I set the FILES variable I was getting an error saying something along the lines of, "Files/directories were installed but not shipped."  Then I more or less just guessed a directory and set my FILES variable to it and then the recipe finished bitbaking with no errors.

But now that the kvm.ko file isn't even being created, I am wondering if it might be because I set the FILES variable wrong?  What does the word "ship" mean?  And along those lines what exactly is a "package" in the setting of Yocto project?

-Thanks!, Wayne Li

[-- Attachment #2: Type: text/html, Size: 7461 bytes --]

Re: [yocto] Question about shipping files to package.

[McKay, Sean]

[-- Attachment #1: Type: text/plain, Size: 4009 bytes --]

I’m not an expert on compiling kernel modules, but I can at least answer some of your questions to get started. I’m also not an actual expert, but just another user, so it’s entirely possible that someone more knowledgeable will come along and tell us that I’m wrong.

With that said, first, I have to ask the simple stuff: when you say that no .ko is being created, where are you looking? If you’re not sure where you should be looking, I’d recommend by doing a find inside of your ${WORKDIR} to see whether or not you’ve got any .ko files. They won’t quite be out in plain sight.

In general (though again, I can’t speak for the module class), compilations take place in a build directory inside of the recipe’s workdir (${B} in bitbake parlance). The install process then copies the created files to a staging directory (${D} in bitbake parlance) that’s usually ${WORKDIR}/image.

Packaging is the process of taking those output files from ${D} and packaging them into one of a few well defined packaging formats. If you’re using the defaults, this would be RPM. A single recipe can (and often does) produce multiple packages. When bitbake moves onto the packaging step (I’m simplifying a bit), it creates directories for each package that’s going to be generated inside of ${WORKDIR}/packages-split. It then looks at the FILES variable for the particular package it’s working on and copies any files that match the patterns in that variable into that package’s directory in packages-split. When it’s done copying files, each of those directories are turned into the appropriate package (probably .rpm) file.
So, for a somewhat more concrete example, when you said ‘FILES_${PN} += "/lib/modules/4.1.8-rt8+gbd51baf"’, you told bitbake that anything that matched /lib/modules/4.1.8-rt8+gbd51baf should get put in the ${PN} (main) package.

In the context of the error message you were previously receiving, “installed” is the do_install step, which is referring to that staging process (analogous to running ‘make install’ on something you just downloaded from the internet). Something is ‘shipped’ if it is packaged into one of the final rpm files.

Cheers!
-Sean McKay

From: yocto@lists.yoctoproject.org <yocto@lists.yoctoproject.org> On Behalf Of Wayne Li
Sent: Thursday, December 5, 2019 2:26 PM
To: bitbake-devel <bitbake-devel@lists.openembedded.org>; Yocto Project Discussion <yocto@yoctoproject.org>
Subject: [yocto] Question about shipping files to package.

Dear Yocto Developers,

So I created a bitbake recipe to integrate kvm into my image as an out-of-tree kernel module.  Here's my recipe right now:

LICENSE = "GPLv2"
LIC_FILES_CHKSUM = "file://COPYING;md5=c616d0e7924e9e78ee192d99a3b26fbd"

inherit module

SRC_URI = "file:///homead/QorIQ-SDK-V2.0-20160527-yocto/sources/meta-virtualization/recipes-kernel/kvm-kmodule/kvm-kmod-3.10.21.tar.bz2"

S = "${WORKDIR}/kvm-kmod-3.10.21"

do_configure() {
    ./configure --arch=ppc64 --kerneldir=/homead/QorIQ-SDK-V2.0-20160527-yocto/build_t4240rdb-64b/tmp/work/t4240rdb_64b-fsl-linux/kernel-devsrc/1.0-r0/image/usr/src/kernel
}

FILES_${PN} += "/lib/modules/4.1.8-rt8+gbd51baf"

Bitbaking this recipe completes with no problems but no kernel module is created (compiling the source code should create a kernel module file kvm.ko).  I was wondering if the problem might be because of what I set FILES_${PN} to be.  Before I set the FILES variable I was getting an error saying something along the lines of, "Files/directories were installed but not shipped."  Then I more or less just guessed a directory and set my FILES variable to it and then the recipe finished bitbaking with no errors.

But now that the kvm.ko file isn't even being created, I am wondering if it might be because I set the FILES variable wrong?  What does the word "ship" mean?  And along those lines what exactly is a "package" in the setting of Yocto project?

-Thanks!, Wayne Li

[-- Attachment #2: Type: text/html, Size: 7461 bytes --]

Re: [yocto] Question about shipping files to package.

[Wayne Li]

[-- Attachment #1: Type: text/plain, Size: 4881 bytes --]

I did a find for any files with a kvm and a .ko in its name throughout my
entire workspace and nothing came up.  But anyway, thanks for the
explanation.  Could packaging the files wrong (i.e. setting the FILES
variable wrong) cause the kvm.ko file to never show up at all?  I guess
that couldn't be the case though because the .ko files are created first
before it gets packaged up so if they were created in the first place a
search through my entire workspace would turn them up.  Is that correct?

On Thu, Dec 5, 2019 at 11:14 PM McKay, Sean <sean.mckay@hpe.com> wrote:

> I’m not an expert on compiling kernel modules, but I can at least answer
> some of your questions to get started. I’m also not an actual expert, but
> just another user, so it’s entirely possible that someone more
> knowledgeable will come along and tell us that I’m wrong.
>
>
>
> With that said, first, I have to ask the simple stuff: when you say that
> no .ko is being created, where are you looking? If you’re not sure where
> you should be looking, I’d recommend by doing a find inside of your
> ${WORKDIR} to see whether or not you’ve got any .ko files. They won’t quite
> be out in plain sight.
>
>
>
> In general (though again, I can’t speak for the module class),
> compilations take place in a build directory inside of the recipe’s workdir
> (${B} in bitbake parlance). The install process then copies the created
> files to a staging directory (${D} in bitbake parlance) that’s usually
> ${WORKDIR}/image.
>
>
>
> Packaging is the process of taking those output files from ${D} and
> packaging them into one of a few well defined packaging formats. If you’re
> using the defaults, this would be RPM. A single recipe can (and often does)
> produce multiple packages. When bitbake moves onto the packaging step (I’m
> simplifying a bit), it creates directories for each package that’s going to
> be generated inside of ${WORKDIR}/packages-split. It then looks at the
> FILES variable for the particular package it’s working on and copies any
> files that match the patterns in that variable into that package’s
> directory in packages-split. When it’s done copying files, each of those
> directories are turned into the appropriate package (probably .rpm) file.
>
> So, for a somewhat more concrete example, when you said ‘FILES_${PN} +=
> "/lib/modules/4.1.8-rt8+gbd51baf"’, you told bitbake that anything that
> matched /lib/modules/4.1.8-rt8+gbd51baf should get put in the ${PN} (main)
> package.
>
>
>
> In the context of the error message you were previously receiving,
> “installed” is the do_install step, which is referring to that staging
> process (analogous to running ‘make install’ on something you just
> downloaded from the internet). Something is ‘shipped’ if it is packaged
> into one of the final rpm files.
>
>
>
> Cheers!
>
> -Sean McKay
>
>
>
> *From:* yocto@lists.yoctoproject.org <yocto@lists.yoctoproject.org> *On
> Behalf Of *Wayne Li
> *Sent:* Thursday, December 5, 2019 2:26 PM
> *To:* bitbake-devel <bitbake-devel@lists.openembedded.org>; Yocto Project
> Discussion <yocto@yoctoproject.org>
> *Subject:* [yocto] Question about shipping files to package.
>
>
>
> Dear Yocto Developers,
>
>
>
> So I created a bitbake recipe to integrate kvm into my image as an
> out-of-tree kernel module.  Here's my recipe right now:
>
>
>
> LICENSE = "GPLv2"
> LIC_FILES_CHKSUM = "file://COPYING;md5=c616d0e7924e9e78ee192d99a3b26fbd"
>
> inherit module
>
> SRC_URI = "
> file:///homead/QorIQ-SDK-V2.0-20160527-yocto/sources/meta-virtualization/recipes-kernel/kvm-kmodule/kvm-kmod-3.10.21.tar.bz2
> "
>
> S = "${WORKDIR}/kvm-kmod-3.10.21"
>
> do_configure() {
>     ./configure --arch=ppc64
> --kerneldir=/homead/QorIQ-SDK-V2.0-20160527-yocto/build_t4240rdb-64b/tmp/work/t4240rdb_64b-fsl-linux/kernel-devsrc/1.0-r0/image/usr/src/kernel
> }
>
> FILES_${PN} += "/lib/modules/4.1.8-rt8+gbd51baf"
>
>
>
> Bitbaking this recipe completes with no problems but no kernel module is
> created (compiling the source code should create a kernel module file
> kvm.ko).  I was wondering if the problem might be because of what I set
> FILES_${PN} to be.  Before I set the FILES variable I was getting an error
> saying something along the lines of, "Files/directories were installed but
> not shipped."  Then I more or less just guessed a directory and set my
> FILES variable to it and then the recipe finished bitbaking with no errors.
>
>
>
> But now that the kvm.ko file isn't even being created, I am wondering if
> it might be because I set the FILES variable wrong?  What does the word
> "ship" mean?  And along those lines what exactly is a "package" in the
> setting of Yocto project?
>
>
>
> -Thanks!, Wayne Li
>

[-- Attachment #2: Type: text/html, Size: 6807 bytes --]

Re: [yocto] Question about shipping files to package.

[Wayne Li]

[-- Attachment #1: Type: text/plain, Size: 4881 bytes --]

I did a find for any files with a kvm and a .ko in its name throughout my
entire workspace and nothing came up.  But anyway, thanks for the
explanation.  Could packaging the files wrong (i.e. setting the FILES
variable wrong) cause the kvm.ko file to never show up at all?  I guess
that couldn't be the case though because the .ko files are created first
before it gets packaged up so if they were created in the first place a
search through my entire workspace would turn them up.  Is that correct?

On Thu, Dec 5, 2019 at 11:14 PM McKay, Sean <sean.mckay@hpe.com> wrote:

> I’m not an expert on compiling kernel modules, but I can at least answer
> some of your questions to get started. I’m also not an actual expert, but
> just another user, so it’s entirely possible that someone more
> knowledgeable will come along and tell us that I’m wrong.
>
>
>
> With that said, first, I have to ask the simple stuff: when you say that
> no .ko is being created, where are you looking? If you’re not sure where
> you should be looking, I’d recommend by doing a find inside of your
> ${WORKDIR} to see whether or not you’ve got any .ko files. They won’t quite
> be out in plain sight.
>
>
>
> In general (though again, I can’t speak for the module class),
> compilations take place in a build directory inside of the recipe’s workdir
> (${B} in bitbake parlance). The install process then copies the created
> files to a staging directory (${D} in bitbake parlance) that’s usually
> ${WORKDIR}/image.
>
>
>
> Packaging is the process of taking those output files from ${D} and
> packaging them into one of a few well defined packaging formats. If you’re
> using the defaults, this would be RPM. A single recipe can (and often does)
> produce multiple packages. When bitbake moves onto the packaging step (I’m
> simplifying a bit), it creates directories for each package that’s going to
> be generated inside of ${WORKDIR}/packages-split. It then looks at the
> FILES variable for the particular package it’s working on and copies any
> files that match the patterns in that variable into that package’s
> directory in packages-split. When it’s done copying files, each of those
> directories are turned into the appropriate package (probably .rpm) file.
>
> So, for a somewhat more concrete example, when you said ‘FILES_${PN} +=
> "/lib/modules/4.1.8-rt8+gbd51baf"’, you told bitbake that anything that
> matched /lib/modules/4.1.8-rt8+gbd51baf should get put in the ${PN} (main)
> package.
>
>
>
> In the context of the error message you were previously receiving,
> “installed” is the do_install step, which is referring to that staging
> process (analogous to running ‘make install’ on something you just
> downloaded from the internet). Something is ‘shipped’ if it is packaged
> into one of the final rpm files.
>
>
>
> Cheers!
>
> -Sean McKay
>
>
>
> *From:* yocto@lists.yoctoproject.org <yocto@lists.yoctoproject.org> *On
> Behalf Of *Wayne Li
> *Sent:* Thursday, December 5, 2019 2:26 PM
> *To:* bitbake-devel <bitbake-devel@lists.openembedded.org>; Yocto Project
> Discussion <yocto@yoctoproject.org>
> *Subject:* [yocto] Question about shipping files to package.
>
>
>
> Dear Yocto Developers,
>
>
>
> So I created a bitbake recipe to integrate kvm into my image as an
> out-of-tree kernel module.  Here's my recipe right now:
>
>
>
> LICENSE = "GPLv2"
> LIC_FILES_CHKSUM = "file://COPYING;md5=c616d0e7924e9e78ee192d99a3b26fbd"
>
> inherit module
>
> SRC_URI = "
> file:///homead/QorIQ-SDK-V2.0-20160527-yocto/sources/meta-virtualization/recipes-kernel/kvm-kmodule/kvm-kmod-3.10.21.tar.bz2
> "
>
> S = "${WORKDIR}/kvm-kmod-3.10.21"
>
> do_configure() {
>     ./configure --arch=ppc64
> --kerneldir=/homead/QorIQ-SDK-V2.0-20160527-yocto/build_t4240rdb-64b/tmp/work/t4240rdb_64b-fsl-linux/kernel-devsrc/1.0-r0/image/usr/src/kernel
> }
>
> FILES_${PN} += "/lib/modules/4.1.8-rt8+gbd51baf"
>
>
>
> Bitbaking this recipe completes with no problems but no kernel module is
> created (compiling the source code should create a kernel module file
> kvm.ko).  I was wondering if the problem might be because of what I set
> FILES_${PN} to be.  Before I set the FILES variable I was getting an error
> saying something along the lines of, "Files/directories were installed but
> not shipped."  Then I more or less just guessed a directory and set my
> FILES variable to it and then the recipe finished bitbaking with no errors.
>
>
>
> But now that the kvm.ko file isn't even being created, I am wondering if
> it might be because I set the FILES variable wrong?  What does the word
> "ship" mean?  And along those lines what exactly is a "package" in the
> setting of Yocto project?
>
>
>
> -Thanks!, Wayne Li
>

[-- Attachment #2: Type: text/html, Size: 6807 bytes --]

Re: [yocto] Question about shipping files to package.

[Sean McKay]

[-- Attachment #1: Type: text/plain, Size: 5753 bytes --]

If the issue were just that your FILES_* variables were set incorrectly, you’d at least end up with the .ko file somewhere in your ${WORKDIR}/image (${D}) directory. You’d almost certainly find another one in the build or source directory as well (depending on how your .bb/bbclass files define the location of the compilation).

The fact that you don’t have one anywhere seems to indicate that your do_compile step isn’t actually generating the .ko that you expect it to, so you’ll need to rewind your recipe creation to working on that step and debug there.

Assuming you’re using poky as a base, what version are you on so I can go read the .bbclass you’re using?

-Sean

From: Wayne Li <waynli329@gmail.com>
Sent: Tuesday, December 10, 2019 12:23 PM
To: McKay, Sean <sean.mckay@hpe.com>
Cc: bitbake-devel <bitbake-devel@lists.openembedded.org>; Yocto Project Discussion <yocto@yoctoproject.org>
Subject: Re: [yocto] Question about shipping files to package.

I did a find for any files with a kvm and a .ko in its name throughout my entire workspace and nothing came up.  But anyway, thanks for the explanation.  Could packaging the files wrong (i.e. setting the FILES variable wrong) cause the kvm.ko file to never show up at all?  I guess that couldn't be the case though because the .ko files are created first before it gets packaged up so if they were created in the first place a search through my entire workspace would turn them up.  Is that correct?

On Thu, Dec 5, 2019 at 11:14 PM McKay, Sean <sean.mckay@hpe.com<mailto:sean.mckay@hpe.com>> wrote:
I’m not an expert on compiling kernel modules, but I can at least answer some of your questions to get started. I’m also not an actual expert, but just another user, so it’s entirely possible that someone more knowledgeable will come along and tell us that I’m wrong.

With that said, first, I have to ask the simple stuff: when you say that no .ko is being created, where are you looking? If you’re not sure where you should be looking, I’d recommend by doing a find inside of your ${WORKDIR} to see whether or not you’ve got any .ko files. They won’t quite be out in plain sight.

In general (though again, I can’t speak for the module class), compilations take place in a build directory inside of the recipe’s workdir (${B} in bitbake parlance). The install process then copies the created files to a staging directory (${D} in bitbake parlance) that’s usually ${WORKDIR}/image.

Packaging is the process of taking those output files from ${D} and packaging them into one of a few well defined packaging formats. If you’re using the defaults, this would be RPM. A single recipe can (and often does) produce multiple packages. When bitbake moves onto the packaging step (I’m simplifying a bit), it creates directories for each package that’s going to be generated inside of ${WORKDIR}/packages-split. It then looks at the FILES variable for the particular package it’s working on and copies any files that match the patterns in that variable into that package’s directory in packages-split. When it’s done copying files, each of those directories are turned into the appropriate package (probably .rpm) file.
So, for a somewhat more concrete example, when you said ‘FILES_${PN} += "/lib/modules/4.1.8-rt8+gbd51baf"’, you told bitbake that anything that matched /lib/modules/4.1.8-rt8+gbd51baf should get put in the ${PN} (main) package.

In the context of the error message you were previously receiving, “installed” is the do_install step, which is referring to that staging process (analogous to running ‘make install’ on something you just downloaded from the internet). Something is ‘shipped’ if it is packaged into one of the final rpm files.

Cheers!
-Sean McKay

From: yocto@lists.yoctoproject.org<mailto:yocto@lists.yoctoproject.org> <yocto@lists.yoctoproject.org<mailto:yocto@lists.yoctoproject.org>> On Behalf Of Wayne Li
Sent: Thursday, December 5, 2019 2:26 PM
To: bitbake-devel <bitbake-devel@lists.openembedded.org<mailto:bitbake-devel@lists.openembedded.org>>; Yocto Project Discussion <yocto@yoctoproject.org<mailto:yocto@yoctoproject.org>>
Subject: [yocto] Question about shipping files to package.

Dear Yocto Developers,

So I created a bitbake recipe to integrate kvm into my image as an out-of-tree kernel module.  Here's my recipe right now:

LICENSE = "GPLv2"
LIC_FILES_CHKSUM = "file://COPYING;md5=c616d0e7924e9e78ee192d99a3b26fbd"

inherit module

SRC_URI = "file:///homead/QorIQ-SDK-V2.0-20160527-yocto/sources/meta-virtualization/recipes-kernel/kvm-kmodule/kvm-kmod-3.10.21.tar.bz2"

S = "${WORKDIR}/kvm-kmod-3.10.21"

do_configure() {
    ./configure --arch=ppc64 --kerneldir=/homead/QorIQ-SDK-V2.0-20160527-yocto/build_t4240rdb-64b/tmp/work/t4240rdb_64b-fsl-linux/kernel-devsrc/1.0-r0/image/usr/src/kernel
}

FILES_${PN} += "/lib/modules/4.1.8-rt8+gbd51baf"

Bitbaking this recipe completes with no problems but no kernel module is created (compiling the source code should create a kernel module file kvm.ko).  I was wondering if the problem might be because of what I set FILES_${PN} to be.  Before I set the FILES variable I was getting an error saying something along the lines of, "Files/directories were installed but not shipped."  Then I more or less just guessed a directory and set my FILES variable to it and then the recipe finished bitbaking with no errors.

But now that the kvm.ko file isn't even being created, I am wondering if it might be because I set the FILES variable wrong?  What does the word "ship" mean?  And along those lines what exactly is a "package" in the setting of Yocto project?

-Thanks!, Wayne Li

[-- Attachment #2: Type: text/html, Size: 12019 bytes --]

Re: [yocto] Question about shipping files to package.

[McKay, Sean]

[-- Attachment #1: Type: text/plain, Size: 5753 bytes --]

If the issue were just that your FILES_* variables were set incorrectly, you’d at least end up with the .ko file somewhere in your ${WORKDIR}/image (${D}) directory. You’d almost certainly find another one in the build or source directory as well (depending on how your .bb/bbclass files define the location of the compilation).

The fact that you don’t have one anywhere seems to indicate that your do_compile step isn’t actually generating the .ko that you expect it to, so you’ll need to rewind your recipe creation to working on that step and debug there.

Assuming you’re using poky as a base, what version are you on so I can go read the .bbclass you’re using?

-Sean

From: Wayne Li <waynli329@gmail.com>
Sent: Tuesday, December 10, 2019 12:23 PM
To: McKay, Sean <sean.mckay@hpe.com>
Cc: bitbake-devel <bitbake-devel@lists.openembedded.org>; Yocto Project Discussion <yocto@yoctoproject.org>
Subject: Re: [yocto] Question about shipping files to package.

I did a find for any files with a kvm and a .ko in its name throughout my entire workspace and nothing came up.  But anyway, thanks for the explanation.  Could packaging the files wrong (i.e. setting the FILES variable wrong) cause the kvm.ko file to never show up at all?  I guess that couldn't be the case though because the .ko files are created first before it gets packaged up so if they were created in the first place a search through my entire workspace would turn them up.  Is that correct?

On Thu, Dec 5, 2019 at 11:14 PM McKay, Sean <sean.mckay@hpe.com<mailto:sean.mckay@hpe.com>> wrote:
I’m not an expert on compiling kernel modules, but I can at least answer some of your questions to get started. I’m also not an actual expert, but just another user, so it’s entirely possible that someone more knowledgeable will come along and tell us that I’m wrong.

With that said, first, I have to ask the simple stuff: when you say that no .ko is being created, where are you looking? If you’re not sure where you should be looking, I’d recommend by doing a find inside of your ${WORKDIR} to see whether or not you’ve got any .ko files. They won’t quite be out in plain sight.

In general (though again, I can’t speak for the module class), compilations take place in a build directory inside of the recipe’s workdir (${B} in bitbake parlance). The install process then copies the created files to a staging directory (${D} in bitbake parlance) that’s usually ${WORKDIR}/image.

Packaging is the process of taking those output files from ${D} and packaging them into one of a few well defined packaging formats. If you’re using the defaults, this would be RPM. A single recipe can (and often does) produce multiple packages. When bitbake moves onto the packaging step (I’m simplifying a bit), it creates directories for each package that’s going to be generated inside of ${WORKDIR}/packages-split. It then looks at the FILES variable for the particular package it’s working on and copies any files that match the patterns in that variable into that package’s directory in packages-split. When it’s done copying files, each of those directories are turned into the appropriate package (probably .rpm) file.
So, for a somewhat more concrete example, when you said ‘FILES_${PN} += "/lib/modules/4.1.8-rt8+gbd51baf"’, you told bitbake that anything that matched /lib/modules/4.1.8-rt8+gbd51baf should get put in the ${PN} (main) package.

In the context of the error message you were previously receiving, “installed” is the do_install step, which is referring to that staging process (analogous to running ‘make install’ on something you just downloaded from the internet). Something is ‘shipped’ if it is packaged into one of the final rpm files.

Cheers!
-Sean McKay

From: yocto@lists.yoctoproject.org<mailto:yocto@lists.yoctoproject.org> <yocto@lists.yoctoproject.org<mailto:yocto@lists.yoctoproject.org>> On Behalf Of Wayne Li
Sent: Thursday, December 5, 2019 2:26 PM
To: bitbake-devel <bitbake-devel@lists.openembedded.org<mailto:bitbake-devel@lists.openembedded.org>>; Yocto Project Discussion <yocto@yoctoproject.org<mailto:yocto@yoctoproject.org>>
Subject: [yocto] Question about shipping files to package.

Dear Yocto Developers,

So I created a bitbake recipe to integrate kvm into my image as an out-of-tree kernel module.  Here's my recipe right now:

LICENSE = "GPLv2"
LIC_FILES_CHKSUM = "file://COPYING;md5=c616d0e7924e9e78ee192d99a3b26fbd"

inherit module

SRC_URI = "file:///homead/QorIQ-SDK-V2.0-20160527-yocto/sources/meta-virtualization/recipes-kernel/kvm-kmodule/kvm-kmod-3.10.21.tar.bz2"

S = "${WORKDIR}/kvm-kmod-3.10.21"

do_configure() {
    ./configure --arch=ppc64 --kerneldir=/homead/QorIQ-SDK-V2.0-20160527-yocto/build_t4240rdb-64b/tmp/work/t4240rdb_64b-fsl-linux/kernel-devsrc/1.0-r0/image/usr/src/kernel
}

FILES_${PN} += "/lib/modules/4.1.8-rt8+gbd51baf"

Bitbaking this recipe completes with no problems but no kernel module is created (compiling the source code should create a kernel module file kvm.ko).  I was wondering if the problem might be because of what I set FILES_${PN} to be.  Before I set the FILES variable I was getting an error saying something along the lines of, "Files/directories were installed but not shipped."  Then I more or less just guessed a directory and set my FILES variable to it and then the recipe finished bitbaking with no errors.

But now that the kvm.ko file isn't even being created, I am wondering if it might be because I set the FILES variable wrong?  What does the word "ship" mean?  And along those lines what exactly is a "package" in the setting of Yocto project?

-Thanks!, Wayne Li

[-- Attachment #2: Type: text/html, Size: 12019 bytes --]

Re: [yocto] Question about shipping files to package.

[Wayne Li]

[-- Attachment #1: Type: text/plain, Size: 6183 bytes --]

That makes a lot of sense.  The version I'm on is: PV="4.1" (I just ran
bitbake -e virtual/kernel | grep "^PV").

On Tue, Dec 10, 2019 at 8:54 PM McKay, Sean <sean.mckay@hpe.com> wrote:

> If the issue were *just* that your FILES_* variables were set
> incorrectly, you’d at least end up with the .ko file somewhere in your
> ${WORKDIR}/image (${D}) directory. You’d almost certainly find another one
> in the build or source directory as well (depending on how your .bb/bbclass
> files define the location of the compilation).
>
>
>
> The fact that you don’t have one anywhere seems to indicate that your
> do_compile step isn’t actually generating the .ko that you expect it to, so
> you’ll need to rewind your recipe creation to working on that step and
> debug there.
>
>
>
> Assuming you’re using poky as a base, what version are you on so I can go
> read the .bbclass you’re using?
>
>
>
> -Sean
>
>
>
> *From:* Wayne Li <waynli329@gmail.com>
> *Sent:* Tuesday, December 10, 2019 12:23 PM
> *To:* McKay, Sean <sean.mckay@hpe.com>
> *Cc:* bitbake-devel <bitbake-devel@lists.openembedded.org>; Yocto Project
> Discussion <yocto@yoctoproject.org>
> *Subject:* Re: [yocto] Question about shipping files to package.
>
>
>
> I did a find for any files with a kvm and a .ko in its name throughout my
> entire workspace and nothing came up.  But anyway, thanks for the
> explanation.  Could packaging the files wrong (i.e. setting the FILES
> variable wrong) cause the kvm.ko file to never show up at all?  I guess
> that couldn't be the case though because the .ko files are created first
> before it gets packaged up so if they were created in the first place a
> search through my entire workspace would turn them up.  Is that correct?
>
>
>
> On Thu, Dec 5, 2019 at 11:14 PM McKay, Sean <sean.mckay@hpe.com> wrote:
>
> I’m not an expert on compiling kernel modules, but I can at least answer
> some of your questions to get started. I’m also not an actual expert, but
> just another user, so it’s entirely possible that someone more
> knowledgeable will come along and tell us that I’m wrong.
>
>
>
> With that said, first, I have to ask the simple stuff: when you say that
> no .ko is being created, where are you looking? If you’re not sure where
> you should be looking, I’d recommend by doing a find inside of your
> ${WORKDIR} to see whether or not you’ve got any .ko files. They won’t quite
> be out in plain sight.
>
>
>
> In general (though again, I can’t speak for the module class),
> compilations take place in a build directory inside of the recipe’s workdir
> (${B} in bitbake parlance). The install process then copies the created
> files to a staging directory (${D} in bitbake parlance) that’s usually
> ${WORKDIR}/image.
>
>
>
> Packaging is the process of taking those output files from ${D} and
> packaging them into one of a few well defined packaging formats. If you’re
> using the defaults, this would be RPM. A single recipe can (and often does)
> produce multiple packages. When bitbake moves onto the packaging step (I’m
> simplifying a bit), it creates directories for each package that’s going to
> be generated inside of ${WORKDIR}/packages-split. It then looks at the
> FILES variable for the particular package it’s working on and copies any
> files that match the patterns in that variable into that package’s
> directory in packages-split. When it’s done copying files, each of those
> directories are turned into the appropriate package (probably .rpm) file.
>
> So, for a somewhat more concrete example, when you said ‘FILES_${PN} +=
> "/lib/modules/4.1.8-rt8+gbd51baf"’, you told bitbake that anything that
> matched /lib/modules/4.1.8-rt8+gbd51baf should get put in the ${PN} (main)
> package.
>
>
>
> In the context of the error message you were previously receiving,
> “installed” is the do_install step, which is referring to that staging
> process (analogous to running ‘make install’ on something you just
> downloaded from the internet). Something is ‘shipped’ if it is packaged
> into one of the final rpm files.
>
>
>
> Cheers!
>
> -Sean McKay
>
>
>
> *From:* yocto@lists.yoctoproject.org <yocto@lists.yoctoproject.org> *On
> Behalf Of *Wayne Li
> *Sent:* Thursday, December 5, 2019 2:26 PM
> *To:* bitbake-devel <bitbake-devel@lists.openembedded.org>; Yocto Project
> Discussion <yocto@yoctoproject.org>
> *Subject:* [yocto] Question about shipping files to package.
>
>
>
> Dear Yocto Developers,
>
>
>
> So I created a bitbake recipe to integrate kvm into my image as an
> out-of-tree kernel module.  Here's my recipe right now:
>
>
>
> LICENSE = "GPLv2"
> LIC_FILES_CHKSUM = "file://COPYING;md5=c616d0e7924e9e78ee192d99a3b26fbd"
>
> inherit module
>
> SRC_URI = "
> file:///homead/QorIQ-SDK-V2.0-20160527-yocto/sources/meta-virtualization/recipes-kernel/kvm-kmodule/kvm-kmod-3.10.21.tar.bz2
> "
>
> S = "${WORKDIR}/kvm-kmod-3.10.21"
>
> do_configure() {
>     ./configure --arch=ppc64
> --kerneldir=/homead/QorIQ-SDK-V2.0-20160527-yocto/build_t4240rdb-64b/tmp/work/t4240rdb_64b-fsl-linux/kernel-devsrc/1.0-r0/image/usr/src/kernel
> }
>
> FILES_${PN} += "/lib/modules/4.1.8-rt8+gbd51baf"
>
>
>
> Bitbaking this recipe completes with no problems but no kernel module is
> created (compiling the source code should create a kernel module file
> kvm.ko).  I was wondering if the problem might be because of what I set
> FILES_${PN} to be.  Before I set the FILES variable I was getting an error
> saying something along the lines of, "Files/directories were installed but
> not shipped."  Then I more or less just guessed a directory and set my
> FILES variable to it and then the recipe finished bitbaking with no errors.
>
>
>
> But now that the kvm.ko file isn't even being created, I am wondering if
> it might be because I set the FILES variable wrong?  What does the word
> "ship" mean?  And along those lines what exactly is a "package" in the
> setting of Yocto project?
>
>
>
> -Thanks!, Wayne Li
>
>

[-- Attachment #2: Type: text/html, Size: 9247 bytes --]

Re: [yocto] Question about shipping files to package.

[Wayne Li]

[-- Attachment #1: Type: text/plain, Size: 6183 bytes --]

That makes a lot of sense.  The version I'm on is: PV="4.1" (I just ran
bitbake -e virtual/kernel | grep "^PV").

On Tue, Dec 10, 2019 at 8:54 PM McKay, Sean <sean.mckay@hpe.com> wrote:

> If the issue were *just* that your FILES_* variables were set
> incorrectly, you’d at least end up with the .ko file somewhere in your
> ${WORKDIR}/image (${D}) directory. You’d almost certainly find another one
> in the build or source directory as well (depending on how your .bb/bbclass
> files define the location of the compilation).
>
>
>
> The fact that you don’t have one anywhere seems to indicate that your
> do_compile step isn’t actually generating the .ko that you expect it to, so
> you’ll need to rewind your recipe creation to working on that step and
> debug there.
>
>
>
> Assuming you’re using poky as a base, what version are you on so I can go
> read the .bbclass you’re using?
>
>
>
> -Sean
>
>
>
> *From:* Wayne Li <waynli329@gmail.com>
> *Sent:* Tuesday, December 10, 2019 12:23 PM
> *To:* McKay, Sean <sean.mckay@hpe.com>
> *Cc:* bitbake-devel <bitbake-devel@lists.openembedded.org>; Yocto Project
> Discussion <yocto@yoctoproject.org>
> *Subject:* Re: [yocto] Question about shipping files to package.
>
>
>
> I did a find for any files with a kvm and a .ko in its name throughout my
> entire workspace and nothing came up.  But anyway, thanks for the
> explanation.  Could packaging the files wrong (i.e. setting the FILES
> variable wrong) cause the kvm.ko file to never show up at all?  I guess
> that couldn't be the case though because the .ko files are created first
> before it gets packaged up so if they were created in the first place a
> search through my entire workspace would turn them up.  Is that correct?
>
>
>
> On Thu, Dec 5, 2019 at 11:14 PM McKay, Sean <sean.mckay@hpe.com> wrote:
>
> I’m not an expert on compiling kernel modules, but I can at least answer
> some of your questions to get started. I’m also not an actual expert, but
> just another user, so it’s entirely possible that someone more
> knowledgeable will come along and tell us that I’m wrong.
>
>
>
> With that said, first, I have to ask the simple stuff: when you say that
> no .ko is being created, where are you looking? If you’re not sure where
> you should be looking, I’d recommend by doing a find inside of your
> ${WORKDIR} to see whether or not you’ve got any .ko files. They won’t quite
> be out in plain sight.
>
>
>
> In general (though again, I can’t speak for the module class),
> compilations take place in a build directory inside of the recipe’s workdir
> (${B} in bitbake parlance). The install process then copies the created
> files to a staging directory (${D} in bitbake parlance) that’s usually
> ${WORKDIR}/image.
>
>
>
> Packaging is the process of taking those output files from ${D} and
> packaging them into one of a few well defined packaging formats. If you’re
> using the defaults, this would be RPM. A single recipe can (and often does)
> produce multiple packages. When bitbake moves onto the packaging step (I’m
> simplifying a bit), it creates directories for each package that’s going to
> be generated inside of ${WORKDIR}/packages-split. It then looks at the
> FILES variable for the particular package it’s working on and copies any
> files that match the patterns in that variable into that package’s
> directory in packages-split. When it’s done copying files, each of those
> directories are turned into the appropriate package (probably .rpm) file.
>
> So, for a somewhat more concrete example, when you said ‘FILES_${PN} +=
> "/lib/modules/4.1.8-rt8+gbd51baf"’, you told bitbake that anything that
> matched /lib/modules/4.1.8-rt8+gbd51baf should get put in the ${PN} (main)
> package.
>
>
>
> In the context of the error message you were previously receiving,
> “installed” is the do_install step, which is referring to that staging
> process (analogous to running ‘make install’ on something you just
> downloaded from the internet). Something is ‘shipped’ if it is packaged
> into one of the final rpm files.
>
>
>
> Cheers!
>
> -Sean McKay
>
>
>
> *From:* yocto@lists.yoctoproject.org <yocto@lists.yoctoproject.org> *On
> Behalf Of *Wayne Li
> *Sent:* Thursday, December 5, 2019 2:26 PM
> *To:* bitbake-devel <bitbake-devel@lists.openembedded.org>; Yocto Project
> Discussion <yocto@yoctoproject.org>
> *Subject:* [yocto] Question about shipping files to package.
>
>
>
> Dear Yocto Developers,
>
>
>
> So I created a bitbake recipe to integrate kvm into my image as an
> out-of-tree kernel module.  Here's my recipe right now:
>
>
>
> LICENSE = "GPLv2"
> LIC_FILES_CHKSUM = "file://COPYING;md5=c616d0e7924e9e78ee192d99a3b26fbd"
>
> inherit module
>
> SRC_URI = "
> file:///homead/QorIQ-SDK-V2.0-20160527-yocto/sources/meta-virtualization/recipes-kernel/kvm-kmodule/kvm-kmod-3.10.21.tar.bz2
> "
>
> S = "${WORKDIR}/kvm-kmod-3.10.21"
>
> do_configure() {
>     ./configure --arch=ppc64
> --kerneldir=/homead/QorIQ-SDK-V2.0-20160527-yocto/build_t4240rdb-64b/tmp/work/t4240rdb_64b-fsl-linux/kernel-devsrc/1.0-r0/image/usr/src/kernel
> }
>
> FILES_${PN} += "/lib/modules/4.1.8-rt8+gbd51baf"
>
>
>
> Bitbaking this recipe completes with no problems but no kernel module is
> created (compiling the source code should create a kernel module file
> kvm.ko).  I was wondering if the problem might be because of what I set
> FILES_${PN} to be.  Before I set the FILES variable I was getting an error
> saying something along the lines of, "Files/directories were installed but
> not shipped."  Then I more or less just guessed a directory and set my
> FILES variable to it and then the recipe finished bitbaking with no errors.
>
>
>
> But now that the kvm.ko file isn't even being created, I am wondering if
> it might be because I set the FILES variable wrong?  What does the word
> "ship" mean?  And along those lines what exactly is a "package" in the
> setting of Yocto project?
>
>
>
> -Thanks!, Wayne Li
>
>

[-- Attachment #2: Type: text/html, Size: 9247 bytes --]