Best practice for files copied directly into sysroot?

Best practice for files copied directly into sysroot?

[Michael Habibi]

[-- Attachment #1: Type: text/plain, Size: 555 bytes --]

I am working on potentially migrating our distribution to Yocto (will be a
long, long process). We have an area of our filesystem where any file
placed will go directly into the rootfs of the target. This works well with
our product, as we have certain binaries and scripts that need to be copied
directly, and there's no other real step involved (no compilation, etc). Is
there a best practice or existing class that handles this? I saw mention of
a binary/ directory but I wasn't sure if that was for this or something
else.

Thanks,
Michael

[-- Attachment #2: Type: text/html, Size: 608 bytes --]

Re: Best practice for files copied directly into sysroot?

[Bryan Evenson]

Michael,

> -----Original Message-----
> From: yocto-bounces@yoctoproject.org [mailto:yocto-
> bounces@yoctoproject.org] On Behalf Of Michael Habibi
> Sent: Tuesday, December 01, 2015 10:26 AM
> To: yocto@yoctoproject.org
> Subject: [yocto] Best practice for files copied directly into sysroot?
> 
> I am working on potentially migrating our distribution to Yocto (will be a long,
> long process). We have an area of our filesystem where any file placed will go
> directly into the rootfs of the target. This works well with our product, as we
> have certain binaries and scripts that need to be copied directly, and there's
> no other real step involved (no compilation, etc). Is there a best practice or
> existing class that handles this? I saw mention of a binary/ directory but I
> wasn't sure if that was for this or something else.
> 
> Thanks,
> Michael

I'm glad I decided to answer your question because I think I found an easier way to do it than I have been doing this in the past.  If you want to be able to update the files on your system at a later date, you will want to make your own recipe which installs the binaries and scripts onto the rootfs.  In general you create a recipe which does not have do_configure or do_compile tasks.  There is a class that does this already called "bin_package" that does this for you.  See the "Packaging Externally Produced Binaries" section of the manual here: http://www.yoctoproject.org/docs/2.0/mega-manual/mega-manual.html#new-recipe-testing-examples.   So if you include the line "inherit bin_package" and follow the instructions in that section of the manual, it should install all the files to the rootfs that are included in the recipe.

Regards,
Bryan

Re: Best practice for files copied directly into sysroot?

[Michael Habibi]

[-- Attachment #1: Type: text/plain, Size: 1938 bytes --]

Thanks Bryan!

On Tue, Dec 1, 2015 at 1:50 PM, Bryan Evenson <bevenson@melinkcorp.com>
wrote:

> Michael,
>
> > -----Original Message-----
> > From: yocto-bounces@yoctoproject.org [mailto:yocto-
> > bounces@yoctoproject.org] On Behalf Of Michael Habibi
> > Sent: Tuesday, December 01, 2015 10:26 AM
> > To: yocto@yoctoproject.org
> > Subject: [yocto] Best practice for files copied directly into sysroot?
> >
> > I am working on potentially migrating our distribution to Yocto (will be
> a long,
> > long process). We have an area of our filesystem where any file placed
> will go
> > directly into the rootfs of the target. This works well with our
> product, as we
> > have certain binaries and scripts that need to be copied directly, and
> there's
> > no other real step involved (no compilation, etc). Is there a best
> practice or
> > existing class that handles this? I saw mention of a binary/ directory
> but I
> > wasn't sure if that was for this or something else.
> >
> > Thanks,
> > Michael
>
> I'm glad I decided to answer your question because I think I found an
> easier way to do it than I have been doing this in the past.  If you want
> to be able to update the files on your system at a later date, you will
> want to make your own recipe which installs the binaries and scripts onto
> the rootfs.  In general you create a recipe which does not have
> do_configure or do_compile tasks.  There is a class that does this already
> called "bin_package" that does this for you.  See the "Packaging Externally
> Produced Binaries" section of the manual here:
> http://www.yoctoproject.org/docs/2.0/mega-manual/mega-manual.html#new-recipe-testing-examples.
>  So if you include the line "inherit bin_package" and follow the
> instructions in that section of the manual, it should install all the files
> to the rootfs that are included in the recipe.
>
> Regards,
> Bryan
>
>

[-- Attachment #2: Type: text/html, Size: 2675 bytes --]

Re: Best practice for files copied directly into sysroot?

[Michael Habibi]

[-- Attachment #1: Type: text/plain, Size: 2442 bytes --]

Bryan,

I looked through the documentation and I'm a bit unclear on how to use
bin_package, and haven't been able to find a good example. Where do I
actually place my rootfs files/binaries to get copied to my image rootfs?
Under ${THISDIR}/${PN} like other recipe artifacts? How does it copy them
to ${S}? Or is this something I need to do?

Thanks,
Michael

On Tue, Dec 1, 2015 at 2:03 PM, Michael Habibi <mikehabibi@gmail.com> wrote:

> Thanks Bryan!
>
> On Tue, Dec 1, 2015 at 1:50 PM, Bryan Evenson <bevenson@melinkcorp.com>
> wrote:
>
>> Michael,
>>
>> > -----Original Message-----
>> > From: yocto-bounces@yoctoproject.org [mailto:yocto-
>> > bounces@yoctoproject.org] On Behalf Of Michael Habibi
>> > Sent: Tuesday, December 01, 2015 10:26 AM
>> > To: yocto@yoctoproject.org
>> > Subject: [yocto] Best practice for files copied directly into sysroot?
>> >
>> > I am working on potentially migrating our distribution to Yocto (will
>> be a long,
>> > long process). We have an area of our filesystem where any file placed
>> will go
>> > directly into the rootfs of the target. This works well with our
>> product, as we
>> > have certain binaries and scripts that need to be copied directly, and
>> there's
>> > no other real step involved (no compilation, etc). Is there a best
>> practice or
>> > existing class that handles this? I saw mention of a binary/ directory
>> but I
>> > wasn't sure if that was for this or something else.
>> >
>> > Thanks,
>> > Michael
>>
>> I'm glad I decided to answer your question because I think I found an
>> easier way to do it than I have been doing this in the past.  If you want
>> to be able to update the files on your system at a later date, you will
>> want to make your own recipe which installs the binaries and scripts onto
>> the rootfs.  In general you create a recipe which does not have
>> do_configure or do_compile tasks.  There is a class that does this already
>> called "bin_package" that does this for you.  See the "Packaging Externally
>> Produced Binaries" section of the manual here:
>> http://www.yoctoproject.org/docs/2.0/mega-manual/mega-manual.html#new-recipe-testing-examples.
>>  So if you include the line "inherit bin_package" and follow the
>> instructions in that section of the manual, it should install all the files
>> to the rootfs that are included in the recipe.
>>
>> Regards,
>> Bryan
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 3563 bytes --]

Re: Best practice for files copied directly into sysroot?

[Bryan Evenson]

Michael,

> -----Original Message-----
> From: Michael Habibi [mailto:mikehabibi@gmail.com]
> Sent: Thursday, December 03, 2015 5:54 PM
> To: Bryan Evenson <bevenson@melinkcorp.com>
> Cc: yocto@yoctoproject.org
> Subject: Re: [yocto] Best practice for files copied directly into sysroot?
> 
> Bryan,
> 
> I looked through the documentation and I'm a bit unclear on how to use
> bin_package, and haven't been able to find a good example. Where do I
> actually place my rootfs files/binaries to get copied to my image rootfs?
> Under ${THISDIR}/${PN} like other recipe artifacts? How does it copy them to
> ${S}? Or is this something I need to do?
> 

I just modified one of my recipes yesterday to use bin_package yesterday and found out a few nuances that make it easier to use.  Proper setup of the files to copy is key.  The manual states " The bin_package class works well when the files extracted into ${S} are already laid out in the way they should be laid out on the target."  By default, any files listed as "file://" in SRC_URI are copied to ${WORKDIR}, and if they are a compressed archive (.tar.gz, .zip) they are uncompressed under ${WORKDIR}.  If you create a tarball with the files in the locations (and permissions!) that you want on the final image, then the bin_package class will do the rest of the work.  However, you need the files in the tarball to be one directory level down so that they are installed on the image properly. 

Let's assume you have a recipe under meta-mylayer/recipes-mine/my-app/ called my-app_1.0.0.bb.  You have an executable called my-app that you want to install in /usr/bin/, a configuration file called my-app.conf that you want to install at /usr/share/my-app/ and documentation called my-app.pdf you want to install at /usr/share/info/.  You want all this to be under a directory so that when the tarball gets unpacked under ${WORKDIR} the files and directories you want on the final image are under ${S}.  ${S} defaults to ${WORKDIR}/${BP}, which in this case would be ${WORKDIR}/my-app-1.0.0.  So if you created the following directory structure:

my-app-1.0.0/
    usr/
        bin/
            my-app
        info/
            my-app.pdf
        my-app/
            my-app.conf

Again, remember to set the directory and file permissions as you want on the final image.  Then create a tarball from the directory you laid out called my-app-1.0.0.tar.gz and place it under meta-mylayer/recipes-mine/my-app/files/.  Then create your recipe as: 

SUMMARY = "summary here"
SECTION = "proper section name here"
LICENSE = "license name here"
LIC_FILES_CHKSUM = "license path and MD5sum here"

SRC_URI = "file://${BP}.tar.gz"
inherit bin_package

The system will then package your files and install them onto the image where they belong.  These are the only lines you need in your recipe other than the standard header variables.  In the future, if you have a new version then just recreate the tarball with the updated files and rename the tarball to match the new version.  Although I am having issues with the license; I haven't figured out where to put it and what to set LIC_FILES_CHKSUM to in the recipe so that bitbake can find the license.  If anyone has a good answer for that, let me know.

Regards,
Bryan

> Thanks,
> Michael
> 
> On Tue, Dec 1, 2015 at 2:03 PM, Michael Habibi <mikehabibi@gmail.com
> <mailto:mikehabibi@gmail.com> > wrote:
> 
> 
> 	Thanks Bryan!
> 
> 	On Tue, Dec 1, 2015 at 1:50 PM, Bryan Evenson
> <bevenson@melinkcorp.com <mailto:bevenson@melinkcorp.com> > wrote:
> 
> 
> 		Michael,
> 
> 
> 		> -----Original Message-----
> 		> From: yocto-bounces@yoctoproject.org <mailto:yocto-
> bounces@yoctoproject.org>  [mailto:yocto- <mailto:yocto->
> 		> bounces@yoctoproject.org
> <mailto:bounces@yoctoproject.org> ] On Behalf Of Michael Habibi
> 		> Sent: Tuesday, December 01, 2015 10:26 AM
> 		> To: yocto@yoctoproject.org
> <mailto:yocto@yoctoproject.org>
> 		> Subject: [yocto] Best practice for files copied directly into
> sysroot?
> 		>
> 		> I am working on potentially migrating our distribution to
> Yocto (will be a long,
> 		> long process). We have an area of our filesystem where
> any file placed will go
> 		> directly into the rootfs of the target. This works well with
> our product, as we
> 		> have certain binaries and scripts that need to be copied
> directly, and there's
> 		> no other real step involved (no compilation, etc). Is there a
> best practice or
> 		> existing class that handles this? I saw mention of a binary/
> directory but I
> 		> wasn't sure if that was for this or something else.
> 		>
> 		> Thanks,
> 		> Michael
> 
> 
> 		I'm glad I decided to answer your question because I think I
> found an easier way to do it than I have been doing this in the past.  If you
> want to be able to update the files on your system at a later date, you will
> want to make your own recipe which installs the binaries and scripts onto the
> rootfs.  In general you create a recipe which does not have do_configure or
> do_compile tasks.  There is a class that does this already called "bin_package"
> that does this for you.  See the "Packaging Externally Produced Binaries"
> section of the manual here: http://www.yoctoproject.org/docs/2.0/mega-
> manual/mega-manual.html#new-recipe-testing-examples.   So if you include
> the line "inherit bin_package" and follow the instructions in that section of
> the manual, it should install all the files to the rootfs that are included in the
> recipe.
> 
> 		Regards,
> 		Bryan
> 
> 
> 
> 

Re: Best practice for files copied directly into sysroot?

[Michael Habibi]

[-- Attachment #1: Type: text/plain, Size: 6348 bytes --]

This is perfect, thanks Bryan.

On Fri, Dec 4, 2015 at 8:03 AM, Bryan Evenson <bevenson@melinkcorp.com>
wrote:

> Michael,
>
> > -----Original Message-----
> > From: Michael Habibi [mailto:mikehabibi@gmail.com]
> > Sent: Thursday, December 03, 2015 5:54 PM
> > To: Bryan Evenson <bevenson@melinkcorp.com>
> > Cc: yocto@yoctoproject.org
> > Subject: Re: [yocto] Best practice for files copied directly into
> sysroot?
> >
> > Bryan,
> >
> > I looked through the documentation and I'm a bit unclear on how to use
> > bin_package, and haven't been able to find a good example. Where do I
> > actually place my rootfs files/binaries to get copied to my image rootfs?
> > Under ${THISDIR}/${PN} like other recipe artifacts? How does it copy
> them to
> > ${S}? Or is this something I need to do?
> >
>
> I just modified one of my recipes yesterday to use bin_package yesterday
> and found out a few nuances that make it easier to use.  Proper setup of
> the files to copy is key.  The manual states " The bin_package class works
> well when the files extracted into ${S} are already laid out in the way
> they should be laid out on the target."  By default, any files listed as
> "file://" in SRC_URI are copied to ${WORKDIR}, and if they are a compressed
> archive (.tar.gz, .zip) they are uncompressed under ${WORKDIR}.  If you
> create a tarball with the files in the locations (and permissions!) that
> you want on the final image, then the bin_package class will do the rest of
> the work.  However, you need the files in the tarball to be one directory
> level down so that they are installed on the image properly.
>
> Let's assume you have a recipe under meta-mylayer/recipes-mine/my-app/
> called my-app_1.0.0.bb.  You have an executable called my-app that you
> want to install in /usr/bin/, a configuration file called my-app.conf that
> you want to install at /usr/share/my-app/ and documentation called
> my-app.pdf you want to install at /usr/share/info/.  You want all this to
> be under a directory so that when the tarball gets unpacked under
> ${WORKDIR} the files and directories you want on the final image are under
> ${S}.  ${S} defaults to ${WORKDIR}/${BP}, which in this case would be
> ${WORKDIR}/my-app-1.0.0.  So if you created the following directory
> structure:
>
> my-app-1.0.0/
>     usr/
>         bin/
>             my-app
>         info/
>             my-app.pdf
>         my-app/
>             my-app.conf
>
> Again, remember to set the directory and file permissions as you want on
> the final image.  Then create a tarball from the directory you laid out
> called my-app-1.0.0.tar.gz and place it under
> meta-mylayer/recipes-mine/my-app/files/.  Then create your recipe as:
>
> SUMMARY = "summary here"
> SECTION = "proper section name here"
> LICENSE = "license name here"
> LIC_FILES_CHKSUM = "license path and MD5sum here"
>
> SRC_URI = "file://${BP}.tar.gz"
> inherit bin_package
>
> The system will then package your files and install them onto the image
> where they belong.  These are the only lines you need in your recipe other
> than the standard header variables.  In the future, if you have a new
> version then just recreate the tarball with the updated files and rename
> the tarball to match the new version.  Although I am having issues with the
> license; I haven't figured out where to put it and what to set
> LIC_FILES_CHKSUM to in the recipe so that bitbake can find the license.  If
> anyone has a good answer for that, let me know.
>
> Regards,
> Bryan
>
> > Thanks,
> > Michael
> >
> > On Tue, Dec 1, 2015 at 2:03 PM, Michael Habibi <mikehabibi@gmail.com
> > <mailto:mikehabibi@gmail.com> > wrote:
> >
> >
> >       Thanks Bryan!
> >
> >       On Tue, Dec 1, 2015 at 1:50 PM, Bryan Evenson
> > <bevenson@melinkcorp.com <mailto:bevenson@melinkcorp.com> > wrote:
> >
> >
> >               Michael,
> >
> >
> >               > -----Original Message-----
> >               > From: yocto-bounces@yoctoproject.org <mailto:yocto-
> > bounces@yoctoproject.org>  [mailto:yocto- <mailto:yocto->
> >               > bounces@yoctoproject.org
> > <mailto:bounces@yoctoproject.org> ] On Behalf Of Michael Habibi
> >               > Sent: Tuesday, December 01, 2015 10:26 AM
> >               > To: yocto@yoctoproject.org
> > <mailto:yocto@yoctoproject.org>
> >               > Subject: [yocto] Best practice for files copied directly
> into
> > sysroot?
> >               >
> >               > I am working on potentially migrating our distribution to
> > Yocto (will be a long,
> >               > long process). We have an area of our filesystem where
> > any file placed will go
> >               > directly into the rootfs of the target. This works well
> with
> > our product, as we
> >               > have certain binaries and scripts that need to be copied
> > directly, and there's
> >               > no other real step involved (no compilation, etc). Is
> there a
> > best practice or
> >               > existing class that handles this? I saw mention of a
> binary/
> > directory but I
> >               > wasn't sure if that was for this or something else.
> >               >
> >               > Thanks,
> >               > Michael
> >
> >
> >               I'm glad I decided to answer your question because I think
> I
> > found an easier way to do it than I have been doing this in the past.
> If you
> > want to be able to update the files on your system at a later date, you
> will
> > want to make your own recipe which installs the binaries and scripts
> onto the
> > rootfs.  In general you create a recipe which does not have do_configure
> or
> > do_compile tasks.  There is a class that does this already called
> "bin_package"
> > that does this for you.  See the "Packaging Externally Produced Binaries"
> > section of the manual here: http://www.yoctoproject.org/docs/2.0/mega-
> > manual/mega-manual.html#new-recipe-testing-examples.   So if you include
> > the line "inherit bin_package" and follow the instructions in that
> section of
> > the manual, it should install all the files to the rootfs that are
> included in the
> > recipe.
> >
> >               Regards,
> >               Bryan
> >
> >
> >
> >
>
>

[-- Attachment #2: Type: text/html, Size: 8461 bytes --]

Re: Best practice for files copied directly into sysroot?

[Paul Eggleton]

Hi Michael,

On Tue, 01 Dec 2015 09:26:15 Michael Habibi wrote:
> I am working on potentially migrating our distribution to Yocto (will be a
> long, long process). We have an area of our filesystem where any file
> placed will go directly into the rootfs of the target. This works well with
> our product, as we have certain binaries and scripts that need to be copied
> directly, and there's no other real step involved (no compilation, etc). Is
> there a best practice or existing class that handles this? I saw mention of
> a binary/ directory but I wasn't sure if that was for this or something
> else.

I see you got your answer - but I wanted to point out when you wrote "sysroot" 
you meant "rootfs" - these two aren't the same thing. The sysroot is the area 
used to share files between recipes (e.g. libraries and headers); whilst it can 
look a bit similar to what ends up in the rootfs it's actually completely 
separate.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Best practice for files copied directly into sysroot?

[Michael Habibi]

[-- Attachment #1: Type: text/plain, Size: 1245 bytes --]

Yes you're right thanks for correcting - I meant rootfs!

On Sat, Dec 5, 2015 at 3:23 PM, Paul Eggleton <paul.eggleton@linux.intel.com
> wrote:

> Hi Michael,
>
> On Tue, 01 Dec 2015 09:26:15 Michael Habibi wrote:
> > I am working on potentially migrating our distribution to Yocto (will be
> a
> > long, long process). We have an area of our filesystem where any file
> > placed will go directly into the rootfs of the target. This works well
> with
> > our product, as we have certain binaries and scripts that need to be
> copied
> > directly, and there's no other real step involved (no compilation, etc).
> Is
> > there a best practice or existing class that handles this? I saw mention
> of
> > a binary/ directory but I wasn't sure if that was for this or something
> > else.
>
> I see you got your answer - but I wanted to point out when you wrote
> "sysroot"
> you meant "rootfs" - these two aren't the same thing. The sysroot is the
> area
> used to share files between recipes (e.g. libraries and headers); whilst
> it can
> look a bit similar to what ends up in the rootfs it's actually completely
> separate.
>
> Cheers,
> Paul
>
> --
>
> Paul Eggleton
> Intel Open Source Technology Centre
>

[-- Attachment #2: Type: text/html, Size: 1714 bytes --]

Re: Best practice for files copied directly into sysroot?

[Trevor Woerner]

On 12/04/15 09:03, Bryan Evenson wrote:
> Although I am having issues with the license; I haven't figured out where to put it and what to set LIC_FILES_CHKSUM to in the recipe so that bitbake can find the license.  If anyone has a good answer for that, let me know.

You could try:
    LICENSE = "CLOSED"

then you don't need a checksum line.

Re: Best practice for files copied directly into sysroot?

[Paul Eggleton]

On Mon, 07 Dec 2015 12:48:29 Trevor Woerner wrote:
> On 12/04/15 09:03, Bryan Evenson wrote:
> > Although I am having issues with the license; I haven't figured out where
> > to put it and what to set LIC_FILES_CHKSUM to in the recipe so that
> > bitbake can find the license.  If anyone has a good answer for that, let
> > me know.
> You could try:
>     LICENSE = "CLOSED"
> 
> then you don't need a checksum line.

This is legitimate of course only when the source is in fact closed (and even 
then, there may be redistribution conditions that you might want to track).

There are several more appropriate ways to solve this issue:

1) The proper way is to point to a file in the fetched/unpacked sources. To do 
this, LIC_FILES_CHKSUM should specify one or more paths relative to ${S} (i.e. 
where the sources are unpacked to) each prefixed by file:// .

2) If there is no file in the sources describing the license, then add one in 
SRC_URI, or point to a common one in ${COMMON_LICENSE_DIR} if appropriate. 
These are then copied into ${WORKDIR} automatically and can be referred to in 
LIC_FILES_CHKSUM from there.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Best practice for files copied directly into sysroot?

[Bryan Evenson]

Paul,

> -----Original Message-----
> From: Paul Eggleton [mailto:paul.eggleton@linux.intel.com]
> Sent: Monday, December 07, 2015 2:50 PM
> To: Bryan Evenson <bevenson@melinkcorp.com>
> Cc: yocto@yoctoproject.org; Trevor Woerner <twoerner@gmail.com>;
> Michael Habibi <mikehabibi@gmail.com>
> Subject: Re: [yocto] Best practice for files copied directly into sysroot?
> 
> On Mon, 07 Dec 2015 12:48:29 Trevor Woerner wrote:
> > On 12/04/15 09:03, Bryan Evenson wrote:
> > > Although I am having issues with the license; I haven't figured out where
> > > to put it and what to set LIC_FILES_CHKSUM to in the recipe so that
> > > bitbake can find the license.  If anyone has a good answer for that, let
> > > me know.
> > You could try:
> >     LICENSE = "CLOSED"
> >
> > then you don't need a checksum line.
> 
> This is legitimate of course only when the source is in fact closed (and even
> then, there may be redistribution conditions that you might want to track).
> 
> There are several more appropriate ways to solve this issue:
> 
> 1) The proper way is to point to a file in the fetched/unpacked sources. To do
> this, LIC_FILES_CHKSUM should specify one or more paths relative to ${S}
> (i.e.
> where the sources are unpacked to) each prefixed by file:// .
> 

Ah, now I see the example in the manual: "Specifying the LIC_FILES_CHKSUM Variable".  In my case I have the license as a separate file in my files/ directory for the recipe. So in my case I now set:

LIC_FILES_CHECKSUM = "file://${WORKDIR}/license.txt;md5=blahblah"
SRC_URI = "file://${BP}.tar.gz \
    file://license.txt \
    "

Now the do_license step is finding the license in the WORKDIR and I no longer have any warnings about the license.

Thanks!
Bryan

> 2) If there is no file in the sources describing the license, then add one in
> SRC_URI, or point to a common one in ${COMMON_LICENSE_DIR} if
> appropriate.
> These are then copied into ${WORKDIR} automatically and can be referred to
> in
> LIC_FILES_CHKSUM from there.
> 
> Cheers,
> Paul
> 
> --
> 
> Paul Eggleton
> Intel Open Source Technology Centre