My stuff is missing from rootfs

My stuff is missing from rootfs

[Paul D. DeRocco]

I've done exactly this in a different Yocto-based project, and it worked.
Now I'm trying to do the same thing in a Gumstix build, and it's not
working. I have a dumb little recipe that merely copies some files into
particlar places in the rootfs. It adds a systemd service unit, as well as
.bashrc and .inputrc to /home/root.

The build logs show the recipe being processed, including the do_install
task which copies the files. No errors are produced. If I rummage through
build/tmp/work, I can find the fragment of the rootfs containing the
/home/root and /etc/systemd/system directories with my files in them. Yet no
matter what I try, these things never wind up in the final rootfs.

I've tried clean and cleansstate on the recipe, as well as on my top-level
recipe. I've bumped PR from r0 to r1. It dutifully reprocesses my recipe,
with no errors, and I end up with a perfectly functioning rootfs without
these particular files.

This is a slightly modified version of gumstix-console-image. I believe it's
based on Danny, as the gumstix Dylan stuff is still a work in progress.

What could conceivably be wrong?

-- 

Ciao,               Paul D. DeRocco
Paul                mailto:pderocco@ix.netcom.com 
 

Re: My stuff is missing from rootfs

[Saul Wold]

On 08/15/2013 11:37 AM, Paul D. DeRocco wrote:
> I've done exactly this in a different Yocto-based project, and it worked.
> Now I'm trying to do the same thing in a Gumstix build, and it's not
> working. I have a dumb little recipe that merely copies some files into
> particlar places in the rootfs. It adds a systemd service unit, as well as
> .bashrc and .inputrc to /home/root.
>
> The build logs show the recipe being processed, including the do_install
> task which copies the files. No errors are produced. If I rummage through
> build/tmp/work, I can find the fragment of the rootfs containing the
> /home/root and /etc/systemd/system directories with my files in them. Yet no
> matter what I try, these things never wind up in the final rootfs.
>
> I've tried clean and cleansstate on the recipe, as well as on my top-level
> recipe. I've bumped PR from r0 to r1. It dutifully reprocesses my recipe,
> with no errors, and I end up with a perfectly functioning rootfs without
> these particular files.
>
> This is a slightly modified version of gumstix-console-image. I believe it's
> based on Danny, as the gumstix Dylan stuff is still a work in progress.
>
> What could conceivably be wrong?
>
Where do you add your recipe's generated packages to the image, this 
could be in your custom image with an RDEPENDS or via something in 
local.conf like CORE_IMAGE_EXTRA_INSTALL_append = " <packagename>".

Do you have other recipes that DEPEND or RDEPEND on your recipe?

That might point you in the right direction.

Sau!

Re: My stuff is missing from rootfs

[Paul D. DeRocco]

[-- Attachment #1: Type: text/plain, Size: 2256 bytes --]

> From: Saul Wold
> 
> > On 08/15/2013 11:37 AM, Paul D. DeRocco wrote:
> > I've done exactly this in a different Yocto-based project, 
> > and it worked.
> > Now I'm trying to do the same thing in a Gumstix build, and it's not
> > working. I have a dumb little recipe that merely copies 
> > some files into
> > particlar places in the rootfs. It adds a systemd service 
> > unit, as well as
> > .bashrc and .inputrc to /home/root.
> >
> > The build logs show the recipe being processed, including 
> > the do_install
> > task which copies the files. No errors are produced. If I 
> > rummage through
> > build/tmp/work, I can find the fragment of the rootfs containing the
> > /home/root and /etc/systemd/system directories with my 
> > files in them. Yet no
> > matter what I try, these things never wind up in the final rootfs.
> >
> > I've tried clean and cleansstate on the recipe, as well as 
> > on my top-level
> > recipe. I've bumped PR from r0 to r1. It dutifully 
> > reprocesses my recipe,
> > with no errors, and I end up with a perfectly functioning 
> > rootfs without
> > these particular files.
> >
> > This is a slightly modified version of 
> > gumstix-console-image. I believe it's
> > based on Danny, as the gumstix Dylan stuff is still a work 
> > in progress.
> >
> > What could conceivably be wrong?
> >
> Where do you add your recipe's generated packages to the image, this 
> could be in your custom image with an RDEPENDS or via something in 
> local.conf like CORE_IMAGE_EXTRA_INSTALL_append = " <packagename>".
> 
> Do you have other recipes that DEPEND or RDEPEND on your recipe?
> 
> That might point you in the right direction.

My top level recipe uses IMAGE_INSTALL to add a bunch of packages, including
one whose name matches the name of the recipe that's being processed but
whose output is being ignored. This is exactly what I did in a different
Yocto project, to get a similar recipe to install some similar files, and it
all worked fine. 

I've attached the top level recipe and the problematic one, only changing
the project name to "foo" for proprietary reasons.

-- 

Ciao,               Paul D. DeRocco
Paul                mailto:pderocco@ix.netcom.com 

[-- Attachment #2: gumstix-foo-pygtk-image.bb --]
[-- Type: application/octet-stream, Size: 1263 bytes --]

DESCRIPTION = "The Gumstix console image, with more Python, Samba, X"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
PR = "r0"

require recipes-images/gumstix/gumstix-console-image.bb

IMAGE_INSTALL += "                  \
    packagegroup-core-x11-base      \
    python-numeric                  \
    python-numpy                    \
    python-pyserial                 \
    python-pygtk                    \
    python-pyqt                     \
    python-matplotlib               \
    python-pyalsaaudio              \
    foo                             \
    samba                           \
    tzdata                          \
    xinput-calibrator               \
    "

# This sets the timezone to US Pacific time
set_local_timezone() {
    ln -sf /usr/share/zoneinfo/America/Los_Angeles ${IMAGE_ROOTFS}/etc/localtime
}

# This sets the root password to "root" in the /etc/shadow file.
set_root_password () {
	sed 's%^root:[^:]*:%root:trHg5jf38krOE:%' < ${IMAGE_ROOTFS}/etc/shadow >${IMAGE_ROOTFS}/etc/shadow.new
	mv ${IMAGE_ROOTFS}/etc/shadow.new ${IMAGE_ROOTFS}/etc/shadow
} 

ROOTFS_POSTPROCESS_COMMAND =+ "set_local_timezone ; set_root_password ; "

[-- Attachment #3: foo_1.0.bb --]
[-- Type: application/octet-stream, Size: 1559 bytes --]

DESCRIPTION = "Foo project odds and ends"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
PR = "r1"

SRC_URI = "                         \
    file://dummyxsession.service    \
    file://.bashrc                  \
    file://.inputrc                 \
    "

S = "${WORKDIR}"

FILES_${PN} += "                                                                \
    ${sysconfdir}/                                                              \
    ${sysconfdir}/systemd/                                                      \
    ${sysconfdir}/systemd/system/                                               \
    ${sysconfdir}/systemd/system/dummyxsession.service                          \
    ${sysconfdir}/systemd/system/multi-user.target.wants/                       \
    ${sysconfdir}/systemd/system/multi-user.target.wants/dummyxsession.service  \
    /home/root/.bashrc                                                          \
    /home/root/.inputrc                                                         \
    "

# This installs the statically defined files
do_install() {
    install -d ${D}${sysconfdir}/systemd/system/multi-user.target.wants
    install -m 0755 dummyxsession.service ${D}${sysconfdir}/systemd/system
    ln -s ${sysconfdir}/systemd/system/dummyxsession.service ${D}${sysconfdir}/systemd/system/multi-user.target.wants
    install -d ${D}/home/root
    install -m 0755 .bashrc ${D}/home/root
    install -m 0755 .inputrc ${D}/home/root
}

Re: My stuff is missing from rootfs

[Saul Wold]

On 08/15/2013 12:30 PM, Paul D. DeRocco wrote:
>> From: Saul Wold
>>
>>> On 08/15/2013 11:37 AM, Paul D. DeRocco wrote:
>>> I've done exactly this in a different Yocto-based project,
>>> and it worked.
>>> Now I'm trying to do the same thing in a Gumstix build, and it's not
>>> working. I have a dumb little recipe that merely copies
>>> some files into
>>> particlar places in the rootfs. It adds a systemd service
>>> unit, as well as
>>> .bashrc and .inputrc to /home/root.
>>>
>>> The build logs show the recipe being processed, including
>>> the do_install
>>> task which copies the files. No errors are produced. If I
>>> rummage through
>>> build/tmp/work, I can find the fragment of the rootfs containing the
>>> /home/root and /etc/systemd/system directories with my
>>> files in them. Yet no
>>> matter what I try, these things never wind up in the final rootfs.
>>>
>>> I've tried clean and cleansstate on the recipe, as well as
>>> on my top-level
>>> recipe. I've bumped PR from r0 to r1. It dutifully
>>> reprocesses my recipe,
>>> with no errors, and I end up with a perfectly functioning
>>> rootfs without
>>> these particular files.
>>>
>>> This is a slightly modified version of
>>> gumstix-console-image. I believe it's
>>> based on Danny, as the gumstix Dylan stuff is still a work
>>> in progress.
>>>
>>> What could conceivably be wrong?
>>>
>> Where do you add your recipe's generated packages to the image, this
>> could be in your custom image with an RDEPENDS or via something in
>> local.conf like CORE_IMAGE_EXTRA_INSTALL_append = " <packagename>".
>>
>> Do you have other recipes that DEPEND or RDEPEND on your recipe?
>>
>> That might point you in the right direction.
>
> My top level recipe uses IMAGE_INSTALL to add a bunch of packages, including
> one whose name matches the name of the recipe that's being processed but
> whose output is being ignored. This is exactly what I did in a different
> Yocto project, to get a similar recipe to install some similar files, and it
> all worked fine.
>
> I've attached the top level recipe and the problematic one, only changing
> the project name to "foo" for proprietary reasons.
>
Interesting, did you verify that the files are in the 
tmp/work/.../foo/packages-split/foo directory.  You can also look in the 
tmp/work/.../gumstix-foo-pyygtk-image/1.0-r0/installed_pkgs.txt file to 
ensure your foo package is there.

You can also look in the image temp dir for the log.do_rootfs and see if 
there are any issues in it or it's missing your package.

Note in this case recipename == packagename, this is not always the case.

Home this helps.

	Sau!

Re: My stuff is missing from rootfs

[Paul D. DeRocco]

> From: Saul Wold
> 
> > My top level recipe uses IMAGE_INSTALL to add a bunch of 
> > packages, including
> > one whose name matches the name of the recipe that's being 
> > processed but
> > whose output is being ignored. This is exactly what I did 
> > in a different
> > Yocto project, to get a similar recipe to install some 
> > similar files, and it
> > all worked fine.
> >
> > I've attached the top level recipe and the problematic one, 
> > only changing
> > the project name to "foo" for proprietary reasons.
> >
> Interesting, did you verify that the files are in the 
> tmp/work/.../foo/packages-split/foo directory.  You can also 
> look in the 
> tmp/work/.../gumstix-foo-pyygtk-image/1.0-r0/installed_pkgs.tx
> t file to 
> ensure your foo package is there.
> 
> You can also look in the image temp dir for the log.do_rootfs 
> and see if 
> there are any issues in it or it's missing your package.

The files are in tmp/work/.../foo_1.0-r1/packages-split/foo, and also in
tmp/work/.../foo_1.0-r1/image, and the package is listed in that text
file. And the logs show no errors.

This smells like one of those situations where nuking tmp and rebuilding
will fix it, and we'll never know what was wrong. I'll let you know if
that fixes it.

-- 

Ciao,               Paul D. DeRocco
Paul                mailto:pderocco@ix.netcom.com 

Re: My stuff is missing from rootfs

[Paul Eggleton]

On Thursday 15 August 2013 13:01:54 Paul D. DeRocco wrote:
> This smells like one of those situations where nuking tmp and rebuilding
> will fix it, and we'll never know what was wrong. I'll let you know if
> that fixes it.

If you keep on doing this we'll never figure out what the problem is. Wiping 
out tmp just removes any way we might have to diagnose the issue.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: My stuff is missing from rootfs

[Paul D. DeRocco]

> From: Paul Eggleton
> 
> On Thursday 15 August 2013 13:01:54 Paul D. DeRocco wrote:
> > This smells like one of those situations where nuking tmp 
> > and rebuilding
> > will fix it, and we'll never know what was wrong. I'll let 
> > you know if
> > that fixes it.
> 
> If you keep on doing this we'll never figure out what the 
> problem is. Wiping 
> out tmp just removes any way we might have to diagnose the issue.

I didn't wipe it out, I renamed it to something else, so I can switch back
to it if I need to. I certainly don't have the faintest idea about how to
diagnose it, beyond the naive steps I've already taken. And I need to get
back to my real job, which is writing apps for this system, instead of
building the distro. If there's anything you can think of that I should
try, I'd be happy to do so.

That said, rebuilding tmp (preserving downloads and sstate-cache) did
indeed fix that problem, but it was replaced with another. Now, do_rootfs
is failing, complaining that it "cannot satisfy the following dependencies
for samba: libpam (>= 1.1.6)". Yet there's libpam, version 1.1.6, right
where it's always been in the metadata. Doing a clean and a cleansstate on
libpam didn't help. Now I'm trying rebuilding samba, but that's a huge
package so it will take a while.

The only way I can get any real work done at the moment is to make manual
changes to the last good build I had.

-- 

Ciao,               Paul D. DeRocco
Paul                mailto:pderocco@ix.netcom.com 

Re: My stuff is missing from rootfs

[Mark Hatle]

On 8/15/13 2:30 PM, Paul D. DeRocco wrote:
>> From: Saul Wold
>>
>>> On 08/15/2013 11:37 AM, Paul D. DeRocco wrote:
>>> I've done exactly this in a different Yocto-based project,
>>> and it worked.
>>> Now I'm trying to do the same thing in a Gumstix build, and it's not
>>> working. I have a dumb little recipe that merely copies
>>> some files into
>>> particlar places in the rootfs. It adds a systemd service
>>> unit, as well as
>>> .bashrc and .inputrc to /home/root.
>>>
>>> The build logs show the recipe being processed, including
>>> the do_install
>>> task which copies the files. No errors are produced. If I
>>> rummage through
>>> build/tmp/work, I can find the fragment of the rootfs containing the
>>> /home/root and /etc/systemd/system directories with my
>>> files in them. Yet no
>>> matter what I try, these things never wind up in the final rootfs.
>>>
>>> I've tried clean and cleansstate on the recipe, as well as
>>> on my top-level
>>> recipe. I've bumped PR from r0 to r1. It dutifully
>>> reprocesses my recipe,
>>> with no errors, and I end up with a perfectly functioning
>>> rootfs without
>>> these particular files.
>>>
>>> This is a slightly modified version of
>>> gumstix-console-image. I believe it's
>>> based on Danny, as the gumstix Dylan stuff is still a work
>>> in progress.
>>>
>>> What could conceivably be wrong?
>>>
>> Where do you add your recipe's generated packages to the image, this
>> could be in your custom image with an RDEPENDS or via something in
>> local.conf like CORE_IMAGE_EXTRA_INSTALL_append = " <packagename>".
>>
>> Do you have other recipes that DEPEND or RDEPEND on your recipe?
>>
>> That might point you in the right direction.
>
> My top level recipe uses IMAGE_INSTALL to add a bunch of packages, including
> one whose name matches the name of the recipe that's being processed but
> whose output is being ignored. This is exactly what I did in a different
> Yocto project, to get a similar recipe to install some similar files, and it
> all worked fine.
>
> I've attached the top level recipe and the problematic one, only changing
> the project name to "foo" for proprietary reasons.
>

A simple way to diagnose if your package is even in the install list is to do 
bitbake -e <image>, then scan the output for "PACKAGE_INSTALL".  If your package 
is not listed there, then something has either cleared your configuration or you 
have a typo.

IMAGE_INSTALL_append = " my_package" should work, and generally won't be cleared 
by a recipe.

(Note you should modify IMAGE_INSTALL, which is transformed by the system into 
PACKAGE_INSTALL... modifying PACKAGE_INSTALL can lead to problems.)

--Mark

>
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>

Re: My stuff is missing from rootfs

[Paul D. DeRocco]

> From: Mark Hatle
> 
> A simple way to diagnose if your package is even in the 
> install list is to do 
> bitbake -e <image>, then scan the output for 
> "PACKAGE_INSTALL".  If your package 
> is not listed there, then something has either cleared your 
> configuration or you 
> have a typo.
> 
> IMAGE_INSTALL_append = " my_package" should work, and 
> generally won't be cleared 
> by a recipe.

It's there, and it's all working now. I've had things break in odd ways
before, and recover when I rebuilt. This time it took a couple of tries.

What makes it frustrating is that I'm building on a wimpy Atom system.
I've been on the verge of buying a killer system just to do builds, but I
keep thinking, maybe I'll only need to do this another dozen or so times
and then I'll be done, in which case it wouldn't be a good investment.

> (Note you should modify IMAGE_INSTALL, which is transformed 
> by the system into 
> PACKAGE_INSTALL... modifying PACKAGE_INSTALL can lead to problems.)

I don't think that all the various ways to append stuff will ever make
sense to me. Currently, I'm using IMAGE_INSTALL += "..." in my top level
recipe, and it works. I don't know what IMAGE_INSTALL_append does
differently.

-- 

Ciao,               Paul D. DeRocco
Paul                mailto:pderocco@ix.netcom.com 

Re: My stuff is missing from rootfs

[Paul D. DeRocco]

> From: yocto-bounces@yoctoproject.org 
> 
> That said, rebuilding tmp (preserving downloads and sstate-cache) did
> indeed fix that problem, but it was replaced with another. 
> Now, do_rootfs
> is failing, complaining that it "cannot satisfy the following 
> dependencies
> for samba: libpam (>= 1.1.6)". Yet there's libpam, version 
> 1.1.6, right
> where it's always been in the metadata. Doing a clean and a 
> cleansstate on
> libpam didn't help. Now I'm trying rebuilding samba, but that's a huge
> package so it will take a while.

Not surprisingly, rebuilding Samba cleared out that spurious issue. So I'm
back to normal.

-- 

Ciao,               Paul D. DeRocco
Paul                mailto:pderocco@ix.netcom.com 

Re: My stuff is missing from rootfs

[Paul Eggleton]

On Thursday 15 August 2013 15:38:37 Paul D. DeRocco wrote:
> > On Thursday 15 August 2013 13:01:54 Paul D. DeRocco wrote:
> > > This smells like one of those situations where nuking tmp
> > > and rebuilding will fix it, and we'll never know what was wrong. I'll let
> > > you know if that fixes it.
> > 
> > If you keep on doing this we'll never figure out what the
> > problem is. Wiping out tmp just removes any way we might have to diagnose
> > the issue.
> 
> I didn't wipe it out, I renamed it to something else, so I can switch back
> to it if I need to. 

Apologies, I just assumed that's what you meant by "nuking". If there's a bug 
or otherwise undesirable behaviour that is causing this problem I'd really 
like to figure out what it is so we can fix it for everyone.

> I certainly don't have the faintest idea about how to
> diagnose it, beyond the naive steps I've already taken. And I need to get
> back to my real job, which is writing apps for this system, instead of
> building the distro. If there's anything you can think of that I should
> try, I'd be happy to do so.

You didn't mention in your reply to Saul whether the foo package was mentioned 
in log.do_rootfs or installed_pkgs.txt files in your old tmpdir; was it?

> That said, rebuilding tmp (preserving downloads and sstate-cache) did
> indeed fix that problem, but it was replaced with another. Now, do_rootfs
> is failing, complaining that it "cannot satisfy the following dependencies
> for samba: libpam (>= 1.1.6)". Yet there's libpam, version 1.1.6, right
> where it's always been in the metadata. Doing a clean and a cleansstate on
> libpam didn't help. 

Does samba.inc used by the samba recipe you are building have a PACKAGECONFIG 
line referring to pam? I think that was added recently in meta-oe master (and 
shortly to be merged into the meta-oe dylan branch). Without it there will be 
a floating dependency on pam, which may account for this latter issue.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: My stuff is missing from rootfs

[Mark Hatle]

On 8/15/13 7:20 PM, Paul D. DeRocco wrote:
>> From: Mark Hatle
>>
>> A simple way to diagnose if your package is even in the
>> install list is to do
>> bitbake -e <image>, then scan the output for
>> "PACKAGE_INSTALL".  If your package
>> is not listed there, then something has either cleared your
>> configuration or you
>> have a typo.
>>
>> IMAGE_INSTALL_append = " my_package" should work, and
>> generally won't be cleared
>> by a recipe.
>
> It's there, and it's all working now. I've had things break in odd ways
> before, and recover when I rebuilt. This time it took a couple of tries.
>
> What makes it frustrating is that I'm building on a wimpy Atom system.
> I've been on the verge of buying a killer system just to do builds, but I
> keep thinking, maybe I'll only need to do this another dozen or so times
> and then I'll be done, in which case it wouldn't be a good investment.
>
>> (Note you should modify IMAGE_INSTALL, which is transformed
>> by the system into
>> PACKAGE_INSTALL... modifying PACKAGE_INSTALL can lead to problems.)
>
> I don't think that all the various ways to append stuff will ever make
> sense to me. Currently, I'm using IMAGE_INSTALL += "..." in my top level
> recipe, and it works. I don't know what IMAGE_INSTALL_append does
> differently.
>

Difference between IMAGE_INSTALL += and IMAGE_INSTALL_append is when the process 
happens.

IMAGE_INSTALL += happens immediately, and if something later in the resolution 
does "IMAGE_INSTALL = " then you can lose the value.

IMAGE_INSTALL_append happens at the very end of variable resolution.. i.e.:

IMAGE_INSTALL_append = " mypkg"
IMAGE_INSTALL = "foo"
IMAGE_INSTALL += "bar"
IMAGE_INSTALL = "foobar"

Your end result will be "foobar mypkg"...

(USUALLY the '=' in the example above actually indicates a bug of some kind.. 
but it does happen.)

--Mark

Re: My stuff is missing from rootfs

[Paul D. DeRocco]

> From: Paul Eggleton
> 
> You didn't mention in your reply to Saul whether the foo 
> package was mentioned 
> in log.do_rootfs or installed_pkgs.txt files in your old 
> tmpdir; was it?

It's in both places. Yet NONE of the various files that were supposed to
be part of that recipe ended up in the rootfs, after several tries, even
though the logs showed that recipe being built.

I think the problem may have been caused by once hitting Ctrl-C at the
beginning of do_rootfs, because I remembered that I had to tweak one of
the files, and I didn't want to wait the fifteen minutes for that to
complete. (I'm running on an Atom.) As it happens, Ctrl-C just sets a flag
that interrupts the process after the task completes, so it didn't really
help, but I think that's when the problem started. Maybe that's a clue.

> > That said, rebuilding tmp (preserving downloads and 
> > sstate-cache) did
> > indeed fix that problem, but it was replaced with another. 
> > Now, do_rootfs
> > is failing, complaining that it "cannot satisfy the 
> > following dependencies
> > for samba: libpam (>= 1.1.6)". Yet there's libpam, version 
> > 1.1.6, right
> > where it's always been in the metadata. Doing a clean and a 
> > cleansstate on
> > libpam didn't help. 
> 
> Does samba.inc used by the samba recipe you are building have 
> a PACKAGECONFIG 
> line referring to pam? I think that was added recently in 
> meta-oe master (and 
> shortly to be merged into the meta-oe dylan branch). Without 
> it there will be 
> a floating dependency on pam, which may account for this latter issue.

No, it's not mentioned in samba.inc, samba-basic.inc or in the recipe
itself.

And cleaning samba and rebuilding fixed the problem. So I suspect what's
been done in the Dylan branch has dealt with that problem, and for now
everything in the Danny branch works fine as long as things get done in
the normal order.

-- 

Ciao,               Paul D. DeRocco
Paul                mailto:pderocco@ix.netcom.com