Switching between multiple DISTROs without "contamination"

Switching between multiple DISTROs without "contamination"

[Nicolas Jeker]

Hi all,

I'm currently using an additional layer and image to differentiate
between a release and development build (enabling NFS, SSH, root login,
etc.). To create a development build, I manually add the layer to
bblayers.conf. This works quite well, but feels a bit clumsy to
integrate into a CI/CD pipeline.

Per these past discussions here [1][2], I'm now trying to migrate to
multiple DISTROs, something like "mydistro" and "mydistro-dev".

While migrating some of the changes, I discovered that I run into
caching(?) issues. I have a recipe for an internal application and want
to include additional systemd service files in the development image.

What I did was this:

Added "application-dbg.service" to recipes-internal/application/files

Adapted application.bb recipe:

SRC_URI:append:mydistro-dev = " file://application-dbg.service"

do_install {
    # ...snip...
    # systemd service
    install -d ${D}${systemd_system_unitdir}
    install -m 0644 ${WORKDIR}/application.service
${D}${systemd_system_unitdir}
}

do_install:append:mydistro-dev() {
    # debug systemd services
    install -d ${D}${systemd_system_unitdir}
    install -m 0644 ${WORKDIR}/application-dbg.service
${D}${systemd_system_unitdir}
}


When I run "DISTRO=mydistro-dev bitbake application" followed by
"DISTRO=mydistro bitbake application", the debug service file is still
present in the package. This seems to be caused by the "image"
directory in the recipe WORKDIR not being cleaned between subsequent
do_install runs. Is this expected behaviour? What's the best solution?

Kind regards,
Nicolas

[1]:
https://lore.kernel.org/yocto/CAH9dsRiArf_9GgQS4hCg5=J_Jk6cd3eiGaOiQd788+iSLTuU+g@mail.gmail.com/
[2]:
https://lore.kernel.org/yocto/VI1PR0602MB3549F83AC93A53785DE48677D3FD9@VI1PR0602MB3549.eurprd06.prod.outlook.com/

AW: [yocto] Switching between multiple DISTROs without "contamination"

[Martin Weber]

Hi Nicolas,

I stumbled over this issue as well. Think of it this way, the package gets built in one environment and then it's built. Now ask to build it in another environment -- but it's built already, why do It again? Yes, in theory it would contain a different set of files / services / ... but it doesn't differentiate, because it's package A in both environments, not package A and package B. So in a sense, even though your (and my) expectations differed, I believe this works as intended.

I use to solve this issue by creating an A.inc file containing the shared stuff, a A.bb recipe building for the "default" environment, and A-XX.bb recpies for building the "XX" environment(s). Your A-XX package will be distinct from A. Now in your image setup, you can add A-XX to the XX sort of image. But keep in mind that your image is also just another recipe in the end. I believe building both in the same build dir with different distros again asks for surprises. 

So, note that sharing the build dir still isn't a good idea. Richard said this is the use-case for the shared sstate so that multiple environments can share build results quickly and efficiently, and thus you wouldn't practically want a shared build dir (once you've setup sstate sharing), because you still get the benefits of not rebuilding unchanged packages, but don't step on your toes if you want to be odd one day and even another.

Best regards,

Martin Weber
Research & Development - Embedded Software Engineer

B&R Industrial Automation GmbH, B&R Straße 1, 5142 Eggelsberg, Austria, www.br-automation.com
Phone +43 7748 6586 - 0

-----Ursprüngliche Nachricht-----
Von: yocto@lists.yoctoproject.org <yocto@lists.yoctoproject.org> Im Auftrag von Nicolas Jeker via lists.yoctoproject.org
Gesendet: Dienstag, 12. Juli 2022 15:38
An: yocto@lists.yoctoproject.org
Betreff: [yocto] Switching between multiple DISTROs without "contamination"

BeSecure!       This email comes from outside of ABB. Make sure you verify the sender before clicking any links or downloading/opening attachments.
If this email looks suspicious, report it by clicking 'Report Phishing' button in Outlook or raising a ticket on MyIS.


Hi all,

I'm currently using an additional layer and image to differentiate
between a release and development build (enabling NFS, SSH, root login,
etc.). To create a development build, I manually add the layer to
bblayers.conf. This works quite well, but feels a bit clumsy to
integrate into a CI/CD pipeline.

Per these past discussions here [1][2], I'm now trying to migrate to
multiple DISTROs, something like "mydistro" and "mydistro-dev".

While migrating some of the changes, I discovered that I run into
caching(?) issues. I have a recipe for an internal application and want
to include additional systemd service files in the development image.

What I did was this:

Added "application-dbg.service" to recipes-internal/application/files

Adapted application.bb recipe:

SRC_URI:append:mydistro-dev = " file://application-dbg.service"

do_install {
    # ...snip...
    # systemd service
    install -d ${D}${systemd_system_unitdir}
    install -m 0644 ${WORKDIR}/application.service
${D}${systemd_system_unitdir}
}

do_install:append:mydistro-dev() {
    # debug systemd services
    install -d ${D}${systemd_system_unitdir}
    install -m 0644 ${WORKDIR}/application-dbg.service
${D}${systemd_system_unitdir}
}


When I run "DISTRO=mydistro-dev bitbake application" followed by
"DISTRO=mydistro bitbake application", the debug service file is still
present in the package. This seems to be caused by the "image"
directory in the recipe WORKDIR not being cleaned between subsequent
do_install runs. Is this expected behaviour? What's the best solution?

Kind regards,
Nicolas

[1]:
https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.kernel.org%2Fyocto%2FCAH9dsRiArf_9GgQS4hCg5%3DJ_Jk6cd3eiGaOiQd788%2BiSLTuU%2Bg%40mail.gmail.com%2F&amp;data=05%7C01%7Cmartin.weber%40br-automation.com%7Cb4c2ffc9851847b6d7c208da640bb064%7C372ee9e09ce04033a64ac07073a91ecd%7C0%7C0%7C637932298663056785%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=STA1V1w3ExlYrXkoYb7DDtkAWlkAeKhtFAkjjkUlsoQ%3D&amp;reserved=0
[2]:
https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.kernel.org%2Fyocto%2FVI1PR0602MB3549F83AC93A53785DE48677D3FD9%40VI1PR0602MB3549.eurprd06.prod.outlook.com%2F&amp;data=05%7C01%7Cmartin.weber%40br-automation.com%7Cb4c2ffc9851847b6d7c208da640bb064%7C372ee9e09ce04033a64ac07073a91ecd%7C0%7C0%7C637932298663056785%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=dT4DkNsm0f0L%2BrCgF7f1Q8XBAN1j2bLYZ0%2FkbKr6nuo%3D&amp;reserved=0

Re: [yocto] Switching between multiple DISTROs without "contamination"

[Mike Looijmans]

Quick answer: Don't build multiple distros in one build directory.

You might get away with setting TMPDIR = "tmp-${DISTRO}" to give each 
its own.

But I'd rather advice to set up two separate builds and just point the 
downloads and sstate-cache to the same location. It'll be faster than 
the TMPDIR option.

Or figure out how to put the difference in the IMAGE only. Then you can 
just build both images (in parallel, woot), which is faster, more 
convenient and saves on diskspace.

What I often do is have my-application.bb generate a 
my-application-utils package that only gets installed in the "dev" image 
but not in the production one, which only installs "my-application".

You could also create a "my-application-dev.bb" recipe that includes 
my-application.bb and just changes what it needs to be different.




Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Products B.V.
Materiaalweg 4, 5681 RJ Best
The Netherlands

T: +31 (0) 499 33 69 69
E: mike.looijmans@topicproducts.com
W: www.topic.nl

Please consider the environment before printing this e-mail
On 12-07-2022 15:37, Nicolas Jeker via lists.yoctoproject.org wrote:
> Hi all,
>
> I'm currently using an additional layer and image to differentiate
> between a release and development build (enabling NFS, SSH, root login,
> etc.). To create a development build, I manually add the layer to
> bblayers.conf. This works quite well, but feels a bit clumsy to
> integrate into a CI/CD pipeline.
>
> Per these past discussions here [1][2], I'm now trying to migrate to
> multiple DISTROs, something like "mydistro" and "mydistro-dev".
>
> While migrating some of the changes, I discovered that I run into
> caching(?) issues. I have a recipe for an internal application and want
> to include additional systemd service files in the development image.
>
> What I did was this:
>
> Added "application-dbg.service" to recipes-internal/application/files
>
> Adapted application.bb recipe:
>
> SRC_URI:append:mydistro-dev = " file://application-dbg.service"
>
> do_install {
>      # ...snip...
>      # systemd service
>      install -d ${D}${systemd_system_unitdir}
>      install -m 0644 ${WORKDIR}/application.service
> ${D}${systemd_system_unitdir}
> }
>
> do_install:append:mydistro-dev() {
>      # debug systemd services
>      install -d ${D}${systemd_system_unitdir}
>      install -m 0644 ${WORKDIR}/application-dbg.service
> ${D}${systemd_system_unitdir}
> }
>
>
> When I run "DISTRO=mydistro-dev bitbake application" followed by
> "DISTRO=mydistro bitbake application", the debug service file is still
> present in the package. This seems to be caused by the "image"
> directory in the recipe WORKDIR not being cleaned between subsequent
> do_install runs. Is this expected behaviour? What's the best solution?
>
> Kind regards,
> Nicolas
>
> [1]:
> https://lore.kernel.org/yocto/CAH9dsRiArf_9GgQS4hCg5=J_Jk6cd3eiGaOiQd788+iSLTuU+g@mail.gmail.com/
> [2]:
> https://lore.kernel.org/yocto/VI1PR0602MB3549F83AC93A53785DE48677D3FD9@VI1PR0602MB3549.eurprd06.prod.outlook.com/
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#57508): https://lists.yoctoproject.org/g/yocto/message/57508
> Mute This Topic: https://lists.yoctoproject.org/mt/92332946/3618446
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [mike.looijmans@topic.nl]
> -=-=-=-=-=-=-=-=-=-=-=-
>

-- 
Mike Looijmans

Re: [yocto] Switching between multiple DISTROs without "contamination"

[Khoi Dinh Trinh]

[-- Attachment #1: Type: text/plain, Size: 4481 bytes --]

Thank you Nicolas for asking this question since I will probably run into
this issue soon if not for this email thread. The answers so far have been
very helpful but I just want to clarify a bit more on why doesn't the
package get rebuilt? From my understanding, Yocto should rerun a task when
the signature of the task changes and since do_install has an override on
mydistro-dev, shouldn't the content and thus the signature of do_install
change when switching distro and so Yocto should rerun it? I have a lot of
tasks with override not just on DISTRO but other things like MACHINE or
custom variables so I want to understand the rebuild mechanism as best I
can.

Best,
Khoi Trinh

On Tue, Jul 12, 2022 at 8:05 AM Mike Looijmans <mike.looijmans@topic.nl>
wrote:

> Quick answer: Don't build multiple distros in one build directory.
>
> You might get away with setting TMPDIR = "tmp-${DISTRO}" to give each
> its own.
>
> But I'd rather advice to set up two separate builds and just point the
> downloads and sstate-cache to the same location. It'll be faster than
> the TMPDIR option.
>
> Or figure out how to put the difference in the IMAGE only. Then you can
> just build both images (in parallel, woot), which is faster, more
> convenient and saves on diskspace.
>
> What I often do is have my-application.bb generate a
> my-application-utils package that only gets installed in the "dev" image
> but not in the production one, which only installs "my-application".
>
> You could also create a "my-application-dev.bb" recipe that includes
> my-application.bb and just changes what it needs to be different.
>
>
>
>
> Met vriendelijke groet / kind regards,
>
> Mike Looijmans
> System Expert
>
>
> TOPIC Embedded Products B.V.
> Materiaalweg 4, 5681 RJ Best
> The Netherlands
>
> T: +31 (0) 499 33 69 69
> E: mike.looijmans@topicproducts.com
> W: www.topic.nl
>
> Please consider the environment before printing this e-mail
> On 12-07-2022 15:37, Nicolas Jeker via lists.yoctoproject.org wrote:
> > Hi all,
> >
> > I'm currently using an additional layer and image to differentiate
> > between a release and development build (enabling NFS, SSH, root login,
> > etc.). To create a development build, I manually add the layer to
> > bblayers.conf. This works quite well, but feels a bit clumsy to
> > integrate into a CI/CD pipeline.
> >
> > Per these past discussions here [1][2], I'm now trying to migrate to
> > multiple DISTROs, something like "mydistro" and "mydistro-dev".
> >
> > While migrating some of the changes, I discovered that I run into
> > caching(?) issues. I have a recipe for an internal application and want
> > to include additional systemd service files in the development image.
> >
> > What I did was this:
> >
> > Added "application-dbg.service" to recipes-internal/application/files
> >
> > Adapted application.bb recipe:
> >
> > SRC_URI:append:mydistro-dev = " file://application-dbg.service"
> >
> > do_install {
> >      # ...snip...
> >      # systemd service
> >      install -d ${D}${systemd_system_unitdir}
> >      install -m 0644 ${WORKDIR}/application.service
> > ${D}${systemd_system_unitdir}
> > }
> >
> > do_install:append:mydistro-dev() {
> >      # debug systemd services
> >      install -d ${D}${systemd_system_unitdir}
> >      install -m 0644 ${WORKDIR}/application-dbg.service
> > ${D}${systemd_system_unitdir}
> > }
> >
> >
> > When I run "DISTRO=mydistro-dev bitbake application" followed by
> > "DISTRO=mydistro bitbake application", the debug service file is still
> > present in the package. This seems to be caused by the "image"
> > directory in the recipe WORKDIR not being cleaned between subsequent
> > do_install runs. Is this expected behaviour? What's the best solution?
> >
> > Kind regards,
> > Nicolas
> >
> > [1]:
> >
> https://lore.kernel.org/yocto/CAH9dsRiArf_9GgQS4hCg5=J_Jk6cd3eiGaOiQd788+iSLTuU+g@mail.gmail.com/
> > [2]:
> >
> https://lore.kernel.org/yocto/VI1PR0602MB3549F83AC93A53785DE48677D3FD9@VI1PR0602MB3549.eurprd06.prod.outlook.com/
> >
> >
> >
>
> --
> Mike Looijmans
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#57515):
> https://lists.yoctoproject.org/g/yocto/message/57515
> Mute This Topic: https://lists.yoctoproject.org/mt/92332946/6433300
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [
> khoidinhtrinh@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

-- 
Best,
Khoi Trinh

[-- Attachment #2: Type: text/html, Size: 6814 bytes --]

Re: [yocto] Switching between multiple DISTROs without "contamination"

[Nicolas Jeker]

Thanks Martin and Mike for your explanations and tips.

So, I've done a lot of testing today and it seems I simplified the
example in my first email a bit too much. The example as-is works fine
when switching DISTROs as far as I can tell. The problem only arises
when wildcards are used.

Changing my initial example like this should trigger the behaviour I've
initially described:

SRC_URI:append:mydistro-dev = " file://application-dbg.service"

do_install {
    # ...snip...
    # systemd service
    install -d ${D}${systemd_system_unitdir}
    install -m 0644 ${WORKDIR}/*.service ${D}${systemd_system_unitdir}
}

do_install:append:mydistro-dev() {
    # debug systemd services
    install -d ${D}${systemd_system_unitdir}
    install -m 0644 ${WORKDIR}/application-dbg.service
${D}${systemd_system_unitdir}
}

Notice the *.service in do_install.

From my testing, this is how contamination happens:

1) Build with 'DISTRO=mydistro bitbake application'. All tasks for the
recipe are run and the directories in WORKDIR are populated, including
the "application.service" file.
2) Build with 'DISTRO=mydistro-dev bitbake application'. do_unpack is
rerun and places the additional "application-dbg.service" file in
WORKDIR.
3) Switching back to 'mydistro' will get the recipe from sstate cache,
which works fine.
4) Changing application.bb and rebuilding with 'DISTRO=mydistro bitbake
application' reruns do_install (as expected). This leads to the
packages do_install picking up the additional "application-dbg.service"
file left behind by the invocation in step 2).

Mike, Martin: Do you remember in which cases you encountered problems
when sharing the build directory?

On Tue, 2022-07-12 at 09:15 -0700, Khoi Dinh Trinh wrote:
> Thank you Nicolas for asking this question since I will probably run
> into this issue soon if not for this email thread. The answers so far
> have been very helpful but I just want to clarify a bit more on why
> doesn't the package get rebuilt? From my understanding, Yocto should
> rerun a task when the signature of the task changes and since
> do_install has an override on mydistro-dev, shouldn't the content and
> thus the signature of do_install change when switching distro and so
> Yocto should rerun it? 

As far as I can tell, all the relevant tasks are rerun correctly when
something is changed. Relevant tasks meaning only the tasks that are
actually different.

The specific issue I experienced is due to the WORKDIR not being
cleaned between different task invocations and the recipe (probably
wrongfully) relying on wildcards to gather files, see example above.

> I have a lot of tasks with override not just on DISTRO but other
> things like MACHINE or custom variables so I want to understand the
> rebuild mechanism as best I can.
> 

There's surely someone more knowledgeable here that could clarify the
inner workings of this mechanism a lot better than me.

> Best,
> Khoi Trinh
> 
> On Tue, Jul 12, 2022 at 8:05 AM Mike Looijmans
> <mike.looijmans@topic.nl> wrote:
> > Quick answer: Don't build multiple distros in one build directory.
> > 

It's really a bummer that it's not reliably possible to switch between
DISTROs inside one build directory.

> > You might get away with setting TMPDIR = "tmp-${DISTRO}" to give
> > each 
> > its own.
> > 
> > But I'd rather advice to set up two separate builds and just point
> > the 
> > downloads and sstate-cache to the same location. It'll be faster
> > than 
> > the TMPDIR option.
> > 
> > Or figure out how to put the difference in the IMAGE only. Then you
> > can 
> > just build both images (in parallel, woot), which is faster, more 
> > convenient and saves on diskspace.

As I'm currently working with something close to what you describe, I
think I'll try to stay away from multiple DISTROs if possible and
improve on what I'm already doing.

> > 
> > What I often do is have my-application.bb generate a 
> > my-application-utils package that only gets installed in the "dev"
> > image 
> > but not in the production one, which only installs "my-
> > application".

This is probably what Martin meant with his A.bb and A-xx.bb example.
It's so far one of the best approaches I've seen, thanks.

> > 
> > You could also create a "my-application-dev.bb" recipe that
> > includes 
> > my-application.bb and just changes what it needs to be different.
> > 
> > 
> > 
> > 
> > Met vriendelijke groet / kind regards,
> > 
> > Mike Looijmans
> > System Expert
> > 
> > 
> > TOPIC Embedded Products B.V.
> > Materiaalweg 4, 5681 RJ Best
> > The Netherlands
> > 
> > T: +31 (0) 499 33 69 69
> > E: mike.looijmans@topicproducts.com
> > W: www.topic.nl
> > 
> > Please consider the environment before printing this e-mail
> > On 12-07-2022 15:37, Nicolas Jeker via lists.yoctoproject.org
> > wrote:
> > > Hi all,
> > > 
> > > I'm currently using an additional layer and image to
> > > differentiate
> > > between a release and development build (enabling NFS, SSH, root
> > > login,
> > > etc.). To create a development build, I manually add the layer to
> > > bblayers.conf. This works quite well, but feels a bit clumsy to
> > > integrate into a CI/CD pipeline.
> > > 
> > > Per these past discussions here [1][2], I'm now trying to migrate
> > > to
> > > multiple DISTROs, something like "mydistro" and "mydistro-dev".
> > > 
> > > While migrating some of the changes, I discovered that I run into
> > > caching(?) issues. I have a recipe for an internal application
> > > and want
> > > to include additional systemd service files in the development
> > > image.
> > > 
> > > What I did was this:
> > > 
> > > Added "application-dbg.service" to recipes-
> > > internal/application/files
> > > 
> > > Adapted application.bb recipe:
> > > 
> > > SRC_URI:append:mydistro-dev = " file://application-dbg.service"
> > > 
> > > do_install {
> > >       # ...snip...
> > >       # systemd service
> > >       install -d ${D}${systemd_system_unitdir}
> > >       install -m 0644 ${WORKDIR}/application.service
> > > ${D}${systemd_system_unitdir}
> > > }
> > > 
> > > do_install:append:mydistro-dev() {
> > >       # debug systemd services
> > >       install -d ${D}${systemd_system_unitdir}
> > >       install -m 0644 ${WORKDIR}/application-dbg.service
> > > ${D}${systemd_system_unitdir}
> > > }
> > > 
> > > 
> > > When I run "DISTRO=mydistro-dev bitbake application" followed by
> > > "DISTRO=mydistro bitbake application", the debug service file is
> > > still
> > > present in the package. This seems to be caused by the "image"
> > > directory in the recipe WORKDIR not being cleaned between
> > > subsequent
> > > do_install runs. Is this expected behaviour? What's the best
> > > solution?
> > > 
> > > Kind regards,
> > > Nicolas
> > > 
> > > [1]:
> > > https://lore.kernel.org/yocto/CAH9dsRiArf_9GgQS4hCg5=J_Jk6cd3eiGaOiQd788+iSLTuU+g@mail.gmail.com/
> > > [2]:
> > > https://lore.kernel.org/yocto/VI1PR0602MB3549F83AC93A53785DE48677D3FD9@VI1PR0602MB3549.eurprd06.prod.outlook.com/
> > > 
> > > 
> > > 
> > 
> > -- 
> > Mike Looijmans
> > 
> > 
> > 
> > 
> 

Re: [yocto] Switching between multiple DISTROs without "contamination"

[Mike Looijmans]

Hi Nicolas,

I guess part of your problem is that DISTRO does not end up in any 
package grouping, unlike MACHINE and ARCH and so.

So if OE/Yocto does The Right Thing, this means that it will have to 
remove a bunch of packages from your feed and sysroot, and add them 
again (from sstate-cache or by building) every time you switch DISTRO 
(which, on a build server, would be each and every build).
Hence my advice to use a separate build directory for each, otherwise 
Yocto will have to juggle with several packages on each and every build. 
With accidents waiting to happen.

Hence my advice remains, you can have "my-image-dev" and even 
"my-machine-dev" in the same build directory, but don't share 
"my-distro-dev" that way, it's an invitation to trouble, even though It 
Should Work In Theory...

Mike.


Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Products B.V.
Materiaalweg 4, 5681 RJ Best
The Netherlands

T: +31 (0) 499 33 69 69
E: mike.looijmans@topicproducts.com
W: www.topic.nl

Please consider the environment before printing this e-mail
On 13-07-2022 15:32, Nicolas Jeker wrote:
> Thanks Martin and Mike for your explanations and tips.
>
> So, I've done a lot of testing today and it seems I simplified the
> example in my first email a bit too much. The example as-is works fine
> when switching DISTROs as far as I can tell. The problem only arises
> when wildcards are used.
>
> Changing my initial example like this should trigger the behaviour I've
> initially described:
>
> SRC_URI:append:mydistro-dev = " file://application-dbg.service"
>
> do_install {
>      # ...snip...
>      # systemd service
>      install -d ${D}${systemd_system_unitdir}
>      install -m 0644 ${WORKDIR}/*.service ${D}${systemd_system_unitdir}
> }
>
> do_install:append:mydistro-dev() {
>      # debug systemd services
>      install -d ${D}${systemd_system_unitdir}
>      install -m 0644 ${WORKDIR}/application-dbg.service
> ${D}${systemd_system_unitdir}
> }
>
> Notice the *.service in do_install.
>
>  From my testing, this is how contamination happens:
>
> 1) Build with 'DISTRO=mydistro bitbake application'. All tasks for the
> recipe are run and the directories in WORKDIR are populated, including
> the "application.service" file.
> 2) Build with 'DISTRO=mydistro-dev bitbake application'. do_unpack is
> rerun and places the additional "application-dbg.service" file in
> WORKDIR.
> 3) Switching back to 'mydistro' will get the recipe from sstate cache,
> which works fine.
> 4) Changing application.bb and rebuilding with 'DISTRO=mydistro bitbake
> application' reruns do_install (as expected). This leads to the
> packages do_install picking up the additional "application-dbg.service"
> file left behind by the invocation in step 2).
>
> Mike, Martin: Do you remember in which cases you encountered problems
> when sharing the build directory?
>
> On Tue, 2022-07-12 at 09:15 -0700, Khoi Dinh Trinh wrote:
>> Thank you Nicolas for asking this question since I will probably run
>> into this issue soon if not for this email thread. The answers so far
>> have been very helpful but I just want to clarify a bit more on why
>> doesn't the package get rebuilt? From my understanding, Yocto should
>> rerun a task when the signature of the task changes and since
>> do_install has an override on mydistro-dev, shouldn't the content and
>> thus the signature of do_install change when switching distro and so
>> Yocto should rerun it?
> As far as I can tell, all the relevant tasks are rerun correctly when
> something is changed. Relevant tasks meaning only the tasks that are
> actually different.
>
> The specific issue I experienced is due to the WORKDIR not being
> cleaned between different task invocations and the recipe (probably
> wrongfully) relying on wildcards to gather files, see example above.
>
>> I have a lot of tasks with override not just on DISTRO but other
>> things like MACHINE or custom variables so I want to understand the
>> rebuild mechanism as best I can.
>>
> There's surely someone more knowledgeable here that could clarify the
> inner workings of this mechanism a lot better than me.
>
>> Best,
>> Khoi Trinh
>>
>> On Tue, Jul 12, 2022 at 8:05 AM Mike Looijmans
>> <mike.looijmans@topic.nl> wrote:
>>> Quick answer: Don't build multiple distros in one build directory.
>>>
> It's really a bummer that it's not reliably possible to switch between
> DISTROs inside one build directory.
>
>>> You might get away with setting TMPDIR = "tmp-${DISTRO}" to give
>>> each
>>> its own.
>>>
>>> But I'd rather advice to set up two separate builds and just point
>>> the
>>> downloads and sstate-cache to the same location. It'll be faster
>>> than
>>> the TMPDIR option.
>>>
>>> Or figure out how to put the difference in the IMAGE only. Then you
>>> can
>>> just build both images (in parallel, woot), which is faster, more
>>> convenient and saves on diskspace.
> As I'm currently working with something close to what you describe, I
> think I'll try to stay away from multiple DISTROs if possible and
> improve on what I'm already doing.
>
>>> What I often do is have my-application.bb generate a
>>> my-application-utils package that only gets installed in the "dev"
>>> image
>>> but not in the production one, which only installs "my-
>>> application".
> This is probably what Martin meant with his A.bb and A-xx.bb example.
> It's so far one of the best approaches I've seen, thanks.
>
>>> You could also create a "my-application-dev.bb" recipe that
>>> includes
>>> my-application.bb and just changes what it needs to be different.
>>>
>>>
>>>
>>>
>>> Met vriendelijke groet / kind regards,
>>>
>>> Mike Looijmans
>>> System Expert
>>>
>>>
>>> TOPIC Embedded Products B.V.
>>> Materiaalweg 4, 5681 RJ Best
>>> The Netherlands
>>>
>>> T: +31 (0) 499 33 69 69
>>> E: mike.looijmans@topicproducts.com
>>> W: www.topic.nl
>>>
>>> Please consider the environment before printing this e-mail
>>> On 12-07-2022 15:37, Nicolas Jeker via lists.yoctoproject.org
>>> wrote:
>>>> Hi all,
>>>>
>>>> I'm currently using an additional layer and image to
>>>> differentiate
>>>> between a release and development build (enabling NFS, SSH, root
>>>> login,
>>>> etc.). To create a development build, I manually add the layer to
>>>> bblayers.conf. This works quite well, but feels a bit clumsy to
>>>> integrate into a CI/CD pipeline.
>>>>
>>>> Per these past discussions here [1][2], I'm now trying to migrate
>>>> to
>>>> multiple DISTROs, something like "mydistro" and "mydistro-dev".
>>>>
>>>> While migrating some of the changes, I discovered that I run into
>>>> caching(?) issues. I have a recipe for an internal application
>>>> and want
>>>> to include additional systemd service files in the development
>>>> image.
>>>>
>>>> What I did was this:
>>>>
>>>> Added "application-dbg.service" to recipes-
>>>> internal/application/files
>>>>
>>>> Adapted application.bb recipe:
>>>>
>>>> SRC_URI:append:mydistro-dev = " file://application-dbg.service"
>>>>
>>>> do_install {
>>>>        # ...snip...
>>>>        # systemd service
>>>>        install -d ${D}${systemd_system_unitdir}
>>>>        install -m 0644 ${WORKDIR}/application.service
>>>> ${D}${systemd_system_unitdir}
>>>> }
>>>>
>>>> do_install:append:mydistro-dev() {
>>>>        # debug systemd services
>>>>        install -d ${D}${systemd_system_unitdir}
>>>>        install -m 0644 ${WORKDIR}/application-dbg.service
>>>> ${D}${systemd_system_unitdir}
>>>> }
>>>>
>>>>
>>>> When I run "DISTRO=mydistro-dev bitbake application" followed by
>>>> "DISTRO=mydistro bitbake application", the debug service file is
>>>> still
>>>> present in the package. This seems to be caused by the "image"
>>>> directory in the recipe WORKDIR not being cleaned between
>>>> subsequent
>>>> do_install runs. Is this expected behaviour? What's the best
>>>> solution?
>>>>
>>>> Kind regards,
>>>> Nicolas
>>>>
>>>> [1]:
>>>> https://lore.kernel.org/yocto/CAH9dsRiArf_9GgQS4hCg5=J_Jk6cd3eiGaOiQd788+iSLTuU+g@mail.gmail.com/
>>>> [2]:
>>>> https://lore.kernel.org/yocto/VI1PR0602MB3549F83AC93A53785DE48677D3FD9@VI1PR0602MB3549.eurprd06.prod.outlook.com/
>>>>
>>>>
>>>>
>>> -- 
>>> Mike Looijmans
>>>
>>>
>>>
>>>

-- 
Mike Looijmans

Re: [yocto] Switching between multiple DISTROs without "contamination"

[Richard Purdie]

On Wed, 2022-07-13 at 15:32 +0200, Nicolas Jeker wrote:
> Thanks Martin and Mike for your explanations and tips.
> 
> So, I've done a lot of testing today and it seems I simplified the
> example in my first email a bit too much. The example as-is works fine
> when switching DISTROs as far as I can tell. The problem only arises
> when wildcards are used.
> 
> Changing my initial example like this should trigger the behaviour I've
> initially described:
> 
> SRC_URI:append:mydistro-dev = " file://application-dbg.service"
> 
> do_install {
>     # ...snip...
>     # systemd service
>     install -d ${D}${systemd_system_unitdir}
>     install -m 0644 ${WORKDIR}/*.service ${D}${systemd_system_unitdir}
> }
> 
> do_install:append:mydistro-dev() {
>     # debug systemd services
>     install -d ${D}${systemd_system_unitdir}
>     install -m 0644 ${WORKDIR}/application-dbg.service
> ${D}${systemd_system_unitdir}
> }
> 
> Notice the *.service in do_install.
> 
> From my testing, this is how contamination happens:
> 
> 1) Build with 'DISTRO=mydistro bitbake application'. All tasks for the
> recipe are run and the directories in WORKDIR are populated, including
> the "application.service" file.
> 2) Build with 'DISTRO=mydistro-dev bitbake application'. do_unpack is
> rerun and places the additional "application-dbg.service" file in
> WORKDIR.
> 3) Switching back to 'mydistro' will get the recipe from sstate cache,
> which works fine.
> 4) Changing application.bb and rebuilding with 'DISTRO=mydistro bitbake
> application' reruns do_install (as expected). This leads to the
> packages do_install picking up the additional "application-dbg.service"
> file left behind by the invocation in step 2).
> 
> Mike, Martin: Do you remember in which cases you encountered problems
> when sharing the build directory?

This is unfortunately a known issue and is one reason I'd like to stop
recipes downloading files to WORKDIR. Sadly changing that to fix this
bug is very invasive and painful but I think we do need to do so
somehow.

Cheers,

Richard