Working on Dependent Recipes in eSDK

Working on Dependent Recipes in eSDK

[Aaron_Wright]

My team recently started using the eSDK for day-to-day development. 
Previously, they were using the bitbake command to build, which works, but 
has overhead. Yesterday three different people came up to me a question 
about workflow with the eSDK, and they all had to do with dependent 
recipes in the eSDK.

For example, a developer would sit down to make something new that relies 
on a library provided by a recipe in the eSDK. So they "devtool 
sdk-install needed-library", and get to work. While working, they 
discover, I need to add something to this library I depend on so that it 
does what I want in the way I want. So they "devtool modify 
needed-library" and change the source of that library to do what they 
want, and build that library using its build system (CMake or Autotools). 
Then they return to working on the new something and find that none of 
their changes to the library they depend on are available. 

At this point, they are lost. They just start trying things to make it 
work, like "devtool build needed-library", which successfully completes, 
but doesn't help. Some people also try to use the build system of the 
library they modified to install it into the machine sysroot of the eSDK 
with a command like, "make install 
DESTDIR=<ESDK_PATH>/tmp/sysroots/<MACHINE>", and that actually does help. 
They can now use the changes they made to the library in the new something 
they are working on. However, when they use the "devtool sdk-install" 
command again, their changes to the needed library that they installed 
into the machine sysroot are reverted.

So--what is the workflow they are supposed to use? 

Pushing the changes to the library upstream, updating the recipe, and 
waiting for a new eSDK is not practical given that the developers may have 
to iterate on the changes to the library quite a bit before they get it 
just right for their new something.

Is there a way to update the eSDK sysroots with recipes that are in the 
workspace? Could "devtool build" also update the eSDK sysroots? Or is 
there a different way to get this done?

Re: Working on Dependent Recipes in eSDK

[Alexander Kanavin]

I don’t think esdk is designed for this kind of ‘full stack’ development. You can go back to working directly with yocto layers, and skip the esdk altogether. Devtool will work exactly same (minus the sdk specific commands).

Alex

> On 18 Oct 2018, at 19.37, Aaron_Wright@selinc.com wrote:
> 
> My team recently started using the eSDK for day-to-day development. 
> Previously, they were using the bitbake command to build, which works, but 
> has overhead. Yesterday three different people came up to me a question 
> about workflow with the eSDK, and they all had to do with dependent 
> recipes in the eSDK.
> 
> For example, a developer would sit down to make something new that relies 
> on a library provided by a recipe in the eSDK. So they "devtool 
> sdk-install needed-library", and get to work. While working, they 
> discover, I need to add something to this library I depend on so that it 
> does what I want in the way I want. So they "devtool modify 
> needed-library" and change the source of that library to do what they 
> want, and build that library using its build system (CMake or Autotools). 
> Then they return to working on the new something and find that none of 
> their changes to the library they depend on are available. 
> 
> At this point, they are lost. They just start trying things to make it 
> work, like "devtool build needed-library", which successfully completes, 
> but doesn't help. Some people also try to use the build system of the 
> library they modified to install it into the machine sysroot of the eSDK 
> with a command like, "make install 
> DESTDIR=<ESDK_PATH>/tmp/sysroots/<MACHINE>", and that actually does help. 
> They can now use the changes they made to the library in the new something 
> they are working on. However, when they use the "devtool sdk-install" 
> command again, their changes to the needed library that they installed 
> into the machine sysroot are reverted.
> 
> So--what is the workflow they are supposed to use? 
> 
> Pushing the changes to the library upstream, updating the recipe, and 
> waiting for a new eSDK is not practical given that the developers may have 
> to iterate on the changes to the library quite a bit before they get it 
> just right for their new something.
> 
> Is there a way to update the eSDK sysroots with recipes that are in the 
> workspace? Could "devtool build" also update the eSDK sysroots? Or is 
> there a different way to get this done?
> -- 
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto

Re: Working on Dependent Recipes in eSDK

[Aaron_Wright]

> Alexander Kanavin <alex.kanavin@gmail.com> wrote on 10/18/2018 11:30:17 
AM:
>
> I don’t think esdk is designed for this kind of ‘full stack’ 
> development. You can go back to working directly with yocto layers, 
> and skip the esdk altogether. Devtool will work exactly same (minus 
> the sdk specific commands).
> 
> Alex

I'm trying to get the developers away from using bitbake and devtool to 
build their code, as those tools have overhead and don't easily allow the 
developers to iterate quickly or allow them to run other make targets for 
unit testing, etc. I really want the developers at the "make" level for 
the majority of their time. When it comes time for recipes and image 
creation, then they'll use devtool.

> > On 18 Oct 2018, at 19.37, Aaron_Wright@selinc.com wrote:
> > 
> > My team recently started using the eSDK for day-to-day development. 
> > Previously, they were using the bitbake command to build, which works, 
but 
> > has overhead. Yesterday three different people came up to me a 
question 
> > about workflow with the eSDK, and they all had to do with dependent 
> > recipes in the eSDK.
> > 
> > For example, a developer would sit down to make something new that 
relies 
> > on a library provided by a recipe in the eSDK. So they "devtool 
> > sdk-install needed-library", and get to work. While working, they 
> > discover, I need to add something to this library I depend on so that 
it 
> > does what I want in the way I want. So they "devtool modify 
> > needed-library" and change the source of that library to do what they 
> > want, and build that library using its build system (CMake or 
Autotools). 
> > Then they return to working on the new something and find that none of 

> > their changes to the library they depend on are available. 
> > 
> > At this point, they are lost. They just start trying things to make it 

> > work, like "devtool build needed-library", which successfully 
completes, 
> > but doesn't help. Some people also try to use the build system of the 
> > library they modified to install it into the machine sysroot of the 
eSDK 
> > with a command like, "make install 
> > DESTDIR=<ESDK_PATH>/tmp/sysroots/<MACHINE>", and that actually does 
help. 
> > They can now use the changes they made to the library in the new 
something 
> > they are working on. However, when they use the "devtool sdk-install" 
> > command again, their changes to the needed library that they installed 

> > into the machine sysroot are reverted.
> > 
> > So--what is the workflow they are supposed to use? 
> > 
> > Pushing the changes to the library upstream, updating the recipe, and 
> > waiting for a new eSDK is not practical given that the developers may 
have 
> > to iterate on the changes to the library quite a bit before they get 
it 
> > just right for their new something.
> > 
> > Is there a way to update the eSDK sysroots with recipes that are in 
the 
> > workspace? Could "devtool build" also update the eSDK sysroots? Or is 
> > there a different way to get this done?
> > -- 
> > _______________________________________________
> > yocto mailing list
> > yocto@yoctoproject.org
> > https://urldefense.proofpoint.com/v2/url?
> u=https-3A__lists.yoctoproject.org_listinfo_yocto&d=DwIFaQ&c=-
> _uRSsrpJskZgEkGwdW-
> sXvhn_FXVaEGsm0EI46qilk&r=GBJubmJUwwrr1B4TJFRqESMrRsDipvY66vPwm4-
> gtNQ&m=-
> 
C-70rjnGksaogzIkgE5MdoY3IfA3x62MZVOwbZmw4c&s=T5PxZ3bNB429M76sWwBqgrtMsMhB-
> kz1BXkdY8hOIRY&e=

Re: Working on Dependent Recipes in eSDK

[Paul Eggleton]

I think we would *like* this to be possible, from what I can tell I think the 
issue is that the copies of the library in the sysroots of other dependent 
recipes aren't being updated. The eSDK (and devtool) were originally written 
when there weren't recipe-specific sysroots, and this stuff was a bit simpler.

Moving back to the full build system won't automatically fix this, though it 
does make workarounds possible - you'd need to rebuild the dependent recipes 
to get their sysroots updated.

Cheers,
Paul

On Friday, 19 October 2018 7:30:17 AM NZDT Alexander Kanavin wrote:
> I don’t think esdk is designed for this kind of ‘full stack’ development.
> You can go back to working directly with yocto layers, and skip the esdk
> altogether. Devtool will work exactly same (minus the sdk specific
> commands).
> 
> Alex
> 
> > On 18 Oct 2018, at 19.37, Aaron_Wright@selinc.com wrote:
> > 
> > My team recently started using the eSDK for day-to-day development. 
> > Previously, they were using the bitbake command to build, which works, but 
> > has overhead. Yesterday three different people came up to me a question 
> > about workflow with the eSDK, and they all had to do with dependent 
> > recipes in the eSDK.
> > 
> > For example, a developer would sit down to make something new that relies 
> > on a library provided by a recipe in the eSDK. So they "devtool 
> > sdk-install needed-library", and get to work. While working, they 
> > discover, I need to add something to this library I depend on so that it 
> > does what I want in the way I want. So they "devtool modify 
> > needed-library" and change the source of that library to do what they 
> > want, and build that library using its build system (CMake or Autotools). 
> > Then they return to working on the new something and find that none of 
> > their changes to the library they depend on are available. 
> > 
> > At this point, they are lost. They just start trying things to make it 
> > work, like "devtool build needed-library", which successfully completes, 
> > but doesn't help. Some people also try to use the build system of the 
> > library they modified to install it into the machine sysroot of the eSDK 
> > with a command like, "make install 
> > DESTDIR=<ESDK_PATH>/tmp/sysroots/<MACHINE>", and that actually does help. 
> > They can now use the changes they made to the library in the new something 
> > they are working on. However, when they use the "devtool sdk-install" 
> > command again, their changes to the needed library that they installed 
> > into the machine sysroot are reverted.
> > 
> > So--what is the workflow they are supposed to use? 
> > 
> > Pushing the changes to the library upstream, updating the recipe, and 
> > waiting for a new eSDK is not practical given that the developers may have 
> > to iterate on the changes to the library quite a bit before they get it 
> > just right for their new something.
> > 
> > Is there a way to update the eSDK sysroots with recipes that are in the 
> > workspace? Could "devtool build" also update the eSDK sysroots? Or is 
> > there a different way to get this done?
> 


-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Working on Dependent Recipes in eSDK

[Aaron_Wright]

> Paul Eggleton <paul.eggleton@linux.intel.com> wrote on 10/18/2018 
12:30:07 PM:
>
> I think we would *like* this to be possible, from what I can tell I 
think the 
> issue is that the copies of the library in the sysroots of other 
dependent 
> recipes aren't being updated. The eSDK (and devtool) were originally 
written 
> when there weren't recipe-specific sysroots, and this stuff was a bit 
simpler.
> 
> Moving back to the full build system won't automatically fix this, 
though it 
> does make workarounds possible - you'd need to rebuild the dependent 
recipes 
> to get their sysroots updated.
> 
> Cheers,
> Paul
> ... snip ...

I put together a devtool plugin to almost do what I want:

> from devtool import exec_build_env_command, DevtoolError
> 
> def sdk_sysroot(args, config, basepath, workspace):
>     """Entry point for the devtool sdk-sysroot command"""
> 
>     try:
>         exec_build_env_command(config.init_path, basepath, 'bitbake 
build-sysroots', watch=True)
>     except bb.process.ExecutionError as e:
>         raise DevtoolError('Failed to bitbake build-sysroots:\n%s' % 
(str(e)))
> 
> def register_commands(subparsers, context):
>     """Register devtool subcommands from the sdk plugin"""
>     if context.fixed_setup:
>         parser_sdk_sysroot = subparsers.add_parser(
>             'sdk-sysroot',
>             help='Update the SDK sysroot',
>             description='Updates the SDK sysroot so that changes made to 
recipes in the workspace will be available,',
>             group='sdk')
>         parser_sdk_sysroot.set_defaults(func=sdk_sysroot)

This allows me to "devtool modify <DEPENDENT_RECIPE>", edit the code, 
"devtool build <DEPENDENT_RECIPE>", "devtool sdk-sysroot", and use the new 
code from the dependent recipe immediately.

This makes me wonder if "bitbake build-sysroots" should not be a part of 
"devtool build", like it is for "devtool sdk-install" (where I stole it 
from).

Re: Working on Dependent Recipes in eSDK

[ChenQi]

Hi Aaron,

The main pain for now is that `devtool sdk-install <DEPEND_RECIPE>' does 
not work as expected, right?
I just checked the codes of sdk-install, and found that it does not fit 
the current RSS design.
I'd like to check with you if this is the major concern from your team?
Do you have some other concerns?
I'll open enhancements/bugs on bugzilla and hopefully will work on them 
in the 2.7 development cycle.

Best Regards,
Chen Qi


On 10/19/2018 04:59 AM, Aaron_Wright@selinc.com wrote:
>> Paul Eggleton <paul.eggleton@linux.intel.com> wrote on 10/18/2018
> 12:30:07 PM:
>> I think we would *like* this to be possible, from what I can tell I
> think the
>> issue is that the copies of the library in the sysroots of other
> dependent
>> recipes aren't being updated. The eSDK (and devtool) were originally
> written
>> when there weren't recipe-specific sysroots, and this stuff was a bit
> simpler.
>> Moving back to the full build system won't automatically fix this,
> though it
>> does make workarounds possible - you'd need to rebuild the dependent
> recipes
>> to get their sysroots updated.
>>
>> Cheers,
>> Paul
>> ... snip ...
> I put together a devtool plugin to almost do what I want:
>
>> from devtool import exec_build_env_command, DevtoolError
>>
>> def sdk_sysroot(args, config, basepath, workspace):
>>      """Entry point for the devtool sdk-sysroot command"""
>>
>>      try:
>>          exec_build_env_command(config.init_path, basepath, 'bitbake
> build-sysroots', watch=True)
>>      except bb.process.ExecutionError as e:
>>          raise DevtoolError('Failed to bitbake build-sysroots:\n%s' %
> (str(e)))
>> def register_commands(subparsers, context):
>>      """Register devtool subcommands from the sdk plugin"""
>>      if context.fixed_setup:
>>          parser_sdk_sysroot = subparsers.add_parser(
>>              'sdk-sysroot',
>>              help='Update the SDK sysroot',
>>              description='Updates the SDK sysroot so that changes made to
> recipes in the workspace will be available,',
>>              group='sdk')
>>          parser_sdk_sysroot.set_defaults(func=sdk_sysroot)
> This allows me to "devtool modify <DEPENDENT_RECIPE>", edit the code,
> "devtool build <DEPENDENT_RECIPE>", "devtool sdk-sysroot", and use the new
> code from the dependent recipe immediately.
>
> This makes me wonder if "bitbake build-sysroots" should not be a part of
> "devtool build", like it is for "devtool sdk-install" (where I stole it
> from).

Re: Working on Dependent Recipes in eSDK

[Aaron_Wright]

> ChenQi <Qi.Chen@windriver.com> wrote on 10/22/2018 07:04:32 PM:
> 
> Hi Aaron,
> 
> The main pain for now is that `devtool sdk-install <DEPEND_RECIPE>' does 

> not work as expected, right?
> I just checked the codes of sdk-install, and found that it does not fit 
> the current RSS design.
> I'd like to check with you if this is the major concern from your team?
> Do you have some other concerns?
> I'll open enhancements/bugs on bugzilla and hopefully will work on them 
> in the 2.7 development cycle.
> 
> Best Regards,
> Chen Qi
> 
> ... snip ...

Well, I'm not quite sure how "devtool sdk-install <DEPENDENT_RECIPE>" 
relates after "devtool modify <DEPENDENT_RECIPE>" has been called. 

My developers use devtool to install dependent recipes into the eSDK when
they start working on something. Often they find that they need to make a
change in the library code that a dependent recipe provides. In effect,
they are developing a patch for the dependent recipe. So they use devtool
to say they're going to modify that dependent recipe, and they go and
make the changes they want. Then what? How do they get those changes,
that patch, available to the eSDK?

Any changes made to the library code provided by the dependent recipe are
not available to the eSDK, except through devtool commands. In effect
there are two different worlds at play, the eSDK sysroots versus the
recipe sysroots. The devtool commands seem to make use of the recipe
sysroots, but in my mind that defeats the purpose of an eSDK. So I made
a plug-in for devtool were I can force the eSDK sysroots to be updated.
I'm not sure how that works, but it seems to propagate the workspace 
(recipe sysroot land) to the eSDK, so changes that developers make are
now available for use.

I want my developers to use the toolchain (cmake, make, gcc, etc) 
provided by the eSDK and by things installed into it 
(devtool sdk-install), rather than having to use devtool as their
iterative build command.

Ultimately, I would like to be able to skip my devtool plug-in, as it 
requires the use of "devtool build <DEPENDENT_RECIPE>" and 
"devtool sdk-sysroot" each time a dependent recipe is modified. But I am
not sure how that would work, other than the devtool build command 
should be updating the eSDK sysroots itself.

Re: Working on Dependent Recipes in eSDK

[Khem Raj]

On Thu, Oct 25, 2018 at 8:29 AM <Aaron_Wright@selinc.com> wrote:
>
> > ChenQi <Qi.Chen@windriver.com> wrote on 10/22/2018 07:04:32 PM:
> >
> > Hi Aaron,
> >
> > The main pain for now is that `devtool sdk-install <DEPEND_RECIPE>' does
>
> > not work as expected, right?
> > I just checked the codes of sdk-install, and found that it does not fit
> > the current RSS design.
> > I'd like to check with you if this is the major concern from your team?
> > Do you have some other concerns?
> > I'll open enhancements/bugs on bugzilla and hopefully will work on them
> > in the 2.7 development cycle.
> >
> > Best Regards,
> > Chen Qi
> >
> > ... snip ...
>
> Well, I'm not quite sure how "devtool sdk-install <DEPENDENT_RECIPE>"
> relates after "devtool modify <DEPENDENT_RECIPE>" has been called.
>
> My developers use devtool to install dependent recipes into the eSDK when
> they start working on something. Often they find that they need to make a
> change in the library code that a dependent recipe provides. In effect,
> they are developing a patch for the dependent recipe. So they use devtool
> to say they're going to modify that dependent recipe, and they go and
> make the changes they want. Then what? How do they get those changes,
> that patch, available to the eSDK?
>
> Any changes made to the library code provided by the dependent recipe are
> not available to the eSDK, except through devtool commands. In effect
> there are two different worlds at play, the eSDK sysroots versus the
> recipe sysroots. The devtool commands seem to make use of the recipe
> sysroots, but in my mind that defeats the purpose of an eSDK. So I made
> a plug-in for devtool were I can force the eSDK sysroots to be updated.
> I'm not sure how that works, but it seems to propagate the workspace
> (recipe sysroot land) to the eSDK, so changes that developers make are
> now available for use.
>
> I want my developers to use the toolchain (cmake, make, gcc, etc)
> provided by the eSDK and by things installed into it
> (devtool sdk-install), rather than having to use devtool as their
> iterative build command.
>
> Ultimately, I would like to be able to skip my devtool plug-in, as it
> requires the use of "devtool build <DEPENDENT_RECIPE>" and
> "devtool sdk-sysroot" each time a dependent recipe is modified. But I am
> not sure how that would work, other than the devtool build command
> should be updating the eSDK sysroots itself.

I think you have two options here

1. Feedback the tweaks to packages back into source and re-generate
SDK and if you setup SDK feeds then users/devs can simply do
devtool sdk-update
and this time the package in question will get updated in their SDKs
with the change that was initiated.

second option could be to generate a downstream derivative SDK using
devtool build-sdk and use this SDK


> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto