way to make do_devshell changes permanent?

way to make do_devshell changes permanent?

[Kent Dorfman]

I am dealing with a vendor supplied yocto release where they have
setup the repos as local gitshallow or tar.xz files.  I asked them to
do this in accordance with our strict air-gapped build policies.

They provide a driver and supplement utilities as a gitshallow copy of
their own repo and I can do_devshell on the recipe to extract the code
and play with it.

What is the "yocto way" of making my changes permanent so that they
survive a do_fetch in the future and are integrated into the target
image?  I've noted that changes made during a devshell session do not
make it into the target image at all at this point, but that
do_devshell is simply a subshell sandbox to play with expanded repo.

Re: [yocto] way to make do_devshell changes permanent?

[Joel A Cohen]

[-- Attachment #1: Type: text/plain, Size: 1569 bytes --]

devshell is good for playing around, but I don't know of any blessed way to
just have it somehow modify recipes and create patches magically. I'd be
interested to hear if there is something like that.

The closest to what I think you want is the "devtool" workflow
<https://www.yoctoproject.org/docs/2.3/dev-manual/dev-manual.html#using-devtool-in-your-workflow>


Basically:
1) devtool modify <recipe>
2) hack source in $BUILDDIR/workspace/sources/<recipe>, test as desired by
using "bitbake <recipe>" or "bitbake <image containing recipe>"
# Once satisfied with changes:
3) git add $BUILDDIR/workspace/sources/<recipe> && git commit -m "Added
cool changes to recipe"
4) devtool finish <recipe> <your_layer_name>

--Aaron

On Tue, Mar 3, 2020 at 3:59 PM Kent Dorfman <kent.dorfman766@gmail.com>
wrote:

> I am dealing with a vendor supplied yocto release where they have
> setup the repos as local gitshallow or tar.xz files.  I asked them to
> do this in accordance with our strict air-gapped build policies.
>
> They provide a driver and supplement utilities as a gitshallow copy of
> their own repo and I can do_devshell on the recipe to extract the code
> and play with it.
>
> What is the "yocto way" of making my changes permanent so that they
> survive a do_fetch in the future and are integrated into the target
> image?  I've noted that changes made during a devshell session do not
> make it into the target image at all at this point, but that
> do_devshell is simply a subshell sandbox to play with expanded repo.
> 
>

[-- Attachment #2: Type: text/html, Size: 2109 bytes --]

Re: [yocto] way to make do_devshell changes permanent?

[Kent Dorfman]

On 3/3/20, Aaron Cohen <aaron@assonance.org> wrote:
> devshell is good for playing around, but I don't know of any blessed way to
> just have it somehow modify recipes and create patches magically. I'd be
> interested to hear if there is something like that.
>
> The closest to what I think you want is the "devtool" workflow
> <https://www.yoctoproject.org/docs/2.3/dev-manual/dev-manual.html#using-devtool-in-your-workflow>
>
>
> Basically:
> 1) devtool modify <recipe>
> 2) hack source in $BUILDDIR/workspace/sources/<recipe>, test as desired by
> using "bitbake <recipe>" or "bitbake <image containing recipe>"
> # Once satisfied with changes:
> 3) git add $BUILDDIR/workspace/sources/<recipe> && git commit -m "Added
> cool changes to recipe"
> 4) devtool finish <recipe> <your_layer_name>

I'm not sure that the git add is relevant for all recipes though.
Some of them are simply tarball sources and others are shallow git
archives.  Should I be confident that yocto will know the difference
and treat each case correctly?  or more likely I'll assume that
something was registered and then it doesn't find it's way into the
build but slips through the cracks?

So the question is whether devtool finish is smart enough to save the
changes, regardless of whether the source repo is git or tarball.

Re: [yocto] way to make do_devshell changes permanent?

[Joel A Cohen]

[-- Attachment #1: Type: text/plain, Size: 1837 bytes --]

The “git commit” step is just how you tell devtool about your changes. Each
commit will result in a patch file in your layer (conversely, each patch
file ends up as a commit in the sources directory after your initial
“devtool modify”)

Notably, this git repo is only loosely connected to the repo or tarball
from the SRC_URI

—Aaron

On Tue, Mar 3, 2020 at 7:29 PM Kent Dorfman <kent.dorfman766@gmail.com>
wrote:

> On 3/3/20, Aaron Cohen <aaron@assonance.org> wrote:
> > devshell is good for playing around, but I don't know of any blessed way
> to
> > just have it somehow modify recipes and create patches magically. I'd be
> > interested to hear if there is something like that.
> >
> > The closest to what I think you want is the "devtool" workflow
> > <
> https://www.yoctoproject.org/docs/2.3/dev-manual/dev-manual.html#using-devtool-in-your-workflow
> >
> >
> >
> > Basically:
> > 1) devtool modify <recipe>
> > 2) hack source in $BUILDDIR/workspace/sources/<recipe>, test as desired
> by
> > using "bitbake <recipe>" or "bitbake <image containing recipe>"
> > # Once satisfied with changes:
> > 3) git add $BUILDDIR/workspace/sources/<recipe> && git commit -m "Added
> > cool changes to recipe"
> > 4) devtool finish <recipe> <your_layer_name>
>
> I'm not sure that the git add is relevant for all recipes though.
> Some of them are simply tarball sources and others are shallow git
> archives.  Should I be confident that yocto will know the difference
> and treat each case correctly?  or more likely I'll assume that
> something was registered and then it doesn't find it's way into the
> build but slips through the cracks?
>
> So the question is whether devtool finish is smart enough to save the
> changes, regardless of whether the source repo is git or tarball.
>

[-- Attachment #2: Type: text/html, Size: 2620 bytes --]

Re: [yocto] way to make do_devshell changes permanent?

[Kent Dorfman]

On 3/3/20, Aaron Cohen <aaron@assonance.org> wrote:
> The “git commit” step is just how you tell devtool about your changes. Each
> commit will result in a patch file in your layer (conversely, each patch
> file ends up as a commit in the sources directory after your initial
> “devtool modify”)
>
>> > using "bitbake <recipe>" or "bitbake <image containing recipe>"
>> > # Once satisfied with changes:
>> > 3) git add $BUILDDIR/workspace/sources/<recipe> && git commit -m "Added

Does the git add have to be done as specified above?  I made changes
and inside the directory where the changes occured I just did a git
add -am "my message", but it didn't save my changes when I rebuilt the
image.

Also, the devtool finish seems really horky.  devtool modify <recipe>
checks out the package, but why is it necessary to explicitly specify
a layer when you finish?  It's hard enough knowing what layer a recipe
is coming from in the first place, with all the inheritance, layer
priorities, and bbappends. D'oh!  In some ways seems needlessly
complex: ie flexibility at the expense of elegance.

Re: [yocto] way to make do_devshell changes permanent?

[Joel A Cohen]

[-- Attachment #1: Type: text/plain, Size: 2062 bytes --]

On Wed, Mar 4, 2020 at 10:53 PM Kent Dorfman <kent.dorfman766@gmail.com>
wrote:

> On 3/3/20, Aaron Cohen <aaron@assonance.org> wrote:
> >> > # Once satisfied with changes:
> >> > 3) git add $BUILDDIR/workspace/sources/<recipe> && git commit -m
> "Added
>
> Does the git add have to be done as specified above?  I made changes
> and inside the directory where the changes occured I just did a git
> add -am "my message", but it didn't save my changes when I rebuilt the
> image.
>

I guess this should be more like:
3) cd $BUILDDIR/workspace/sources/<recipe> && git add ... && git commit ...

because I don't think git works correctly from outside the repository.


> Also, the devtool finish seems really horky.  devtool modify <recipe>
> checks out the package, but why is it necessary to explicitly specify
> a layer when you finish?  It's hard enough knowing what layer a recipe
> is coming from in the first place, with all the inheritance, layer
> priorities, and bbappends. D'oh!  In some ways seems needlessly
> complex: ie flexibility at the expense of elegance.
>

You actually don't want to "devtool finish" while you're still playing with
the recipe.

While you're in the "devtool modify" state, bitbake will prefer the source
code within your workspace (you shouldn't need to "git commit" either while
at this stage). "devtool finish" tries to automatically update your recipe,
but also removes the association with the source code in your workspace
directory (it doesn't delete the directory within your workspace as a
safety measure though).

You have to specify the layer to save your changes to, because ideally, you
are either the owner of the recipe, in which case you should know exactly
which layer to save changes, or if you're not the owner of the recipe, the
"correct" thing to do is make a bbappend in your own layer with patch files
for the source code changes. If you're not the owner, your life will be
easier in the end if you don't change the recipe in your copy of the layer.

--Aaron

[-- Attachment #2: Type: text/html, Size: 2910 bytes --]