automatic recipe upgrades - making them better and sweeter

automatic recipe upgrades - making them better and sweeter

[Alexander Kanavin]

Hello,

I've been asked by RP to look into making automatic package upgrades 
work better, so here's what I'd like to improve:

1) when using 'devtool upgrade', remove the need to specify version and 
source revision on the command line, if we want to upgrade to the latest 
release. There is now distrodata/checkpkg code in place to determine 
both, so 'devtool upgrade <recipe>' could simply default to that.

2) help with LICENSE checksum changes a bit more; both devtool and AUH 
have some code for this, so I want to review what they do, and 
consolidate it in a single place. Maintainers should be able to see a 
diff of what the changes are.

3) 'devtool finish' currently does not handle the recipe upgrade 
situation well: the new recipe and patches are placed side by side with 
the current recipe. The old stuff should be removed, and a tentative 
commit should be created. So maybe add a 'devtool finish-upgrade'.

4) AUH should be stripped to its core business: sending useful emails to 
maintainers, and leave the rest to devtool. If 'devtool upgrade' 
succeeded, then corresponding patches should be attached, otherwise (if 
patch rebase failed, or a fetch issue), at least it could tell what 
happened and suggest a devtool upgrade command to get started. Any 
duplicate functionality for upgrading (stuff that is also in devtool) 
should be removed.

Anything I missed? Comments?

Thanks,
Alex

Re: automatic recipe upgrades - making them better and sweeter

[Paul Eggleton]

Hi Alex,

On Tuesday, 7 November 2017 12:50:05 AM NZDT Alexander Kanavin wrote:
> I've been asked by RP to look into making automatic package upgrades 
> work better, so here's what I'd like to improve:
> 
> 1) when using 'devtool upgrade', remove the need to specify version and 
> source revision on the command line, if we want to upgrade to the latest 
> release. There is now distrodata/checkpkg code in place to determine 
> both, so 'devtool upgrade <recipe>' could simply default to that.

You could certainly have it dynamically inherit distrodata for the recipe (via 
the workspace bbappend) and then use that to get the required information. I'd 
avoided that extra complexity earlier but it may be that now it's actually 
easier than it was before. The only thing I would say is we need to produce 
clear error messages if we don't have the needed information to automatically 
figure out what the upgrade version should be.

> 2) help with LICENSE checksum changes a bit more; both devtool and AUH 
> have some code for this, so I want to review what they do, and 
> consolidate it in a single place. Maintainers should be able to see a 
> diff of what the changes are.

Agreed. For everyone else's benefit, Alex filed a bug for this a while ago:

  https://bugzilla.yoctoproject.org/show_bug.cgi?id=10801

I am keeping an eye on this one.

> 3) 'devtool finish' currently does not handle the recipe upgrade 
> situation well: the new recipe and patches are placed side by side with 
> the current recipe. The old stuff should be removed, and a tentative 
> commit should be created. So maybe add a 'devtool finish-upgrade'.

That's not what's supposed to happen - it should already be deleting the 
original files (it takes a record of the copied files when you run devtool 
upgrade, and then when you run devtool finish it deletes those and moves the 
new ones across from the workspace). If that's not working it's a bug and I 
would appreciate a reproducer. I'd certainly prefer that we keep to one finish 
command that does its best to "do the right thing" depending on the situation.
 
> 4) AUH should be stripped to its core business: sending useful emails to 
> maintainers, and leave the rest to devtool. If 'devtool upgrade' 
> succeeded, then corresponding patches should be attached, otherwise (if 
> patch rebase failed, or a fetch issue), at least it could tell what 
> happened and suggest a devtool upgrade command to get started. Any 
> duplicate functionality for upgrading (stuff that is also in devtool) 
> should be removed.

Agreed. There's a bug open for this one too:

  https://bugzilla.yoctoproject.org/show_bug.cgi?id=10598

BTW, you have impeccable timing as I am currently in the middle of preparing a 
fairly substantial changeset for devtool including implementing a dry-run mode 
for devtool finish (so you can see what it's going to do) and fixing the 
following bugs, among others:

  https://bugzilla.yoctoproject.org/show_bug.cgi?id=10939
  https://bugzilla.yoctoproject.org/show_bug.cgi?id=11948
  https://bugzilla.yoctoproject.org/show_bug.cgi?id=11516

Current WIP is on paule/devtool31 in poky-contrib.

Cheers,
Paul

-- 
Paul Eggleton
Intel Open Source Technology Centre

Re: automatic recipe upgrades - making them better and sweeter

[Alexander Kanavin]

On 11/06/2017 10:04 PM, Paul Eggleton wrote:
>> 3) 'devtool finish' currently does not handle the recipe upgrade
>> situation well: the new recipe and patches are placed side by side with
>> the current recipe. The old stuff should be removed, and a tentative
>> commit should be created. So maybe add a 'devtool finish-upgrade'.
> 
> That's not what's supposed to happen - it should already be deleting the
> original files (it takes a record of the copied files when you run devtool
> upgrade, and then when you run devtool finish it deletes those and moves the
> new ones across from the workspace). If that's not working it's a bug and I
> would appreciate a reproducer. I'd certainly prefer that we keep to one finish
> command that does its best to "do the right thing" depending on the situation.

For me it happens every time. I run 'devtool finish <recipe> ../meta' 
and while the new files are placed into the correct dir in the layer, 
the old ones all remain (which is especially annoying if I then need to 
manually sort the added/deleted patch files).

Can you run a quick check to see if it happens on your side? Something 
trivial, like ffmpeg would do. I can then file a bug.

Also, there shouldn't be a need to specify the destination at all, if 
the recipe is modified or upgraded.


Alex

Re: automatic recipe upgrades - making them better and sweeter

[Paul Eggleton]

On Wednesday, 8 November 2017 4:20:18 AM NZDT Alexander Kanavin wrote:
> On 11/06/2017 10:04 PM, Paul Eggleton wrote:
> >> 3) 'devtool finish' currently does not handle the recipe upgrade
> >> situation well: the new recipe and patches are placed side by side with
> >> the current recipe. The old stuff should be removed, and a tentative
> >> commit should be created. So maybe add a 'devtool finish-upgrade'.
> > 
> > That's not what's supposed to happen - it should already be deleting the
> > original files (it takes a record of the copied files when you run devtool
> > upgrade, and then when you run devtool finish it deletes those and moves
> > the new ones across from the workspace). If that's not working it's a bug
> > and I would appreciate a reproducer. I'd certainly prefer that we keep to
> > one finish command that does its best to "do the right thing" depending on
> > the situation.
> 
> For me it happens every time. I run 'devtool finish <recipe> ../meta' 
> and while the new files are placed into the correct dir in the layer, 
> the old ones all remain (which is especially annoying if I then need to 
> manually sort the added/deleted patch files).
> 
> Can you run a quick check to see if it happens on your side? Something 
> trivial, like ffmpeg would do. I can then file a bug.

You're right, I tried upgrading ffmpeg from 3.3.3 to 3.3.5 and it didn't 
remove the old files. I've gone ahead and filed a bug and will look at it 
today:

  https://bugzilla.yoctoproject.org/show_bug.cgi?id=12318
 
> Also, there shouldn't be a need to specify the destination at all, if 
> the recipe is modified or upgraded.

So there is a method to this madness, but it's less about your use case than 
it is people making customisations that they are best advised to put somewhere 
else - I'm trying to make the user think about where the changes should go 
beforehand, because the nature of devtool finish is that you can't easily go 
back and shift the changes elsewhere after you've run it. It would be trivial 
to add an option that put the recipe back where it came from without you 
having to be explicit though, so we could add that; potentially we could have 
a configuration option to make that the default for you and others who more 
commonly update the base metadata. Otherwise, I'm open to alternative 
suggestions, but preferably I still want to avoid making it easy for users to 
make a mistake and update the original metadata when they really should have 
pointed it at their own layer.

Cheers,
Paul

-- 
Paul Eggleton
Intel Open Source Technology Centre

Re: automatic recipe upgrades - making them better and sweeter

[Alexander Kanavin]

On 11/07/2017 10:17 PM, Paul Eggleton wrote:

> So there is a method to this madness, but it's less about your use case than
> it is people making customisations that they are best advised to put somewhere
> else - I'm trying to make the user think about where the changes should go
> beforehand, because the nature of devtool finish is that you can't easily go
> back and shift the changes elsewhere after you've run it. It would be trivial
> to add an option that put the recipe back where it came from without you
> having to be explicit though, so we could add that; potentially we could have
> a configuration option to make that the default for you and others who more
> commonly update the base metadata. Otherwise, I'm open to alternative
> suggestions, but preferably I still want to avoid making it easy for users to
> make a mistake and update the original metadata when they really should have
> pointed it at their own layer.

That's fine with me. Unlike specifying version/revision on the command 
line (which at the moment need to be manually determined, and change 
every time), the layer destination is more or less static and can be 
hardcoded in a wrapper (or finger memory).

Alex