Re: [oe-commits] org.oe.dev rootfs_ipk: as per OE policy: remove feed management tools

Re: [oe-commits] org.oe.dev rootfs_ipk: as per OE policy: remove feed management tools

[Michael 'Mickey' Lauer]

koen commit wrote:
> rootfs_ipk: as per OE policy: remove feed management tools

Guys, please don't play games. Since it is a handy feature indeed,
can we perhaps agree on making this a seperate bbclass to keep both of
you satisfied?

:M:
-- 
Michael 'Mickey' Lauer | IT-Freelancer | http://www.vanille-media.de

Re: [oe-commits] org.oe.dev rootfs_ipk: as per OE policy: remove feed management tools

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Michael 'Mickey' Lauer schreef:
> koen commit wrote:
>> rootfs_ipk: as per OE policy: remove feed management tools
> 
> Guys, please don't play games. Since it is a handy feature indeed,
> can we perhaps agree on making this a seperate bbclass to keep both of
> you satisfied?

All feed management was removed a long time ago from OE because

a) it's out of scope
b) can't be done right in an automated fashion

And after discovering some bugs in ipkg-utils a while ago it seems you have to manually
munge the Packages file to get something that ipkg will handle correctly when upgrading stuff.

So whichever way you look at it it's going to add broken behaviour to OE. I don't want to
explain to users why their 'feeds' don't work because they disabled common sense because
'OE generated it'.

But but but but but... OE generates feeds now as well!! Right, deploy/ipk contains
subdirectories people could abuse as feeds if they want, but I wouldn't recommend it.

So *if* people *need* OE to generate feeds, there you have them. However, I strongly
oppose adding know broken behaviour to satisfy lazy people.

regards,

Koen

PS: find */ -name  "*.ipk" -exec mv  '{}'  ./ \;


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFF5v34MkyGM64RGpERAjI0AJ4tbIeCUQoeurXBhlJ8vdFmQ0AD0QCZAVSW
EUO+7SF5AFM4BwLZyTj89tc=
=xAzY
-----END PGP SIGNATURE-----

Re: [oe-commits] org.oe.dev rootfs_ipk: as per OE policy: remove feed management tools

[Hans Henry von Tresckow]

Koen, Is there some documentation available on how to properly generate
feeds by hand then? I whould like to do things "correctly", but references
to "munging" things by hand don't help me much at this point.

Thank you very much for any pointers.

On 3/1/07, Koen Kooi <koen@dominion.kabel.utwente.nl> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Michael 'Mickey' Lauer schreef:
> > koen commit wrote:
> >> rootfs_ipk: as per OE policy: remove feed management tools
> >
> > Guys, please don't play games. Since it is a handy feature indeed,
> > can we perhaps agree on making this a seperate bbclass to keep both of
> > you satisfied?
>
> All feed management was removed a long time ago from OE because
>
> a) it's out of scope
> b) can't be done right in an automated fashion
>
> And after discovering some bugs in ipkg-utils a while ago it seems you
> have to manually
> munge the Packages file to get something that ipkg will handle correctly
> when upgrading stuff.
>
> So whichever way you look at it it's going to add broken behaviour to OE.
> I don't want to
> explain to users why their 'feeds' don't work because they disabled common
> sense because
> 'OE generated it'.
>
> But but but but but... OE generates feeds now as well!! Right, deploy/ipk
> contains
> subdirectories people could abuse as feeds if they want, but I wouldn't
> recommend it.
>
> So *if* people *need* OE to generate feeds, there you have them. However,
> I strongly
> oppose adding know broken behaviour to satisfy lazy people.
>
> regards,
>
> Koen
>
> PS: find */ -name  "*.ipk" -exec mv  '{}'  ./ \;
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.5 (Darwin)
>
> iD8DBQFF5v34MkyGM64RGpERAjI0AJ4tbIeCUQoeurXBhlJ8vdFmQ0AD0QCZAVSW
> EUO+7SF5AFM4BwLZyTj89tc=
> =xAzY
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>

Re: [oe-commits] org.oe.dev rootfs_ipk: as per OE policy: remove feed management tools

[Hans Henry von Tresckow]

On 3/1/07, Koen Kooi <koen@dominion.kabel.utwente.nl> wrote:
>
>
> And after discovering some bugs in ipkg-utils a while ago it seems you
> have to manually
> munge the Packages file to get something that ipkg will handle correctly
> when upgrading stuff.


Koen, could you point me towards the bugs you are refereing to? I would like
to take a look at ipkg-utils to see if I can find some sort of solution.

Thanks,
Henry

regards,
>
> Koen
>
>
>

Re: [oe-commits] org.oe.dev rootfs_ipk: as per OE policy: remove feed management tools

[Richard Purdie]

Hi,

On Thu, 2007-03-01 at 17:23 +0100, Koen Kooi wrote:
> Michael 'Mickey' Lauer schreef:
> > koen commit wrote:
> >> rootfs_ipk: as per OE policy: remove feed management tools
> > 
> > Guys, please don't play games. Since it is a handy feature indeed,
> > can we perhaps agree on making this a seperate bbclass to keep both of
> > you satisfied?

Mickey has a point here. There are a number of people upset by the
recent deploy directory changes. I would like a calm rational discussion
to take place before any further commits are made on this subject.

Personally, I'm currently undecided. I'd like to hear more from both
sides. If there is a convincing reason for breaking deploy as a feed
behaviour, I'm listening. If we can use deploy as a feed and just need
to fix some broken tools, we should consider allowing that (even if its
a case of user beware and not officially supported).

Moving on to Koen's comments:

> All feed management was removed a long time ago from OE because
> 
> a) it's out of scope
> b) can't be done right in an automated fashion

Can you explain b) a bit more?

> And after discovering some bugs in ipkg-utils a while ago it seems you have to manually
> munge the Packages file to get something that ipkg will handle correctly when upgrading stuff.

Does this not mean we need to fix ipkg or ipkg-utils? Can you explain
what the problem is as a lot of people would like to know. We might even
the find someone to fix it...

> So whichever way you look at it it's going to add broken behaviour to OE. I don't want to
> explain to users why their 'feeds' don't work because they disabled common sense because
> 'OE generated it'.
>
> But but but but but... OE generates feeds now as well!! Right, deploy/ipk contains
> subdirectories people could abuse as feeds if they want, but I wouldn't recommend it.

Something a little more technical would be good here. 

We have a spectrum of users with differing abilities. I do things I'd
not recommend people did all the time. In a lot of those cases, if
someone's patches stopped me doing them for no good reason, I'd want to
fix that.

> So *if* people *need* OE to generate feeds, there you have them. However, I strongly
> oppose adding know broken behaviour to satisfy lazy people.

What is this known broken behaviour you keep talking about?

I will add that feed generation is something that most OE users have to
deal with at some point. It would be nice to have a wiki page or
something showing some best practise in managing them. I know I'll have
to face this with poky at some point and others are in a similar
position. I don't think feed management should dictate how OE's
directory structure works but I'm also not sure saying feed management
is something external is a good idea either since we're all going to
have to deal with it at some point.

Cheers,

Richard

Re: [oe-commits] org.oe.dev rootfs_ipk: as per OE policy: remove feed management tools

[Matthias Hentges]

Am Donnerstag, den 01.03.2007, 17:23 +0100 schrieb Koen Kooi:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Michael 'Mickey' Lauer schreef:
> > koen commit wrote:
> >> rootfs_ipk: as per OE policy: remove feed management tools
> > 
> > Guys, please don't play games. Since it is a handy feature indeed,
> > can we perhaps agree on making this a seperate bbclass to keep both of
> > you satisfied?
> 
> All feed management was removed a long time ago from OE because

Bullshit. Explain meta/package-index.bb (I know you can't as you ignored
my previous email due to lack of arguments)

> a) it's out of scope

So you alone define the "scope" of OpenEmbedded and it's usage?
Interesting indeed! I guess I didn't get the memo about that!

> b) can't be done right in an automated fashion

Bullshit again. Because _you_ can't do this doesn't mean anybody else
ever had any problems with it. It was working fine for _years_.

> And after discovering some bugs in ipkg-utils a while ago it seems you have to manually
> munge the Packages file to get something that ipkg will handle correctly when upgrading stuff.

Bullshit part three. I was managing a distribution which aimed to make
any updates available via "ipkg upgrade". Ignoring corner-cases like
kernel upgrades it worked as expected. Always. I never _ever_ had to
edit Packages. Sounds like you FUBAR'ed some packages and then blame the
fault on ipkg.

There are problems with corner-cases on ipkg-upgrade but I prefer
solving them to sticking my head into the ground.

> So whichever way you look at it it's going to add broken behaviour to OE. I don't want to
> explain to users why their 'feeds' don't work because they disabled common sense because
> 'OE generated it'.

It isn't broken, never was. And people compiling their images w/ OE from
scratch can be expected to handle their own feeds. We are talking
developers here, not mouse-pushing windows-hugging endusers!

> But but but but but... OE generates feeds now as well!! Right, deploy/ipk contains
> subdirectories people could abuse as feeds if they want, but I wouldn't recommend it.

No one even asked you about your opinion! You blindly removed a very
useful feature of OE for _bullshit_ reasons after a _bullshit_ RFC that
noone even had a chance to reply to in time. 

You even reverted (!) my compromise cset which added a completely
optional an OFF BY DEFAULT oe-feed quoting a _bullshit_ policy that is
no where to be found!

> So *if* people *need* OE to generate feeds, there you have them. However, I strongly
> oppose adding know broken behaviour to satisfy lazy people.

Oh I'm lazy all right. I don't see the point that 90% of all OE users
suddenly are forced to use custom feed-creation scripts (pointless
post-processing).

> regards,
> 
> Koen
> 
> PS: find */ -name  "*.ipk" -exec mv  '{}'  ./ \;
> 

Needless post-processing that is only required due to your selfish and
ignorant commit to rootfs_ipk and complete and utter disregard of other
peoples interest.

Re: [oe-commits] org.oe.dev rootfs_ipk: as per OE policy: remove feed management tools

[Rod Whitby]

">" = RP, ">>" = Koen ...

> Mickey has a point here. There are a number of people upset by the
> recent deploy directory changes. I would like a calm rational discussion
> to take place before any further commits are made on this subject.

The voices of reason.  Thank you.

> Personally, I'm currently undecided. I'd like to hear more from both
> sides. If there is a convincing reason for breaking deploy as a feed
> behaviour, I'm listening. If we can use deploy as a feed and just need
> to fix some broken tools, we should consider allowing that (even if its
> a case of user beware and not officially supported).

Unslung and SlugOS has been using an rsync of the tmp/deploy/ipk
directory (with no changes) for over two years, without a single
complaint from either developers or users.

>> All feed management was removed a long time ago from OE because
>>
>> a) it's out of scope

I want it to be in scope.  Who makes the decision as to what's in scope
and what's out of scope.  If I want to bring something that was
previously out of scope (and we've seen no documentation that OE feeds
falls in this category, so I don't concede that at all), and it does not
affect any existing use of OE, then who makes the final decision on
whether it can be brought in scope or not?  I truly hope it's not just
the will of a single developer, or a single distro ...

>> b) can't be done right in an automated fashion
> 
> Can you explain b) a bit more?

Seconded.  As stated above, Unslung and SlugOS have been doing this for
two years in an automated way with nothing more than "rsync" and what's
already in OE.  No complaints from developers or users.  No upgrade
problems.  No bugs reported.  That doesn't mean it works for any other
distros, but we haven't been given the technical information of how it
breaks for anyone yet ...

> I will add that feed generation is something that most OE users have to
> deal with at some point. It would be nice to have a wiki page or
> something showing some best practise in managing them. I know I'll have
> to face this with poky at some point and others are in a similar
> position. I don't think feed management should dictate how OE's
> directory structure works but I'm also not sure saying feed management
> is something external is a good idea either since we're all going to
> have to deal with it at some point.

If OE wants to ensure that developers and users handle feeds correctly,
then the support should (IMHO) be added to OE, so that any bugs in that
handling are fixed in a central place, rather than each distro or
maintainer or developer handling feed generation in multiple
incompatible ways ...

-- Rod

Re: [oe-commits] org.oe.dev rootfs_ipk: as per OE, policy: remove feed management tools

[Øyvind Repvik]

On Sunday 04 March 2007 16:01, Koen Kooi wrote:
> Matthias Hentges schreef:
> > Am Sonntag, den 04.03.2007, 12:31 +0100 schrieb Koen Kooi:
> >> Assuming the above didn't break, what about this:
> >>
> >> 'bitbake package-index' will generate a deploy/bogofeed, everything else
> >> stays as it is now.
> >
> > My reverted cset did exactly the above, it didn't even so much as touch
> > your holy new ipk directory.
>
> But you put it in do_rootfs, where it doesn't belong.

Well, why didn't you say that in the first place, instead of all this silly 
bickering? This argument has been extremely un-constructive, and the only 
reason I can see for your arguments when they are all summed up is that you 
made a mistake and you're trying to cover your ass. Get over it already, move 
it away from do_rootfs, but make it possible to choose.

Regards,
Øyvind Repvik

Re: [oe-commits] org.oe.dev rootfs_ipk: as per OE, policy: remove feed management tools

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Matthias Hentges schreef:
> Am Sonntag, den 04.03.2007, 12:31 +0100 schrieb Koen Kooi:

>> Assuming the above didn't break, what about this:
>>
>> 'bitbake package-index' will generate a deploy/bogofeed, everything else stays as it is now.
> 
> My reverted cset did exactly the above, it didn't even so much as touch
> your holy new ipk directory.

But you put it in do_rootfs, where it doesn't belong.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFF6t8yMkyGM64RGpERAtJyAKCLvpQRXnPVj5fN1A6mmbHtCGfBbgCgqEHU
gPSso5b2k0TVv/uT9cFlRfk=
=DODQ
-----END PGP SIGNATURE-----

Re: [oe-commits] org.oe.dev rootfs_ipk: as per OE, policy: remove feed management tools

[Matthias Hentges]

Am Sonntag, den 04.03.2007, 12:31 +0100 schrieb Koen Kooi:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Richard Purdie schreef:
> > On Sun, 2007-03-04 at 09:43 +0100, Koen Kooi wrote:
> 
> >> So it always required an extra command (either 'bitbake <foo>-image' or 'bitbake
> >> package-index'), since the index wasn't refreshed automagically after you built a package.
> > 
> > Nobody has a problem with that.
> 
> People did seem to have problems with that, citing 'needless post processing' when I said
> you need to run one (1) command to get a feed! But we'll from now on assume people have no
> problems with it.

I have no problem running package-index to re(!)-generate a feed. I've
been using package-index for exactly this purpose for a long time.

> > OE is not just about image generation!
> 
> I didn't say that, I meant that people wanting a Packages.* in deploy/ipk _most likely_
> have more packages there than <foo>-image will generate and hence move out of do_rootfs
> territory.

Indeed. As soon as we leave do_rootfs territory, we kinda sorta enter
feed-generation territory, do we not? What good is a truckload of
compiled stuff if you don't have a sane way of installing it.

> FWIW, image generation is ~5% of what I use OE for.
> 
> 
> > I accept there is a fine line between this and supporting feed
> > generation from OE. The thing is people have been happily using that
> > directory as a feed, not for users but for development work. I don't
> > think its fair to break that when we don't need to.
> 
> It didn't break, it _moved_, the subdirectories can still be used as development feeds.
> Instead of saying
> 
> echo 'src/gz devfeed http://buildbox/tmp/deploy/ipk' > /etc/ipkg/devfeed.conf
> 
> you will now say
> 
> echo 'src/gz devfeed1 http://buildbox/tmp/deploy/ipk/ppc603e' > /etc/ipkg/devfeed1.conf
> echo 'src/gz devfeed2 http://buildbox/tmp/deploy/ipk/efika' > /etc/ipkg/devfeed2.conf

Well, it's 3 feeds for slugos and openmo feed. Basically you get
ipk/ARCH, ipk/MACHINE, and ipk/all.

In addition you'll get a bogus empty x86 feed and an empty ipk/Packages
file....

> Right? Or did the cset *really* break that as people keep claiming?

The cset broke the long-used single feed that many of us were using for
years (!) for the sole benefit of the multi-machine guys. Without a way
to revert to the old way.

That's the whole point of this entire thread. I never had a problem with
the change, only the way it has been implemented.

> Assuming the above didn't break, what about this:
> 
> 'bitbake package-index' will generate a deploy/bogofeed, everything else stays as it is now.

My reverted cset did exactly the above, it didn't even so much as touch
your holy new ipk directory.

Re: [oe-commits] org.oe.dev rootfs_ipk: as per OE, policy: remove feed management tools

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Richard Purdie schreef:
> On Sun, 2007-03-04 at 09:43 +0100, Koen Kooi wrote:

>> So it always required an extra command (either 'bitbake <foo>-image' or 'bitbake
>> package-index'), since the index wasn't refreshed automagically after you built a package.
> 
> Nobody has a problem with that.

People did seem to have problems with that, citing 'needless post processing' when I said
you need to run one (1) command to get a feed! But we'll from now on assume people have no
problems with it.

> OE is not just about image generation!

I didn't say that, I meant that people wanting a Packages.* in deploy/ipk _most likely_
have more packages there than <foo>-image will generate and hence move out of do_rootfs
territory.
FWIW, image generation is ~5% of what I use OE for.


> I accept there is a fine line between this and supporting feed
> generation from OE. The thing is people have been happily using that
> directory as a feed, not for users but for development work. I don't
> think its fair to break that when we don't need to.

It didn't break, it _moved_, the subdirectories can still be used as development feeds.
Instead of saying

echo 'src/gz devfeed http://buildbox/tmp/deploy/ipk' > /etc/ipkg/devfeed.conf

you will now say

echo 'src/gz devfeed1 http://buildbox/tmp/deploy/ipk/ppc603e' > /etc/ipkg/devfeed1.conf
echo 'src/gz devfeed2 http://buildbox/tmp/deploy/ipk/efika' > /etc/ipkg/devfeed2.conf

Right? Or did the cset *really* break that as people keep claiming?

Assuming the above didn't break, what about this:

'bitbake package-index' will generate a deploy/bogofeed, everything else stays as it is now.


regards,

Koen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFF6q4TMkyGM64RGpERAgN/AJ9fb59es7p6p+rJ4Uhwsn+2QUJ6QwCePf/f
Sk9wVjfSY4bvmVEs0jfAeOo=
=wBAr
-----END PGP SIGNATURE-----

Re: [oe-commits] org.oe.dev rootfs_ipk: as per OE, policy: remove feed management tools

[Richard Purdie]

On Sun, 2007-03-04 at 09:43 +0100, Koen Kooi wrote:
> Richard Purdie schreef:
> > On Fri, 2007-03-02 at 10:47 +0100, Koen Kooi wrote:
> >> Again: the ability to use deploy as a feed is still present, it only moved to subdirectories.
> >
> > Which changed a behaviour people were relying on.
> >
> > My view on this is currently that:
> >
> > * Being able to use tmp/deploy/ipk as a feed is a useful feature
> > * we should continue to allow that (as developer use only, not for
> > building real feeds off)
> 
> Sure, but people wanting that can do 'bitbake -b contrib/oe-ipkg-feed.bb'. Building a feed
> not used for rootfs assembling during do_rootfs is plain wrong. Can we at least agree on
> it being out of scope for do_rootfs?

do_rootfs requires a feed at the moment. If its going to create a feed,
we might as well make that feed useful. If a new do_rootfs is created
that doesn't require a feed, I will accept that it need not generate a
feed. We will have package-index for that which is intended to setup
tmp/deploy/ipk as a feed, effectively.

Sticking .bb files in contrib is just plain silly, especially when it
should be doing the same thing as package-index.bb.

I accept there is a fine line between this and supporting feed
generation from OE. The thing is people have been happily using that
directory as a feed, not for users but for development work. I don't
think its fair to break that when we don't need to.

> > Since a number of people want the older single feed behaviour, I think
> > we should have some way of allowing that.
> 
> 'bitbake -b contrib/oe-ipkg-feed.bb' which is only a few chars more as 'bitbake
> package-index'. Same amount of commands and 'post-processing'

But why make this so difficult for people?

> > I understand why people want
> > the split feed behaviour too and we should allow people to select that
> > too. Which should be default I don't really care about.
> 
> People keep saying that do_rootfs should build a feed that can be used as a feed, but what
> is the point in that? OE builds only stuff that's put in the images, so the feed would
> almost be a 1:1 copy of the image you install on your device. 

Eh? I can type bitbake something, then the something packages appear in
deploy. OE isn't just about image generation, that's just what a lot of
people happen to use it for. Or are we going to break anything that
isn't an image target next?

> If you start building more
> packages and run do_rootfs again you're getting into the territory of what you called
> "long standing distro feeds" and associated bugs.

Then that would be a bug which we need to fix in OE. I build tons of
packages every day which are not included in my rootfs and nothing
appears to break. If it did, I would fix the bug.

OE is not just about image generation!

> Think about the situation for a moment:
> 
> * OE builds packages and puts those somewhere
> * OE uses said packages to make a rootfs

Yes, long established behaviour.

> As a side effect you get something you could use a feed after do_rootfs has run.
> Or you ran manually ran 'bitbake package-index'.

Yes, long established behaviour.

> So it always required an extra command (either 'bitbake <foo>-image' or 'bitbake
> package-index'), since the index wasn't refreshed automagically after you built a package.

Nobody has a problem with that.

> However, there is one thing that breaks, which is the upload scripts people were using. My
> suggestion that scripts can be changed didn't go over to well, so I don't have a solution
> for that.

People wanted the old behaviour to be selectable. I don't see why its a
problem. It doesn't appear to hurt anything. Yes, you need to beware how
you use it but that was the case before and is still the case.

> I don't disagree with having a way in OE to generate a feed out of the contents in
> deploy/, but given the problems raised OE should make clear you'll have no warranty and it
> will eat your dog. What about having an OE way that creates 'deploy/bogofeed', in the
> spirit of linux' bogomips?

I think this is now more than well documented on the mailing list and we
can refer anyone having problems to them.

Cheers,

Richard

Re: [oe-commits] org.oe.dev rootfs_ipk: as per OE, policy: remove feed management tools

[Richard Purdie]

On Sun, 2007-03-04 at 01:08 +0100, Matthias Hentges wrote:
> Using a very large ipk feed does indeed work nicely to generate images.
> However, generating Packages can take ages ( and so do image rebuilds
> because of this ). That's about the only problem I see w/ One Giant Feed

We speeded that up quite a bit and there is another speedup planned
which should remove that obstacle. Its been discussed here in the past.

> > My personal experience says multimachine works well and I use if for
> > everything as standard these days. Have you any specific examples or
> > problems? I have been dealing with them as and when I've seen them but I
> > haven't seen any in quite a whiie... 
> 
> I wasn't exactly running into real ipkg bugs. I had problems with
> packages using machine-specific stuff for some models ( say all
> sl-c3x00) and hence having their ARCH set as $MACHINE, but not for other
> models ( like collie, poodle ) having their ARCH set as $ARCH. 
> 
> It's a packaging bug, not an ipkg bug but annoying in any case ;)

Some packages work like this as a feature, ts-conf now being one of
them. The standard one can be overridden by a machine specific package.
It should just require that the ipkg Architectures are in the right
order. There could also be an issue for ipkg deciding whether it should
use an  arch specific package or latest version in a case where packages
that used to be machine specific no longer are. If ipkg can't handle
that, I'd say ipkg needs fixing.

Cheers,

Richard

Re: [oe-commits] org.oe.dev rootfs_ipk: as per OE, policy: remove feed management tools

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Richard Purdie schreef:
> On Fri, 2007-03-02 at 10:47 +0100, Koen Kooi wrote:

>> Again: the ability to use deploy as a feed is still present, it only moved to subdirectories.
>
> Which changed a behaviour people were relying on.
>
> My view on this is currently that:
>
> * Being able to use tmp/deploy/ipk as a feed is a useful feature
> * we should continue to allow that (as developer use only, not for
> building real feeds off)

Sure, but people wanting that can do 'bitbake -b contrib/oe-ipkg-feed.bb'. Building a feed
not used for rootfs assembling during do_rootfs is plain wrong. Can we at least agree on
it being out of scope for do_rootfs?

> Since a number of people want the older single feed behaviour, I think
> we should have some way of allowing that.

'bitbake -b contrib/oe-ipkg-feed.bb' which is only a few chars more as 'bitbake
package-index'. Same amount of commands and 'post-processing'

> I understand why people want
> the split feed behaviour too and we should allow people to select that
> too. Which should be default I don't really care about.

People keep saying that do_rootfs should build a feed that can be used as a feed, but what
is the point in that? OE builds only stuff that's put in the images, so the feed would
almost be a 1:1 copy of the image you install on your device. If you start building more
packages and run do_rootfs again you're getting into the territory of what you called
"long standing distro feeds" and associated bugs.

Think about the situation for a moment:

* OE builds packages and puts those somewhere
* OE uses said packages to make a rootfs

As a side effect you get something you could use a feed after do_rootfs has run.
Or you ran manually ran 'bitbake package-index'.

So it always required an extra command (either 'bitbake <foo>-image' or 'bitbake
package-index'), since the index wasn't refreshed automagically after you built a package.

However, there is one thing that breaks, which is the upload scripts people were using. My
suggestion that scripts can be changed didn't go over to well, so I don't have a solution
for that.

I don't disagree with having a way in OE to generate a feed out of the contents in
deploy/, but given the problems raised OE should make clear you'll have no warranty and it
will eat your dog. What about having an OE way that creates 'deploy/bogofeed', in the
spirit of linux' bogomips?

regards,

Koen

PS: Now I think of it, not using TARGET_VENDOR breaks host = target situations....

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFF6oa+MkyGM64RGpERAmotAJ0ZG5yAEwSgnHmnYSWo06gSPIUGIwCgmZWV
LgJLBfxNp9e1y6A1ipfti7M=
=D0Av
-----END PGP SIGNATURE-----

Re: [oe-commits] org.oe.dev rootfs_ipk: as per OE, policy: remove feed management tools

[Matthias Hentges]

Am Samstag, den 03.03.2007, 22:47 +0000 schrieb Richard Purdie:
> On Sat, 2007-03-03 at 18:19 +0100, Matthias Hentges wrote:
> > > Rod Whitby schreef:
> > > > Stelios Koroneos wrote:
> >  
> > Ahh, finally someone with _real_ arguments in favor of the change. I do
> > feel your pain, I really do. I can see that the change makes life easier
> > for multi-machine / multi-arch builders. I don't have any problems with
> > that at all, only with the way it was implemented.
> 
> Technically a giant pool of ipks should work and if it doesn't it points
> at bugs that need fixing.
> 
> Equally, I don't mind having splitting implemented as per my last mail.
> Lets not try and brush ipkg bugs under the mat though...

Using a very large ipk feed does indeed work nicely to generate images.
However, generating Packages can take ages ( and so do image rebuilds
because of this ). That's about the only problem I see w/ One Giant Feed


> My personal experience says multimachine works well and I use if for
> everything as standard these days. Have you any specific examples or
> problems? I have been dealing with them as and when I've seen them but I
> haven't seen any in quite a whiie... 

I wasn't exactly running into real ipkg bugs. I had problems with
packages using machine-specific stuff for some models ( say all
sl-c3x00) and hence having their ARCH set as $MACHINE, but not for other
models ( like collie, poodle ) having their ARCH set as $ARCH. 

It's a packaging bug, not an ipkg bug but annoying in any case ;)

> > > Rootfs generation continued to work.
> > > However, rootfs_ipk.bbclass is *not* the place to put in feed management. That class is
> > > supposed to build a rootfs from ipkgs, which it does by treating deploy/ipk/<arch> as
> > > feeds. I can think of ways to do rootfs generation without Packages.* (e.g sqlite), and if
> > > they prove to be faster and more robust I _will_ replace the old methods.
> > 
> > The current feed-style "installation" of the rootfs has the huge
> > advantage of checking the feed for broken packages at _image-generation
> > time_.
> > 
> > Removing this important check will likely cause endless pains to feed
> > and distro maintainers for barely any benefit.
> 
> I must admit I fail to see what's wrong with the current image
> generation methods. Yes, we can optimise some of it further but its not
> that bad at the moment. If you don't like ipkg, you can also do it using
> apt now...

Exactly.

Re: [oe-commits] org.oe.dev rootfs_ipk: as per OE, policy: remove feed management tools

[Richard Purdie]

On Sat, 2007-03-03 at 18:19 +0100, Matthias Hentges wrote:
> > Rod Whitby schreef:
> > > Stelios Koroneos wrote:
> > >> I am not sure how many of you are building multimachine distro's but the old
> > >> system was not a very good solution.
> > >> We are building a distro for a number of machines x86 arch (i486, i585,
> > >> i686), a number of powerpc targets  and recently mipsel and puttinng
> > >> everything in a giant ipk pool is not the best of solutions.
> 
> Ahh, finally someone with _real_ arguments in favor of the change. I do
> feel your pain, I really do. I can see that the change makes life easier
> for multi-machine / multi-arch builders. I don't have any problems with
> that at all, only with the way it was implemented.

Technically a giant pool of ipks should work and if it doesn't it points
at bugs that need fixing.

Equally, I don't mind having splitting implemented as per my last mail.
Lets not try and brush ipkg bugs under the mat though...

> Correct again. It is however my _personal_ experience that
> multi-machine.bb causes more problem than it solves (of course YMMV).
> Most of them are caused by faulty/problematic package .bb's (not OE or
> ipkg bugs) but IMO these errors are very hard to catch and it is almost
> impossible to review all 4336 .bb's currently residing in OE.
>
> The only way to build clean and (almost) guaranteed-working feeds is to
> build each MACHINE in its own tmpdir. You'll get many duplicated
> packages for sure but the feed will be consistent.

My personal experience says multimachine works well and I use if for
everything as standard these days. Have you any specific examples or
problems? I have been dealing with them as and when I've seen them but I
haven't seen any in quite a whiie... 

> > Rootfs generation continued to work.
> > However, rootfs_ipk.bbclass is *not* the place to put in feed management. That class is
> > supposed to build a rootfs from ipkgs, which it does by treating deploy/ipk/<arch> as
> > feeds. I can think of ways to do rootfs generation without Packages.* (e.g sqlite), and if
> > they prove to be faster and more robust I _will_ replace the old methods.
> 
> The current feed-style "installation" of the rootfs has the huge
> advantage of checking the feed for broken packages at _image-generation
> time_.
> 
> Removing this important check will likely cause endless pains to feed
> and distro maintainers for barely any benefit.

I must admit I fail to see what's wrong with the current image
generation methods. Yes, we can optimise some of it further but its not
that bad at the moment. If you don't like ipkg, you can also do it using
apt now...

Regards,

Richard

Re: [oe-commits] org.oe.dev rootfs_ipk: as per OE, policy: remove feed management tools

[Richard Purdie]

On Fri, 2007-03-02 at 10:47 +0100, Koen Kooi wrote:
> As I said, OE still generates 'feeds' in subdirs of deploy/ipk, in contrary of what people
> seem to claim on the mailinglist.

What people are claiming in that the format of deploy/ipk changes from a
single to multiple feeds. This is a behaviour change and people relied
on the old behaviour.

> Using deploy/ipk/ as feed will give you the following problems:
> 
> 1) ipkg doesn't handle upgrades properly if the versions between multiple archtectures
> don't match

An ipkg bug which should be filed somewhere...

> 2) ipkg doesn't handle upgrades proberly when you don't use '-m' (OE does now, but I can
> see people 'optimizing' it and removing '-m' to get more speed)

What does the -m option to ipkg do?

> 3) if you rebuild a package and forget to regenerate Packages your target will freak out
> due to mismatched md5sums

That is a usage problem and one that is easily fixed...

> 4) Packages.filelist can only be generated from scratch, so the former
> deploy/ipk/Packages.filelist was broken

What uses that file?

> That are the easy things, lets look at other problems:
> 
> a) shlib renaming madness. Every now and then something goes wrong with shlib renaming,
> e.g. an app starts shipping an extra util and renaming doesn't work anymore. So you end up
> with 'jpeg_6b.ipk' instead of 'libjpeg6_6b.ipk'. This means you have to edit your Packages
> file to add 'Provides: jpeg' since OE will shlib rename 'RPROVIDES += jpeg' to 'Provides:
> libjpeg6'

This is a problem although it mainly affects long standing distro feeds
rather than people using tmp/deploy as a feed which will probably be
unaffected by such a problem.

> b) Packaging changes without PR bump. If you just blindly rsync, different users will get
> different contents of what appears to be the same package.

Again this is a usage problem. I think people using the feeds directly
will be intelligent enough to be able to deal with this. 

> We aren't even mentioning stripping 'Source:' fields to make ipkg consume less RAM and
> storage.

We could probably improve the OE image feed handling to address that.

> So, knowing all this, do people still want to pretend that a single deploy/ipk feed has no
> problems at all and OE should support it?

Nobody is pretending it has no problems, there are some. People are
saying they're prepared to live with those problems as they need the
feature more.

> On the feed-management tools:
> 
> The splitting scripts (to split feeds into gpe/ and opie/ and varous other ways) got
> removed and I was told "it's out of scope" by Chris (not to mention the surprise when
> 'bitbake world' mangles deploy/ipk).

That kind of feed splitting is out of scope. It would belong in contrib.

>  package-index.bb, however serves a different purpose for me: being able to regenerate 
> Packages so I can look at several control files at once,
> instead of using dpkg-deb or ar -x to extract single ones. I'll happily remove it Matthias
> keeps claiming it's a feed management tool.

IMO, it should stay, it is a useful tool.

> Again: the ability to use deploy as a feed is still present, it only moved to subdirectories.

Which changed a behaviour people were relying on.

My view on this is currently that:

* Being able to use tmp/deploy/ipk as a feed is a useful feature
* we should continue to allow that (as developer use only, not for
building real feeds off)

Since a number of people want the older single feed behaviour, I think
we should have some way of allowing that. I understand why people want
the split feed behaviour too and we should allow people to select that
too. Which should be default I don't really care about.

Cheers,

Richard

Re: [oe-commits] org.oe.dev rootfs_ipk: as per OE, policy: remove feed management tools

[Matthias Hentges]

I'm going to spare you a detailed reply to your previous email
regarding problems w/ ipkg feeds (all of which are either OE or ipkg
bugs, very few of which can be effectively caught via any type of
post-processing, rendering yet again your whole argument mute)

Am Freitag, den 02.03.2007, 11:00 +0100 schrieb Koen Kooi:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Rod Whitby schreef:
> > Stelios Koroneos wrote:
> >> I am not sure how many of you are building multimachine distro's but the old
> >> system was not a very good solution.
> >> We are building a distro for a number of machines x86 arch (i486, i585,
> >> i686), a number of powerpc targets  and recently mipsel and puttinng
> >> everything in a giant ipk pool is not the best of solutions.

Ahh, finally someone with _real_ arguments in favor of the change. I do
feel your pain, I really do. I can see that the change makes life easier
for multi-machine / multi-arch builders. I don't have any problems with
that at all, only with the way it was implemented.

> >> As far as i can understand its a case where  single machine distro's made
> >> use of this OE feature while multimachine distros had to look for other ways
> >> (i.e post processing).

Exactly.

> > Perhaps when multimachine was added, the deploy ipk feed support should
> > have been updated to continue to work, instead of later breaking single
> > machine feed support.

Correct again. It is however my _personal_ experience that
multi-machine.bb causes more problem than it solves (of course YMMV).
Most of them are caused by faulty/problematic package .bb's (not OE or
ipkg bugs) but IMO these errors are very hard to catch and it is almost
impossible to review all 4336 .bb's currently residing in OE.

The only way to build clean and (almost) guaranteed-working feeds is to
build each MACHINE in its own tmpdir. You'll get many duplicated
packages for sure but the feed will be consistent.

> Again, it didn't break anything that was *supported*. 

Who decides what is "supported" and what isn't? 
Where can I read black on white what is supported or not?

> Rootfs generation continued to work.
> However, rootfs_ipk.bbclass is *not* the place to put in feed management. That class is
> supposed to build a rootfs from ipkgs, which it does by treating deploy/ipk/<arch> as
> feeds. I can think of ways to do rootfs generation without Packages.* (e.g sqlite), and if
> they prove to be faster and more robust I _will_ replace the old methods.

The current feed-style "installation" of the rootfs has the huge
advantage of checking the feed for broken packages at _image-generation
time_.

Removing this important check will likely cause endless pains to feed
and distro maintainers for barely any benefit.

> So my proposal:
> 
> Make a contrib/deploy-ipkg-feed.bb that basically does:
> 
> cd ${DEPLOY_DIR_IPK}
> find */ -name  "*.ipk" -exec cp  '{}'  ./ \;
> rm Packages*
> ipkg-make-index -p Packages -l Packages.filelist -m .
> 
> and add the problems mentioned in other mails as comments.
> 
> This way people wanting a single feed get it with a simple 'bitbake -b
> contrib/deploy-ipkg-feed.bb' without the impression it's officially supported in OE.
> 
> Does anyone have problems with this proposal?

Well, since you asked...

- Putting comments inside a .bb file is less-than-ideal as nobody opens
up a .bb when it's working as expected. oe.warn() might be what you are
looking for. Not that I agree that there is anything a user should be
"warned" of when building a feed. 

- Even if you call it a ".bb", it is just a script for _manual_
post-processing. Exactly how is this different than requiring the user
to run as shell script to generate the feed? Hint: It isn't.

- The only acceptable way is an automatic method, integrated into the
image-generation process as before, which the user can enable/disable if
he choose to.

Not that there is much of a reason to disable this perfectly fine
feature since oe-feed could live happily side-by-side w/ deploy/ipk....

But for some reason you (and as I understand it, it is _only_ you) have
a problem with that. 

You reverted my _optional_ OFF BY DEFAULT _compromise_ cset, you even
revoked my commit access to OE ( until someone intervened, that is )

For every one else who would like his once-working oe feed back, I might
suggest applying the oe-feed patch at
http://stuff.pimp-your-penguin.org/oe-patches/dev/

It is basically the cset koen reverted, usage instructions inside.
Should even work fine for multi-machine and multi-arch builds w/ barely
any added time to the feed-generation process. Oh, it lives in
deploy/oe-feed and is triggered by package-index.bb and image creation.

Re: [oe-commits] org.oe.dev rootfs_ipk: as per OE, policy: remove feed management tools

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Rod Whitby schreef:
> Stelios Koroneos wrote:
>> I am not sure how many of you are building multimachine distro's but the old
>> system was not a very good solution.
>> We are building a distro for a number of machines x86 arch (i486, i585,
>> i686), a number of powerpc targets  and recently mipsel and puttinng
>> everything in a giant ipk pool is not the best of solutions.
>>
>> As far as i can understand its a case where  single machine distro's made
>> use of this OE feature while multimachine distros had to look for other ways
>> (i.e post processing).
> 
> Perhaps when multimachine was added, the deploy ipk feed support should
> have been updated to continue to work, instead of later breaking single
> machine feed support.

Again, it didn't break anything that was *supported*. Rootfs generation continued to work.
However, rootfs_ipk.bbclass is *not* the place to put in feed management. That class is
supposed to build a rootfs from ipkgs, which it does by treating deploy/ipk/<arch> as
feeds. I can think of ways to do rootfs generation without Packages.* (e.g sqlite), and if
they prove to be faster and more robust I _will_ replace the old methods.

So my proposal:

Make a contrib/deploy-ipkg-feed.bb that basically does:

cd ${DEPLOY_DIR_IPK}
find */ -name  "*.ipk" -exec cp  '{}'  ./ \;
rm Packages*
ipkg-make-index -p Packages -l Packages.filelist -m .

and add the problems mentioned in other mails as comments.

This way people wanting a single feed get it with a simple 'bitbake -b
contrib/deploy-ipkg-feed.bb' without the impression it's officially supported in OE.

Does anyone have problems with this proposal?

regards,

Koen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFF5/XJMkyGM64RGpERAgtvAKChIZPrfuunIzm/pk7t83yim5zo0gCfQMli
OfEiT2MTSgKrxADF0pZhdw4=
=eWES
-----END PGP SIGNATURE-----

Re: [oe-commits] org.oe.dev rootfs_ipk: as per OE, policy: remove feed management tools

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Andy Wilcox schreef:
> Mickey writes:
>> Guys, please don't play games. Since it is a handy feature indeed,
>> can we perhaps agree on making this a seperate bbclass to keep both of
>> you satisfied?
>>   
> Agreed.  Any big behavior change like this should have much more discussion.
> It might be wrong, but sometimes bugs become features.  Get used to it.
> 
> Feed generation is one way I sold OE internally and one reason why
> it is so terribly useful.  If it has to come out for some policy, then 
> perhaps the
> policy should be revisited.

As I said, OE still generates 'feeds' in subdirs of deploy/ipk, in contrary of what people
seem to claim on the mailinglist.

Using deploy/ipk/ as feed will give you the following problems:

1) ipkg doesn't handle upgrades properly if the versions between multiple archtectures
don't match
2) ipkg doesn't handle upgrades proberly when you don't use '-m' (OE does now, but I can
see people 'optimizing' it and removing '-m' to get more speed)
3) if you rebuild a package and forget to regenerate Packages your target will freak out
due to mismatched md5sums
4) Packages.filelist can only be generated from scratch, so the former
deploy/ipk/Packages.filelist was broken

That are the easy things, lets look at other problems:

a) shlib renaming madness. Every now and then something goes wrong with shlib renaming,
e.g. an app starts shipping an extra util and renaming doesn't work anymore. So you end up
with 'jpeg_6b.ipk' instead of 'libjpeg6_6b.ipk'. This means you have to edit your Packages
file to add 'Provides: jpeg' since OE will shlib rename 'RPROVIDES += jpeg' to 'Provides:
libjpeg6'
b) Packaging changes without PR bump. If you just blindly rsync, different users will get
different contents of what appears to be the same package.

We aren't even mentioning stripping 'Source:' fields to make ipkg consume less RAM and
storage.

So, knowing all this, do people still want to pretend that a single deploy/ipk feed has no
problems at all and OE should support it?

On the feed-management tools:

The splitting scripts (to split feeds into gpe/ and opie/ and varous other ways) got
removed and I was told "it's out of scope" by Chris (not to mention the surprise when
'bitbake world' mangles deploy/ipk). package-index.bb, however serves a different purpose
for me: being able to regenerate Packages so I can look at several control files at once,
instead of using dpkg-deb or ar -x to extract single ones. I'll happily remove it Matthias
keeps claiming it's a feed management tool.

Again: the ability to use deploy as a feed is still present, it only moved to subdirectories.

regards,

Koen

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFF5/LEMkyGM64RGpERAlvcAJ0cMcDEBXGznkAUTbV+aBA4+HmJrgCgs6J5
iwQP4Ko2H18hcZTB/VnLoN4=
=Zdv6
-----END PGP SIGNATURE-----

Re: [oe-commits] org.oe.dev rootfs_ipk: as per OE, policy: remove feed management tools

[Rod Whitby]

Stelios Koroneos wrote:
> I am not sure how many of you are building multimachine distro's but the old
> system was not a very good solution.
> We are building a distro for a number of machines x86 arch (i486, i585,
> i686), a number of powerpc targets  and recently mipsel and puttinng
> everything in a giant ipk pool is not the best of solutions.
> 
> As far as i can understand its a case where  single machine distro's made
> use of this OE feature while multimachine distros had to look for other ways
> (i.e post processing).

Perhaps when multimachine was added, the deploy ipk feed support should
have been updated to continue to work, instead of later breaking single
machine feed support.

Just because it works for some, doesn't mean it works for all.  Just
because it doesn't work for some, doesn't mean it's not useful for others.

> From what i read so far there seems to be 2 solutions
> Either allow for the old behaviur of OE through some 'switch' and let the
> distro mainter decide what do to or document the new way of handling feeds.
> In my case i would prefer the second option, as both single and multimachine
> distro's could use the same mechanism to update their feeds

The third (and my preferred) option is not just to document the new way
of handling feeds, but fix OE to handle feeds in the multimachine case.
 OE is meant to make things easier and more consistent, not meant to
force people to *all* do postprocessing to do a common action (like
having a development feed or a simply rsync'ed public feed).

If a feature is commonly used, and fervently wanted by those who use it,
then breaking or deleting it is not a way to keep developers using OE
... especially in corporate situations like Andy recounted, where ease
of development feeds is a selling feature internally against other
home-grown build solutions.

-- Rod

Re: [oe-commits] org.oe.dev rootfs_ipk: as per OE, policy: remove feed management tools

[Stelios Koroneos]

Just my 2 euro cents

I am not sure how many of you are building multimachine distro's but the old
system was not a very good solution.
We are building a distro for a number of machines x86 arch (i486, i585,
i686), a number of powerpc targets  and recently mipsel and puttinng
everything in a giant ipk pool is not the best of solutions.

As far as i can understand its a case where  single machine distro's made
use of this OE feature while multimachine distros had to look for other ways
(i.e post processing).
With the recent changes it seems that single machine distro's also need to
do some post-processing in order to get going and people are upset about
that

From what i read so far there seems to be 2 solutions
Either allow for the old behaviur of OE through some 'switch' and let the
distro mainter decide what do to or document the new way of handling feeds.
In my case i would prefer the second option, as both single and multimachine
distro's could use the same mechanism to update their feeds

Stelios

Re: [oe-commits] org.oe.dev rootfs_ipk: as per OE, policy: remove feed management tools

[Andy Wilcox]

Mickey writes:
>
> Guys, please don't play games. Since it is a handy feature indeed,
> can we perhaps agree on making this a seperate bbclass to keep both of
> you satisfied?
>   
Agreed.  Any big behavior change like this should have much more discussion.
It might be wrong, but sometimes bugs become features.  Get used to it.

Feed generation is one way I sold OE internally and one reason why
it is so terribly useful.  If it has to come out for some policy, then 
perhaps the
policy should be revisited.

Kind Regards,
Andy