RFE: make the init manager an image feature (again)

RFE: make the init manager an image feature (again)

[Enrico Scholz]

Hello,

it would be nice when the decision to make the init manager a distribution
feature will be reverted to the old oe-meta mechanism.

Being a distribution feature means, that packages are created in such a
way that it is impossible to split off unwanted and heavy weighted
functionality at image creation time.

E.g. on most of my systems, I create two kinds of images: a full
featured, systemd based one and a very minimal rescue system with
busybox and some filesystem utilities.  With recent systemd packaging
change, the rescue image size grow up from 5.9 MiB to 27 MiB because
systemd dependencies are hardcoded in mandatory packages.

Formerly, systemd dependencies could be avoided by adding the -systemd
packages to BAD_RECOMMENDATIONS (e.g. due to busybox-syslog ->
busybox-syslog-systemd rrecommend).

I am aware that initscripts were always part of the main package.  But
sysvinit was very lightweighted and the extra space either negligible or
easy to recover by removing some files in IMAGE_PREPROCESS_COMMAND.

Hence my recommendation: make the init manager an image feature again
and create -systemd and -sysv packages with the corresponding scripts.
OpenEmbedded is still for embedded devices where size matters.


Of course, systemd can be still a distribution feature to enable things
like socket activation as part of PACKAGE_CONFIG.  But dependencies on
init system packages should be RRECOMMENDS which can be overridden
easily at image creation time.



Enrico

Re: RFE: make the init manager an image feature (again)

[Otavio Salvador]

On Fri, Feb 15, 2013 at 4:19 PM, Enrico Scholz
<enrico.scholz@sigma-chemnitz.de> wrote:
> Hello,
>
> it would be nice when the decision to make the init manager a distribution
> feature will be reverted to the old oe-meta mechanism.
>
> Being a distribution feature means, that packages are created in such a
> way that it is impossible to split off unwanted and heavy weighted
> functionality at image creation time.
>
> E.g. on most of my systems, I create two kinds of images: a full
> featured, systemd based one and a very minimal rescue system with
> busybox and some filesystem utilities.  With recent systemd packaging
> change, the rescue image size grow up from 5.9 MiB to 27 MiB because
> systemd dependencies are hardcoded in mandatory packages.
>
> Formerly, systemd dependencies could be avoided by adding the -systemd
> packages to BAD_RECOMMENDATIONS (e.g. due to busybox-syslog ->
> busybox-syslog-systemd rrecommend).
>
> I am aware that initscripts were always part of the main package.  But
> sysvinit was very lightweighted and the extra space either negligible or
> easy to recover by removing some files in IMAGE_PREPROCESS_COMMAND.
>
> Hence my recommendation: make the init manager an image feature again
> and create -systemd and -sysv packages with the corresponding scripts.
> OpenEmbedded is still for embedded devices where size matters.
>
>
> Of course, systemd can be still a distribution feature to enable things
> like socket activation as part of PACKAGE_CONFIG.  But dependencies on
> init system packages should be RRECOMMENDS which can be overridden
> easily at image creation time.

I fully support this! I also want this flexibility back (in fact I see
no reason why it has been dropped).

-- 
Otavio Salvador                             O.S. Systems
E-mail: otavio@ossystems.com.br  http://www.ossystems.com.br
Mobile: +55 53 9981-7854              http://projetos.ossystems.com.br

Re: RFE: make the init manager an image feature (again)

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1979 bytes --]

On Fri, Feb 15, 2013 at 04:47:37PM -0200, Otavio Salvador wrote:
> On Fri, Feb 15, 2013 at 4:19 PM, Enrico Scholz
> <enrico.scholz@sigma-chemnitz.de> wrote:
> > Hello,
> >
> > it would be nice when the decision to make the init manager a distribution
> > feature will be reverted to the old oe-meta mechanism.
> >
> > Being a distribution feature means, that packages are created in such a
> > way that it is impossible to split off unwanted and heavy weighted
> > functionality at image creation time.
> >
> > E.g. on most of my systems, I create two kinds of images: a full
> > featured, systemd based one and a very minimal rescue system with
> > busybox and some filesystem utilities.  With recent systemd packaging
> > change, the rescue image size grow up from 5.9 MiB to 27 MiB because
> > systemd dependencies are hardcoded in mandatory packages.
> >
> > Formerly, systemd dependencies could be avoided by adding the -systemd
> > packages to BAD_RECOMMENDATIONS (e.g. due to busybox-syslog ->
> > busybox-syslog-systemd rrecommend).
> >
> > I am aware that initscripts were always part of the main package.  But
> > sysvinit was very lightweighted and the extra space either negligible or
> > easy to recover by removing some files in IMAGE_PREPROCESS_COMMAND.
> >
> > Hence my recommendation: make the init manager an image feature again
> > and create -systemd and -sysv packages with the corresponding scripts.
> > OpenEmbedded is still for embedded devices where size matters.
> >
> >
> > Of course, systemd can be still a distribution feature to enable things
> > like socket activation as part of PACKAGE_CONFIG.  But dependencies on
> > init system packages should be RRECOMMENDS which can be overridden
> > easily at image creation time.
> 
> I fully support this! I also want this flexibility back (in fact I see
> no reason why it has been dropped).

me too

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: RFE: make the init manager an image feature (again)

[Richard Purdie]

On Fri, 2013-02-15 at 19:19 +0100, Enrico Scholz wrote:
> it would be nice when the decision to make the init manager a distribution
> feature will be reverted to the old oe-meta mechanism.
> 
> Being a distribution feature means, that packages are created in such a
> way that it is impossible to split off unwanted and heavy weighted
> functionality at image creation time.
> 
> E.g. on most of my systems, I create two kinds of images: a full
> featured, systemd based one and a very minimal rescue system with
> busybox and some filesystem utilities.  With recent systemd packaging
> change, the rescue image size grow up from 5.9 MiB to 27 MiB because
> systemd dependencies are hardcoded in mandatory packages.
> 
> Formerly, systemd dependencies could be avoided by adding the -systemd
> packages to BAD_RECOMMENDATIONS (e.g. due to busybox-syslog ->
> busybox-syslog-systemd rrecommend).
> 
> I am aware that initscripts were always part of the main package.  But
> sysvinit was very lightweighted and the extra space either negligible or
> easy to recover by removing some files in IMAGE_PREPROCESS_COMMAND.
> 
> Hence my recommendation: make the init manager an image feature again
> and create -systemd and -sysv packages with the corresponding scripts.
> OpenEmbedded is still for embedded devices where size matters.
>
> Of course, systemd can be still a distribution feature to enable things
> like socket activation as part of PACKAGE_CONFIG.  But dependencies on
> init system packages should be RRECOMMENDS which can be overridden
> easily at image creation time.

The trouble is that by making it an "image feature", people will expect
*everything* to work properly and to be able to have fully functional
sysvinit and systemd variants of images. We already see this
expectation. Trying to explain to people what the limitations are, what
is expected to work and what isn't will be difficult. I know you
understand this but going forward most people will simply not and I
think it will be a nightmare for usability.

For that reason I'd rather see this done in a different way, for example
blacklisting the problematic systemd dependencies at image generation
time with some kind of stronger BAD_RECOMMENDS code. Yes, this means
there would be some systemd config files lying around but you'd stop the
main bulk of it coming in (and as you said, some small config files
aren't an issue).

Cheers,

Richard

Re: RFE: make the init manager an image feature (again)

[Otavio Salvador]

On Sat, Feb 16, 2013 at 7:15 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Fri, 2013-02-15 at 19:19 +0100, Enrico Scholz wrote:
>> it would be nice when the decision to make the init manager a distribution
>> feature will be reverted to the old oe-meta mechanism.
>>
>> Being a distribution feature means, that packages are created in such a
>> way that it is impossible to split off unwanted and heavy weighted
>> functionality at image creation time.
>>
>> E.g. on most of my systems, I create two kinds of images: a full
>> featured, systemd based one and a very minimal rescue system with
>> busybox and some filesystem utilities.  With recent systemd packaging
>> change, the rescue image size grow up from 5.9 MiB to 27 MiB because
>> systemd dependencies are hardcoded in mandatory packages.
>>
>> Formerly, systemd dependencies could be avoided by adding the -systemd
>> packages to BAD_RECOMMENDATIONS (e.g. due to busybox-syslog ->
>> busybox-syslog-systemd rrecommend).
>>
>> I am aware that initscripts were always part of the main package.  But
>> sysvinit was very lightweighted and the extra space either negligible or
>> easy to recover by removing some files in IMAGE_PREPROCESS_COMMAND.
>>
>> Hence my recommendation: make the init manager an image feature again
>> and create -systemd and -sysv packages with the corresponding scripts.
>> OpenEmbedded is still for embedded devices where size matters.
>>
>> Of course, systemd can be still a distribution feature to enable things
>> like socket activation as part of PACKAGE_CONFIG.  But dependencies on
>> init system packages should be RRECOMMENDS which can be overridden
>> easily at image creation time.
>
> The trouble is that by making it an "image feature", people will expect
> *everything* to work properly and to be able to have fully functional
> sysvinit and systemd variants of images. We already see this
> expectation. Trying to explain to people what the limitations are, what
> is expected to work and what isn't will be difficult. I know you
> understand this but going forward most people will simply not and I
> think it will be a nightmare for usability.

You said you already see it so what is the change? For me less
flexibility is more problem and the result are more ugly hacks. I've
been using the systemd system in meta-oe in product for long time and
it works fine. In same distro I build other product with is sysvinit
based and it also works fine.

Now I have a nightmare in upgrade path, a change in the way my
customer work and as a plus the need to make two different
distributions for same product. No sense to me.

Final result? This product won't go out of danny as I won't be able to
provide an usable upgrade path without forking OE or having an insane
amount of bbappend hacks.

> For that reason I'd rather see this done in a different way, for example
> blacklisting the problematic systemd dependencies at image generation
> time with some kind of stronger BAD_RECOMMENDS code. Yes, this means
> there would be some systemd config files lying around but you'd stop the
> main bulk of it coming in (and as you said, some small config files
> aren't an issue).

Check the amount of changes for do_install_append has sent today. This
was all not need with the code we had and which works fine. This is
just duplicated code all over recipes. Seriously the systemd
integration was done in complete wrong way in my POV. It broght up
problems we have fixed and do major improvements in meta-oe ... at no
profit.

I know it is not the better thing to read but I think it is nice when
we receive feedback as it allows for rethink of our path of work.

Regards,

-- 
Otavio Salvador                             O.S. Systems
E-mail: otavio@ossystems.com.br  http://www.ossystems.com.br
Mobile: +55 53 9981-7854              http://projetos.ossystems.com.br

Re: RFE: make the init manager an image feature (again)

[Enrico Scholz]

Richard Purdie <richard.purdie@linuxfoundation.org> writes:

>> it would be nice when the decision to make the init manager a distribution
>> feature will be reverted to the old oe-meta mechanism.
>
> The trouble is that by making it an "image feature", people will
> expect *everything* to work properly and to be able to have fully
> functional sysvinit and systemd variants of images.

I do not see an obvious reason why fully functional sysvinit, systemd and
perhaps upstart image variants based on the same distribution/package set
are impossible.

Of course, not "everything" will work.  But initmgr being a distribution
feature makes some things completely impossible.


> We already see this expectation.

IMO, removal of features just to lower expectations is the completely
wrong way.


> Trying to explain to people what the limitations are, what is expected
> to work and what isn't will be difficult.

OpenEmbedded is not an end-user distribution but for people who are
willing to invest some learning effort.  Trying to limit ourself on the
lowest common ground is not desirable imo.


> For that reason I'd rather see this done in a different way, for
> example blacklisting the problematic systemd dependencies at image
> generation time with some kind of stronger BAD_RECOMMENDS code.

Assuming we are able to break the hard dependencies, what is with package
scripts which require programs, files or directories from these deps?  Do
we need a way to differ between good and bad script failures then?

Sounds extremely hacky and fragile...


Enrico

Re: RFE: make the init manager an image feature (again)

[Richard Purdie]

On Sat, 2013-02-16 at 12:57 +0100, Enrico Scholz wrote:
> Richard Purdie <richard.purdie@linuxfoundation.org> writes:
> >> it would be nice when the decision to make the init manager a distribution
> >> feature will be reverted to the old oe-meta mechanism.
> >
> > The trouble is that by making it an "image feature", people will
> > expect *everything* to work properly and to be able to have fully
> > functional sysvinit and systemd variants of images.
> 
> I do not see an obvious reason why fully functional sysvinit, systemd and
> perhaps upstart image variants based on the same distribution/package set
> are impossible.
> 
> Of course, not "everything" will work.  But initmgr being a distribution
> feature makes some things completely impossible.
>
> > We already see this expectation.
> 
> IMO, removal of features just to lower expectations is the completely
> wrong way.

meta-oe earned a *horrendous* reputation because of the way systemd was
implemented there. I believe (as do a number of others) that it has
damaged OE's reputation and usability and actively hurts new users. Yes,
the people who use systemd and meta-oe were happy. People who didn't
want systemd were not. There continues to be a fairly toxic mix of
distro policy mingled in with the recipes in there although good
progress is being made in fixing that and I'm grateful to the people
who've noticed and taken on that work (and done previous work like the
separation of meta-systemd).

It was clear systemd needed to move into the core but also become more
configurable to work for everyone. I don't want to remove features, I do
want to talk about whether there is a better way we can fulfil certain
uses cases, particularly with a focus on usability.

> > Trying to explain to people what the limitations are, what is expected
> > to work and what isn't will be difficult.
> 
> OpenEmbedded is not an end-user distribution but for people who are
> willing to invest some learning effort.  Trying to limit ourself on the
> lowest common ground is not desirable imo.

I did not say we're not going to support your use case. I'm asking if we
can summarise exactly what the problem is and whether there is another
way we can get there which isn't going to surprise as many people and be
easier to use.

I'm actually moderately annoyed that throughout the various discussions
about systemd and how we'd get it into OE-Core nobody actually mentioned
these specific requirements until very late in the implementation.

At the bare minimum, we actually need to document the usecases people
are using and require. Yes, you know your usecase but you need to take
some responsibility for ensuring its documented and known about else it
will continue to get broken time and time again.

> > For that reason I'd rather see this done in a different way, for
> > example blacklisting the problematic systemd dependencies at image
> > generation time with some kind of stronger BAD_RECOMMENDS code.
> 
> Assuming we are able to break the hard dependencies, what is with package
> scripts which require programs, files or directories from these deps?  Do
> we need a way to differ between good and bad script failures then?

At the technical level there is little difference from packaging these
things separately to avoid specific dependencies and explicitly telling
the package manager to avoid that dependency instead.

Equally there are other cases where people do want to break specific
dependencies so this could help us in that area too.

Perhaps there is a dependency we need to drop to a Recommends level,
then use BAD_RECOMMENDS to handy that. We do need to document why we do
that and how it is expected to be used.

> Sounds extremely hacky and fragile...

It depends on the implementation.

Having initscripts in individual packages and having to play tricks to
ensure the right sets of packages get installed is rather hacky and
fragile.

Cheers,

Richard

Re: RFE: make the init manager an image feature (again)

[Richard Purdie]

On Sat, 2013-02-16 at 08:47 -0200, Otavio Salvador wrote:
> On Sat, Feb 16, 2013 at 7:15 AM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > On Fri, 2013-02-15 at 19:19 +0100, Enrico Scholz wrote:
> >> it would be nice when the decision to make the init manager a distribution
> >> feature will be reverted to the old oe-meta mechanism.
> >>
> >> Being a distribution feature means, that packages are created in such a
> >> way that it is impossible to split off unwanted and heavy weighted
> >> functionality at image creation time.
> >>
> >> E.g. on most of my systems, I create two kinds of images: a full
> >> featured, systemd based one and a very minimal rescue system with
> >> busybox and some filesystem utilities.  With recent systemd packaging
> >> change, the rescue image size grow up from 5.9 MiB to 27 MiB because
> >> systemd dependencies are hardcoded in mandatory packages.
> >>
> >> Formerly, systemd dependencies could be avoided by adding the -systemd
> >> packages to BAD_RECOMMENDATIONS (e.g. due to busybox-syslog ->
> >> busybox-syslog-systemd rrecommend).
> >>
> >> I am aware that initscripts were always part of the main package.  But
> >> sysvinit was very lightweighted and the extra space either negligible or
> >> easy to recover by removing some files in IMAGE_PREPROCESS_COMMAND.
> >>
> >> Hence my recommendation: make the init manager an image feature again
> >> and create -systemd and -sysv packages with the corresponding scripts.
> >> OpenEmbedded is still for embedded devices where size matters.
> >>
> >> Of course, systemd can be still a distribution feature to enable things
> >> like socket activation as part of PACKAGE_CONFIG.  But dependencies on
> >> init system packages should be RRECOMMENDS which can be overridden
> >> easily at image creation time.
> >
> > The trouble is that by making it an "image feature", people will expect
> > *everything* to work properly and to be able to have fully functional
> > sysvinit and systemd variants of images. We already see this
> > expectation. Trying to explain to people what the limitations are, what
> > is expected to work and what isn't will be difficult. I know you
> > understand this but going forward most people will simply not and I
> > think it will be a nightmare for usability.
> 
> You said you already see it so what is the change? For me less
> flexibility is more problem and the result are more ugly hacks. I've
> been using the systemd system in meta-oe in product for long time and
> it works fine. In same distro I build other product with is sysvinit
> based and it also works fine.
> 
> Now I have a nightmare in upgrade path, a change in the way my
> customer work and as a plus the need to make two different
> distributions for same product. No sense to me.
> 
> Final result? This product won't go out of danny as I won't be able to
> provide an usable upgrade path without forking OE or having an insane
> amount of bbappend hacks.

When systemd was added to meta-oe, it was clearly mentioned that it was
a proof of concept to experiment with integrating it and subject to
change until it made it into OE-Core. The fact is is changing is
therefore not a surprise. We do need to think about making sure we have
equivalent functionality, we also need to think about migration but the
fact changes happened with some experimental work should not surprise
people. I appreciate that isn't ideal but it is the way it is.

> > For that reason I'd rather see this done in a different way, for example
> > blacklisting the problematic systemd dependencies at image generation
> > time with some kind of stronger BAD_RECOMMENDS code. Yes, this means
> > there would be some systemd config files lying around but you'd stop the
> > main bulk of it coming in (and as you said, some small config files
> > aren't an issue).
> 
> Check the amount of changes for do_install_append has sent today. This
> was all not need with the code we had and which works fine. This is
> just duplicated code all over recipes. Seriously the systemd
> integration was done in complete wrong way in my POV. It broght up
> problems we have fixed and do major improvements in meta-oe ... at no
> profit.

Sorry you feel that way. I'm equally frustrated with the way systemd was
added to meta-oe and the damage it did to the overall usability of the
OE ecosystem. I'm trying to take positive actions from that such as
ensuring we never do something like it again. 

As for the integration into the core, I haven't been too involved with
that and perhaps I should be. The trouble is I alone don't scale and I'm
trying to ensure we have more people taking on this kind of work. This
means letting others take it on and learn. There is a requirement from
the community to ensure people working on improvements like this have
all the needed data and that is something which I don't think happened
well here. My point is that I think there have been failings in various
places and the community needs to take some responsibility too.

On the specifics of the do_install_append, you've seen my comments about
how we're not learning from past mistakes with the way the do_install in
the class was written. I note Phil also agreed with them, both of us
remembering some of the horrors we've dealt with in the past (and
binconfig.bbclass is still around, sadly).

One of the goals of OE-Core is to improve quality. I appreciate in this
case you disagree about the "quality" of those bbappends but I still
believe that approach will be better in the long term as upstream
software starts shipping systemd support. Short term its uglier, I
totally agree. Part of my role in things is to look beyond the short
term, even if people don't want to hear that.

> I know it is not the better thing to read but I think it is nice when
> we receive feedback as it allows for rethink of our path of work.

See above as there is some feedback there too ;-)

I'd also note that its ELC next week and I'll be travelling. Lack of
response doesn't mean I don't care about this, I do. There are only so
many hours in the day though and whilst at ELC, I will need to focus
primarily on that.

Cheers,

Richard

Re: RFE: make the init manager an image feature (again)

[Otavio Salvador]

On Sat, Feb 16, 2013 at 10:34 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Sat, 2013-02-16 at 12:57 +0100, Enrico Scholz wrote:
>> Richard Purdie <richard.purdie@linuxfoundation.org> writes:
>> >> it would be nice when the decision to make the init manager a distribution
>> >> feature will be reverted to the old oe-meta mechanism.
>> >
>> > The trouble is that by making it an "image feature", people will
>> > expect *everything* to work properly and to be able to have fully
>> > functional sysvinit and systemd variants of images.
>>
>> I do not see an obvious reason why fully functional sysvinit, systemd and
>> perhaps upstart image variants based on the same distribution/package set
>> are impossible.
>>
>> Of course, not "everything" will work.  But initmgr being a distribution
>> feature makes some things completely impossible.
>>
>> > We already see this expectation.
>>
>> IMO, removal of features just to lower expectations is the completely
>> wrong way.
>
> meta-oe earned a *horrendous* reputation because of the way systemd was
> implemented there. I believe (as do a number of others) that it has
> damaged OE's reputation and usability and actively hurts new users. Yes,
> the people who use systemd and meta-oe were happy. People who didn't
> want systemd were not. There continues to be a fairly toxic mix of
> distro policy mingled in with the recipes in there although good
> progress is being made in fixing that and I'm grateful to the people
> who've noticed and taken on that work (and done previous work like the
> separation of meta-systemd).

In the begging yes. This has been fixed in meta-oe/meta-systemd split
and it allowed for both Worlds to be in peace again.

> It was clear systemd needed to move into the core but also become more
> configurable to work for everyone. I don't want to remove features, I do
> want to talk about whether there is a better way we can fulfil certain
> uses cases, particularly with a focus on usability.

Agreed; but removing the -systemd packages this removed features and
broke upgrade path for everyone using it. Plus it enforces the
deployment of systemd things in every image so now we cannot have
images with sysv OR systemd anymore.

>> > Trying to explain to people what the limitations are, what is expected
>> > to work and what isn't will be difficult.
>>
>> OpenEmbedded is not an end-user distribution but for people who are
>> willing to invest some learning effort.  Trying to limit ourself on the
>> lowest common ground is not desirable imo.
>
> I did not say we're not going to support your use case. I'm asking if we
> can summarise exactly what the problem is and whether there is another
> way we can get there which isn't going to surprise as many people and be
> easier to use.
>
> I'm actually moderately annoyed that throughout the various discussions
> about systemd and how we'd get it into OE-Core nobody actually mentioned
> these specific requirements until very late in the implementation.

Sorry Richard, I talked to people about it. If you check the first
version of patchset I complained about it but it was ignored.

> At the bare minimum, we actually need to document the usecases people
> are using and require. Yes, you know your usecase but you need to take
> some responsibility for ensuring its documented and known about else it
> will continue to get broken time and time again.

The use-case were covered by meta-oe/meta-systemd very well. So it was
documented and working.

-- 
Otavio Salvador                             O.S. Systems
E-mail: otavio@ossystems.com.br  http://www.ossystems.com.br
Mobile: +55 53 9981-7854              http://projetos.ossystems.com.br

Re: RFE: make the init manager an image feature (again)

[Otavio Salvador]

On Sat, Feb 16, 2013 at 10:53 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Sat, 2013-02-16 at 08:47 -0200, Otavio Salvador wrote:
>> On Sat, Feb 16, 2013 at 7:15 AM, Richard Purdie
>> <richard.purdie@linuxfoundation.org> wrote:
>> > On Fri, 2013-02-15 at 19:19 +0100, Enrico Scholz wrote:
>> >> it would be nice when the decision to make the init manager a distribution
>> >> feature will be reverted to the old oe-meta mechanism.
>> >>
>> >> Being a distribution feature means, that packages are created in such a
>> >> way that it is impossible to split off unwanted and heavy weighted
>> >> functionality at image creation time.
>> >>
>> >> E.g. on most of my systems, I create two kinds of images: a full
>> >> featured, systemd based one and a very minimal rescue system with
>> >> busybox and some filesystem utilities.  With recent systemd packaging
>> >> change, the rescue image size grow up from 5.9 MiB to 27 MiB because
>> >> systemd dependencies are hardcoded in mandatory packages.
>> >>
>> >> Formerly, systemd dependencies could be avoided by adding the -systemd
>> >> packages to BAD_RECOMMENDATIONS (e.g. due to busybox-syslog ->
>> >> busybox-syslog-systemd rrecommend).
>> >>
>> >> I am aware that initscripts were always part of the main package.  But
>> >> sysvinit was very lightweighted and the extra space either negligible or
>> >> easy to recover by removing some files in IMAGE_PREPROCESS_COMMAND.
>> >>
>> >> Hence my recommendation: make the init manager an image feature again
>> >> and create -systemd and -sysv packages with the corresponding scripts.
>> >> OpenEmbedded is still for embedded devices where size matters.
>> >>
>> >> Of course, systemd can be still a distribution feature to enable things
>> >> like socket activation as part of PACKAGE_CONFIG.  But dependencies on
>> >> init system packages should be RRECOMMENDS which can be overridden
>> >> easily at image creation time.
>> >
>> > The trouble is that by making it an "image feature", people will expect
>> > *everything* to work properly and to be able to have fully functional
>> > sysvinit and systemd variants of images. We already see this
>> > expectation. Trying to explain to people what the limitations are, what
>> > is expected to work and what isn't will be difficult. I know you
>> > understand this but going forward most people will simply not and I
>> > think it will be a nightmare for usability.
>>
>> You said you already see it so what is the change? For me less
>> flexibility is more problem and the result are more ugly hacks. I've
>> been using the systemd system in meta-oe in product for long time and
>> it works fine. In same distro I build other product with is sysvinit
>> based and it also works fine.
>>
>> Now I have a nightmare in upgrade path, a change in the way my
>> customer work and as a plus the need to make two different
>> distributions for same product. No sense to me.
>>
>> Final result? This product won't go out of danny as I won't be able to
>> provide an usable upgrade path without forking OE or having an insane
>> amount of bbappend hacks.
>
> When systemd was added to meta-oe, it was clearly mentioned that it was
> a proof of concept to experiment with integrating it and subject to
> change until it made it into OE-Core. The fact is is changing is
> therefore not a surprise. We do need to think about making sure we have
> equivalent functionality, we also need to think about migration but the
> fact changes happened with some experimental work should not surprise
> people. I appreciate that isn't ideal but it is the way it is.

Sure; I'd expect a change for the better ... besides all the work done
in meta-oe lead it to a proper and good implementation (having it
being split and drop the contamination of images we had first
implementation). Having it as a distro feature is a good option too
but all package split, logic and implementation of rest were good and
flexible.

>> > For that reason I'd rather see this done in a different way, for example
>> > blacklisting the problematic systemd dependencies at image generation
>> > time with some kind of stronger BAD_RECOMMENDS code. Yes, this means
>> > there would be some systemd config files lying around but you'd stop the
>> > main bulk of it coming in (and as you said, some small config files
>> > aren't an issue).
>>
>> Check the amount of changes for do_install_append has sent today. This
>> was all not need with the code we had and which works fine. This is
>> just duplicated code all over recipes. Seriously the systemd
>> integration was done in complete wrong way in my POV. It broght up
>> problems we have fixed and do major improvements in meta-oe ... at no
>> profit.
>
> Sorry you feel that way. I'm equally frustrated with the way systemd was
> added to meta-oe and the damage it did to the overall usability of the
> OE ecosystem. I'm trying to take positive actions from that such as
> ensuring we never do something like it again.

Agreed. This was fix when we moved it to meta-oe/meta-systemd.

> As for the integration into the core, I haven't been too involved with
> that and perhaps I should be. The trouble is I alone don't scale and I'm

Agreed. Another problem is I didn't see *any* public discussion how it
would be done *before* the patchsets. The patchset were done and post
as RFC but I'd expect a wider discussion on how it would and should be
done before it.

> trying to ensure we have more people taking on this kind of work. This
> means letting others take it on and learn. There is a requirement from
> the community to ensure people working on improvements like this have
> all the needed data and that is something which I don't think happened
> well here. My point is that I think there have been failings in various
> places and the community needs to take some responsibility too.

Agreed. But bear on mind I complained in mailing list about this in
past (when the systemd patchset were send so it not new I'd prefer it
in another way).

http://thread.gmane.org/gmane.comp.handhelds.openembedded.core/31067/focus=31090

> On the specifics of the do_install_append, you've seen my comments about
> how we're not learning from past mistakes with the way the do_install in
> the class was written. I note Phil also agreed with them, both of us
> remembering some of the horrors we've dealt with in the past (and
> binconfig.bbclass is still around, sadly).

So you prefer same code to copy same type of files all over the recipes?

> One of the goals of OE-Core is to improve quality. I appreciate in this
> case you disagree about the "quality" of those bbappends but I still
> believe that approach will be better in the long term as upstream
> software starts shipping systemd support. Short term its uglier, I
> totally agree. Part of my role in things is to look beyond the short
> term, even if people don't want to hear that.

Quality is relative. For me avoid code duplication is one of quality
aspects I like.

>> I know it is not the better thing to read but I think it is nice when
>> we receive feedback as it allows for rethink of our path of work.
>
> See above as there is some feedback there too ;-)

heh :-)

> I'd also note that its ELC next week and I'll be travelling. Lack of
> response doesn't mean I don't care about this, I do. There are only so
> many hours in the day though and whilst at ELC, I will need to focus
> primarily on that.

Sure; we all have commitments. I'd appreciate if people comment on this as well.

I belive more people that *use* systemd in products or development
should make clear if they expect to be able to build images with
sysv-only and systemd-only support without having to maintain two
distros. For me I even need it for initramfs installers which with
systemd are huge.

Regards,

--
Otavio Salvador                             O.S. Systems
E-mail: otavio@ossystems.com.br  http://www.ossystems.com.br
Mobile: +55 53 9981-7854              http://projetos.ossystems.com.br

Re: RFE: make the init manager an image feature (again)

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 6352 bytes --]

On Sat, Feb 16, 2013 at 12:34:58PM +0000, Richard Purdie wrote:
> On Sat, 2013-02-16 at 12:57 +0100, Enrico Scholz wrote:
> > Richard Purdie <richard.purdie@linuxfoundation.org> writes:
> > >> it would be nice when the decision to make the init manager a distribution
> > >> feature will be reverted to the old oe-meta mechanism.
> > >
> > > The trouble is that by making it an "image feature", people will
> > > expect *everything* to work properly and to be able to have fully
> > > functional sysvinit and systemd variants of images.
> > 
> > I do not see an obvious reason why fully functional sysvinit, systemd and
> > perhaps upstart image variants based on the same distribution/package set
> > are impossible.
> > 
> > Of course, not "everything" will work.  But initmgr being a distribution
> > feature makes some things completely impossible.
> >
> > > We already see this expectation.
> > 
> > IMO, removal of features just to lower expectations is the completely
> > wrong way.
> 
> meta-oe earned a *horrendous* reputation because of the way systemd was
> implemented there. I believe (as do a number of others) that it has
> damaged OE's reputation and usability and actively hurts new users. Yes,
> the people who use systemd and meta-oe were happy. People who didn't
> want systemd were not. There continues to be a fairly toxic mix of
> distro policy mingled in with the recipes in there although good
> progress is being made in fixing that and I'm grateful to the people
> who've noticed and taken on that work (and done previous work like the
> separation of meta-systemd).

I was using sysvinit images even with systemd in meta-oe, setting 2
VIRTUAL_RUNTIME variables and 1 .bbappend was enough to keep sysvinit
images working.

I agree that we were missing documentation for that and new users
weren't able to figure it out for themselves.
 
> It was clear systemd needed to move into the core but also become more
> configurable to work for everyone. I don't want to remove features, I do
> want to talk about whether there is a better way we can fulfil certain
> uses cases, particularly with a focus on usability.

I was talking with Ross about systemd as DISTRO_FEATURE, I agreed that
it's fine to have it as DISTRO_FEATURE but I didn't expect that it also
means that he wants to move PN-systemd to PN without upgrade path and
possibility to create image without PN-systemd packages.

My expectation was that systemd in DISTRO_FEATURES will enable
systemd.bbclass functionality (same functionality as
meta-systemd/system.bbclass had), not that it will force systemd and
only systemd.

Like with "3g" in DISTRO_FEATURES we expect that DISTRO supports 3g but
not that every possible image and MACHINE will provide 3g functionality.

I can imagine distribution with sysvinit+upstart+systemd in
DISTRO_FEATURES if they are careful enough to prepare images with only
right packages.

> > > Trying to explain to people what the limitations are, what is expected
> > > to work and what isn't will be difficult.
> > 
> > OpenEmbedded is not an end-user distribution but for people who are
> > willing to invest some learning effort.  Trying to limit ourself on the
> > lowest common ground is not desirable imo.
> 
> I did not say we're not going to support your use case. I'm asking if we
> can summarise exactly what the problem is and whether there is another
> way we can get there which isn't going to surprise as many people and be
> easier to use.
> 
> I'm actually moderately annoyed that throughout the various discussions
> about systemd and how we'd get it into OE-Core nobody actually mentioned
> these specific requirements until very late in the implementation.

I think we all replied on first patch to move from PN-systemd to PN.
 
> At the bare minimum, we actually need to document the usecases people
> are using and require. Yes, you know your usecase but you need to take
> some responsibility for ensuring its documented and known about else it
> will continue to get broken time and time again.
> 
> > > For that reason I'd rather see this done in a different way, for
> > > example blacklisting the problematic systemd dependencies at image
> > > generation time with some kind of stronger BAD_RECOMMENDS code.
> > 
> > Assuming we are able to break the hard dependencies, what is with package
> > scripts which require programs, files or directories from these deps?  Do
> > we need a way to differ between good and bad script failures then?
> 
> At the technical level there is little difference from packaging these
> things separately to avoid specific dependencies and explicitly telling
> the package manager to avoid that dependency instead.
> 
> Equally there are other cases where people do want to break specific
> dependencies so this could help us in that area too.
> 
> Perhaps there is a dependency we need to drop to a Recommends level,
> then use BAD_RECOMMENDS to handy that. We do need to document why we do
> that and how it is expected to be used.

I think the problem Enrico describes is that systemctl calls were
contained to PN-systemd postinst/prerm scripts, now with systemd support
moved from PN-systemd to PN you have to move postinst/prerm too and if
you allow runtime dependency on systemd package to be broken, then you
need to wrap each postinst/prerm script with check to see if e.g. systemctl 
exists on target image.

> > Sounds extremely hacky and fragile...
> 
> It depends on the implementation.
> 
> Having initscripts in individual packages and having to play tricks to
> ensure the right sets of packages get installed is rather hacky and
> fragile.

Automatic RRECOMMENDS from systemd.class was working fine, only other
place where you needed to make sure all bits are in place was
packagegroup-core-boot.

I wasn't even using automatic RRECOMMEND but small packagegroup recipe
to pull only systemd support for packages where I wanted it. It was more
flexible and allowed me for example to install only ntpdate without
systemd support:

ntpdate works fine for people who wants to occasionally sync their time
ntpdate-systemd is better for people who want to sync on every boot

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: RFE: make the init manager an image feature (again)

[Otavio Salvador]

On Sat, Feb 16, 2013 at 5:40 PM, Martin Jansa <martin.jansa@gmail.com> wrote:
> On Sat, Feb 16, 2013 at 12:34:58PM +0000, Richard Purdie wrote:
>> On Sat, 2013-02-16 at 12:57 +0100, Enrico Scholz wrote:
>> > Richard Purdie <richard.purdie@linuxfoundation.org> writes:
>> > >> it would be nice when the decision to make the init manager a distribution
>> > >> feature will be reverted to the old oe-meta mechanism.
>> > >
>> > > The trouble is that by making it an "image feature", people will
>> > > expect *everything* to work properly and to be able to have fully
>> > > functional sysvinit and systemd variants of images.
>> >
>> > I do not see an obvious reason why fully functional sysvinit, systemd and
>> > perhaps upstart image variants based on the same distribution/package set
>> > are impossible.
>> >
>> > Of course, not "everything" will work.  But initmgr being a distribution
>> > feature makes some things completely impossible.
>> >
>> > > We already see this expectation.
>> >
>> > IMO, removal of features just to lower expectations is the completely
>> > wrong way.
>>
>> meta-oe earned a *horrendous* reputation because of the way systemd was
>> implemented there. I believe (as do a number of others) that it has
>> damaged OE's reputation and usability and actively hurts new users. Yes,
>> the people who use systemd and meta-oe were happy. People who didn't
>> want systemd were not. There continues to be a fairly toxic mix of
>> distro policy mingled in with the recipes in there although good
>> progress is being made in fixing that and I'm grateful to the people
>> who've noticed and taken on that work (and done previous work like the
>> separation of meta-systemd).
>
> I was using sysvinit images even with systemd in meta-oe, setting 2
> VIRTUAL_RUNTIME variables and 1 .bbappend was enough to keep sysvinit
> images working.
>
> I agree that we were missing documentation for that and new users
> weren't able to figure it out for themselves.
>
>> It was clear systemd needed to move into the core but also become more
>> configurable to work for everyone. I don't want to remove features, I do
>> want to talk about whether there is a better way we can fulfil certain
>> uses cases, particularly with a focus on usability.
>
> I was talking with Ross about systemd as DISTRO_FEATURE, I agreed that
> it's fine to have it as DISTRO_FEATURE but I didn't expect that it also
> means that he wants to move PN-systemd to PN without upgrade path and
> possibility to create image without PN-systemd packages.
>
> My expectation was that systemd in DISTRO_FEATURES will enable
> systemd.bbclass functionality (same functionality as
> meta-systemd/system.bbclass had), not that it will force systemd and
> only systemd.
>
> Like with "3g" in DISTRO_FEATURES we expect that DISTRO supports 3g but
> not that every possible image and MACHINE will provide 3g functionality.
>
> I can imagine distribution with sysvinit+upstart+systemd in
> DISTRO_FEATURES if they are careful enough to prepare images with only
> right packages.
>
>> > > Trying to explain to people what the limitations are, what is expected
>> > > to work and what isn't will be difficult.
>> >
>> > OpenEmbedded is not an end-user distribution but for people who are
>> > willing to invest some learning effort.  Trying to limit ourself on the
>> > lowest common ground is not desirable imo.
>>
>> I did not say we're not going to support your use case. I'm asking if we
>> can summarise exactly what the problem is and whether there is another
>> way we can get there which isn't going to surprise as many people and be
>> easier to use.
>>
>> I'm actually moderately annoyed that throughout the various discussions
>> about systemd and how we'd get it into OE-Core nobody actually mentioned
>> these specific requirements until very late in the implementation.
>
> I think we all replied on first patch to move from PN-systemd to PN.
>
>> At the bare minimum, we actually need to document the usecases people
>> are using and require. Yes, you know your usecase but you need to take
>> some responsibility for ensuring its documented and known about else it
>> will continue to get broken time and time again.
>>
>> > > For that reason I'd rather see this done in a different way, for
>> > > example blacklisting the problematic systemd dependencies at image
>> > > generation time with some kind of stronger BAD_RECOMMENDS code.
>> >
>> > Assuming we are able to break the hard dependencies, what is with package
>> > scripts which require programs, files or directories from these deps?  Do
>> > we need a way to differ between good and bad script failures then?
>>
>> At the technical level there is little difference from packaging these
>> things separately to avoid specific dependencies and explicitly telling
>> the package manager to avoid that dependency instead.
>>
>> Equally there are other cases where people do want to break specific
>> dependencies so this could help us in that area too.
>>
>> Perhaps there is a dependency we need to drop to a Recommends level,
>> then use BAD_RECOMMENDS to handy that. We do need to document why we do
>> that and how it is expected to be used.
>
> I think the problem Enrico describes is that systemctl calls were
> contained to PN-systemd postinst/prerm scripts, now with systemd support
> moved from PN-systemd to PN you have to move postinst/prerm too and if
> you allow runtime dependency on systemd package to be broken, then you
> need to wrap each postinst/prerm script with check to see if e.g. systemctl
> exists on target image.
>
>> > Sounds extremely hacky and fragile...
>>
>> It depends on the implementation.
>>
>> Having initscripts in individual packages and having to play tricks to
>> ensure the right sets of packages get installed is rather hacky and
>> fragile.
>
> Automatic RRECOMMENDS from systemd.class was working fine, only other
> place where you needed to make sure all bits are in place was
> packagegroup-core-boot.
>
> I wasn't even using automatic RRECOMMEND but small packagegroup recipe
> to pull only systemd support for packages where I wanted it. It was more
> flexible and allowed me for example to install only ntpdate without
> systemd support:
>
> ntpdate works fine for people who wants to occasionally sync their time
> ntpdate-systemd is better for people who want to sync on every boot

+1

Yocto/OpenEmbedded needs to give priority to flexibility not
simplicity. The default settings need to work for general use case but
drop power from users is the wrong route in my point of view.

-- 
Otavio Salvador                             O.S. Systems
E-mail: otavio@ossystems.com.br  http://www.ossystems.com.br
Mobile: +55 53 9981-7854              http://projetos.ossystems.com.br

Re: RFE: make the init manager an image feature (again)

[Enrico Scholz]

Richard Purdie <richard.purdie@linuxfoundation.org> writes:

> meta-oe earned a *horrendous* reputation because of the way systemd was
> implemented there.

Can you point me to the corresponding discussion resp. which aspects of
the meta-oe implementation were criticized?

I can image only two problems with splitted -sysv/-systemd packages:

1. it enlarges the package database

2. there does not exist a good mechanism to select automatically the
   correct subpackage; e.g. so that when somebody makes 'opkg install
   foo', the -sysv subpackage gets installed on sysv systems and the
   -systemd subpackage on systemd systems.  Ditto with rpm and dpkg
   package managers.

   meta-oe implementation defined a default with help of RRECOMMENDS
   which can be overridden at image build time with BAD_RECOMMENDS.


For me, point 1 is negligible because pkg management is disabled for
images where every byte matters.

Point 2 needs some discussion but I am pretty sure that some clean
technical solution can be found.  The most trivial one might be the
moving of -<initsys> packages into an own 'all-<initsys>' architecture
whose package feed is enabled at image build time. This does not scale
well and violates orthogonality of buildsystem somehow, but works now.

Another approach might adding RRECOMMENDS on all supported -<initsys>
packages and let the package manager choose this with the lowest costs.
That's not trivial and requires probably changes in opkg.

Blacklisting packages with a certain, configurable pattern in its name
or (better) rprovides (e.g. implementing BAD_RECOMMENDS in opkg, smart,
...) is another way requiring changes in the package manager.
  

> I believe (as do a number of others) that it has damaged OE's reputation
> and usability and actively hurts new users.

The current oe-core implementation hurts active, long time users.


> It was clear systemd needed to move into the core but also become more
> configurable to work for everyone.

Exactly this configurability was taken away.


> I don't want to remove features, I do want to talk about whether there
> is a better way we can fulfil certain uses cases, particularly with a
> focus on usability.

For me, a good usability of a build system is defined by a small,
orthogonal and powerful set of tools and rules.

Implementing hacks for specific use cases (e.g. magically remove the
busybox or util-linux -> systemd dep for "rescue systems") violates
this and will probably cause bad surprises (e.g. for people who want
'lighttp' in the rescue system and when the hack has not been applied
to this package).


> I'm actually moderately annoyed that throughout the various discussions
> about systemd and how we'd get it into OE-Core nobody actually mentioned
> these specific requirements until very late in the implementation.

That's a general problem with oe release model.  Parts of systemd (in
-core) were changed without adapting the other ones (in -meta).  This
made the whole oe unbuildable for me for one week and I did not saw the
problem with the rescue image hence.


>> Assuming we are able to break the hard dependencies, what is with package
>> scripts which require programs, files or directories from these deps?  Do
>> we need a way to differ between good and bad script failures then?
>
> At the technical level there is little difference from packaging these
> things separately to avoid specific dependencies and explicitly telling
> the package manager to avoid that dependency instead.

see Martin's comment about post/pre scripts


> Equally there are other cases where people do want to break specific
> dependencies so this could help us in that area too.

Dependencies and recommends are good and moving things into subpackages
is *the* way to break them.



Enrico

Re: RFE: make the init manager an image feature (again)

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 768 bytes --]

On Sat, Feb 16, 2013 at 12:53:07PM +0000, Richard Purdie wrote:
> On the specifics of the do_install_append, you've seen my comments about
> how we're not learning from past mistakes with the way the do_install in
> the class was written. I note Phil also agreed with them, both of us
> remembering some of the horrors we've dealt with in the past (and
> binconfig.bbclass is still around, sadly).

But it doesn't need to be as dangerous as binconfig.bbclass, because we
already list .service or .socket files in SYSTEMD_SERVICE
so we can improve that "find" call to install only stuff we know we need
and error out if it's not found or if there are multiple versions of
that file in WORKDIR.

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: RFE: make the init manager an image feature (again)

[Enrico Scholz]

Martin Jansa <martin.jansa-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
writes:

>> On the specifics of the do_install_append, you've seen my comments
>> about how we're not learning from past mistakes with the way the
>> do_install in the class was written. I note Phil also agreed with
>> them, both of us remembering some of the horrors we've dealt with in
>> the past (and binconfig.bbclass is still around, sadly).
>
> But it doesn't need to be as dangerous as binconfig.bbclass, because
> we already list .service or .socket files in SYSTEMD_SERVICE so we can
> improve that "find" call

Why is 'find' required at all?  afaik, only files from $SRC_URI are
affected.  So we can

1. create an (overridable) SYSTEMD_EXTRA_SERVICES variable

2. fill this variable in systemd.bbclass with .service, .target, .socket
   + .mount files from $SRC_URI

3. modify systemd's do_install so that files above are copied after
   doing some some sanity checks (e.g. checks that no previous version
   from 'make install' exists or that it are really systemd files).


Enrico

Re: RFE: make the init manager an image feature (again)

[Burton, Ross]

On 18 February 2013 10:17, Enrico Scholz
<enrico.scholz@sigma-chemnitz.de> wrote:
>> But it doesn't need to be as dangerous as binconfig.bbclass, because
>> we already list .service or .socket files in SYSTEMD_SERVICE so we can
>> improve that "find" call
>
> Why is 'find' required at all?  afaik, only files from $SRC_URI are
> affected.  So we can
>
> 1. create an (overridable) SYSTEMD_EXTRA_SERVICES variable
>
> 2. fill this variable in systemd.bbclass with .service, .target, .socket
>    + .mount files from $SRC_URI
>
> 3. modify systemd's do_install so that files above are copied after
>    doing some some sanity checks (e.g. checks that no previous version
>    from 'make install' exists or that it are really systemd files).

As I've said before you'll need to be pre-processing these files
anyway, so either the class gets *even more* logic or you just deal
with it.  Over time more and more upstreams will get systemd support
so we'll be carrying overcomplicated logic for no reason.  We're
really just talking about one conditional and a sed call here.

Ross

Re: RFE: make the init manager an image feature (again)

[Enrico Scholz]

"Burton, Ross" <ross.burton@intel.com> writes:

>>> But it doesn't need to be as dangerous as binconfig.bbclass, because
>>> we already list .service or .socket files in SYSTEMD_SERVICE so we
>>> can improve that "find" call
>>
>> Why is 'find' required at all?  afaik, only files from $SRC_URI are
>> affected.  So we can
>>
>> 1. create an (overridable) SYSTEMD_EXTRA_SERVICES variable
>>
>> 2. fill this variable in systemd.bbclass with .service, .target, .socket
>>    + .mount files from $SRC_URI
>>
>> 3. modify systemd's do_install so that files above are copied after
>>    doing some some sanity checks (e.g. checks that no previous version
>>    from 'make install' exists or that it are really systemd files).
>
> As I've said before you'll need to be pre-processing these files
> anyway,

oh... this means khem's "meta-systemd: Append ${PN} to SYSTEMD_SERVICE"
patch series is incomplete and all the do_install_append() need to get
yet more complicated....

It is really time, to move the additional-service-file installation back
into the class.


> so either the class gets *even more* logic or you just deal with it.
> Over time more and more upstreams will get systemd support so we'll
> be carrying overcomplicated logic for no reason.  We're really just
> talking about one conditional and a sed call here.

By applying some general rules (e.g. using autoconf like '@bindir@'
templates), this can/should be done by the class and not in every
recipe.



Enrico

Re: RFE: make the init manager an image feature (again)

[Burton, Ross]

On 21 February 2013 10:34, Enrico Scholz
<enrico.scholz@sigma-chemnitz.de> wrote:
> oh... this means khem's "meta-systemd: Append ${PN} to SYSTEMD_SERVICE"
> patch series is incomplete and all the do_install_append() need to get
> yet more complicated....

What was originally in meta-systemd, and what is there by simply
adding a do_install_append that just does install, is plain wrong.
The moment someone changes the directory structure. it's broken.

You only have appends where you don't do the work in the real class.
Remember that as oe-core supports systemd, all other layers can
support it too.  meta-systemd can disappear, and more upstream over
time will also integrate systemd unit files directly.

Ross

Re: RFE: make the init manager an image feature (again)

[Enrico Scholz]

"Burton, Ross" <ross.burton-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
writes:

> more upstream over time will also integrate systemd unit files
> directly.

When in 5 or 10 years everybody switched to systemd and installs its
service files by itself, we can mark the relevant code in the class as
deprecated and remove it.

But now, the oe-core systemd implementation causes redundant implementations
of non trivial code in several packages.


Enrico

Re: RFE: make the init manager an image feature (again)

[Otavio Salvador]

On Thu, Feb 21, 2013 at 8:34 AM, Enrico Scholz
<enrico.scholz@sigma-chemnitz.de> wrote:
> "Burton, Ross" <ross.burton-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> writes:
>
>> more upstream over time will also integrate systemd unit files
>> directly.
>
> When in 5 or 10 years everybody switched to systemd and installs its
> service files by itself, we can mark the relevant code in the class as
> deprecated and remove it.
>
> But now, the oe-core systemd implementation causes redundant implementations
> of non trivial code in several packages.

I fully agree; we have many cases where classes workaround system
issues/limitations to avoid code duplications so I see no reason why
this needs to be different with systemd.

-- 
Otavio Salvador                             O.S. Systems
E-mail: otavio@ossystems.com.br  http://www.ossystems.com.br
Mobile: +55 53 9981-7854              http://projetos.ossystems.com.br

Re: RFE: make the init manager an image feature (again)

[Phil Blundell]

On Thu, 2013-02-21 at 08:50 -0300, Otavio Salvador wrote:
> I fully agree; we have many cases where classes workaround system
> issues/limitations to avoid code duplications so I see no reason why
> this needs to be different with systemd.

If you wanted to propose an addition to the class that purely abstracted
out the duplication (without adding any potential bear-traps) then I
would have thought that should be fine. 

For example, something along the approximate lines of:

install_systemd_collateral_file() {
   if ${@base_contains('DISTRO_FEATURES', 'systemd', 'true', 'false')}; then
      sed 's/@bindir@/${bindir}/' < $1 > $2
   fi
}

seems like it would address most of the concerns about code duplication
without introducing any hairy heuristics or unpredictably spontaneous
behaviour in the class.

p.

Re: RFE: make the init manager an image feature (again)

[Burton, Ross]

Hi,

This thread started sprawling, so I'll do my best to cover all the
points.  This is also mainly an attempt to get more information as to
how people are using init managers, as it's still not very clear what
people want beyond "choice".

"With recent systemd packaging change, the rescue image size grow up
from 5.9 MiB to 27 MiB because systemd dependencies are hardcoded in
mandatory packages."

This certainly can happen.  core-image-minimal-initramfs went from 19M
up to 35M if you turn on systemd with the (unmerged) busybox enabling
turned on.  Investigating this shows that the cause was busybox
recommending busybox-syslog which depends on systemd, which isn't
useful in the initramfs.  My branch adds busybox-syslog to
BAD_RECOMMENDS and it's back down to 19M.  This works as this image
has a custom /sbin/init so it doesn't use an init manager at all.

When people are building images with and without systemd, how are the
non-systemd images booting?  If sysvinit, what's the rationale for
this?  Will you consider switching to systemd for everything if the
disk increase was only a new MB, on the grounds of consistent booting
behaviour?  I can imagine many rescue or initramfs images are using a
custom /init, but as these don't/shouldn't actually have any daemons
installed this is practically a solved problem.

systemd as integrated is currently too big for very constrained
environments, and I'm not denying that.  Using this
core-image-minimal-initramfs as a test whilst pulling in systemd I've
been cleaning up spurious dependencies and have managed to trim around
4MB from the footprint, but there's massive low-hanging fruit like a
4MB /lib/udev/hwdb.d.  Rescue images likely don't need this, so we can
split it out and not always install it (now down to a 5MB footprint
compared to no init system).

One massive problem with making init system an image time choice with
packages is what happens when you have a feed with both pn-sysv and
pn-systemd packages in.  If after the initial construction you want to
install a daemon, you'll have to know to manually install the right
init package too.

Supporting multiple init systems in the same packaging would be
achievable, but would people who want this be happy with pulling the
systemd libraries into sysvinit images?  The alternative is that if
sysvinit and systemd are enabled we can't do any real systemd
enabling.   The good thing is that we could certainly make
systemd.bbclass inject a recommends instead of dependency on systemd
and add guards around the systemd calls, ditto for update-rcd.bbclass.

Regards,
Ross

Re: RFE: make the init manager an image feature (again)

[Otavio Salvador]

On Thu, Feb 21, 2013 at 12:35 PM, Burton, Ross <ross.burton@intel.com> wrote:
> Hi,
>
> This thread started sprawling, so I'll do my best to cover all the
> points.  This is also mainly an attempt to get more information as to
> how people are using init managers, as it's still not very clear what
> people want beyond "choice".

Awesome. I agree it is difficult to track in a such big thread.

> "With recent systemd packaging change, the rescue image size grow up
> from 5.9 MiB to 27 MiB because systemd dependencies are hardcoded in
> mandatory packages."
>
> This certainly can happen.  core-image-minimal-initramfs went from 19M
> up to 35M if you turn on systemd with the (unmerged) busybox enabling
> turned on.  Investigating this shows that the cause was busybox
> recommending busybox-syslog which depends on systemd, which isn't
> useful in the initramfs.  My branch adds busybox-syslog to
> BAD_RECOMMENDS and it's back down to 19M.  This works as this image
> has a custom /sbin/init so it doesn't use an init manager at all.

Right. However BAD_RECOMMENDS is a little *manual* ... more on this later ...

> When people are building images with and without systemd, how are the
> non-systemd images booting?  If sysvinit, what's the rationale for

Last time I tested no. systemd-udevd would need to be split from
systemd package and have init scripts as an option.

> this?  Will you consider switching to systemd for everything if the
> disk increase was only a new MB, on the grounds of consistent booting
> behaviour?  I can imagine many rescue or initramfs images are using a
> custom /init, but as these don't/shouldn't actually have any daemons
> installed this is practically a solved problem.

No; one of our products is build with an initramfs (using
initramfs-framework) and it is much easier for this than the systemd.
In some cases we use the init.d scripts provided by packages to allow
reuse in initramfs-framework and it works fine.

> systemd as integrated is currently too big for very constrained
> environments, and I'm not denying that.  Using this
> core-image-minimal-initramfs as a test whilst pulling in systemd I've
> been cleaning up spurious dependencies and have managed to trim around
> 4MB from the footprint, but there's massive low-hanging fruit like a
> 4MB /lib/udev/hwdb.d.  Rescue images likely don't need this, so we can
> split it out and not always install it (now down to a 5MB footprint
> compared to no init system).

Yes; this is a regression from the status we had in meta-oe *before*
the merge in oe-core. It was more flexible.

> One massive problem with making init system an image time choice with
> packages is what happens when you have a feed with both pn-sysv and
> pn-systemd packages in.  If after the initial construction you want to
> install a daemon, you'll have to know to manually install the right
> init package too.

Yes; this is one problem. I think we might think about havig something
in package manager that try to fullfil a virtual provide (as we had
for locales).

> Supporting multiple init systems in the same packaging would be
> achievable, but would people who want this be happy with pulling the
> systemd libraries into sysvinit images?  The alternative is that if
> sysvinit and systemd are enabled we can't do any real systemd
> enabling.   The good thing is that we could certainly make
> systemd.bbclass inject a recommends instead of dependency on systemd
> and add guards around the systemd calls, ditto for update-rcd.bbclass.

It depends. If I have 'systemd' enabled in the distro I am sure
expecting it to happen. If I don't, I don't expect it to happen.

-- 
Otavio Salvador                             O.S. Systems
E-mail: otavio@ossystems.com.br  http://www.ossystems.com.br
Mobile: +55 53 9981-7854              http://projetos.ossystems.com.br

Re: RFE: make the init manager an image feature (again)

[Enrico Scholz]

"Burton, Ross" <ross.burton-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
writes:

> "With recent systemd packaging change, the rescue image size grow up
> from 5.9 MiB to 27 MiB because systemd dependencies are hardcoded in
> mandatory packages."
>
> This certainly can happen.  core-image-minimal-initramfs went from 19M
> up to 35M if you turn on systemd with the (unmerged) busybox enabling
> turned on.  Investigating this shows that the cause was busybox
> recommending busybox-syslog which depends on systemd, which isn't
> useful in the initramfs.  My branch adds busybox-syslog to
> BAD_RECOMMENDS and it's back down to 19M.  

I want busybox-syslogd in the rescue image. And an ntp + dhcp client
and http server.  Next might be tools from util-linux which depends on
systemd too.


> When people are building images with and without systemd, how are the
> non-systemd images booting?

Custom /init which executes busybox's /sbin/init finally.


> If sysvinit, what's the rationale for this?

It allows modularization (starting new services without editing /init)
with zero costs (busybox-init is already there).  I need full control
over things like attaching/detecting ubi volumes or sd cards and I feel
much better, when every line of relevant code was written by me ;)


> Will you consider switching to systemd for everything if the disk
> increase was only a new MB, on the grounds of consistent booting
> behaviour?

Definitively not.  There is far too much logic in usual systemd/udev
systems (e.g. accessing block devices and other hardware during udev
scanning) which contradicts with purposes of a rescue system.


> One massive problem with making init system an image time choice with
> packages is what happens when you have a feed with both pn-sysv and
> pn-systemd packages in.

I wrote some ideas here[1]. It starts with moving init script subpackages
into separate 'all-<initsys>' feeds till implementing the BAD_RECOMMENDS
mechanism in the depsolvers.

For me, it would be also okay, when runtime package management gets ignored
at all (rescue system are initramfs images without pkg management) and
making the initmgr in DISTRO_FEATURES selecting the default RRECOMMENDS.
The actual BAD_RECOMMENDATIONS mechanism works perfect in this case; adding
interesting -sysv subpackages manually when creating the rescue image would
be ok for me.


> The alternative is that if sysvinit and systemd are enabled we can't
> do any real systemd enabling.

Why not?  The functions in libsystemd are robust enough so that daemons
are still working in non-systemd systems.



Enrico

Footnotes: 
[1]  http://lists.linuxtogo.org/pipermail/openembedded-core/2013-February/035998.html

Re: RFE: make the init manager an image feature (again)

[Khem Raj]

On (16/02/13 11:41), Otavio Salvador wrote:
> On Sat, Feb 16, 2013 at 10:53 AM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > On Sat, 2013-02-16 at 08:47 -0200, Otavio Salvador wrote:
> >> On Sat, Feb 16, 2013 at 7:15 AM, Richard Purdie
> >> <richard.purdie@linuxfoundation.org> wrote:
> >> > On Fri, 2013-02-15 at 19:19 +0100, Enrico Scholz wrote:
> >> >> it would be nice when the decision to make the init manager a distribution
> >> >> feature will be reverted to the old oe-meta mechanism.
> >> >>
> >> >> Being a distribution feature means, that packages are created in such a
> >> >> way that it is impossible to split off unwanted and heavy weighted
> >> >> functionality at image creation time.
> >> >>
> >> >> E.g. on most of my systems, I create two kinds of images: a full
> >> >> featured, systemd based one and a very minimal rescue system with
> >> >> busybox and some filesystem utilities.  With recent systemd packaging
> >> >> change, the rescue image size grow up from 5.9 MiB to 27 MiB because
> >> >> systemd dependencies are hardcoded in mandatory packages.
> >> >>
> >> >> Formerly, systemd dependencies could be avoided by adding the -systemd
> >> >> packages to BAD_RECOMMENDATIONS (e.g. due to busybox-syslog ->
> >> >> busybox-syslog-systemd rrecommend).
> >> >>
> >> >> I am aware that initscripts were always part of the main package.  But
> >> >> sysvinit was very lightweighted and the extra space either negligible or
> >> >> easy to recover by removing some files in IMAGE_PREPROCESS_COMMAND.
> >> >>
> >> >> Hence my recommendation: make the init manager an image feature again
> >> >> and create -systemd and -sysv packages with the corresponding scripts.
> >> >> OpenEmbedded is still for embedded devices where size matters.
> >> >>
> >> >> Of course, systemd can be still a distribution feature to enable things
> >> >> like socket activation as part of PACKAGE_CONFIG.  But dependencies on
> >> >> init system packages should be RRECOMMENDS which can be overridden
> >> >> easily at image creation time.
> >> >
> >> > The trouble is that by making it an "image feature", people will expect
> >> > *everything* to work properly and to be able to have fully functional
> >> > sysvinit and systemd variants of images. We already see this
> >> > expectation. Trying to explain to people what the limitations are, what
> >> > is expected to work and what isn't will be difficult. I know you
> >> > understand this but going forward most people will simply not and I
> >> > think it will be a nightmare for usability.
> >>
> >> You said you already see it so what is the change? For me less
> >> flexibility is more problem and the result are more ugly hacks. I've
> >> been using the systemd system in meta-oe in product for long time and
> >> it works fine. In same distro I build other product with is sysvinit
> >> based and it also works fine.
> >>
> >> Now I have a nightmare in upgrade path, a change in the way my
> >> customer work and as a plus the need to make two different
> >> distributions for same product. No sense to me.
> >>
> >> Final result? This product won't go out of danny as I won't be able to
> >> provide an usable upgrade path without forking OE or having an insane
> >> amount of bbappend hacks.
> >
> > When systemd was added to meta-oe, it was clearly mentioned that it was
> > a proof of concept to experiment with integrating it and subject to
> > change until it made it into OE-Core. The fact is is changing is
> > therefore not a surprise. We do need to think about making sure we have
> > equivalent functionality, we also need to think about migration but the
> > fact changes happened with some experimental work should not surprise
> > people. I appreciate that isn't ideal but it is the way it is.
> 
> Sure; I'd expect a change for the better ... besides all the work done
> in meta-oe lead it to a proper and good implementation (having it
> being split and drop the contamination of images we had first
> implementation). Having it as a distro feature is a good option too
> but all package split, logic and implementation of rest were good and
> flexible.
> 
> >> > For that reason I'd rather see this done in a different way, for example
> >> > blacklisting the problematic systemd dependencies at image generation
> >> > time with some kind of stronger BAD_RECOMMENDS code. Yes, this means
> >> > there would be some systemd config files lying around but you'd stop the
> >> > main bulk of it coming in (and as you said, some small config files
> >> > aren't an issue).
> >>
> >> Check the amount of changes for do_install_append has sent today. This
> >> was all not need with the code we had and which works fine. This is
> >> just duplicated code all over recipes. Seriously the systemd
> >> integration was done in complete wrong way in my POV. It broght up
> >> problems we have fixed and do major improvements in meta-oe ... at no
> >> profit.
> >
> > Sorry you feel that way. I'm equally frustrated with the way systemd was
> > added to meta-oe and the damage it did to the overall usability of the
> > OE ecosystem. I'm trying to take positive actions from that such as
> > ensuring we never do something like it again.
> 
> Agreed. This was fix when we moved it to meta-oe/meta-systemd.
> 
> > As for the integration into the core, I haven't been too involved with
> > that and perhaps I should be. The trouble is I alone don't scale and I'm
> 
> Agreed. Another problem is I didn't see *any* public discussion how it
> would be done *before* the patchsets. The patchset were done and post
> as RFC but I'd expect a wider discussion on how it would and should be
> done before it.
> 
> > trying to ensure we have more people taking on this kind of work. This
> > means letting others take it on and learn. There is a requirement from
> > the community to ensure people working on improvements like this have
> > all the needed data and that is something which I don't think happened
> > well here. My point is that I think there have been failings in various
> > places and the community needs to take some responsibility too.
> 
> Agreed. But bear on mind I complained in mailing list about this in
> past (when the systemd patchset were send so it not new I'd prefer it
> in another way).
> 
> http://thread.gmane.org/gmane.comp.handhelds.openembedded.core/31067/focus=31090
> 
> > On the specifics of the do_install_append, you've seen my comments about
> > how we're not learning from past mistakes with the way the do_install in
> > the class was written. I note Phil also agreed with them, both of us
> > remembering some of the horrors we've dealt with in the past (and
> > binconfig.bbclass is still around, sadly).
> 
> So you prefer same code to copy same type of files all over the recipes?

duplicating usecase is better even though it may sound
copying here now think if you add this to a bbclass and works ok for
a package where we glue the unitfiles fast forward 6mos and the next
version of the package has added the unitfiles into the package itself
since systemd is so cool. We will be silently installing our own
unit files without knowing. Worst if the files from package are
different. Having individual install append gives you an opportunity
to delete this append once the package gets its own support for systemd
and you can easily migrate as the systemd support comes in.
> 
> > One of the goals of OE-Core is to improve quality. I appreciate in this
> > case you disagree about the "quality" of those bbappends but I still
> > believe that approach will be better in the long term as upstream
> > software starts shipping systemd support. Short term its uglier, I
> > totally agree. Part of my role in things is to look beyond the short
> > term, even if people don't want to hear that.
> 
> Quality is relative. For me avoid code duplication is one of quality
> aspects I like.

yes it is. But this just isnt the case where it weighs more.

Re: RFE: make the init manager an image feature (again)

[Ross Burton]

Hi,

Just brainstorming out loud, but here's a suggestion that might just please everyone:

A virtual provider for the init manager, which can be overridden per-image (for main / rescue images).

DISTRO_FEATURES contains the init script *style* that you want: sysvinit or systemd.  These are not mutually exclusive so specifying both will get you both directly in packages that support both.  I'm still not convinced we need to split these out into separate packages, the size impact is practically negligible and the dependencies are effectively redundant as you'll have trouble booting without an init system.  Instead the postinsts should wrap the initscript fragments in checks.

Those distro features will also be used to enable/disable features in the source, so enabling systemd may well lead to libsystemd-* libraries sneaking into your rescue image.  The systemd libraries will have to be reviewed to verify that installing one of them doesn't pull in systemd itself.

Here's the fun bit - we can't have two builds of udev in the same distro, and that's what could happen, so I think we'll need to cut down to one udev instead of two.

I'm not entirely convinced this is a good plan yet myself though…

Ross

Re: RFE: make the init manager an image feature (again)

[Ross Burton]

On Sunday, 24 February 2013 at 10:37, Ross Burton wrote:
> Here's the fun bit - we can't have two builds of udev in the same distro, and that's what could happen, so I think we'll need to cut down to one udev instead of two.

Ignore this, we can continue to key off the systemd distro feature to pick between udev and systemd as the source for udevd. 

Ross

Re: RFE: make the init manager an image feature (again)

[Otavio Salvador]

On Sun, Feb 24, 2013 at 7:37 AM, Ross Burton <ross.burton@intel.com> wrote:
> Hi,
>
> Just brainstorming out loud, but here's a suggestion that might just please everyone:
>
> A virtual provider for the init manager, which can be overridden per-image (for main / rescue images).
>
> DISTRO_FEATURES contains the init script *style* that you want: sysvinit or systemd.  These are not mutually exclusive so specifying both will get you both directly in packages that support both.  I'm still not convinced we need to split these out into separate packages, the size impact is practically negligible and the dependencies are effectively redundant as you'll have trouble booting without an init system.  Instead the postinsts should wrap the initscript fragments in checks.

The size impact it not negligible; specially for initramfs images but
what concerns me even more is the upgrade path from previous users of
meta-oe systemd.

> Those distro features will also be used to enable/disable features in the source, so enabling systemd may well lead to libsystemd-* libraries sneaking into your rescue image.  The systemd libraries will have to be reviewed to verify that installing one of them doesn't pull in systemd itself.
>
> Here's the fun bit - we can't have two builds of udev in the same distro, and that's what could happen, so I think we'll need to cut down to one udev instead of two.

And add init scripts to this udev as well

> I'm not entirely convinced this is a good plan yet myself though…

I'd like to have an upgrade path.

-- 
Otavio Salvador                             O.S. Systems
E-mail: otavio@ossystems.com.br  http://www.ossystems.com.br
Mobile: +55 53 9981-7854              http://projetos.ossystems.com.br

Re: RFE: make the init manager an image feature (again)

[Otavio Salvador]

On Sun, Feb 24, 2013 at 5:50 AM, Khem Raj <raj.khem@gmail.com> wrote:
> On (16/02/13 11:41), Otavio Salvador wrote:
>> On Sat, Feb 16, 2013 at 10:53 AM, Richard Purdie
>> <richard.purdie@linuxfoundation.org> wrote:
>> > On Sat, 2013-02-16 at 08:47 -0200, Otavio Salvador wrote:
>> >> On Sat, Feb 16, 2013 at 7:15 AM, Richard Purdie
>> >> <richard.purdie@linuxfoundation.org> wrote:
>> >> > On Fri, 2013-02-15 at 19:19 +0100, Enrico Scholz wrote:
>> >> >> it would be nice when the decision to make the init manager a distribution
>> >> >> feature will be reverted to the old oe-meta mechanism.
>> >> >>
>> >> >> Being a distribution feature means, that packages are created in such a
>> >> >> way that it is impossible to split off unwanted and heavy weighted
>> >> >> functionality at image creation time.
>> >> >>
>> >> >> E.g. on most of my systems, I create two kinds of images: a full
>> >> >> featured, systemd based one and a very minimal rescue system with
>> >> >> busybox and some filesystem utilities.  With recent systemd packaging
>> >> >> change, the rescue image size grow up from 5.9 MiB to 27 MiB because
>> >> >> systemd dependencies are hardcoded in mandatory packages.
>> >> >>
>> >> >> Formerly, systemd dependencies could be avoided by adding the -systemd
>> >> >> packages to BAD_RECOMMENDATIONS (e.g. due to busybox-syslog ->
>> >> >> busybox-syslog-systemd rrecommend).
>> >> >>
>> >> >> I am aware that initscripts were always part of the main package.  But
>> >> >> sysvinit was very lightweighted and the extra space either negligible or
>> >> >> easy to recover by removing some files in IMAGE_PREPROCESS_COMMAND.
>> >> >>
>> >> >> Hence my recommendation: make the init manager an image feature again
>> >> >> and create -systemd and -sysv packages with the corresponding scripts.
>> >> >> OpenEmbedded is still for embedded devices where size matters.
>> >> >>
>> >> >> Of course, systemd can be still a distribution feature to enable things
>> >> >> like socket activation as part of PACKAGE_CONFIG.  But dependencies on
>> >> >> init system packages should be RRECOMMENDS which can be overridden
>> >> >> easily at image creation time.
>> >> >
>> >> > The trouble is that by making it an "image feature", people will expect
>> >> > *everything* to work properly and to be able to have fully functional
>> >> > sysvinit and systemd variants of images. We already see this
>> >> > expectation. Trying to explain to people what the limitations are, what
>> >> > is expected to work and what isn't will be difficult. I know you
>> >> > understand this but going forward most people will simply not and I
>> >> > think it will be a nightmare for usability.
>> >>
>> >> You said you already see it so what is the change? For me less
>> >> flexibility is more problem and the result are more ugly hacks. I've
>> >> been using the systemd system in meta-oe in product for long time and
>> >> it works fine. In same distro I build other product with is sysvinit
>> >> based and it also works fine.
>> >>
>> >> Now I have a nightmare in upgrade path, a change in the way my
>> >> customer work and as a plus the need to make two different
>> >> distributions for same product. No sense to me.
>> >>
>> >> Final result? This product won't go out of danny as I won't be able to
>> >> provide an usable upgrade path without forking OE or having an insane
>> >> amount of bbappend hacks.
>> >
>> > When systemd was added to meta-oe, it was clearly mentioned that it was
>> > a proof of concept to experiment with integrating it and subject to
>> > change until it made it into OE-Core. The fact is is changing is
>> > therefore not a surprise. We do need to think about making sure we have
>> > equivalent functionality, we also need to think about migration but the
>> > fact changes happened with some experimental work should not surprise
>> > people. I appreciate that isn't ideal but it is the way it is.
>>
>> Sure; I'd expect a change for the better ... besides all the work done
>> in meta-oe lead it to a proper and good implementation (having it
>> being split and drop the contamination of images we had first
>> implementation). Having it as a distro feature is a good option too
>> but all package split, logic and implementation of rest were good and
>> flexible.
>>
>> >> > For that reason I'd rather see this done in a different way, for example
>> >> > blacklisting the problematic systemd dependencies at image generation
>> >> > time with some kind of stronger BAD_RECOMMENDS code. Yes, this means
>> >> > there would be some systemd config files lying around but you'd stop the
>> >> > main bulk of it coming in (and as you said, some small config files
>> >> > aren't an issue).
>> >>
>> >> Check the amount of changes for do_install_append has sent today. This
>> >> was all not need with the code we had and which works fine. This is
>> >> just duplicated code all over recipes. Seriously the systemd
>> >> integration was done in complete wrong way in my POV. It broght up
>> >> problems we have fixed and do major improvements in meta-oe ... at no
>> >> profit.
>> >
>> > Sorry you feel that way. I'm equally frustrated with the way systemd was
>> > added to meta-oe and the damage it did to the overall usability of the
>> > OE ecosystem. I'm trying to take positive actions from that such as
>> > ensuring we never do something like it again.
>>
>> Agreed. This was fix when we moved it to meta-oe/meta-systemd.
>>
>> > As for the integration into the core, I haven't been too involved with
>> > that and perhaps I should be. The trouble is I alone don't scale and I'm
>>
>> Agreed. Another problem is I didn't see *any* public discussion how it
>> would be done *before* the patchsets. The patchset were done and post
>> as RFC but I'd expect a wider discussion on how it would and should be
>> done before it.
>>
>> > trying to ensure we have more people taking on this kind of work. This
>> > means letting others take it on and learn. There is a requirement from
>> > the community to ensure people working on improvements like this have
>> > all the needed data and that is something which I don't think happened
>> > well here. My point is that I think there have been failings in various
>> > places and the community needs to take some responsibility too.
>>
>> Agreed. But bear on mind I complained in mailing list about this in
>> past (when the systemd patchset were send so it not new I'd prefer it
>> in another way).
>>
>> http://thread.gmane.org/gmane.comp.handhelds.openembedded.core/31067/focus=31090
>>
>> > On the specifics of the do_install_append, you've seen my comments about
>> > how we're not learning from past mistakes with the way the do_install in
>> > the class was written. I note Phil also agreed with them, both of us
>> > remembering some of the horrors we've dealt with in the past (and
>> > binconfig.bbclass is still around, sadly).
>>
>> So you prefer same code to copy same type of files all over the recipes?
>
> duplicating usecase is better even though it may sound
> copying here now think if you add this to a bbclass and works ok for
> a package where we glue the unitfiles fast forward 6mos and the next
> version of the package has added the unitfiles into the package itself
> since systemd is so cool. We will be silently installing our own
> unit files without knowing. Worst if the files from package are
> different. Having individual install append gives you an opportunity
> to delete this append once the package gets its own support for systemd
> and you can easily migrate as the systemd support comes in.

I fail to see how duplicating the code helps to spot the new package
would have the service files. You'll keep installing them until you
discover it has the services files there.

>> > One of the goals of OE-Core is to improve quality. I appreciate in this
>> > case you disagree about the "quality" of those bbappends but I still
>> > believe that approach will be better in the long term as upstream
>> > software starts shipping systemd support. Short term its uglier, I
>> > totally agree. Part of my role in things is to look beyond the short
>> > term, even if people don't want to hear that.
>>
>> Quality is relative. For me avoid code duplication is one of quality
>> aspects I like.
>
> yes it is. But this just isnt the case where it weighs more.

That's way I said it is relative. All depends on which quality aspect
you're thinking about.

-- 
Otavio Salvador                             O.S. Systems
E-mail: otavio@ossystems.com.br  http://www.ossystems.com.br
Mobile: +55 53 9981-7854              http://projetos.ossystems.com.br

Re: RFE: make the init manager an image feature (again)

[Ross Burton]

On Sunday, 24 February 2013 at 14:06, Otavio Salvador wrote:
> > DISTRO_FEATURES contains the init script *style* that you want: sysvinit or systemd. These are not mutually exclusive so specifying both will get you both directly in packages that support both. I'm still not convinced we need to split these out into separate packages, the size impact is practically negligible and the dependencies are effectively redundant as you'll have trouble booting without an init system. Instead the postinsts should wrap the initscript fragments in checks.
>  
>  
> The size impact it not negligible; specially for initramfs images but
> what concerns me even more is the upgrade path from previous users of
> meta-oe systemd.


I obviously didn't make myself clear - the size impact is negligible when you're talking about just the init script - the dependencies on systemd/updatercd could be recommends at most, as the postinst scripts could check what init system they have before calling any tools.  Either way a rescue image that boots using busybox init shouldn't have systemd, clearly.
> I'd like to have an upgrade path.

Well, strictly speaking oe-core itself doesn't have an upgrade path to consider… Why can't any distros that shipped with meta-systemd (or just in meta-systemd) have an include that injects the RPROVIDES/RREPLACES/RCONFLICTS for the packages that they enabled?

Ross

Re: RFE: make the init manager an image feature (again)

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1929 bytes --]

On Sun, Feb 24, 2013 at 10:04:42PM +0000, Ross Burton wrote:
> On Sunday, 24 February 2013 at 14:06, Otavio Salvador wrote:
> > > DISTRO_FEATURES contains the init script *style* that you want: sysvinit or systemd. These are not mutually exclusive so specifying both will get you both directly in packages that support both. I'm still not convinced we need to split these out into separate packages, the size impact is practically negligible and the dependencies are effectively redundant as you'll have trouble booting without an init system. Instead the postinsts should wrap the initscript fragments in checks.
> >  
> >  
> > The size impact it not negligible; specially for initramfs images but
> > what concerns me even more is the upgrade path from previous users of
> > meta-oe systemd.
> 
> 
> I obviously didn't make myself clear - the size impact is negligible when you're talking about just the init script - the dependencies on systemd/updatercd could be recommends at most, as the postinst scripts could check what init system they have before calling any tools.  Either way a rescue image that boots using busybox init shouldn't have systemd, clearly.
> > I'd like to have an upgrade path.
> 
> Well, strictly speaking oe-core itself doesn't have an upgrade path to consider… Why can't any distros that shipped with meta-systemd (or just in meta-systemd) have an include that injects the RPROVIDES/RREPLACES/RCONFLICTS for the packages that they enabled?

That would force multiple distro layers to maintain many .bbappends with
just RPROVIDES/RREPLACES/RCONFLICTS and PRINC bump maintaned forever.

It was pain to rename .bbappends in meta-systemd after each .bb upgrade,
so if we have to do that, then we should rather keep meta-systemd as
layer so that we can share .bbappend maintenance for all distributions in
one place.

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: RFE: make the init manager an image feature (again)

[Andreas Müller]

On Sun, Feb 24, 2013 at 11:04 PM, Ross Burton <ross.burton@intel.com> wrote:
>> The size impact it not negligible; specially for initramfs images but
>> what concerns me even more is the upgrade path from previous users of
>> meta-oe systemd.
>
>
> I obviously didn't make myself clear - the size impact is negligible when you're talking about just the init script - the >dependencies on systemd/updatercd could be recommends at most, as the postinst scripts could check what init system >they have before calling any tools.  Either way a rescue image that boots using busybox init shouldn't have systemd, clearly.
Maybe I missed something but I have seen some arguments for splitting
initscripts into single packages - but I can't remember one for having
all in one package. Maybe somebody can help me out here.

Andreas

Re: RFE: make the init manager an image feature (again)

[Enrico Scholz]

Khem Raj <raj.khem-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:

>> > On the specifics of the do_install_append, you've seen my comments about
>> > how we're not learning from past mistakes with the way the do_install in
>> > the class was written. I note Phil also agreed with them, both of us
>> > remembering some of the horrors we've dealt with in the past (and
>> > binconfig.bbclass is still around, sadly).
>> 
>> So you prefer same code to copy same type of files all over the recipes?
>
> duplicating usecase is better even though it may sound copying here
> now think if you add this to a bbclass and works ok for a package
> where we glue the unitfiles fast forward 6mos and the next version of
> the package has added the unitfiles into the package itself since
> systemd is so cool. We will be silently installing our own unit files
> without knowing.

The recent .bbappends with their own do_install_append have this problem
too. But due to the code replication the problem must be fixed at a lot
of different places instead of at a central one.

The .bbclass can do some sanity checks (e.g. making the copy operation
fail when the unit file already exists).


> Worst if the files from package are different. Having individual
> install append gives you an opportunity to this append

I do not see how this is better than removing the local .service from
SRC_URI.



Enrico

Re: RFE: make the init manager an image feature (again)

[Enrico Scholz]

Ross Burton <ross.burton-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
writes:

> the source, so enabling systemd may well lead to libsystemd-* libraries
> sneaking into your rescue image.

for socket activation and sd_notify(), only libsystemd-daemon is required
which is

-rwxr-xr-x    1 root     root         11644 Feb 25 09:27 /lib/libsystemd-daemon.so.0.0.7

~ # ldd /lib/libsystemd-daemon.so.0.*
        libdl.so.2 => /lib/libdl.so.2 (0x441e0000)
        librt.so.1 => /lib/librt.so.1 (0x441c8000)
        libc.so.6 => /lib/libc.so.6 (0x44040000)
        /lib/ld-linux.so.3 (0x44010000)
        libpthread.so.0 => /lib/libpthread.so.0 (0x441a0000)

~ # ldd -u -r /lib/libsystemd-daemon.so.0.*
Unused direct dependencies:
        /lib/libdl.so.2


Ok, it needs more space than nothing and small things like this can sum
up easily.  But it is still a significant difference between having
libsystemd-daemon in the image or the whole systemd.


> The systemd libraries will have to be reviewed to verify that installing
> one of them doesn't pull in systemd itself.

afair, libsystemd-daemon was designed to be as light weighted as possible
and will never depend on the rest of systemd.



Enrico

Re: RFE: make the init manager an image feature (again)

[Otavio Salvador]

On Sun, Feb 24, 2013 at 7:04 PM, Ross Burton <ross.burton@intel.com> wrote:
> On Sunday, 24 February 2013 at 14:06, Otavio Salvador wrote:
>> > DISTRO_FEATURES contains the init script *style* that you want: sysvinit or systemd. These are not mutually exclusive so specifying both will get you both directly in packages that support both. I'm still not convinced we need to split these out into separate packages, the size impact is practically negligible and the dependencies are effectively redundant as you'll have trouble booting without an init system. Instead the postinsts should wrap the initscript fragments in checks.
>>
>>
>> The size impact it not negligible; specially for initramfs images but
>> what concerns me even more is the upgrade path from previous users of
>> meta-oe systemd.
>
>
> I obviously didn't make myself clear - the size impact is negligible when you're talking about just the init script - the dependencies on systemd/updatercd could be recommends at most, as the postinst scripts could check what init system they have before calling any tools.  Either way a rescue image that boots using busybox init shouldn't have systemd, clearly.
>> I'd like to have an upgrade path.
>
> Well, strictly speaking oe-core itself doesn't have an upgrade path to consider… Why can't any distros that shipped with meta-systemd (or just in meta-systemd) have an include that injects the RPROVIDES/RREPLACES/RCONFLICTS for the packages that they enabled?

Well, in this way why meta-oe or meta-systemd fork systemd code and do
it as previously? I think this is the wrong route.

-- 
Otavio Salvador                             O.S. Systems
E-mail: otavio@ossystems.com.br  http://www.ossystems.com.br
Mobile: +55 53 9981-7854              http://projetos.ossystems.com.br

Re: RFE: make the init manager an image feature (again)

[Khem Raj]

On Sun, Feb 24, 2013 at 2:37 AM, Ross Burton <ross.burton@intel.com> wrote:
> Hi,
>
> Just brainstorming out loud, but here's a suggestion that might just please everyone:
>
> A virtual provider for the init manager, which can be overridden per-image (for main / rescue images).

Yes I was thinking about it too but it has to be made one off case.
Since we wont be able to support all combinations


>
> DISTRO_FEATURES contains the init script *style* that you want: sysvinit or systemd.  These are not mutually exclusive so specifying both will get you both directly in packages that support both.  I'm still not convinced we need to split these out into separate packages, the size impact is practically negligible and the dependencies are effectively redundant as you'll have trouble booting without an init system.  Instead the postinsts should wrap the initscript fragments in checks.

systemd lives fine with initscripts. In some cases the adhoc unit
files just call these initscripts so in a way systemd does
expect the initscripts to be around. So systemd is sort of addon and
thats why ${PN}-systemd worked well. So we can have
a perfect sysvinit based system and then we replace it with systemd
and add the ${PN}-systemd packages into image and it becomes
a systemd based image. I think as porting happens this case of
wrapping initscripts into systemd service files will continue
unless individual packages get their own unit files which are
rewritten and do not use initscripts anymore.

>
> Those distro features will also be used to enable/disable features in the source, so enabling systemd may well lead to libsystemd-* libraries sneaking into your rescue image.  The systemd libraries will have to be reviewed to verify that installing one of them doesn't pull in systemd itself.
>
> Here's the fun bit - we can't have two builds of udev in the same distro, and that's what could happen, so I think we'll need to cut down to one udev instead of two.

well we can make a udev recipe which uses exact sources used by systemd.

>
> I'm not entirely convinced this is a good plan yet myself though…
>
> Ross
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

RFE: make the init manager an image feature (again)

[Daniel Lazzari]

Sorry I didn't pipe up earlier but we are still working from denzil so I hadn't noticed the steady move away from supporting both sysvinit and systemd in a single distro. We have a similar use case to Otavio where we are hoping to move to systemd this year, but we'll need to use sysvinit for our rescue images since they operate out of a ramfs where space is precious. These images are all built out of the same distro and it seems like it would be a nightmare to try and support 2 distros just so we could have our rescue images.

Daniel Lazzari Jr.
Firmware Engineer
dlazzari@leapfrog.com