OEM supplied yocto image is impossible to work with

OEM supplied yocto image is impossible to work with

[Kent Dorfman]

Oh, and this is the THIRD time I've tried posting this...you certainly
don't make the yocto mailing lists easy to subscribe to.

Lemme appologize in avance because parts of this are going to come
across as rants directed at both our OEM and yocto.  Hopefully "yinz"
have thick skins.

I've been doing embedded system design since the beginning of the
millenium (mostly VXworks, with some linux intel embedded, PPC
cross-dev chain stuff, and bare metal C/C++ microcontrollers) and now
I'm tasked with implementing a very mission critical set of embedded
processors in an aerospace project.  I've not used yocto before.  The
closest thing being buildroot.

Anyway, the OEM for our boards was chosen before I came on board and
while I can find no fault with the zynq based hardware, their software
SDK and support is fracking terrible.

Like most OEMs, their prefered model is to give the customer just
enough information to try and force us to pay them to build a turnkey
solution on our behalf, which is not an option in our enterprise.

Their minimal yocto based SDK and reference implementation:

1) as is, isn't suitable for our mission needs, where we must make
changes to the base image,  kernel, and initramfs.  They seem to
expect customers to only add on apps to the UBI rootfs and not screw
with anything else.

2) has many closed source packages in it that will only build from
source when in their local intranet.  Deleting the cache and
attempting a complete source level rebuild consistently fails.

3) isn't documented at all, and they will only answer direct, well
phrased questions, instead of volunteering information that meets our
stated goals.

4) them being a multinational company causes additional legal,
information sharing, and logistical problems

Questions/problems in yocto where the documentation kind of sucks, are:

1) the necessity to "clean build" is inherent in any software
endeavour, yet the simple equivalent of "make clean && make image" is
nowhere to be easily found in yocto.  It is implied that deleting the
tmp/ and sstate_cache/ directories should have the same affect, but is
that safe?  What will it break?  I need to be confident in the process
when I go scream at the OEM, telling them that their build tool is
incomplete for our air-gapped environment.

2) related to above, yocto needs a much better description of the
expected directory tree within a project.

3) the relationship between bitbake and devtool needs to be better
documented and both utilities need to be better documented themselves.
trying to run bitbake manually causes path errors (null entry in path)
when in fact, bitbake itself is setting a path somewhere incorrectly.
my path has no null entries or "." in it.

4) yocto documentation fails in presenting a good explanation of the
difference between packages, layers, recipes, and images.  Also, I've
seen cases where virtual/kernel is used to check-out the kernel to
work on it, yet no virtual/kernel directory exists in the layers
directory tree, so a better explanation of the mapping between real
directories and virtual names is warranted. ... and yes, I know what
BB files are for.

5) a big area that seems to be lacking is the ability to inquire in a
yocto build as to what is included in it.  This is especially
important if the responsible party didn't actually create the build,
as is our case.  We're left with trying to guess what the OEM did, and
why.

6) the seeming inability of yocto to build reproducable binary images
is a serious shortcoming in IA environments.  The first step in
development for us is to baseline the reference build provided by the
OEM and then make incremental changes to it, but yocto doesn't seem to
have a way to validate that the build done in-house is identical to
the pre-built images supplied on the board.  Is there a way to "forced
sequentialize" the yocto process so that images can be reproducable?

7) a large part of me wants to throw away the OEM build and start from
scratch, but I have no information about how to correctly import their
"blob only" packages into a separate yocto project.  Hell, I am not
convinced that their blobs will remain persistent if I delete the
cache because I don't know whether they were created in their network
and  expected to always live in the cache, thus negating the ability
of a customer to do a "clean build"

Anyway, yes, I've read what docs I can find on yocto, and aside from
falling asleep several times in the process, the docs really are not
helping me with the problem at hand: using an existing yocto build
provided by a possibly unscrupulous vendor.

Re: [yocto] OEM supplied yocto image is impossible to work with

[Josef Holzmayr]

Hello Kent!

Sorry to hear that your presumably first contact with Yocto respectively
OpenEmbedded technology is being so painful. Please find some thought
and comments inline.

On Fri, Dec 27, 2019 at 12:00:13PM -0500, Kent Dorfman wrote:
> Oh, and this is the THIRD time I've tried posting this...you certainly
> don't make the yocto mailing lists easy to subscribe to.
> 
> Lemme appologize in avance because parts of this are going to come
> across as rants directed at both our OEM and yocto.  Hopefully "yinz"
> have thick skins.

We have, by now.

> I've been doing embedded system design since the beginning of the
> millenium (mostly VXworks, with some linux intel embedded, PPC
> cross-dev chain stuff, and bare metal C/C++ microcontrollers) and now
> I'm tasked with implementing a very mission critical set of embedded
> processors in an aerospace project.  I've not used yocto before.  The
> closest thing being buildroot.

"close" is really somewhat relative here. The words that trigger my
alarm here are "mission critical" and "aerospace". Anything that I will
write below is no legal advice, and not fit for any form of
certification or audit. If these are your requirements, then there are
companies in the Yocto ecosystem that are willing to offer their
services.
 
> Anyway, the OEM for our boards was chosen before I came on board and
> while I can find no fault with the zynq based hardware, their software
> SDK and support is fracking terrible.

BSPs are a constant cause of pain for us giving Yocto support too. If
a board vendor screws up, we get to pick up the pieces too many times.

> Like most OEMs, their prefered model is to give the customer just
> enough information to try and force us to pay them to build a turnkey
> solution on our behalf, which is not an option in our enterprise.

Little to add, besides... that it might have been a bad choice or
negotiation, if you're only now learning that they want to charge you
additionally. Thats something we obviously can't help you with.
 
> Their minimal yocto based SDK and reference implementation:
> 
> 1) as is, isn't suitable for our mission needs, where we must make
> changes to the base image,  kernel, and initramfs.  They seem to
> expect customers to only add on apps to the UBI rootfs and not screw
> with anything else.

If they are not willing to hand out all sources plus metadata layers,
then thats a total red flag. Yet again, not something we could change.

> 2) has many closed source packages in it that will only build from
> source when in their local intranet.  Deleting the cache and
> attempting a complete source level rebuild consistently fails.

Red flag. Once again.

> 3) isn't documented at all, and they will only answer direct, well
> phrased questions, instead of volunteering information that meets our
> stated goals.

And another one.

> 4) them being a multinational company causes additional legal,
> information sharing, and logistical problems

Sorry, I mean... didn't you request at least some form of eval kit? Demo
boards? To actually understand what you are buying?

Up to this point, yes, I can feel your frustration, but there's little
to say other than that you probably have a bad partnership by now.

So now, on to things that I can hopefully say more positive things.

> Questions/problems in yocto where the documentation kind of sucks, are:
> 
> 1) the necessity to "clean build" is inherent in any software
> endeavour, yet the simple equivalent of "make clean && make image" is
> nowhere to be easily found in yocto.  It is implied that deleting the
> tmp/ and sstate_cache/ directories should have the same affect, but is
> that safe?  What will it break?  I need to be confident in the process
> when I go scream at the OEM, telling them that their build tool is
> incomplete for our air-gapped environment.

The cleanest without throwing away the whole build that you can do is
remove everything in the build directory besides "conf". Yet there are
probably two sides to your actual question. a "bitbake -c clean
$YOURRECIPE" is the perfect equivalent to make clean, "bitbake
$YOURRECIPE" equivalates to "make". Wiping tmp and sstate cache is
pretty close to a full clean rebuild of the whole system, and no, it
shall not break. If it does, there is either something wrong with your
build setup (can happen) or you hit a bug (happens less often, but
happens too). The second side is actually running a full system rebuild,
which includes the complete build setup. Actually there is no
Yocto-inhernt way to do it, as different people have different needs -
and nobody managed a one size fits all solution until now. You can look
at the autobuilder as provided by the Yocto Project, and at the various
approaches here [1];

> 2) related to above, yocto needs a much better description of the
> expected directory tree within a project.

Within a project? Could you please elaborate? 
 
> 3) the relationship between bitbake and devtool needs to be better
> documented and both utilities need to be better documented themselves.
> trying to run bitbake manually causes path errors (null entry in path)
> when in fact, bitbake itself is setting a path somewhere incorrectly.
> my path has no null entries or "." in it.

bitbake and devtool can and should be used side by side. Running bitbake
manually, as you put it, is the definitively primary way of doing
things. So if that eeks out for, there is probably something strange
with your setup. If you can provide a more extensive log or error
message, we will happily...erm, at least honestly try to help.

> 4) yocto documentation fails in presenting a good explanation of the
> difference between packages, layers, recipes, and images.  Also, I've
> seen cases where virtual/kernel is used to check-out the kernel to
> work on it, yet no virtual/kernel directory exists in the layers
> directory tree, so a better explanation of the mapping between real
> directories and virtual names is warranted. ... and yes, I know what
> BB files are for.

I tend to agree on the mapping part.

> 5) a big area that seems to be lacking is the ability to inquire in a
> yocto build as to what is included in it.  This is especially
> important if the responsible party didn't actually create the build,
> as is our case.  We're left with trying to guess what the OEM did, and
> why.

Now this is certainly not true. Even in the utmost default setup, a
manifest file will be created along the image which tells you which
packages including version and license that went into it.

bitbake -g $TARGETOFYOURCHOICE will give you a dot file to inspect the
complete dependency graph of your chosen target. This is a bit too much
in-depth at times, but extremely powerful.

And last but not least, its a good practise to have buildhistory enabled
[2]. This offers extremely detailed information about each and every
package and dependency that went into your build, down to each single
file, package size,...

> 6) the seeming inability of yocto to build reproducable binary images
> is a serious shortcoming in IA environments.  The first step in
> development for us is to baseline the reference build provided by the
> OEM and then make incremental changes to it, but yocto doesn't seem to
> have a way to validate that the build done in-house is identical to
> the pre-built images supplied on the board.  Is there a way to "forced
> sequentialize" the yocto process so that images can be reproducable?

There is work being done on reproductible builds, but we're not there
yet. Might not be the answer you want or need, but the honest state of
affairs.

> 7) a large part of me wants to throw away the OEM build and start from
> scratch, but I have no information about how to correctly import their
> "blob only" packages into a separate yocto project.  Hell, I am not
> convinced that their blobs will remain persistent if I delete the
> cache because I don't know whether they were created in their network
> and  expected to always live in the cache, thus negating the ability
> of a customer to do a "clean build"

Including pre-produced blobs in a Yocto build is not pretty, but often
handy or impossible to avoid. Please see [3] for the documentation on
it.
 
> Anyway, yes, I've read what docs I can find on yocto, and aside from
> falling asleep several times in the process, the docs really are not
> helping me with the problem at hand: using an existing yocto build
> provided by a possibly unscrupulous vendor.

In a nutshell: whenever somebody hands you a binary build, without the
corresponding set of sources and metadata, then you are out of luck and
in for a lot of pain. That exactly the way of how to NOT use yocto when
selling hardware. Some do, and unfortunately we, as the Yocto community
can't do much about it, other than try and help the folks who struggle
with it. And go buy our hardware in another place.

Having said all this. Yes, we know that we have a very steep learning
curve and lots of places to go wrong - yet on the other hand if you make
it through *AND* if it fits your needs, then you are awarded with an
amount of power that is beyond comparison to about any other build
system. But those are two big ifs, yes, and the latter we cannot answer
for you. Only help in the first one.

This might not be the best time of the year as most of us are on
vacation, but feel free to head into #yocto on the freenode network,
where you can easily get in touch with us directly, and we really try to
offer good help and advice. Starting from Jan 7th, you can also expect
better response times :)

And while maybe not an exact fit for you, there still might be
interesting sessions in this playlist for you [4].

With that, I hope I could give a somewhat helpful but certainly
honest answer.

Greetz

[1] https://stackoverflow.com/questions/58863254/how-to-manage-meta-layers-for-a-yocto-project-build-configs-in-git/58865947#58865947
[2] https://www.yoctoproject.org/docs/3.0.1/dev-manual/dev-manual.html#maintaining-build-output-quality
[3] https://www.yoctoproject.org/docs/3.0.1/dev-manual/dev-manual.html#packaging-externally-produced-binaries
[4] https://www.youtube.com/playlist?list=PLD4M5FoHz-TxMfBFrDKfIS_GLY25Qsfyj

-- 
———————————————
Josef Holzmayr
Software Developer Embedded Systems

Tel: +49 8444 9204-48
Fax: +49 8444 9204-50

R-S-I Elektrotechnik GmbH & Co. KG
Woelkestrasse 11
D-85301 Schweitenkirchen
www.rsi-elektrotechnik.de
———————————————
Amtsgericht Ingolstadt – GmbH: HRB 191328 – KG: HRA 170393
Geschäftsführer: Dr.-Ing. Michael Sorg, Dipl.-Ing. Franz Sorg
Ust-IdNr: DE 128592548 

_____________________________________________________________
Amtsgericht Ingolstadt - GmbH: HRB 191328 - KG: HRA 170363
Geschäftsführer: Dr.-Ing. Michael Sorg, Dipl.-Ing. Franz Sorg
USt-IdNr.: DE 128592548

Re: [yocto] OEM supplied yocto image is impossible to work with

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 5784 bytes --]

I would like to add that it helps enormously if you introduce yourself to
the community properly. When you do not state what your company is, who is
your OEM, and write from a gmail account using what looks like a pseudonym,
I personally wasn’t entirely sure if you honestly are seeking help or just
aiming to start a flame war, and so abstained from answering. Just my two
euro cents.

Alex

On Fri, 27 Dec 2019 at 20.00, Kent Dorfman <kent.dorfman766@gmail.com>
wrote:

> Oh, and this is the THIRD time I've tried posting this...you certainly
> don't make the yocto mailing lists easy to subscribe to.
>
> Lemme appologize in avance because parts of this are going to come
> across as rants directed at both our OEM and yocto.  Hopefully "yinz"
> have thick skins.
>
> I've been doing embedded system design since the beginning of the
> millenium (mostly VXworks, with some linux intel embedded, PPC
> cross-dev chain stuff, and bare metal C/C++ microcontrollers) and now
> I'm tasked with implementing a very mission critical set of embedded
> processors in an aerospace project.  I've not used yocto before.  The
> closest thing being buildroot.
>
> Anyway, the OEM for our boards was chosen before I came on board and
> while I can find no fault with the zynq based hardware, their software
> SDK and support is fracking terrible.
>
> Like most OEMs, their prefered model is to give the customer just
> enough information to try and force us to pay them to build a turnkey
> solution on our behalf, which is not an option in our enterprise.
>
> Their minimal yocto based SDK and reference implementation:
>
> 1) as is, isn't suitable for our mission needs, where we must make
> changes to the base image,  kernel, and initramfs.  They seem to
> expect customers to only add on apps to the UBI rootfs and not screw
> with anything else.
>
> 2) has many closed source packages in it that will only build from
> source when in their local intranet.  Deleting the cache and
> attempting a complete source level rebuild consistently fails.
>
> 3) isn't documented at all, and they will only answer direct, well
> phrased questions, instead of volunteering information that meets our
> stated goals.
>
> 4) them being a multinational company causes additional legal,
> information sharing, and logistical problems
>
> Questions/problems in yocto where the documentation kind of sucks, are:
>
> 1) the necessity to "clean build" is inherent in any software
> endeavour, yet the simple equivalent of "make clean && make image" is
> nowhere to be easily found in yocto.  It is implied that deleting the
> tmp/ and sstate_cache/ directories should have the same affect, but is
> that safe?  What will it break?  I need to be confident in the process
> when I go scream at the OEM, telling them that their build tool is
> incomplete for our air-gapped environment.
>
> 2) related to above, yocto needs a much better description of the
> expected directory tree within a project.
>
> 3) the relationship between bitbake and devtool needs to be better
> documented and both utilities need to be better documented themselves.
> trying to run bitbake manually causes path errors (null entry in path)
> when in fact, bitbake itself is setting a path somewhere incorrectly.
> my path has no null entries or "." in it.
>
> 4) yocto documentation fails in presenting a good explanation of the
> difference between packages, layers, recipes, and images.  Also, I've
> seen cases where virtual/kernel is used to check-out the kernel to
> work on it, yet no virtual/kernel directory exists in the layers
> directory tree, so a better explanation of the mapping between real
> directories and virtual names is warranted. ... and yes, I know what
> BB files are for.
>
> 5) a big area that seems to be lacking is the ability to inquire in a
> yocto build as to what is included in it.  This is especially
> important if the responsible party didn't actually create the build,
> as is our case.  We're left with trying to guess what the OEM did, and
> why.
>
> 6) the seeming inability of yocto to build reproducable binary images
> is a serious shortcoming in IA environments.  The first step in
> development for us is to baseline the reference build provided by the
> OEM and then make incremental changes to it, but yocto doesn't seem to
> have a way to validate that the build done in-house is identical to
> the pre-built images supplied on the board.  Is there a way to "forced
> sequentialize" the yocto process so that images can be reproducable?
>
> 7) a large part of me wants to throw away the OEM build and start from
> scratch, but I have no information about how to correctly import their
> "blob only" packages into a separate yocto project.  Hell, I am not
> convinced that their blobs will remain persistent if I delete the
> cache because I don't know whether they were created in their network
> and  expected to always live in the cache, thus negating the ability
> of a customer to do a "clean build"
>
> Anyway, yes, I've read what docs I can find on yocto, and aside from
> falling asleep several times in the process, the docs really are not
> helping me with the problem at hand: using an existing yocto build
> provided by a possibly unscrupulous vendor.
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
>
> View/Reply Online (#47824):
> https://lists.yoctoproject.org/g/yocto/message/47824
> Mute This Topic: https://lists.yoctoproject.org/mt/69288358/1686489
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub  [
> alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

[-- Attachment #2: Type: text/html, Size: 6889 bytes --]

Re: [yocto] OEM supplied yocto image is impossible to work with

[Kent Dorfman]

On 12/27/19, Josef Holzmayr <holzmayr@rsi-elektrotechnik.de> wrote:
> Hello Kent!

thanks for the followup!

> "close" is really somewhat relative here. The words that trigger my
> alarm here are "mission critical" and "aerospace". Anything that I will
> write below is no legal advice, and not fit for any form of
> certification or audit. If these are your requirements, then there are
> companies in the Yocto ecosystem that are willing to offer their
> services.

yeah...no legal advice is expected online...just a forum to vent in, sometimes.



> BSPs are a constant cause of pain for us giving Yocto support too. If
> a board vendor screws up, we get to pick up the pieces too many times.

Yup...been there, done that.



> Little to add, besides... that it might have been a bad choice or
> negotiation, if you're only now learning that they want to charge you
> additionally. Thats something we obviously can't help you with.

Yes, very bad choice in a field with no notably good choices.

> If they are not willing to hand out all sources plus metadata layers,
> then thats a total red flag. Yet again, not something we could change.

OK, to be clear, I have access to meta-layers, but a large percentage
of the BB files point to private git repos, and they don't volunteer
the gitshallow  for a repo unless I know enough to specifically ask
them for that repo, and assuming it's not something they are claiming
is IP.


> Sorry, I mean... didn't you request at least some form of eval kit? Demo
> boards? To actually understand what you are buying?

bad decisions were made before I came on-board, and I did ask them for
access to the SDK before we took dilivery of hardware, and they balked
at that idea...which justifiably left me with an uneasy feeling.  Of
course I would have denied delivery if I had seen the state of their
reference SDK beforehand.


> Up to this point, yes, I can feel your frustration, but there's little
> to say other than that you probably have a bad partnership by now.

Oh yeah, I think we're about to head into the "grudgingly conformant"
phase of the relationship.


> The cleanest without throwing away the whole build that you can do is
> remove everything in the build directory besides "conf". Yet there are
> probably two sides to your actual question. a "bitbake -c clean
> $YOURRECIPE" is the perfect equivalent to make clean, "bitbake
> $YOURRECIPE" equivalates to "make". Wiping tmp and sstate cache is
> pretty close to a full clean rebuild of the whole system, and no, it
> shall not break. If it does, there is either something wrong with your
> build setup (can happen) or you hit a bug (happens less often, but
> happens too). The second side is actually running a full system rebuild,
> which includes the complete build setup. Actually there is no
> Yocto-inhernt way to do it, as different people have different needs -
> and nobody managed a one size fits all solution until now. You can look
> at the autobuilder as provided by the Yocto Project, and at the various
> approaches here [1];

I empty the conf/ sstate_cache/ and tmp/ directories and try to do a
devtool build-image, but failures ensue based on it trying to load
from private git repos, and some jibberish about locked signatures not
matching.  I just need to know that I'm reverting to a source level
build "the right way".


>> 2) related to above, yocto needs a much better description of the
>> expected directory tree within a project.
>
> Within a project? Could you please elaborate?

They provided and "ark" embedded SDK, that was presumably generated
from yocto.  I'm not differentiating between projects because there is
only one, so I'm mean within the SDK directory tree.


> bitbake and devtool can and should be used side by side. Running bitbake
> manually, as you put it, is the definitively primary way of doing
> things. So if that eeks out for, there is probably something strange
> with your setup. If you can provide a more extensive log or error
> message, we will happily...erm, at least honestly try to help.

Thanks.  bitbake fails when run manually, telling me that there is a
"." or null path element when I know for a fact that there is not one.
Apparently when devtool calls bitbake it sets up the path correctly,
but when it is called manually (it's not in the shell path), it
doesn't like the PATH var.  There are some online references to this
problem but no authoritative solution or description of why.



>> 5) a big area that seems to be lacking is the ability to inquire in a
>> yocto build as to what is included in it.  This is especially
>> important if the responsible party didn't actually create the build,
>> as is our case.  We're left with trying to guess what the OEM did, and
>> why.
>
> Now this is certainly not true. Even in the utmost default setup, a
> manifest file will be created along the image which tells you which
> packages including version and license that went into it.

A manifest shows packages, not layers/recipes/packages, nor a
breakdown of what is included as a consequence of what layers or
recipes it exists in.  This makes trying to modify a build that
someone else has created, more difficult.


> bitbake -g $TARGETOFYOURCHOICE will give you a dot file to inspect the
> complete dependency graph of your chosen target. This is a bit too much
> in-depth at times, but extremely powerful.

TARGETOFYOURCHOICE?  What is the glue between "target-name" and
available meta-layers and recipes?


> And last but not least, its a good practise to have buildhistory enabled
> [2]. This offers extremely detailed information about each and every
> package and dependency that went into your build, down to each single
> file, package size,...

Agreed.  I think their SDK has it enabled by default but I need to check.



> Including pre-produced blobs in a Yocto build is not pretty, but often
> handy or impossible to avoid. Please see [3] for the documentation on
> it.

Well, unless they give us git access I don't see any way around it.


> In a nutshell: whenever somebody hands you a binary build, without the
> corresponding set of sources and metadata, then you are out of luck and
> in for a lot of pain. That exactly the way of how to NOT use yocto when
> selling hardware. Some do, and unfortunately we, as the Yocto community
> can't do much about it, other than try and help the folks who struggle
> with it. And go buy our hardware in another place.

I wish I did have options, but if you know business, they are all
about free, as long as it doesn't pose any encumberance on them.  I
need to see just how far the OEM is will to go, and if not far enough,
then I've to evaluate other options.


> With that, I hope I could give a somewhat helpful but certainly
> honest answer.


Gracias, Amigo!

Re: [yocto] OEM supplied yocto image is impossible to work with

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 8203 bytes --]

To me it seems that the SDK is the wrong kind of delivery to begin with (as
it is basically a frozen target image configuration without the ability to
customize the entire system build; Yocto SDKs are meant for application
development). What you need is a hardware BSP meta-layer with ability to
integrate the layer into your private build configurations.

If you can get the OEM to agree to this, we can discuss acceptance criteria
for such a layer.

Alex

On Sat, 28 Dec 2019 at 00:52, Kent Dorfman <kent.dorfman766@gmail.com>
wrote:

> On 12/27/19, Josef Holzmayr <holzmayr@rsi-elektrotechnik.de> wrote:
> > Hello Kent!
>
> thanks for the followup!
>
> > "close" is really somewhat relative here. The words that trigger my
> > alarm here are "mission critical" and "aerospace". Anything that I will
> > write below is no legal advice, and not fit for any form of
> > certification or audit. If these are your requirements, then there are
> > companies in the Yocto ecosystem that are willing to offer their
> > services.
>
> yeah...no legal advice is expected online...just a forum to vent in,
> sometimes.
>
>
>
> > BSPs are a constant cause of pain for us giving Yocto support too. If
> > a board vendor screws up, we get to pick up the pieces too many times.
>
> Yup...been there, done that.
>
>
>
> > Little to add, besides... that it might have been a bad choice or
> > negotiation, if you're only now learning that they want to charge you
> > additionally. Thats something we obviously can't help you with.
>
> Yes, very bad choice in a field with no notably good choices.
>
> > If they are not willing to hand out all sources plus metadata layers,
> > then thats a total red flag. Yet again, not something we could change.
>
> OK, to be clear, I have access to meta-layers, but a large percentage
> of the BB files point to private git repos, and they don't volunteer
> the gitshallow  for a repo unless I know enough to specifically ask
> them for that repo, and assuming it's not something they are claiming
> is IP.
>
>
> > Sorry, I mean... didn't you request at least some form of eval kit? Demo
> > boards? To actually understand what you are buying?
>
> bad decisions were made before I came on-board, and I did ask them for
> access to the SDK before we took dilivery of hardware, and they balked
> at that idea...which justifiably left me with an uneasy feeling.  Of
> course I would have denied delivery if I had seen the state of their
> reference SDK beforehand.
>
>
> > Up to this point, yes, I can feel your frustration, but there's little
> > to say other than that you probably have a bad partnership by now.
>
> Oh yeah, I think we're about to head into the "grudgingly conformant"
> phase of the relationship.
>
>
> > The cleanest without throwing away the whole build that you can do is
> > remove everything in the build directory besides "conf". Yet there are
> > probably two sides to your actual question. a "bitbake -c clean
> > $YOURRECIPE" is the perfect equivalent to make clean, "bitbake
> > $YOURRECIPE" equivalates to "make". Wiping tmp and sstate cache is
> > pretty close to a full clean rebuild of the whole system, and no, it
> > shall not break. If it does, there is either something wrong with your
> > build setup (can happen) or you hit a bug (happens less often, but
> > happens too). The second side is actually running a full system rebuild,
> > which includes the complete build setup. Actually there is no
> > Yocto-inhernt way to do it, as different people have different needs -
> > and nobody managed a one size fits all solution until now. You can look
> > at the autobuilder as provided by the Yocto Project, and at the various
> > approaches here [1];
>
> I empty the conf/ sstate_cache/ and tmp/ directories and try to do a
> devtool build-image, but failures ensue based on it trying to load
> from private git repos, and some jibberish about locked signatures not
> matching.  I just need to know that I'm reverting to a source level
> build "the right way".
>
>
> >> 2) related to above, yocto needs a much better description of the
> >> expected directory tree within a project.
> >
> > Within a project? Could you please elaborate?
>
> They provided and "ark" embedded SDK, that was presumably generated
> from yocto.  I'm not differentiating between projects because there is
> only one, so I'm mean within the SDK directory tree.
>
>
> > bitbake and devtool can and should be used side by side. Running bitbake
> > manually, as you put it, is the definitively primary way of doing
> > things. So if that eeks out for, there is probably something strange
> > with your setup. If you can provide a more extensive log or error
> > message, we will happily...erm, at least honestly try to help.
>
> Thanks.  bitbake fails when run manually, telling me that there is a
> "." or null path element when I know for a fact that there is not one.
> Apparently when devtool calls bitbake it sets up the path correctly,
> but when it is called manually (it's not in the shell path), it
> doesn't like the PATH var.  There are some online references to this
> problem but no authoritative solution or description of why.
>
>
>
> >> 5) a big area that seems to be lacking is the ability to inquire in a
> >> yocto build as to what is included in it.  This is especially
> >> important if the responsible party didn't actually create the build,
> >> as is our case.  We're left with trying to guess what the OEM did, and
> >> why.
> >
> > Now this is certainly not true. Even in the utmost default setup, a
> > manifest file will be created along the image which tells you which
> > packages including version and license that went into it.
>
> A manifest shows packages, not layers/recipes/packages, nor a
> breakdown of what is included as a consequence of what layers or
> recipes it exists in.  This makes trying to modify a build that
> someone else has created, more difficult.
>
>
> > bitbake -g $TARGETOFYOURCHOICE will give you a dot file to inspect the
> > complete dependency graph of your chosen target. This is a bit too much
> > in-depth at times, but extremely powerful.
>
> TARGETOFYOURCHOICE?  What is the glue between "target-name" and
> available meta-layers and recipes?
>
>
> > And last but not least, its a good practise to have buildhistory enabled
> > [2]. This offers extremely detailed information about each and every
> > package and dependency that went into your build, down to each single
> > file, package size,...
>
> Agreed.  I think their SDK has it enabled by default but I need to check.
>
>
>
> > Including pre-produced blobs in a Yocto build is not pretty, but often
> > handy or impossible to avoid. Please see [3] for the documentation on
> > it.
>
> Well, unless they give us git access I don't see any way around it.
>
>
> > In a nutshell: whenever somebody hands you a binary build, without the
> > corresponding set of sources and metadata, then you are out of luck and
> > in for a lot of pain. That exactly the way of how to NOT use yocto when
> > selling hardware. Some do, and unfortunately we, as the Yocto community
> > can't do much about it, other than try and help the folks who struggle
> > with it. And go buy our hardware in another place.
>
> I wish I did have options, but if you know business, they are all
> about free, as long as it doesn't pose any encumberance on them.  I
> need to see just how far the OEM is will to go, and if not far enough,
> then I've to evaluate other options.
>
>
> > With that, I hope I could give a somewhat helpful but certainly
> > honest answer.
>
>
> Gracias, Amigo!
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
>
> View/Reply Online (#47827):
> https://lists.yoctoproject.org/g/yocto/message/47827
> Mute This Topic: https://lists.yoctoproject.org/mt/69288358/1686489
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub  [
> alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

[-- Attachment #2: Type: text/html, Size: 9880 bytes --]

Re: [yocto] OEM supplied yocto image is impossible to work with

[Nicholas Krause]

[-- Attachment #1: Type: text/plain, Size: 9703 bytes --]



On 12/27/19 5:17 PM, Alexander Kanavin wrote:
> To me it seems that the SDK is the wrong kind of delivery to begin 
> with (as it is basically a frozen target image configuration without 
> the ability to customize the entire system build; Yocto SDKs are meant 
> for application development). What you need is a hardware BSP 
> meta-layer with ability to integrate the layer into your private build 
> configurations.
>
> If you can get the OEM to agree to this, we can discuss acceptance 
> criteria for such a layer.
>
> Alex
Alex,
Isn't that even mentioned is the manual as best practices? Odd a OEM 
would not
even follow that :(.

Nick
>
> On Sat, 28 Dec 2019 at 00:52, Kent Dorfman <kent.dorfman766@gmail.com 
> <mailto:kent.dorfman766@gmail.com>> wrote:
>
>     On 12/27/19, Josef Holzmayr <holzmayr@rsi-elektrotechnik.de
>     <mailto:holzmayr@rsi-elektrotechnik.de>> wrote:
>     > Hello Kent!
>
>     thanks for the followup!
>
>     > "close" is really somewhat relative here. The words that trigger my
>     > alarm here are "mission critical" and "aerospace". Anything that
>     I will
>     > write below is no legal advice, and not fit for any form of
>     > certification or audit. If these are your requirements, then
>     there are
>     > companies in the Yocto ecosystem that are willing to offer their
>     > services.
>
>     yeah...no legal advice is expected online...just a forum to vent
>     in, sometimes.
>
>
>
>     > BSPs are a constant cause of pain for us giving Yocto support
>     too. If
>     > a board vendor screws up, we get to pick up the pieces too many
>     times.
>
>     Yup...been there, done that.
>
>
>
>     > Little to add, besides... that it might have been a bad choice or
>     > negotiation, if you're only now learning that they want to
>     charge you
>     > additionally. Thats something we obviously can't help you with.
>
>     Yes, very bad choice in a field with no notably good choices.
>
>     > If they are not willing to hand out all sources plus metadata
>     layers,
>     > then thats a total red flag. Yet again, not something we could
>     change.
>
>     OK, to be clear, I have access to meta-layers, but a large percentage
>     of the BB files point to private git repos, and they don't volunteer
>     the gitshallow  for a repo unless I know enough to specifically ask
>     them for that repo, and assuming it's not something they are claiming
>     is IP.
>
>
>     > Sorry, I mean... didn't you request at least some form of eval
>     kit? Demo
>     > boards? To actually understand what you are buying?
>
>     bad decisions were made before I came on-board, and I did ask them for
>     access to the SDK before we took dilivery of hardware, and they balked
>     at that idea...which justifiably left me with an uneasy feeling.  Of
>     course I would have denied delivery if I had seen the state of their
>     reference SDK beforehand.
>
>
>     > Up to this point, yes, I can feel your frustration, but there's
>     little
>     > to say other than that you probably have a bad partnership by now.
>
>     Oh yeah, I think we're about to head into the "grudgingly conformant"
>     phase of the relationship.
>
>
>     > The cleanest without throwing away the whole build that you can
>     do is
>     > remove everything in the build directory besides "conf". Yet
>     there are
>     > probably two sides to your actual question. a "bitbake -c clean
>     > $YOURRECIPE" is the perfect equivalent to make clean, "bitbake
>     > $YOURRECIPE" equivalates to "make". Wiping tmp and sstate cache is
>     > pretty close to a full clean rebuild of the whole system, and no, it
>     > shall not break. If it does, there is either something wrong
>     with your
>     > build setup (can happen) or you hit a bug (happens less often, but
>     > happens too). The second side is actually running a full system
>     rebuild,
>     > which includes the complete build setup. Actually there is no
>     > Yocto-inhernt way to do it, as different people have different
>     needs -
>     > and nobody managed a one size fits all solution until now. You
>     can look
>     > at the autobuilder as provided by the Yocto Project, and at the
>     various
>     > approaches here [1];
>
>     I empty the conf/ sstate_cache/ and tmp/ directories and try to do a
>     devtool build-image, but failures ensue based on it trying to load
>     from private git repos, and some jibberish about locked signatures not
>     matching.  I just need to know that I'm reverting to a source level
>     build "the right way".
>
>
>     >> 2) related to above, yocto needs a much better description of the
>     >> expected directory tree within a project.
>     >
>     > Within a project? Could you please elaborate?
>
>     They provided and "ark" embedded SDK, that was presumably generated
>     from yocto.  I'm not differentiating between projects because there is
>     only one, so I'm mean within the SDK directory tree.
>
>
>     > bitbake and devtool can and should be used side by side. Running
>     bitbake
>     > manually, as you put it, is the definitively primary way of doing
>     > things. So if that eeks out for, there is probably something strange
>     > with your setup. If you can provide a more extensive log or error
>     > message, we will happily...erm, at least honestly try to help.
>
>     Thanks.  bitbake fails when run manually, telling me that there is a
>     "." or null path element when I know for a fact that there is not one.
>     Apparently when devtool calls bitbake it sets up the path correctly,
>     but when it is called manually (it's not in the shell path), it
>     doesn't like the PATH var.  There are some online references to this
>     problem but no authoritative solution or description of why.
>
>
>
>     >> 5) a big area that seems to be lacking is the ability to
>     inquire in a
>     >> yocto build as to what is included in it.  This is especially
>     >> important if the responsible party didn't actually create the
>     build,
>     >> as is our case.  We're left with trying to guess what the OEM
>     did, and
>     >> why.
>     >
>     > Now this is certainly not true. Even in the utmost default setup, a
>     > manifest file will be created along the image which tells you which
>     > packages including version and license that went into it.
>
>     A manifest shows packages, not layers/recipes/packages, nor a
>     breakdown of what is included as a consequence of what layers or
>     recipes it exists in.  This makes trying to modify a build that
>     someone else has created, more difficult.
>
>
>     > bitbake -g $TARGETOFYOURCHOICE will give you a dot file to
>     inspect the
>     > complete dependency graph of your chosen target. This is a bit
>     too much
>     > in-depth at times, but extremely powerful.
>
>     TARGETOFYOURCHOICE?  What is the glue between "target-name" and
>     available meta-layers and recipes?
>
>
>     > And last but not least, its a good practise to have buildhistory
>     enabled
>     > [2]. This offers extremely detailed information about each and every
>     > package and dependency that went into your build, down to each
>     single
>     > file, package size,...
>
>     Agreed.  I think their SDK has it enabled by default but I need to
>     check.
>
>
>
>     > Including pre-produced blobs in a Yocto build is not pretty, but
>     often
>     > handy or impossible to avoid. Please see [3] for the
>     documentation on
>     > it.
>
>     Well, unless they give us git access I don't see any way around it.
>
>
>     > In a nutshell: whenever somebody hands you a binary build,
>     without the
>     > corresponding set of sources and metadata, then you are out of
>     luck and
>     > in for a lot of pain. That exactly the way of how to NOT use
>     yocto when
>     > selling hardware. Some do, and unfortunately we, as the Yocto
>     community
>     > can't do much about it, other than try and help the folks who
>     struggle
>     > with it. And go buy our hardware in another place.
>
>     I wish I did have options, but if you know business, they are all
>     about free, as long as it doesn't pose any encumberance on them.  I
>     need to see just how far the OEM is will to go, and if not far enough,
>     then I've to evaluate other options.
>
>
>     > With that, I hope I could give a somewhat helpful but certainly
>     > honest answer.
>
>
>     Gracias, Amigo!
>     -=-=-=-=-=-=-=-=-=-=-=-
>     Links: You receive all messages sent to this group.
>
>     View/Reply Online (#47827):
>     https://lists.yoctoproject.org/g/yocto/message/47827
>     Mute This Topic: https://lists.yoctoproject.org/mt/69288358/1686489
>     Group Owner: yocto+owner@lists.yoctoproject.org
>     <mailto:yocto%2Bowner@lists.yoctoproject.org>
>     Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub
>     [alex.kanavin@gmail.com <mailto:alex.kanavin@gmail.com>]
>     -=-=-=-=-=-=-=-=-=-=-=-
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
>
> View/Reply Online (#47828): https://lists.yoctoproject.org/g/yocto/message/47828
> Mute This Topic: https://lists.yoctoproject.org/mt/69288358/3618653
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub  [xerofoify@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-


[-- Attachment #2: Type: text/html, Size: 14568 bytes --]

Re: [yocto] OEM supplied yocto image is impossible to work with

[Kent Dorfman]

On 12/27/19, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
> To me it seems that the SDK is the wrong kind of delivery to begin with (as
> it is basically a frozen target image configuration without the ability to
> customize the entire system build; Yocto SDKs are meant for application
> development). What you need is a hardware BSP meta-layer with ability to
> integrate the layer into your private build configurations.
>
> If you can get the OEM to agree to this, we can discuss acceptance criteria
> for such a layer.
>
> Alex

I think that was their intention from the start.  they wanted to only
give their customer the ability to produce user level apps, without
modifying the system, but for our enterprise that is unrealistic.

They do provide a meta-vendor layer, but the stuff in it points to
private git repos and stuff like systemd startup scripts, so if it
ends up not in the cache then the build will fail unless the build is
done from within their corporate intranet.

So no, they would definitely NOT meet any real acceptance criteria.

Re: [yocto] OEM supplied yocto image is impossible to work with

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1651 bytes --]

On Sat, 28 Dec 2019 at 01:25, Kent Dorfman <kent.dorfman766@gmail.com>
wrote:

> On 12/27/19, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
> > To me it seems that the SDK is the wrong kind of delivery to begin with
> (as
> > it is basically a frozen target image configuration without the ability
> to
> > customize the entire system build; Yocto SDKs are meant for application
> > development). What you need is a hardware BSP meta-layer with ability to
> > integrate the layer into your private build configurations.
> >
> > If you can get the OEM to agree to this, we can discuss acceptance
> criteria
> > for such a layer.
> >
> > Alex
>
> I think that was their intention from the start.  they wanted to only
> give their customer the ability to produce user level apps, without
> modifying the system, but for our enterprise that is unrealistic.
>
> They do provide a meta-vendor layer, but the stuff in it points to
> private git repos and stuff like systemd startup scripts, so if it
> ends up not in the cache then the build will fail unless the build is
> done from within their corporate intranet.
>
> So no, they would definitely NOT meet any real acceptance criteria.
>

There are ways to adapt such a proprietary layer. If the intranet location
is set with a bitbake variable, then you can adjust the variable to point
to *your* intranet, and mirror all the needed pieces. But I can imagine
that you first need to discuss with your management what the problem is,
then explain to the OEM what you, the customer, expect from them, and
somehow codify it in a signed contract kind of thing.

Alex

[-- Attachment #2: Type: text/html, Size: 2125 bytes --]

Re: [yocto] OEM supplied yocto image is impossible to work with

[Kent Dorfman]

On 12/27/19, Alexander Kanavin <alex.kanavin@gmail.com> wrote:


>
> There are ways to adapt such a proprietary layer. If the intranet location
> is set with a bitbake variable, then you can adjust the variable to point
> to *your* intranet, and mirror all the needed pieces. But I can imagine
> that you first need to discuss with your management what the problem is,
> then explain to the OEM what you, the customer, expect from them, and
> somehow codify it in a signed contract kind of thing.

Agreed.  That is kind of where we are at.  Yes, these closed source
layers/packages simply point to closed source repos.  I need them to
prepare pre-compiled archives for these packages that can be overlaid
into our rootfs, and to make the build process use these local
archives instead of attempting real builds on those packages.  Then I
don't believe it is any longer an issue.

They gave us the kernel source but there are sysadmin apps and
libraries that talk to their FPGA that they haven't shared the
contents thereof.