Submission of 'emit-buildvars' class?

Submission of 'emit-buildvars' class?

[Enrico Scholz]

Hello,

we are using for some time an 'emit-buildvars' class which exports
bitbake variables into makefile fragments.  This class is used for for
developing of kernel, u-boot and other low level software and for
generic packaging.

The class supports exporting images over userspace NFS and is available
at

  https://github.com/sigma-embedded/meta-de.sigma-chemnitz/blob/sumo/classes/elito-emit-buildvars.bbclass
  https://github.com/sigma-embedded/meta-de.sigma-chemnitz/tree/sumo/scripts/buildvars
  https://github.com/sigma-embedded/meta-de.sigma-chemnitz/blob/sumo/classes/elito-nfs-export.bbclass

It is much faster than 'devshell' and can be used in a scripted
environment.  A complete SDK is far too heavy weight for our tasks
and not flexible enough.

Is there common interest in such a functionality? If so, I would split
it out from our layer and submit it for core.



A more detailed description:

The generated makefile fragment contains information about the per-package
sysroot and can be used to set e.g. $PATH.  The system requires a small
Makefile to instantiate these variables; e.g. for common cases something
like

 ---- workspace/Makefile.common ---
|  # --*- makefile -*--
| WORKSPACE_DIR := $(abspath $(dir $(lastword $(MAKEFILE_LIST))))
| DEPLOY_DIR ?= ${WORKSPACE_DIR}/../build/tmp/deploy
| MACHINE ?= imx8m-var-dart
| 
| _BUILDVAR_STYLE ?= _generic_
| include ${DEPLOY_DIR}/buildvars/${MACHINE}/$R.mk
| 
| %:
| 	${MAKE} $@

is enough.  For kernel development, the makefile contains some more
information about the image which is going to be exported over NFS.

 ---- workspace/Makefile.kernel ---
| # --*- makefile -*--
| WORKSPACE_DIR := $(abspath $(dir $(lastword $(MAKEFILE_LIST))))
| DEPLOY_DIR ?= ${WORKSPACE_DIR}/../build/tmp/deploy
| MACHINE ?= imx8m-var-dart
| 
| _BUILDVAR_STYLE := kernel
| IMAGE_RECIPE = camtest-image
| 
| HAVE_NFSROOT=1
| include ${WORKSPACE_DIR}/../sources/de.sigma-chemnitz.core/mk/nfs-opt.mk
| include ${DEPLOY_DIR}/buildvars/${MACHINE}/linux-mainline.mk


Now, generate the '.mk' fragments either manually:

| $ bitbake busybox -c emit_buildvars

or enable it within the recipe (recommended when working on it)

| BUILDVARS_EMIT = "true"

and build the package.


When you try to fix something in 'busybox', extract the sources
somewhere (e.g. in workspace/) and call

| make -f ../Makefile.common R=busybox all

to build it.


Else, see the class for more details.



Enrico

Re: Submission of 'emit-buildvars' class?

[Alexander Kanavin]

2018-08-20 16:01 GMT+02:00 Enrico Scholz via Openembedded-core
<openembedded-core@lists.openembedded.org>:
> we are using for some time an 'emit-buildvars' class which exports
> bitbake variables into makefile fragments.  This class is used for for
> developing of kernel, u-boot and other low level software and for
> generic packaging.

I don't quite understand how this is useful? Can you provide specific
usage scenarios please?

Alex

Re: Submission of 'emit-buildvars' class?

[Enrico Scholz]

Alexander Kanavin <alex.kanavin@gmail.com> writes:

>> we are using for some time an 'emit-buildvars' class which exports
>> bitbake variables into makefile fragments.  This class is used for for
>> developing of kernel, u-boot and other low level software and for
>> generic packaging.
>
> I don't quite understand how this is useful? Can you provide specific
> usage scenarios please?

I ma doing perhaps 70% of my development for kernel, u-boot and barebox.
Building them with the actual toolchain requires only a

| make -f ../Makefile.kernel tftp

Result is available after a few seconds on the TFTP server; when using a
specialized variant ('kbuild-file=...'), even faster so that it can be
used in Emacs flymake mode.


Ditto for other programs (testsuites or so); you write your recipe with
all the 'DEPENDS', execute the 'emit_buildvars' task and build your
application with

| make -f ../Makefile.common R=my-app

When you see that something is missing in DEPENDS, add it, reexecute
'emit_buildvars' and it will be in the per-package sysroot.  Excluding
the build time of the dependency, this won't take more than 30 seconds.


Or, it is trivial to run unfsd in the enviornment of the image recipe
and to build kernel modules with

| make -f ../Makefile.kernel modules modules-install

These modules will be installed there and can be used immediately on the
target system:

| # rmmod foo-module
| # modprobe foo-module


I do not know, how this can be done with plain OE.  As said, SDKs are too
heavy weighted, 'devshell' too slow and can not be scripted, 'externalsrc'
is broken, slow and can not be scripted.


Enrico

Re: Submission of 'emit-buildvars' class?

[Alexander Kanavin]

2018-08-20 17:58 GMT+02:00 Enrico Scholz <enrico.scholz@sigma-chemnitz.de>:
> I do not know, how this can be done with plain OE.  As said, SDKs are too
> heavy weighted, 'devshell' too slow and can not be scripted, 'externalsrc'
> is broken, slow and can not be scripted.

For better or worse, Yocto Project has standardized on devtool (not
devshell!) and externalsrc for these kind of local development
workflows. You are very welcome to improve and add to them, so they
are not broken or slow, but a wholly different mechanism is not likely
to be accepted.

Alex

Re: Submission of 'emit-buildvars' class?

[Enrico Scholz]

Alexander Kanavin <alex.kanavin@gmail.com> writes:

>> I do not know, how this can be done with plain OE.  As said, SDKs are too
>> heavy weighted, 'devshell' too slow and can not be scripted, 'externalsrc'
>> is broken, slow and can not be scripted.
>
> For better or worse, Yocto Project has standardized on devtool (not
> devshell!)

'devtool' is for writing recipes, not for writing software.


> and externalsrc for these kind of local development workflows. You are
> very welcome to improve and add to them, so they are not broken

I tried 'externalsrc' after the per-package sysroot changes but just run
into bugs (class tried to write into read-only mounted source dir, every
2nd 'do_fetch' failed because sysroot was already populated).


Doing a 'make -f ../Makefile.common install' which writes data directly
into the NFS root is much faster than doing all the steps of the dev*
tools.


> or slow,

They *are* slow.

| $ cd workspace/kernel
| 
| $ git commit -m 'empty' --allow-empty
| 
| $ time bitbake linux-mainline
| real    4m7.417s
| user    0m0.763s
| sys     0m0.110s

vs.

| time make -f ../Makefile.kernel  -j12
| real    0m13.204s
| user    0m54.888s
| sys     0m17.970s



Even a plain 'bitbake'  is takes 0.5 sec:

| $ time bitbake
| Nothing to do.  Use 'bitbake world' to build everything, or run 'bitbake --help' for usage information.
| 
| real    0m0.588s
| user    0m0.255s
| sys     0m0.023s

vs.

| $ time make
| real    0m0.007s
| user    0m0.002s
| sys     0m0.005s



And some things are impossible because they are not supported by the
design.  E.g. how would would you implement the flymake 'check-syntax'
target[1]

| check-syntax:
|         gcc -o /dev/null -S ${CHK_SOURCES} || true

with devtool/devshell/externalsrc?


Deciding which task to execute (compile, do_compile_kernelmodules) and
installation into NFS root is not supported by dev* either.



Enrico


Footnotes: 
[1]  https://www.gnu.org/software/emacs/manual/html_mono/flymake.html#Example_002d_002d_002dConfiguring-a-tool-called-via-make

-- 
SIGMA Chemnitz GmbH       Registergericht:   Amtsgericht Chemnitz HRB 1750
Am Erlenwald 13           Geschaeftsfuehrer: Grit Freitag, Frank Pyritz
09128 Chemnitz