[Buildroot] Report from the Buildroot Summer Camp

[Buildroot] Report from the Buildroot Summer Camp

[Thomas Petazzoni]

Hello all,

As was announced on the list, six core/regular Buildroot developers
met from July 1 to July 5 in Toulouse, France, for the "Buildroot
Summer Camp". Peter Korsgaard, Samuel Martin, Yann E. Morin, Romain
Naour, Maxime Hadjinlian and myself participated to this event. Thanks
to spending 5 days full time on Buildroot, we managed to work on a
significant number of topics.

First of all, most of our effort was concentrated on reducing the
backlog of patches in patchwork.

 - We started the meeting with about ~250 patches in patchwork.

 - We ended the meeting with only 98 patches remaining in patchwork,
   which is the lowest number that has been reached in many years. We
   hope to continue this effort in the coming weeks, and welcome your
   help to review and test the patches that remain in the backlog.

 - We merged a total of 357 patches (from
   3d7aa870a162b7b4cbb840f79a56e1f2c0e6d89c to
   563c693fdce1363ea6a66af9b8f73987de8b6ff9).

 - 1402 e-mails have been exchanged on the list during the meeting.

Noticeable things that have been merged:

 - Bump of the SELinux stack. Thanks to Adam Duskett for providing the
   patches.

 - Addition of Qemu defconfig for MIPS r6. Thanks to Vicente Olivert
   Riera for providing the patches.

 - Many new packages added: python-engineio, python-socketio,
   python-tomako, openblas, python-xlwt, python-xlutils, libuio,
   python-scapy3k, babeld, systemd-bootchart, acpitool, libaacs,
   libbdplus, libvdpau, ninja, supertuxkart, acpica, runc,
   docker-containerd, jemalloc, texinfo, rs485conf, circus, dante.

 - Better support for Fortran was added, including support for Fortran
   in the external toolchains, and the addition of a hidden boolean
   that tells packages whether Fortran is available or not. Several
   packages started using it. Thanks to Samuel Martin and Vicente
   Olivert Riera for working on this topic.

 - Addition of BR2_REPRODUCIBLE, which will be used progressively to
   make Buildroot builds byte-to-byte reproducible. Thanks to Gilles
   Chanteperdrix for starting this effort.

 - A significant amount of systemd patches have been merged, mainly
   from Gave Evans. Thanks to Maxime Hadjinlian and Yann E. Morin for
   their review and additional patches on the topic.

 - Support for fetching git submodules in packages has been added. Use
   <pkg>_GIT_SUBMODULES = YES in your package to enable the fetching
   of git submodules. Thanks to Yann E. Morin for working on this
   topic.

 - The automatic "derivation" of host package dependencies from their
   corresponding target dependencies has been removed. This was too
   "obscure" and caused a number of mistakes over time. We thought it
   was better to make the dependencies of the target and host packages
   separate and explicit. A large number of patches was needed to
   achieve this goal. Thanks to Julien Floret for his work on the
   topic.

 - The package infrastructure has been changed to not attempt to
   download when no <pkg>_VERSION variable is set (instead of defining
   it to "undefined"). This allowed to clean-up a number of packages
   for which the source code is in Buildroot, or when there is no
   source code at all (virtual packages for example).

 - Support for building the "system emulation" mode in the host-qemu
   package has been added. Thanks to Simon Maes for providing the
   initial patches.

 - The dependencies on BR2_PACKAGE_BUSYBOX_SHOW_OTHERS have been moved
   from package/Config.in to their respective packages.

 - libgpg-error was finally bumped (to version 1.23). This was long
   overdue, but wasn't a simple bump since libgpg-error now has some
   architecture specific bits, so the architecture dependency had to
   be propagated to all reverse dependencies.

 - A number of tweaks have been made on how the options of some
   packages are displayed. However, the complete series from Yann
   E. Morin that moved more packages to use "menuconfig" has not been
   merged. Both Peter and I felt that it was not really improving the
   usability.

 - Support for the Blackfin architecture in the internal toolchain has
   been added. Thanks to Waldemar Brodkorb for working on this topic.

 - A host-texinfo package has been added, and is now used as a
   dependency of binutils and gdb when needed. This should hopefully
   fix the longstanding "makeinfo missing" build failures.

 - Packages can now register "target finalize hooks" using
   <pkg>_TARGET_FINALIZE_HOOKS rather than the global
   TARGET_FINALIZE_HOOKS variable. This makes sure the package hooks
   are only called when the package is enabled.

 - We have added a new option, BR2_ROOTFS_POST_FAKEROOT_SCRIPT, which
   allows to pass to Buildroot some scripts that will be called
   *within* the fakeroot environment. It started from a patch provided
   by Cam Hutchison that proposed to support rootfs overlays passed as
   tarball, and extracted within the fakeroot environment (so that
   permissions and owernship are preserved). We replaced Cam's
   proposal by a more general solution, similar to the post-build and
   post-image script, but that runs within the fakeroot environment,
   right before the root filesystem image is produced.

Other topics:

 - We decided to reject the "merged defconfig" proposal from Sam
   Bobroff, since we believe this should be handled outside of
   Buildroot
   (http://lists.busybox.net/pipermail/buildroot/2016-June/165384.html).

 - As mentionned above, we decided to not merge a significant part of
   Yann E. Morin's proposal to use 'menuconfig' in many more places,
   though we took some of the patches
   (http://lists.busybox.net/pipermail/buildroot/2016-July/166512.html).

 - We started discussing the series from Yann E. Morin that splits the
   skeleton to improve systemd support, and adds support for using a
   read-only rootfs with systemd. Some of the preparatory patches have
   been merged. I'll sum up the discussion as a reply to the relevant
   thread.

I think I can say that all of us really enjoyed this Summer Camp, and
we look forward to the next Buildroot meeting next to the Embedded
Linux Conference Europe.

I would like to thank all the participants who joined this Summer Camp
and participated. I would also like to thank OpenWide/Smile [1] for
sponsoring the participation of Romain Naour to the Summer Camp.

Best regards,

Thomas

[1] http://ingenierie.openwide.fr/
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] Report from the Buildroot Summer Camp

[Yann E. MORIN]

Thomas, All,

Chiming in a bit late...

On 2016-07-06 22:17 +0200, Thomas Petazzoni spake thusly:
> As was announced on the list, six core/regular Buildroot developers
> met from July 1 to July 5 in Toulouse, France, for the "Buildroot
> Summer Camp". Peter Korsgaard, Samuel Martin, Yann E. Morin, Romain
> Naour, Maxime Hadjinlian and myself participated to this event. Thanks
> to spending 5 days full time on Buildroot, we managed to work on a
> significant number of topics.

Yes, this was a very intense WE. A little bit sleep-deprived, but
nothing that can't be compensated by the pelasure to do all that work!

> Noticeable things that have been merged:
[--SNIP--]

All that thanks to Thomas and Peter who were around so we could poke
them until they applied such or such patches! Good job! ;-)

Thanks to all participants, it was again fun!

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] Report from the Buildroot Summer Camp

[Thomas Petazzoni]

Hello,

From Saturday to Wednesday, 7 Buildroot developers gathered in
Brittany, France for a 5-day hackaton. Peter Korsgaard, Yann Morin,
Arnout Vandecappelle, Samuel Martin, Romain Naour, Maxime Hadjinlian
and myself participated to this event. Fabrice Fontaine, a Buildroot
contributor from the area, also joined us for a barbecue.

You had anyway probably already discovered that "something" was going
on by looking at the mailing list and commit activity.

In total, we applied 384 patches, and made useful progress on a number
of topics, even though they haven't all been merged yet.

Here are the main highlights of things that have been merged:

 - Improvements to the runtime test infrastructure:

   * Patches from Ricardo that make the tests executable in parallel,
     as well as other improvements, have been merged.

   * The runtime tests are now executed daily by the Gitlab CI
     infrastructure (see .gitlab-ci.yml for details). This was done by
     Thomas Petazzoni.

     Therefore, Gitlab now sends an e-mail every day to the mailing
     list with the result of those tests. Unfortunately, Gitlab
     notification e-mails do not contain a To: field in their headers,
     so this might break filtering for people filtering on To/Cc
     instead of using the List-Id header.

   * A few tests have been added to test external toolchain related
     features

 - Utilities directly useful to the user have been moved from
   support/scripts/ to the top-level utils/ directory. It currently
   contains check-package, get-developers, scancpan, scanpypi,
   size-stats-compare, test-pkg, and the new brmake.

   The latter is a small make wrapper that you can use to get a
   cleaner output of Buildroot builds, while keeping the entire build
   log into a file.

   The patches, and brmake script, were provided by Yann E. Morin.

 - We no longer download patches from Github, and generally changed
   our policy to only use the per-package <pkg>_PATCH variable when
   downloaded patches are guaranteed to be stable. Indeed, patches
   from Github are generated on the fly, and their contents can change
   slightly over time, making the hash of such patches invalid. We
   have therefore modified all packages downloading patches from
   Github so that they contain the patches directly in the package
   directory.

   This was done by Thomas Petazzoni, after discussion with Peter
   Korsgaard and Arnout Vandecappelle.

 - Support for storing the hash of license files inside the .hash file
   has been added. The idea is that this will allow us to detect when
   the contents of license files are changed by upstream developers.

   Patches were provided by Yann E. Morin.

 - Support for building Busybox as individual binaries has finally
   been merged. It's a small thing, but it was the last missing piece
   for basic SELinux support, an effort that was started many years
   ago by folks from Rockwell Collins. Thanks to Arnout Vandecapelle
   for doing the final polishing on those patches.

 - The ext2 filesystem image creation logic has been changed to use
   mkfs.ext2/3/4 from e2fsprogs, rather than our mke2img wrapper +
   genext2fs. This allows to directly generate ext4 images, benefit
   from automated optimal block size calculation, and more.

   Thanks to Samuel Martin, who picked up the patches from Yann
   E. Morin and S?bastien Szymanski, merged them together, and did the
   final polishing.

 - A major rework of the gettext handling has been made. We now rely
   on uClibc-ng stub libintl implementation, which allows to avoid the
   need for building gettext if you don't need internationalization. A
   BR2_SYSTEM_ENABLE_NLS option has been added to enable Native
   Language Support, which is disabled by default.

   This also allows to avoid lots of static linking issues with
   libintl, and many libintl related workarounds have already been
   removed from the tree. Indeed, we no longer support providing
   internationalization in static linking configurations.

   This was done by Thomas Petazzoni, with lots of input and
   discussion from Arnout Vandecappelle.

 - The $(HOST_DIR)/usr has been removed, and everything moved one
   level up into $(HOST_DIR). We have kept a $(HOST_DIR)/usr symbolic
   link pointing to $(HOST_DIR) to preserve backward compatibility.

   This was all done by Arnout Vandecappelle.

 - We have merged a few patches from Yann E. Morin preparing the way
   for improvements in systemd support. Yann patches create a
   ifupdown-scripts package to contain the ifupdown related files,
   instead of having them in the skeleton. They also fix a number of
   issues with timezone handling.

 - Peter Korsgaard has handled a number of security updates, and
   published the maintenance releases 2017.02.4 and 2017.05.1.

 - Obviously, many, many package updates have been merged.

Here are the highlights of other topics being worked on and/or
discussed, but that haven't been completed/merged yet:

 - Maxime Hadjinlian has worked on a patch series reworking the
   download logic, with two goals:

    1. Store the downloaded files in per-package sub-folders under
       $(DL_DIR). Instead of having all downloaded files in a flat
       directory, they would be stored in $(DL_DIR)/<package>/. This
       will avoid collisions if two packages download a file of the
       same name.

    2. Implement Git caching, to avoid re-cloning entire Git
       repositories when just the version of a package has
       changed. The idea is that when fetching from Git, we keep a
       bare repository in $(DL_DIR)/<package>/git/, and use it
       whenever new downloads of the same package are needed, even for
       different versions. This will greatly speed up the download of
       large packages like the Linux kernel, as it won't re-clone the
       entire Git repository each time the version changes.

   Maxime has sent one iteration of the patches on the mailing list,
   but more work is needed to polish them.

 - Yann E. Morin has worked on his patch series to split the skeleton
   package into different packages depending on the init system
   (busybox, sysv, systemd) and his support for using systemd on a
   read-only filesystem.

   Yann had already sent such a work on the mailing list a while
   ago. Hopefully, with the work done during the last days, a new
   iteration will be available soon.

 - Romain Naour has worked on LLVM/Clang support, in order to support
   llvm-pipe in Mesa, as well as OpenCL. He has posted his patches for
   review and comments on the mailing list, as there are some
   non-trivial remaining issues.

 - Discussion about the libressl integration and libssl/ssl virtual
   package has occured. We have decide to simply add libssl as a
   regular standalone package, with a "depends on
   !BR2_PACKAGE_OPENSSL". Packages that want to use OpenSSL or
   LibreSSL will have to explicitly support both. Depending on the
   results, we will see later if a virtual package is useful or not.

 - While not present at the hackaton, Wolfgang Grandegger has worked
   very hard on improving the reuse of the toolchain as a relocatable
   SDK. This work has tow main components. The first is to rewrite the
   binaries so they refer to their dynamic libraries as a relative
   path.  This is done with the fix-rpath script and the patchelf
   program. The other part is a script that should be run when the
   toolchain is moved to a different location, to update all the text
   files (e.g. .pc files).  This series is already in its 7th
   iteration but is not yet entirely ready.

We have also discussed the next Buildroot Developers meeting, which
traditionally takes place around the Embedded Linux Conference
Europe. Since this year ELCE takes place from Monday 23 to Wednesday
25 of October, the Buildroot Developers meeting will take place on
Saturday 21 and Sunday 22 of October. We will send a separate e-mail
about this.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com