[Buildroot] Buildroot docker image

[Buildroot] Buildroot docker image

[Kilian Zinnecker via buildroot]

Hello Yann, Arnout, all,

I am looking for a docker image for buildroot to setup a CI. (I am actually 
not very familiar with docker to this day.) I found one on dockerhub, which 
looks like it was created by you: https://hub.docker.com/r/buildroot/base Is 
this some kind of "official" buildroot docker image?

Best regards,
Kilian


_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Buildroot docker image

[Arnout Vandecappelle via buildroot]

On 01/09/2023 08:51, Kilian Zinnecker wrote:
> Hello Yann, Arnout, all,
> 
> I am looking for a docker image for buildroot to setup a CI. (I am actually
> not very familiar with docker to this day.) I found one on dockerhub, which
> looks like it was created by you: https://hub.docker.com/r/buildroot/base Is
> this some kind of "official" buildroot docker image?

  Not really, no.

  First of all, the one on dockerhub is no longer updated. The images we 
actually use are on registry.gitlab.com/buildroot.org/buildroot/base

  But even those images are not really "for user consumption". They are the 
images we use for CI, and they therefore have a bit more stuff in them than 
really needed, e.g. qemu. They are also not updated (on the OS level) - no 
security updates or anything are applied on it.

  It would probably be better if we would also create a small image that people 
can use for just building in their CI. A small image based on e.g. Alpine or 
Arch, and that is updated e.g. daily on dockerhub. However, nobody ever 
contributed such an image so we don't have it.

  Regards,
  Arnout
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Buildroot docker image

[Thomas Petazzoni via buildroot]

On Fri, 1 Sep 2023 09:19:02 +0200
Arnout Vandecappelle via buildroot <buildroot@buildroot.org> wrote:

>   First of all, the one on dockerhub is no longer updated. The images we 
> actually use are on registry.gitlab.com/buildroot.org/buildroot/base
> 
>   But even those images are not really "for user consumption". They are the 
> images we use for CI, and they therefore have a bit more stuff in them than 
> really needed, e.g. qemu. They are also not updated (on the OS level) - no 
> security updates or anything are applied on it.
> 
>   It would probably be better if we would also create a small image that people 
> can use for just building in their CI. A small image based on e.g. Alpine or 
> Arch, and that is updated e.g. daily on dockerhub. However, nobody ever 
> contributed such an image so we don't have it.

Yann has been working on and off on a set of Docker images for
Buildroot. See his branch at
https://gitlab.com/ymorin/buildroot/-/commits/yem/dockers/. I indeed
shared with him recently on IRC that it would be nice to have a really
minimal image, separate from the image used in CI (which has more stuff
enabled, most notably python3).

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Buildroot docker image

[Kilian Zinnecker via buildroot]

Hello Thomas, Arnout and Yann,

[--SNIP--]

> Yann has been working on and off on a set of Docker images for
> Buildroot. See his branch at
> https://gitlab.com/ymorin/buildroot/-/commits/yem/dockers/. I indeed
> shared with him recently on IRC that it would be nice to have a really
> minimal image, separate from the image used in CI (which has more stuff
> enabled, most notably python3).

I agree. This would for example be useful if someone has their own out-of-tree 
customized board support and wants to setup a CI for that.

I started to try to setup a docker (please note again, that I am not very 
familiar with anything regarding docker). As Arnout suggested, I tried Alpine 
as a base. See below a patch, where I add the dockerfile to the support folder. 
It downloads dependencies, then buildroot. When running the docker it tries to 
build a board support. However, during compilation of fakeroot, it throws the 
following error:

======
libfakeroot.c:147:13: error: conflicting types for 'id_t'; have 'int'
  147 | typedef int id_t;
      |             ^~~~
In file included from /usr/include/sys/types.h:57,
                 from communicate.h:84:
/usr/include/bits/alltypes.h:240:18: note: previous declaration of 'id_t' with 
type 'id_t' {aka 'unsigned int'}
  240 | typedef unsigned id_t;
      |                  ^~~~
/bin/bash ./libtool  --tag=CC   --mode=link /usr/bin/gcc  -O2 -I/home/br-user/
buildroot/output/host/include  -L/home/br-user/buildroot/output/host/lib -Wl,-
rpath,/home/br-user/buildroot/output/host/lib -o libfakeroot_time64.la  
libfakeroot_time64_la-libfakeroot_time64.lo  -ldl
libfakeroot.c: In function 'common_setxattr':
libfakeroot.c:140:33: warning: implicit declaration of function 
'SEND_GET_XATTR'; did you mean 'SEND_GET_XATTR64'? [-Wimplicit-function-
declaration]
  140 | #define INT_SEND_GET_XATTR(a,b) SEND_GET_XATTR(a,b,_STAT_VER)
      |                                 ^~~~~~~~~~~~~~
libfakeroot.c:1717:3: note: in expansion of macro 'INT_SEND_GET_XATTR'
 1717 |   INT_SEND_GET_XATTR(st, &xattr);
      |   ^~~~~~~~~~~~~~~~~~
make[3]: *** [Makefile:686: libfakeroot.lo] Error 1
======

I tried to research this. The issue seems to be related is this patch, added 
to an openwrt repository, speaking about fixing the "conflicting types for 
'id_t'" and the "SEND_GET_XATTR: symbol not found" issue:

https://gitea.auro.re/jeltz/openwrt/commit/
a162fe9a96fc83cb890944d40dadaf1091084f98

Now I checked the current soure code used for fakeroot, an it is obtained from 
the fakeroot_1.32.1.orig.tar.gz from here: https://snapshot.debian.org/
archive/debian/20230724T160429Z/pool/main/f/fakeroot/
And indeed it seems to not contain the changes from the openwrt patch. So when 
I download this patch and put it into the package/fakeroot folder, the docker 
continues beyond building the fakeroot tool. However, it is slow and currently 
still building, so I can't yet tell, whether there are other issues. So what 
do you think? Shall we then apply this patch to the fakeroot package? If so, 
the raw fakeroot patch does not contain a "signed-off-by". Would that be a 
problem?

And of course I appreciate feedback regarding the alpine dockerfile. The 
dockerfile downloads the fakeroot patch and puts it into the fakeroot folder 
for now. See the patch below

Best regards,
Kilian

====== Patch below ======

From 7016be22cf4b32e7b359bab5e6434991e1f0d13b Mon Sep 17 00:00:00 2001
From: Kilian Zinnecker <kilian.zinnecker@mail.de>
Date: Sun, 3 Dec 2023 12:10:09 +0100
Subject: [PATCH 1/1] Add alpine based dockerfile

The dockerfile builds a docker based on alpine. The docker builds a
board support with buildroot. It is work in progress to create a
docker based on alpine, containing buildroot for use in CIs.

Signed-off-by: Kilian Zinnecker <kilian.zinnecker@mail.de>
---
 support/docker/alpine-latest/Dockerfile       | 62 +++++++++++++++++++
 .../docker/alpine-latest/build-buildroot.sh   |  6 ++
 2 files changed, 68 insertions(+)
 create mode 100644 support/docker/alpine-latest/Dockerfile
 create mode 100755 support/docker/alpine-latest/build-buildroot.sh

diff --git a/support/docker/alpine-latest/Dockerfile b/support/docker/alpine-
latest/Dockerfile
new file mode 100644
index 0000000000..cafd069766
--- /dev/null
+++ b/support/docker/alpine-latest/Dockerfile
@@ -0,0 +1,62 @@
+FROM alpine:latest
+
+RUN apk add --no-cache wget
+RUN apk add --no-cache linux-headers
+RUN apk add --no-cache which
+RUN apk add --no-cache sed
+RUN apk add --no-cache make
+RUN apk add --no-cache binutils
+RUN apk add --no-cache dpkg
+RUN apk add --no-cache libc6-compat
+RUN apk add --no-cache fakeroot
+RUN apk add --no-cache diffutils
+RUN apk add --no-cache gcc
+RUN apk add --no-cache g++
+RUN apk add --no-cache bash
+RUN apk add --no-cache patch
+RUN apk add --no-cache gzip
+RUN apk add --no-cache bzip2
+RUN apk add --no-cache perl
+RUN apk add --no-cache tar
+RUN apk add --no-cache cpio
+RUN apk add --no-cache unzip
+RUN apk add --no-cache rsync
+RUN apk add --no-cache file
+RUN apk add --no-cache bc
+RUN apk add --no-cache findutils
+
+RUN apk add --no-cache python3
+RUN apk add --no-cache ncurses
+RUN apk add --no-cache qt5-qtbase
+RUN apk add --no-cache glib
+RUN apk add --no-cache gtk+2.0
+RUN apk add --no-cache glade
+RUN apk add --no-cache cvs
+RUN apk add --no-cache git
+RUN apk add --no-cache mercurial
+RUN apk add --no-cache rsync
+RUN apk add --no-cache dropbear-scp
+RUN apk add --no-cache openssh-sftp-server
+RUN apk add --no-cache subversion
+RUN apk add --no-cache openjdk8
+
+RUN apk add --no-cache asciidoc
+RUN apk add --no-cache w3m
+RUN apk add --no-cache graphviz
+RUN apk add --no-cache py3-matplotlib
+
+RUN addgroup --gid 1000 br-user
+RUN adduser --disabled-password --ingroup "br-user" --uid "1000" br-user
+
+USER root
+WORKDIR /home/br-user
+ADD https://github.com/buildroot/buildroot/archive/refs/heads/master.tar.gz /
home/br-user/buildroot.tar.gz
+RUN chown -R br-user /home/br-user
+RUN chgrp -R br-user /home/br-user
+USER br-user
+RUN mkdir /home/br-user/buildroot
+RUN tar -xzvf /home/br-user/buildroot.tar.gz -C /home/br-user/buildroot --
strip-components=1
+RUN wget https://gitea.auro.re/jeltz/openwrt/raw/commit/
7a70f78fb7da4dc3e8c58e7662f4dc623274a78b/tools/fakeroot/patches/400-alpine-
libc.musl-fix.patch -P /home/br-user/buildroot/package/fakeroot/
+ENV HOME /home/br-user
+COPY ./build-buildroot.sh /home/br-user/build-buildroot.sh
+CMD ["/home/br-user/build-buildroot.sh"]
diff --git a/support/docker/alpine-latest/build-buildroot.sh b/support/docker/
alpine-latest/build-buildroot.sh
new file mode 100755
index 0000000000..39d31646ec
--- /dev/null
+++ b/support/docker/alpine-latest/build-buildroot.sh
@@ -0,0 +1,6 @@
+#!/bin/bash
+
+cd buildroot
+make clean distclean clean
+make rock5b_defconfig
+make
--
2.34.1




_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Buildroot docker image

[Arnout Vandecappelle via buildroot]

  Hi Kilian,

  Thanks for looking at this!

On 03/12/2023 12:47, Kilian Zinnecker wrote:
> Hello Thomas, Arnout and Yann,
> 
> [--SNIP--]
> 
>> Yann has been working on and off on a set of Docker images for
>> Buildroot. See his branch at
>> https://gitlab.com/ymorin/buildroot/-/commits/yem/dockers/. I indeed
>> shared with him recently on IRC that it would be nice to have a really
>> minimal image, separate from the image used in CI (which has more stuff
>> enabled, most notably python3).
> 
> I agree. This would for example be useful if someone has their own out-of-tree
> customized board support and wants to setup a CI for that.
> 
> I started to try to setup a docker (please note again, that I am not very
> familiar with anything regarding docker). As Arnout suggested, I tried Alpine

  In retrospect, this was probably a mistake :-) Alpine is based on the musl 
libc, and as you noticed below, there are host tools that don't work with musl. 
Clearly, nobody is using Buildroot on a musl-based build machine at the moment, 
since nobody ever complained about this (and fakeroot always needs to be built).

  So, if we do go for Alpine as the official docker image, we can expect quite a 
lot of breakage... Therefore, before going there, I think we should go for a 
"normal" distro after all for the time being. Of all the traditional distros, 
ubuntu:22.04 has by far the smallest image at 80MB (this actually came as quite 
the surprise to me... I have no idea why Fedora would be twice as large! But it 
does already include Python, so after installing all necessary packages it may 
even out again...).

  So I think it is best to have an Ubuntu-based one as the official one, and 
perhaps a "minimal" variant based on Alpine (that we can use in the autobuilders 
to scrub out problems).

> as a base. See below a patch, where I add the dockerfile to the support folder.
> It downloads dependencies, then buildroot. When running the docker it tries to
> build a board support. However, during compilation of fakeroot, it throws the
> following error:
> 
> ======
> libfakeroot.c:147:13: error: conflicting types for 'id_t'; have 'int'
>    147 | typedef int id_t;
>        |             ^~~~
> In file included from /usr/include/sys/types.h:57,
>                   from communicate.h:84:
> /usr/include/bits/alltypes.h:240:18: note: previous declaration of 'id_t' with
> type 'id_t' {aka 'unsigned int'}
>    240 | typedef unsigned id_t;
>        |                  ^~~~
> /bin/bash ./libtool  --tag=CC   --mode=link /usr/bin/gcc  -O2 -I/home/br-user/
> buildroot/output/host/include  -L/home/br-user/buildroot/output/host/lib -Wl,-
> rpath,/home/br-user/buildroot/output/host/lib -o libfakeroot_time64.la
> libfakeroot_time64_la-libfakeroot_time64.lo  -ldl
> libfakeroot.c: In function 'common_setxattr':
> libfakeroot.c:140:33: warning: implicit declaration of function
> 'SEND_GET_XATTR'; did you mean 'SEND_GET_XATTR64'? [-Wimplicit-function-
> declaration]
>    140 | #define INT_SEND_GET_XATTR(a,b) SEND_GET_XATTR(a,b,_STAT_VER)
>        |                                 ^~~~~~~~~~~~~~
> libfakeroot.c:1717:3: note: in expansion of macro 'INT_SEND_GET_XATTR'
>   1717 |   INT_SEND_GET_XATTR(st, &xattr);
>        |   ^~~~~~~~~~~~~~~~~~
> make[3]: *** [Makefile:686: libfakeroot.lo] Error 1
> ======
> 
> I tried to research this. The issue seems to be related is this patch, added

  "Related to" sounds like it is caused by that patch, but it's actually _fixed_ 
by that patch.

> to an openwrt repository, speaking about fixing the "conflicting types for
> 'id_t'" and the "SEND_GET_XATTR: symbol not found" issue:
> 
> https://gitea.auro.re/jeltz/openwrt/commit/
> a162fe9a96fc83cb890944d40dadaf1091084f98

  Why do you use a fork instead of OpenWRT's upstream repository,
https://git.openwrt.org/openwrt/openwrt.git, or its official github mirror, 
https://github.com/openwrt/openwrt ?

  Also, that commit is cherry-picked from 7a70f78fb7da4dc3e8c58e7662f4dc623274a78b.

> 
> Now I checked the current soure code used for fakeroot, an it is obtained from
> the fakeroot_1.32.1.orig.tar.gz from here: https://snapshot.debian.org/
> archive/debian/20230724T160429Z/pool/main/f/fakeroot/
> And indeed it seems to not contain the changes from the openwrt patch. 

  Yeah, OpenWRT is _very_ bad at upstreaming patches.

> So when
> I download this patch and put it into the package/fakeroot folder, the docker
> continues beyond building the fakeroot tool. However, it is slow and currently
> still building,

  Do you mean it is slower in the container than without it? That's not good...

> so I can't yet tell, whether there are other issues. So what
> do you think? Shall we then apply this patch to the fakeroot package?

  Regardless of whether or not we create an Alpine Dockerfile, it's a good idea 
to have this patch in Buildroot - and to send it upstream!


> If so,
> the raw fakeroot patch does not contain a "signed-off-by". Would that be a
> problem?

  The commit that adds it to OpenWRT _does_ have a signed-off-by, so that's fine.

  In the Buildroot patch, you should refer to the source of the patch (this is 
_not_ the Upstream: tag). You should also send the patch to upstream (i.e. 
Debian) and refer to this submission in the Upstream tag. You should also put 
the original author in Cc of the latter. I'm not entirely sure if you should add 
the original author's signoff or not - I think not though.


> And of course I appreciate feedback regarding the alpine dockerfile. The
> dockerfile downloads the fakeroot patch and puts it into the fakeroot folder
> for now. See the patch below
> 
> Best regards,
> Kilian
> 
> ====== Patch below ======
> 
>  From 7016be22cf4b32e7b359bab5e6434991e1f0d13b Mon Sep 17 00:00:00 2001
> From: Kilian Zinnecker <kilian.zinnecker@mail.de>
> Date: Sun, 3 Dec 2023 12:10:09 +0100
> Subject: [PATCH 1/1] Add alpine based dockerfile
> 
> The dockerfile builds a docker based on alpine. The docker builds a
> board support with buildroot. It is work in progress to create a
> docker based on alpine, containing buildroot for use in CIs.
> 
> Signed-off-by: Kilian Zinnecker <kilian.zinnecker@mail.de>
> ---
>   support/docker/alpine-latest/Dockerfile       | 62 +++++++++++++++++++
>   .../docker/alpine-latest/build-buildroot.sh   |  6 ++
>   2 files changed, 68 insertions(+)
>   create mode 100644 support/docker/alpine-latest/Dockerfile
>   create mode 100755 support/docker/alpine-latest/build-buildroot.sh
> 
> diff --git a/support/docker/alpine-latest/Dockerfile b/support/docker/alpine-
> latest/Dockerfile
> new file mode 100644
> index 0000000000..cafd069766
> --- /dev/null
> +++ b/support/docker/alpine-latest/Dockerfile
> @@ -0,0 +1,62 @@
> +FROM alpine:latest
> +
> +RUN apk add --no-cache wget

  Is wget really needed? Alpine has busybox installed, isn't its wget good 
enough? If for some reason it is not, I think we should first look if we can't 
change our wget calls to fit with busybox wget.

> +RUN apk add --no-cache linux-headers

  Why is linux-headers needed?

> +RUN apk add --no-cache which
> +RUN apk add --no-cache sed

  which and sed are also part of busybox.

> +RUN apk add --no-cache make
> +RUN apk add --no-cache binutils

  I think it's better to have a single command for all the fetches. Otherwise, 
you're creating a lot of separate layers, which is not very efficient. Also, 
keep things alphabetic, if possible.

RUN apk add --no-cache \
	binutils \
	...

> +RUN apk add --no-cache dpkg

  Why is dpkg needed?

  (Repeat for many others...)

> +RUN apk add --no-cache libc6-compat
> +RUN apk add --no-cache fakeroot
> +RUN apk add --no-cache diffutils
> +RUN apk add --no-cache gcc
> +RUN apk add --no-cache g++
> +RUN apk add --no-cache bash
> +RUN apk add --no-cache patch
> +RUN apk add --no-cache gzip
> +RUN apk add --no-cache bzip2
> +RUN apk add --no-cache perl
> +RUN apk add --no-cache tar
> +RUN apk add --no-cache cpio
> +RUN apk add --no-cache unzip
> +RUN apk add --no-cache rsync
> +RUN apk add --no-cache file
> +RUN apk add --no-cache bc

  Actually, we check for bc, but I don't think it's needed for anything... The 
alpine container is a good way to test for this kind of stuff (all normal build 
machines will have bc installed for other reasons).

> +RUN apk add --no-cache findutils
> +
> +RUN apk add --no-cache python3
> +RUN apk add --no-cache ncurses
> +RUN apk add --no-cache qt5-qtbase

  I guess this is for qconfig? Isn't there some qt5-qtgui kind of package needed 
as well?

  But anyway, qconfig is not going to work in a container (at least not easily), 
so I wouldn't bother with it.

> +RUN apk add --no-cache glib

  If anythin needs glib, it should be pulled in automatically as a dependency.

> +RUN apk add --no-cache gtk+2.0
> +RUN apk add --no-cache glade

  Same as for qt, I don't think we need this.

> +RUN apk add --no-cache cvs

  I _think_ host-cvs gets built automatically if it is needed. But perhaps we 
should kill cvs support entirely :-)

  Oh, it actually doesn't get built automatically. It is only needed, however, 
if you want to use a cvs package. So I don't think it should be in the minimal 
docker image.

> +RUN apk add --no-cache git
> +RUN apk add --no-cache mercurial
> +RUN apk add --no-cache rsync
> +RUN apk add --no-cache dropbear-scp
> +RUN apk add --no-cache openssh-sftp-server

  Why do you need this? normally you'd use volumes to "copy" stuff into the 
container.

> +RUN apk add --no-cache subversion
> +RUN apk add --no-cache openjdk8

  We _really_ shouldn't need openjdk, and definitely not such an old version.

> +
> +RUN apk add --no-cache asciidoc
> +RUN apk add --no-cache w3m
> +RUN apk add --no-cache graphviz
> +RUN apk add --no-cache py3-matplotlib
> +
> +RUN addgroup --gid 1000 br-user
> +RUN adduser --disabled-password --ingroup "br-user" --uid "1000" br-user
> +
> +USER root
> +WORKDIR /home/br-user
> +ADD https://github.com/buildroot/buildroot/archive/refs/heads/master.tar.gz /

  Don't use the github mirror, use the actual upstream:
http://gitlab.com/buildroot.org/buildroot

  However, I don't think we should pull the repository here. Instead, we should 
use the already-checked-out buildroot from which this Dockerfile is built. Or we 
shouldn't have any buildroot source in the container and just mount the volume, 
like is done in docker-run.

> home/br-user/buildroot.tar.gz
> +RUN chown -R br-user /home/br-user
> +RUN chgrp -R br-user /home/br-user

  You can do chown -R br-user.br-user to do both at the same time.

> +USER br-user
> +RUN mkdir /home/br-user/buildroot
> +RUN tar -xzvf /home/br-user/buildroot.tar.gz -C /home/br-user/buildroot --
> strip-components=1

  _If_ we put a buildroot inside the container, it should be a git clone, not an 
extracted tarball.

> +RUN wget https://gitea.auro.re/jeltz/openwrt/raw/commit/
> 7a70f78fb7da4dc3e8c58e7662f4dc623274a78b/tools/fakeroot/patches/400-alpine-
> libc.musl-fix.patch -P /home/br-user/buildroot/package/fakeroot/

  This patch should be part of Buildroot itself, not downloaded here.

> +ENV HOME /home/br-user
> +COPY ./build-buildroot.sh /home/br-user/build-buildroot.sh
> +CMD ["/home/br-user/build-buildroot.sh"]

  I think it makes more sense to use

ENTRYPOINT make

That way you can run e.g. "docker run buildroot libcurl-rebuild world"


  Also, current dir should be the buildroot directory.

> diff --git a/support/docker/alpine-latest/build-buildroot.sh b/support/docker/
> alpine-latest/build-buildroot.sh
> new file mode 100755
> index 0000000000..39d31646ec
> --- /dev/null
> +++ b/support/docker/alpine-latest/build-buildroot.sh
> @@ -0,0 +1,6 @@
> +#!/bin/bash
> +
> +cd buildroot
> +make clean distclean clean
> +make rock5b_defconfig
> +make

  This is clearly not super useful as the default entrypoint. If we can't think 
of anything useful, better make it just empty (i.e. the shell).


  Regards,
  Arnout
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Buildroot docker image

[Thomas Petazzoni via buildroot]

On Sat, 9 Dec 2023 22:29:32 +0100
Arnout Vandecappelle <arnout@mind.be> wrote:

>   So, if we do go for Alpine as the official docker image, we can expect quite a 
> lot of breakage... Therefore, before going there, I think we should go for a 
> "normal" distro after all for the time being. Of all the traditional distros, 
> ubuntu:22.04 has by far the smallest image at 80MB (this actually came as quite 
> the surprise to me... I have no idea why Fedora would be twice as large! But it 
> does already include Python, so after installing all necessary packages it may 
> even out again...).

An advantage of Fedora is that their Perl packaging is much more
fine-grained than in Debian/Ubuntu, which allows us to better detect
when we have missing Perl modules.

> > +RUN apk add --no-cache bc  
> 
>   Actually, we check for bc, but I don't think it's needed for anything... The 
> alpine container is a good way to test for this kind of stuff (all normal build 
> machines will have bc installed for other reasons).

I think bc is needed to build the Linux kernel, and we don't build
host-bc as a dependency of the linux package.


> > +USER root
> > +WORKDIR /home/br-user
> > +ADD https://github.com/buildroot/buildroot/archive/refs/heads/master.tar.gz /  
> 
>   Don't use the github mirror, use the actual upstream:
> http://gitlab.com/buildroot.org/buildroot
> 
>   However, I don't think we should pull the repository here. Instead, we should 
> use the already-checked-out buildroot from which this Dockerfile is built. Or we 
> shouldn't have any buildroot source in the container and just mount the volume, 
> like is done in docker-run.
> 
> > home/br-user/buildroot.tar.gz
> > +RUN chown -R br-user /home/br-user
> > +RUN chgrp -R br-user /home/br-user  
> 
>   You can do chown -R br-user.br-user to do both at the same time.

[...]

What is the goal of this new Dockerfile, compared to the one we already
have at support/docker/Dockerfile ?

support/docker/Dockerfile already has a good setup in terms of how the
br-user should be created, what the Dockerfile should use, etc. Why
should we do things differently?

I had one two things to suggest improving compared to our existing
Dockerfile:

- We should have a more minimal Dockerfile that installs really the
  strict minimal requirements that Buildroot needs to do a build, and
  then a second Dockerfile than inherits from the first, and adding
  what's needed so that it can be used as our CI Docker

- We should similarly have combos of Docker file (one minimal, one CI)
  that use different distributions;

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Buildroot docker image

[Yann E. MORIN]

Thomas, All,

On 2023-09-02 09:42 +0200, Thomas Petazzoni via buildroot spake thusly:
> On Fri, 1 Sep 2023 09:19:02 +0200
> Arnout Vandecappelle via buildroot <buildroot@buildroot.org> wrote:
> >   First of all, the one on dockerhub is no longer updated. The images we 
> > actually use are on registry.gitlab.com/buildroot.org/buildroot/base
> > 
> >   But even those images are not really "for user consumption". They are the 
> > images we use for CI, and they therefore have a bit more stuff in them than 
> > really needed, e.g. qemu. They are also not updated (on the OS level) - no 
> > security updates or anything are applied on it.
> > 
> >   It would probably be better if we would also create a small image that people 
> > can use for just building in their CI. A small image based on e.g. Alpine or 
> > Arch, and that is updated e.g. daily on dockerhub. However, nobody ever 
> > contributed such an image so we don't have it.
> 
> Yann has been working on and off on a set of Docker images for
> Buildroot. See his branch at
> https://gitlab.com/ymorin/buildroot/-/commits/yem/dockers/. I indeed
> shared with him recently on IRC that it would be nice to have a really
> minimal image, separate from the image used in CI (which has more stuff
> enabled, most notably python3).

I was looking at this right now, and in fact I believe the existing
image is the perfect fir as the minimal image to use.

Indeed, let's take a look at what we have there, and why we do:

  MTSCR
  ×     bc
  ×     build-essential
        bzr
  ×     ca-certificates
   ×    cmake
  ×     cpio
  ×     cvs
  ×     file
  ×     g++-multilib
  ×     git
   ×    libc6:i386
  ×     libncurses5-dev
  ×     locales
        mercurial
      × openssh-server
     ×× python3
     ×  python3-flake8
     ×  python3-magic
      × python3-nose2
      × python3-pexpect
      × python3-pytest
      × qemu-system-arm
      × qemu-system-misc
      × qemu-system-x86
  ×     rsync
     ×  shellcheck
        subversion
  ×     unzip
  ×     wget

M: strictly mandatory
T: optional, but helps use pre-built toolchains
S: optional, speeds up the build
C: needed to run check-package et al.
R: needed to run the runtime test-suite

Bazaar, mercurial and subversion are optional, and only required when a
package is bzr-, hg-, or svn-hosted, but if we want this image to be
usable generally, it should have all three.

Since mercurial is a python package, we can't drop python3, and the
remaining modules are relatively small in comparison to the rest.

Getting a more minimal image would be just about dropping qemu. That
would make for an image that we could not use in the CI, and that would
not be usable to test the changes done to Buildroot. I don't think that
would mkae for a good image.

The size of this image is currently ~330MiB to download, and ~975MiB of
storage; building an image without qemu would not make much of a
difference.

So, I think this is the best minimal we can get to, as it contains
everything that people should need to build and to run the tooling.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Buildroot docker image

[Yann E. MORIN]

Re.

On 2023-12-10 14:48 +0100, Yann E. MORIN spake thusly:
> Thomas, All,
> 
> On 2023-09-02 09:42 +0200, Thomas Petazzoni via buildroot spake thusly:
> > On Fri, 1 Sep 2023 09:19:02 +0200
> > Arnout Vandecappelle via buildroot <buildroot@buildroot.org> wrote:
> > >   First of all, the one on dockerhub is no longer updated. The images we 
> > > actually use are on registry.gitlab.com/buildroot.org/buildroot/base
> > > 
> > >   But even those images are not really "for user consumption". They are the 
> > > images we use for CI, and they therefore have a bit more stuff in them than 
> > > really needed, e.g. qemu. They are also not updated (on the OS level) - no 
> > > security updates or anything are applied on it.
> > > 
> > >   It would probably be better if we would also create a small image that people 
> > > can use for just building in their CI. A small image based on e.g. Alpine or 
> > > Arch, and that is updated e.g. daily on dockerhub. However, nobody ever 
> > > contributed such an image so we don't have it.
> > 
> > Yann has been working on and off on a set of Docker images for
> > Buildroot. See his branch at
> > https://gitlab.com/ymorin/buildroot/-/commits/yem/dockers/. I indeed
> > shared with him recently on IRC that it would be nice to have a really
> > minimal image, separate from the image used in CI (which has more stuff
> > enabled, most notably python3).
> 
> I was looking at this right now, and in fact I believe the existing
> image is the perfect fir as the minimal image to use.

s/fir/fit/

> Indeed, let's take a look at what we have there, and why we do:
> 
>   MTSCR
>   ×     bc
>   ×     build-essential
>         bzr
>   ×     ca-certificates
>    ×    cmake

cmake should have been in the 'S' column.

Regards,
Yann E. MORIN.

>   ×     cpio
>   ×     cvs
>   ×     file
>   ×     g++-multilib
>   ×     git
>    ×    libc6:i386
>   ×     libncurses5-dev
>   ×     locales
>         mercurial
>       × openssh-server
>      ×× python3
>      ×  python3-flake8
>      ×  python3-magic
>       × python3-nose2
>       × python3-pexpect
>       × python3-pytest
>       × qemu-system-arm
>       × qemu-system-misc
>       × qemu-system-x86
>   ×     rsync
>      ×  shellcheck
>         subversion
>   ×     unzip
>   ×     wget
> 
> M: strictly mandatory
> T: optional, but helps use pre-built toolchains
> S: optional, speeds up the build
> C: needed to run check-package et al.
> R: needed to run the runtime test-suite
> 
> Bazaar, mercurial and subversion are optional, and only required when a
> package is bzr-, hg-, or svn-hosted, but if we want this image to be
> usable generally, it should have all three.
> 
> Since mercurial is a python package, we can't drop python3, and the
> remaining modules are relatively small in comparison to the rest.
> 
> Getting a more minimal image would be just about dropping qemu. That
> would make for an image that we could not use in the CI, and that would
> not be usable to test the changes done to Buildroot. I don't think that
> would mkae for a good image.
> 
> The size of this image is currently ~330MiB to download, and ~975MiB of
> storage; building an image without qemu would not make much of a
> difference.
> 
> So, I think this is the best minimal we can get to, as it contains
> everything that people should need to build and to run the tooling.
> 
> Regards,
> Yann E. MORIN.
> 
> -- 
> .-----------------.--------------------.------------------.--------------------.
> |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
> | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
> | +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
> '------------------------------^-------^------------------^--------------------'
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Buildroot docker image

[Thomas Petazzoni via buildroot]

On Sun, 10 Dec 2023 14:48:14 +0100
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> M: strictly mandatory
> T: optional, but helps use pre-built toolchains
> S: optional, speeds up the build
> C: needed to run check-package et al.
> R: needed to run the runtime test-suite
> 
> Bazaar, mercurial and subversion are optional, and only required when a
> package is bzr-, hg-, or svn-hosted, but if we want this image to be
> usable generally, it should have all three.
> 
> Since mercurial is a python package, we can't drop python3, and the
> remaining modules are relatively small in comparison to the rest.
> 
> Getting a more minimal image would be just about dropping qemu. That
> would make for an image that we could not use in the CI, and that would
> not be usable to test the changes done to Buildroot. I don't think that
> would mkae for a good image.

It's mainly Python3 that is important to drop. Python3 is not part of
our mandatory requirements, and it's a very frequent sources, as people
don't have an easily accessible container to test Buildroot builds
without Python3, so they forget BR2_TARGET_UBOOT_NEEDS_PYTHON3 or
similar options.

We have exactly two packages that use Mercurial.

pygame has the comment "stable 1.9.1 release requires V4L which has
been wiped out of recent Linux kernels, so use latest mercurial
revision until next stable release is out.", but there's a recent
pygame 2.5.2 release from September 2023. And pygame is now on Github:
https://github.com/pygame/pygame.

The other one is dvb-apps, which hasn't seen a single commit since
March 2014: https://linuxtv.org/hg/dvb-apps. We could consider it
obsolete/unmaintained maybe? :-)

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Buildroot docker image

[Yann E. MORIN]

Thomas, All,

On 2023-12-10 18:02 +0100, Thomas Petazzoni spake thusly:
> On Sun, 10 Dec 2023 14:48:14 +0100
> "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
> > M: strictly mandatory
> > T: optional, but helps use pre-built toolchains
> > S: optional, speeds up the build
> > C: needed to run check-package et al.
> > R: needed to run the runtime test-suite
> > 
> > Bazaar, mercurial and subversion are optional, and only required when a
> > package is bzr-, hg-, or svn-hosted, but if we want this image to be
> > usable generally, it should have all three.
> > 
> > Since mercurial is a python package, we can't drop python3, and the
> > remaining modules are relatively small in comparison to the rest.
> > 
> > Getting a more minimal image would be just about dropping qemu. That
> > would make for an image that we could not use in the CI, and that would
> > not be usable to test the changes done to Buildroot. I don't think that
> > would mkae for a good image.
> 
> It's mainly Python3 that is important to drop. Python3 is not part of
> our mandatory requirements, and it's a very frequent sources, as people
> don't have an easily accessible container to test Buildroot builds
> without Python3, so they forget BR2_TARGET_UBOOT_NEEDS_PYTHON3 or
> similar options.

Sure, but removing python3 means removing all the testing infrastructure
as well, and the ability to run check-package and flake8, and then we
would complain people are not running the linters (or are running them
on their host system without the proper versions).

Note also that, if we go the route your suggest, which is:

  - base image without python3 et al. for people to test-build,

  - CI image based on the above, plus all the testing infra;

then the CI would not be able to catch the issues such as a missing
BR2_TARGET_UBOOT_NEEDS_PYTHON3, because python3 would be part of the CI
image.

Also, I believe it is very optimistic to expect all contributors to
build-test in the docker image before they submit their changes. Some
will do (and some already do!), most won't.

> We have exactly two packages that use Mercurial.

The goal of that base image was also supposedly for people to reuse for
themselves, possibly with their own packages, not just for Buildroot.

So, we need three images:

  - an image with only the strictly required tools, which means some
    features will not be available: none of the VCS tools are mandatory,
    so they all must be excluded, yes, that also means git [0]: we can't
    exclude Hg and include git. Either the image is exactly right just
    the minimal with only the mandatory packages, or it is not;

  - an image that has all we need to run our tooling in the CI [1];

  - an image that people can reuse for themselves as a reference image
    that "just works".

I'm arguing that the two latter images are exactly the same, in fact, and
are the image we already have.

I am also arguing that the first image, without all of the optional
packages, is not going to be very useful in practice. Indeed, let's
take uboot as an example: one can chose an arbitrary version from an
arbitrary git tree, and thus you need git, so we can't build a lot of
our defconfig files in an image that has only the strictly mandatory
packages, and this is the image that would be able to catch the python3
issue...

Of course, we could make git part of the mandatory packages. Or we could
introduce host-git and treat it like host-lzip; and so on for host-svn,
host-cvs, host-mercurial, host-bazaar...

And maybe that is what we should do:

 1. promote git from optional to mandatory, like wget is: indeed, it is
    so common and pervasive that it does not make sense to keep it
    optional; then, and only then, it would make sense to have the first
    image with only the mandatory tools;

 2. add host variants of the other VCS tools (where that is not too
    difficult), so that we can use them if missing on the build machine.

[0] the manual states that rsync is both mandatory and optional, I'll
send a patch to fix that.

[1] ideally, it would also be used by the autobuilders, but that's
another topic entirely...

Regards,
Yann E. MORIN.

> pygame has the comment "stable 1.9.1 release requires V4L which has
> been wiped out of recent Linux kernels, so use latest mercurial
> revision until next stable release is out.", but there's a recent
> pygame 2.5.2 release from September 2023. And pygame is now on Github:
> https://github.com/pygame/pygame.
> 
> The other one is dvb-apps, which hasn't seen a single commit since
> March 2014: https://linuxtv.org/hg/dvb-apps. We could consider it
> obsolete/unmaintained maybe? :-)

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Buildroot docker image

[Peter Korsgaard]

>>>>> "Arnout" == Arnout Vandecappelle via buildroot <buildroot@buildroot.org> writes:

Hi,

 >  In retrospect, this was probably a mistake :-) Alpine is based on the
 >  musl libc, and as you noticed below, there are host tools that don't
 >  work with musl. Clearly, nobody is using Buildroot on a musl-based
 >  build machine at the moment, since nobody ever complained about this
 > (and fakeroot always needs to be built).

Yes, lets stick to a "normal" glibc based distribution.

 >  So, if we do go for Alpine as the official docker image, we can
 >  expect quite a lot of breakage... Therefore, before going there, I
 >  think we should go for a "normal" distro after all for the time
 >  being. Of all the traditional distros, ubuntu:22.04 has by far the
 >  smallest image at 80MB (this actually came as quite the surprise to
 >  me... I have no idea why Fedora would be twice as large! But it does
 >  already include Python, so after installing all necessary packages it
 >  may even out again...).

I think Fedora is the outlier here, Debian is similar to Ubuntu:

docker image ls debian:12.2\*
REPOSITORY   TAG         IMAGE ID       CREATED       SIZE
debian       12.2-slim   31d5e503c34f   2 weeks ago   74.8MB
debian       12.2        0ce03c8a15ec   2 weeks ago   117MB

Given that we already use Debian for the "CI" container, it IMHO makes
sense to use it here as well.

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot