* [Qemu-devel] [PATCH v3 0/3] Docker cross-compile targets and user build support @ 2017-02-14 10:07 Alex Bennée 2017-02-14 10:07 ` [Qemu-devel] [PATCH v3 1/3] tests/docker: add basic user mapping support Alex Bennée ` (2 more replies) 0 siblings, 3 replies; 16+ messages in thread From: Alex Bennée @ 2017-02-14 10:07 UTC (permalink / raw) To: famz; +Cc: qemu-devel, Alex Bennée Hi Fam, Here is the latest iteration of the docker changes without the random guff from the last posting. I've re-arranged the docker images so now the cross-builds targets are explicitly built from the Debian base. This means a standard cross build can be triggered as normal with: make docker-test-build@debian-armhf-cross J=9 \ EXTRA_CONFIGURE_OPTS="--cross-prefix=arm-linux-gnueabihf-" \ TARGET_LIST="arm-softmmu,arm-linux-user" I've included the shippable.yml which exercises the cross build docker recipes and runs a limited build with them. I'm happy to include this in my next Travis related pull once the underlying Docker bits are merged. Regards, Alex. Alex Bennée (3): tests/docker: add basic user mapping support new: debian docker targets for cross-compiling .shippable.yml: new CI provider .shippable.yml | 23 ++++++++++++++++++++ MAINTAINERS | 1 + tests/docker/Makefile.include | 6 ++++++ tests/docker/docker.py | 15 +++++++++++-- tests/docker/dockerfiles/debian-arm64-cross.docker | 12 +++++++++++ tests/docker/dockerfiles/debian-armhf-cross.docker | 12 +++++++++++ tests/docker/dockerfiles/debian.docker | 25 ++++++++++++++++++++++ 7 files changed, 92 insertions(+), 2 deletions(-) create mode 100644 .shippable.yml create mode 100644 tests/docker/dockerfiles/debian-arm64-cross.docker create mode 100644 tests/docker/dockerfiles/debian-armhf-cross.docker create mode 100644 tests/docker/dockerfiles/debian.docker -- 2.11.0 ^ permalink raw reply [flat|nested] 16+ messages in thread
* [Qemu-devel] [PATCH v3 1/3] tests/docker: add basic user mapping support 2017-02-14 10:07 [Qemu-devel] [PATCH v3 0/3] Docker cross-compile targets and user build support Alex Bennée @ 2017-02-14 10:07 ` Alex Bennée 2017-02-14 14:22 ` Fam Zheng 2017-02-14 10:07 ` [Qemu-devel] [PATCH v3 2/3] new: debian docker targets for cross-compiling Alex Bennée 2017-02-14 10:07 ` [Qemu-devel] [PATCH v3 3/3] .shippable.yml: new CI provider Alex Bennée 2 siblings, 1 reply; 16+ messages in thread From: Alex Bennée @ 2017-02-14 10:07 UTC (permalink / raw) To: famz; +Cc: qemu-devel, Alex Bennée Currently all docker builds are done by exporting a tarball to the docker container and running the build as the containers root user. Other use cases are possible however and it is possible to map a part of users file-system to the container. This is useful for example for doing cross-builds of arbitrary source trees. For this to work smoothly the container needs to have a user created that maps cleanly to the host system. This adds a -u option to the docker script so that: DEB_ARCH=armhf DEB_TYPE=stable ./tests/docker/docker.py build \ -u --include-executable=arm-linux-user/qemu-arm \ debian:armhf ./tests/docker/dockerfiles/debian-bootstrap.docker Will build a container that can then be run like: docker run --rm -it -v /home/alex/lsrc/qemu/risu.git/:/src \ --user=alex:alex -w /src/ debian:armhf \ sh -c "make clean && ./configure -s && make" Alternatively specifying USER will add the appropriate stanza when building via the Makefile: make docker-image-debian-armhf-cross USER=1 Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Fam Zheng <famz@redhat.com> --- v2 - write the useradd directly - change long option to --add-current-user v3 - images -> image's - add r-b - add USER to Makefile --- tests/docker/Makefile.include | 2 ++ tests/docker/docker.py | 15 +++++++++++++-- 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include index 3f15d5aea8..8db55e43aa 100644 --- a/tests/docker/Makefile.include +++ b/tests/docker/Makefile.include @@ -50,6 +50,7 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker $(call quiet-command,\ $(SRC_PATH)/tests/docker/docker.py build qemu:$* $< \ $(if $V,,--quiet) $(if $(NOCACHE),--no-cache) \ + $(if $(USER),--add-current-user,) \ $(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)),\ "BUILD","$*") @@ -99,6 +100,7 @@ docker: @echo ' (default is 1)' @echo ' DEBUG=1 Stop and drop to shell in the created container' @echo ' before running the command.' + @echo ' USER=1 Add current user to docker image.' @echo ' NOCACHE=1 Ignore cache when build images.' @echo ' EXECUTABLE=<path> Include executable in image.' diff --git a/tests/docker/docker.py b/tests/docker/docker.py index 37d83199e7..3e6e7c6460 100755 --- a/tests/docker/docker.py +++ b/tests/docker/docker.py @@ -25,6 +25,7 @@ import signal from tarfile import TarFile, TarInfo from StringIO import StringIO from shutil import copy, rmtree +from pwd import getpwuid DEVNULL = open(os.devnull, 'wb') @@ -149,13 +150,20 @@ class Docker(object): labels = json.loads(resp)[0]["Config"].get("Labels", {}) return labels.get("com.qemu.dockerfile-checksum", "") - def build_image(self, tag, docker_dir, dockerfile, quiet=True, argv=None): + def build_image(self, tag, docker_dir, dockerfile, + quiet=True, user=False, argv=None): if argv == None: argv = [] tmp_df = tempfile.NamedTemporaryFile(dir=docker_dir, suffix=".docker") tmp_df.write(dockerfile) + if user: + uid = os.getuid() + uname = getpwuid(uid).pw_name + tmp_df.write("\n") + tmp_df.write("RUN useradd -u %d -U %s" % (uid, uname)) + tmp_df.write("\n") tmp_df.write("LABEL com.qemu.dockerfile-checksum=%s" % _text_checksum(dockerfile)) @@ -225,6 +233,9 @@ class BuildCommand(SubCommand): help="""Specify a binary that will be copied to the container together with all its dependent libraries""") + parser.add_argument("--add-current-user", "-u", dest="user", + action="store_true", + help="Add the current user to image's passwd") parser.add_argument("tag", help="Image Tag") parser.add_argument("dockerfile", @@ -261,7 +272,7 @@ class BuildCommand(SubCommand): docker_dir) dkr.build_image(tag, docker_dir, dockerfile, - quiet=args.quiet, argv=argv) + quiet=args.quiet, user=args.user, argv=argv) rmtree(docker_dir) -- 2.11.0 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] [PATCH v3 1/3] tests/docker: add basic user mapping support 2017-02-14 10:07 ` [Qemu-devel] [PATCH v3 1/3] tests/docker: add basic user mapping support Alex Bennée @ 2017-02-14 14:22 ` Fam Zheng 2017-02-14 15:11 ` Alex Bennée 0 siblings, 1 reply; 16+ messages in thread From: Fam Zheng @ 2017-02-14 14:22 UTC (permalink / raw) To: Alex Bennée; +Cc: qemu-devel On Tue, 02/14 10:07, Alex Bennée wrote: > Currently all docker builds are done by exporting a tarball to the > docker container and running the build as the containers root user. > Other use cases are possible however and it is possible to map a part > of users file-system to the container. This is useful for example for > doing cross-builds of arbitrary source trees. For this to work > smoothly the container needs to have a user created that maps cleanly > to the host system. > > This adds a -u option to the docker script so that: > > DEB_ARCH=armhf DEB_TYPE=stable ./tests/docker/docker.py build \ > -u --include-executable=arm-linux-user/qemu-arm \ > debian:armhf ./tests/docker/dockerfiles/debian-bootstrap.docker > > Will build a container that can then be run like: > > docker run --rm -it -v /home/alex/lsrc/qemu/risu.git/:/src \ > --user=alex:alex -w /src/ debian:armhf \ > sh -c "make clean && ./configure -s && make" > > Alternatively specifying USER will add the appropriate stanza when > building via the Makefile: > > make docker-image-debian-armhf-cross USER=1 > > Signed-off-by: Alex Bennée <alex.bennee@linaro.org> > Reviewed-by: Fam Zheng <famz@redhat.com> > > --- > v2 > - write the useradd directly > - change long option to --add-current-user > v3 > - images -> image's > - add r-b > - add USER to Makefile > --- > tests/docker/Makefile.include | 2 ++ > tests/docker/docker.py | 15 +++++++++++++-- > 2 files changed, 15 insertions(+), 2 deletions(-) > > diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include > index 3f15d5aea8..8db55e43aa 100644 > --- a/tests/docker/Makefile.include > +++ b/tests/docker/Makefile.include > @@ -50,6 +50,7 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker > $(call quiet-command,\ > $(SRC_PATH)/tests/docker/docker.py build qemu:$* $< \ > $(if $V,,--quiet) $(if $(NOCACHE),--no-cache) \ > + $(if $(USER),--add-current-user,) \ > $(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)),\ > "BUILD","$*") > > @@ -99,6 +100,7 @@ docker: > @echo ' (default is 1)' > @echo ' DEBUG=1 Stop and drop to shell in the created container' > @echo ' before running the command.' > + @echo ' USER=1 Add current user to docker image.' This is a bad choice of name, $USER is "$(id -un)", maybe ADD_CUR_USER? Fam > @echo ' NOCACHE=1 Ignore cache when build images.' > @echo ' EXECUTABLE=<path> Include executable in image.' > ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] [PATCH v3 1/3] tests/docker: add basic user mapping support 2017-02-14 14:22 ` Fam Zheng @ 2017-02-14 15:11 ` Alex Bennée 2017-02-14 15:20 ` Fam Zheng 0 siblings, 1 reply; 16+ messages in thread From: Alex Bennée @ 2017-02-14 15:11 UTC (permalink / raw) To: Fam Zheng; +Cc: qemu-devel Fam Zheng <famz@redhat.com> writes: > On Tue, 02/14 10:07, Alex Bennée wrote: >> Currently all docker builds are done by exporting a tarball to the >> docker container and running the build as the containers root user. >> Other use cases are possible however and it is possible to map a part >> of users file-system to the container. This is useful for example for >> doing cross-builds of arbitrary source trees. For this to work >> smoothly the container needs to have a user created that maps cleanly >> to the host system. >> >> This adds a -u option to the docker script so that: >> >> DEB_ARCH=armhf DEB_TYPE=stable ./tests/docker/docker.py build \ >> -u --include-executable=arm-linux-user/qemu-arm \ >> debian:armhf ./tests/docker/dockerfiles/debian-bootstrap.docker >> >> Will build a container that can then be run like: >> >> docker run --rm -it -v /home/alex/lsrc/qemu/risu.git/:/src \ >> --user=alex:alex -w /src/ debian:armhf \ >> sh -c "make clean && ./configure -s && make" >> >> Alternatively specifying USER will add the appropriate stanza when >> building via the Makefile: >> >> make docker-image-debian-armhf-cross USER=1 >> >> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> >> Reviewed-by: Fam Zheng <famz@redhat.com> >> >> --- >> v2 >> - write the useradd directly >> - change long option to --add-current-user >> v3 >> - images -> image's >> - add r-b >> - add USER to Makefile >> --- >> tests/docker/Makefile.include | 2 ++ >> tests/docker/docker.py | 15 +++++++++++++-- >> 2 files changed, 15 insertions(+), 2 deletions(-) >> >> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include >> index 3f15d5aea8..8db55e43aa 100644 >> --- a/tests/docker/Makefile.include >> +++ b/tests/docker/Makefile.include >> @@ -50,6 +50,7 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker >> $(call quiet-command,\ >> $(SRC_PATH)/tests/docker/docker.py build qemu:$* $< \ >> $(if $V,,--quiet) $(if $(NOCACHE),--no-cache) \ >> + $(if $(USER),--add-current-user,) \ >> $(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)),\ >> "BUILD","$*") >> >> @@ -99,6 +100,7 @@ docker: >> @echo ' (default is 1)' >> @echo ' DEBUG=1 Stop and drop to shell in the created container' >> @echo ' before running the command.' >> + @echo ' USER=1 Add current user to docker image.' > > This is a bad choice of name, $USER is "$(id -un)", maybe > ADD_CUR_USER? Or we could just enable it by default? Is there a reason not to enable the current user for these images? > > Fam > > >> @echo ' NOCACHE=1 Ignore cache when build images.' >> @echo ' EXECUTABLE=<path> Include executable in image.' >> -- Alex Bennée ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] [PATCH v3 1/3] tests/docker: add basic user mapping support 2017-02-14 15:11 ` Alex Bennée @ 2017-02-14 15:20 ` Fam Zheng 0 siblings, 0 replies; 16+ messages in thread From: Fam Zheng @ 2017-02-14 15:20 UTC (permalink / raw) To: Alex Bennée; +Cc: qemu-devel On Tue, 02/14 15:11, Alex Bennée wrote: > > Fam Zheng <famz@redhat.com> writes: > > > On Tue, 02/14 10:07, Alex Bennée wrote: > >> Currently all docker builds are done by exporting a tarball to the > >> docker container and running the build as the containers root user. > >> Other use cases are possible however and it is possible to map a part > >> of users file-system to the container. This is useful for example for > >> doing cross-builds of arbitrary source trees. For this to work > >> smoothly the container needs to have a user created that maps cleanly > >> to the host system. > >> > >> This adds a -u option to the docker script so that: > >> > >> DEB_ARCH=armhf DEB_TYPE=stable ./tests/docker/docker.py build \ > >> -u --include-executable=arm-linux-user/qemu-arm \ > >> debian:armhf ./tests/docker/dockerfiles/debian-bootstrap.docker > >> > >> Will build a container that can then be run like: > >> > >> docker run --rm -it -v /home/alex/lsrc/qemu/risu.git/:/src \ > >> --user=alex:alex -w /src/ debian:armhf \ > >> sh -c "make clean && ./configure -s && make" > >> > >> Alternatively specifying USER will add the appropriate stanza when > >> building via the Makefile: > >> > >> make docker-image-debian-armhf-cross USER=1 > >> > >> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> > >> Reviewed-by: Fam Zheng <famz@redhat.com> > >> > >> --- > >> v2 > >> - write the useradd directly > >> - change long option to --add-current-user > >> v3 > >> - images -> image's > >> - add r-b > >> - add USER to Makefile > >> --- > >> tests/docker/Makefile.include | 2 ++ > >> tests/docker/docker.py | 15 +++++++++++++-- > >> 2 files changed, 15 insertions(+), 2 deletions(-) > >> > >> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include > >> index 3f15d5aea8..8db55e43aa 100644 > >> --- a/tests/docker/Makefile.include > >> +++ b/tests/docker/Makefile.include > >> @@ -50,6 +50,7 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker > >> $(call quiet-command,\ > >> $(SRC_PATH)/tests/docker/docker.py build qemu:$* $< \ > >> $(if $V,,--quiet) $(if $(NOCACHE),--no-cache) \ > >> + $(if $(USER),--add-current-user,) \ > >> $(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)),\ > >> "BUILD","$*") > >> > >> @@ -99,6 +100,7 @@ docker: > >> @echo ' (default is 1)' > >> @echo ' DEBUG=1 Stop and drop to shell in the created container' > >> @echo ' before running the command.' > >> + @echo ' USER=1 Add current user to docker image.' > > > > This is a bad choice of name, $USER is "$(id -un)", maybe > > ADD_CUR_USER? > > Or we could just enable it by default? Is there a reason not to enable > the current user for these images? I think that's fine, but then docker.py needs to tolerate user already existing in the image, such as when you build qemu:debian-cross-$foo on top of qemu:debian. Fam ^ permalink raw reply [flat|nested] 16+ messages in thread
* [Qemu-devel] [PATCH v3 2/3] new: debian docker targets for cross-compiling 2017-02-14 10:07 [Qemu-devel] [PATCH v3 0/3] Docker cross-compile targets and user build support Alex Bennée 2017-02-14 10:07 ` [Qemu-devel] [PATCH v3 1/3] tests/docker: add basic user mapping support Alex Bennée @ 2017-02-14 10:07 ` Alex Bennée 2017-02-14 14:00 ` Fam Zheng 2017-02-14 14:30 ` Peter Maydell 2017-02-14 10:07 ` [Qemu-devel] [PATCH v3 3/3] .shippable.yml: new CI provider Alex Bennée 2 siblings, 2 replies; 16+ messages in thread From: Alex Bennée @ 2017-02-14 10:07 UTC (permalink / raw) To: famz; +Cc: qemu-devel, Alex Bennée This provides a basic Debian install with access to the emdebian cross compilers. The debian-armhf-cross and debian-arm64-cross targets build on the basic Debian image to allow cross compiling to those targets. To build against the cross compiler a cross-compile prefix needs to be set in the configure options, for example: make docker-test-build@debian-arm64-cross \ EXTRA_CONFIGURE_OPTS="--cross-prefix=aarch64-linux-gnu-" \ TARGET_LIST="aarch64-softmmu,aarch64-linux-user" Signed-off-by: Alex Bennée <alex.bennee@linaro.org> --- v2 - add clang (keep shippable happy) - rm adduser code (done direct now) - add aptitude (useful for debugging package clashes) v3 - split into debian, debian-armhf-cross and debian-aarch64-cross --- tests/docker/Makefile.include | 4 ++++ tests/docker/dockerfiles/debian-arm64-cross.docker | 12 +++++++++++ tests/docker/dockerfiles/debian-armhf-cross.docker | 12 +++++++++++ tests/docker/dockerfiles/debian.docker | 25 ++++++++++++++++++++++ 4 files changed, 53 insertions(+) create mode 100644 tests/docker/dockerfiles/debian-arm64-cross.docker create mode 100644 tests/docker/dockerfiles/debian-armhf-cross.docker create mode 100644 tests/docker/dockerfiles/debian.docker diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include index 8db55e43aa..dbd12d9042 100644 --- a/tests/docker/Makefile.include +++ b/tests/docker/Makefile.include @@ -54,6 +54,10 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker $(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)),\ "BUILD","$*") +# Enforce dependancies for composite images +docker-image-debian-armhf-cross: docker-image-debian +docker-image-debian-arm64-cross: docker-image-debian + # Expand all the pre-requistes for each docker image and test combination $(foreach i,$(DOCKER_IMAGES), \ $(foreach t,$(DOCKER_TESTS) $(DOCKER_TOOLS), \ diff --git a/tests/docker/dockerfiles/debian-arm64-cross.docker b/tests/docker/dockerfiles/debian-arm64-cross.docker new file mode 100644 index 0000000000..04212629f7 --- /dev/null +++ b/tests/docker/dockerfiles/debian-arm64-cross.docker @@ -0,0 +1,12 @@ +# +# Docker arm64 cross-compiler target +# +# This docker target builds on the base debian image. +# +FROM qemu:debian + +# Add the foreign architecture we want and install dependacies +RUN dpkg --add-architecture arm64 +RUN apt update +RUN apt install -yy crossbuild-essential-arm64 +RUN apt-get build-dep -yy -a arm64 qemu diff --git a/tests/docker/dockerfiles/debian-armhf-cross.docker b/tests/docker/dockerfiles/debian-armhf-cross.docker new file mode 100644 index 0000000000..1377641e3e --- /dev/null +++ b/tests/docker/dockerfiles/debian-armhf-cross.docker @@ -0,0 +1,12 @@ +# +# Docker armhf cross-compiler target +# +# This docker target builds on the base debian image. +# +FROM qemu:debian + +# Add the foreign architecture we want and install dependacies +RUN dpkg --add-architecture armhf +RUN apt update +RUN apt install -yy crossbuild-essential-armhf +RUN apt-get build-dep -yy -a armhf qemu diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker new file mode 100644 index 0000000000..52bd79938e --- /dev/null +++ b/tests/docker/dockerfiles/debian.docker @@ -0,0 +1,25 @@ +# +# Docker multiarch cross-compiler target +# +# This docker target is builds on Debian and Emdebian's cross compiler targets +# to build distro with a selection of cross compilers for building test binaries. +# +# On its own you can't build much but the docker-foo-cross targets +# build on top of the base debian image. +# +FROM debian:stable-slim + +# Setup some basic tools we need +RUN apt update +RUN apt install -yy curl aptitude + +# Setup Emdebian +RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list +RUN curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add - + +# Duplicate deb line as deb-src +RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list + +# Install common build utilities +RUN apt update +RUN apt install -yy build-essential clang -- 2.11.0 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] [PATCH v3 2/3] new: debian docker targets for cross-compiling 2017-02-14 10:07 ` [Qemu-devel] [PATCH v3 2/3] new: debian docker targets for cross-compiling Alex Bennée @ 2017-02-14 14:00 ` Fam Zheng 2017-02-14 14:27 ` Fam Zheng 2017-02-14 14:30 ` Peter Maydell 1 sibling, 1 reply; 16+ messages in thread From: Fam Zheng @ 2017-02-14 14:00 UTC (permalink / raw) To: Alex Bennée; +Cc: qemu-devel On Tue, 02/14 10:07, Alex Bennée wrote: > This provides a basic Debian install with access to the emdebian cross > compilers. The debian-armhf-cross and debian-arm64-cross targets build > on the basic Debian image to allow cross compiling to those targets. > > To build against the cross compiler a cross-compile prefix needs to be > set in the configure options, for example: > > make docker-test-build@debian-arm64-cross \ > EXTRA_CONFIGURE_OPTS="--cross-prefix=aarch64-linux-gnu-" \ > TARGET_LIST="aarch64-softmmu,aarch64-linux-user" If cross-compile is the only meaningful way to use this image, we probably should bake the configure option into the dockerfile and pick it up automatically in common.rc. What do you think? Fam ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] [PATCH v3 2/3] new: debian docker targets for cross-compiling 2017-02-14 14:00 ` Fam Zheng @ 2017-02-14 14:27 ` Fam Zheng 2017-02-14 15:10 ` Alex Bennée 0 siblings, 1 reply; 16+ messages in thread From: Fam Zheng @ 2017-02-14 14:27 UTC (permalink / raw) To: Alex Bennée; +Cc: qemu-devel On Tue, 02/14 22:00, Fam Zheng wrote: > On Tue, 02/14 10:07, Alex Bennée wrote: > > This provides a basic Debian install with access to the emdebian cross > > compilers. The debian-armhf-cross and debian-arm64-cross targets build > > on the basic Debian image to allow cross compiling to those targets. > > > > To build against the cross compiler a cross-compile prefix needs to be > > set in the configure options, for example: > > > > make docker-test-build@debian-arm64-cross \ > > EXTRA_CONFIGURE_OPTS="--cross-prefix=aarch64-linux-gnu-" \ > > TARGET_LIST="aarch64-softmmu,aarch64-linux-user" > > If cross-compile is the only meaningful way to use this image, we probably > should bake the configure option into the dockerfile and pick it up > automatically in common.rc. What do you think? The example change is like this: diff --git a/tests/docker/common.rc b/tests/docker/common.rc index 21657e87c6..6865689bb5 100755 --- a/tests/docker/common.rc +++ b/tests/docker/common.rc @@ -29,7 +29,7 @@ build_qemu() config_opts="--enable-werror \ ${TARGET_LIST:+--target-list=${TARGET_LIST}} \ --prefix=$PWD/install \ - $EXTRA_CONFIGURE_OPTS \ + $QEMU_CONFIGURE_OPTS $EXTRA_CONFIGURE_OPTS \ $@" echo "Configure options:" echo $config_opts diff --git a/tests/docker/dockerfiles/debian-arm64-cross.docker b/tests/docker/dockerfiles/debian-arm64-cross.docker index 04212629f7..fc1c135e74 100644 --- a/tests/docker/dockerfiles/debian-arm64-cross.docker +++ b/tests/docker/dockerfiles/debian-arm64-cross.docker @@ -10,3 +10,4 @@ RUN dpkg --add-architecture arm64 RUN apt update RUN apt install -yy crossbuild-essential-arm64 RUN apt-get build-dep -yy -a arm64 qemu +ENV QEMU_CONFIGURE_OPTS --cross-prefix=aarch64-linux-gnu- ^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] [PATCH v3 2/3] new: debian docker targets for cross-compiling 2017-02-14 14:27 ` Fam Zheng @ 2017-02-14 15:10 ` Alex Bennée 0 siblings, 0 replies; 16+ messages in thread From: Alex Bennée @ 2017-02-14 15:10 UTC (permalink / raw) To: Fam Zheng; +Cc: qemu-devel Fam Zheng <famz@redhat.com> writes: > On Tue, 02/14 22:00, Fam Zheng wrote: >> On Tue, 02/14 10:07, Alex Bennée wrote: >> > This provides a basic Debian install with access to the emdebian cross >> > compilers. The debian-armhf-cross and debian-arm64-cross targets build >> > on the basic Debian image to allow cross compiling to those targets. >> > >> > To build against the cross compiler a cross-compile prefix needs to be >> > set in the configure options, for example: >> > >> > make docker-test-build@debian-arm64-cross \ >> > EXTRA_CONFIGURE_OPTS="--cross-prefix=aarch64-linux-gnu-" \ >> > TARGET_LIST="aarch64-softmmu,aarch64-linux-user" >> >> If cross-compile is the only meaningful way to use this image, we probably >> should bake the configure option into the dockerfile and pick it up >> automatically in common.rc. What do you think? > > The example change is like this: > > diff --git a/tests/docker/common.rc b/tests/docker/common.rc > index 21657e87c6..6865689bb5 100755 > --- a/tests/docker/common.rc > +++ b/tests/docker/common.rc > @@ -29,7 +29,7 @@ build_qemu() > config_opts="--enable-werror \ > ${TARGET_LIST:+--target-list=${TARGET_LIST}} \ > --prefix=$PWD/install \ > - $EXTRA_CONFIGURE_OPTS \ > + $QEMU_CONFIGURE_OPTS $EXTRA_CONFIGURE_OPTS \ > $@" > echo "Configure options:" > echo $config_opts > diff --git a/tests/docker/dockerfiles/debian-arm64-cross.docker b/tests/docker/dockerfiles/debian-arm64-cross.docker > index 04212629f7..fc1c135e74 100644 > --- a/tests/docker/dockerfiles/debian-arm64-cross.docker > +++ b/tests/docker/dockerfiles/debian-arm64-cross.docker > @@ -10,3 +10,4 @@ RUN dpkg --add-architecture arm64 > RUN apt update > RUN apt install -yy crossbuild-essential-arm64 > RUN apt-get build-dep -yy -a arm64 qemu > +ENV QEMU_CONFIGURE_OPTS --cross-prefix=aarch64-linux-gnu- Yeah that seems like a good idea. Then a make docker-test-build@debian-armhf-cross will work as expected ;-) -- Alex Bennée ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] [PATCH v3 2/3] new: debian docker targets for cross-compiling 2017-02-14 10:07 ` [Qemu-devel] [PATCH v3 2/3] new: debian docker targets for cross-compiling Alex Bennée 2017-02-14 14:00 ` Fam Zheng @ 2017-02-14 14:30 ` Peter Maydell 2017-02-14 15:09 ` Alex Bennée 1 sibling, 1 reply; 16+ messages in thread From: Peter Maydell @ 2017-02-14 14:30 UTC (permalink / raw) To: Alex Bennée; +Cc: Fam Zheng, QEMU Developers On 14 February 2017 at 10:07, Alex Bennée <alex.bennee@linaro.org> wrote: > This provides a basic Debian install with access to the emdebian cross > compilers. The debian-armhf-cross and debian-arm64-cross targets build > on the basic Debian image to allow cross compiling to those targets. Is there a particular reason for creating different docker images for each cross target rather than just having one image with all the cross compilers in it? thanks -- PMM ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] [PATCH v3 2/3] new: debian docker targets for cross-compiling 2017-02-14 14:30 ` Peter Maydell @ 2017-02-14 15:09 ` Alex Bennée 2017-02-14 15:18 ` Fam Zheng 0 siblings, 1 reply; 16+ messages in thread From: Alex Bennée @ 2017-02-14 15:09 UTC (permalink / raw) To: Peter Maydell; +Cc: Fam Zheng, QEMU Developers Peter Maydell <peter.maydell@linaro.org> writes: > On 14 February 2017 at 10:07, Alex Bennée <alex.bennee@linaro.org> wrote: >> This provides a basic Debian install with access to the emdebian cross >> compilers. The debian-armhf-cross and debian-arm64-cross targets build >> on the basic Debian image to allow cross compiling to those targets. > > Is there a particular reason for creating different docker images > for each cross target rather than just having one image with > all the cross compilers in it? Mainly the clashing of build-dependencies. Debian's multi-arch is pretty good but you still can't generally have two arches worth of complex dependencies installed at once. I did originally have one base docker image with all the compilers and intended to do the dependencies step as part of test-build. However when running as builders the docker images don't actually have network ability so couldn't download them. -- Alex Bennée ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] [PATCH v3 2/3] new: debian docker targets for cross-compiling 2017-02-14 15:09 ` Alex Bennée @ 2017-02-14 15:18 ` Fam Zheng 0 siblings, 0 replies; 16+ messages in thread From: Fam Zheng @ 2017-02-14 15:18 UTC (permalink / raw) To: Alex Bennée; +Cc: Peter Maydell, QEMU Developers On Tue, 02/14 15:09, Alex Bennée wrote: > > Peter Maydell <peter.maydell@linaro.org> writes: > > > On 14 February 2017 at 10:07, Alex Bennée <alex.bennee@linaro.org> wrote: > >> This provides a basic Debian install with access to the emdebian cross > >> compilers. The debian-armhf-cross and debian-arm64-cross targets build > >> on the basic Debian image to allow cross compiling to those targets. > > > > Is there a particular reason for creating different docker images > > for each cross target rather than just having one image with > > all the cross compilers in it? > > Mainly the clashing of build-dependencies. Debian's multi-arch is pretty > good but you still can't generally have two arches worth of complex > dependencies installed at once. > > I did originally have one base docker image with all the compilers and > intended to do the dependencies step as part of test-build. However when > running as builders the docker images don't actually have network > ability so couldn't download them. Yes, having to "prepare", especially to download things, for each test invocation is not resource friendly for CI too. Fam ^ permalink raw reply [flat|nested] 16+ messages in thread
* [Qemu-devel] [PATCH v3 3/3] .shippable.yml: new CI provider 2017-02-14 10:07 [Qemu-devel] [PATCH v3 0/3] Docker cross-compile targets and user build support Alex Bennée 2017-02-14 10:07 ` [Qemu-devel] [PATCH v3 1/3] tests/docker: add basic user mapping support Alex Bennée 2017-02-14 10:07 ` [Qemu-devel] [PATCH v3 2/3] new: debian docker targets for cross-compiling Alex Bennée @ 2017-02-14 10:07 ` Alex Bennée 2017-02-14 14:11 ` Fam Zheng 2 siblings, 1 reply; 16+ messages in thread From: Alex Bennée @ 2017-02-14 10:07 UTC (permalink / raw) To: famz; +Cc: qemu-devel, Alex Bennée Ostensibly Shippable offers a similar set of services as Travis. However they are focused on Docker container based work-flows so we can use our existing containers to run a few extra builds - in this case a bunch of cross-compiled targets on a Debian multiarch system. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> --- v3 - reduce matrix to armhf/arm64 which currently work - use the make docker-image-* build stanzas - add TARGET_LIST to each build --- .shippable.yml | 23 +++++++++++++++++++++++ MAINTAINERS | 1 + 2 files changed, 24 insertions(+) create mode 100644 .shippable.yml diff --git a/.shippable.yml b/.shippable.yml new file mode 100644 index 0000000000..e4fa159481 --- /dev/null +++ b/.shippable.yml @@ -0,0 +1,23 @@ +language: c +env: + matrix: + - IMAGE=debian-armhf-cross + CROSS_PREFIX=arm-linux-gnueabihf- + TARGET_LIST=arm-softmmu,arm-linux-user + - IMAGE=debian-arm64-cross + CROSS_PREFIX=aarch64-linux-gnu- + TARGET_LIST=aarch64-softmmu,aarch64-linux-user + - IMAGE=centos6 + TARGET_LIST=i386-softmmu,x86_64-softmmu +build: + pre_ci: + - make docker-image-${IMAGE} + pre_ci_boot: + image_name: qemu + image_tag: ${IMAGE} + pull: false + options: "-e HOME=/root" + ci: + - unset CC + - ./configure --cross-prefix=${CROSS_PREFIX} --target-list=${TARGET_LIST} + - make -j2 diff --git a/MAINTAINERS b/MAINTAINERS index 7afbadaa15..578888d32d 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1805,6 +1805,7 @@ M: Alex Bennée <alex.bennee@linaro.org> L: qemu-devel@nongnu.org S: Supported F: .travis.yml +F: .shippable.yml Documentation ------------- -- 2.11.0 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] [PATCH v3 3/3] .shippable.yml: new CI provider 2017-02-14 10:07 ` [Qemu-devel] [PATCH v3 3/3] .shippable.yml: new CI provider Alex Bennée @ 2017-02-14 14:11 ` Fam Zheng 2017-02-14 15:56 ` Alex Bennée 0 siblings, 1 reply; 16+ messages in thread From: Fam Zheng @ 2017-02-14 14:11 UTC (permalink / raw) To: Alex Bennée; +Cc: qemu-devel On Tue, 02/14 10:07, Alex Bennée wrote: > Ostensibly Shippable offers a similar set of services as Travis. > However they are focused on Docker container based work-flows so we > can use our existing containers to run a few extra builds - in this > case a bunch of cross-compiled targets on a Debian multiarch system. > > Signed-off-by: Alex Bennée <alex.bennee@linaro.org> > > --- > v3 > - reduce matrix to armhf/arm64 which currently work > - use the make docker-image-* build stanzas > - add TARGET_LIST to each build > --- > .shippable.yml | 23 +++++++++++++++++++++++ > MAINTAINERS | 1 + > 2 files changed, 24 insertions(+) > create mode 100644 .shippable.yml > > diff --git a/.shippable.yml b/.shippable.yml > new file mode 100644 > index 0000000000..e4fa159481 > --- /dev/null > +++ b/.shippable.yml > @@ -0,0 +1,23 @@ > +language: c > +env: > + matrix: > + - IMAGE=debian-armhf-cross > + CROSS_PREFIX=arm-linux-gnueabihf- > + TARGET_LIST=arm-softmmu,arm-linux-user > + - IMAGE=debian-arm64-cross > + CROSS_PREFIX=aarch64-linux-gnu- > + TARGET_LIST=aarch64-softmmu,aarch64-linux-user > + - IMAGE=centos6 > + TARGET_LIST=i386-softmmu,x86_64-softmmu > +build: > + pre_ci: > + - make docker-image-${IMAGE} > + pre_ci_boot: > + image_name: qemu > + image_tag: ${IMAGE} > + pull: false > + options: "-e HOME=/root" > + ci: > + - unset CC > + - ./configure --cross-prefix=${CROSS_PREFIX} --target-list=${TARGET_LIST} > + - make -j2 Looks cool! > diff --git a/MAINTAINERS b/MAINTAINERS > index 7afbadaa15..578888d32d 100644 > --- a/MAINTAINERS > +++ b/MAINTAINERS > @@ -1805,6 +1805,7 @@ M: Alex Bennée <alex.bennee@linaro.org> > L: qemu-devel@nongnu.org > S: Supported > F: .travis.yml > +F: .shippable.yml > > Documentation > ------------- > -- > 2.11.0 > > May I propose we merge 'docker testing' section of MAINTAINERS into 'build and test automation' section? I don't know as much about travis (and shippable) but I'm totally fine if you want to have docker tests under your umbrella, and it seems a logical step judging from their names. Fam ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] [PATCH v3 3/3] .shippable.yml: new CI provider 2017-02-14 14:11 ` Fam Zheng @ 2017-02-14 15:56 ` Alex Bennée 2017-02-15 0:51 ` Fam Zheng 0 siblings, 1 reply; 16+ messages in thread From: Alex Bennée @ 2017-02-14 15:56 UTC (permalink / raw) To: Fam Zheng; +Cc: qemu-devel Fam Zheng <famz@redhat.com> writes: > On Tue, 02/14 10:07, Alex Bennée wrote: >> Ostensibly Shippable offers a similar set of services as Travis. >> However they are focused on Docker container based work-flows so we >> can use our existing containers to run a few extra builds - in this >> case a bunch of cross-compiled targets on a Debian multiarch system. >> >> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> >> >> --- >> v3 >> - reduce matrix to armhf/arm64 which currently work >> - use the make docker-image-* build stanzas >> - add TARGET_LIST to each build >> --- >> .shippable.yml | 23 +++++++++++++++++++++++ >> MAINTAINERS | 1 + >> 2 files changed, 24 insertions(+) >> create mode 100644 .shippable.yml >> >> diff --git a/.shippable.yml b/.shippable.yml >> new file mode 100644 >> index 0000000000..e4fa159481 >> --- /dev/null >> +++ b/.shippable.yml >> @@ -0,0 +1,23 @@ >> +language: c >> +env: >> + matrix: >> + - IMAGE=debian-armhf-cross >> + CROSS_PREFIX=arm-linux-gnueabihf- >> + TARGET_LIST=arm-softmmu,arm-linux-user >> + - IMAGE=debian-arm64-cross >> + CROSS_PREFIX=aarch64-linux-gnu- >> + TARGET_LIST=aarch64-softmmu,aarch64-linux-user >> + - IMAGE=centos6 >> + TARGET_LIST=i386-softmmu,x86_64-softmmu >> +build: >> + pre_ci: >> + - make docker-image-${IMAGE} >> + pre_ci_boot: >> + image_name: qemu >> + image_tag: ${IMAGE} >> + pull: false >> + options: "-e HOME=/root" >> + ci: >> + - unset CC >> + - ./configure --cross-prefix=${CROSS_PREFIX} --target-list=${TARGET_LIST} >> + - make -j2 > > Looks cool! > >> diff --git a/MAINTAINERS b/MAINTAINERS >> index 7afbadaa15..578888d32d 100644 >> --- a/MAINTAINERS >> +++ b/MAINTAINERS >> @@ -1805,6 +1805,7 @@ M: Alex Bennée <alex.bennee@linaro.org> >> L: qemu-devel@nongnu.org >> S: Supported >> F: .travis.yml >> +F: .shippable.yml >> >> Documentation >> ------------- >> -- >> 2.11.0 >> >> > > May I propose we merge 'docker testing' section of MAINTAINERS into 'build and > test automation' section? I don't know as much about travis (and shippable) but > I'm totally fine if you want to have docker tests under your umbrella, and > it seems a logical step judging from their names. So: Build and test automation ------------------------- M: Alex Bennée <alex.bennee@linaro.org> M: Fam Zheng <famz@redhat.com> L: qemu-devel@nongnu.org S: Maintained F: .travis.yml F: .shippable.yml F: tests/docker/ Or did you want to drop out of the review cycle and concentrate on patchew? -- Alex Bennée ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] [PATCH v3 3/3] .shippable.yml: new CI provider 2017-02-14 15:56 ` Alex Bennée @ 2017-02-15 0:51 ` Fam Zheng 0 siblings, 0 replies; 16+ messages in thread From: Fam Zheng @ 2017-02-15 0:51 UTC (permalink / raw) To: Alex Bennée; +Cc: qemu-devel On Tue, 02/14 15:56, Alex Bennée wrote: > > May I propose we merge 'docker testing' section of MAINTAINERS into 'build and > > test automation' section? I don't know as much about travis (and shippable) but > > I'm totally fine if you want to have docker tests under your umbrella, and > > it seems a logical step judging from their names. > > So: > > Build and test automation > ------------------------- > M: Alex Bennée <alex.bennee@linaro.org> > M: Fam Zheng <famz@redhat.com> > L: qemu-devel@nongnu.org > S: Maintained > F: .travis.yml > F: .shippable.yml > F: tests/docker/ Yes, looks good. Please include a patch in v4. > > Or did you want to drop out of the review cycle and concentrate on patchew? I would still keep my eyes on tests/docker/, the intent is just for being more compact on MAINTAINERS. Fam ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2017-02-15 0:51 UTC | newest] Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-02-14 10:07 [Qemu-devel] [PATCH v3 0/3] Docker cross-compile targets and user build support Alex Bennée 2017-02-14 10:07 ` [Qemu-devel] [PATCH v3 1/3] tests/docker: add basic user mapping support Alex Bennée 2017-02-14 14:22 ` Fam Zheng 2017-02-14 15:11 ` Alex Bennée 2017-02-14 15:20 ` Fam Zheng 2017-02-14 10:07 ` [Qemu-devel] [PATCH v3 2/3] new: debian docker targets for cross-compiling Alex Bennée 2017-02-14 14:00 ` Fam Zheng 2017-02-14 14:27 ` Fam Zheng 2017-02-14 15:10 ` Alex Bennée 2017-02-14 14:30 ` Peter Maydell 2017-02-14 15:09 ` Alex Bennée 2017-02-14 15:18 ` Fam Zheng 2017-02-14 10:07 ` [Qemu-devel] [PATCH v3 3/3] .shippable.yml: new CI provider Alex Bennée 2017-02-14 14:11 ` Fam Zheng 2017-02-14 15:56 ` Alex Bennée 2017-02-15 0:51 ` Fam Zheng
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.