* [PATCH v1 0/6] testing/next (docker binfmt tests)
@ 2021-01-19 17:52 Alex Bennée
2021-01-19 17:52 ` [PATCH v1 1/6] tests/docker: Fix _get_so_libs() for docker-binfmt-image Alex Bennée
` (5 more replies)
0 siblings, 6 replies; 8+ messages in thread
From: Alex Bennée @ 2021-01-19 17:52 UTC (permalink / raw)
To: qemu-devel; +Cc: Alex Bennée
Hi,
The testing updates keep on rolling. Promoted by some fixes from
Phillipe I decided to expand the testing of the binfmt code. This at
least ensures that any binary copied into the docker container has
enough bits to be functional.
The following need review:
- tests/docker: add a docker-exec-copy-test
- tests/docker: alias docker-help target for consistency
- tests/docker: preserve original name when copying libs
- tests/docker: make _copy_with_mkdir accept missing files
Alex Bennée (4):
tests/docker: make _copy_with_mkdir accept missing files
tests/docker: preserve original name when copying libs
tests/docker: alias docker-help target for consistency
tests/docker: add a docker-exec-copy-test
Philippe Mathieu-Daudé (2):
tests/docker: Fix _get_so_libs() for docker-binfmt-image
tests/docker: Fix typo in help message
tests/docker/Makefile.include | 26 +++++++++++++++++++++++---
tests/docker/docker.py | 23 +++++++++++++++++------
tests/docker/dockerfiles/empty.docker | 8 ++++++++
3 files changed, 48 insertions(+), 9 deletions(-)
create mode 100644 tests/docker/dockerfiles/empty.docker
--
2.20.1
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH v1 1/6] tests/docker: Fix _get_so_libs() for docker-binfmt-image
2021-01-19 17:52 [PATCH v1 0/6] testing/next (docker binfmt tests) Alex Bennée
@ 2021-01-19 17:52 ` Alex Bennée
2021-01-19 17:52 ` [PATCH v1 2/6] tests/docker: Fix typo in help message Alex Bennée
` (4 subsequent siblings)
5 siblings, 0 replies; 8+ messages in thread
From: Alex Bennée @ 2021-01-19 17:52 UTC (permalink / raw)
To: qemu-devel
Cc: Fam Zheng, Philippe Mathieu-Daudé,
Alex Bennée, Philippe Mathieu-Daudé
From: Philippe Mathieu-Daudé <f4bug@amsat.org>
Fix a variable rename mistake from commit 5e33f7fead5:
Traceback (most recent call last):
File "./tests/docker/docker.py", line 710, in <module>
sys.exit(main())
File "./tests/docker/docker.py", line 706, in main
return args.cmdobj.run(args, argv)
File "./tests/docker/docker.py", line 489, in run
_copy_binary_with_libs(args.include_executable,
File "./tests/docker/docker.py", line 149, in _copy_binary_with_libs
libs = _get_so_libs(src)
File "./tests/docker/docker.py", line 123, in _get_so_libs
libs.append(s.group(1))
NameError: name 's' is not defined
Fixes: 5e33f7fead5 ("tests/docker: better handle symlinked libs")
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210119050149.516910-1-f4bug@amsat.org>
---
tests/docker/docker.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 884dfeb29c..0b4f6167b3 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -120,7 +120,7 @@ def _get_so_libs(executable):
search = ldd_re.search(line)
if search:
try:
- libs.append(s.group(1))
+ libs.append(search.group(1))
except IndexError:
pass
except subprocess.CalledProcessError:
--
2.20.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH v1 2/6] tests/docker: Fix typo in help message
2021-01-19 17:52 [PATCH v1 0/6] testing/next (docker binfmt tests) Alex Bennée
2021-01-19 17:52 ` [PATCH v1 1/6] tests/docker: Fix _get_so_libs() for docker-binfmt-image Alex Bennée
@ 2021-01-19 17:52 ` Alex Bennée
2021-01-19 17:52 ` [PATCH v1 3/6] tests/docker: make _copy_with_mkdir accept missing files Alex Bennée
` (3 subsequent siblings)
5 siblings, 0 replies; 8+ messages in thread
From: Alex Bennée @ 2021-01-19 17:52 UTC (permalink / raw)
To: qemu-devel
Cc: Fam Zheng, Philippe Mathieu-Daudé,
Alex Bennée, Philippe Mathieu-Daudé
From: Philippe Mathieu-Daudé <f4bug@amsat.org>
To have the variable properly passed, we need to set it,
ie. NOUSER=1. Fix the message displayed by 'make docker'.
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210119052120.522069-1-f4bug@amsat.org>
---
tests/docker/Makefile.include | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 0779dab5b9..bdc53ddfcf 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -209,7 +209,7 @@ endif
@echo ' before running the command.'
@echo ' NETWORK=1 Enable virtual network interface with default backend.'
@echo ' NETWORK=$$BACKEND Enable virtual network interface with $$BACKEND.'
- @echo ' NOUSER Define to disable adding current user to containers passwd.'
+ @echo ' NOUSER=1 Define to disable adding current user to containers passwd.'
@echo ' NOCACHE=1 Ignore cache when build images.'
@echo ' EXECUTABLE=<path> Include executable in image.'
@echo ' EXTRA_FILES="<path> [... <path>]"'
--
2.20.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH v1 3/6] tests/docker: make _copy_with_mkdir accept missing files
2021-01-19 17:52 [PATCH v1 0/6] testing/next (docker binfmt tests) Alex Bennée
2021-01-19 17:52 ` [PATCH v1 1/6] tests/docker: Fix _get_so_libs() for docker-binfmt-image Alex Bennée
2021-01-19 17:52 ` [PATCH v1 2/6] tests/docker: Fix typo in help message Alex Bennée
@ 2021-01-19 17:52 ` Alex Bennée
2021-01-19 18:56 ` Philippe Mathieu-Daudé
2021-01-19 17:52 ` [PATCH v1 4/6] tests/docker: preserve original name when copying libs Alex Bennée
` (2 subsequent siblings)
5 siblings, 1 reply; 8+ messages in thread
From: Alex Bennée @ 2021-01-19 17:52 UTC (permalink / raw)
To: qemu-devel; +Cc: Fam Zheng, Philippe Mathieu-Daudé, Alex Bennée
Depending on the linker/ldd setup we might get a file with no path.
Typically this is the psuedo library linux-vdso.so which doesn't
actually exist on the disk. Rather than try and catch these distro
specific edge cases just shout about it and try and continue.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/docker/docker.py | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 0b4f6167b3..fb3de41c0b 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -103,7 +103,12 @@ def _copy_with_mkdir(src, root_dir, sub_path='.'):
pass
dest_file = "%s/%s" % (dest_dir, os.path.basename(src))
- copy(src, dest_file)
+
+ try:
+ copy(src, dest_file)
+ except FileNotFoundError:
+ print("Couldn't copy %s to %s" % (src, dest_file))
+ pass
def _get_so_libs(executable):
--
2.20.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH v1 4/6] tests/docker: preserve original name when copying libs
2021-01-19 17:52 [PATCH v1 0/6] testing/next (docker binfmt tests) Alex Bennée
` (2 preceding siblings ...)
2021-01-19 17:52 ` [PATCH v1 3/6] tests/docker: make _copy_with_mkdir accept missing files Alex Bennée
@ 2021-01-19 17:52 ` Alex Bennée
2021-01-19 17:52 ` [PATCH v1 5/6] tests/docker: alias docker-help target for consistency Alex Bennée
2021-01-19 17:52 ` [PATCH v1 6/6] tests/docker: add a docker-exec-copy-test Alex Bennée
5 siblings, 0 replies; 8+ messages in thread
From: Alex Bennée @ 2021-01-19 17:52 UTC (permalink / raw)
To: qemu-devel; +Cc: Fam Zheng, Philippe Mathieu-Daudé, Alex Bennée
While it is important we chase down the symlinks to copy the correct
data we can confuse the kernel by renaming the interpreter to what is
in the binary. Extend _copy_with_mkdir to preserve the original name
of the file when asked.
Fixes: 5e33f7fead ("tests/docker: better handle symlinked libs")
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/docker/docker.py | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index fb3de41c0b..39da3fefcf 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -93,7 +93,7 @@ def _guess_engine_command():
commands_txt)
-def _copy_with_mkdir(src, root_dir, sub_path='.'):
+def _copy_with_mkdir(src, root_dir, sub_path='.', name=None):
"""Copy src into root_dir, creating sub_path as needed."""
dest_dir = os.path.normpath("%s/%s" % (root_dir, sub_path))
try:
@@ -102,7 +102,7 @@ def _copy_with_mkdir(src, root_dir, sub_path='.'):
# we can safely ignore already created directories
pass
- dest_file = "%s/%s" % (dest_dir, os.path.basename(src))
+ dest_file = "%s/%s" % (dest_dir, name if name else os.path.basename(src))
try:
copy(src, dest_file)
@@ -155,8 +155,9 @@ def _copy_binary_with_libs(src, bin_dest, dest_dir):
if libs:
for l in libs:
so_path = os.path.dirname(l)
+ name = os.path.basename(l)
real_l = os.path.realpath(l)
- _copy_with_mkdir(real_l, dest_dir, so_path)
+ _copy_with_mkdir(real_l, dest_dir, so_path, name)
def _check_binfmt_misc(executable):
--
2.20.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH v1 5/6] tests/docker: alias docker-help target for consistency
2021-01-19 17:52 [PATCH v1 0/6] testing/next (docker binfmt tests) Alex Bennée
` (3 preceding siblings ...)
2021-01-19 17:52 ` [PATCH v1 4/6] tests/docker: preserve original name when copying libs Alex Bennée
@ 2021-01-19 17:52 ` Alex Bennée
2021-01-19 17:52 ` [PATCH v1 6/6] tests/docker: add a docker-exec-copy-test Alex Bennée
5 siblings, 0 replies; 8+ messages in thread
From: Alex Bennée @ 2021-01-19 17:52 UTC (permalink / raw)
To: qemu-devel; +Cc: Fam Zheng, Philippe Mathieu-Daudé, Alex Bennée
We have a bunch of -help targets so this will save some cognitive
dissonance. Keep the original for those with muscle memory.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/docker/Makefile.include | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index bdc53ddfcf..a5c1e4a615 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -1,6 +1,6 @@
# Makefile for Docker tests
-.PHONY: docker docker-test docker-clean docker-image docker-qemu-src
+.PHONY: docker docker-help docker-test docker-clean docker-image docker-qemu-src
NULL :=
SPACE := $(NULL) #
@@ -218,6 +218,8 @@ endif
@echo ' Specify which container engine to run.'
@echo ' REGISTRY=url Cache builds from registry (default:$(DOCKER_REGISTRY))'
+docker-help: docker
+
# This rule if for directly running against an arbitrary docker target.
# It is called by the expanded docker targets (e.g. make
# docker-test-foo@bar) which will do additional verification.
--
2.20.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH v1 6/6] tests/docker: add a docker-exec-copy-test
2021-01-19 17:52 [PATCH v1 0/6] testing/next (docker binfmt tests) Alex Bennée
` (4 preceding siblings ...)
2021-01-19 17:52 ` [PATCH v1 5/6] tests/docker: alias docker-help target for consistency Alex Bennée
@ 2021-01-19 17:52 ` Alex Bennée
5 siblings, 0 replies; 8+ messages in thread
From: Alex Bennée @ 2021-01-19 17:52 UTC (permalink / raw)
To: qemu-devel; +Cc: Fam Zheng, Philippe Mathieu-Daudé, Alex Bennée
This provides test machinery for checking the QEMU copying logic works
properly. It takes considerably less time to run than starting a
debootstrap only for it to fail later. I considered adding a remove
command to docker.py but figured that might be gold plating given the
relative size of the containers compared to the ones with actual stuff
in them.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/docker/Makefile.include | 20 +++++++++++++++++++-
tests/docker/docker.py | 7 ++++++-
tests/docker/dockerfiles/empty.docker | 8 ++++++++
3 files changed, 33 insertions(+), 2 deletions(-)
create mode 100644 tests/docker/dockerfiles/empty.docker
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index a5c1e4a615..93b29ad823 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -11,7 +11,7 @@ HOST_ARCH = $(if $(ARCH),$(ARCH),$(shell uname -m))
DOCKER_SUFFIX := .docker
DOCKER_FILES_DIR := $(SRC_PATH)/tests/docker/dockerfiles
# we don't run tests on intermediate images (used as base by another image)
-DOCKER_PARTIAL_IMAGES := debian10 debian11 debian-bootstrap
+DOCKER_PARTIAL_IMAGES := debian10 debian11 debian-bootstrap empty
DOCKER_IMAGES := $(sort $(notdir $(basename $(wildcard $(DOCKER_FILES_DIR)/*.docker))))
DOCKER_TARGETS := $(patsubst %,docker-image-%,$(DOCKER_IMAGES))
# Use a global constant ccache directory to speed up repetitive builds
@@ -92,6 +92,24 @@ docker-binfmt-image-debian-%: $(DOCKER_FILES_DIR)/debian-bootstrap.docker
{ echo "You will need to build $(EXECUTABLE)"; exit 1;},\
"CHECK", "debian-$* exists"))
+# These are test targets
+USER_TCG_TARGETS=$(patsubst %-linux-user,qemu-%,$(filter %-linux-user,$(TARGET_DIRS)))
+EXEC_COPY_TESTS=$(patsubst %,docker-exec-copy-test-%, $(USER_TCG_TARGETS))
+
+$(EXEC_COPY_TESTS): docker-exec-copy-test-%: $(DOCKER_FILES_DIR)/empty.docker
+ $(call quiet-command, \
+ $(DOCKER_SCRIPT) build -t qemu/exec-copy-test-$* -f $< \
+ $(if $V,,--quiet) --no-cache \
+ --include-executable=$* \
+ --skip-binfmt, \
+ "TEST","copy $* to container")
+ $(call quiet-command, \
+ $(DOCKER_SCRIPT) run qemu/exec-copy-test-$* \
+ /$* -version > tests/docker-exec-copy-test-$*.out, \
+ "TEST","check $* works in container")
+
+docker-exec-copy-test: $(EXEC_COPY_TESTS)
+
endif
# Enforce dependencies for composite images
diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 39da3fefcf..d28df4c140 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -438,6 +438,9 @@ class BuildCommand(SubCommand):
help="""Specify a binary that will be copied to the
container together with all its dependent
libraries""")
+ parser.add_argument("--skip-binfmt",
+ action="store_true",
+ help="""Skip binfmt entry check (used for testing)""")
parser.add_argument("--extra-files", nargs='*',
help="""Specify files that will be copied in the
Docker image, fulfilling the ADD directive from the
@@ -466,7 +469,9 @@ class BuildCommand(SubCommand):
docker_dir = tempfile.mkdtemp(prefix="docker_build")
# Validate binfmt_misc will work
- if args.include_executable:
+ if args.skip_binfmt:
+ qpath = args.include_executable
+ elif args.include_executable:
qpath, enabled = _check_binfmt_misc(args.include_executable)
if not enabled:
return 1
diff --git a/tests/docker/dockerfiles/empty.docker b/tests/docker/dockerfiles/empty.docker
new file mode 100644
index 0000000000..9ba980f1a8
--- /dev/null
+++ b/tests/docker/dockerfiles/empty.docker
@@ -0,0 +1,8 @@
+#
+# Empty Dockerfile
+#
+
+FROM scratch
+
+# Add everything from the context into the container
+ADD . /
--
2.20.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH v1 3/6] tests/docker: make _copy_with_mkdir accept missing files
2021-01-19 17:52 ` [PATCH v1 3/6] tests/docker: make _copy_with_mkdir accept missing files Alex Bennée
@ 2021-01-19 18:56 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 8+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-01-19 18:56 UTC (permalink / raw)
To: Alex Bennée, qemu-devel; +Cc: Fam Zheng
On 1/19/21 6:52 PM, Alex Bennée wrote:
> Depending on the linker/ldd setup we might get a file with no path.
> Typically this is the psuedo library linux-vdso.so which doesn't
> actually exist on the disk. Rather than try and catch these distro
> specific edge cases just shout about it and try and continue.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
> tests/docker/docker.py | 7 ++++++-
> 1 file changed, 6 insertions(+), 1 deletion(-)
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2021-01-19 20:09 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-01-19 17:52 [PATCH v1 0/6] testing/next (docker binfmt tests) Alex Bennée
2021-01-19 17:52 ` [PATCH v1 1/6] tests/docker: Fix _get_so_libs() for docker-binfmt-image Alex Bennée
2021-01-19 17:52 ` [PATCH v1 2/6] tests/docker: Fix typo in help message Alex Bennée
2021-01-19 17:52 ` [PATCH v1 3/6] tests/docker: make _copy_with_mkdir accept missing files Alex Bennée
2021-01-19 18:56 ` Philippe Mathieu-Daudé
2021-01-19 17:52 ` [PATCH v1 4/6] tests/docker: preserve original name when copying libs Alex Bennée
2021-01-19 17:52 ` [PATCH v1 5/6] tests/docker: alias docker-help target for consistency Alex Bennée
2021-01-19 17:52 ` [PATCH v1 6/6] tests/docker: add a docker-exec-copy-test Alex Bennée
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).