[Qemu-devel] [PATCH] tests: make docker-test-debug@fedora run sanitizers

[Qemu-devel] [PATCH] tests: make docker-test-debug@fedora run sanitizers

[Marc-André Lureau]

Since --enable-debug no longer enable sanitizers, we need explicit
--enable-sanitizers.

llvm package is required for llvm-symbolizer, to get symbols in
backtraces.

Add make V=1 to get details about failing tests.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 tests/docker/dockerfiles/fedora.docker | 2 +-
 tests/docker/test-debug                | 6 +++---
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker
index 994a35a332..a00004319e 100644
--- a/tests/docker/dockerfiles/fedora.docker
+++ b/tests/docker/dockerfiles/fedora.docker
@@ -2,7 +2,7 @@ FROM fedora:27
 ENV PACKAGES \
     ccache gettext git tar PyYAML sparse flex bison python3 bzip2 hostname \
     glib2-devel pixman-devel zlib-devel SDL-devel libfdt-devel \
-    gcc gcc-c++ clang make perl which bc findutils libaio-devel \
+    gcc gcc-c++ llvm clang make perl which bc findutils libaio-devel \
     nettle-devel libasan libubsan \
     mingw32-pixman mingw32-glib2 mingw32-gmp mingw32-SDL mingw32-pkg-config \
     mingw32-gtk2 mingw32-gtk3 mingw32-gnutls mingw32-nettle mingw32-libtasn1 \
diff --git a/tests/docker/test-debug b/tests/docker/test-debug
index d020b06917..d3f9f70d01 100755
--- a/tests/docker/test-debug
+++ b/tests/docker/test-debug
@@ -1,6 +1,6 @@
 #!/bin/bash -e
 #
-# Compile and check with clang & --enable-debug.
+# Compile and check with clang & --enable-debug --enable-sanitizers.
 #
 # Copyright (c) 2016-2018 Red Hat Inc.
 #
@@ -19,8 +19,8 @@ requires clang asan
 cd "$BUILD_DIR"
 
 OPTS="--cxx=clang++ --cc=clang --host-cc=clang"
-OPTS="--enable-debug $OPTS"
+OPTS="--enable-debug --enable-sanitizers $OPTS"
 
 build_qemu $OPTS
-make $MAKEFLAGS check
+make $MAKEFLAGS V=1 check
 install_qemu
-- 
2.16.2.346.g9779355e34

Re: [Qemu-devel] [PATCH] tests: make docker-test-debug@fedora run sanitizers

[Marc-André Lureau]

Hi Fam,

It would be great if patchew could run the debug/sanitizer build.
Unfortunately, except x86 target, the run is full of ASAN leaks or
warnings.

Do you think patchew could learn to do regression testing? By that I
mean that patchew could compare a patchset output with the current
master (that output should be cached for multiple tests). The output
to compare here would be the errors reported by ASAN during the build.
(eventually, this could be used for other metrics some day). If so,
would you be willing to help me implementing it?


On Mon, Mar 12, 2018 at 1:08 PM, Marc-André Lureau
<marcandre.lureau@redhat.com> wrote:
> Since --enable-debug no longer enable sanitizers, we need explicit
> --enable-sanitizers.
>
> llvm package is required for llvm-symbolizer, to get symbols in
> backtraces.
>
> Add make V=1 to get details about failing tests.
>
> Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> ---
>  tests/docker/dockerfiles/fedora.docker | 2 +-
>  tests/docker/test-debug                | 6 +++---
>  2 files changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker
> index 994a35a332..a00004319e 100644
> --- a/tests/docker/dockerfiles/fedora.docker
> +++ b/tests/docker/dockerfiles/fedora.docker
> @@ -2,7 +2,7 @@ FROM fedora:27
>  ENV PACKAGES \
>      ccache gettext git tar PyYAML sparse flex bison python3 bzip2 hostname \
>      glib2-devel pixman-devel zlib-devel SDL-devel libfdt-devel \
> -    gcc gcc-c++ clang make perl which bc findutils libaio-devel \
> +    gcc gcc-c++ llvm clang make perl which bc findutils libaio-devel \
>      nettle-devel libasan libubsan \
>      mingw32-pixman mingw32-glib2 mingw32-gmp mingw32-SDL mingw32-pkg-config \
>      mingw32-gtk2 mingw32-gtk3 mingw32-gnutls mingw32-nettle mingw32-libtasn1 \
> diff --git a/tests/docker/test-debug b/tests/docker/test-debug
> index d020b06917..d3f9f70d01 100755
> --- a/tests/docker/test-debug
> +++ b/tests/docker/test-debug
> @@ -1,6 +1,6 @@
>  #!/bin/bash -e
>  #
> -# Compile and check with clang & --enable-debug.
> +# Compile and check with clang & --enable-debug --enable-sanitizers.
>  #
>  # Copyright (c) 2016-2018 Red Hat Inc.
>  #
> @@ -19,8 +19,8 @@ requires clang asan
>  cd "$BUILD_DIR"
>
>  OPTS="--cxx=clang++ --cc=clang --host-cc=clang"
> -OPTS="--enable-debug $OPTS"
> +OPTS="--enable-debug --enable-sanitizers $OPTS"
>
>  build_qemu $OPTS
> -make $MAKEFLAGS check
> +make $MAKEFLAGS V=1 check
>  install_qemu
> --
> 2.16.2.346.g9779355e34
>
>



-- 
Marc-André Lureau

Re: [Qemu-devel] [PATCH] tests: make docker-test-debug@fedora run sanitizers

[Fam Zheng]

On Mon, 03/12 15:11, Marc-André Lureau wrote:
> Hi Fam,
> 
> It would be great if patchew could run the debug/sanitizer build.
> Unfortunately, except x86 target, the run is full of ASAN leaks or
> warnings.
> 
> Do you think patchew could learn to do regression testing? By that I
> mean that patchew could compare a patchset output with the current
> master (that output should be cached for multiple tests). The output
> to compare here would be the errors reported by ASAN during the build.
> (eventually, this could be used for other metrics some day). If so,
> would you be willing to help me implementing it?

The question is can we simply rely on the exit code of cmp or diff for that kind
of comparison? If there is any out-of-order texts, that simple diff won't work.

Anything more intelligent/sensible should go as a policy, and patchew should
continue focus on mechanisms.

I'll send a pull for the patch. Thanks!

Thanks.

Re: [Qemu-devel] [PATCH] tests: make docker-test-debug@fedora run sanitizers

[Marc-André Lureau]

Hi

On Mon, Mar 12, 2018 at 6:11 PM, Fam Zheng <famz@redhat.com> wrote:
> On Mon, 03/12 15:11, Marc-André Lureau wrote:
>> Hi Fam,
>>
>> It would be great if patchew could run the debug/sanitizer build.
>> Unfortunately, except x86 target, the run is full of ASAN leaks or
>> warnings.
>>
>> Do you think patchew could learn to do regression testing? By that I
>> mean that patchew could compare a patchset output with the current
>> master (that output should be cached for multiple tests). The output
>> to compare here would be the errors reported by ASAN during the build.
>> (eventually, this could be used for other metrics some day). If so,
>> would you be willing to help me implementing it?
>
> The question is can we simply rely on the exit code of cmp or diff for that kind
> of comparison? If there is any out-of-order texts, that simple diff won't work.

If the tests are run sequentially, i think it should produce the same
output when run multiple times (to be verified).

However, if there are leaks already, and the address of allocation or
symbols change, this may produce diff on existing lines.

What I would like to see is the diff of added (or removed) lines.

I suppose that should be possible, I'll do some experiments.

> Anything more intelligent/sensible should go as a policy, and patchew should
> continue focus on mechanisms.

hmm ok :)

> I'll send a pull for the patch. Thanks!
>

thanks


-- 
Marc-André Lureau

Re: [Qemu-devel] [PATCH] tests: make docker-test-debug@fedora run sanitizers

[Fam Zheng]

On Mon, 03/12 18:17, Marc-André Lureau wrote:
> Hi
> 
> On Mon, Mar 12, 2018 at 6:11 PM, Fam Zheng <famz@redhat.com> wrote:
> > On Mon, 03/12 15:11, Marc-André Lureau wrote:
> >> Hi Fam,
> >>
> >> It would be great if patchew could run the debug/sanitizer build.
> >> Unfortunately, except x86 target, the run is full of ASAN leaks or
> >> warnings.
> >>
> >> Do you think patchew could learn to do regression testing? By that I
> >> mean that patchew could compare a patchset output with the current
> >> master (that output should be cached for multiple tests). The output
> >> to compare here would be the errors reported by ASAN during the build.
> >> (eventually, this could be used for other metrics some day). If so,
> >> would you be willing to help me implementing it?
> >
> > The question is can we simply rely on the exit code of cmp or diff for that kind
> > of comparison? If there is any out-of-order texts, that simple diff won't work.
> 
> If the tests are run sequentially, i think it should produce the same
> output when run multiple times (to be verified).

In that case the test will last for a long time. Currently patchew isn't very
good as multi-tasking. But that is one option we can explore (like we can devote
one VM for asan test).

> 
> However, if there are leaks already, and the address of allocation or
> symbols change, this may produce diff on existing lines.

Maybe we can come up with a 'make docker-test-debug-filtered@fedora' or a
relatively simple 'make docker-test-debug@fedora | sed -e "..."' command that
can be fed to diff.

> 
> What I would like to see is the diff of added (or removed) lines.
> 
> I suppose that should be possible, I'll do some experiments.
> 
> > Anything more intelligent/sensible should go as a policy, and patchew should
> > continue focus on mechanisms.
> 
> hmm ok :)

Like said above, I think it's possible to postprocess the output to make it easy
to diff/cmp. By mechanism/policy I mean the place to draw a line to code
Patchew.  However patchew has a database where we can write test script snippets
(as configurations) that is "the policy".

Fam