[PATCH] gitlab-ci.yml: Add fuzzer tests

[PATCH] gitlab-ci.yml: Add fuzzer tests

[Thomas Huth]

So far we neither compile-tested nor run any of the new fuzzers in our CI,
which led to some build failures of the fuzzer code in the past weeks.
To avoid this problem, add a job to compile the fuzzer code and run some
loops (which likely don't find any new bugs via fuzzing, but at least we
know that the code can still be run).

A nice side-effect of this test is that the leak tests are enabled here,
so we should now notice some of the memory leaks in our code base earlier.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .gitlab-ci.yml | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 5eeba2791b..e96f8794b9 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -161,9 +161,27 @@ build-clang:
     IMAGE: fedora
     CONFIGURE_ARGS: --cc=clang --cxx=clang++
     TARGETS: alpha-softmmu arm-softmmu m68k-softmmu mips64-softmmu
-      ppc-softmmu s390x-softmmu x86_64-softmmu arm-linux-user
+      ppc-softmmu s390x-softmmu arm-linux-user
     MAKE_CHECK_ARGS: check
 
+build-fuzzer:
+  <<: *native_build_job_definition
+  variables:
+    IMAGE: fedora
+  script:
+    - mkdir build
+    - cd build
+    - ../configure --cc=clang --cxx=clang++ --enable-fuzzing
+                   --target-list=x86_64-softmmu
+    - make -j"$JOBS" all check-build x86_64-softmmu/fuzz
+    - make check
+    - for fuzzer in i440fx-qos-fork-fuzz i440fx-qos-noreset-fuzz
+        i440fx-qtest-reboot-fuzz virtio-scsi-flags-fuzz virtio-scsi-fuzz ; do
+          echo Testing ${fuzzer} ... ;
+          x86_64-softmmu/qemu-fuzz-x86_64 --fuzz-target=${fuzzer} -runs=1000
+            || exit 1 ;
+      done
+
 build-tci:
   <<: *native_build_job_definition
   variables:
-- 
2.18.1

Re: [PATCH] gitlab-ci.yml: Add fuzzer tests

[Thomas Huth]

On 16/07/2020 12.09, Thomas Huth wrote:
> So far we neither compile-tested nor run any of the new fuzzers in our CI,
> which led to some build failures of the fuzzer code in the past weeks.
> To avoid this problem, add a job to compile the fuzzer code and run some
> loops (which likely don't find any new bugs via fuzzing, but at least we
> know that the code can still be run).
> 
> A nice side-effect of this test is that the leak tests are enabled here,
> so we should now notice some of the memory leaks in our code base earlier.
> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  .gitlab-ci.yml | 20 +++++++++++++++++++-
>  1 file changed, 19 insertions(+), 1 deletion(-)

Note: This patch needs two other patches merged first to work correctly:

- 'fuzz: Expect the cmdline in a freeable GString' from Alexander

- 'qom: Plug memory leak in "info qom-tree"' from Markus

Otherwise the test will fail due to detected memory leaks.

 Thomas

Re: [PATCH] gitlab-ci.yml: Add fuzzer tests

[Alexander Bulekov]

On 200716 1209, Thomas Huth wrote:
> So far we neither compile-tested nor run any of the new fuzzers in our CI,
> which led to some build failures of the fuzzer code in the past weeks.
> To avoid this problem, add a job to compile the fuzzer code and run some
> loops (which likely don't find any new bugs via fuzzing, but at least we
> know that the code can still be run).
> 
> A nice side-effect of this test is that the leak tests are enabled here,
> so we should now notice some of the memory leaks in our code base earlier.
> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>

Thank you for this, Thomas. I just sent a patch to add a job that runs
similar tests with the build-script that we use on oss-fuzz
Patch: <20200716163330.29141-1-alxndr@bu.edu>

I know that these jobs have a lot of overlap, but there are enough
quirks in the oss-fuzz build-script that I, personally, don't think
they are redundant.

A couple notes below, and I haven't been able to test on my own fork of
qemu on gitlab, yet due to some pipeline errors, but otherwise

Reviewed-by: Alexander Bulekov <alxndr@bu.edu>

> ---
>  .gitlab-ci.yml | 20 +++++++++++++++++++-
>  1 file changed, 19 insertions(+), 1 deletion(-)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 5eeba2791b..e96f8794b9 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -161,9 +161,27 @@ build-clang:
>      IMAGE: fedora
>      CONFIGURE_ARGS: --cc=clang --cxx=clang++
>      TARGETS: alpha-softmmu arm-softmmu m68k-softmmu mips64-softmmu
> -      ppc-softmmu s390x-softmmu x86_64-softmmu arm-linux-user
> +      ppc-softmmu s390x-softmmu arm-linux-user
>      MAKE_CHECK_ARGS: check
>  
> +build-fuzzer:
> +  <<: *native_build_job_definition
> +  variables:
> +    IMAGE: fedora
> +  script:
> +    - mkdir build
> +    - cd build
> +    - ../configure --cc=clang --cxx=clang++ --enable-fuzzing
> +                   --target-list=x86_64-softmmu

https://lists.nongnu.org/archive/html/qemu-devel/2020-07/msg02310.html
When/if this gets merged, enable-fuzzing won't build with
AddressSanitizer, by default, so I would add --enable-sanitizers, just
to be safe. 

> +    - make -j"$JOBS" all check-build x86_64-softmmu/fuzz
> +    - make check
> +    - for fuzzer in i440fx-qos-fork-fuzz i440fx-qos-noreset-fuzz
> +        i440fx-qtest-reboot-fuzz virtio-scsi-flags-fuzz virtio-scsi-fuzz ; do

Any reason for these particular fuzzers? I know the virtio-net ones find
crashes pretty quickly, but I dont think that causes a non-zero exit.

> +          echo Testing ${fuzzer} ... ;
> +          x86_64-softmmu/qemu-fuzz-x86_64 --fuzz-target=${fuzzer} -runs=1000
> +            || exit 1 ;
> +      done
> +
>  build-tci:
>    <<: *native_build_job_definition
>    variables:
> -- 
> 2.18.1
> 

Re: [PATCH] gitlab-ci.yml: Add fuzzer tests

[Thomas Huth]

On 16/07/2020 18.46, Alexander Bulekov wrote:
> On 200716 1209, Thomas Huth wrote:
>> So far we neither compile-tested nor run any of the new fuzzers in our CI,
>> which led to some build failures of the fuzzer code in the past weeks.
>> To avoid this problem, add a job to compile the fuzzer code and run some
>> loops (which likely don't find any new bugs via fuzzing, but at least we
>> know that the code can still be run).
>>
>> A nice side-effect of this test is that the leak tests are enabled here,
>> so we should now notice some of the memory leaks in our code base earlier.
>>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
> 
> Thank you for this, Thomas. I just sent a patch to add a job that runs
> similar tests with the build-script that we use on oss-fuzz
> Patch: <20200716163330.29141-1-alxndr@bu.edu>

Good idea! ... but it does not work quite yet, I gave it a try and ended
up here:

 https://gitlab.com/huth/qemu/-/jobs/643353500

> I know that these jobs have a lot of overlap, but there are enough
> quirks in the oss-fuzz build-script that I, personally, don't think
> they are redundant.

I think we should finally go with your approach and compile the fuzzing
test via the script. But since that seems to need some more work first,
let's go with my patch now, so that we have something for 5.1-rc1, and
then when your patch is ready, replace my "build-fuzzer" job with yours, ok?

>> +build-fuzzer:
>> +  <<: *native_build_job_definition
>> +  variables:
>> +    IMAGE: fedora
>> +  script:
>> +    - mkdir build
>> +    - cd build
>> +    - ../configure --cc=clang --cxx=clang++ --enable-fuzzing
>> +                   --target-list=x86_64-softmmu
> 
> https://lists.nongnu.org/archive/html/qemu-devel/2020-07/msg02310.html
> When/if this gets merged, enable-fuzzing won't build with
> AddressSanitizer, by default, so I would add --enable-sanitizers, just
> to be safe. 

Ok, thanks, I'll add that.

>> +    - make -j"$JOBS" all check-build x86_64-softmmu/fuzz
>> +    - make check
>> +    - for fuzzer in i440fx-qos-fork-fuzz i440fx-qos-noreset-fuzz
>> +        i440fx-qtest-reboot-fuzz virtio-scsi-flags-fuzz virtio-scsi-fuzz ; do
> 
> Any reason for these particular fuzzers? I know the virtio-net ones find
> crashes pretty quickly, but I dont think that causes a non-zero exit.

I did not include the virtio-net fuzzers because I read that warning
"May result in network traffic emitted from the  process. Run in an
isolated network environment." in the help text ... so I was not sure
whether they are really suitable for the CI on the gitlab machines?

 Thomas