* [PATCH] gitlab-ci.yml: Add fuzzer tests @ 2020-07-16 10:09 Thomas Huth 2020-07-16 10:29 ` Thomas Huth 2020-07-16 16:46 ` Alexander Bulekov 0 siblings, 2 replies; 4+ messages in thread From: Thomas Huth @ 2020-07-16 10:09 UTC (permalink / raw) To: qemu-devel, Alexander Bulekov Cc: Philippe Mathieu-Daudé, Markus Armbruster, Wainer dos Santos Moschetta, Bandan Das, Stefan Hajnoczi, Paolo Bonzini, Alex Bennée So far we neither compile-tested nor run any of the new fuzzers in our CI, which led to some build failures of the fuzzer code in the past weeks. To avoid this problem, add a job to compile the fuzzer code and run some loops (which likely don't find any new bugs via fuzzing, but at least we know that the code can still be run). A nice side-effect of this test is that the leak tests are enabled here, so we should now notice some of the memory leaks in our code base earlier. Signed-off-by: Thomas Huth <thuth@redhat.com> --- .gitlab-ci.yml | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 5eeba2791b..e96f8794b9 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -161,9 +161,27 @@ build-clang: IMAGE: fedora CONFIGURE_ARGS: --cc=clang --cxx=clang++ TARGETS: alpha-softmmu arm-softmmu m68k-softmmu mips64-softmmu - ppc-softmmu s390x-softmmu x86_64-softmmu arm-linux-user + ppc-softmmu s390x-softmmu arm-linux-user MAKE_CHECK_ARGS: check +build-fuzzer: + <<: *native_build_job_definition + variables: + IMAGE: fedora + script: + - mkdir build + - cd build + - ../configure --cc=clang --cxx=clang++ --enable-fuzzing + --target-list=x86_64-softmmu + - make -j"$JOBS" all check-build x86_64-softmmu/fuzz + - make check + - for fuzzer in i440fx-qos-fork-fuzz i440fx-qos-noreset-fuzz + i440fx-qtest-reboot-fuzz virtio-scsi-flags-fuzz virtio-scsi-fuzz ; do + echo Testing ${fuzzer} ... ; + x86_64-softmmu/qemu-fuzz-x86_64 --fuzz-target=${fuzzer} -runs=1000 + || exit 1 ; + done + build-tci: <<: *native_build_job_definition variables: -- 2.18.1 ^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH] gitlab-ci.yml: Add fuzzer tests 2020-07-16 10:09 [PATCH] gitlab-ci.yml: Add fuzzer tests Thomas Huth @ 2020-07-16 10:29 ` Thomas Huth 2020-07-16 16:46 ` Alexander Bulekov 1 sibling, 0 replies; 4+ messages in thread From: Thomas Huth @ 2020-07-16 10:29 UTC (permalink / raw) To: qemu-devel Cc: Alex Bennée, Markus Armbruster, Wainer dos Santos Moschetta, Alexander Bulekov, Bandan Das, Stefan Hajnoczi, Paolo Bonzini, Philippe Mathieu-Daudé On 16/07/2020 12.09, Thomas Huth wrote: > So far we neither compile-tested nor run any of the new fuzzers in our CI, > which led to some build failures of the fuzzer code in the past weeks. > To avoid this problem, add a job to compile the fuzzer code and run some > loops (which likely don't find any new bugs via fuzzing, but at least we > know that the code can still be run). > > A nice side-effect of this test is that the leak tests are enabled here, > so we should now notice some of the memory leaks in our code base earlier. > > Signed-off-by: Thomas Huth <thuth@redhat.com> > --- > .gitlab-ci.yml | 20 +++++++++++++++++++- > 1 file changed, 19 insertions(+), 1 deletion(-) Note: This patch needs two other patches merged first to work correctly: - 'fuzz: Expect the cmdline in a freeable GString' from Alexander - 'qom: Plug memory leak in "info qom-tree"' from Markus Otherwise the test will fail due to detected memory leaks. Thomas ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH] gitlab-ci.yml: Add fuzzer tests 2020-07-16 10:09 [PATCH] gitlab-ci.yml: Add fuzzer tests Thomas Huth 2020-07-16 10:29 ` Thomas Huth @ 2020-07-16 16:46 ` Alexander Bulekov 2020-07-17 5:33 ` Thomas Huth 1 sibling, 1 reply; 4+ messages in thread From: Alexander Bulekov @ 2020-07-16 16:46 UTC (permalink / raw) To: Thomas Huth Cc: Philippe Mathieu-Daudé, qemu-devel, Wainer dos Santos Moschetta, Markus Armbruster, Bandan Das, Stefan Hajnoczi, Paolo Bonzini, Alex Bennée On 200716 1209, Thomas Huth wrote: > So far we neither compile-tested nor run any of the new fuzzers in our CI, > which led to some build failures of the fuzzer code in the past weeks. > To avoid this problem, add a job to compile the fuzzer code and run some > loops (which likely don't find any new bugs via fuzzing, but at least we > know that the code can still be run). > > A nice side-effect of this test is that the leak tests are enabled here, > so we should now notice some of the memory leaks in our code base earlier. > > Signed-off-by: Thomas Huth <thuth@redhat.com> Thank you for this, Thomas. I just sent a patch to add a job that runs similar tests with the build-script that we use on oss-fuzz Patch: <20200716163330.29141-1-alxndr@bu.edu> I know that these jobs have a lot of overlap, but there are enough quirks in the oss-fuzz build-script that I, personally, don't think they are redundant. A couple notes below, and I haven't been able to test on my own fork of qemu on gitlab, yet due to some pipeline errors, but otherwise Reviewed-by: Alexander Bulekov <alxndr@bu.edu> > --- > .gitlab-ci.yml | 20 +++++++++++++++++++- > 1 file changed, 19 insertions(+), 1 deletion(-) > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml > index 5eeba2791b..e96f8794b9 100644 > --- a/.gitlab-ci.yml > +++ b/.gitlab-ci.yml > @@ -161,9 +161,27 @@ build-clang: > IMAGE: fedora > CONFIGURE_ARGS: --cc=clang --cxx=clang++ > TARGETS: alpha-softmmu arm-softmmu m68k-softmmu mips64-softmmu > - ppc-softmmu s390x-softmmu x86_64-softmmu arm-linux-user > + ppc-softmmu s390x-softmmu arm-linux-user > MAKE_CHECK_ARGS: check > > +build-fuzzer: > + <<: *native_build_job_definition > + variables: > + IMAGE: fedora > + script: > + - mkdir build > + - cd build > + - ../configure --cc=clang --cxx=clang++ --enable-fuzzing > + --target-list=x86_64-softmmu https://lists.nongnu.org/archive/html/qemu-devel/2020-07/msg02310.html When/if this gets merged, enable-fuzzing won't build with AddressSanitizer, by default, so I would add --enable-sanitizers, just to be safe. > + - make -j"$JOBS" all check-build x86_64-softmmu/fuzz > + - make check > + - for fuzzer in i440fx-qos-fork-fuzz i440fx-qos-noreset-fuzz > + i440fx-qtest-reboot-fuzz virtio-scsi-flags-fuzz virtio-scsi-fuzz ; do Any reason for these particular fuzzers? I know the virtio-net ones find crashes pretty quickly, but I dont think that causes a non-zero exit. > + echo Testing ${fuzzer} ... ; > + x86_64-softmmu/qemu-fuzz-x86_64 --fuzz-target=${fuzzer} -runs=1000 > + || exit 1 ; > + done > + > build-tci: > <<: *native_build_job_definition > variables: > -- > 2.18.1 > ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH] gitlab-ci.yml: Add fuzzer tests 2020-07-16 16:46 ` Alexander Bulekov @ 2020-07-17 5:33 ` Thomas Huth 0 siblings, 0 replies; 4+ messages in thread From: Thomas Huth @ 2020-07-17 5:33 UTC (permalink / raw) To: Alexander Bulekov Cc: Philippe Mathieu-Daudé, qemu-devel, Wainer dos Santos Moschetta, Markus Armbruster, Bandan Das, Stefan Hajnoczi, Paolo Bonzini, Alex Bennée On 16/07/2020 18.46, Alexander Bulekov wrote: > On 200716 1209, Thomas Huth wrote: >> So far we neither compile-tested nor run any of the new fuzzers in our CI, >> which led to some build failures of the fuzzer code in the past weeks. >> To avoid this problem, add a job to compile the fuzzer code and run some >> loops (which likely don't find any new bugs via fuzzing, but at least we >> know that the code can still be run). >> >> A nice side-effect of this test is that the leak tests are enabled here, >> so we should now notice some of the memory leaks in our code base earlier. >> >> Signed-off-by: Thomas Huth <thuth@redhat.com> > > Thank you for this, Thomas. I just sent a patch to add a job that runs > similar tests with the build-script that we use on oss-fuzz > Patch: <20200716163330.29141-1-alxndr@bu.edu> Good idea! ... but it does not work quite yet, I gave it a try and ended up here: https://gitlab.com/huth/qemu/-/jobs/643353500 > I know that these jobs have a lot of overlap, but there are enough > quirks in the oss-fuzz build-script that I, personally, don't think > they are redundant. I think we should finally go with your approach and compile the fuzzing test via the script. But since that seems to need some more work first, let's go with my patch now, so that we have something for 5.1-rc1, and then when your patch is ready, replace my "build-fuzzer" job with yours, ok? >> +build-fuzzer: >> + <<: *native_build_job_definition >> + variables: >> + IMAGE: fedora >> + script: >> + - mkdir build >> + - cd build >> + - ../configure --cc=clang --cxx=clang++ --enable-fuzzing >> + --target-list=x86_64-softmmu > > https://lists.nongnu.org/archive/html/qemu-devel/2020-07/msg02310.html > When/if this gets merged, enable-fuzzing won't build with > AddressSanitizer, by default, so I would add --enable-sanitizers, just > to be safe. Ok, thanks, I'll add that. >> + - make -j"$JOBS" all check-build x86_64-softmmu/fuzz >> + - make check >> + - for fuzzer in i440fx-qos-fork-fuzz i440fx-qos-noreset-fuzz >> + i440fx-qtest-reboot-fuzz virtio-scsi-flags-fuzz virtio-scsi-fuzz ; do > > Any reason for these particular fuzzers? I know the virtio-net ones find > crashes pretty quickly, but I dont think that causes a non-zero exit. I did not include the virtio-net fuzzers because I read that warning "May result in network traffic emitted from the process. Run in an isolated network environment." in the help text ... so I was not sure whether they are really suitable for the CI on the gitlab machines? Thomas ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2020-07-17 5:35 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-07-16 10:09 [PATCH] gitlab-ci.yml: Add fuzzer tests Thomas Huth 2020-07-16 10:29 ` Thomas Huth 2020-07-16 16:46 ` Alexander Bulekov 2020-07-17 5:33 ` Thomas Huth
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.