[PATCH] gitlab-ci.yml: Add oss-fuzz build tests

[PATCH] gitlab-ci.yml: Add oss-fuzz build tests

[Alexander Bulekov]

This tries to build and run the fuzzers with the same build-script used
by oss-fuzz. This doesn't guarantee that the builds on oss-fuzz will
also succeed, since oss-fuzz provides its own compiler and fuzzer vars,
but it can catch changes that are not compatible with the the
./scripts/oss-fuzz/build.sh script.
The strange way of finding fuzzer binaries stems from the method used by
oss-fuzz:
https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-runner/targets_list

Signed-off-by: Alexander Bulekov <alxndr@bu.edu>
---

Similar to Thomas' patch:

> Note: This patch needs two other patches merged first to work correctly:

> - 'fuzz: Expect the cmdline in a freeable GString' from Alexander

> - 'qom: Plug memory leak in "info qom-tree"' from Markus

Otherwise the test will fail due to detected memory leaks.

Fair warning: I haven't been able to trigger this new job yet. I tried
to run the pipeline with these changes on my forked repo on gitlab, but
did not reach the build-oss-fuzz. I think this is due to some failures
in the Containers-layer-2 stage:

...
Error response from daemon: manifest for
registry.gitlab.com/a1xndr/qemu/qemu/debian-all-test-cross:latest not
found: manifest unknown: manifest unknown
#2 [internal] load .dockerignore
#2 transferring context:
#2 transferring context: 2B 0.1s done
#2 DONE 0.1s
#1 [internal] load build definition from tmpg8j4xoop.docker
#1 transferring dockerfile: 2.21kB 0.1s done
#1 DONE 0.2s
#3 [internal] load metadata for docker.io/qemu/debian10:latest
#3 ERROR: pull access denied, repository does not exist or may require
authorization: server message: insufficient_scope: authorization failed
#4 [1/3] FROM docker.io/qemu/debian10:latest
#4 resolve docker.io/qemu/debian10:latest 0.1s done
#4 ERROR: pull access denied, repository does not exist or may require
authorization: server message: insufficient_scope: authorization failed
------
 > [internal] load metadata for docker.io/qemu/debian10:latest:
------
------
 > [1/3] FROM docker.io/qemu/debian10:latest:
------
failed to solve with frontend dockerfile.v0: failed to build LLB: failed
to load cache key: pull access denied, repository does not exist or may
require authorization: server message: insufficient_scope: authorization
failed
...

 .gitlab-ci.yml | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index e96f8794b9..a50df420c9 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -182,6 +182,20 @@ build-fuzzer:
             || exit 1 ;
       done
 
+build-oss-fuzz:
+  <<: *native_build_job_definition
+  variables:
+    IMAGE: fedora
+  script:
+    - OUT_DIR="./build" CC=clang-9 CXX=clang++-9 CFLAGS="-fsanitize=address"
+      LIB_FUZZING_ENGINE="-fsanitize=fuzzer" CFL
+      ./scripts/oss-fuzz/build.sh
+    - for fuzzer in $(find ./build-oss-fuzz/DEST_DIR/ -executable -type f); do
+        grep "LLVMFuzzerTestOneInput" ${fuzzer} > /dev/null 2>&1 || continue ;
+        echo Testing ${fuzzer} ... ;
+        "${fuzzer}" -runs=1000 || exit 1 ;
+      done
+
 build-tci:
   <<: *native_build_job_definition
   variables:
-- 
2.26.2

Re: [PATCH] gitlab-ci.yml: Add oss-fuzz build tests

[Thomas Huth]

On 16/07/2020 18.33, Alexander Bulekov wrote:
> This tries to build and run the fuzzers with the same build-script used
> by oss-fuzz. This doesn't guarantee that the builds on oss-fuzz will
> also succeed, since oss-fuzz provides its own compiler and fuzzer vars,
> but it can catch changes that are not compatible with the the
> ./scripts/oss-fuzz/build.sh script.
> The strange way of finding fuzzer binaries stems from the method used by
> oss-fuzz:
> https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-runner/targets_list
> 
> Signed-off-by: Alexander Bulekov <alxndr@bu.edu>
> ---
> 
> Similar to Thomas' patch:
> 
>> Note: This patch needs two other patches merged first to work correctly:
> 
>> - 'fuzz: Expect the cmdline in a freeable GString' from Alexander
> 
>> - 'qom: Plug memory leak in "info qom-tree"' from Markus
> 
> Otherwise the test will fail due to detected memory leaks.
> 
> Fair warning: I haven't been able to trigger this new job yet. I tried
> to run the pipeline with these changes on my forked repo on gitlab, but
> did not reach the build-oss-fuzz. I think this is due to some failures
> in the Containers-layer-2 stage:
> 
> ...
> Error response from daemon: manifest for
> registry.gitlab.com/a1xndr/qemu/qemu/debian-all-test-cross:latest not
> found: manifest unknown: manifest unknown
> #2 [internal] load .dockerignore
> #2 transferring context:
> #2 transferring context: 2B 0.1s done
> #2 DONE 0.1s
> #1 [internal] load build definition from tmpg8j4xoop.docker
> #1 transferring dockerfile: 2.21kB 0.1s done
> #1 DONE 0.2s
> #3 [internal] load metadata for docker.io/qemu/debian10:latest
> #3 ERROR: pull access denied, repository does not exist or may require
> authorization: server message: insufficient_scope: authorization failed

These look like the problems that we've seen with the main repo until
two days ago, too, e.g.:

 https://gitlab.com/qemu-project/qemu/-/jobs/640410842

Maybe Alex (Bennée) can comment on how to resolve them?

> 
>  .gitlab-ci.yml | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index e96f8794b9..a50df420c9 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -182,6 +182,20 @@ build-fuzzer:
>              || exit 1 ;
>        done

As mentioned in my other mail, I think you can replace my build-fuzzer
job once this is working.

> +build-oss-fuzz:
> +  <<: *native_build_job_definition
> +  variables:
> +    IMAGE: fedora
> +  script:
> +    - OUT_DIR="./build" CC=clang-9 CXX=clang++-9 CFLAGS="-fsanitize=address"
> +      LIB_FUZZING_ENGINE="-fsanitize=fuzzer" CFL

That "CFL" at the end seems to be a typo (leftover from "CFLAGS")?

Also the fedora container does not have clang-9 :

 https://gitlab.com/huth/qemu/-/jobs/643383032#L28

I think it is at clang 10 already, so maybe just use CC=clang (without
version number)?

> +      ./scripts/oss-fuzz/build.sh
> +    - for fuzzer in $(find ./build-oss-fuzz/DEST_DIR/ -executable -type f); do
> +        grep "LLVMFuzzerTestOneInput" ${fuzzer} > /dev/null 2>&1 || continue ;
> +        echo Testing ${fuzzer} ... ;
> +        "${fuzzer}" -runs=1000 || exit 1 ;
> +      done

Should we exclude the virtio-net tests, since they could leak network
traffic to the host?

 Thomas

Re: [PATCH] gitlab-ci.yml: Add oss-fuzz build tests

[Thomas Huth]

On 17/07/2020 07.40, Thomas Huth wrote:
> On 16/07/2020 18.33, Alexander Bulekov wrote:
>> This tries to build and run the fuzzers with the same build-script used
>> by oss-fuzz. This doesn't guarantee that the builds on oss-fuzz will
>> also succeed, since oss-fuzz provides its own compiler and fuzzer vars,
>> but it can catch changes that are not compatible with the the
>> ./scripts/oss-fuzz/build.sh script.
>> The strange way of finding fuzzer binaries stems from the method used by
>> oss-fuzz:
>> https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-runner/targets_list
>>
>> Signed-off-by: Alexander Bulekov <alxndr@bu.edu>
>> ---
>>
>> Similar to Thomas' patch:
>>
>>> Note: This patch needs two other patches merged first to work correctly:
>>
>>> - 'fuzz: Expect the cmdline in a freeable GString' from Alexander
>>
>>> - 'qom: Plug memory leak in "info qom-tree"' from Markus
>>
>> Otherwise the test will fail due to detected memory leaks.
>>
>> Fair warning: I haven't been able to trigger this new job yet. I tried
>> to run the pipeline with these changes on my forked repo on gitlab, but
>> did not reach the build-oss-fuzz. I think this is due to some failures
>> in the Containers-layer-2 stage:

I think I've got it basically working like this:

build-oss-fuzz:
  <<: *native_build_job_definition
  variables:
    IMAGE: fedora
  script:
    - mkdir build-oss-fuzz
    - CC="clang" CXX="clang++" CFLAGS="-fsanitize=address"
      ./scripts/oss-fuzz/build.sh
    - for fuzzer in $(find build-oss-fuzz/DEST_DIR/ -executable -type f
                      | grep -v slirp); do
        grep "LLVMFuzzerTestOneInput" ${fuzzer} > /dev/null 2>&1 ||
continue ;
        echo Testing ${fuzzer} ... ;
        "${fuzzer}" -runs=1000 || exit 1 ;
      done

However, it still triggered a memory leak and thus the pipeline failed:

 https://gitlab.com/huth/qemu/-/jobs/643472695

... and this was with all the other "leak fix" patches already applied.
Is there an easy way to debug that issue? That information from the
LeakSanitizer looks pretty sparse...

 Thomas

Re: [PATCH] gitlab-ci.yml: Add oss-fuzz build tests

[Alexander Bulekov]

On 200717 0951, Thomas Huth wrote:
> On 17/07/2020 07.40, Thomas Huth wrote:
> > On 16/07/2020 18.33, Alexander Bulekov wrote:
> >> This tries to build and run the fuzzers with the same build-script used
> >> by oss-fuzz. This doesn't guarantee that the builds on oss-fuzz will
> >> also succeed, since oss-fuzz provides its own compiler and fuzzer vars,
> >> but it can catch changes that are not compatible with the the
> >> ./scripts/oss-fuzz/build.sh script.
> >> The strange way of finding fuzzer binaries stems from the method used by
> >> oss-fuzz:
> >> https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-runner/targets_list
> >>
> >> Signed-off-by: Alexander Bulekov <alxndr@bu.edu>
> >> ---
> >>
> >> Similar to Thomas' patch:
> >>
> >>> Note: This patch needs two other patches merged first to work correctly:
> >>
> >>> - 'fuzz: Expect the cmdline in a freeable GString' from Alexander
> >>
> >>> - 'qom: Plug memory leak in "info qom-tree"' from Markus
> >>
> >> Otherwise the test will fail due to detected memory leaks.
> >>
> >> Fair warning: I haven't been able to trigger this new job yet. I tried
> >> to run the pipeline with these changes on my forked repo on gitlab, but
> >> did not reach the build-oss-fuzz. I think this is due to some failures
> >> in the Containers-layer-2 stage:
> 
> I think I've got it basically working like this:
> 
> build-oss-fuzz:
>   <<: *native_build_job_definition
>   variables:
>     IMAGE: fedora
>   script:
>     - mkdir build-oss-fuzz
>     - CC="clang" CXX="clang++" CFLAGS="-fsanitize=address"
>       ./scripts/oss-fuzz/build.sh
>     - for fuzzer in $(find build-oss-fuzz/DEST_DIR/ -executable -type f
>                       | grep -v slirp); do
>         grep "LLVMFuzzerTestOneInput" ${fuzzer} > /dev/null 2>&1 ||
> continue ;
>         echo Testing ${fuzzer} ... ;
>         "${fuzzer}" -runs=1000 || exit 1 ;
>       done
> 
> However, it still triggered a memory leak and thus the pipeline failed:
> 
>  https://gitlab.com/huth/qemu/-/jobs/643472695
> 
> ... and this was with all the other "leak fix" patches already applied.
> Is there an easy way to debug that issue? That information from the
> LeakSanitizer looks pretty sparse...

Strange... Especially since Philippe didn't get the same error. We
should probably add -seed=1 after -runs=1000, to make sure the fuzzer is
using the same rng seed. That said, I just ran the fuzzer for 50k
iterations, and still could not reproduce the leak...

This environment variable should fix the partial stacktrace, so we don't
have to guess next time:
ASAN_OPTIONS="fast_unwind_on_malloc=0"
-Alex

>  Thomas
> 

Re: [PATCH] gitlab-ci.yml: Add oss-fuzz build tests

[Thomas Huth]

On 17/07/2020 15.20, Alexander Bulekov wrote:
> On 200717 0951, Thomas Huth wrote:
>> On 17/07/2020 07.40, Thomas Huth wrote:
[...]
>> I think I've got it basically working like this:
>>
>> build-oss-fuzz:
>>   <<: *native_build_job_definition
>>   variables:
>>     IMAGE: fedora
>>   script:
>>     - mkdir build-oss-fuzz
>>     - CC="clang" CXX="clang++" CFLAGS="-fsanitize=address"
>>       ./scripts/oss-fuzz/build.sh
>>     - for fuzzer in $(find build-oss-fuzz/DEST_DIR/ -executable -type f
>>                       | grep -v slirp); do
>>         grep "LLVMFuzzerTestOneInput" ${fuzzer} > /dev/null 2>&1 ||
>> continue ;
>>         echo Testing ${fuzzer} ... ;
>>         "${fuzzer}" -runs=1000 || exit 1 ;
>>       done
>>
>> However, it still triggered a memory leak and thus the pipeline failed:
>>
>>  https://gitlab.com/huth/qemu/-/jobs/643472695
>>
>> ... and this was with all the other "leak fix" patches already applied.
>> Is there an easy way to debug that issue? That information from the
>> LeakSanitizer looks pretty sparse...
> 
> Strange... Especially since Philippe didn't get the same error. We
> should probably add -seed=1 after -runs=1000, to make sure the fuzzer is
> using the same rng seed. That said, I just ran the fuzzer for 50k
> iterations, and still could not reproduce the leak...
> 
> This environment variable should fix the partial stacktrace, so we don't
> have to guess next time:
> ASAN_OPTIONS="fast_unwind_on_malloc=0"

Thanks, that did the trick:

 https://gitlab.com/huth/qemu/-/jobs/644354706#L3616

... that also explains why I haven't seen it in my other tests where I
am using the --fuzz-target parameter : The leak only happens if the
fuzz-target is encoded in the program name - looks like we have to free
the memory returned by g_path_get_dirname() there...

 Thomas

Re: [PATCH] gitlab-ci.yml: Add oss-fuzz build tests

[Alex Bennée]

Thomas Huth <thuth@redhat.com> writes:

> On 16/07/2020 18.33, Alexander Bulekov wrote:
>> This tries to build and run the fuzzers with the same build-script used
>> by oss-fuzz. This doesn't guarantee that the builds on oss-fuzz will
>> also succeed, since oss-fuzz provides its own compiler and fuzzer vars,
>> but it can catch changes that are not compatible with the the
>> ./scripts/oss-fuzz/build.sh script.
>> The strange way of finding fuzzer binaries stems from the method used by
>> oss-fuzz:
>> https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-runner/targets_list
>> 
>> Signed-off-by: Alexander Bulekov <alxndr@bu.edu>
>> ---
>> 
>> Similar to Thomas' patch:
>> 
>>> Note: This patch needs two other patches merged first to work correctly:
>> 
>>> - 'fuzz: Expect the cmdline in a freeable GString' from Alexander
>> 
>>> - 'qom: Plug memory leak in "info qom-tree"' from Markus
>> 
>> Otherwise the test will fail due to detected memory leaks.
>> 
>> Fair warning: I haven't been able to trigger this new job yet. I tried
>> to run the pipeline with these changes on my forked repo on gitlab, but
>> did not reach the build-oss-fuzz. I think this is due to some failures
>> in the Containers-layer-2 stage:
>> 
>> ...
>> Error response from daemon: manifest for
>> registry.gitlab.com/a1xndr/qemu/qemu/debian-all-test-cross:latest not
>> found: manifest unknown: manifest unknown
>> #2 [internal] load .dockerignore
>> #2 transferring context:
>> #2 transferring context: 2B 0.1s done
>> #2 DONE 0.1s
>> #1 [internal] load build definition from tmpg8j4xoop.docker
>> #1 transferring dockerfile: 2.21kB 0.1s done
>> #1 DONE 0.2s
>> #3 [internal] load metadata for docker.io/qemu/debian10:latest
>> #3 ERROR: pull access denied, repository does not exist or may require
>> authorization: server message: insufficient_scope: authorization failed
>
> These look like the problems that we've seen with the main repo until
> two days ago, too, e.g.:
>
>  https://gitlab.com/qemu-project/qemu/-/jobs/640410842
>
> Maybe Alex (Bennée) can comment on how to resolve them?

It all should be working now the qemu-project container repository has
been properly seeded:

  https://gitlab.com/qemu-project/qemu/container_registry

>
>> 
>>  .gitlab-ci.yml | 14 ++++++++++++++
>>  1 file changed, 14 insertions(+)
>> 
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index e96f8794b9..a50df420c9 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -182,6 +182,20 @@ build-fuzzer:
>>              || exit 1 ;
>>        done
>
> As mentioned in my other mail, I think you can replace my build-fuzzer
> job once this is working.
>
>> +build-oss-fuzz:
>> +  <<: *native_build_job_definition
>> +  variables:
>> +    IMAGE: fedora
>> +  script:
>> +    - OUT_DIR="./build" CC=clang-9 CXX=clang++-9 CFLAGS="-fsanitize=address"
>> +      LIB_FUZZING_ENGINE="-fsanitize=fuzzer" CFL
>
> That "CFL" at the end seems to be a typo (leftover from "CFLAGS")?
>
> Also the fedora container does not have clang-9 :
>
>  https://gitlab.com/huth/qemu/-/jobs/643383032#L28
>
> I think it is at clang 10 already, so maybe just use CC=clang (without
> version number)?

I think all the clang-10 fixes are in now so yes.

>
>> +      ./scripts/oss-fuzz/build.sh
>> +    - for fuzzer in $(find ./build-oss-fuzz/DEST_DIR/ -executable -type f); do
>> +        grep "LLVMFuzzerTestOneInput" ${fuzzer} > /dev/null 2>&1 || continue ;
>> +        echo Testing ${fuzzer} ... ;
>> +        "${fuzzer}" -runs=1000 || exit 1 ;
>> +      done
>
> Should we exclude the virtio-net tests, since they could leak network
> traffic to the host?
>
>  Thomas


-- 
Alex Bennée

Re: [PATCH] gitlab-ci.yml: Add oss-fuzz build tests

[Alexander Bulekov]

On 200717 0740, Thomas Huth wrote:
> On 16/07/2020 18.33, Alexander Bulekov wrote:
> > This tries to build and run the fuzzers with the same build-script used
> > by oss-fuzz. This doesn't guarantee that the builds on oss-fuzz will
> > also succeed, since oss-fuzz provides its own compiler and fuzzer vars,
> > but it can catch changes that are not compatible with the the
> > ./scripts/oss-fuzz/build.sh script.
> > The strange way of finding fuzzer binaries stems from the method used by
> > oss-fuzz:
> > https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-runner/targets_list
> > 
> > Signed-off-by: Alexander Bulekov <alxndr@bu.edu>
> > ---
> > 
> > Similar to Thomas' patch:
> > 
> >> Note: This patch needs two other patches merged first to work correctly:
> > 
> >> - 'fuzz: Expect the cmdline in a freeable GString' from Alexander
> > 
> >> - 'qom: Plug memory leak in "info qom-tree"' from Markus
> > 
> > Otherwise the test will fail due to detected memory leaks.
> > 
> > Fair warning: I haven't been able to trigger this new job yet. I tried
> > to run the pipeline with these changes on my forked repo on gitlab, but
> > did not reach the build-oss-fuzz. I think this is due to some failures
> > in the Containers-layer-2 stage:
> > 
> > ...
> > Error response from daemon: manifest for
> > registry.gitlab.com/a1xndr/qemu/qemu/debian-all-test-cross:latest not
> > found: manifest unknown: manifest unknown
> > #2 [internal] load .dockerignore
> > #2 transferring context:
> > #2 transferring context: 2B 0.1s done
> > #2 DONE 0.1s
> > #1 [internal] load build definition from tmpg8j4xoop.docker
> > #1 transferring dockerfile: 2.21kB 0.1s done
> > #1 DONE 0.2s
> > #3 [internal] load metadata for docker.io/qemu/debian10:latest
> > #3 ERROR: pull access denied, repository does not exist or may require
> > authorization: server message: insufficient_scope: authorization failed
> 
> These look like the problems that we've seen with the main repo until
> two days ago, too, e.g.:
> 
>  https://gitlab.com/qemu-project/qemu/-/jobs/640410842
> 
> Maybe Alex (Bennée) can comment on how to resolve them?
> 
> > 
> >  .gitlab-ci.yml | 14 ++++++++++++++
> >  1 file changed, 14 insertions(+)
> > 
> > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > index e96f8794b9..a50df420c9 100644
> > --- a/.gitlab-ci.yml
> > +++ b/.gitlab-ci.yml
> > @@ -182,6 +182,20 @@ build-fuzzer:
> >              || exit 1 ;
> >        done
> 
> As mentioned in my other mail, I think you can replace my build-fuzzer
> job once this is working.
> 
> > +build-oss-fuzz:
> > +  <<: *native_build_job_definition
> > +  variables:
> > +    IMAGE: fedora
> > +  script:
> > +    - OUT_DIR="./build" CC=clang-9 CXX=clang++-9 CFLAGS="-fsanitize=address"
> > +      LIB_FUZZING_ENGINE="-fsanitize=fuzzer" CFL
> 
> That "CFL" at the end seems to be a typo (leftover from "CFLAGS")?

oops...

> Also the fedora container does not have clang-9 :
> 
>  https://gitlab.com/huth/qemu/-/jobs/643383032#L28
> 
> I think it is at clang 10 already, so maybe just use CC=clang (without
> version number)?
> 
For some reason my local machine doesn't have symlinks for clang and
clang++, and I forgot to remove the versions when I copied the command.

> > +      ./scripts/oss-fuzz/build.sh
> > +    - for fuzzer in $(find ./build-oss-fuzz/DEST_DIR/ -executable -type f); do
> > +        grep "LLVMFuzzerTestOneInput" ${fuzzer} > /dev/null 2>&1 || continue ;
> > +        echo Testing ${fuzzer} ... ;
> > +        "${fuzzer}" -runs=1000 || exit 1 ;
> > +      done
> 
> Should we exclude the virtio-net tests, since they could leak network
> traffic to the host?

Ah good point. I doubt that 1000 runs is enough to generate something
that slirp will forward, but we should probably skip over these targets
just to be on the safe side.

-Alex

>  Thomas
>