[PATCH] gitlab-ci.yml: Run check-qtest and check-unit at the end of the fuzzer job

[PATCH] gitlab-ci.yml: Run check-qtest and check-unit at the end of the fuzzer job

[Thomas Huth]

The fuzzer job finishes quite early, so we can run the unit tests and
qtests with -fsanitize=address here without extending the total test time.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .gitlab-ci.yml | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 17f1f8fad9..417fda6909 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -256,13 +256,14 @@ build-oss-fuzz:
     - mkdir build-oss-fuzz
     - CC="clang" CXX="clang++" CFLAGS="-fsanitize=address"
       ./scripts/oss-fuzz/build.sh
+    - export ASAN_OPTIONS="fast_unwind_on_malloc=0"
     - for fuzzer in $(find ./build-oss-fuzz/DEST_DIR/ -executable -type f
                       | grep -v slirp); do
         grep "LLVMFuzzerTestOneInput" ${fuzzer} > /dev/null 2>&1 || continue ;
         echo Testing ${fuzzer} ... ;
-        ASAN_OPTIONS="fast_unwind_on_malloc=0"
-         "${fuzzer}" -runs=1000 -seed=1 || exit 1 ;
+        "${fuzzer}" -runs=1000 -seed=1 || exit 1 ;
       done
+    - cd build-oss-fuzz && make check-qtest-i386 check-unit
 
 build-tci:
   <<: *native_build_job_definition
-- 
2.18.2

Re: [PATCH] gitlab-ci.yml: Run check-qtest and check-unit at the end of the fuzzer job

[Alexander Bulekov]

On 200831 1732, Thomas Huth wrote:
> The fuzzer job finishes quite early, so we can run the unit tests and
> qtests with -fsanitize=address here without extending the total test time.
> 

Sounds good to me, though its too bad that this is limited to i386 and
we aren't using the --enable-sanitizers configure argument due to the
strange oss-fuzz build script.

Speaking of testing with sanitizers, does it make sense to have a job
that does check-qtest with --enable-tsan, now that we have
ThreadSanitizer support? 

Can --enable-sanitizers --enable-tsan be added to some existing job,
without severely increasing the pipeline's exeuction time?

Reviewed-by: Alexander Bulekov <alxndr@bu.edu>

> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  .gitlab-ci.yml | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 17f1f8fad9..417fda6909 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -256,13 +256,14 @@ build-oss-fuzz:
>      - mkdir build-oss-fuzz
>      - CC="clang" CXX="clang++" CFLAGS="-fsanitize=address"
>        ./scripts/oss-fuzz/build.sh
> +    - export ASAN_OPTIONS="fast_unwind_on_malloc=0"
>      - for fuzzer in $(find ./build-oss-fuzz/DEST_DIR/ -executable -type f
>                        | grep -v slirp); do
>          grep "LLVMFuzzerTestOneInput" ${fuzzer} > /dev/null 2>&1 || continue ;
>          echo Testing ${fuzzer} ... ;
> -        ASAN_OPTIONS="fast_unwind_on_malloc=0"
> -         "${fuzzer}" -runs=1000 -seed=1 || exit 1 ;
> +        "${fuzzer}" -runs=1000 -seed=1 || exit 1 ;
>        done
> +    - cd build-oss-fuzz && make check-qtest-i386 check-unit
>  
>  build-tci:
>    <<: *native_build_job_definition
> -- 
> 2.18.2
> 

Re: [PATCH] gitlab-ci.yml: Run check-qtest and check-unit at the end of the fuzzer job

[Philippe Mathieu-Daudé]

[-- Attachment #1: Type: text/plain, Size: 1414 bytes --]

Hi Thomas,

Le lun. 31 août 2020 17:33, Thomas Huth <thuth@redhat.com> a écrit :

> The fuzzer job finishes quite early, so we can run the unit tests and
> qtests with -fsanitize=address here without extending the total test time.
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  .gitlab-ci.yml | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 17f1f8fad9..417fda6909 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -256,13 +256,14 @@ build-oss-fuzz:
>      - mkdir build-oss-fuzz
>      - CC="clang" CXX="clang++" CFLAGS="-fsanitize=address"
>        ./scripts/oss-fuzz/build.sh
> +    - export ASAN_OPTIONS="fast_unwind_on_malloc=0"
>      - for fuzzer in $(find ./build-oss-fuzz/DEST_DIR/ -executable -type f
>                        | grep -v slirp); do
>          grep "LLVMFuzzerTestOneInput" ${fuzzer} > /dev/null 2>&1 ||
> continue ;
>          echo Testing ${fuzzer} ... ;
> -        ASAN_OPTIONS="fast_unwind_on_malloc=0"
> -         "${fuzzer}" -runs=1000 -seed=1 || exit 1 ;
> +        "${fuzzer}" -runs=1000 -seed=1 || exit 1 ;
>        done
> +    - cd build-oss-fuzz && make check-qtest-i386 check-unit
>

As this does not use the fuzzer main entry point, what is the point of
running that?


>  build-tci:
>    <<: *native_build_job_definition
> --
> 2.18.2
>
>
>

[-- Attachment #2: Type: text/html, Size: 2225 bytes --]

Re: [PATCH] gitlab-ci.yml: Run check-qtest and check-unit at the end of the fuzzer job

[Thomas Huth]

On 31/08/2020 23.31, Philippe Mathieu-Daudé wrote:
> Hi Thomas,
> 
> Le lun. 31 août 2020 17:33, Thomas Huth <thuth@redhat.com
> <mailto:thuth@redhat.com>> a écrit :
> 
>     The fuzzer job finishes quite early, so we can run the unit tests and
>     qtests with -fsanitize=address here without extending the total test
>     time.
> 
>     Signed-off-by: Thomas Huth <thuth@redhat.com <mailto:thuth@redhat.com>>
>     ---
>      .gitlab-ci.yml | 5 +++--
>      1 file changed, 3 insertions(+), 2 deletions(-)
> 
>     diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>     index 17f1f8fad9..417fda6909 100644
>     --- a/.gitlab-ci.yml
>     +++ b/.gitlab-ci.yml
>     @@ -256,13 +256,14 @@ build-oss-fuzz:
>          - mkdir build-oss-fuzz
>          - CC="clang" CXX="clang++" CFLAGS="-fsanitize=address"
>            ./scripts/oss-fuzz/build.sh
>     +    - export ASAN_OPTIONS="fast_unwind_on_malloc=0"
>          - for fuzzer in $(find ./build-oss-fuzz/DEST_DIR/ -executable
>     -type f
>                            | grep -v slirp); do
>              grep "LLVMFuzzerTestOneInput" ${fuzzer} > /dev/null 2>&1 ||
>     continue ;
>              echo Testing ${fuzzer} ... ;
>     -        ASAN_OPTIONS="fast_unwind_on_malloc=0"
>     -         "${fuzzer}" -runs=1000 -seed=1 || exit 1 ;
>     +        "${fuzzer}" -runs=1000 -seed=1 || exit 1 ;
>            done
>     +    - cd build-oss-fuzz && make check-qtest-i386 check-unit
> 
> 
> As this does not use the fuzzer main entry point, what is the point of
> running that?

Read the friendly patch description ;-) - the idea here is that we run
the qtests and unit tests with -fsanitize=address in one of the
pipelines. We could also use a different task for this, but since the
fuzzer tests finish quite early and we already compiled the code with
-fsanitize=address here anyway, it's IMHO simply a good fit here.

 Thomas

Re: [PATCH] gitlab-ci.yml: Run check-qtest and check-unit at the end of the fuzzer job

[Thomas Huth]

On 31/08/2020 18.40, Alexander Bulekov wrote:
> On 200831 1732, Thomas Huth wrote:
>> The fuzzer job finishes quite early, so we can run the unit tests and
>> qtests with -fsanitize=address here without extending the total test time.
>>
> 
> Sounds good to me, though its too bad that this is limited to i386 and
> we aren't using the --enable-sanitizers configure argument due to the
> strange oss-fuzz build script.
> 
> Speaking of testing with sanitizers, does it make sense to have a job
> that does check-qtest with --enable-tsan, now that we have
> ThreadSanitizer support? 
> 
> Can --enable-sanitizers --enable-tsan be added to some existing job,
> without severely increasing the pipeline's exeuction time?

Good idea, I'll have a try when I've got some spare time again (probably
not this week, though, it's a pretty busy one for me).

> Reviewed-by: Alexander Bulekov <alxndr@bu.edu>

Thanks!

 Thomas

Re: [PATCH] gitlab-ci.yml: Run check-qtest and check-unit at the end of the fuzzer job

[Philippe Mathieu-Daudé]

On Tue, Sep 1, 2020 at 8:05 AM Thomas Huth <thuth@redhat.com> wrote:
>
> On 31/08/2020 23.31, Philippe Mathieu-Daudé wrote:
> > Hi Thomas,
> >
> > Le lun. 31 août 2020 17:33, Thomas Huth <thuth@redhat.com
> > <mailto:thuth@redhat.com>> a écrit :
> >
> >     The fuzzer job finishes quite early, so we can run the unit tests and
> >     qtests with -fsanitize=address here without extending the total test
> >     time.
> >
> >     Signed-off-by: Thomas Huth <thuth@redhat.com <mailto:thuth@redhat.com>>
> >     ---
> >      .gitlab-ci.yml | 5 +++--
> >      1 file changed, 3 insertions(+), 2 deletions(-)
> >
> >     diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> >     index 17f1f8fad9..417fda6909 100644
> >     --- a/.gitlab-ci.yml
> >     +++ b/.gitlab-ci.yml
> >     @@ -256,13 +256,14 @@ build-oss-fuzz:
> >          - mkdir build-oss-fuzz
> >          - CC="clang" CXX="clang++" CFLAGS="-fsanitize=address"
> >            ./scripts/oss-fuzz/build.sh
> >     +    - export ASAN_OPTIONS="fast_unwind_on_malloc=0"
> >          - for fuzzer in $(find ./build-oss-fuzz/DEST_DIR/ -executable
> >     -type f
> >                            | grep -v slirp); do
> >              grep "LLVMFuzzerTestOneInput" ${fuzzer} > /dev/null 2>&1 ||
> >     continue ;
> >              echo Testing ${fuzzer} ... ;
> >     -        ASAN_OPTIONS="fast_unwind_on_malloc=0"
> >     -         "${fuzzer}" -runs=1000 -seed=1 || exit 1 ;
> >     +        "${fuzzer}" -runs=1000 -seed=1 || exit 1 ;
> >            done
> >     +    - cd build-oss-fuzz && make check-qtest-i386 check-unit
> >
> >
> > As this does not use the fuzzer main entry point, what is the point of
> > running that?
>
> Read the friendly patch description ;-) - the idea here is that we run
> the qtests and unit tests with -fsanitize=address in one of the
> pipelines.

Sorry, the description was not that obvious to me.
Can you add a comment before the 'make check*' line?

  # Unrelated to fuzzer: run tests with -fsanitize=address

> We could also use a different task for this, but since the
> fuzzer tests finish quite early and we already compiled the code with
> -fsanitize=address here anyway, it's IMHO simply a good fit here.
>
>  Thomas
>
>

Re: [PATCH] gitlab-ci.yml: Run check-qtest and check-unit at the end of the fuzzer job

[Thomas Huth]

On 01/09/2020 08.12, Philippe Mathieu-Daudé wrote:
> On Tue, Sep 1, 2020 at 8:05 AM Thomas Huth <thuth@redhat.com> wrote:
>>
>> On 31/08/2020 23.31, Philippe Mathieu-Daudé wrote:
>>> Hi Thomas,
>>>
>>> Le lun. 31 août 2020 17:33, Thomas Huth <thuth@redhat.com
>>> <mailto:thuth@redhat.com>> a écrit :
>>>
>>>     The fuzzer job finishes quite early, so we can run the unit tests and
>>>     qtests with -fsanitize=address here without extending the total test
>>>     time.
>>>
>>>     Signed-off-by: Thomas Huth <thuth@redhat.com <mailto:thuth@redhat.com>>
>>>     ---
>>>      .gitlab-ci.yml | 5 +++--
>>>      1 file changed, 3 insertions(+), 2 deletions(-)
>>>
>>>     diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>>>     index 17f1f8fad9..417fda6909 100644
>>>     --- a/.gitlab-ci.yml
>>>     +++ b/.gitlab-ci.yml
>>>     @@ -256,13 +256,14 @@ build-oss-fuzz:
>>>          - mkdir build-oss-fuzz
>>>          - CC="clang" CXX="clang++" CFLAGS="-fsanitize=address"
>>>            ./scripts/oss-fuzz/build.sh
>>>     +    - export ASAN_OPTIONS="fast_unwind_on_malloc=0"
>>>          - for fuzzer in $(find ./build-oss-fuzz/DEST_DIR/ -executable
>>>     -type f
>>>                            | grep -v slirp); do
>>>              grep "LLVMFuzzerTestOneInput" ${fuzzer} > /dev/null 2>&1 ||
>>>     continue ;
>>>              echo Testing ${fuzzer} ... ;
>>>     -        ASAN_OPTIONS="fast_unwind_on_malloc=0"
>>>     -         "${fuzzer}" -runs=1000 -seed=1 || exit 1 ;
>>>     +        "${fuzzer}" -runs=1000 -seed=1 || exit 1 ;
>>>            done
>>>     +    - cd build-oss-fuzz && make check-qtest-i386 check-unit
>>>
>>>
>>> As this does not use the fuzzer main entry point, what is the point of
>>> running that?
>>
>> Read the friendly patch description ;-) - the idea here is that we run
>> the qtests and unit tests with -fsanitize=address in one of the
>> pipelines.
> 
> Sorry, the description was not that obvious to me.
> Can you add a comment before the 'make check*' line?
> 
>   # Unrelated to fuzzer: run tests with -fsanitize=address

Sure, will do.

 Thomas