Re: [PATCH v3 4/5] Added build and install scripts

[<samcacc@amazon.com>]

On 6/27/19 6:57 PM, Alexander Graf wrote:
> 
> 
> On 24.06.19 16:24, Sam Caccavale wrote:
>> install_afl.sh installs AFL locally and emits AFLPATH,
>> build.sh, and run.sh build and run respectively
>>
>> ---
>>
>> v1 -> v2:
>>   - Introduced this patch
>>
>> v2 -> v3:
>>   - Moved non-essential development scripts to a later patch
>>
>> Signed-off-by: Sam Caccavale <samcacc@amazon.de>
>> ---
>>   tools/fuzz/x86ie/scripts/afl-many       | 31 +++++++++++++++++++++++
>>   tools/fuzz/x86ie/scripts/build.sh       | 33 +++++++++++++++++++++++++
>>   tools/fuzz/x86ie/scripts/install_afl.sh | 17 +++++++++++++
>>   tools/fuzz/x86ie/scripts/run.sh         | 10 ++++++++
>>   4 files changed, 91 insertions(+)
>>   create mode 100755 tools/fuzz/x86ie/scripts/afl-many
>>   create mode 100755 tools/fuzz/x86ie/scripts/build.sh
>>   create mode 100755 tools/fuzz/x86ie/scripts/install_afl.sh
>>   create mode 100755 tools/fuzz/x86ie/scripts/run.sh
>>
>> diff --git a/tools/fuzz/x86ie/scripts/afl-many
>> b/tools/fuzz/x86ie/scripts/afl-many
>> new file mode 100755
>> index 000000000000..e55ff115a777
>> --- /dev/null
>> +++ b/tools/fuzz/x86ie/scripts/afl-many
>> @@ -0,0 +1,31 @@
>> +#!/bin/bash
>> +# SPDX-License-Identifier: GPL-2.0+
>> +# This is for running AFL over NPROC or `nproc` cores with normal AFL
>> options ex:
>> +# ulimit -Sv $[21999999999 << 10];
>> ./tools/fuzz/x86ie/scripts/afl-many -m 22000000000 -i $FUZZDIR/in -o
>> $FUZZDIR/out tools/fuzz/x86ie/afl-harness @@
>> +
>> +export AFL_NO_AFFINITY=1
>> +
>> +while [ -z "$sync_dir" ]; do
>> +  while getopts ":o:" opt; do
>> +    case "${opt}" in
>> +      o)
>> +        sync_dir="${OPTARG}"
>> +        ;;
>> +      *)
>> +        ;;
>> +    esac
>> +  done
>> +  ((OPTIND++))
>> +  [ $OPTIND -gt $# ] && break
>> +done
>> +
>> +# AFL/linux do some weird stuff with core affinity and will often run
>> +# N processes over < N virtual cores.  In order to avoid that, we
>> taskset
>> +# each process to its own core.
>> +for i in $(seq 1 $(( ${NPROC:-$(nproc)} - 1)) ); do
>> +    taskset -c "$i" ./afl-fuzz -S "slave$i" $@ >/dev/null 2>&1 &
>> +done
>> +taskset -c 0 ./afl-fuzz -M master $@ >/dev/null 2>&1 &
>> +
>> +watch -n1 "echo \"Executing '$AFLPATH/afl-fuzz $@' on
>> ${NPROC:-$(nproc)} cores.\" && $AFLPATH/afl-whatsup -s ${sync_dir}"
>> +pkill afl-fuzz
>> diff --git a/tools/fuzz/x86ie/scripts/build.sh
>> b/tools/fuzz/x86ie/scripts/build.sh
>> new file mode 100755
>> index 000000000000..032762bf56ef
>> --- /dev/null
>> +++ b/tools/fuzz/x86ie/scripts/build.sh
>> @@ -0,0 +1,33 @@
>> +#!/bin/bash
>> +# SPDX-License-Identifier: GPL-2.0+
>> +# Run from root of linux via `./tools/fuzz/x86ie/scripts/build.sh`
>> +
>> +kernel_objects="arch/x86/kvm/emulate.o arch/x86/lib/retpoline.o
>> lib/find_bit.o"
>> +
>> +disable() { sed -i -r "/\b$1\b/c\# $1" .config; }
>> +enable() { sed -i -r "/\b$1\b/c\\$1=y" .config; }
>> +
>> +make ${CC:+ "CC=$CC"} ${DEBUG:+ "DEBUG=1"} defconfig
>> +
>> +enable "CONFIG_DEBUG_INFO"
>> +enable "CONFIG_STACKPROTECTOR"
>> +
>> +yes ' ' | make ${CC:+ "CC=$CC"} ${DEBUG:+ "DEBUG=1"} $kernel_objects
>> +
>> +omit_arg () { args=$(echo "$args" | sed "s/ $1//g"); }
>> +add_arg () { args+=" $1"; }
>> +
>> +rebuild () {
>> +  args="$(head -1 $(dirname $1)/.$(basename $1).cmd | sed -e 's/.*:=
>> //g')"
>> +  omit_arg "-mcmodel=kernel"
>> +  omit_arg "-mpreferred-stack-boundary=3"
>> +  add_arg "-fsanitize=address"
>> +  echo -e "Rebuilding $1 with \n$args"
>> +  eval "$args"
>> +}
>> +
>> +for object in $kernel_objects; do
>> +  rebuild $object
>> +done
>> +
>> +make ${CC:+ "CC=$CC"} ${DEBUG:+ "DEBUG=1"} tools/fuzz
>> diff --git a/tools/fuzz/x86ie/scripts/install_afl.sh
>> b/tools/fuzz/x86ie/scripts/install_afl.sh
>> new file mode 100755
>> index 000000000000..3bdbdf2a040b
>> --- /dev/null
>> +++ b/tools/fuzz/x86ie/scripts/install_afl.sh
>> @@ -0,0 +1,17 @@
>> +#!/bin/bash
>> +# SPDX-License-Identifier: GPL-2.0+
>> +# Can be run where ever, but usually run from linux root:
>> +# `source ./tools/fuzz/x86ie/scripts/install_afl.sh`
>> +# (must be sourced to get the AFLPATH envvar, otherwise set manually)
>> +
>> +wget http://lcamtuf.coredump.cx/afl/releases/afl-latest.tgz
>> +mkdir -p afl
>> +tar xzf afl-latest.tgz -C afl --strip-components 1
>> +
>> +pushd afl
>> +set AFL_USE_ASAN
>> +make clean all
>> +export AFLPATH="$(pwd)"
>> +popd
>> +
>> +sudo bash -c "echo core >/proc/sys/kernel/core_pattern"
> 
> What is this? :)
> 
> Surely if it's important to generate core dumps, it's not only important
> during installation, no?

Yep... missed this.  I'll move it to run.sh right before alf-many is
invoked.  It would be nice to not have to sudo but it seems the only
alternative is an envvar AFL_I_DONT_CARE_ABOUT_MISSING_CRASHES which
just ignores AFL's warning if your system isn't going to produce core
dumps (which will cause AFL to miss some crashes, as the name suggests).

Thanks for all the feedback thusfar,
Sam

> 
> Alex
> 
>> diff --git a/tools/fuzz/x86ie/scripts/run.sh
>> b/tools/fuzz/x86ie/scripts/run.sh
>> new file mode 100755
>> index 000000000000..0571cd524c01
>> --- /dev/null
>> +++ b/tools/fuzz/x86ie/scripts/run.sh
>> @@ -0,0 +1,10 @@
>> +#!/bin/bash
>> +# SPDX-License-Identifier: GPL-2.0+
>> +
>> +FUZZDIR="${FUZZDIR:-$(pwd)/fuzz}"
>> +
>> +mkdir -p $FUZZDIR/in
>> +cp tools/fuzz/x86ie/rand_sample.bin $FUZZDIR/in
>> +mkdir -p $FUZZDIR/out
>> +
>> +screen bash -c "ulimit -Sv $[21999999999 << 10];
>> ./tools/fuzz/x86ie/scripts/afl-many -m 22000000000 -i $FUZZDIR/in -o
>> $FUZZDIR/out tools/fuzz/x86ie/afl-harness @@"
>>