Re: [PATCH v5 2/2] ci: Introduce travis builds for github repositories

From: Thomas Monjalon <thomas@monjalon.net>
To: Michael Santana <msantana@redhat.com>, Aaron Conole <aconole@redhat.com>
Cc: dev@dpdk.org, Bruce Richardson <bruce.richardson@intel.com>,
	Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
Subject: Re: [PATCH v5 2/2] ci: Introduce travis builds for github repositories
Date: Wed, 27 Feb 2019 14:56:28 +0100	[thread overview]
Message-ID: <2766343.Bn91GvS4A1@xps> (raw)
In-Reply-To: <20190207220114.8020-3-msantana@redhat.com>

Hi,

07/02/2019 23:01, Michael Santana:
> GitHub is a service used by developers to store repositories.  GitHub
> provides service integrations that allow 3rd party services to access
> developer repositories and perform actions.  One of these services is
> Travis-CI, a simple continuous integration platform.
> 
> This is a simple initial implementation of a travis build for the DPDK
> project.  It doesn't require any changes from individual developers to
> enable, but will allow those developers who opt-in to GitHub and the
> travis service to get automatic builds for every push they make.
> 
> Additionally, the travis service will send an email to the test-report
> list informing anyone interested in the automated build (including a
> result).
> 
> Signed-off-by: Aaron Conole <aconole@redhat.com>
> Signed-off-by: Michael Santana <msantana@redhat.com>
> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>

I think it is a good idea to provide some recipes for some well known CI.

I have some concerns about the organization of what is generic/specific
and about the automation of reports.
Detailed comments below:

> --- /dev/null
> +++ b/.ci/linux-build.sh
> +# Just used for the 'classic' configuration system (ie: make)

I am not sure about supporting the legacy system in a new CI.

> +set_conf() {
> +    echo "[BUILT WITH $2 SET TO $3]"

In order to improve readability, I suggest to document the parameters
on the first line. Example:

set_conf() { # <dir> <key> <value>

> +    c="$1/.config"
> +    shift
> +
> +    if grep -q "$1" "$c"; then
> +        sed -i "s:^$1=.*$:$1=$2:g" $c

You don't need the "g" as there is only one option per line.

> +    else
> +        echo $1=$2 >> "$c"

The else branch is useless because .config includes all options.

> +    fi
> +}
> +
> +BUILD_ARCH="x86_64-native-linuxapp-"

We could have some native Arm compilation.

> +
> +if [ "${AARCH64}" == "1" ]; then
> +    # convert the arch specifier
> +    BUILD_ARCH="arm64-armv8a-linuxapp-"
> +    AARCH64_TOOL="linaro-arm-tool"

What is this directory? It looks really specific to Travis.

> +    export PATH=$PATH:$(pwd)/${AARCH64_TOOL}/bin
> +fi
> +
> +
> +if [ "${NINJABUILD}" == "1" ]; then
> +    OPTS=""
> +
> +    DEF_LIB="static"
> +    if [ "${SHARED}" == "1" ]; then
> +        DEF_LIB="shared"
> +    fi
> +
> +    if [ "${DISABLE_KERNEL_MODULES}" == "1" ]; then
> +        OPTS="-Denable_kmods=false"
> +    fi

Isn't it possible to directly provide the meson options in travis.yml
instead of doing a translation with new option names?

> +
> +    if [ "${AARCH64}" == "1" ]; then
> +        OPTS="${OPTS} --cross-file config/arm/arm64_armv8_linuxapp_gcc"
> +    fi
> +
> +    OPTS="$OPTS --default-library=$DEF_LIB"
> +    meson build --werror -Dexamples=all ${OPTS}
> +    ninja -C build
> +else
> +    EXTRA_OPTS=""
> +
> +    make config T="${BUILD_ARCH}${CC}"
> +
> +    set_conf build CONFIG_RTE_KNI_KMOD n
> +    set_conf build CONFIG_RTE_EAL_IGB_UIO n

Why these options are fixed?

> +
> +    if dpkg --list | grep -q zlib1g ; then
> +        set_conf build CONFIG_RTE_LIBRTE_PMD_ZLIB y
> +    fi

This is Debian-specific.

> +
> +    if dpkg --list | grep -q libpcap-dev ; then
> +        set_conf build CONFIG_RTE_PORT_PCAP y
> +    fi
> +
> +    if [ "${SHARED}" == "1" ]; then
> +        set_conf build CONFIG_RTE_BUILD_SHARED_LIB y
> +    fi
> +
> +    if [ "${DISABLE_KERNEL_MODULES}" == "1" ]; then
> +        echo Unsupported kernel builds at the moment
> +    fi
> +
> +    if [ "${AARCH64}" == "1" ]; then
> +        EXTRA_OPTS="CROSS=aarch64-linux-gnu-"
> +
> +        # need to turn off these extras
> +        set_conf build CONFIG_RTE_PORT_PCAP n
> +        set_conf build CONFIG_RTE_LIBRTE_PMD_ZLIB n
> +
> +        # convert the CC/CXX variables
> +        export CC=aarch64-linux-gnu-${CC}
> +        export CXX=aarch64-linux-gnu-${CXX}
> +        export AR=aarch64-linux-gnu-ar
> +        export STRIP=aarch64-linux-gnu-strip
> +    fi
> +
> +    make all ${EXTRA_OPTS}
> +fi
> diff --git a/.ci/linux-setup.sh b/.ci/linux-setup.sh
> new file mode 100755
> index 000000000..8dfd5aa49
> --- /dev/null
> +++ b/.ci/linux-setup.sh
> @@ -0,0 +1,31 @@
> +#!/bin/bash
> +
> +python3.5 -m pip install --upgrade meson --user

Which distributions have python3.5?
It looks very specific.

> +
> +echo "AARCH64 is [ ${AARCH64} ]"
> +
> +if [ "${AARCH64}" == "1" ]; then
> +    # need to build & install libnuma

Why is it needed? linbnuma is optional.
I think this file can be dropped or renamed to
something like install-libnuma-for-cross-arm.sh

> +    # This will only be minimal support for now.
> +    AARCH64_TOOL_URL='https://releases.linaro.org/components/toolchain/binaries/latest-7/aarch64-linux-gnu/gcc-linaro-7.4.1-2019.02-x86_64_aarch64-linux-gnu.tar.xz'
> +    AARCH64_TOOL="linaro-arm-tool"
> +    NUMA_GIT_URL="https://github.com/numactl/numactl.git"
> +
> +    wget -O "${AARCH64_TOOL}.tar.xz" "${AARCH64_TOOL_URL}"
> +    tar -xf "${AARCH64_TOOL}.tar.xz"
> +    mv gcc-linaro* "${AARCH64_TOOL}"
> +    export PATH=$PATH:$(pwd)/${AARCH64_TOOL}/bin
> +    git clone "${NUMA_GIT_URL}"
> +    cd numactl
> +    git checkout v2.0.11
> +    ./autogen.sh
> +    autoconf -i
> +    mkdir numa_bin
> +    ./configure --host=aarch64-linux-gnu CC=aarch64-linux-gnu-gcc \
> +                --prefix=$(pwd)/numa_bin
> +    make install # install numa
> +    cd ..
> +    cp numactl/numa_bin/include/numa*.h "${AARCH64_TOOL}/aarch64-linux-gnu/libc/usr/include/"
> +    cp numactl/numa_bin/lib/libnuma.* "${AARCH64_TOOL}/aarch64-linux-gnu/lib64/"
> +    cp numactl/numa_bin/lib/libnuma.* "${AARCH64_TOOL}/lib/"
> +fi
> diff --git a/.travis.yml b/.travis.yml
> new file mode 100644
> index 000000000..0faa35770
> --- /dev/null
> +++ b/.travis.yml
> @@ -0,0 +1,159 @@
> +language: c
> +compiler:
> +  - gcc
> +  - clang
> +
> +os:
> +  - linux
> +
> +addons:
> +  apt:
> +    sources:
> +      - deadsnakes  #Repo for python 3.5
> +      - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
> +    packages:
> +      - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, ninja-build]
> +
> +before_install: ./.ci/${TRAVIS_OS_NAME}-setup.sh
> +
> +sudo: false
> +
> +env:
> +  - SHARED=1
> +  - DISABLE_KERNEL_MODULES=1
> +  - SHARED=1 DISABLE_KERNEL_MODULES=1
> +  - NINJABUILD=1
> +  - NINJABUILD=1 SHARED=1
> +  - NINJABUILD=1 DISABLE_KERNEL_MODULES=1
> +  - NINJABUILD=1 SHARED=1 DISABLE_KERNEL_MODULES=1
> +
> +matrix:
> +  include:
> +  - env: SHARED=1
> +    compiler: gcc
> +    addons:
> +      apt:
> +        sources:
> +          - deadsnakes  #Repo for python 3.5
> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
> +        packages:
> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
> +          - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, ninja-build]
> +  - env: DISABLE_KERNEL_MODULES=1
> +    compiler: gcc
> +    addons:
> +      apt:
> +        sources:
> +          - deadsnakes  #Repo for python 3.5
> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
> +        packages:
> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
> +          - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, ninja-build]
> +  - env: SHARED=1 DISABLE_KERNEL_MODULES=1
> +    compiler: gcc
> +    addons:
> +      apt:
> +        sources:
> +          - deadsnakes  #Repo for python 3.5
> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
> +        packages:
> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
> +          - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, ninja-build]
> +  - env: SHARED=1
> +    compiler: clang
> +    addons:
> +      apt:
> +        sources:
> +          - deadsnakes  #Repo for python 3.5
> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
> +        packages:
> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
> +          - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, ninja-build]
> +  - env: DISABLE_KERNEL_MODULES=1
> +    compiler: clang
> +    addons:
> +      apt:
> +        sources:
> +          - deadsnakes  #Repo for python 3.5
> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
> +        packages:
> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
> +          - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, ninja-build]
> +  - env: SHARED=1 DISABLE_KERNEL_MODULES=1
> +    compiler: clang
> +    addons:
> +      apt:
> +        sources:
> +          - deadsnakes  #Repo for python 3.5
> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
> +        packages:
> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
> +          - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, ninja-build]
> +  - env: AARCH64=1
> +    compiler: gcc
> +    addons:
> +      apt:
> +        sources:
> +          - deadsnakes  #Repo for python 3.5
> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
> +        packages:
> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
> +          - [libnuma-dev, linux-headers-$(uname -r), libtool, python3.5, python3-pip]
> +  - env: AARCH64=1 NINJABUILD=1
> +    compiler: gcc
> +    addons:
> +      apt:
> +        sources:
> +          - deadsnakes  #Repo for python 3.5
> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
> +        packages:
> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
> +          - [linux-headers-$(uname -r), libtool, python3.5, python3-pip, ninja-build]
> +  - env: NINJABUILD=1
> +    compiler: gcc
> +    addons:
> +      apt:
> +        sources:
> +          - deadsnakes  #Repo for python 3.5
> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
> +        packages:
> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
> +          - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, ninja-build]
> +  - env: NINJABUILD=1 SHARED=1
> +    compiler: gcc
> +    addons:
> +      apt:
> +        sources:
> +          - deadsnakes  #Repo for python 3.5
> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
> +        packages:
> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
> +          - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, ninja-build]
> +  - env: NINJABUILD=1 DISABLE_KERNEL_MODULES=1
> +    compiler: gcc
> +    addons:
> +      apt:
> +        sources:
> +          - deadsnakes  #Repo for python 3.5
> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
> +        packages:
> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
> +          - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, ninja-build]
> +  - env: NINJABUILD=1 SHARED=1 DISABLE_KERNEL_MODULES=1
> +    compiler: gcc
> +    addons:
> +      apt:
> +        sources:
> +          - deadsnakes  #Repo for python 3.5
> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
> +        packages:
> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
> +          - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, ninja-build]
> +
> +
> +script: ./.ci/${TRAVIS_OS_NAME}-build.sh
> +
> +notifications:
> +  email:
> +    recipients:
> +      - test-report@dpdk.org

The idea of this mailing list is to receive reports about
the upstream development. When doing a private development,
reports should not be sent. How can it be disabled?