From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: From: Date: Fri, 4 Jun 2021 19:13:56 +0530 Message-ID: <20210604134356.1607-1-venkata.pyla@toshiba-tsip.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Subject: [Fuego] Use common script to install fuego on host or docker system List-Id: Mailing list for the Fuego test framework List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Tim.Bird@sony.com, tbird20d@yahoo.com Cc: fuego@lists.linuxfoundation.org From: venkata pyla Modified the install scripts to use common scripts for both types of installation, this way it will help to make sure both the environments are same and if any future modification will have single place of change. Now, for installing fuego on docker system the 'install.sh' script will use common script in Dockefile to install fuego related stuff. sameway for installing fuego on host system it can use the same common script to install fuego related stuff. Below are the common scripts that are used in both the installation types for each debian distribution * install-stretch.sh * install-buster.sh so finally, To install fuego on docker system use the below script with options $ ./install.sh image_name e.g: ./install.sh debian_stretch 8090 debian stretch or ./install.sh debian_buster 8090 debian buster or ./install.sh --nojenkins debian_buster_nojenkins 8090 debian buster To install fuego on host system use the below script with options $ ./install-fuego.sh <-d distribution> <-s suite> <-p port> e.g: ./install-fuego.sh -d debian -s stretch -p 8090 or ./install-fuego.sh -d debian -s buster -p 8090 or ./install-fuego.sh -d debian -s buster -p 8090 --nojenkins Signed-off-by: venkata pyla --- Dockerfile | 303 ++--------------- Dockerfile.nojenkins | 151 --------- README | 29 +- entrypoint.sh | 7 + fuego-host-scripts/docker-build-image.sh | 11 +- install-buster.sh | 294 ++++++++++++++++ install-debian.sh | 410 ++++++++++++++--------- install-fuego.sh | 109 ++++++ install-stretch.sh | 288 ++++++++++++++++ install.sh | 12 +- uninstall-fuego.sh | 47 +++ 11 files changed, 1057 insertions(+), 604 deletions(-) delete mode 100644 Dockerfile.nojenkins create mode 100755 entrypoint.sh create mode 100755 install-buster.sh create mode 100755 install-fuego.sh create mode 100755 install-stretch.sh create mode 100644 uninstall-fuego.sh diff --git a/Dockerfile b/Dockerfile index 7f63798..ac57cb2 100644 --- a/Dockerfile +++ b/Dockerfile @@ -2,9 +2,11 @@ # WARNING: this Dockerfile assumes that the container will be created with # several bind mounts (see docker-create-container.sh) # ============================================================================== -# FIXTHIS: build this as an extension of the nonjenkins image -FROM debian:stretch-slim +ARG SUITE=stretch +ARG DIST=debian + +FROM ${DIST}:${SUITE}-slim MAINTAINER tim.bird@sony.com # ============================================================================== @@ -16,299 +18,32 @@ ENV http_proxy ${HTTP_PROXY} ENV https_proxy ${HTTP_PROXY} # ============================================================================== -# Prepare basic image +# Install fuego with the script # ============================================================================== -ARG DEBIAN_FRONTEND=noninteractive - -WORKDIR / -RUN echo deb http://deb.debian.org/debian stretch main non-free > /etc/apt/sources.list -RUN echo deb http://security.debian.org/debian-security stretch/updates main >> /etc/apt/sources.list -RUN if [ -n "$HTTP_PROXY" ]; then echo 'Acquire::http::proxy "'$HTTP_PROXY'";' > /etc/apt/apt.conf.d/80proxy; fi - -# Fuego python dependencies -# - python-lxml: ftc, loggen -# - python-simplejson: ftc -# - python-yaml: ftc -# - python-openpyxl: ftc (also LTP) -# - python-requests: ftc (also fuego_release_test) -# - python-reportlab: ftc -# - python-parsedatetime: ftc -# - python-pexpect: ssh_exec (part of ttc) -# - python-pip: to install filelock, flake8 -# - filelock: parser -RUN mkdir -p /usr/share/man/man1 -RUN apt-get update -q=2 && apt-get -q=2 -V --no-install-recommends install \ - python-lxml python-simplejson python-yaml python-openpyxl \ - python-requests python-reportlab python-parsedatetime \ - python-pexpect python-pip python-setuptools python-wheel -RUN pip install filelock - -# Fuego command dependencies -RUN apt-get -q=2 -V --no-install-recommends install \ - git sshpass openssh-client sudo net-tools wget curl lava-tool \ - bash-completion iproute2 - -# Default SDK for testing locally or on an x86 board -RUN apt-get -q=2 -V --no-install-recommends install \ - build-essential cmake bison flex automake kmod libtool \ - libelf-dev libssl-dev libsdl1.2-dev libcairo2-dev libxmu-dev \ - libxmuu-dev libglib2.0-dev libaio-dev pkg-config rsync u-boot-tools - -# Default test host dependencies -# - iperf iperf3 netperf: used as servers on the host -# - bzip2 bc: used for local tests by Functional.bzip2/bc -# - python-matplotlib: Benchmark.iperf3 parser -# - python-xmltodict: AGL tests -# - flake8: Functional.fuego_lint -# - netpipe-tcp - used by Benchmark.netpipe (provides the netpipe server) -# - iputils-ping - for /bin/ping command, used by some tests -# FIXTHIS: install dependencies dynamically on the tests that need them -RUN apt-get -q=2 -V --no-install-recommends install \ - iperf iperf3 netperf bzip2 bc python-matplotlib python-xmltodict \ - netpipe-tcp iputils-ping -RUN pip install flake8 - -# miscelaneous packages: -# python-serial - used by serio -# diffstat and vim - used by Tim -# time - useful for timing command duration -RUN apt-get -q=2 -V --no-install-recommends install \ - python-serial \ - diffstat \ - vim \ - time - -# install packages used by NuttX SDK -RUN apt-get -q=2 -V --no-install-recommends install \ - genromfs - -# FIXTHIS: determine if these tools are really necessary -#RUN apt-get -q=2 -V --no-install-recommends install \ -# apt-utils python-paramiko \ -# xmlstarlet rsync \ -# inotify-tools gettext netpipe-tcp \ -# at minicom lzop bsdmainutils \ -# mc netcat openssh-server - -# bsdmainutils provides hexdump, which is needed for netperf test -RUN apt-get -q=2 -V --no-install-recommends install \ - bsdmainutils - -RUN /bin/bash -c 'echo "dash dash/sh boolean false" | debconf-set-selections ; dpkg-reconfigure dash' -RUN if [ -n "$HTTP_PROXY" ]; then echo "use_proxy = on" >> /etc/wgetrc; fi -RUN if [ -n "$HTTP_PROXY" ]; then echo -e "http_proxy=$HTTP_PROXY\nhttps_proxy=$HTTP_PROXY" >> /etc/environment; fi - -# ============================================================================== -# Install Jenkins with the same UID/GID as the host user -# ============================================================================== +RUN mkdir /fuego +COPY . /fuego/ +WORKDIR /fuego -ARG user=jenkins -ARG group=jenkins -ARG uid=1000 -ARG gid=${uid} ARG JENKINS_PORT=8090 -# see https://updates.jenkins.io/download/war/ for latest Jenkins versions -#ARG JENKINS_VERSION=2.164.2 -#ARG JENKINS_SHA=4536f43f61b1fca6c58bd91040fa09304eea96ab -#ARG JENKINS_VERSION=2.235.3 -#ARG JENKINS_SHA=89a37ed1b3c8c14ff94cc66c3f2254f281b60794 -ARG JENKINS_VERSION=2.249.3 -ARG JENKINS_SHA=534014c007edbb533a1833fe6f2dc115faf3faa2 - -ARG JENKINS_URL=https://pkg.jenkins.io/debian-stable/binary/jenkins_${JENKINS_VERSION}_all.deb -ARG JENKINS_UC=https://updates.jenkins.io -ARG REF=/var/lib/jenkins/plugins +ARG NO_JENKINS= +ARG SUITE=stretch +ARG DIST=debian +ARG uid= +ARG gid= ENV JENKINS_HOME=/var/lib/jenkins ENV JENKINS_PORT=$JENKINS_PORT +RUN ./install-$DIST.sh -s $SUITE -p $JENKINS_PORT -u $uid -g $gid $NO_JENKINS -# Jenkins dependencies -RUN apt-get -q=2 -V --no-install-recommends install \ - default-jdk daemon psmisc adduser procps unzip -RUN pip install python-jenkins==1.4.0 - -RUN echo -e "JENKINS_PORT=$JENKINS_PORT" >> /etc/environment -RUN getent group ${gid} >/dev/null || groupadd -g ${gid} ${group} -RUN useradd -l -m -d "${JENKINS_HOME}" -u ${uid} -g ${gid} -G sudo -s /bin/bash ${user} -RUN wget -nv ${JENKINS_URL} -RUN echo "${JENKINS_SHA} jenkins_${JENKINS_VERSION}_all.deb" | sha1sum -c - -# allow Jenkins to start and install plugins, as part of dpkg installation -RUN printf "#!/bin/sh\nexit 0" > /usr/sbin/policy-rc.d -RUN dpkg -i jenkins_${JENKINS_VERSION}_all.deb -RUN rm jenkins_${JENKINS_VERSION}_all.deb - -# ============================================================================== -# get ttc script and helpers -# ============================================================================== -RUN git clone https://github.com/tbird20d/ttc.git /usr/local/src/ttc -RUN /usr/local/src/ttc/install.sh /usr/local/bin -# as of ttc version 2.0.2, this should not be needed -#RUN perl -p -i -e "s#config_dir = \"/etc\"#config_dir = \"/fuego-ro/conf\"#" /usr/local/bin/ttc - -# ============================================================================== -# get ebf script and helpers -# ============================================================================== -RUN apt-get -q=2 -V --no-install-recommends install jq -RUN git clone https://github.com/TimesysGit/board-farm-rest-api.git /usr/local/src/board-farm-rest-api ; \ - cp /usr/local/src/board-farm-rest-api/cli/ebf /usr/local/bin/ebf ; \ - chmod a+x /usr/local/bin/ebf - -# ============================================================================== -# Serial Config -# ============================================================================== - -RUN git clone https://github.com/frowand/serio.git /usr/local/src/serio -COPY frontend-install/0001-Fix-host-parsing-for-serial-device-with-in-name.patch \ - frontend-install/0002-Output-data-from-port-during-command-execution.patch \ - /tmp/ -RUN /bin/bash -c 'patch -d /usr/local/src/serio -p1 > /root/.bashrc - -# ============================================================================== -# Lava -# ============================================================================== - -RUN ln -s /fuego-ro/scripts/fuego-lava-target-setup /usr/local/bin -RUN ln -s /fuego-ro/scripts/fuego-lava-target-teardown /usr/local/bin +# Delete the files that are copied for installation +RUN rm -rf /fuego +WORKDIR / # ============================================================================== # Setup startup command # ============================================================================== - # FIXTHIS: when running multiple Fuego containers, or if the host is already # running the netperf server, netperf will complain because the port is taken -ENTRYPOINT service jenkins start && service netperf start && iperf3 -V -s -D -f M && /bin/bash +ENTRYPOINT entrypoint.sh $NO_JENKINS && /bin/bash diff --git a/Dockerfile.nojenkins b/Dockerfile.nojenkins deleted file mode 100644 index 56de98a..0000000 --- a/Dockerfile.nojenkins +++ /dev/null @@ -1,151 +0,0 @@ -# ============================================================================== -# WARNING: this Dockerfile assumes that the container will be created with -# several bind mounts (see docker-create-container.sh) -# ============================================================================== -# Usage: -# host$ vi fuego-ro/conf/fuego.conf -# jenkins_enabled=0 -# host$ ./fuego-host-scripts/docker-build-image.sh fuego-nojenkins 8090 Dockerfile.nojenkins -# host$ ./fuego-host-scripts/docker-create-container.sh fuego-nojenkins fuego-nojenkins-container -# host$ ./fuego-host-scripts/docker-start-container.sh fuego-nojenkins-container -# docker# ftc list-boards -# docker, bbb, ... -# docker# ftc list-tests -# Functional.hello_world, ... -# docker# ftc run-test -b docker -t Functional.hello_world -# log_compare: pattern 'SUCCESS' found 1 times (expected greater or equal than 1) - -FROM debian:stretch-slim -MAINTAINER tim.bird@sony.com - -# ============================================================================== -# Proxy variables -# ============================================================================== - -ARG HTTP_PROXY -ENV http_proxy ${HTTP_PROXY} -ENV https_proxy ${HTTP_PROXY} - -# ============================================================================== -# Prepare basic image -# ============================================================================== - -ARG DEBIAN_FRONTEND=noninteractive - -WORKDIR / -RUN echo deb http://deb.debian.org/debian stretch main non-free > /etc/apt/sources.list -RUN echo deb http://security.debian.org/debian-security stretch/updates main >> /etc/apt/sources.list -RUN if [ -n "$HTTP_PROXY" ]; then echo 'Acquire::http::proxy "'$HTTP_PROXY'";' > /etc/apt/apt.conf.d/80proxy; fi - -# Fuego python dependencies -# - python-lxml: ftc, loggen -# - python-simplejson: ftc -# - python-yaml: ftc -# - python-openpyxl: ftc (also LTP) -# - python-requests: ftc (also fuego_release_test) -# - python-reportlab: ftc -# - python-parsedatetime: ftc -# - python-pexpect: ssh_exec (part of ttc) -# - python-pip: to install filelock, flake8 -# - filelock: parser -RUN mkdir -p /usr/share/man/man1 -RUN apt-get update -q=2 && apt-get -q=2 -V --no-install-recommends install \ - python-lxml python-simplejson python-yaml python-openpyxl \ - python-requests python-reportlab python-parsedatetime \ - python-pexpect python-pip python-setuptools python-wheel -RUN pip install filelock - -# Fuego command dependencies -RUN apt-get -q=2 -V --no-install-recommends install \ - git sshpass openssh-client sudo net-tools wget curl lava-tool \ - bash-completion iproute2 - -# Default SDK for testing locally or on an x86 board -RUN apt-get -q=2 -V --no-install-recommends install \ - build-essential cmake bison flex automake kmod libtool \ - libelf-dev libssl-dev libsdl1.2-dev libcairo2-dev libxmu-dev \ - libxmuu-dev libglib2.0-dev libaio-dev pkg-config rsync u-boot-tools - -# Default test host dependencies -# - iperf iperf3 netperf: used as servers on the host -# - bzip2 bc: used for local tests by Functional.bzip2/bc -# - python-matplotlib: Benchmark.iperf3 parser -# - python-xmltodict: AGL tests -# - flake8: Functional.fuego_lint -# - netpipe-tcp - used by Benchmark.netpipe (provides the netpipe server) -# - iputils-ping - for /bin/ping command, used by some tests -# FIXTHIS: install dependencies dynamically on the tests that need them -RUN apt-get -q=2 -V --no-install-recommends install \ - iperf iperf3 netperf bzip2 bc python-matplotlib python-xmltodict \ - netpipe-tcp iputils-ping -RUN pip install flake8 - -# miscelaneous packages: -# python-serial - used by serio -# diffstat and vim - used by Tim -# time - useful for timing command duration -RUN apt-get -q=2 -V --no-install-recommends install \ - python-serial \ - diffstat \ - vim \ - time - -# FIXTHIS: determine if these tools are really necessary -# RUN apt-get -q=2 -V --no-install-recommends install \ -# apt-utils python-paramiko \ -# xmlstarlet rsync \ -# inotify-tools gettext netpipe-tcp \ -# at minicom lzop bsdmainutils \ -# mc netcat openssh-server - -# bsdmainutils provides hexdump, which is needed for netperf test -RUN apt-get -q=2 -V --no-install-recommends install \ - bsdmainutils - -RUN /bin/bash -c 'echo "dash dash/sh boolean false" | debconf-set-selections ; dpkg-reconfigure dash' -RUN if [ -n "$HTTP_PROXY" ]; then echo "use_proxy = on" >> /etc/wgetrc; fi -RUN if [ -n "$HTTP_PROXY" ]; then echo -e "http_proxy=$HTTP_PROXY\nhttps_proxy=$HTTP_PROXY" >> /etc/environment; fi - -# ============================================================================== -# get ttc script and helpers -# ============================================================================== -RUN git clone https://github.com/tbird20d/ttc.git /usr/local/src/ttc -RUN /usr/local/src/ttc/install.sh /usr/local/bin -# as of ttc version 2.0.2, this should not be needed -#RUN perl -p -i -e "s#config_dir = \"/etc\"#config_dir = \"/fuego-ro/conf\"#" /usr/local/bin/ttc - -# ============================================================================== -# Serial Config -# ============================================================================== - -RUN /bin/bash -c 'git clone https://github.com/frowand/serio.git /usr/local/src/serio ; cp /usr/local/src/serio/serio /usr/local/bin/ ; ln -s /usr/local/bin/serio /usr/local/bin/sercp ; ln -s /usr/local/bin/serio /usr/local/bin/sersh' - -RUN /bin/bash -c 'git clone https://github.com/tbird20d/serlogin.git /usr/local/src/serlogin ; cp /usr/local/src/serlogin/serlogin /usr/local/bin' - -# ============================================================================== -# fserver -# ============================================================================== - -RUN /bin/bash -c 'git clone https://github.com/tbird20d/fserver.git /usr/local/lib/fserver ; ln -s /usr/local/lib/fserver/start_local_bg_server /usr/local/bin/start_local_bg_server' - -# ============================================================================== -# ftc post installation -# ============================================================================== - -RUN ln -s /fuego-core/scripts/ftc /usr/local/bin/ -COPY fuego-core/scripts/ftc_completion.sh /etc/bash_completion.d/ftc -RUN echo ". /etc/bash_completion" >> /root/.bashrc - -# ============================================================================== -# Lava -# ============================================================================== - -RUN ln -s /fuego-ro/scripts/fuego-lava-target-setup /usr/local/bin -RUN ln -s /fuego-ro/scripts/fuego-lava-target-teardown /usr/local/bin - -# ============================================================================== -# Setup startup command -# ============================================================================== - -# FIXTHIS: when running multiple Fuego containers netperf will complain because the port is taken -ENTRYPOINT service netperf start && iperf3 -V -s -D -f M && /bin/bash diff --git a/README b/README index 882d84e..ce42e1e 100644 --- a/README +++ b/README @@ -44,6 +44,29 @@ You can change the name of the docker image and the port used by Jenkins by passing the corresponding parameters to install.sh. ./install.sh fuego-8082 8082 +To run fuego on different distribution versions like Debian(stretch, buster, etc.) + pass appropriate arguments to the install.sh script + ./install.sh + +By default install.sh script will install fuego on Debian stretch version + to install fuego on Debian buster version pass below options + ./install.sh fuego_buster 8090 debian buster + +[Note] It is currently verified on Debian versions (stretch and buster), +and yet to provide install scripts for other distributions + +Installing Fuego to a host system +------------------------------------------------------------------------------- +Run install-fuego.sh to install fuego on host system, by default it assume the +host system is Debian stretch + ./install-fuego.sh + + to install on Debian buster version + ./install-fuego.sh -d debian -s buster -p 8090 + + to install without jenkins pass --nojenkins option to the script + ./install-fuego.sh --nojenkins + Running ------------------------------------------------------------------------------- To start your Fuego container, issue the following command top directory: @@ -85,9 +108,9 @@ Updating the Jenkins version ------------------------------------------------------------------------------- Modify the version and checksum - $ vi Dockerfile - ARG JENKINS_VERSION=2.32.1 - ARG JENKINS_SHA=3cbe9b6d4d82c975642ea4409e21e087 + $ vi install-debian.sh + JENKINS_VERSION=2.32.1 + JENKINS_SHA=3cbe9b6d4d82c975642ea4409e21e087 Re-build the flot plugin $ sudo apt-get install maven diff --git a/entrypoint.sh b/entrypoint.sh new file mode 100755 index 0000000..6eac895 --- /dev/null +++ b/entrypoint.sh @@ -0,0 +1,7 @@ +#!/bin/bash + +[ "$1" != "--nojenkins" ] && service jenkins start + +#TODO: netperf not present in buster, remove this condition once alternative found +[ -f /etc/init.d/netperf ] && service netperf start +iperf3 -V -s -D -f M diff --git a/fuego-host-scripts/docker-build-image.sh b/fuego-host-scripts/docker-build-image.sh index 5823357..6889c47 100755 --- a/fuego-host-scripts/docker-build-image.sh +++ b/fuego-host-scripts/docker-build-image.sh @@ -2,9 +2,10 @@ # $1 - name for the docker image (default: fuego) # $2 - port for jenkins (default: 8090) # $3 - Dockerfile or Dockerfile.nojenkins +# $4 - Debian code name on which to install (default: stretch) # # Example: -# ./fuego-host-scripts/docker-build-image.sh --no-cache myfuegoimg 8082 Dockerfile.nojenkins +# ./fuego-host-scripts/docker-build-image.sh --no-cache myfuegoimg 8082 debian stretch --nojenkins # if [ "$1" = "--no-cache" ]; then NO_CACHE=--no-cache @@ -13,7 +14,10 @@ fi DOCKERIMAGE=${1:-fuego} JENKINS_PORT=${2:-8090} -DOCKERFILE=${3:-Dockerfile} +DISTRIBUTION=${3:-debian} +SUITE=${4:-stretch} +NO_JENKINS=${5:""} +DOCKERFILE="Dockerfile" if [ "$(id -u)" == "0" ]; then JENKINS_UID=$(id -u $SUDO_USER) @@ -27,4 +31,7 @@ echo "Using Port $JENKINS_PORT" sudo docker build ${NO_CACHE} -t ${DOCKERIMAGE} --build-arg HTTP_PROXY=$http_proxy \ --build-arg uid=$JENKINS_UID --build-arg gid=$JENKINS_GID \ + --build-arg DIST=$DISTRIBUTION \ + --build-arg SUITE=$SUITE \ + --build-arg NO_JENKINS=$NO_JENKINS \ --build-arg JENKINS_PORT=$JENKINS_PORT -f $DOCKERFILE . diff --git a/install-buster.sh b/install-buster.sh new file mode 100755 index 0000000..f4ce65b --- /dev/null +++ b/install-buster.sh @@ -0,0 +1,294 @@ +#!/bin/bash +# 2019 (c) Toshiba corp. +# +# Usage: +# $ sudo ./install-buster.sh [--help|-h] [--nojenkins] [] +# +usage(){ + +cat <] + +Install fuego on the host Debian filesystem. + +options: + --help Show usage help + --nojenkins Install Fuego without Jenkins +HERE +} + +if [[ $EUID -ne 0 ]]; then + echo "Sorry, you need root permissions" + exit 1 +fi + +nojenkins=0 + +POSITIONAL=() +while [[ $# -gt 0 ]] ; do + case $1 in + --nojenkins) + nojenkins=1 + shift + ;; + --help | -h) + usage + exit 0 + ;; + *) + POSITIONAL+=("$1") # save argument for later + shift + esac +done +set -- "${POSITIONAL[@]}" # restore positional arguments + +port=${2:-8090} + +# ============================================================================== +# Install Fuego dependencies +# ============================================================================== + +mkdir -p /usr/share/man/man1 + +# netperf is in non-free +echo deb http://deb.debian.org/debian buster main non-free > /etc/apt/sources.list +echo deb http://security.debian.org/debian-security buster/updates main >> /etc/apt/sources.list + +apt-get update -q=2 + +# Fuego python dependencies +apt-get -q=2 -V --no-install-recommends install \ + python-lxml python-simplejson python-yaml python-openpyxl \ + python-requests python-reportlab python-parsedatetime \ + python-pexpect python-pip python-setuptools python-wheel +pip install filelock + +# Fuego command dependencies +apt-get -q=2 -V --no-install-recommends install \ + git sshpass openssh-client sudo net-tools wget curl lava-tool \ + bash-completion iproute2 + +# Default SDK for testing locally or on an x86 board +apt-get -q=2 -V --no-install-recommends install \ + build-essential cmake bison flex kmod automake libtool \ + libelf-dev libssl-dev libsdl1.2-dev libcairo2-dev libxmu-dev \ + libxmuu-dev libglib2.0-dev libaio-dev pkg-config rsync u-boot-tools + +# Default test host dependencies +apt-get -q=2 -V --no-install-recommends install \ + iperf iperf3 bzip2 bc python-matplotlib python-xmltodict \ + netpipe-tcp iputils-ping +pip install flake8 + +# miscelaneous packages: +# python-serial - used by serio +# diffstat and vim - used by Tim +# time - useful for timing command duration +apt-get -q=2 -V --no-install-recommends install \ + python-serial \ + diffstat \ + vim \ + time + +# install packages used by NuttX SDK +apt-get -q=2 -V --no-install-recommends install \ + genromfs + +# bsdmainutils provides hexdump, which is needed for netperf test +#apt-get -q=2 -V --no-install-recommends install \ +# bsdmainutils + +echo "dash dash/sh boolean false" | debconf-set-selections +DEBIAN_FRONTEND=noninteractive dpkg-reconfigure dash + +# TODO: check if proxy settings can set outside docker container +if [ -n "$http_proxy" ]; then + sed -i -e 's/#use_proxy = on/use_proxy = on/g' /etc/wgetrc + echo -e "http_proxy=$http_proxy\nhttps_proxy=$https_proxy" >> /etc/environment +fi + +# ============================================================================== +# Install Jenkins +# ============================================================================== + +if [ $nojenkins -eq 0 ]; then + #JENKINS_VERSION=2.164.2 + #JENKINS_SHA=4536f43f61b1fca6c58bd91040fa09304eea96ab + #JENKINS_VERSION=2.249.3 + #JENKINS_SHA=534014c007edbb533a1833fe6f2dc115faf3faa2 + JENKINS_VERSION=2.164.3 + JENKINS_SHA=27f14f0ccca207660765205317295a5c29f4dd53 + JENKINS_URL=https://pkg.jenkins.io/debian-stable/binary/jenkins_${JENKINS_VERSION}_all.deb + export JENKINS_UC=https://updates.jenkins.io + REF=/var/lib/jenkins/plugins + JENKINS_HOME=/var/lib/jenkins + JENKINS_PORT=$port + + # Jenkins dependencies + apt-get -q=2 -V --no-install-recommends install \ + default-jdk daemon psmisc adduser procps unzip + pip install python-jenkins==1.4.0 + + echo -e "JENKINS_PORT=$JENKINS_PORT" >> /etc/environment + export JENKINS_PORT=$port + + groupadd jenkins + useradd -l -m -d "${JENKINS_HOME}" -g jenkins -G sudo -s /bin/bash jenkins + + wget -nv ${JENKINS_URL} + echo "${JENKINS_SHA} jenkins_${JENKINS_VERSION}_all.deb" | sha1sum -c - + + # allow Jenkins to start and install plugins, as part of dpkg installation + printf "#!/bin/sh\nexit 0" > /usr/sbin/policy-rc.d + dpkg -i jenkins_${JENKINS_VERSION}_all.deb + rm jenkins_${JENKINS_VERSION}_all.deb + + # fix for Jenkins Java version check error + JAVA_VERSION_CHECK="\$\(\$JAVA -version 2\>\&1 \| sed -n \';s/.* version \"\\\(.*\\\)\\\.\\\(.*\\\)\\\..\*\\\..\*\".\*\/\\\1\\\2\/p;\'\)" + sed -i -e "s#^JAVA_VERSION=.*#JAVA_VERSION\=\"${JAVA_VERSION_CHECK}\"#g" /etc/init.d/jenkins + + source /etc/default/jenkins && \ + JENKINS_ARGS="$JENKINS_ARGS --prefix=/fuego" && \ + sed -i -e "s#JENKINS_ARGS.*#JENKINS_ARGS\=\"${JENKINS_ARGS}\"#g" /etc/default/jenkins + + source /etc/default/jenkins && \ + JAVA_ARGS="$JAVA_ARGS -Djenkins.install.runSetupWizard=false -Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true" && \ + if [ -n "$http_proxy" ]; then \ + PROXYSERVER=$(echo $http_proxy | sed -E 's/^http://' | sed -E 's/\///g' | sed -E 's/(.*):(.*)/\1/') && \ + PROXYPORT=$(echo $http_proxy | sed -E 's/^http://' | sed -E 's/\///g' | sed -E 's/(.*):(.*)/\2/') && \ + JAVA_ARGS="$JAVA_ARGS -Dhttp.proxyHost="${PROXYSERVER}" -Dhttp.proxyPort="${PROXYPORT}" -Dhttps.proxyHost="${PROXYSERVER}" -Dhttps.proxyPort="${PROXYPORT}; \ + fi && \ + sed -i -e "s#^JAVA_ARGS.*#JAVA_ARGS\=\"${JAVA_ARGS}\"#g" /etc/default/jenkins; + + sed -i -e "s#8080#$JENKINS_PORT#g" /etc/default/jenkins + + # set up Jenkins plugins + cp frontend-install/install-plugins.sh \ + frontend-install/jenkins-support \ + frontend-install/clitest \ + /usr/local/bin/ + + mkdir -p $JENKINS_HOME/userContent/docs + cp frontend-install/config.xml $JENKINS_HOME/config.xml + ln -s /fuego-rw/logs $JENKINS_HOME/userContent/fuego.logs + cp docs/fuego-docs.pdf $JENKINS_HOME/userContent/docs/fuego-docs.pdf + cp frontend-install/jenkins.model.JenkinsLocationConfiguration.xml $JENKINS_HOME/jenkins.model.JenkinsLocationConfiguration.xml + sed -i -e "s#8080#$JENKINS_PORT#g" $JENKINS_HOME/jenkins.model.JenkinsLocationConfiguration.xml + chown -R jenkins:jenkins $JENKINS_HOME/ + + # start and stop jenkins to pre-populate some settings + service jenkins start && \ + sleep 30 && \ + service jenkins stop + + # install other plugins from Jenkins update center + # NOTE: not sure all of these are needed, but keep list + # compatible with 1.2.1 release for now + #/usr/local/bin/install-plugins.sh \ + # ant:1.9 \ + # antisamy-markup-formatter:1.5 \ + # bouncycastle-api:2.17 \ + # command-launcher:1.3 \ + # description-setter:1.10 \ + # display-url-api:2.3.1 \ + # external-monitor-job:1.7 \ + # greenballs:1.15 \ + # icon-shim:2.0.3 \ + # javadoc:1.5 \ + # jdk-tool:1.2 \ + # junit:1.27 \ + # ldap:1.20 \ + # mailer:1.23 \ + # matrix-auth:2.3 \ + # matrix-project:1.14 \ + # pam-auth:1.4 \ + # pegdown-formatter:1.3 \ + # structs:1.17 \ + # windows-slaves:1.4 + + # install plugins for latest LTS Jenkins (currently 2.249.3) + /usr/local/bin/install-plugins.sh \ + script-security \ + structs \ + workflow-step-api \ + workflow-api \ + junit \ + scm-api \ + ant \ + antisamy-markup-formatter \ + bouncycastle-api \ + command-launcher \ + description-setter \ + display-url-api \ + external-monitor-job \ + greenballs \ + icon-shim \ + javadoc \ + jdk-tool \ + mailer \ + matrix-auth \ + matrix-project \ + pam-auth \ + pegdown-formatter \ + windows-slaves + + cp frontend-install/plugins/flot-plotter-plugin/flot.hpi $JENKINS_HOME/plugins/flot.jpi + + # make the mod.js symlink well after flot is installed + service jenkins start && sleep 30 && \ + rm $JENKINS_HOME/plugins/flot/flot/mod.js && \ + ln -s /fuego-core/scripts/mod.js $JENKINS_HOME/plugins/flot/flot/mod.js + + chown -R jenkins:jenkins $JENKINS_HOME/ +fi + +# ============================================================================== +# get ttc script and helpers +# ============================================================================== +git clone https://github.com/tbird20d/ttc.git /usr/local/src/ttc +/usr/local/src/ttc/install.sh /usr/local/bin +perl -p -i -e "s#config_dir = \"/etc\"#config_dir = \"/fuego-ro/conf\"#" /usr/local/bin/ttc + +# ============================================================================== +# Serial Config +# ============================================================================== +if [ $nojenkins -eq 0 ]; then + /bin/bash -c 'git clone https://github.com/frowand/serio.git /usr/local/src/serio ; chown -R jenkins /usr/local/src/serio ; cp /usr/local/src/serio/serio /usr/local/bin/ ; ln -s /usr/local/bin/serio /usr/local/bin/sercp ; ln -s /usr/local/bin/serio /usr/local/bin/sersh' + /bin/bash -c 'git clone https://github.com/tbird20d/serlogin.git /usr/local/src/serlogin ; chown -R jenkins /usr/local/src/serlogin ; cp /usr/local/src/serlogin/serlogin /usr/local/bin' +else + /bin/bash -c 'git clone https://github.com/frowand/serio.git /usr/local/src/serio ; cp /usr/local/src/serio/serio /usr/local/bin/ ; ln -s /usr/local/bin/serio /usr/local/bin/sercp ; ln -s /usr/local/bin/serio /usr/local/bin/sersh' + /bin/bash -c 'git clone https://github.com/tbird20d/serlogin.git /usr/local/src/serlogin ; cp /usr/local/src/serlogin/serlogin /usr/local/bin' +fi +cp frontend-install/0001-Fix-host-parsing-for-serial-device-with-in-name.patch \ + frontend-install/0002-Output-data-from-port-during-command-execution.patch \ + /tmp/ +/bin/bash -c 'patch -d /usr/local/src/serio -p1 > /root/.bashrc + +# ============================================================================== +# Lava +# ============================================================================== +ln -s /fuego-ro/scripts/fuego-lava-target-setup /usr/local/bin +ln -s /fuego-ro/scripts/fuego-lava-target-teardown /usr/local/bin diff --git a/install-debian.sh b/install-debian.sh index ccca032..2d3ffd3 100755 --- a/install-debian.sh +++ b/install-debian.sh @@ -2,102 +2,163 @@ # 2019 (c) Toshiba corp. # # Usage: -# $ sudo ./install-debian.sh [--help|-h] [--nojenkins] [] +# $ sudo ./install-debian.sh [--help|-h] [-s ] [-p ] [--nojenkins] # -if [ -n "$1" ]; then - if [ "$1" = "--help" -o "$1" = "-h" ]; then - cat <] - -Installs fuego on the host Debian filesystem. - -options: - --help Show usage help - --nojenkins Install Fuego without Jenkins -HERE - exit 0 - fi -fi + +usage(){ + echo "" + echo "Usage: sudo ./install-debian.sh [--help|-h] [-s ] [-p ] [--nojenkins]" + echo "Install fuego on Debian filesystem." + echo "" + echo "options:" + echo " --help | -h Show usage help" + echo " --suite | -s Specify Debian suite" + echo " --port | -p Jenkins port" + echo " --nojenkins Install Fuego without Jenkins" + echo "" +} if [[ $EUID -ne 0 ]]; then echo "Sorry, you need root permissions" exit 1 fi -if [ "$1" = "--nojenkins" ]; then - nojenkins=1 +nojenkins=0 +port=8090 +suite=stretch +user=jenkins +group=jenkins +uid=1000 +gid=1000 + +while [ "$1" != "" ]; do + case $1 in + -s | --suite ) + shift + suite=$1 + ;; + -p | --port ) + shift + port=$1 + ;; + -u | --uid ) + shift + uid=$1 + ;; + -g | --gid ) + shift + gid=$1 + ;; + --nojenkins ) + nojenkins=1 + ;; + -h | --help ) + usage + exit 0 + ;; + * ) + break + esac shift -else - nojenkins=0 - port=${1:-8090} -fi +done # ============================================================================== # Install Fuego dependencies # ============================================================================== # netperf is in non-free -echo deb http://deb.debian.org/debian stretch main non-free > /etc/apt/sources.list -echo deb http://security.debian.org/debian-security stretch/updates main >> /etc/apt/sources.list - -apt-get update -q=2 +echo deb http://deb.debian.org/debian $suite main non-free > /etc/apt/sources.list +echo deb http://security.debian.org/debian-security $suite/updates main >> /etc/apt/sources.list # Fuego python dependencies -apt-get -q=2 -V --no-install-recommends install \ +# - python-lxml: ftc, loggen +# - python-simplejson: ftc +# - python-yaml: ftc +# - python-openpyxl: ftc (also LTP) +# - python-requests: ftc (also fuego_release_test) +# - python-reportlab: ftc +# - python-parsedatetime: ftc +# - python-pexpect: ssh_exec (part of ttc) +# - python-pip: to install filelock, flake8 +# - filelock: parser +mkdir -p /usr/share/man/man1 +apt-get update -q=2 && apt-get -q=2 -V --no-install-recommends install \ python-lxml python-simplejson python-yaml python-openpyxl \ python-requests python-reportlab python-parsedatetime \ - python-pip + python-pexpect python-pip python-setuptools python-wheel pip install filelock # Fuego command dependencies apt-get -q=2 -V --no-install-recommends install \ git sshpass openssh-client sudo net-tools wget curl lava-tool \ - bash-completion + bash-completion iproute2 # Default SDK for testing locally or on an x86 board apt-get -q=2 -V --no-install-recommends install \ build-essential cmake bison flex automake kmod libtool \ libelf-dev libssl-dev libsdl1.2-dev libcairo2-dev libxmu-dev \ - libxmuu-dev libglib2.0-dev libaio-dev u-boot-tools pkg-config + libxmuu-dev libglib2.0-dev libaio-dev pkg-config rsync u-boot-tools # Default test host dependencies +# - iperf iperf3 netperf: used as servers on the host +# - bzip2 bc: used for local tests by Functional.bzip2/bc +# - python-matplotlib: Benchmark.iperf3 parser +# - python-xmltodict: AGL tests +# - flake8: Functional.fuego_lint +# - netpipe-tcp - used by Benchmark.netpipe (provides the netpipe server) +# - iputils-ping - for /bin/ping command, used by some tests +# FIXTHIS: install dependencies dynamically on the tests that need them apt-get -q=2 -V --no-install-recommends install \ - iperf iperf3 netperf bzip2 bc python-matplotlib python-xmltodict -pip install flake8 bsdmainutils + iperf iperf3 bzip2 bc python-matplotlib python-xmltodict \ + netpipe-tcp iputils-ping -echo "dash dash/sh boolean false" | debconf-set-selections -DEBIAN_FRONTEND=noninteractive dpkg-reconfigure dash -if [ -n "$http_proxy" ]; then - sed -i -e 's/#use_proxy = on/use_proxy = on/g' /etc/wgetrc - echo -e "http_proxy=$http_proxy\nhttps_proxy=$https_proxy" >> /etc/environment +#TODO: find alternate package for netperf in buster version +if [ $suite != "buster" ];then + apt-get -q=2 -V --no-install-recommends install netperf fi +pip install flake8 -# ============================================================================== -# Clone fuego and fuego-core -# ============================================================================== -if [ ! -d "/fuego" ]; then - cd / - git clone --branch next --depth=1 https://bitbucket.org/nirrognas/fuego.git - ln -s /fuego/fuego-ro /fuego-ro - ln -s /fuego/fuego-rw /fuego-rw - cd fuego - git clone --branch next --depth=1 https://bitbucket.org/nirrognas/fuego-core.git - ln -s /fuego/fuego-core /fuego-core -fi +# miscelaneous packages: +# python-serial - used by serio +# diffstat and vim - used by Tim +# time - useful for timing command duration +apt-get -q=2 -V --no-install-recommends install \ + python-serial \ + diffstat \ + vim \ + time + +# install packages used by NuttX SDK +apt-get -q=2 -V --no-install-recommends install \ + genromfs + +# FIXTHIS: determine if these tools are really necessary +#RUN apt-get -q=2 -V --no-install-recommends install \ +# apt-utils python-paramiko \ +# xmlstarlet rsync \ +# inotify-tools gettext netpipe-tcp \ +# at minicom lzop bsdmainutils \ +# mc netcat openssh-server + +# bsdmainutils provides hexdump, which is needed for netperf test +apt-get -q=2 -V --no-install-recommends install \ + bsdmainutils + +echo "dash dash/sh boolean false" | debconf-set-selections +DEBIAN_FRONTEND=noninteractive dpkg-reconfigure dash # ============================================================================== # Install Jenkins # ============================================================================== - if [ $nojenkins -eq 0 ]; then #JENKINS_VERSION=2.164.2 #JENKINS_SHA=4536f43f61b1fca6c58bd91040fa09304eea96ab JENKINS_VERSION=2.249.3 JENKINS_SHA=534014c007edbb533a1833fe6f2dc115faf3faa2 JENKINS_URL=https://pkg.jenkins.io/debian-stable/binary/jenkins_${JENKINS_VERSION}_all.deb - JENKINS_UC=https://updates.jenkins.io - REF=/var/lib/jenkins/plugins - JENKINS_HOME=/var/lib/jenkins + export JENKINS_UC=https://updates.jenkins.io + export REF=/var/lib/jenkins/plugins + export JENKINS_HOME=/var/lib/jenkins JENKINS_PORT=$port # Jenkins dependencies @@ -106,10 +167,12 @@ if [ $nojenkins -eq 0 ]; then pip install python-jenkins==1.4.0 echo -e "JENKINS_PORT=$JENKINS_PORT" >> /etc/environment - groupadd jenkins - useradd -l -m -d "${JENKINS_HOME}" -g jenkins -G sudo -s /bin/bash jenkins - cd - wget ${JENKINS_URL} + export JENKINS_PORT=$port + + getent group ${gid} >/dev/null || groupadd -g ${gid} ${group} + useradd -l -m -d "${JENKINS_HOME}" -u ${uid} -g ${gid} -G sudo -s /bin/bash ${user} + + wget -nv ${JENKINS_URL} echo "${JENKINS_SHA} jenkins_${JENKINS_VERSION}_all.deb" | sha1sum -c - # allow Jenkins to start and install plugins, as part of dpkg installation @@ -117,11 +180,60 @@ if [ $nojenkins -eq 0 ]; then dpkg -i jenkins_${JENKINS_VERSION}_all.deb rm jenkins_${JENKINS_VERSION}_all.deb - # update ownership - chown -R jenkins:jenkins /fuego - chown -R jenkins:jenkins /fuego-ro - chown -R jenkins:jenkins /fuego-rw - chown -R jenkins:jenkins /fuego-core +fi + +# ============================================================================== +# get ttc script and helpers +# ============================================================================== +git clone https://github.com/tbird20d/ttc.git /usr/local/src/ttc +/usr/local/src/ttc/install.sh /usr/local/bin +# as of ttc version 2.0.2, this should not be needed +#perl -p -i -e "s#config_dir = \"/etc\"#config_dir = \"/fuego-ro/conf\"#" /usr/local/bin/ttc + +# ============================================================================== +# get ebf script and helpers +# ============================================================================== +if [ $nojenkins -eq 0 ]; then + apt-get -q=2 -V --no-install-recommends install jq + git clone https://github.com/TimesysGit/board-farm-rest-api.git /usr/local/src/board-farm-rest-api + cp /usr/local/src/board-farm-rest-api/cli/ebf /usr/local/bin/ebf + chmod a+x /usr/local/bin/ebf +fi + +# ============================================================================== +# Serial Config +# ============================================================================== +git clone https://github.com/frowand/serio.git /usr/local/src/serio +if [ $nojenkins -eq 0 ]; then + cp frontend-install/0001-Fix-host-parsing-for-serial-device-with-in-name.patch \ + frontend-install/0002-Output-data-from-port-during-command-execution.patch \ + /tmp/ + patch -d /usr/local/src/serio -p1 > /root/.bashrc # ============================================================================== @@ -255,14 +358,3 @@ echo ". /etc/bash_completion" >> /root/.bashrc # ============================================================================== ln -s /fuego-ro/scripts/fuego-lava-target-setup /usr/local/bin ln -s /fuego-ro/scripts/fuego-lava-target-teardown /usr/local/bin - -# ============================================================================== -# Small guide -# ============================================================================== -echo "Run 'service netperf start' to start a netperf server" -echo "Run 'iperf3 -V -s -D -f M' to start an iperf3 server" -echo "Run 'ftc list-boards' to see the available boards" -echo "Run 'ftc list-tests' to see the available tests" -echo "Run 'ftc run-test -b local -t Functional.hello_world' to run a hello world" -echo "Run 'ftc run-test -b local -t Benchmark.Dhrystone -s 500M' to run Dhrystone" -echo "Run 'ftc gen-report' to get results" diff --git a/install-fuego.sh b/install-fuego.sh new file mode 100755 index 0000000..7b3e010 --- /dev/null +++ b/install-fuego.sh @@ -0,0 +1,109 @@ +#!/bin/bash +# 2019 (c) Toshiba corp. +# +# Usage: +# $ sudo ./install-fuego.sh [--help|-h] [-d ] [-s ] [-p ] [--nojenkins] +# + +usage(){ + echo "" + echo "Usage: sudo ./install-fuego.sh [--help|-h] [-d ] [-s ] [-p ] [--nojenkins]" + echo "Install fuego on Debian filesystem." + echo "" + echo "options:" + echo " --help | -h Show usage help" + echo " --dist | -d Specify distribution" + echo " --suite | -s Specify suite" + echo " --port | -p Jenkins port" + echo " --nojenkins Install Fuego without Jenkins" + echo "" +} + +if [[ $EUID -ne 0 ]]; then + echo "Sorry, you need root permissions" + exit 1 +fi + +nojenkins= +port=8090 +dist=debian +suite=stretch +fuego_dir=$(pwd) + +while [ "$1" != "" ]; do + case $1 in + -d | --dist ) + shift + dist=$1 + ;; + -s | --suite ) + shift + suite=$1 + ;; + -p | --port ) + shift + port=$1 + ;; + --nojenkins ) + nojenkins="--nojenkins" + ;; + -h | --help ) + usage + exit 0 + ;; + * ) + break + esac + shift +done + +# ============================================================================== +# Clone fuego-core +# ============================================================================== +if [ ! -f "fuego-core/scripts/ftc" ]; then + # set fuego-core branch to same as current fuego branch + # get current git branch + set -o noglob + while IFS=" " read -r part1 part2 ; do + if [ $part1 = "*" ] ; then + branch=$part2 + fi + done < <(git branch) + set +o noglob + git clone -b $branch https://bitbucket.org/fuegotest/fuego-core.git +fi + +# ============================================================================== +# Install fuego on host system +# ============================================================================== +./install-$dist.sh -s $suite -p $port $nojenkins + +systemctl daemon-reload +service jenkins restart + +# provide permission to current user +usermod -aG jenkins $USER +chmod -R 775 $fuego_dir/fuego-rw + +chown -R jenkins:jenkins $fuego_dir/fuego-ro +chown -R jenkins:jenkins $fuego_dir/fuego-rw +chown -R jenkins:jenkins $fuego_dir/fuego-core + +# ============================================================================== +# Create Links to fuego folders +# ============================================================================== +ln -s $fuego_dir/fuego-ro / +ln -s $fuego_dir/fuego-rw / +ln -s $fuego_dir/fuego-core / + +# ============================================================================== +# Small guide +# ============================================================================== +echo "Logout and Login current user session to work properly" +echo "Run 'service netperf start' to start a netperf server" +echo "Run 'iperf3 -V -s -D -f M' to start an iperf3 server" +echo "Run 'ftc list-boards' to see the available boards" +echo "Run 'ftc list-tests' to see the available tests" +echo "Run 'ftc run-test -b local -t Functional.hello_world' to run a hello world" +echo "Run 'ftc run-test -b local -t Benchmark.Dhrystone -s 500M' to run Dhrystone" +echo "Run 'ftc gen-report' to get results" diff --git a/install-stretch.sh b/install-stretch.sh new file mode 100755 index 0000000..4f6f724 --- /dev/null +++ b/install-stretch.sh @@ -0,0 +1,288 @@ +#!/bin/bash +# 2019 (c) Toshiba corp. +# +# Usage: +# $ sudo ./install-.sh [--help|-h] [--nojenkins] [] +# +usage(){ + +cat <.sh [--help|-h] [--nojenkins] [] + +Install fuego on the host Debian filesystem. + +options: + --help Show usage help + --nojenkins Install Fuego without Jenkins +HERE +} + +if [[ $EUID -ne 0 ]]; then + echo "Sorry, you need root permissions" + exit 1 +fi + +nojenkins=0 + +POSITIONAL=() +while [[ $# -gt 0 ]] ; do + case $1 in + --nojenkins) + nojenkins=1 + shift + ;; + --help | -h) + usage + exit 0 + ;; + *) + POSITIONAL+=("$1") # save argument for later + shift + esac +done +set -- "${POSITIONAL[@]}" # restore positional arguments + +port=${2:-8090} + +# ============================================================================== +# Install Fuego dependencies +# ============================================================================== + +mkdir -p /usr/share/man/man1 + +# netperf is in non-free +echo deb http://deb.debian.org/debian stretch main non-free > /etc/apt/sources.list +echo deb http://security.debian.org/debian-security stretch/updates main >> /etc/apt/sources.list + +apt-get update -q=2 + +# Fuego python dependencies +apt-get -q=2 -V --no-install-recommends install \ + python-lxml python-simplejson python-yaml python-openpyxl \ + python-requests python-reportlab python-parsedatetime \ + python-pexpect python-pip python-setuptools python-wheel +pip install filelock + +# Fuego command dependencies +apt-get -q=2 -V --no-install-recommends install \ + git sshpass openssh-client sudo net-tools wget curl lava-tool \ + bash-completion iproute2 + +# Default SDK for testing locally or on an x86 board +apt-get -q=2 -V --no-install-recommends install \ + build-essential cmake bison flex kmod automake libtool \ + libelf-dev libssl-dev libsdl1.2-dev libcairo2-dev libxmu-dev \ + libxmuu-dev libglib2.0-dev libaio-dev pkg-config rsync u-boot-tools + +# Default test host dependencies +apt-get -q=2 -V --no-install-recommends install \ + iperf iperf3 netperf bzip2 bc python-matplotlib python-xmltodict \ + netpipe-tcp iputils-ping +pip install flake8 + +# miscelaneous packages: +# python-serial - used by serio +# diffstat and vim - used by Tim +# time - useful for timing command duration +apt-get -q=2 -V --no-install-recommends install \ + python-serial \ + diffstat \ + vim \ + time + +# install packages used by NuttX SDK +apt-get -q=2 -V --no-install-recommends install \ + genromfs + +# bsdmainutils provides hexdump, which is needed for netperf test +apt-get -q=2 -V --no-install-recommends install \ + bsdmainutils + +echo "dash dash/sh boolean false" | debconf-set-selections +DEBIAN_FRONTEND=noninteractive dpkg-reconfigure dash + +# TODO: check if proxy settings can set outside docker container +#if [ -n "$http_proxy" ]; then +# sed -i -e 's/#use_proxy = on/use_proxy = on/g' /etc/wgetrc +# echo -e "http_proxy=$http_proxy\nhttps_proxy=$https_proxy" >> /etc/environment +#fi + +# ============================================================================== +# Install Jenkins +# ============================================================================== + +if [ $nojenkins -eq 0 ]; then + #JENKINS_VERSION=2.164.2 + #JENKINS_SHA=4536f43f61b1fca6c58bd91040fa09304eea96ab + JENKINS_VERSION=2.249.3 + JENKINS_SHA=534014c007edbb533a1833fe6f2dc115faf3faa2 + JENKINS_URL=https://pkg.jenkins.io/debian-stable/binary/jenkins_${JENKINS_VERSION}_all.deb + export JENKINS_UC=https://updates.jenkins.io + REF=/var/lib/jenkins/plugins + JENKINS_HOME=/var/lib/jenkins + JENKINS_PORT=$port + + # Jenkins dependencies + apt-get -q=2 -V --no-install-recommends install \ + default-jdk daemon psmisc adduser procps unzip + pip install python-jenkins==1.4.0 + + echo -e "JENKINS_PORT=$JENKINS_PORT" >> /etc/environment + export JENKINS_PORT=$port + + groupadd jenkins + useradd -l -m -d "${JENKINS_HOME}" -g jenkins -G sudo -s /bin/bash jenkins + + wget -nv ${JENKINS_URL} + echo "${JENKINS_SHA} jenkins_${JENKINS_VERSION}_all.deb" | sha1sum -c - + + # allow Jenkins to start and install plugins, as part of dpkg installation + printf "#!/bin/sh\nexit 0" > /usr/sbin/policy-rc.d + dpkg -i jenkins_${JENKINS_VERSION}_all.deb + rm jenkins_${JENKINS_VERSION}_all.deb + + source /etc/default/jenkins && \ + JENKINS_ARGS="$JENKINS_ARGS --prefix=/fuego" && \ + sed -i -e "s#JENKINS_ARGS.*#JENKINS_ARGS\=\"${JENKINS_ARGS}\"#g" /etc/default/jenkins + + source /etc/default/jenkins && \ + JAVA_ARGS="$JAVA_ARGS -Djenkins.install.runSetupWizard=false -Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true" && \ + if [ -n "$http_proxy" ]; then \ + PROXYSERVER=$(echo $http_proxy | sed -E 's/^http://' | sed -E 's/\///g' | sed -E 's/(.*):(.*)/\1/') && \ + PROXYPORT=$(echo $http_proxy | sed -E 's/^http://' | sed -E 's/\///g' | sed -E 's/(.*):(.*)/\2/') && \ + JAVA_ARGS="$JAVA_ARGS -Dhttp.proxyHost="${PROXYSERVER}" -Dhttp.proxyPort="${PROXYPORT}" -Dhttps.proxyHost="${PROXYSERVER}" -Dhttps.proxyPort="${PROXYPORT}; \ + fi && \ + sed -i -e "s#^JAVA_ARGS.*#JAVA_ARGS\=\"${JAVA_ARGS}\"#g" /etc/default/jenkins; + + sed -i -e "s#8080#$JENKINS_PORT#g" /etc/default/jenkins + + # set up Jenkins plugins + cp frontend-install/install-plugins.sh \ + frontend-install/jenkins-support \ + frontend-install/clitest \ + /usr/local/bin/ + + mkdir -p $JENKINS_HOME/userContent/docs + cp frontend-install/config.xml $JENKINS_HOME/config.xml + ln -s /fuego-rw/logs $JENKINS_HOME/userContent/fuego.logs + cp docs/fuego-docs.pdf $JENKINS_HOME/userContent/docs/fuego-docs.pdf + cp frontend-install/jenkins.model.JenkinsLocationConfiguration.xml $JENKINS_HOME/jenkins.model.JenkinsLocationConfiguration.xml + sed -i -e "s#8080#$JENKINS_PORT#g" $JENKINS_HOME/jenkins.model.JenkinsLocationConfiguration.xml + chown -R jenkins:jenkins $JENKINS_HOME/ + + # start and stop jenkins to pre-populate some settings + service jenkins start && \ + sleep 30 && \ + service jenkins stop + + # install other plugins from Jenkins update center + # NOTE: not sure all of these are needed, but keep list + # compatible with 1.2.1 release for now + #/usr/local/bin/install-plugins.sh \ + # ant:1.9 \ + # antisamy-markup-formatter:1.5 \ + # bouncycastle-api:2.17 \ + # command-launcher:1.3 \ + # description-setter:1.10 \ + # display-url-api:2.3.1 \ + # external-monitor-job:1.7 \ + # greenballs:1.15 \ + # icon-shim:2.0.3 \ + # javadoc:1.5 \ + # jdk-tool:1.2 \ + # junit:1.27 \ + # ldap:1.20 \ + # mailer:1.23 \ + # matrix-auth:2.3 \ + # matrix-project:1.14 \ + # pam-auth:1.4 \ + # pegdown-formatter:1.3 \ + # structs:1.17 \ + # windows-slaves:1.4 + + # install plugins for latest LTS Jenkins (currently 2.249.3) + /usr/local/bin/install-plugins.sh \ + script-security \ + structs \ + workflow-step-api \ + workflow-api \ + junit \ + scm-api \ + ant \ + antisamy-markup-formatter \ + bouncycastle-api \ + command-launcher \ + description-setter \ + display-url-api \ + external-monitor-job \ + greenballs \ + icon-shim \ + javadoc \ + jdk-tool \ + mailer \ + matrix-auth \ + matrix-project \ + pam-auth \ + pegdown-formatter \ + windows-slaves + + cp frontend-install/plugins/flot-plotter-plugin/flot.hpi $JENKINS_HOME/plugins/flot.jpi + + # make the mod.js symlink well after flot is installed + service jenkins start && sleep 30 && \ + rm $JENKINS_HOME/plugins/flot/flot/mod.js && \ + ln -s /fuego-core/scripts/mod.js $JENKINS_HOME/plugins/flot/flot/mod.js + + chown -R jenkins:jenkins $JENKINS_HOME/ +fi + +# ============================================================================== +# get ttc script and helpers +# ============================================================================== +git clone https://github.com/tbird20d/ttc.git /usr/local/src/ttc +/usr/local/src/ttc/install.sh /usr/local/bin +perl -p -i -e "s#config_dir = \"/etc\"#config_dir = \"/fuego-ro/conf\"#" /usr/local/bin/ttc + +# ============================================================================== +# Serial Config +# ============================================================================== +if [ $nojenkins -eq 0 ]; then + /bin/bash -c 'git clone https://github.com/frowand/serio.git /usr/local/src/serio ; chown -R jenkins /usr/local/src/serio ; cp /usr/local/src/serio/serio /usr/local/bin/ ; ln -s /usr/local/bin/serio /usr/local/bin/sercp ; ln -s /usr/local/bin/serio /usr/local/bin/sersh' + /bin/bash -c 'git clone https://github.com/tbird20d/serlogin.git /usr/local/src/serlogin ; chown -R jenkins /usr/local/src/serlogin ; cp /usr/local/src/serlogin/serlogin /usr/local/bin' +else + /bin/bash -c 'git clone https://github.com/frowand/serio.git /usr/local/src/serio ; cp /usr/local/src/serio/serio /usr/local/bin/ ; ln -s /usr/local/bin/serio /usr/local/bin/sercp ; ln -s /usr/local/bin/serio /usr/local/bin/sersh' + /bin/bash -c 'git clone https://github.com/tbird20d/serlogin.git /usr/local/src/serlogin ; cp /usr/local/src/serlogin/serlogin /usr/local/bin' +fi +cp frontend-install/0001-Fix-host-parsing-for-serial-device-with-in-name.patch \ + frontend-install/0002-Output-data-from-port-during-command-execution.patch \ + /tmp/ +/bin/bash -c 'patch -d /usr/local/src/serio -p1 > /root/.bashrc + +# ============================================================================== +# Lava +# ============================================================================== +ln -s /fuego-ro/scripts/fuego-lava-target-setup /usr/local/bin +ln -s /fuego-ro/scripts/fuego-lava-target-teardown /usr/local/bin diff --git a/install.sh b/install.sh index a5a90b0..259c887 100755 --- a/install.sh +++ b/install.sh @@ -1,13 +1,13 @@ #!/bin/bash # -# install.sh [--help] [--priv] [--no-cache] [--nojenkins] [] [] +# install.sh [--help] [--priv] [--no-cache] [--nojenkins] [] [] [] [] # # $1 is the exit code after usage is shown usage() { exit_code="$1" cat <] [] +Usage: install.sh [--help] [--priv] [--no-cache] [--nojenkins] [] [] [] [] Create the docker image and container with the Fuego test distribution. If no is provided, the image will be named 'fuego'. @@ -34,7 +34,7 @@ fi priv=0 NOCACHE="" -dockerfile="Dockerfile" +NOJENKINS="" POSITIONAL=() while [[ $# -gt 0 ]] ; do @@ -48,7 +48,7 @@ while [[ $# -gt 0 ]] ; do shift ;; --nojenkins) - dockerfile="Dockerfile.nojenkins" + NOJENKINS="--nojenkins" shift ;; *) @@ -60,6 +60,8 @@ set -- "${POSITIONAL[@]}" # restore positional arguments image_name=${1:-fuego} jenkins_port=${2:-8090} +distribution=${3:-debian} +suite=${4:-stretch} container_name="${image_name}-container" @@ -79,7 +81,7 @@ fi set -e -source fuego-host-scripts/docker-build-image.sh $NOCACHE ${image_name} ${jenkins_port} ${dockerfile} +source fuego-host-scripts/docker-build-image.sh $NOCACHE ${image_name} ${jenkins_port} ${distribution} ${suite} ${NOJENKINS} if [ "$priv" == "0" ]; then fuego-host-scripts/docker-create-container.sh ${image_name} ${container_name} else diff --git a/uninstall-fuego.sh b/uninstall-fuego.sh new file mode 100644 index 0000000..b899e7c --- /dev/null +++ b/uninstall-fuego.sh @@ -0,0 +1,47 @@ +#!/bin/bash + +# Remove Jenkins plugins +[ -d /usr/share/jenkins/ref/plugins/ ] && rm -rf /usr/share/jenkins/ref/plugins + +# uninstall Jenkins +service jenkins stop +dpkg -P jenkins + +[ -f /usr/local/bin/install-plugins.sh ] && rm -f /usr/local/bin/install-plugins.sh +[ -f /usr/local/bin/jenkins-support ] && rm -rf /usr/local/bin/jenkins-support +[ -f /usr/local/bin/clitest ] && rm -rf /usr/local/bin/clitest +[ -d /var/lib/jenkins ] && rm -rf /var/lib/jenkins +sed -i '/^JENKINS_PORT=/d' /etc/environment + +# uninstall ttc script +[ -d /usr/local/src/ttc ] && rm -rf /usr/local/src/ttc + +# uninstall ebf script +[ -d /usr/local/src/board-farm-rest-api ] && rm -rf /usr/local/src/board-farm-rest-api +[ -f /usr/local/bin/ebf ] && rm -f /usr/local/bin/ebf + +# uninstall serial config +[ -d /usr/local/src/serio ] && rm -rf /usr/local/src/serio +[ -f /usr/local/bin/serio ] && rm -rf /usr/local/bin/serio +[ -f /usr/local/bin/sercp ] && rm -rf /usr/local/bin/sercp +[ -f /usr/local/bin/sersh ] && rm -rf /usr/local/bin/sersh +[ -d /usr/local/src/serlogin ] && rm -rf /usr/local/src/serlogin +[ -f /usr/local/bin/serlogin ] && rm -rf /usr/local/bin/serlogin + +# uninstall fserver +[ -d /usr/local/lib/fserver ] && rm -f /usr/local/bin/fserver +[ -f /usr/local/bin/start_local_bg_server ] && rm -rf /usr/local/bin/start_local_bg_server + +# uninstall ftc +[ -f /usr/local/bin/ftc ] && rm -rf /usr/local/bin/ftc +[ -f /etc/bash_completion.d/ftc ] && rm -rf /etc/bash_completion.d/ftc + +# remove lava target files +[ -f /usr/local/bin/fuego-lava-target-setup ] && rm -rf /usr/local/bin/fuego-lava-target-setup +[ -f /usr/local/bin/fuego-lava-target-teardown ] && rm -rf /usr/local/bin/fuego-lava-target-teardown + +# remove links +[ -d /fuego-ro ] && rm /fuego-ro +[ -d /fuego-rw ] && rm /fuego-rw +[ -d /fuego-core ] && rm /fuego-core + -- 2.20.1 The information contained in this e-mail message and in any attachments/annexure/appendices is confidential to the recipient and may contain privileged information. If you are not the intended recipient, please notify the sender and delete the message along with any attachments/annexure/appendices. You should not disclose, copy or otherwise use the information contained in the message or any annexure. Any views expressed in this e-mail are those of the individual sender except where the sender specifically states them to be the views of Toshiba Software India Pvt. Ltd. (TSIP),Bangalore. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by Toshiba Embedded Software India Pvt. Ltd, for any loss or damage arising in any way from its use.