From: <Tim.Bird@sony.com>
To: venkata.pyla@toshiba-tsip.com, tbird20d@yahoo.com
Cc: fuego@lists.linuxfoundation.org
Subject: Re: [Fuego] Use common script to install fuego on host or docker system
Date: Tue, 15 Jun 2021 00:08:45 +0000 [thread overview]
Message-ID: <BYAPR13MB25037C97DF64D58B5DA0855DFD309@BYAPR13MB2503.namprd13.prod.outlook.com> (raw)
In-Reply-To: <20210604134356.1607-1-venkata.pyla@toshiba-tsip.com>
Please see comments inline below.
I tried to apply this patch from the e-mail list using 'git am', but got a failure.
That made it a hard to experiment with and view the changes in
context.
What version of the code is this patch based on?
> -----Original Message-----
> From: venkata.pyla@toshiba-tsip.com <venkata.pyla@toshiba-tsip.com>
> Subject: [Fuego] Use common script to install fuego on host or docker system
>
> From: venkata pyla <venkata.pyla@toshiba-tsip.com>
>
> Modified the install scripts to use common scripts for both types of
> installation, this way it will help to make sure both the environments
> are same and if any future modification will have single place of change.
>
> Now, for installing fuego on docker system the 'install.sh' script will
> use common script in Dockefile to install fuego related stuff.
>
> sameway for installing fuego on host system it can use the same common
> script to install fuego related stuff.
>
> Below are the common scripts that are used in both the installation types
> for each debian distribution
> * install-stretch.sh
> * install-buster.sh
>
> so finally,
> To install fuego on docker system use the below script with options
> $ ./install.sh <options> image_name <port> <distribution> <suite>
> e.g: ./install.sh debian_stretch 8090 debian stretch
> or
> ./install.sh debian_buster 8090 debian buster
> or
> ./install.sh --nojenkins debian_buster_nojenkins 8090 debian buster
This is a bit wordy, but OK.
If we don't support any other distributions besides Debian at the moment,
then I'm not sure we need the distribution argument. Do you have any
non-debian distribution install scripts ready to submit soon?
I'm not sure I follow calling the Debian release name the "<suite>".
>
> To install fuego on host system use the below script with options
> $ ./install-fuego.sh <options> <-d distribution> <-s suite> <-p port>
"suite" seems like an odd name. Should this be distro "version" or "release"?
> e.g: ./install-fuego.sh -d debian -s stretch -p 8090
> or
> ./install-fuego.sh -d debian -s buster -p 8090
> or
> ./install-fuego.sh -d debian -s buster -p 8090 --nojenkins
>
> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
> ---
> Dockerfile | 303 ++---------------
> Dockerfile.nojenkins | 151 ---------
> README | 29 +-
> entrypoint.sh | 7 +
> fuego-host-scripts/docker-build-image.sh | 11 +-
> install-buster.sh | 294 ++++++++++++++++
> install-debian.sh | 410 ++++++++++++++---------
> install-fuego.sh | 109 ++++++
> install-stretch.sh | 288 ++++++++++++++++
Is install-debian.sh the common parts for a Debian installation, with
buster and stretch having the parts that are unique to each of
those distros respectively? If we're going to refactor this, I
want to eliminate any duplication, so that package names are
not used in multiple places. This has been a headache to
maintain.
It appears there is lots of duplication between install-buster.sh and
install-stretch.sh. Can we possible put this duplication into an
"install-debian-common.sh", and reserve install-buster.sh and
install-stretch.sh for only the parts unique to those distros?
> install.sh | 12 +-
> uninstall-fuego.sh | 47 +++
Ooo - I like the sound of this uninstall script.
> 11 files changed, 1057 insertions(+), 604 deletions(-)
> delete mode 100644 Dockerfile.nojenkins
> create mode 100755 entrypoint.sh
> create mode 100755 install-buster.sh
> create mode 100755 install-fuego.sh
> create mode 100755 install-stretch.sh
> create mode 100644 uninstall-fuego.sh
>
> diff --git a/Dockerfile b/Dockerfile
> index 7f63798..ac57cb2 100644
> --- a/Dockerfile
> +++ b/Dockerfile
> @@ -2,9 +2,11 @@
> # WARNING: this Dockerfile assumes that the container will be created with
> # several bind mounts (see docker-create-container.sh)
> # ==============================================================================
> -# FIXTHIS: build this as an extension of the nonjenkins image
>
> -FROM debian:stretch-slim
> +ARG SUITE=stretch
> +ARG DIST=debian
> +
> +FROM ${DIST}:${SUITE}-slim
> MAINTAINER tim.bird@sony.com
>
> # ==============================================================================
> @@ -16,299 +18,32 @@ ENV http_proxy ${HTTP_PROXY}
> ENV https_proxy ${HTTP_PROXY}
>
> # ==============================================================================
> -# Prepare basic image
> +# Install fuego with the script
> # ==============================================================================
>
> -ARG DEBIAN_FRONTEND=noninteractive
> -
> -WORKDIR /
> -RUN echo deb http://deb.debian.org/debian stretch main non-free > /etc/apt/sources.list
> -RUN echo deb http://security.debian.org/debian-security stretch/updates main >> /etc/apt/sources.list
> -RUN if [ -n "$HTTP_PROXY" ]; then echo 'Acquire::http::proxy "'$HTTP_PROXY'";' > /etc/apt/apt.conf.d/80proxy; fi
> -
> -# Fuego python dependencies
> -# - python-lxml: ftc, loggen
> -# - python-simplejson: ftc
> -# - python-yaml: ftc
> -# - python-openpyxl: ftc (also LTP)
> -# - python-requests: ftc (also fuego_release_test)
> -# - python-reportlab: ftc
> -# - python-parsedatetime: ftc
> -# - python-pexpect: ssh_exec (part of ttc)
> -# - python-pip: to install filelock, flake8
> -# - filelock: parser
> -RUN mkdir -p /usr/share/man/man1
> -RUN apt-get update -q=2 && apt-get -q=2 -V --no-install-recommends install \
> - python-lxml python-simplejson python-yaml python-openpyxl \
> - python-requests python-reportlab python-parsedatetime \
> - python-pexpect python-pip python-setuptools python-wheel
> -RUN pip install filelock
> -
> -# Fuego command dependencies
> -RUN apt-get -q=2 -V --no-install-recommends install \
> - git sshpass openssh-client sudo net-tools wget curl lava-tool \
> - bash-completion iproute2
> -
> -# Default SDK for testing locally or on an x86 board
> -RUN apt-get -q=2 -V --no-install-recommends install \
> - build-essential cmake bison flex automake kmod libtool \
> - libelf-dev libssl-dev libsdl1.2-dev libcairo2-dev libxmu-dev \
> - libxmuu-dev libglib2.0-dev libaio-dev pkg-config rsync u-boot-tools
> -
> -# Default test host dependencies
> -# - iperf iperf3 netperf: used as servers on the host
> -# - bzip2 bc: used for local tests by Functional.bzip2/bc
> -# - python-matplotlib: Benchmark.iperf3 parser
> -# - python-xmltodict: AGL tests
> -# - flake8: Functional.fuego_lint
> -# - netpipe-tcp - used by Benchmark.netpipe (provides the netpipe server)
> -# - iputils-ping - for /bin/ping command, used by some tests
> -# FIXTHIS: install dependencies dynamically on the tests that need them
> -RUN apt-get -q=2 -V --no-install-recommends install \
> - iperf iperf3 netperf bzip2 bc python-matplotlib python-xmltodict \
> - netpipe-tcp iputils-ping
> -RUN pip install flake8
> -
> -# miscelaneous packages:
> -# python-serial - used by serio
> -# diffstat and vim - used by Tim
> -# time - useful for timing command duration
> -RUN apt-get -q=2 -V --no-install-recommends install \
> - python-serial \
> - diffstat \
> - vim \
> - time
> -
> -# install packages used by NuttX SDK
> -RUN apt-get -q=2 -V --no-install-recommends install \
> - genromfs
> -
> -# FIXTHIS: determine if these tools are really necessary
> -#RUN apt-get -q=2 -V --no-install-recommends install \
> -# apt-utils python-paramiko \
> -# xmlstarlet rsync \
> -# inotify-tools gettext netpipe-tcp \
> -# at minicom lzop bsdmainutils \
> -# mc netcat openssh-server
> -
> -# bsdmainutils provides hexdump, which is needed for netperf test
> -RUN apt-get -q=2 -V --no-install-recommends install \
> - bsdmainutils
> -
> -RUN /bin/bash -c 'echo "dash dash/sh boolean false" | debconf-set-selections ; dpkg-reconfigure dash'
> -RUN if [ -n "$HTTP_PROXY" ]; then echo "use_proxy = on" >> /etc/wgetrc; fi
> -RUN if [ -n "$HTTP_PROXY" ]; then echo -e "http_proxy=$HTTP_PROXY\nhttps_proxy=$HTTP_PROXY" >> /etc/environment; fi
> -
> -# ==============================================================================
> -# Install Jenkins with the same UID/GID as the host user
> -# ==============================================================================
> +RUN mkdir /fuego
> +COPY . /fuego/
> +WORKDIR /fuego
>
> -ARG user=jenkins
> -ARG group=jenkins
> -ARG uid=1000
> -ARG gid=${uid}
> ARG JENKINS_PORT=8090
> -# see https://updates.jenkins.io/download/war/ for latest Jenkins versions
> -#ARG JENKINS_VERSION=2.164.2
> -#ARG JENKINS_SHA=4536f43f61b1fca6c58bd91040fa09304eea96ab
> -#ARG JENKINS_VERSION=2.235.3
> -#ARG JENKINS_SHA=89a37ed1b3c8c14ff94cc66c3f2254f281b60794
> -ARG JENKINS_VERSION=2.249.3
> -ARG JENKINS_SHA=534014c007edbb533a1833fe6f2dc115faf3faa2
> -
> -ARG JENKINS_URL=https://pkg.jenkins.io/debian-stable/binary/jenkins_${JENKINS_VERSION}_all.deb
> -ARG JENKINS_UC=https://updates.jenkins.io
> -ARG REF=/var/lib/jenkins/plugins
> +ARG NO_JENKINS=
> +ARG SUITE=stretch
> +ARG DIST=debian
> +ARG uid=
> +ARG gid=
> ENV JENKINS_HOME=/var/lib/jenkins
> ENV JENKINS_PORT=$JENKINS_PORT
> +RUN ./install-$DIST.sh -s $SUITE -p $JENKINS_PORT -u $uid -g $gid $NO_JENKINS
One question about this overall approach. Does this retain the ability to cache
operations, so that if something goes wrong with an install, you don't have to
start over from scratch. When building with docker, with multiple steps, it
caches the image at each step. When rebuilding, or when an error occurs,
this can save a lot of time. It also can introduce subtle bugs, but that's
a separate issue.
This setup seems like it performs the bulk of the install as a single docker
step, is that right?
> -# Jenkins dependencies
> -RUN apt-get -q=2 -V --no-install-recommends install \
> - default-jdk daemon psmisc adduser procps unzip
> -RUN pip install python-jenkins==1.4.0
> -
> -RUN echo -e "JENKINS_PORT=$JENKINS_PORT" >> /etc/environment
> -RUN getent group ${gid} >/dev/null || groupadd -g ${gid} ${group}
> -RUN useradd -l -m -d "${JENKINS_HOME}" -u ${uid} -g ${gid} -G sudo -s /bin/bash ${user}
> -RUN wget -nv ${JENKINS_URL}
> -RUN echo "${JENKINS_SHA} jenkins_${JENKINS_VERSION}_all.deb" | sha1sum -c -
> -# allow Jenkins to start and install plugins, as part of dpkg installation
> -RUN printf "#!/bin/sh\nexit 0" > /usr/sbin/policy-rc.d
> -RUN dpkg -i jenkins_${JENKINS_VERSION}_all.deb
> -RUN rm jenkins_${JENKINS_VERSION}_all.deb
> -
> -# ==============================================================================
> -# get ttc script and helpers
> -# ==============================================================================
> -RUN git clone https://github.com/tbird20d/ttc.git /usr/local/src/ttc
> -RUN /usr/local/src/ttc/install.sh /usr/local/bin
> -# as of ttc version 2.0.2, this should not be needed
> -#RUN perl -p -i -e "s#config_dir = \"/etc\"#config_dir = \"/fuego-ro/conf\"#" /usr/local/bin/ttc
> -
> -# ==============================================================================
> -# get ebf script and helpers
> -# ==============================================================================
> -RUN apt-get -q=2 -V --no-install-recommends install jq
> -RUN git clone https://github.com/TimesysGit/board-farm-rest-api.git /usr/local/src/board-farm-rest-api ; \
> - cp /usr/local/src/board-farm-rest-api/cli/ebf /usr/local/bin/ebf ; \
> - chmod a+x /usr/local/bin/ebf
> -
> -# ==============================================================================
> -# Serial Config
> -# ==============================================================================
> -
> -RUN git clone https://github.com/frowand/serio.git /usr/local/src/serio
> -COPY frontend-install/0001-Fix-host-parsing-for-serial-device-with-in-name.patch \
> - frontend-install/0002-Output-data-from-port-during-command-execution.patch \
> - /tmp/
> -RUN /bin/bash -c 'patch -d /usr/local/src/serio -p1 </tmp/0001-Fix-host-parsing-for-serial-device-with-in-name.patch ; \
> - patch -d /usr/local/src/serio -p1 </tmp/0002-Output-data-from-port-during-command-execution.patch ; \
> - chown -R jenkins /usr/local/src/serio ; \
> - cp /usr/local/src/serio/serio /usr/local/bin/ ; \
> - ln -s /usr/local/bin/serio /usr/local/bin/sercp ; \
> - ln -s /usr/local/bin/serio /usr/local/bin/sersh'
> -
> -RUN /bin/bash -c 'git clone https://github.com/tbird20d/serlogin.git /usr/local/src/serlogin ; chown -R jenkins /usr/local/src/serlogin ; cp
> /usr/local/src/serlogin/serlogin /usr/local/bin'
> -
> -# ==============================================================================
> -# fserver
> -# ==============================================================================
> -
> -RUN /bin/bash -c 'git clone https://github.com/tbird20d/fserver.git /usr/local/lib/fserver ; \
> - ln -s /usr/local/lib/fserver/start_local_bg_server /usr/local/bin/start_local_bg_server'
> -
> -# ==============================================================================
> -# Jenkins post installation
> -# ==============================================================================
> -
> -RUN source /etc/default/jenkins && \
> - JENKINS_ARGS="$JENKINS_ARGS --prefix=/fuego" && \
> - sed -i -e "s#JENKINS_ARGS.*#JENKINS_ARGS\=\"${JENKINS_ARGS}\"#g" /etc/default/jenkins
> -
> -RUN source /etc/default/jenkins && \
> - JAVA_ARGS="$JAVA_ARGS -Djenkins.install.runSetupWizard=false -
> Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true" && \
> - if [ -n "$HTTP_PROXY" ]; then \
> - PROXYSERVER=$(echo $http_proxy | sed -E 's/^http://' | sed -E 's/\///g' | sed -E 's/(.*):(.*)/\1/') && \
> - PROXYPORT=$(echo $http_proxy | sed -E 's/^http://' | sed -E 's/\///g' | sed -E 's/(.*):(.*)/\2/') && \
> - JAVA_ARGS="$JAVA_ARGS -Dhttp.proxyHost="${PROXYSERVER}" -Dhttp.proxyPort="${PROXYPORT}" -
> Dhttps.proxyHost="${PROXYSERVER}" -Dhttps.proxyPort="${PROXYPORT}; \
> - fi && \
> - sed -i -e "s#^JAVA_ARGS.*#JAVA_ARGS\=\"${JAVA_ARGS}\"#g" /etc/default/jenkins;
> -
> -RUN sed -i -e "s#8080#$JENKINS_PORT#g" /etc/default/jenkins
> -
> -# set up Jenkins plugins
> -COPY frontend-install/install-plugins.sh \
> - frontend-install/jenkins-support \
> - frontend-install/clitest \
> - /usr/local/bin/
> -
> -# start Jenkins onces to initialize directories?
> -RUN service jenkins start && \
> - sleep 30 && \
> - service jenkins stop
> -
> -# install plugins - these versions are for Jenkins version 2.164.2
> -# Explicitly install script-security v1.68, otherwise
> -# v1.74 will automatically be installed as a dependency of
> -# the junit plugin. Make sure to install before junit plugin.
> -# Explicitly install junit:1.27
> -
> -# install other plugins from Jenkins update center
> -# NOTE: not sure all of these are needed, but keep list
> -# as compatible as possible with 1.2.1 release for now
> -# Do NOT change the order of the plugins, unless you know what
> -# you are doing. Otherwise, the install-plugins.sh script will
> -# install the wrong versions of other plugins based on dependency
> -# information in the plugin files.
> -
> -#RUN /usr/local/bin/install-plugins.sh \
> -# script-security:1.68 \
> -# structs:1.20 \
> -# workflow-step-api:2.22 \
> -# workflow-api:2.40 \
> -# junit:1.27 \
> -# scm-api:2.6.3 \
> -# ant:1.9 \
> -# antisamy-markup-formatter:1.5 \
> -# bouncycastle-api:2.17 \
> -# command-launcher:1.3 \
> -# description-setter:1.10 \
> -# display-url-api:2.3.1 \
> -# external-monitor-job:1.7 \
> -# greenballs:1.15 \
> -# icon-shim:2.0.3 \
> -# javadoc:1.5 \
> -# jdk-tool:1.2 \
> -# ldap:1.20 \
> -# mailer:1.23 \
> -# matrix-auth:2.3 \
> -# matrix-project:1.14 \
> -# pam-auth:1.5 \
> -# pegdown-formatter:1.3 \
> -# windows-slaves:1.4
> -
> -# install plugins for latest LTS Jenkins (currently 2.249.3)
> -RUN /usr/local/bin/install-plugins.sh \
> - script-security \
> - structs \
> - workflow-step-api \
> - workflow-api \
> - junit \
> - scm-api \
> - ant \
> - antisamy-markup-formatter \
> - bouncycastle-api \
> - command-launcher \
> - description-setter \
> - display-url-api \
> - external-monitor-job \
> - greenballs \
> - icon-shim \
> - javadoc \
> - jdk-tool \
> - mailer \
> - matrix-auth \
> - matrix-project \
> - pam-auth \
> - pegdown-formatter \
> - windows-slaves
> +COPY entrypoint.sh /usr/bin/
>
> -COPY frontend-install/plugins/flot-plotter-plugin/flot.hpi $JENKINS_HOME/plugins/flot.jpi
> -
> -# jenkins should automatically unzip any plugins in the plugin dir
> -# make a symlink for mod.js well after flot is installed
> -RUN service jenkins start && sleep 30 && \
> - rm $JENKINS_HOME/plugins/flot/flot/mod.js && \
> - ln -s /fuego-core/scripts/mod.js $JENKINS_HOME/plugins/flot/flot/mod.js
> -
> -RUN ln -s /fuego-rw/logs $JENKINS_HOME/userContent/fuego.logs
> -COPY docs/fuego-docs.pdf $JENKINS_HOME/userContent/docs/fuego-docs.pdf
> -
> -COPY frontend-install/config.xml $JENKINS_HOME/config.xml
> -COPY frontend-install/jenkins.model.JenkinsLocationConfiguration.xml $JENKINS_HOME/jenkins.model.JenkinsLocationConfiguration.xml
> -RUN sed -i -e "s#8080#$JENKINS_PORT#g" $JENKINS_HOME/jenkins.model.JenkinsLocationConfiguration.xml
> -
> -RUN chown -R jenkins:jenkins $JENKINS_HOME/
> -
> -# ==============================================================================
> -# ftc post installation
> -# ==============================================================================
> -
> -RUN ln -s /fuego-core/scripts/ftc /usr/local/bin/
> -COPY fuego-core/scripts/ftc_completion.sh /etc/bash_completion.d/ftc
> -RUN echo ". /etc/bash_completion" >> /root/.bashrc
> -
> -# ==============================================================================
> -# Lava
> -# ==============================================================================
> -
> -RUN ln -s /fuego-ro/scripts/fuego-lava-target-setup /usr/local/bin
> -RUN ln -s /fuego-ro/scripts/fuego-lava-target-teardown /usr/local/bin
> +# Delete the files that are copied for installation
> +RUN rm -rf /fuego
I'm not sure what this is doing. It looks scary. Please explain.
> +WORKDIR /
>
> # ==============================================================================
> # Setup startup command
> # ==============================================================================
> -
> # FIXTHIS: when running multiple Fuego containers, or if the host is already
> # running the netperf server, netperf will complain because the port is taken
> -ENTRYPOINT service jenkins start && service netperf start && iperf3 -V -s -D -f M && /bin/bash
> +ENTRYPOINT entrypoint.sh $NO_JENKINS && /bin/bash
Isolating this to a separate script is nice!
> diff --git a/Dockerfile.nojenkins b/Dockerfile.nojenkins
> deleted file mode 100644
> index 56de98a..0000000
> --- a/Dockerfile.nojenkins
> +++ /dev/null
Getting rid of this separate Dockerfile is nice.
> @@ -1,151 +0,0 @@
> -# ==============================================================================
> -# WARNING: this Dockerfile assumes that the container will be created with
> -# several bind mounts (see docker-create-container.sh)
> -# ==============================================================================
> -# Usage:
> -# host$ vi fuego-ro/conf/fuego.conf
> -# jenkins_enabled=0
> -# host$ ./fuego-host-scripts/docker-build-image.sh fuego-nojenkins 8090 Dockerfile.nojenkins
> -# host$ ./fuego-host-scripts/docker-create-container.sh fuego-nojenkins fuego-nojenkins-container
> -# host$ ./fuego-host-scripts/docker-start-container.sh fuego-nojenkins-container
> -# docker# ftc list-boards
> -# docker, bbb, ...
> -# docker# ftc list-tests
> -# Functional.hello_world, ...
> -# docker# ftc run-test -b docker -t Functional.hello_world
> -# log_compare: pattern 'SUCCESS' found 1 times (expected greater or equal than 1)
> -
> -FROM debian:stretch-slim
> -MAINTAINER tim.bird@sony.com
> -
> -# ==============================================================================
> -# Proxy variables
> -# ==============================================================================
> -
> -ARG HTTP_PROXY
> -ENV http_proxy ${HTTP_PROXY}
> -ENV https_proxy ${HTTP_PROXY}
> -
> -# ==============================================================================
> -# Prepare basic image
> -# ==============================================================================
> -
> -ARG DEBIAN_FRONTEND=noninteractive
> -
> -WORKDIR /
> -RUN echo deb http://deb.debian.org/debian stretch main non-free > /etc/apt/sources.list
> -RUN echo deb http://security.debian.org/debian-security stretch/updates main >> /etc/apt/sources.list
> -RUN if [ -n "$HTTP_PROXY" ]; then echo 'Acquire::http::proxy "'$HTTP_PROXY'";' > /etc/apt/apt.conf.d/80proxy; fi
> -
> -# Fuego python dependencies
> -# - python-lxml: ftc, loggen
> -# - python-simplejson: ftc
> -# - python-yaml: ftc
> -# - python-openpyxl: ftc (also LTP)
> -# - python-requests: ftc (also fuego_release_test)
> -# - python-reportlab: ftc
> -# - python-parsedatetime: ftc
> -# - python-pexpect: ssh_exec (part of ttc)
> -# - python-pip: to install filelock, flake8
> -# - filelock: parser
> -RUN mkdir -p /usr/share/man/man1
> -RUN apt-get update -q=2 && apt-get -q=2 -V --no-install-recommends install \
> - python-lxml python-simplejson python-yaml python-openpyxl \
> - python-requests python-reportlab python-parsedatetime \
> - python-pexpect python-pip python-setuptools python-wheel
> -RUN pip install filelock
> -
> -# Fuego command dependencies
> -RUN apt-get -q=2 -V --no-install-recommends install \
> - git sshpass openssh-client sudo net-tools wget curl lava-tool \
> - bash-completion iproute2
> -
> -# Default SDK for testing locally or on an x86 board
> -RUN apt-get -q=2 -V --no-install-recommends install \
> - build-essential cmake bison flex automake kmod libtool \
> - libelf-dev libssl-dev libsdl1.2-dev libcairo2-dev libxmu-dev \
> - libxmuu-dev libglib2.0-dev libaio-dev pkg-config rsync u-boot-tools
> -
> -# Default test host dependencies
> -# - iperf iperf3 netperf: used as servers on the host
> -# - bzip2 bc: used for local tests by Functional.bzip2/bc
> -# - python-matplotlib: Benchmark.iperf3 parser
> -# - python-xmltodict: AGL tests
> -# - flake8: Functional.fuego_lint
> -# - netpipe-tcp - used by Benchmark.netpipe (provides the netpipe server)
> -# - iputils-ping - for /bin/ping command, used by some tests
> -# FIXTHIS: install dependencies dynamically on the tests that need them
> -RUN apt-get -q=2 -V --no-install-recommends install \
> - iperf iperf3 netperf bzip2 bc python-matplotlib python-xmltodict \
> - netpipe-tcp iputils-ping
> -RUN pip install flake8
> -
> -# miscelaneous packages:
> -# python-serial - used by serio
> -# diffstat and vim - used by Tim
> -# time - useful for timing command duration
> -RUN apt-get -q=2 -V --no-install-recommends install \
> - python-serial \
> - diffstat \
> - vim \
> - time
> -
> -# FIXTHIS: determine if these tools are really necessary
> -# RUN apt-get -q=2 -V --no-install-recommends install \
> -# apt-utils python-paramiko \
> -# xmlstarlet rsync \
> -# inotify-tools gettext netpipe-tcp \
> -# at minicom lzop bsdmainutils \
> -# mc netcat openssh-server
> -
> -# bsdmainutils provides hexdump, which is needed for netperf test
> -RUN apt-get -q=2 -V --no-install-recommends install \
> - bsdmainutils
> -
> -RUN /bin/bash -c 'echo "dash dash/sh boolean false" | debconf-set-selections ; dpkg-reconfigure dash'
> -RUN if [ -n "$HTTP_PROXY" ]; then echo "use_proxy = on" >> /etc/wgetrc; fi
> -RUN if [ -n "$HTTP_PROXY" ]; then echo -e "http_proxy=$HTTP_PROXY\nhttps_proxy=$HTTP_PROXY" >> /etc/environment; fi
> -
> -# ==============================================================================
> -# get ttc script and helpers
> -# ==============================================================================
> -RUN git clone https://github.com/tbird20d/ttc.git /usr/local/src/ttc
> -RUN /usr/local/src/ttc/install.sh /usr/local/bin
> -# as of ttc version 2.0.2, this should not be needed
> -#RUN perl -p -i -e "s#config_dir = \"/etc\"#config_dir = \"/fuego-ro/conf\"#" /usr/local/bin/ttc
> -
> -# ==============================================================================
> -# Serial Config
> -# ==============================================================================
> -
> -RUN /bin/bash -c 'git clone https://github.com/frowand/serio.git /usr/local/src/serio ; cp /usr/local/src/serio/serio /usr/local/bin/ ; ln -s
> /usr/local/bin/serio /usr/local/bin/sercp ; ln -s /usr/local/bin/serio /usr/local/bin/sersh'
> -
> -RUN /bin/bash -c 'git clone https://github.com/tbird20d/serlogin.git /usr/local/src/serlogin ; cp /usr/local/src/serlogin/serlogin
> /usr/local/bin'
> -
> -# ==============================================================================
> -# fserver
> -# ==============================================================================
> -
> -RUN /bin/bash -c 'git clone https://github.com/tbird20d/fserver.git /usr/local/lib/fserver ; ln -s /usr/local/lib/fserver/start_local_bg_server
> /usr/local/bin/start_local_bg_server'
> -
> -# ==============================================================================
> -# ftc post installation
> -# ==============================================================================
> -
> -RUN ln -s /fuego-core/scripts/ftc /usr/local/bin/
> -COPY fuego-core/scripts/ftc_completion.sh /etc/bash_completion.d/ftc
> -RUN echo ". /etc/bash_completion" >> /root/.bashrc
> -
> -# ==============================================================================
> -# Lava
> -# ==============================================================================
> -
> -RUN ln -s /fuego-ro/scripts/fuego-lava-target-setup /usr/local/bin
> -RUN ln -s /fuego-ro/scripts/fuego-lava-target-teardown /usr/local/bin
> -
> -# ==============================================================================
> -# Setup startup command
> -# ==============================================================================
> -
> -# FIXTHIS: when running multiple Fuego containers netperf will complain because the port is taken
> -ENTRYPOINT service netperf start && iperf3 -V -s -D -f M && /bin/bash
> diff --git a/README b/README
> index 882d84e..ce42e1e 100644
> --- a/README
> +++ b/README
> @@ -44,6 +44,29 @@ You can change the name of the docker image and the port used by Jenkins
> by passing the corresponding parameters to install.sh.
> ./install.sh fuego-8082 8082
>
> +To run fuego on different distribution versions like Debian(stretch, buster, etc.)
> + pass appropriate arguments to the install.sh script
> + ./install.sh <image name> <Jenkins port> <Distribution name> <suite>
should probably be:
./install.sh [[<image name>] [[<Jenkins port>] [<Distribution name> <suite>]]]
> +
> +By default install.sh script will install fuego on Debian stretch version
> + to install fuego on Debian buster version pass below options
> + ./install.sh fuego_buster 8090 debian buster
> +
> +[Note] It is currently verified on Debian versions (stretch and buster),
> +and yet to provide install scripts for other distributions
> +
> +Installing Fuego to a host system
> +-------------------------------------------------------------------------------
> +Run install-fuego.sh to install fuego on host system, by default it assume the
> +host system is Debian stretch
> + ./install-fuego.sh
> +
> + to install on Debian buster version
> + ./install-fuego.sh -d debian -s buster -p 8090
> +
> + to install without jenkins pass --nojenkins option to the script
> + ./install-fuego.sh --nojenkins
> +
> Running
> -------------------------------------------------------------------------------
> To start your Fuego container, issue the following command top directory:
> @@ -85,9 +108,9 @@ Updating the Jenkins version
> -------------------------------------------------------------------------------
>
> Modify the version and checksum
> - $ vi Dockerfile
> - ARG JENKINS_VERSION=2.32.1
> - ARG JENKINS_SHA=3cbe9b6d4d82c975642ea4409e21e087
> + $ vi install-debian.sh
> + JENKINS_VERSION=2.32.1
> + JENKINS_SHA=3cbe9b6d4d82c975642ea4409e21e087
>
> Re-build the flot plugin
> $ sudo apt-get install maven
> diff --git a/entrypoint.sh b/entrypoint.sh
> new file mode 100755
> index 0000000..6eac895
> --- /dev/null
> +++ b/entrypoint.sh
> @@ -0,0 +1,7 @@
> +#!/bin/bash
> +
> +[ "$1" != "--nojenkins" ] && service jenkins start
> +
> +#TODO: netperf not present in buster, remove this condition once alternative found
> +[ -f /etc/init.d/netperf ] && service netperf start
> +iperf3 -V -s -D -f M
> diff --git a/fuego-host-scripts/docker-build-image.sh b/fuego-host-scripts/docker-build-image.sh
> index 5823357..6889c47 100755
> --- a/fuego-host-scripts/docker-build-image.sh
> +++ b/fuego-host-scripts/docker-build-image.sh
> @@ -2,9 +2,10 @@
> # $1 - name for the docker image (default: fuego)
> # $2 - port for jenkins (default: 8090)
> # $3 - Dockerfile or Dockerfile.nojenkins
> +# $4 - Debian code name on which to install (default: stretch)
> #
> # Example:
> -# ./fuego-host-scripts/docker-build-image.sh --no-cache myfuegoimg 8082 Dockerfile.nojenkins
> +# ./fuego-host-scripts/docker-build-image.sh --no-cache myfuegoimg 8082 debian stretch --nojenkins
> #
> if [ "$1" = "--no-cache" ]; then
> NO_CACHE=--no-cache
> @@ -13,7 +14,10 @@ fi
>
> DOCKERIMAGE=${1:-fuego}
> JENKINS_PORT=${2:-8090}
> -DOCKERFILE=${3:-Dockerfile}
> +DISTRIBUTION=${3:-debian}
> +SUITE=${4:-stretch}
> +NO_JENKINS=${5:""}
> +DOCKERFILE="Dockerfile"
>
> if [ "$(id -u)" == "0" ]; then
> JENKINS_UID=$(id -u $SUDO_USER)
> @@ -27,4 +31,7 @@ echo "Using Port $JENKINS_PORT"
>
> sudo docker build ${NO_CACHE} -t ${DOCKERIMAGE} --build-arg HTTP_PROXY=$http_proxy \
> --build-arg uid=$JENKINS_UID --build-arg gid=$JENKINS_GID \
> + --build-arg DIST=$DISTRIBUTION \
> + --build-arg SUITE=$SUITE \
> + --build-arg NO_JENKINS=$NO_JENKINS \
> --build-arg JENKINS_PORT=$JENKINS_PORT -f $DOCKERFILE .
> diff --git a/install-buster.sh b/install-buster.sh
> new file mode 100755
> index 0000000..f4ce65b
> --- /dev/null
> +++ b/install-buster.sh
> @@ -0,0 +1,294 @@
> +#!/bin/bash
> +# 2019 (c) Toshiba corp. <daniel.sangorrin@toshiba.co.jp>
> +#
> +# Usage:
> +# $ sudo ./install-buster.sh [--help|-h] [--nojenkins] [<port>]
> +#
> +usage(){
> +
> +cat <<HERE
> +Usage: sudo ./install-buster.sh [--help|-h] [--nojenkins] [<port>]
> +
> +Install fuego on the host Debian filesystem.
> +
> +options:
> + --help Show usage help
> + --nojenkins Install Fuego without Jenkins
> +HERE
> +}
> +
> +if [[ $EUID -ne 0 ]]; then
> + echo "Sorry, you need root permissions"
> + exit 1
> +fi
> +
> +nojenkins=0
> +
> +POSITIONAL=()
> +while [[ $# -gt 0 ]] ; do
> + case $1 in
> + --nojenkins)
> + nojenkins=1
> + shift
> + ;;
> + --help | -h)
> + usage
> + exit 0
> + ;;
> + *)
> + POSITIONAL+=("$1") # save argument for later
> + shift
> + esac
> +done
> +set -- "${POSITIONAL[@]}" # restore positional arguments
> +
> +port=${2:-8090}
> +
> +# ==============================================================================
> +# Install Fuego dependencies
> +# ==============================================================================
> +
> +mkdir -p /usr/share/man/man1
> +
> +# netperf is in non-free
> +echo deb http://deb.debian.org/debian buster main non-free > /etc/apt/sources.list
> +echo deb http://security.debian.org/debian-security buster/updates main >> /etc/apt/sources.list
> +
> +apt-get update -q=2
> +
> +# Fuego python dependencies
> +apt-get -q=2 -V --no-install-recommends install \
> + python-lxml python-simplejson python-yaml python-openpyxl \
> + python-requests python-reportlab python-parsedatetime \
> + python-pexpect python-pip python-setuptools python-wheel
> +pip install filelock
> +
Can we put this stuff that is common between install-buster.sh and install-stretch.sh
into an "install-debian-common.sh"?
> +# Fuego command dependencies
> +apt-get -q=2 -V --no-install-recommends install \
> + git sshpass openssh-client sudo net-tools wget curl lava-tool \
> + bash-completion iproute2
> +
> +# Default SDK for testing locally or on an x86 board
> +apt-get -q=2 -V --no-install-recommends install \
> + build-essential cmake bison flex kmod automake libtool \
> + libelf-dev libssl-dev libsdl1.2-dev libcairo2-dev libxmu-dev \
> + libxmuu-dev libglib2.0-dev libaio-dev pkg-config rsync u-boot-tools
> +
> +# Default test host dependencies
> +apt-get -q=2 -V --no-install-recommends install \
> + iperf iperf3 bzip2 bc python-matplotlib python-xmltodict \
> + netpipe-tcp iputils-ping
> +pip install flake8
> +
> +# miscelaneous packages:
> +# python-serial - used by serio
> +# diffstat and vim - used by Tim
> +# time - useful for timing command duration
> +apt-get -q=2 -V --no-install-recommends install \
> + python-serial \
> + diffstat \
> + vim \
> + time
> +
> +# install packages used by NuttX SDK
> +apt-get -q=2 -V --no-install-recommends install \
> + genromfs
> +
> +# bsdmainutils provides hexdump, which is needed for netperf test
> +#apt-get -q=2 -V --no-install-recommends install \
> +# bsdmainutils
> +
> +echo "dash dash/sh boolean false" | debconf-set-selections
> +DEBIAN_FRONTEND=noninteractive dpkg-reconfigure dash
> +
> +# TODO: check if proxy settings can set outside docker container
> +if [ -n "$http_proxy" ]; then
> + sed -i -e 's/#use_proxy = on/use_proxy = on/g' /etc/wgetrc
> + echo -e "http_proxy=$http_proxy\nhttps_proxy=$https_proxy" >> /etc/environment
> +fi
This doesn't seem specific to Debian (/etc/wgetrc and /etc/environment).
Should this be moved to an 'install-common.sh' script?
> +
> +# ==============================================================================
> +# Install Jenkins
> +# ==============================================================================
> +
> +if [ $nojenkins -eq 0 ]; then
> + #JENKINS_VERSION=2.164.2
> + #JENKINS_SHA=4536f43f61b1fca6c58bd91040fa09304eea96ab
> + #JENKINS_VERSION=2.249.3
> + #JENKINS_SHA=534014c007edbb533a1833fe6f2dc115faf3faa2
> + JENKINS_VERSION=2.164.3
> + JENKINS_SHA=27f14f0ccca207660765205317295a5c29f4dd53
> + JENKINS_URL=https://pkg.jenkins.io/debian-stable/binary/jenkins_${JENKINS_VERSION}_all.deb
> + export JENKINS_UC=https://updates.jenkins.io
> + REF=/var/lib/jenkins/plugins
> + JENKINS_HOME=/var/lib/jenkins
> + JENKINS_PORT=$port
> +
> + # Jenkins dependencies
> + apt-get -q=2 -V --no-install-recommends install \
> + default-jdk daemon psmisc adduser procps unzip
> + pip install python-jenkins==1.4.0
> +
> + echo -e "JENKINS_PORT=$JENKINS_PORT" >> /etc/environment
> + export JENKINS_PORT=$port
> +
> + groupadd jenkins
> + useradd -l -m -d "${JENKINS_HOME}" -g jenkins -G sudo -s /bin/bash jenkins
> +
> + wget -nv ${JENKINS_URL}
> + echo "${JENKINS_SHA} jenkins_${JENKINS_VERSION}_all.deb" | sha1sum -c -
> +
> + # allow Jenkins to start and install plugins, as part of dpkg installation
> + printf "#!/bin/sh\nexit 0" > /usr/sbin/policy-rc.d
> + dpkg -i jenkins_${JENKINS_VERSION}_all.deb
> + rm jenkins_${JENKINS_VERSION}_all.deb
> +
> + # fix for Jenkins Java version check error
> + JAVA_VERSION_CHECK="\$\(\$JAVA -version 2\>\&1 \| sed -n \';s/.* version
> \"\\\(.*\\\)\\\.\\\(.*\\\)\\\..\*\\\..\*\".\*\/\\\1\\\2\/p;\'\)"
> + sed -i -e "s#^JAVA_VERSION=.*#JAVA_VERSION\=\"${JAVA_VERSION_CHECK}\"#g" /etc/init.d/jenkins
> +
> + source /etc/default/jenkins && \
> + JENKINS_ARGS="$JENKINS_ARGS --prefix=/fuego" && \
> + sed -i -e "s#JENKINS_ARGS.*#JENKINS_ARGS\=\"${JENKINS_ARGS}\"#g" /etc/default/jenkins
> +
> + source /etc/default/jenkins && \
> + JAVA_ARGS="$JAVA_ARGS -Djenkins.install.runSetupWizard=false -
> Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true" && \
> + if [ -n "$http_proxy" ]; then \
> + PROXYSERVER=$(echo $http_proxy | sed -E 's/^http://' | sed -E 's/\///g' | sed -E 's/(.*):(.*)/\1/') && \
> + PROXYPORT=$(echo $http_proxy | sed -E 's/^http://' | sed -E 's/\///g' | sed -E 's/(.*):(.*)/\2/') && \
> + JAVA_ARGS="$JAVA_ARGS -Dhttp.proxyHost="${PROXYSERVER}" -Dhttp.proxyPort="${PROXYPORT}" -
> Dhttps.proxyHost="${PROXYSERVER}" -Dhttps.proxyPort="${PROXYPORT}; \
> + fi && \
> + sed -i -e "s#^JAVA_ARGS.*#JAVA_ARGS\=\"${JAVA_ARGS}\"#g" /etc/default/jenkins;
> +
> + sed -i -e "s#8080#$JENKINS_PORT#g" /etc/default/Jenkins
The above looks like there's a lot of non-Debian-specific stuff intermingled with Debian-specific stuff.
I'm not sure what other distributions are intended to be supported, but wouldn't the user and
group manipulation be the same for those?
> +
> + # set up Jenkins plugins
> + cp frontend-install/install-plugins.sh \
> + frontend-install/jenkins-support \
> + frontend-install/clitest \
> + /usr/local/bin/
> +
> + mkdir -p $JENKINS_HOME/userContent/docs
> + cp frontend-install/config.xml $JENKINS_HOME/config.xml
> + ln -s /fuego-rw/logs $JENKINS_HOME/userContent/fuego.logs
> + cp docs/fuego-docs.pdf $JENKINS_HOME/userContent/docs/fuego-docs.pdf
> + cp frontend-install/jenkins.model.JenkinsLocationConfiguration.xml
> $JENKINS_HOME/jenkins.model.JenkinsLocationConfiguration.xml
> + sed -i -e "s#8080#$JENKINS_PORT#g" $JENKINS_HOME/jenkins.model.JenkinsLocationConfiguration.xml
> + chown -R jenkins:jenkins $JENKINS_HOME/
> +
> + # start and stop jenkins to pre-populate some settings
> + service jenkins start && \
> + sleep 30 && \
> + service jenkins stop
> +
> + # install other plugins from Jenkins update center
> + # NOTE: not sure all of these are needed, but keep list
> + # compatible with 1.2.1 release for now
> + #/usr/local/bin/install-plugins.sh \
> + # ant:1.9 \
> + # antisamy-markup-formatter:1.5 \
> + # bouncycastle-api:2.17 \
> + # command-launcher:1.3 \
> + # description-setter:1.10 \
> + # display-url-api:2.3.1 \
> + # external-monitor-job:1.7 \
> + # greenballs:1.15 \
> + # icon-shim:2.0.3 \
> + # javadoc:1.5 \
> + # jdk-tool:1.2 \
> + # junit:1.27 \
> + # ldap:1.20 \
> + # mailer:1.23 \
> + # matrix-auth:2.3 \
> + # matrix-project:1.14 \
> + # pam-auth:1.4 \
> + # pegdown-formatter:1.3 \
> + # structs:1.17 \
> + # windows-slaves:1.4
> +
> + # install plugins for latest LTS Jenkins (currently 2.249.3)
> + /usr/local/bin/install-plugins.sh \
> + script-security \
> + structs \
> + workflow-step-api \
> + workflow-api \
> + junit \
> + scm-api \
> + ant \
> + antisamy-markup-formatter \
> + bouncycastle-api \
> + command-launcher \
> + description-setter \
> + display-url-api \
> + external-monitor-job \
> + greenballs \
> + icon-shim \
> + javadoc \
> + jdk-tool \
> + mailer \
> + matrix-auth \
> + matrix-project \
> + pam-auth \
> + pegdown-formatter \
> + windows-slaves
> +
> + cp frontend-install/plugins/flot-plotter-plugin/flot.hpi $JENKINS_HOME/plugins/flot.jpi
> +
> + # make the mod.js symlink well after flot is installed
> + service jenkins start && sleep 30 && \
> + rm $JENKINS_HOME/plugins/flot/flot/mod.js && \
> + ln -s /fuego-core/scripts/mod.js $JENKINS_HOME/plugins/flot/flot/mod.js
> +
> + chown -R jenkins:jenkins $JENKINS_HOME/
> +fi
> +
> +# ==============================================================================
> +# get ttc script and helpers
> +# ==============================================================================
> +git clone https://github.com/tbird20d/ttc.git /usr/local/src/ttc
> +/usr/local/src/ttc/install.sh /usr/local/bin
> +perl -p -i -e "s#config_dir = \"/etc\"#config_dir = \"/fuego-ro/conf\"#" /usr/local/bin/ttc
> +
The above and following are definitely not unique to Debian, or a specific version of Debian.
They should be put into an "install-common.sh"
> +# ==============================================================================
> +# Serial Config
> +# ==============================================================================
> +if [ $nojenkins -eq 0 ]; then
> + /bin/bash -c 'git clone https://github.com/frowand/serio.git /usr/local/src/serio ; chown -R jenkins /usr/local/src/serio ; cp
> /usr/local/src/serio/serio /usr/local/bin/ ; ln -s /usr/local/bin/serio /usr/local/bin/sercp ; ln -s /usr/local/bin/serio /usr/local/bin/sersh'
> + /bin/bash -c 'git clone https://github.com/tbird20d/serlogin.git /usr/local/src/serlogin ; chown -R jenkins /usr/local/src/serlogin ;
> cp /usr/local/src/serlogin/serlogin /usr/local/bin'
> +else
> + /bin/bash -c 'git clone https://github.com/frowand/serio.git /usr/local/src/serio ; cp /usr/local/src/serio/serio /usr/local/bin/ ; ln -
> s /usr/local/bin/serio /usr/local/bin/sercp ; ln -s /usr/local/bin/serio /usr/local/bin/sersh'
> + /bin/bash -c 'git clone https://github.com/tbird20d/serlogin.git /usr/local/src/serlogin ; cp /usr/local/src/serlogin/serlogin
> /usr/local/bin'
> +fi
> +cp frontend-install/0001-Fix-host-parsing-for-serial-device-with-in-name.patch \
> + frontend-install/0002-Output-data-from-port-during-command-execution.patch \
> + /tmp/
> +/bin/bash -c 'patch -d /usr/local/src/serio -p1 </tmp/0001-Fix-host-parsing-for-serial-device-with-in-name.patch ; \
> + patch -d /usr/local/src/serio -p1 </tmp/0002-Output-data-from-port-during-command-execution.patch ;'
> +
> +
> +# ==============================================================================
> +# get ebf script and helpers
> +# ==============================================================================
> +apt-get -q=2 -V --no-install-recommends install jq
> +git clone https://github.com/TimesysGit/board-farm-rest-api.git /usr/local/src/board-farm-rest-api
> +cp /usr/local/src/board-farm-rest-api/cli/ebf /usr/local/bin/ebf
> +chmod a+x /usr/local/bin/ebf
> +
> +# ==============================================================================
> +# fserver
> +# ==============================================================================
> +git clone https://github.com/tbird20d/fserver.git /usr/local/lib/fserver
> +ln -s /usr/local/lib/fserver/start_local_bg_server /usr/local/bin/start_local_bg_server
> +
> +# ==============================================================================
> +# ftc post installation
> +# ==============================================================================
> +ln -s /fuego-core/scripts/ftc /usr/local/bin/
> +cp fuego-core/scripts/ftc_completion.sh /etc/bash_completion.d/ftc
> +echo ". /etc/bash_completion" >> /root/.bashrc
> +
> +# ==============================================================================
> +# Lava
> +# ==============================================================================
> +ln -s /fuego-ro/scripts/fuego-lava-target-setup /usr/local/bin
> +ln -s /fuego-ro/scripts/fuego-lava-target-teardown /usr/local/bin
> diff --git a/install-debian.sh b/install-debian.sh
> index ccca032..2d3ffd3 100755
> --- a/install-debian.sh
> +++ b/install-debian.sh
> @@ -2,102 +2,163 @@
> # 2019 (c) Toshiba corp. <daniel.sangorrin@toshiba.co.jp>
> #
> # Usage:
> -# $ sudo ./install-debian.sh [--help|-h] [--nojenkins] [<port>]
> +# $ sudo ./install-debian.sh [--help|-h] [-s <suite>] [-p <jenkins port>] [--nojenkins]
> #
> -if [ -n "$1" ]; then
> - if [ "$1" = "--help" -o "$1" = "-h" ]; then
> - cat <<HERE
> -Usage: sudo ./install-debian.sh [--help|-h] [--nojenkins] [<port>]
> -
> -Installs fuego on the host Debian filesystem.
> -
> -options:
> - --help Show usage help
> - --nojenkins Install Fuego without Jenkins
> -HERE
> - exit 0
> - fi
> -fi
> +
> +usage(){
> + echo ""
> + echo "Usage: sudo ./install-debian.sh [--help|-h] [-s <suite>] [-p <jenkins port>] [--nojenkins]"
> + echo "Install fuego on Debian filesystem."
> + echo ""
> + echo "options:"
> + echo " --help | -h Show usage help"
> + echo " --suite | -s <suite> Specify Debian suite"
> + echo " --port | -p <port> Jenkins port"
> + echo " --nojenkins Install Fuego without Jenkins"
> + echo ""
> +}
>
> if [[ $EUID -ne 0 ]]; then
> echo "Sorry, you need root permissions"
> exit 1
> fi
>
> -if [ "$1" = "--nojenkins" ]; then
> - nojenkins=1
> +nojenkins=0
> +port=8090
> +suite=stretch
> +user=jenkins
> +group=jenkins
> +uid=1000
> +gid=1000
> +
> +while [ "$1" != "" ]; do
> + case $1 in
> + -s | --suite )
> + shift
> + suite=$1
> + ;;
> + -p | --port )
> + shift
> + port=$1
> + ;;
> + -u | --uid )
> + shift
> + uid=$1
> + ;;
> + -g | --gid )
> + shift
> + gid=$1
> + ;;
> + --nojenkins )
> + nojenkins=1
> + ;;
> + -h | --help )
> + usage
> + exit 0
> + ;;
> + * )
> + break
> + esac
> shift
> -else
> - nojenkins=0
> - port=${1:-8090}
> -fi
> +done
>
> # ==============================================================================
> # Install Fuego dependencies
> # ==============================================================================
>
> # netperf is in non-free
> -echo deb http://deb.debian.org/debian stretch main non-free > /etc/apt/sources.list
> -echo deb http://security.debian.org/debian-security stretch/updates main >> /etc/apt/sources.list
> -
> -apt-get update -q=2
> +echo deb http://deb.debian.org/debian $suite main non-free > /etc/apt/sources.list
> +echo deb http://security.debian.org/debian-security $suite/updates main >> /etc/apt/sources.list
>
> # Fuego python dependencies
> -apt-get -q=2 -V --no-install-recommends install \
> +# - python-lxml: ftc, loggen
> +# - python-simplejson: ftc
> +# - python-yaml: ftc
> +# - python-openpyxl: ftc (also LTP)
> +# - python-requests: ftc (also fuego_release_test)
> +# - python-reportlab: ftc
> +# - python-parsedatetime: ftc
> +# - python-pexpect: ssh_exec (part of ttc)
> +# - python-pip: to install filelock, flake8
> +# - filelock: parser
> +mkdir -p /usr/share/man/man1
> +apt-get update -q=2 && apt-get -q=2 -V --no-install-recommends install \
> python-lxml python-simplejson python-yaml python-openpyxl \
> python-requests python-reportlab python-parsedatetime \
> - python-pip
> + python-pexpect python-pip python-setuptools python-wheel
> pip install filelock
>
> # Fuego command dependencies
> apt-get -q=2 -V --no-install-recommends install \
> git sshpass openssh-client sudo net-tools wget curl lava-tool \
> - bash-completion
> + bash-completion iproute2
>
> # Default SDK for testing locally or on an x86 board
> apt-get -q=2 -V --no-install-recommends install \
> build-essential cmake bison flex automake kmod libtool \
> libelf-dev libssl-dev libsdl1.2-dev libcairo2-dev libxmu-dev \
> - libxmuu-dev libglib2.0-dev libaio-dev u-boot-tools pkg-config
> + libxmuu-dev libglib2.0-dev libaio-dev pkg-config rsync u-boot-tools
>
> # Default test host dependencies
> +# - iperf iperf3 netperf: used as servers on the host
> +# - bzip2 bc: used for local tests by Functional.bzip2/bc
> +# - python-matplotlib: Benchmark.iperf3 parser
> +# - python-xmltodict: AGL tests
> +# - flake8: Functional.fuego_lint
> +# - netpipe-tcp - used by Benchmark.netpipe (provides the netpipe server)
> +# - iputils-ping - for /bin/ping command, used by some tests
> +# FIXTHIS: install dependencies dynamically on the tests that need them
> apt-get -q=2 -V --no-install-recommends install \
> - iperf iperf3 netperf bzip2 bc python-matplotlib python-xmltodict
> -pip install flake8 bsdmainutils
> + iperf iperf3 bzip2 bc python-matplotlib python-xmltodict \
> + netpipe-tcp iputils-ping
>
> -echo "dash dash/sh boolean false" | debconf-set-selections
> -DEBIAN_FRONTEND=noninteractive dpkg-reconfigure dash
> -if [ -n "$http_proxy" ]; then
> - sed -i -e 's/#use_proxy = on/use_proxy = on/g' /etc/wgetrc
> - echo -e "http_proxy=$http_proxy\nhttps_proxy=$https_proxy" >> /etc/environment
> +#TODO: find alternate package for netperf in buster version
> +if [ $suite != "buster" ];then
> + apt-get -q=2 -V --no-install-recommends install netperf
> fi
> +pip install flake8
>
> -# ==============================================================================
> -# Clone fuego and fuego-core
> -# ==============================================================================
> -if [ ! -d "/fuego" ]; then
> - cd /
> - git clone --branch next --depth=1 https://bitbucket.org/nirrognas/fuego.git
> - ln -s /fuego/fuego-ro /fuego-ro
> - ln -s /fuego/fuego-rw /fuego-rw
> - cd fuego
> - git clone --branch next --depth=1 https://bitbucket.org/nirrognas/fuego-core.git
> - ln -s /fuego/fuego-core /fuego-core
> -fi
> +# miscelaneous packages:
> +# python-serial - used by serio
> +# diffstat and vim - used by Tim
> +# time - useful for timing command duration
> +apt-get -q=2 -V --no-install-recommends install \
> + python-serial \
> + diffstat \
> + vim \
> + time
> +
> +# install packages used by NuttX SDK
> +apt-get -q=2 -V --no-install-recommends install \
> + genromfs
> +
> +# FIXTHIS: determine if these tools are really necessary
> +#RUN apt-get -q=2 -V --no-install-recommends install \
> +# apt-utils python-paramiko \
> +# xmlstarlet rsync \
> +# inotify-tools gettext netpipe-tcp \
> +# at minicom lzop bsdmainutils \
> +# mc netcat openssh-server
> +
> +# bsdmainutils provides hexdump, which is needed for netperf test
> +apt-get -q=2 -V --no-install-recommends install \
> + bsdmainutils
> +
> +echo "dash dash/sh boolean false" | debconf-set-selections
> +DEBIAN_FRONTEND=noninteractive dpkg-reconfigure dash
>
> # ==============================================================================
> # Install Jenkins
> # ==============================================================================
> -
> if [ $nojenkins -eq 0 ]; then
> #JENKINS_VERSION=2.164.2
> #JENKINS_SHA=4536f43f61b1fca6c58bd91040fa09304eea96ab
> JENKINS_VERSION=2.249.3
> JENKINS_SHA=534014c007edbb533a1833fe6f2dc115faf3faa2
> JENKINS_URL=https://pkg.jenkins.io/debian-stable/binary/jenkins_${JENKINS_VERSION}_all.deb
> - JENKINS_UC=https://updates.jenkins.io
> - REF=/var/lib/jenkins/plugins
> - JENKINS_HOME=/var/lib/jenkins
> + export JENKINS_UC=https://updates.jenkins.io
> + export REF=/var/lib/jenkins/plugins
> + export JENKINS_HOME=/var/lib/jenkins
> JENKINS_PORT=$port
>
> # Jenkins dependencies
> @@ -106,10 +167,12 @@ if [ $nojenkins -eq 0 ]; then
> pip install python-jenkins==1.4.0
>
> echo -e "JENKINS_PORT=$JENKINS_PORT" >> /etc/environment
> - groupadd jenkins
> - useradd -l -m -d "${JENKINS_HOME}" -g jenkins -G sudo -s /bin/bash jenkins
> - cd
> - wget ${JENKINS_URL}
> + export JENKINS_PORT=$port
> +
> + getent group ${gid} >/dev/null || groupadd -g ${gid} ${group}
> + useradd -l -m -d "${JENKINS_HOME}" -u ${uid} -g ${gid} -G sudo -s /bin/bash ${user}
> +
> + wget -nv ${JENKINS_URL}
> echo "${JENKINS_SHA} jenkins_${JENKINS_VERSION}_all.deb" | sha1sum -c -
>
> # allow Jenkins to start and install plugins, as part of dpkg installation
> @@ -117,11 +180,60 @@ if [ $nojenkins -eq 0 ]; then
> dpkg -i jenkins_${JENKINS_VERSION}_all.deb
> rm jenkins_${JENKINS_VERSION}_all.deb
>
> - # update ownership
> - chown -R jenkins:jenkins /fuego
> - chown -R jenkins:jenkins /fuego-ro
> - chown -R jenkins:jenkins /fuego-rw
> - chown -R jenkins:jenkins /fuego-core
> +fi
> +
> +# ==============================================================================
> +# get ttc script and helpers
> +# ==============================================================================
> +git clone https://github.com/tbird20d/ttc.git /usr/local/src/ttc
> +/usr/local/src/ttc/install.sh /usr/local/bin
> +# as of ttc version 2.0.2, this should not be needed
> +#perl -p -i -e "s#config_dir = \"/etc\"#config_dir = \"/fuego-ro/conf\"#" /usr/local/bin/ttc
> +
> +# ==============================================================================
> +# get ebf script and helpers
> +# ==============================================================================
> +if [ $nojenkins -eq 0 ]; then
> + apt-get -q=2 -V --no-install-recommends install jq
> + git clone https://github.com/TimesysGit/board-farm-rest-api.git /usr/local/src/board-farm-rest-api
> + cp /usr/local/src/board-farm-rest-api/cli/ebf /usr/local/bin/ebf
> + chmod a+x /usr/local/bin/ebf
> +fi
> +
> +# ==============================================================================
> +# Serial Config
> +# ==============================================================================
> +git clone https://github.com/frowand/serio.git /usr/local/src/serio
> +if [ $nojenkins -eq 0 ]; then
> + cp frontend-install/0001-Fix-host-parsing-for-serial-device-with-in-name.patch \
> + frontend-install/0002-Output-data-from-port-during-command-execution.patch \
> + /tmp/
> + patch -d /usr/local/src/serio -p1 </tmp/0001-Fix-host-parsing-for-serial-device-with-in-name.patch
> + patch -d /usr/local/src/serio -p1 </tmp/0002-Output-data-from-port-during-command-execution.patch
> + chown -R jenkins /usr/local/src/serio
> +fi
> +cp /usr/local/src/serio/serio /usr/local/bin/
> +ln -s /usr/local/bin/serio /usr/local/bin/sercp
> +ln -s /usr/local/bin/serio /usr/local/bin/sersh
> +
> +git clone https://github.com/tbird20d/serlogin.git /usr/local/src/serlogin
> +[ $nojenkins -eq 0 ] && chown -R jenkins /usr/local/src/serlogin
> +cp /usr/local/src/serlogin/serlogin /usr/local/bin
> +
> +# ==============================================================================
> +# fserver
> +# ==============================================================================
> +git clone https://github.com/tbird20d/fserver.git /usr/local/lib/fserver
> +ln -s /usr/local/lib/fserver/start_local_bg_server /usr/local/bin/start_local_bg_server
> +
> +# ==============================================================================
> +# Jenkins post installation
> +# ==============================================================================
> +if [ $nojenkins -eq 0 ]; then
> +
> + # Fix jenkins issue, Java version check fails when java version is 11
> + JAVA_VERSION_CHECK="\$\(\$JAVA -version 2\>\&1 \| sed -n \';s/.* version \"\\\(\[^.-\]*\\\)\\\.\\\(\[^.-
> \]*\\\).*\\\".*/\\\1\\\2\/p;')"
> + sed -i -e "s#^JAVA_VERSION=.*#JAVA_VERSION\=\"${JAVA_VERSION_CHECK}\"#g" /etc/init.d/jenkins
>
> source /etc/default/jenkins && \
> JENKINS_ARGS="$JENKINS_ARGS --prefix=/fuego" && \
> @@ -138,116 +250,107 @@ if [ $nojenkins -eq 0 ]; then
>
> sed -i -e "s#8080#$JENKINS_PORT#g" /etc/default/jenkins
>
> - cp /fuego/frontend-install/install-plugins.sh \
> - /fuego/frontend-install/jenkins-support \
> - /fuego/frontend-install/clitest \
> + # set up Jenkins plugins
> + cp frontend-install/install-plugins.sh \
> + frontend-install/jenkins-support \
> + frontend-install/clitest \
> /usr/local/bin/
>
> - cp /fuego/frontend-install/config.xml $JENKINS_HOME/config.xml
> - ln -s /fuego-rw/logs $JENKINS_HOME/userContent/fuego.logs
> - mkdir $JENKINS_HOME/userContent/docs
> - cp /fuego/docs/fuego-docs.pdf $JENKINS_HOME/userContent/docs/fuego-docs.pdf
> - jenkins cp /fuego/frontend-install/jenkins.model.JenkinsLocationConfiguration.xml
> $JENKINS_HOME/jenkins.model.JenkinsLocationConfiguration.xml
> - sed -i -e "s#8080#$JENKINS_PORT#g" $JENKINS_HOME/jenkins.model.JenkinsLocationConfiguration.xml
> - chown -R jenkins:jenkins $JENKINS_HOME/
> -
> # start and stop jenkins to pre-populate some settings
> service jenkins start && \
> sleep 30 && \
> service jenkins stop
>
> + # install plugins - these versions are for Jenkins version 2.164.2
> + # Explicitly install script-security v1.68, otherwise
> + # v1.74 will automatically be installed as a dependency of
> + # the junit plugin. Make sure to install before junit plugin.
> + # Explicitly install junit:1.27
> +
> # install other plugins from Jenkins update center
> # NOTE: not sure all of these are needed, but keep list
> - # compatible with 1.2.1 release for now
> - #/usr/local/bin/install-plugins.sh \
> - # ant:1.9 \
> - # antisamy-markup-formatter:1.5 \
> - # bouncycastle-api:2.17 \
> - # command-launcher:1.3 \
> - # description-setter:1.10 \
> - # display-url-api:2.3.1 \
> - # external-monitor-job:1.7 \
> - # greenballs:1.15 \
> - # icon-shim:2.0.3 \
> - # javadoc:1.5 \
> - # jdk-tool:1.2 \
> - # junit:1.27 \
> - # ldap:1.20 \
> - # mailer:1.23 \
> - # matrix-auth:2.3 \
> - # matrix-project:1.14 \
> - # pam-auth:1.4 \
> - # pegdown-formatter:1.3 \
> - # structs:1.17 \
> - # windows-slaves:1.4
> -
> - # install plugins for latest LTS Jenkins (currently 2.249.3)
> - /usr/local/bin/install-plugins.sh \
> - script-security \
> - structs \
> - workflow-step-api \
> - workflow-api \
> - junit \
> - scm-api \
> - ant \
> - antisamy-markup-formatter \
> - bouncycastle-api \
> - command-launcher \
> - description-setter \
> - display-url-api \
> - external-monitor-job \
> - greenballs \
> - icon-shim \
> - javadoc \
> - jdk-tool \
> - mailer \
> - matrix-auth \
> - matrix-project \
> - pam-auth \
> - pegdown-formatter \
> - windows-slaves
> -
> - cp frontend-install/plugins/flot-plotter-plugin/flot.hpi $JENKINS_HOME/plugins/flot.jpi
> + # as compatible as possible with 1.2.1 release for now
> + # Do NOT change the order of the plugins, unless you know what
> + # you are doing. Otherwise, the install-plugins.sh script will
> + # install the wrong versions of other plugins based on dependency
> + # information in the plugin files.
>
> + #/usr/local/bin/install-plugins.sh \
> + # script-security:1.68 \
> + # structs:1.20 \
> + # workflow-step-api:2.22 \
> + # workflow-api:2.40 \
> + # junit:1.27 \
> + # scm-api:2.6.3 \
> + # ant:1.9 \
> + # antisamy-markup-formatter:1.5 \
> + # bouncycastle-api:2.17 \
> + # command-launcher:1.3 \
> + # description-setter:1.10 \
> + # display-url-api:2.3.1 \
> + # external-monitor-job:1.7 \
> + # greenballs:1.15 \
> + # icon-shim:2.0.3 \
> + # javadoc:1.5 \
> + # jdk-tool:1.2 \
> + # ldap:1.20 \
> + # mailer:1.23 \
> + # matrix-auth:2.3 \
> + # matrix-project:1.14 \
> + # pam-auth:1.5 \
> + # pegdown-formatter:1.3 \
> + # windows-slaves:1.4
> +
> + # install plugins for latest LTS Jenkins (currently 2.249.3)
> + /usr/local/bin/install-plugins.sh \
> + script-security \
> + structs \
> + workflow-step-api \
> + workflow-api \
> + junit \
> + scm-api \
> + ant \
> + antisamy-markup-formatter \
> + bouncycastle-api \
> + command-launcher \
> + description-setter \
> + display-url-api \
> + external-monitor-job \
> + greenballs \
> + icon-shim \
> + javadoc \
> + jdk-tool \
> + mailer \
> + matrix-auth \
> + matrix-project \
> + pam-auth \
> + pegdown-formatter \
> + windows-slaves
> +
> + cp frontend-install/plugins/flot-plotter-plugin/flot.hpi $JENKINS_HOME/plugins/flot.jpi
> +
> + # jenkins should automatically unzip any plugins in the plugin dir
> # make the mod.js symlink well after flot is installed
> service jenkins start && sleep 30 && \
> rm $JENKINS_HOME/plugins/flot/flot/mod.js && \
> ln -s /fuego-core/scripts/mod.js $JENKINS_HOME/plugins/flot/flot/mod.js
>
> - chown -R jenkins:jenkins $JENKINS_HOME/
> -else
> - sed -i -e 's/jenkins_enabled=1/jenkins_enabled=0/g' /fuego-ro/conf/fuego.conf
> -fi
> + mkdir -p $JENKINS_HOME/userContent/docs
> + ln -s /fuego-rw/logs $JENKINS_HOME/userContent/fuego.logs
> + cp docs/fuego-docs.pdf $JENKINS_HOME/userContent/docs/fuego-docs.pdf
>
> -# ==============================================================================
> -# get ttc script and helpers
> -# ==============================================================================
> -git clone https://github.com/tbird20d/ttc.git /usr/local/src/ttc
> -/usr/local/src/ttc/install.sh /usr/local/bin
> -perl -p -i -e "s#config_dir = \"/etc\"#config_dir = \"/fuego-ro/conf\"#" /usr/local/bin/ttc
> + cp frontend-install/config.xml $JENKINS_HOME/config.xml
> + cp frontend-install/jenkins.model.JenkinsLocationConfiguration.xml
> $JENKINS_HOME/jenkins.model.JenkinsLocationConfiguration.xml
> + sed -i -e "s#8080#$JENKINS_PORT#g" $JENKINS_HOME/jenkins.model.JenkinsLocationConfiguration.xml
>
> -# ==============================================================================
> -# Serial Config
> -# ==============================================================================
> -if [ $nojenkins -eq 0 ]; then
> - /bin/bash -c 'git clone https://github.com/frowand/serio.git /usr/local/src/serio ; chown -R jenkins /usr/local/src/serio ; cp
> /usr/local/src/serio/serio /usr/local/bin/ ; ln -s /usr/local/bin/serio /usr/local/bin/sercp ; ln -s /usr/local/bin/serio /usr/local/bin/sersh'
> - /bin/bash -c 'git clone https://github.com/tbird20d/serlogin.git /usr/local/src/serlogin ; chown -R jenkins /usr/local/src/serlogin ;
> cp /usr/local/src/serlogin/serlogin /usr/local/bin'
> -else
> - /bin/bash -c 'git clone https://github.com/frowand/serio.git /usr/local/src/serio ; cp /usr/local/src/serio/serio /usr/local/bin/ ; ln -
> s /usr/local/bin/serio /usr/local/bin/sercp ; ln -s /usr/local/bin/serio /usr/local/bin/sersh'
> - /bin/bash -c 'git clone https://github.com/tbird20d/serlogin.git /usr/local/src/serlogin ; cp /usr/local/src/serlogin/serlogin
> /usr/local/bin'
> + chown -R jenkins:jenkins $JENKINS_HOME/
> fi
>
> -# ==============================================================================
> -# fserver
> -# ==============================================================================
> -git clone https://github.com/tbird20d/fserver.git /usr/local/lib/fserver
> -ln -s /usr/local/lib/fserver/start_local_bg_server /usr/local/bin/start_local_bg_server
> -
> # ==============================================================================
> # ftc post installation
> # ==============================================================================
> ln -s /fuego-core/scripts/ftc /usr/local/bin/
> -cp /fuego-core/scripts/ftc_completion.sh /etc/bash_completion.d/ftc
> +cp fuego-core/scripts/ftc_completion.sh /etc/bash_completion.d/ftc
> echo ". /etc/bash_completion" >> /root/.bashrc
>
> # ==============================================================================
> @@ -255,14 +358,3 @@ echo ". /etc/bash_completion" >> /root/.bashrc
> # ==============================================================================
> ln -s /fuego-ro/scripts/fuego-lava-target-setup /usr/local/bin
> ln -s /fuego-ro/scripts/fuego-lava-target-teardown /usr/local/bin
> -
> -# ==============================================================================
> -# Small guide
> -# ==============================================================================
> -echo "Run 'service netperf start' to start a netperf server"
> -echo "Run 'iperf3 -V -s -D -f M' to start an iperf3 server"
> -echo "Run 'ftc list-boards' to see the available boards"
> -echo "Run 'ftc list-tests' to see the available tests"
> -echo "Run 'ftc run-test -b local -t Functional.hello_world' to run a hello world"
> -echo "Run 'ftc run-test -b local -t Benchmark.Dhrystone -s 500M' to run Dhrystone"
> -echo "Run 'ftc gen-report' to get results"
> diff --git a/install-fuego.sh b/install-fuego.sh
> new file mode 100755
> index 0000000..7b3e010
> --- /dev/null
> +++ b/install-fuego.sh
> @@ -0,0 +1,109 @@
> +#!/bin/bash
> +# 2019 (c) Toshiba corp. <daniel.sangorrin@toshiba.co.jp>
> +#
> +# Usage:
> +# $ sudo ./install-fuego.sh [--help|-h] [-d <distribution>] [-s <suite>] [-p <jenkins port>] [--nojenkins]
> +#
> +
> +usage(){
> + echo ""
> + echo "Usage: sudo ./install-fuego.sh [--help|-h] [-d <distribution>] [-s <suite>] [-p <jenkins port>] [--nojenkins]"
> + echo "Install fuego on Debian filesystem."
> + echo ""
> + echo "options:"
> + echo " --help | -h Show usage help"
> + echo " --dist | -d <distribution> Specify distribution"
> + echo " --suite | -s <suite> Specify suite"
> + echo " --port | -p <port> Jenkins port"
> + echo " --nojenkins Install Fuego without Jenkins"
> + echo ""
> +}
> +
> +if [[ $EUID -ne 0 ]]; then
> + echo "Sorry, you need root permissions"
> + exit 1
> +fi
> +
> +nojenkins=
> +port=8090
> +dist=debian
> +suite=stretch
> +fuego_dir=$(pwd)
> +
> +while [ "$1" != "" ]; do
> + case $1 in
> + -d | --dist )
> + shift
> + dist=$1
> + ;;
> + -s | --suite )
> + shift
> + suite=$1
> + ;;
> + -p | --port )
> + shift
> + port=$1
> + ;;
> + --nojenkins )
> + nojenkins="--nojenkins"
> + ;;
> + -h | --help )
> + usage
> + exit 0
> + ;;
> + * )
> + break
> + esac
> + shift
> +done
> +
> +# ==============================================================================
> +# Clone fuego-core
> +# ==============================================================================
> +if [ ! -f "fuego-core/scripts/ftc" ]; then
> + # set fuego-core branch to same as current fuego branch
> + # get current git branch
> + set -o noglob
> + while IFS=" " read -r part1 part2 ; do
> + if [ $part1 = "*" ] ; then
> + branch=$part2
> + fi
> + done < <(git branch)
> + set +o noglob
> + git clone -b $branch https://bitbucket.org/fuegotest/fuego-core.git
> +fi
> +
> +# ==============================================================================
> +# Install fuego on host system
> +# ==============================================================================
> +./install-$dist.sh -s $suite -p $port $nojenkins
> +
> +systemctl daemon-reload
> +service jenkins restart
> +
> +# provide permission to current user
> +usermod -aG jenkins $USER
> +chmod -R 775 $fuego_dir/fuego-rw
> +
> +chown -R jenkins:jenkins $fuego_dir/fuego-ro
> +chown -R jenkins:jenkins $fuego_dir/fuego-rw
> +chown -R jenkins:jenkins $fuego_dir/fuego-core
> +
> +# ==============================================================================
> +# Create Links to fuego folders
> +# ==============================================================================
> +ln -s $fuego_dir/fuego-ro /
> +ln -s $fuego_dir/fuego-rw /
> +ln -s $fuego_dir/fuego-core /
> +
> +# ==============================================================================
> +# Small guide
> +# ==============================================================================
> +echo "Logout and Login current user session to work properly"
> +echo "Run 'service netperf start' to start a netperf server"
> +echo "Run 'iperf3 -V -s -D -f M' to start an iperf3 server"
> +echo "Run 'ftc list-boards' to see the available boards"
> +echo "Run 'ftc list-tests' to see the available tests"
> +echo "Run 'ftc run-test -b local -t Functional.hello_world' to run a hello world"
> +echo "Run 'ftc run-test -b local -t Benchmark.Dhrystone -s 500M' to run Dhrystone"
> +echo "Run 'ftc gen-report' to get results"
> diff --git a/install-stretch.sh b/install-stretch.sh
> new file mode 100755
> index 0000000..4f6f724
> --- /dev/null
> +++ b/install-stretch.sh
> @@ -0,0 +1,288 @@
> +#!/bin/bash
> +# 2019 (c) Toshiba corp. <daniel.sangorrin@toshiba.co.jp>
> +#
> +# Usage:
> +# $ sudo ./install-<debian-suit>.sh [--help|-h] [--nojenkins] [<port>]
> +#
> +usage(){
> +
> +cat <<HERE
> +Usage: sudo ./install-<debian-suit>.sh [--help|-h] [--nojenkins] [<port>]
> +
> +Install fuego on the host Debian filesystem.
> +
> +options:
> + --help Show usage help
> + --nojenkins Install Fuego without Jenkins
> +HERE
> +}
> +
> +if [[ $EUID -ne 0 ]]; then
> + echo "Sorry, you need root permissions"
> + exit 1
> +fi
> +
> +nojenkins=0
> +
> +POSITIONAL=()
> +while [[ $# -gt 0 ]] ; do
> + case $1 in
> + --nojenkins)
> + nojenkins=1
> + shift
> + ;;
> + --help | -h)
> + usage
> + exit 0
> + ;;
> + *)
> + POSITIONAL+=("$1") # save argument for later
> + shift
> + esac
> +done
> +set -- "${POSITIONAL[@]}" # restore positional arguments
> +
> +port=${2:-8090}
> +
> +# ==============================================================================
> +# Install Fuego dependencies
> +# ==============================================================================
> +
> +mkdir -p /usr/share/man/man1
> +
> +# netperf is in non-free
> +echo deb http://deb.debian.org/debian stretch main non-free > /etc/apt/sources.list
> +echo deb http://security.debian.org/debian-security stretch/updates main >> /etc/apt/sources.list
> +
> +apt-get update -q=2
> +
> +# Fuego python dependencies
> +apt-get -q=2 -V --no-install-recommends install \
> + python-lxml python-simplejson python-yaml python-openpyxl \
> + python-requests python-reportlab python-parsedatetime \
> + python-pexpect python-pip python-setuptools python-wheel
> +pip install filelock
> +
> +# Fuego command dependencies
> +apt-get -q=2 -V --no-install-recommends install \
> + git sshpass openssh-client sudo net-tools wget curl lava-tool \
> + bash-completion iproute2
> +
> +# Default SDK for testing locally or on an x86 board
> +apt-get -q=2 -V --no-install-recommends install \
> + build-essential cmake bison flex kmod automake libtool \
> + libelf-dev libssl-dev libsdl1.2-dev libcairo2-dev libxmu-dev \
> + libxmuu-dev libglib2.0-dev libaio-dev pkg-config rsync u-boot-tools
> +
> +# Default test host dependencies
> +apt-get -q=2 -V --no-install-recommends install \
> + iperf iperf3 netperf bzip2 bc python-matplotlib python-xmltodict \
> + netpipe-tcp iputils-ping
> +pip install flake8
> +
> +# miscelaneous packages:
> +# python-serial - used by serio
> +# diffstat and vim - used by Tim
> +# time - useful for timing command duration
> +apt-get -q=2 -V --no-install-recommends install \
> + python-serial \
> + diffstat \
> + vim \
> + time
> +
> +# install packages used by NuttX SDK
> +apt-get -q=2 -V --no-install-recommends install \
> + genromfs
> +
> +# bsdmainutils provides hexdump, which is needed for netperf test
> +apt-get -q=2 -V --no-install-recommends install \
> + bsdmainutils
> +
> +echo "dash dash/sh boolean false" | debconf-set-selections
> +DEBIAN_FRONTEND=noninteractive dpkg-reconfigure dash
> +
> +# TODO: check if proxy settings can set outside docker container
> +#if [ -n "$http_proxy" ]; then
> +# sed -i -e 's/#use_proxy = on/use_proxy = on/g' /etc/wgetrc
> +# echo -e "http_proxy=$http_proxy\nhttps_proxy=$https_proxy" >> /etc/environment
> +#fi
> +
> +# ==============================================================================
> +# Install Jenkins
> +# ==============================================================================
> +
> +if [ $nojenkins -eq 0 ]; then
> + #JENKINS_VERSION=2.164.2
> + #JENKINS_SHA=4536f43f61b1fca6c58bd91040fa09304eea96ab
> + JENKINS_VERSION=2.249.3
> + JENKINS_SHA=534014c007edbb533a1833fe6f2dc115faf3faa2
> + JENKINS_URL=https://pkg.jenkins.io/debian-stable/binary/jenkins_${JENKINS_VERSION}_all.deb
> + export JENKINS_UC=https://updates.jenkins.io
> + REF=/var/lib/jenkins/plugins
> + JENKINS_HOME=/var/lib/jenkins
> + JENKINS_PORT=$port
> +
> + # Jenkins dependencies
> + apt-get -q=2 -V --no-install-recommends install \
> + default-jdk daemon psmisc adduser procps unzip
> + pip install python-jenkins==1.4.0
> +
> + echo -e "JENKINS_PORT=$JENKINS_PORT" >> /etc/environment
> + export JENKINS_PORT=$port
> +
> + groupadd jenkins
> + useradd -l -m -d "${JENKINS_HOME}" -g jenkins -G sudo -s /bin/bash jenkins
> +
> + wget -nv ${JENKINS_URL}
> + echo "${JENKINS_SHA} jenkins_${JENKINS_VERSION}_all.deb" | sha1sum -c -
> +
> + # allow Jenkins to start and install plugins, as part of dpkg installation
> + printf "#!/bin/sh\nexit 0" > /usr/sbin/policy-rc.d
> + dpkg -i jenkins_${JENKINS_VERSION}_all.deb
> + rm jenkins_${JENKINS_VERSION}_all.deb
> +
> + source /etc/default/jenkins && \
> + JENKINS_ARGS="$JENKINS_ARGS --prefix=/fuego" && \
> + sed -i -e "s#JENKINS_ARGS.*#JENKINS_ARGS\=\"${JENKINS_ARGS}\"#g" /etc/default/jenkins
> +
> + source /etc/default/jenkins && \
> + JAVA_ARGS="$JAVA_ARGS -Djenkins.install.runSetupWizard=false -
> Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true" && \
> + if [ -n "$http_proxy" ]; then \
> + PROXYSERVER=$(echo $http_proxy | sed -E 's/^http://' | sed -E 's/\///g' | sed -E 's/(.*):(.*)/\1/') && \
> + PROXYPORT=$(echo $http_proxy | sed -E 's/^http://' | sed -E 's/\///g' | sed -E 's/(.*):(.*)/\2/') && \
> + JAVA_ARGS="$JAVA_ARGS -Dhttp.proxyHost="${PROXYSERVER}" -Dhttp.proxyPort="${PROXYPORT}" -
> Dhttps.proxyHost="${PROXYSERVER}" -Dhttps.proxyPort="${PROXYPORT}; \
> + fi && \
> + sed -i -e "s#^JAVA_ARGS.*#JAVA_ARGS\=\"${JAVA_ARGS}\"#g" /etc/default/jenkins;
> +
> + sed -i -e "s#8080#$JENKINS_PORT#g" /etc/default/jenkins
> +
> + # set up Jenkins plugins
> + cp frontend-install/install-plugins.sh \
> + frontend-install/jenkins-support \
> + frontend-install/clitest \
> + /usr/local/bin/
> +
> + mkdir -p $JENKINS_HOME/userContent/docs
> + cp frontend-install/config.xml $JENKINS_HOME/config.xml
> + ln -s /fuego-rw/logs $JENKINS_HOME/userContent/fuego.logs
> + cp docs/fuego-docs.pdf $JENKINS_HOME/userContent/docs/fuego-docs.pdf
> + cp frontend-install/jenkins.model.JenkinsLocationConfiguration.xml
> $JENKINS_HOME/jenkins.model.JenkinsLocationConfiguration.xml
> + sed -i -e "s#8080#$JENKINS_PORT#g" $JENKINS_HOME/jenkins.model.JenkinsLocationConfiguration.xml
> + chown -R jenkins:jenkins $JENKINS_HOME/
> +
> + # start and stop jenkins to pre-populate some settings
> + service jenkins start && \
> + sleep 30 && \
> + service jenkins stop
> +
> + # install other plugins from Jenkins update center
> + # NOTE: not sure all of these are needed, but keep list
> + # compatible with 1.2.1 release for now
> + #/usr/local/bin/install-plugins.sh \
> + # ant:1.9 \
> + # antisamy-markup-formatter:1.5 \
> + # bouncycastle-api:2.17 \
> + # command-launcher:1.3 \
> + # description-setter:1.10 \
> + # display-url-api:2.3.1 \
> + # external-monitor-job:1.7 \
> + # greenballs:1.15 \
> + # icon-shim:2.0.3 \
> + # javadoc:1.5 \
> + # jdk-tool:1.2 \
> + # junit:1.27 \
> + # ldap:1.20 \
> + # mailer:1.23 \
> + # matrix-auth:2.3 \
> + # matrix-project:1.14 \
> + # pam-auth:1.4 \
> + # pegdown-formatter:1.3 \
> + # structs:1.17 \
> + # windows-slaves:1.4
> +
> + # install plugins for latest LTS Jenkins (currently 2.249.3)
> + /usr/local/bin/install-plugins.sh \
> + script-security \
> + structs \
> + workflow-step-api \
> + workflow-api \
> + junit \
> + scm-api \
> + ant \
> + antisamy-markup-formatter \
> + bouncycastle-api \
> + command-launcher \
> + description-setter \
> + display-url-api \
> + external-monitor-job \
> + greenballs \
> + icon-shim \
> + javadoc \
> + jdk-tool \
> + mailer \
> + matrix-auth \
> + matrix-project \
> + pam-auth \
> + pegdown-formatter \
> + windows-slaves
> +
> + cp frontend-install/plugins/flot-plotter-plugin/flot.hpi $JENKINS_HOME/plugins/flot.jpi
> +
> + # make the mod.js symlink well after flot is installed
> + service jenkins start && sleep 30 && \
> + rm $JENKINS_HOME/plugins/flot/flot/mod.js && \
> + ln -s /fuego-core/scripts/mod.js $JENKINS_HOME/plugins/flot/flot/mod.js
> +
> + chown -R jenkins:jenkins $JENKINS_HOME/
> +fi
> +
> +# ==============================================================================
> +# get ttc script and helpers
> +# ==============================================================================
> +git clone https://github.com/tbird20d/ttc.git /usr/local/src/ttc
> +/usr/local/src/ttc/install.sh /usr/local/bin
> +perl -p -i -e "s#config_dir = \"/etc\"#config_dir = \"/fuego-ro/conf\"#" /usr/local/bin/ttc
> +
> +# ==============================================================================
> +# Serial Config
> +# ==============================================================================
> +if [ $nojenkins -eq 0 ]; then
> + /bin/bash -c 'git clone https://github.com/frowand/serio.git /usr/local/src/serio ; chown -R jenkins /usr/local/src/serio ; cp
> /usr/local/src/serio/serio /usr/local/bin/ ; ln -s /usr/local/bin/serio /usr/local/bin/sercp ; ln -s /usr/local/bin/serio /usr/local/bin/sersh'
> + /bin/bash -c 'git clone https://github.com/tbird20d/serlogin.git /usr/local/src/serlogin ; chown -R jenkins /usr/local/src/serlogin ;
> cp /usr/local/src/serlogin/serlogin /usr/local/bin'
> +else
> + /bin/bash -c 'git clone https://github.com/frowand/serio.git /usr/local/src/serio ; cp /usr/local/src/serio/serio /usr/local/bin/ ; ln -
> s /usr/local/bin/serio /usr/local/bin/sercp ; ln -s /usr/local/bin/serio /usr/local/bin/sersh'
> + /bin/bash -c 'git clone https://github.com/tbird20d/serlogin.git /usr/local/src/serlogin ; cp /usr/local/src/serlogin/serlogin
> /usr/local/bin'
> +fi
> +cp frontend-install/0001-Fix-host-parsing-for-serial-device-with-in-name.patch \
> + frontend-install/0002-Output-data-from-port-during-command-execution.patch \
> + /tmp/
> +/bin/bash -c 'patch -d /usr/local/src/serio -p1 </tmp/0001-Fix-host-parsing-for-serial-device-with-in-name.patch ; \
> + patch -d /usr/local/src/serio -p1 </tmp/0002-Output-data-from-port-during-command-execution.patch ;'
> +
> +
> +# ==============================================================================
> +# get ebf script and helpers
> +# ==============================================================================
> +apt-get -q=2 -V --no-install-recommends install jq
> +git clone https://github.com/TimesysGit/board-farm-rest-api.git /usr/local/src/board-farm-rest-api
> +cp /usr/local/src/board-farm-rest-api/cli/ebf /usr/local/bin/ebf
> +chmod a+x /usr/local/bin/ebf
> +
> +# ==============================================================================
> +# fserver
> +# ==============================================================================
> +git clone https://github.com/tbird20d/fserver.git /usr/local/lib/fserver
> +ln -s /usr/local/lib/fserver/start_local_bg_server /usr/local/bin/start_local_bg_server
> +
> +# ==============================================================================
> +# ftc post installation
> +# ==============================================================================
> +ln -s /fuego-core/scripts/ftc /usr/local/bin/
> +cp fuego-core/scripts/ftc_completion.sh /etc/bash_completion.d/ftc
> +echo ". /etc/bash_completion" >> /root/.bashrc
> +
> +# ==============================================================================
> +# Lava
> +# ==============================================================================
> +ln -s /fuego-ro/scripts/fuego-lava-target-setup /usr/local/bin
> +ln -s /fuego-ro/scripts/fuego-lava-target-teardown /usr/local/bin
> diff --git a/install.sh b/install.sh
> index a5a90b0..259c887 100755
> --- a/install.sh
> +++ b/install.sh
> @@ -1,13 +1,13 @@
> #!/bin/bash
> #
> -# install.sh [--help] [--priv] [--no-cache] [--nojenkins] [<image_name>] [<port>]
> +# install.sh [--help] [--priv] [--no-cache] [--nojenkins] [<image_name>] [<port>] [<distribution>] [<suit>]
> #
>
> # $1 is the exit code after usage is shown
> usage() {
> exit_code="$1"
> cat <<HERE
> -Usage: install.sh [--help] [--priv] [--no-cache] [--nojenkins] [<image_name>] [<port>]
> +Usage: install.sh [--help] [--priv] [--no-cache] [--nojenkins] [<image_name>] [<port>] [<distribution>] [<suit>]
>
> Create the docker image and container with the Fuego test distribution.
> If no <image_name> is provided, the image will be named 'fuego'.
> @@ -34,7 +34,7 @@ fi
>
> priv=0
> NOCACHE=""
> -dockerfile="Dockerfile"
> +NOJENKINS=""
>
> POSITIONAL=()
> while [[ $# -gt 0 ]] ; do
> @@ -48,7 +48,7 @@ while [[ $# -gt 0 ]] ; do
> shift
> ;;
> --nojenkins)
> - dockerfile="Dockerfile.nojenkins"
> + NOJENKINS="--nojenkins"
> shift
> ;;
> *)
> @@ -60,6 +60,8 @@ set -- "${POSITIONAL[@]}" # restore positional arguments
>
> image_name=${1:-fuego}
> jenkins_port=${2:-8090}
> +distribution=${3:-debian}
> +suite=${4:-stretch}
>
> container_name="${image_name}-container"
>
> @@ -79,7 +81,7 @@ fi
>
> set -e
>
> -source fuego-host-scripts/docker-build-image.sh $NOCACHE ${image_name} ${jenkins_port} ${dockerfile}
> +source fuego-host-scripts/docker-build-image.sh $NOCACHE ${image_name} ${jenkins_port} ${distribution} ${suite} ${NOJENKINS}
> if [ "$priv" == "0" ]; then
> fuego-host-scripts/docker-create-container.sh ${image_name} ${container_name}
> else
> diff --git a/uninstall-fuego.sh b/uninstall-fuego.sh
> new file mode 100644
> index 0000000..b899e7c
> --- /dev/null
> +++ b/uninstall-fuego.sh
> @@ -0,0 +1,47 @@
> +#!/bin/bash
> +
> +# Remove Jenkins plugins
> +[ -d /usr/share/jenkins/ref/plugins/ ] && rm -rf /usr/share/jenkins/ref/plugins
> +
> +# uninstall Jenkins
> +service jenkins stop
> +dpkg -P jenkins
> +
> +[ -f /usr/local/bin/install-plugins.sh ] && rm -f /usr/local/bin/install-plugins.sh
> +[ -f /usr/local/bin/jenkins-support ] && rm -rf /usr/local/bin/jenkins-support
> +[ -f /usr/local/bin/clitest ] && rm -rf /usr/local/bin/clitest
> +[ -d /var/lib/jenkins ] && rm -rf /var/lib/jenkins
> +sed -i '/^JENKINS_PORT=/d' /etc/environment
> +
> +# uninstall ttc script
> +[ -d /usr/local/src/ttc ] && rm -rf /usr/local/src/ttc
> +
> +# uninstall ebf script
> +[ -d /usr/local/src/board-farm-rest-api ] && rm -rf /usr/local/src/board-farm-rest-api
> +[ -f /usr/local/bin/ebf ] && rm -f /usr/local/bin/ebf
> +
> +# uninstall serial config
> +[ -d /usr/local/src/serio ] && rm -rf /usr/local/src/serio
> +[ -f /usr/local/bin/serio ] && rm -rf /usr/local/bin/serio
> +[ -f /usr/local/bin/sercp ] && rm -rf /usr/local/bin/sercp
> +[ -f /usr/local/bin/sersh ] && rm -rf /usr/local/bin/sersh
> +[ -d /usr/local/src/serlogin ] && rm -rf /usr/local/src/serlogin
> +[ -f /usr/local/bin/serlogin ] && rm -rf /usr/local/bin/serlogin
> +
> +# uninstall fserver
> +[ -d /usr/local/lib/fserver ] && rm -f /usr/local/bin/fserver
> +[ -f /usr/local/bin/start_local_bg_server ] && rm -rf /usr/local/bin/start_local_bg_server
> +
> +# uninstall ftc
> +[ -f /usr/local/bin/ftc ] && rm -rf /usr/local/bin/ftc
> +[ -f /etc/bash_completion.d/ftc ] && rm -rf /etc/bash_completion.d/ftc
> +
> +# remove lava target files
> +[ -f /usr/local/bin/fuego-lava-target-setup ] && rm -rf /usr/local/bin/fuego-lava-target-setup
> +[ -f /usr/local/bin/fuego-lava-target-teardown ] && rm -rf /usr/local/bin/fuego-lava-target-teardown
> +
> +# remove links
> +[ -d /fuego-ro ] && rm /fuego-ro
> +[ -d /fuego-rw ] && rm /fuego-rw
> +[ -d /fuego-core ] && rm /fuego-core
> +
> --
> 2.20.1
>
Overall - this looks like a good start. But I'd like to see the duplicate sections of install-buster.sh
and install-stretch.sh put into a single location "install-debian-common.sh". Also, I'd like to see
the non-debian stuff (ttc, serio, ebf, etc.) put into "install-common.sh".
Can you please re-do this with that refactoring in mind?
Thanks,
-- Tim
next prev parent reply other threads:[~2021-06-15 0:08 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-06-04 13:43 [Fuego] Use common script to install fuego on host or docker system venkata.pyla
2021-06-04 14:25 ` Venkata.Pyla
2021-06-09 3:35 ` Tim.Bird
2021-06-15 0:08 ` Tim.Bird [this message]
2021-06-15 3:46 ` Venkata.Pyla
2021-06-16 17:47 ` Tim.Bird
2021-06-17 7:01 ` Venkata.Pyla
2021-06-17 20:53 ` Tim.Bird
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=BYAPR13MB25037C97DF64D58B5DA0855DFD309@BYAPR13MB2503.namprd13.prod.outlook.com \
--to=tim.bird@sony.com \
--cc=fuego@lists.linuxfoundation.org \
--cc=tbird20d@yahoo.com \
--cc=venkata.pyla@toshiba-tsip.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.