Re: [Fuego] RFC: Fuego's version up and other changes

Re: [Fuego] RFC: Fuego's version up and other changes

[Jan-Simon Möller]

Hi Daniel, hi Tim,

@Daniel:
I tried your next branch and I think the uprev is well done.
I'll try to port my modifications to it. An idea of the timeline
or should we collaborate on-top of a -next branch in upstream ?

@Tim:
I'm working on porting AGLs jta<->lava support shim to fuego.
Also I'm looking at your ttc transport commits. That might be a nice way to 
add different mechanisms for the actual connection to the boards.

Best,
--
Jan-Simon Möller
dl9pf@gmx.de

Re: [Fuego] RFC: Fuego's version up and other changes

[Bird, Timothy]

> -----Original Message-----
> From: Jan-Simon Möller on  Monday, January 23, 2017 9:28 AM
>
> @Daniel:
> I tried your next branch and I think the uprev is well done.
> I'll try to port my modifications to it. An idea of the timeline
> or should we collaborate on-top of a -next branch in upstream ?
>
We should start discussing this.  It would be nice to either have this
done by ELC, or get together and make the plan for this at ELC.
 
Actually, I think I'll put out a general call to see who's involved
with Fuego and coming to ELC, and try to put together a dedicated
session for discussing the status and roadmap.
 
> @Tim:
> I'm working on porting AGLs jta<->lava support shim to fuego.
> Also I'm looking at your ttc transport commits. That might be a nice way to
> add different mechanisms for the actual connection to the boards.

Fuego integration with LAVA has been on my to-do list.  I know that 
AGL-CIAT does this, but haven't had time to study it.  I am looking forward
to your ELC presentation on Fuego and Lava!

Yes - I believe that adding support for LAVA at the transport level could
be accomplished with ttc, but there's only 3 commands, and I wouldn't
be surprised if you could do it directly.  Frankly, I thought that only Sony
was using ttc, but we'd probably be pretty happy to add the ability for ttc
to access and control LAVA-based boards, and that would give us LAVA
support in fuego for free since we've already integrated ttc into fuego.

Note that we've got support for serial transport almost working.  It's mostly
working but with a few bugs we're ironing out.  The final transport I want to
add directly into fuego is adb, for Android-based boards.  I think it will
be trivial - I just don't have a board running Android that up and running
at the moment.

Would it be worth having a discussion offline for you to educate me on how
AGL-CIAT is using LAVA in conjunction with Fuego, before ELC?  I've been
meaning to join the AGL-CIAT meetings, but last time I got confused on the
time (I misread the timezone and missed the last call).  I'm not sure if that's
the right venue to discuss this, or we should do something separate.

Let me know what you think.

Thanks,
 - - Tim

Re: [Fuego] RFC: Fuego's version up and other changes

[Daniel Sangorrin]

> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird@am.sony.com]
> Sent: Friday, January 13, 2017 12:35 PM
> To: Daniel Sangorrin; Ibe.Kengo@bx.MitsubishiElectric.co.jp
> Cc: fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] RFC: Fuego's version up and other changes
> 
> > -----Original Message-----
> > From: Daniel Sangorrin on Thursday, January 12, 2017 6:06 PM
> >
> > It seems to be related to this (docker build hangs/crashes when useradd with
> > large UID):
> > https://github.com/docker/docker/issues/5419
> >
> > Can you add the -l option to useradd and see if it fixes it?
> 
> That fixes the problem.  Thanks!
> 
> I'll do some more testing and tell you anything more I find.

OK, thank you!!
I'm also testing it and already found some bugs. I will push them as quickly as I can.

Thanks
Daniel

Re: [Fuego] RFC: Fuego's version up and other changes

[Bird, Timothy]

> -----Original Message-----
> From: Daniel Sangorrin on Thursday, January 12, 2017 6:06 PM
>
> It seems to be related to this (docker build hangs/crashes when useradd with
> large UID):
> https://github.com/docker/docker/issues/5419
> 
> Can you add the -l option to useradd and see if it fixes it?

That fixes the problem.  Thanks!

I'll do some more testing and tell you anything more I find.
 -- Tim

Re: [Fuego] RFC: Fuego's version up and other changes

[Daniel Sangorrin]

Hi Tim,

> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird@am.sony.com]
> Sent: Friday, January 13, 2017 10:58 AM
> To: Daniel Sangorrin; Ibe.Kengo@bx.MitsubishiElectric.co.jp
> Cc: fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] RFC: Fuego's version up and other changes
> 
> OK - I think there's some bug in docker.  I added some debug lines to the Dockerfile,
> and split the 'groupadd...' line.  I found that the hang is with the 'useradd' line,
> which is weird as this should be a fairly straightforward operation inside the image.
> 
> Here's what I get:
> Step 20 : ENV JENKINS_HOME /var/lib/jenkins
>  ---> Using cache
>  ---> b26e2d078353
> Step 21 : RUN echo "uid=${uid}"
>  ---> Running in b22a2010fff2
> uid=2060932290
>  ---> 5dcc36f214af
> Removing intermediate container b22a2010fff2
> Step 22 : RUN echo "gid=${gid}"
>  ---> Running in ed562da957b6
> gid=2060452353
>  ---> 228948aad0df
> Removing intermediate container ed562da957b6
> Step 23 : RUN groupadd -g ${gid} ${group}
>  ---> Running in 7db50dee926e
>  ---> 33a79d51f82e
> Removing intermediate container 7db50dee926e
> Step 24 : RUN useradd -m -d "${JENKINS_HOME}" -u ${uid} -g ${gid} -G sudo -s /bin/bash ${user}
>  ---> Running in f75c9c00ef86
> (hangs here, with dockerd at 100% of CPU)
> ^C1000072457@tlinux:~/work/daniel-fuego/fuego$
> 
> I'm not sure what's going on here.  But if I comment out the 'RUN useradd...' line, the image
> builds and I get what appear to be default (auto-generated) uid and gid numbers for the 'jenkins' user
> in the image.
> 

It seems to be related to this (docker build hangs/crashes when useradd with large UID):
https://github.com/docker/docker/issues/5419

Can you add the -l option to useradd and see if it fixes it?

> I'll just do that for now, and try some other operations to see if the image is working as intended.
> Or will the permission difference between container and host give me more problems?

If the folders are owned by jenkins inside docker there shouldn't be any problem.

Thanks,
Daniel

>  -- Tim
> 
> 
> > -----Original Message-----
> > From: Daniel Sangorrin [mailto:daniel.sangorrin@toshiba.co.jp]
> > Sent: Thursday, January 12, 2017 5:45 PM
> > To: Bird, Timothy <Tim.Bird@am.sony.com>;
> > Ibe.Kengo@bx.MitsubishiElectric.co.jp
> > Cc: fuego@lists.linuxfoundation.org
> > Subject: RE: [Fuego] RFC: Fuego's version up and other changes
> >
> > oops, sorry I wrote it too quickly. I just fixed it.
> >
> > > -----Original Message-----
> > > From: Bird, Timothy [mailto:Tim.Bird@am.sony.com]
> > > Sent: Friday, January 13, 2017 10:38 AM
> > > To: Daniel Sangorrin; Ibe.Kengo@bx.MitsubishiElectric.co.jp
> > > Cc: fuego@lists.linuxfoundation.org
> > > Subject: RE: [Fuego] RFC: Fuego's version up and other changes
> > >
> > > I just pulled it.  Is this right?
> > >
> > > if [ "$(id -u)" == "0" ]; then
> > >  	JENKINS_UID=$(id -u $SUDO_USER)
> > > +	JENKINS_GID=$(id -g $SUDO_USER)
> > >  else
> > >  	JENKINS_UID=$(id -u $USER)
> > > +	JENKINS_GID=$(id -g $SUDO_USER)
> > >  fi
> > >
> > > I'm not sure exactly what's going on, but shouldn't that second
> > > changed line be?
> > >                JENKINS_GID=$(id -g $USER)
> > >
> > >  -- Tim
> > >
> > > > -----Original Message-----
> > > > From: Daniel Sangorrin [mailto:daniel.sangorrin@toshiba.co.jp]
> > > > Sent: Thursday, January 12, 2017 5:30 PM
> > > > To: Bird, Timothy <Tim.Bird@am.sony.com>;
> > > > Ibe.Kengo@bx.MitsubishiElectric.co.jp
> > > > Cc: fuego@lists.linuxfoundation.org
> > > > Subject: RE: [Fuego] RFC: Fuego's version up and other changes
> > > >
> > > > Hi Tim,
> > > >
> > > > > -----Original Message-----
> > > > > From: Bird, Timothy [mailto:Tim.Bird@am.sony.com]
> > > > > Sent: Friday, January 13, 2017 10:02 AM
> > > > > To: Daniel Sangorrin; Ibe.Kengo@bx.MitsubishiElectric.co.jp
> > > > > Cc: fuego@lists.linuxfoundation.org
> > > > > Subject: RE: [Fuego] RFC: Fuego's version up and other changes
> > > > >
> > > > > Daniel,
> > > > >
> > > > > Thanks for all this work.  I'm having some issues, which may be related
> > to
> > > > the user and group, so
> > > > > I'm replying to the message rather than start a new thread.
> > > > >
> > > > > Basically I'm having two problems, both during ./install.sh.
> > > > >
> > > > > 1) I very often have the debian update and extra file installation fail
> > (Step
> > > > 10 in the Docker image build).
> > > > > (noninteractive apt-get update && apt-get -yV install ...)
> > > > >
> > > > > There are 352 'get' operations during this secondary installation, and if
> > even
> > > > a single one fails,
> > > > > the docker build fails.  I kept having problems getting all the way
> > through
> > > > this step.  I would
> > > > > have network failures from the debian servers.  It seemed to be
> > > > intermittent, but overall it
> > > > > required that I try to build the docker container about 10 times for it to
> > get
> > > > past that step.
> > > > > Here's a sample set of messages:
> > > > >
> > > > > Get:95 http://httpredir.debian.org/debian/ jessie/main libatk-wrapper-
> > > > java all 0.30.5-1 [30.3 kB]
> > > > > Err http://httpredir.debian.org/debian/ jessie/main libavahi-glib1
> > amd64
> > > > 0.6.31-5
> > > > >   Error reading from server. Remote end closed connection [IP:
> > 5.153.231.35
> > > > 80]
> > > >
> > > > Yes, I had the same problem yesterday when I installed it on a different
> > > > machine.
> > > > It seems that httpredir.debian.org is not as reliable as using a local mirror
> > > > directly.
> > > > I'm not sure if I should add retries or put a default debian server as it used
> > to
> > > > be.
> > > > Could you change httpredir by the US mirror
> > > > (http://ftp.us.debian.org/debian) and try again?
> > > >
> > > > > And, at the end of the failed docker build, this:
> > > > > -------
> > > > > Fetched 231 MB in 1min 43s (2226 kB/s)
> > > > > E: Failed to fetch
> > > > http://httpredir.debian.org/debian/pool/main/a/avahi/libavahi-
> > glib1_0.6.31-
> > > > 5_amd64.deb  Error reading from
> > > > > server. Remote end closed connection [IP: 5.153.231.35 80]
> > > > >
> > > > > E: Unable to fetch some archives, maybe run apt-get update or try with
> > --
> > > > fix-missing?
> > > > > The command '/bin/sh -c DEBIAN_FRONTEND=noninteractive apt-get
> > > > update && apt-get -yV install    apt-utils daemon gcc make
> > > > > python-paramiko python-lxml python-simplejson         python-
> > matplotlib
> > > > libtool xmlstarlet autoconf automake rsync openjdk-7-jre
> > > > > openjdk-7-jdk iperf netperf netpipe-tcp sshpass wget git        diffstat
> > sudo
> > > > net-tools vim curl inotify-tools python-openpyxl  g++ bzip2
> > > > > bc libaio-dev gettext pkg-config libglib2.0-dev' returned a non-zero
> > code:
> > > > 100
> > > > > --------
> > > > >
> > > > > It was not the same package failing each time - but the times it failed
> > some
> > > > of the
> > > > > same packages occurred (so, it's hard to pin it on the package or the
> > > > server).
> > > > >
> > > > > Maybe this is just some bad networking at my site, but it did cause me
> > > > problems.
> > > > > (Or maybe the Debian servers are just having a bad day today).
> > > > >
> > > > > More problematical is a hang at Step 21:
> > > > > Removing intermediate container a522905e9fc2
> > > > > Step 21 : RUN groupadd -g ${gid} ${group} 	&& useradd -m -d
> > > > "${JENKINS_HOME}" -u ${uid} -g ${gid} -G sudo -s /bin/bash ${user}
> > > > >  ---> Running in 106f8bf82d61
> > > > >
> > > > > The build just hangs there, every time.  The dockerd is running on my
> > > > desktop,
> > > > > and consuming 110% cpu.
> > > > >
> > > > > I'm not sure if this is hung in the 'groupadd' step or the 'wget -nv
> > > > ${JENKINS_URL}' step.
> > > > >
> > > > > I'm on Ubuntu 14.04.5 LTS using docker version 1.12.1.
> > > >
> > > > The wget does take a couple of minutes. But I hadn't experienced the
> > > > groupadd hang.
> > > >
> > > > > dockerd appears to be stuck waiting for a futex.
> > > > >
> > > > > I'm not sure that the line 'ARG uid=1000' is going to work in my case.
> > Here's
> > > > my setup:
> > > > > $ id
> > > > > uid=2060932290(1000072457) gid=2060452353(domain_users)
> > > > >
> > > >
> > groups=2060452353(domain_users),4(adm),20(dialout),24(cdrom),27(sudo),
> > > > 46(plugdev),108(lpadmin),124(sambashare),999(docker)
> > > > >
> > > > > Don’t ask - this is due to my Linux machine being previously
> > administered
> > > > by corporate types.
> > > >
> > > > Ok, I think I know what the problem is. I had assumed that the gid would
> > be
> > > > the same as the uid
> > > > in the general case, and other users could just pass their uid/gid in the
> > > > docker-build-image.sh
> > > > script. I have fixed docker-build-image.sh and added the definition of gid.
> > > > Please, could you do a git pull and try again?
> > > >
> > > >
> > https://bitbucket.org/nirrognas/fuego/commits/9d8d1426c326575af8467f73
> > > > bce7773d1f45a76c
> > > >
> > > > > By the way, when I kill the build, dockerd continues to dominate my
> > CPU
> > > > usage in top
> > > > > (and I can hear my desktop cpu fan running).
> > > > >
> > > > > I think this might be a docker bug, but I'm not sure.
> > > >
> > > > Sorry about that!
> > > >
> > > > Thanks for your feedback!!
> > > > Daniel
> > > >
> > > > >
> > > > > I'll play around with the Dockerfile (maybe altering the user
> > configuration),
> > > > but I thought I should report my status so far.
> > > > >
> > > > > Any ideas?
> > > > >  -- Tim
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> > > > > > bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> > > > > > Sent: Wednesday, January 11, 2017 10:45 PM
> > > > > > To: Ibe.Kengo@bx.MitsubishiElectric.co.jp
> > > > > > Cc: fuego@lists.linuxfoundation.org
> > > > > > Subject: Re: [Fuego] RFC: Fuego's version up and other changes
> > > > > >
> > > > > > Hi Kengo-san,
> > > > > >
> > > > > > Thanks a lot for your feedback.
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Ibe.Kengo@bx.MitsubishiElectric.co.jp
> > > > > > [mailto:Ibe.Kengo@bx.MitsubishiElectric.co.jp]
> > > > > > > Sent: Thursday, January 12, 2017 2:52 PM
> > > > > > > To: Daniel Sangorrin
> > > > > > > Cc: fuego@lists.linuxfoundation.org
> > > > > > > Subject: Re: [Fuego] RFC: Fuego's version up and other changes
> > > > > > >
> > > > > > > Hi, Daniel-san.
> > > > > > >
> > > > > > > I’m very interested in your work!
> > > > > > > So I tried to run your build scripts to create Docker image today,
> > > > > > > then I could run new Fuego.
> > > > > > > And, I added Docker node and test cases to use following
> > command.
> > > > > > > ==
> > > > > > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego.git
> > > > > > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego-
> > > > core.git
> > > > > > >
> > > > > > >   $ cd fuego/
> > > > > > >   $ ./install.sh
> > > > > > >   $ ./fuego-host-scripts/docker-create-container.sh
> > > > > > >   $ ./fuego-host-scripts/docker-start-container.sh
> > > > > > >   docker# fuego-create-node --board docker
> > > > > > >   docker# fuego-create-jobs --board docker --testplan
> > testplan_docker
> > > > > > > --distrib nosyslogd.dist
> > > > > > > ==
> > > > > > >
> > > > > > > I can see  new node and new test jobs.
> > > > > > > But, I can’t run tests like Dhrystone on Docker.
> > > > > > >
> > > > > > > So far, I had one small problem that is “Failed to mkdirs: /fuego-rw/
> > > > > > > buildzone”
> > > > > > > This reason is the permission or owner of “fuego-rw”,
> > > > > > > so I added the write permission to “fuego-rw” diredtory.
> > > > > >
> > > > > > Humm You should not need to change any permissions.
> > > > > > The script is supposed to use your user's uid/gid for the jenkins
> > daemon.
> > > > > > However, I have only tested it on my pc which has a single user.
> > > > > > Do you have more than one user? Could you check that the jenkins
> > > > uid/gid
> > > > > > is the same as yours? (you can check on /etc/passwd inside and
> > outside
> > > > > > docker)
> > > > > > Please also check that the fuego-rw's owner is your user (e.g.: don't
> > git
> > > > clone
> > > > > > with
> > > > > > a different user or root)
> > > > > >
> > > > > > > I have solved this problem.
> > > > > > >
> > > > > > > After that, I got the error message like following in console log.
> > > > > > > ====
> > > > > > > Started by user anonymous
> > > > > > > Building remotely on docker in workspace /fuego-rw/buildzone
> > > > > > > [buildzone] $ /bin/sh -xe /tmp/hudson1255554151521556820.sh
> > > > > > > + export Reboot=false
> > > > > > > + Reboot=false
> > > > > > > + export Rebuild=true
> > > > > > > + Rebuild=true
> > > > > > > + export Target_Cleanup=true
> > > > > > > + Target_Cleanup=true
> > > > > > > + export TESTDIR=Benchmark.Dhrystone
> > > > > > > + TESTDIR=Benchmark.Dhrystone
> > > > > > > + export TESTNAME=Dhrystone
> > > > > > > + TESTNAME=Dhrystone
> > > > > > > + export DISTRIB=nosyslogd.dist
> > > > > > > + DISTRIB=nosyslogd.dist
> > > > > > > + export TESTPLAN=testplan_docker
> > > > > > > + TESTPLAN=testplan_docker
> > > > > > > + timeout --signal=9 3m /bin/bash
> > > > > > > /fuego-core/engine/tests/Benchmark.Dhrystone/Dhrystone.sh
> > > > > > > board overlay: boards/docker.board
> > > > > > > using boards/docker.board board overlay
> > > > > > > using nosyslogd.dist overlay
> > > > > > > BATCH_TESTPLAN is not set, using testplan_docker.json testplan
> > > > > > > Traceback (most recent call last):
> > > > > > >   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 615, in
> > > > <module
> > > > > > > >
> > > > > > >     run()
> > > > > > >   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 607, in run
> > > > > > >     generateProlog(output, ofcls, classes, testPlans, testSpecs)
> > > > > > >   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 375, in
> > > > > > > generateProlog
> > > > > > >     outfile = open(outFilePath, "w")
> > > > > > > IOError: [Errno 2] No such file or directory: '/fuego-
> > rw/work/docker_
> > > > > > > prolog.sh'
> > > > > > >
> > > > > > > *** ABORTED ***
> > > > > > >
> > > > > > > Fuego error reason: Error while prolog.sh file generation
> > > > > > >
> > > > > > > Build step 'Execute shell' marked build as failure
> > > > > > > [description-setter] Description set:
> > > > > > > Finished: FAILURE
> > > > > > > ====
> > > > > > >
> > > > > > > Did I miss to run test cases on Docker node?
> > > > > > > If you know this solution, could you give me some advice?
> > > > > >
> > > > > > Could you check that fuego-rw is correctly mounted inside docker?
> > > > > > outside-docker$ touch fuego-rw/hoge
> > > > > > inside-docker$ ls /fuego-rw
> > > > > > hoge <-- does it appear?
> > > > > >
> > > > > > On my PC (ubuntu 16.04, docker version 1.12.3 ) it's always succesful
> > so I
> > > > will
> > > > > > try on a different one. Which OS and docker version are you using?
> > > > > >
> > > > > > Thanks,
> > > > > > Daniel
> > > > > >
> > > > > > > Thank you.
> > > > > > >
> > > > > > > Best Regard.
> > > > > > > Kengo Ibe
> > > > > > >
> > > > > > > ----- Original Message -----
> > > > > > > 件名: [Fuego] RFC: Fuego's version up and other changes
> > > > > > > 送信者: "Daniel Sangorrin" <daniel.sangorrin@toshiba.co.jp>
> > > > > > > 送信日時: 2017年01月11日(水) 09:28:07
> > > > > > > 宛先: "Bird, Timothy" <Tim.Bird@am.sony.com>, fuego@lists.
> > > > > > > linuxfoundation.org
> > > > > > >
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > I've been working on Fuego's upgrade for a few weeks, and in the
> > > > > > > process I
> > > > > > > > have made quite a few changes.
> > > > > > > > You can check it by following these instructions. I'd really
> > > > > > > appreciate
> > > > > > > > your feedback:
> > > > > > > >
> > > > > > > > Quickstart:
> > > > > > > >   $ git clone --branch next
> > https://bitbucket.org/nirrognas/fuego.git
> > > > > > > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego-
> > > > core.git
> > > > > > >
> > > > > > > >   $ cd fuego/
> > > > > > > >   $ ./install.sh
> > > > > > > >   $ ./fuego-host-scripts/docker-create-container.sh
> > > > > > > >   $ ./fuego-host-scripts/docker-start-container.sh
> > > > > > > >   docker# fuego-create-node --board docker
> > > > > > > >   docker# fuego-create-jobs --board docker --testplan
> > > > testplan_docker
> > > > > > > > --distrib nosyslogd.dist
> > > > > > > >   Then, start the "docker.testplan_docker.batch" job which will
> > > > > > > trigger
> > > > > > > > the rest of jobs.
> > > > > > > >
> > > > > > > > Here is a list of the most fundamental modifications I've added:
> > > > > > > > - I automatized the version upgrade process. So far I have
> > updated
> > > > > > > Jenkins
> > > > > > > > 3 times quite smoothly.
> > > > > > > >    The current version is the latest LTS version (2.31.1 LTS)
> > > > > > > > - I reduced plugin dependencies to the minimum (only 2, and they
> > can
> > > > > > > be
> > > > > > > > made optional).
> > > > > > > > - I removed dependencies on all groovy scripts.
> > > > > > > > - Userdata is gone. Instead we now have fuego-ro and fuego-rw,
> > > > which
> > > > > > > > together with fuego-core
> > > > > > > >    are mounted as external docker volumes. Only fuego-rw, which
> > > > > > > contains
> > > > > > > > logs and build folders,
> > > > > > > >    is mounted in RW mode. I also fixed the uid/gid of jenkins so
> > that
> > > > > > > it
> > > > > > > > matches the one in the host.
> > > > > > > >    This means you can develop comfortably on your host, with GUI
> > > > tools,
> > > > > > > > and don't fear a buggy
> > > > > > > >    script deleting your folders.
> > > > > > > > - I added a  tool for generating jobs from testplans.  Creating jobs
> > > > > > > from
> > > > > > > > testplans is powerful.
> > > > > > > >    It allows you, for example, to easily specify the timeout for each
> > > > > > > test
> > > > > > > > on your board.
> > > > > > > >    The tool also generates the corresponding batch job. For
> > example,
> > > > > > > you
> > > > > > > > can easily trigger all
> > > > > > > >    tests for your board and a specific testplan periodically.
> > > > > > > >    [Note] This will also enable the creation of scripts for comparing
> > > > > > > test
> > > > > > > > results across
> > > > > > > >    different boards and test parameters (on my TODO list).
> > > > > > > >
> > > > > > > > Other less fundamental changes include:
> > > > > > > > - I added support for using the docker container itself as a target
> > > > > > > board
> > > > > > > > (e.g.: for a quickstart or unit tests)
> > > > > > > > - I fixed the flot plugin (java and javascript fixes)
> > > > > > > > - Now you can click on a node and see which jobs (and testplans)
> > are
> > > > > > > > assigned to your board. This was
> > > > > > > >   one of my old feature requests and makes Fuego's GUI easier to
> > > > use.
> > > > > > > > - I added quite a few fixes and improvements to the core engine
> > > > > > > scripts
> > > > > > > > and tests (too detailed to describe here).
> > > > > > > > - I added Excel output support for IOzone
> > > > > > > > - Faster docker build time (ARMhf compiler installation is now
> > > > > > > optional,
> > > > > > > > no latex..)
> > > > > > > > - I put a fixed name to the fuego container, instead of the
> > > > > > > > "last_container_id" file.
> > > > > > > > - I removed the inotify script
> > > > > > > > - I added scripts for easily removing the docker container/image.
> > > > > > > >
> > > > > > > > I have to split the current "huge" patch into smaller chunks and
> > test
> > > > > > > it
> > > > > > > > further but it would
> > > > > > > > be nice to get your feedback about the approach in general.
> > > > > > > >
> > > > > > > > Tim: I haven't updated ftc nor your "Christmas commits" yet.
> > > > > > > >
> > > > > > > > Best regards
> > > > > > > > Daniel
> > > > > > > >
> > > > > > > >
> > > > > > > > _______________________________________________
> > > > > > > > Fuego mailing list
> > > > > > > > Fuego@lists.linuxfoundation.org
> > > > > > > > https://lists.linuxfoundation.org/mailman/listinfo/fuego
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > Fuego mailing list
> > > > > > Fuego@lists.linuxfoundation.org
> > > > > > https://lists.linuxfoundation.org/mailman/listinfo/fuego
> > > >
> >
> >

Re: [Fuego] RFC: Fuego's version up and other changes

[Bird, Timothy]

OK - I think there's some bug in docker.  I added some debug lines to the Dockerfile,
and split the 'groupadd...' line.  I found that the hang is with the 'useradd' line,
which is weird as this should be a fairly straightforward operation inside the image.

Here's what I get:
Step 20 : ENV JENKINS_HOME /var/lib/jenkins
 ---> Using cache
 ---> b26e2d078353
Step 21 : RUN echo "uid=${uid}"
 ---> Running in b22a2010fff2
uid=2060932290
 ---> 5dcc36f214af
Removing intermediate container b22a2010fff2
Step 22 : RUN echo "gid=${gid}"
 ---> Running in ed562da957b6
gid=2060452353
 ---> 228948aad0df
Removing intermediate container ed562da957b6
Step 23 : RUN groupadd -g ${gid} ${group}
 ---> Running in 7db50dee926e
 ---> 33a79d51f82e
Removing intermediate container 7db50dee926e
Step 24 : RUN useradd -m -d "${JENKINS_HOME}" -u ${uid} -g ${gid} -G sudo -s /bin/bash ${user}
 ---> Running in f75c9c00ef86
(hangs here, with dockerd at 100% of CPU)
^C1000072457@tlinux:~/work/daniel-fuego/fuego$ 

I'm not sure what's going on here.  But if I comment out the 'RUN useradd...' line, the image
builds and I get what appear to be default (auto-generated) uid and gid numbers for the 'jenkins' user
in the image.

I'll just do that for now, and try some other operations to see if the image is working as intended.
Or will the permission difference between container and host give me more problems?
 -- Tim


> -----Original Message-----
> From: Daniel Sangorrin [mailto:daniel.sangorrin@toshiba.co.jp]
> Sent: Thursday, January 12, 2017 5:45 PM
> To: Bird, Timothy <Tim.Bird@am.sony.com>;
> Ibe.Kengo@bx.MitsubishiElectric.co.jp
> Cc: fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] RFC: Fuego's version up and other changes
> 
> oops, sorry I wrote it too quickly. I just fixed it.
> 
> > -----Original Message-----
> > From: Bird, Timothy [mailto:Tim.Bird@am.sony.com]
> > Sent: Friday, January 13, 2017 10:38 AM
> > To: Daniel Sangorrin; Ibe.Kengo@bx.MitsubishiElectric.co.jp
> > Cc: fuego@lists.linuxfoundation.org
> > Subject: RE: [Fuego] RFC: Fuego's version up and other changes
> >
> > I just pulled it.  Is this right?
> >
> > if [ "$(id -u)" == "0" ]; then
> >  	JENKINS_UID=$(id -u $SUDO_USER)
> > +	JENKINS_GID=$(id -g $SUDO_USER)
> >  else
> >  	JENKINS_UID=$(id -u $USER)
> > +	JENKINS_GID=$(id -g $SUDO_USER)
> >  fi
> >
> > I'm not sure exactly what's going on, but shouldn't that second
> > changed line be?
> >                JENKINS_GID=$(id -g $USER)
> >
> >  -- Tim
> >
> > > -----Original Message-----
> > > From: Daniel Sangorrin [mailto:daniel.sangorrin@toshiba.co.jp]
> > > Sent: Thursday, January 12, 2017 5:30 PM
> > > To: Bird, Timothy <Tim.Bird@am.sony.com>;
> > > Ibe.Kengo@bx.MitsubishiElectric.co.jp
> > > Cc: fuego@lists.linuxfoundation.org
> > > Subject: RE: [Fuego] RFC: Fuego's version up and other changes
> > >
> > > Hi Tim,
> > >
> > > > -----Original Message-----
> > > > From: Bird, Timothy [mailto:Tim.Bird@am.sony.com]
> > > > Sent: Friday, January 13, 2017 10:02 AM
> > > > To: Daniel Sangorrin; Ibe.Kengo@bx.MitsubishiElectric.co.jp
> > > > Cc: fuego@lists.linuxfoundation.org
> > > > Subject: RE: [Fuego] RFC: Fuego's version up and other changes
> > > >
> > > > Daniel,
> > > >
> > > > Thanks for all this work.  I'm having some issues, which may be related
> to
> > > the user and group, so
> > > > I'm replying to the message rather than start a new thread.
> > > >
> > > > Basically I'm having two problems, both during ./install.sh.
> > > >
> > > > 1) I very often have the debian update and extra file installation fail
> (Step
> > > 10 in the Docker image build).
> > > > (noninteractive apt-get update && apt-get -yV install ...)
> > > >
> > > > There are 352 'get' operations during this secondary installation, and if
> even
> > > a single one fails,
> > > > the docker build fails.  I kept having problems getting all the way
> through
> > > this step.  I would
> > > > have network failures from the debian servers.  It seemed to be
> > > intermittent, but overall it
> > > > required that I try to build the docker container about 10 times for it to
> get
> > > past that step.
> > > > Here's a sample set of messages:
> > > >
> > > > Get:95 http://httpredir.debian.org/debian/ jessie/main libatk-wrapper-
> > > java all 0.30.5-1 [30.3 kB]
> > > > Err http://httpredir.debian.org/debian/ jessie/main libavahi-glib1
> amd64
> > > 0.6.31-5
> > > >   Error reading from server. Remote end closed connection [IP:
> 5.153.231.35
> > > 80]
> > >
> > > Yes, I had the same problem yesterday when I installed it on a different
> > > machine.
> > > It seems that httpredir.debian.org is not as reliable as using a local mirror
> > > directly.
> > > I'm not sure if I should add retries or put a default debian server as it used
> to
> > > be.
> > > Could you change httpredir by the US mirror
> > > (http://ftp.us.debian.org/debian) and try again?
> > >
> > > > And, at the end of the failed docker build, this:
> > > > -------
> > > > Fetched 231 MB in 1min 43s (2226 kB/s)
> > > > E: Failed to fetch
> > > http://httpredir.debian.org/debian/pool/main/a/avahi/libavahi-
> glib1_0.6.31-
> > > 5_amd64.deb  Error reading from
> > > > server. Remote end closed connection [IP: 5.153.231.35 80]
> > > >
> > > > E: Unable to fetch some archives, maybe run apt-get update or try with
> --
> > > fix-missing?
> > > > The command '/bin/sh -c DEBIAN_FRONTEND=noninteractive apt-get
> > > update && apt-get -yV install    apt-utils daemon gcc make
> > > > python-paramiko python-lxml python-simplejson         python-
> matplotlib
> > > libtool xmlstarlet autoconf automake rsync openjdk-7-jre
> > > > openjdk-7-jdk iperf netperf netpipe-tcp sshpass wget git        diffstat
> sudo
> > > net-tools vim curl inotify-tools python-openpyxl  g++ bzip2
> > > > bc libaio-dev gettext pkg-config libglib2.0-dev' returned a non-zero
> code:
> > > 100
> > > > --------
> > > >
> > > > It was not the same package failing each time - but the times it failed
> some
> > > of the
> > > > same packages occurred (so, it's hard to pin it on the package or the
> > > server).
> > > >
> > > > Maybe this is just some bad networking at my site, but it did cause me
> > > problems.
> > > > (Or maybe the Debian servers are just having a bad day today).
> > > >
> > > > More problematical is a hang at Step 21:
> > > > Removing intermediate container a522905e9fc2
> > > > Step 21 : RUN groupadd -g ${gid} ${group} 	&& useradd -m -d
> > > "${JENKINS_HOME}" -u ${uid} -g ${gid} -G sudo -s /bin/bash ${user}
> > > >  ---> Running in 106f8bf82d61
> > > >
> > > > The build just hangs there, every time.  The dockerd is running on my
> > > desktop,
> > > > and consuming 110% cpu.
> > > >
> > > > I'm not sure if this is hung in the 'groupadd' step or the 'wget -nv
> > > ${JENKINS_URL}' step.
> > > >
> > > > I'm on Ubuntu 14.04.5 LTS using docker version 1.12.1.
> > >
> > > The wget does take a couple of minutes. But I hadn't experienced the
> > > groupadd hang.
> > >
> > > > dockerd appears to be stuck waiting for a futex.
> > > >
> > > > I'm not sure that the line 'ARG uid=1000' is going to work in my case.
> Here's
> > > my setup:
> > > > $ id
> > > > uid=2060932290(1000072457) gid=2060452353(domain_users)
> > > >
> > >
> groups=2060452353(domain_users),4(adm),20(dialout),24(cdrom),27(sudo),
> > > 46(plugdev),108(lpadmin),124(sambashare),999(docker)
> > > >
> > > > Don’t ask - this is due to my Linux machine being previously
> administered
> > > by corporate types.
> > >
> > > Ok, I think I know what the problem is. I had assumed that the gid would
> be
> > > the same as the uid
> > > in the general case, and other users could just pass their uid/gid in the
> > > docker-build-image.sh
> > > script. I have fixed docker-build-image.sh and added the definition of gid.
> > > Please, could you do a git pull and try again?
> > >
> > >
> https://bitbucket.org/nirrognas/fuego/commits/9d8d1426c326575af8467f73
> > > bce7773d1f45a76c
> > >
> > > > By the way, when I kill the build, dockerd continues to dominate my
> CPU
> > > usage in top
> > > > (and I can hear my desktop cpu fan running).
> > > >
> > > > I think this might be a docker bug, but I'm not sure.
> > >
> > > Sorry about that!
> > >
> > > Thanks for your feedback!!
> > > Daniel
> > >
> > > >
> > > > I'll play around with the Dockerfile (maybe altering the user
> configuration),
> > > but I thought I should report my status so far.
> > > >
> > > > Any ideas?
> > > >  -- Tim
> > > >
> > > > > -----Original Message-----
> > > > > From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> > > > > bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> > > > > Sent: Wednesday, January 11, 2017 10:45 PM
> > > > > To: Ibe.Kengo@bx.MitsubishiElectric.co.jp
> > > > > Cc: fuego@lists.linuxfoundation.org
> > > > > Subject: Re: [Fuego] RFC: Fuego's version up and other changes
> > > > >
> > > > > Hi Kengo-san,
> > > > >
> > > > > Thanks a lot for your feedback.
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Ibe.Kengo@bx.MitsubishiElectric.co.jp
> > > > > [mailto:Ibe.Kengo@bx.MitsubishiElectric.co.jp]
> > > > > > Sent: Thursday, January 12, 2017 2:52 PM
> > > > > > To: Daniel Sangorrin
> > > > > > Cc: fuego@lists.linuxfoundation.org
> > > > > > Subject: Re: [Fuego] RFC: Fuego's version up and other changes
> > > > > >
> > > > > > Hi, Daniel-san.
> > > > > >
> > > > > > I’m very interested in your work!
> > > > > > So I tried to run your build scripts to create Docker image today,
> > > > > > then I could run new Fuego.
> > > > > > And, I added Docker node and test cases to use following
> command.
> > > > > > ==
> > > > > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego.git
> > > > > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego-
> > > core.git
> > > > > >
> > > > > >   $ cd fuego/
> > > > > >   $ ./install.sh
> > > > > >   $ ./fuego-host-scripts/docker-create-container.sh
> > > > > >   $ ./fuego-host-scripts/docker-start-container.sh
> > > > > >   docker# fuego-create-node --board docker
> > > > > >   docker# fuego-create-jobs --board docker --testplan
> testplan_docker
> > > > > > --distrib nosyslogd.dist
> > > > > > ==
> > > > > >
> > > > > > I can see  new node and new test jobs.
> > > > > > But, I can’t run tests like Dhrystone on Docker.
> > > > > >
> > > > > > So far, I had one small problem that is “Failed to mkdirs: /fuego-rw/
> > > > > > buildzone”
> > > > > > This reason is the permission or owner of “fuego-rw”,
> > > > > > so I added the write permission to “fuego-rw” diredtory.
> > > > >
> > > > > Humm You should not need to change any permissions.
> > > > > The script is supposed to use your user's uid/gid for the jenkins
> daemon.
> > > > > However, I have only tested it on my pc which has a single user.
> > > > > Do you have more than one user? Could you check that the jenkins
> > > uid/gid
> > > > > is the same as yours? (you can check on /etc/passwd inside and
> outside
> > > > > docker)
> > > > > Please also check that the fuego-rw's owner is your user (e.g.: don't
> git
> > > clone
> > > > > with
> > > > > a different user or root)
> > > > >
> > > > > > I have solved this problem.
> > > > > >
> > > > > > After that, I got the error message like following in console log.
> > > > > > ====
> > > > > > Started by user anonymous
> > > > > > Building remotely on docker in workspace /fuego-rw/buildzone
> > > > > > [buildzone] $ /bin/sh -xe /tmp/hudson1255554151521556820.sh
> > > > > > + export Reboot=false
> > > > > > + Reboot=false
> > > > > > + export Rebuild=true
> > > > > > + Rebuild=true
> > > > > > + export Target_Cleanup=true
> > > > > > + Target_Cleanup=true
> > > > > > + export TESTDIR=Benchmark.Dhrystone
> > > > > > + TESTDIR=Benchmark.Dhrystone
> > > > > > + export TESTNAME=Dhrystone
> > > > > > + TESTNAME=Dhrystone
> > > > > > + export DISTRIB=nosyslogd.dist
> > > > > > + DISTRIB=nosyslogd.dist
> > > > > > + export TESTPLAN=testplan_docker
> > > > > > + TESTPLAN=testplan_docker
> > > > > > + timeout --signal=9 3m /bin/bash
> > > > > > /fuego-core/engine/tests/Benchmark.Dhrystone/Dhrystone.sh
> > > > > > board overlay: boards/docker.board
> > > > > > using boards/docker.board board overlay
> > > > > > using nosyslogd.dist overlay
> > > > > > BATCH_TESTPLAN is not set, using testplan_docker.json testplan
> > > > > > Traceback (most recent call last):
> > > > > >   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 615, in
> > > <module
> > > > > > >
> > > > > >     run()
> > > > > >   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 607, in run
> > > > > >     generateProlog(output, ofcls, classes, testPlans, testSpecs)
> > > > > >   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 375, in
> > > > > > generateProlog
> > > > > >     outfile = open(outFilePath, "w")
> > > > > > IOError: [Errno 2] No such file or directory: '/fuego-
> rw/work/docker_
> > > > > > prolog.sh'
> > > > > >
> > > > > > *** ABORTED ***
> > > > > >
> > > > > > Fuego error reason: Error while prolog.sh file generation
> > > > > >
> > > > > > Build step 'Execute shell' marked build as failure
> > > > > > [description-setter] Description set:
> > > > > > Finished: FAILURE
> > > > > > ====
> > > > > >
> > > > > > Did I miss to run test cases on Docker node?
> > > > > > If you know this solution, could you give me some advice?
> > > > >
> > > > > Could you check that fuego-rw is correctly mounted inside docker?
> > > > > outside-docker$ touch fuego-rw/hoge
> > > > > inside-docker$ ls /fuego-rw
> > > > > hoge <-- does it appear?
> > > > >
> > > > > On my PC (ubuntu 16.04, docker version 1.12.3 ) it's always succesful
> so I
> > > will
> > > > > try on a different one. Which OS and docker version are you using?
> > > > >
> > > > > Thanks,
> > > > > Daniel
> > > > >
> > > > > > Thank you.
> > > > > >
> > > > > > Best Regard.
> > > > > > Kengo Ibe
> > > > > >
> > > > > > ----- Original Message -----
> > > > > > 件名: [Fuego] RFC: Fuego's version up and other changes
> > > > > > 送信者: "Daniel Sangorrin" <daniel.sangorrin@toshiba.co.jp>
> > > > > > 送信日時: 2017年01月11日(水) 09:28:07
> > > > > > 宛先: "Bird, Timothy" <Tim.Bird@am.sony.com>, fuego@lists.
> > > > > > linuxfoundation.org
> > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > I've been working on Fuego's upgrade for a few weeks, and in the
> > > > > > process I
> > > > > > > have made quite a few changes.
> > > > > > > You can check it by following these instructions. I'd really
> > > > > > appreciate
> > > > > > > your feedback:
> > > > > > >
> > > > > > > Quickstart:
> > > > > > >   $ git clone --branch next
> https://bitbucket.org/nirrognas/fuego.git
> > > > > > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego-
> > > core.git
> > > > > >
> > > > > > >   $ cd fuego/
> > > > > > >   $ ./install.sh
> > > > > > >   $ ./fuego-host-scripts/docker-create-container.sh
> > > > > > >   $ ./fuego-host-scripts/docker-start-container.sh
> > > > > > >   docker# fuego-create-node --board docker
> > > > > > >   docker# fuego-create-jobs --board docker --testplan
> > > testplan_docker
> > > > > > > --distrib nosyslogd.dist
> > > > > > >   Then, start the "docker.testplan_docker.batch" job which will
> > > > > > trigger
> > > > > > > the rest of jobs.
> > > > > > >
> > > > > > > Here is a list of the most fundamental modifications I've added:
> > > > > > > - I automatized the version upgrade process. So far I have
> updated
> > > > > > Jenkins
> > > > > > > 3 times quite smoothly.
> > > > > > >    The current version is the latest LTS version (2.31.1 LTS)
> > > > > > > - I reduced plugin dependencies to the minimum (only 2, and they
> can
> > > > > > be
> > > > > > > made optional).
> > > > > > > - I removed dependencies on all groovy scripts.
> > > > > > > - Userdata is gone. Instead we now have fuego-ro and fuego-rw,
> > > which
> > > > > > > together with fuego-core
> > > > > > >    are mounted as external docker volumes. Only fuego-rw, which
> > > > > > contains
> > > > > > > logs and build folders,
> > > > > > >    is mounted in RW mode. I also fixed the uid/gid of jenkins so
> that
> > > > > > it
> > > > > > > matches the one in the host.
> > > > > > >    This means you can develop comfortably on your host, with GUI
> > > tools,
> > > > > > > and don't fear a buggy
> > > > > > >    script deleting your folders.
> > > > > > > - I added a  tool for generating jobs from testplans.  Creating jobs
> > > > > > from
> > > > > > > testplans is powerful.
> > > > > > >    It allows you, for example, to easily specify the timeout for each
> > > > > > test
> > > > > > > on your board.
> > > > > > >    The tool also generates the corresponding batch job. For
> example,
> > > > > > you
> > > > > > > can easily trigger all
> > > > > > >    tests for your board and a specific testplan periodically.
> > > > > > >    [Note] This will also enable the creation of scripts for comparing
> > > > > > test
> > > > > > > results across
> > > > > > >    different boards and test parameters (on my TODO list).
> > > > > > >
> > > > > > > Other less fundamental changes include:
> > > > > > > - I added support for using the docker container itself as a target
> > > > > > board
> > > > > > > (e.g.: for a quickstart or unit tests)
> > > > > > > - I fixed the flot plugin (java and javascript fixes)
> > > > > > > - Now you can click on a node and see which jobs (and testplans)
> are
> > > > > > > assigned to your board. This was
> > > > > > >   one of my old feature requests and makes Fuego's GUI easier to
> > > use.
> > > > > > > - I added quite a few fixes and improvements to the core engine
> > > > > > scripts
> > > > > > > and tests (too detailed to describe here).
> > > > > > > - I added Excel output support for IOzone
> > > > > > > - Faster docker build time (ARMhf compiler installation is now
> > > > > > optional,
> > > > > > > no latex..)
> > > > > > > - I put a fixed name to the fuego container, instead of the
> > > > > > > "last_container_id" file.
> > > > > > > - I removed the inotify script
> > > > > > > - I added scripts for easily removing the docker container/image.
> > > > > > >
> > > > > > > I have to split the current "huge" patch into smaller chunks and
> test
> > > > > > it
> > > > > > > further but it would
> > > > > > > be nice to get your feedback about the approach in general.
> > > > > > >
> > > > > > > Tim: I haven't updated ftc nor your "Christmas commits" yet.
> > > > > > >
> > > > > > > Best regards
> > > > > > > Daniel
> > > > > > >
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > Fuego mailing list
> > > > > > > Fuego@lists.linuxfoundation.org
> > > > > > > https://lists.linuxfoundation.org/mailman/listinfo/fuego
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > Fuego mailing list
> > > > > Fuego@lists.linuxfoundation.org
> > > > > https://lists.linuxfoundation.org/mailman/listinfo/fuego
> > >
> 
> 

Re: [Fuego] RFC: Fuego's version up and other changes

[Daniel Sangorrin]

oops, sorry I wrote it too quickly. I just fixed it.

> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird@am.sony.com]
> Sent: Friday, January 13, 2017 10:38 AM
> To: Daniel Sangorrin; Ibe.Kengo@bx.MitsubishiElectric.co.jp
> Cc: fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] RFC: Fuego's version up and other changes
> 
> I just pulled it.  Is this right?
> 
> if [ "$(id -u)" == "0" ]; then
>  	JENKINS_UID=$(id -u $SUDO_USER)
> +	JENKINS_GID=$(id -g $SUDO_USER)
>  else
>  	JENKINS_UID=$(id -u $USER)
> +	JENKINS_GID=$(id -g $SUDO_USER)
>  fi
> 
> I'm not sure exactly what's going on, but shouldn't that second
> changed line be?
>                JENKINS_GID=$(id -g $USER)
> 
>  -- Tim
> 
> > -----Original Message-----
> > From: Daniel Sangorrin [mailto:daniel.sangorrin@toshiba.co.jp]
> > Sent: Thursday, January 12, 2017 5:30 PM
> > To: Bird, Timothy <Tim.Bird@am.sony.com>;
> > Ibe.Kengo@bx.MitsubishiElectric.co.jp
> > Cc: fuego@lists.linuxfoundation.org
> > Subject: RE: [Fuego] RFC: Fuego's version up and other changes
> >
> > Hi Tim,
> >
> > > -----Original Message-----
> > > From: Bird, Timothy [mailto:Tim.Bird@am.sony.com]
> > > Sent: Friday, January 13, 2017 10:02 AM
> > > To: Daniel Sangorrin; Ibe.Kengo@bx.MitsubishiElectric.co.jp
> > > Cc: fuego@lists.linuxfoundation.org
> > > Subject: RE: [Fuego] RFC: Fuego's version up and other changes
> > >
> > > Daniel,
> > >
> > > Thanks for all this work.  I'm having some issues, which may be related to
> > the user and group, so
> > > I'm replying to the message rather than start a new thread.
> > >
> > > Basically I'm having two problems, both during ./install.sh.
> > >
> > > 1) I very often have the debian update and extra file installation fail (Step
> > 10 in the Docker image build).
> > > (noninteractive apt-get update && apt-get -yV install ...)
> > >
> > > There are 352 'get' operations during this secondary installation, and if even
> > a single one fails,
> > > the docker build fails.  I kept having problems getting all the way through
> > this step.  I would
> > > have network failures from the debian servers.  It seemed to be
> > intermittent, but overall it
> > > required that I try to build the docker container about 10 times for it to get
> > past that step.
> > > Here's a sample set of messages:
> > >
> > > Get:95 http://httpredir.debian.org/debian/ jessie/main libatk-wrapper-
> > java all 0.30.5-1 [30.3 kB]
> > > Err http://httpredir.debian.org/debian/ jessie/main libavahi-glib1 amd64
> > 0.6.31-5
> > >   Error reading from server. Remote end closed connection [IP: 5.153.231.35
> > 80]
> >
> > Yes, I had the same problem yesterday when I installed it on a different
> > machine.
> > It seems that httpredir.debian.org is not as reliable as using a local mirror
> > directly.
> > I'm not sure if I should add retries or put a default debian server as it used to
> > be.
> > Could you change httpredir by the US mirror
> > (http://ftp.us.debian.org/debian) and try again?
> >
> > > And, at the end of the failed docker build, this:
> > > -------
> > > Fetched 231 MB in 1min 43s (2226 kB/s)
> > > E: Failed to fetch
> > http://httpredir.debian.org/debian/pool/main/a/avahi/libavahi-glib1_0.6.31-
> > 5_amd64.deb  Error reading from
> > > server. Remote end closed connection [IP: 5.153.231.35 80]
> > >
> > > E: Unable to fetch some archives, maybe run apt-get update or try with --
> > fix-missing?
> > > The command '/bin/sh -c DEBIAN_FRONTEND=noninteractive apt-get
> > update && apt-get -yV install    apt-utils daemon gcc make
> > > python-paramiko python-lxml python-simplejson         python-matplotlib
> > libtool xmlstarlet autoconf automake rsync openjdk-7-jre
> > > openjdk-7-jdk iperf netperf netpipe-tcp sshpass wget git        diffstat sudo
> > net-tools vim curl inotify-tools python-openpyxl  g++ bzip2
> > > bc libaio-dev gettext pkg-config libglib2.0-dev' returned a non-zero code:
> > 100
> > > --------
> > >
> > > It was not the same package failing each time - but the times it failed some
> > of the
> > > same packages occurred (so, it's hard to pin it on the package or the
> > server).
> > >
> > > Maybe this is just some bad networking at my site, but it did cause me
> > problems.
> > > (Or maybe the Debian servers are just having a bad day today).
> > >
> > > More problematical is a hang at Step 21:
> > > Removing intermediate container a522905e9fc2
> > > Step 21 : RUN groupadd -g ${gid} ${group} 	&& useradd -m -d
> > "${JENKINS_HOME}" -u ${uid} -g ${gid} -G sudo -s /bin/bash ${user}
> > >  ---> Running in 106f8bf82d61
> > >
> > > The build just hangs there, every time.  The dockerd is running on my
> > desktop,
> > > and consuming 110% cpu.
> > >
> > > I'm not sure if this is hung in the 'groupadd' step or the 'wget -nv
> > ${JENKINS_URL}' step.
> > >
> > > I'm on Ubuntu 14.04.5 LTS using docker version 1.12.1.
> >
> > The wget does take a couple of minutes. But I hadn't experienced the
> > groupadd hang.
> >
> > > dockerd appears to be stuck waiting for a futex.
> > >
> > > I'm not sure that the line 'ARG uid=1000' is going to work in my case.  Here's
> > my setup:
> > > $ id
> > > uid=2060932290(1000072457) gid=2060452353(domain_users)
> > >
> > groups=2060452353(domain_users),4(adm),20(dialout),24(cdrom),27(sudo),
> > 46(plugdev),108(lpadmin),124(sambashare),999(docker)
> > >
> > > Don’t ask - this is due to my Linux machine being previously administered
> > by corporate types.
> >
> > Ok, I think I know what the problem is. I had assumed that the gid would be
> > the same as the uid
> > in the general case, and other users could just pass their uid/gid in the
> > docker-build-image.sh
> > script. I have fixed docker-build-image.sh and added the definition of gid.
> > Please, could you do a git pull and try again?
> >
> > https://bitbucket.org/nirrognas/fuego/commits/9d8d1426c326575af8467f73
> > bce7773d1f45a76c
> >
> > > By the way, when I kill the build, dockerd continues to dominate my CPU
> > usage in top
> > > (and I can hear my desktop cpu fan running).
> > >
> > > I think this might be a docker bug, but I'm not sure.
> >
> > Sorry about that!
> >
> > Thanks for your feedback!!
> > Daniel
> >
> > >
> > > I'll play around with the Dockerfile (maybe altering the user configuration),
> > but I thought I should report my status so far.
> > >
> > > Any ideas?
> > >  -- Tim
> > >
> > > > -----Original Message-----
> > > > From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> > > > bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> > > > Sent: Wednesday, January 11, 2017 10:45 PM
> > > > To: Ibe.Kengo@bx.MitsubishiElectric.co.jp
> > > > Cc: fuego@lists.linuxfoundation.org
> > > > Subject: Re: [Fuego] RFC: Fuego's version up and other changes
> > > >
> > > > Hi Kengo-san,
> > > >
> > > > Thanks a lot for your feedback.
> > > >
> > > > > -----Original Message-----
> > > > > From: Ibe.Kengo@bx.MitsubishiElectric.co.jp
> > > > [mailto:Ibe.Kengo@bx.MitsubishiElectric.co.jp]
> > > > > Sent: Thursday, January 12, 2017 2:52 PM
> > > > > To: Daniel Sangorrin
> > > > > Cc: fuego@lists.linuxfoundation.org
> > > > > Subject: Re: [Fuego] RFC: Fuego's version up and other changes
> > > > >
> > > > > Hi, Daniel-san.
> > > > >
> > > > > I’m very interested in your work!
> > > > > So I tried to run your build scripts to create Docker image today,
> > > > > then I could run new Fuego.
> > > > > And, I added Docker node and test cases to use following command.
> > > > > ==
> > > > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego.git
> > > > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego-
> > core.git
> > > > >
> > > > >   $ cd fuego/
> > > > >   $ ./install.sh
> > > > >   $ ./fuego-host-scripts/docker-create-container.sh
> > > > >   $ ./fuego-host-scripts/docker-start-container.sh
> > > > >   docker# fuego-create-node --board docker
> > > > >   docker# fuego-create-jobs --board docker --testplan testplan_docker
> > > > > --distrib nosyslogd.dist
> > > > > ==
> > > > >
> > > > > I can see  new node and new test jobs.
> > > > > But, I can’t run tests like Dhrystone on Docker.
> > > > >
> > > > > So far, I had one small problem that is “Failed to mkdirs: /fuego-rw/
> > > > > buildzone”
> > > > > This reason is the permission or owner of “fuego-rw”,
> > > > > so I added the write permission to “fuego-rw” diredtory.
> > > >
> > > > Humm You should not need to change any permissions.
> > > > The script is supposed to use your user's uid/gid for the jenkins daemon.
> > > > However, I have only tested it on my pc which has a single user.
> > > > Do you have more than one user? Could you check that the jenkins
> > uid/gid
> > > > is the same as yours? (you can check on /etc/passwd inside and outside
> > > > docker)
> > > > Please also check that the fuego-rw's owner is your user (e.g.: don't git
> > clone
> > > > with
> > > > a different user or root)
> > > >
> > > > > I have solved this problem.
> > > > >
> > > > > After that, I got the error message like following in console log.
> > > > > ====
> > > > > Started by user anonymous
> > > > > Building remotely on docker in workspace /fuego-rw/buildzone
> > > > > [buildzone] $ /bin/sh -xe /tmp/hudson1255554151521556820.sh
> > > > > + export Reboot=false
> > > > > + Reboot=false
> > > > > + export Rebuild=true
> > > > > + Rebuild=true
> > > > > + export Target_Cleanup=true
> > > > > + Target_Cleanup=true
> > > > > + export TESTDIR=Benchmark.Dhrystone
> > > > > + TESTDIR=Benchmark.Dhrystone
> > > > > + export TESTNAME=Dhrystone
> > > > > + TESTNAME=Dhrystone
> > > > > + export DISTRIB=nosyslogd.dist
> > > > > + DISTRIB=nosyslogd.dist
> > > > > + export TESTPLAN=testplan_docker
> > > > > + TESTPLAN=testplan_docker
> > > > > + timeout --signal=9 3m /bin/bash
> > > > > /fuego-core/engine/tests/Benchmark.Dhrystone/Dhrystone.sh
> > > > > board overlay: boards/docker.board
> > > > > using boards/docker.board board overlay
> > > > > using nosyslogd.dist overlay
> > > > > BATCH_TESTPLAN is not set, using testplan_docker.json testplan
> > > > > Traceback (most recent call last):
> > > > >   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 615, in
> > <module
> > > > > >
> > > > >     run()
> > > > >   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 607, in run
> > > > >     generateProlog(output, ofcls, classes, testPlans, testSpecs)
> > > > >   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 375, in
> > > > > generateProlog
> > > > >     outfile = open(outFilePath, "w")
> > > > > IOError: [Errno 2] No such file or directory: '/fuego-rw/work/docker_
> > > > > prolog.sh'
> > > > >
> > > > > *** ABORTED ***
> > > > >
> > > > > Fuego error reason: Error while prolog.sh file generation
> > > > >
> > > > > Build step 'Execute shell' marked build as failure
> > > > > [description-setter] Description set:
> > > > > Finished: FAILURE
> > > > > ====
> > > > >
> > > > > Did I miss to run test cases on Docker node?
> > > > > If you know this solution, could you give me some advice?
> > > >
> > > > Could you check that fuego-rw is correctly mounted inside docker?
> > > > outside-docker$ touch fuego-rw/hoge
> > > > inside-docker$ ls /fuego-rw
> > > > hoge <-- does it appear?
> > > >
> > > > On my PC (ubuntu 16.04, docker version 1.12.3 ) it's always succesful so I
> > will
> > > > try on a different one. Which OS and docker version are you using?
> > > >
> > > > Thanks,
> > > > Daniel
> > > >
> > > > > Thank you.
> > > > >
> > > > > Best Regard.
> > > > > Kengo Ibe
> > > > >
> > > > > ----- Original Message -----
> > > > > 件名: [Fuego] RFC: Fuego's version up and other changes
> > > > > 送信者: "Daniel Sangorrin" <daniel.sangorrin@toshiba.co.jp>
> > > > > 送信日時: 2017年01月11日(水) 09:28:07
> > > > > 宛先: "Bird, Timothy" <Tim.Bird@am.sony.com>, fuego@lists.
> > > > > linuxfoundation.org
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > I've been working on Fuego's upgrade for a few weeks, and in the
> > > > > process I
> > > > > > have made quite a few changes.
> > > > > > You can check it by following these instructions. I'd really
> > > > > appreciate
> > > > > > your feedback:
> > > > > >
> > > > > > Quickstart:
> > > > > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego.git
> > > > > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego-
> > core.git
> > > > >
> > > > > >   $ cd fuego/
> > > > > >   $ ./install.sh
> > > > > >   $ ./fuego-host-scripts/docker-create-container.sh
> > > > > >   $ ./fuego-host-scripts/docker-start-container.sh
> > > > > >   docker# fuego-create-node --board docker
> > > > > >   docker# fuego-create-jobs --board docker --testplan
> > testplan_docker
> > > > > > --distrib nosyslogd.dist
> > > > > >   Then, start the "docker.testplan_docker.batch" job which will
> > > > > trigger
> > > > > > the rest of jobs.
> > > > > >
> > > > > > Here is a list of the most fundamental modifications I've added:
> > > > > > - I automatized the version upgrade process. So far I have updated
> > > > > Jenkins
> > > > > > 3 times quite smoothly.
> > > > > >    The current version is the latest LTS version (2.31.1 LTS)
> > > > > > - I reduced plugin dependencies to the minimum (only 2, and they can
> > > > > be
> > > > > > made optional).
> > > > > > - I removed dependencies on all groovy scripts.
> > > > > > - Userdata is gone. Instead we now have fuego-ro and fuego-rw,
> > which
> > > > > > together with fuego-core
> > > > > >    are mounted as external docker volumes. Only fuego-rw, which
> > > > > contains
> > > > > > logs and build folders,
> > > > > >    is mounted in RW mode. I also fixed the uid/gid of jenkins so that
> > > > > it
> > > > > > matches the one in the host.
> > > > > >    This means you can develop comfortably on your host, with GUI
> > tools,
> > > > > > and don't fear a buggy
> > > > > >    script deleting your folders.
> > > > > > - I added a  tool for generating jobs from testplans.  Creating jobs
> > > > > from
> > > > > > testplans is powerful.
> > > > > >    It allows you, for example, to easily specify the timeout for each
> > > > > test
> > > > > > on your board.
> > > > > >    The tool also generates the corresponding batch job. For example,
> > > > > you
> > > > > > can easily trigger all
> > > > > >    tests for your board and a specific testplan periodically.
> > > > > >    [Note] This will also enable the creation of scripts for comparing
> > > > > test
> > > > > > results across
> > > > > >    different boards and test parameters (on my TODO list).
> > > > > >
> > > > > > Other less fundamental changes include:
> > > > > > - I added support for using the docker container itself as a target
> > > > > board
> > > > > > (e.g.: for a quickstart or unit tests)
> > > > > > - I fixed the flot plugin (java and javascript fixes)
> > > > > > - Now you can click on a node and see which jobs (and testplans) are
> > > > > > assigned to your board. This was
> > > > > >   one of my old feature requests and makes Fuego's GUI easier to
> > use.
> > > > > > - I added quite a few fixes and improvements to the core engine
> > > > > scripts
> > > > > > and tests (too detailed to describe here).
> > > > > > - I added Excel output support for IOzone
> > > > > > - Faster docker build time (ARMhf compiler installation is now
> > > > > optional,
> > > > > > no latex..)
> > > > > > - I put a fixed name to the fuego container, instead of the
> > > > > > "last_container_id" file.
> > > > > > - I removed the inotify script
> > > > > > - I added scripts for easily removing the docker container/image.
> > > > > >
> > > > > > I have to split the current "huge" patch into smaller chunks and test
> > > > > it
> > > > > > further but it would
> > > > > > be nice to get your feedback about the approach in general.
> > > > > >
> > > > > > Tim: I haven't updated ftc nor your "Christmas commits" yet.
> > > > > >
> > > > > > Best regards
> > > > > > Daniel
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > Fuego mailing list
> > > > > > Fuego@lists.linuxfoundation.org
> > > > > > https://lists.linuxfoundation.org/mailman/listinfo/fuego
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > Fuego mailing list
> > > > Fuego@lists.linuxfoundation.org
> > > > https://lists.linuxfoundation.org/mailman/listinfo/fuego
> >

Re: [Fuego] RFC: Fuego's version up and other changes

[Bird, Timothy]

I just pulled it.  Is this right?

if [ "$(id -u)" == "0" ]; then
 	JENKINS_UID=$(id -u $SUDO_USER)
+	JENKINS_GID=$(id -g $SUDO_USER)
 else
 	JENKINS_UID=$(id -u $USER)
+	JENKINS_GID=$(id -g $SUDO_USER)
 fi

I'm not sure exactly what's going on, but shouldn't that second
changed line be?
               JENKINS_GID=$(id -g $USER)

 -- Tim

> -----Original Message-----
> From: Daniel Sangorrin [mailto:daniel.sangorrin@toshiba.co.jp]
> Sent: Thursday, January 12, 2017 5:30 PM
> To: Bird, Timothy <Tim.Bird@am.sony.com>;
> Ibe.Kengo@bx.MitsubishiElectric.co.jp
> Cc: fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] RFC: Fuego's version up and other changes
> 
> Hi Tim,
> 
> > -----Original Message-----
> > From: Bird, Timothy [mailto:Tim.Bird@am.sony.com]
> > Sent: Friday, January 13, 2017 10:02 AM
> > To: Daniel Sangorrin; Ibe.Kengo@bx.MitsubishiElectric.co.jp
> > Cc: fuego@lists.linuxfoundation.org
> > Subject: RE: [Fuego] RFC: Fuego's version up and other changes
> >
> > Daniel,
> >
> > Thanks for all this work.  I'm having some issues, which may be related to
> the user and group, so
> > I'm replying to the message rather than start a new thread.
> >
> > Basically I'm having two problems, both during ./install.sh.
> >
> > 1) I very often have the debian update and extra file installation fail (Step
> 10 in the Docker image build).
> > (noninteractive apt-get update && apt-get -yV install ...)
> >
> > There are 352 'get' operations during this secondary installation, and if even
> a single one fails,
> > the docker build fails.  I kept having problems getting all the way through
> this step.  I would
> > have network failures from the debian servers.  It seemed to be
> intermittent, but overall it
> > required that I try to build the docker container about 10 times for it to get
> past that step.
> > Here's a sample set of messages:
> >
> > Get:95 http://httpredir.debian.org/debian/ jessie/main libatk-wrapper-
> java all 0.30.5-1 [30.3 kB]
> > Err http://httpredir.debian.org/debian/ jessie/main libavahi-glib1 amd64
> 0.6.31-5
> >   Error reading from server. Remote end closed connection [IP: 5.153.231.35
> 80]
> 
> Yes, I had the same problem yesterday when I installed it on a different
> machine.
> It seems that httpredir.debian.org is not as reliable as using a local mirror
> directly.
> I'm not sure if I should add retries or put a default debian server as it used to
> be.
> Could you change httpredir by the US mirror
> (http://ftp.us.debian.org/debian) and try again?
> 
> > And, at the end of the failed docker build, this:
> > -------
> > Fetched 231 MB in 1min 43s (2226 kB/s)
> > E: Failed to fetch
> http://httpredir.debian.org/debian/pool/main/a/avahi/libavahi-glib1_0.6.31-
> 5_amd64.deb  Error reading from
> > server. Remote end closed connection [IP: 5.153.231.35 80]
> >
> > E: Unable to fetch some archives, maybe run apt-get update or try with --
> fix-missing?
> > The command '/bin/sh -c DEBIAN_FRONTEND=noninteractive apt-get
> update && apt-get -yV install    apt-utils daemon gcc make
> > python-paramiko python-lxml python-simplejson         python-matplotlib
> libtool xmlstarlet autoconf automake rsync openjdk-7-jre
> > openjdk-7-jdk iperf netperf netpipe-tcp sshpass wget git        diffstat sudo
> net-tools vim curl inotify-tools python-openpyxl  g++ bzip2
> > bc libaio-dev gettext pkg-config libglib2.0-dev' returned a non-zero code:
> 100
> > --------
> >
> > It was not the same package failing each time - but the times it failed some
> of the
> > same packages occurred (so, it's hard to pin it on the package or the
> server).
> >
> > Maybe this is just some bad networking at my site, but it did cause me
> problems.
> > (Or maybe the Debian servers are just having a bad day today).
> >
> > More problematical is a hang at Step 21:
> > Removing intermediate container a522905e9fc2
> > Step 21 : RUN groupadd -g ${gid} ${group} 	&& useradd -m -d
> "${JENKINS_HOME}" -u ${uid} -g ${gid} -G sudo -s /bin/bash ${user}
> >  ---> Running in 106f8bf82d61
> >
> > The build just hangs there, every time.  The dockerd is running on my
> desktop,
> > and consuming 110% cpu.
> >
> > I'm not sure if this is hung in the 'groupadd' step or the 'wget -nv
> ${JENKINS_URL}' step.
> >
> > I'm on Ubuntu 14.04.5 LTS using docker version 1.12.1.
> 
> The wget does take a couple of minutes. But I hadn't experienced the
> groupadd hang.
> 
> > dockerd appears to be stuck waiting for a futex.
> >
> > I'm not sure that the line 'ARG uid=1000' is going to work in my case.  Here's
> my setup:
> > $ id
> > uid=2060932290(1000072457) gid=2060452353(domain_users)
> >
> groups=2060452353(domain_users),4(adm),20(dialout),24(cdrom),27(sudo),
> 46(plugdev),108(lpadmin),124(sambashare),999(docker)
> >
> > Don’t ask - this is due to my Linux machine being previously administered
> by corporate types.
> 
> Ok, I think I know what the problem is. I had assumed that the gid would be
> the same as the uid
> in the general case, and other users could just pass their uid/gid in the
> docker-build-image.sh
> script. I have fixed docker-build-image.sh and added the definition of gid.
> Please, could you do a git pull and try again?
> 
> https://bitbucket.org/nirrognas/fuego/commits/9d8d1426c326575af8467f73
> bce7773d1f45a76c
> 
> > By the way, when I kill the build, dockerd continues to dominate my CPU
> usage in top
> > (and I can hear my desktop cpu fan running).
> >
> > I think this might be a docker bug, but I'm not sure.
> 
> Sorry about that!
> 
> Thanks for your feedback!!
> Daniel
> 
> >
> > I'll play around with the Dockerfile (maybe altering the user configuration),
> but I thought I should report my status so far.
> >
> > Any ideas?
> >  -- Tim
> >
> > > -----Original Message-----
> > > From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> > > bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> > > Sent: Wednesday, January 11, 2017 10:45 PM
> > > To: Ibe.Kengo@bx.MitsubishiElectric.co.jp
> > > Cc: fuego@lists.linuxfoundation.org
> > > Subject: Re: [Fuego] RFC: Fuego's version up and other changes
> > >
> > > Hi Kengo-san,
> > >
> > > Thanks a lot for your feedback.
> > >
> > > > -----Original Message-----
> > > > From: Ibe.Kengo@bx.MitsubishiElectric.co.jp
> > > [mailto:Ibe.Kengo@bx.MitsubishiElectric.co.jp]
> > > > Sent: Thursday, January 12, 2017 2:52 PM
> > > > To: Daniel Sangorrin
> > > > Cc: fuego@lists.linuxfoundation.org
> > > > Subject: Re: [Fuego] RFC: Fuego's version up and other changes
> > > >
> > > > Hi, Daniel-san.
> > > >
> > > > I’m very interested in your work!
> > > > So I tried to run your build scripts to create Docker image today,
> > > > then I could run new Fuego.
> > > > And, I added Docker node and test cases to use following command.
> > > > ==
> > > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego.git
> > > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego-
> core.git
> > > >
> > > >   $ cd fuego/
> > > >   $ ./install.sh
> > > >   $ ./fuego-host-scripts/docker-create-container.sh
> > > >   $ ./fuego-host-scripts/docker-start-container.sh
> > > >   docker# fuego-create-node --board docker
> > > >   docker# fuego-create-jobs --board docker --testplan testplan_docker
> > > > --distrib nosyslogd.dist
> > > > ==
> > > >
> > > > I can see  new node and new test jobs.
> > > > But, I can’t run tests like Dhrystone on Docker.
> > > >
> > > > So far, I had one small problem that is “Failed to mkdirs: /fuego-rw/
> > > > buildzone”
> > > > This reason is the permission or owner of “fuego-rw”,
> > > > so I added the write permission to “fuego-rw” diredtory.
> > >
> > > Humm You should not need to change any permissions.
> > > The script is supposed to use your user's uid/gid for the jenkins daemon.
> > > However, I have only tested it on my pc which has a single user.
> > > Do you have more than one user? Could you check that the jenkins
> uid/gid
> > > is the same as yours? (you can check on /etc/passwd inside and outside
> > > docker)
> > > Please also check that the fuego-rw's owner is your user (e.g.: don't git
> clone
> > > with
> > > a different user or root)
> > >
> > > > I have solved this problem.
> > > >
> > > > After that, I got the error message like following in console log.
> > > > ====
> > > > Started by user anonymous
> > > > Building remotely on docker in workspace /fuego-rw/buildzone
> > > > [buildzone] $ /bin/sh -xe /tmp/hudson1255554151521556820.sh
> > > > + export Reboot=false
> > > > + Reboot=false
> > > > + export Rebuild=true
> > > > + Rebuild=true
> > > > + export Target_Cleanup=true
> > > > + Target_Cleanup=true
> > > > + export TESTDIR=Benchmark.Dhrystone
> > > > + TESTDIR=Benchmark.Dhrystone
> > > > + export TESTNAME=Dhrystone
> > > > + TESTNAME=Dhrystone
> > > > + export DISTRIB=nosyslogd.dist
> > > > + DISTRIB=nosyslogd.dist
> > > > + export TESTPLAN=testplan_docker
> > > > + TESTPLAN=testplan_docker
> > > > + timeout --signal=9 3m /bin/bash
> > > > /fuego-core/engine/tests/Benchmark.Dhrystone/Dhrystone.sh
> > > > board overlay: boards/docker.board
> > > > using boards/docker.board board overlay
> > > > using nosyslogd.dist overlay
> > > > BATCH_TESTPLAN is not set, using testplan_docker.json testplan
> > > > Traceback (most recent call last):
> > > >   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 615, in
> <module
> > > > >
> > > >     run()
> > > >   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 607, in run
> > > >     generateProlog(output, ofcls, classes, testPlans, testSpecs)
> > > >   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 375, in
> > > > generateProlog
> > > >     outfile = open(outFilePath, "w")
> > > > IOError: [Errno 2] No such file or directory: '/fuego-rw/work/docker_
> > > > prolog.sh'
> > > >
> > > > *** ABORTED ***
> > > >
> > > > Fuego error reason: Error while prolog.sh file generation
> > > >
> > > > Build step 'Execute shell' marked build as failure
> > > > [description-setter] Description set:
> > > > Finished: FAILURE
> > > > ====
> > > >
> > > > Did I miss to run test cases on Docker node?
> > > > If you know this solution, could you give me some advice?
> > >
> > > Could you check that fuego-rw is correctly mounted inside docker?
> > > outside-docker$ touch fuego-rw/hoge
> > > inside-docker$ ls /fuego-rw
> > > hoge <-- does it appear?
> > >
> > > On my PC (ubuntu 16.04, docker version 1.12.3 ) it's always succesful so I
> will
> > > try on a different one. Which OS and docker version are you using?
> > >
> > > Thanks,
> > > Daniel
> > >
> > > > Thank you.
> > > >
> > > > Best Regard.
> > > > Kengo Ibe
> > > >
> > > > ----- Original Message -----
> > > > 件名: [Fuego] RFC: Fuego's version up and other changes
> > > > 送信者: "Daniel Sangorrin" <daniel.sangorrin@toshiba.co.jp>
> > > > 送信日時: 2017年01月11日(水) 09:28:07
> > > > 宛先: "Bird, Timothy" <Tim.Bird@am.sony.com>, fuego@lists.
> > > > linuxfoundation.org
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > I've been working on Fuego's upgrade for a few weeks, and in the
> > > > process I
> > > > > have made quite a few changes.
> > > > > You can check it by following these instructions. I'd really
> > > > appreciate
> > > > > your feedback:
> > > > >
> > > > > Quickstart:
> > > > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego.git
> > > > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego-
> core.git
> > > >
> > > > >   $ cd fuego/
> > > > >   $ ./install.sh
> > > > >   $ ./fuego-host-scripts/docker-create-container.sh
> > > > >   $ ./fuego-host-scripts/docker-start-container.sh
> > > > >   docker# fuego-create-node --board docker
> > > > >   docker# fuego-create-jobs --board docker --testplan
> testplan_docker
> > > > > --distrib nosyslogd.dist
> > > > >   Then, start the "docker.testplan_docker.batch" job which will
> > > > trigger
> > > > > the rest of jobs.
> > > > >
> > > > > Here is a list of the most fundamental modifications I've added:
> > > > > - I automatized the version upgrade process. So far I have updated
> > > > Jenkins
> > > > > 3 times quite smoothly.
> > > > >    The current version is the latest LTS version (2.31.1 LTS)
> > > > > - I reduced plugin dependencies to the minimum (only 2, and they can
> > > > be
> > > > > made optional).
> > > > > - I removed dependencies on all groovy scripts.
> > > > > - Userdata is gone. Instead we now have fuego-ro and fuego-rw,
> which
> > > > > together with fuego-core
> > > > >    are mounted as external docker volumes. Only fuego-rw, which
> > > > contains
> > > > > logs and build folders,
> > > > >    is mounted in RW mode. I also fixed the uid/gid of jenkins so that
> > > > it
> > > > > matches the one in the host.
> > > > >    This means you can develop comfortably on your host, with GUI
> tools,
> > > > > and don't fear a buggy
> > > > >    script deleting your folders.
> > > > > - I added a  tool for generating jobs from testplans.  Creating jobs
> > > > from
> > > > > testplans is powerful.
> > > > >    It allows you, for example, to easily specify the timeout for each
> > > > test
> > > > > on your board.
> > > > >    The tool also generates the corresponding batch job. For example,
> > > > you
> > > > > can easily trigger all
> > > > >    tests for your board and a specific testplan periodically.
> > > > >    [Note] This will also enable the creation of scripts for comparing
> > > > test
> > > > > results across
> > > > >    different boards and test parameters (on my TODO list).
> > > > >
> > > > > Other less fundamental changes include:
> > > > > - I added support for using the docker container itself as a target
> > > > board
> > > > > (e.g.: for a quickstart or unit tests)
> > > > > - I fixed the flot plugin (java and javascript fixes)
> > > > > - Now you can click on a node and see which jobs (and testplans) are
> > > > > assigned to your board. This was
> > > > >   one of my old feature requests and makes Fuego's GUI easier to
> use.
> > > > > - I added quite a few fixes and improvements to the core engine
> > > > scripts
> > > > > and tests (too detailed to describe here).
> > > > > - I added Excel output support for IOzone
> > > > > - Faster docker build time (ARMhf compiler installation is now
> > > > optional,
> > > > > no latex..)
> > > > > - I put a fixed name to the fuego container, instead of the
> > > > > "last_container_id" file.
> > > > > - I removed the inotify script
> > > > > - I added scripts for easily removing the docker container/image.
> > > > >
> > > > > I have to split the current "huge" patch into smaller chunks and test
> > > > it
> > > > > further but it would
> > > > > be nice to get your feedback about the approach in general.
> > > > >
> > > > > Tim: I haven't updated ftc nor your "Christmas commits" yet.
> > > > >
> > > > > Best regards
> > > > > Daniel
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > Fuego mailing list
> > > > > Fuego@lists.linuxfoundation.org
> > > > > https://lists.linuxfoundation.org/mailman/listinfo/fuego
> > > > >
> > > >
> > >
> > >
> > >
> > > _______________________________________________
> > > Fuego mailing list
> > > Fuego@lists.linuxfoundation.org
> > > https://lists.linuxfoundation.org/mailman/listinfo/fuego
> 

Re: [Fuego] RFC: Fuego's version up and other changes

[Daniel Sangorrin]

Hi Tim,

> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird@am.sony.com]
> Sent: Friday, January 13, 2017 10:02 AM
> To: Daniel Sangorrin; Ibe.Kengo@bx.MitsubishiElectric.co.jp
> Cc: fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] RFC: Fuego's version up and other changes
> 
> Daniel,
> 
> Thanks for all this work.  I'm having some issues, which may be related to the user and group, so
> I'm replying to the message rather than start a new thread.
> 
> Basically I'm having two problems, both during ./install.sh.
> 
> 1) I very often have the debian update and extra file installation fail (Step 10 in the Docker image build).
> (noninteractive apt-get update && apt-get -yV install ...)
> 
> There are 352 'get' operations during this secondary installation, and if even a single one fails,
> the docker build fails.  I kept having problems getting all the way through this step.  I would
> have network failures from the debian servers.  It seemed to be intermittent, but overall it
> required that I try to build the docker container about 10 times for it to get past that step.
> Here's a sample set of messages:
> 
> Get:95 http://httpredir.debian.org/debian/ jessie/main libatk-wrapper-java all 0.30.5-1 [30.3 kB]
> Err http://httpredir.debian.org/debian/ jessie/main libavahi-glib1 amd64 0.6.31-5
>   Error reading from server. Remote end closed connection [IP: 5.153.231.35 80]

Yes, I had the same problem yesterday when I installed it on a different machine.
It seems that httpredir.debian.org is not as reliable as using a local mirror directly.
I'm not sure if I should add retries or put a default debian server as it used to be.
Could you change httpredir by the US mirror (http://ftp.us.debian.org/debian) and try again?

> And, at the end of the failed docker build, this:
> -------
> Fetched 231 MB in 1min 43s (2226 kB/s)
> E: Failed to fetch http://httpredir.debian.org/debian/pool/main/a/avahi/libavahi-glib1_0.6.31-5_amd64.deb  Error reading from
> server. Remote end closed connection [IP: 5.153.231.35 80]
> 
> E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
> The command '/bin/sh -c DEBIAN_FRONTEND=noninteractive apt-get update && apt-get -yV install    apt-utils daemon gcc make
> python-paramiko python-lxml python-simplejson         python-matplotlib libtool xmlstarlet autoconf automake rsync openjdk-7-jre
> openjdk-7-jdk iperf netperf netpipe-tcp sshpass wget git        diffstat sudo net-tools vim curl inotify-tools python-openpyxl  g++ bzip2
> bc libaio-dev gettext pkg-config libglib2.0-dev' returned a non-zero code: 100
> --------
> 
> It was not the same package failing each time - but the times it failed some of the
> same packages occurred (so, it's hard to pin it on the package or the server).
> 
> Maybe this is just some bad networking at my site, but it did cause me problems.
> (Or maybe the Debian servers are just having a bad day today).
> 
> More problematical is a hang at Step 21:
> Removing intermediate container a522905e9fc2
> Step 21 : RUN groupadd -g ${gid} ${group} 	&& useradd -m -d "${JENKINS_HOME}" -u ${uid} -g ${gid} -G sudo -s /bin/bash ${user}
>  ---> Running in 106f8bf82d61
> 
> The build just hangs there, every time.  The dockerd is running on my desktop,
> and consuming 110% cpu.
> 
> I'm not sure if this is hung in the 'groupadd' step or the 'wget -nv ${JENKINS_URL}' step.
> 
> I'm on Ubuntu 14.04.5 LTS using docker version 1.12.1.

The wget does take a couple of minutes. But I hadn't experienced the groupadd hang.

> dockerd appears to be stuck waiting for a futex.
> 
> I'm not sure that the line 'ARG uid=1000' is going to work in my case.  Here's my setup:
> $ id
> uid=2060932290(1000072457) gid=2060452353(domain_users)
> groups=2060452353(domain_users),4(adm),20(dialout),24(cdrom),27(sudo),46(plugdev),108(lpadmin),124(sambashare),999(docker)
> 
> Don’t ask - this is due to my Linux machine being previously administered by corporate types.

Ok, I think I know what the problem is. I had assumed that the gid would be the same as the uid
in the general case, and other users could just pass their uid/gid in the docker-build-image.sh
script. I have fixed docker-build-image.sh and added the definition of gid.
Please, could you do a git pull and try again?

https://bitbucket.org/nirrognas/fuego/commits/9d8d1426c326575af8467f73bce7773d1f45a76c

> By the way, when I kill the build, dockerd continues to dominate my CPU usage in top
> (and I can hear my desktop cpu fan running).
> 
> I think this might be a docker bug, but I'm not sure.

Sorry about that!

Thanks for your feedback!!
Daniel

> 
> I'll play around with the Dockerfile (maybe altering the user configuration), but I thought I should report my status so far.
> 
> Any ideas?
>  -- Tim
> 
> > -----Original Message-----
> > From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> > bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> > Sent: Wednesday, January 11, 2017 10:45 PM
> > To: Ibe.Kengo@bx.MitsubishiElectric.co.jp
> > Cc: fuego@lists.linuxfoundation.org
> > Subject: Re: [Fuego] RFC: Fuego's version up and other changes
> >
> > Hi Kengo-san,
> >
> > Thanks a lot for your feedback.
> >
> > > -----Original Message-----
> > > From: Ibe.Kengo@bx.MitsubishiElectric.co.jp
> > [mailto:Ibe.Kengo@bx.MitsubishiElectric.co.jp]
> > > Sent: Thursday, January 12, 2017 2:52 PM
> > > To: Daniel Sangorrin
> > > Cc: fuego@lists.linuxfoundation.org
> > > Subject: Re: [Fuego] RFC: Fuego's version up and other changes
> > >
> > > Hi, Daniel-san.
> > >
> > > I’m very interested in your work!
> > > So I tried to run your build scripts to create Docker image today,
> > > then I could run new Fuego.
> > > And, I added Docker node and test cases to use following command.
> > > ==
> > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego.git
> > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego-core.git
> > >
> > >   $ cd fuego/
> > >   $ ./install.sh
> > >   $ ./fuego-host-scripts/docker-create-container.sh
> > >   $ ./fuego-host-scripts/docker-start-container.sh
> > >   docker# fuego-create-node --board docker
> > >   docker# fuego-create-jobs --board docker --testplan testplan_docker
> > > --distrib nosyslogd.dist
> > > ==
> > >
> > > I can see  new node and new test jobs.
> > > But, I can’t run tests like Dhrystone on Docker.
> > >
> > > So far, I had one small problem that is “Failed to mkdirs: /fuego-rw/
> > > buildzone”
> > > This reason is the permission or owner of “fuego-rw”,
> > > so I added the write permission to “fuego-rw” diredtory.
> >
> > Humm You should not need to change any permissions.
> > The script is supposed to use your user's uid/gid for the jenkins daemon.
> > However, I have only tested it on my pc which has a single user.
> > Do you have more than one user? Could you check that the jenkins uid/gid
> > is the same as yours? (you can check on /etc/passwd inside and outside
> > docker)
> > Please also check that the fuego-rw's owner is your user (e.g.: don't git clone
> > with
> > a different user or root)
> >
> > > I have solved this problem.
> > >
> > > After that, I got the error message like following in console log.
> > > ====
> > > Started by user anonymous
> > > Building remotely on docker in workspace /fuego-rw/buildzone
> > > [buildzone] $ /bin/sh -xe /tmp/hudson1255554151521556820.sh
> > > + export Reboot=false
> > > + Reboot=false
> > > + export Rebuild=true
> > > + Rebuild=true
> > > + export Target_Cleanup=true
> > > + Target_Cleanup=true
> > > + export TESTDIR=Benchmark.Dhrystone
> > > + TESTDIR=Benchmark.Dhrystone
> > > + export TESTNAME=Dhrystone
> > > + TESTNAME=Dhrystone
> > > + export DISTRIB=nosyslogd.dist
> > > + DISTRIB=nosyslogd.dist
> > > + export TESTPLAN=testplan_docker
> > > + TESTPLAN=testplan_docker
> > > + timeout --signal=9 3m /bin/bash
> > > /fuego-core/engine/tests/Benchmark.Dhrystone/Dhrystone.sh
> > > board overlay: boards/docker.board
> > > using boards/docker.board board overlay
> > > using nosyslogd.dist overlay
> > > BATCH_TESTPLAN is not set, using testplan_docker.json testplan
> > > Traceback (most recent call last):
> > >   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 615, in <module
> > > >
> > >     run()
> > >   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 607, in run
> > >     generateProlog(output, ofcls, classes, testPlans, testSpecs)
> > >   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 375, in
> > > generateProlog
> > >     outfile = open(outFilePath, "w")
> > > IOError: [Errno 2] No such file or directory: '/fuego-rw/work/docker_
> > > prolog.sh'
> > >
> > > *** ABORTED ***
> > >
> > > Fuego error reason: Error while prolog.sh file generation
> > >
> > > Build step 'Execute shell' marked build as failure
> > > [description-setter] Description set:
> > > Finished: FAILURE
> > > ====
> > >
> > > Did I miss to run test cases on Docker node?
> > > If you know this solution, could you give me some advice?
> >
> > Could you check that fuego-rw is correctly mounted inside docker?
> > outside-docker$ touch fuego-rw/hoge
> > inside-docker$ ls /fuego-rw
> > hoge <-- does it appear?
> >
> > On my PC (ubuntu 16.04, docker version 1.12.3 ) it's always succesful so I will
> > try on a different one. Which OS and docker version are you using?
> >
> > Thanks,
> > Daniel
> >
> > > Thank you.
> > >
> > > Best Regard.
> > > Kengo Ibe
> > >
> > > ----- Original Message -----
> > > 件名: [Fuego] RFC: Fuego's version up and other changes
> > > 送信者: "Daniel Sangorrin" <daniel.sangorrin@toshiba.co.jp>
> > > 送信日時: 2017年01月11日(水) 09:28:07
> > > 宛先: "Bird, Timothy" <Tim.Bird@am.sony.com>, fuego@lists.
> > > linuxfoundation.org
> > >
> > > > Hi everyone,
> > > >
> > > > I've been working on Fuego's upgrade for a few weeks, and in the
> > > process I
> > > > have made quite a few changes.
> > > > You can check it by following these instructions. I'd really
> > > appreciate
> > > > your feedback:
> > > >
> > > > Quickstart:
> > > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego.git
> > > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego-core.git
> > >
> > > >   $ cd fuego/
> > > >   $ ./install.sh
> > > >   $ ./fuego-host-scripts/docker-create-container.sh
> > > >   $ ./fuego-host-scripts/docker-start-container.sh
> > > >   docker# fuego-create-node --board docker
> > > >   docker# fuego-create-jobs --board docker --testplan testplan_docker
> > > > --distrib nosyslogd.dist
> > > >   Then, start the "docker.testplan_docker.batch" job which will
> > > trigger
> > > > the rest of jobs.
> > > >
> > > > Here is a list of the most fundamental modifications I've added:
> > > > - I automatized the version upgrade process. So far I have updated
> > > Jenkins
> > > > 3 times quite smoothly.
> > > >    The current version is the latest LTS version (2.31.1 LTS)
> > > > - I reduced plugin dependencies to the minimum (only 2, and they can
> > > be
> > > > made optional).
> > > > - I removed dependencies on all groovy scripts.
> > > > - Userdata is gone. Instead we now have fuego-ro and fuego-rw, which
> > > > together with fuego-core
> > > >    are mounted as external docker volumes. Only fuego-rw, which
> > > contains
> > > > logs and build folders,
> > > >    is mounted in RW mode. I also fixed the uid/gid of jenkins so that
> > > it
> > > > matches the one in the host.
> > > >    This means you can develop comfortably on your host, with GUI tools,
> > > > and don't fear a buggy
> > > >    script deleting your folders.
> > > > - I added a  tool for generating jobs from testplans.  Creating jobs
> > > from
> > > > testplans is powerful.
> > > >    It allows you, for example, to easily specify the timeout for each
> > > test
> > > > on your board.
> > > >    The tool also generates the corresponding batch job. For example,
> > > you
> > > > can easily trigger all
> > > >    tests for your board and a specific testplan periodically.
> > > >    [Note] This will also enable the creation of scripts for comparing
> > > test
> > > > results across
> > > >    different boards and test parameters (on my TODO list).
> > > >
> > > > Other less fundamental changes include:
> > > > - I added support for using the docker container itself as a target
> > > board
> > > > (e.g.: for a quickstart or unit tests)
> > > > - I fixed the flot plugin (java and javascript fixes)
> > > > - Now you can click on a node and see which jobs (and testplans) are
> > > > assigned to your board. This was
> > > >   one of my old feature requests and makes Fuego's GUI easier to use.
> > > > - I added quite a few fixes and improvements to the core engine
> > > scripts
> > > > and tests (too detailed to describe here).
> > > > - I added Excel output support for IOzone
> > > > - Faster docker build time (ARMhf compiler installation is now
> > > optional,
> > > > no latex..)
> > > > - I put a fixed name to the fuego container, instead of the
> > > > "last_container_id" file.
> > > > - I removed the inotify script
> > > > - I added scripts for easily removing the docker container/image.
> > > >
> > > > I have to split the current "huge" patch into smaller chunks and test
> > > it
> > > > further but it would
> > > > be nice to get your feedback about the approach in general.
> > > >
> > > > Tim: I haven't updated ftc nor your "Christmas commits" yet.
> > > >
> > > > Best regards
> > > > Daniel
> > > >
> > > >
> > > > _______________________________________________
> > > > Fuego mailing list
> > > > Fuego@lists.linuxfoundation.org
> > > > https://lists.linuxfoundation.org/mailman/listinfo/fuego
> > > >
> > >
> >
> >
> >
> > _______________________________________________
> > Fuego mailing list
> > Fuego@lists.linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] RFC: Fuego's version up and other changes

[Bird, Timothy]

Daniel,

Thanks for all this work.  I'm having some issues, which may be related to the user and group, so 
I'm replying to the message rather than start a new thread.

Basically I'm having two problems, both during ./install.sh.

1) I very often have the debian update and extra file installation fail (Step 10 in the Docker image build).
(noninteractive apt-get update && apt-get -yV install ...)

There are 352 'get' operations during this secondary installation, and if even a single one fails, 
the docker build fails.  I kept having problems getting all the way through this step.  I would
have network failures from the debian servers.  It seemed to be intermittent, but overall it
required that I try to build the docker container about 10 times for it to get past that step.
Here's a sample set of messages:

Get:95 http://httpredir.debian.org/debian/ jessie/main libatk-wrapper-java all 0.30.5-1 [30.3 kB]
Err http://httpredir.debian.org/debian/ jessie/main libavahi-glib1 amd64 0.6.31-5
  Error reading from server. Remote end closed connection [IP: 5.153.231.35 80]

And, at the end of the failed docker build, this:
-------
Fetched 231 MB in 1min 43s (2226 kB/s)
E: Failed to fetch http://httpredir.debian.org/debian/pool/main/a/avahi/libavahi-glib1_0.6.31-5_amd64.deb  Error reading from server. Remote end closed connection [IP: 5.153.231.35 80]

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
The command '/bin/sh -c DEBIAN_FRONTEND=noninteractive apt-get update && apt-get -yV install    apt-utils daemon gcc make python-paramiko python-lxml python-simplejson         python-matplotlib libtool xmlstarlet autoconf automake rsync openjdk-7-jre      openjdk-7-jdk iperf netperf netpipe-tcp sshpass wget git        diffstat sudo net-tools vim curl inotify-tools python-openpyxl  g++ bzip2 bc libaio-dev gettext pkg-config libglib2.0-dev' returned a non-zero code: 100
--------

It was not the same package failing each time - but the times it failed some of the
same packages occurred (so, it's hard to pin it on the package or the server).

Maybe this is just some bad networking at my site, but it did cause me problems.
(Or maybe the Debian servers are just having a bad day today).

More problematical is a hang at Step 21:
Removing intermediate container a522905e9fc2
Step 21 : RUN groupadd -g ${gid} ${group} 	&& useradd -m -d "${JENKINS_HOME}" -u ${uid} -g ${gid} -G sudo -s /bin/bash ${user}
 ---> Running in 106f8bf82d61

The build just hangs there, every time.  The dockerd is running on my desktop, 
and consuming 110% cpu.

I'm not sure if this is hung in the 'groupadd' step or the 'wget -nv ${JENKINS_URL}' step.

I'm on Ubuntu 14.04.5 LTS using docker version 1.12.1.

dockerd appears to be stuck waiting for a futex.

I'm not sure that the line 'ARG uid=1000' is going to work in my case.  Here's my setup:
$ id
uid=2060932290(1000072457) gid=2060452353(domain_users) groups=2060452353(domain_users),4(adm),20(dialout),24(cdrom),27(sudo),46(plugdev),108(lpadmin),124(sambashare),999(docker)

Don’t ask - this is due to my Linux machine being previously administered by corporate types.

By the way, when I kill the build, dockerd continues to dominate my CPU usage in top
(and I can hear my desktop cpu fan running).

I think this might be a docker bug, but I'm not sure.

I'll play around with the Dockerfile (maybe altering the user configuration), but I thought I should report my status so far.

Any ideas?
 -- Tim

> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> Sent: Wednesday, January 11, 2017 10:45 PM
> To: Ibe.Kengo@bx.MitsubishiElectric.co.jp
> Cc: fuego@lists.linuxfoundation.org
> Subject: Re: [Fuego] RFC: Fuego's version up and other changes
> 
> Hi Kengo-san,
> 
> Thanks a lot for your feedback.
> 
> > -----Original Message-----
> > From: Ibe.Kengo@bx.MitsubishiElectric.co.jp
> [mailto:Ibe.Kengo@bx.MitsubishiElectric.co.jp]
> > Sent: Thursday, January 12, 2017 2:52 PM
> > To: Daniel Sangorrin
> > Cc: fuego@lists.linuxfoundation.org
> > Subject: Re: [Fuego] RFC: Fuego's version up and other changes
> >
> > Hi, Daniel-san.
> >
> > I’m very interested in your work!
> > So I tried to run your build scripts to create Docker image today,
> > then I could run new Fuego.
> > And, I added Docker node and test cases to use following command.
> > ==
> >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego.git
> >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego-core.git
> >
> >   $ cd fuego/
> >   $ ./install.sh
> >   $ ./fuego-host-scripts/docker-create-container.sh
> >   $ ./fuego-host-scripts/docker-start-container.sh
> >   docker# fuego-create-node --board docker
> >   docker# fuego-create-jobs --board docker --testplan testplan_docker
> > --distrib nosyslogd.dist
> > ==
> >
> > I can see  new node and new test jobs.
> > But, I can’t run tests like Dhrystone on Docker.
> >
> > So far, I had one small problem that is “Failed to mkdirs: /fuego-rw/
> > buildzone”
> > This reason is the permission or owner of “fuego-rw”,
> > so I added the write permission to “fuego-rw” diredtory.
> 
> Humm You should not need to change any permissions.
> The script is supposed to use your user's uid/gid for the jenkins daemon.
> However, I have only tested it on my pc which has a single user.
> Do you have more than one user? Could you check that the jenkins uid/gid
> is the same as yours? (you can check on /etc/passwd inside and outside
> docker)
> Please also check that the fuego-rw's owner is your user (e.g.: don't git clone
> with
> a different user or root)
> 
> > I have solved this problem.
> >
> > After that, I got the error message like following in console log.
> > ====
> > Started by user anonymous
> > Building remotely on docker in workspace /fuego-rw/buildzone
> > [buildzone] $ /bin/sh -xe /tmp/hudson1255554151521556820.sh
> > + export Reboot=false
> > + Reboot=false
> > + export Rebuild=true
> > + Rebuild=true
> > + export Target_Cleanup=true
> > + Target_Cleanup=true
> > + export TESTDIR=Benchmark.Dhrystone
> > + TESTDIR=Benchmark.Dhrystone
> > + export TESTNAME=Dhrystone
> > + TESTNAME=Dhrystone
> > + export DISTRIB=nosyslogd.dist
> > + DISTRIB=nosyslogd.dist
> > + export TESTPLAN=testplan_docker
> > + TESTPLAN=testplan_docker
> > + timeout --signal=9 3m /bin/bash
> > /fuego-core/engine/tests/Benchmark.Dhrystone/Dhrystone.sh
> > board overlay: boards/docker.board
> > using boards/docker.board board overlay
> > using nosyslogd.dist overlay
> > BATCH_TESTPLAN is not set, using testplan_docker.json testplan
> > Traceback (most recent call last):
> >   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 615, in <module
> > >
> >     run()
> >   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 607, in run
> >     generateProlog(output, ofcls, classes, testPlans, testSpecs)
> >   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 375, in
> > generateProlog
> >     outfile = open(outFilePath, "w")
> > IOError: [Errno 2] No such file or directory: '/fuego-rw/work/docker_
> > prolog.sh'
> >
> > *** ABORTED ***
> >
> > Fuego error reason: Error while prolog.sh file generation
> >
> > Build step 'Execute shell' marked build as failure
> > [description-setter] Description set:
> > Finished: FAILURE
> > ====
> >
> > Did I miss to run test cases on Docker node?
> > If you know this solution, could you give me some advice?
> 
> Could you check that fuego-rw is correctly mounted inside docker?
> outside-docker$ touch fuego-rw/hoge
> inside-docker$ ls /fuego-rw
> hoge <-- does it appear?
> 
> On my PC (ubuntu 16.04, docker version 1.12.3 ) it's always succesful so I will
> try on a different one. Which OS and docker version are you using?
> 
> Thanks,
> Daniel
> 
> > Thank you.
> >
> > Best Regard.
> > Kengo Ibe
> >
> > ----- Original Message -----
> > 件名: [Fuego] RFC: Fuego's version up and other changes
> > 送信者: "Daniel Sangorrin" <daniel.sangorrin@toshiba.co.jp>
> > 送信日時: 2017年01月11日(水) 09:28:07
> > 宛先: "Bird, Timothy" <Tim.Bird@am.sony.com>, fuego@lists.
> > linuxfoundation.org
> >
> > > Hi everyone,
> > >
> > > I've been working on Fuego's upgrade for a few weeks, and in the
> > process I
> > > have made quite a few changes.
> > > You can check it by following these instructions. I'd really
> > appreciate
> > > your feedback:
> > >
> > > Quickstart:
> > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego.git
> > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego-core.git
> >
> > >   $ cd fuego/
> > >   $ ./install.sh
> > >   $ ./fuego-host-scripts/docker-create-container.sh
> > >   $ ./fuego-host-scripts/docker-start-container.sh
> > >   docker# fuego-create-node --board docker
> > >   docker# fuego-create-jobs --board docker --testplan testplan_docker
> > > --distrib nosyslogd.dist
> > >   Then, start the "docker.testplan_docker.batch" job which will
> > trigger
> > > the rest of jobs.
> > >
> > > Here is a list of the most fundamental modifications I've added:
> > > - I automatized the version upgrade process. So far I have updated
> > Jenkins
> > > 3 times quite smoothly.
> > >    The current version is the latest LTS version (2.31.1 LTS)
> > > - I reduced plugin dependencies to the minimum (only 2, and they can
> > be
> > > made optional).
> > > - I removed dependencies on all groovy scripts.
> > > - Userdata is gone. Instead we now have fuego-ro and fuego-rw, which
> > > together with fuego-core
> > >    are mounted as external docker volumes. Only fuego-rw, which
> > contains
> > > logs and build folders,
> > >    is mounted in RW mode. I also fixed the uid/gid of jenkins so that
> > it
> > > matches the one in the host.
> > >    This means you can develop comfortably on your host, with GUI tools,
> > > and don't fear a buggy
> > >    script deleting your folders.
> > > - I added a  tool for generating jobs from testplans.  Creating jobs
> > from
> > > testplans is powerful.
> > >    It allows you, for example, to easily specify the timeout for each
> > test
> > > on your board.
> > >    The tool also generates the corresponding batch job. For example,
> > you
> > > can easily trigger all
> > >    tests for your board and a specific testplan periodically.
> > >    [Note] This will also enable the creation of scripts for comparing
> > test
> > > results across
> > >    different boards and test parameters (on my TODO list).
> > >
> > > Other less fundamental changes include:
> > > - I added support for using the docker container itself as a target
> > board
> > > (e.g.: for a quickstart or unit tests)
> > > - I fixed the flot plugin (java and javascript fixes)
> > > - Now you can click on a node and see which jobs (and testplans) are
> > > assigned to your board. This was
> > >   one of my old feature requests and makes Fuego's GUI easier to use.
> > > - I added quite a few fixes and improvements to the core engine
> > scripts
> > > and tests (too detailed to describe here).
> > > - I added Excel output support for IOzone
> > > - Faster docker build time (ARMhf compiler installation is now
> > optional,
> > > no latex..)
> > > - I put a fixed name to the fuego container, instead of the
> > > "last_container_id" file.
> > > - I removed the inotify script
> > > - I added scripts for easily removing the docker container/image.
> > >
> > > I have to split the current "huge" patch into smaller chunks and test
> > it
> > > further but it would
> > > be nice to get your feedback about the approach in general.
> > >
> > > Tim: I haven't updated ftc nor your "Christmas commits" yet.
> > >
> > > Best regards
> > > Daniel
> > >
> > >
> > > _______________________________________________
> > > Fuego mailing list
> > > Fuego@lists.linuxfoundation.org
> > > https://lists.linuxfoundation.org/mailman/listinfo/fuego
> > >
> >
> 
> 
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] RFC: Fuego's version up and other changes

[Daniel Sangorrin]

> From: Ibe.Kengo@bx.MitsubishiElectric.co.jp [mailto:Ibe.Kengo@bx.MitsubishiElectric.co.jp]
> Hi, Daniel-san,
> 
> Thank you for prompt help.
> I was able to run tests in my Docker!! :)
> 
> I'm going to try to use some function of your Fuego.
> If I have some question, let me ask you some question.

Ok, that's great!
I'll be glad to answer your questions or ideas for improving it.

Thanks,
Daniel

> ----- Original Message -----
> 件名: RE: [Fuego] RFC: Fuego's version up and other changes
> 送信者: "Daniel Sangorrin" <daniel.sangorrin@toshiba.co.jp>
> 送信日時: 2017年01月12日(木) 17:45:20
> 宛先: Ibe.Kengo@bx.MitsubishiElectric.co.jp
> Cc: fuego@lists.linuxfoundation.org
> 
> > Hi Kengo-san,
> >
> > I'm sorry, it seems that fuego-rw didn't get pushed. The reason is
> because it's included in .gitignore!!
> > I have pushed it now. Please remove your current fuego-rw and do a git
> pull. It should work properly this time.
> >
> > Thanks
> > Daniel
> >
> >
> > > -----Original Message-----
> > > From: Ibe.Kengo@bx.MitsubishiElectric.co.jp [mailto:Ibe.Kengo@bx.
> MitsubishiElectric.co.jp]
> > > Sent: Thursday, January 12, 2017 5:16 PM
> > > To: Daniel Sangorrin
> > > Cc: fuego@lists.linuxfoundation.org
> > > Subject: Re: [Fuego] RFC: Fuego's version up and other changes
> > >
> > > Hi, Daniel-san
> > >
> > > Thank you for your prompt reply.
> > > I have more a few question.
> > >
> > > > Humm You should not need to change any permissions.
> > > OK.
> > > I removed the Fuego and the Fuego-core,
> > > after that, I redownloaded the Fuego and the Fuego-core and
> rebuilded
> > > and start Docker.
> > > I can see following permissions of directories. Is it correct?
> > > ==
> > > melco@debian8:~/$ ./fuego/fuego-host-scripts/docker-start-container.
> sh
> > > Starting Fuego container (fuego-container)
> > > [ ok ] Starting Jenkins Continuous Integration Server: jenkins.
> > >
> > > root@debian8:/# ls -al
> > > total 84
> > > drwxr-xr-x  65 root    root    4096 Jan 12 07:26 .
> > > drwxr-xr-x  65 root    root    4096 Jan 12 07:26 ..
> > > -rwxr-xr-x   1 root    root       0 Jan 12 07:26 .dockerenv
> > > drwxr-xr-x   2 root    root    4096 Jan 12 00:53 bin
> > > drwxr-xr-x   2 root    root    4096 Sep 12 04:09 boot
> > > drwxr-xr-x   5 root    root     380 Jan 12 07:26 dev
> > > drwxr-xr-x  96 root    root    4096 Jan 12 07:26 etc
> > > drwxr-xr-x   4 jenkins jenkins 4096 Jan 12 07:07 fuego-core
> > > drwxr-xr-x   5 jenkins jenkins 4096 Jan 12 07:03 fuego-ro
> > > drwxr-xr-x   2 root    root    4096 Jan 12 07:26 fuego-rw
> > > drwxr-xr-x   2 root    root    4096 Sep 12 04:09 home
> > > ~
> > >
> > > melco@debian8:~/$ ls -al fuego
> > > 合計 68
> > > drwxr-xr-x 8 melco melco 4096  1月 12 16:26 .
> > > drwxr-xr-x 4 melco melco 4096  1月 12 16:02 ..
> > > -rw-r--r-- 1 melco melco   42  1月 12 16:03 .dockerignore
> > > drwxr-xr-x 8 melco melco 4096  1月 12 16:03 .git
> > > -rw-r--r-- 1 melco melco    9  1月 12 16:03 .gitignore
> > > -rw-r--r-- 1 melco melco 1482  1月 12 16:03 CHANGELOG
> > > -rw-r--r-- 1 melco melco 4810  1月 12 16:03 Dockerfile
> > > -rw-r--r-- 1 melco melco 1492  1月 12 16:03 LICENSE
> > > -rw-r--r-- 1 melco melco 3414  1月 12 16:03 README
> > > -rw-r--r-- 1 melco melco   85  1月 12 16:03 VERSION
> > > drwxr-xr-x 3 melco melco 4096  1月 12 16:03 docs
> > > drwxr-xr-x 3 melco melco 4096  1月 12 16:03 frontend-install
> > > drwxr-xr-x 2 melco melco 4096  1月 12 16:03 fuego-host-scripts
> > > drwxr-xr-x 5 melco melco 4096  1月 12 16:03 fuego-ro
> > > drwxr-xr-x 2 root  root  4096  1月 12 16:26 fuego-rw
> > > -rwxr-xr-x 1 melco melco   48  1月 12 16:03 install.sh
> > >
> > > ==
> > > “melco” user is only user on host PC which is Debian8.
> > > I’m concerned about fuego-rw permissions and owner.
> > >
> > > > The script is supposed to use your user's uid/gid for the jenkins
> > > daemon.
> > > Do you mean that I need to make “Jenkins” user on host PC?
> > > Or
> > > Can I use “melco” user inside Docker?
> > >
> > > > However, I have only tested it on my pc which has a single user.
> > > > Do you have more than one user?
> > > I have one user. I'v not made "jenkins" user on host PC.
> > >
> > > I’ll check other point following your advice.
> > > Thank you so much.
> > > Kengo
> > >
> > >
> > > ----- Original Message -----
> > > 件名: RE: [Fuego] RFC: Fuego's version up and other changes
> > > 送信者: "Daniel Sangorrin" <daniel.sangorrin@toshiba.co.jp>
> > > 送信日時: 2017年01月12日(木) 15:44:34
> > > 宛先: Ibe.Kengo@bx.MitsubishiElectric.co.jp
> > > Cc: fuego@lists.linuxfoundation.org
> > >
> > > > Hi Kengo-san,
> > > >
> > > > Thanks a lot for your feedback.
> > > >
> > > > > -----Original Message-----
> > > > > From: Ibe.Kengo@bx.MitsubishiElectric.co.jp [mailto:Ibe.Kengo@bx.
> > > MitsubishiElectric.co.jp]
> > > > > Sent: Thursday, January 12, 2017 2:52 PM
> > > > > To: Daniel Sangorrin
> > > > > Cc: fuego@lists.linuxfoundation.org
> > > > > Subject: Re: [Fuego] RFC: Fuego's version up and other changes
> > > > >
> > > > > Hi, Daniel-san.
> > > > >
> > > > > I’m very interested in your work!
> > > > > So I tried to run your build scripts to create Docker image
> today,
> > > > > then I could run new Fuego.
> > > > > And, I added Docker node and test cases to use following command.
> > > > > ==
> > > > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego.git
> 
> > >
> > > > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego-core.git
> 
> > >
> > > > >
> > > > >   $ cd fuego/
> > > > >   $ ./install.sh
> > > > >   $ ./fuego-host-scripts/docker-create-container.sh
> > > > >   $ ./fuego-host-scripts/docker-start-container.sh
> > > > >   docker# fuego-create-node --board docker
> > > > >   docker# fuego-create-jobs --board docker --testplan testplan_
> > > docker
> > > > > --distrib nosyslogd.dist
> > > > > ==
> > > > >
> > > > > I can see  new node and new test jobs.
> > > > > But, I can’t run tests like Dhrystone on Docker.
> > > > >
> > > > > So far, I had one small problem that is “Failed to mkdirs: /
> fuego-rw
> > > /
> > > > > buildzone”
> > > > > This reason is the permission or owner of “fuego-rw”,
> > > > > so I added the write permission to “fuego-rw” diredtory.
> > > >
> > > > Humm You should not need to change any permissions.
> > > > The script is supposed to use your user's uid/gid for the jenkins
> > > daemon.
> > > > However, I have only tested it on my pc which has a single user.
> > > > Do you have more than one user? Could you check that the jenkins
> uid/
> > > gid
> > > > is the same as yours? (you can check on /etc/passwd inside and
> outside
> > > docker)
> > > > Please also check that the fuego-rw's owner is your user (e.g.:
> don't
> > > git clone with
> > > > a different user or root)
> > > >
> > > > > I have solved this problem.
> > > > >
> > > > > After that, I got the error message like following in console
> log.
> > > > > ====
> > > > > Started by user anonymous
> > > > > Building remotely on docker in workspace /fuego-rw/buildzone
> > > > > [buildzone] $ /bin/sh -xe /tmp/hudson1255554151521556820.sh
> > > > > + export Reboot=false
> > > > > + Reboot=false
> > > > > + export Rebuild=true
> > > > > + Rebuild=true
> > > > > + export Target_Cleanup=true
> > > > > + Target_Cleanup=true
> > > > > + export TESTDIR=Benchmark.Dhrystone
> > > > > + TESTDIR=Benchmark.Dhrystone
> > > > > + export TESTNAME=Dhrystone
> > > > > + TESTNAME=Dhrystone
> > > > > + export DISTRIB=nosyslogd.dist
> > > > > + DISTRIB=nosyslogd.dist
> > > > > + export TESTPLAN=testplan_docker
> > > > > + TESTPLAN=testplan_docker
> > > > > + timeout --signal=9 3m /bin/bash
> > > > > /fuego-core/engine/tests/Benchmark.Dhrystone/Dhrystone.sh
> > > > > board overlay: boards/docker.board
> > > > > using boards/docker.board board overlay
> > > > > using nosyslogd.dist overlay
> > > > > BATCH_TESTPLAN is not set, using testplan_docker.json testplan
> > > > > Traceback (most recent call last):
> > > > >   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 615, in
> <
> > > module
> > > > > >
> > > > >     run()
> > > > >   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 607, in
> run
> > > > >     generateProlog(output, ofcls, classes, testPlans, testSpecs)
> > > > >   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 375, in
> > > > > generateProlog
> > > > >     outfile = open(outFilePath, "w")
> > > > > IOError: [Errno 2] No such file or directory: '/fuego-rw/work/
> docker
> > > _
> > > > > prolog.sh'
> > > > >
> > > > > *** ABORTED ***
> > > > >
> > > > > Fuego error reason: Error while prolog.sh file generation
> > > > >
> > > > > Build step 'Execute shell' marked build as failure
> > > > > [description-setter] Description set:
> > > > > Finished: FAILURE
> > > > > ====
> > > > >
> > > > > Did I miss to run test cases on Docker node?
> > > > > If you know this solution, could you give me some advice?
> > > >
> > > > Could you check that fuego-rw is correctly mounted inside docker?
> > > > outside-docker$ touch fuego-rw/hoge
> > > > inside-docker$ ls /fuego-rw
> > > > hoge <-- does it appear?
> > > >
> > > > On my PC (ubuntu 16.04, docker version 1.12.3 ) it's always
> succesful
> > > so I will
> > > > try on a different one. Which OS and docker version are you using?
> > > >
> > > > Thanks,
> > > > Daniel
> > > >
> > > > > Thank you.
> > > > >
> > > > > Best Regard.
> > > > > Kengo Ibe
> > > > >
> > > > > ----- Original Message -----
> > > > > 件名: [Fuego] RFC: Fuego's version up and other changes
> > > > > 送信者: "Daniel Sangorrin" <daniel.sangorrin@toshiba.co.jp>
> > > > > 送信日時: 2017年01月11日(水) 09:28:07
> > > > > 宛先: "Bird, Timothy" <Tim.Bird@am.sony.com>, fuego@lists.
> > > > > linuxfoundation.org
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > I've been working on Fuego's upgrade for a few weeks, and in
> the
> > > > > process I
> > > > > > have made quite a few changes.
> > > > > > You can check it by following these instructions. I'd really
> > > > > appreciate
> > > > > > your feedback:
> > > > > >
> > > > > > Quickstart:
> > > > > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego.git
> 
> > >
> > > > > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego-core.git
> 
> > >
> > > > >
> > > > > >   $ cd fuego/
> > > > > >   $ ./install.sh
> > > > > >   $ ./fuego-host-scripts/docker-create-container.sh
> > > > > >   $ ./fuego-host-scripts/docker-start-container.sh
> > > > > >   docker# fuego-create-node --board docker
> > > > > >   docker# fuego-create-jobs --board docker --testplan testplan
> _
> > > docker
> > > > > > --distrib nosyslogd.dist
> > > > > >   Then, start the "docker.testplan_docker.batch" job which
> will
> > > > > trigger
> > > > > > the rest of jobs.
> > > > > >
> > > > > > Here is a list of the most fundamental modifications I've
> added:
> > > > > > - I automatized the version upgrade process. So far I have
> updated
> > > > > Jenkins
> > > > > > 3 times quite smoothly.
> > > > > >    The current version is the latest LTS version (2.31.1 LTS)
> > > > > > - I reduced plugin dependencies to the minimum (only 2, and
> they
> > > can
> > > > > be
> > > > > > made optional).
> > > > > > - I removed dependencies on all groovy scripts.
> > > > > > - Userdata is gone. Instead we now have fuego-ro and fuego-rw,
> > > which
> > > > > > together with fuego-core
> > > > > >    are mounted as external docker volumes. Only fuego-rw,
> which
> > > > > contains
> > > > > > logs and build folders,
> > > > > >    is mounted in RW mode. I also fixed the uid/gid of jenkins
> so
> > > that
> > > > > it
> > > > > > matches the one in the host.
> > > > > >    This means you can develop comfortably on your host, with
> GUI
> > > tools,
> > > > > > and don't fear a buggy
> > > > > >    script deleting your folders.
> > > > > > - I added a  tool for generating jobs from testplans.
> Creating
> > > jobs
> > > > > from
> > > > > > testplans is powerful.
> > > > > >    It allows you, for example, to easily specify the timeout
> for
> > > each
> > > > > test
> > > > > > on your board.
> > > > > >    The tool also generates the corresponding batch job. For
> > > example,
> > > > > you
> > > > > > can easily trigger all
> > > > > >    tests for your board and a specific testplan periodically.
> > > > > >    [Note] This will also enable the creation of scripts for
> > > comparing
> > > > > test
> > > > > > results across
> > > > > >    different boards and test parameters (on my TODO list).
> > > > > >
> > > > > > Other less fundamental changes include:
> > > > > > - I added support for using the docker container itself as a
> > > target
> > > > > board
> > > > > > (e.g.: for a quickstart or unit tests)
> > > > > > - I fixed the flot plugin (java and javascript fixes)
> > > > > > - Now you can click on a node and see which jobs (and
> testplans)
> > > are
> > > > > > assigned to your board. This was
> > > > > >   one of my old feature requests and makes Fuego's GUI easier
> to
> > > use.
> > > > > > - I added quite a few fixes and improvements to the core
> engine
> > > > > scripts
> > > > > > and tests (too detailed to describe here).
> > > > > > - I added Excel output support for IOzone
> > > > > > - Faster docker build time (ARMhf compiler installation is now
> > > > > optional,
> > > > > > no latex..)
> > > > > > - I put a fixed name to the fuego container, instead of the
> > > > > > "last_container_id" file.
> > > > > > - I removed the inotify script
> > > > > > - I added scripts for easily removing the docker container/
> image.
> > > > > >
> > > > > > I have to split the current "huge" patch into smaller chunks
> and
> > > test
> > > > > it
> > > > > > further but it would
> > > > > > be nice to get your feedback about the approach in general.
> > > > > >
> > > > > > Tim: I haven't updated ftc nor your "Christmas commits" yet.
> > > > > >
> > > > > > Best regards
> > > > > > Daniel
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > Fuego mailing list
> > > > > > Fuego@lists.linuxfoundation.org
> > > > > > https://lists.linuxfoundation.org/mailman/listinfo/fuego
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
> >
> >
> >
> 

Re: [Fuego] RFC: Fuego's version up and other changes

[Ibe.Kengo]

Hi, Daniel-san,

Thank you for prompt help.
I was able to run tests in my Docker!! :)

I'm going to try to use some function of your Fuego.
If I have some question, let me ask you some question.

Thanks,
Kengo

----- Original Message -----
件名: RE: [Fuego] RFC: Fuego's version up and other changes
送信者: "Daniel Sangorrin" <daniel.sangorrin@toshiba.co.jp>
送信日時: 2017年01月12日(木) 17:45:20
宛先: Ibe.Kengo@bx.MitsubishiElectric.co.jp
Cc: fuego@lists.linuxfoundation.org

> Hi Kengo-san,
>
> I'm sorry, it seems that fuego-rw didn't get pushed. The reason is
because it's included in .gitignore!!
> I have pushed it now. Please remove your current fuego-rw and do a git
pull. It should work properly this time.
>
> Thanks
> Daniel
>
>
> > -----Original Message-----
> > From: Ibe.Kengo@bx.MitsubishiElectric.co.jp [mailto:Ibe.Kengo@bx.
MitsubishiElectric.co.jp]
> > Sent: Thursday, January 12, 2017 5:16 PM
> > To: Daniel Sangorrin
> > Cc: fuego@lists.linuxfoundation.org
> > Subject: Re: [Fuego] RFC: Fuego's version up and other changes
> >
> > Hi, Daniel-san
> >
> > Thank you for your prompt reply.
> > I have more a few question.
> >
> > > Humm You should not need to change any permissions.
> > OK.
> > I removed the Fuego and the Fuego-core,
> > after that, I redownloaded the Fuego and the Fuego-core and
rebuilded
> > and start Docker.
> > I can see following permissions of directories. Is it correct?
> > ==
> > melco@debian8:~/$ ./fuego/fuego-host-scripts/docker-start-container.
sh
> > Starting Fuego container (fuego-container)
> > [ ok ] Starting Jenkins Continuous Integration Server: jenkins.
> >
> > root@debian8:/# ls -al
> > total 84
> > drwxr-xr-x  65 root    root    4096 Jan 12 07:26 .
> > drwxr-xr-x  65 root    root    4096 Jan 12 07:26 ..
> > -rwxr-xr-x   1 root    root       0 Jan 12 07:26 .dockerenv
> > drwxr-xr-x   2 root    root    4096 Jan 12 00:53 bin
> > drwxr-xr-x   2 root    root    4096 Sep 12 04:09 boot
> > drwxr-xr-x   5 root    root     380 Jan 12 07:26 dev
> > drwxr-xr-x  96 root    root    4096 Jan 12 07:26 etc
> > drwxr-xr-x   4 jenkins jenkins 4096 Jan 12 07:07 fuego-core
> > drwxr-xr-x   5 jenkins jenkins 4096 Jan 12 07:03 fuego-ro
> > drwxr-xr-x   2 root    root    4096 Jan 12 07:26 fuego-rw
> > drwxr-xr-x   2 root    root    4096 Sep 12 04:09 home
> > ~
> >
> > melco@debian8:~/$ ls -al fuego
> > 合計 68
> > drwxr-xr-x 8 melco melco 4096  1月 12 16:26 .
> > drwxr-xr-x 4 melco melco 4096  1月 12 16:02 ..
> > -rw-r--r-- 1 melco melco   42  1月 12 16:03 .dockerignore
> > drwxr-xr-x 8 melco melco 4096  1月 12 16:03 .git
> > -rw-r--r-- 1 melco melco    9  1月 12 16:03 .gitignore
> > -rw-r--r-- 1 melco melco 1482  1月 12 16:03 CHANGELOG
> > -rw-r--r-- 1 melco melco 4810  1月 12 16:03 Dockerfile
> > -rw-r--r-- 1 melco melco 1492  1月 12 16:03 LICENSE
> > -rw-r--r-- 1 melco melco 3414  1月 12 16:03 README
> > -rw-r--r-- 1 melco melco   85  1月 12 16:03 VERSION
> > drwxr-xr-x 3 melco melco 4096  1月 12 16:03 docs
> > drwxr-xr-x 3 melco melco 4096  1月 12 16:03 frontend-install
> > drwxr-xr-x 2 melco melco 4096  1月 12 16:03 fuego-host-scripts
> > drwxr-xr-x 5 melco melco 4096  1月 12 16:03 fuego-ro
> > drwxr-xr-x 2 root  root  4096  1月 12 16:26 fuego-rw
> > -rwxr-xr-x 1 melco melco   48  1月 12 16:03 install.sh
> >
> > ==
> > “melco” user is only user on host PC which is Debian8.
> > I’m concerned about fuego-rw permissions and owner.
> >
> > > The script is supposed to use your user's uid/gid for the jenkins
> > daemon.
> > Do you mean that I need to make “Jenkins” user on host PC?
> > Or
> > Can I use “melco” user inside Docker?
> >
> > > However, I have only tested it on my pc which has a single user.
> > > Do you have more than one user?
> > I have one user. I'v not made "jenkins" user on host PC.
> >
> > I’ll check other point following your advice.
> > Thank you so much.
> > Kengo
> >
> >
> > ----- Original Message -----
> > 件名: RE: [Fuego] RFC: Fuego's version up and other changes
> > 送信者: "Daniel Sangorrin" <daniel.sangorrin@toshiba.co.jp>
> > 送信日時: 2017年01月12日(木) 15:44:34
> > 宛先: Ibe.Kengo@bx.MitsubishiElectric.co.jp
> > Cc: fuego@lists.linuxfoundation.org
> >
> > > Hi Kengo-san,
> > >
> > > Thanks a lot for your feedback.
> > >
> > > > -----Original Message-----
> > > > From: Ibe.Kengo@bx.MitsubishiElectric.co.jp [mailto:Ibe.Kengo@bx.
> > MitsubishiElectric.co.jp]
> > > > Sent: Thursday, January 12, 2017 2:52 PM
> > > > To: Daniel Sangorrin
> > > > Cc: fuego@lists.linuxfoundation.org
> > > > Subject: Re: [Fuego] RFC: Fuego's version up and other changes
> > > >
> > > > Hi, Daniel-san.
> > > >
> > > > I’m very interested in your work!
> > > > So I tried to run your build scripts to create Docker image
today,
> > > > then I could run new Fuego.
> > > > And, I added Docker node and test cases to use following command.
> > > > ==
> > > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego.git

> >
> > > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego-core.git

> >
> > > >
> > > >   $ cd fuego/
> > > >   $ ./install.sh
> > > >   $ ./fuego-host-scripts/docker-create-container.sh
> > > >   $ ./fuego-host-scripts/docker-start-container.sh
> > > >   docker# fuego-create-node --board docker
> > > >   docker# fuego-create-jobs --board docker --testplan testplan_
> > docker
> > > > --distrib nosyslogd.dist
> > > > ==
> > > >
> > > > I can see  new node and new test jobs.
> > > > But, I can’t run tests like Dhrystone on Docker.
> > > >
> > > > So far, I had one small problem that is “Failed to mkdirs: /
fuego-rw
> > /
> > > > buildzone”
> > > > This reason is the permission or owner of “fuego-rw”,
> > > > so I added the write permission to “fuego-rw” diredtory.
> > >
> > > Humm You should not need to change any permissions.
> > > The script is supposed to use your user's uid/gid for the jenkins
> > daemon.
> > > However, I have only tested it on my pc which has a single user.
> > > Do you have more than one user? Could you check that the jenkins
uid/
> > gid
> > > is the same as yours? (you can check on /etc/passwd inside and
outside
> > docker)
> > > Please also check that the fuego-rw's owner is your user (e.g.:
don't
> > git clone with
> > > a different user or root)
> > >
> > > > I have solved this problem.
> > > >
> > > > After that, I got the error message like following in console
log.
> > > > ====
> > > > Started by user anonymous
> > > > Building remotely on docker in workspace /fuego-rw/buildzone
> > > > [buildzone] $ /bin/sh -xe /tmp/hudson1255554151521556820.sh
> > > > + export Reboot=false
> > > > + Reboot=false
> > > > + export Rebuild=true
> > > > + Rebuild=true
> > > > + export Target_Cleanup=true
> > > > + Target_Cleanup=true
> > > > + export TESTDIR=Benchmark.Dhrystone
> > > > + TESTDIR=Benchmark.Dhrystone
> > > > + export TESTNAME=Dhrystone
> > > > + TESTNAME=Dhrystone
> > > > + export DISTRIB=nosyslogd.dist
> > > > + DISTRIB=nosyslogd.dist
> > > > + export TESTPLAN=testplan_docker
> > > > + TESTPLAN=testplan_docker
> > > > + timeout --signal=9 3m /bin/bash
> > > > /fuego-core/engine/tests/Benchmark.Dhrystone/Dhrystone.sh
> > > > board overlay: boards/docker.board
> > > > using boards/docker.board board overlay
> > > > using nosyslogd.dist overlay
> > > > BATCH_TESTPLAN is not set, using testplan_docker.json testplan
> > > > Traceback (most recent call last):
> > > >   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 615, in
<
> > module
> > > > >
> > > >     run()
> > > >   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 607, in
run
> > > >     generateProlog(output, ofcls, classes, testPlans, testSpecs)
> > > >   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 375, in
> > > > generateProlog
> > > >     outfile = open(outFilePath, "w")
> > > > IOError: [Errno 2] No such file or directory: '/fuego-rw/work/
docker
> > _
> > > > prolog.sh'
> > > >
> > > > *** ABORTED ***
> > > >
> > > > Fuego error reason: Error while prolog.sh file generation
> > > >
> > > > Build step 'Execute shell' marked build as failure
> > > > [description-setter] Description set:
> > > > Finished: FAILURE
> > > > ====
> > > >
> > > > Did I miss to run test cases on Docker node?
> > > > If you know this solution, could you give me some advice?
> > >
> > > Could you check that fuego-rw is correctly mounted inside docker?
> > > outside-docker$ touch fuego-rw/hoge
> > > inside-docker$ ls /fuego-rw
> > > hoge <-- does it appear?
> > >
> > > On my PC (ubuntu 16.04, docker version 1.12.3 ) it's always
succesful
> > so I will
> > > try on a different one. Which OS and docker version are you using?
> > >
> > > Thanks,
> > > Daniel
> > >
> > > > Thank you.
> > > >
> > > > Best Regard.
> > > > Kengo Ibe
> > > >
> > > > ----- Original Message -----
> > > > 件名: [Fuego] RFC: Fuego's version up and other changes
> > > > 送信者: "Daniel Sangorrin" <daniel.sangorrin@toshiba.co.jp>
> > > > 送信日時: 2017年01月11日(水) 09:28:07
> > > > 宛先: "Bird, Timothy" <Tim.Bird@am.sony.com>, fuego@lists.
> > > > linuxfoundation.org
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > I've been working on Fuego's upgrade for a few weeks, and in
the
> > > > process I
> > > > > have made quite a few changes.
> > > > > You can check it by following these instructions. I'd really
> > > > appreciate
> > > > > your feedback:
> > > > >
> > > > > Quickstart:
> > > > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego.git

> >
> > > > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego-core.git

> >
> > > >
> > > > >   $ cd fuego/
> > > > >   $ ./install.sh
> > > > >   $ ./fuego-host-scripts/docker-create-container.sh
> > > > >   $ ./fuego-host-scripts/docker-start-container.sh
> > > > >   docker# fuego-create-node --board docker
> > > > >   docker# fuego-create-jobs --board docker --testplan testplan
_
> > docker
> > > > > --distrib nosyslogd.dist
> > > > >   Then, start the "docker.testplan_docker.batch" job which
will
> > > > trigger
> > > > > the rest of jobs.
> > > > >
> > > > > Here is a list of the most fundamental modifications I've
added:
> > > > > - I automatized the version upgrade process. So far I have
updated
> > > > Jenkins
> > > > > 3 times quite smoothly.
> > > > >    The current version is the latest LTS version (2.31.1 LTS)
> > > > > - I reduced plugin dependencies to the minimum (only 2, and
they
> > can
> > > > be
> > > > > made optional).
> > > > > - I removed dependencies on all groovy scripts.
> > > > > - Userdata is gone. Instead we now have fuego-ro and fuego-rw,
> > which
> > > > > together with fuego-core
> > > > >    are mounted as external docker volumes. Only fuego-rw,
which
> > > > contains
> > > > > logs and build folders,
> > > > >    is mounted in RW mode. I also fixed the uid/gid of jenkins
so
> > that
> > > > it
> > > > > matches the one in the host.
> > > > >    This means you can develop comfortably on your host, with
GUI
> > tools,
> > > > > and don't fear a buggy
> > > > >    script deleting your folders.
> > > > > - I added a  tool for generating jobs from testplans.
Creating
> > jobs
> > > > from
> > > > > testplans is powerful.
> > > > >    It allows you, for example, to easily specify the timeout
for
> > each
> > > > test
> > > > > on your board.
> > > > >    The tool also generates the corresponding batch job. For
> > example,
> > > > you
> > > > > can easily trigger all
> > > > >    tests for your board and a specific testplan periodically.
> > > > >    [Note] This will also enable the creation of scripts for
> > comparing
> > > > test
> > > > > results across
> > > > >    different boards and test parameters (on my TODO list).
> > > > >
> > > > > Other less fundamental changes include:
> > > > > - I added support for using the docker container itself as a
> > target
> > > > board
> > > > > (e.g.: for a quickstart or unit tests)
> > > > > - I fixed the flot plugin (java and javascript fixes)
> > > > > - Now you can click on a node and see which jobs (and
testplans)
> > are
> > > > > assigned to your board. This was
> > > > >   one of my old feature requests and makes Fuego's GUI easier
to
> > use.
> > > > > - I added quite a few fixes and improvements to the core
engine
> > > > scripts
> > > > > and tests (too detailed to describe here).
> > > > > - I added Excel output support for IOzone
> > > > > - Faster docker build time (ARMhf compiler installation is now
> > > > optional,
> > > > > no latex..)
> > > > > - I put a fixed name to the fuego container, instead of the
> > > > > "last_container_id" file.
> > > > > - I removed the inotify script
> > > > > - I added scripts for easily removing the docker container/
image.
> > > > >
> > > > > I have to split the current "huge" patch into smaller chunks
and
> > test
> > > > it
> > > > > further but it would
> > > > > be nice to get your feedback about the approach in general.
> > > > >
> > > > > Tim: I haven't updated ftc nor your "Christmas commits" yet.
> > > > >
> > > > > Best regards
> > > > > Daniel
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > Fuego mailing list
> > > > > Fuego@lists.linuxfoundation.org
> > > > > https://lists.linuxfoundation.org/mailman/listinfo/fuego
> > > > >
> > > >
> > >
> > >
> > >
> > >
> >
>
>
>
>

Re: [Fuego] RFC: Fuego's version up and other changes

[Daniel Sangorrin]

Hi Kengo-san,

I'm sorry, it seems that fuego-rw didn't get pushed. The reason is because it's included in .gitignore!!
I have pushed it now. Please remove your current fuego-rw and do a git pull. It should work properly this time.

Thanks
Daniel


> -----Original Message-----
> From: Ibe.Kengo@bx.MitsubishiElectric.co.jp [mailto:Ibe.Kengo@bx.MitsubishiElectric.co.jp]
> Sent: Thursday, January 12, 2017 5:16 PM
> To: Daniel Sangorrin
> Cc: fuego@lists.linuxfoundation.org
> Subject: Re: [Fuego] RFC: Fuego's version up and other changes
> 
> Hi, Daniel-san
> 
> Thank you for your prompt reply.
> I have more a few question.
> 
> > Humm You should not need to change any permissions.
> OK.
> I removed the Fuego and the Fuego-core,
> after that, I redownloaded the Fuego and the Fuego-core and rebuilded
> and start Docker.
> I can see following permissions of directories. Is it correct?
> ==
> melco@debian8:~/$ ./fuego/fuego-host-scripts/docker-start-container.sh
> Starting Fuego container (fuego-container)
> [ ok ] Starting Jenkins Continuous Integration Server: jenkins.
> 
> root@debian8:/# ls -al
> total 84
> drwxr-xr-x  65 root    root    4096 Jan 12 07:26 .
> drwxr-xr-x  65 root    root    4096 Jan 12 07:26 ..
> -rwxr-xr-x   1 root    root       0 Jan 12 07:26 .dockerenv
> drwxr-xr-x   2 root    root    4096 Jan 12 00:53 bin
> drwxr-xr-x   2 root    root    4096 Sep 12 04:09 boot
> drwxr-xr-x   5 root    root     380 Jan 12 07:26 dev
> drwxr-xr-x  96 root    root    4096 Jan 12 07:26 etc
> drwxr-xr-x   4 jenkins jenkins 4096 Jan 12 07:07 fuego-core
> drwxr-xr-x   5 jenkins jenkins 4096 Jan 12 07:03 fuego-ro
> drwxr-xr-x   2 root    root    4096 Jan 12 07:26 fuego-rw
> drwxr-xr-x   2 root    root    4096 Sep 12 04:09 home
> ~
> 
> melco@debian8:~/$ ls -al fuego
> 合計 68
> drwxr-xr-x 8 melco melco 4096  1月 12 16:26 .
> drwxr-xr-x 4 melco melco 4096  1月 12 16:02 ..
> -rw-r--r-- 1 melco melco   42  1月 12 16:03 .dockerignore
> drwxr-xr-x 8 melco melco 4096  1月 12 16:03 .git
> -rw-r--r-- 1 melco melco    9  1月 12 16:03 .gitignore
> -rw-r--r-- 1 melco melco 1482  1月 12 16:03 CHANGELOG
> -rw-r--r-- 1 melco melco 4810  1月 12 16:03 Dockerfile
> -rw-r--r-- 1 melco melco 1492  1月 12 16:03 LICENSE
> -rw-r--r-- 1 melco melco 3414  1月 12 16:03 README
> -rw-r--r-- 1 melco melco   85  1月 12 16:03 VERSION
> drwxr-xr-x 3 melco melco 4096  1月 12 16:03 docs
> drwxr-xr-x 3 melco melco 4096  1月 12 16:03 frontend-install
> drwxr-xr-x 2 melco melco 4096  1月 12 16:03 fuego-host-scripts
> drwxr-xr-x 5 melco melco 4096  1月 12 16:03 fuego-ro
> drwxr-xr-x 2 root  root  4096  1月 12 16:26 fuego-rw
> -rwxr-xr-x 1 melco melco   48  1月 12 16:03 install.sh
> 
> ==
> “melco” user is only user on host PC which is Debian8.
> I’m concerned about fuego-rw permissions and owner.
> 
> > The script is supposed to use your user's uid/gid for the jenkins
> daemon.
> Do you mean that I need to make “Jenkins” user on host PC?
> Or
> Can I use “melco” user inside Docker?
> 
> > However, I have only tested it on my pc which has a single user.
> > Do you have more than one user?
> I have one user. I'v not made "jenkins" user on host PC.
> 
> I’ll check other point following your advice.
> Thank you so much.
> Kengo
> 
> 
> ----- Original Message -----
> 件名: RE: [Fuego] RFC: Fuego's version up and other changes
> 送信者: "Daniel Sangorrin" <daniel.sangorrin@toshiba.co.jp>
> 送信日時: 2017年01月12日(木) 15:44:34
> 宛先: Ibe.Kengo@bx.MitsubishiElectric.co.jp
> Cc: fuego@lists.linuxfoundation.org
> 
> > Hi Kengo-san,
> >
> > Thanks a lot for your feedback.
> >
> > > -----Original Message-----
> > > From: Ibe.Kengo@bx.MitsubishiElectric.co.jp [mailto:Ibe.Kengo@bx.
> MitsubishiElectric.co.jp]
> > > Sent: Thursday, January 12, 2017 2:52 PM
> > > To: Daniel Sangorrin
> > > Cc: fuego@lists.linuxfoundation.org
> > > Subject: Re: [Fuego] RFC: Fuego's version up and other changes
> > >
> > > Hi, Daniel-san.
> > >
> > > I’m very interested in your work!
> > > So I tried to run your build scripts to create Docker image today,
> > > then I could run new Fuego.
> > > And, I added Docker node and test cases to use following command.
> > > ==
> > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego.git
> 
> > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego-core.git
> 
> > >
> > >   $ cd fuego/
> > >   $ ./install.sh
> > >   $ ./fuego-host-scripts/docker-create-container.sh
> > >   $ ./fuego-host-scripts/docker-start-container.sh
> > >   docker# fuego-create-node --board docker
> > >   docker# fuego-create-jobs --board docker --testplan testplan_
> docker
> > > --distrib nosyslogd.dist
> > > ==
> > >
> > > I can see  new node and new test jobs.
> > > But, I can’t run tests like Dhrystone on Docker.
> > >
> > > So far, I had one small problem that is “Failed to mkdirs: /fuego-rw
> /
> > > buildzone”
> > > This reason is the permission or owner of “fuego-rw”,
> > > so I added the write permission to “fuego-rw” diredtory.
> >
> > Humm You should not need to change any permissions.
> > The script is supposed to use your user's uid/gid for the jenkins
> daemon.
> > However, I have only tested it on my pc which has a single user.
> > Do you have more than one user? Could you check that the jenkins uid/
> gid
> > is the same as yours? (you can check on /etc/passwd inside and outside
> docker)
> > Please also check that the fuego-rw's owner is your user (e.g.: don't
> git clone with
> > a different user or root)
> >
> > > I have solved this problem.
> > >
> > > After that, I got the error message like following in console log.
> > > ====
> > > Started by user anonymous
> > > Building remotely on docker in workspace /fuego-rw/buildzone
> > > [buildzone] $ /bin/sh -xe /tmp/hudson1255554151521556820.sh
> > > + export Reboot=false
> > > + Reboot=false
> > > + export Rebuild=true
> > > + Rebuild=true
> > > + export Target_Cleanup=true
> > > + Target_Cleanup=true
> > > + export TESTDIR=Benchmark.Dhrystone
> > > + TESTDIR=Benchmark.Dhrystone
> > > + export TESTNAME=Dhrystone
> > > + TESTNAME=Dhrystone
> > > + export DISTRIB=nosyslogd.dist
> > > + DISTRIB=nosyslogd.dist
> > > + export TESTPLAN=testplan_docker
> > > + TESTPLAN=testplan_docker
> > > + timeout --signal=9 3m /bin/bash
> > > /fuego-core/engine/tests/Benchmark.Dhrystone/Dhrystone.sh
> > > board overlay: boards/docker.board
> > > using boards/docker.board board overlay
> > > using nosyslogd.dist overlay
> > > BATCH_TESTPLAN is not set, using testplan_docker.json testplan
> > > Traceback (most recent call last):
> > >   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 615, in <
> module
> > > >
> > >     run()
> > >   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 607, in run
> > >     generateProlog(output, ofcls, classes, testPlans, testSpecs)
> > >   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 375, in
> > > generateProlog
> > >     outfile = open(outFilePath, "w")
> > > IOError: [Errno 2] No such file or directory: '/fuego-rw/work/docker
> _
> > > prolog.sh'
> > >
> > > *** ABORTED ***
> > >
> > > Fuego error reason: Error while prolog.sh file generation
> > >
> > > Build step 'Execute shell' marked build as failure
> > > [description-setter] Description set:
> > > Finished: FAILURE
> > > ====
> > >
> > > Did I miss to run test cases on Docker node?
> > > If you know this solution, could you give me some advice?
> >
> > Could you check that fuego-rw is correctly mounted inside docker?
> > outside-docker$ touch fuego-rw/hoge
> > inside-docker$ ls /fuego-rw
> > hoge <-- does it appear?
> >
> > On my PC (ubuntu 16.04, docker version 1.12.3 ) it's always succesful
> so I will
> > try on a different one. Which OS and docker version are you using?
> >
> > Thanks,
> > Daniel
> >
> > > Thank you.
> > >
> > > Best Regard.
> > > Kengo Ibe
> > >
> > > ----- Original Message -----
> > > 件名: [Fuego] RFC: Fuego's version up and other changes
> > > 送信者: "Daniel Sangorrin" <daniel.sangorrin@toshiba.co.jp>
> > > 送信日時: 2017年01月11日(水) 09:28:07
> > > 宛先: "Bird, Timothy" <Tim.Bird@am.sony.com>, fuego@lists.
> > > linuxfoundation.org
> > >
> > > > Hi everyone,
> > > >
> > > > I've been working on Fuego's upgrade for a few weeks, and in the
> > > process I
> > > > have made quite a few changes.
> > > > You can check it by following these instructions. I'd really
> > > appreciate
> > > > your feedback:
> > > >
> > > > Quickstart:
> > > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego.git
> 
> > > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego-core.git
> 
> > >
> > > >   $ cd fuego/
> > > >   $ ./install.sh
> > > >   $ ./fuego-host-scripts/docker-create-container.sh
> > > >   $ ./fuego-host-scripts/docker-start-container.sh
> > > >   docker# fuego-create-node --board docker
> > > >   docker# fuego-create-jobs --board docker --testplan testplan_
> docker
> > > > --distrib nosyslogd.dist
> > > >   Then, start the "docker.testplan_docker.batch" job which will
> > > trigger
> > > > the rest of jobs.
> > > >
> > > > Here is a list of the most fundamental modifications I've added:
> > > > - I automatized the version upgrade process. So far I have updated
> > > Jenkins
> > > > 3 times quite smoothly.
> > > >    The current version is the latest LTS version (2.31.1 LTS)
> > > > - I reduced plugin dependencies to the minimum (only 2, and they
> can
> > > be
> > > > made optional).
> > > > - I removed dependencies on all groovy scripts.
> > > > - Userdata is gone. Instead we now have fuego-ro and fuego-rw,
> which
> > > > together with fuego-core
> > > >    are mounted as external docker volumes. Only fuego-rw, which
> > > contains
> > > > logs and build folders,
> > > >    is mounted in RW mode. I also fixed the uid/gid of jenkins so
> that
> > > it
> > > > matches the one in the host.
> > > >    This means you can develop comfortably on your host, with GUI
> tools,
> > > > and don't fear a buggy
> > > >    script deleting your folders.
> > > > - I added a  tool for generating jobs from testplans.  Creating
> jobs
> > > from
> > > > testplans is powerful.
> > > >    It allows you, for example, to easily specify the timeout for
> each
> > > test
> > > > on your board.
> > > >    The tool also generates the corresponding batch job. For
> example,
> > > you
> > > > can easily trigger all
> > > >    tests for your board and a specific testplan periodically.
> > > >    [Note] This will also enable the creation of scripts for
> comparing
> > > test
> > > > results across
> > > >    different boards and test parameters (on my TODO list).
> > > >
> > > > Other less fundamental changes include:
> > > > - I added support for using the docker container itself as a
> target
> > > board
> > > > (e.g.: for a quickstart or unit tests)
> > > > - I fixed the flot plugin (java and javascript fixes)
> > > > - Now you can click on a node and see which jobs (and testplans)
> are
> > > > assigned to your board. This was
> > > >   one of my old feature requests and makes Fuego's GUI easier to
> use.
> > > > - I added quite a few fixes and improvements to the core engine
> > > scripts
> > > > and tests (too detailed to describe here).
> > > > - I added Excel output support for IOzone
> > > > - Faster docker build time (ARMhf compiler installation is now
> > > optional,
> > > > no latex..)
> > > > - I put a fixed name to the fuego container, instead of the
> > > > "last_container_id" file.
> > > > - I removed the inotify script
> > > > - I added scripts for easily removing the docker container/image.
> > > >
> > > > I have to split the current "huge" patch into smaller chunks and
> test
> > > it
> > > > further but it would
> > > > be nice to get your feedback about the approach in general.
> > > >
> > > > Tim: I haven't updated ftc nor your "Christmas commits" yet.
> > > >
> > > > Best regards
> > > > Daniel
> > > >
> > > >
> > > > _______________________________________________
> > > > Fuego mailing list
> > > > Fuego@lists.linuxfoundation.org
> > > > https://lists.linuxfoundation.org/mailman/listinfo/fuego
> > > >
> > >
> >
> >
> >
> >
> 

Re: [Fuego] RFC: Fuego's version up and other changes

[Ibe.Kengo]

Hi, Daniel-san

Thank you for your prompt reply.
I have more a few question.

> Humm You should not need to change any permissions.
OK.
I removed the Fuego and the Fuego-core,
after that, I redownloaded the Fuego and the Fuego-core and rebuilded
and start Docker.
I can see following permissions of directories. Is it correct?
==
melco@debian8:~/$ ./fuego/fuego-host-scripts/docker-start-container.sh
Starting Fuego container (fuego-container)
[ ok ] Starting Jenkins Continuous Integration Server: jenkins.

root@debian8:/# ls -al
total 84
drwxr-xr-x  65 root    root    4096 Jan 12 07:26 .
drwxr-xr-x  65 root    root    4096 Jan 12 07:26 ..
-rwxr-xr-x   1 root    root       0 Jan 12 07:26 .dockerenv
drwxr-xr-x   2 root    root    4096 Jan 12 00:53 bin
drwxr-xr-x   2 root    root    4096 Sep 12 04:09 boot
drwxr-xr-x   5 root    root     380 Jan 12 07:26 dev
drwxr-xr-x  96 root    root    4096 Jan 12 07:26 etc
drwxr-xr-x   4 jenkins jenkins 4096 Jan 12 07:07 fuego-core
drwxr-xr-x   5 jenkins jenkins 4096 Jan 12 07:03 fuego-ro
drwxr-xr-x   2 root    root    4096 Jan 12 07:26 fuego-rw
drwxr-xr-x   2 root    root    4096 Sep 12 04:09 home
~

melco@debian8:~/$ ls -al fuego
合計 68
drwxr-xr-x 8 melco melco 4096  1月 12 16:26 .
drwxr-xr-x 4 melco melco 4096  1月 12 16:02 ..
-rw-r--r-- 1 melco melco   42  1月 12 16:03 .dockerignore
drwxr-xr-x 8 melco melco 4096  1月 12 16:03 .git
-rw-r--r-- 1 melco melco    9  1月 12 16:03 .gitignore
-rw-r--r-- 1 melco melco 1482  1月 12 16:03 CHANGELOG
-rw-r--r-- 1 melco melco 4810  1月 12 16:03 Dockerfile
-rw-r--r-- 1 melco melco 1492  1月 12 16:03 LICENSE
-rw-r--r-- 1 melco melco 3414  1月 12 16:03 README
-rw-r--r-- 1 melco melco   85  1月 12 16:03 VERSION
drwxr-xr-x 3 melco melco 4096  1月 12 16:03 docs
drwxr-xr-x 3 melco melco 4096  1月 12 16:03 frontend-install
drwxr-xr-x 2 melco melco 4096  1月 12 16:03 fuego-host-scripts
drwxr-xr-x 5 melco melco 4096  1月 12 16:03 fuego-ro
drwxr-xr-x 2 root  root  4096  1月 12 16:26 fuego-rw
-rwxr-xr-x 1 melco melco   48  1月 12 16:03 install.sh

==
“melco” user is only user on host PC which is Debian8.
I’m concerned about fuego-rw permissions and owner.

> The script is supposed to use your user's uid/gid for the jenkins
daemon.
Do you mean that I need to make “Jenkins” user on host PC?
Or
Can I use “melco” user inside Docker?

> However, I have only tested it on my pc which has a single user.
> Do you have more than one user?
I have one user. I'v not made "jenkins" user on host PC.

I’ll check other point following your advice.
Thank you so much.
Kengo


----- Original Message -----
件名: RE: [Fuego] RFC: Fuego's version up and other changes
送信者: "Daniel Sangorrin" <daniel.sangorrin@toshiba.co.jp>
送信日時: 2017年01月12日(木) 15:44:34
宛先: Ibe.Kengo@bx.MitsubishiElectric.co.jp
Cc: fuego@lists.linuxfoundation.org

> Hi Kengo-san,
>
> Thanks a lot for your feedback.
>
> > -----Original Message-----
> > From: Ibe.Kengo@bx.MitsubishiElectric.co.jp [mailto:Ibe.Kengo@bx.
MitsubishiElectric.co.jp]
> > Sent: Thursday, January 12, 2017 2:52 PM
> > To: Daniel Sangorrin
> > Cc: fuego@lists.linuxfoundation.org
> > Subject: Re: [Fuego] RFC: Fuego's version up and other changes
> >
> > Hi, Daniel-san.
> >
> > I’m very interested in your work!
> > So I tried to run your build scripts to create Docker image today,
> > then I could run new Fuego.
> > And, I added Docker node and test cases to use following command.
> > ==
> >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego.git

> >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego-core.git

> >
> >   $ cd fuego/
> >   $ ./install.sh
> >   $ ./fuego-host-scripts/docker-create-container.sh
> >   $ ./fuego-host-scripts/docker-start-container.sh
> >   docker# fuego-create-node --board docker
> >   docker# fuego-create-jobs --board docker --testplan testplan_
docker
> > --distrib nosyslogd.dist
> > ==
> >
> > I can see  new node and new test jobs.
> > But, I can’t run tests like Dhrystone on Docker.
> >
> > So far, I had one small problem that is “Failed to mkdirs: /fuego-rw
/
> > buildzone”
> > This reason is the permission or owner of “fuego-rw”,
> > so I added the write permission to “fuego-rw” diredtory.
>
> Humm You should not need to change any permissions.
> The script is supposed to use your user's uid/gid for the jenkins
daemon.
> However, I have only tested it on my pc which has a single user.
> Do you have more than one user? Could you check that the jenkins uid/
gid
> is the same as yours? (you can check on /etc/passwd inside and outside
docker)
> Please also check that the fuego-rw's owner is your user (e.g.: don't
git clone with
> a different user or root)
>
> > I have solved this problem.
> >
> > After that, I got the error message like following in console log.
> > ====
> > Started by user anonymous
> > Building remotely on docker in workspace /fuego-rw/buildzone
> > [buildzone] $ /bin/sh -xe /tmp/hudson1255554151521556820.sh
> > + export Reboot=false
> > + Reboot=false
> > + export Rebuild=true
> > + Rebuild=true
> > + export Target_Cleanup=true
> > + Target_Cleanup=true
> > + export TESTDIR=Benchmark.Dhrystone
> > + TESTDIR=Benchmark.Dhrystone
> > + export TESTNAME=Dhrystone
> > + TESTNAME=Dhrystone
> > + export DISTRIB=nosyslogd.dist
> > + DISTRIB=nosyslogd.dist
> > + export TESTPLAN=testplan_docker
> > + TESTPLAN=testplan_docker
> > + timeout --signal=9 3m /bin/bash
> > /fuego-core/engine/tests/Benchmark.Dhrystone/Dhrystone.sh
> > board overlay: boards/docker.board
> > using boards/docker.board board overlay
> > using nosyslogd.dist overlay
> > BATCH_TESTPLAN is not set, using testplan_docker.json testplan
> > Traceback (most recent call last):
> >   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 615, in <
module
> > >
> >     run()
> >   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 607, in run
> >     generateProlog(output, ofcls, classes, testPlans, testSpecs)
> >   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 375, in
> > generateProlog
> >     outfile = open(outFilePath, "w")
> > IOError: [Errno 2] No such file or directory: '/fuego-rw/work/docker
_
> > prolog.sh'
> >
> > *** ABORTED ***
> >
> > Fuego error reason: Error while prolog.sh file generation
> >
> > Build step 'Execute shell' marked build as failure
> > [description-setter] Description set:
> > Finished: FAILURE
> > ====
> >
> > Did I miss to run test cases on Docker node?
> > If you know this solution, could you give me some advice?
>
> Could you check that fuego-rw is correctly mounted inside docker?
> outside-docker$ touch fuego-rw/hoge
> inside-docker$ ls /fuego-rw
> hoge <-- does it appear?
>
> On my PC (ubuntu 16.04, docker version 1.12.3 ) it's always succesful
so I will
> try on a different one. Which OS and docker version are you using?
>
> Thanks,
> Daniel
>
> > Thank you.
> >
> > Best Regard.
> > Kengo Ibe
> >
> > ----- Original Message -----
> > 件名: [Fuego] RFC: Fuego's version up and other changes
> > 送信者: "Daniel Sangorrin" <daniel.sangorrin@toshiba.co.jp>
> > 送信日時: 2017年01月11日(水) 09:28:07
> > 宛先: "Bird, Timothy" <Tim.Bird@am.sony.com>, fuego@lists.
> > linuxfoundation.org
> >
> > > Hi everyone,
> > >
> > > I've been working on Fuego's upgrade for a few weeks, and in the
> > process I
> > > have made quite a few changes.
> > > You can check it by following these instructions. I'd really
> > appreciate
> > > your feedback:
> > >
> > > Quickstart:
> > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego.git

> > >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego-core.git

> >
> > >   $ cd fuego/
> > >   $ ./install.sh
> > >   $ ./fuego-host-scripts/docker-create-container.sh
> > >   $ ./fuego-host-scripts/docker-start-container.sh
> > >   docker# fuego-create-node --board docker
> > >   docker# fuego-create-jobs --board docker --testplan testplan_
docker
> > > --distrib nosyslogd.dist
> > >   Then, start the "docker.testplan_docker.batch" job which will
> > trigger
> > > the rest of jobs.
> > >
> > > Here is a list of the most fundamental modifications I've added:
> > > - I automatized the version upgrade process. So far I have updated
> > Jenkins
> > > 3 times quite smoothly.
> > >    The current version is the latest LTS version (2.31.1 LTS)
> > > - I reduced plugin dependencies to the minimum (only 2, and they
can
> > be
> > > made optional).
> > > - I removed dependencies on all groovy scripts.
> > > - Userdata is gone. Instead we now have fuego-ro and fuego-rw,
which
> > > together with fuego-core
> > >    are mounted as external docker volumes. Only fuego-rw, which
> > contains
> > > logs and build folders,
> > >    is mounted in RW mode. I also fixed the uid/gid of jenkins so
that
> > it
> > > matches the one in the host.
> > >    This means you can develop comfortably on your host, with GUI
tools,
> > > and don't fear a buggy
> > >    script deleting your folders.
> > > - I added a  tool for generating jobs from testplans.  Creating
jobs
> > from
> > > testplans is powerful.
> > >    It allows you, for example, to easily specify the timeout for
each
> > test
> > > on your board.
> > >    The tool also generates the corresponding batch job. For
example,
> > you
> > > can easily trigger all
> > >    tests for your board and a specific testplan periodically.
> > >    [Note] This will also enable the creation of scripts for
comparing
> > test
> > > results across
> > >    different boards and test parameters (on my TODO list).
> > >
> > > Other less fundamental changes include:
> > > - I added support for using the docker container itself as a
target
> > board
> > > (e.g.: for a quickstart or unit tests)
> > > - I fixed the flot plugin (java and javascript fixes)
> > > - Now you can click on a node and see which jobs (and testplans)
are
> > > assigned to your board. This was
> > >   one of my old feature requests and makes Fuego's GUI easier to
use.
> > > - I added quite a few fixes and improvements to the core engine
> > scripts
> > > and tests (too detailed to describe here).
> > > - I added Excel output support for IOzone
> > > - Faster docker build time (ARMhf compiler installation is now
> > optional,
> > > no latex..)
> > > - I put a fixed name to the fuego container, instead of the
> > > "last_container_id" file.
> > > - I removed the inotify script
> > > - I added scripts for easily removing the docker container/image.
> > >
> > > I have to split the current "huge" patch into smaller chunks and
test
> > it
> > > further but it would
> > > be nice to get your feedback about the approach in general.
> > >
> > > Tim: I haven't updated ftc nor your "Christmas commits" yet.
> > >
> > > Best regards
> > > Daniel
> > >
> > >
> > > _______________________________________________
> > > Fuego mailing list
> > > Fuego@lists.linuxfoundation.org
> > > https://lists.linuxfoundation.org/mailman/listinfo/fuego
> > >
> >
>
>
>
>

Re: [Fuego] RFC: Fuego's version up and other changes

[Daniel Sangorrin]

Hi Kengo-san,

Thanks a lot for your feedback.

> -----Original Message-----
> From: Ibe.Kengo@bx.MitsubishiElectric.co.jp [mailto:Ibe.Kengo@bx.MitsubishiElectric.co.jp]
> Sent: Thursday, January 12, 2017 2:52 PM
> To: Daniel Sangorrin
> Cc: fuego@lists.linuxfoundation.org
> Subject: Re: [Fuego] RFC: Fuego's version up and other changes
> 
> Hi, Daniel-san.
> 
> I’m very interested in your work!
> So I tried to run your build scripts to create Docker image today,
> then I could run new Fuego.
> And, I added Docker node and test cases to use following command.
> ==
>   $ git clone --branch next https://bitbucket.org/nirrognas/fuego.git
>   $ git clone --branch next https://bitbucket.org/nirrognas/fuego-core.git
> 
>   $ cd fuego/
>   $ ./install.sh
>   $ ./fuego-host-scripts/docker-create-container.sh
>   $ ./fuego-host-scripts/docker-start-container.sh
>   docker# fuego-create-node --board docker
>   docker# fuego-create-jobs --board docker --testplan testplan_docker
> --distrib nosyslogd.dist
> ==
> 
> I can see  new node and new test jobs.
> But, I can’t run tests like Dhrystone on Docker.
> 
> So far, I had one small problem that is “Failed to mkdirs: /fuego-rw/
> buildzone”
> This reason is the permission or owner of “fuego-rw”,
> so I added the write permission to “fuego-rw” diredtory.

Humm You should not need to change any permissions.
The script is supposed to use your user's uid/gid for the jenkins daemon.
However, I have only tested it on my pc which has a single user.
Do you have more than one user? Could you check that the jenkins uid/gid
is the same as yours? (you can check on /etc/passwd inside and outside docker)
Please also check that the fuego-rw's owner is your user (e.g.: don't git clone with
a different user or root)

> I have solved this problem.
> 
> After that, I got the error message like following in console log.
> ====
> Started by user anonymous
> Building remotely on docker in workspace /fuego-rw/buildzone
> [buildzone] $ /bin/sh -xe /tmp/hudson1255554151521556820.sh
> + export Reboot=false
> + Reboot=false
> + export Rebuild=true
> + Rebuild=true
> + export Target_Cleanup=true
> + Target_Cleanup=true
> + export TESTDIR=Benchmark.Dhrystone
> + TESTDIR=Benchmark.Dhrystone
> + export TESTNAME=Dhrystone
> + TESTNAME=Dhrystone
> + export DISTRIB=nosyslogd.dist
> + DISTRIB=nosyslogd.dist
> + export TESTPLAN=testplan_docker
> + TESTPLAN=testplan_docker
> + timeout --signal=9 3m /bin/bash
> /fuego-core/engine/tests/Benchmark.Dhrystone/Dhrystone.sh
> board overlay: boards/docker.board
> using boards/docker.board board overlay
> using nosyslogd.dist overlay
> BATCH_TESTPLAN is not set, using testplan_docker.json testplan
> Traceback (most recent call last):
>   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 615, in <module
> >
>     run()
>   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 607, in run
>     generateProlog(output, ofcls, classes, testPlans, testSpecs)
>   File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 375, in
> generateProlog
>     outfile = open(outFilePath, "w")
> IOError: [Errno 2] No such file or directory: '/fuego-rw/work/docker_
> prolog.sh'
> 
> *** ABORTED ***
> 
> Fuego error reason: Error while prolog.sh file generation
> 
> Build step 'Execute shell' marked build as failure
> [description-setter] Description set:
> Finished: FAILURE
> ====
> 
> Did I miss to run test cases on Docker node?
> If you know this solution, could you give me some advice?

Could you check that fuego-rw is correctly mounted inside docker?
outside-docker$ touch fuego-rw/hoge
inside-docker$ ls /fuego-rw
hoge <-- does it appear?

On my PC (ubuntu 16.04, docker version 1.12.3 ) it's always succesful so I will
try on a different one. Which OS and docker version are you using?

Thanks,
Daniel
 
> Thank you.
> 
> Best Regard.
> Kengo Ibe
> 
> ----- Original Message -----
> 件名: [Fuego] RFC: Fuego's version up and other changes
> 送信者: "Daniel Sangorrin" <daniel.sangorrin@toshiba.co.jp>
> 送信日時: 2017年01月11日(水) 09:28:07
> 宛先: "Bird, Timothy" <Tim.Bird@am.sony.com>, fuego@lists.
> linuxfoundation.org
> 
> > Hi everyone,
> >
> > I've been working on Fuego's upgrade for a few weeks, and in the
> process I
> > have made quite a few changes.
> > You can check it by following these instructions. I'd really
> appreciate
> > your feedback:
> >
> > Quickstart:
> >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego.git
> >   $ git clone --branch next https://bitbucket.org/nirrognas/fuego-core.git
> 
> >   $ cd fuego/
> >   $ ./install.sh
> >   $ ./fuego-host-scripts/docker-create-container.sh
> >   $ ./fuego-host-scripts/docker-start-container.sh
> >   docker# fuego-create-node --board docker
> >   docker# fuego-create-jobs --board docker --testplan testplan_docker
> > --distrib nosyslogd.dist
> >   Then, start the "docker.testplan_docker.batch" job which will
> trigger
> > the rest of jobs.
> >
> > Here is a list of the most fundamental modifications I've added:
> > - I automatized the version upgrade process. So far I have updated
> Jenkins
> > 3 times quite smoothly.
> >    The current version is the latest LTS version (2.31.1 LTS)
> > - I reduced plugin dependencies to the minimum (only 2, and they can
> be
> > made optional).
> > - I removed dependencies on all groovy scripts.
> > - Userdata is gone. Instead we now have fuego-ro and fuego-rw, which
> > together with fuego-core
> >    are mounted as external docker volumes. Only fuego-rw, which
> contains
> > logs and build folders,
> >    is mounted in RW mode. I also fixed the uid/gid of jenkins so that
> it
> > matches the one in the host.
> >    This means you can develop comfortably on your host, with GUI tools,
> > and don't fear a buggy
> >    script deleting your folders.
> > - I added a  tool for generating jobs from testplans.  Creating jobs
> from
> > testplans is powerful.
> >    It allows you, for example, to easily specify the timeout for each
> test
> > on your board.
> >    The tool also generates the corresponding batch job. For example,
> you
> > can easily trigger all
> >    tests for your board and a specific testplan periodically.
> >    [Note] This will also enable the creation of scripts for comparing
> test
> > results across
> >    different boards and test parameters (on my TODO list).
> >
> > Other less fundamental changes include:
> > - I added support for using the docker container itself as a target
> board
> > (e.g.: for a quickstart or unit tests)
> > - I fixed the flot plugin (java and javascript fixes)
> > - Now you can click on a node and see which jobs (and testplans) are
> > assigned to your board. This was
> >   one of my old feature requests and makes Fuego's GUI easier to use.
> > - I added quite a few fixes and improvements to the core engine
> scripts
> > and tests (too detailed to describe here).
> > - I added Excel output support for IOzone
> > - Faster docker build time (ARMhf compiler installation is now
> optional,
> > no latex..)
> > - I put a fixed name to the fuego container, instead of the
> > "last_container_id" file.
> > - I removed the inotify script
> > - I added scripts for easily removing the docker container/image.
> >
> > I have to split the current "huge" patch into smaller chunks and test
> it
> > further but it would
> > be nice to get your feedback about the approach in general.
> >
> > Tim: I haven't updated ftc nor your "Christmas commits" yet.
> >
> > Best regards
> > Daniel
> >
> >
> > _______________________________________________
> > Fuego mailing list
> > Fuego@lists.linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/fuego
> >
> 

Re: [Fuego] RFC: Fuego's version up and other changes

[Ibe.Kengo]

Hi, Daniel-san.

I’m very interested in your work!
So I tried to run your build scripts to create Docker image today,
then I could run new Fuego.
And, I added Docker node and test cases to use following command.
==
  $ git clone --branch next https://bitbucket.org/nirrognas/fuego.git
  $ git clone --branch next https://bitbucket.org/nirrognas/fuego-core.git

  $ cd fuego/
  $ ./install.sh
  $ ./fuego-host-scripts/docker-create-container.sh
  $ ./fuego-host-scripts/docker-start-container.sh
  docker# fuego-create-node --board docker
  docker# fuego-create-jobs --board docker --testplan testplan_docker
--distrib nosyslogd.dist
==

I can see  new node and new test jobs.
But, I can’t run tests like Dhrystone on Docker.

So far, I had one small problem that is “Failed to mkdirs: /fuego-rw/
buildzone”
This reason is the permission or owner of “fuego-rw”,
so I added the write permission to “fuego-rw” diredtory.
I have solved this problem.

After that, I got the error message like following in console log.
====
Started by user anonymous
Building remotely on docker in workspace /fuego-rw/buildzone
[buildzone] $ /bin/sh -xe /tmp/hudson1255554151521556820.sh
+ export Reboot=false
+ Reboot=false
+ export Rebuild=true
+ Rebuild=true
+ export Target_Cleanup=true
+ Target_Cleanup=true
+ export TESTDIR=Benchmark.Dhrystone
+ TESTDIR=Benchmark.Dhrystone
+ export TESTNAME=Dhrystone
+ TESTNAME=Dhrystone
+ export DISTRIB=nosyslogd.dist
+ DISTRIB=nosyslogd.dist
+ export TESTPLAN=testplan_docker
+ TESTPLAN=testplan_docker
+ timeout --signal=9 3m /bin/bash
/fuego-core/engine/tests/Benchmark.Dhrystone/Dhrystone.sh
board overlay: boards/docker.board
using boards/docker.board board overlay
using nosyslogd.dist overlay
BATCH_TESTPLAN is not set, using testplan_docker.json testplan
Traceback (most recent call last):
  File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 615, in <module
>
    run()
  File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 607, in run
    generateProlog(output, ofcls, classes, testPlans, testSpecs)
  File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 375, in
generateProlog
    outfile = open(outFilePath, "w")
IOError: [Errno 2] No such file or directory: '/fuego-rw/work/docker_
prolog.sh'

*** ABORTED ***

Fuego error reason: Error while prolog.sh file generation

Build step 'Execute shell' marked build as failure
[description-setter] Description set:
Finished: FAILURE
====

Did I miss to run test cases on Docker node?
If you know this solution, could you give me some advice?

Thank you.

Best Regard.
Kengo Ibe

----- Original Message -----
件名: [Fuego] RFC: Fuego's version up and other changes
送信者: "Daniel Sangorrin" <daniel.sangorrin@toshiba.co.jp>
送信日時: 2017年01月11日(水) 09:28:07
宛先: "Bird, Timothy" <Tim.Bird@am.sony.com>, fuego@lists.
linuxfoundation.org

> Hi everyone,
>
> I've been working on Fuego's upgrade for a few weeks, and in the
process I
> have made quite a few changes.
> You can check it by following these instructions. I'd really
appreciate
> your feedback:
>
> Quickstart:
>   $ git clone --branch next https://bitbucket.org/nirrognas/fuego.git
>   $ git clone --branch next https://bitbucket.org/nirrognas/fuego-core.git

>   $ cd fuego/
>   $ ./install.sh
>   $ ./fuego-host-scripts/docker-create-container.sh
>   $ ./fuego-host-scripts/docker-start-container.sh
>   docker# fuego-create-node --board docker
>   docker# fuego-create-jobs --board docker --testplan testplan_docker
> --distrib nosyslogd.dist
>   Then, start the "docker.testplan_docker.batch" job which will
trigger
> the rest of jobs.
>
> Here is a list of the most fundamental modifications I've added:
> - I automatized the version upgrade process. So far I have updated
Jenkins
> 3 times quite smoothly.
>    The current version is the latest LTS version (2.31.1 LTS)
> - I reduced plugin dependencies to the minimum (only 2, and they can
be
> made optional).
> - I removed dependencies on all groovy scripts.
> - Userdata is gone. Instead we now have fuego-ro and fuego-rw, which
> together with fuego-core
>    are mounted as external docker volumes. Only fuego-rw, which
contains
> logs and build folders,
>    is mounted in RW mode. I also fixed the uid/gid of jenkins so that
it
> matches the one in the host.
>    This means you can develop comfortably on your host, with GUI tools,
> and don't fear a buggy
>    script deleting your folders.
> - I added a  tool for generating jobs from testplans.  Creating jobs
from
> testplans is powerful.
>    It allows you, for example, to easily specify the timeout for each
test
> on your board.
>    The tool also generates the corresponding batch job. For example,
you
> can easily trigger all
>    tests for your board and a specific testplan periodically.
>    [Note] This will also enable the creation of scripts for comparing
test
> results across
>    different boards and test parameters (on my TODO list).
>
> Other less fundamental changes include:
> - I added support for using the docker container itself as a target
board
> (e.g.: for a quickstart or unit tests)
> - I fixed the flot plugin (java and javascript fixes)
> - Now you can click on a node and see which jobs (and testplans) are
> assigned to your board. This was
>   one of my old feature requests and makes Fuego's GUI easier to use.
> - I added quite a few fixes and improvements to the core engine
scripts
> and tests (too detailed to describe here).
> - I added Excel output support for IOzone
> - Faster docker build time (ARMhf compiler installation is now
optional,
> no latex..)
> - I put a fixed name to the fuego container, instead of the
> "last_container_id" file.
> - I removed the inotify script
> - I added scripts for easily removing the docker container/image.
>
> I have to split the current "huge" patch into smaller chunks and test
it
> further but it would
> be nice to get your feedback about the approach in general.
>
> Tim: I haven't updated ftc nor your "Christmas commits" yet.
>
> Best regards
> Daniel
>
>
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego
>

[Fuego] RFC: Fuego's version up and other changes

[Daniel Sangorrin]

Hi everyone,

I've been working on Fuego's upgrade for a few weeks, and in the process I have made quite a few changes.
You can check it by following these instructions. I'd really appreciate your feedback:

Quickstart:
  $ git clone --branch next https://bitbucket.org/nirrognas/fuego.git
  $ git clone --branch next https://bitbucket.org/nirrognas/fuego-core.git
  $ cd fuego/
  $ ./install.sh
  $ ./fuego-host-scripts/docker-create-container.sh
  $ ./fuego-host-scripts/docker-start-container.sh
  docker# fuego-create-node --board docker
  docker# fuego-create-jobs --board docker --testplan testplan_docker --distrib nosyslogd.dist
  Then, start the "docker.testplan_docker.batch" job which will trigger the rest of jobs.

Here is a list of the most fundamental modifications I've added:
- I automatized the version upgrade process. So far I have updated Jenkins 3 times quite smoothly.
   The current version is the latest LTS version (2.31.1 LTS)
- I reduced plugin dependencies to the minimum (only 2, and they can be made optional).
- I removed dependencies on all groovy scripts.
- Userdata is gone. Instead we now have fuego-ro and fuego-rw, which together with fuego-core
   are mounted as external docker volumes. Only fuego-rw, which contains logs and build folders,  
   is mounted in RW mode. I also fixed the uid/gid of jenkins so that it matches the one in the host.
   This means you can develop comfortably on your host, with GUI tools, and don't fear a buggy
   script deleting your folders.
- I added a  tool for generating jobs from testplans.  Creating jobs from testplans is powerful. 
   It allows you, for example, to easily specify the timeout for each test on your board.
   The tool also generates the corresponding batch job. For example, you can easily trigger all 
   tests for your board and a specific testplan periodically.
   [Note] This will also enable the creation of scripts for comparing test results across 
   different boards and test parameters (on my TODO list).

Other less fundamental changes include:
- I added support for using the docker container itself as a target board (e.g.: for a quickstart or unit tests)
- I fixed the flot plugin (java and javascript fixes)
- Now you can click on a node and see which jobs (and testplans) are assigned to your board. This was 
  one of my old feature requests and makes Fuego's GUI easier to use. 
- I added quite a few fixes and improvements to the core engine scripts and tests (too detailed to describe here).
- I added Excel output support for IOzone
- Faster docker build time (ARMhf compiler installation is now optional, no latex..)
- I put a fixed name to the fuego container, instead of the "last_container_id" file.
- I removed the inotify script
- I added scripts for easily removing the docker container/image.

I have to split the current "huge" patch into smaller chunks and test it further but it would 
be nice to get your feedback about the approach in general.

Tim: I haven't updated ftc nor your "Christmas commits" yet.

Best regards
Daniel