[Fuego] Fuego docker container size keeps growing and hits maximum storage capacity

[Fuego] Fuego docker container size keeps growing and hits maximum storage capacity

[Dhinakar Kalyanasundaram]

[-- Attachment #1: Type: text/html, Size: 4900 bytes --]

[-- Attachment #2: 201602111742151_N3WZA6X7.png --]
[-- Type: image/png, Size: 33527 bytes --]

Re: [Fuego] Fuego docker container size keeps growing and hits maximum storage capacity

[Tim.Bird]

> -----Original Message-----
> From: Dhinakar Kalyanasundaram 
> 
> Dear Fuego Users,
> 
> I am using Fuego version 1.3.0 (see below) in one of my Linux system
> (Ubuntu 16.04 LTS).
> 
> # This project adheres to Semantic Versioning.  See http://semver.org/
> # The codename for this release is "Draft"
> VERSION=1.3.0

Can you do: "git describe" in your fuego-core directory, and tell me what
it says?

> 
> I noticed that the fuego container size keeps growing continuously and hits
> the maximum storage capacity in a day or two.
> 
This is a serious problem.  I think it's likely that there's some runaway logging
going on inside the container.  It would be good to isolate this and see what's
going on.  It is possible some recent Docker changes have introduced a
new program that logs more than it should.
Here are some questions and some advice for finding the problem.

If you exit the container and restart it, does the size reset?
Do any files outside the container get bigger (e.g. in fuego-ro or fuego-rw 
on your host system)?

Please try to find the file inside the container that is getting big.
First look at logging, and if it's not something obvious in there, then
look other places.

Inside the container, do:
 $ cd /var/log
 $ du -shx * | sort -h

You can use the 'du | sort' sequence in various directories to try to track
down where the size is accumulating.  Since your container is at 45G, this will take
a long time if you run it at the root of the container filesystem, but that's probably
the best place to start.

> 
> It grows at the rate of 1GB per minute
> 
> root@userl2:/ <mailto:root@userl2:/> # docker ps -s
> CONTAINER ID        IMAGE                        COMMAND                  CREATED
> STATUS              PORTS               NAMES                                         SIZE
> 632433hg24324        fuego-docker-image   "/bin/sh -c 'serviceâ¦"   19 hours
> ago        Up 2 hours                              fuego-docker-container-12thNov2018
> 40.9GB (virtual 42.7GB)
> 
> 
> In the next 30 mins I see below:
> root@userl2:/ <mailto:root@userl2:/> # docker ps -s
> CONTAINER ID        IMAGE                        COMMAND                  CREATED
> STATUS              PORTS               NAMES                                         SIZE
> 6bdacdafd2343        fuego-docker-image   "/bin/sh -c 'serviceâ¦"   19 hours
> ago        Up 3 hours                              fuego-docker-container-12thNov2018
> 45.9GB (virtual 47.7GB)
> 
> 
> I am not sure if this issue is related to current version of Fuego framework
> but I don't see this issue in previous Fuego versions installed on my other
> Ubuntu systems.
> 
> I am using Ubuntu 16.04 LTS version across all my Linux systems.

I'm using Ubuntu 16.04 LTS on my Fuego host, and the latest Fuego version,
and I haven't seen this.  My guess is that some error is triggering some repeated
message to a log.
Once we identify the file or directory that is growing inside the container,
we should be able to pinpoint the issue.
 -- Tim

Re: [Fuego] Fuego docker container size keeps growing and hits maximum storage capacity

[Dhinakar Kalyanasundaram]

[-- Attachment #1: Type: text/html, Size: 15480 bytes --]

[-- Attachment #2: 201602111742151_N3WZA6X7.png --]
[-- Type: image/png, Size: 33527 bytes --]

Re: [Fuego] Fuego docker container size keeps growing and hits maximum storage capacity

[Tim.Bird]

> -----Original Message-----
> From: Dhinakar Kalyanasundaram 
> 
> Hi Tim,
> 
> Thanks for your prompt response.
> 
> Please find below the information that you asked for.
> 
> >> Can you do: "git describe" in your fuego-core directory, and tell me what it
> says?
> 
> 
> dhinakar@userl2:~/fuego-core$ <mailto:dhinakar@userl2:~/fuego-core$>
> git describe
> Draft-126-gb011f54
> 
> 
> Now the size of the container is 78GB
> 
> 
> 
> root@userl2:/var/log <mailto:root@userl2:/var/log> # du -shx * | sort -h
> 0       btmp
> 0       netserver.debug_79
> 0       netserver.debug_84
> 0       netserver.debug_85
> 0       wtmp
> 4.0K    dmesg
> 4.0K    exim4
> 4.0K    faillog
> 4.0K    fontconfig.log
> 4.0K    unattended-upgrades
> 12K     fsck
> 32K     lastlog
> 36K     alternatives.log
> 184K    apt
> 352K    dpkg.log
> 74G     Jenkins
> 
> 
> Looks like jenkins.log under /var/log/Jenkins folder is the culprit.
> 
> 
> 
> root@userl2:/var/log/jenkins <mailto:root@userl2:/var/log/jenkins> # ls -l
> total 77020892
> -rw-r--r-- 1 jenkins jenkins 78869378191 Nov 14 05:20 jenkins.log
> 
> 
> The truncated contents of it are listed below:
> 
> Since the file is too big I could not send you the whole contents.
> 
> Looks like some network logging is going on. I hope you get some clue based
> on the below data.
> 
> ..........
> 
>         [AllRecords@453309601 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords@1649968098 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name:
> 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords@557489375 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: MacShare.local.]
>         [AllRecords@252773764 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: MacShare.local.]
>         [AllRecords@645997782 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: MacShare.local.]
>         [AllRecords@1754886489 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: MacShare.local.]
>         [AllRecords@836812725 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords@441064872 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 184.56.108.107.in-addr.arpa.]
>         [AllRecords@2011237907 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: 184.56.108.107.in-addr.arpa.]
>         [AllRecords@1482899169 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name:
> 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords@1538609948 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: 184.56.108.107.in-addr.arpa.]
>         [AllRecords@1566802509 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: MacShare.local.]
>         [AllRecords@805289249 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords@1054464549 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name:
> 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords@1350619999 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name:
> 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords@968174403 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 184.56.108.107.in-addr.arpa.]
>         [AllRecords@623597902 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 184.56.108.107.in-addr.arpa.]
>         [AllRecords@1089741554 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: 184.56.108.107.in-addr.arpa.]
>         [AllRecords@1545756462 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name:
> 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords@1419179075 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: MacShare.local.]
>         [AllRecords@41846296 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 184.56.108.107.in-addr.arpa.]
>         [AllRecords@1083544303 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: MacShare.local.]
>         [AllRecords@467773961 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords@2086814507 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: MacShare.local.]
>         [AllRecords@388496345 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 184.56.108.107.in-addr.arpa.]
>         [AllRecords@1107783535 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: MacShare.local.]
>         [AllRecords@1982136902 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: MacShare.local.]
>         [AllRecords@2008579650 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name:
> 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords@1902730357 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: 184.56.108.107.in-addr.arpa.]
>         [AllRecords@1010122622 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name:
> 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords@1272678419 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: MacShare.local.]
>         [AllRecords@969064013 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords@542887643 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 184.56.108.107.in-addr.arpa.]
>         [AllRecords@1802083445 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: MacShare.local.]
>         [AllRecords@1654586474 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: 184.56.108.107.in-addr.arpa.]
>         [AllRecords@869123515 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: MacShare.local.]
>         [AllRecords@175149711 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 184.56.108.107.in-addr.arpa.]
>         [AllRecords@1688596387 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: MacShare.local.]
>         [AllRecords@1243081204 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: MacShare.local.]
>         [AllRecords@1744877612 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: 184.56.108.107.in-addr.arpa.]
>         [AllRecords@121341452 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: MacShare.local.]
>         [AllRecords@1423802761 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name:
> 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords@442725606 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords@2005906834 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name:
> 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords@2077868432 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: MacShare.local.]
>         [AllRecords@355669110 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: MacShare.local.]
>         [AllRecords@1283742169 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: MacShare.local.]
>         [AllRecords@86432974 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords@980048646 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 24.56.108.107.in-addr.arpa.]
>         [AllRecords@478020832 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords@359799997 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 24.56.108.107.in-addr.arpa.]
>         [AllRecords@1992083274 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: 184.56.108.107.in-addr.arpa.]
> ............

Looks like it could be related to this bug in Jenkins, having to do with multicast:
https://issues.jenkins-ci.org/browse/JENKINS-25369

I would truncate the log file, stop the container, and see if there's some way to
disable DNSMultiCast.  Something appears to have gone wrong with your network
configuration.  It could be the docker virtual network, or maybe something has
changed about your physical networks.

 -- Tim

Re: [Fuego] Fuego docker container size keeps growing and hits maximum storage capacity

[Tim.Bird]

This seems like a band-aid rather than a fix, but here's some information
about turning down the logging for Jenkins.  A proper fix would be to figure
out what error in your networking setup is causing the DNS errors, and fix
that.  But disabling logging might help.

Here's the logging level fix:
https://stackoverflow.com/questions/31719756/how-to-stop-jenkins-log-from-becoming-huge

See the last answer in that thread.  It has a workaround to turn off DNSMultiCast,
which might be the cause of the problem.
 --  Tim



> > -----Original Message-----
...

> 
> Looks like it could be related to this bug in Jenkins, having to do with
> multicast:
> https://issues.jenkins-ci.org/browse/JENKINS-25369
> 
> I would truncate the log file, stop the container, and see if there's some way
> to
> disable DNSMultiCast.  Something appears to have gone wrong with your
> network
> configuration.  It could be the docker virtual network, or maybe something
> has
> changed about your physical networks.
> 
>  -- Tim
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] Fuego docker container size keeps growing and hits maximum storage capacity

[Dhinakar Kalyanasundaram]

[-- Attachment #1: Type: text/html, Size: 6612 bytes --]

[-- Attachment #2: 201602111742151_N3WZA6X7.png --]
[-- Type: image/png, Size: 33527 bytes --]

Re: [Fuego] Fuego docker container size keeps growing and hits maximum storage capacity

[Dhinakar Kalyanasundaram]

[-- Attachment #1: Type: text/html, Size: 7964 bytes --]

[-- Attachment #2: 201602111742151_N3WZA6X7.png --]
[-- Type: image/png, Size: 33527 bytes --]

Re: [Fuego] Fuego docker container size keeps growing and hits maximum storage capacity

[Tim.Bird]

[-- Attachment #1.1: Type: text/plain, Size: 5509 bytes --]

Wow. I didn't know you could copy a file to a non-running container! That's handy to know. I'm glad you were able to find a workaround. We should add that to our docket tips page: http://fuegotest.org/wiki/Docker_Tips

- - Tim


-------- Original Message --------
Subject: RE: RE: RE: [Fuego] Fuego docker container size keeps growing and hits maximum storage capacity
From: Dhinakar Kalyanasundaram <dhinakar.k@samsung.com>
Date: Nov 16, 2018, 3:43 AM
To: "Bird, Timothy" <Tim.Bird@sony.com>,fuego@lists.linuxfoundation.org

Hi Tim,



Just an update to the issue that I mentioned at the end of my email.

I was able to find a way to fix the issue that I made because of which the container was not starting.

I got the container id via 'docker ps -a' command and replaced the 'etc/default/jenkins' file with the working one using 'docker cp' command as shown below.

Then I was able to start the docker container successfully.



$ sudo docker cp jenkins 6b4e6e63rfg7:/etc/default/



Regards,

Dhinakar





--------- Original Message ---------

Sender : Dhinakar Kalyanasundaram <dhinakar.k@samsung.com> Staff Engineer/FDS SW /SSIR/Samsung Electronics

Date : 2018-11-14 18:57 (GMT+5:30)

Title : RE: RE: [Fuego] Fuego docker container size keeps growing and hits maximum storage capacity

To : null<Tim.Bird@sony.com>, null<fuego@lists.linuxfoundation.org>

CC : Dhinakar Kalyanasundaram<dhinakar.k@samsung.com>



Hi Tim,



I have disabled 'DNSMultiCast' temporarily via Manage Jenkins -> System Log -> Log Levels (on the left)
and also deleted the /var/log/jenkins/jenkins.log file.

I tried a few things to permanently disable 'DNSMultiCast' inside the container (/etc/default/jenkins file)

Add the below JAVA_ARGS and restarted fuego but did not see it getting added under http://107.108.34.124:8080/fuego/systemInfo

JAVA_ARGS="-Djava.awt.headless=true -Dhudson.DNSMultiCast.disabled=true"



I also tried -> JAVA_ARGS="-Dhudson.DNSMultiCast.disabled=true" but did not work. Only Manage Jenkins -> System Log -> Log Levels (on the left) works.

Is there any way to permanently disable it within the container, so that each time I restart the container I don't have to do this -> Manage Jenkins -> System Log -> Log Levels (on the left)

Also, I tried setting the timezone to 'Asia/Kolkata' in /etc/default/jenkins file (add below arguments) but it also never gets updated

JAVA_ARGS="-Dorg.apache.commons.jelly.tags.fmt.timeZone=Asia/Kolkata"

Please let me know how to set timezone permanent as well.



Basically I want to keep everything updated in the container so that I can easily port it across machines.



During the whole process there was an interesting observation detailed below.

To edit timezone to 'Asia/Kolkata' I tried many things in /etc/default/jenkins file.

In one such case, I appended the below phrase to 'JENKINS_ARGS' in /etc/default/jenkins file, which seems erroneous,

JENKINS_ARGS="--webroot=/var/cache/jenkins/war --httpPort=8080 --prefix=/storm --Dorg.apache.commons.jelly.tags.fmt.timeZone=Asia/Kolkata"

saved the file, exited fuego, and tried to restart the container but it failed to start as shown below,



dhinakar@userl2:~/fuego$<mailto:dhinakar@userl2:~/fuego$> ./fuego-host-scripts/docker-start-container.sh fuego-docker-container-12thNov2018

Starting Fuego container (fuego-docker-container-12thNov2018)
[FAIL] Starting Jenkins Continuous Integration Server: jenkins failed!
Please create Fuego docker container via docker-create-container.sh script



So the question is, if we make such mistake to /etc/default/jenkins file and not able to start the container, is there a way to fix it from outside the container?

Because in my case I had to create a new container, so lost all changes done inside my previous container.

Regards,

Dhinakar





--------- Original Message ---------

Sender : Tim.Bird@sony.com <Tim.Bird@sony.com>

Date : 2018-11-14 13:13 (GMT+5:30)

Title : RE: [Fuego] Fuego docker container size keeps growing and hits maximum storage capacity

To : Tim.Bird@sony.com, Dhinakar Kalyanasundaram<dhinakar.k@samsung.com>, fuego@lists.linuxfoundation.org



This seems like a band-aid rather than a fix, but here's some information
about turning down the logging for Jenkins.  A proper fix would be to figure
out what error in your networking setup is causing the DNS errors, and fix
that.  But disabling logging might help.

Here's the logging level fix:
https://stackoverflow.com/questions/31719756/how-to-stop-jenkins-log-from-becoming-huge

See the last answer in that thread.  It has a workaround to turn off DNSMultiCast,
which might be the cause of the problem.
 --  Tim



> > -----Original Message-----
...

>
> Looks like it could be related to this bug in Jenkins, having to do with
> multicast:
> https://issues.jenkins-ci.org/browse/JENKINS-25369
>
> I would truncate the log file, stop the container, and see if there's some way
> to
> disable DNSMultiCast.  Something appears to have gone wrong with your
> network
> configuration.  It could be the docker virtual network, or maybe something
> has
> changed about your physical networks.
>
>  -- Tim
>
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego


[cid:XOK0LK7CT9SZ@namo.co.kr]




[-- Attachment #1.2: Type: text/html, Size: 8489 bytes --]

[-- Attachment #2: 201602111742151_N3WZA6X7.png --]
[-- Type: image/png, Size: 33527 bytes --]

Re: [Fuego] Fuego docker container size keeps growing and hits maximum storage capacity

[Dhinakar Kalyanasundaram]

[-- Attachment #1: Type: text/html, Size: 10262 bytes --]

[-- Attachment #2: 201602111742151_N3WZA6X7.png --]
[-- Type: image/png, Size: 33527 bytes --]

Re: [Fuego] Fuego docker container size keeps growing and hits maximum storage capacity

[Tim.Bird]

Thanks - this looks good.
 -- Tim


> -----Original Message-----
> From: Dhinakar Kalyanasundaram [mailto:dhinakar.k@samsung.com]
> Sent: Sunday, November 18, 2018 9:07 PM
> To: Bird, Timothy <Tim.Bird@sony.com>; fuego@lists.linuxfoundation.org
> Cc: Dhinakar Kalyanasundaram <dhinakar.k@samsung.com>
> Subject: RE: RE: RE: RE: [Fuego] Fuego docker container size keeps growing
> and hits maximum storage capacity
> 
> Hi Tim,
> 
> 
> 
> I have added it to 'docker tips' (contents pasted below).
> 
> Please feel free to edit it if you find anything inappropriate.
> 
> 
> 
> = Copy/Replace a file into a non-running container =
> Background:
> Consider a case where you make some changes to '/etc/default/jenkins' file
> when your container is running and restarted the container. Unfortunately
> your container did not start because of an issue in the '/etc/default/jenkins'
> file. How do you fix it as the container itself is not running?
> 
> Solution:
> Get the container id (of the non-running container) via
> {{{
> $ 'docker ps -a' command
> }}}
> 
> Replace the faulty file with original/corrected one via 'docker cp' command as
> shown in the example below.
> {{{
> $ sudo docker cp jenkins 6b4e6e63rfg7:/etc/default/
> }}}
> where '6b4e6e63rfg7' is the container id of the non-running container
> 
> Now you will able to start the docker container successfully.
> 
> 
> Regards,
> 
> Dhinakar
> 
> 
> 
> 
> 
> --------- Original Message ---------
> 
> Sender : Tim.Bird@sony.com <Tim.Bird@sony.com>
> 
> Date : 2018-11-17 05:15 (GMT+5:30)
> 
> Title : RE: RE: RE: [Fuego] Fuego docker container size keeps growing and hits
> maximum storage capacity
> 
> To : Dhinakar Kalyanasundaram<dhinakar.k@samsung.com>,
> fuego@lists.linuxfoundation.org
> 
> 
> 
> Wow. I didn't know you could copy a file to a non-running container! That's
> handy to know. I'm glad you were able to find a workaround. We should add
> that to our docket tips page: http://fuegotest.org/wiki/Docker_Tips
> 
> - - Tim
> 
> 
> -------- Original Message --------
> Subject: RE: RE: RE: [Fuego] Fuego docker container size keeps growing and
> hits maximum storage capacity
> From: Dhinakar Kalyanasundaram <dhinakar.k@samsung.com>
> Date: Nov 16, 2018, 3:43 AM
> To: "Bird, Timothy" <Tim.Bird@sony.com>,fuego@lists.linuxfoundation.org
> 
> Hi Tim,
> 
> 
> 
> Just an update to the issue that I mentioned at the end of my email.
> 
> I was able to find a way to fix the issue that I made because of which the
> container was not starting.
> 
> I got the container id via 'docker ps -a' command and replaced the
> 'etc/default/jenkins' file with the working one using 'docker cp' command as
> shown below.
> 
> Then I was able to start the docker container successfully.
> 
> 
> 
> $ sudo docker cp jenkins 6b4e6e63rfg7:/etc/default/
> 
> 
> 
> Regards,
> 
> Dhinakar
> 
> 
> 
> 
> 
> --------- Original Message ---------
> 
> Sender : Dhinakar Kalyanasundaram <dhinakar.k@samsung.com> Staff
> Engineer/FDS SW /SSIR/Samsung Electronics
> 
> Date : 2018-11-14 18:57 (GMT+5:30)
> 
> Title : RE: RE: [Fuego] Fuego docker container size keeps growing and hits
> maximum storage capacity
> 
> To : null<Tim.Bird@sony.com>, null<fuego@lists.linuxfoundation.org>
> 
> CC : Dhinakar Kalyanasundaram<dhinakar.k@samsung.com>
> 
> 
> 
> Hi Tim,
> 
> 
> 
> I have disabled 'DNSMultiCast' temporarily via Manage Jenkins -> System Log
> -> Log Levels (on the left)
> and also deleted the /var/log/jenkins/jenkins.log file.
> 
> I tried a few things to permanently disable 'DNSMultiCast' inside the
> container (/etc/default/jenkins file)
> 
> Add the below JAVA_ARGS and restarted fuego but did not see it getting
> added under http://107.108.34.124:8080/fuego/systemInfo
> 
> JAVA_ARGS="-Djava.awt.headless=true -
> Dhudson.DNSMultiCast.disabled=true"
> 
> 
> 
> I also tried -> JAVA_ARGS="-Dhudson.DNSMultiCast.disabled=true" but did
> not work. Only Manage Jenkins -> System Log -> Log Levels (on the left)
> works.
> 
> Is there any way to permanently disable it within the container, so that each
> time I restart the container I don't have to do this -> Manage Jenkins ->
> System Log -> Log Levels (on the left)
> 
> 
> Also, I tried setting the timezone to 'Asia/Kolkata' in /etc/default/jenkins file
> (add below arguments) but it also never gets updated
> 
> JAVA_ARGS="-
> Dorg.apache.commons.jelly.tags.fmt.timeZone=Asia/Kolkata"
> 
> Please let me know how to set timezone permanent as well.
> 
> 
> 
> Basically I want to keep everything updated in the container so that I can
> easily port it across machines.
> 
> 
> 
> During the whole process there was an interesting observation detailed
> below.
> 
> To edit timezone to 'Asia/Kolkata' I tried many things in /etc/default/jenkins
> file.
> 
> In one such case, I appended the below phrase to 'JENKINS_ARGS' in
> /etc/default/jenkins file, which seems erroneous,
> 
> JENKINS_ARGS="--webroot=/var/cache/jenkins/war --httpPort=8080 --
> prefix=/storm --
> Dorg.apache.commons.jelly.tags.fmt.timeZone=Asia/Kolkata"
> 
> saved the file, exited fuego, and tried to restart the container but it failed to
> start as shown below,
> 
> 
> 
> dhinakar@userl2:~/fuego$ <mailto:dhinakar@userl2:~/fuego$>  ./fuego-
> host-scripts/docker-start-container.sh fuego-docker-container-12thNov2018
> 
> Starting Fuego container (fuego-docker-container-12thNov2018)
> [FAIL] Starting Jenkins Continuous Integration Server: jenkins failed!
> Please create Fuego docker container via docker-create-container.sh script
> 
> 
> 
> So the question is, if we make such mistake to /etc/default/jenkins file and
> not able to start the container, is there a way to fix it from outside the
> container?
> 
> Because in my case I had to create a new container, so lost all changes done
> inside my previous container.
> 
> 
> Regards,
> 
> Dhinakar
> 
> 
> 
> 
> 
> --------- Original Message ---------
> 
> Sender : Tim.Bird@sony.com <Tim.Bird@sony.com>
> 
> Date : 2018-11-14 13:13 (GMT+5:30)
> 
> Title : RE: [Fuego] Fuego docker container size keeps growing and hits
> maximum storage capacity
> 
> To : Tim.Bird@sony.com, Dhinakar
> Kalyanasundaram<dhinakar.k@samsung.com>,
> fuego@lists.linuxfoundation.org
> 
> 
> 
> This seems like a band-aid rather than a fix, but here's some information
> about turning down the logging for Jenkins.  A proper fix would be to figure
> out what error in your networking setup is causing the DNS errors, and fix
> that.  But disabling logging might help.
> 
> Here's the logging level fix:
> https://stackoverflow.com/questions/31719756/how-to-stop-jenkins-log-
> from-becoming-huge
> 
> See the last answer in that thread.  It has a workaround to turn off
> DNSMultiCast,
> which might be the cause of the problem.
>  --  Tim
> 
> 
> 
> > > -----Original Message-----
> ...
> 
> >
> > Looks like it could be related to this bug in Jenkins, having to do with
> > multicast:
> > https://issues.jenkins-ci.org/browse/JENKINS-25369
> >
> > I would truncate the log file, stop the container, and see if there's some
> way
> > to
> > disable DNSMultiCast.  Something appears to have gone wrong with your
> > network
> > configuration.  It could be the docker virtual network, or maybe something
> > has
> > changed about your physical networks.
> >
> >  -- Tim
> >
> > _______________________________________________
> > Fuego mailing list
> > Fuego@lists.linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/fuego
> 
> 
> 
> <http://ext.samsung.net/mail/ext/v1/external/status/update?userid=dhina
> kar.k&do=bWFpbElEPTIwMTgxMTE5MDUwNzAyZXBjbXM1cDgxNGUwZGUx
> ODRkZmUzZTY3ZDlmOTZhNWU5MjM1NTRiNiZyZWNpcGllbnRBZGRyZXNzPV
> RpbS5CaXJkQHNvbnkuY29t>