[Fuego] Dockerfile vs install-debian.sh

[Fuego] Dockerfile vs install-debian.sh

[Georg Piewald]

Hi guys,

I was just looking into how to install Fuego without a Docker container 
(since the CHANGELOG says it's possible). Apparently, that's what 
install-debian.sh is good for (or is there something else, that I missed?).

But honestly, this script seems like a halfhearted attempt to me and 
strongly violates the DRY principle. It is clearly descendant from 
Dockerfile, but did not evolve with it. Several later modifications in 
Dockerfile did not make it to install-debian.sh. How do you plan to keep 
these files in sync?

I propose to only let install-debian.sh have all the code that sets up 
the OS and installs all requirements. The Dockerfile instead should do 
(almost) nothing but execute install-debian.sh.

An additional benefit of that method is that it will be much easier to 
also keep Dockerfile.nojenkins in sync with the standard Dockerfile. 
Because here we have the same problem of lots of copied code, that is 
slowly drifting apart. We might even not need Dockerfile.nojenkins 
anymore at all.

I can offer to implement this, unless there are strong objections by anyone.

Regards, Georg

Re: [Fuego] Dockerfile vs install-debian.sh

[daniel.sangorrin]

Hi Georg
Tim: I have a question of your at the end.

> -----Original Message-----
> From: Fuego <fuego-bounces@lists.linuxfoundation.org> On Behalf Of Georg Piewald
> Sent: Monday, July 27, 2020 8:14 PM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] Dockerfile vs install-debian.sh
> 
> Hi guys,
> 
> I was just looking into how to install Fuego without a Docker container (since the CHANGELOG says it's possible). Apparently, that's what
> install-debian.sh is good for (or is there something else, that I missed?).

Yes, I created it in 2019 and it worked at that time. I run it together with LAVA.
Unfortunately, I haven't been able to maintain it since then for private reasons. Now I am back but I can't spend as much time as I'd like.
https://bkk19.sched.com/event/Li6u/bkk19-tr08-how-to-integrate-fuego-automated-testing-tool-in-your-ci-loop
You have the slides there (and the presentation is on youtube)

> But honestly, this script seems like a halfhearted attempt to me and strongly violates the DRY principle. It is clearly descendant from
> Dockerfile, but did not evolve with it. Several later modifications in Dockerfile did not make it to install-debian.sh. How do you plan to keep
> these files in sync?

You are right. This was just a fast prototype to prove a point (that Fuego can work with LAVA, Ktest, Linaro test definitions, and other test frameworks) on that conference.
It is waiting for proper integration and maintenance.
 
> I propose to only let install-debian.sh have all the code that sets up the OS and installs all requirements. The Dockerfile instead should do
> (almost) nothing but execute install-debian.sh.

Yes, that was my plan as well.
 
> An additional benefit of that method is that it will be much easier to also keep Dockerfile.nojenkins in sync with the standard Dockerfile.
> Because here we have the same problem of lots of copied code, that is slowly drifting apart. We might even not need Dockerfile.nojenkins
> anymore at all.

Exactly.

> I can offer to implement this, unless there are strong objections by anyone.

That would super awesome. It would be good if you see my video on Youtube or at least read the slides to get a bit of background of what I wanted to do with it.
Also if you have any question about why i did that, or if there is a missing piece of code please let me know.

Tim: do you agree?

Best regards,
Daniel

Re: [Fuego] Dockerfile vs install-debian.sh

[daniel.sangorrin]

 -----Original Message-----
> From: sangorrin daniel(サンゴリン ダニエル □ＳＷＣ◯ＡＣＴ)
> Hi Georg
> Tim: I have a question of your at the end.

of your -> for you

^^U

> 
> > -----Original Message-----
> > From: Fuego <fuego-bounces@lists.linuxfoundation.org> On Behalf Of
> > Georg Piewald
> > Sent: Monday, July 27, 2020 8:14 PM
> > To: fuego@lists.linuxfoundation.org
> > Subject: [Fuego] Dockerfile vs install-debian.sh
> >
> > Hi guys,
> >
> > I was just looking into how to install Fuego without a Docker
> > container (since the CHANGELOG says it's possible). Apparently, that's what install-debian.sh is good for (or is there something else, that I
> missed?).
> 
> Yes, I created it in 2019 and it worked at that time. I run it together with LAVA.
> Unfortunately, I haven't been able to maintain it since then for private reasons. Now I am back but I can't spend as much time as I'd like.
> https://bkk19.sched.com/event/Li6u/bkk19-tr08-how-to-integrate-fuego-automated-testing-tool-in-your-ci-loop
> You have the slides there (and the presentation is on youtube)
> 
> > But honestly, this script seems like a halfhearted attempt to me and
> > strongly violates the DRY principle. It is clearly descendant from
> > Dockerfile, but did not evolve with it. Several later modifications in Dockerfile did not make it to install-debian.sh. How do you plan to
> keep these files in sync?
> 
> You are right. This was just a fast prototype to prove a point (that Fuego can work with LAVA, Ktest, Linaro test definitions, and other test
> frameworks) on that conference.
> It is waiting for proper integration and maintenance.
> 
> > I propose to only let install-debian.sh have all the code that sets up
> > the OS and installs all requirements. The Dockerfile instead should do
> > (almost) nothing but execute install-debian.sh.
> 
> Yes, that was my plan as well.
> 
> > An additional benefit of that method is that it will be much easier to also keep Dockerfile.nojenkins in sync with the standard Dockerfile.
> > Because here we have the same problem of lots of copied code, that is
> > slowly drifting apart. We might even not need Dockerfile.nojenkins anymore at all.
> 
> Exactly.
> 
> > I can offer to implement this, unless there are strong objections by anyone.
> 
> That would super awesome. It would be good if you see my video on Youtube or at least read the slides to get a bit of background of what I
> wanted to do with it.
> Also if you have any question about why i did that, or if there is a missing piece of code please let me know.
> 
> Tim: do you agree?
> 
> Best regards,
> Daniel

Re: [Fuego] Dockerfile vs install-debian.sh

[Bird, Tim]

> -----Original Message-----
> From: daniel.sangorrin@toshiba.co.jp 

> Hi Georg
> Tim: I have a question of your at the end.
> 
> > -----Original Message-----
> > From: Fuego <fuego-bounces@lists.linuxfoundation.org> On Behalf Of Georg Piewald
....
> That would super awesome. It would be good if you see my video on Youtube or at least read the slides to get a bit of background of what
> I wanted to do with it.
> Also if you have any question about why i did that, or if there is a missing piece of code please let me know.
> 
> Tim: do you agree?
Yes.  See my other mail.

Another item came up recently - which was the bit rot of the Fuego release test
(Functional.fuego_release_test).  That test uses (or is supposed to use) Dockerfile.test.

We don't have a documented way to build an "extra-beefy" container with the extra
stuff needed for selftesting (e.g. with chromium, selenium, and other bits).  I'm not
sure whether to integrate this into install.sh (with a new --test option), and whether
the code should go into install.sh or be a separate script like install-debian.sh.

Anyway - let's see what Georg comes up with and we'll take it from there.
 -- Tim

Re: [Fuego] Dockerfile vs install-debian.sh

[Georg Piewald]

Hi Daniel,

On 27.07.2020 13:52, daniel.sangorrin@toshiba.co.jp wrote:
> That would super awesome. It would be good if you see my video on
> Youtube or at least read the slides to get a bit of background of
> what I wanted to do with it.

Thanks for the video! I learned quite some things from it!

> Also if you have any question about why i did that, or if there is a
> missing piece of code please let me know.

I am currently working my way through it, trying to keep the 
modifications as small as possible (still, I'm afraid it won't be too 
easy for you to review). I understand most of what's happening there, 
but this line confuses me:

https://bitbucket.org/fuegotest/fuego/src/master/install-debian.sh#lines-148

What's the "jenkins cp" command?

Greets, Georg

Re: [Fuego] Dockerfile vs install-debian.sh

[daniel.sangorrin]

Hi Georg,

> -----Original Message-----
> From: Georg Piewald <gpi@mailbox.org>
> Sent: Tuesday, July 28, 2020 5:25 PM
> To: sangorrin daniel(サンゴリン ダニエル □ＳＷＣ◯ＡＣＴ) <daniel.sangorrin@toshiba.co.jp>; fuego@lists.linuxfoundation.org;
> Tim.Bird@sony.com
> Subject: Re: [Fuego] Dockerfile vs install-debian.sh
> 
> Hi Daniel,
> 
> On 27.07.2020 13:52, daniel.sangorrin@toshiba.co.jp wrote:
> > That would super awesome. It would be good if you see my video on
> > Youtube or at least read the slides to get a bit of background of what
> > I wanted to do with it.
> 
> Thanks for the video! I learned quite some things from it!
> 
> > Also if you have any question about why i did that, or if there is a
> > missing piece of code please let me know.
> 
> I am currently working my way through it, trying to keep the modifications as small as possible (still, I'm afraid it won't be too easy for you
> to review). I understand most of what's happening there, but this line confuses me:
> 
> https://bitbucket.org/fuegotest/fuego/src/master/install-debian.sh#lines-148
> 
> What's the "jenkins cp" command?

It looks like a typo, good catch. It should be "cp" only.
I am not sure how the word jenkins ended up there to be honest.

Kind regards,
Daniel



> 
> Greets, Georg

Re: [Fuego] Dockerfile vs install-debian.sh

[Bird, Tim]

> -----Original Message-----
> From: Georg Piewald
> 
> Hi guys,
> 
> I was just looking into how to install Fuego without a Docker container
> (since the CHANGELOG says it's possible). Apparently, that's what
> install-debian.sh is good for (or is there something else, that I missed?).

I believe that's it.  But Daniel would know.

> 
> But honestly, this script seems like a halfhearted attempt to me and
> strongly violates the DRY principle. It is clearly descendant from
> Dockerfile, but did not evolve with it. Several later modifications in
> Dockerfile did not make it to install-debian.sh. How do you plan to keep
> these files in sync?

To be brutally honest, I don't think we have a plan.  So making it so
that the information is in one place sounds like a great improvement.

> 
> I propose to only let install-debian.sh have all the code that sets up
> the OS and installs all requirements. The Dockerfile instead should do
> (almost) nothing but execute install-debian.sh.
> 
> An additional benefit of that method is that it will be much easier to
> also keep Dockerfile.nojenkins in sync with the standard Dockerfile.
> Because here we have the same problem of lots of copied code, that is
> slowly drifting apart. We might even not need Dockerfile.nojenkins
> anymore at all.
> 
> I can offer to implement this, unless there are strong objections by anyone.

No objections - and strong support.

Thanks very much for the offer. Any help is appreciated! :-)
 -- Tim

Re: [Fuego] Dockerfile vs install-debian.sh

[Georg Piewald]

Hi Tim and Daniel,

I pushed a first commit on a feature branch to Bitbucket. It's work in 
progress and far from done. But I must admit that I feel slightly 
uncomfortable, doing such a massive refactoring in your codebase, and 
I'm not sure if you'll approve that.

So maybe just follow along, and if you feel that it's going in a wrong 
direction, give me a stop sign, please.

Regards, Georg

Re: [Fuego] Dockerfile vs install-debian.sh

[daniel.sangorrin]

Hello Georg,

Sorry for the long delay answering your e-mail.

I wasn't sure where your bitbucket feature branch was.
Actually, could you send the commit(s) you have to the this mailing list for review?

Also please let me know if you want to continue the work. Otherwise, I can do it instead.

Thanks,
Daniel

> -----Original Message-----
> From: Fuego <fuego-bounces@lists.linuxfoundation.org> On Behalf Of Georg Piewald via Fuego
> Sent: Friday, July 31, 2020 6:39 AM
> To: fuego@lists.linuxfoundation.org
> Subject: Re: [Fuego] Dockerfile vs install-debian.sh
> 
> Hi Tim and Daniel,
> 
> I pushed a first commit on a feature branch to Bitbucket. It's work in
> progress and far from done. But I must admit that I feel slightly
> uncomfortable, doing such a massive refactoring in your codebase, and
> I'm not sure if you'll approve that.
> 
> So maybe just follow along, and if you feel that it's going in a wrong
> direction, give me a stop sign, please.
> 
> Regards, Georg
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego