Re: [Fuego] [LTSI-dev] First post, LTSI TEST questions

Re: [Fuego] [LTSI-dev] First post, LTSI TEST questions

[Bird, Timothy]

> -----Original Message-----
> From: ltsi-dev-bounces@lists.linuxfoundation.org [mailto:ltsi-dev-
>
> Our systems team is evaluating if the LTSI test system might be applicable to our
> kernel QA process.
> If so we would be pleased to be users and perhaps, in time, contributors.
> 
> Is there a catalog of the “60+” tests currently in the LTSI Test System that
> documents the details of what each test accomplishes?

Unfortunately, no.  Each test is supposed to come with a description field
filled in (in it's config.xml file).  But I was recently examining them, and the
descriptions are often missing, and sometimes quite minimal.  IMHO this is
something we should really fix up.

One feature I'd like to add to the command line tool (which is under construction
at the moment) is the ability to list the descriptions in a kind-of catalog form,
so people can peruse these more easily.

And finally, I'm hoping in the very long term to put each test in it's own package,
and have such information easily accessible via a standard package manager tool.

Sorry - I know this doesn't help you in your evaluation in the short term.

> Thanks very much in advance!

Thanks for taking the time to evaluate the tool.  As we receive feedback from people
it helps us prioritize the things we're working on.
 -- Tim

Re: [Fuego] [LTSI-dev] First post, LTSI TEST questions

[Bird, Timothy]

> -----Original Message-----
> From: Doug Crawford [mailto:dcrawford@zonoff.com]
> Sent: Wednesday, September 21, 2016 11:34 AM
> To: Bird, Timothy <Tim.Bird@am.sony.com>
> Cc: ltsi-dev@lists.linuxfoundation.org; fuego@lists.linuxfoundation.org
> Subject: Re: [LTSI-dev] First post, LTSI TEST questions
> 
> 
> > On Sep 20, 2016, at 8:23 PM, Bird, Timothy <Tim.Bird@am.sony.com> wrote:
> >
> >> -----Original Message-----
> >> From: ltsi-dev-bounces@lists.linuxfoundation.org [mailto:ltsi-dev-
> >>
> >> Our systems team is evaluating if the LTSI test system might be applicable to
> our
> >> kernel QA process.
> >> If so we would be pleased to be users and perhaps, in time, contributors.
> >>
> >> Is there a catalog of the “60+” tests currently in the LTSI Test System that
> >> documents the details of what each test accomplishes?
> >
> > Unfortunately, no.  Each test is supposed to come with a description field
> > filled in (in it's config.xml file).  But I was recently examining them, and the
> > descriptions are often missing, and sometimes quite minimal.  IMHO this is
> > something we should really fix up.
> >
> > One feature I'd like to add to the command line tool (which is under
> construction
> > at the moment) is the ability to list the descriptions in a kind-of catalog form,
> > so people can peruse these more easily.
> >
> > And finally, I'm hoping in the very long term to put each test in it's own
> package,
> > and have such information easily accessible via a standard package manager
> tool.
> >
> > Sorry - I know this doesn't help you in your evaluation in the short term.
> >
> >> Thanks very much in advance!
> >
> > Thanks for taking the time to evaluate the tool.  As we receive feedback from
> people
> > it helps us prioritize the things we're working on.
> > -- Tim
> >
> 
> Tim:
> Thanks so much for the heads up.  I appreciate you mentioning the plans for
> providing content info.
> I will continue to work through getting the system running.
> In the mean time, can you point me to the source files for the tests themselves?

The tests consist of multiple files, but the main "base script" for each test, which
is the one that builds the source, deploys it to target, executes it and collects
and analyses the result is in each test's "home" directory under /home/jenkins/tests
(inside the container).

So, for example, the base script for bonnie is in /home/Jenkins/tests/Benchmark.bonnie,
and is called bonnie++.sh

See http://bird.org/fuego/Adding_a_test#Files for a description of the different files
that are part of a test definition, and where they are in the system (inside the container).

Because there are symlinks, you can find the files via multiple paths:
/home/jenkins/tests/{test_name}
or
/home/jenkins/fuego/engine/tests/{test_name}

Note that /home/jenkins/fuego corresponds to the git repository fuego-core.
(See https://bitbucket.org/tbird20d/fuego-core/)

> I looked around, expecting to find .c or .cpp files within the jaa-public directory
> and the docker volume, no luck.
> Perhaps these come in on a further installation step, but I don’t seem to see that
> happening in
> the steps of the jta document pdf.

It sounds like you're using jta instead of fuego (and an older version of jta at that).
You might want to try using fuego instead, from this repository:
https://bitbucket.org/tbird20d/fuego/

I'm not sure how far behind fuego jta is, but jta is deprecated now.
 -- Tim

Re: [Fuego] [LTSI-dev] First post, LTSI TEST questions

[Doug Crawford]

[-- Attachment #1: Type: text/plain, Size: 664 bytes --]


> On Sep 21, 2016, at 7:33 PM, Bird, Timothy <Tim.Bird@am.sony.com> wrote:
> 
> 
> 
.
.
.
> It sounds like you're using jta instead of fuego (and an older version of jta at that).
> You might want to try using fuego instead, from this repository:
> https://bitbucket.org/tbird20d/fuego/
> 
> I'm not sure how far behind fuego jta is, but jta is deprecated now.
> -- Tim
> 
> 

OK I’ll start over with Fuego.
Should the  “Download LTSI Test Environment” link on web page http://ltsi.linuxfoundation.org/ltsi-test-project <http://ltsi.linuxfoundation.org/ltsi-test-project>
point to your fuego instead of the JTA version?  

-Doug C



[-- Attachment #2: Type: text/html, Size: 1494 bytes --]

Re: [Fuego] [LTSI-dev] First post, LTSI TEST questions

[Doug Crawford]

> 
> It sounds like you're using jta instead of fuego (and an older version of jta at that).
> You might want to try using fuego instead, from this repository:
> https://bitbucket.org/tbird20d/fuego/
> 
> I'm not sure how far behind fuego jta is, but jta is deprecated now.
> -- Tim
> 
> 

I repeated the install with fuego from the repository and that went great!
Thanks so much for pointing me in the right direction.
The instructions on the repository page were spot on and an improvement on the 
Installation section fo fuego-docs.pdf .

So  now have the server up and accessible.
That’s a long way from where I was last week!  :)

The fuego-docs.pdf goes into great detail on creating tests and customizing the system for a new embedded system.
I think I need something a little different-  like a canned configuration for running the package
against something standardized, like a Raspberry Pi 3.
Can you "fast track” me somehow to achieving that?  If so I think I’ll then have what I need to
intelligently evaluate the test suite to our needs.

(Perhaps this is something that would be useful in general- inclusion of one standardized example
like the Pi)

— Doug C.

Re: [Fuego] [LTSI-dev] First post, LTSI TEST questions

[Bird, Timothy]

> -----Original Message-----
> From: Doug Crawford [mailto:dcrawford@zonoff.com]
> Sent: Thursday, September 22, 2016 5:01 AM
> To: Bird, Timothy <Tim.Bird@am.sony.com>
> Cc: ltsi-dev@lists.linuxfoundation.org; fuego@lists.linuxfoundation.org
> Subject: Re: [LTSI-dev] First post, LTSI TEST questions
> 
> 
> 	On Sep 21, 2016, at 7:33 PM, Bird, Timothy <Tim.Bird@am.sony.com
> <mailto:Tim.Bird@am.sony.com> > wrote:
> 
> 
> 
> 
> .
> .
> .
> 
> 
> 	It sounds like you're using jta instead of fuego (and an older version of
> jta at that).
> 	You might want to try using fuego instead, from this repository:
> 	https://bitbucket.org/tbird20d/fuego/
> 
> 	I'm not sure how far behind fuego jta is, but jta is deprecated now.
> 	-- Tim
> 
> 
> 
> 
> 
> OK I’ll start over with Fuego.
> Should the  “Download LTSI Test Environment” link on web page
> http://ltsi.linuxfoundation.org/ltsi-test-project
> point to your fuego instead of the JTA version?

Yes, it should.  Sorry about that.  I thought that might be the link you used... :-)
I don't own that page, but I'll try to get it changed.

If anyone from the LTSI project is watching this thread - can you change the link on
http://ltsi.linuxfoundation.org/ltsi-test-project
for the wording:  "Download LTSI Test Environment"
from
https://bitbucket.org/cogentembedded/jta-public/
to
https://bitbucket.org/tbird20d/fuego
?

As an aside, I'd like to get a pre-built fuego image on docker.com that people can try out
without having to do the initial build.

Thanks,
 -- Tim

Re: [Fuego] [LTSI-dev] First post, LTSI TEST questions

[Bird, Timothy]

> -----Original Message-----
> From: ltsi-dev-bounces@lists.linuxfoundation.org [mailto:ltsi-dev-
> bounces@lists.linuxfoundation.org] On Behalf Of Doug Crawford via LTSI-dev
> Sent: Thursday, September 22, 2016 7:11 AM
...
> The fuego-docs.pdf goes into great detail on creating tests and customizing the
> system for a new embedded system.
> I think I need something a little different-  like a canned configuration for
> running the package
> against something standardized, like a Raspberry Pi 3.
> Can you "fast track” me somehow to achieving that?  If so I think I’ll then have
> what I need to
> intelligently evaluate the test suite to our needs.
> 
The only things you can do right now are 1) add a new board, or 2) use
the qemuarm board.  Unfortunately, there's a bug right now that the
pre-installed ARM toolchain doesn't work with the default Ubuntu ARM qemu image.
(Don't ask - it's a long story).

Adding a board shouldn't be too difficult.  Try the instructions at:
http://bird.org/fuego/Adding_a_board


> (Perhaps this is something that would be useful in general- inclusion of one
> standardized example
> like the Pi)

That sounds like a good idea.  The "boards" we currently have in there are not very common.
I do most of my work on a beaglebone black, and maybe I should add that board as one of our examples, since I think it's fairly popular.  Also, adding a raspberry pi would be good.

I don't want the target list to be too cluttered on first install, but it would be good, IMHO to have a few more examples - especially popular ones.

I don't have a raspberry pi.  If you follow the instructions and get one installed, would you mind sending me the files for it?

I would need the following:
/userdata/conf/boards/rasberrypi.board (or whatever you call it)
and
/userdata/conf/config.xml

I assume you are running raspbian on the board?  I assume that's the most popular distro.

Let me know if you can send me something.
 -- Tim

Re: [Fuego] [LTSI-dev] First post, LTSI TEST questions

[Doug Crawford]

[-- Attachment #1: Type: text/plain, Size: 3407 bytes --]


> On Sep 22, 2016, at 1:45 PM, Bird, Timothy <Tim.Bird@am.sony.com> wrote:
> 
> 
> 
>> -----Original Message-----
>> From: ltsi-dev-bounces@lists.linuxfoundation.org [mailto:ltsi-dev-
>> bounces@lists.linuxfoundation.org] On Behalf Of Doug Crawford via LTSI-dev
>> Sent: Thursday, September 22, 2016 7:11 AM
> ...
>> The fuego-docs.pdf goes into great detail on creating tests and customizing the
>> system for a new embedded system.
>> I think I need something a little different-  like a canned configuration for
>> running the package
>> against something standardized, like a Raspberry Pi 3.
>> Can you "fast track” me somehow to achieving that?  If so I think I’ll then have
>> what I need to
>> intelligently evaluate the test suite to our needs.
>> 
> The only things you can do right now are 1) add a new board, or 2) use
> the qemuarm board.  Unfortunately, there's a bug right now that the
> pre-installed ARM toolchain doesn't work with the default Ubuntu ARM qemu image.
> (Don't ask - it's a long story).
> 
> Adding a board shouldn't be too difficult.  Try the instructions at:
> http://bird.org/fuego/Adding_a_board
> 
> 
>> (Perhaps this is something that would be useful in general- inclusion of one
>> standardized example
>> like the Pi)
> 
> That sounds like a good idea.  The "boards" we currently have in there are not very common.
> I do most of my work on a beaglebone black, and maybe I should add that board as one of our examples, since I think it's fairly popular.  Also, adding a raspberry pi would be good.
> 
> I don't want the target list to be too cluttered on first install, but it would be good, IMHO to have a few more examples - especially popular ones.
> 
> I don't have a raspberry pi.  If you follow the instructions and get one installed, would you mind sending me the files for it?
> 
> I would need the following:
> /userdata/conf/boards/rasberrypi.board (or whatever you call it)
> and
> /userdata/conf/config.xml
> 
> I assume you are running raspbian on the board?  I assume that's the most popular distro.
> 
> Let me know if you can send me something.
> -- Tim


Let me look over the wiki and "the add a board" link see if I “get it” in pursuing adding the Pi 3.
Yes I would be using Raspian.  Is that a problem with ARM toolchain issue?
No big deal to use a different distro for this exercise.
As a fallback,  I would be open to acquiring a Beagle Bone Black in order to flip this around use your BBB config files instead.

Interestingly, the first thing that led me here was this presentation that set up a Pi 2.
http://events.linuxfoundation.org/sites/events/files/slides/ELCE2015-LTSI_Test_Project.pdf <http://events.linuxfoundation.org/sites/events/files/slides/ELCE2015-LTSI_Test_Project.pdf>
If this process is valid for Fuego, I should be able to get through adding the Pi3 by following it.
On the other hand, it may mean that there are already working files on the Pi somewhere within your organization too.

Supporting the Pi within the distribution would grease the skids for wide evaluation of the test suite since there is said to be some 3 million units sold.
It would also be useful as way to prove a working hardware and software set before turning attention to a new proprietary unknown embedded system.

Thank you very much for your kind and speedy response.

-Doug C

[-- Attachment #2: Type: text/html, Size: 4999 bytes --]