From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Sony.onmicrosoft.com; s=selector1-Sony-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=8P9+1jo9Qp1bXQPQ1Wy24Il7oWrPK7FYk0LZKu0Vu9c=; b=k2SWOhIl64RMkvi3MlF4agPVLqWP9JUkjzt/uiXerHSSW3WWGLUY9jmmwd4SDSTRS+d8vPuXNuFmcKmlBt+xl6mM/qw7HdZxHtZLlZJgj6KcejjQgf6JOCtOUI9ltfWjJ0D6ysjMpXRBLZM69JpF0/eiYBAdjygpk50hh/8KkvQ= From: "Bird, Timothy" Date: Tue, 9 May 2017 19:04:22 +0000 Message-ID: References: In-Reply-To: Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [Fuego] First steps with fuego List-Id: Mailing list for the Fuego test framework List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Rafael Gago Castano , "fuego@lists.linuxfoundation.org" > From: Rafael Gago Castano on Tuesday, May 09, 2017 2:49 AM > > We are a group that is using a LAVA setup, but we are not completely happ= y > with it and we are trying alternatives. We like from fuego that it seems = to be > more hackable, it runs the server from a container (hopefully being able = to > develop tests locally) and doesn't try to bring in a lot of Web > technologies/languages. Thanks for taking a look at Fuego! I hope we can answer your questions and address any issues you have. >=20 > So far I have some questions. >=20 > I have built the .pdf under fuego/docs (fuego-docs.pdf) and I'm trying to > follow it on Ubuntu 16.04. I'm now blocked at step 4. The jenkins Web > interface is not showing anything specific to fuego, I see just a vanill= a Jenkins > server with no tabs, not something like this: >=20 > http://bird.org/fuego-files/fuego-dashboard-history.png Yeah - the PDF doc is unfortunately a bit out of date. We had a major cha= nge in the jenkins interface, and in the method of populating the interface, in= our 1.1 release, which was released in March. I'm sorry this didn't get updated fo= r that release. That file on our wiki that you point to refers to the old, pre v1= .1 interface. The best location for information on getting up and running is the wiki. T= his page: http://bird.org/fuego/Fuego_Quickstart_Guide hopefully has some good information you can use (and up-to-date screenshots= ). There are also some screenshots at: http://bird.org/fuego/Jenkins_User_Interface >=20 > I was able to submit some tests (Initial test section on the README file)= by > using the ftc tool and I saw some of them failing > (docker.default.Functional.LTP, docker.default.Functional.glib), but the > Jenkins interface doesn't let me do anything. I'm not very well versed on > Jenkins and Docker setups so I might be doing something wrong. What > should I check? Actually, docker is a special target for playing around with Fuego by running tests on the Fuego docker container. We actually discovered a few real bugs in docker with Fuego, that we haven't been able to fix yet. So the LTP bug you saw is real. There are currently 3 errors from LTP that should be passing but are currently failing with the "docker" board, that are under investigation now. We may just blacklist those particular sub-tests, so that they don't cause undue anxiety running Fuego.= =20 In the future, our plan is to have an 'ftc' sub-command you can run to save your current results (including test failures that you feel you can ignore)= , as the new "reference" results that future tests on your hardware will compare against to determine success or failure. There was a tool for this in the past, but it was an awk script, and had some usability problems, and we have just recently refactored a lot of our results parsing in conjunctio= n with a unified results output and processing, across all tests (both Benchmark and Functional). Unfortunately, this area of Fuego is undergoing some churn at the moment. You can detect errors with Fuego, using the Jenkins interface, and you can detect functional test regressions and benchmark results (performance) regressions. But you can't use the Jenkins interface to blacklist the failing tests (indicate that you don't want to run them, or indicate that you want to ignore those results). And actually investigating the cause of an error is still left as an exercise to the user (although our links to the test logs should help). We are still investigating the docker bug with the Functional.glib test, to see what the actual problem is. I hope to have this resolved and/or explained in Fuego version 1.2. To be honest, the 1.1 release was a bit rushed, and we probably should have removed Functional.glib from the testplans for the docker board, to avoid these issues (and then re-enabled it in the testplans when the issue was resolved.) You bring up an important point - which is that we should probably have some release notes that explain known problems, especially with one of our "demo" boards (the 'docker' pseudo-board). One thing we are striving for is a good out-of-the box experience, and it looks like we've failed wit= h the 1.1 release. >=20 > Then I saw no reference about what in LAVA is called multinode tests (jus= t > for someone reading this that doesn't know LAVA: a test using more than > one board involved to be able to test e.g. communication hardware). If th= is is > achievable with Fuego, how would be do? In a way, every test in Fuego is a multi-node test, with the host being one end of the test. The way Fuego is structured, the base script for the test runs on the host, inside the docker container, and communicates with the board. For many tests, the base script just: 1) builds the software, 2) deploys it to target, 3) executes it on the board, 4) collects the resul= ts and 5) analyzes it. However, we do have a few tests that perform other actions on the host, or rely on processes running on the host. For example, the Functional.netperf test relies on a netperf server running= on the host (an instance is started in the container if one is not already run= ning). A test can execute whatever steps it wants to, including communicating with nodes, machines, or test equipment that is not on the target board. These steps would be put into the base script's test_pre_check() or test_ru= n() functions. test_pre_check is usually reserved to checking that test pre-re= quisites are met, which in the case of a multi-node test might include reserving off= -target testing hardware (such as video scanners), or it might include setting up the connections for things like bus testing (e.g. establishing the connecti= ons to the device under test for CAN bus testing or USB testing). Then the test_run() function should do things like initiating the actual sa= ving of results, or starting and stopping communication from off-board endpoints= . See http://bird.org/fuego/Adding_a_test for more information about the functions in a test base script. One concept in Fuego related to this, that I think is important, is that we intend the contents of the Fuego docker container to be a "test distribution", or in other words, a distribution of Linux, based on Debian, that is specifically geared for testing. Not only does Fuego come with actual test programs, and the test packages themselves include source (if needed) to build the tests for the target boards. But also, the Fuego test distribution is intended to include host-side software as well for testing, as well as (potentially) the software needed to drive external devices and test equipment. It is a pain for the average developer to collect, install, and integrate this software onto their desktop machines, and that's one aspect of testing that we'd like to share between developers (test program setup, hardware and tools configuration, etc.) You can see the first steps in this direction by the inclusion in Fuego of the netperf server, and several utilities (like serio and ttc) to handle different board communication methods. We'll be expanding this in the years to come, so that more and more tests can be run with just the single docker installation of Fuego, requiring much less pre-test configuration and specialized expertise by testers. I hope this answers your questions. Thanks again for experimenting with Fuego and providing feedback. -- Tim