From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Sony.onmicrosoft.com; s=selector1-Sony-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=gDFuD+0kVQeL+aqUSNr79kFKKKm2jI6n/uJUAAiJ14E=; b=CQh1TwPU6WoCKlkaRjJnQPDLCp9u8i5k805riObWHqPv2N2XbpuFdztKxTiu1xhHHx0cZau4zDZwiqX2H965LwawnYkyE8AfmsWO1P4RJdS/dnUHWfGwUSJF1kRYYai+2K1tlQTEYkh+IwWHEHu2TAc0Vmgv/RjxO+eDri9D+DY= From: "Bird, Timothy" Date: Wed, 10 May 2017 19:30:21 +0000 Message-ID: References: , In-Reply-To: Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [Fuego] First steps with fuego List-Id: Mailing list for the Fuego test framework List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Rafael Gago Castano , "fuego@lists.linuxfoundation.org" > -----Original Message----- > From: Rafael Gago on Wednesday, May 10, 2017 6:12 AM > I got some time today to keep investigating today, it was trivial to inte= grate > with one of our boards, so the Wiki it's up to date and good. It was a ve= ry > easy and trouble free setup. Thanks. That's good to hear. =20 > Then I wanted as an exercise to write test a for rs485 (the same case > happens with CAN) but I have been looking at "fuego-core/engine/tests" > and I couldn't answer the questions myself. >=20 > 1.) Our current setup does device poweon/off the device (custom > commands in LAVA), fetches a kernel, dtb and rootfs from our build server= s > and copies the images to the DUT's RAM for RAM booting (tftp + Uboot > commands integrated in LAVA). How does these steps are best integrated in > fuego? >=20 > LAVA has this feature integrated, but we already have an internal Python > program that handles this, so we don't need fuego to talk with UBoot, jus= t to > know the best place to launch the tool. Spontaneusly and with what I know= I > only can think about doing it on the "test_pre_check" function or as the = first > (dummy) test of a DUT specific tesplan, but both would have its drawbacks= . It depends on what the actual test is. If you are trying to determine if s= omething is going wrong with those steps themselves, then it would be appropriate to put them in the "test_run()" function. If these are just pre-cursors to= prepare for the actual testing, and the "actual' test is of some other functionalit= y, then they should go in "test_pre_check()", probably with something to validate t= hat they succeeded before proceeding with the actual test. If this is for setup for a sequence of tests, then I think making a dummy t= est, as you indicated, is the right thing. You could add it as the first step o= f a Fuego testplan, or you could make your own Jenkins job to handle the job sequence= . There are two Jenkins plugins which can handle this type of thing: https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Trigger+Plugin https://wiki.jenkins-ci.org/display/JENKINS/Join+Plugin Both of these allow for jobs to be chained together, as precursors or succe= ssors to other jobs. I'm am not a LAVA expert, but my understanding is that LAVA always does a r= eboot between tests. (I may be confusing this with KernelCI) Fuego does not hav= e this model by default, but you can construct Jenkins jobs to perform this type o= f system-level sequence of 'build, deploy and reboot' if desired.=20 This presentation might be helpful: Continuous Integration and Autotest Environment Using Fuego - by Kenji Tada= no & Kengo Ibe (Mitsubishi Electric), Oct 2016 at ELCE 2016 You can find the links for the PDF and video at: http://bird.org/fuego/Presentations#2016 If lots of people do testing this way, we should probably improve our suppo= rt for it, or come up with guidelines for a good way to do it. > 2.) Let's say that we have the desired image running on the device and we > want to test RS485 TX/RX at different baudrates. Now with LAVA we boot, > wait until both machines are booted (using the "lava-send" and "lava-wait= " > synchronization primitives) set the baudrate in both sides, start the rec= eiver > side (either host or DUT), synchronize with the sender side (either host = or > DUT) , start sending and then we start the cycle again with another baudr= ate > - TX/RX cfg. We do the same type of test for CAN too (using cansequence). Hmmm. The 'ftc' command does have a 'wait-for' sub-command for this type of thing. However, I don't know of any tests that use this, and I don't k= now how it compares to the lava-send/lava-wait protocols. When I've used it in the past I've done simple things like check for the existence of a file, and th= e "signaling" side has done something like "touch /tmp/target_ready" It currently only checks conditions on the host. This is a deficiency. It= should also check for conditions on the target. Something could be jury-rigged, but it= would require some multi-processing in the test_run function. > I haven't figured out how I would implement such test with fuego. Both > sender and receiver would need to be able to signal a failure, but there = is > only a "run_test" function and it's running on the DUT. Is there any way = to > implement this or any sane workaround? test_run() is running on the host, and can start and stop both sides. But it is not threaded. It would need to start separate processes on the sending side and the receiving side of the test, and keep them in sync. If one side is the host, then 'ftc wait-for' could be used for this. You could have a sequence in test_run that looked like the code below. I'm just brainstorming this, and haven't tried any of this, but here's some code: This assumes that the sender on the host can detect a failure or will time out, and will do "touch /tmp/sender_done" when it is finished. # start the test log on the board report "echo Starting baud_rate test" for baud_rate in $FUNCTIONAL_RS485_RXTX_BAUD_RATE_LIST ; do=20 # start a listener on the target, and append it's stdout to the test log report_append "start_listener $baud_rate" # start a sender on the host, and collect it's stdout locally (on the ho= st) $TEST_HOME/start_sender $baud_rate >/tmp/sender_log # wait (up to 100 seconds) for process on host to signal completion ftc wait-for -t 100 "test -f /tmp/sender_done" # kill sender on host (should check ftc exit code to see if this is need= ed) pkill sender # terminate listener on board, if it hasn't already exited kill_procs listener # append the log from the host to the board's test log put /tmp/sender_log /tmp report_append "cat /tmp/sender_log" done BAUD_RATE_LIST would be in the spec file for the test (named here as Functional.RS485_rxtx). start_sender is a made-up program that runs on the host to=20 start one side of the connection. start_listener is a made-up program that runs on the target board to start the other side of the connection. This code assumes it starts a process called 'listener' on the board, and this code kills that listener when the test is over, if it is still running. start_listener is deployed to the tar= get in test_deploy, and start_sender is executed directly from the test's home directory (e.g. fuego-core/engine/tests/Functional.RS485_rx= tx) OK - just going through this mental exercise has shown some deficiencies in Fuego's host/target logging model and synchronization that we should correct. There should definitely be a method of adding=20 something from the host to the log, in one step. And we should extend ftc to support checking the condition of something on target. I'm not sure if the logging shown here will work or not. start_listener needs to return immediately, and leave the 'listener' process executing on the board, still outputting to the log. If this doesn't work by default= , another layer of output capture and appending to the log could be done, but that's more awkward. The function "report()" starts the log on the board, and "report_append()" adds more material to it. Since we also want information from the host (or from a 3rd machine), we have to collect that ourselves and add it to the log directly. The tmp file (sender_log) on the host should definitely be put into a more unique location (test specific, and with a unique temp filename), to avoid collisions between tests running on multiple boards. But this shows proof of concept for how this can be done. >=20 > 3.) This one is very easy to add/contribute, but some inbuilt support to > specify inside a test a git + branch + commit sha instead of a tarball wo= uld be > nice to avoid duplication of tests for different library versions (e.g. w= hen > testing previous image releases that get backports). That's a good use case to support. Actually, we're adding support for getting source for a test from git in Fu= ego v1.2. Some preliminary support is in the 'next' branch, but it still needs= a=20 bit of work (It doesn't support specifying the commit ID yet). See the function "unpack" in fuego-core/engine/scripts/functions.sh. =20 -- Tim