Re: [Fuego] Board setup

[Michal Simek <monstr@monstr.eu>]

[-- Attachment #1.1: Type: text/plain, Size: 13527 bytes --]

On 12.4.2017 20:01, Bird, Timothy wrote:
> 
> 
>> -----Original Message-----
>> From: Michal Simek on Wednesday, April 12, 2017 2:22 AM
>> On 11.4.2017 20:25, Bird, Timothy wrote:
>>>> -----Original Message-----
>>>> From: Michal Simek on Tuesday, April 11, 2017 2:18 AM
>>>> Hi,
>>>>
>>>> I have started to play with fuego with one arm64 board I have here.
>>>> I have a question about handling board itself. I have multiple
>>>> boards(home boardfarm) connected to my PC and every board needs to
>> get
>>>> certain configuration before starts. At least power on.
>>>> And when test is done power off too.
>>>> How is this handled? I didn't find any hint how to do it.
>>>
>>> Well, there *are* some hooks for things like this, but they're a bit kludgy,
>> and
>>> indeed, not document hardly at all.  We are planning on improving this in
>> the
>>> next release (1.2) with some API changes to make board setup and
>> teardown
>>> more straightforward and orthogonal.
>>
>> What's the schedule for 1.2? I see you have tagged 1.1.1 some hours ago.
> 
> The hope is to get 1.2 done by the end of May - in time for Open Souce Summit Japan
> (and the Automotive Linux Summit).  However, I'm taking bugfixes for 1.1 in the meantime,
> and I've already started a 'next' branch for the 1.2 release.  If features that fix your
> problem make it into my 'next' branch, you can certainly use that prior to the actual 1.2
> release.  For example, I already put in the line about TARGET_TEARDOWN_LINK into
> my next branch (as a placeholder for the more complete solution.)

ok. Just keep in your mind one thing. I am coming from fpga space. It
means one thing is board itself and the second is configuration in
programmable logic. It means one board can have unlimited number of
configurations. And different bitstream should be loaded for that
configuration.
This can be handled by setting up different boards and maybe different
testplans.


>>>
>>> What we have now is the following:
>>> During the pre_test phase, if the board has defined a function that does
>>> board setup, and you set the name of that function in the variable
>>> TARGET_SETUP_LINK, it will be executed, before any communication is
>>> attempted with the board.  So you can define this function and set this
>>> variable in your board file, and use it to power on your board, and/or
>>> set up communication channels for the board.
>>>
>>> The only documentation reference I could find to this was at:
>>> http://bird.org/fuego/function_pre_test
>>> and it's pretty cursory.
>>>
>>> During the post_test phase, there is no similar board-specific function
>>> called.  We are planning on adding one in the 1.2 release.  You can add
>>> one yourself in the meantime.  To do this, in fuego-
>> core/engine/scripts/functions.sh
>>> in the function post_test,  before the end of the function, but after all
>> communication
>>> with the board is complete (logs are downloaded, etc.)  add the following
>>> line:
>>>
>>> [ -n "$TARGET_TEARDOWN_LINK" ] && $TARGET_TEARDOWN_LINK
>>>
>>> I would put this right before the call to "make_run_file".
>>>
>>> Then you can define a function, and in your board file set the
>>> variable TARGET_TEARDOWN_LINK to the name of that function,
>>> and it will get called prior to test exit.
>>>
>>> This system is crude, and would end up power-cycling the board
>>> on every test.  We will be working out a more elegant solution in
>>> the future, but this should get things working for you for now.
>>
>> This is working fine. Thanks for pointers. In bigger boardfarm it will
>> take take time to ask for a board and get an access to it.
>>
>>
>>>>
>>>>
>>>> Also default docker is not supporting /dev/serial/* which is elegant way
>>>> how to handle connecting to multiple boards connected to one PC.
>>>>
>>>> Because what you need to really do is to connect to board via this link
>>>> SERIAL="/dev/serial/by-path/pci-0000:00:1d.0-usb-0:1.7.1.2.4.4:1.0-port0"
>>>> instead of just
>>>> SERIAL="/dev/ttyUSB10"
>>>> which is changing all the time.
>>>>
>>>> Do you know what to do?
>>> We have a separate "docker-create-container" script for handling ports
>>> that docker needs permission to, and that can change dynamically
>>> on the host at runtime.  It is called
>>> docker-create-usb-privileged-container.sh, and it's in the fuego/fuego-
>> host-scripts
>>> directory.  It makes certain device nodes on the host available inside the
>> target.
>>>
>>> I don't have experience with /dev/serial/by-path, but what this script
>>> does is volume-mount certain directories from the host into the container
>> filesystem.
>>> Maybe it would work to add a line like the following to this script:
>>>
>>>    -v /dev/serial/by-path:/dev/serial/by-path \
>>>
>>> and use that script to create the container.
>>
>> This is working but the problem is with sercp which reports
>> ERROR: Port must be configured before it can be used.
>>
>> sersh is fine with this serial setup
>>
>> SERIAL="/dev/serial/by-path/pci-0000:00:1d.0-usb-0:1.7.1.2.4.4:1.0-port0"
>>
>> Both are pointing to the same file that's why I expect that issue is there.
>>
>> In serio.git there is
>> 9daaf498e0fc4a086fd50e32c2edd3d7410d47ae
>>
>> You have this in commit message.
>>     You can _not_ use /dev/ttyACM0 for host1 and host2 because the '/'
>>     before a ':' is how scp determines that you are trying to copy
>>     a file with a ':' in the name instead of specifying a host name.
>>
>> Frank: Can you please comment?
> 
> I took a look at this, and indeed serio (the base command for both sersh and sercp) parses
> the hostname such that it can't have an embedded slash for the sersh command.
> (see lines 674 to 678).  I'm not sure how Frank would like to handle this.  It would be relatively
> simple to just change the conditional here to allow slashes in the host, but disallow colons in
> the file paths, but I'm pretty sure Frank won't go for this as a long-term solution.  
> 
> The path you have to the serial port has both slashes AND colons, making it especially problematical.
> Maybe it would be better to support an actual --host argument that supercedes anything parsed
> from the source or destination string, and you'd need to have some other indicator for which argument
> specifies the local vs. the remote side of the copy.  (Maybe use --srchost vs. --desthost, and allow only
> one of them to be used.)
> 
> Frank - any ideas?

There is probably an option to select appropriate ttyUSBx by checking
symlinks based on path but this can stop to work when you have flaky
usb-serial cable.

[by-path]$ pwd
/dev/serial/by-path
[by-path]$ ls -la
total 0
drwxr-xr-x 2 root root 140 dub 12 22:17 .
drwxr-xr-x 4 root root  80 dub 12 22:17 ..
lrwxrwxrwx 1 root root  13 dub 12 22:17
pci-0000:00:1d.0-usb-0:1.7.1.2.3:1.0-port0 -> ../../ttyUSB0
lrwxrwxrwx 1 root root  13 dub 12 22:17
pci-0000:00:1d.0-usb-0:1.7.1.2.3:1.1-port0 -> ../../ttyUSB1
lrwxrwxrwx 1 root root  13 dub 12 22:17
pci-0000:00:1d.0-usb-0:1.7.1.2.3:1.2-port0 -> ../../ttyUSB2
lrwxrwxrwx 1 root root  13 dub 12 22:17
pci-0000:00:1d.0-usb-0:1.7.1.2.3:1.3-port0 -> ../../ttyUSB3
lrwxrwxrwx 1 root root  13 dub 12 22:17
pci-0000:00:1d.0-usb-0:1.7.1.2.4.4:1.0-port0 -> ../../ttyUSB4
[by-path]$


> 
>>
>>>> I am playing with serial setup and can you please tell me how tests are
>>>> moved from container to target?
>>>> Is there any tree structure expected on the target which should be
>> followed?
>>>
>>> Each test moves test material to the target in its test_deploy function.
>>> We strongly suggest that all tests put their materials in the following
>>> location:
>>> $BOARD_TESTDIR/fuego.$TESTDIR
>>> Fuego removes this directory when the test is complete.
>>>
>>> BOARD_TESTDIR is defined in the board file, and can be anywhere
>>> the user would like to configure it. Common places are "/test" and
>>> "/home/test".  I use "/home/a" just to keep the directory name shorter.
>>> $TESTDIR corresponds to the full name of the test: e.g.
>> "Functional.hello_world"
>>>
>>> http://bird.org/fuego/function_test_deploy for information about
>>> the test_deploy function.
>>
>> I found that stuff a little bit later that serio is that magic serial
>> tool which is doing sersh or sercp and copy that stuff over serial line.
>>
>> It will be worth to separate load channel from command channel. The
>> reason is simple suspend tests which I am playing with.
>> It is not a problem to move files to target via ethernet/wifi but if you
>> want to test wakeup from uart you need to have this channel setup.
> 
> Can you tell me more about the tests you are doing?  It sounds like
> you are testing suspend/resume and wakeup from uart.  It will be
> difficult to use the uart for both the transport mechanism and also
> as part of the test.

I was playing with it and it is not that bad but far from ideal solution.
What I have done is simple script which is
#!/bin/sh

sleep XX
<commands>

and you will copy it over serial and run it as background task.
Because it is on background you get shell back which looks like it is
requirement for sersh to finish.

Then next task is to send whatever you need via uart - not sure how to
do it properly. Just let one command over serial to fail and then next
command can check dmesg.

> If you have network, why are you using the serial port for any Fuego activity?

If this is supposed to be framework for testing embedded linux stuff
than you need to test serial itself too. ltp-ddt has for example uart
loopback test. This can be internal to IP, it can be external to IP when
you connect rx/tx in programmable logic and can be also use as in this
case as wake up source from suspend.
You also need to have testcases where you want to test different
baudrates and doing this just via loopback is not enough because it is
not able to detect issues related to incorrect input clock setting and
baudrate where both side won't understand each other.

For suspend you can use different wakeup sources like rtc or gpio which
also should be tested.


> Do you have sshd on the target?

yep sshd is there but this is just one channel. Also not sure if this is
the best channel for sending command if you want to test different
ethernet setting - full/half duplex, 10/100/1000Mbit/s, jumbo frames, etc.
That's why sometimes having communication with board out of network is
necessary.

> 
> We have one test already that handles the target going away and coming back,
> which is our reboot test.  This is different than a suspend test, because we don't
> expect anything that was running on the target at the beginning of the test to be
> running after the reboot.  For a suspend, you could have a long-lived process on
> the target doing timings or checking things after the resume.

Not sure if this is covering all that cases but yes - it is something to
look into. Also drivers have options to have save and restore functions
which should be tested.


> However, the reboot test has a few peculiarities with dealing with the machine
> coming back to life (handling timeouts and retries), that might be common with your
> use case.

Yep, I was looking how reboot is implemented. There is definitely a need
to test that serial case because at least for me with 2 days fuego
experience it is failing. Via ssh it is working fine.

> In general Fuego as it stands today (in the 1.1 release) doesn't handle tests that completely
> wedge the machine.  It currently expects to be able to do a software reboot of the machine.
> Another item on our 1.2 list is to add an API for hardware reboot, for people who have
> power control over their board, so it can be called mid-test, if needed.  Is this something
> that you need?

Right now I see big gap in platform setup. I have one small board farm
at home and then working with big one remotely and this requires
complete layer to separate it.
Also testcases expect that system is up and running but I didn't see any
build script which build selected kernel and run that.

> Finally (sorry this is so long), we are investigating splitting the deployment, so that a board
> can use client-initiated operations, rather than host initiated operations to support some
> tricky network configurations.

From my point of view would be good to have an option. Build images
against whatever Linux branch I have with whatever rootfs I want (cpio,
sd, usb, sata, emmc, etc). Copy this to the target. Boot it and check
errors in bootlog.
And then start to execute tests. You have a lot of generic tests but a
lot of other tests needs to be added. For example module testing. When
you need to run the same tests when driver is compiled in the kernel or
it is like a module. Reload modules several times and repeat tests.

And also for reference boards you should have an option to test
different boot modes because bootloader can setup problematic stuff
which breaks Linux. But I expect this can covered by different board setup.

Thanks,
Michal

-- 
Michal Simek, Ing. (M.Eng), OpenPGP -> KeyID: FE3D1F91
w: www.monstr.eu p: +42-0-721842854
Maintainer of Linux kernel - Xilinx Microblaze
Maintainer of Linux kernel - Xilinx Zynq ARM and ZynqMP ARM64 SoCs
U-Boot custodian - Xilinx Microblaze/Zynq/ZynqMP SoCs



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]