[Fuego] user experience of fserver

[Fuego] user experience of fserver

[Wang, Mingyu]

[-- Attachment #1: Type: text/plain, Size: 918 bytes --]

hi Tim,

I tried the fserver according to the user manual.
I was able to successfully use one local server to send requirements to another local server and get results.
It has not been verified whether the remote server can perform smoothly.

By using it, there are some inconveniences  as follows:
1. The directory of the log file cannot be customized
2. No information such as request and runs can be seen on the web interface (I don't know if I'm wrong)
3. The results are not intuitive, a lot of information is confusing

by WangMingyu

  --
Best Regards
---------------------------------------------------
Wang Mingyu
Development Dept.I
Nanjing Fujitsu Nanda Software Tech. Co., Ltd.(FNST) No. 6 Wenzhu Road, Nanjing, 210012, China
TEL: +86+25-86630566-8568
COINS: 79988548
FAX: +86+25-83317685
MAIL: wangmy@cn.fujitsu.com<mailto:wangmy@cn.fujitsu.com>
http://www.fujitsu.com/cn/fnst/




[-- Attachment #2: Type: text/html, Size: 6441 bytes --]

Re: [Fuego] user experience of fserver

[Bird, Tim]

> -----Original Message-----
> From: Fuego <fuego-bounces@lists.linuxfoundation.org> On Behalf Of Wang, Mingyu
> 
> hi Tim,
>  
> I tried the fserver according to the user manual.
> 
> I was able to successfully use one local server to send requirements to another local server and get results.
> 
> It has not been verified whether the remote server can perform smoothly.
> 
> 
> 
> By using it, there are some inconveniences  as follows:
> 
> 1. The directory of the log file cannot be customized
> 
> 2. No information such as request and runs can be seen on the web interface (I don't know if I'm wrong)
> 
> 3. The results are not intuitive, a lot of information is confusing
> 

Thank you very much for this feedback.  I feel a little bad because I know the
code is not production-grade yet, but I was happy to hear that at least some
things worked for you.  I had hoped we could get some utility out of the
feature, even though it is not complete (see below).

What did you mean: "The directory of the log file cannot be customized" ?
Are you talking about on the side that pulls the run information from the
server, or on the server itself?

Items 2 and 3 are correct.  The feature needs more work to be ready for 
full use as a Fuego feature.  I'm still not sure if placing the results into the
local Jenkins interface is the right thing to do or not.  Right now there is
no place for the results to land, because there is no local Jenkins node for
the remote board.  We could make a "proxy" node for a remote board, so
that results that were performed by a board in another
lab could be put into the interface for that proxy.  This would require that
I enhance the ability to put non-local data into the Jenkins build area, which
is something I probably need to do anyway.  This is something that will take
a bit of time.

However, let me describe how I'd like to use this feature in the short term,
to see if how I envision using it would work for you.

One problem I have as a developer of Fuego is that when I get a test
that does not apply to any of my boards I have no way to "test the test".
I would like to execute it on a board to which is does apply, and examine
how it works, and what issues it might have being generalized for other boards.

To do this, I would like to be able to run a test on a board in another lab.
Specifically, if Fujitsu submits a test that runs on one of their boards or
distributions of Linux, I would like to be able to execute the test remotely,
and get the results back and examine them.

I am thinking the flow would be like this:
1) Fujitsu submits a new test to Fuego, to the Fuego mailing list
1a) Fujitsu provides the name of a lab and board that is available for executing the test
1b) Fujitsu would execute poll_requests.sh, to automatically poll for jobs for that board.
2) I review the test, and if I have issues or cannot test it, I execute the test on their board
This means that I will:
2a) ftc put-test the test to the public fserver (on fuegotest.org)
2b) Fujitsu's lab's poll_requests.sh would execute the test and put the results on the
public fserver.
2c) I would collect the results and use it to validate the test
3) I may modify the test to experiment with different alternatives for executing it
For example I may add extra debugging steps, or I may change the flow of execution.
After this, I would repeat step 2 until I was satisfied, or until I completed the review
and returned feedback to Fujitsu about the test.

The biggest problem I see with this flow, is that in step 3 I can introduce any arbitrary
change to the test, including something that (inadvertently) wiped out the board, or
(if I were malicious) probed the Fujitsu network.  I can imagine that it would be difficult for
Fujitsu (or any company) to allow running a test it obtained from outside their network,
inside their network.

Alternatives to this flow, that give me the same benefit of being able to execute the
test on a Fujitsu board:
1) Have Fujitsu send me detailed execution logs (with FUEGO_DEBUG=255) for 
tests that are submitted, so I can see how they execute on your systems.
I request manual changes to the test, and someone at Fujitsu changes the
code, runs the code against their boards, and sends the results (this is just the
same thing we're doing now, but with debug test logs attached.)
2) Get a Fujitsu board (and distro?) and put it in my lab locally.  That way Fujitsu
doesn't have to run a test from outside their private network.
3) Do this process manually, instead of automatically.  I could still do the
put-request, but then a Fujitsu person would need to execute commands
to download the test, execute it, and put the results back on the fserver.
There could be a step where they manually checked that the test didn't
do anything weird or dangerous.
It wouldn't be real-time for me, which would delay my review of the test,
but it would still provide me more detailed information, and allow me to
test alternatives.

Let me know what you think.
 -- Tim

Re: [Fuego] user experience of fserver

[Wang, Mingyu]

Hi Tim,

>What did you mean: "The directory of the log file cannot be customized" ?
I can't reproduce this problem now. I'll report it later.

I executed the following command:
ftc run-request request-2020-02-17_05:01:32.91-fuego-wangmy:myboard --put-run

There are two problems:
1. The test set is finished after execution. There is no put-run step. Is there any problem with the operation？
2. In addition, when I use it, I find that this step will be performed after every run request:
Request request-2020-02-17_05:01:32.91-fuego-wangmy:myboard was removed from the server
Then the request is deleted. Is it necessary to execute?


After reading your ideas on how to use fserver, my ideas are as follows:
1.
>1) Fujitsu submits a new test to Fuego, to the Fuego mailing list
>1a) Fujitsu provides the name of a lab and board that is available for executing the test
>1b) Fujitsu would execute poll_requests.sh, to automatically poll for jobs for that board.

Because we only have one board for testing, and it is used for other purpose, we can’t use it only for fuego testing.

2.
>2) Get a Fujitsu board (and distro?) and put it in my lab locally. That way Fujitsu doesn't have to run a test from outside their private network.
You have to prepare the board by yourself as we only got one currently.
Currently, we can directly provide you our latest binary for testing.

We have mentioned the idea of open source distro before. 
If it becomes open source code, you can build it by yourself.
We can provide detailed usage.
And I need to remind you that building our distro requires more than 100G.

I'll confirm if we can offer you another board to test Fuego.
And we will continue to discuss the plan. If there are any new ideas, we will let you know in time.

by Wangmy


-----Original Message-----
From: Bird, Tim [mailto:Tim.Bird@sony.com] 
Sent: Thursday, February 13, 2020 8:48 PM
To: Wang, Mingyu/王 鸣瑜 <wangmy@cn.fujitsu.com>; fuego@lists.linuxfoundation.org
Subject: RE: user experience of fserver



> -----Original Message-----
> From: Fuego <fuego-bounces@lists.linuxfoundation.org> On Behalf Of 
> Wang, Mingyu
> 
> hi Tim,
>  
> I tried the fserver according to the user manual.
> 
> I was able to successfully use one local server to send requirements to another local server and get results.
> 
> It has not been verified whether the remote server can perform smoothly.
> 
> 
> 
> By using it, there are some inconveniences  as follows:
> 
> 1. The directory of the log file cannot be customized
> 
> 2. No information such as request and runs can be seen on the web 
> interface (I don't know if I'm wrong)
> 
> 3. The results are not intuitive, a lot of information is confusing
> 

Thank you very much for this feedback.  I feel a little bad because I know the code is not production-grade yet, but I was happy to hear that at least some things worked for you.  I had hoped we could get some utility out of the feature, even though it is not complete (see below).

What did you mean: "The directory of the log file cannot be customized" ?
Are you talking about on the side that pulls the run information from the server, or on the server itself?

Items 2 and 3 are correct.  The feature needs more work to be ready for full use as a Fuego feature.  I'm still not sure if placing the results into the local Jenkins interface is the right thing to do or not.  Right now there is no place for the results to land, because there is no local Jenkins node for the remote board.  We could make a "proxy" node for a remote board, so that results that were performed by a board in another lab could be put into the interface for that proxy.  This would require that I enhance the ability to put non-local data into the Jenkins build area, which is something I probably need to do anyway.  This is something that will take a bit of time.

However, let me describe how I'd like to use this feature in the short term, to see if how I envision using it would work for you.

One problem I have as a developer of Fuego is that when I get a test that does not apply to any of my boards I have no way to "test the test".
I would like to execute it on a board to which is does apply, and examine how it works, and what issues it might have being generalized for other boards.

To do this, I would like to be able to run a test on a board in another lab.
Specifically, if Fujitsu submits a test that runs on one of their boards or distributions of Linux, I would like to be able to execute the test remotely, and get the results back and examine them.

I am thinking the flow would be like this:
1) Fujitsu submits a new test to Fuego, to the Fuego mailing list
1a) Fujitsu provides the name of a lab and board that is available for executing the test
1b) Fujitsu would execute poll_requests.sh, to automatically poll for jobs for that board.
2) I review the test, and if I have issues or cannot test it, I execute the test on their board This means that I will:
2a) ftc put-test the test to the public fserver (on fuegotest.org)
2b) Fujitsu's lab's poll_requests.sh would execute the test and put the results on the public fserver.
2c) I would collect the results and use it to validate the test
3) I may modify the test to experiment with different alternatives for executing it For example I may add extra debugging steps, or I may change the flow of execution.
After this, I would repeat step 2 until I was satisfied, or until I completed the review and returned feedback to Fujitsu about the test.

The biggest problem I see with this flow, is that in step 3 I can introduce any arbitrary change to the test, including something that (inadvertently) wiped out the board, or (if I were malicious) probed the Fujitsu network.  I can imagine that it would be difficult for Fujitsu (or any company) to allow running a test it obtained from outside their network, inside their network.

Alternatives to this flow, that give me the same benefit of being able to execute the test on a Fujitsu board:
1) Have Fujitsu send me detailed execution logs (with FUEGO_DEBUG=255) for tests that are submitted, so I can see how they execute on your systems.
I request manual changes to the test, and someone at Fujitsu changes the code, runs the code against their boards, and sends the results (this is just the same thing we're doing now, but with debug test logs attached.)
2) Get a Fujitsu board (and distro?) and put it in my lab locally.  That way Fujitsu doesn't have to run a test from outside their private network.
3) Do this process manually, instead of automatically.  I could still do the put-request, but then a Fujitsu person would need to execute commands to download the test, execute it, and put the results back on the fserver.
There could be a step where they manually checked that the test didn't do anything weird or dangerous.
It wouldn't be real-time for me, which would delay my review of the test, but it would still provide me more detailed information, and allow me to test alternatives.

Let me know what you think.
 -- Tim