From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Sony.onmicrosoft.com; s=selector2-Sony-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2brJwoXIDrkWD3R9p3AwDh+ky0IaFIIzLklJycU9IOg=; b=MR9Z5AjdZKLYL3+/vFE9ANO/HdzSnAWnrSqnvBaPwF7CdXGq9Nq6OunoIj9hTSHNMDqnTUaOGybBKuyQDzrQhPL49sJYuF8VS8k/TcQP2wb3B0JGJia9QY4I8+HPQVsVPy5y8M9IctXBaCVDYl0xKnj6AjWuwHvRlkwz+4t0XLU= From: "Bird, Tim" Date: Thu, 13 Feb 2020 12:47:57 +0000 Message-ID: References: In-Reply-To: Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [Fuego] user experience of fserver List-Id: Mailing list for the Fuego test framework List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Wang, Mingyu" , "fuego@lists.linuxfoundation.org" > -----Original Message----- > From: Fuego On Behalf Of Wang, = Mingyu >=20 > hi Tim, > =20 > I tried the fserver according to the user manual. >=20 > I was able to successfully use one local server to send requirements to a= nother local server and get results. >=20 > It has not been verified whether the remote server can perform smoothly. >=20 >=20 >=20 > By using it, there are some inconveniences as follows: >=20 > 1. The directory of the log file cannot be customized >=20 > 2. No information such as request and runs can be seen on the web interfa= ce (I don't know if I'm wrong) >=20 > 3. The results are not intuitive, a lot of information is confusing >=20 Thank you very much for this feedback. I feel a little bad because I know = the code is not production-grade yet, but I was happy to hear that at least som= e things worked for you. I had hoped we could get some utility out of the feature, even though it is not complete (see below). What did you mean: "The directory of the log file cannot be customized" ? Are you talking about on the side that pulls the run information from the server, or on the server itself? Items 2 and 3 are correct. The feature needs more work to be ready for=20 full use as a Fuego feature. I'm still not sure if placing the results int= o the local Jenkins interface is the right thing to do or not. Right now there i= s no place for the results to land, because there is no local Jenkins node fo= r the remote board. We could make a "proxy" node for a remote board, so that results that were performed by a board in another lab could be put into the interface for that proxy. This would require tha= t I enhance the ability to put non-local data into the Jenkins build area, wh= ich is something I probably need to do anyway. This is something that will tak= e a bit of time. However, let me describe how I'd like to use this feature in the short term= , to see if how I envision using it would work for you. One problem I have as a developer of Fuego is that when I get a test that does not apply to any of my boards I have no way to "test the test". I would like to execute it on a board to which is does apply, and examine how it works, and what issues it might have being generalized for other boa= rds. To do this, I would like to be able to run a test on a board in another lab= . Specifically, if Fujitsu submits a test that runs on one of their boards or distributions of Linux, I would like to be able to execute the test remotel= y, and get the results back and examine them. I am thinking the flow would be like this: 1) Fujitsu submits a new test to Fuego, to the Fuego mailing list 1a) Fujitsu provides the name of a lab and board that is available for exec= uting the test 1b) Fujitsu would execute poll_requests.sh, to automatically poll for jobs = for that board. 2) I review the test, and if I have issues or cannot test it, I execute the= test on their board This means that I will: 2a) ftc put-test the test to the public fserver (on fuegotest.org) 2b) Fujitsu's lab's poll_requests.sh would execute the test and put the res= ults on the public fserver. 2c) I would collect the results and use it to validate the test 3) I may modify the test to experiment with different alternatives for exec= uting it For example I may add extra debugging steps, or I may change the flow of ex= ecution. After this, I would repeat step 2 until I was satisfied, or until I complet= ed the review and returned feedback to Fujitsu about the test. The biggest problem I see with this flow, is that in step 3 I can introduce= any arbitrary change to the test, including something that (inadvertently) wiped out the = board, or (if I were malicious) probed the Fujitsu network. I can imagine that it wo= uld be difficult for Fujitsu (or any company) to allow running a test it obtained from outside t= heir network, inside their network. Alternatives to this flow, that give me the same benefit of being able to e= xecute the test on a Fujitsu board: 1) Have Fujitsu send me detailed execution logs (with FUEGO_DEBUG=3D255) fo= r=20 tests that are submitted, so I can see how they execute on your systems. I request manual changes to the test, and someone at Fujitsu changes the code, runs the code against their boards, and sends the results (this is ju= st the same thing we're doing now, but with debug test logs attached.) 2) Get a Fujitsu board (and distro?) and put it in my lab locally. That wa= y Fujitsu doesn't have to run a test from outside their private network. 3) Do this process manually, instead of automatically. I could still do th= e put-request, but then a Fujitsu person would need to execute commands to download the test, execute it, and put the results back on the fserver. There could be a step where they manually checked that the test didn't do anything weird or dangerous. It wouldn't be real-time for me, which would delay my review of the test, but it would still provide me more detailed information, and allow me to test alternatives. Let me know what you think. -- Tim