Re: [Fuego] Please switch server to fuegotest.org (was RE: poll request scheduling)

[Pooja Sanjay More <pooja.sm@pathpartnertech.com>]

[-- Attachment #1.1.1: Type: text/plain, Size: 14204 bytes --]

Hi Tim,

We checked with the IT team on the result upload issue and it was indeed a
firewall
configuration which was blocking it. This issue has been resolved now and
we have
successfully verified the result object updates on the server.

Logs for a test :

root@cpu-366U:~# ftc put-request pptlab:rpi Functional.hello_world
Request was accepted by the server.
request_id=request-2020-12-24_12:39:23.93-pptlab:rpi

root@cpu-366U:~# ftc run-request --put-run
request-2020-12-24_12:39:23.93-pptlab:rpi
Trying to get request 'request-2020-12-24_12:39:23.93-pptlab:rpi' from
server
Executing test Functional.hello_world on board rpi (using spec default)
Running test 'Functional.hello_world' on board 'rpi' using spec 'default'
===== doing fuego phase: pre_test =====
===== doing fuego phase: build =====
The test is already built
Fuego test_build duration=0 seconds
===== doing fuego phase: deploy =====
===== doing fuego phase: snapshot =====
Firmware revision: 5.4.83-v7-52+
===== doing fuego phase: run =====
-------------------------------------------------
Hello World!

This is a test program, to demonstrate test success and failure
Test result: SUCCESS
-------------------------------------------------
===== doing fuego phase: post_test =====
===== doing fuego phase: processing =====
log_compare: pattern 'SUCCESS' found 1 times (expected greater or equal
than 1)
chart config not found. Using default values.
Fuego: requested test phases complete!
Packaging run 'Functional.hello_world-default-130-rpi'
run/
run/machine-snapshot.txt
run/syslog.before.txt
run/devlog.txt
run/testlog.txt
run/run.json
run/syslog.after.txt
run/prolog.sh
run/spec.json
run/build.xml
run/consolelog.txt
Run packaged successfully, and is at:
/tmp/run-Functional.hello_world-default-130-on-pptlab:rpi.frp
Run package run-Functional.hello_world-default-130-on-pptlab:rpi.frp was
accepted by the server.
Run Functional.hello_world-default-130-rpi put to server

Fserver Output:[image: image.png]
[image: image.png]

We ran multiple tests and found that when there are tests which do not have
test.yaml files
give an error for  "eprint".

[image: image.png]

eprint() was not defined in ftc. We added it to ftc then it is working fine.

Changes in ftc:

diff --git a/scripts/ftc b/scripts/ftc
index ae9ab2f..e70847d 100755
--- a/scripts/ftc
+++ b/scripts/ftc
@@ -616,6 +616,9 @@ def wprint(msg):
     global ll_warning
     if ll_warning:
         print("### WARNING: " + msg)
+# always print errors
+def eprint(msg):
+    print("!!! ERROR: " + msg)

 class config_class:
     def __init__(self, config_path):

Patch file is attached below.

*Thanks and regards:*
* Pooja More*





On Wed, Dec 23, 2020 at 5:02 AM <Tim.Bird@sony.com> wrote:

> Pooja,
>
> Thanks for the updates.
>
> I spent quite a bit of time trying to debug this.  From what I can tell,
> this is a problem on the client side.  The CGI script (fserver.py) calls
> the python module 'cgi' to parse the form data for the POST operation
> which uploads the run data to the server.  Inside the cgi module,
> in the cgi.FieldStorage() class initializer, the code reads from the
> CGI program's standard input.  In some cases, the standard input
> appears truncated, and the code cannot read the full data in order
> to create the file.  When this happens, the code returns the
> a message about not being able to find the 'file1' attribute in
> order to process the file.  In other cases, the standard input appears
> to be completely empty.  This causes the cgi.FieldStorage() to block
> internally on a read request. Apparently in this case the web
> server eventually times out the request, and the client receives
> no result data, which ends up showing the message:
>
> Server returned message:
>
> (with no message).
>
> I did notice that the 'requests' module used in your setup is different
> from the
> one I'm using.  The client string for mine is: "python-requests/2.12.4",
> whereas
> the client string for your submissions is: "python-requests/2.25.1". I
> don't think
> this should make a difference, but if you could try installing the
> specific 2.12.4
> version of the 'requests' python module, it would help us eliminate that
> difference
> between the two setups.  My other theory is that there's a firewall that's
> messing
> up the data (somehow truncating the http request body in the POST message)
> between the client and server.  I don't have
> the capability to sniff the network traffic at the server.  Can you try
> to run wireshark or some other packet tracer, and see if the data being
> sent
> by ftc to the network looks correct?  The http request should include the
> data for
> the packaged run file, in a multipart section in the body of the request.
>
> Finally - I saw one other thing that was quite strange.  When the server
> CGI
> script hangs (blocks reading the request data), I would have expected that
> 'ftc'
> would wait for the response before proceeding.  But I often see the next
> operation from ftc (after the put_run operation), which is an
> update_request,
> come to the server within less than a second.  I'm not sure why this is,
> and
> I have no explanation for it.  The calls to the 'request' module by ftc
> should
> be completely synchronous - so I should not be seeing the update_request
> operation at the server until the client times out on the previous
> 'put_run'
> request.
>
> Anyway - I've run out of time to work on this for now, and I don't really
> have
> a solution.  Based on what I've seen so far, I don't think this is a
> server-side issue.
> Please check on your side and see if the version of the request module,
> something to do with the firewall, or examining the network packet data,
> can
> provide any additional insights into the problem.
>
>  -- Tim
>
>
> > -----Original Message-----
> > From: Pooja Sanjay More <pooja.sm@pathpartnertech.com>
> > Hi Tim,
> >
> > Run test still doesn't  update the result object on fserver.
> >
> > poll request log:
> >
> > root@cpu-366U:/# fuego-core/scripts/poll_requests.sh
> > == Polling server "fuegotest.org/cgi-bin <http://fuegotest.org/cgi-bin>
> " for requests ==
> > Processing requests for any board
> > Waiting 60 seconds between polls of the server.
> > Running forever (no specified duration)
> > Type Ctrl-C to exit
> > Checking
> > Running request: request-2020-12-21_11:22:58.84-pptlab:rpi
> > Trying to get request 'request-2020-12-21_11:22:58.84-pptlab:rpi' from
> server
> > Executing test Functional.hello_world on board rpi (using spec default)
> > Running test 'Functional.hello_world' on board 'rpi' using spec 'default'
> > ===== doing fuego phase: pre_test =====
> > ===== doing fuego phase: build =====
> > The test is already built
> > Fuego test_build duration=0 seconds
> > ===== doing fuego phase: deploy =====
> > ===== doing fuego phase: snapshot =====
> > Firmware revision: 5.4.83-v7-51+
> > ===== doing fuego phase: run =====
> > -------------------------------------------------
> > Hello World!
> >
> > This is a test program, to demonstrate test success and failure
> > Test result: SUCCESS
> > -------------------------------------------------
> > ===== doing fuego phase: post_test =====
> > ===== doing fuego phase: processing =====
> > log_compare: pattern 'SUCCESS' found 1 times (expected greater or equal
> than 1)
> > chart config not found. Using default values.
> > Fuego: requested test phases complete!
> > Packaging run 'Functional.hello_world-default-104-rpi'
> > run/
> > run/machine-snapshot.txt
> > run/syslog.before.txt
> > run/devlog.txt
> > run/testlog.txt
> > run/run.json
> > run/syslog.after.txt
> > run/prolog.sh
> > run/spec.json
> > run/build.xml
> > run/consolelog.txt
> > Run packaged successfully, and is at:
> /tmp/run-Functional.hello_world-default-104-on-pptlab:rpi.frp
> > Can't put run to server
> > Server returned message:
> > Run Functional.hello_world-default-104-rpi put to server
> > Checking Waiting ..................................
> >
> >
> > Also we have installed "ttc" outside the container and configured it for
> basic commands like
> >
> > console and login, it's working for rpi board.
> >
> >
> >
> > Thanks and regards,
> > Pooja More
> >
> >
> > On Wed, Dec 16, 2020 at 9:16 PM Pooja Sanjay More <
> pooja.sm@pathpartnertech.com <mailto:pooja.sm@pathpartnertech.com> >
> > wrote:
> >
> >
> >       Hi Tim,
> >
> >       We have configured the fuego to fuegotest.org <
> http://fuegotest.org>  server.
> >       We tried put-request and run-request for rpi board after
> >       configuring it to fuegotest.org <http://fuegotest.org> , it's
> working fine.
> >       But the test run  does not create a link for test result object on
> fserver.
> >       We have scheduled a poll request for the rpi board.
> >
> >
> >       Thanks and Regards,
> >       Pooja More
> >
> >
> >       On Wed, Dec 16, 2020 at 8:59 AM Bird, Tim <Tim.Bird@sony.com
> <mailto:Tim.Bird@sony.com> > wrote:
> >
> >
> >               My Internet provider currently has some routing issues
> that prevent me
> >               from accessing birdcloud.org <http://birdcloud.org> .
> People outside my network have no problems,
> >               but I can't access from my home network.  I've been
> working to get the
> >               issues resolved, but currently, I cannot submit jobs to
> the Fuego server
> >               on birdcloud.org <http://birdcloud.org> .
> >
> >               Therefore, I have transitioned my work to the main public
> Fuego
> >               server on fuegotest.org <http://fuegotest.org> .
> >
> >               Can you please switch your Fuego configuration to use the
> fuegotest.org <http://fuegotest.org>
> >               server, instead of birdcloud.org <http://birdcloud.org> ?
> >
> >               Please change the value of "server_domain" in your
> fuego-ro/conf/fuego.conf
> >               file to the following (which is the original default value
> for a Fuego installation):
> >
> >               server_domain=fuegotest.org/cgi-bin <
> http://fuegotest.org/cgi-bin>
> >
> >               I have added the boards humanshu_wks:rpi and pptlab:rpi to
> that server.
> >               There was also a board for fuegohost:rpi, which looks like
> it may have been
> >               related to your lab.  Can you let me know what's going on
> with the humanshu_wks
> >               host entry and board?  Is that someone from your team?
> >
> >               (If not, it could be time to roll out some security for
> the server.)
> >
> >               Let me know when you have switched your configuration, and
> I will send
> >               some test requests your way. Actually, I'll post a test
> request on fuegotest.org <http://fuegotest.org>
> >               (for pptlab:rpi) to see if your host picks it up.
> >
> >               Thanks.  Sorry about this issue.
> >                -- Tim
> >
> >
> >               > -----Original Message-----
> >               > From: Pooja Sanjay More <pooja.sm@pathpartnertech.com
> <mailto:pooja.sm@pathpartnertech.com> >
> >               >
> >               > Hi Tim,
> >               >
> >               > Jenkins job has been added to schedule a poll request
> to run it for 12 hrs on a daily basis and 15 minutes  wait time  is
> > given. Previously
> >               > rpireboot.py  was failing through jenkins job because of
> permission issues. Now jenkins is added to the dialout group so
> > reboot is also
> >               > working as jenkins job.
> >               >
> >               >
> >               >
> >               >
> >               >
> >               >
> >               >
> >               >
> >               >
> >               >
> >               > Thanks and regards,
> >               > Pooja More
> >               >
> >               >
> >               >
> >               >
> >               > This message contains confidential information and is
> intended only for the individual(s) named. If you are not the
> > intended recipient, you
> >               > are notified that disclosing, copying, distributing or
> taking any action in reliance on the contents of this mail and
> > attached file/s is strictly
> >               > prohibited. Please notify the sender immediately and
> delete this e-mail from your system. E-mail transmission cannot be
> > guaranteed to be
> >               > secured or error-free as information could be
> intercepted, corrupted, lost, destroyed, arrive late or incomplete, or
> > contain viruses. The
> >               > sender therefore does not accept liability for any
> errors or omissions in the contents of this message, which arise as a
> > result of e-mail
> >               > transmission.
> >
> >
> >
> >
> > This message contains confidential information and is intended only for
> the individual(s) named. If you are not the intended recipient, you
> > are notified that disclosing, copying, distributing or taking any action
> in reliance on the contents of this mail and attached file/s is strictly
> > prohibited. Please notify the sender immediately and delete this e-mail
> from your system. E-mail transmission cannot be guaranteed to be
> > secured or error-free as information could be intercepted, corrupted,
> lost, destroyed, arrive late or incomplete, or contain viruses. The
> > sender therefore does not accept liability for any errors or omissions
> in the contents of this message, which arise as a result of e-mail
> > transmission.
>
>

-- 






This
message contains confidential information and is intended only 
for the
individual(s) named. If you are not the intended
recipient, you are 
notified that disclosing, copying, distributing or taking any
action in 
reliance on the contents of this mail and attached file/s is strictly
prohibited. Please notify the
sender immediately and delete this e-mail 
from your system. E-mail transmission
cannot be guaranteed to be secured or 
error-free as information could be
intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain
viruses. The sender therefore does 
not accept liability for any errors or
omissions in the contents of this 
message, which arise as a result of e-mail
transmission.

[-- Attachment #1.1.2: Type: text/html, Size: 19560 bytes --]

[-- Attachment #1.2: image.png --]
[-- Type: image/png, Size: 39370 bytes --]

[-- Attachment #1.3: image.png --]
[-- Type: image/png, Size: 57277 bytes --]

[-- Attachment #1.4: image.png --]
[-- Type: image/png, Size: 97484 bytes --]

[-- Attachment #2: eprint_define.patch --]
[-- Type: application/octet-stream, Size: 357 bytes --]

diff --git a/scripts/ftc b/scripts/ftc
index ae9ab2f..e70847d 100755
--- a/scripts/ftc
+++ b/scripts/ftc
@@ -616,6 +616,9 @@ def wprint(msg):
     global ll_warning
     if ll_warning:
         print("### WARNING: " + msg)
+# always print errors
+def eprint(msg):
+    print("!!! ERROR: " + msg)
 
 class config_class:
     def __init__(self, config_path):