[Fuego] Please switch server to fuegotest.org (was RE: poll request scheduling)

[Fuego] Please switch server to fuegotest.org (was RE: poll request scheduling)

[Bird, Tim]

My Internet provider currently has some routing issues that prevent me
from accessing birdcloud.org.  People outside my network have no problems,
but I can't access from my home network.  I've been working to get the
issues resolved, but currently, I cannot submit jobs to the Fuego server
on birdcloud.org.

Therefore, I have transitioned my work to the main public Fuego 
server on fuegotest.org.

Can you please switch your Fuego configuration to use the fuegotest.org
server, instead of birdcloud.org?

Please change the value of "server_domain" in your fuego-ro/conf/fuego.conf
file to the following (which is the original default value for a Fuego installation):

server_domain=fuegotest.org/cgi-bin

I have added the boards humanshu_wks:rpi and pptlab:rpi to that server.
There was also a board for fuegohost:rpi, which looks like it may have been
related to your lab.  Can you let me know what's going on with the humanshu_wks
host entry and board?  Is that someone from your team?

(If not, it could be time to roll out some security for the server.)

Let me know when you have switched your configuration, and I will send
some test requests your way. Actually, I'll post a test request on fuegotest.org
(for pptlab:rpi) to see if your host picks it up.

Thanks.  Sorry about this issue.
 -- Tim


> -----Original Message-----
> From: Pooja Sanjay More <pooja.sm@pathpartnertech.com>
> 
> Hi Tim,
> 
> Jenkins job has been added to schedule a poll request  to run it for 12 hrs on a daily basis and 15 minutes  wait time  is given. Previously
> rpireboot.py  was failing through jenkins job because of permission issues. Now jenkins is added to the dialout group so reboot is also
> working as jenkins job.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Thanks and regards,
> Pooja More
> 
> 
> 
> 
> This message contains confidential information and is intended only for the individual(s) named. If you are not the intended recipient, you
> are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this mail and attached file/s is strictly
> prohibited. Please notify the sender immediately and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be
> secured or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The
> sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail
> transmission.

Re: [Fuego] Please switch server to fuegotest.org (was RE: poll request scheduling)

[Srivatsan S]

[-- Attachment #1: Type: text/plain, Size: 3823 bytes --]

Tim,

Replies inline...


On Wed, Dec 16, 2020 at 9:00 AM Bird, Tim <Tim.Bird@sony.com> wrote:

> My Internet provider currently has some routing issues that prevent me
> from accessing birdcloud.org.  People outside my network have no problems,
> but I can't access from my home network.  I've been working to get the
> issues resolved, but currently, I cannot submit jobs to the Fuego server
> on birdcloud.org.
>
> Therefore, I have transitioned my work to the main public Fuego
> server on fuegotest.org.
>
> Can you please switch your Fuego configuration to use the fuegotest.org
> server, instead of birdcloud.org?
>
> Please change the value of "server_domain" in your fuego-ro/conf/fuego.conf
> file to the following (which is the original default value for a Fuego
> installation):
>
> server_domain=fuegotest.org/cgi-bin

[Srivatsan]
Sure.. We will update you after changing the configuration.


>
>
> I have added the boards humanshu_wks:rpi and pptlab:rpi to that server.
> There was also a board for fuegohost:rpi, which looks like it may have been
> related to your lab.  Can you let me know what's going on with the
> humanshu_wks
> host entry and board?  Is that someone from your team?
>
[Srivatsan]
Yes.. This is a request from one of our team members to test the
poll_requests.

Thanks,
Srivatsan


>
> (If not, it could be time to roll out some security for the server.)
>
> Let me know when you have switched your configuration, and I will send
> some test requests your way. Actually, I'll post a test request on
> fuegotest.org
> (for pptlab:rpi) to see if your host picks it up.
>
> Thanks.  Sorry about this issue.
>  -- Tim
>
>
> > -----Original Message-----
> > From: Pooja Sanjay More <pooja.sm@pathpartnertech.com>
> >
> > Hi Tim,
> >
> > Jenkins job has been added to schedule a poll request  to run it for 12
> hrs on a daily basis and 15 minutes  wait time  is given. Previously
> > rpireboot.py  was failing through jenkins job because of permission
> issues. Now jenkins is added to the dialout group so reboot is also
> > working as jenkins job.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Thanks and regards,
> > Pooja More
> >
> >
> >
> >
> > This message contains confidential information and is intended only for
> the individual(s) named. If you are not the intended recipient, you
> > are notified that disclosing, copying, distributing or taking any action
> in reliance on the contents of this mail and attached file/s is strictly
> > prohibited. Please notify the sender immediately and delete this e-mail
> from your system. E-mail transmission cannot be guaranteed to be
> > secured or error-free as information could be intercepted, corrupted,
> lost, destroyed, arrive late or incomplete, or contain viruses. The
> > sender therefore does not accept liability for any errors or omissions
> in the contents of this message, which arise as a result of e-mail
> > transmission.
>
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego
>

-- 






This
message contains confidential information and is intended only 
for the
individual(s) named. If you are not the intended
recipient, you are 
notified that disclosing, copying, distributing or taking any
action in 
reliance on the contents of this mail and attached file/s is strictly
prohibited. Please notify the
sender immediately and delete this e-mail 
from your system. E-mail transmission
cannot be guaranteed to be secured or 
error-free as information could be
intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain
viruses. The sender therefore does 
not accept liability for any errors or
omissions in the contents of this 
message, which arise as a result of e-mail
transmission.

[-- Attachment #2: Type: text/html, Size: 5674 bytes --]

Re: [Fuego] Please switch server to fuegotest.org (was RE: poll request scheduling)

[Pooja Sanjay More]

[-- Attachment #1: Type: text/plain, Size: 3776 bytes --]

Hi Tim,

We have configured the fuego to fuegotest.org server.
We tried put-request and run-request for rpi board after
configuring it to fuegotest.org, it's working fine.
But the test run  does not create a link for test result object on fserver.
We have scheduled a poll request for the rpi board.


Thanks and Regards,
Pooja More


On Wed, Dec 16, 2020 at 8:59 AM Bird, Tim <Tim.Bird@sony.com> wrote:

> My Internet provider currently has some routing issues that prevent me
> from accessing birdcloud.org.  People outside my network have no problems,
> but I can't access from my home network.  I've been working to get the
> issues resolved, but currently, I cannot submit jobs to the Fuego server
> on birdcloud.org.
>
> Therefore, I have transitioned my work to the main public Fuego
> server on fuegotest.org.
>
> Can you please switch your Fuego configuration to use the fuegotest.org
> server, instead of birdcloud.org?
>
> Please change the value of "server_domain" in your fuego-ro/conf/fuego.conf
> file to the following (which is the original default value for a Fuego
> installation):
>
> server_domain=fuegotest.org/cgi-bin
>
> I have added the boards humanshu_wks:rpi and pptlab:rpi to that server.
> There was also a board for fuegohost:rpi, which looks like it may have been
> related to your lab.  Can you let me know what's going on with the
> humanshu_wks
> host entry and board?  Is that someone from your team?
>
> (If not, it could be time to roll out some security for the server.)
>
> Let me know when you have switched your configuration, and I will send
> some test requests your way. Actually, I'll post a test request on
> fuegotest.org
> (for pptlab:rpi) to see if your host picks it up.
>
> Thanks.  Sorry about this issue.
>  -- Tim
>
>
> > -----Original Message-----
> > From: Pooja Sanjay More <pooja.sm@pathpartnertech.com>
> >
> > Hi Tim,
> >
> > Jenkins job has been added to schedule a poll request  to run it for 12
> hrs on a daily basis and 15 minutes  wait time  is given. Previously
> > rpireboot.py  was failing through jenkins job because of permission
> issues. Now jenkins is added to the dialout group so reboot is also
> > working as jenkins job.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Thanks and regards,
> > Pooja More
> >
> >
> >
> >
> > This message contains confidential information and is intended only for
> the individual(s) named. If you are not the intended recipient, you
> > are notified that disclosing, copying, distributing or taking any action
> in reliance on the contents of this mail and attached file/s is strictly
> > prohibited. Please notify the sender immediately and delete this e-mail
> from your system. E-mail transmission cannot be guaranteed to be
> > secured or error-free as information could be intercepted, corrupted,
> lost, destroyed, arrive late or incomplete, or contain viruses. The
> > sender therefore does not accept liability for any errors or omissions
> in the contents of this message, which arise as a result of e-mail
> > transmission.
>
>

-- 






This
message contains confidential information and is intended only 
for the
individual(s) named. If you are not the intended
recipient, you are 
notified that disclosing, copying, distributing or taking any
action in 
reliance on the contents of this mail and attached file/s is strictly
prohibited. Please notify the
sender immediately and delete this e-mail 
from your system. E-mail transmission
cannot be guaranteed to be secured or 
error-free as information could be
intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain
viruses. The sender therefore does 
not accept liability for any errors or
omissions in the contents of this 
message, which arise as a result of e-mail
transmission.

[-- Attachment #2: Type: text/html, Size: 5229 bytes --]

Re: [Fuego] Please switch server to fuegotest.org (was RE: poll request scheduling)

[Pooja Sanjay More]

[-- Attachment #1: Type: text/plain, Size: 6012 bytes --]

Hi Tim,

Run test still doesn't  update the result object on fserver.

*poll request log:*

root@cpu-366U:/# fuego-core/scripts/poll_requests.sh
== Polling server "fuegotest.org/cgi-bin" for requests ==
Processing requests for any board
Waiting 60 seconds between polls of the server.
Running forever (no specified duration)
Type Ctrl-C to exit
Checking
Running request: request-2020-12-21_11:22:58.84-pptlab:rpi
Trying to get request 'request-2020-12-21_11:22:58.84-pptlab:rpi' from
server
Executing test Functional.hello_world on board rpi (using spec default)
Running test 'Functional.hello_world' on board 'rpi' using spec 'default'
===== doing fuego phase: pre_test =====
===== doing fuego phase: build =====
The test is already built
Fuego test_build duration=0 seconds
===== doing fuego phase: deploy =====
===== doing fuego phase: snapshot =====
Firmware revision: 5.4.83-v7-51+
===== doing fuego phase: run =====
-------------------------------------------------
Hello World!

This is a test program, to demonstrate test success and failure
Test result: SUCCESS
-------------------------------------------------
===== doing fuego phase: post_test =====
===== doing fuego phase: processing =====
log_compare: pattern 'SUCCESS' found 1 times (expected greater or equal
than 1)
chart config not found. Using default values.
Fuego: requested test phases complete!
Packaging run 'Functional.hello_world-default-104-rpi'
run/
run/machine-snapshot.txt
run/syslog.before.txt
run/devlog.txt
run/testlog.txt
run/run.json
run/syslog.after.txt
run/prolog.sh
run/spec.json
run/build.xml
run/consolelog.txt
Run packaged successfully, and is at:
/tmp/run-Functional.hello_world-default-104-on-pptlab:rpi.frp
Can't put run to server
Server returned message:
Run Functional.hello_world-default-104-rpi put to server
Checking Waiting ..................................


Also we have installed "ttc" outside the container and configured it for
basic commands like
console and login, it's working for rpi board.


Thanks and regards,
Pooja More

On Wed, Dec 16, 2020 at 9:16 PM Pooja Sanjay More <
pooja.sm@pathpartnertech.com> wrote:

> Hi Tim,
>
> We have configured the fuego to fuegotest.org server.
> We tried put-request and run-request for rpi board after
> configuring it to fuegotest.org, it's working fine.
> But the test run  does not create a link for test result object on fserver.
> We have scheduled a poll request for the rpi board.
>
>
> Thanks and Regards,
> Pooja More
>
>
> On Wed, Dec 16, 2020 at 8:59 AM Bird, Tim <Tim.Bird@sony.com> wrote:
>
>> My Internet provider currently has some routing issues that prevent me
>> from accessing birdcloud.org.  People outside my network have no
>> problems,
>> but I can't access from my home network.  I've been working to get the
>> issues resolved, but currently, I cannot submit jobs to the Fuego server
>> on birdcloud.org.
>>
>> Therefore, I have transitioned my work to the main public Fuego
>> server on fuegotest.org.
>>
>> Can you please switch your Fuego configuration to use the fuegotest.org
>> server, instead of birdcloud.org?
>>
>> Please change the value of "server_domain" in your
>> fuego-ro/conf/fuego.conf
>> file to the following (which is the original default value for a Fuego
>> installation):
>>
>> server_domain=fuegotest.org/cgi-bin
>>
>> I have added the boards humanshu_wks:rpi and pptlab:rpi to that server.
>> There was also a board for fuegohost:rpi, which looks like it may have
>> been
>> related to your lab.  Can you let me know what's going on with the
>> humanshu_wks
>> host entry and board?  Is that someone from your team?
>>
>> (If not, it could be time to roll out some security for the server.)
>>
>> Let me know when you have switched your configuration, and I will send
>> some test requests your way. Actually, I'll post a test request on
>> fuegotest.org
>> (for pptlab:rpi) to see if your host picks it up.
>>
>> Thanks.  Sorry about this issue.
>>  -- Tim
>>
>>
>> > -----Original Message-----
>> > From: Pooja Sanjay More <pooja.sm@pathpartnertech.com>
>> >
>> > Hi Tim,
>> >
>> > Jenkins job has been added to schedule a poll request  to run it for 12
>> hrs on a daily basis and 15 minutes  wait time  is given. Previously
>> > rpireboot.py  was failing through jenkins job because of permission
>> issues. Now jenkins is added to the dialout group so reboot is also
>> > working as jenkins job.
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > Thanks and regards,
>> > Pooja More
>> >
>> >
>> >
>> >
>> > This message contains confidential information and is intended only for
>> the individual(s) named. If you are not the intended recipient, you
>> > are notified that disclosing, copying, distributing or taking any
>> action in reliance on the contents of this mail and attached file/s is
>> strictly
>> > prohibited. Please notify the sender immediately and delete this e-mail
>> from your system. E-mail transmission cannot be guaranteed to be
>> > secured or error-free as information could be intercepted, corrupted,
>> lost, destroyed, arrive late or incomplete, or contain viruses. The
>> > sender therefore does not accept liability for any errors or omissions
>> in the contents of this message, which arise as a result of e-mail
>> > transmission.
>>
>>

-- 






This
message contains confidential information and is intended only 
for the
individual(s) named. If you are not the intended
recipient, you are 
notified that disclosing, copying, distributing or taking any
action in 
reliance on the contents of this mail and attached file/s is strictly
prohibited. Please notify the
sender immediately and delete this e-mail 
from your system. E-mail transmission
cannot be guaranteed to be secured or 
error-free as information could be
intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain
viruses. The sender therefore does 
not accept liability for any errors or
omissions in the contents of this 
message, which arise as a result of e-mail
transmission.

[-- Attachment #2: Type: text/html, Size: 8174 bytes --]

Re: [Fuego] Please switch server to fuegotest.org (was RE: poll request scheduling)

[Tim.Bird]

Pooja,

Thanks for the updates.

I spent quite a bit of time trying to debug this.  From what I can tell,
this is a problem on the client side.  The CGI script (fserver.py) calls
the python module 'cgi' to parse the form data for the POST operation
which uploads the run data to the server.  Inside the cgi module,
in the cgi.FieldStorage() class initializer, the code reads from the
CGI program's standard input.  In some cases, the standard input
appears truncated, and the code cannot read the full data in order
to create the file.  When this happens, the code returns the
a message about not being able to find the 'file1' attribute in
order to process the file.  In other cases, the standard input appears
to be completely empty.  This causes the cgi.FieldStorage() to block
internally on a read request. Apparently in this case the web
server eventually times out the request, and the client receives
no result data, which ends up showing the message:

Server returned message:

(with no message).

I did notice that the 'requests' module used in your setup is different from the
one I'm using.  The client string for mine is: "python-requests/2.12.4", whereas
the client string for your submissions is: "python-requests/2.25.1". I don't think
this should make a difference, but if you could try installing the specific 2.12.4
version of the 'requests' python module, it would help us eliminate that difference
between the two setups.  My other theory is that there's a firewall that's messing
up the data (somehow truncating the http request body in the POST message)
between the client and server.  I don't have
the capability to sniff the network traffic at the server.  Can you try
to run wireshark or some other packet tracer, and see if the data being sent
by ftc to the network looks correct?  The http request should include the data for
the packaged run file, in a multipart section in the body of the request.

Finally - I saw one other thing that was quite strange.  When the server CGI
script hangs (blocks reading the request data), I would have expected that 'ftc'
would wait for the response before proceeding.  But I often see the next
operation from ftc (after the put_run operation), which is an update_request,
come to the server within less than a second.  I'm not sure why this is, and
I have no explanation for it.  The calls to the 'request' module by ftc should
be completely synchronous - so I should not be seeing the update_request
operation at the server until the client times out on the previous 'put_run'
request.

Anyway - I've run out of time to work on this for now, and I don't really have
a solution.  Based on what I've seen so far, I don't think this is a server-side issue.
Please check on your side and see if the version of the request module,
something to do with the firewall, or examining the network packet data, can
provide any additional insights into the problem.

 -- Tim


> -----Original Message-----
> From: Pooja Sanjay More <pooja.sm@pathpartnertech.com>
> Hi Tim,
> 
> Run test still doesn't  update the result object on fserver.
> 
> poll request log:
> 
> root@cpu-366U:/# fuego-core/scripts/poll_requests.sh
> == Polling server "fuegotest.org/cgi-bin <http://fuegotest.org/cgi-bin> " for requests ==
> Processing requests for any board
> Waiting 60 seconds between polls of the server.
> Running forever (no specified duration)
> Type Ctrl-C to exit
> Checking
> Running request: request-2020-12-21_11:22:58.84-pptlab:rpi
> Trying to get request 'request-2020-12-21_11:22:58.84-pptlab:rpi' from server
> Executing test Functional.hello_world on board rpi (using spec default)
> Running test 'Functional.hello_world' on board 'rpi' using spec 'default'
> ===== doing fuego phase: pre_test =====
> ===== doing fuego phase: build =====
> The test is already built
> Fuego test_build duration=0 seconds
> ===== doing fuego phase: deploy =====
> ===== doing fuego phase: snapshot =====
> Firmware revision: 5.4.83-v7-51+
> ===== doing fuego phase: run =====
> -------------------------------------------------
> Hello World!
> 
> This is a test program, to demonstrate test success and failure
> Test result: SUCCESS
> -------------------------------------------------
> ===== doing fuego phase: post_test =====
> ===== doing fuego phase: processing =====
> log_compare: pattern 'SUCCESS' found 1 times (expected greater or equal than 1)
> chart config not found. Using default values.
> Fuego: requested test phases complete!
> Packaging run 'Functional.hello_world-default-104-rpi'
> run/
> run/machine-snapshot.txt
> run/syslog.before.txt
> run/devlog.txt
> run/testlog.txt
> run/run.json
> run/syslog.after.txt
> run/prolog.sh
> run/spec.json
> run/build.xml
> run/consolelog.txt
> Run packaged successfully, and is at: /tmp/run-Functional.hello_world-default-104-on-pptlab:rpi.frp
> Can't put run to server
> Server returned message:
> Run Functional.hello_world-default-104-rpi put to server
> Checking Waiting ..................................
> 
> 
> Also we have installed "ttc" outside the container and configured it for basic commands like
> 
> console and login, it's working for rpi board.
> 
> 
> 
> Thanks and regards,
> Pooja More
> 
> 
> On Wed, Dec 16, 2020 at 9:16 PM Pooja Sanjay More <pooja.sm@pathpartnertech.com <mailto:pooja.sm@pathpartnertech.com> >
> wrote:
> 
> 
> 	Hi Tim,
> 
> 	We have configured the fuego to fuegotest.org <http://fuegotest.org>  server.
> 	We tried put-request and run-request for rpi board after
> 	configuring it to fuegotest.org <http://fuegotest.org> , it's working fine.
> 	But the test run  does not create a link for test result object on fserver.
> 	We have scheduled a poll request for the rpi board.
> 
> 
> 	Thanks and Regards,
> 	Pooja More
> 
> 
> 	On Wed, Dec 16, 2020 at 8:59 AM Bird, Tim <Tim.Bird@sony.com <mailto:Tim.Bird@sony.com> > wrote:
> 
> 
> 		My Internet provider currently has some routing issues that prevent me
> 		from accessing birdcloud.org <http://birdcloud.org> .  People outside my network have no problems,
> 		but I can't access from my home network.  I've been working to get the
> 		issues resolved, but currently, I cannot submit jobs to the Fuego server
> 		on birdcloud.org <http://birdcloud.org> .
> 
> 		Therefore, I have transitioned my work to the main public Fuego
> 		server on fuegotest.org <http://fuegotest.org> .
> 
> 		Can you please switch your Fuego configuration to use the fuegotest.org <http://fuegotest.org>
> 		server, instead of birdcloud.org <http://birdcloud.org> ?
> 
> 		Please change the value of "server_domain" in your fuego-ro/conf/fuego.conf
> 		file to the following (which is the original default value for a Fuego installation):
> 
> 		server_domain=fuegotest.org/cgi-bin <http://fuegotest.org/cgi-bin>
> 
> 		I have added the boards humanshu_wks:rpi and pptlab:rpi to that server.
> 		There was also a board for fuegohost:rpi, which looks like it may have been
> 		related to your lab.  Can you let me know what's going on with the humanshu_wks
> 		host entry and board?  Is that someone from your team?
> 
> 		(If not, it could be time to roll out some security for the server.)
> 
> 		Let me know when you have switched your configuration, and I will send
> 		some test requests your way. Actually, I'll post a test request on fuegotest.org <http://fuegotest.org>
> 		(for pptlab:rpi) to see if your host picks it up.
> 
> 		Thanks.  Sorry about this issue.
> 		 -- Tim
> 
> 
> 		> -----Original Message-----
> 		> From: Pooja Sanjay More <pooja.sm@pathpartnertech.com <mailto:pooja.sm@pathpartnertech.com> >
> 		>
> 		> Hi Tim,
> 		>
> 		> Jenkins job has been added to schedule a poll request  to run it for 12 hrs on a daily basis and 15 minutes  wait time  is
> given. Previously
> 		> rpireboot.py  was failing through jenkins job because of permission issues. Now jenkins is added to the dialout group so
> reboot is also
> 		> working as jenkins job.
> 		>
> 		>
> 		>
> 		>
> 		>
> 		>
> 		>
> 		>
> 		>
> 		>
> 		> Thanks and regards,
> 		> Pooja More
> 		>
> 		>
> 		>
> 		>
> 		> This message contains confidential information and is intended only for the individual(s) named. If you are not the
> intended recipient, you
> 		> are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this mail and
> attached file/s is strictly
> 		> prohibited. Please notify the sender immediately and delete this e-mail from your system. E-mail transmission cannot be
> guaranteed to be
> 		> secured or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or
> contain viruses. The
> 		> sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a
> result of e-mail
> 		> transmission.
> 
> 
> 
> 
> This message contains confidential information and is intended only for the individual(s) named. If you are not the intended recipient, you
> are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this mail and attached file/s is strictly
> prohibited. Please notify the sender immediately and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be
> secured or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The
> sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail
> transmission.

Re: [Fuego] Please switch server to fuegotest.org (was RE: poll request scheduling)

[Pooja Sanjay More]

[-- Attachment #1.1.1: Type: text/plain, Size: 14204 bytes --]

Hi Tim,

We checked with the IT team on the result upload issue and it was indeed a
firewall
configuration which was blocking it. This issue has been resolved now and
we have
successfully verified the result object updates on the server.

Logs for a test :

root@cpu-366U:~# ftc put-request pptlab:rpi Functional.hello_world
Request was accepted by the server.
request_id=request-2020-12-24_12:39:23.93-pptlab:rpi

root@cpu-366U:~# ftc run-request --put-run
request-2020-12-24_12:39:23.93-pptlab:rpi
Trying to get request 'request-2020-12-24_12:39:23.93-pptlab:rpi' from
server
Executing test Functional.hello_world on board rpi (using spec default)
Running test 'Functional.hello_world' on board 'rpi' using spec 'default'
===== doing fuego phase: pre_test =====
===== doing fuego phase: build =====
The test is already built
Fuego test_build duration=0 seconds
===== doing fuego phase: deploy =====
===== doing fuego phase: snapshot =====
Firmware revision: 5.4.83-v7-52+
===== doing fuego phase: run =====
-------------------------------------------------
Hello World!

This is a test program, to demonstrate test success and failure
Test result: SUCCESS
-------------------------------------------------
===== doing fuego phase: post_test =====
===== doing fuego phase: processing =====
log_compare: pattern 'SUCCESS' found 1 times (expected greater or equal
than 1)
chart config not found. Using default values.
Fuego: requested test phases complete!
Packaging run 'Functional.hello_world-default-130-rpi'
run/
run/machine-snapshot.txt
run/syslog.before.txt
run/devlog.txt
run/testlog.txt
run/run.json
run/syslog.after.txt
run/prolog.sh
run/spec.json
run/build.xml
run/consolelog.txt
Run packaged successfully, and is at:
/tmp/run-Functional.hello_world-default-130-on-pptlab:rpi.frp
Run package run-Functional.hello_world-default-130-on-pptlab:rpi.frp was
accepted by the server.
Run Functional.hello_world-default-130-rpi put to server

Fserver Output:[image: image.png]
[image: image.png]

We ran multiple tests and found that when there are tests which do not have
test.yaml files
give an error for  "eprint".

[image: image.png]

eprint() was not defined in ftc. We added it to ftc then it is working fine.

Changes in ftc:

diff --git a/scripts/ftc b/scripts/ftc
index ae9ab2f..e70847d 100755
--- a/scripts/ftc
+++ b/scripts/ftc
@@ -616,6 +616,9 @@ def wprint(msg):
     global ll_warning
     if ll_warning:
         print("### WARNING: " + msg)
+# always print errors
+def eprint(msg):
+    print("!!! ERROR: " + msg)

 class config_class:
     def __init__(self, config_path):

Patch file is attached below.

*Thanks and regards:*
* Pooja More*





On Wed, Dec 23, 2020 at 5:02 AM <Tim.Bird@sony.com> wrote:

> Pooja,
>
> Thanks for the updates.
>
> I spent quite a bit of time trying to debug this.  From what I can tell,
> this is a problem on the client side.  The CGI script (fserver.py) calls
> the python module 'cgi' to parse the form data for the POST operation
> which uploads the run data to the server.  Inside the cgi module,
> in the cgi.FieldStorage() class initializer, the code reads from the
> CGI program's standard input.  In some cases, the standard input
> appears truncated, and the code cannot read the full data in order
> to create the file.  When this happens, the code returns the
> a message about not being able to find the 'file1' attribute in
> order to process the file.  In other cases, the standard input appears
> to be completely empty.  This causes the cgi.FieldStorage() to block
> internally on a read request. Apparently in this case the web
> server eventually times out the request, and the client receives
> no result data, which ends up showing the message:
>
> Server returned message:
>
> (with no message).
>
> I did notice that the 'requests' module used in your setup is different
> from the
> one I'm using.  The client string for mine is: "python-requests/2.12.4",
> whereas
> the client string for your submissions is: "python-requests/2.25.1". I
> don't think
> this should make a difference, but if you could try installing the
> specific 2.12.4
> version of the 'requests' python module, it would help us eliminate that
> difference
> between the two setups.  My other theory is that there's a firewall that's
> messing
> up the data (somehow truncating the http request body in the POST message)
> between the client and server.  I don't have
> the capability to sniff the network traffic at the server.  Can you try
> to run wireshark or some other packet tracer, and see if the data being
> sent
> by ftc to the network looks correct?  The http request should include the
> data for
> the packaged run file, in a multipart section in the body of the request.
>
> Finally - I saw one other thing that was quite strange.  When the server
> CGI
> script hangs (blocks reading the request data), I would have expected that
> 'ftc'
> would wait for the response before proceeding.  But I often see the next
> operation from ftc (after the put_run operation), which is an
> update_request,
> come to the server within less than a second.  I'm not sure why this is,
> and
> I have no explanation for it.  The calls to the 'request' module by ftc
> should
> be completely synchronous - so I should not be seeing the update_request
> operation at the server until the client times out on the previous
> 'put_run'
> request.
>
> Anyway - I've run out of time to work on this for now, and I don't really
> have
> a solution.  Based on what I've seen so far, I don't think this is a
> server-side issue.
> Please check on your side and see if the version of the request module,
> something to do with the firewall, or examining the network packet data,
> can
> provide any additional insights into the problem.
>
>  -- Tim
>
>
> > -----Original Message-----
> > From: Pooja Sanjay More <pooja.sm@pathpartnertech.com>
> > Hi Tim,
> >
> > Run test still doesn't  update the result object on fserver.
> >
> > poll request log:
> >
> > root@cpu-366U:/# fuego-core/scripts/poll_requests.sh
> > == Polling server "fuegotest.org/cgi-bin <http://fuegotest.org/cgi-bin>
> " for requests ==
> > Processing requests for any board
> > Waiting 60 seconds between polls of the server.
> > Running forever (no specified duration)
> > Type Ctrl-C to exit
> > Checking
> > Running request: request-2020-12-21_11:22:58.84-pptlab:rpi
> > Trying to get request 'request-2020-12-21_11:22:58.84-pptlab:rpi' from
> server
> > Executing test Functional.hello_world on board rpi (using spec default)
> > Running test 'Functional.hello_world' on board 'rpi' using spec 'default'
> > ===== doing fuego phase: pre_test =====
> > ===== doing fuego phase: build =====
> > The test is already built
> > Fuego test_build duration=0 seconds
> > ===== doing fuego phase: deploy =====
> > ===== doing fuego phase: snapshot =====
> > Firmware revision: 5.4.83-v7-51+
> > ===== doing fuego phase: run =====
> > -------------------------------------------------
> > Hello World!
> >
> > This is a test program, to demonstrate test success and failure
> > Test result: SUCCESS
> > -------------------------------------------------
> > ===== doing fuego phase: post_test =====
> > ===== doing fuego phase: processing =====
> > log_compare: pattern 'SUCCESS' found 1 times (expected greater or equal
> than 1)
> > chart config not found. Using default values.
> > Fuego: requested test phases complete!
> > Packaging run 'Functional.hello_world-default-104-rpi'
> > run/
> > run/machine-snapshot.txt
> > run/syslog.before.txt
> > run/devlog.txt
> > run/testlog.txt
> > run/run.json
> > run/syslog.after.txt
> > run/prolog.sh
> > run/spec.json
> > run/build.xml
> > run/consolelog.txt
> > Run packaged successfully, and is at:
> /tmp/run-Functional.hello_world-default-104-on-pptlab:rpi.frp
> > Can't put run to server
> > Server returned message:
> > Run Functional.hello_world-default-104-rpi put to server
> > Checking Waiting ..................................
> >
> >
> > Also we have installed "ttc" outside the container and configured it for
> basic commands like
> >
> > console and login, it's working for rpi board.
> >
> >
> >
> > Thanks and regards,
> > Pooja More
> >
> >
> > On Wed, Dec 16, 2020 at 9:16 PM Pooja Sanjay More <
> pooja.sm@pathpartnertech.com <mailto:pooja.sm@pathpartnertech.com> >
> > wrote:
> >
> >
> >       Hi Tim,
> >
> >       We have configured the fuego to fuegotest.org <
> http://fuegotest.org>  server.
> >       We tried put-request and run-request for rpi board after
> >       configuring it to fuegotest.org <http://fuegotest.org> , it's
> working fine.
> >       But the test run  does not create a link for test result object on
> fserver.
> >       We have scheduled a poll request for the rpi board.
> >
> >
> >       Thanks and Regards,
> >       Pooja More
> >
> >
> >       On Wed, Dec 16, 2020 at 8:59 AM Bird, Tim <Tim.Bird@sony.com
> <mailto:Tim.Bird@sony.com> > wrote:
> >
> >
> >               My Internet provider currently has some routing issues
> that prevent me
> >               from accessing birdcloud.org <http://birdcloud.org> .
> People outside my network have no problems,
> >               but I can't access from my home network.  I've been
> working to get the
> >               issues resolved, but currently, I cannot submit jobs to
> the Fuego server
> >               on birdcloud.org <http://birdcloud.org> .
> >
> >               Therefore, I have transitioned my work to the main public
> Fuego
> >               server on fuegotest.org <http://fuegotest.org> .
> >
> >               Can you please switch your Fuego configuration to use the
> fuegotest.org <http://fuegotest.org>
> >               server, instead of birdcloud.org <http://birdcloud.org> ?
> >
> >               Please change the value of "server_domain" in your
> fuego-ro/conf/fuego.conf
> >               file to the following (which is the original default value
> for a Fuego installation):
> >
> >               server_domain=fuegotest.org/cgi-bin <
> http://fuegotest.org/cgi-bin>
> >
> >               I have added the boards humanshu_wks:rpi and pptlab:rpi to
> that server.
> >               There was also a board for fuegohost:rpi, which looks like
> it may have been
> >               related to your lab.  Can you let me know what's going on
> with the humanshu_wks
> >               host entry and board?  Is that someone from your team?
> >
> >               (If not, it could be time to roll out some security for
> the server.)
> >
> >               Let me know when you have switched your configuration, and
> I will send
> >               some test requests your way. Actually, I'll post a test
> request on fuegotest.org <http://fuegotest.org>
> >               (for pptlab:rpi) to see if your host picks it up.
> >
> >               Thanks.  Sorry about this issue.
> >                -- Tim
> >
> >
> >               > -----Original Message-----
> >               > From: Pooja Sanjay More <pooja.sm@pathpartnertech.com
> <mailto:pooja.sm@pathpartnertech.com> >
> >               >
> >               > Hi Tim,
> >               >
> >               > Jenkins job has been added to schedule a poll request
> to run it for 12 hrs on a daily basis and 15 minutes  wait time  is
> > given. Previously
> >               > rpireboot.py  was failing through jenkins job because of
> permission issues. Now jenkins is added to the dialout group so
> > reboot is also
> >               > working as jenkins job.
> >               >
> >               >
> >               >
> >               >
> >               >
> >               >
> >               >
> >               >
> >               >
> >               >
> >               > Thanks and regards,
> >               > Pooja More
> >               >
> >               >
> >               >
> >               >
> >               > This message contains confidential information and is
> intended only for the individual(s) named. If you are not the
> > intended recipient, you
> >               > are notified that disclosing, copying, distributing or
> taking any action in reliance on the contents of this mail and
> > attached file/s is strictly
> >               > prohibited. Please notify the sender immediately and
> delete this e-mail from your system. E-mail transmission cannot be
> > guaranteed to be
> >               > secured or error-free as information could be
> intercepted, corrupted, lost, destroyed, arrive late or incomplete, or
> > contain viruses. The
> >               > sender therefore does not accept liability for any
> errors or omissions in the contents of this message, which arise as a
> > result of e-mail
> >               > transmission.
> >
> >
> >
> >
> > This message contains confidential information and is intended only for
> the individual(s) named. If you are not the intended recipient, you
> > are notified that disclosing, copying, distributing or taking any action
> in reliance on the contents of this mail and attached file/s is strictly
> > prohibited. Please notify the sender immediately and delete this e-mail
> from your system. E-mail transmission cannot be guaranteed to be
> > secured or error-free as information could be intercepted, corrupted,
> lost, destroyed, arrive late or incomplete, or contain viruses. The
> > sender therefore does not accept liability for any errors or omissions
> in the contents of this message, which arise as a result of e-mail
> > transmission.
>
>

-- 






This
message contains confidential information and is intended only 
for the
individual(s) named. If you are not the intended
recipient, you are 
notified that disclosing, copying, distributing or taking any
action in 
reliance on the contents of this mail and attached file/s is strictly
prohibited. Please notify the
sender immediately and delete this e-mail 
from your system. E-mail transmission
cannot be guaranteed to be secured or 
error-free as information could be
intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain
viruses. The sender therefore does 
not accept liability for any errors or
omissions in the contents of this 
message, which arise as a result of e-mail
transmission.

[-- Attachment #1.1.2: Type: text/html, Size: 19560 bytes --]

[-- Attachment #1.2: image.png --]
[-- Type: image/png, Size: 39370 bytes --]

[-- Attachment #1.3: image.png --]
[-- Type: image/png, Size: 57277 bytes --]

[-- Attachment #1.4: image.png --]
[-- Type: image/png, Size: 97484 bytes --]

[-- Attachment #2: eprint_define.patch --]
[-- Type: application/octet-stream, Size: 357 bytes --]

diff --git a/scripts/ftc b/scripts/ftc
index ae9ab2f..e70847d 100755
--- a/scripts/ftc
+++ b/scripts/ftc
@@ -616,6 +616,9 @@ def wprint(msg):
     global ll_warning
     if ll_warning:
         print("### WARNING: " + msg)
+# always print errors
+def eprint(msg):
+    print("!!! ERROR: " + msg)
 
 class config_class:
     def __init__(self, config_path):

Re: [Fuego] Please switch server to fuegotest.org (was RE: poll request scheduling)

[Tim.Bird]

Hey everyone,

I'm back from vacation.  Sorry this took so long to respond to,
but I was away from work since Dec. 22.  Please see my comments
inline below.

> -----Original Message-----
> From: Pooja Sanjay More <pooja.sm@pathpartnertech.com>
> 
> Hi Tim,
> 
> We checked with the IT team on the result upload issue and it was indeed a firewall
> configuration which was blocking it. This issue has been resolved now and we have
> successfully verified the result object updates on the server.

That is very good to hear!  Thanks for getting this resolved on your end.

> 
> 
> Logs for a test :
> 
> 
> root@cpu-366U:~# ftc put-request pptlab:rpi Functional.hello_world
> Request was accepted by the server.
> request_id=request-2020-12-24_12:39:23.93-pptlab:rpi
> 
> root@cpu-366U:~# ftc run-request --put-run request-2020-12-24_12:39:23.93-pptlab:rpi
> Trying to get request 'request-2020-12-24_12:39:23.93-pptlab:rpi' from server
> Executing test Functional.hello_world on board rpi (using spec default)
> Running test 'Functional.hello_world' on board 'rpi' using spec 'default'
> ===== doing fuego phase: pre_test =====
> ===== doing fuego phase: build =====
> The test is already built
> Fuego test_build duration=0 seconds
> ===== doing fuego phase: deploy =====
> ===== doing fuego phase: snapshot =====
> Firmware revision: 5.4.83-v7-52+
> ===== doing fuego phase: run =====
> -------------------------------------------------
> Hello World!
> 
> This is a test program, to demonstrate test success and failure
> Test result: SUCCESS
> -------------------------------------------------
> ===== doing fuego phase: post_test =====
> ===== doing fuego phase: processing =====
> log_compare: pattern 'SUCCESS' found 1 times (expected greater or equal than 1)
> chart config not found. Using default values.
> Fuego: requested test phases complete!
> Packaging run 'Functional.hello_world-default-130-rpi'
> run/
> run/machine-snapshot.txt
> run/syslog.before.txt
> run/devlog.txt
> run/testlog.txt
> run/run.json
> run/syslog.after.txt
> run/prolog.sh
> run/spec.json
> run/build.xml
> run/consolelog.txt
> Run packaged successfully, and is at: /tmp/run-Functional.hello_world-default-130-on-pptlab:rpi.frp
> Run package run-Functional.hello_world-default-130-on-pptlab:rpi.frp was accepted by the server.
> Run Functional.hello_world-default-130-rpi put to server

This looks good.

> 
> Fserver Output:
> 
> 
> 
> 
> We ran multiple tests and found that when there are tests which do not have test.yaml files
> give an error for  "eprint".
> 
> 
> 
> 
> 
> eprint() was not defined in ftc. We added it to ftc then it is working fine.
> 
> Changes in ftc:
> 
> diff --git a/scripts/ftc b/scripts/ftc
> index ae9ab2f..e70847d 100755
> --- a/scripts/ftc
> +++ b/scripts/ftc
> @@ -616,6 +616,9 @@ def wprint(msg):
>      global ll_warning
>      if ll_warning:
>          print("### WARNING: " + msg)
> +# always print errors
> +def eprint(msg):
> +    print("!!! ERROR: " + msg)
> 
>  class config_class:
>      def __init__(self, config_path):
> 
> Patch file is attached below.

I applied the patch.  It was not a well-formed patch for integration with
the project.  It was missing the author, commit text, signed-off-by line
and other meta-information required for a good commit.

I added this material, and added my own "Signed-off-by" line.  Please
note that I added a "Signed-off-by" line for you (Pooja).  Please let
me know if there is any problem with this.  Since the code looks like it
was taken from parser/common.py, I guessed that it was OK to add
your signed-off-by for the patch.

I found some other bugs in my testing, and added some other patches.
These have all been pushed to the master branch on bitbucket.

Please 'git pull' the latest material, and let me know if you see any problems.

It is really nice to get these bugs fixed.  Thanks
for the testing and the patch!!

I have some other comments that I'll put in another thread.

Thanks,
 -- Tim

Re: [Fuego] Please switch server to fuegotest.org (was RE: poll request scheduling)

[Pooja Sanjay More]

[-- Attachment #1: Type: text/plain, Size: 5182 bytes --]

Hi,

We have tested test-requests from fserver after pulling the latest changes.
Tests are working fine and also updates test results on fserver.

Thanks and regards,
Pooja More

On Tue, Jan 5, 2021 at 4:02 AM <Tim.Bird@sony.com> wrote:

> Hey everyone,
>
> I'm back from vacation.  Sorry this took so long to respond to,
> but I was away from work since Dec. 22.  Please see my comments
> inline below.
>
> > -----Original Message-----
> > From: Pooja Sanjay More <pooja.sm@pathpartnertech.com>
> >
> > Hi Tim,
> >
> > We checked with the IT team on the result upload issue and it was indeed
> a firewall
> > configuration which was blocking it. This issue has been resolved now
> and we have
> > successfully verified the result object updates on the server.
>
> That is very good to hear!  Thanks for getting this resolved on your end.
>
> >
> >
> > Logs for a test :
> >
> >
> > root@cpu-366U:~# ftc put-request pptlab:rpi Functional.hello_world
> > Request was accepted by the server.
> > request_id=request-2020-12-24_12:39:23.93-pptlab:rpi
> >
> > root@cpu-366U:~# ftc run-request --put-run
> request-2020-12-24_12:39:23.93-pptlab:rpi
> > Trying to get request 'request-2020-12-24_12:39:23.93-pptlab:rpi' from
> server
> > Executing test Functional.hello_world on board rpi (using spec default)
> > Running test 'Functional.hello_world' on board 'rpi' using spec 'default'
> > ===== doing fuego phase: pre_test =====
> > ===== doing fuego phase: build =====
> > The test is already built
> > Fuego test_build duration=0 seconds
> > ===== doing fuego phase: deploy =====
> > ===== doing fuego phase: snapshot =====
> > Firmware revision: 5.4.83-v7-52+
> > ===== doing fuego phase: run =====
> > -------------------------------------------------
> > Hello World!
> >
> > This is a test program, to demonstrate test success and failure
> > Test result: SUCCESS
> > -------------------------------------------------
> > ===== doing fuego phase: post_test =====
> > ===== doing fuego phase: processing =====
> > log_compare: pattern 'SUCCESS' found 1 times (expected greater or equal
> than 1)
> > chart config not found. Using default values.
> > Fuego: requested test phases complete!
> > Packaging run 'Functional.hello_world-default-130-rpi'
> > run/
> > run/machine-snapshot.txt
> > run/syslog.before.txt
> > run/devlog.txt
> > run/testlog.txt
> > run/run.json
> > run/syslog.after.txt
> > run/prolog.sh
> > run/spec.json
> > run/build.xml
> > run/consolelog.txt
> > Run packaged successfully, and is at:
> /tmp/run-Functional.hello_world-default-130-on-pptlab:rpi.frp
> > Run package run-Functional.hello_world-default-130-on-pptlab:rpi.frp was
> accepted by the server.
> > Run Functional.hello_world-default-130-rpi put to server
>
> This looks good.
>
> >
> > Fserver Output:
> >
> >
> >
> >
> > We ran multiple tests and found that when there are tests which do not
> have test.yaml files
> > give an error for  "eprint".
> >
> >
> >
> >
> >
> > eprint() was not defined in ftc. We added it to ftc then it is working
> fine.
> >
> > Changes in ftc:
> >
> > diff --git a/scripts/ftc b/scripts/ftc
> > index ae9ab2f..e70847d 100755
> > --- a/scripts/ftc
> > +++ b/scripts/ftc
> > @@ -616,6 +616,9 @@ def wprint(msg):
> >      global ll_warning
> >      if ll_warning:
> >          print("### WARNING: " + msg)
> > +# always print errors
> > +def eprint(msg):
> > +    print("!!! ERROR: " + msg)
> >
> >  class config_class:
> >      def __init__(self, config_path):
> >
> > Patch file is attached below.
>
> I applied the patch.  It was not a well-formed patch for integration with
> the project.  It was missing the author, commit text, signed-off-by line
> and other meta-information required for a good commit.
>
> I added this material, and added my own "Signed-off-by" line.  Please
> note that I added a "Signed-off-by" line for you (Pooja).  Please let
> me know if there is any problem with this.  Since the code looks like it
> was taken from parser/common.py, I guessed that it was OK to add
> your signed-off-by for the patch.
>
> I found some other bugs in my testing, and added some other patches.
> These have all been pushed to the master branch on bitbucket.
>
> Please 'git pull' the latest material, and let me know if you see any
> problems.
>
> It is really nice to get these bugs fixed.  Thanks
> for the testing and the patch!!
>
> I have some other comments that I'll put in another thread.
>
> Thanks,
>  -- Tim
>
>

-- 






This
message contains confidential information and is intended only 
for the
individual(s) named. If you are not the intended
recipient, you are 
notified that disclosing, copying, distributing or taking any
action in 
reliance on the contents of this mail and attached file/s is strictly
prohibited. Please notify the
sender immediately and delete this e-mail 
from your system. E-mail transmission
cannot be guaranteed to be secured or 
error-free as information could be
intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain
viruses. The sender therefore does 
not accept liability for any errors or
omissions in the contents of this 
message, which arise as a result of e-mail
transmission.

[-- Attachment #2: Type: text/html, Size: 6484 bytes --]