[Fuego] Problems with the serial transport.

[Fuego] Problems with the serial transport.

[Rafael Gago Castano]

Hello!

I have a Fuego test that is used for image flashing that uses 
TARGET_SETUP_LINK. All works fine with ssh, but it would be more practical
for us to use the serial transport, as we have all the serial infrastructure 
ready on our LAVA rig and it would allow to run iperf on the network interface
which would othewise be used for ssh.

I have an utility that attaches to Uboot after flashing, prints the kernel log and 
detects when the device is ready to login, then I manually login (via stty + cat)
into the device as according to the sersh (serio) known problem list sersh is not
able to Login onto the device.

The problem is that sersh is not functioning correctly, it seems to hang on the
last line below (first sersh usage):

+++ echo 'Board setup complete'
Board setup complete
++ '[' true = true ']'
++ target_cleanup Functional.SLP_board_setup
++ local fuego_test_tmp=/tmp/fuego.Functional.SLP_board_setup
++ cmd 'rm -rf /home/root/fuego.Functional.SLP_board_setup /tmp/fuego.Functional.SLP_board_setup'
++ report_devlog 'cmd: rm -rf /home/root/fuego.Functional.SLP_board_setup /tmp/fuego.Functional.SLP_board_setup'
++ echo 'cmd: rm -rf /home/root/fuego.Functional.SLP_board_setup /tmp/fuego.Functional.SLP_board_setup'
++ ov_transport_cmd 'rm -rf /home/root/fuego.Functional.SLP_board_setup /tmp/fuego.Functional.SLP_board_setup'
++ case "$TRANSPORT" in
++ run_python_quiet /usr/local/bin/sersh -T 0.2 -b 115200 root@ttyS0 'rm -rf /home/root/fuego.Functional.SLP_board_setup /tmp/fuego.Functional.SLP_board_setup'
++ '[' '!' -z /usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games ']'
++ PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
++ python /usr/local/bin/sersh -T 0.2 -b 115200 root@ttyS0 'rm -rf /home/root/fuego.Functional.SLP_board_setup /tmp/fuego.Functional.SLP_board_setup'

If I cancel the test and I just run the failing command from the docker 
container:

> python /usr/local/bin/sersh -T 0.2 -b 115200 root@ttyS0 'rm -rf 
> home/root/fuego.Functional.SLP_board_setup /tmp/fuego.Functional.SLP_board_setup'

I consistently get this error:

go.Functional.SLP_board_setup' ; r=$?; echo '
ERROR: invalid literal for int() with base 10: '[[&&serio_cmd_done&&]]'

If I then run:

> python /usr/local/bin/sersh -T 0.2 -b 115200 root@ttyS0 ls /

It consistently works and outputs the rootfs listing. At this stage I can run
the failing command above (rm -rf ...) and it still fails consistently in the
same way.

So far I get two different behaviors, a failing test (by 20 min timeout) because
sersh doesn't seem to return or fail and an error when I try to debug from 
the console.

I have tried to change IO_TIME_SERIAL up to 0.2 with no success. From the 
command line works with much lower values, e.g. 0.5.

How are you using the serial transport? Any other thing I could try?

Our shell is Busybox.

Re: [Fuego] Problems with the serial transport.

[Dhinakar Kalyanasundaram]

[-- Attachment #1: Type: text/html, Size: 5476 bytes --]

[-- Attachment #2: 201602111742151_N3WZA6X7.png --]
[-- Type: image/png, Size: 33527 bytes --]

Re: [Fuego] Problems with the serial transport.

[Rafael Gago Castano]

Thanks for your response.

Unfortunately I don't know if it will be OK for us to just change to bash.

I have found two things:
-When the test hangs is because of the shell waiting for closing commas
 because of "weird" output. Both issues are being caused by broken 
 terminal behavior. It just hangs when the commands are broken in a way
 that they leave open commas.
-If I run "stty cols 4096" on the device I can successfully run all the long
 commands from the command line but they still fail when running from
 Fuego.

I will post my findings (if any).



 

Re: [Fuego] Problems with the serial transport.

[Rafael Gago Castano]

[-- Attachment #1: Type: text/plain, Size: 169 bytes --]

If I turn off line wrapping on the terminal:


> printf "printf %b '\033[?7l'\r" > $SERIAL_DEV


All the commands run successfully, but then sercp fails.





[-- Attachment #2: Type: text/html, Size: 796 bytes --]

Re: [Fuego] Problems with the serial transport.

[Rafael Gago Castano]

It looks like sercp --basic (using echo and cat) works, so it was just a matter of disabling line wrapping on the terminal and everything works on the BusyBox shell too. I guess that sercp failed because our device has no uuencode.

Re: [Fuego] Problems with the serial transport.

[Bird, Timothy]

> -----Original Message-----
> From: Rafael Gago Castano on Friday, May 26, 2017 8:42 PM
> 
> It looks like sercp --basic (using echo and cat) works, so it was just a matter of
> disabling line wrapping on the terminal and everything works on the BusyBox
> shell too. I guess that sercp failed because our device has no uuencode.

That's good to hear.  I didn't have time to figure out the problem using the busybox
shell, but it's nice to hear there's an easy workaround (to make the column length long
using stty).

It's too bad you had to figure out the missing 'uuencode' issue yourself.

IMHO it would be good to detect these automatically and adjust serio (or
Fuego's use of serio) to automatically compensate.  I have a test in the 'next'
branch that checks various features (presence of programs and other items),
and reports if there are any problems.  I'll look at adding something to this test
so that at least people will get a warning if there might be issues with the serial
transport.  Also I'll think about doing some extra steps in Fuego, or about
enhancing serio so that it will automatically deal with these issues.

Thanks for the report!
 - - Tim

Re: [Fuego] Problems with the serial transport.

[Rafael Gago Castano]

[-- Attachment #1: Type: text/plain, Size: 2155 bytes --]

Well, what seemed to fix the issue was turning off the line wrapping (using a VT100 control sequence), not the column length:


> printf "printf '\033[?7l'\r" > $SERIAL_DEV


But it doesn't seem to work consistently, e.g. it did work with our images flashed on NAND but not with our image booted for RAM booting. Yesterday it stopped working (it may be my mistake).


It would be nice to document these issues for the serial transport:


-Login must be handled separately (it's documented on the serio sources).

-serio has issues with Busybox adding newlines (it's documented on the serio sources).


I'm still blocked on the Busybox serial issues. I'll post here if I find something.

________________________________
From: Bird, Timothy <Tim.Bird@sony.com>
Sent: Tuesday, May 30, 2017 9:08:31 AM
To: Rafael Gago Castano; fuego@lists.linuxfoundation.org; dhinakar.k@samsung.com
Subject: RE: [Fuego] Problems with the serial transport.

> -----Original Message-----
> From: Rafael Gago Castano on Friday, May 26, 2017 8:42 PM
>
> It looks like sercp --basic (using echo and cat) works, so it was just a matter of
> disabling line wrapping on the terminal and everything works on the BusyBox
> shell too. I guess that sercp failed because our device has no uuencode.

That's good to hear.  I didn't have time to figure out the problem using the busybox
shell, but it's nice to hear there's an easy workaround (to make the column length long
using stty).

It's too bad you had to figure out the missing 'uuencode' issue yourself.

IMHO it would be good to detect these automatically and adjust serio (or
Fuego's use of serio) to automatically compensate.  I have a test in the 'next'
branch that checks various features (presence of programs and other items),
and reports if there are any problems.  I'll look at adding something to this test
so that at least people will get a warning if there might be issues with the serial
transport.  Also I'll think about doing some extra steps in Fuego, or about
enhancing serio so that it will automatically deal with these issues.

Thanks for the report!
 - - Tim


[-- Attachment #2: Type: text/html, Size: 5177 bytes --]

Re: [Fuego] Problems with the serial transport.

[Bird, Timothy]

> -----Original Message-----
> From: Rafael Gago Castano on Tuesday, May 30, 2017 4:29 PM
> 
> Well, what seemed to fix the issue was turning off the line wrapping (using a
> VT100 control sequence), not the column length:
> 
> > printf "printf '\033[?7l'\r" > $SERIAL_DEV

OK - sorry.  I misread the thread.
> 
> But it doesn't seem to work consistently, e.g. it did work with our images
> flashed on NAND but not with our image booted for RAM booting. Yesterday
> it stopped working (it may be my mistake).
>  
OK - please let me know if you get more information about the
inconsistent behavior.

> 
> It would be nice to document these issues for the serial transport:
> 
> -Login must be handled separately (it's documented on the serio sources).
I have added a separate 'serlogin' program in the 'next' branch which is intended
to deal with this issue.  It works for me, but is a bit finicky.  But the goal is to
have it completely handle this particular issue.

See https://github.com/tbird20d/serlogin
and
https://bitbucket.org/tbird20d/fuego-core/src/805adb067afc492382ee23bc9178c059b90c043e/engine/overlays/base/base-board.fuegoclass?at=next

> -serio has issues with Busybox adding newlines (it's documented on the serio
> sources).

I'd like to see if this can be fixed in serio.  serio used to turn off
echoing, and we experimented with using a different mechanism
to detect end of program output.

I'm not sure if either of these would affect this issue, but 
I'll try to find some time to look at this before the 1.2 release.

> 
> I'm still blocked on the Busybox serial issues. I'll post here if I find something.
OK - thanks.
 -- Tim

> ________________________________
> 
> From: Bird, Timothy <Tim.Bird@sony.com>
> Sent: Tuesday, May 30, 2017 9:08:31 AM
> To: Rafael Gago Castano; fuego@lists.linuxfoundation.org;
> dhinakar.k@samsung.com
> Subject: RE: [Fuego] Problems with the serial transport.
> 
> > -----Original Message-----
> > From: Rafael Gago Castano on Friday, May 26, 2017 8:42 PM
> >
> > It looks like sercp --basic (using echo and cat) works, so it was just a matter
> of
> > disabling line wrapping on the terminal and everything works on the
> BusyBox
> > shell too. I guess that sercp failed because our device has no uuencode.
> 
> That's good to hear.  I didn't have time to figure out the problem using the
> busybox
> shell, but it's nice to hear there's an easy workaround (to make the column
> length long
> using stty).
> 
> It's too bad you had to figure out the missing 'uuencode' issue yourself.
> 
> IMHO it would be good to detect these automatically and adjust serio (or
> Fuego's use of serio) to automatically compensate.  I have a test in the 'next'
> branch that checks various features (presence of programs and other items),
> and reports if there are any problems.  I'll look at adding something to this
> test
> so that at least people will get a warning if there might be issues with the
> serial
> transport.  Also I'll think about doing some extra steps in Fuego, or about
> enhancing serio so that it will automatically deal with these issues.
> 
> Thanks for the report!
>  - - Tim
> 

Re: [Fuego] Problems with the serial transport.

[Rafael Gago Castano]

[-- Attachment #1: Type: text/plain, Size: 3702 bytes --]

Well, today our lead developer came back from a short vacation and it seems that our tests images weren't using bash because of some Yocto misconfiguration. We have moved to bash and everything works.


________________________________
From: Bird, Timothy <Tim.Bird@sony.com>
Sent: Tuesday, May 30, 2017 10:29:16 AM
To: Rafael Gago Castano; fuego@lists.linuxfoundation.org; dhinakar.k@samsung.com
Cc: Frank Rowand
Subject: RE: [Fuego] Problems with the serial transport.



> -----Original Message-----
> From: Rafael Gago Castano on Tuesday, May 30, 2017 4:29 PM
>
> Well, what seemed to fix the issue was turning off the line wrapping (using a
> VT100 control sequence), not the column length:
>
> > printf "printf '\033[?7l'\r" > $SERIAL_DEV

OK - sorry.  I misread the thread.
>
> But it doesn't seem to work consistently, e.g. it did work with our images
> flashed on NAND but not with our image booted for RAM booting. Yesterday
> it stopped working (it may be my mistake).
>
OK - please let me know if you get more information about the
inconsistent behavior.

>
> It would be nice to document these issues for the serial transport:
>
> -Login must be handled separately (it's documented on the serio sources).
I have added a separate 'serlogin' program in the 'next' branch which is intended
to deal with this issue.  It works for me, but is a bit finicky.  But the goal is to
have it completely handle this particular issue.

See https://github.com/tbird20d/serlogin
and
https://bitbucket.org/tbird20d/fuego-core/src/805adb067afc492382ee23bc9178c059b90c043e/engine/overlays/base/base-board.fuegoclass?at=next

> -serio has issues with Busybox adding newlines (it's documented on the serio
> sources).

I'd like to see if this can be fixed in serio.  serio used to turn off
echoing, and we experimented with using a different mechanism
to detect end of program output.

I'm not sure if either of these would affect this issue, but
I'll try to find some time to look at this before the 1.2 release.

>
> I'm still blocked on the Busybox serial issues. I'll post here if I find something.
OK - thanks.
 -- Tim

> ________________________________
>
> From: Bird, Timothy <Tim.Bird@sony.com>
> Sent: Tuesday, May 30, 2017 9:08:31 AM
> To: Rafael Gago Castano; fuego@lists.linuxfoundation.org;
> dhinakar.k@samsung.com
> Subject: RE: [Fuego] Problems with the serial transport.
>
> > -----Original Message-----
> > From: Rafael Gago Castano on Friday, May 26, 2017 8:42 PM
> >
> > It looks like sercp --basic (using echo and cat) works, so it was just a matter
> of
> > disabling line wrapping on the terminal and everything works on the
> BusyBox
> > shell too. I guess that sercp failed because our device has no uuencode.
>
> That's good to hear.  I didn't have time to figure out the problem using the
> busybox
> shell, but it's nice to hear there's an easy workaround (to make the column
> length long
> using stty).
>
> It's too bad you had to figure out the missing 'uuencode' issue yourself.
>
> IMHO it would be good to detect these automatically and adjust serio (or
> Fuego's use of serio) to automatically compensate.  I have a test in the 'next'
> branch that checks various features (presence of programs and other items),
> and reports if there are any problems.  I'll look at adding something to this
> test
> so that at least people will get a warning if there might be issues with the
> serial
> transport.  Also I'll think about doing some extra steps in Fuego, or about
> enhancing serio so that it will automatically deal with these issues.
>
> Thanks for the report!
>  - - Tim
>


[-- Attachment #2: Type: text/html, Size: 5502 bytes --]