[Fuego] ssh timeout problem - was RE: [LTSI-dev] Fuego installation

All of lore.kernel.org
 help / color / mirror / Atom feed

* [Fuego] ssh timeout problem - was RE: [LTSI-dev] Fuego installation
@ 2016-10-21 19:44 Bird, Timothy
  2016-10-24 14:45 ` anirudh vasudevan
  0 siblings, 1 reply; 6+ messages in thread
From: Bird, Timothy @ 2016-10-21 19:44 UTC (permalink / raw)
  To: Bird, Timothy, anirudh vasudevan, Daniel Sangorrin; +Cc: ltsi-dev, fuego



> -----Original Message-----
> From: ltsi-dev-bounces@lists.linuxfoundation.org [mailto:ltsi-dev-
> bounces@lists.linuxfoundation.org] On Behalf Of Bird, Timothy
> Sent: Friday, October 21, 2016 12:10 PM
> To: anirudh vasudevan <vasuanirudh@gmail.com>
> Cc: ltsi-dev@lists.linuxfoundation.org; fuego@lists.linuxfoundation.org
> Subject: Re: [LTSI-dev] [Fuego] Fuego installation
> 
> 
> 
> > -----Original Message-----
> > From: ltsi-dev-bounces@lists.linuxfoundation.org [mailto:ltsi-dev-
> > bounces@lists.linuxfoundation.org] On Behalf Of anirudh vasudevan
> > Sent: Friday, October 21, 2016 11:52 AM
> > To: Bird, Timothy <Tim.Bird@am.sony.com>
> > Cc: ltsi-dev@lists.linuxfoundation.org; fuego@lists.linuxfoundation.org
> > Subject: Re: [LTSI-dev] [Fuego] Fuego installation
> >
> > Hi Tim,
> >
> >
> > I had already changed the timeout to 45 and ran and found no luck Now I
> > tried changing it to 80 and I was left with the same result.
> >
> 
> Hmm.  I found a reference on the internet to changing the value of UseDNS
> on the
> target.  Can you do the following:
> 
> On your target, in the file /etc/ssh/sshd_config, add a line containing:
> UseDNS no
> 
> Then do: "sudo service ssh restart"
> on the target.
> 
> Then re-run the test and see if you're still getting the timeout.
> 
> I'm setting up an environment to test this on my side, but it will take a while.
> (I have to reconstruct my setup from the tech showcase.)
> But if you could try this out real quick and let me know if it helps, that would
> be useful.

Just to report back.  I am testing a beaglebone black, using it's Ethernet connection,
on a private LAN (no connection to an external network for either the host or target).

I followed those instructions (setting 'UseDNS no') and the running Functional.bc
on the board took only 7.7 seconds, as opposed to the 4 minutes and 23 seconds
it was taking before the change.

So, this solves this particular timeout problem for me.  Please let me know if
it solves your timeout problem.

I believe we should have some "health check" tests in the test suite, to check
a variety of items relating to the target (/proc filesystem availability, ssh
timing test, etc.) and the host/target connection, that would help us identify
and cure these problems.

Let me know what you think.
 -- Tim


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [Fuego] ssh timeout problem - was RE: [LTSI-dev] Fuego installation
  2016-10-21 19:44 [Fuego] ssh timeout problem - was RE: [LTSI-dev] Fuego installation Bird, Timothy
@ 2016-10-24 14:45 ` anirudh vasudevan
  2016-10-25  1:10   ` [Fuego] [LTSI-dev] ssh timeout problem - was " Bird, Timothy
  0 siblings, 1 reply; 6+ messages in thread
From: anirudh vasudevan @ 2016-10-24 14:45 UTC (permalink / raw)
  To: Bird, Timothy; +Cc: ltsi-dev, fuego

[-- Attachment #1: Type: text/plain, Size: 3375 bytes --]

Hi Tim,

I am still facing the issue. The target is running on Debian distribution,
I did add the UseDNS and also had the target and testing machine on a
private network. Please see the below error I got from the console o/p.



+++ echo 'cmd: echo "Firmware revision:" 4.7.4-IMX6.VAR_R17'
+++ ov_transport_cmd 'echo "Firmware revision:" 4.7.4-IMX6.VAR_R17'
+++ case "$TRANSPORT" in

+++ sshpass -e ssh -o ServerAliveInterval=30000 -o
StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
ConnectTimeout=30000 -p 22 root@192.168.1.56 'echo "Firmware
revision:" 4.7.4-IMX6.VAR_R17'

ssh: connect to host 192.168.1.56 port 22: Connection timed out
+++ abort_job 'Error while ROOTFS_FWVER command execution on target'
+++ set +x

*** ABORTED ***

Fuego error reason: Error while ROOTFS_FWVER command execution on target




Thanks,

Anirudh


On Fri, Oct 21, 2016 at 3:44 PM, Bird, Timothy <Tim.Bird@am.sony.com> wrote:

>
>
> > -----Original Message-----
> > From: ltsi-dev-bounces@lists.linuxfoundation.org [mailto:ltsi-dev-
> > bounces@lists.linuxfoundation.org] On Behalf Of Bird, Timothy
> > Sent: Friday, October 21, 2016 12:10 PM
> > To: anirudh vasudevan <vasuanirudh@gmail.com>
> > Cc: ltsi-dev@lists.linuxfoundation.org; fuego@lists.linuxfoundation.org
> > Subject: Re: [LTSI-dev] [Fuego] Fuego installation
> >
> >
> >
> > > -----Original Message-----
> > > From: ltsi-dev-bounces@lists.linuxfoundation.org [mailto:ltsi-dev-
> > > bounces@lists.linuxfoundation.org] On Behalf Of anirudh vasudevan
> > > Sent: Friday, October 21, 2016 11:52 AM
> > > To: Bird, Timothy <Tim.Bird@am.sony.com>
> > > Cc: ltsi-dev@lists.linuxfoundation.org; fuego@lists.linuxfoundation.
> org
> > > Subject: Re: [LTSI-dev] [Fuego] Fuego installation
> > >
> > > Hi Tim,
> > >
> > >
> > > I had already changed the timeout to 45 and ran and found no luck Now I
> > > tried changing it to 80 and I was left with the same result.
> > >
> >
> > Hmm.  I found a reference on the internet to changing the value of UseDNS
> > on the
> > target.  Can you do the following:
> >
> > On your target, in the file /etc/ssh/sshd_config, add a line containing:
> > UseDNS no
> >
> > Then do: "sudo service ssh restart"
> > on the target.
> >
> > Then re-run the test and see if you're still getting the timeout.
> >
> > I'm setting up an environment to test this on my side, but it will take
> a while.
> > (I have to reconstruct my setup from the tech showcase.)
> > But if you could try this out real quick and let me know if it helps,
> that would
> > be useful.
>
> Just to report back.  I am testing a beaglebone black, using it's Ethernet
> connection,
> on a private LAN (no connection to an external network for either the host
> or target).
>
> I followed those instructions (setting 'UseDNS no') and the running
> Functional.bc
> on the board took only 7.7 seconds, as opposed to the 4 minutes and 23
> seconds
> it was taking before the change.
>
> So, this solves this particular timeout problem for me.  Please let me
> know if
> it solves your timeout problem.
>
> I believe we should have some "health check" tests in the test suite, to
> check
> a variety of items relating to the target (/proc filesystem availability,
> ssh
> timing test, etc.) and the host/target connection, that would help us
> identify
> and cure these problems.
>
> Let me know what you think.
>  -- Tim
>
>

[-- Attachment #2: Type: text/html, Size: 4992 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [Fuego] [LTSI-dev] ssh timeout problem - was RE: Fuego installation
  2016-10-24 14:45 ` anirudh vasudevan
@ 2016-10-25  1:10   ` Bird, Timothy
  2016-10-25 15:36     ` anirudh vasudevan
  0 siblings, 1 reply; 6+ messages in thread
From: Bird, Timothy @ 2016-10-25  1:10 UTC (permalink / raw)
  To: anirudh vasudevan; +Cc: ltsi-dev, fuego



> -----Original Message-----
> From: anirudh vasudevan on Monday, October 24, 2016 7:46 AM
>
> I am still facing the issue. The target is running on Debian distribution, I did
> add the UseDNS and also had the target and testing machine on a private
> network. Please see the below error I got from the console o/p.
> 
> +++ echo 'cmd: echo "Firmware revision:" 4.7.4-IMX6.VAR_R17'
> +++ ov_transport_cmd 'echo "Firmware revision:" 4.7.4-IMX6.VAR_R17'
> +++ case "$TRANSPORT" in
> +++ sshpass -e ssh -o ServerAliveInterval=30000 -o
> StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
> ConnectTimeout=30000 -p 22 root@192.168.1.56 <mailto:root@192.168.1.56>
> 'echo "Firmware revision:" 4.7.4-IMX6.VAR_R17'
> ssh: connect to host 192.168.1.56 port 22: Connection timed out
> +++ abort_job 'Error while ROOTFS_FWVER command execution on target'
> +++ set +x
> 
> *** ABORTED ***
> 
> Fuego error reason: Error while ROOTFS_FWVER command execution on
> target.

OK - let's see if we can pin down the problem.

You've said that you have no problem accessing the board using SSH from your
host.  There must be some problem with the networking from inside the
docker container to your network where the board is attached.

Can you try the following:

On your target do:
$ ifconfig 

On your host do:
$ ifconfig

Do the following command from your host (at a regular shell prompt, outside
the docker container):
$ ssh -vvv  -o ServerAliveInterval=30 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=20  -p 22 root@192.168.1.56

Then, as root, inside the docker container, do the same command:
$ ssh -vvv  -o ServerAliveInterval=30 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=20  -p 22 root@192.168.1.56

Then, inside the docker container, add the '-vvv' command line option to the ssh line in /home/jenkins/fuego/engine/overlays/base/base-params.fuegoclass.

The line should look something like this:
SSH="sshpass -e ssh -vvv -o ServerAliveInterval=30 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=30 -p $SSH_PORT $LOGIN@"

Now, run a test and report the results.

Thanks,
 -- Tim





^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [Fuego] [LTSI-dev] ssh timeout problem - was RE: Fuego installation
  2016-10-25  1:10   ` [Fuego] [LTSI-dev] ssh timeout problem - was " Bird, Timothy
@ 2016-10-25 15:36     ` anirudh vasudevan
  2016-10-25 19:02       ` Bird, Timothy
  0 siblings, 1 reply; 6+ messages in thread
From: anirudh vasudevan @ 2016-10-25 15:36 UTC (permalink / raw)
  To: Bird, Timothy; +Cc: ltsi-dev, fuego

[-- Attachment #1: Type: text/plain, Size: 15196 bytes --]

Here is test you had asked for

Host : ifconfig

docker0   Link encap:Ethernet
          inet addr:172.17.0.1  Bcast:0.0.0.0  Mask:255.255.0.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth0   Link encap:Ethernet
          inet addr:192.168.1.27  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::96a4:8e81:cf71:7a2c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:5360 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4990 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2956455 (2.9 MB)  TX bytes:648438 (648.4 KB)
          Interrupt:17



Target: VAR



ap0       Link encap:Ethernet
          inet6 addr: fe80::be66:41ff:fe30:a456/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:286 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:17772 (17.3 KiB)

br0       Link encap:Ethernet
          inet addr:10.15.140.230  Bcast:10.255.255.255  Mask:255.0.0.0
          inet6 addr: fe80::8625:3fff:fe0f:8ce6/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3189 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4094 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:215833 (210.7 KiB)  TX bytes:4057937 (3.8 MiB)

eth0      Link encap:Ethernet
          inet addr:192.168.1.56  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::fadc:7aff:fe05:6d00/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4988 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3993 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:4402058 (4.1 MiB)  TX bytes:372918 (364.1 KiB)



ssh command with verbose which you told me to run outside and inside of the
docker container works, am able to ssh to the target by providing the
password.

Below results are the ones from the console after I added verbose on the
base-params.fuegoclass scripts.

There are series of ssh commands which takes place and it is successfully
getting the firmware version from the board it is failing after which it
fetches the firmware version.

+++ target_cleanup Functional.bc
+++ cmd 'rm -rf /home/a/* /tmp/* /home/a/.local'
+++ report_devlog 'cmd: rm -rf /home/a/* /tmp/* /home/a/.local'
+++ mkdir -p /home/jenkins/logs/Functional.bc/devlogs
+++ echo 'cmd: rm -rf /home/a/* /tmp/* /home/a/.local'
+++ ov_transport_cmd 'rm -rf /home/a/* /tmp/* /home/a/.local'
+++ case "$TRANSPORT" in
+++ sshpass -e ssh -vvv -o ServerAliveInterval=30 -o
StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
ConnectTimeout=35 -p 22 root@192.168.1.56 'rm -rf /home/a/* /tmp/*
/home/a/.local'
OpenSSH_6.7p1 Debian-5+deb8u3, OpenSSL 1.0.1t  3 May 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 192.168.1.56 [192.168.1.56] port 22.
debug2: fd 3 setting O_NONBLOCK
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug3: timeout: 35000 ms remain after connect
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u3
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.7p1
Debian-5+deb8u3
debug1: match: OpenSSH_6.7p1 Debian-5+deb8u3 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "192.168.1.56" from file
"/dev/null"
debug3: load_hostkeys: loaded 0 keys
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: curve25519-sha256@libssh.org
,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,
aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,
aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com,
hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,
hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com
,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com,
hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,
hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com
,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: setup umac-64-etm@openssh.com
debug1: kex: server->client aes128-ctr umac-64-etm@openssh.com none
debug2: mac_setup: setup umac-64-etm@openssh.com
debug1: kex: client->server aes128-ctr umac-64-etm@openssh.com none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA
81:cb:85:b4:3f:f5:26:22:a0:d6:63:85:c2:75:ca:d2
debug3: load_hostkeys: loading entries for host "192.168.1.56" from file
"/dev/null"
debug3: load_hostkeys: loaded 0 keys
Warning: Permanently added '192.168.1.56' (ECDSA) to the list of known
hosts.
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /var/lib/jenkins/.ssh/id_rsa ((nil)),
debug2: key: /var/lib/jenkins/.ssh/id_dsa ((nil)),
debug2: key: /var/lib/jenkins/.ssh/id_ecdsa ((nil)),
debug2: key: /var/lib/jenkins/.ssh/id_ed25519 ((nil)),
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred
gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /var/lib/jenkins/.ssh/id_rsa
debug3: no such identity: /var/lib/jenkins/.ssh/id_rsa: No such file or
directory
debug1: Trying private key: /var/lib/jenkins/.ssh/id_dsa
debug3: no such identity: /var/lib/jenkins/.ssh/id_dsa: No such file or
directory
debug1: Trying private key: /var/lib/jenkins/.ssh/id_ecdsa
debug3: no such identity: /var/lib/jenkins/.ssh/id_ecdsa: No such file or
directory
debug1: Trying private key: /var/lib/jenkins/.ssh/id_ed25519
debug3: no such identity: /var/lib/jenkins/.ssh/id_ed25519: No such file or
directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
debug2: we sent a password packet, wait for reply
debug1: Authentication succeeded (password).
Authenticated to 192.168.1.56 ([192.168.1.56]:22).
debug2: fd 4 setting O_NONBLOCK
debug2: fd 5 setting O_NONBLOCK
debug3: fd 6 is O_NONBLOCK
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug2: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: packet_set_tos: set IP_TOS 0x08
debug2: client_session2_setup: id 0
debug1: Sending environment.
debug3: Ignored env BUILD_URL
debug3: Ignored env variableEnvFunctional.bc0
debug3: Ignored env HUDSON_SERVER_COOKIE
debug3: Ignored env SHELL
debug3: Ignored env TERM
debug3: Ignored env BOARD_OVERLAY
debug3: Ignored env BUILD_TAG
debug3: Ignored env Rebuild
debug3: Ignored env TESTPLAN
debug3: Ignored env Device
debug3: Ignored env WORKSPACE
debug3: Ignored env FUEGO_ENGINE_PATH
debug3: Ignored env JOB_URL
debug3: Ignored env USER
debug3: Ignored env FUEGO_LOGS_PATH
debug3: Ignored env JENKINS_HOME
debug3: Ignored env PATH
debug3: Ignored env MAIL
debug3: Ignored env PWD
debug3: Ignored env HUDSON_URL
debug3: Ignored env JOB_NAME
debug3: Ignored env FUEGO_SCRIPTS_PATH
debug3: Ignored env JENKINS_URL
debug3: Ignored env BUILD_ID
debug3: Ignored env HOME
debug3: Ignored env SHLVL
debug3: Ignored env SSHPASS
debug3: Ignored env $JOB_NAME
debug3: Ignored env JENKINS_SERVER_COOKIE
debug3: Ignored env EXECUTOR_NUMBER
debug3: Ignored env NODE_LABELS
debug3: Ignored env LOGNAME
debug3: Ignored env HUDSON_HOME
debug3: Ignored env FUEGO_PARSER_PATH
debug3: Ignored env FUEGO_TESTS_PATH
debug3: Ignored env SLAVEJAR_URL
debug3: Ignored env NODE_NAME
debug3: Ignored env BUILD_NUMBER
debug3: Ignored env HUDSON_COOKIE
debug3: Ignored env Reboot
debug3: Ignored env Target_Cleanup
debug3: Ignored env DISTRIB
debug3: Ignored env _
debug1: Sending command: rm -rf /home/a/* /tmp/* /home/a/.local
debug2: channel 0: request exec confirm 1
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel 0: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 0
debug2: exec request accepted on channel 0
debug2: channel 0: read<=0 rfd 4 len 0
debug2: channel 0: read failed
debug2: channel 0: close_read
debug2: channel 0: input open -> drain
debug2: channel 0: ibuf empty
debug2: channel 0: send eof
debug2: channel 0: input drain -> closed
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug2: channel 0: obuf empty
debug2: channel 0: close_write
debug2: channel 0: output drain -> closed
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug2: channel 0: rcvd close
debug3: channel 0: will not send data after close
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
  #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)

debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
debug3: fd 2 is not O_NONBLOCK
Transferred: sent 2500, received 1476 bytes, in 0.1 seconds
Bytes per second: sent 32307.3, received 19074.2

SSH command failing here

+++ sshpass -e ssh -vvv -o ServerAliveInterval=30 -o
StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
ConnectTimeout=35 -p 22 root@192.168.1.56 'echo "Firmware revision:"
4.7.4-IMX6.VAR_R17'
OpenSSH_6.7p1 Debian-5+deb8u3, OpenSSL 1.0.1t  3 May 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 192.168.1.56 [192.168.1.56] port 22.
debug2: fd 3 setting O_NONBLOCK

debug1: connect to address 192.168.1.56 port 22: Connection timed out
ssh: connect to host 192.168.1.56 port 22: Connection timed out
+++ abort_job 'Error while ROOTFS_FWVER command execution on target'
+++ set +x



Thanks,
Anirudh Vasudevan

On Mon, Oct 24, 2016 at 9:10 PM, Bird, Timothy <Tim.Bird@am.sony.com> wrote:

>
>
> > -----Original Message-----
> > From: anirudh vasudevan on Monday, October 24, 2016 7:46 AM
> >
> > I am still facing the issue. The target is running on Debian
> distribution, I did
> > add the UseDNS and also had the target and testing machine on a private
> > network. Please see the below error I got from the console o/p.
> >
> > +++ echo 'cmd: echo "Firmware revision:" 4.7.4-IMX6.VAR_R17'
> > +++ ov_transport_cmd 'echo "Firmware revision:" 4.7.4-IMX6.VAR_R17'
> > +++ case "$TRANSPORT" in
> > +++ sshpass -e ssh -o ServerAliveInterval=30000 -o
> > StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
> > ConnectTimeout=30000 -p 22 root@192.168.1.56 <mailto:root@192.168.1.56>
> > 'echo "Firmware revision:" 4.7.4-IMX6.VAR_R17'
> > ssh: connect to host 192.168.1.56 port 22: Connection timed out
> > +++ abort_job 'Error while ROOTFS_FWVER command execution on target'
> > +++ set +x
> >
> > *** ABORTED ***
> >
> > Fuego error reason: Error while ROOTFS_FWVER command execution on
> > target.
>
> OK - let's see if we can pin down the problem.
>
> You've said that you have no problem accessing the board using SSH from
> your
> host.  There must be some problem with the networking from inside the
> docker container to your network where the board is attached.
>
> Can you try the following:
>
> On your target do:
> $ ifconfig
>
> On your host do:
> $ ifconfig
>
> Do the following command from your host (at a regular shell prompt, outside
> the docker container):
> $ ssh -vvv  -o ServerAliveInterval=30 -o StrictHostKeyChecking=no -o
> UserKnownHostsFile=/dev/null -o ConnectTimeout=20  -p 22 root@192.168.1.56
>
> Then, as root, inside the docker container, do the same command:
> $ ssh -vvv  -o ServerAliveInterval=30 -o StrictHostKeyChecking=no -o
> UserKnownHostsFile=/dev/null -o ConnectTimeout=20  -p 22 root@192.168.1.56
>
> Then, inside the docker container, add the '-vvv' command line option to
> the ssh line in /home/jenkins/fuego/engine/overlays/base/base-params.
> fuegoclass.
>
> The line should look something like this:
> SSH="sshpass -e ssh -vvv -o ServerAliveInterval=30 -o
> StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
> ConnectTimeout=30 -p $SSH_PORT $LOGIN@"
>
> Now, run a test and report the results.
>
> Thanks,
>  -- Tim
>
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 18879 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [Fuego] [LTSI-dev] ssh timeout problem - was RE: Fuego installation
  2016-10-25 15:36     ` anirudh vasudevan
@ 2016-10-25 19:02       ` Bird, Timothy
  2016-10-25 19:20         ` anirudh vasudevan
  0 siblings, 1 reply; 6+ messages in thread
From: Bird, Timothy @ 2016-10-25 19:02 UTC (permalink / raw)
  To: anirudh vasudevan; +Cc: ltsi-dev, fuego

anirudh,

(Sorry for the duplicate message.  I forgot to copy the lists in my reply)

These logs are not the full logs for each command.  They're missing the lines I'm interested in.
In particular, in wanted to see the "debug1: sending command" and "debug1: Exit status" lines
for each ssh invocation.

See more comments/questions inline below.

> -----Original Message-----
> From: anirudh vasudevan [mailto:vasuanirudh@gmail.com]
> Sent: Tuesday, October 25, 2016 8:37 AM
> To: Bird, Timothy <Tim.Bird@am.sony.com>
> Cc: ltsi-dev@lists.linuxfoundation.org; fuego@lists.linuxfoundation.org
> Subject: Re: [LTSI-dev] ssh timeout problem - was RE: [Fuego] Fuego
> installation
> 
> Here is test you had asked for
> 
> Host : ifconfig
> 
> docker0   Link encap:Ethernet
>           inet addr:172.17.0.1  Bcast:0.0.0.0  Mask:255.255.0.0
>           UP BROADCAST MULTICAST  MTU:1500  Metric:1
>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
> 
> eth0   Link encap:Ethernet
>           inet addr:192.168.1.27  Bcast:192.168.1.255  Mask:255.255.255.0
>           inet6 addr: fe80::96a4:8e81:cf71:7a2c/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:5360 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:4990 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:2956455 (2.9 MB)  TX bytes:648438 (648.4 KB)
>           Interrupt:17
> 
> 
> 
> Target: VAR
> 
> 
> 
> ap0       Link encap:Ethernet
>           inet6 addr: fe80::be66:41ff:fe30:a456/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:286 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:0 (0.0 B)  TX bytes:17772 (17.3 KiB)
> 
> br0       Link encap:Ethernet
>           inet addr:10.15.140.230  Bcast:10.255.255.255  Mask:255.0.0.0
>           inet6 addr: fe80::8625:3fff:fe0f:8ce6/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:3189 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:4094 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:215833 (210.7 KiB)  TX bytes:4057937 (3.8 MiB)
> 
> eth0      Link encap:Ethernet
>           inet addr:192.168.1.56  Bcast:192.168.1.255  Mask:255.255.255.0
>           inet6 addr: fe80::fadc:7aff:fe05:6d00/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:4988 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:3993 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:4402058 (4.1 MiB)  TX bytes:372918 (364.1 KiB)
> 
> 
> 
> 
> ssh command with verbose which you told me to run outside and inside of
> the docker container works, am able to ssh to the target by providing the
> password.


OK
> 
> 
> Below results are the ones from the console after I added verbose on the
> base-params.fuegoclass scripts.
> 
> 
> There are series of ssh commands which takes place and it is successfully
> getting the firmware version from the board it is failing after which it fetches
> the firmware version.
> 
> +++ target_cleanup Functional.bc
> +++ cmd 'rm -rf /home/a/* /tmp/* /home/a/.local'
> +++ report_devlog 'cmd: rm -rf /home/a/* /tmp/* /home/a/.local'
> +++ mkdir -p /home/jenkins/logs/Functional.bc/devlogs
> +++ echo 'cmd: rm -rf /home/a/* /tmp/* /home/a/.local'
> +++ ov_transport_cmd 'rm -rf /home/a/* /tmp/* /home/a/.local'
> +++ case "$TRANSPORT" in
> +++ sshpass -e ssh -vvv -o ServerAliveInterval=30 -o
> StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
> ConnectTimeout=35 -p 22 root@192.168.1.56 <mailto:root@192.168.1.56>
> 'rm -rf /home/a/* /tmp/* /home/a/.local'
> OpenSSH_6.7p1 Debian-5+deb8u3, OpenSSL 1.0.1t  3 May 2016
> debug1: Reading configuration data /etc/ssh/ssh_config
> debug1: /etc/ssh/ssh_config line 19: Applying options for *
> debug2: ssh_connect: needpriv 0
> debug1: Connecting to 192.168.1.56 [192.168.1.56] port 22.
> debug2: fd 3 setting O_NONBLOCK
> debug1: fd 3 clearing O_NONBLOCK
> debug1: Connection established.
> debug3: timeout: 35000 ms remain after connect
> debug1: key_load_public: No such file or directory
> debug1: identity file /var/lib/jenkins/.ssh/id_rsa type -1
> debug1: key_load_public: No such file or directory
> debug1: identity file /var/lib/jenkins/.ssh/id_rsa-cert type -1
> debug1: key_load_public: No such file or directory
> debug1: identity file /var/lib/jenkins/.ssh/id_dsa type -1
> debug1: key_load_public: No such file or directory
> debug1: identity file /var/lib/jenkins/.ssh/id_dsa-cert type -1
> debug1: key_load_public: No such file or directory
> debug1: identity file /var/lib/jenkins/.ssh/id_ecdsa type -1
> debug1: key_load_public: No such file or directory
> debug1: identity file /var/lib/jenkins/.ssh/id_ecdsa-cert type -1
> debug1: key_load_public: No such file or directory
> debug1: identity file /var/lib/jenkins/.ssh/id_ed25519 type -1
> debug1: key_load_public: No such file or directory
> debug1: identity file /var/lib/jenkins/.ssh/id_ed25519-cert type -1
> debug1: Enabling compatibility mode for protocol 2.0
> debug1: Local version string SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u3
> debug1: Remote protocol version 2.0, remote software version
> OpenSSH_6.7p1 Debian-5+deb8u3
> debug1: match: OpenSSH_6.7p1 Debian-5+deb8u3 pat OpenSSH* compat
> 0x04000000
> debug2: fd 3 setting O_NONBLOCK
> debug3: load_hostkeys: loading entries for host "192.168.1.56" from file
> "/dev/null"
> debug3: load_hostkeys: loaded 0 keys
> debug1: SSH2_MSG_KEXINIT sent
> debug1: SSH2_MSG_KEXINIT received
> debug2: kex_parse_kexinit:
> debug2: kex_parse_kexinit:
> debug2: kex_parse_kexinit: first_kex_follows 0
> debug2: kex_parse_kexinit: reserved 0
> debug2: kex_parse_kexinit: curve25519-sha256@libssh.org
> <mailto:curve25519-sha256@libssh.org> ,ecdh-sha2-nistp256,ecdh-sha2-
> nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-
> hellman-group14-sha1
> debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-
> ed25519
> debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes128-
> gcm@openssh.com <mailto:aes128-gcm@openssh.com> ,aes256-
> gcm@openssh.com <mailto:aes256-gcm@openssh.com> ,chacha20-
> poly1305@openssh.com <mailto:chacha20-poly1305@openssh.com>
> debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes128-
> gcm@openssh.com <mailto:aes128-gcm@openssh.com> ,aes256-
> gcm@openssh.com <mailto:aes256-gcm@openssh.com> ,chacha20-
> poly1305@openssh.com <mailto:chacha20-poly1305@openssh.com>
> debug2: kex_parse_kexinit: umac-64-etm@openssh.com <mailto:umac-64-
> etm@openssh.com> ,umac-128-etm@openssh.com <mailto:umac-128-
> etm@openssh.com> ,hmac-sha2-256-etm@openssh.com <mailto:hmac-
> sha2-256-etm@openssh.com> ,hmac-sha2-512-etm@openssh.com
> <mailto:hmac-sha2-512-etm@openssh.com> ,hmac-sha1-
> etm@openssh.com <mailto:hmac-sha1-etm@openssh.com> ,umac-
> 64@openssh.com <mailto:umac-64@openssh.com> ,umac-
> 128@openssh.com <mailto:umac-128@openssh.com> ,hmac-sha2-
> 256,hmac-sha2-512,hmac-sha1
> debug2: kex_parse_kexinit: umac-64-etm@openssh.com <mailto:umac-64-
> etm@openssh.com> ,umac-128-etm@openssh.com <mailto:umac-128-
> etm@openssh.com> ,hmac-sha2-256-etm@openssh.com <mailto:hmac-
> sha2-256-etm@openssh.com> ,hmac-sha2-512-etm@openssh.com
> <mailto:hmac-sha2-512-etm@openssh.com> ,hmac-sha1-
> etm@openssh.com <mailto:hmac-sha1-etm@openssh.com> ,umac-
> 64@openssh.com <mailto:umac-64@openssh.com> ,umac-
> 128@openssh.com <mailto:umac-128@openssh.com> ,hmac-sha2-
> 256,hmac-sha2-512,hmac-sha1
> debug2: kex_parse_kexinit: none,zlib@openssh.com
> <mailto:zlib@openssh.com>
> debug2: kex_parse_kexinit: none,zlib@openssh.com
> <mailto:zlib@openssh.com>
> debug2: kex_parse_kexinit:
> debug2: kex_parse_kexinit:
> debug2: kex_parse_kexinit: first_kex_follows 0
> debug2: kex_parse_kexinit: reserved 0
> debug2: mac_setup: setup umac-64-etm@openssh.com <mailto:umac-64-
> etm@openssh.com>
> debug1: kex: server->client aes128-ctr umac-64-etm@openssh.com
> <mailto:umac-64-etm@openssh.com>  none
> debug2: mac_setup: setup umac-64-etm@openssh.com <mailto:umac-64-
> etm@openssh.com>
> debug1: kex: client->server aes128-ctr umac-64-etm@openssh.com
> <mailto:umac-64-etm@openssh.com>  none
> debug1: sending SSH2_MSG_KEX_ECDH_INIT
> debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
> debug1: Server host key: ECDSA
> 81:cb:85:b4:3f:f5:26:22:a0:d6:63:85:c2:75:ca:d2
> debug3: load_hostkeys: loading entries for host "192.168.1.56" from file
> "/dev/null"
> debug3: load_hostkeys: loaded 0 keys
> Warning: Permanently added '192.168.1.56' (ECDSA) to the list of known
> hosts.
> debug2: kex_derive_keys
> debug2: set_newkeys: mode 1
> debug1: SSH2_MSG_NEWKEYS sent
> debug1: expecting SSH2_MSG_NEWKEYS
> debug2: set_newkeys: mode 0
> debug1: SSH2_MSG_NEWKEYS received
> debug1: SSH2_MSG_SERVICE_REQUEST sent
> debug2: service_accept: ssh-userauth
> debug1: SSH2_MSG_SERVICE_ACCEPT received
> debug2: key: /var/lib/jenkins/.ssh/id_rsa ((nil)),
> debug2: key: /var/lib/jenkins/.ssh/id_dsa ((nil)),
> debug2: key: /var/lib/jenkins/.ssh/id_ecdsa ((nil)),
> debug2: key: /var/lib/jenkins/.ssh/id_ed25519 ((nil)),
> debug1: Authentications that can continue: publickey,password
> debug3: start over, passed a different list publickey,password
> debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-
> interactive,password
> debug3: authmethod_lookup publickey
> debug3: remaining preferred: keyboard-interactive,password
> debug3: authmethod_is_enabled publickey
> debug1: Next authentication method: publickey
> debug1: Trying private key: /var/lib/jenkins/.ssh/id_rsa
> debug3: no such identity: /var/lib/jenkins/.ssh/id_rsa: No such file or
> directory
> debug1: Trying private key: /var/lib/jenkins/.ssh/id_dsa
> debug3: no such identity: /var/lib/jenkins/.ssh/id_dsa: No such file or
> directory
> debug1: Trying private key: /var/lib/jenkins/.ssh/id_ecdsa
> debug3: no such identity: /var/lib/jenkins/.ssh/id_ecdsa: No such file or
> directory
> debug1: Trying private key: /var/lib/jenkins/.ssh/id_ed25519
> debug3: no such identity: /var/lib/jenkins/.ssh/id_ed25519: No such file or
> directory
> debug2: we did not send a packet, disable method
> debug3: authmethod_lookup password
> debug3: remaining preferred: ,password
> debug3: authmethod_is_enabled password
> debug1: Next authentication method: password
> debug2: we sent a password packet, wait for reply
> debug1: Authentication succeeded (password).
> Authenticated to 192.168.1.56 ([192.168.1.56]:22).
> debug2: fd 4 setting O_NONBLOCK
> debug2: fd 5 setting O_NONBLOCK
> debug3: fd 6 is O_NONBLOCK
> debug1: channel 0: new [client-session]
> debug3: ssh_session2_open: channel_new: 0
> debug2: channel 0: send open
> debug1: Requesting no-more-sessions@openssh.com <mailto:no-more-
> sessions@openssh.com>
> debug1: Entering interactive session.
> debug2: callback start
> debug2: fd 3 setting TCP_NODELAY
> debug3: packet_set_tos: set IP_TOS 0x08
> debug2: client_session2_setup: id 0
> debug1: Sending environment.
> debug3: Ignored env BUILD_URL
> debug3: Ignored env variableEnvFunctional.bc0
> debug3: Ignored env HUDSON_SERVER_COOKIE
> debug3: Ignored env SHELL
> debug3: Ignored env TERM
> debug3: Ignored env BOARD_OVERLAY
> debug3: Ignored env BUILD_TAG
> debug3: Ignored env Rebuild
> debug3: Ignored env TESTPLAN
> debug3: Ignored env Device
> debug3: Ignored env WORKSPACE
> debug3: Ignored env FUEGO_ENGINE_PATH
> debug3: Ignored env JOB_URL
> debug3: Ignored env USER
> debug3: Ignored env FUEGO_LOGS_PATH
> debug3: Ignored env JENKINS_HOME
> debug3: Ignored env PATH
> debug3: Ignored env MAIL
> debug3: Ignored env PWD
> debug3: Ignored env HUDSON_URL
> debug3: Ignored env JOB_NAME
> debug3: Ignored env FUEGO_SCRIPTS_PATH
> debug3: Ignored env JENKINS_URL
> debug3: Ignored env BUILD_ID
> debug3: Ignored env HOME
> debug3: Ignored env SHLVL
> debug3: Ignored env SSHPASS
> debug3: Ignored env $JOB_NAME
> debug3: Ignored env JENKINS_SERVER_COOKIE
> debug3: Ignored env EXECUTOR_NUMBER
> debug3: Ignored env NODE_LABELS
> debug3: Ignored env LOGNAME
> debug3: Ignored env HUDSON_HOME
> debug3: Ignored env FUEGO_PARSER_PATH
> debug3: Ignored env FUEGO_TESTS_PATH
> debug3: Ignored env SLAVEJAR_URL
> debug3: Ignored env NODE_NAME
> debug3: Ignored env BUILD_NUMBER
> debug3: Ignored env HUDSON_COOKIE
> debug3: Ignored env Reboot
> debug3: Ignored env Target_Cleanup
> debug3: Ignored env DISTRIB
> debug3: Ignored env _
> debug1: Sending command: rm -rf /home/a/* /tmp/* /home/a/.local
> debug2: channel 0: request exec confirm 1
> debug2: callback done
> debug2: channel 0: open confirm rwindow 0 rmax 32768
> debug2: channel 0: rcvd adjust 2097152
> debug2: channel_input_status_confirm: type 99 id 0
> debug2: exec request accepted on channel 0
> debug2: channel 0: read<=0 rfd 4 len 0
> debug2: channel 0: read failed
> debug2: channel 0: close_read
> debug2: channel 0: input open -> drain
> debug2: channel 0: ibuf empty
> debug2: channel 0: send eof
> debug2: channel 0: input drain -> closed
> debug2: channel 0: rcvd eof
> debug2: channel 0: output open -> drain
> debug2: channel 0: obuf empty
> debug2: channel 0: close_write
> debug2: channel 0: output drain -> closed
> debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
> debug2: channel 0: rcvd close
> debug3: channel 0: will not send data after close
> debug2: channel 0: almost dead
> debug2: channel 0: gc: notify user
> debug2: channel 0: gc: user detached
> debug2: channel 0: send close
> debug2: channel 0: is dead
> debug2: channel 0: garbage collecting
> debug1: channel 0: free: client-session, nchannels 1
> debug3: channel 0: status: The following connections are open:
>   #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)
> 
> debug1: fd 0 clearing O_NONBLOCK
> debug1: fd 1 clearing O_NONBLOCK
> debug3: fd 2 is not O_NONBLOCK
> Transferred: sent 2500, received 1476 bytes, in 0.1 seconds
> Bytes per second: sent 32307.3, received 19074.2
>

What is the exit status of this command?

What about for the 'true' command, which is also run during the pre-test phase?
 
> 
> SSH command failing here
Is this commenting on the ssh operation above or below?  I presume below...

> 
> 
> +++ sshpass -e ssh -vvv -o ServerAliveInterval=30 -o
> StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
> ConnectTimeout=35 -p 22 root@192.168.1.56 <mailto:root@192.168.1.56>
> 'echo "Firmware revision:" 4.7.4-IMX6.VAR_R17'
> OpenSSH_6.7p1 Debian-5+deb8u3, OpenSSL 1.0.1t  3 May 2016
> debug1: Reading configuration data /etc/ssh/ssh_config
> debug1: /etc/ssh/ssh_config line 19: Applying options for *
> debug2: ssh_connect: needpriv 0
> debug1: Connecting to 192.168.1.56 [192.168.1.56] port 22.
> debug2: fd 3 setting O_NONBLOCK
> 
> debug1: connect to address 192.168.1.56 port 22: Connection timed out
> ssh: connect to host 192.168.1.56 port 22: Connection timed out
> +++ abort_job 'Error while ROOTFS_FWVER command execution on target'
> +++ set +x
> 

There are 3 commands before the 'echo "Firmware revision" ...' command.
 * rm -rf /home/a/* /tmp/* /home/a/.local
 * true
 * uname -r | xargs echo

I presume that all these commands run OK on the target, and that:
 * 'echo "Firmware revision:" 4.7.4-IMX6.VAR_R17'
is the first one that times out?

Is this consistent?  Does the timeout occur on any other commands or is it
always this one?

Given what I see above, I think you have an honest-to-goodness networking
problem between your host and your target.  The TCP connection (before
doing all the authentication stuff for SSH) is not getting established on this execution
of ssh.  You should be seeing the following two lines, after the lines trying to 
stabling the connection:
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established

After this error occurs, is there anything in the kernel messages on the target about
the Ethernet link dropping?

If you run the test again, do the first three commands work again?

If the board has fundamental networking problems (ie the networking is flaky),
then we can't really do any testing with it, with current Fuego.  We have plans
to work on support for operating on boards that only have serial console support,
but that work won't be done for a few months.

Sorry to not be able to help more.
 -- Tim


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [Fuego] [LTSI-dev] ssh timeout problem - was RE: Fuego installation
  2016-10-25 19:02       ` Bird, Timothy
@ 2016-10-25 19:20         ` anirudh vasudevan
  0 siblings, 0 replies; 6+ messages in thread
From: anirudh vasudevan @ 2016-10-25 19:20 UTC (permalink / raw)
  To: Bird, Timothy; +Cc: ltsi-dev, fuego

[-- Attachment #1: Type: text/plain, Size: 54616 bytes --]

Tim,

Please let me know if these are ok? the entire log file


+++ case "$TRANSPORT" in
+++ sshpass -e ssh -vvv -o ServerAliveInterval=30 -o
StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
ConnectTimeout=35 -p 22 root@192.168.1.56 'rm -rf /home/a/* /tmp/*
/home/a/.local'
OpenSSH_6.7p1 Debian-5+deb8u3, OpenSSL 1.0.1t  3 May 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 192.168.1.56 [192.168.1.56] port 22.
debug2: fd 3 setting O_NONBLOCK
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug3: timeout: 35000 ms remain after connect
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u3
debug1: Remote protocol version 2.0, remote software version
OpenSSH_6.7p1 Debian-5+deb8u3
debug1: match: OpenSSH_6.7p1 Debian-5+deb8u3 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "192.168.1.56" from
file "/dev/null"
debug3: load_hostkeys: loaded 0 keys
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit:
curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit:
ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit:
aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit:
aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit:
umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit:
umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit:
curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
debug2: kex_parse_kexinit:
aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
debug2: kex_parse_kexinit:
aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
debug2: kex_parse_kexinit:
umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: kex_parse_kexinit:
umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: setup umac-64-etm@openssh.com
debug1: kex: server->client aes128-ctr umac-64-etm@openssh.com none
debug2: mac_setup: setup umac-64-etm@openssh.com
debug1: kex: client->server aes128-ctr umac-64-etm@openssh.com none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 81:cb:85:b4:3f:f5:26:22:a0:d6:63:85:c2:75:ca:d2
debug3: load_hostkeys: loading entries for host "192.168.1.56" from
file "/dev/null"
debug3: load_hostkeys: loaded 0 keys
Warning: Permanently added '192.168.1.56' (ECDSA) to the list of known hosts.
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /var/lib/jenkins/.ssh/id_rsa ((nil)),
debug2: key: /var/lib/jenkins/.ssh/id_dsa ((nil)),
debug2: key: /var/lib/jenkins/.ssh/id_ecdsa ((nil)),
debug2: key: /var/lib/jenkins/.ssh/id_ed25519 ((nil)),
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred
gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /var/lib/jenkins/.ssh/id_rsa
debug3: no such identity: /var/lib/jenkins/.ssh/id_rsa: No such file
or directory
debug1: Trying private key: /var/lib/jenkins/.ssh/id_dsa
debug3: no such identity: /var/lib/jenkins/.ssh/id_dsa: No such file
or directory
debug1: Trying private key: /var/lib/jenkins/.ssh/id_ecdsa
debug3: no such identity: /var/lib/jenkins/.ssh/id_ecdsa: No such file
or directory
debug1: Trying private key: /var/lib/jenkins/.ssh/id_ed25519
debug3: no such identity: /var/lib/jenkins/.ssh/id_ed25519: No such
file or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
debug2: we sent a password packet, wait for reply
debug1: Authentication succeeded (password).
Authenticated to 192.168.1.56 ([192.168.1.56]:22).
debug2: fd 4 setting O_NONBLOCK
debug2: fd 5 setting O_NONBLOCK
debug3: fd 6 is O_NONBLOCK
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug2: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: packet_set_tos: set IP_TOS 0x08
debug2: client_session2_setup: id 0
debug1: Sending environment.
debug3: Ignored env BUILD_URL
debug3: Ignored env variableEnvFunctional.bc0
debug3: Ignored env HUDSON_SERVER_COOKIE
debug3: Ignored env SHELL
debug3: Ignored env TERM
debug3: Ignored env BOARD_OVERLAY
debug3: Ignored env BUILD_TAG
debug3: Ignored env Rebuild
debug3: Ignored env TESTPLAN
debug3: Ignored env Device
debug3: Ignored env WORKSPACE
debug3: Ignored env FUEGO_ENGINE_PATH
debug3: Ignored env JOB_URL
debug3: Ignored env USER
debug3: Ignored env FUEGO_LOGS_PATH
debug3: Ignored env JENKINS_HOME
debug3: Ignored env PATH
debug3: Ignored env MAIL
debug3: Ignored env PWD
debug3: Ignored env HUDSON_URL
debug3: Ignored env JOB_NAME
debug3: Ignored env FUEGO_SCRIPTS_PATH
debug3: Ignored env JENKINS_URL
debug3: Ignored env BUILD_ID
debug3: Ignored env HOME
debug3: Ignored env SHLVL
debug3: Ignored env SSHPASS
debug3: Ignored env $JOB_NAME
debug3: Ignored env JENKINS_SERVER_COOKIE
debug3: Ignored env EXECUTOR_NUMBER
debug3: Ignored env NODE_LABELS
debug3: Ignored env LOGNAME
debug3: Ignored env HUDSON_HOME
debug3: Ignored env FUEGO_PARSER_PATH
debug3: Ignored env FUEGO_TESTS_PATH
debug3: Ignored env SLAVEJAR_URL
debug3: Ignored env NODE_NAME
debug3: Ignored env BUILD_NUMBER
debug3: Ignored env HUDSON_COOKIE
debug3: Ignored env Reboot
debug3: Ignored env Target_Cleanup
debug3: Ignored env DISTRIB
debug3: Ignored env _
debug1: Sending command: rm -rf /home/a/* /tmp/* /home/a/.local
debug2: channel 0: request exec confirm 1
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel 0: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 0
debug2: exec request accepted on channel 0
debug2: channel 0: read<=0 rfd 4 len 0
debug2: channel 0: read failed
debug2: channel 0: close_read
debug2: channel 0: input open -> drain
debug2: channel 0: ibuf empty
debug2: channel 0: send eof
debug2: channel 0: input drain -> closed
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug2: channel 0: obuf empty
debug2: channel 0: close_write
debug2: channel 0: output drain -> closed
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug2: channel 0: rcvd close
debug3: channel 0: will not send data after close
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
  #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)

debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
debug3: fd 2 is not O_NONBLOCK
Transferred: sent 2500, received 1476 bytes, in 0.1 seconds
Bytes per second: sent 47811.2, received 28227.7
debug1: Exit status 0
+++ cmd true
+++ report_devlog 'cmd: true'
+++ mkdir -p /home/jenkins/logs/Functional.bc/devlogs
+++ devlog_file=/home/jenkins/logs/Functional.bc/devlogs/xxx.2016-10-25_18-33-48.7.txt
+++ echo 'cmd: true'
+++ ov_transport_cmd true
+++ case "$TRANSPORT" in
+++ sshpass -e ssh -vvv -o ServerAliveInterval=30 -o
StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
ConnectTimeout=35 -p 22 root@192.168.1.56 true
OpenSSH_6.7p1 Debian-5+deb8u3, OpenSSL 1.0.1t  3 May 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 192.168.1.56 [192.168.1.56] port 22.
debug2: fd 3 setting O_NONBLOCK
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug3: timeout: 34998 ms remain after connect
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u3
debug1: Remote protocol version 2.0, remote software version
OpenSSH_6.7p1 Debian-5+deb8u3
debug1: match: OpenSSH_6.7p1 Debian-5+deb8u3 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "192.168.1.56" from
file "/dev/null"
debug3: load_hostkeys: loaded 0 keys
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit:
curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit:
ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit:
aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit:
aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit:
umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit:
umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit:
curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
debug2: kex_parse_kexinit:
aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
debug2: kex_parse_kexinit:
aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
debug2: kex_parse_kexinit:
umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: kex_parse_kexinit:
umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: setup umac-64-etm@openssh.com
debug1: kex: server->client aes128-ctr umac-64-etm@openssh.com none
debug2: mac_setup: setup umac-64-etm@openssh.com
debug1: kex: client->server aes128-ctr umac-64-etm@openssh.com none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 81:cb:85:b4:3f:f5:26:22:a0:d6:63:85:c2:75:ca:d2
debug3: load_hostkeys: loading entries for host "192.168.1.56" from
file "/dev/null"
debug3: load_hostkeys: loaded 0 keys
Warning: Permanently added '192.168.1.56' (ECDSA) to the list of known hosts.
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /var/lib/jenkins/.ssh/id_rsa ((nil)),
debug2: key: /var/lib/jenkins/.ssh/id_dsa ((nil)),
debug2: key: /var/lib/jenkins/.ssh/id_ecdsa ((nil)),
debug2: key: /var/lib/jenkins/.ssh/id_ed25519 ((nil)),
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred
gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /var/lib/jenkins/.ssh/id_rsa
debug3: no such identity: /var/lib/jenkins/.ssh/id_rsa: No such file
or directory
debug1: Trying private key: /var/lib/jenkins/.ssh/id_dsa
debug3: no such identity: /var/lib/jenkins/.ssh/id_dsa: No such file
or directory
debug1: Trying private key: /var/lib/jenkins/.ssh/id_ecdsa
debug3: no such identity: /var/lib/jenkins/.ssh/id_ecdsa: No such file
or directory
debug1: Trying private key: /var/lib/jenkins/.ssh/id_ed25519
debug3: no such identity: /var/lib/jenkins/.ssh/id_ed25519: No such
file or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
debug2: we sent a password packet, wait for reply
debug1: Authentication succeeded (password).
Authenticated to 192.168.1.56 ([192.168.1.56]:22).
debug2: fd 4 setting O_NONBLOCK
debug2: fd 5 setting O_NONBLOCK
debug3: fd 6 is O_NONBLOCK
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug2: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: packet_set_tos: set IP_TOS 0x08
debug2: client_session2_setup: id 0
debug1: Sending environment.
debug3: Ignored env BUILD_URL
debug3: Ignored env variableEnvFunctional.bc0
debug3: Ignored env HUDSON_SERVER_COOKIE
debug3: Ignored env SHELL
debug3: Ignored env TERM
debug3: Ignored env BOARD_OVERLAY
debug3: Ignored env BUILD_TAG
debug3: Ignored env Rebuild
debug3: Ignored env TESTPLAN
debug3: Ignored env Device
debug3: Ignored env WORKSPACE
debug3: Ignored env FUEGO_ENGINE_PATH
debug3: Ignored env JOB_URL
debug3: Ignored env USER
debug3: Ignored env FUEGO_LOGS_PATH
debug3: Ignored env JENKINS_HOME
debug3: Ignored env PATH
debug3: Ignored env MAIL
debug3: Ignored env PWD
debug3: Ignored env HUDSON_URL
debug3: Ignored env JOB_NAME
debug3: Ignored env FUEGO_SCRIPTS_PATH
debug3: Ignored env JENKINS_URL
debug3: Ignored env BUILD_ID
debug3: Ignored env HOME
debug3: Ignored env SHLVL
debug3: Ignored env SSHPASS
debug3: Ignored env $JOB_NAME
debug3: Ignored env JENKINS_SERVER_COOKIE
debug3: Ignored env EXECUTOR_NUMBER
debug3: Ignored env NODE_LABELS
debug3: Ignored env LOGNAME
debug3: Ignored env HUDSON_HOME
debug3: Ignored env FUEGO_PARSER_PATH
debug3: Ignored env FUEGO_TESTS_PATH
debug3: Ignored env SLAVEJAR_URL
debug3: Ignored env NODE_NAME
debug3: Ignored env BUILD_NUMBER
debug3: Ignored env HUDSON_COOKIE
debug3: Ignored env Reboot
debug3: Ignored env Target_Cleanup
debug3: Ignored env DISTRIB
debug3: Ignored env _
debug1: Sending command: true
debug2: channel 0: request exec confirm 1
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel 0: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 0
debug2: exec request accepted on channel 0
debug2: channel 0: read<=0 rfd 4 len 0
debug2: channel 0: read failed
debug2: channel 0: close_read
debug2: channel 0: input open -> drain
debug2: channel 0: ibuf empty
debug2: channel 0: send eof
debug2: channel 0: input drain -> closed
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug2: channel 0: obuf empty
debug2: channel 0: close_write
debug2: channel 0: output drain -> closed
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug2: channel 0: rcvd close
debug3: channel 0: will not send data after close
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
  #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)

debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
debug3: fd 2 is not O_NONBLOCK
Transferred: sent 2468, received 1476 bytes, in 0.0 seconds
Bytes per second: sent 51514.3, received 30808.4
debug1: Exit status 0
+++ mkdir -p /home/jenkins/logs/Functional.bc/testlogs
+++ mkdir -p /home/jenkins/logs/Functional.bc/systemlogs
+++ mkdir -p /home/jenkins/logs/Functional.bc/devlogs
+++ firmware
+++ ov_get_firmware
++++ cmd 'uname -r | xargs echo '
++++ report_devlog 'cmd: uname -r | xargs echo '
++++ mkdir -p /home/jenkins/logs/Functional.bc/devlogs
++++ devlog_file=/home/jenkins/logs/Functional.bc/devlogs/xxx.2016-10-25_18-33-48.7.txt
++++ echo 'cmd: uname -r | xargs echo '
++++ ov_transport_cmd 'uname -r | xargs echo '
++++ case "$TRANSPORT" in
++++ sshpass -e ssh -vvv -o ServerAliveInterval=30 -o
StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
ConnectTimeout=35 -p 22 root@192.168.1.56 'uname -r | xargs echo '
OpenSSH_6.7p1 Debian-5+deb8u3, OpenSSL 1.0.1t  3 May 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 192.168.1.56 [192.168.1.56] port 22.
debug2: fd 3 setting O_NONBLOCK
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug3: timeout: 35000 ms remain after connect
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u3
debug1: Remote protocol version 2.0, remote software version
OpenSSH_6.7p1 Debian-5+deb8u3
debug1: match: OpenSSH_6.7p1 Debian-5+deb8u3 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "192.168.1.56" from
file "/dev/null"
debug3: load_hostkeys: loaded 0 keys
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit:
curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit:
ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit:
aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit:
aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit:
umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit:
umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit:
curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
debug2: kex_parse_kexinit:
aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
debug2: kex_parse_kexinit:
aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
debug2: kex_parse_kexinit:
umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: kex_parse_kexinit:
umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: setup umac-64-etm@openssh.com
debug1: kex: server->client aes128-ctr umac-64-etm@openssh.com none
debug2: mac_setup: setup umac-64-etm@openssh.com
debug1: kex: client->server aes128-ctr umac-64-etm@openssh.com none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 81:cb:85:b4:3f:f5:26:22:a0:d6:63:85:c2:75:ca:d2
debug3: load_hostkeys: loading entries for host "192.168.1.56" from
file "/dev/null"
debug3: load_hostkeys: loaded 0 keys
Warning: Permanently added '192.168.1.56' (ECDSA) to the list of known hosts.
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /var/lib/jenkins/.ssh/id_rsa ((nil)),
debug2: key: /var/lib/jenkins/.ssh/id_dsa ((nil)),
debug2: key: /var/lib/jenkins/.ssh/id_ecdsa ((nil)),
debug2: key: /var/lib/jenkins/.ssh/id_ed25519 ((nil)),
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred
gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /var/lib/jenkins/.ssh/id_rsa
debug3: no such identity: /var/lib/jenkins/.ssh/id_rsa: No such file
or directory
debug1: Trying private key: /var/lib/jenkins/.ssh/id_dsa
debug3: no such identity: /var/lib/jenkins/.ssh/id_dsa: No such file
or directory
debug1: Trying private key: /var/lib/jenkins/.ssh/id_ecdsa
debug3: no such identity: /var/lib/jenkins/.ssh/id_ecdsa: No such file
or directory
debug1: Trying private key: /var/lib/jenkins/.ssh/id_ed25519
debug3: no such identity: /var/lib/jenkins/.ssh/id_ed25519: No such
file or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
debug2: we sent a password packet, wait for reply
debug1: Authentication succeeded (password).
Authenticated to 192.168.1.56 ([192.168.1.56]:22).
debug2: fd 4 setting O_NONBLOCK
debug2: fd 5 setting O_NONBLOCK
debug2: fd 6 setting O_NONBLOCK
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug2: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: packet_set_tos: set IP_TOS 0x08
debug2: client_session2_setup: id 0
debug1: Sending environment.
debug3: Ignored env BUILD_URL
debug3: Ignored env variableEnvFunctional.bc0
debug3: Ignored env HUDSON_SERVER_COOKIE
debug3: Ignored env SHELL
debug3: Ignored env TERM
debug3: Ignored env BOARD_OVERLAY
debug3: Ignored env BUILD_TAG
debug3: Ignored env Rebuild
debug3: Ignored env TESTPLAN
debug3: Ignored env Device
debug3: Ignored env WORKSPACE
debug3: Ignored env FUEGO_ENGINE_PATH
debug3: Ignored env JOB_URL
debug3: Ignored env USER
debug3: Ignored env FUEGO_LOGS_PATH
debug3: Ignored env JENKINS_HOME
debug3: Ignored env PATH
debug3: Ignored env MAIL
debug3: Ignored env PWD
debug3: Ignored env HUDSON_URL
debug3: Ignored env JOB_NAME
debug3: Ignored env FUEGO_SCRIPTS_PATH
debug3: Ignored env JENKINS_URL
debug3: Ignored env BUILD_ID
debug3: Ignored env HOME
debug3: Ignored env SHLVL
debug3: Ignored env SSHPASS
debug3: Ignored env $JOB_NAME
debug3: Ignored env JENKINS_SERVER_COOKIE
debug3: Ignored env EXECUTOR_NUMBER
debug3: Ignored env NODE_LABELS
debug3: Ignored env LOGNAME
debug3: Ignored env HUDSON_HOME
debug3: Ignored env FUEGO_PARSER_PATH
debug3: Ignored env FUEGO_TESTS_PATH
debug3: Ignored env SLAVEJAR_URL
debug3: Ignored env NODE_NAME
debug3: Ignored env BUILD_NUMBER
debug3: Ignored env HUDSON_COOKIE
debug3: Ignored env Reboot
debug3: Ignored env Target_Cleanup
debug3: Ignored env DISTRIB
debug3: Ignored env _
debug1: Sending command: uname -r | xargs echo
debug2: channel 0: request exec confirm 1
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel 0: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 0
debug2: exec request accepted on channel 0
debug2: channel 0: read<=0 rfd 4 len 0
debug2: channel 0: read failed
debug2: channel 0: close_read
debug2: channel 0: input open -> drain
debug2: channel 0: ibuf empty
debug2: channel 0: send eof
debug2: channel 0: input drain -> closed
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug2: channel 0: obuf empty
debug2: channel 0: close_write
debug2: channel 0: output drain -> closed
debug2: channel 0: rcvd close
debug3: channel 0: will not send data after close
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
  #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)

debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
debug1: fd 2 clearing O_NONBLOCK
Transferred: sent 2484, received 1528 bytes, in 0.1 seconds
Bytes per second: sent 38178.4, received 23484.9
debug1: Exit status 0
+++ FW=4.7.4-IMX6.VAR_R17
+++ export FWVER=4.7.4-IMX6.VAR_R17
+++ FWVER=4.7.4-IMX6.VAR_R17
+++ cmd 'echo "Firmware revision:" 4.7.4-IMX6.VAR_R17'
+++ report_devlog 'cmd: echo "Firmware revision:" 4.7.4-IMX6.VAR_R17'
+++ mkdir -p /home/jenkins/logs/Functional.bc/devlogs
+++ devlog_file=/home/jenkins/logs/Functional.bc/devlogs/xxxx.2016-10-25_18-33-48.7.txt
+++ echo 'cmd: echo "Firmware revision:" 4.7.4-IMX6.VAR_R17'
+++ ov_transport_cmd 'echo "Firmware revision:" 4.7.4-IMX6.VAR_R17'
+++ case "$TRANSPORT" in
+++ sshpass -e ssh -vvv -o ServerAliveInterval=30 -o
StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
ConnectTimeout=35 -p 22 root@192.168.1.56 'echo "Firmware revision:"
4.7.4-IMX6.VAR_R17'
OpenSSH_6.7p1 Debian-5+deb8u3, OpenSSL 1.0.1t  3 May 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 192.168.1.56 [192.168.1.56] port 22.
debug2: fd 3 setting O_NONBLOCK

debug1: connect to address 192.168.1.56 port 22: Connection timed out
ssh: connect to host 192.168.1.56 port 22: Connection timed out
+++ abort_job 'Error while ROOTFS_FWVER command execution on target'
+++ set +x

*** ABORTED ***

Fuego error reason: Error while ROOTFS_FWVER command execution on target

Build step 'Execute shell' marked build as failure
Description set:
Performing Post build task...
Match found for :ssh : True
Logical operation result is TRUE
Running script  : source $FUEGO_SCRIPTS_PATH/functions.sh;
post_test $TESTDIR


Thanks,

Anirudh


On Tue, Oct 25, 2016 at 3:02 PM, Bird, Timothy <Tim.Bird@am.sony.com> wrote:

> anirudh,
>
> (Sorry for the duplicate message.  I forgot to copy the lists in my reply)
>
> These logs are not the full logs for each command.  They're missing the
> lines I'm interested in.
> In particular, in wanted to see the "debug1: sending command" and "debug1:
> Exit status" lines
> for each ssh invocation.
>
> See more comments/questions inline below.
>
> > -----Original Message-----
> > From: anirudh vasudevan [mailto:vasuanirudh@gmail.com]
> > Sent: Tuesday, October 25, 2016 8:37 AM
> > To: Bird, Timothy <Tim.Bird@am.sony.com>
> > Cc: ltsi-dev@lists.linuxfoundation.org; fuego@lists.linuxfoundation.org
> > Subject: Re: [LTSI-dev] ssh timeout problem - was RE: [Fuego] Fuego
> > installation
> >
> > Here is test you had asked for
> >
> > Host : ifconfig
> >
> > docker0   Link encap:Ethernet
> >           inet addr:172.17.0.1  Bcast:0.0.0.0  Mask:255.255.0.0
> >           UP BROADCAST MULTICAST  MTU:1500  Metric:1
> >           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
> >           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
> >           collisions:0 txqueuelen:0
> >           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
> >
> > eth0   Link encap:Ethernet
> >           inet addr:192.168.1.27  Bcast:192.168.1.255  Mask:255.255.255.0
> >           inet6 addr: fe80::96a4:8e81:cf71:7a2c/64 Scope:Link
> >           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
> >           RX packets:5360 errors:0 dropped:0 overruns:0 frame:0
> >           TX packets:4990 errors:0 dropped:0 overruns:0 carrier:0
> >           collisions:0 txqueuelen:1000
> >           RX bytes:2956455 (2.9 MB)  TX bytes:648438 (648.4 KB)
> >           Interrupt:17
> >
> >
> >
> > Target: VAR
> >
> >
> >
> > ap0       Link encap:Ethernet
> >           inet6 addr: fe80::be66:41ff:fe30:a456/64 Scope:Link
> >           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
> >           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
> >           TX packets:286 errors:0 dropped:0 overruns:0 carrier:0
> >           collisions:0 txqueuelen:1000
> >           RX bytes:0 (0.0 B)  TX bytes:17772 (17.3 KiB)
> >
> > br0       Link encap:Ethernet
> >           inet addr:10.15.140.230  Bcast:10.255.255.255  Mask:255.0.0.0
> >           inet6 addr: fe80::8625:3fff:fe0f:8ce6/64 Scope:Link
> >           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
> >           RX packets:3189 errors:0 dropped:0 overruns:0 frame:0
> >           TX packets:4094 errors:0 dropped:0 overruns:0 carrier:0
> >           collisions:0 txqueuelen:1000
> >           RX bytes:215833 (210.7 KiB)  TX bytes:4057937 (3.8 MiB)
> >
> > eth0      Link encap:Ethernet
> >           inet addr:192.168.1.56  Bcast:192.168.1.255  Mask:255.255.255.0
> >           inet6 addr: fe80::fadc:7aff:fe05:6d00/64 Scope:Link
> >           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
> >           RX packets:4988 errors:0 dropped:0 overruns:0 frame:0
> >           TX packets:3993 errors:0 dropped:0 overruns:0 carrier:0
> >           collisions:0 txqueuelen:1000
> >           RX bytes:4402058 (4.1 MiB)  TX bytes:372918 (364.1 KiB)
> >
> >
> >
> >
> > ssh command with verbose which you told me to run outside and inside of
> > the docker container works, am able to ssh to the target by providing the
> > password.
>
>
> OK
> >
> >
> > Below results are the ones from the console after I added verbose on the
> > base-params.fuegoclass scripts.
> >
> >
> > There are series of ssh commands which takes place and it is successfully
> > getting the firmware version from the board it is failing after which it
> fetches
> > the firmware version.
> >
> > +++ target_cleanup Functional.bc
> > +++ cmd 'rm -rf /home/a/* /tmp/* /home/a/.local'
> > +++ report_devlog 'cmd: rm -rf /home/a/* /tmp/* /home/a/.local'
> > +++ mkdir -p /home/jenkins/logs/Functional.bc/devlogs
> > +++ echo 'cmd: rm -rf /home/a/* /tmp/* /home/a/.local'
> > +++ ov_transport_cmd 'rm -rf /home/a/* /tmp/* /home/a/.local'
> > +++ case "$TRANSPORT" in
> > +++ sshpass -e ssh -vvv -o ServerAliveInterval=30 -o
> > StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
> > ConnectTimeout=35 -p 22 root@192.168.1.56 <mailto:root@192.168.1.56>
> > 'rm -rf /home/a/* /tmp/* /home/a/.local'
> > OpenSSH_6.7p1 Debian-5+deb8u3, OpenSSL 1.0.1t  3 May 2016
> > debug1: Reading configuration data /etc/ssh/ssh_config
> > debug1: /etc/ssh/ssh_config line 19: Applying options for *
> > debug2: ssh_connect: needpriv 0
> > debug1: Connecting to 192.168.1.56 [192.168.1.56] port 22.
> > debug2: fd 3 setting O_NONBLOCK
> > debug1: fd 3 clearing O_NONBLOCK
> > debug1: Connection established.
> > debug3: timeout: 35000 ms remain after connect
> > debug1: key_load_public: No such file or directory
> > debug1: identity file /var/lib/jenkins/.ssh/id_rsa type -1
> > debug1: key_load_public: No such file or directory
> > debug1: identity file /var/lib/jenkins/.ssh/id_rsa-cert type -1
> > debug1: key_load_public: No such file or directory
> > debug1: identity file /var/lib/jenkins/.ssh/id_dsa type -1
> > debug1: key_load_public: No such file or directory
> > debug1: identity file /var/lib/jenkins/.ssh/id_dsa-cert type -1
> > debug1: key_load_public: No such file or directory
> > debug1: identity file /var/lib/jenkins/.ssh/id_ecdsa type -1
> > debug1: key_load_public: No such file or directory
> > debug1: identity file /var/lib/jenkins/.ssh/id_ecdsa-cert type -1
> > debug1: key_load_public: No such file or directory
> > debug1: identity file /var/lib/jenkins/.ssh/id_ed25519 type -1
> > debug1: key_load_public: No such file or directory
> > debug1: identity file /var/lib/jenkins/.ssh/id_ed25519-cert type -1
> > debug1: Enabling compatibility mode for protocol 2.0
> > debug1: Local version string SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u3
> > debug1: Remote protocol version 2.0, remote software version
> > OpenSSH_6.7p1 Debian-5+deb8u3
> > debug1: match: OpenSSH_6.7p1 Debian-5+deb8u3 pat OpenSSH* compat
> > 0x04000000
> > debug2: fd 3 setting O_NONBLOCK
> > debug3: load_hostkeys: loading entries for host "192.168.1.56" from file
> > "/dev/null"
> > debug3: load_hostkeys: loaded 0 keys
> > debug1: SSH2_MSG_KEXINIT sent
> > debug1: SSH2_MSG_KEXINIT received
> > debug2: kex_parse_kexinit:
> > debug2: kex_parse_kexinit:
> > debug2: kex_parse_kexinit: first_kex_follows 0
> > debug2: kex_parse_kexinit: reserved 0
> > debug2: kex_parse_kexinit: curve25519-sha256@libssh.org
> > <mailto:curve25519-sha256@libssh.org> ,ecdh-sha2-nistp256,ecdh-sha2-
> > nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-
> > hellman-group14-sha1
> > debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-
> > ed25519
> > debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes128-
> > gcm@openssh.com <mailto:aes128-gcm@openssh.com> ,aes256-
> > gcm@openssh.com <mailto:aes256-gcm@openssh.com> ,chacha20-
> > poly1305@openssh.com <mailto:chacha20-poly1305@openssh.com>
> > debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes128-
> > gcm@openssh.com <mailto:aes128-gcm@openssh.com> ,aes256-
> > gcm@openssh.com <mailto:aes256-gcm@openssh.com> ,chacha20-
> > poly1305@openssh.com <mailto:chacha20-poly1305@openssh.com>
> > debug2: kex_parse_kexinit: umac-64-etm@openssh.com <mailto:umac-64-
> > etm@openssh.com> ,umac-128-etm@openssh.com <mailto:umac-128-
> > etm@openssh.com> ,hmac-sha2-256-etm@openssh.com <mailto:hmac-
> > sha2-256-etm@openssh.com> ,hmac-sha2-512-etm@openssh.com
> > <mailto:hmac-sha2-512-etm@openssh.com> ,hmac-sha1-
> > etm@openssh.com <mailto:hmac-sha1-etm@openssh.com> ,umac-
> > 64@openssh.com <mailto:umac-64@openssh.com> ,umac-
> > 128@openssh.com <mailto:umac-128@openssh.com> ,hmac-sha2-
> > 256,hmac-sha2-512,hmac-sha1
> > debug2: kex_parse_kexinit: umac-64-etm@openssh.com <mailto:umac-64-
> > etm@openssh.com> ,umac-128-etm@openssh.com <mailto:umac-128-
> > etm@openssh.com> ,hmac-sha2-256-etm@openssh.com <mailto:hmac-
> > sha2-256-etm@openssh.com> ,hmac-sha2-512-etm@openssh.com
> > <mailto:hmac-sha2-512-etm@openssh.com> ,hmac-sha1-
> > etm@openssh.com <mailto:hmac-sha1-etm@openssh.com> ,umac-
> > 64@openssh.com <mailto:umac-64@openssh.com> ,umac-
> > 128@openssh.com <mailto:umac-128@openssh.com> ,hmac-sha2-
> > 256,hmac-sha2-512,hmac-sha1
> > debug2: kex_parse_kexinit: none,zlib@openssh.com
> > <mailto:zlib@openssh.com>
> > debug2: kex_parse_kexinit: none,zlib@openssh.com
> > <mailto:zlib@openssh.com>
> > debug2: kex_parse_kexinit:
> > debug2: kex_parse_kexinit:
> > debug2: kex_parse_kexinit: first_kex_follows 0
> > debug2: kex_parse_kexinit: reserved 0
> > debug2: mac_setup: setup umac-64-etm@openssh.com <mailto:umac-64-
> > etm@openssh.com>
> > debug1: kex: server->client aes128-ctr umac-64-etm@openssh.com
> > <mailto:umac-64-etm@openssh.com>  none
> > debug2: mac_setup: setup umac-64-etm@openssh.com <mailto:umac-64-
> > etm@openssh.com>
> > debug1: kex: client->server aes128-ctr umac-64-etm@openssh.com
> > <mailto:umac-64-etm@openssh.com>  none
> > debug1: sending SSH2_MSG_KEX_ECDH_INIT
> > debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
> > debug1: Server host key: ECDSA
> > 81:cb:85:b4:3f:f5:26:22:a0:d6:63:85:c2:75:ca:d2
> > debug3: load_hostkeys: loading entries for host "192.168.1.56" from file
> > "/dev/null"
> > debug3: load_hostkeys: loaded 0 keys
> > Warning: Permanently added '192.168.1.56' (ECDSA) to the list of known
> > hosts.
> > debug2: kex_derive_keys
> > debug2: set_newkeys: mode 1
> > debug1: SSH2_MSG_NEWKEYS sent
> > debug1: expecting SSH2_MSG_NEWKEYS
> > debug2: set_newkeys: mode 0
> > debug1: SSH2_MSG_NEWKEYS received
> > debug1: SSH2_MSG_SERVICE_REQUEST sent
> > debug2: service_accept: ssh-userauth
> > debug1: SSH2_MSG_SERVICE_ACCEPT received
> > debug2: key: /var/lib/jenkins/.ssh/id_rsa ((nil)),
> > debug2: key: /var/lib/jenkins/.ssh/id_dsa ((nil)),
> > debug2: key: /var/lib/jenkins/.ssh/id_ecdsa ((nil)),
> > debug2: key: /var/lib/jenkins/.ssh/id_ed25519 ((nil)),
> > debug1: Authentications that can continue: publickey,password
> > debug3: start over, passed a different list publickey,password
> > debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-
> > interactive,password
> > debug3: authmethod_lookup publickey
> > debug3: remaining preferred: keyboard-interactive,password
> > debug3: authmethod_is_enabled publickey
> > debug1: Next authentication method: publickey
> > debug1: Trying private key: /var/lib/jenkins/.ssh/id_rsa
> > debug3: no such identity: /var/lib/jenkins/.ssh/id_rsa: No such file or
> > directory
> > debug1: Trying private key: /var/lib/jenkins/.ssh/id_dsa
> > debug3: no such identity: /var/lib/jenkins/.ssh/id_dsa: No such file or
> > directory
> > debug1: Trying private key: /var/lib/jenkins/.ssh/id_ecdsa
> > debug3: no such identity: /var/lib/jenkins/.ssh/id_ecdsa: No such file
> or
> > directory
> > debug1: Trying private key: /var/lib/jenkins/.ssh/id_ed25519
> > debug3: no such identity: /var/lib/jenkins/.ssh/id_ed25519: No such
> file or
> > directory
> > debug2: we did not send a packet, disable method
> > debug3: authmethod_lookup password
> > debug3: remaining preferred: ,password
> > debug3: authmethod_is_enabled password
> > debug1: Next authentication method: password
> > debug2: we sent a password packet, wait for reply
> > debug1: Authentication succeeded (password).
> > Authenticated to 192.168.1.56 ([192.168.1.56]:22).
> > debug2: fd 4 setting O_NONBLOCK
> > debug2: fd 5 setting O_NONBLOCK
> > debug3: fd 6 is O_NONBLOCK
> > debug1: channel 0: new [client-session]
> > debug3: ssh_session2_open: channel_new: 0
> > debug2: channel 0: send open
> > debug1: Requesting no-more-sessions@openssh.com <mailto:no-more-
> > sessions@openssh.com>
> > debug1: Entering interactive session.
> > debug2: callback start
> > debug2: fd 3 setting TCP_NODELAY
> > debug3: packet_set_tos: set IP_TOS 0x08
> > debug2: client_session2_setup: id 0
> > debug1: Sending environment.
> > debug3: Ignored env BUILD_URL
> > debug3: Ignored env variableEnvFunctional.bc0
> > debug3: Ignored env HUDSON_SERVER_COOKIE
> > debug3: Ignored env SHELL
> > debug3: Ignored env TERM
> > debug3: Ignored env BOARD_OVERLAY
> > debug3: Ignored env BUILD_TAG
> > debug3: Ignored env Rebuild
> > debug3: Ignored env TESTPLAN
> > debug3: Ignored env Device
> > debug3: Ignored env WORKSPACE
> > debug3: Ignored env FUEGO_ENGINE_PATH
> > debug3: Ignored env JOB_URL
> > debug3: Ignored env USER
> > debug3: Ignored env FUEGO_LOGS_PATH
> > debug3: Ignored env JENKINS_HOME
> > debug3: Ignored env PATH
> > debug3: Ignored env MAIL
> > debug3: Ignored env PWD
> > debug3: Ignored env HUDSON_URL
> > debug3: Ignored env JOB_NAME
> > debug3: Ignored env FUEGO_SCRIPTS_PATH
> > debug3: Ignored env JENKINS_URL
> > debug3: Ignored env BUILD_ID
> > debug3: Ignored env HOME
> > debug3: Ignored env SHLVL
> > debug3: Ignored env SSHPASS
> > debug3: Ignored env $JOB_NAME
> > debug3: Ignored env JENKINS_SERVER_COOKIE
> > debug3: Ignored env EXECUTOR_NUMBER
> > debug3: Ignored env NODE_LABELS
> > debug3: Ignored env LOGNAME
> > debug3: Ignored env HUDSON_HOME
> > debug3: Ignored env FUEGO_PARSER_PATH
> > debug3: Ignored env FUEGO_TESTS_PATH
> > debug3: Ignored env SLAVEJAR_URL
> > debug3: Ignored env NODE_NAME
> > debug3: Ignored env BUILD_NUMBER
> > debug3: Ignored env HUDSON_COOKIE
> > debug3: Ignored env Reboot
> > debug3: Ignored env Target_Cleanup
> > debug3: Ignored env DISTRIB
> > debug3: Ignored env _
> > debug1: Sending command: rm -rf /home/a/* /tmp/* /home/a/.local
> > debug2: channel 0: request exec confirm 1
> > debug2: callback done
> > debug2: channel 0: open confirm rwindow 0 rmax 32768
> > debug2: channel 0: rcvd adjust 2097152
> > debug2: channel_input_status_confirm: type 99 id 0
> > debug2: exec request accepted on channel 0
> > debug2: channel 0: read<=0 rfd 4 len 0
> > debug2: channel 0: read failed
> > debug2: channel 0: close_read
> > debug2: channel 0: input open -> drain
> > debug2: channel 0: ibuf empty
> > debug2: channel 0: send eof
> > debug2: channel 0: input drain -> closed
> > debug2: channel 0: rcvd eof
> > debug2: channel 0: output open -> drain
> > debug2: channel 0: obuf empty
> > debug2: channel 0: close_write
> > debug2: channel 0: output drain -> closed
> > debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
> > debug2: channel 0: rcvd close
> > debug3: channel 0: will not send data after close
> > debug2: channel 0: almost dead
> > debug2: channel 0: gc: notify user
> > debug2: channel 0: gc: user detached
> > debug2: channel 0: send close
> > debug2: channel 0: is dead
> > debug2: channel 0: garbage collecting
> > debug1: channel 0: free: client-session, nchannels 1
> > debug3: channel 0: status: The following connections are open:
> >   #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)
> >
> > debug1: fd 0 clearing O_NONBLOCK
> > debug1: fd 1 clearing O_NONBLOCK
> > debug3: fd 2 is not O_NONBLOCK
> > Transferred: sent 2500, received 1476 bytes, in 0.1 seconds
> > Bytes per second: sent 32307.3, received 19074.2
> >
>
> What is the exit status of this command?
>
> What about for the 'true' command, which is also run during the pre-test
> phase?
>
> >
> > SSH command failing here
> Is this commenting on the ssh operation above or below?  I presume below...
>
> >
> >
> > +++ sshpass -e ssh -vvv -o ServerAliveInterval=30 -o
> > StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
> > ConnectTimeout=35 -p 22 root@192.168.1.56 <mailto:root@192.168.1.56>
> > 'echo "Firmware revision:" 4.7.4-IMX6.VAR_R17'
> > OpenSSH_6.7p1 Debian-5+deb8u3, OpenSSL 1.0.1t  3 May 2016
> > debug1: Reading configuration data /etc/ssh/ssh_config
> > debug1: /etc/ssh/ssh_config line 19: Applying options for *
> > debug2: ssh_connect: needpriv 0
> > debug1: Connecting to 192.168.1.56 [192.168.1.56] port 22.
> > debug2: fd 3 setting O_NONBLOCK
> >
> > debug1: connect to address 192.168.1.56 port 22: Connection timed out
> > ssh: connect to host 192.168.1.56 port 22: Connection timed out
> > +++ abort_job 'Error while ROOTFS_FWVER command execution on target'
> > +++ set +x
> >
>
> There are 3 commands before the 'echo "Firmware revision" ...' command.
>  * rm -rf /home/a/* /tmp/* /home/a/.local
>  * true
>  * uname -r | xargs echo
>
> I presume that all these commands run OK on the target, and that:
>  * 'echo "Firmware revision:" 4.7.4-IMX6.VAR_R17'
> is the first one that times out?
>
> Is this consistent?  Does the timeout occur on any other commands or is it
> always this one?
>
> Given what I see above, I think you have an honest-to-goodness networking
> problem between your host and your target.  The TCP connection (before
> doing all the authentication stuff for SSH) is not getting established on
> this execution
> of ssh.  You should be seeing the following two lines, after the lines
> trying to
> stabling the connection:
> debug1: fd 3 clearing O_NONBLOCK
> debug1: Connection established
>
> After this error occurs, is there anything in the kernel messages on the
> target about
> the Ethernet link dropping?
>
> If you run the test again, do the first three commands work again?
>
> If the board has fundamental networking problems (ie the networking is
> flaky),
> then we can't really do any testing with it, with current Fuego.  We have
> plans
> to work on support for operating on boards that only have serial console
> support,
> but that work won't be done for a few months.
>
> Sorry to not be able to help more.
>  -- Tim
>
>

[-- Attachment #2: Type: text/html, Size: 71771 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2016-10-25 19:20 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-10-21 19:44 [Fuego] ssh timeout problem - was RE: [LTSI-dev] Fuego installation Bird, Timothy
2016-10-24 14:45 ` anirudh vasudevan
2016-10-25  1:10   ` [Fuego] [LTSI-dev] ssh timeout problem - was " Bird, Timothy
2016-10-25 15:36     ` anirudh vasudevan
2016-10-25 19:02       ` Bird, Timothy
2016-10-25 19:20         ` anirudh vasudevan

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.