[Fuego] pdudaemon support for Fuego

[Fuego] pdudaemon support for Fuego

[Daniel Sangorrin]

Hello Tim,

One of the results from the Automated Testing Summit in Scotland was
an agreement on using pdudaemon to control PDUs (Power Distribution Units).

This patch adds such support:
[PATCH] ftc: add a new command to interact with pdudaemon

This is a very short quickstart:

- Prepare pdudaemon (for an ip9258 PDU in this example)

host$ git clone https://github.com/pdudaemon/pdudaemon
[Note] I have made a few changes and sent a pull request, so temporarily you may want to use https://github.com/sangorrin/pdudaemon
host$ cd pdudaemon
host$ vi share/pdudaemon.conf
{
	"daemon": {
		"hostname": "0.0.0.0",
		"port": 16421,
		"dbname": "pdudaemon",
		"logging_level": "DEBUG",
		"listener": "http"
	},
	"pdus": {
		"192.168.11.68": {
			"driver": "ip9258"
		}
	}
}
host$ sudo docker build -t pdudaemon --build-arg HTTP_PROXY=$http_proxy -f Dockerfile.dockerhub .
host$ docker run --rm -it -e http_proxy=$http_proxy -e https_proxy=$https_proxy -e NO_PROXY="$no_proxy" --net="host" pdudaemon:latest

- Check that it works manually

$ curl "http://localhost:16421/power/control/on?hostname=192.168.11.68&port=1"
$ curl "http://localhost:16421/power/control/on?hostname=192.168.11.68&port=1"

- Check that it works from Fuego

$ vi myboard.board
	PDU_HOSTNAME="192.168.11.68"
	PDU_PORT="3"
$ ftc pdu -b myboard.board --command on
$ ftc pdu -b myboard.board --command off

If you have any question about pdudaemon please let me know.

[Note] Please feel free to add this to the wiki.

Thanks,
Daniel

[Fuego] [PATCH] ftc: add a new command to interact with pdudaemon

[Daniel Sangorrin]

This patch adds the "pdu" command to send requests for pdudaemon
to power-on/off or reboot a board.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 scripts/ftc               | 80 +++++++++++++++++++++++++++++++++++++++++++++++
 scripts/ftc_completion.sh |  5 +++
 2 files changed, 85 insertions(+)

diff --git a/scripts/ftc b/scripts/ftc
index 7153c01..30ea470 100755
--- a/scripts/ftc
+++ b/scripts/ftc
@@ -426,6 +426,29 @@ A message and the exit code indicate if the resource was released.
 
    ex: ftc release-resource -b beaglebone"""),
 
+"pdu": ("Send a request to pdudaemon.",
+    """Usage: ftc pdu -b <board> --command on|off|reboot [--pdudaemon_hostname localhost --pdudaemon_port 16421]
+
+This command is used to send a request for pdudaemon to power-on, power-off or reboot a board.
+
+The command accepts these arguments:
+ - board file: bbb, raspberrypi3, etc (see ftc list-boards)
+ - command: on|off|reboot
+ - pdudaemon_hostname: hostname or IP address where pdudaemon is running (default: localhost)
+ - pdudaemon_port: port where pdudaemon is listening (default: 16421)
+   [Note] make sure that pdudaemon is configured to accept http requests
+
+Additionally, you need to define two variables in your board file:
+ - PDU_HOSTNAME: the PDU hostname or IP address used in the pdudaemon configuration file
+ - PDU_PORT: the PDU port number (a plug port, not a tcp port)
+
+You need to install pdudaemon by yourself
+  $ git clone https://github.com/pdudaemon/pdudaemon
+  $ cd pdudaemon
+  $ less README.md
+
+Example: ftc pdu -b raspberrypi3 --command on"""),
+
 "package-test": ("Package a test.",
     """Usage: ftc package-test <test_name> [options]
 
@@ -4783,6 +4806,59 @@ def do_release_resource(conf, options):
 
     return 0
 
+def do_pdu(bvars, conf, options):
+    # pdudaemon hostname
+    pdudaemon_hostname = 'localhost'
+    if '--pdudaemon_hostname' in options:
+        try:
+            pdudaemon_hostname = options[options.index('--pdudaemon_hostname')+1]
+        except IndexError:
+            error_out('Value not provided after --pdudaemon_hostname')
+        options.remove(pdudaemon_hostname)
+        options.remove('--pdudaemon_hostname')
+
+    # pdudaemon port
+    pdudaemon_port = '16421'
+    if '--pdudaemon_port' in options:
+        try:
+            pdudaemon_port = options[options.index('--pdudaemon_port')+1]
+        except IndexError:
+            error_out('Value not provided after --pdudaemon_port')
+        options.remove(pdudaemon_port)
+        options.remove('--pdudaemon_port')
+
+    # pdu hostname
+    if 'PDU_HOSTNAME' in bvars:
+        pdu_hostname = bvars['PDU_HOSTNAME']
+    else:
+        error_out('PDU_HOSTNAME not specified in board file')
+
+    # pdu port
+    if 'PDU_PORT' in bvars:
+        pdu_port = bvars['PDU_PORT']
+    else:
+        error_out('PDU_PORT not specified in board file')
+
+    # command (on|off|reboot)
+    if '--command' in options:
+        try:
+            command = options[options.index('--command')+1]
+        except IndexError:
+            error_out('Value not provided after --command')
+        options.remove(command)
+        options.remove('--command')
+        if command not in ['on', 'off', 'reboot']:
+            error_out('Unrecognized PDU command %s' % command)
+    else:
+        error_out('Command not supplied')
+
+    query='&'.join(['hostname='+pdu_hostname, 'port='+pdu_port])
+    url = 'http://%s:%s/power/control/%s?%s' % (pdudaemon_hostname, pdudaemon_port, command, query)
+    resp = requests.get(url)
+    if resp.status_code != 200:
+        error_out("pdudaemon did not accept the request")
+    return 0
+
 def main():
     # use global module names
     global re, time, copy2, subprocess, signal, fcntl, requests, json
@@ -5094,6 +5170,10 @@ def main():
         do_delete_var(bvars, options)
         sys.exit(0)
 
+    if command == "pdu":
+        rcode = do_pdu(bvars, conf, options)
+        sys.exit(rcode)
+
     error_out("Unknown command %s" % command)
 
 
diff --git a/scripts/ftc_completion.sh b/scripts/ftc_completion.sh
index 984e003..2cfb77a 100755
--- a/scripts/ftc_completion.sh
+++ b/scripts/ftc_completion.sh
@@ -55,6 +55,7 @@ _ftc()
         ["list-tests"]="-q"
         ["package-run"]="-o -f"
         ["package-test"]="-o -f"
+        ["pdu"]="-b --command --pdudaemon_hostname --pdudaemon_port"
         ["put-request"]="-R -s"
         ["put-run"]=""
         ["put-test"]=""
@@ -123,6 +124,10 @@ _ftc()
             COMPREPLY=( $( compgen -W "true false" -- "$cur" ) )
             return 0
             ;;
+        '--command')
+            COMPREPLY=( $( compgen -W "on off reboot" -- "$cur" ) )
+            return 0
+            ;;
         'put-run')
             COMPREPLY=( $( compgen -W "$(ftc list-runs -q)" -- "$cur" ) )
             return 0
-- 
2.7.4

Re: [Fuego] pdudaemon support for Fuego

[Tim.Bird]

Thanks for working on this.  I have a few questions.
See below.

> -----Original Message-----
> From: Daniel Sangorrin
> 
> Hello Tim,
> 
> One of the results from the Automated Testing Summit in Scotland was
> an agreement on using pdudaemon to control PDUs (Power Distribution
> Units).
> 
> This patch adds such support:
> [PATCH] ftc: add a new command to interact with pdudaemon
> 
> This is a very short quickstart:
> 
> - Prepare pdudaemon (for an ip9258 PDU in this example)
> 
> host$ git clone https://github.com/pdudaemon/pdudaemon
> [Note] I have made a few changes and sent a pull request, so temporarily
> you may want to use https://github.com/sangorrin/pdudaemon
> host$ cd pdudaemon
> host$ vi share/pdudaemon.conf
> {
> 	"daemon": {
> 		"hostname": "0.0.0.0",
> 		"port": 16421,
> 		"dbname": "pdudaemon",
> 		"logging_level": "DEBUG",
> 		"listener": "http"
> 	},
> 	"pdus": {
> 		"192.168.11.68": {
> 			"driver": "ip9258"
> 		}
> 	}
> }
> host$ sudo docker build -t pdudaemon --build-arg HTTP_PROXY=$http_proxy
> -f Dockerfile.dockerhub .
> host$ docker run --rm -it -e http_proxy=$http_proxy -e
> https_proxy=$https_proxy -e NO_PROXY="$no_proxy" --net="host"
> pdudaemon:latest
> 
> - Check that it works manually

OK - I'm just trying to double-check my understanding of the situation here.
So, apparently the way to use pdudaemon is to build a docker container for it,
and communicate with the daemon inside that container via curl (that is, using
HTTP over a network). Is that right?

Would it make sense to clone the pdudaemon code into the Fuego docker container,
and start the daemon there, rather than in its own container? 

I can think of pros and cons to each approach.

Pros:
  * It eliminates separate steps for installing and starting the daemon (well, technically
      it would combine the steps with Fuego's container creation)
      * In this case, we might try to put the pdudaemon.conf into Fuego's fuego-ro/conf directory.

Cons:
  * It may be confusing to disable a Fuego-integrated pdudaemon, if someone already
      has one installed and is using that one (you might transfer the configuration to the Fuego
      instance of the daemon, or you might disable the integrated one and just configure
      Fuego to use the pre-existing one.

At the Automated Testing Summit, we didn't have time to discuss how to coordinate
how different pieces would fit together, particularly when they are in their own containers.
I'm not sure if this is something that needs to be done on a case-by-case basis, or if
there is some design pattern we should be following for this, that we want to encourage
tools to adopt to make this easier.

In the case of 'ttc', which provides similar functionality, it's integrated into the Dockerfile,
and into Fuego's transport layer.  And the IPC used is Unix command line invocation.


> 
> $ curl
> "http://localhost:16421/power/control/on?hostname=192.168.11.68&port=1
> "
> $ curl
> "http://localhost:16421/power/control/on?hostname=192.168.11.68&port=1
> "

Does the 'pdudaemon' project include their own command line utility?
If so, I'd rather use that than add something to Fuego.
 
> - Check that it works from Fuego
> 
> $ vi myboard.board
> 	PDU_HOSTNAME="192.168.11.68"
> 	PDU_PORT="3"
> $ ftc pdu -b myboard.board --command on
> $ ftc pdu -b myboard.board --command off
> 
> If you have any question about pdudaemon please let me know.

See above.
 -- Tim

Re: [Fuego] [PATCH] ftc: add a new command to interact with pdudaemon

[Tim.Bird]

Here are some overall comments:

Rather than add a new command to ftc for managing pdudaemon operations,
I was expecting to install support for pdudaemon at Fuego's board management (/board_control)
layer.  That is, I was expecting to see something like this:

diff --git a/overlays/base/base-board.fuegoclass b/overlays/base/base-board.fuegoclass
index ae35464..8f34922 100644
--- a/overlays/base/base-board.fuegoclass
+++ b/overlays/base/base-board.fuegoclass
@@ -299,6 +299,9 @@ function ov_board_control_reboot() {
   "ttc")
     $TTC $TTC_TARGET reboot
     ;;
+  "pdudaemon"):
+    curl "http://$PDUDAEMON_HOST:$PDUDAEMON_PORT/power/control/reboot?hostname=$PDU_HOSTNAME&port=$PDU_PORT"
+    ;;
   *)
     abort_job "Error reason: unsupported BOARD_CONTROL ${BOARD_CONTROL}"
     ;;
----

Note that I don't know if pdudaemon supports the 'reboot' control option, or if I would have
to synthesize something by sending 'on' and 'off'.  But you get the idea, I think.
I'm not sure if the hostname in the URL is supposed to be the board name or the pdu name,
but presumably PDUDAEMON_HOST and PDUDAEMON_PORT would come from 
fuego.conf, and PDU_HOSTNAME and PDU_PORT would come from the board file?

If more fine-grained control is desired, than just the 'reboot' operation, 
we could add more ov_board_control() functions if needed, and have Fuego use those.
e.g. ov_board_control_on() and ov_board_control_off().

Once we've plugged pdudaemon into the board layer, I'd prefer to have a generic
set of operations that someone can use from ftc, like:
   ftc reboot-board -b <board>
   ftc power-on-board -b <board>
   ftc power-off-board -b <board>
and maybe even
   ftc show-status-of-board <board>

In some labs, there may be a layer between pdudaemon and Fuego (for example libvirt,
or labgrid or even LAVA).

I think I'd rather have an abstraction layer between Fuego and pdudaemon, rather than
integrating it directly into our user interface.

Thoughts?
 -- tim


> -----Original Message-----
> From: Daniel Sangorrin
> 
> This patch adds the "pdu" command to send requests for pdudaemon
> to power-on/off or reboot a board.
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  scripts/ftc               | 80
> +++++++++++++++++++++++++++++++++++++++++++++++
>  scripts/ftc_completion.sh |  5 +++
>  2 files changed, 85 insertions(+)
> 
> diff --git a/scripts/ftc b/scripts/ftc
> index 7153c01..30ea470 100755
> --- a/scripts/ftc
> +++ b/scripts/ftc
> @@ -426,6 +426,29 @@ A message and the exit code indicate if the resource
> was released.
> 
>     ex: ftc release-resource -b beaglebone"""),
> 
> +"pdu": ("Send a request to pdudaemon.",
> +    """Usage: ftc pdu -b <board> --command on|off|reboot [--
> pdudaemon_hostname localhost --pdudaemon_port 16421]
> +
> +This command is used to send a request for pdudaemon to power-on,
> power-off or reboot a board.
> +
> +The command accepts these arguments:
> + - board file: bbb, raspberrypi3, etc (see ftc list-boards)
> + - command: on|off|reboot
> + - pdudaemon_hostname: hostname or IP address where pdudaemon is
> running (default: localhost)
> + - pdudaemon_port: port where pdudaemon is listening (default: 16421)
> +   [Note] make sure that pdudaemon is configured to accept http requests
> +
> +Additionally, you need to define two variables in your board file:
> + - PDU_HOSTNAME: the PDU hostname or IP address used in the
> pdudaemon configuration file
> + - PDU_PORT: the PDU port number (a plug port, not a tcp port)
> +
> +You need to install pdudaemon by yourself
> +  $ git clone https://github.com/pdudaemon/pdudaemon
> +  $ cd pdudaemon
> +  $ less README.md
> +
> +Example: ftc pdu -b raspberrypi3 --command on"""),
> +
>  "package-test": ("Package a test.",
>      """Usage: ftc package-test <test_name> [options]
> 
> @@ -4783,6 +4806,59 @@ def do_release_resource(conf, options):
> 
>      return 0
> 
> +def do_pdu(bvars, conf, options):
> +    # pdudaemon hostname
> +    pdudaemon_hostname = 'localhost'
> +    if '--pdudaemon_hostname' in options:
> +        try:
> +            pdudaemon_hostname = options[options.index('--
> pdudaemon_hostname')+1]
> +        except IndexError:
> +            error_out('Value not provided after --pdudaemon_hostname')
> +        options.remove(pdudaemon_hostname)
> +        options.remove('--pdudaemon_hostname')
> +
> +    # pdudaemon port
> +    pdudaemon_port = '16421'
> +    if '--pdudaemon_port' in options:
> +        try:
> +            pdudaemon_port = options[options.index('--pdudaemon_port')+1]
> +        except IndexError:
> +            error_out('Value not provided after --pdudaemon_port')
> +        options.remove(pdudaemon_port)
> +        options.remove('--pdudaemon_port')
> +
> +    # pdu hostname
> +    if 'PDU_HOSTNAME' in bvars:
> +        pdu_hostname = bvars['PDU_HOSTNAME']
> +    else:
> +        error_out('PDU_HOSTNAME not specified in board file')
> +
> +    # pdu port
> +    if 'PDU_PORT' in bvars:
> +        pdu_port = bvars['PDU_PORT']
> +    else:
> +        error_out('PDU_PORT not specified in board file')
> +
> +    # command (on|off|reboot)
> +    if '--command' in options:
> +        try:
> +            command = options[options.index('--command')+1]
> +        except IndexError:
> +            error_out('Value not provided after --command')
> +        options.remove(command)
> +        options.remove('--command')
> +        if command not in ['on', 'off', 'reboot']:
> +            error_out('Unrecognized PDU command %s' % command)
> +    else:
> +        error_out('Command not supplied')
> +
> +    query='&'.join(['hostname='+pdu_hostname, 'port='+pdu_port])
> +    url = 'http://%s:%s/power/control/%s?%s' % (pdudaemon_hostname,
> pdudaemon_port, command, query)
> +    resp = requests.get(url)
> +    if resp.status_code != 200:
> +        error_out("pdudaemon did not accept the request")
> +    return 0
> +
>  def main():
>      # use global module names
>      global re, time, copy2, subprocess, signal, fcntl, requests, json
> @@ -5094,6 +5170,10 @@ def main():
>          do_delete_var(bvars, options)
>          sys.exit(0)
> 
> +    if command == "pdu":
> +        rcode = do_pdu(bvars, conf, options)
> +        sys.exit(rcode)
> +
>      error_out("Unknown command %s" % command)
> 
> 
> diff --git a/scripts/ftc_completion.sh b/scripts/ftc_completion.sh
> index 984e003..2cfb77a 100755
> --- a/scripts/ftc_completion.sh
> +++ b/scripts/ftc_completion.sh
> @@ -55,6 +55,7 @@ _ftc()
>          ["list-tests"]="-q"
>          ["package-run"]="-o -f"
>          ["package-test"]="-o -f"
> +        ["pdu"]="-b --command --pdudaemon_hostname --pdudaemon_port"
>          ["put-request"]="-R -s"
>          ["put-run"]=""
>          ["put-test"]=""
> @@ -123,6 +124,10 @@ _ftc()
>              COMPREPLY=( $( compgen -W "true false" -- "$cur" ) )
>              return 0
>              ;;
> +        '--command')
> +            COMPREPLY=( $( compgen -W "on off reboot" -- "$cur" ) )
> +            return 0
> +            ;;
>          'put-run')
>              COMPREPLY=( $( compgen -W "$(ftc list-runs -q)" -- "$cur" ) )
>              return 0
> --
> 2.7.4
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] pdudaemon support for Fuego

[daniel.sangorrin]

Hi Tim,

Thanks for checking the patches despite being busy.

> -----Original Message-----
> From: Tim.Bird@sony.com <Tim.Bird@sony.com>
> Sent: Thursday, March 14, 2019 4:33 AM
> To: sangorrin daniel(サンゴリン ダニエル ○ＳＷＣ□ＯＳＴ) <daniel.sangorrin@toshiba.co.jp>;
> fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] pdudaemon support for Fuego
> 
> Thanks for working on this.  I have a few questions.
> See below.
> 
> > -----Original Message-----
> > From: Daniel Sangorrin
> >
> > Hello Tim,
> >
> > One of the results from the Automated Testing Summit in Scotland was
> > an agreement on using pdudaemon to control PDUs (Power Distribution
> > Units).
> >
> > This patch adds such support:
> > [PATCH] ftc: add a new command to interact with pdudaemon
> >
> > This is a very short quickstart:
> >
> > - Prepare pdudaemon (for an ip9258 PDU in this example)
> >
> > host$ git clone https://github.com/pdudaemon/pdudaemon
> > [Note] I have made a few changes and sent a pull request, so temporarily
> > you may want to use https://github.com/sangorrin/pdudaemon
> > host$ cd pdudaemon
> > host$ vi share/pdudaemon.conf
> > {
> > 	"daemon": {
> > 		"hostname": "0.0.0.0",
> > 		"port": 16421,
> > 		"dbname": "pdudaemon",
> > 		"logging_level": "DEBUG",
> > 		"listener": "http"
> > 	},
> > 	"pdus": {
> > 		"192.168.11.68": {
> > 			"driver": "ip9258"
> > 		}
> > 	}
> > }
> > host$ sudo docker build -t pdudaemon --build-arg HTTP_PROXY=$http_proxy
> > -f Dockerfile.dockerhub .
> > host$ docker run --rm -it -e http_proxy=$http_proxy -e
> > https_proxy=$https_proxy -e NO_PROXY="$no_proxy" --net="host"
> > pdudaemon:latest
> >
> > - Check that it works manually
> 
> OK - I'm just trying to double-check my understanding of the situation here.
> So, apparently the way to use pdudaemon is to build a docker container for it,
> and communicate with the daemon inside that container via curl (that is, using
> HTTP over a network). Is that right?

Yes, that's what I did. You can have pdudaemon on another node in your lab.

> Would it make sense to clone the pdudaemon code into the Fuego docker container,
> and start the daemon there, rather than in its own container?

It is possible.

I didn't do it because when I tried installing pdudaemon, the Dockerfile based on Debian Jessie didn't work for me and I had to upgrade it to Debian Stretch. I am still waiting for the maintainer to answer my pull request
https://github.com/pdudaemon/pdudaemon/pull/57

> I can think of pros and cons to each approach.
> 
> Pros:
>   * It eliminates separate steps for installing and starting the daemon (well, technically
>       it would combine the steps with Fuego's container creation)
>       * In this case, we might try to put the pdudaemon.conf into Fuego's fuego-ro/conf directory.
> 
> Cons:
>   * It may be confusing to disable a Fuego-integrated pdudaemon, if someone already
>       has one installed and is using that one (you might transfer the configuration to the Fuego
>       instance of the daemon, or you might disable the integrated one and just configure
>       Fuego to use the pre-existing one.

I will add the maintenance of dependencies to the list. 

> At the Automated Testing Summit, we didn't have time to discuss how to coordinate
> how different pieces would fit together, particularly when they are in their own containers.
> I'm not sure if this is something that needs to be done on a case-by-case basis, or if
> there is some design pattern we should be following for this, that we want to encourage
> tools to adopt to make this easier.
> 
> In the case of 'ttc', which provides similar functionality, it's integrated into the Dockerfile,
> and into Fuego's transport layer.  And the IPC used is Unix command line invocation.

I have never tried ttc, does it support controlling IP9258 PDUs?

> > $ curl
> > "http://localhost:16421/power/control/on?hostname=192.168.11.68&port=1
> > "
> > $ curl
> > "http://localhost:16421/power/control/on?hostname=192.168.11.68&port=1
> > "
> 
> Does the 'pdudaemon' project include their own command line utility?
> If so, I'd rather use that than add something to Fuego.

Yes, it does. The client (pduclient) uses tcp instead of http. I didn't want to use because it will pull dependencies into Fuego.

Thanks,
Daniel
 
> > - Check that it works from Fuego
> >
> > $ vi myboard.board
> > 	PDU_HOSTNAME="192.168.11.68"
> > 	PDU_PORT="3"
> > $ ftc pdu -b myboard.board --command on
> > $ ftc pdu -b myboard.board --command off
> >
> > If you have any question about pdudaemon please let me know.
> 
> See above.
>  -- Tim

Re: [Fuego] [PATCH] ftc: add a new command to interact with pdudaemon

[daniel.sangorrin]

Hi Tim, 

Thanks again for reviewing in your busy time.

> -----Original Message-----
> From: Tim.Bird@sony.com <Tim.Bird@sony.com>
> Sent: Thursday, March 14, 2019 4:55 AM
> To: sangorrin daniel(サンゴリン ダニエル ○ＳＷＣ□ＯＳＴ) <daniel.sangorrin@toshiba.co.jp>;
> fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] [PATCH] ftc: add a new command to interact with pdudaemon
> 
> Here are some overall comments:
> 
> Rather than add a new command to ftc for managing pdudaemon operations,
> I was expecting to install support for pdudaemon at Fuego's board management (/board_control)
> layer.  That is, I was expecting to see something like this:
> 
> diff --git a/overlays/base/base-board.fuegoclass b/overlays/base/base-board.fuegoclass
> index ae35464..8f34922 100644
> --- a/overlays/base/base-board.fuegoclass
> +++ b/overlays/base/base-board.fuegoclass
> @@ -299,6 +299,9 @@ function ov_board_control_reboot() {
>    "ttc")
>      $TTC $TTC_TARGET reboot
>      ;;
> +  "pdudaemon"):
> +    curl
> "http://$PDUDAEMON_HOST:$PDUDAEMON_PORT/power/control/reboot?hostname=$PDU_HOSTNAME&por
> t=$PDU_PORT"
> +    ;;
>    *)
>      abort_job "Error reason: unsupported BOARD_CONTROL ${BOARD_CONTROL}"
>      ;;
> ----

Alright, that sounds like a good idea. I will resend.

> Note that I don't know if pdudaemon supports the 'reboot' control option, or if I would have
reboot is supported

> to synthesize something by sending 'on' and 'off'.  But you get the idea, I think.
> I'm not sure if the hostname in the URL is supposed to be the board name or the pdu name,
PDU_HOSTNAME is the PDU's IP address/hostname.
PDUDAEMON_HOSTNAME is the IP address/hostname of the computer where pdudaemon is running.
The board doesn't know that it can be "killed" at anytime by the PDU.

> but presumably PDUDAEMON_HOST and PDUDAEMON_PORT would come from
> fuego.conf, and PDU_HOSTNAME and PDU_PORT would come from the board file?
From the board file because each board can be in a different place. These two parameters are passed to the pdudaemon. Pdudaemon then internally uses them to connect with the PDU by the supported protocol (e.g.: SNMTP).

> If more fine-grained control is desired, than just the 'reboot' operation,
> we could add more ov_board_control() functions if needed, and have Fuego use those.
> e.g. ov_board_control_on() and ov_board_control_off().

Yes, I think we need on off.
In particular, I was thinking about Fuego testplans that would work like this:

ftc pdu -b bbb --command on
ftc expect -b bbb --regex "u-boot>" --send SIGINT
ftc send -b bbb --command "run tftpboot"
ftc expect -b bbb --regex "mypromptlinuxprompt" --send "root"
echo ftc pdu -b bbb --command off | at now + 1 minute
ftc run-test -b bbb -t Functional.StressFilesystem
sleep 60
ftc pdu -b bbb --command on
ftc expect -b bbb --regex myprompt
ftc run-test -b bbb -t Functional.ext4
ftc pdu -b bbb --command off

And we can make more complicated things.
[Note] The ftc expect is something I am designing now to interact with the board in a flexible way, and provide LAVA-like functionality. 

> Once we've plugged pdudaemon into the board layer, I'd prefer to have a generic
> set of operations that someone can use from ftc, like:
>    ftc reboot-board -b <board>
>    ftc power-on-board -b <board>
>    ftc power-off-board -b <board>
> and maybe even
>    ftc show-status-of-board <board>

Maybe we can use subcommands like this.
ftc power reboot -b bbb
ftc power on -b board
ftc power off -b board
ftc power status -b board

Or put hyphens, but it would rather have the same 1st work for bash autocomplete to be more intuitive.

> In some labs, there may be a layer between pdudaemon and Fuego (for example libvirt,
> or labgrid or even LAVA).
> 
> I think I'd rather have an abstraction layer between Fuego and pdudaemon, rather than
> integrating it directly into our user interface.
> 
> Thoughts?

See above.
Thanks,
Daniel

>  -- tim
> 
> 
> > -----Original Message-----
> > From: Daniel Sangorrin
> >
> > This patch adds the "pdu" command to send requests for pdudaemon
> > to power-on/off or reboot a board.
> >
> > Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> > ---
> >  scripts/ftc               | 80
> > +++++++++++++++++++++++++++++++++++++++++++++++
> >  scripts/ftc_completion.sh |  5 +++
> >  2 files changed, 85 insertions(+)
> >
> > diff --git a/scripts/ftc b/scripts/ftc
> > index 7153c01..30ea470 100755
> > --- a/scripts/ftc
> > +++ b/scripts/ftc
> > @@ -426,6 +426,29 @@ A message and the exit code indicate if the resource
> > was released.
> >
> >     ex: ftc release-resource -b beaglebone"""),
> >
> > +"pdu": ("Send a request to pdudaemon.",
> > +    """Usage: ftc pdu -b <board> --command on|off|reboot [--
> > pdudaemon_hostname localhost --pdudaemon_port 16421]
> > +
> > +This command is used to send a request for pdudaemon to power-on,
> > power-off or reboot a board.
> > +
> > +The command accepts these arguments:
> > + - board file: bbb, raspberrypi3, etc (see ftc list-boards)
> > + - command: on|off|reboot
> > + - pdudaemon_hostname: hostname or IP address where pdudaemon is
> > running (default: localhost)
> > + - pdudaemon_port: port where pdudaemon is listening (default: 16421)
> > +   [Note] make sure that pdudaemon is configured to accept http requests
> > +
> > +Additionally, you need to define two variables in your board file:
> > + - PDU_HOSTNAME: the PDU hostname or IP address used in the
> > pdudaemon configuration file
> > + - PDU_PORT: the PDU port number (a plug port, not a tcp port)
> > +
> > +You need to install pdudaemon by yourself
> > +  $ git clone https://github.com/pdudaemon/pdudaemon
> > +  $ cd pdudaemon
> > +  $ less README.md
> > +
> > +Example: ftc pdu -b raspberrypi3 --command on"""),
> > +
> >  "package-test": ("Package a test.",
> >      """Usage: ftc package-test <test_name> [options]
> >
> > @@ -4783,6 +4806,59 @@ def do_release_resource(conf, options):
> >
> >      return 0
> >
> > +def do_pdu(bvars, conf, options):
> > +    # pdudaemon hostname
> > +    pdudaemon_hostname = 'localhost'
> > +    if '--pdudaemon_hostname' in options:
> > +        try:
> > +            pdudaemon_hostname = options[options.index('--
> > pdudaemon_hostname')+1]
> > +        except IndexError:
> > +            error_out('Value not provided after --pdudaemon_hostname')
> > +        options.remove(pdudaemon_hostname)
> > +        options.remove('--pdudaemon_hostname')
> > +
> > +    # pdudaemon port
> > +    pdudaemon_port = '16421'
> > +    if '--pdudaemon_port' in options:
> > +        try:
> > +            pdudaemon_port = options[options.index('--pdudaemon_port')+1]
> > +        except IndexError:
> > +            error_out('Value not provided after --pdudaemon_port')
> > +        options.remove(pdudaemon_port)
> > +        options.remove('--pdudaemon_port')
> > +
> > +    # pdu hostname
> > +    if 'PDU_HOSTNAME' in bvars:
> > +        pdu_hostname = bvars['PDU_HOSTNAME']
> > +    else:
> > +        error_out('PDU_HOSTNAME not specified in board file')
> > +
> > +    # pdu port
> > +    if 'PDU_PORT' in bvars:
> > +        pdu_port = bvars['PDU_PORT']
> > +    else:
> > +        error_out('PDU_PORT not specified in board file')
> > +
> > +    # command (on|off|reboot)
> > +    if '--command' in options:
> > +        try:
> > +            command = options[options.index('--command')+1]
> > +        except IndexError:
> > +            error_out('Value not provided after --command')
> > +        options.remove(command)
> > +        options.remove('--command')
> > +        if command not in ['on', 'off', 'reboot']:
> > +            error_out('Unrecognized PDU command %s' % command)
> > +    else:
> > +        error_out('Command not supplied')
> > +
> > +    query='&'.join(['hostname='+pdu_hostname, 'port='+pdu_port])
> > +    url = 'http://%s:%s/power/control/%s?%s' % (pdudaemon_hostname,
> > pdudaemon_port, command, query)
> > +    resp = requests.get(url)
> > +    if resp.status_code != 200:
> > +        error_out("pdudaemon did not accept the request")
> > +    return 0
> > +
> >  def main():
> >      # use global module names
> >      global re, time, copy2, subprocess, signal, fcntl, requests, json
> > @@ -5094,6 +5170,10 @@ def main():
> >          do_delete_var(bvars, options)
> >          sys.exit(0)
> >
> > +    if command == "pdu":
> > +        rcode = do_pdu(bvars, conf, options)
> > +        sys.exit(rcode)
> > +
> >      error_out("Unknown command %s" % command)
> >
> >
> > diff --git a/scripts/ftc_completion.sh b/scripts/ftc_completion.sh
> > index 984e003..2cfb77a 100755
> > --- a/scripts/ftc_completion.sh
> > +++ b/scripts/ftc_completion.sh
> > @@ -55,6 +55,7 @@ _ftc()
> >          ["list-tests"]="-q"
> >          ["package-run"]="-o -f"
> >          ["package-test"]="-o -f"
> > +        ["pdu"]="-b --command --pdudaemon_hostname --pdudaemon_port"
> >          ["put-request"]="-R -s"
> >          ["put-run"]=""
> >          ["put-test"]=""
> > @@ -123,6 +124,10 @@ _ftc()
> >              COMPREPLY=( $( compgen -W "true false" -- "$cur" ) )
> >              return 0
> >              ;;
> > +        '--command')
> > +            COMPREPLY=( $( compgen -W "on off reboot" -- "$cur" ) )
> > +            return 0
> > +            ;;
> >          'put-run')
> >              COMPREPLY=( $( compgen -W "$(ftc list-runs -q)" -- "$cur" ) )
> >              return 0
> > --
> > 2.7.4
> >
> > _______________________________________________
> > Fuego mailing list
> > Fuego@lists.linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] [PATCH] ftc: add a new command to interact with pdudaemon

[daniel.sangorrin]

Hello Tim,

> -----Original Message-----
> From: Tim.Bird@sony.com <Tim.Bird@sony.com>
> Sent: Thursday, March 14, 2019 4:55 AM
> To: sangorrin daniel(サンゴリン ダニエル ○ＳＷＣ□ＯＳＴ) <daniel.sangorrin@toshiba.co.jp>;
> fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] [PATCH] ftc: add a new command to interact with pdudaemon
> 
> Here are some overall comments:
> 
> Rather than add a new command to ftc for managing pdudaemon operations,
> I was expecting to install support for pdudaemon at Fuego's board management (/board_control)
> layer.  That is, I was expecting to see something like this:
> 
> diff --git a/overlays/base/base-board.fuegoclass b/overlays/base/base-board.fuegoclass
> index ae35464..8f34922 100644
> --- a/overlays/base/base-board.fuegoclass
> +++ b/overlays/base/base-board.fuegoclass
> @@ -299,6 +299,9 @@ function ov_board_control_reboot() {
>    "ttc")
>      $TTC $TTC_TARGET reboot
>      ;;
> +  "pdudaemon"):
> +    curl
> "http://$PDUDAEMON_HOST:$PDUDAEMON_PORT/power/control/reboot?hostname=$PDU_HOSTNAME&por
> t=$PDU_PORT"
> +    ;;
>    *)
>      abort_job "Error reason: unsupported BOARD_CONTROL ${BOARD_CONTROL}"
>      ;;
> ----

I have tried to do it this way but I met some difficulties.

Suppose that I call
ftc power-on-board -b bbb

Then, to be able to execute ov_board_control_reboot (or ov_board_control_power_on) I would need to do like ftc run-test and export lots of variables and call the overlays generator etc. This is probably a hint that the architecture is not correct.

If we do the control from ftc things will be smoother. Summarizing, this is the patch that i plan to submit:

- add three commands to ftc
ftc power-on-board
ftc power-off-board
ftc reboot-board
[Note] depending on the BOARD_CONTROL variable (bvars) use ttc or pdudaemon
- remove ov_board_control_reboot
- remove calls to ov_board_control_reboot during pre_test and post_test
   - option 1: ask the user to call ftc power-on-board, and then ftc wait-for (e.g. wait for the prompt or something else)
   - option 2: substitute it by ftc reboot-board and add a new variable BOARD_CONTROL_AUTOMATIC to the board file (true: reboot board before and after test, false: I will call ftc reboot-board whenever I want)
- remove calls to ov_board_control_reboot from Benchmark.reboot
   - substitute it by a nested call to "ftc reboot-board -b $NODE_NAME"

Any thoughts?

Thanks,
Daniel


> 
> Note that I don't know if pdudaemon supports the 'reboot' control option, or if I would have
> to synthesize something by sending 'on' and 'off'.  But you get the idea, I think.
> I'm not sure if the hostname in the URL is supposed to be the board name or the pdu name,
> but presumably PDUDAEMON_HOST and PDUDAEMON_PORT would come from
> fuego.conf, and PDU_HOSTNAME and PDU_PORT would come from the board file?
> 
> If more fine-grained control is desired, than just the 'reboot' operation,
> we could add more ov_board_control() functions if needed, and have Fuego use those.
> e.g. ov_board_control_on() and ov_board_control_off().
> 
> Once we've plugged pdudaemon into the board layer, I'd prefer to have a generic
> set of operations that someone can use from ftc, like:
>    ftc reboot-board -b <board>
>    ftc power-on-board -b <board>
>    ftc power-off-board -b <board>
> and maybe even
>    ftc show-status-of-board <board>
> 
> In some labs, there may be a layer between pdudaemon and Fuego (for example libvirt,
> or labgrid or even LAVA).
> 
> I think I'd rather have an abstraction layer between Fuego and pdudaemon, rather than
> integrating it directly into our user interface.
> 
> Thoughts?
>  -- tim
> 
> 
> > -----Original Message-----
> > From: Daniel Sangorrin
> >
> > This patch adds the "pdu" command to send requests for pdudaemon
> > to power-on/off or reboot a board.
> >
> > Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> > ---
> >  scripts/ftc               | 80
> > +++++++++++++++++++++++++++++++++++++++++++++++
> >  scripts/ftc_completion.sh |  5 +++
> >  2 files changed, 85 insertions(+)
> >
> > diff --git a/scripts/ftc b/scripts/ftc
> > index 7153c01..30ea470 100755
> > --- a/scripts/ftc
> > +++ b/scripts/ftc
> > @@ -426,6 +426,29 @@ A message and the exit code indicate if the resource
> > was released.
> >
> >     ex: ftc release-resource -b beaglebone"""),
> >
> > +"pdu": ("Send a request to pdudaemon.",
> > +    """Usage: ftc pdu -b <board> --command on|off|reboot [--
> > pdudaemon_hostname localhost --pdudaemon_port 16421]
> > +
> > +This command is used to send a request for pdudaemon to power-on,
> > power-off or reboot a board.
> > +
> > +The command accepts these arguments:
> > + - board file: bbb, raspberrypi3, etc (see ftc list-boards)
> > + - command: on|off|reboot
> > + - pdudaemon_hostname: hostname or IP address where pdudaemon is
> > running (default: localhost)
> > + - pdudaemon_port: port where pdudaemon is listening (default: 16421)
> > +   [Note] make sure that pdudaemon is configured to accept http requests
> > +
> > +Additionally, you need to define two variables in your board file:
> > + - PDU_HOSTNAME: the PDU hostname or IP address used in the
> > pdudaemon configuration file
> > + - PDU_PORT: the PDU port number (a plug port, not a tcp port)
> > +
> > +You need to install pdudaemon by yourself
> > +  $ git clone https://github.com/pdudaemon/pdudaemon
> > +  $ cd pdudaemon
> > +  $ less README.md
> > +
> > +Example: ftc pdu -b raspberrypi3 --command on"""),
> > +
> >  "package-test": ("Package a test.",
> >      """Usage: ftc package-test <test_name> [options]
> >
> > @@ -4783,6 +4806,59 @@ def do_release_resource(conf, options):
> >
> >      return 0
> >
> > +def do_pdu(bvars, conf, options):
> > +    # pdudaemon hostname
> > +    pdudaemon_hostname = 'localhost'
> > +    if '--pdudaemon_hostname' in options:
> > +        try:
> > +            pdudaemon_hostname = options[options.index('--
> > pdudaemon_hostname')+1]
> > +        except IndexError:
> > +            error_out('Value not provided after --pdudaemon_hostname')
> > +        options.remove(pdudaemon_hostname)
> > +        options.remove('--pdudaemon_hostname')
> > +
> > +    # pdudaemon port
> > +    pdudaemon_port = '16421'
> > +    if '--pdudaemon_port' in options:
> > +        try:
> > +            pdudaemon_port = options[options.index('--pdudaemon_port')+1]
> > +        except IndexError:
> > +            error_out('Value not provided after --pdudaemon_port')
> > +        options.remove(pdudaemon_port)
> > +        options.remove('--pdudaemon_port')
> > +
> > +    # pdu hostname
> > +    if 'PDU_HOSTNAME' in bvars:
> > +        pdu_hostname = bvars['PDU_HOSTNAME']
> > +    else:
> > +        error_out('PDU_HOSTNAME not specified in board file')
> > +
> > +    # pdu port
> > +    if 'PDU_PORT' in bvars:
> > +        pdu_port = bvars['PDU_PORT']
> > +    else:
> > +        error_out('PDU_PORT not specified in board file')
> > +
> > +    # command (on|off|reboot)
> > +    if '--command' in options:
> > +        try:
> > +            command = options[options.index('--command')+1]
> > +        except IndexError:
> > +            error_out('Value not provided after --command')
> > +        options.remove(command)
> > +        options.remove('--command')
> > +        if command not in ['on', 'off', 'reboot']:
> > +            error_out('Unrecognized PDU command %s' % command)
> > +    else:
> > +        error_out('Command not supplied')
> > +
> > +    query='&'.join(['hostname='+pdu_hostname, 'port='+pdu_port])
> > +    url = 'http://%s:%s/power/control/%s?%s' % (pdudaemon_hostname,
> > pdudaemon_port, command, query)
> > +    resp = requests.get(url)
> > +    if resp.status_code != 200:
> > +        error_out("pdudaemon did not accept the request")
> > +    return 0
> > +
> >  def main():
> >      # use global module names
> >      global re, time, copy2, subprocess, signal, fcntl, requests, json
> > @@ -5094,6 +5170,10 @@ def main():
> >          do_delete_var(bvars, options)
> >          sys.exit(0)
> >
> > +    if command == "pdu":
> > +        rcode = do_pdu(bvars, conf, options)
> > +        sys.exit(rcode)
> > +
> >      error_out("Unknown command %s" % command)
> >
> >
> > diff --git a/scripts/ftc_completion.sh b/scripts/ftc_completion.sh
> > index 984e003..2cfb77a 100755
> > --- a/scripts/ftc_completion.sh
> > +++ b/scripts/ftc_completion.sh
> > @@ -55,6 +55,7 @@ _ftc()
> >          ["list-tests"]="-q"
> >          ["package-run"]="-o -f"
> >          ["package-test"]="-o -f"
> > +        ["pdu"]="-b --command --pdudaemon_hostname --pdudaemon_port"
> >          ["put-request"]="-R -s"
> >          ["put-run"]=""
> >          ["put-test"]=""
> > @@ -123,6 +124,10 @@ _ftc()
> >              COMPREPLY=( $( compgen -W "true false" -- "$cur" ) )
> >              return 0
> >              ;;
> > +        '--command')
> > +            COMPREPLY=( $( compgen -W "on off reboot" -- "$cur" ) )
> > +            return 0
> > +            ;;
> >          'put-run')
> >              COMPREPLY=( $( compgen -W "$(ftc list-runs -q)" -- "$cur" ) )
> >              return 0
> > --
> > 2.7.4
> >
> > _______________________________________________
> > Fuego mailing list
> > Fuego@lists.linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] [PATCH] ftc: add a new command to interact with pdudaemon

[Tim.Bird]

> -----Original Message-----
> From: daniel.sangorrin@toshiba.co.jp
> 
> Hello Tim,
> 
> > -----Original Message-----
> > From: Tim.Bird@sony.com <Tim.Bird@sony.com>
> >
> > Here are some overall comments:
> >
> > Rather than add a new command to ftc for managing pdudaemon
> operations,
> > I was expecting to install support for pdudaemon at Fuego's board
> management (/board_control)
> > layer.  That is, I was expecting to see something like this:
> >
> > diff --git a/overlays/base/base-board.fuegoclass b/overlays/base/base-
> board.fuegoclass
> > index ae35464..8f34922 100644
> > --- a/overlays/base/base-board.fuegoclass
> > +++ b/overlays/base/base-board.fuegoclass
> > @@ -299,6 +299,9 @@ function ov_board_control_reboot() {
> >    "ttc")
> >      $TTC $TTC_TARGET reboot
> >      ;;
> > +  "pdudaemon"):
> > +    curl
> >
> "http://$PDUDAEMON_HOST:$PDUDAEMON_PORT/power/control/reboot?
> hostname=$PDU_HOSTNAME&por
> > t=$PDU_PORT"
> > +    ;;
> >    *)
> >      abort_job "Error reason: unsupported BOARD_CONTROL
> ${BOARD_CONTROL}"
> >      ;;
> > ----
> 
> I have tried to do it this way but I met some difficulties.
> 
> Suppose that I call
> ftc power-on-board -b bbb
> 
> Then, to be able to execute ov_board_control_reboot (or
> ov_board_control_power_on) I would need to do like ftc run-test and
> export lots of variables and call the overlays generator etc. This is probably a
> hint that the architecture is not correct.

You are right.  I didn't give this enough thought.  I mirrored the way we
did abstractions for transports, by putting the abstraction in the
overlay layer.  Quite frankly, that's an abstraction layer I haven't ever
been happy with.  It's way more complicated than it needs to be, 
and behaves in ways that are unexpected to the user.  (It violates
the rule of least surprise).  The other problems with it are
too numerous to list here, but it has several issues.

Just to be clear, that layer was created before I became involved with
this project.

Having a shell-based abstraction layer made sense when the tests
were executed directly as shell scripts. 'ftc' didn't exist when the
overlay layer was created.

ftc allows for the notion that you may want to allow sub-operations
to be called from the command line.  We do this for nested tests, as
well as for reading Fuego and board variables.  Performing sub-operations
can occur either outside or inside the context of a test. 
That is, someone may want to manually perform a sub-operation at
the command line, or use 'ftc' to perform the sub-operation from inside a test.

By way of contrast, we don't have a mechanism for doing TRANSPORT
operations directly using ttc, but it would obviously be useful.
(For example, it would be nice to be able to do get or put operations with
'ftc put -b bbb somefile /tmp' or (or, with syntactic sugar, make that:
'ftc put somefile bbb:/tmp').  And it would be nice to be able to do:
'ftc cmd -b bbb <some_command>')

The same might be said for our 'DISTRO' abstractions.  But now I'm getting
off-topic.  Getting back to just the board control operations...

I personally haven't needed manual board control operations from the
'ftc' command line because I use 'ttc'.  So that's probably been a blind
spot for me in the architecture.  I've been hesitating on this, because it's
not clear to me that we want to define a new, Fuego-specific set of operations
for board control.  But we have to have something that manages the abstractions
(so we're not hardcoded to pdudaemon, labgrid, LAVA or ttc) and if we're going
to consolidate our abstractions in one place, it should probably be in 'ftc'.

I haven't answered your other e-mail.  I'll try to go back and do that today.
But I'll put some comments that I had intended for that e-mail below, along
with some notes.

> 
> If we do the control from ftc things will be smoother. Summarizing, this is the
> patch that i plan to submit:
> 
> - add three commands to ftc
> ftc power-on-board
> ftc power-off-board
> ftc reboot-board
Just a note on naming.  I'm trying to make the naming for 'ftc' consistent by having
the command names be <verb>-<object>.  In the case of the the 'power' commands
this results in somewhat awkward names, as there is no single word commonly used
in English that I am aware of to "turn on" or apply power to an item.  So these turn into
phrases, which is less than ideal.  But hopefully that explains the names I suggested.

> [Note] depending on the BOARD_CONTROL variable (bvars) use ttc or
> pdudaemon
This will still require reading the board file internally.  But 1) ftc has internal
functions for that already, and 2) it might not require processing the
overlay as you could just use specific variables (ie "BOARD_CONTROL")
without having to perform the overlay operation.  The overlay
operation is only needed if you are drilling down through the distro
or transport layer to get to a function.  This assumes that you don't allow
the user or test to override the BOARD_CONTROL variable with a stored variable.
(See http://fuegotest.org/wiki/Test_variables#Stored_variables), which I think
is a reasonable limitation.

> - remove ov_board_control_reboot

I don't want to hardcode the board management layer inside 'ftc'.
I was planning on using the board management layer not just for power
control, but also for provisioning, and possibly state control of the bootup sequence
(like labgrid does, and in support of that integrating Fuego and labgrid).

So, getting more specific.  That call to 'curl' that is used to communicate with
pdudaemon has to be coded somewhere.  If it's not in the overlay system,
or in 'ftc' proper, then there needs to be a new plugin system for 
providing those commands, that could be selected (de-multiplexed) based
on the BOARD_CONTROL variable.

I have some ideas for this, but it seems like the choices would be to implement
this as 1) a python module, 2) some shell-based system, 3) some configuration-based
system, allowing simple multi-line commands.  Possibly, we could move forward
with putting the code into ftc, and then create the plugin layer afterwards, but
if possible I'd like to do this at the same time that we introduce these ftc commands
and wire up our first abstraction.

> - remove calls to ov_board_control_reboot during pre_test and post_test
>    - option 1: ask the user to call ftc power-on-board, and then ftc wait-for
> (e.g. wait for the prompt or something else)
>    - option 2: substitute it by ftc reboot-board and add a new variable
> BOARD_CONTROL_AUTOMATIC to the board file (true: reboot board before
> and after test, false: I will call ftc reboot-board whenever I want)

I think I understand some (but not all) of the use cases for this new variable.
We might need to discuss this more.

> - remove calls to ov_board_control_reboot from Benchmark.reboot
>    - substitute it by a nested call to "ftc reboot-board -b $NODE_NAME"
It may be desirable to have a Fuego shell function to wrap that, like 'do_hardware_reboot' or
something, but that's an implementation detail.

Sorry this is so long-winded.

I think I agree in principle with the proposed changes, but  we may need to
have more discussion about the pros and cons of having the board control
layer be pluggable.
 -- Tim


> Any thoughts?
> 
> Thanks,
> Daniel
> 
> 
> >
> > Note that I don't know if pdudaemon supports the 'reboot' control option,
> or if I would have
> > to synthesize something by sending 'on' and 'off'.  But you get the idea, I
> think.
> > I'm not sure if the hostname in the URL is supposed to be the board name
> or the pdu name,
> > but presumably PDUDAEMON_HOST and PDUDAEMON_PORT would come
> from
> > fuego.conf, and PDU_HOSTNAME and PDU_PORT would come from the
> board file?
> >
> > If more fine-grained control is desired, than just the 'reboot' operation,
> > we could add more ov_board_control() functions if needed, and have
> Fuego use those.
> > e.g. ov_board_control_on() and ov_board_control_off().
> >
> > Once we've plugged pdudaemon into the board layer, I'd prefer to have a
> generic
> > set of operations that someone can use from ftc, like:
> >    ftc reboot-board -b <board>
> >    ftc power-on-board -b <board>
> >    ftc power-off-board -b <board>
> > and maybe even
> >    ftc show-status-of-board <board>
> >
> > In some labs, there may be a layer between pdudaemon and Fuego (for
> example libvirt,
> > or labgrid or even LAVA).
> >
> > I think I'd rather have an abstraction layer between Fuego and
> pdudaemon, rather than
> > integrating it directly into our user interface.
> >
> > Thoughts?
> >  -- tim
> >
> >
> > > -----Original Message-----
> > > From: Daniel Sangorrin
> > >
> > > This patch adds the "pdu" command to send requests for pdudaemon
> > > to power-on/off or reboot a board.
> > >
> > > Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> > > ---
> > >  scripts/ftc               | 80
> > > +++++++++++++++++++++++++++++++++++++++++++++++
> > >  scripts/ftc_completion.sh |  5 +++
> > >  2 files changed, 85 insertions(+)
> > >
> > > diff --git a/scripts/ftc b/scripts/ftc
> > > index 7153c01..30ea470 100755
> > > --- a/scripts/ftc
> > > +++ b/scripts/ftc
> > > @@ -426,6 +426,29 @@ A message and the exit code indicate if the
> resource
> > > was released.
> > >
> > >     ex: ftc release-resource -b beaglebone"""),
> > >
> > > +"pdu": ("Send a request to pdudaemon.",
> > > +    """Usage: ftc pdu -b <board> --command on|off|reboot [--
> > > pdudaemon_hostname localhost --pdudaemon_port 16421]
> > > +
> > > +This command is used to send a request for pdudaemon to power-on,
> > > power-off or reboot a board.
> > > +
> > > +The command accepts these arguments:
> > > + - board file: bbb, raspberrypi3, etc (see ftc list-boards)
> > > + - command: on|off|reboot
> > > + - pdudaemon_hostname: hostname or IP address where pdudaemon is
> > > running (default: localhost)
> > > + - pdudaemon_port: port where pdudaemon is listening (default: 16421)
> > > +   [Note] make sure that pdudaemon is configured to accept http
> requests
> > > +
> > > +Additionally, you need to define two variables in your board file:
> > > + - PDU_HOSTNAME: the PDU hostname or IP address used in the
> > > pdudaemon configuration file
> > > + - PDU_PORT: the PDU port number (a plug port, not a tcp port)
> > > +
> > > +You need to install pdudaemon by yourself
> > > +  $ git clone https://github.com/pdudaemon/pdudaemon
> > > +  $ cd pdudaemon
> > > +  $ less README.md
> > > +
> > > +Example: ftc pdu -b raspberrypi3 --command on"""),
> > > +
> > >  "package-test": ("Package a test.",
> > >      """Usage: ftc package-test <test_name> [options]
> > >
> > > @@ -4783,6 +4806,59 @@ def do_release_resource(conf, options):
> > >
> > >      return 0
> > >
> > > +def do_pdu(bvars, conf, options):
> > > +    # pdudaemon hostname
> > > +    pdudaemon_hostname = 'localhost'
> > > +    if '--pdudaemon_hostname' in options:
> > > +        try:
> > > +            pdudaemon_hostname = options[options.index('--
> > > pdudaemon_hostname')+1]
> > > +        except IndexError:
> > > +            error_out('Value not provided after --pdudaemon_hostname')
> > > +        options.remove(pdudaemon_hostname)
> > > +        options.remove('--pdudaemon_hostname')
> > > +
> > > +    # pdudaemon port
> > > +    pdudaemon_port = '16421'
> > > +    if '--pdudaemon_port' in options:
> > > +        try:
> > > +            pdudaemon_port = options[options.index('--
> pdudaemon_port')+1]
> > > +        except IndexError:
> > > +            error_out('Value not provided after --pdudaemon_port')
> > > +        options.remove(pdudaemon_port)
> > > +        options.remove('--pdudaemon_port')
> > > +
> > > +    # pdu hostname
> > > +    if 'PDU_HOSTNAME' in bvars:
> > > +        pdu_hostname = bvars['PDU_HOSTNAME']
> > > +    else:
> > > +        error_out('PDU_HOSTNAME not specified in board file')
> > > +
> > > +    # pdu port
> > > +    if 'PDU_PORT' in bvars:
> > > +        pdu_port = bvars['PDU_PORT']
> > > +    else:
> > > +        error_out('PDU_PORT not specified in board file')
> > > +
> > > +    # command (on|off|reboot)
> > > +    if '--command' in options:
> > > +        try:
> > > +            command = options[options.index('--command')+1]
> > > +        except IndexError:
> > > +            error_out('Value not provided after --command')
> > > +        options.remove(command)
> > > +        options.remove('--command')
> > > +        if command not in ['on', 'off', 'reboot']:
> > > +            error_out('Unrecognized PDU command %s' % command)
> > > +    else:
> > > +        error_out('Command not supplied')
> > > +
> > > +    query='&'.join(['hostname='+pdu_hostname, 'port='+pdu_port])
> > > +    url = 'http://%s:%s/power/control/%s?%s' %
> (pdudaemon_hostname,
> > > pdudaemon_port, command, query)
> > > +    resp = requests.get(url)
> > > +    if resp.status_code != 200:
> > > +        error_out("pdudaemon did not accept the request")
> > > +    return 0
> > > +
> > >  def main():
> > >      # use global module names
> > >      global re, time, copy2, subprocess, signal, fcntl, requests, json
> > > @@ -5094,6 +5170,10 @@ def main():
> > >          do_delete_var(bvars, options)
> > >          sys.exit(0)
> > >
> > > +    if command == "pdu":
> > > +        rcode = do_pdu(bvars, conf, options)
> > > +        sys.exit(rcode)
> > > +
> > >      error_out("Unknown command %s" % command)
> > >
> > >
> > > diff --git a/scripts/ftc_completion.sh b/scripts/ftc_completion.sh
> > > index 984e003..2cfb77a 100755
> > > --- a/scripts/ftc_completion.sh
> > > +++ b/scripts/ftc_completion.sh
> > > @@ -55,6 +55,7 @@ _ftc()
> > >          ["list-tests"]="-q"
> > >          ["package-run"]="-o -f"
> > >          ["package-test"]="-o -f"
> > > +        ["pdu"]="-b --command --pdudaemon_hostname --
> pdudaemon_port"
> > >          ["put-request"]="-R -s"
> > >          ["put-run"]=""
> > >          ["put-test"]=""
> > > @@ -123,6 +124,10 @@ _ftc()
> > >              COMPREPLY=( $( compgen -W "true false" -- "$cur" ) )
> > >              return 0
> > >              ;;
> > > +        '--command')
> > > +            COMPREPLY=( $( compgen -W "on off reboot" -- "$cur" ) )
> > > +            return 0
> > > +            ;;
> > >          'put-run')
> > >              COMPREPLY=( $( compgen -W "$(ftc list-runs -q)" -- "$cur" ) )
> > >              return 0
> > > --
> > > 2.7.4
> > >
> > > _______________________________________________
> > > Fuego mailing list
> > > Fuego@lists.linuxfoundation.org
> > > https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] [PATCH] ftc: add a new command to interact with pdudaemon

[Tim.Bird]

More feedback on this topic...  see below.

> -----Original Message-----
> From: daniel.sangorrin@toshiba.co.jp
> 
> Hi Tim,
> 
> Thanks again for reviewing in your busy time.
> 
> > -----Original Message-----
> > From: Tim.Bird@sony.com <Tim.Bird@sony.com>
> > Sent: Thursday, March 14, 2019 4:55 AM
> > To: sangorrin daniel(サンゴリン ダニエル ○ＳＷＣ□ＯＳＴ)
> <daniel.sangorrin@toshiba.co.jp>;
> > fuego@lists.linuxfoundation.org
> > Subject: RE: [Fuego] [PATCH] ftc: add a new command to interact with
> pdudaemon
> >
> > Here are some overall comments:
> >
> > Rather than add a new command to ftc for managing pdudaemon
> operations,
> > I was expecting to install support for pdudaemon at Fuego's board
> management (/board_control)
> > layer.  That is, I was expecting to see something like this:
> >
> > diff --git a/overlays/base/base-board.fuegoclass b/overlays/base/base-
> board.fuegoclass
> > index ae35464..8f34922 100644
> > --- a/overlays/base/base-board.fuegoclass
> > +++ b/overlays/base/base-board.fuegoclass
> > @@ -299,6 +299,9 @@ function ov_board_control_reboot() {
> >    "ttc")
> >      $TTC $TTC_TARGET reboot
> >      ;;
> > +  "pdudaemon"):
> > +    curl
> >
> "http://$PDUDAEMON_HOST:$PDUDAEMON_PORT/power/control/reboot?
> hostname=$PDU_HOSTNAME&por
> > t=$PDU_PORT"
> > +    ;;
> >    *)
> >      abort_job "Error reason: unsupported BOARD_CONTROL
> ${BOARD_CONTROL}"
> >      ;;
> > ----
> 
> Alright, that sounds like a good idea. I will resend.
> 
> > Note that I don't know if pdudaemon supports the 'reboot' control option,
> or if I would have
> reboot is supported
> 
> > to synthesize something by sending 'on' and 'off'.  But you get the idea, I
> think.
> > I'm not sure if the hostname in the URL is supposed to be the board name
> or the pdu name,
> PDU_HOSTNAME is the PDU's IP address/hostname.
> PDUDAEMON_HOSTNAME is the IP address/hostname of the computer
> where pdudaemon is running.
> The board doesn't know that it can be "killed" at anytime by the PDU.
> 
> > but presumably PDUDAEMON_HOST and PDUDAEMON_PORT would come
> from
> > fuego.conf, and PDU_HOSTNAME and PDU_PORT would come from the
> board file?
> From the board file because each board can be in a different place. These
> two parameters are passed to the pdudaemon. Pdudaemon then internally
> uses them to connect with the PDU by the supported protocol (e.g.:
> SNMTP).

OK. As discussed in the other thread, this means that the board file needs
to be parsed by ftc to execute these, to get these board-related test variables.
For now I believe it's OK to ignore processing overlays and stored variables for
this.  However, I think the ftc internal function get_board_vars() does all this processing
anyway, so it might just be easier to use that.

> 
> > If more fine-grained control is desired, than just the 'reboot' operation,
> > we could add more ov_board_control() functions if needed, and have
> Fuego use those.
> > e.g. ov_board_control_on() and ov_board_control_off().
> 
> Yes, I think we need on off.
> In particular, I was thinking about Fuego testplans that would work like this:
> 
> ftc pdu -b bbb --command on
> ftc expect -b bbb --regex "u-boot>" --send SIGINT
> ftc send -b bbb --command "run tftpboot"
> ftc expect -b bbb --regex "mypromptlinuxprompt" --send "root"
> echo ftc pdu -b bbb --command off | at now + 1 minute
> ftc run-test -b bbb -t Functional.StressFilesystem
> sleep 60
> ftc pdu -b bbb --command on
> ftc expect -b bbb --regex myprompt
> ftc run-test -b bbb -t Functional.ext4
> ftc pdu -b bbb --command off

A couple of notes here:
1) It looks like get_board_arg() looks for FUEGO_BOARD
in the environment, and uses that if present.
I'm pretty sure you can do: export FUEGO_BOARD=bbb, and then omit the '-b bbb' on every line.
From inside a test, maybe we should make this automatic, and set FUEGO_BOARD from
NODE_NAME when we do run-test.

2) 'ftc expect' looks nice, and it shouldn't be too hard to do since python has an 'expect' module
(famous last words... :-)

3) We should discuss the high-level operations we want to do, and make
ftc operations for those.   For some tests, the details of how the board is booted
and provisioned should be opaque to the test.


One issue I see with your low-level steps, is that they encode
things that are board-specific (like the boot prompt regexes) into
the commands
I'm not opposed, to low-level operations being encoded into tests.
It's important, though, to try to isolate board-specific items so that the
tests don't become dependent on a specific board, bootloader, distro, 
board control layer, or even operating system.  (e.g. I've started work
on supporting Nuttx testing with Fuego).

Maybe you intend to parameterize those regexes from the board file.
But this still encapsulates too much of the way a system boots and
this particular example I believe requires that you have a serial port.
That wouldn't be true, for example, with a mobile phone, where you
have adb access, but often not a serial port.

Here's how I've used ttc for board control and provisioning:
ttc setenv bbb  # this allows me to omit 'bbb' on every following command line
ttc get_kernel
ttc set_config CONFIG_SOME_KCONFIG=y;CONFIG_SOME_VALUE=255
ttc kbuild
ttc kinstall
ttc reboot
ttc wait-for "ttc run 'echo alive'"
ttc cp testfiles target:/test_directory
ttc run "some commands"
ttc cp target:/test_directory/testlogs host_directory
> 
> And we can make more complicated things.
> [Note] The ftc expect is something I am designing now to interact with the
> board in a flexible way, and provide LAVA-like functionality.
I'd like to have a discussion with you next week about the best
approach in Fuego to handle provisioning, and building up complex test
pipelines.  I think some sub-operations should be in ftc, and maybe
some should be put into tests that call sub-tests.

 -- Tim

> 
> > Once we've plugged pdudaemon into the board layer, I'd prefer to have a
> generic
> > set of operations that someone can use from ftc, like:
> >    ftc reboot-board -b <board>
> >    ftc power-on-board -b <board>
> >    ftc power-off-board -b <board>
> > and maybe even
> >    ftc show-status-of-board <board>
> 
> Maybe we can use subcommands like this.
> ftc power reboot -b bbb
> ftc power on -b board
> ftc power off -b board
> ftc power status -b board
> 
> Or put hyphens, but it would rather have the same 1st work for bash
> autocomplete to be more intuitive.
> 
> > In some labs, there may be a layer between pdudaemon and Fuego (for
> example libvirt,
> > or labgrid or even LAVA).
> >
> > I think I'd rather have an abstraction layer between Fuego and
> pdudaemon, rather than
> > integrating it directly into our user interface.
> >
> > Thoughts?
> 
> See above.
> Thanks,
> Daniel
> 
> >  -- tim
> >
> >
> > > -----Original Message-----
> > > From: Daniel Sangorrin
> > >
> > > This patch adds the "pdu" command to send requests for pdudaemon
> > > to power-on/off or reboot a board.
> > >
> > > Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> > > ---
> > >  scripts/ftc               | 80
> > > +++++++++++++++++++++++++++++++++++++++++++++++
> > >  scripts/ftc_completion.sh |  5 +++
> > >  2 files changed, 85 insertions(+)
> > >
> > > diff --git a/scripts/ftc b/scripts/ftc
> > > index 7153c01..30ea470 100755
> > > --- a/scripts/ftc
> > > +++ b/scripts/ftc
> > > @@ -426,6 +426,29 @@ A message and the exit code indicate if the
> resource
> > > was released.
> > >
> > >     ex: ftc release-resource -b beaglebone"""),
> > >
> > > +"pdu": ("Send a request to pdudaemon.",
> > > +    """Usage: ftc pdu -b <board> --command on|off|reboot [--
> > > pdudaemon_hostname localhost --pdudaemon_port 16421]
> > > +
> > > +This command is used to send a request for pdudaemon to power-on,
> > > power-off or reboot a board.
> > > +
> > > +The command accepts these arguments:
> > > + - board file: bbb, raspberrypi3, etc (see ftc list-boards)
> > > + - command: on|off|reboot
> > > + - pdudaemon_hostname: hostname or IP address where pdudaemon is
> > > running (default: localhost)
> > > + - pdudaemon_port: port where pdudaemon is listening (default: 16421)
> > > +   [Note] make sure that pdudaemon is configured to accept http
> requests
> > > +
> > > +Additionally, you need to define two variables in your board file:
> > > + - PDU_HOSTNAME: the PDU hostname or IP address used in the
> > > pdudaemon configuration file
> > > + - PDU_PORT: the PDU port number (a plug port, not a tcp port)
> > > +
> > > +You need to install pdudaemon by yourself
> > > +  $ git clone https://github.com/pdudaemon/pdudaemon
> > > +  $ cd pdudaemon
> > > +  $ less README.md
> > > +
> > > +Example: ftc pdu -b raspberrypi3 --command on"""),
> > > +
> > >  "package-test": ("Package a test.",
> > >      """Usage: ftc package-test <test_name> [options]
> > >
> > > @@ -4783,6 +4806,59 @@ def do_release_resource(conf, options):
> > >
> > >      return 0
> > >
> > > +def do_pdu(bvars, conf, options):
> > > +    # pdudaemon hostname
> > > +    pdudaemon_hostname = 'localhost'
> > > +    if '--pdudaemon_hostname' in options:
> > > +        try:
> > > +            pdudaemon_hostname = options[options.index('--
> > > pdudaemon_hostname')+1]
> > > +        except IndexError:
> > > +            error_out('Value not provided after --pdudaemon_hostname')
> > > +        options.remove(pdudaemon_hostname)
> > > +        options.remove('--pdudaemon_hostname')
> > > +
> > > +    # pdudaemon port
> > > +    pdudaemon_port = '16421'
> > > +    if '--pdudaemon_port' in options:
> > > +        try:
> > > +            pdudaemon_port = options[options.index('--
> pdudaemon_port')+1]
> > > +        except IndexError:
> > > +            error_out('Value not provided after --pdudaemon_port')
> > > +        options.remove(pdudaemon_port)
> > > +        options.remove('--pdudaemon_port')
> > > +
> > > +    # pdu hostname
> > > +    if 'PDU_HOSTNAME' in bvars:
> > > +        pdu_hostname = bvars['PDU_HOSTNAME']
> > > +    else:
> > > +        error_out('PDU_HOSTNAME not specified in board file')
> > > +
> > > +    # pdu port
> > > +    if 'PDU_PORT' in bvars:
> > > +        pdu_port = bvars['PDU_PORT']
> > > +    else:
> > > +        error_out('PDU_PORT not specified in board file')
> > > +
> > > +    # command (on|off|reboot)
> > > +    if '--command' in options:
> > > +        try:
> > > +            command = options[options.index('--command')+1]
> > > +        except IndexError:
> > > +            error_out('Value not provided after --command')
> > > +        options.remove(command)
> > > +        options.remove('--command')
> > > +        if command not in ['on', 'off', 'reboot']:
> > > +            error_out('Unrecognized PDU command %s' % command)
> > > +    else:
> > > +        error_out('Command not supplied')
> > > +
> > > +    query='&'.join(['hostname='+pdu_hostname, 'port='+pdu_port])
> > > +    url = 'http://%s:%s/power/control/%s?%s' %
> (pdudaemon_hostname,
> > > pdudaemon_port, command, query)
> > > +    resp = requests.get(url)
> > > +    if resp.status_code != 200:
> > > +        error_out("pdudaemon did not accept the request")
> > > +    return 0
> > > +
> > >  def main():
> > >      # use global module names
> > >      global re, time, copy2, subprocess, signal, fcntl, requests, json
> > > @@ -5094,6 +5170,10 @@ def main():
> > >          do_delete_var(bvars, options)
> > >          sys.exit(0)
> > >
> > > +    if command == "pdu":
> > > +        rcode = do_pdu(bvars, conf, options)
> > > +        sys.exit(rcode)
> > > +
> > >      error_out("Unknown command %s" % command)
> > >
> > >
> > > diff --git a/scripts/ftc_completion.sh b/scripts/ftc_completion.sh
> > > index 984e003..2cfb77a 100755
> > > --- a/scripts/ftc_completion.sh
> > > +++ b/scripts/ftc_completion.sh
> > > @@ -55,6 +55,7 @@ _ftc()
> > >          ["list-tests"]="-q"
> > >          ["package-run"]="-o -f"
> > >          ["package-test"]="-o -f"
> > > +        ["pdu"]="-b --command --pdudaemon_hostname --
> pdudaemon_port"
> > >          ["put-request"]="-R -s"
> > >          ["put-run"]=""
> > >          ["put-test"]=""
> > > @@ -123,6 +124,10 @@ _ftc()
> > >              COMPREPLY=( $( compgen -W "true false" -- "$cur" ) )
> > >              return 0
> > >              ;;
> > > +        '--command')
> > > +            COMPREPLY=( $( compgen -W "on off reboot" -- "$cur" ) )
> > > +            return 0
> > > +            ;;
> > >          'put-run')
> > >              COMPREPLY=( $( compgen -W "$(ftc list-runs -q)" -- "$cur" ) )
> > >              return 0
> > > --
> > > 2.7.4
> > >
> > > _______________________________________________
> > > Fuego mailing list
> > > Fuego@lists.linuxfoundation.org
> > > https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] [PATCH] ftc: add a new command to interact with pdudaemon

[daniel.sangorrin]

Hi Tim,

> -----Original Message-----
> From: Tim.Bird@sony.com <Tim.Bird@sony.com>
> Sent: Wednesday, March 27, 2019 2:08 AM
> To: sangorrin daniel(サンゴリン ダニエル ○ＳＷＣ□ＯＳＴ) <daniel.sangorrin@toshiba.co.jp>;
> fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] [PATCH] ftc: add a new command to interact with pdudaemon
> 
> > -----Original Message-----
> > From: daniel.sangorrin@toshiba.co.jp
> >
> > Hello Tim,
> >
> > > -----Original Message-----
> > > From: Tim.Bird@sony.com <Tim.Bird@sony.com>
> > >
> > > Here are some overall comments:
> > >
> > > Rather than add a new command to ftc for managing pdudaemon
> > operations,
> > > I was expecting to install support for pdudaemon at Fuego's board
> > management (/board_control)
> > > layer.  That is, I was expecting to see something like this:
> > >
> > > diff --git a/overlays/base/base-board.fuegoclass b/overlays/base/base-
> > board.fuegoclass
> > > index ae35464..8f34922 100644
> > > --- a/overlays/base/base-board.fuegoclass
> > > +++ b/overlays/base/base-board.fuegoclass
> > > @@ -299,6 +299,9 @@ function ov_board_control_reboot() {
> > >    "ttc")
> > >      $TTC $TTC_TARGET reboot
> > >      ;;
> > > +  "pdudaemon"):
> > > +    curl
> > >
> > "http://$PDUDAEMON_HOST:$PDUDAEMON_PORT/power/control/reboot?
> > hostname=$PDU_HOSTNAME&por
> > > t=$PDU_PORT"
> > > +    ;;
> > >    *)
> > >      abort_job "Error reason: unsupported BOARD_CONTROL
> > ${BOARD_CONTROL}"
> > >      ;;
> > > ----
> >
> > I have tried to do it this way but I met some difficulties.
> >
> > Suppose that I call
> > ftc power-on-board -b bbb
> >
> > Then, to be able to execute ov_board_control_reboot (or
> > ov_board_control_power_on) I would need to do like ftc run-test and
> > export lots of variables and call the overlays generator etc. This is probably a
> > hint that the architecture is not correct.
> 
> You are right.  I didn't give this enough thought.  I mirrored the way we
> did abstractions for transports, by putting the abstraction in the
> overlay layer.  Quite frankly, that's an abstraction layer I haven't ever
> been happy with.  It's way more complicated than it needs to be,
> and behaves in ways that are unexpected to the user.  (It violates
> the rule of least surprise).  The other problems with it are
> too numerous to list here, but it has several issues.
> 
> Just to be clear, that layer was created before I became involved with
> this project.
> 
> Having a shell-based abstraction layer made sense when the tests
> were executed directly as shell scripts. 'ftc' didn't exist when the
> overlay layer was created.
> 
> ftc allows for the notion that you may want to allow sub-operations
> to be called from the command line.  We do this for nested tests, as
> well as for reading Fuego and board variables.  Performing sub-operations
> can occur either outside or inside the context of a test.
> That is, someone may want to manually perform a sub-operation at
> the command line, or use 'ftc' to perform the sub-operation from inside a test.
> 
> By way of contrast, we don't have a mechanism for doing TRANSPORT
> operations directly using ttc, but it would obviously be useful.
> (For example, it would be nice to be able to do get or put operations with
> 'ftc put -b bbb somefile /tmp' or (or, with syntactic sugar, make that:
> 'ftc put somefile bbb:/tmp').  And it would be nice to be able to do:
> 'ftc cmd -b bbb <some_command>')

Good point.
Looking at 'ttc' it sounds like you already had all these stuff figured out a long time ago.
It looks like 'ttc' could be integrated into Fuego, what do you think?

> The same might be said for our 'DISTRO' abstractions.  But now I'm getting
> off-topic.  Getting back to just the board control operations...
> 
> I personally haven't needed manual board control operations from the
> 'ftc' command line because I use 'ttc'.  So that's probably been a blind
> spot for me in the architecture.  I've been hesitating on this, because it's
> not clear to me that we want to define a new, Fuego-specific set of operations
> for board control.  But we have to have something that manages the abstractions
> (so we're not hardcoded to pdudaemon, labgrid, LAVA or ttc) and if we're going
> to consolidate our abstractions in one place, it should probably be in 'ftc'.

Yes, we need an abstraction either in ftc or in ttc.
Pdudaemon is easy to integrate.
I would like to also add virtual machine support (vagrant down, vagrant up etc)

> I haven't answered your other e-mail.  I'll try to go back and do that today.
> But I'll put some comments that I had intended for that e-mail below, along
> with some notes.
> 
> >
> > If we do the control from ftc things will be smoother. Summarizing, this is the
> > patch that i plan to submit:
> >
> > - add three commands to ftc
> > ftc power-on-board
> > ftc power-off-board
> > ftc reboot-board
> Just a note on naming.  I'm trying to make the naming for 'ftc' consistent by having
> the command names be <verb>-<object>.  In the case of the the 'power' commands
> this results in somewhat awkward names, as there is no single word commonly used
> in English that I am aware of to "turn on" or apply power to an item.  So these turn into
> phrases, which is less than ideal.  But hopefully that explains the names I suggested.
> 
> > [Note] depending on the BOARD_CONTROL variable (bvars) use ttc or
> > pdudaemon
> This will still require reading the board file internally.  
> But 1) ftc has internal functions for that already, 

Yes, this is what I meant with 'bvars'.

> and 2) it might not require processing the
> overlay as you could just use specific variables (ie "BOARD_CONTROL")
> without having to perform the overlay operation.  The overlay
> operation is only needed if you are drilling down through the distro
> or transport layer to get to a function.  This assumes that you don't allow
> the user or test to override the BOARD_CONTROL variable with a stored variable.
> (See http://fuegotest.org/wiki/Test_variables#Stored_variables), which I think
> is a reasonable limitation.

Correct.

> > - remove ov_board_control_reboot
> 
> I don't want to hardcode the board management layer inside 'ftc'.
> I was planning on using the board management layer not just for power
> control, but also for provisioning, and possibly state control of the bootup sequence
> (like labgrid does, and in support of that integrating Fuego and labgrid).

I have to study labgrid, but it looks like using ttc wait-for (or the ftc expect that I proposed) we can already provision boards. For example, "ttc wait-for" can wait for u-boot's prompt on the serial port and write the necessary tftpboot commands to provision the board.
In the future, we could even place a job.yaml layer that could be decomposed into ftc operations and support both imperative and declarative test plans.

> So, getting more specific.  That call to 'curl' that is used to communicate with
> pdudaemon has to be coded somewhere.  If it's not in the overlay system,
> or in 'ftc' proper, then there needs to be a new plugin system for
> providing those commands, that could be selected (de-multiplexed) based
> on the BOARD_CONTROL variable.
> 
> I have some ideas for this, but it seems like the choices would be to implement
> this as 1) a python module, 2) some shell-based system, 3) some configuration-based
> system, allowing simple multi-line commands.  Possibly, we could move forward
> with putting the code into ftc, and then create the plugin layer afterwards, but
> if possible I'd like to do this at the same time that we introduce these ftc commands
> and wire up our first abstraction.

It would be nice if BOARD_CONTROL accepts a set of predefined values (pdudaemon, ttc, labgrid, custom). This values would be multiplexed in ftc and the code for each value would be in a separate python module. The "custom" value would require one more board variable ("BOARD_CONTROL_CMD") to define a shell script (e.g. my_board_control.sh COMMAND), where COMMAND can be "on, off, reboot".

> > - remove calls to ov_board_control_reboot during pre_test and post_test
> >    - option 1: ask the user to call ftc power-on-board, and then ftc wait-for
> > (e.g. wait for the prompt or something else)
> >    - option 2: substitute it by ftc reboot-board and add a new variable
> > BOARD_CONTROL_AUTOMATIC to the board file (true: reboot board before
> > and after test, false: I will call ftc reboot-board whenever I want)
> 
> I think I understand some (but not all) of the use cases for this new variable.
> We might need to discuss this more.

OK. What I mean is that in its current state, if we define BOARD_CONTROL Fuego is going to automatically power on and off the board by default. This can be annoying if you want to keep the board on, for example during a testplan.

> > - remove calls to ov_board_control_reboot from Benchmark.reboot
> >    - substitute it by a nested call to "ftc reboot-board -b $NODE_NAME"
> It may be desirable to have a Fuego shell function to wrap that, like 'do_hardware_reboot' or
> something, but that's an implementation detail.

Good idea.

Thanks,
Daniel

> Sorry this is so long-winded.
> 
> I think I agree in principle with the proposed changes, but  we may need to
> have more discussion about the pros and cons of having the board control
> layer be pluggable.
>  -- Tim
> 
> 
> > Any thoughts?
> >
> > Thanks,
> > Daniel
> >
> >
> > >
> > > Note that I don't know if pdudaemon supports the 'reboot' control option,
> > or if I would have
> > > to synthesize something by sending 'on' and 'off'.  But you get the idea, I
> > think.
> > > I'm not sure if the hostname in the URL is supposed to be the board name
> > or the pdu name,
> > > but presumably PDUDAEMON_HOST and PDUDAEMON_PORT would come
> > from
> > > fuego.conf, and PDU_HOSTNAME and PDU_PORT would come from the
> > board file?
> > >
> > > If more fine-grained control is desired, than just the 'reboot' operation,
> > > we could add more ov_board_control() functions if needed, and have
> > Fuego use those.
> > > e.g. ov_board_control_on() and ov_board_control_off().
> > >
> > > Once we've plugged pdudaemon into the board layer, I'd prefer to have a
> > generic
> > > set of operations that someone can use from ftc, like:
> > >    ftc reboot-board -b <board>
> > >    ftc power-on-board -b <board>
> > >    ftc power-off-board -b <board>
> > > and maybe even
> > >    ftc show-status-of-board <board>
> > >
> > > In some labs, there may be a layer between pdudaemon and Fuego (for
> > example libvirt,
> > > or labgrid or even LAVA).
> > >
> > > I think I'd rather have an abstraction layer between Fuego and
> > pdudaemon, rather than
> > > integrating it directly into our user interface.
> > >
> > > Thoughts?
> > >  -- tim
> > >
> > >
> > > > -----Original Message-----
> > > > From: Daniel Sangorrin
> > > >
> > > > This patch adds the "pdu" command to send requests for pdudaemon
> > > > to power-on/off or reboot a board.
> > > >
> > > > Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> > > > ---
> > > >  scripts/ftc               | 80
> > > > +++++++++++++++++++++++++++++++++++++++++++++++
> > > >  scripts/ftc_completion.sh |  5 +++
> > > >  2 files changed, 85 insertions(+)
> > > >
> > > > diff --git a/scripts/ftc b/scripts/ftc
> > > > index 7153c01..30ea470 100755
> > > > --- a/scripts/ftc
> > > > +++ b/scripts/ftc
> > > > @@ -426,6 +426,29 @@ A message and the exit code indicate if the
> > resource
> > > > was released.
> > > >
> > > >     ex: ftc release-resource -b beaglebone"""),
> > > >
> > > > +"pdu": ("Send a request to pdudaemon.",
> > > > +    """Usage: ftc pdu -b <board> --command on|off|reboot [--
> > > > pdudaemon_hostname localhost --pdudaemon_port 16421]
> > > > +
> > > > +This command is used to send a request for pdudaemon to power-on,
> > > > power-off or reboot a board.
> > > > +
> > > > +The command accepts these arguments:
> > > > + - board file: bbb, raspberrypi3, etc (see ftc list-boards)
> > > > + - command: on|off|reboot
> > > > + - pdudaemon_hostname: hostname or IP address where pdudaemon is
> > > > running (default: localhost)
> > > > + - pdudaemon_port: port where pdudaemon is listening (default: 16421)
> > > > +   [Note] make sure that pdudaemon is configured to accept http
> > requests
> > > > +
> > > > +Additionally, you need to define two variables in your board file:
> > > > + - PDU_HOSTNAME: the PDU hostname or IP address used in the
> > > > pdudaemon configuration file
> > > > + - PDU_PORT: the PDU port number (a plug port, not a tcp port)
> > > > +
> > > > +You need to install pdudaemon by yourself
> > > > +  $ git clone https://github.com/pdudaemon/pdudaemon
> > > > +  $ cd pdudaemon
> > > > +  $ less README.md
> > > > +
> > > > +Example: ftc pdu -b raspberrypi3 --command on"""),
> > > > +
> > > >  "package-test": ("Package a test.",
> > > >      """Usage: ftc package-test <test_name> [options]
> > > >
> > > > @@ -4783,6 +4806,59 @@ def do_release_resource(conf, options):
> > > >
> > > >      return 0
> > > >
> > > > +def do_pdu(bvars, conf, options):
> > > > +    # pdudaemon hostname
> > > > +    pdudaemon_hostname = 'localhost'
> > > > +    if '--pdudaemon_hostname' in options:
> > > > +        try:
> > > > +            pdudaemon_hostname = options[options.index('--
> > > > pdudaemon_hostname')+1]
> > > > +        except IndexError:
> > > > +            error_out('Value not provided after --pdudaemon_hostname')
> > > > +        options.remove(pdudaemon_hostname)
> > > > +        options.remove('--pdudaemon_hostname')
> > > > +
> > > > +    # pdudaemon port
> > > > +    pdudaemon_port = '16421'
> > > > +    if '--pdudaemon_port' in options:
> > > > +        try:
> > > > +            pdudaemon_port = options[options.index('--
> > pdudaemon_port')+1]
> > > > +        except IndexError:
> > > > +            error_out('Value not provided after --pdudaemon_port')
> > > > +        options.remove(pdudaemon_port)
> > > > +        options.remove('--pdudaemon_port')
> > > > +
> > > > +    # pdu hostname
> > > > +    if 'PDU_HOSTNAME' in bvars:
> > > > +        pdu_hostname = bvars['PDU_HOSTNAME']
> > > > +    else:
> > > > +        error_out('PDU_HOSTNAME not specified in board file')
> > > > +
> > > > +    # pdu port
> > > > +    if 'PDU_PORT' in bvars:
> > > > +        pdu_port = bvars['PDU_PORT']
> > > > +    else:
> > > > +        error_out('PDU_PORT not specified in board file')
> > > > +
> > > > +    # command (on|off|reboot)
> > > > +    if '--command' in options:
> > > > +        try:
> > > > +            command = options[options.index('--command')+1]
> > > > +        except IndexError:
> > > > +            error_out('Value not provided after --command')
> > > > +        options.remove(command)
> > > > +        options.remove('--command')
> > > > +        if command not in ['on', 'off', 'reboot']:
> > > > +            error_out('Unrecognized PDU command %s' % command)
> > > > +    else:
> > > > +        error_out('Command not supplied')
> > > > +
> > > > +    query='&'.join(['hostname='+pdu_hostname, 'port='+pdu_port])
> > > > +    url = 'http://%s:%s/power/control/%s?%s' %
> > (pdudaemon_hostname,
> > > > pdudaemon_port, command, query)
> > > > +    resp = requests.get(url)
> > > > +    if resp.status_code != 200:
> > > > +        error_out("pdudaemon did not accept the request")
> > > > +    return 0
> > > > +
> > > >  def main():
> > > >      # use global module names
> > > >      global re, time, copy2, subprocess, signal, fcntl, requests, json
> > > > @@ -5094,6 +5170,10 @@ def main():
> > > >          do_delete_var(bvars, options)
> > > >          sys.exit(0)
> > > >
> > > > +    if command == "pdu":
> > > > +        rcode = do_pdu(bvars, conf, options)
> > > > +        sys.exit(rcode)
> > > > +
> > > >      error_out("Unknown command %s" % command)
> > > >
> > > >
> > > > diff --git a/scripts/ftc_completion.sh b/scripts/ftc_completion.sh
> > > > index 984e003..2cfb77a 100755
> > > > --- a/scripts/ftc_completion.sh
> > > > +++ b/scripts/ftc_completion.sh
> > > > @@ -55,6 +55,7 @@ _ftc()
> > > >          ["list-tests"]="-q"
> > > >          ["package-run"]="-o -f"
> > > >          ["package-test"]="-o -f"
> > > > +        ["pdu"]="-b --command --pdudaemon_hostname --
> > pdudaemon_port"
> > > >          ["put-request"]="-R -s"
> > > >          ["put-run"]=""
> > > >          ["put-test"]=""
> > > > @@ -123,6 +124,10 @@ _ftc()
> > > >              COMPREPLY=( $( compgen -W "true false" -- "$cur" ) )
> > > >              return 0
> > > >              ;;
> > > > +        '--command')
> > > > +            COMPREPLY=( $( compgen -W "on off reboot" -- "$cur" ) )
> > > > +            return 0
> > > > +            ;;
> > > >          'put-run')
> > > >              COMPREPLY=( $( compgen -W "$(ftc list-runs -q)" -- "$cur" ) )
> > > >              return 0
> > > > --
> > > > 2.7.4
> > > >
> > > > _______________________________________________
> > > > Fuego mailing list
> > > > Fuego@lists.linuxfoundation.org
> > > > https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] [PATCH] ftc: add a new command to interact with pdudaemon

[daniel.sangorrin]

> -----Original Message-----
> From: Tim.Bird@sony.com <Tim.Bird@sony.com>
> Sent: Wednesday, March 27, 2019 6:09 AM
> To: sangorrin daniel(サンゴリン ダニエル ○ＳＷＣ□ＯＳＴ) <daniel.sangorrin@toshiba.co.jp>;
> fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] [PATCH] ftc: add a new command to interact with pdudaemon
> 
> More feedback on this topic...  see below.
> 
> > -----Original Message-----
> > From: daniel.sangorrin@toshiba.co.jp
> >
> > Hi Tim,
> >
> > Thanks again for reviewing in your busy time.
> >
> > > -----Original Message-----
> > > From: Tim.Bird@sony.com <Tim.Bird@sony.com>
> > > Sent: Thursday, March 14, 2019 4:55 AM
> > > To: sangorrin daniel(サンゴリン ダニエル ○ＳＷＣ□ＯＳＴ)
> > <daniel.sangorrin@toshiba.co.jp>;
> > > fuego@lists.linuxfoundation.org
> > > Subject: RE: [Fuego] [PATCH] ftc: add a new command to interact with
> > pdudaemon
> > >
> > > Here are some overall comments:
> > >
> > > Rather than add a new command to ftc for managing pdudaemon
> > operations,
> > > I was expecting to install support for pdudaemon at Fuego's board
> > management (/board_control)
> > > layer.  That is, I was expecting to see something like this:
> > >
> > > diff --git a/overlays/base/base-board.fuegoclass b/overlays/base/base-
> > board.fuegoclass
> > > index ae35464..8f34922 100644
> > > --- a/overlays/base/base-board.fuegoclass
> > > +++ b/overlays/base/base-board.fuegoclass
> > > @@ -299,6 +299,9 @@ function ov_board_control_reboot() {
> > >    "ttc")
> > >      $TTC $TTC_TARGET reboot
> > >      ;;
> > > +  "pdudaemon"):
> > > +    curl
> > >
> > "http://$PDUDAEMON_HOST:$PDUDAEMON_PORT/power/control/reboot?
> > hostname=$PDU_HOSTNAME&por
> > > t=$PDU_PORT"
> > > +    ;;
> > >    *)
> > >      abort_job "Error reason: unsupported BOARD_CONTROL
> > ${BOARD_CONTROL}"
> > >      ;;
> > > ----
> >
> > Alright, that sounds like a good idea. I will resend.
> >
> > > Note that I don't know if pdudaemon supports the 'reboot' control option,
> > or if I would have
> > reboot is supported
> >
> > > to synthesize something by sending 'on' and 'off'.  But you get the idea, I
> > think.
> > > I'm not sure if the hostname in the URL is supposed to be the board name
> > or the pdu name,
> > PDU_HOSTNAME is the PDU's IP address/hostname.
> > PDUDAEMON_HOSTNAME is the IP address/hostname of the computer
> > where pdudaemon is running.
> > The board doesn't know that it can be "killed" at anytime by the PDU.
> >
> > > but presumably PDUDAEMON_HOST and PDUDAEMON_PORT would come
> > from
> > > fuego.conf, and PDU_HOSTNAME and PDU_PORT would come from the
> > board file?
> > From the board file because each board can be in a different place. These
> > two parameters are passed to the pdudaemon. Pdudaemon then internally
> > uses them to connect with the PDU by the supported protocol (e.g.:
> > SNMTP).
> 
> OK. As discussed in the other thread, this means that the board file needs
> to be parsed by ftc to execute these, to get these board-related test variables.
> For now I believe it's OK to ignore processing overlays and stored variables for
> this.  However, I think the ftc internal function get_board_vars() does all this processing
> anyway, so it might just be easier to use that.

Yes, this works fine. I already have it implemented and it's ready for demultiplexing.

> > > If more fine-grained control is desired, than just the 'reboot' operation,
> > > we could add more ov_board_control() functions if needed, and have
> > Fuego use those.
> > > e.g. ov_board_control_on() and ov_board_control_off().
> >
> > Yes, I think we need on off.
> > In particular, I was thinking about Fuego testplans that would work like this:
> >
> > ftc pdu -b bbb --command on
> > ftc expect -b bbb --regex "u-boot>" --send SIGINT
> > ftc send -b bbb --command "run tftpboot"
> > ftc expect -b bbb --regex "mypromptlinuxprompt" --send "root"
> > echo ftc pdu -b bbb --command off | at now + 1 minute
> > ftc run-test -b bbb -t Functional.StressFilesystem
> > sleep 60
> > ftc pdu -b bbb --command on
> > ftc expect -b bbb --regex myprompt
> > ftc run-test -b bbb -t Functional.ext4
> > ftc pdu -b bbb --command off
> 
> A couple of notes here:
> 1) It looks like get_board_arg() looks for FUEGO_BOARD
> in the environment, and uses that if present.
> I'm pretty sure you can do: export FUEGO_BOARD=bbb, and then omit the '-b bbb' on every line.
> From inside a test, maybe we should make this automatic, and set FUEGO_BOARD from
> NODE_NAME when we do run-test.

I didn't know.
I also had the idea to add a patch that would assume the board 'local' if -b was not provided.

> 2) 'ftc expect' looks nice, and it shouldn't be too hard to do since python has an 'expect' module
> (famous last words... :-)

Actually, after looking at your ttc application it looks like you already have an implementation (ttc wait-for). I guess it should be quite similar.

> 3) We should discuss the high-level operations we want to do, and make
> ftc operations for those.   For some tests, the details of how the board is booted
> and provisioned should be opaque to the test.

Yes, sure.
I think that we are arriving to what LAVA defines in yaml:
-power on
-wait for u-boot
-send commands to u-boot to provision the board
-wait for login
-run the test
...

So probably Fuego users will end up defining "test jobs" (similar to LAVA jobs) where they prepare a sequence of ftc instructions (power-on-board, wait-for, run-test, run-test again, power-off-board, et..). And we could even put a yaml layer on top of that (similar to LAVA syntax).

> One issue I see with your low-level steps, is that they encode
> things that are board-specific (like the boot prompt regexes) into
> the commands
> I'm not opposed, to low-level operations being encoded into tests.
> It's important, though, to try to isolate board-specific items so that the
> tests don't become dependent on a specific board, bootloader, distro,
> board control layer, or even operating system.  (e.g. I've started work
> on supporting Nuttx testing with Fuego).

Sure, we can create BOARD variables as necessary, and still allow users to override them, optionally, on the command line for quick testing.

> Maybe you intend to parameterize those regexes from the board file.
> But this still encapsulates too much of the way a system boots and
> this particular example I believe requires that you have a serial port.
> That wouldn't be true, for example, with a mobile phone, where you
> have adb access, but often not a serial port.

I think we need to consider boards that have a combination of serial and ssh access.
Serial could be used for booting/provisioning and SSH for deploying/getting results/running commands etc.

Also, what I am talking about is not a "test" but rather a "test job" with a sequence of ftc commands that would be customized by users and not necessarily shared.

> Here's how I've used ttc for board control and provisioning:
> ttc setenv bbb  # this allows me to omit 'bbb' on every following command line
> ttc get_kernel
> ttc set_config CONFIG_SOME_KCONFIG=y;CONFIG_SOME_VALUE=255
> ttc kbuild
> ttc kinstall
> ttc reboot
> ttc wait-for "ttc run 'echo alive'"
> ttc cp testfiles target:/test_directory
> ttc run "some commands"
> ttc cp target:/test_directory/testlogs host_directory

This sequence is what I am calling a "test job" (not to confuse with Jenkins jobs) and I think it works pretty well.
If we were going to integrate that with LAVA, most of the steps would be done by LAVA and we would only need the "ttc run ..." part.

> > And we can make more complicated things.
> > [Note] The ftc expect is something I am designing now to interact with the
> > board in a flexible way, and provide LAVA-like functionality.
> I'd like to have a discussion with you next week about the best
> approach in Fuego to handle provisioning, and building up complex test
> pipelines.  I think some sub-operations should be in ftc, and maybe
> some should be put into tests that call sub-tests.

Oh "test pipelines" sounds better than "test jobs". Sure, let's discuss that.

Thanks,
Daniel

> 
>  -- Tim
> 
> >
> > > Once we've plugged pdudaemon into the board layer, I'd prefer to have a
> > generic
> > > set of operations that someone can use from ftc, like:
> > >    ftc reboot-board -b <board>
> > >    ftc power-on-board -b <board>
> > >    ftc power-off-board -b <board>
> > > and maybe even
> > >    ftc show-status-of-board <board>
> >
> > Maybe we can use subcommands like this.
> > ftc power reboot -b bbb
> > ftc power on -b board
> > ftc power off -b board
> > ftc power status -b board
> >
> > Or put hyphens, but it would rather have the same 1st work for bash
> > autocomplete to be more intuitive.
> >
> > > In some labs, there may be a layer between pdudaemon and Fuego (for
> > example libvirt,
> > > or labgrid or even LAVA).
> > >
> > > I think I'd rather have an abstraction layer between Fuego and
> > pdudaemon, rather than
> > > integrating it directly into our user interface.
> > >
> > > Thoughts?
> >
> > See above.
> > Thanks,
> > Daniel
> >
> > >  -- tim
> > >
> > >
> > > > -----Original Message-----
> > > > From: Daniel Sangorrin
> > > >
> > > > This patch adds the "pdu" command to send requests for pdudaemon
> > > > to power-on/off or reboot a board.
> > > >
> > > > Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> > > > ---
> > > >  scripts/ftc               | 80
> > > > +++++++++++++++++++++++++++++++++++++++++++++++
> > > >  scripts/ftc_completion.sh |  5 +++
> > > >  2 files changed, 85 insertions(+)
> > > >
> > > > diff --git a/scripts/ftc b/scripts/ftc
> > > > index 7153c01..30ea470 100755
> > > > --- a/scripts/ftc
> > > > +++ b/scripts/ftc
> > > > @@ -426,6 +426,29 @@ A message and the exit code indicate if the
> > resource
> > > > was released.
> > > >
> > > >     ex: ftc release-resource -b beaglebone"""),
> > > >
> > > > +"pdu": ("Send a request to pdudaemon.",
> > > > +    """Usage: ftc pdu -b <board> --command on|off|reboot [--
> > > > pdudaemon_hostname localhost --pdudaemon_port 16421]
> > > > +
> > > > +This command is used to send a request for pdudaemon to power-on,
> > > > power-off or reboot a board.
> > > > +
> > > > +The command accepts these arguments:
> > > > + - board file: bbb, raspberrypi3, etc (see ftc list-boards)
> > > > + - command: on|off|reboot
> > > > + - pdudaemon_hostname: hostname or IP address where pdudaemon is
> > > > running (default: localhost)
> > > > + - pdudaemon_port: port where pdudaemon is listening (default: 16421)
> > > > +   [Note] make sure that pdudaemon is configured to accept http
> > requests
> > > > +
> > > > +Additionally, you need to define two variables in your board file:
> > > > + - PDU_HOSTNAME: the PDU hostname or IP address used in the
> > > > pdudaemon configuration file
> > > > + - PDU_PORT: the PDU port number (a plug port, not a tcp port)
> > > > +
> > > > +You need to install pdudaemon by yourself
> > > > +  $ git clone https://github.com/pdudaemon/pdudaemon
> > > > +  $ cd pdudaemon
> > > > +  $ less README.md
> > > > +
> > > > +Example: ftc pdu -b raspberrypi3 --command on"""),
> > > > +
> > > >  "package-test": ("Package a test.",
> > > >      """Usage: ftc package-test <test_name> [options]
> > > >
> > > > @@ -4783,6 +4806,59 @@ def do_release_resource(conf, options):
> > > >
> > > >      return 0
> > > >
> > > > +def do_pdu(bvars, conf, options):
> > > > +    # pdudaemon hostname
> > > > +    pdudaemon_hostname = 'localhost'
> > > > +    if '--pdudaemon_hostname' in options:
> > > > +        try:
> > > > +            pdudaemon_hostname = options[options.index('--
> > > > pdudaemon_hostname')+1]
> > > > +        except IndexError:
> > > > +            error_out('Value not provided after --pdudaemon_hostname')
> > > > +        options.remove(pdudaemon_hostname)
> > > > +        options.remove('--pdudaemon_hostname')
> > > > +
> > > > +    # pdudaemon port
> > > > +    pdudaemon_port = '16421'
> > > > +    if '--pdudaemon_port' in options:
> > > > +        try:
> > > > +            pdudaemon_port = options[options.index('--
> > pdudaemon_port')+1]
> > > > +        except IndexError:
> > > > +            error_out('Value not provided after --pdudaemon_port')
> > > > +        options.remove(pdudaemon_port)
> > > > +        options.remove('--pdudaemon_port')
> > > > +
> > > > +    # pdu hostname
> > > > +    if 'PDU_HOSTNAME' in bvars:
> > > > +        pdu_hostname = bvars['PDU_HOSTNAME']
> > > > +    else:
> > > > +        error_out('PDU_HOSTNAME not specified in board file')
> > > > +
> > > > +    # pdu port
> > > > +    if 'PDU_PORT' in bvars:
> > > > +        pdu_port = bvars['PDU_PORT']
> > > > +    else:
> > > > +        error_out('PDU_PORT not specified in board file')
> > > > +
> > > > +    # command (on|off|reboot)
> > > > +    if '--command' in options:
> > > > +        try:
> > > > +            command = options[options.index('--command')+1]
> > > > +        except IndexError:
> > > > +            error_out('Value not provided after --command')
> > > > +        options.remove(command)
> > > > +        options.remove('--command')
> > > > +        if command not in ['on', 'off', 'reboot']:
> > > > +            error_out('Unrecognized PDU command %s' % command)
> > > > +    else:
> > > > +        error_out('Command not supplied')
> > > > +
> > > > +    query='&'.join(['hostname='+pdu_hostname, 'port='+pdu_port])
> > > > +    url = 'http://%s:%s/power/control/%s?%s' %
> > (pdudaemon_hostname,
> > > > pdudaemon_port, command, query)
> > > > +    resp = requests.get(url)
> > > > +    if resp.status_code != 200:
> > > > +        error_out("pdudaemon did not accept the request")
> > > > +    return 0
> > > > +
> > > >  def main():
> > > >      # use global module names
> > > >      global re, time, copy2, subprocess, signal, fcntl, requests, json
> > > > @@ -5094,6 +5170,10 @@ def main():
> > > >          do_delete_var(bvars, options)
> > > >          sys.exit(0)
> > > >
> > > > +    if command == "pdu":
> > > > +        rcode = do_pdu(bvars, conf, options)
> > > > +        sys.exit(rcode)
> > > > +
> > > >      error_out("Unknown command %s" % command)
> > > >
> > > >
> > > > diff --git a/scripts/ftc_completion.sh b/scripts/ftc_completion.sh
> > > > index 984e003..2cfb77a 100755
> > > > --- a/scripts/ftc_completion.sh
> > > > +++ b/scripts/ftc_completion.sh
> > > > @@ -55,6 +55,7 @@ _ftc()
> > > >          ["list-tests"]="-q"
> > > >          ["package-run"]="-o -f"
> > > >          ["package-test"]="-o -f"
> > > > +        ["pdu"]="-b --command --pdudaemon_hostname --
> > pdudaemon_port"
> > > >          ["put-request"]="-R -s"
> > > >          ["put-run"]=""
> > > >          ["put-test"]=""
> > > > @@ -123,6 +124,10 @@ _ftc()
> > > >              COMPREPLY=( $( compgen -W "true false" -- "$cur" ) )
> > > >              return 0
> > > >              ;;
> > > > +        '--command')
> > > > +            COMPREPLY=( $( compgen -W "on off reboot" -- "$cur" ) )
> > > > +            return 0
> > > > +            ;;
> > > >          'put-run')
> > > >              COMPREPLY=( $( compgen -W "$(ftc list-runs -q)" -- "$cur" ) )
> > > >              return 0
> > > > --
> > > > 2.7.4
> > > >
> > > > _______________________________________________
> > > > Fuego mailing list
> > > > Fuego@lists.linuxfoundation.org
> > > > https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] [PATCH] ftc: add a new command to interact with pdudaemon

[Tim.Bird]

Daniel,

I've kind of lost track of where we are with this.  Can you update me on the status
of new ftc commands, and their integration with pdudaemon?

Are we waiting for pdudaemon to change its API, or are we moving ahead with
the interface to is via 'curl'?

And where are we putting the connection between board and (pdu and port)?

Do we need a new kind of test object?   I've been thinking of having test objects
that are used for different things, like Monitors, Stressors, and Provisioners.  A monitor
would be executed in parallel with a test, to gather information from another device
like a power meter.  A Stressor would also run in parallel, to put a known workload
on a device while another test is running.  And  a Provisioner would be run prior to
a test, in order to prepare the board for the test by deploying the software under
test and booting it.

So, I'm starting work on the API for starting, checking the running status of, and stopping
these other test processes.  Let me know what you think, or if you have some example
APIs I should look at for inspiration.

Thanks,
 -- Tim

> -----Original Message-----
> From: daniel.sangorrin@toshiba.co.jp
> 
> > -----Original Message-----
> > From: Tim.Bird@sony.com <Tim.Bird@sony.com>
> > Sent: Wednesday, March 27, 2019 6:09 AM
> > To: sangorrin daniel(サンゴリン ダニエル ○ＳＷＣ□ＯＳＴ)
> <daniel.sangorrin@toshiba.co.jp>;
> > fuego@lists.linuxfoundation.org
> > Subject: RE: [Fuego] [PATCH] ftc: add a new command to interact with
> pdudaemon
> >
> > More feedback on this topic...  see below.
> >
> > > -----Original Message-----
> > > From: daniel.sangorrin@toshiba.co.jp
> > >
> > > Hi Tim,
> > >
> > > Thanks again for reviewing in your busy time.
> > >
> > > > -----Original Message-----
> > > > From: Tim.Bird@sony.com <Tim.Bird@sony.com>
> > > > Sent: Thursday, March 14, 2019 4:55 AM
> > > > To: sangorrin daniel(サンゴリン ダニエル ○ＳＷＣ□ＯＳＴ)
> > > <daniel.sangorrin@toshiba.co.jp>;
> > > > fuego@lists.linuxfoundation.org
> > > > Subject: RE: [Fuego] [PATCH] ftc: add a new command to interact with
> > > pdudaemon
> > > >
> > > > Here are some overall comments:
> > > >
> > > > Rather than add a new command to ftc for managing pdudaemon
> > > operations,
> > > > I was expecting to install support for pdudaemon at Fuego's board
> > > management (/board_control)
> > > > layer.  That is, I was expecting to see something like this:
> > > >
> > > > diff --git a/overlays/base/base-board.fuegoclass
> b/overlays/base/base-
> > > board.fuegoclass
> > > > index ae35464..8f34922 100644
> > > > --- a/overlays/base/base-board.fuegoclass
> > > > +++ b/overlays/base/base-board.fuegoclass
> > > > @@ -299,6 +299,9 @@ function ov_board_control_reboot() {
> > > >    "ttc")
> > > >      $TTC $TTC_TARGET reboot
> > > >      ;;
> > > > +  "pdudaemon"):
> > > > +    curl
> > > >
> > >
> "http://$PDUDAEMON_HOST:$PDUDAEMON_PORT/power/control/reboot?
> > > hostname=$PDU_HOSTNAME&por
> > > > t=$PDU_PORT"
> > > > +    ;;
> > > >    *)
> > > >      abort_job "Error reason: unsupported BOARD_CONTROL
> > > ${BOARD_CONTROL}"
> > > >      ;;
> > > > ----
> > >
> > > Alright, that sounds like a good idea. I will resend.
> > >
> > > > Note that I don't know if pdudaemon supports the 'reboot' control
> option,
> > > or if I would have
> > > reboot is supported
> > >
> > > > to synthesize something by sending 'on' and 'off'.  But you get the idea,
> I
> > > think.
> > > > I'm not sure if the hostname in the URL is supposed to be the board
> name
> > > or the pdu name,
> > > PDU_HOSTNAME is the PDU's IP address/hostname.
> > > PDUDAEMON_HOSTNAME is the IP address/hostname of the computer
> > > where pdudaemon is running.
> > > The board doesn't know that it can be "killed" at anytime by the PDU.
> > >
> > > > but presumably PDUDAEMON_HOST and PDUDAEMON_PORT would
> come
> > > from
> > > > fuego.conf, and PDU_HOSTNAME and PDU_PORT would come from
> the
> > > board file?
> > > From the board file because each board can be in a different place. These
> > > two parameters are passed to the pdudaemon. Pdudaemon then
> internally
> > > uses them to connect with the PDU by the supported protocol (e.g.:
> > > SNMTP).
> >
> > OK. As discussed in the other thread, this means that the board file needs
> > to be parsed by ftc to execute these, to get these board-related test
> variables.
> > For now I believe it's OK to ignore processing overlays and stored variables
> for
> > this.  However, I think the ftc internal function get_board_vars() does all
> this processing
> > anyway, so it might just be easier to use that.
> 
> Yes, this works fine. I already have it implemented and it's ready for
> demultiplexing.
> 
> > > > If more fine-grained control is desired, than just the 'reboot' operation,
> > > > we could add more ov_board_control() functions if needed, and have
> > > Fuego use those.
> > > > e.g. ov_board_control_on() and ov_board_control_off().
> > >
> > > Yes, I think we need on off.
> > > In particular, I was thinking about Fuego testplans that would work like
> this:
> > >
> > > ftc pdu -b bbb --command on
> > > ftc expect -b bbb --regex "u-boot>" --send SIGINT
> > > ftc send -b bbb --command "run tftpboot"
> > > ftc expect -b bbb --regex "mypromptlinuxprompt" --send "root"
> > > echo ftc pdu -b bbb --command off | at now + 1 minute
> > > ftc run-test -b bbb -t Functional.StressFilesystem
> > > sleep 60
> > > ftc pdu -b bbb --command on
> > > ftc expect -b bbb --regex myprompt
> > > ftc run-test -b bbb -t Functional.ext4
> > > ftc pdu -b bbb --command off
> >
> > A couple of notes here:
> > 1) It looks like get_board_arg() looks for FUEGO_BOARD
> > in the environment, and uses that if present.
> > I'm pretty sure you can do: export FUEGO_BOARD=bbb, and then omit the
> '-b bbb' on every line.
> > From inside a test, maybe we should make this automatic, and set
> FUEGO_BOARD from
> > NODE_NAME when we do run-test.
> 
> I didn't know.
> I also had the idea to add a patch that would assume the board 'local' if -b
> was not provided.
> 
> > 2) 'ftc expect' looks nice, and it shouldn't be too hard to do since python
> has an 'expect' module
> > (famous last words... :-)
> 
> Actually, after looking at your ttc application it looks like you already have an
> implementation (ttc wait-for). I guess it should be quite similar.
> 
> > 3) We should discuss the high-level operations we want to do, and make
> > ftc operations for those.   For some tests, the details of how the board is
> booted
> > and provisioned should be opaque to the test.
> 
> Yes, sure.
> I think that we are arriving to what LAVA defines in yaml:
> -power on
> -wait for u-boot
> -send commands to u-boot to provision the board
> -wait for login
> -run the test
> ...
> 
> So probably Fuego users will end up defining "test jobs" (similar to LAVA
> jobs) where they prepare a sequence of ftc instructions (power-on-board,
> wait-for, run-test, run-test again, power-off-board, et..). And we could even
> put a yaml layer on top of that (similar to LAVA syntax).
> 
> > One issue I see with your low-level steps, is that they encode
> > things that are board-specific (like the boot prompt regexes) into
> > the commands
> > I'm not opposed, to low-level operations being encoded into tests.
> > It's important, though, to try to isolate board-specific items so that the
> > tests don't become dependent on a specific board, bootloader, distro,
> > board control layer, or even operating system.  (e.g. I've started work
> > on supporting Nuttx testing with Fuego).
> 
> Sure, we can create BOARD variables as necessary, and still allow users to
> override them, optionally, on the command line for quick testing.
> 
> > Maybe you intend to parameterize those regexes from the board file.
> > But this still encapsulates too much of the way a system boots and
> > this particular example I believe requires that you have a serial port.
> > That wouldn't be true, for example, with a mobile phone, where you
> > have adb access, but often not a serial port.
> 
> I think we need to consider boards that have a combination of serial and ssh
> access.
> Serial could be used for booting/provisioning and SSH for deploying/getting
> results/running commands etc.
> 
> Also, what I am talking about is not a "test" but rather a "test job" with a
> sequence of ftc commands that would be customized by users and not
> necessarily shared.
> 
> > Here's how I've used ttc for board control and provisioning:
> > ttc setenv bbb  # this allows me to omit 'bbb' on every following command
> line
> > ttc get_kernel
> > ttc set_config CONFIG_SOME_KCONFIG=y;CONFIG_SOME_VALUE=255
> > ttc kbuild
> > ttc kinstall
> > ttc reboot
> > ttc wait-for "ttc run 'echo alive'"
> > ttc cp testfiles target:/test_directory
> > ttc run "some commands"
> > ttc cp target:/test_directory/testlogs host_directory
> 
> This sequence is what I am calling a "test job" (not to confuse with Jenkins
> jobs) and I think it works pretty well.
> If we were going to integrate that with LAVA, most of the steps would be
> done by LAVA and we would only need the "ttc run ..." part.
> 
> > > And we can make more complicated things.
> > > [Note] The ftc expect is something I am designing now to interact with
> the
> > > board in a flexible way, and provide LAVA-like functionality.
> > I'd like to have a discussion with you next week about the best
> > approach in Fuego to handle provisioning, and building up complex test
> > pipelines.  I think some sub-operations should be in ftc, and maybe
> > some should be put into tests that call sub-tests.
> 
> Oh "test pipelines" sounds better than "test jobs". Sure, let's discuss that.
> 
> Thanks,
> Daniel
> 
> >
> >  -- Tim
> >
> > >
> > > > Once we've plugged pdudaemon into the board layer, I'd prefer to
> have a
> > > generic
> > > > set of operations that someone can use from ftc, like:
> > > >    ftc reboot-board -b <board>
> > > >    ftc power-on-board -b <board>
> > > >    ftc power-off-board -b <board>
> > > > and maybe even
> > > >    ftc show-status-of-board <board>
> > >
> > > Maybe we can use subcommands like this.
> > > ftc power reboot -b bbb
> > > ftc power on -b board
> > > ftc power off -b board
> > > ftc power status -b board
> > >
> > > Or put hyphens, but it would rather have the same 1st work for bash
> > > autocomplete to be more intuitive.
> > >
> > > > In some labs, there may be a layer between pdudaemon and Fuego
> (for
> > > example libvirt,
> > > > or labgrid or even LAVA).
> > > >
> > > > I think I'd rather have an abstraction layer between Fuego and
> > > pdudaemon, rather than
> > > > integrating it directly into our user interface.
> > > >
> > > > Thoughts?
> > >
> > > See above.
> > > Thanks,
> > > Daniel
> > >
> > > >  -- tim
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Daniel Sangorrin
> > > > >
> > > > > This patch adds the "pdu" command to send requests for pdudaemon
> > > > > to power-on/off or reboot a board.
> > > > >
> > > > > Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> > > > > ---
> > > > >  scripts/ftc               | 80
> > > > > +++++++++++++++++++++++++++++++++++++++++++++++
> > > > >  scripts/ftc_completion.sh |  5 +++
> > > > >  2 files changed, 85 insertions(+)
> > > > >
> > > > > diff --git a/scripts/ftc b/scripts/ftc
> > > > > index 7153c01..30ea470 100755
> > > > > --- a/scripts/ftc
> > > > > +++ b/scripts/ftc
> > > > > @@ -426,6 +426,29 @@ A message and the exit code indicate if the
> > > resource
> > > > > was released.
> > > > >
> > > > >     ex: ftc release-resource -b beaglebone"""),
> > > > >
> > > > > +"pdu": ("Send a request to pdudaemon.",
> > > > > +    """Usage: ftc pdu -b <board> --command on|off|reboot [--
> > > > > pdudaemon_hostname localhost --pdudaemon_port 16421]
> > > > > +
> > > > > +This command is used to send a request for pdudaemon to power-
> on,
> > > > > power-off or reboot a board.
> > > > > +
> > > > > +The command accepts these arguments:
> > > > > + - board file: bbb, raspberrypi3, etc (see ftc list-boards)
> > > > > + - command: on|off|reboot
> > > > > + - pdudaemon_hostname: hostname or IP address where
> pdudaemon is
> > > > > running (default: localhost)
> > > > > + - pdudaemon_port: port where pdudaemon is listening (default:
> 16421)
> > > > > +   [Note] make sure that pdudaemon is configured to accept http
> > > requests
> > > > > +
> > > > > +Additionally, you need to define two variables in your board file:
> > > > > + - PDU_HOSTNAME: the PDU hostname or IP address used in the
> > > > > pdudaemon configuration file
> > > > > + - PDU_PORT: the PDU port number (a plug port, not a tcp port)
> > > > > +
> > > > > +You need to install pdudaemon by yourself
> > > > > +  $ git clone https://github.com/pdudaemon/pdudaemon
> > > > > +  $ cd pdudaemon
> > > > > +  $ less README.md
> > > > > +
> > > > > +Example: ftc pdu -b raspberrypi3 --command on"""),
> > > > > +
> > > > >  "package-test": ("Package a test.",
> > > > >      """Usage: ftc package-test <test_name> [options]
> > > > >
> > > > > @@ -4783,6 +4806,59 @@ def do_release_resource(conf, options):
> > > > >
> > > > >      return 0
> > > > >
> > > > > +def do_pdu(bvars, conf, options):
> > > > > +    # pdudaemon hostname
> > > > > +    pdudaemon_hostname = 'localhost'
> > > > > +    if '--pdudaemon_hostname' in options:
> > > > > +        try:
> > > > > +            pdudaemon_hostname = options[options.index('--
> > > > > pdudaemon_hostname')+1]
> > > > > +        except IndexError:
> > > > > +            error_out('Value not provided after --
> pdudaemon_hostname')
> > > > > +        options.remove(pdudaemon_hostname)
> > > > > +        options.remove('--pdudaemon_hostname')
> > > > > +
> > > > > +    # pdudaemon port
> > > > > +    pdudaemon_port = '16421'
> > > > > +    if '--pdudaemon_port' in options:
> > > > > +        try:
> > > > > +            pdudaemon_port = options[options.index('--
> > > pdudaemon_port')+1]
> > > > > +        except IndexError:
> > > > > +            error_out('Value not provided after --pdudaemon_port')
> > > > > +        options.remove(pdudaemon_port)
> > > > > +        options.remove('--pdudaemon_port')
> > > > > +
> > > > > +    # pdu hostname
> > > > > +    if 'PDU_HOSTNAME' in bvars:
> > > > > +        pdu_hostname = bvars['PDU_HOSTNAME']
> > > > > +    else:
> > > > > +        error_out('PDU_HOSTNAME not specified in board file')
> > > > > +
> > > > > +    # pdu port
> > > > > +    if 'PDU_PORT' in bvars:
> > > > > +        pdu_port = bvars['PDU_PORT']
> > > > > +    else:
> > > > > +        error_out('PDU_PORT not specified in board file')
> > > > > +
> > > > > +    # command (on|off|reboot)
> > > > > +    if '--command' in options:
> > > > > +        try:
> > > > > +            command = options[options.index('--command')+1]
> > > > > +        except IndexError:
> > > > > +            error_out('Value not provided after --command')
> > > > > +        options.remove(command)
> > > > > +        options.remove('--command')
> > > > > +        if command not in ['on', 'off', 'reboot']:
> > > > > +            error_out('Unrecognized PDU command %s' % command)
> > > > > +    else:
> > > > > +        error_out('Command not supplied')
> > > > > +
> > > > > +    query='&'.join(['hostname='+pdu_hostname, 'port='+pdu_port])
> > > > > +    url = 'http://%s:%s/power/control/%s?%s' %
> > > (pdudaemon_hostname,
> > > > > pdudaemon_port, command, query)
> > > > > +    resp = requests.get(url)
> > > > > +    if resp.status_code != 200:
> > > > > +        error_out("pdudaemon did not accept the request")
> > > > > +    return 0
> > > > > +
> > > > >  def main():
> > > > >      # use global module names
> > > > >      global re, time, copy2, subprocess, signal, fcntl, requests, json
> > > > > @@ -5094,6 +5170,10 @@ def main():
> > > > >          do_delete_var(bvars, options)
> > > > >          sys.exit(0)
> > > > >
> > > > > +    if command == "pdu":
> > > > > +        rcode = do_pdu(bvars, conf, options)
> > > > > +        sys.exit(rcode)
> > > > > +
> > > > >      error_out("Unknown command %s" % command)
> > > > >
> > > > >
> > > > > diff --git a/scripts/ftc_completion.sh b/scripts/ftc_completion.sh
> > > > > index 984e003..2cfb77a 100755
> > > > > --- a/scripts/ftc_completion.sh
> > > > > +++ b/scripts/ftc_completion.sh
> > > > > @@ -55,6 +55,7 @@ _ftc()
> > > > >          ["list-tests"]="-q"
> > > > >          ["package-run"]="-o -f"
> > > > >          ["package-test"]="-o -f"
> > > > > +        ["pdu"]="-b --command --pdudaemon_hostname --
> > > pdudaemon_port"
> > > > >          ["put-request"]="-R -s"
> > > > >          ["put-run"]=""
> > > > >          ["put-test"]=""
> > > > > @@ -123,6 +124,10 @@ _ftc()
> > > > >              COMPREPLY=( $( compgen -W "true false" -- "$cur" ) )
> > > > >              return 0
> > > > >              ;;
> > > > > +        '--command')
> > > > > +            COMPREPLY=( $( compgen -W "on off reboot" -- "$cur" ) )
> > > > > +            return 0
> > > > > +            ;;
> > > > >          'put-run')
> > > > >              COMPREPLY=( $( compgen -W "$(ftc list-runs -q)" -- "$cur" ) )
> > > > >              return 0
> > > > > --
> > > > > 2.7.4
> > > > >
> > > > > _______________________________________________
> > > > > Fuego mailing list
> > > > > Fuego@lists.linuxfoundation.org
> > > > > https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] [PATCH] ftc: add a new command to interact with pdudaemon

[daniel.sangorrin]

Hi Tim,

> From: Tim.Bird@sony.com <Tim.Bird@sony.com>
> Sent: Friday, April 12, 2019 9:39 AM
> Daniel,
> 
> I've kind of lost track of where we are with this.  Can you update me on the
> status of new ftc commands, and their integration with pdudaemon?

OK, this is a quick update. I sent this patch in March:
https://lists.linuxfoundation.org/pipermail/fuego/2019-March/003024.html

What that patch does:
- to add a new command to ftc that can turn on/off/reboot a board
- if PDUdaemon is not running on localhost (on a separate container) the user can specify the IP address where PDUdaemon is running
- the board file contains the information that defines what PDU and port it is connected to. This information is sent to the PDUdaemon through the "requests" library.

Limitations of my patch:
- I didn't demultiplex the command request into multiple power-control tools (I was assuming that we would only need PDUdaemon).
- The overlay layer has its own power-control and demultiplexing code, but it is hard to use from ftc.
 
What to do next:
- I could fix the command to support multiple tools (ttc, PDUdaemon..), and modify the overlay layer code so that it calls ftc recursively/nested.

> Are we waiting for pdudaemon to change its API, or are we moving ahead
> with the interface to is via 'curl'?

I think that we can move ahead because PDUdaemon will support both the old and the new API in any case (nothing difficult to achieve).
I suggest that we use the interface below for Fuego and internally we use an API to connect to each power-control tool:

ftc power-on -b bbb
ftc power-off -b bbb
ftc power-cycle -b bbb
[Note] power-reset, power-reboot are also fine.

> And where are we putting the connection between board and (pdu and
> port)?

There are 2 types of variables:
- variables used by the power-control tools (e.g.: IP address of the PDUdaemon(s))
   - I suggest that we store them in the Fuego configuration file, but allow users to override them per-board file. The most common case is for Fuego users to use a single PDUdaemon on the localhost (usually on a separate container) . I suggest that we use that configuration by default.
- variables used for turning on/off the boards (e.g.: strings that identify the particular PDU and hardware port they are connected to)
  - I suggest that we put that information inside each board file. 

The lab operator will need to associate the PDU with a suitable "driver" (code that makes on/off requests to the PDU through SNMP for example) by adjusting the tool's configuration file. Perhaps it would be convenient to have a possibly optional API for users to select the driver and other parameters (delay) on the fly, without having to setup each power-control tool's configuration file.

> Do we need a new kind of test object?   I've been thinking of having test
> objects
> that are used for different things, like Monitors, Stressors, and Provisioners.
> A monitor would be executed in parallel with a test, to gather information
> from another device like a power meter.  A Stressor would also run in
> parallel, to put a known workload on a device while another test is running.
> And  a Provisioner would be run prior to a test, in order to prepare the board
> for the test by deploying the software under test and booting it.

Sounds like a "power-control" class could be used to do the multiplexing.

> So, I'm starting work on the API for starting, checking the running status of,
> and stopping these other test processes.  Let me know what you think, or if
> you have some example APIs I should look at for inspiration.

For the stressors, I think that most tests can work as Stressors. We could also support testplans as stressors (e.g.: we may want to test cyclictest under io stress first, the under scheduler stress and finally under network stress). To do that, we can add a new phase similar to cleanup (how about test_cancel) so that we can execute the test but cancel it whenever we want.

Thanks,
Daniel

> Thanks,
>  -- Tim
> 
> > -----Original Message-----
> > From: daniel.sangorrin@toshiba.co.jp
> >
> > > -----Original Message-----
> > > From: Tim.Bird@sony.com <Tim.Bird@sony.com>
> > > Sent: Wednesday, March 27, 2019 6:09 AM
> > > To: sangorrin daniel(サンゴリン ダニエル ○ＳＷＣ□ＯＳＴ)
> > <daniel.sangorrin@toshiba.co.jp>;
> > > fuego@lists.linuxfoundation.org
> > > Subject: RE: [Fuego] [PATCH] ftc: add a new command to interact with
> > pdudaemon
> > >
> > > More feedback on this topic...  see below.
> > >
> > > > -----Original Message-----
> > > > From: daniel.sangorrin@toshiba.co.jp
> > > >
> > > > Hi Tim,
> > > >
> > > > Thanks again for reviewing in your busy time.
> > > >
> > > > > -----Original Message-----
> > > > > From: Tim.Bird@sony.com <Tim.Bird@sony.com>
> > > > > Sent: Thursday, March 14, 2019 4:55 AM
> > > > > To: sangorrin daniel(サンゴリン ダニエル ○ＳＷＣ□ＯＳＴ)
> > > > <daniel.sangorrin@toshiba.co.jp>;
> > > > > fuego@lists.linuxfoundation.org
> > > > > Subject: RE: [Fuego] [PATCH] ftc: add a new command to interact
> > > > > with
> > > > pdudaemon
> > > > >
> > > > > Here are some overall comments:
> > > > >
> > > > > Rather than add a new command to ftc for managing pdudaemon
> > > > operations,
> > > > > I was expecting to install support for pdudaemon at Fuego's
> > > > > board
> > > > management (/board_control)
> > > > > layer.  That is, I was expecting to see something like this:
> > > > >
> > > > > diff --git a/overlays/base/base-board.fuegoclass
> > b/overlays/base/base-
> > > > board.fuegoclass
> > > > > index ae35464..8f34922 100644
> > > > > --- a/overlays/base/base-board.fuegoclass
> > > > > +++ b/overlays/base/base-board.fuegoclass
> > > > > @@ -299,6 +299,9 @@ function ov_board_control_reboot() {
> > > > >    "ttc")
> > > > >      $TTC $TTC_TARGET reboot
> > > > >      ;;
> > > > > +  "pdudaemon"):
> > > > > +    curl
> > > > >
> > > >
> >
> "http://$PDUDAEMON_HOST:$PDUDAEMON_PORT/power/control/reboot?
> > > > hostname=$PDU_HOSTNAME&por
> > > > > t=$PDU_PORT"
> > > > > +    ;;
> > > > >    *)
> > > > >      abort_job "Error reason: unsupported BOARD_CONTROL
> > > > ${BOARD_CONTROL}"
> > > > >      ;;
> > > > > ----
> > > >
> > > > Alright, that sounds like a good idea. I will resend.
> > > >
> > > > > Note that I don't know if pdudaemon supports the 'reboot'
> > > > > control
> > option,
> > > > or if I would have
> > > > reboot is supported
> > > >
> > > > > to synthesize something by sending 'on' and 'off'.  But you get
> > > > > the idea,
> > I
> > > > think.
> > > > > I'm not sure if the hostname in the URL is supposed to be the
> > > > > board
> > name
> > > > or the pdu name,
> > > > PDU_HOSTNAME is the PDU's IP address/hostname.
> > > > PDUDAEMON_HOSTNAME is the IP address/hostname of the computer
> > > > where pdudaemon is running.
> > > > The board doesn't know that it can be "killed" at anytime by the PDU.
> > > >
> > > > > but presumably PDUDAEMON_HOST and PDUDAEMON_PORT would
> > come
> > > > from
> > > > > fuego.conf, and PDU_HOSTNAME and PDU_PORT would come from
> > the
> > > > board file?
> > > > From the board file because each board can be in a different
> > > > place. These two parameters are passed to the pdudaemon.
> Pdudaemon
> > > > then
> > internally
> > > > uses them to connect with the PDU by the supported protocol (e.g.:
> > > > SNMTP).
> > >
> > > OK. As discussed in the other thread, this means that the board file
> > > needs to be parsed by ftc to execute these, to get these
> > > board-related test
> > variables.
> > > For now I believe it's OK to ignore processing overlays and stored
> > > variables
> > for
> > > this.  However, I think the ftc internal function get_board_vars()
> > > does all
> > this processing
> > > anyway, so it might just be easier to use that.
> >
> > Yes, this works fine. I already have it implemented and it's ready for
> > demultiplexing.
> >
> > > > > If more fine-grained control is desired, than just the 'reboot'
> > > > > operation, we could add more ov_board_control() functions if
> > > > > needed, and have
> > > > Fuego use those.
> > > > > e.g. ov_board_control_on() and ov_board_control_off().
> > > >
> > > > Yes, I think we need on off.
> > > > In particular, I was thinking about Fuego testplans that would
> > > > work like
> > this:
> > > >
> > > > ftc pdu -b bbb --command on
> > > > ftc expect -b bbb --regex "u-boot>" --send SIGINT ftc send -b bbb
> > > > --command "run tftpboot"
> > > > ftc expect -b bbb --regex "mypromptlinuxprompt" --send "root"
> > > > echo ftc pdu -b bbb --command off | at now + 1 minute ftc run-test
> > > > -b bbb -t Functional.StressFilesystem sleep 60 ftc pdu -b bbb
> > > > --command on ftc expect -b bbb --regex myprompt ftc run-test -b
> > > > bbb -t Functional.ext4 ftc pdu -b bbb --command off
> > >
> > > A couple of notes here:
> > > 1) It looks like get_board_arg() looks for FUEGO_BOARD in the
> > > environment, and uses that if present.
> > > I'm pretty sure you can do: export FUEGO_BOARD=bbb, and then omit
> > > the
> > '-b bbb' on every line.
> > > From inside a test, maybe we should make this automatic, and set
> > FUEGO_BOARD from
> > > NODE_NAME when we do run-test.
> >
> > I didn't know.
> > I also had the idea to add a patch that would assume the board 'local'
> > if -b was not provided.
> >
> > > 2) 'ftc expect' looks nice, and it shouldn't be too hard to do since
> > > python
> > has an 'expect' module
> > > (famous last words... :-)
> >
> > Actually, after looking at your ttc application it looks like you
> > already have an implementation (ttc wait-for). I guess it should be quite
> similar.
> >
> > > 3) We should discuss the high-level operations we want to do, and make
> > > ftc operations for those.   For some tests, the details of how the board is
> > booted
> > > and provisioned should be opaque to the test.
> >
> > Yes, sure.
> > I think that we are arriving to what LAVA defines in yaml:
> > -power on
> > -wait for u-boot
> > -send commands to u-boot to provision the board -wait for login -run
> > the test ...
> >
> > So probably Fuego users will end up defining "test jobs" (similar to
> > LAVA
> > jobs) where they prepare a sequence of ftc instructions
> > (power-on-board, wait-for, run-test, run-test again, power-off-board,
> > et..). And we could even put a yaml layer on top of that (similar to LAVA
> syntax).
> >
> > > One issue I see with your low-level steps, is that they encode
> > > things that are board-specific (like the boot prompt regexes) into
> > > the commands I'm not opposed, to low-level operations being encoded
> > > into tests.
> > > It's important, though, to try to isolate board-specific items so
> > > that the tests don't become dependent on a specific board,
> > > bootloader, distro, board control layer, or even operating system.
> > > (e.g. I've started work on supporting Nuttx testing with Fuego).
> >
> > Sure, we can create BOARD variables as necessary, and still allow
> > users to override them, optionally, on the command line for quick testing.
> >
> > > Maybe you intend to parameterize those regexes from the board file.
> > > But this still encapsulates too much of the way a system boots and
> > > this particular example I believe requires that you have a serial port.
> > > That wouldn't be true, for example, with a mobile phone, where you
> > > have adb access, but often not a serial port.
> >
> > I think we need to consider boards that have a combination of serial
> > and ssh access.
> > Serial could be used for booting/provisioning and SSH for
> > deploying/getting results/running commands etc.
> >
> > Also, what I am talking about is not a "test" but rather a "test job"
> > with a sequence of ftc commands that would be customized by users and
> > not necessarily shared.
> >
> > > Here's how I've used ttc for board control and provisioning:
> > > ttc setenv bbb  # this allows me to omit 'bbb' on every following
> > > command
> > line
> > > ttc get_kernel
> > > ttc set_config CONFIG_SOME_KCONFIG=y;CONFIG_SOME_VALUE=255
> > > ttc kbuild
> > > ttc kinstall
> > > ttc reboot
> > > ttc wait-for "ttc run 'echo alive'"
> > > ttc cp testfiles target:/test_directory ttc run "some commands"
> > > ttc cp target:/test_directory/testlogs host_directory
> >
> > This sequence is what I am calling a "test job" (not to confuse with
> > Jenkins
> > jobs) and I think it works pretty well.
> > If we were going to integrate that with LAVA, most of the steps would
> > be done by LAVA and we would only need the "ttc run ..." part.
> >
> > > > And we can make more complicated things.
> > > > [Note] The ftc expect is something I am designing now to interact
> > > > with
> > the
> > > > board in a flexible way, and provide LAVA-like functionality.
> > > I'd like to have a discussion with you next week about the best
> > > approach in Fuego to handle provisioning, and building up complex
> > > test pipelines.  I think some sub-operations should be in ftc, and
> > > maybe some should be put into tests that call sub-tests.
> >
> > Oh "test pipelines" sounds better than "test jobs". Sure, let's discuss that.
> >
> > Thanks,
> > Daniel
> >
> > >
> > >  -- Tim
> > >
> > > >
> > > > > Once we've plugged pdudaemon into the board layer, I'd prefer to
> > have a
> > > > generic
> > > > > set of operations that someone can use from ftc, like:
> > > > >    ftc reboot-board -b <board>
> > > > >    ftc power-on-board -b <board>
> > > > >    ftc power-off-board -b <board> and maybe even
> > > > >    ftc show-status-of-board <board>
> > > >
> > > > Maybe we can use subcommands like this.
> > > > ftc power reboot -b bbb
> > > > ftc power on -b board
> > > > ftc power off -b board
> > > > ftc power status -b board
> > > >
> > > > Or put hyphens, but it would rather have the same 1st work for
> > > > bash autocomplete to be more intuitive.
> > > >
> > > > > In some labs, there may be a layer between pdudaemon and Fuego
> > (for
> > > > example libvirt,
> > > > > or labgrid or even LAVA).
> > > > >
> > > > > I think I'd rather have an abstraction layer between Fuego and
> > > > pdudaemon, rather than
> > > > > integrating it directly into our user interface.
> > > > >
> > > > > Thoughts?
> > > >
> > > > See above.
> > > > Thanks,
> > > > Daniel
> > > >
> > > > >  -- tim
> > > > >
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Daniel Sangorrin
> > > > > >
> > > > > > This patch adds the "pdu" command to send requests for
> > > > > > pdudaemon to power-on/off or reboot a board.
> > > > > >
> > > > > > Signed-off-by: Daniel Sangorrin
> > > > > > <daniel.sangorrin@toshiba.co.jp>
> > > > > > ---
> > > > > >  scripts/ftc               | 80
> > > > > > +++++++++++++++++++++++++++++++++++++++++++++++
> > > > > >  scripts/ftc_completion.sh |  5 +++
> > > > > >  2 files changed, 85 insertions(+)
> > > > > >
> > > > > > diff --git a/scripts/ftc b/scripts/ftc index 7153c01..30ea470
> > > > > > 100755
> > > > > > --- a/scripts/ftc
> > > > > > +++ b/scripts/ftc
> > > > > > @@ -426,6 +426,29 @@ A message and the exit code indicate if
> > > > > > the
> > > > resource
> > > > > > was released.
> > > > > >
> > > > > >     ex: ftc release-resource -b beaglebone"""),
> > > > > >
> > > > > > +"pdu": ("Send a request to pdudaemon.",
> > > > > > +    """Usage: ftc pdu -b <board> --command on|off|reboot [--
> > > > > > pdudaemon_hostname localhost --pdudaemon_port 16421]
> > > > > > +
> > > > > > +This command is used to send a request for pdudaemon to
> > > > > > +power-
> > on,
> > > > > > power-off or reboot a board.
> > > > > > +
> > > > > > +The command accepts these arguments:
> > > > > > + - board file: bbb, raspberrypi3, etc (see ftc list-boards)
> > > > > > + - command: on|off|reboot
> > > > > > + - pdudaemon_hostname: hostname or IP address where
> > pdudaemon is
> > > > > > running (default: localhost)
> > > > > > + - pdudaemon_port: port where pdudaemon is listening (default:
> > 16421)
> > > > > > +   [Note] make sure that pdudaemon is configured to accept
> > > > > > + http
> > > > requests
> > > > > > +
> > > > > > +Additionally, you need to define two variables in your board file:
> > > > > > + - PDU_HOSTNAME: the PDU hostname or IP address used in the
> > > > > > pdudaemon configuration file
> > > > > > + - PDU_PORT: the PDU port number (a plug port, not a tcp
> > > > > > + port)
> > > > > > +
> > > > > > +You need to install pdudaemon by yourself
> > > > > > +  $ git clone https://github.com/pdudaemon/pdudaemon
> > > > > > +  $ cd pdudaemon
> > > > > > +  $ less README.md
> > > > > > +
> > > > > > +Example: ftc pdu -b raspberrypi3 --command on"""),
> > > > > > +
> > > > > >  "package-test": ("Package a test.",
> > > > > >      """Usage: ftc package-test <test_name> [options]
> > > > > >
> > > > > > @@ -4783,6 +4806,59 @@ def do_release_resource(conf, options):
> > > > > >
> > > > > >      return 0
> > > > > >
> > > > > > +def do_pdu(bvars, conf, options):
> > > > > > +    # pdudaemon hostname
> > > > > > +    pdudaemon_hostname = 'localhost'
> > > > > > +    if '--pdudaemon_hostname' in options:
> > > > > > +        try:
> > > > > > +            pdudaemon_hostname = options[options.index('--
> > > > > > pdudaemon_hostname')+1]
> > > > > > +        except IndexError:
> > > > > > +            error_out('Value not provided after --
> > pdudaemon_hostname')
> > > > > > +        options.remove(pdudaemon_hostname)
> > > > > > +        options.remove('--pdudaemon_hostname')
> > > > > > +
> > > > > > +    # pdudaemon port
> > > > > > +    pdudaemon_port = '16421'
> > > > > > +    if '--pdudaemon_port' in options:
> > > > > > +        try:
> > > > > > +            pdudaemon_port = options[options.index('--
> > > > pdudaemon_port')+1]
> > > > > > +        except IndexError:
> > > > > > +            error_out('Value not provided after --pdudaemon_port')
> > > > > > +        options.remove(pdudaemon_port)
> > > > > > +        options.remove('--pdudaemon_port')
> > > > > > +
> > > > > > +    # pdu hostname
> > > > > > +    if 'PDU_HOSTNAME' in bvars:
> > > > > > +        pdu_hostname = bvars['PDU_HOSTNAME']
> > > > > > +    else:
> > > > > > +        error_out('PDU_HOSTNAME not specified in board file')
> > > > > > +
> > > > > > +    # pdu port
> > > > > > +    if 'PDU_PORT' in bvars:
> > > > > > +        pdu_port = bvars['PDU_PORT']
> > > > > > +    else:
> > > > > > +        error_out('PDU_PORT not specified in board file')
> > > > > > +
> > > > > > +    # command (on|off|reboot)
> > > > > > +    if '--command' in options:
> > > > > > +        try:
> > > > > > +            command = options[options.index('--command')+1]
> > > > > > +        except IndexError:
> > > > > > +            error_out('Value not provided after --command')
> > > > > > +        options.remove(command)
> > > > > > +        options.remove('--command')
> > > > > > +        if command not in ['on', 'off', 'reboot']:
> > > > > > +            error_out('Unrecognized PDU command %s' % command)
> > > > > > +    else:
> > > > > > +        error_out('Command not supplied')
> > > > > > +
> > > > > > +    query='&'.join(['hostname='+pdu_hostname, 'port='+pdu_port])
> > > > > > +    url = 'http://%s:%s/power/control/%s?%s' %
> > > > (pdudaemon_hostname,
> > > > > > pdudaemon_port, command, query)
> > > > > > +    resp = requests.get(url)
> > > > > > +    if resp.status_code != 200:
> > > > > > +        error_out("pdudaemon did not accept the request")
> > > > > > +    return 0
> > > > > > +
> > > > > >  def main():
> > > > > >      # use global module names
> > > > > >      global re, time, copy2, subprocess, signal, fcntl,
> > > > > > requests, json @@ -5094,6 +5170,10 @@ def main():
> > > > > >          do_delete_var(bvars, options)
> > > > > >          sys.exit(0)
> > > > > >
> > > > > > +    if command == "pdu":
> > > > > > +        rcode = do_pdu(bvars, conf, options)
> > > > > > +        sys.exit(rcode)
> > > > > > +
> > > > > >      error_out("Unknown command %s" % command)
> > > > > >
> > > > > >
> > > > > > diff --git a/scripts/ftc_completion.sh
> > > > > > b/scripts/ftc_completion.sh index 984e003..2cfb77a 100755
> > > > > > --- a/scripts/ftc_completion.sh
> > > > > > +++ b/scripts/ftc_completion.sh
> > > > > > @@ -55,6 +55,7 @@ _ftc()
> > > > > >          ["list-tests"]="-q"
> > > > > >          ["package-run"]="-o -f"
> > > > > >          ["package-test"]="-o -f"
> > > > > > +        ["pdu"]="-b --command --pdudaemon_hostname --
> > > > pdudaemon_port"
> > > > > >          ["put-request"]="-R -s"
> > > > > >          ["put-run"]=""
> > > > > >          ["put-test"]=""
> > > > > > @@ -123,6 +124,10 @@ _ftc()
> > > > > >              COMPREPLY=( $( compgen -W "true false" -- "$cur" ) )
> > > > > >              return 0
> > > > > >              ;;
> > > > > > +        '--command')
> > > > > > +            COMPREPLY=( $( compgen -W "on off reboot" -- "$cur" ) )
> > > > > > +            return 0
> > > > > > +            ;;
> > > > > >          'put-run')
> > > > > >              COMPREPLY=( $( compgen -W "$(ftc list-runs -q)" -- "$cur" ) )
> > > > > >              return 0
> > > > > > --
> > > > > > 2.7.4
> > > > > >
> > > > > > _______________________________________________
> > > > > > Fuego mailing list
> > > > > > Fuego@lists.linuxfoundation.org
> > > > > > https://lists.linuxfoundation.org/mailman/listinfo/fuego