[Fuego] How to add a Power-Controller and Console Server to FUEGO?

[Fuego] How to add a Power-Controller and Console Server to FUEGO?

[Heinrich.Toews]

Hey altogether,

at our company we are willing to use FUEGO for our Linux Kernel Testing.

We already have installed Racks ready with our own MODBUS TCP/UDP 
power-controllers and console servers interfacing with TELNET/SSH.

What would be the best practice to add such support to the FUEGO code? 
Is there something ready that we could just reuse?

Thanks!

Heinrich Toews

Re: [Fuego] How to add a Power-Controller and Console Server to FUEGO?

[daniel.sangorrin]

Hello Heinrich,

> From: fuego-bounces@lists.linuxfoundation.org <fuego-bounces@lists.linuxfoundation.org> On Behalf Of
> Heinrich.Toews@wago.com
> Sent: Tuesday, June 18, 2019 6:45 PM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] How to add a Power-Controller and Console Server to FUEGO?
> 
> Hey altogether,
> 
> at our company we are willing to use FUEGO for our Linux Kernel Testing.
> 
> We already have installed Racks ready with our own MODBUS TCP/UDP
> power-controllers and console servers interfacing with TELNET/SSH.

I will assume that this is what you want to do, otherwise please let me know:

[FUEGO HOST] --TCP--> [MODBUS TCP/UDP]--power on/off--> [boards under test]
             `---SSH--> [Console Server]--serial port-----------^

> What would be the best practice to add such support to the FUEGO code?

For the power-controllers, I sent a patch to Fuego that adds support for PDUdaemon [1]. Unfortunately, it got stalled and never made it into Fuego. Perhaps I can revive that thread.
https://lists.linuxfoundation.org/pipermail/fuego/2019-March/thread.html

In parallel, could you check if PDUdaemon supports your power controllers?.
https://github.com/pdudaemon/pdudaemon/tree/master/pdudaemon/drivers

For the console server, I guess that you can use the SSH transport support available in Fuego. I have never used a Console server. Does it use a different SSH port for each of the underlying serial port connections?. If that is the case, you will need to define the SSH_PORT on the corresponding board file at `fuego-ro/boards`. If you are using different user names instead, then you would need to set the SSH_KEY or the SSH user and password (LOGIN, PASSWORD) in the board file.

> Is there something ready that we could just reuse?

Hopefully, you don't need to add anything.

Thanks,
Daniel

> Thanks!
> 
> Heinrich Toews
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] How to add a Power-Controller and Console Server to FUEGO?

[Heinrich.Toews]

Hello Daniel,

On 19.06.19 02:47, daniel.sangorrin@toshiba.co.jp wrote:
> Hello Heinrich,
> 
>> From: fuego-bounces@lists.linuxfoundation.org <fuego-bounces@lists.linuxfoundation.org> On Behalf Of
>> Heinrich.Toews@wago.com
>> Sent: Tuesday, June 18, 2019 6:45 PM
>> To: fuego@lists.linuxfoundation.org
>> Subject: [Fuego] How to add a Power-Controller and Console Server to FUEGO?
>>
>> Hey altogether,
>>
>> at our company we are willing to use FUEGO for our Linux Kernel Testing.
>>
>> We already have installed Racks ready with our own MODBUS TCP/UDP
>> power-controllers and console servers interfacing with TELNET/SSH.
> 
> I will assume that this is what you want to do, otherwise please let me know:
> 
> [FUEGO HOST] --TCP--> [MODBUS TCP/UDP]--power on/off--> [boards under test]
>               `---SSH--> [Console Server]--serial port-----------^

Yea, you're right that's exactly what I want to do!

> 
>> What would be the best practice to add such support to the FUEGO code?
> 
> For the power-controllers, I sent a patch to Fuego that adds support for PDUdaemon [1]. Unfortunately, it got stalled and never made it into Fuego. Perhaps I can revive that thread.
> https://lists.linuxfoundation.org/pipermail/fuego/2019-March/thread.html

Thanks! I walked briefly through the thread and I find it reasonable to 
use PDUdaemon for this purpose .. It would be great if it gets revived!

I think many will have this requirement ..

> In parallel, could you check if PDUdaemon supports your power controllers?.
> https://github.com/pdudaemon/pdudaemon/tree/master/pdudaemon/drivers

A Modbus driver is not included yet but it would be an easy task to add it.

> For the console server, I guess that you can use the SSH transport support available in Fuego. I have never used a Console server. Does it use a different SSH port for each of the underlying serial port connections?. If that is the case, you will need to define the SSH_PORT on the corresponding board file at `fuego-ro/boards`. If you are using different user names instead, then you would need to set the SSH_KEY or the SSH user and password (LOGIN, PASSWORD) in the board file.

Okay, thanks - I will try this. We are using ConnectPort LTS32 .. SSH 
should be possible somehow. Until now we only used TELNET.

>> Is there something ready that we could just reuse?
> 
> Hopefully, you don't need to add anything.

Just the modbus driver :)

Thanks & have a nice day!

Heinrich

Re: [Fuego] How to add a Power-Controller and Console Server to FUEGO?

[daniel.sangorrin]

> From: Heinrich.Toews@wago.com <Heinrich.Toews@wago.com>
> On 19.06.19 02:47, daniel.sangorrin@toshiba.co.jp wrote:
> > Hello Heinrich,
> >
> >> From: fuego-bounces@lists.linuxfoundation.org <fuego-bounces@lists.linuxfoundation.org> On Behalf Of
> >> Heinrich.Toews@wago.com
> >> Sent: Tuesday, June 18, 2019 6:45 PM
> >> To: fuego@lists.linuxfoundation.org
> >> Subject: [Fuego] How to add a Power-Controller and Console Server to FUEGO?
> >>
> >> Hey altogether,
> >>
> >> at our company we are willing to use FUEGO for our Linux Kernel Testing.
> >>
> >> We already have installed Racks ready with our own MODBUS TCP/UDP
> >> power-controllers and console servers interfacing with TELNET/SSH.
> >
> > I will assume that this is what you want to do, otherwise please let me know:
> >
> > [FUEGO HOST] --TCP--> [MODBUS TCP/UDP]--power on/off--> [boards under test]
> >               `---SSH--> [Console Server]--serial port-----------^
> 
> Yea, you're right that's exactly what I want to do!

Oops I missed something important. 
Are the boards under test connected to a LAN?
I ask that for two reasons:

1) You probably want to install the latest kernel and file system image before you start testing. For example, you can do that by booting from tftpboot, or you could use some more advanced solution such as LAVA. Both of them require a LAN connection. 
[Note] Alternatively, you can install the latest kernel/image manually.

2) You also need to install the tests that you want to run. Most Fuego built-in tests assume that there is a connection with the board to install test software (a transport layer). If you don't have such a connection, you will need to build your tests in your image and skip the build and deploy phase in Fuego (using ftc command arguments). 

> >> What would be the best practice to add such support to the FUEGO code?
> >
> > For the power-controllers, I sent a patch to Fuego that adds support for PDUdaemon [1]. Unfortunately, it got
> stalled and never made it into Fuego. Perhaps I can revive that thread.
> > https://lists.linuxfoundation.org/pipermail/fuego/2019-March/thread.html
> 
> Thanks! I walked briefly through the thread and I find it reasonable to
> use PDUdaemon for this purpose .. It would be great if it gets revived!
> 
> I think many will have this requirement ..

OK, give me a bit of time and I will re-send the patches.

> > In parallel, could you check if PDUdaemon supports your power controllers?.
> > https://github.com/pdudaemon/pdudaemon/tree/master/pdudaemon/drivers
> 
> A Modbus driver is not included yet but it would be an easy task to add it.

OK, good.
[Note] Another option is to add new BOARD_CONTROL type or override ov_board_control_power_xxx functions in your board file.

> > For the console server, I guess that you can use the SSH transport support available in Fuego. I have never
> used a Console server. Does it use a different SSH port for each of the underlying serial port connections?. If that
> is the case, you will need to define the SSH_PORT on the corresponding board file at `fuego-ro/boards`. If you
> are using different user names instead, then you would need to set the SSH_KEY or the SSH user and password
> (LOGIN, PASSWORD) in the board file.
> 
> Okay, thanks - I will try this. We are using ConnectPort LTS32 .. SSH
> should be possible somehow. Until now we only used TELNET.

SORRY, I was probably half-sleep when I said that. Probably, you CANNOT use Fuego's SSH transport because you are using serial ports on the boards' side.

Your setup is probably closer to what we call "Serial" transport. In other words, I believe that the console server provides you only with a serial console, not a session where you can send or receive files to and from the board. 

If you can install your own software (e.g. serio) on the console server, then you can probably use the the Fuego transport "ssh2serial". I believe that this transport uses SSH to get to the console server and then uses "serio" to put/get files through the serial port.
Tim: could you confirm that?

But as I said above, this is not the ideal setup or use case for Fuego. Fuego works much better when it has direct SSH access to the board.

For your architecture, I think that LAVA and LKFT test definitions might suit you better.

Thanks,
Daniel

> >> Is there something ready that we could just reuse?
> >
> > Hopefully, you don't need to add anything.
> 
> Just the modbus driver :)
> 
> Thanks & have a nice day!
> 
> Heinrich

Re: [Fuego] How to add a Power-Controller and Console Server to FUEGO?

[daniel.sangorrin]

> -----Original Message-----
> From: sangorrin daniel(サンゴリン ダニエル ○ＳＷＣ□ＯＳＴ)
f 
> 1) You probably want to install the latest kernel and file system image before you start testing. For example, you
> can do that by booting from tftpboot, or you could use some more advanced solution such as LAVA. Both of them
> require a LAN connection.

Ah sorry, LAVA only needs a network connection on the dispatcher. The boards only need serial. So that would be perfect for your use case.

Regards,
Daniel

Re: [Fuego] How to add a Power-Controller and Console Server to FUEGO?

[daniel.sangorrin]

> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org <fuego-
> 
> > -----Original Message-----
> > From: sangorrin daniel(サンゴリン ダニエル ○ＳＷＣ□ＯＳＴ)
> f
> > 1) You probably want to install the latest kernel and file system image before you start testing. For example,
> you
> > can do that by booting from tftpboot, or you could use some more advanced solution such as LAVA. Both of
> them
> > require a LAN connection.
> 
> Ah sorry, LAVA only needs a network connection on the dispatcher. The boards only need serial. So that would
> be perfect for your use case.

Sorry, to be more accurate the dispatcher (a board or PC that executes LAVA slave) usually requires a network connection to download the OS image, kernel and test binaries. Then, the dispatcher provisions the board through a local ethernet connection or some other method (e.g., an SDCard gadget). In other words, boards usually (not necessarily if you have this SDCard gadget) require a LAN connection to talk to the dispatcher but the boards don't need to be connected to the Internet or to other segments (although sometimes having them connected to Internet could facilitate some tests).

In summary:
- If you want to reuse Fuego built-in tests as they are, you probably want to forget about the console server and just connect the boards to a LAN together with the Fuego host. You will not be able to do boot tests where you need to see the kernel boot messages, Fuego is not focused on that.
[Note] you may be able to run some of them by skipping the build/deploy phase as I mentioned before.
- If you want to write your own tests, then you can use Fuego with your setup. You would just need to add Telnet as a transpor to Fuego (I can help you with that) and then specify the Telnet port on your board files (I assume that telnet port identifies the serial port to use). Then you would need to have a separate tool for provisioning the board with the latest image and test binaries (e.g., burn the sdcard by hand, or use tftpboot etc). Your tests would just send commands, read the results on the serial connection (telnet) and parse them.
- You may also want to consider using a LAVA/LKFT setup instead of Fuego. This might be harder to setup though.

Thanks,
Daniel

> 
> Regards,
> Daniel
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] How to add a Power-Controller and Console Server to FUEGO?

[Tim.Bird]

> -----Original Message-----
> From: daniel.sangorrin@toshiba.co.jp
> 
> > From: Heinrich.Toews@wago.com <Heinrich.Toews@wago.com>
> > On 19.06.19 02:47, daniel.sangorrin@toshiba.co.jp wrote:
> > > Hello Heinrich,
> > >
> > >> From: fuego-bounces@lists.linuxfoundation.org <fuego-
> bounces@lists.linuxfoundation.org> On Behalf Of
> > >> Heinrich.Toews@wago.com
> > >>
> > >> Hey altogether,
> > >>
> > >> at our company we are willing to use FUEGO for our Linux Kernel
> Testing.
> > >>
> > >> We already have installed Racks ready with our own MODBUS TCP/UDP
> > >> power-controllers and console servers interfacing with TELNET/SSH.
> > >
> > > I will assume that this is what you want to do, otherwise please let me
> know:
> > >
> > > [FUEGO HOST] --TCP--> [MODBUS TCP/UDP]--power on/off--> [boards
> under test]
> > >               `---SSH--> [Console Server]--serial port-----------^
> >
> > Yea, you're right that's exactly what I want to do!
> 
> Oops I missed something important.
> Are the boards under test connected to a LAN?
> I ask that for two reasons:
> 
> 1) You probably want to install the latest kernel and file system image before
> you start testing. For example, you can do that by booting from tftpboot, or
> you could use some more advanced solution such as LAVA. Both of them
> require a LAN connection.
> [Note] Alternatively, you can install the latest kernel/image manually.
> 
> 2) You also need to install the tests that you want to run. Most Fuego built-in
> tests assume that there is a connection with the board to install test software
> (a transport layer). If you don't have such a connection, you will need to build
> your tests in your image and skip the build and deploy phase in Fuego (using
> ftc command arguments).
> 
> > >> What would be the best practice to add such support to the FUEGO
> code?
> > >
> > > For the power-controllers, I sent a patch to Fuego that adds support for
> PDUdaemon [1]. Unfortunately, it got
> > stalled and never made it into Fuego. Perhaps I can revive that thread.
> > > https://lists.linuxfoundation.org/pipermail/fuego/2019-
> March/thread.html
> >
> > Thanks! I walked briefly through the thread and I find it reasonable to
> > use PDUdaemon for this purpose .. It would be great if it gets revived!
> >
> > I think many will have this requirement ..
> 
> OK, give me a bit of time and I will re-send the patches.

I don't recall why the patches stalled last time, but 
I'm committed to having Fuego support PDUdaemon, and I'm confident
we can work out a solution.  I have been wanting to avoid having
Fuego manage the provisioning layer, but I think we need to add
something for those users who don't want to install something heavy
like LAVA.

I believe we kicked around some ideas for this last Fall that I thought
were promising.

> 
> > > In parallel, could you check if PDUdaemon supports your power
> controllers?.
> > >
> https://github.com/pdudaemon/pdudaemon/tree/master/pdudaemon/driv
> ers
> >
> > A Modbus driver is not included yet but it would be an easy task to add it.
> 
> OK, good.
> [Note] Another option is to add new BOARD_CONTROL type or override
> ov_board_control_power_xxx functions in your board file.
> 
> > > For the console server, I guess that you can use the SSH transport
> support available in Fuego. I have never
> > used a Console server. Does it use a different SSH port for each of the
> underlying serial port connections?. If that
> > is the case, you will need to define the SSH_PORT on the corresponding
> board file at `fuego-ro/boards`. If you
> > are using different user names instead, then you would need to set the
> SSH_KEY or the SSH user and password
> > (LOGIN, PASSWORD) in the board file.
> >
> > Okay, thanks - I will try this. We are using ConnectPort LTS32 .. SSH
> > should be possible somehow. Until now we only used TELNET.
> 
> SORRY, I was probably half-sleep when I said that. Probably, you CANNOT
> use Fuego's SSH transport because you are using serial ports on the boards'
> side.
> 
> Your setup is probably closer to what we call "Serial" transport. In other
> words, I believe that the console server provides you only with a serial
> console, not a session where you can send or receive files to and from the
> board.

Agreed.  The diagram is not complete.  If the board has telnet, then that
implies networking, which was not in Daniel's initial diagram.  The most
common transport used by Fuego is network-based (ssh), but others
are available, and we could probably adapt an existing one to your use case
if needed.
> 
> If you can install your own software (e.g. serio) on the console server, then
> you can probably use the the Fuego transport "ssh2serial". I believe that this
> transport uses SSH to get to the console server and then uses "serio" to
> put/get files through the serial port.
> Tim: could you confirm that?
Yes.  The key is that ssh needs to be able to execute serio on the console server,
so it has to be an open system.  The group that was using ssh2serial was using
a general computer system (I think a raspberry pi), so they could install the needed
software on the intermediary machine.

> But as I said above, this is not the ideal setup or use case for Fuego. Fuego
> works much better when it has direct SSH access to the board.

Any network access that provides command execution and file transfer
are fine.  Most Fuego users use ssh and scp, but telnet and rcp or ftp
would work.  There are some issues (mostly to do with performance)
with using serial, or ssh2serial as transports, but they can be made to work
in most circumstances.
 
 -- Tim

Re: [Fuego] How to add a Power-Controller and Console Server to FUEGO?

[Tim.Bird]

Sorry for the slow response.  I was on vacation last week.

> -----Original Message-----
> From: Heinrich.Toews@wago.com
> 
> Hey altogether,
> 
> at our company we are willing to use FUEGO for our Linux Kernel Testing.
> 
> We already have installed Racks ready with our own MODBUS TCP/UDP
> power-controllers and console servers interfacing with TELNET/SSH.
> 
> What would be the best practice to add such support to the FUEGO code?
> Is there something ready that we could just reuse?

When you say "add such support" to the Fuego code, are you
referring to just the interfaces to the board management hardware
(your power controllers and console servers)?  Or are you referring
to your actual hardware under test?

I think Daniel already stated that Fuego usually talks to the hardware under
test using network protocols (ssh and scp), but that other transport
mechanisms are available, or can be written quickly.

And I think that you mentioned in this thread that you could probably
write a driver for PDUdaemon for your power controller relatively
easily.

Are the console servers used for board control (ie, are they used
during the software re-installation process?), or just for monitoring
outputs from the board serial console during a test?

The reason I ask is that these are two separate areas in Fuego.
Fuego currently doesn't have much integrated support for board
provisioning (the process of getting new software-under-test on
a board).  This is why Daniel recommended you look at LAVA, which 
does have a lot of board provisioning support.
Note that board provisioning  may end up being a focus for Fuego's
next release (which I hope to start soon).

If the console servers are used for monitoring kernel/board output
during a test, that would go through a different Fuego sub-system.
There has been some work in the 1.5 release to add support for
multiple monitored output streams from the target, but this has
not been released yet.  However, if it would solve problems for you,
it might become a focus to finish this work up.

Thanks,
 -- Tim

> 
> Thanks!
> 
> Heinrich Toews
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] How to add a Power-Controller and Console Server to FUEGO?

[daniel.sangorrin]

Hi Tim,

> From: Tim.Bird@sony.com <Tim.Bird@sony.com>
[...]
> > > > For the power-controllers, I sent a patch to Fuego that adds support for
> > PDUdaemon [1]. Unfortunately, it got
> > > stalled and never made it into Fuego. Perhaps I can revive that thread.
> > > > https://lists.linuxfoundation.org/pipermail/fuego/2019-
> > March/thread.html
> > >
> > > Thanks! I walked briefly through the thread and I find it reasonable to
> > > use PDUdaemon for this purpose .. It would be great if it gets revived!
> > >
> > > I think many will have this requirement ..
> >
> > OK, give me a bit of time and I will re-send the patches.
> 
> I don't recall why the patches stalled last time, but
> I'm committed to having Fuego support PDUdaemon, and I'm confident
> we can work out a solution.  

The main discussion was about where to put the power control abstraction [1]. At the moment, ttc functions are referenced from the base-board.fuegoclass script (ov_board_control_reboot). The problem is that to use all those functions from "ftc power-on" we would need to export a lot of variables required by Fuego.

Instead, I proposed that we demultiplex "ftc power-on/off/reboot" within "ftc", by using a switch on the $BOARD_CONTROL tool (e.g. pdudaemon, ttc, adb), and let scripts call ftc power control functions in a nested manner.

There was another element of discussion [2] related to including PDUdaemon in Fuego or leaving it outside. After talks with PDUdaemon's maintainer Matt, I think that leaving it outside Fuego (running on its own docker container) is more flexible, and easier to integrate and maintain.

> I have been wanting to avoid having
> Fuego manage the provisioning layer, but I think we need to add
> something for those users who don't want to install something heavy
> like LAVA.
> 
> I believe we kicked around some ideas for this last Fall that I thought
> were promising.

Yeah, you can see some of the ideas in the same thread as the PDUdaemon series [3].
We can discuss that during the Fuego jamboree.

[1] https://lists.linuxfoundation.org/pipermail/fuego/2019-March/003079.html
[2] https://lists.linuxfoundation.org/pipermail/fuego/2019-March/003040.html
[3] https://lists.linuxfoundation.org/pipermail/fuego/2019-March/003082.html

[...]
> > Your setup is probably closer to what we call "Serial" transport. In other
> > words, I believe that the console server provides you only with a serial
> > console, not a session where you can send or receive files to and from the
> > board.
> 
> Agreed.  The diagram is not complete.  If the board has telnet, then that
> implies networking, which was not in Daniel's initial diagram. 

I think that the telnet connection Heinrich was talking about is between the host and the console server.
For Fuego, the best would be to drop the console server and substitute it by a switching hub.
In the future, we may want to have ssh AND serial (e.g. for bootlogs). In that case, the console server can be useful.

> The most
> common transport used by Fuego is network-based (ssh), but others
> are available, and we could probably adapt an existing one to your use case
> if needed.
> >
> > If you can install your own software (e.g. serio) on the console server, then
> > you can probably use the the Fuego transport "ssh2serial". I believe that this
> > transport uses SSH to get to the console server and then uses "serio" to
> > put/get files through the serial port.
> > Tim: could you confirm that?
> Yes.  The key is that ssh needs to be able to execute serio on the console server,
> so it has to be an open system.  The group that was using ssh2serial was using
> a general computer system (I think a raspberry pi), so they could install the needed
> software on the intermediary machine.

The console server that he is using seems to provide some programming functionality but it doesn't look as generic as using something like a raspberry pi or a PC as intermediary machine.

> 
> > But as I said above, this is not the ideal setup or use case for Fuego. Fuego
> > works much better when it has direct SSH access to the board.
> 
> Any network access that provides command execution and file transfer
> are fine.  Most Fuego users use ssh and scp, but telnet and rcp or ftp
> would work.  There are some issues (mostly to do with performance)
> with using serial, or ssh2serial as transports, but they can be made to work
> in most circumstances.

I think that telnet is used just as a "remote serial" port that allows receiving serial port messeges remotely.

Thanks,
Daniel

Re: [Fuego] How to add a Power-Controller and Console Server to FUEGO?

[Tim.Bird]

> -----Original Message-----
> From: daniel.sangorrin@toshiba.co.jp
> 
> Hi Tim,
> 
> > From: Tim.Bird@sony.com <Tim.Bird@sony.com>
> [...]
> > > > > For the power-controllers, I sent a patch to Fuego that adds support
> for
> > > PDUdaemon [1]. Unfortunately, it got
> > > > stalled and never made it into Fuego. Perhaps I can revive that thread.
> > > > > https://lists.linuxfoundation.org/pipermail/fuego/2019-
> > > March/thread.html
> > > >
> > > > Thanks! I walked briefly through the thread and I find it reasonable to
> > > > use PDUdaemon for this purpose .. It would be great if it gets revived!
> > > >
> > > > I think many will have this requirement ..
> > >
> > > OK, give me a bit of time and I will re-send the patches.
> >
> > I don't recall why the patches stalled last time, but
> > I'm committed to having Fuego support PDUdaemon, and I'm confident
> > we can work out a solution.
> 
> The main discussion was about where to put the power control abstraction
> [1]. At the moment, ttc functions are referenced from the base-
> board.fuegoclass script (ov_board_control_reboot). The problem is that to
> use all those functions from "ftc power-on" we would need to export a lot of
> variables required by Fuego.

Yeah, that's not good.

> 
> Instead, I proposed that we demultiplex "ftc power-on/off/reboot" within
> "ftc", by using a switch on the $BOARD_CONTROL tool (e.g. pdudaemon, ttc,
> adb), and let scripts call ftc power control functions in a nested manner.

I think my objection to this was that I don't want ftc to become some kind
of super-tool, and making it the abstraction layer for Fuego doesn't help
with defragmenting this layer across multiple tools.  We could start there,
and try to develop some ideas for handling the board management
abstraction with multiple different board management layers, to see what
shakes out as a good API here.

> 
> There was another element of discussion [2] related to including
> PDUdaemon in Fuego or leaving it outside. After talks with PDUdaemon's
> maintainer Matt, I think that leaving it outside Fuego (running on its own
> docker container) is more flexible, and easier to integrate and maintain.

I'm OK with this, as long as it's relatively easy to run (and I can do it from inside
or outside the Fuego container).  I do like having Fuego be a "batteries included"
system, so that someone starting from scratch doesn't have to go out and 
learn and integrate multiple systems just to get started.
 
> > I have been wanting to avoid having
> > Fuego manage the provisioning layer, but I think we need to add
> > something for those users who don't want to install something heavy
> > like LAVA.
> >
> > I believe we kicked around some ideas for this last Fall that I thought
> > were promising.
> 
> Yeah, you can see some of the ideas in the same thread as the PDUdaemon
> series [3].
> We can discuss that during the Fuego jamboree.

Sounds good. Lets discuss all of this at OSSJ/ALS and FJ3.

Are you going to OSSJ?
 -- Tim


> 
> [1] https://lists.linuxfoundation.org/pipermail/fuego/2019-
> March/003079.html
> [2] https://lists.linuxfoundation.org/pipermail/fuego/2019-
> March/003040.html
> [3] https://lists.linuxfoundation.org/pipermail/fuego/2019-
> March/003082.html
> 
> [...]
> > > Your setup is probably closer to what we call "Serial" transport. In other
> > > words, I believe that the console server provides you only with a serial
> > > console, not a session where you can send or receive files to and from
> the
> > > board.
> >
> > Agreed.  The diagram is not complete.  If the board has telnet, then that
> > implies networking, which was not in Daniel's initial diagram.
> 
> I think that the telnet connection Heinrich was talking about is between the
> host and the console server.

OK - that's right.  I didn't interpret that correctly.

> For Fuego, the best would be to drop the console server and substitute it by
> a switching hub.
> In the future, we may want to have ssh AND serial (e.g. for bootlogs). In that
> case, the console server can be useful.
> 
> > The most
> > common transport used by Fuego is network-based (ssh), but others
> > are available, and we could probably adapt an existing one to your use case
> > if needed.
> > >
> > > If you can install your own software (e.g. serio) on the console server,
> then
> > > you can probably use the the Fuego transport "ssh2serial". I believe that
> this
> > > transport uses SSH to get to the console server and then uses "serio" to
> > > put/get files through the serial port.
> > > Tim: could you confirm that?
> > Yes.  The key is that ssh needs to be able to execute serio on the console
> server,
> > so it has to be an open system.  The group that was using ssh2serial was
> using
> > a general computer system (I think a raspberry pi), so they could install the
> needed
> > software on the intermediary machine.
> 
> The console server that he is using seems to provide some programming
> functionality but it doesn't look as generic as using something like a raspberry
> pi or a PC as intermediary machine.

I thought that might be the case.  It's not something you could ssh through.
For commands it might be possible to do some kind of telnet2serial transport,
but I don't think you could make file operations (get and put) work.

> 
> >
> > > But as I said above, this is not the ideal setup or use case for Fuego.
> Fuego
> > > works much better when it has direct SSH access to the board.
> >
> > Any network access that provides command execution and file transfer
> > are fine.  Most Fuego users use ssh and scp, but telnet and rcp or ftp
> > would work.  There are some issues (mostly to do with performance)
> > with using serial, or ssh2serial as transports, but they can be made to work
> > in most circumstances.
> 
> I think that telnet is used just as a "remote serial" port that allows receiving
> serial port messeges remotely.
Agreed.

 -- Tim

Re: [Fuego] How to add a Power-Controller and Console Server to FUEGO?

[daniel.sangorrin]

Hi Tim,

> From: Tim.Bird@sony.com <Tim.Bird@sony.com>
> > Instead, I proposed that we demultiplex "ftc power-on/off/reboot" within
> > "ftc", by using a switch on the $BOARD_CONTROL tool (e.g. pdudaemon, ttc,
> > adb), and let scripts call ftc power control functions in a nested manner.
> 
> I think my objection to this was that I don't want ftc to become some kind
> of super-tool, and making it the abstraction layer for Fuego doesn't help
> with defragmenting this layer across multiple tools.  We could start there,
> and try to develop some ideas for handling the board management
> abstraction with multiple different board management layers, to see what
> shakes out as a good API here.

I have upgraded my patches to the latest version of Fuego and sent them to the list separately with a few warnings and questions. Note that the patches are only for discussion at this stage. BTW, I had problems installing fuego ("GPG error"), hopefully it will clean up tomorrow.

One of the questions I ask there is: how can I move from the patches I sent to the board management abstraction that you envision.

> Sounds good. Lets discuss all of this at OSSJ/ALS and FJ3.
> 
> Are you going to OSSJ?

Yes.

Thanks,
Daniel

Re: [Fuego] How to add a Power-Controller and Console Server to FUEGO?

[Heinrich.Toews]

Hi Tim,

sorry for this late answer .. We were discussing a lot of things in between.

On 26.06.19 03:16, Tim.Bird@sony.com wrote:
> Sorry for the slow response.  I was on vacation last week.
> 
>> -----Original Message-----
>> From: Heinrich.Toews@wago.com
>>
>> Hey altogether,
>>
>> at our company we are willing to use FUEGO for our Linux Kernel Testing.
>>
>> We already have installed Racks ready with our own MODBUS TCP/UDP
>> power-controllers and console servers interfacing with TELNET/SSH.
>>
>> What would be the best practice to add such support to the FUEGO code?
>> Is there something ready that we could just reuse?
> 
> When you say "add such support" to the Fuego code, are you
> referring to just the interfaces to the board management hardware
> (your power controllers and console servers)?  Or are you referring
> to your actual hardware under test?

My original intention was to see both handled by FUEGO as it is also 
done by LAVA or labgrid.

Below I try to summarize our current thoughts on this.

> 
> I think Daniel already stated that Fuego usually talks to the hardware under
> test using network protocols (ssh and scp), but that other transport
> mechanisms are available, or can be written quickly.

Our console server ships a proprietary RealPort Linux driver which 
creates TTY nodes and so allows us to use standard serial console tools. 
For this reason we use in FUEGO 'serial' as transport mechanism.

For the tests we use the serial interface only as a command/log 
interface for the DUT target and a network interface for file transfer. 
FUEGO intended the transport mechanism to be used for command AND data 
communication so when using 'serial' as transport mechanism we patched 
it not to be used for put and get operations to speed up the test 
deployment. That's ugly but it helps for now to satisfy our 
requirements. It takes too long to transfer a 5 MiB file  through a 
115200 bps serial interface.

> 
> And I think that you mentioned in this thread that you could probably
> write a driver for PDUdaemon for your power controller relatively
> easily.

Yes, I did that by using pyModbus and it works very well.

I applied Daniel's patchset. Thanks for this!

It's really great to have a 'ftc' pdu command to be able to control the 
power, especially the reboot command.

> 
> Are the console servers used for board control (ie, are they used
> during the software re-installation process?), or just for monitoring
> outputs from the board serial console during a test?

We intended them to be used in both cases.

> 
> The reason I ask is that these are two separate areas in Fuego.
> Fuego currently doesn't have much integrated support for board
> provisioning (the process of getting new software-under-test on
> a board).  This is why Daniel recommended you look at LAVA, which
> does have a lot of board provisioning support.
> Note that board provisioning  may end up being a focus for Fuego's
> next release (which I hope to start soon).

We had both in mind. Concerning LAVA: we used it for a while but there 
were some issues that we didn't want to follow-up with. I need to look 
at it again to pick up the details. One thing I remember is that we 
needed to patch our firmware images for some reason. At this moment we 
decided not to use LAVA for provisioning.

Instead for provisioning we are thinking about implementing our own 
scripts for firmware deployment (over adb, fastboot, tftp/scp) that can 
be triggered by different Test-Frameworks. Similar to LAVA we would 
instruct the bootloader (we use barebox) to load the different artifacts 
through network using dhcp/tftp. In this context we would need to 
re-power the DUT. Here we need to decide if we want to call 'ftc pdu' 
here our just using the pyModbus module directly. FUEGO needs to provide 
the configuration parameters like IP address and port number. What we 
also think of is to check prior for the firmware and linux kernel 
version before re-installing everything again.

> 
> If the console servers are used for monitoring kernel/board output
> during a test, that would go through a different Fuego sub-system.
> There has been some work in the 1.5 release to add support for
> multiple monitored output streams from the target, but this has
> not been released yet.  However, if it would solve problems for you,
> it might become a focus to finish this work up.

I'm not sure if I'm getting it right .. I think we would not need 
multiple output stream monitoring as we only are watching the serial 
interface.

We decided to monitor the serial interface because during some rt tests 
it can happen, that the network interface gets freezed due to a kernel 
dump. Such things happened in the past and we have to react on it.


Thanks,

Heinrich