[Redfish] VirtualMedia

[Redfish] VirtualMedia

[Rapkiewicz, Pawel]

Hi,

I've reviewed some email threads about implementation and usage of Virtual Media, and I tend to agree, that Virtual Media schema was not designed for usage like web socket proxy. I'm planning to propose some OEM extension that will make Virtual Media closer to our needs, and I'd like to ask you what would be your usage of NBD over web sockets, are you planning to mount NBD device only in BMC, make it available to HOST over i.e. usb gadget, or both depending on use case (Any other?).

I was thinking about implementation of Virtual Media Service, that will allow those actions, but before that I'd like to gather more input.

I'm attaching link to nbd-proxy functionality implementation in bmcweb: https://gerrit.openbmc-project.xyz/#/c/openbmc/bmcweb/+/17749/ 
There are two proposals on how nbd-client can be spawned upon web socket connection:
- systemd watching Unix Socket files
- VirtualMediaService triggered by bmcweb

Systemd tends to be easier, but with less control
VirtualMediaService allows more (like usb gadget), but would be better designed if I'd receive some input on your use cases.

Regards,
Pawel Rapkiewicz
--------------------------------------------------------------------

Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.

Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek
przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by
others is strictly prohibited.

Re: [Redfish] VirtualMedia

[Brad Bishop]

On Mon, Feb 04, 2019 at 10:12:46PM +0000, Rapkiewicz, Pawel wrote:
>Hi,
>
>I've reviewed some email threads about implementation and usage of Virtual Media, and I tend to agree, that Virtual Media schema was not designed for usage like web socket proxy. I'm planning to propose some OEM extension that will make Virtual Media closer to our needs, and I'd like to ask you what would be your usage of NBD over web sockets, are you planning to mount NBD device only in BMC, make it available to HOST over i.e. usb gadget, or both depending on use case (Any other?).

We are planning on making available to the host via usb gadget.  Just
curious - what are some use cases for mounting the file locally on the
BMC?

fwiw Adriana and Jeremy had been working to this design here:

https://lists.ozlabs.org/pipermail/openbmc/2018-July/012647.html

Based on the chat in IRC it sounded like Intel might want to tweak that
slightly and get rid of the nbd-proxy as a stand-alone application and
move that functionality into bmcweb.  Did I understand correctly?  I
don't think I'm opposed to that or anything (Jeremy/Adriana please
comment if you have any concerns there) but can you comment on the
motivation for the change?

>
>I was thinking about implementation of Virtual Media Service, that will allow those actions, but before that I'd like to gather more input.
>
>I'm attaching link to nbd-proxy functionality implementation in bmcweb: https://gerrit.openbmc-project.xyz/#/c/openbmc/bmcweb/+/17749/
>There are two proposals on how nbd-client can be spawned upon web socket connection:
>- systemd watching Unix Socket files
>- VirtualMediaService triggered by bmcweb
>
>Systemd tends to be easier, but with less control
>VirtualMediaService allows more (like usb gadget), but would be better designed if I'd receive some input on your use cases.
>
>Regards,
>Pawel Rapkiewicz
>--------------------------------------------------------------------
>
>Intel Technology Poland sp. z o.o.
>ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.
>
>Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek
>przegladanie lub rozpowszechnianie jest zabronione.
>This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by
>others is strictly prohibited.

RE: [Redfish] VirtualMedia

[Rapkiewicz, Pawel]

Brad, Adriana,

Here are answers for your email, at the bottom I'm describing flow we were talking on IRC channel.

> We are planning on making available to the host via usb gadget.  Just curious - what are some use cases for mounting the file locally on the BMC?

I'm not aware of any, but I can imagine some, one of it might be as you mentioned on IRC channel to perform update, not sure if it will be useful,
and how it would work comparing to other update methods, but it's for free. If there is already possibility to expose image through BMC to HOST,
mounting it on BMC is one step less. That's why I was asking about use cases, it complicates a little a flow, i.e. slightly larger VirtualMedia Oem extention,
but might be worth it. In the below flow I skipped that possibility.

> fwiw Adriana and Jeremy had been working to this design here:

> https://lists.ozlabs.org/pipermail/openbmc/2018-July/012647.html

I've read it before and find using NBD great idea.

> Based on the chat in IRC it sounded like Intel might want to tweak that slightly and get rid of the nbd-proxy as a stand-alone application and move that functionality into bmcweb.  Did I understand correctly?  I don't think I'm opposed to that or anything (Jeremy/Adriana please comment if you have any concerns there) but can you comment on the motivation for the change?

First motivation was to integrate NBD solution inside bmcweb (it was before I found other integration patchsets), and inherit all the benefits from it, like all supported authentication methods, TLS, and one server certificate, as far as I know, there would be problems with having websocketd working with same certificate as bmcweb (I might be wrong at this point). Then I've realized that using boost::asio, and boost::beast nbd-proxy may be incorporated into bmcweb, and the libraries would make it simpler. Moreover  it will remove additional communication channel - pipes, which will decrease context switching and number of syscalls for communication over pipes. To be honest I have no idea how big (or little)  is a gain in performance, it might be worth to measure it, and have comparable results. 

I left spawning of nbd-client, but would like to discuss that. The motivation behind that was that bmcweb is large standalone application which has a lot of functionalities - websockets are one of it, and I don't want to implement spawning and process monitoring inside it, I think it shouldn't be its responsibility, that feature is far away from base bmcweb functionality.

After reviewing https://gerrit.openbmc-project.xyz/#/c/openbmc/meta-phosphor/+/17494/2  
I suggest two options of spawning nbd-client and configuring usb gadget, both were already proposed in nbd-proxy patchset, but here in details

1. Using systemd to monitor /tmp/nbd-proxy.X.sock files, and based on any change it will start nbd-client with appropriate options, like:
nbd-client -u /tmp/nbd-proxy.X.sock -b 512 /dev/nbdX
then it will start "state_hook" script from the above commit with appropriate parameters.
Pros:
- It's almost for free, easy shell script is required to determine which nbd-client shall be connected
- there is no need for watching if nbd-client shall be terminated - if local socket will disappear nbd-client would stop working, stopping usb gadget would be required
Cons:
- There is very little control in options for both nbd-client and usb gadget, options cannot be passed explicitly, monitoring script shall determine which are required
- VirtualMedia redfish schema have no real data to be feed, probably nothing except IP of remote host cannot be exposed,

2. Going with the spirit of OpenBMC as I understand it, and implement service for managing VirtualMedia with appropriate DBus interface, I'd advocate for that.
Pros:
- Gives a lot of flexibility in options for mounting devices, like block-size, timeout, etc
- Can monitor nbd-client, connections, etc and do appropriate actions on usb gadget, i.e. if connection is broken
- Will deliver real data for VirtualMedia schema
- Can be extended (if needed) to allow mounting other types of protocols, as described in DMTF's VirtualMedia schema (NFS, CIFS, etc) all having same DBus interface
Cons:
- This is another service, it will not interfere on communication between NBD-server, and nbd-client, once its connected, but consumes flash, RAM, and once for a while context switching time

So based on nbd-proxy.hpp bmcweb patchset(https://gerrit.openbmc-project.xyz/#/c/openbmc/bmcweb/+/17749/), and choosing second option from the above the flow might look like:

1. bmcweb starts and expose /nbd URI
2. From browser user is serving image through nbd-server (JS)
3. bmcweb accepts connection, verifies credentials (i.e. Basic Auth, session, etc)
4. bmcweb Exposes first free local socket from pool (i.e. /tmp/nbd-proxy.0.sock)
<options>:
A) Easier, automatic
5. bmcweb calls appropriate Virtual Media DBus method, 
B) Manual
5.1. bmcweb exposes VirtualMedia redfish schema with connected NBD Server over WS, and allows OEM "Mount to Host" action with parameters
5.2 bmcweb accepts POST on OEM "Mount to Host" action and calls appropriate Dbus method exposed by Virtual Media
</options>
6. VirtualMedia spawns nbd-client from /tmp/nbd-proxy.0.sock and mounts it to first free /dev/nbdX device, i.e. /dev/nbd0, 
7. VirtualMedia starts monitoring nbd-client, and configures usb gadget to expose /dev/nbd0 to HOST

At this point the traffic goes through this stream
NBDServer(JS) <== WSS ==> bmcweb <== local socket ==> nbd-client <== syscalls? ==> usb gadget

Moreover:
- Virtual Media schema contains all the information delivered from Virtual Media Dbus service, it can contain whatever nbd-client is delivering + IP of remote host, local NBD device, usb gadget data
- Virtual Media schema allows to break connection by DELETE method (or eject action) over HTTP to bmcweb forwarded over Dbus to Virtual Media Schema
- Virtual Media reacts on nbd-client stops working, websocket or local socket disconnection, and is able to clean up (mostly usb gadget configuration)

Regards,
Pawel 


--------------------------------------------------------------------

Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.

Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek
przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by
others is strictly prohibited.

Re: [Redfish] VirtualMedia

[Ed Tanous]

On 2/5/19 9:36 AM, Brad Bishop wrote:
> On Mon, Feb 04, 2019 at 10:12:46PM +0000, Rapkiewicz, Pawel wrote:
> 
> Based on the chat in IRC it sounded like Intel might want to tweak that
> slightly and get rid of the nbd-proxy as a stand-alone application and
> move that functionality into bmcweb.  Did I understand correctly?  I
> don't think I'm opposed to that or anything (Jeremy/Adriana please
> comment if you have any concerns there) but can you comment on the
> motivation for the change?
> 

If you're talking about the conversation I had, it was an idea, but I'm
not sure how feasible it is given my new understanding of how the bnd
interface works in the kernel.  I really need to give it more thought.
Pawel understands this a lot better than I do at this point, and
actually has patches to back it up, so lets let his opinion supersede
mine for the moment.

I would still like to see the two options duke it out though.

(grabs popcorn)

RE: [Redfish] VirtualMedia

[Rapkiewicz, Pawel]

Hi,

I'd like to refresh the topic, since I'm seeing some traffic around Virtual Media, I think this is a right time :)

Regards,
Pawel


-----Original Message-----
From: openbmc [mailto:openbmc-bounces+pawel.rapkiewicz=intel.com@lists.ozlabs.org] On Behalf Of Rapkiewicz, Pawel
Sent: Wednesday, February 6, 2019 12:08 AM
To: Brad Bishop <bradleyb@fuzziesquirrel.com>
Cc: Adriana Kobylak <anoo@linux.vnet.ibm.com>; openbmc@lists.ozlabs.org; Tanous, Ed <ed.tanous@intel.com>
Subject: RE: [Redfish] VirtualMedia

Brad, Adriana,

Here are answers for your email, at the bottom I'm describing flow we were talking on IRC channel.

> We are planning on making available to the host via usb gadget.  Just curious - what are some use cases for mounting the file locally on the BMC?

I'm not aware of any, but I can imagine some, one of it might be as you mentioned on IRC channel to perform update, not sure if it will be useful, and how it would work comparing to other update methods, but it's for free. If there is already possibility to expose image through BMC to HOST, mounting it on BMC is one step less. That's why I was asking about use cases, it complicates a little a flow, i.e. slightly larger VirtualMedia Oem extention, but might be worth it. In the below flow I skipped that possibility.

> fwiw Adriana and Jeremy had been working to this design here:

> https://lists.ozlabs.org/pipermail/openbmc/2018-July/012647.html

I've read it before and find using NBD great idea.

> Based on the chat in IRC it sounded like Intel might want to tweak that slightly and get rid of the nbd-proxy as a stand-alone application and move that functionality into bmcweb.  Did I understand correctly?  I don't think I'm opposed to that or anything (Jeremy/Adriana please comment if you have any concerns there) but can you comment on the motivation for the change?

First motivation was to integrate NBD solution inside bmcweb (it was before I found other integration patchsets), and inherit all the benefits from it, like all supported authentication methods, TLS, and one server certificate, as far as I know, there would be problems with having websocketd working with same certificate as bmcweb (I might be wrong at this point). Then I've realized that using boost::asio, and boost::beast nbd-proxy may be incorporated into bmcweb, and the libraries would make it simpler. Moreover  it will remove additional communication channel - pipes, which will decrease context switching and number of syscalls for communication over pipes. To be honest I have no idea how big (or little)  is a gain in performance, it might be worth to measure it, and have comparable results. 

I left spawning of nbd-client, but would like to discuss that. The motivation behind that was that bmcweb is large standalone application which has a lot of functionalities - websockets are one of it, and I don't want to implement spawning and process monitoring inside it, I think it shouldn't be its responsibility, that feature is far away from base bmcweb functionality.

After reviewing https://gerrit.openbmc-project.xyz/#/c/openbmc/meta-phosphor/+/17494/2
I suggest two options of spawning nbd-client and configuring usb gadget, both were already proposed in nbd-proxy patchset, but here in details

1. Using systemd to monitor /tmp/nbd-proxy.X.sock files, and based on any change it will start nbd-client with appropriate options, like:
nbd-client -u /tmp/nbd-proxy.X.sock -b 512 /dev/nbdX then it will start "state_hook" script from the above commit with appropriate parameters.
Pros:
- It's almost for free, easy shell script is required to determine which nbd-client shall be connected
- there is no need for watching if nbd-client shall be terminated - if local socket will disappear nbd-client would stop working, stopping usb gadget would be required
Cons:
- There is very little control in options for both nbd-client and usb gadget, options cannot be passed explicitly, monitoring script shall determine which are required
- VirtualMedia redfish schema have no real data to be feed, probably nothing except IP of remote host cannot be exposed,

2. Going with the spirit of OpenBMC as I understand it, and implement service for managing VirtualMedia with appropriate DBus interface, I'd advocate for that.
Pros:
- Gives a lot of flexibility in options for mounting devices, like block-size, timeout, etc
- Can monitor nbd-client, connections, etc and do appropriate actions on usb gadget, i.e. if connection is broken
- Will deliver real data for VirtualMedia schema
- Can be extended (if needed) to allow mounting other types of protocols, as described in DMTF's VirtualMedia schema (NFS, CIFS, etc) all having same DBus interface
Cons:
- This is another service, it will not interfere on communication between NBD-server, and nbd-client, once its connected, but consumes flash, RAM, and once for a while context switching time

So based on nbd-proxy.hpp bmcweb patchset(https://gerrit.openbmc-project.xyz/#/c/openbmc/bmcweb/+/17749/), and choosing second option from the above the flow might look like:

1. bmcweb starts and expose /nbd URI
2. From browser user is serving image through nbd-server (JS) 3. bmcweb accepts connection, verifies credentials (i.e. Basic Auth, session, etc) 4. bmcweb Exposes first free local socket from pool (i.e. /tmp/nbd-proxy.0.sock)
<options>:
A) Easier, automatic
5. bmcweb calls appropriate Virtual Media DBus method,
B) Manual
5.1. bmcweb exposes VirtualMedia redfish schema with connected NBD Server over WS, and allows OEM "Mount to Host" action with parameters
5.2 bmcweb accepts POST on OEM "Mount to Host" action and calls appropriate Dbus method exposed by Virtual Media </options> 6. VirtualMedia spawns nbd-client from /tmp/nbd-proxy.0.sock and mounts it to first free /dev/nbdX device, i.e. /dev/nbd0, 7. VirtualMedia starts monitoring nbd-client, and configures usb gadget to expose /dev/nbd0 to HOST

At this point the traffic goes through this stream
NBDServer(JS) <== WSS ==> bmcweb <== local socket ==> nbd-client <== syscalls? ==> usb gadget

Moreover:
- Virtual Media schema contains all the information delivered from Virtual Media Dbus service, it can contain whatever nbd-client is delivering + IP of remote host, local NBD device, usb gadget data
- Virtual Media schema allows to break connection by DELETE method (or eject action) over HTTP to bmcweb forwarded over Dbus to Virtual Media Schema
- Virtual Media reacts on nbd-client stops working, websocket or local socket disconnection, and is able to clean up (mostly usb gadget configuration)

Regards,
Pawel 


--------------------------------------------------------------------

Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.

Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by others is strictly prohibited.

--------------------------------------------------------------------

Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.

Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek
przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by
others is strictly prohibited.

RE: [Redfish] VirtualMedia

[Adriana Kobylak]

> I'd like to refresh the topic, since I'm seeing some traffic around
> Virtual Media, I think this is a right time :)

Good time indeed.

> 
> 1. Using systemd to monitor /tmp/nbd-proxy.X.sock files, and based on
> any change it will start nbd-client with appropriate options, like:
> nbd-client -u /tmp/nbd-proxy.X.sock -b 512 /dev/nbdX then it will
> start "state_hook" script from the above commit with appropriate
> parameters.

One advantage of using the nbd-proxy is that it manages the disk udev 
events
and polls to know when to start the "state_hook".

> 
> 5. bmcweb calls appropriate Virtual Media DBus method,

The first point talks about systemd monitoring the socket creation,
which seems like a stand-alone service file. But here we're talking
about a DBus method that theoretically would start nbd-client?

> 
> - Can be extended (if needed) to allow mounting other types of 
> protocols,
as described in DMTF's VirtualMedia schema (NFS, CIFS, etc) all having 
same
DBus interface the Virtual Media Redfish

The Redfish interface was designed to be able to mount a host from a
network share rather than proxying over a websocket, that's the reason
we haven't added the Redfish endpoint yet. Do you see us eventually
implementing the Redfish API or just sticking with websocket 
implementations?

RE: [Redfish] VirtualMedia

[Rapkiewicz, Pawel]

>> 
>> 1. Using systemd to monitor /tmp/nbd-proxy.X.sock files, and based on 
>> any change it will start nbd-client with appropriate options, like:
>> nbd-client -u /tmp/nbd-proxy.X.sock -b 512 /dev/nbdX then it will 
>> start "state_hook" script from the above commit with appropriate 
>> parameters.

> One advantage of using the nbd-proxy is that it manages the disk udev events and polls to know when to start the "state_hook".

I understand that, and definitely I'm not against using udev, I'd like to avoid limiting throughput by proxying data through one more service. 
We can write a mechanism based on core OS features (systemd/udev). I described systemd usage so far, but it can be split for systemd and udev as well. For example:

systemd:
to monitor if new /tmp/nbd-proxy.X.sock file appeared and start nbd-client accordingly

udev:
add a rule in /etc/udev/rules.d, which will watch for new nbd device attached, and start "state_hook"

The above approach seems to be right, however it limits the functionality to exposing nbd device to host only (or whatever state_hook would do in future). Since there are no others usage so far we can stick with that.

The second one described, with DBus allows more flexibility, but requires additional service on DBus.

>> 
>> 5. bmcweb calls appropriate Virtual Media DBus method,

> The first point talks about systemd monitoring the socket creation, which seems like a stand-alone service file. But here we're talking about a DBus method that theoretically would start nbd-client?

Yes the second option is to write a DBus service for spawning nbd-client, and perhaps (in future) deal with Redfish virtual media, like NFS, CIFS, etc. The DBus approach gives a little more flexibility.

>> 
>> - Can be extended (if needed) to allow mounting other types of 
>> protocols,
as described in DMTF's VirtualMedia schema (NFS, CIFS, etc) all having same DBus interface the Virtual Media Redfish

> The Redfish interface was designed to be able to mount a host from a network share rather than proxying over a websocket, that's the reason we haven't added the Redfish endpoint yet. Do you see us eventually implementing the Redfish API or just sticking with websocket implementations?

If we'd like to have Redfish as primary interface (WebUI using redfish endpoints instead of current ones) I think it's desired to implement such support. Even if current Virtual Media schemas does not cover proxying over websockets, this can be added as OEM extension. 

Regards,
Pawel
--------------------------------------------------------------------

Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.

Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek
przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by
others is strictly prohibited.