Virtual Media

Virtual Media

[Matt Spinler]

Hi,

We are preparing to start working on virtual media, as in remote mass 
storage.

I saw there was a discussion about it at the hackathon,
https://docs.google.com/document/d/1lvK2o4-mMHQ5R0rs1gaB71SKw0dCAlJjyxIqh5azm3k.

Has anyone done any more work on this yet, or plan to?

I hope to send out a high level design for RFC soon, or people can chime 
in now
with any ideas they have for it.


Thanks!
Matt

Re: Virtual Media

[Matt Spinler]

Hi,

Does anyone know if supporting multiple virtual media instances at once 
is ever a required/useful customer use case?  If not, we will just plan 
on supporting 1 at a time.

Thanks,

Matt


On 2018-07-18 16:55, Matt Spinler wrote:
> Hi,
> 
> We are preparing to start working on virtual media, as in remote mass 
> storage.
> 
> I saw there was a discussion about it at the hackathon,
> https://docs.google.com/document/d/1lvK2o4-mMHQ5R0rs1gaB71SKw0dCAlJjyxIqh5azm3k.
> 
> Has anyone done any more work on this yet, or plan to?
> 
> I hope to send out a high level design for RFC soon, or people can 
> chime in now
> with any ideas they have for it.
> 
> 
> Thanks!
> Matt

Re: Virtual Media

[Jeremy Kerr]

Hi Matt & all,

> I hope to send out a high level design for RFC soon, or people can
> chime in now with any ideas they have for it.

So, as part of working on that design, I've been doing some prototyping
with the virtual media code, and just got some experiments working
yesterday, that expose a file from my local browser to the host machine
(as a USB mass-storage device).

The overall architecture looks like this:

  - the BMC kernel has a network block device (nbd) driver; this
    provides a block device, where read & write operations on the block
    device end up being sent as requests over a socket interface.

  - the nbd infrastructure allows us to use a unix domain socket (rather
    than a network socket) for that interface.

  - so, I wrote a little proxy that does a some setup and then just
    forwards data between that unix domain socket and stdin/stdout.

  - Using a websocket proxy, we can connect that process' stdio to a
    websocket.

  - I wrote a little javascript nbd server to connect to that websocket
    from a browser session, and serve nbd read requests by reading from
    a local file. This gives us a block device on the BMC, that's backed
    by a file accessed by the browser.

  - On the BMC, we create a USB mass-storage gadget (using the ASPEED
    UDC device), that uses the nbd device as the underlying storage.

Using that, I was able to forward an .iso image from my webbrowser on
one side of the country to a machine in our lab on the other. Speed
wasn't fantastic, but should be fine for most cases.

Let me know what you think. The main caveat of this is that this method
isn't able to serve writeable images, as sensible browsers won't allow
served javascript code to modify files.

If you're curious, here's the prototype code for the nbd channel (proxy
+ javascript server):

   https://github.com/jk-ozlabs/jsnbd

This isn't necessarily the code that we'll want to run on OpenBMC, more
a proof-of-concept to make sure that all the parts work.

In order to implement this on OpenBMC, we'd still need a few things:

  - integrating the nbd-proxy into the proper http server framework

  - integration into the actual OpenBMC web UI

  - adding hooks to automatically start/stop the USB mass-storage gadget
    on nbd session setup

Cheers,


Jeremy

Re: Virtual Media

[Vernon Mauery]

On 27-Jul-2018 05:06 PM, Jeremy Kerr wrote:
>Hi Matt & all,
>
>>I hope to send out a high level design for RFC soon, or people can
>>chime in now with any ideas they have for it.

In past BMC implementations, we offered up to four virtual devices to be 
mounted remotely. I can't remember if this included the one used for 
firmware updates or not. But we will probably need more than one.

>So, as part of working on that design, I've been doing some prototyping
>with the virtual media code, and just got some experiments working
>yesterday, that expose a file from my local browser to the host machine
>(as a USB mass-storage device).
>
>The overall architecture looks like this:
>
> - the BMC kernel has a network block device (nbd) driver; this
>   provides a block device, where read & write operations on the block
>   device end up being sent as requests over a socket interface.
>
> - the nbd infrastructure allows us to use a unix domain socket (rather
>   than a network socket) for that interface.
>
> - so, I wrote a little proxy that does a some setup and then just
>   forwards data between that unix domain socket and stdin/stdout.
>
> - Using a websocket proxy, we can connect that process' stdio to a
>   websocket.
>
> - I wrote a little javascript nbd server to connect to that websocket
>   from a browser session, and serve nbd read requests by reading from
>   a local file. This gives us a block device on the BMC, that's backed
>   by a file accessed by the browser.
>
> - On the BMC, we create a USB mass-storage gadget (using the ASPEED
>   UDC device), that uses the nbd device as the underlying storage.

That sounds pretty good from a high level. Nice job using off-the-self 
parts.

>Using that, I was able to forward an .iso image from my webbrowser on
>one side of the country to a machine in our lab on the other. Speed
>wasn't fantastic, but should be fine for most cases.
>
>Let me know what you think. The main caveat of this is that this method
>isn't able to serve writeable images, as sensible browsers won't allow
>served javascript code to modify files.

RO files should really be the biggest use case for this -- host OS 
installs and such. I really don't know how much people will complain if 
we take their RW files away. We are also not requiring them to install 
Java, so they win one and lose one.

--Vernon

>If you're curious, here's the prototype code for the nbd channel (proxy
>+ javascript server):
>
>  https://github.com/jk-ozlabs/jsnbd
>
>This isn't necessarily the code that we'll want to run on OpenBMC, more
>a proof-of-concept to make sure that all the parts work.
>
>In order to implement this on OpenBMC, we'd still need a few things:
>
> - integrating the nbd-proxy into the proper http server framework
>
> - integration into the actual OpenBMC web UI
>
> - adding hooks to automatically start/stop the USB mass-storage gadget
>   on nbd session setup
>
>Cheers,
>
>
>Jeremy

Re: Virtual Media

[Jeremy Kerr]

Hi Vernon,

> In past BMC implementations, we offered up to four virtual devices to be 
> mounted remotely. I can't remember if this included the one used for 
> firmware updates or not. But we will probably need more than one.

That would definitely be possible with this design, we'd just need to
parameterise the nbd device and USB endpoint.

However, I've personally never encountered a need for more than one
virtual media device. As you say, most of this will be for host OS
installs.

But if we do have a compelling case to implement multiple devices, that
shouldn't be a hassle.

> That sounds pretty good from a high level. Nice job using off-the-self 
> parts.

Thanks! Was nice to find that we didn't need to implement custom
protocols & kernel interfaces for this.

> RO files should really be the biggest use case for this -- host OS 
> installs and such. I really don't know how much people will complain if 
> we take their RW files away.

Yes, I'd agree there. As above, I'm not sure how much this would be
used. The one use-case I can imagine is a small writeable filesystem for
saving debug logs (in our case, from petitboot). However, we also have
actual host network available at that point...

It *could* be possible to do RW media in javascript by storing CoW data
of modified blocks client-side (eg, in an IndexedDB). The UX isn't great
for this though, as the user would need to explicitly "download" and
save the modified file after use (and I guarantee that folks will forget
to do that and lose data).

> We are also not requiring them to install Java,

Hooray! :)

Cheers,


Jeremy

Re: Virtual Media

[Stewart Smith]

Jeremy Kerr <jk@ozlabs.org> writes:
>> RO files should really be the biggest use case for this -- host OS 
>> installs and such. I really don't know how much people will complain if 
>> we take their RW files away.
>
> Yes, I'd agree there. As above, I'm not sure how much this would be
> used. The one use-case I can imagine is a small writeable filesystem for
> saving debug logs (in our case, from petitboot). However, we also have
> actual host network available at that point...

zmodem over the terminal right to the browser!

(only slightly not joking)

-- 
Stewart Smith
OPAL Architect, IBM.

Re: Virtual Media

[Jeremy Kerr]

Hi Stewart,

>> Yes, I'd agree there. As above, I'm not sure how much this would be
>> used. The one use-case I can imagine is a small writeable filesystem for
>> saving debug logs (in our case, from petitboot). However, we also have
>> actual host network available at that point...
> 
> zmodem over the terminal right to the browser!

The browser still can't write to the local filesystem :P

Cheers,


Jeremy

Re: Virtual Media

[Stewart Smith]

Jeremy Kerr <jk@ozlabs.org> writes:
> Hi Stewart,
>
>>> Yes, I'd agree there. As above, I'm not sure how much this would be
>>> used. The one use-case I can imagine is a small writeable filesystem for
>>> saving debug logs (in our case, from petitboot). However, we also have
>>> actual host network available at that point...
>> 
>> zmodem over the terminal right to the browser!
>
> The browser still can't write to the local filesystem :P

But you can download the file and save it, which would solve the debug
log problem.

-- 
Stewart Smith
OPAL Architect, IBM.

Re: Virtual Media

[Adriana Kobylak]

Hi Ed,

>  - the nbd infrastructure allows us to use a unix domain socket (rather
>    than a network socket) for that interface.

>  - Using a websocket proxy, we can connect that process' stdio to a
>    websocket.

As you pointed out, the Redfish schema for Virtual Media may not fit 
with this design, so wanted to get your thoughts on how/if we could 
somehow implement having the client request to mount the device via a 
websocket, like by calling the VirtualMedia/<device_id> api, or could we 
leverage some other schemas, or what would be your suggestion? Thanks!

Reference:
https://gerrit.openbmc-project.xyz/#/c/openbmc/bmcweb/+/16481/
https://gerrit.openbmc-project.xyz/#/c/openbmc/phosphor-rest-server/+/14752/

Re: Virtual Media

[Ed Tanous]

On 12/10/18 9:43 AM, Adriana Kobylak wrote:
> Hi Ed,
>
>>  - the nbd infrastructure allows us to use a unix domain socket (rather
>>    than a network socket) for that interface.
>
>>  - Using a websocket proxy, we can connect that process' stdio to a
>>    websocket.
>
> As you pointed out, the Redfish schema for Virtual Media may not fit
> with this design, so wanted to get your thoughts on how/if we could
> somehow implement having the client request to mount the device via a
> websocket, like by calling the VirtualMedia/<device_id> api, or could
> we leverage some other schemas, or what would be your suggestion? Thanks!
>
> Reference:
> https://gerrit.openbmc-project.xyz/#/c/openbmc/bmcweb/+/16481/
> https://gerrit.openbmc-project.xyz/#/c/openbmc/phosphor-rest-server/+/14752/
>
>
I think the root of the issue here is that the Virtual Media Redfish
interface was designed to be able to mount a host from a network share,
rather than proxying over a websocket.  I think in the long term, if
we're going to support all the use cases we want to, we will eventually
need both mechanisms, but for the moment, if we want to get something
that works up quickly, and iterate, we're probably better off just
continuing to make the websocket implementation work reliably.  In my
head (i.e. I haven't backed this up with actual data) I see websocket as
one implementation of a transport layer, with other transport layers
(samba, NFS, ect) that could be supported at a later date.

I suspect the best approach here is to simply add a url handler under
the existing name (/kvmws0) that does the websocket.  When and if
someone goes to build out the Redfish interface for Virtual Media, we
can go implement the redfish specific urls.  If you'd prefer to do it in
the reverse order, that would work as well.

-Ed

Virtual Media

[Rapkiewicz, Pawel]

[-- Attachment #1: Type: text/plain, Size: 1624 bytes --]

Adriana,

We're currently in the middle of Virtual Media implementation based on design: https://github.com/openbmc/docs/blob/master/designs/VirtualMedia.md
There are few patches in review to bmcweb which enables feature as above.  There are few minor differences which made it not 100% compatible with
yours (i.e. different endpoints), on the other hand it has some benefits, like support for Redfish Virtual Media.

As It's reasonable not to support two different implementations of same functionality, I'd like to ask you to review those patches,
and see if we can agree to have one common implementation in bmcweb.

I'm adding Przemyslaw to the thread since he'll take care of Virtual Media implementation from Intel side.

Thanks in advance,
Regards,
Pawel

--------------------------------------------------------------------

Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.

Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek
przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by
others is strictly prohibited.

[-- Attachment #2: Type: text/html, Size: 4740 bytes --]

Re: Virtual Media

[Adriana Kobylak]

Hi Pawel,


> We're currently in the middle of Virtual Media implementation based on
> design:
> https://github.com/openbmc/docs/blob/master/designs/VirtualMedia.md
> [1]
> 
> There are few patches in review to bmcweb which enables feature as
> above. There are few minor differences which made it not 100%
> compatible with
> 
> yours (i.e. different endpoints), on the other hand it has some
> benefits, like support for Redfish Virtual Media.

Great, agree, the project direction is to move to Redfish.

> 
> As It's reasonable not to support two different implementations of
> same functionality, I'd like to ask you to review those patches,
> 
> and see if we can agree to have one common implementation in bmcweb.

I support the Redfish implementation, let's just make sure that the web 
UI
transitions to the new Virtual Media implementation before deprecating
any existing endpoints.

As for the bmcweb changes, I'm not really an expert there so will defer 
to
the maintainers on approving how it's implemented.

Re: Virtual Media

[Ed Tanous]

On 10/10/19 12:30 PM, Adriana Kobylak wrote:
> 
> I support the Redfish implementation, let's just make sure that the web UI
> transitions to the new Virtual Media implementation before deprecating
> any existing endpoints.

+1

> 
> As for the bmcweb changes, I'm not really an expert there so will defer to
> the maintainers on approving how it's implemented.

I think the key here is whether or not the community is ok with
completely replacing the BMCWEB_ENABLE_VM_WEBSOCKET option with the new
scheme, or if there's a reason to keep the old one around.  The new one
seems to be a superset of what exists in terms of features.  If the
webui is moved over to the new, multiple-slot scheme, is there any issue
with that, or is there a reason to keep the single-slot variant around?