OpenBMC on the OpenWRT

OpenBMC on the OpenWRT

[Anton D. Kachalov]

Hello everybody.

I'm working on a separate project for rack solution which utilize many portions of OpenBMC:

   https://github.com/ya-mouse/openbmc-target

As a system base we choose OpenWRT (flexible kernel-style configurations, package manager,
wide and experienced support for embedded systems with limited resources).
We still use old u-boot that came from OpenSource release of AMI SDK.
Have plans to move to the OpenBMC's version to start using of DTS.
And a few modifications over OpenBMC kernel tree:

1. I2C slave support for the aspeed adapter (it is ugly a bit).
2. I2C slave support for MUX switch (pca954x).
3. DTS overlay support ported from Raspberry Pi with userspace dtoverlay/dtparam/dtmerge tools to manage configs.
4. LuaJIT + Nginx as a HTTP API backend instead of python/micropython.
    We're on internal dissccusion of the API as a mainstream replacement for IPMB/IPMI.
    We use local memcached for fast serving data.
5. Transparent master/slave i2c based IPMB driver with OpenIPMI interface
    (works with regular ipmitool/freeipmi) to send requests to the Nodes.
    To work in a daemon mode kernel part of ipmi_msghandler should be modified a bit
    to receive incoming requests and store it separately while user (opener of the /dev/ipmiX)
    assigned by the outgoing SeqId. For unwanted (from current module's point of view)
    incoming packets there is currently no way to proper store such messages.

Just for reference.
We've compared performance of the several dynamic HTTP configurations such as
python/micropython/lua (LuCI) and embeeded lua within nginx and discover that last one has almost
maximum RPS comparing to the static pages serving.
Static is about 350 RPS while dynamic nginx+luajit with auth checking is at 220RPS.
Simple python's http serves at 40 RPS. LuCI variant is 4-6 RPS.
First request to authentication service occurs via https and the result stored in the
local nginx shared memory storage for the further use (configured for one minute).

-- 
Anton D. Kachalov

Re: OpenBMC on the OpenWRT

[Joel Stanley]

Hey Anton,

On Wed, Aug 10, 2016 at 7:26 PM, Anton D. Kachalov <mouse@yandex-team.ru> wrote:
> Hello everybody.
>
> I'm working on a separate project for rack solution which utilize many portions of OpenBMC:
>
>    https://github.com/ya-mouse/openbmc-target

Nice! I came across your repository the other day.

>
> As a system base we choose OpenWRT (flexible kernel-style configurations, package manager,
> wide and experienced support for embedded systems with limited resources).

I'm not too familiar with OpenWRT aside from using it on my WiFi
router at home. I do have experience with buildroot, which uses
Kconfig and does a good job at creating small, configurable images.

Have you been happy with OpenWRT as a build system?

> We still use old u-boot that came from OpenSource release of AMI SDK.
> Have plans to move to the OpenBMC's version to start using of DTS.

Great! Let me know how you go. I'm happy to apply patches in order to
support your configuration. Our goal is to upstream our patches to
remove the need to fork it all together.

> And a few modifications over OpenBMC kernel tree:
>
> 1. I2C slave support for the aspeed adapter (it is ugly a bit).

Did you see the slave patch that Brendan submitted? Would it work for
your requirements?

  https://github.com/openbmc/linux/commit/090cb02e01906e7a7cf9c210237c4899972a9770

> 2. I2C slave support for MUX switch (pca954x).
> 3. DTS overlay support ported from Raspberry Pi with userspace dtoverlay/dtparam/dtmerge tools to manage configs.
> 4. LuaJIT + Nginx as a HTTP API backend instead of python/micropython.
>     We're on internal dissccusion of the API as a mainstream replacement for IPMB/IPMI.
>     We use local memcached for fast serving data.
> 5. Transparent master/slave i2c based IPMB driver with OpenIPMI interface
>     (works with regular ipmitool/freeipmi) to send requests to the Nodes.
>     To work in a daemon mode kernel part of ipmi_msghandler should be modified a bit
>     to receive incoming requests and store it separately while user (opener of the /dev/ipmiX)
>     assigned by the outgoing SeqId. For unwanted (from current module's point of view)
>     incoming packets there is currently no way to proper store such messages.
>
> Just for reference.
> We've compared performance of the several dynamic HTTP configurations such as
> python/micropython/lua (LuCI) and embeeded lua within nginx and discover that last one has almost
> maximum RPS comparing to the static pages serving.
> Static is about 350 RPS while dynamic nginx+luajit with auth checking is at 220RPS.
> Simple python's http serves at 40 RPS. LuCI variant is 4-6 RPS.
> First request to authentication service occurs via https and the result stored in the
> local nginx shared memory storage for the further use (configured for one minute).

Thanks for sharing your results. How does nginx go with memory usage?

Cheers,

Joel

Re: OpenBMC on the OpenWRT

[Anton D. Kachalov]

Hi, Joel.

10.08.2016, 14:06, "Joel Stanley" <joel@jms.id.au>:
> Hey Anton,
>
> On Wed, Aug 10, 2016 at 7:26 PM, Anton D. Kachalov <mouse@yandex-team.ru> wrote:
>>  Hello everybody.
>>
>>  I'm working on a separate project for rack solution which utilize many portions of OpenBMC:
>>
>>     https://github.com/ya-mouse/openbmc-target
>
> Nice! I came across your repository the other day.
>
>>  As a system base we choose OpenWRT (flexible kernel-style configurations, package manager,
>>  wide and experienced support for embedded systems with limited resources).
>
> I'm not too familiar with OpenWRT aside from using it on my WiFi
> router at home. I do have experience with buildroot, which uses
> Kconfig and does a good job at creating small, configurable images.
>
> Have you been happy with OpenWRT as a build system?

Yep, we happy with it. We rebase other project from micro ubuntu system
(handcrafted files' lists to keep system as small as 4M + additional app packs) to the
OpenWRT on x86 in our HW load testing and servers' pre-deployment environment.
Of course, there are several tricks that needed to be learned :)

>>  We still use old u-boot that came from OpenSource release of AMI SDK.
>>  Have plans to move to the OpenBMC's version to start using of DTS.
>
> Great! Let me know how you go. I'm happy to apply patches in order to
> support your configuration. Our goal is to upstream our patches to
> remove the need to fork it all together.

I'll try to rebase our current code to the latest OpenBMC (pwmfan, pinctrl, i2c slave).

>>  And a few modifications over OpenBMC kernel tree:
>>
>>  1. I2C slave support for the aspeed adapter (it is ugly a bit).
>
> Did you see the slave patch that Brendan submitted? Would it work for
> your requirements?
>
>   https://github.com/openbmc/linux/commit/090cb02e01906e7a7cf9c210237c4899972a9770

It's good! I'll try it. Next thing that we should think is a buffer mode additional to the byte mode transfer.
It's good for big transactions to offload byte ACKing to HW but requires some changes in, for instance,
i2c slave support design. I've started such discussion in kernel's i2c maillist:

    http://www.spinics.net/lists/linux-i2c/msg25656.html

> Thanks for sharing your results. How does nginx go with memory usage?

It is twice more than uhttpd, but still has a little memory footprint (3-4MB).

I would like to start a little discussion around board setup process in arch/arm/mach-aspeed/aspeed.c.
For most cases this process is "flat": just modifying regs one-by-one. Sometimes with applying mask.
I have an idea to add an array of "instructions" to the Device Tree in the following manner
(I hope dtc's preprocess via cpp resolve ORed macro in the correct way):

==========
#include <dt-bindings/soc/aspeed.h>

/ {
    model = "Our HW";
    compatible = "aspeed,ast2400";
    board-setup = <WR, AST_BASE_LPC | 0x9c, 0x02010023, 0>,
                           <WR, AST_BASE_SCU, SCU_PASSWORD, 0>,
                           <WR, AST_BASE_SCU | 0x80, 0xcb000000, 0>,
                           ...
                            /* write(read() | 0x1000) */
                           <OR, AST_BASE_SCU | 0x2c, 0x00001000, 0>,

                           /* write((read() &~ 0x2000) | 0x1000) */
                           <OR, AST_BASE_SCU | 0x2c, 0x00001000, 0x2000>;
==========

or in less talkative way:

==========
                            /* write((read() &~ 0) | 0xcb000000) */
                           <AST_BASE_SCU | 0x80, 0xcb000000, 0>,
                           ...
                            /* write((read() &~ 0x1000) | 0x1000) */
                           <AST_BASE_SCU | 0x2c, 0x00001000, 0x1000>,
==========

to completely eliminate board-specific init functions.

-- 
Anton D. Kachalov

Re: OpenBMC on the OpenWRT

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 3100 bytes --]

On Wed, Aug 10, 2016 at 12:56:57PM +0300, Anton D. Kachalov wrote:
> Hello everybody.

Hi Anton.  Thank you for connecting with us.

> I'm working on a separate project for rack solution which utilize many portions of OpenBMC:
> 
>    https://github.com/ya-mouse/openbmc-target

I would certainly be interested to understand the use-cases you are
solving.  Our intention is that this OpenBMC code base can be used not
only for typical servers but also storage / network controllers or
blade chassis.  Right now we are focused on a standard server but we are
trying not to design ourselves into a dead-end.

> As a system base we choose OpenWRT (flexible kernel-style configurations, package manager,
> wide and experienced support for embedded systems with limited resources).

Is there a technical advantage that you see by using OpenWRT for your
project or just a familiarity?  A few of us have worked on another
project that used Buildroot which is very similar to OpenWRT, but
settled on Yocto for this project.  Yocto certainly has a steeper
learning curve, but it has much better support for "overlays" than
Buildroot/OpenWRT; that solves a major problem we had collaborating with
others on the Buildroot-based project.

> 4. LuaJIT + Nginx as a HTTP API backend instead of python/micropython.
>     We're on internal dissccusion of the API as a mainstream replacement for IPMB/IPMI.
>     We use local memcached for fast serving data.

Our current API is REST-based but it is a simple introspection of the
dbus objects presented by our userspace applications.  All interprocess
communication is done through well-defined dbus interfaces and we simply
expose those out to the user via REST.

We also intended for this to be a replacement for IPMI, but there seems
to be a lot of resistance to that due to Redfish being very similar in
goals.

We are not necessarily stuck on the python implementation, but would
need to understand the CPU vs flash space trade-off.  Most server BMCs
have very limited flash space (32MB of NOR flash) so that is our largest
constraint.

Your use of memcached does spark an interesting optimization in my mind
for our dbus-based REST server though.  We could do something similar to
cache the state of internal dbus objects and flush the cache based on
the object-changed dbus signals.

> Just for reference.
> We've compared performance of the several dynamic HTTP configurations such as
> python/micropython/lua (LuCI) and embeeded lua within nginx and discover that last one has almost
> maximum RPS comparing to the static pages serving.
> Static is about 350 RPS while dynamic nginx+luajit with auth checking is at 220RPS.
> Simple python's http serves at 40 RPS. LuCI variant is 4-6 RPS.
> First request to authentication service occurs via https and the result stored in the
> local nginx shared memory storage for the further use (configured for one minute).

That sounds significantly more performant.  Do you happen to know the
disk space requirements of this config?

-- 
Patrick Williams

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: OpenBMC on the OpenWRT

[Anton D. Kachalov]

10.08.2016, 22:15, "Patrick Williams" <patrick@stwcx.xyz>:
> On Wed, Aug 10, 2016 at 12:56:57PM +0300, Anton D. Kachalov wrote:
>>  Hello everybody.
>
> Hi Anton. Thank you for connecting with us.
>
>>  I'm working on a separate project for rack solution which utilize many portions of OpenBMC:
>>
>>     https://github.com/ya-mouse/openbmc-target
>
> I would certainly be interested to understand the use-cases you are
> solving. Our intention is that this OpenBMC code base can be used not
> only for typical servers but also storage / network controllers or
> blade chassis. Right now we are focused on a standard server but we are
> trying not to design ourselves into a dead-end.

We're wokring on Rack control solution: several boards that talks to nodes (BMC),
collect sensors, drive fans wall and do rack power capping.
As a further move – replace BMC,but it is more difficult task.
There are BIOS integration and ME interaction. KVM part is solvable: virtual USB hub + usb gadgets
and Spice server with Aspeed video frames decoder.

>
>>  As a system base we choose OpenWRT (flexible kernel-style configurations, package manager,
>>  wide and experienced support for embedded systems with limited resources).
>
> Is there a technical advantage that you see by using OpenWRT for your
> project or just a familiarity? A few of us have worked on another
> project that used Buildroot which is very similar to OpenWRT, but
> settled on Yocto for this project. Yocto certainly has a steeper
> learning curve, but it has much better support for "overlays" than
> Buildroot/OpenWRT; that solves a major problem we had collaborating with
> others on the Buildroot-based project.

I found it more intuitive as a build system. Very flexible and a number of makefile helpers
to extend functionality. Also, there is out-of-the box an user Web interface (LuCI).
We mostly focused on target system size and minimum system processes.
Current installation takes just 5M (kmods, ipmitool, nginx, lua, memcached) + kernel (2.5M).

What are the issues with overlays in OpenWRT comparing to Yocto?

>>  4. LuaJIT + Nginx as a HTTP API backend instead of python/micropython.
>>      We're on internal dissccusion of the API as a mainstream replacement for IPMB/IPMI.
>>      We use local memcached for fast serving data.
>
> Our current API is REST-based but it is a simple introspection of the
> dbus objects presented by our userspace applications. All interprocess
> communication is done through well-defined dbus interfaces and we simply
> expose those out to the user via REST.

We're going to expose ubus methods via HTTP. Registered modules (also written in Lua) expose
their methods to the ubus. Events such as Node replacement or other are going through the ubus.

> We also intended for this to be a replacement for IPMI, but there seems
> to be a lot of resistance to that due to Redfish being very similar in
> goals.

From the first look Redfish feels like a WBEM/CIM but use JSON instead of XML.
Too much overloaded to make simple requests.
API have to be kept simple as much as possible.

> We are not necessarily stuck on the python implementation, but would
> need to understand the CPU vs flash space trade-off. Most server BMCs
> have very limited flash space (32MB of NOR flash) so that is our largest
> constraint.

Python waste the space dramatically (not fit 16M).
LuaJIT is only 340k + 340k lib. Plus several bin OpenWRT modules (~100k).
We fit 16M and have several free megs in overlay JFFS2 part.
Lua is a tradeoff between low-level languages such as C (and very easy to write bindings via FFI)
and scripting languages with ability to modify-and-run during development on the target system.

> Your use of memcached does spark an interesting optimization in my mind
> for our dbus-based REST server though. We could do something similar to
> cache the state of internal dbus objects and flush the cache based on
> the object-changed dbus signals.

Yeap. On node replace we're flushing records belongs to the node. All software that need current
sensors data have to request them from memcached instead of directly poll the sensors.

> That sounds significantly more performant. Do you happen to know the
> disk space requirements of this config?

Nginx+LuaJIT takes 860k + 340k shared luajit lib, libpcre (240k), libcrypto (1M) for SSL.
On compressed FS (squashfs + lzma) it takes much less.

-- 
Anton D. Kachalov