thoughts on data and backward and forward compatibility

thoughts on data and backward and forward compatibility

[Brad Bishop]

I would like for OpenBMC to support forward and backward compatibility.  This means changing the code in both directions without having to do a factory reset.  Is anyone opposed to maintaining backward compatibility?

To that ^ end, I would like to talk about application data and where to put it.  The output of the discussion would be a set of guidelines (e.g. something in openbmc/docs) on where to put data to achieve that.  I’m making assertions below but please poke holes.

Most of our applications have data.  Often times it gets stashed in /etc and most times that is probably not the best place to put it.  Some common places to put data on *-nix systems are:

/etc
/usr/share/<package>
/var/lib/<package>
/var/cache/<package>

The following are guidelines and examples to help decide the location in which you place your data.

---
/etc
---
This is static data.  Applications that consume data here should not write to it.  Other applications or end-users may write to it.

Data in /etc is hard to maintain forward and backward compatibility.  When a package that consumes this data is upgraded and it has a new data format, the new package must continue to support the old data format or a package installation process must detect the old data format and migrate it to the new data format.  The latter (migration) makes the possibility of backwards compatibility unlikely.

Further complicating the process, other applications or users may write data here and that data must be “merged” during any migration process.

Given the challenges, avoid placing data in /etc unless it is necessary.

---
/usr/share/<package>
---
This is also static data and applications that consume data here should not write to this data either.

Data where changes from the user or other applications does not need to survive a package upgrade should be placed here.  Applications can change the format of their data found here and need not worry about backwards or forwards compatibility.

—--
/var/lib/<package>
—--
This is dynamic, mutable data.  If your application writes data, it should go here (with one exception below).  Just like in /etc, data here is also hard to maintain forward and backward compatibility, and just like in /etc, applications need to support old and new data formats simultaneously or have data migration.  Applications other than the consumer or end-users should not modify data here so data merging is not required.

—--
/var/cache/<package>
—--
This is also dynamic, mutable data.  If your data can be deleted at any time without impacting the state of the application, the data should go here.  Data here need not be forward/backward compatible - package upgrades or applications can simply remove this data at upgrade or startup time.

thx - brad

Re: thoughts on data and backward and forward compatibility

[Adriana Kobylak]

Thanks Brad for putting this together.


> The following are guidelines and examples to help decide the location
> in which you place your data.

For additional details, refer to the FHS - 
https://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.pdf


> ---
> /etc
> ---
> Further complicating the process, other applications or users may
> write data here and that data must be “merged” during any
> migration process.

Even further, /etc is mounted as an overlayfs that has its own 
restrictions on renaming/deleting files 
(https://en.wikipedia.org/wiki/OverlayFS). The overlay is either mounted 
over / or /etc depending if you want to preserve the rest of the 
filesystem as read-only.


> —--
> /var/lib/<package>
> —--
> This is dynamic, mutable data.  If your application writes data, it
> should go here (with one exception below).  Just like in /etc, data
> here is also hard to maintain forward and backward compatibility, and
> just like in /etc, applications need to support old and new data
> formats simultaneously or have data migration.  Applications other
> than the consumer or end-users should not modify data here so data
> merging is not required.

The cereal package can be used to store data and supports versioning for 
when changes are desired. Multiple apps already use it in OpenBMC.