Openbmc archive at lore.kernel.org
 help / color / Atom feed
* /etc/migration.d
@ 2020-10-14 18:47 Anton Kachalov
  2020-10-16 16:50 ` /etc/migration.d Ed Tanous
  2020-10-16 20:25 ` /etc/migration.d Patrick Williams
  0 siblings, 2 replies; 5+ messages in thread
From: Anton Kachalov @ 2020-10-14 18:47 UTC (permalink / raw)
  To: OpenBMC Maillist


[-- Attachment #1: Type: text/plain, Size: 826 bytes --]

With moving from root-only environment to unprivileged users' space, we
need to ensure a smooth transition. To achieve that we need a mechanism for
one-shot per-package scripts that would take care of migration. That's not
only about groups & owners, but a general approach. It's similar to
firstboot, but has a different purpose.

I'm going to prototype a robust / naive solution to start a service before
everything else in the system with a condition (non-empty /etc/migration.d)
and iterate through all files. Each script has to run at list with "set -e"
to bail out on failures. If the script succeeded -- it will be removed.

The tricky part is: what if the script fails? Keep it, ignore the failure
and proceed with others and then boot the system? Or proceed other scripts
as well and then enter some "failure state"?

[-- Attachment #2: Type: text/html, Size: 937 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: /etc/migration.d
  2020-10-14 18:47 /etc/migration.d Anton Kachalov
@ 2020-10-16 16:50 ` Ed Tanous
  2020-10-16 17:10   ` /etc/migration.d Anton Kachalov
  2020-10-16 20:25 ` /etc/migration.d Patrick Williams
  1 sibling, 1 reply; 5+ messages in thread
From: Ed Tanous @ 2020-10-16 16:50 UTC (permalink / raw)
  To: Anton Kachalov; +Cc: OpenBMC Maillist

On Wed, Oct 14, 2020 at 11:49 AM Anton Kachalov <rnouse@google.com> wrote:
>
> With moving from root-only environment to unprivileged users' space, we need to ensure a smooth transition. To achieve that we need a mechanism for one-shot per-package scripts that would take care of migration. That's not only about groups & owners, but a general approach.

Are there other use cases that necessitate a general approach?  I'm
not against it, but owners and groups seems unique in the regard that
the migration has to run as root.  Most (all?) other migrations don't
seem to, or haven't in the past, and therefore can be run as a
pre-init, or as part of the service itself.  If the service itself
does the migration, the startup dependencies are a lot easier to track
as a maintainer, and running your migrations in a compiled language
likely has a positive effect on boot time, which has been a problem in
the past (still is depending on who you ask).

It should be noted, several apps have done simple migrations of config
file formats in the past, so there's some precedent for it, just not
in a generalist solution.

>  It's similar to firstboot, but has a different purpose.
>
> I'm going to prototype a robust / naive solution to start a service before everything else in the system with a condition (non-empty /etc/migration.d) and iterate through all files. Each script has to run at list with "set -e" to bail out on failures. If the script succeeded -- it will be removed.

The script itself will be removed?  Presumably that means you're
executing the script out of non-volatile?  That seems like a security
gap in that an attacker could inject migration scripts that did
anything, and have the system run them for them.  Maybe just keeping
some kind of external log of "these scripts have completed" or,
preferably, enforcing that migration scripts are idempotent would be
better, and would reduce the possibility of a bad actor getting
permanent execution privileges if they somehow overwrote the scripts?

>
> The tricky part is: what if the script fails? Keep it, ignore the failure and proceed with others and then boot the system? Or proceed other scripts as well and then enter some "failure state"?

Assuming you can have migrations that are interlinked, have to be run
in order, and sometimes can fail, maybe the "best" thing to do is to
simply stop on the failing one, and try to boot the system as well as
it's able to in the degraded state.  This would mean that flakey
scripts would be rerun on the next boot, and hopefully succeed, and
consistently failing scripts could be replaced on a subsequent
firmware update with more robust versions, and rerun.

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: /etc/migration.d
  2020-10-16 16:50 ` /etc/migration.d Ed Tanous
@ 2020-10-16 17:10   ` Anton Kachalov
  0 siblings, 0 replies; 5+ messages in thread
From: Anton Kachalov @ 2020-10-16 17:10 UTC (permalink / raw)
  To: Ed Tanous; +Cc: OpenBMC Maillist


[-- Attachment #1: Type: text/plain, Size: 3929 bytes --]

On Fri, 16 Oct 2020 at 18:50, Ed Tanous <ed@tanous.net> wrote:

> On Wed, Oct 14, 2020 at 11:49 AM Anton Kachalov <rnouse@google.com> wrote:
> >
> > With moving from root-only environment to unprivileged users' space, we
> need to ensure a smooth transition. To achieve that we need a mechanism for
> one-shot per-package scripts that would take care of migration. That's not
> only about groups & owners, but a general approach.
>
> Are there other use cases that necessitate a general approach?  I'm
> not against it, but owners and groups seems unique in the regard that
> the migration has to run as root.  Most (all?) other migrations don't
> seem to, or haven't in the past, and therefore can be run as a
> pre-init, or as part of the service itself.  If the service itself
> does the migration, the startup dependencies are a lot easier to track
> as a maintainer, and running your migrations in a compiled language
> likely has a positive effect on boot time, which has been a problem in
> the past (still is depending on who you ask).
>

For instance, the bmcweb has some internal logic for migration of some
files. The problem with "in-service" compiled code is that the service will
be run with least privileges and could be sandboxed, thus, wouldn't be able
to modify filesystem.

The migration scripts have to be part of the corresponding package, thus,
it will be easy to track & maintain.

The one-time run scripts wouldn't make too much overhead if they run once.

Pre-init like StartExecPre= option in the service file isn't a good choice
because it will run every time and really increase the boot time.


>
> It should be noted, several apps have done simple migrations of config
> file formats in the past, so there's some precedent for it, just not
> in a generalist solution.
>
> >  It's similar to firstboot, but has a different purpose.
> >
> > I'm going to prototype a robust / naive solution to start a service
> before everything else in the system with a condition (non-empty
> /etc/migration.d) and iterate through all files. Each script has to run at
> list with "set -e" to bail out on failures. If the script succeeded -- it
> will be removed.
>
> The script itself will be removed?  Presumably that means you're
> executing the script out of non-volatile?  That seems like a security
> gap in that an attacker could inject migration scripts that did
> anything, and have the system run them for them.  Maybe just keeping
> some kind of external log of "these scripts have completed" or,
> preferably, enforcing that migration scripts are idempotent would be
> better, and would reduce the possibility of a bad actor getting
> permanent execution privileges if they somehow overwrote the scripts?
>

Basically, we can have a SHA-sum with a list of approved scripts to run.
Such lists have to be placed on the read-only part. The way how to mark the
succeeded scripts may vary. We can have scripts available on the same
read-only partition and touch files somewhere under /var/lib for the
succeeded ones. As well as make the scripts re-entrant in case of
losing state files.


>
> >
> > The tricky part is: what if the script fails? Keep it, ignore the
> failure and proceed with others and then boot the system? Or proceed other
> scripts as well and then enter some "failure state"?
>
> Assuming you can have migrations that are interlinked, have to be run
> in order, and sometimes can fail, maybe the "best" thing to do is to
> simply stop on the failing one, and try to boot the system as well as
> it's able to in the degraded state.  This would mean that flakey
> scripts would be rerun on the next boot, and hopefully succeed, and
> consistently failing scripts could be replaced on a subsequent
> firmware update with more robust versions, and rerun.
>

Are there any read-to-use API / scripts to make the system boot in degraded
mode? Otherwise, we can just add this functionality later.

[-- Attachment #2: Type: text/html, Size: 4968 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: /etc/migration.d
  2020-10-14 18:47 /etc/migration.d Anton Kachalov
  2020-10-16 16:50 ` /etc/migration.d Ed Tanous
@ 2020-10-16 20:25 ` Patrick Williams
  2020-10-16 21:01   ` /etc/migration.d Anton Kachalov
  1 sibling, 1 reply; 5+ messages in thread
From: Patrick Williams @ 2020-10-16 20:25 UTC (permalink / raw)
  To: Anton Kachalov; +Cc: OpenBMC Maillist


[-- Attachment #1: Type: text/plain, Size: 2732 bytes --]

On Wed, Oct 14, 2020 at 08:47:57PM +0200, Anton Kachalov wrote:
> With moving from root-only environment to unprivileged users' space, we
> need to ensure a smooth transition. To achieve that we need a mechanism for
> one-shot per-package scripts that would take care of migration. That's not
> only about groups & owners, but a general approach. It's similar to
> firstboot, but has a different purpose.
> 
> I'm going to prototype a robust / naive solution to start a service before
> everything else in the system with a condition (non-empty /etc/migration.d)
> and iterate through all files. Each script has to run at list with "set -e"
> to bail out on failures. If the script succeeded -- it will be removed.
> 
> The tricky part is: what if the script fails? Keep it, ignore the failure
> and proceed with others and then boot the system? Or proceed other scripts
> as well and then enter some "failure state"?

Hi Anton,

I have some high-level questions / ideas about this.

* Would these migrations be restricted to just useradd/groupadd operations?  Or
  are you trying to create a general framework for "upgrade scripts"?

* Have you looked at any existing support by Yocto or systemd to provide
  what you need?  Yocto has USERADD_PACKAGES, postinst_intercept.
  Systemd has firstboot.  There might be other mechanisms I'm not
  remembering as well.  (I guess you mentioned firstboot).  There is
  hacky override to install a "@reboot" directive in the crontab.

* How long would a "migration" be kept around for?  Are we expecting
  that packages provide them forever?

* How do we handle downgrades?  Some systems are set up with a "golden
  image" which is locked at manufacturing.  Maybe simple
  useradd/groupadd calls are innately backwards compatible but I worry
  about a general framework falling apart.

* Is there some mechanism we should do to run the migrations as part of
  the upgrade process instead of waiting to the next boot?  The
  migrations could be included in the image tarball and thus be signed.
  That would save time on reboots for checking if the migrations are
  done.

* Rather than have a single migration script that runs before everything
  else (and is thus serial), you might create a template service
  (phosphor-migration-@.service) that can be depended on by the services
  needing the migration results.  (ie. service foo depends on
  migration-foo).

* In a follow up email you mentioned something about hashing.  I was
  going to ask how you know when a particular migration has been
  executed.  Maybe there are some tricks of recording hash values in
  the RWFS could prevent multiple executions.

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: /etc/migration.d
  2020-10-16 20:25 ` /etc/migration.d Patrick Williams
@ 2020-10-16 21:01   ` Anton Kachalov
  0 siblings, 0 replies; 5+ messages in thread
From: Anton Kachalov @ 2020-10-16 21:01 UTC (permalink / raw)
  To: Patrick Williams; +Cc: OpenBMC Maillist


[-- Attachment #1: Type: text/plain, Size: 4949 bytes --]

Hello, Patrick.

On Fri, 16 Oct 2020 at 22:25, Patrick Williams <patrick@stwcx.xyz> wrote:

> On Wed, Oct 14, 2020 at 08:47:57PM +0200, Anton Kachalov wrote:
> > With moving from root-only environment to unprivileged users' space, we
> > need to ensure a smooth transition. To achieve that we need a mechanism
> for
> > one-shot per-package scripts that would take care of migration. That's
> not
> > only about groups & owners, but a general approach. It's similar to
> > firstboot, but has a different purpose.
> >
> > I'm going to prototype a robust / naive solution to start a service
> before
> > everything else in the system with a condition (non-empty
> /etc/migration.d)
> > and iterate through all files. Each script has to run at list with "set
> -e"
> > to bail out on failures. If the script succeeded -- it will be removed.
> >
> > The tricky part is: what if the script fails? Keep it, ignore the failure
> > and proceed with others and then boot the system? Or proceed other
> scripts
> > as well and then enter some "failure state"?
>
> Hi Anton,
>
> I have some high-level questions / ideas about this.
>
> * Would these migrations be restricted to just useradd/groupadd
> operations?  Or
>   are you trying to create a general framework for "upgrade scripts"?
>

This might be a general framework.


>
> * Have you looked at any existing support by Yocto or systemd to provide
>   what you need?  Yocto has USERADD_PACKAGES, postinst_intercept.
>   Systemd has firstboot.  There might be other mechanisms I'm not
>   remembering as well.  (I guess you mentioned firstboot).  There is
>   hacky override to install a "@reboot" directive in the crontab.
>

afaik, systemd's firstboot is only about to run special units right after
installation. Once the system is configured, the firstboot units wouldn't
be executed anymore.
This thread I've started to find possible solutions.
The postinst chunks executed during the image formation (as a part of rpm /
deb packages' scripts).


>
> * How long would a "migration" be kept around for?  Are we expecting
>   that packages provide them forever?
>

That is a good question because we don't know how old the firmware is being
upgraded. I suppose, that like one-two-whatever release cycles. Then the
update process should be either using an intermediate firmware version or
forcing the non-volatile storage to be wiped. Regardless of the migration
scripts, we might have some incompatibilities between two releases that
will require NV (overlayfs back partition) cleanup.


>
> * How do we handle downgrades?  Some systems are set up with a "golden
>   image" which is locked at manufacturing.  Maybe simple
>   useradd/groupadd calls are innately backwards compatible but I worry
>   about a general framework falling apart.
>

In general, that's an issue. Golden-image downgrades should be allowed
within a compatible release branch (without wiping data). As above,
golden-images might be incompatible and wouldn't allow downgrades.

The particular migration from root-only users to unprivileged users should
be one way without wiping data. If the downgrade is requested, then it will
be required to wipe the data.


>
> * Is there some mechanism we should do to run the migrations as part of
>   the upgrade process instead of waiting to the next boot?  The
>   migrations could be included in the image tarball and thus be signed.
>   That would save time on reboots for checking if the migrations are
>   done.
>

Yes, it could be done as a set of scripts during the update process. That
is one of the possible approaches. This also could be an approach for
downgrades. I'm only worrying about the effort to support downgrades from
random version to random version. The least effort with incompatible
upgrades / downgrades is to keep special transition firmware allowing
downgrade from current Golden version to the previous Golden version from
incompatible branch. For upgrades the latest version of transition firmware
might not be golden. This will require a separate repo with an
auto-generated set of scripts to be used to build transition fws.



>
> * Rather than have a single migration script that runs before everything
>   else (and is thus serial), you might create a template service
>   (phosphor-migration-@.service) that can be depended on by the services
>   needing the migration results.  (ie. service foo depends on
>   migration-foo).
>

While migration is one-off, it might be safer to run serial one by one.


>
> * In a follow up email you mentioned something about hashing.  I was
>   going to ask how you know when a particular migration has been
>   executed.  Maybe there are some tricks of recording hash values in
>   the RWFS could prevent multiple executions.
>

We can track the succeeded scripts by touching some file in a directory
like /var/lib/migration (e.g. create a file named as sha-sum of the runned
script).


>
> --
> Patrick Williams
>

[-- Attachment #2: Type: text/html, Size: 6819 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, back to index

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-10-14 18:47 /etc/migration.d Anton Kachalov
2020-10-16 16:50 ` /etc/migration.d Ed Tanous
2020-10-16 17:10   ` /etc/migration.d Anton Kachalov
2020-10-16 20:25 ` /etc/migration.d Patrick Williams
2020-10-16 21:01   ` /etc/migration.d Anton Kachalov

Openbmc archive at lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/openbmc/0 openbmc/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 openbmc openbmc/ https://lore.kernel.org/openbmc \
		openbmc@lists.ozlabs.org openbmc@ozlabs.org
	public-inbox-index openbmc

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.ozlabs.lists.openbmc


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git