* [poky][PATCH] package_manager: install versioned postinst scripts
@ 2021-03-22 15:45 Anton Kachalov
2021-03-22 15:54 ` [OE-core] " Richard Purdie
[not found] ` <1b153bce-a66a-45ee-a5c6-963ea6fb1c82.949ef384-8293-46b8-903f-40a477c056ae.f4dbd25e-5dbe-43cd-9759-142455aef0eb@emailsignatures365.codetwo.com>
0 siblings, 2 replies; 6+ messages in thread
From: Anton Kachalov @ 2021-03-22 15:45 UTC (permalink / raw)
To: openembedded-core; +Cc: Anton D. Kachalov
From: "Anton D. Kachalov" <gmouse@google.com>
When running on read-only rootfs backed with overlayfs, the processed scripts
are being marked as deleted on upperdir of overlayfs. When it comes to
upgrade the main read-only image, it might contain new postinst scripts that
are different from the already processed. Introduced suffix (sha256 of
the content) allows to distinct updated scripts of the same package.
Signed-off-by: Anton D. Kachalov <gmouse@google.com>
---
meta/lib/oe/package_manager/deb/rootfs.py | 11 ++++++++++-
meta/lib/oe/package_manager/ipk/rootfs.py | 11 ++++++++++-
meta/lib/oe/package_manager/rpm/__init__.py | 4 +++-
3 files changed, 23 insertions(+), 3 deletions(-)
diff --git a/meta/lib/oe/package_manager/deb/rootfs.py b/meta/lib/oe/package_manager/deb/rootfs.py
index 8fbaca11d6..704d213626 100644
--- a/meta/lib/oe/package_manager/deb/rootfs.py
+++ b/meta/lib/oe/package_manager/deb/rootfs.py
@@ -4,6 +4,7 @@
import re
import shutil
+from hashlib import sha256
from oe.rootfs import Rootfs
from oe.manifest import Manifest
from oe.utils import execute_pre_post_process
@@ -115,8 +116,16 @@ class DpkgOpkgRootfs(Rootfs):
bb.utils.mkdirhier(dst_postinst_dir)
if os.path.exists(os.path.join(src_postinst_dir, p + ".postinst")):
+ csum = sha256()
+ with open(src_postinst_dir, p + ".postinst", "rb") as f:
+ while True:
+ data = f.read(65536)
+ if not data:
+ break
+ csum.update(data)
+
shutil.copy(os.path.join(src_postinst_dir, p + ".postinst"),
- os.path.join(dst_postinst_dir, "%03d-%s" % (num, p)))
+ os.path.join(dst_postinst_dir, "%03d-%s.%s" % (num, p, csum.hexdigest())))
num += 1
diff --git a/meta/lib/oe/package_manager/ipk/rootfs.py b/meta/lib/oe/package_manager/ipk/rootfs.py
index 26dbee6f6a..6ead6ddef3 100644
--- a/meta/lib/oe/package_manager/ipk/rootfs.py
+++ b/meta/lib/oe/package_manager/ipk/rootfs.py
@@ -5,6 +5,7 @@
import re
import filecmp
import shutil
+from hashlib import sha256
from oe.rootfs import Rootfs
from oe.manifest import Manifest
from oe.utils import execute_pre_post_process
@@ -116,8 +117,16 @@ class DpkgOpkgRootfs(Rootfs):
bb.utils.mkdirhier(dst_postinst_dir)
if os.path.exists(os.path.join(src_postinst_dir, p + ".postinst")):
+ csum = sha256()
+ with open(src_postinst_dir, p + ".postinst", "rb") as f:
+ while True:
+ data = f.read(65536)
+ if not data:
+ break
+ csum.update(data)
+
shutil.copy(os.path.join(src_postinst_dir, p + ".postinst"),
- os.path.join(dst_postinst_dir, "%03d-%s" % (num, p)))
+ os.path.join(dst_postinst_dir, "%03d-%s.%s" % (num, p, csum.hexdigest())))
num += 1
diff --git a/meta/lib/oe/package_manager/rpm/__init__.py b/meta/lib/oe/package_manager/rpm/__init__.py
index 6df0092281..4746e96ce8 100644
--- a/meta/lib/oe/package_manager/rpm/__init__.py
+++ b/meta/lib/oe/package_manager/rpm/__init__.py
@@ -4,6 +4,7 @@
import shutil
import subprocess
+from hashlib import sha256
from oe.package_manager import *
class RpmIndexer(Indexer):
@@ -359,7 +360,8 @@ class RpmPM(PackageManager):
target_path = oe.path.join(self.target_rootfs, self.d.expand('${sysconfdir}/rpm-postinsts/'))
bb.utils.mkdirhier(target_path)
num = self._script_num_prefix(target_path)
- saved_script_name = oe.path.join(target_path, "%d-%s" % (num, pkg))
+ csum = sha256(output.encode("utf-8")).hexdigest()
+ saved_script_name = oe.path.join(target_path, "%d-%s.%s" % (num, pkg, csum))
open(saved_script_name, 'w').write(output)
os.chmod(saved_script_name, 0o755)
--
2.31.0.rc2.261.g7f71774620-goog
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [OE-core] [poky][PATCH] package_manager: install versioned postinst scripts
2021-03-22 15:45 [poky][PATCH] package_manager: install versioned postinst scripts Anton Kachalov
@ 2021-03-22 15:54 ` Richard Purdie
[not found] ` <1b153bce-a66a-45ee-a5c6-963ea6fb1c82.949ef384-8293-46b8-903f-40a477c056ae.f4dbd25e-5dbe-43cd-9759-142455aef0eb@emailsignatures365.codetwo.com>
1 sibling, 0 replies; 6+ messages in thread
From: Richard Purdie @ 2021-03-22 15:54 UTC (permalink / raw)
To: gmouse, openembedded-core
On Mon, 2021-03-22 at 16:45 +0100, Anton Kachalov via lists.openembedded.org wrote:
> From: "Anton D. Kachalov" <gmouse@google.com>
>
> When running on read-only rootfs backed with overlayfs, the processed scripts
> are being marked as deleted on upperdir of overlayfs. When it comes to
> upgrade the main read-only image, it might contain new postinst scripts that
> are different from the already processed. Introduced suffix (sha256 of
> the content) allows to distinct updated scripts of the same package.
>
> Signed-off-by: Anton D. Kachalov <gmouse@google.com>
> ---
> meta/lib/oe/package_manager/deb/rootfs.py | 11 ++++++++++-
> meta/lib/oe/package_manager/ipk/rootfs.py | 11 ++++++++++-
> meta/lib/oe/package_manager/rpm/__init__.py | 4 +++-
> 3 files changed, 23 insertions(+), 3 deletions(-)
Isn't this working around an issue with overlayfs?
I'm a bit worried we're adding complexity to all users of the project
for a very specific use case where the issue is really somewhere else?
Cheers,
Richard
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [OE-core] [poky][PATCH] package_manager: install versioned postinst scripts
[not found] ` <1b153bce-a66a-45ee-a5c6-963ea6fb1c82.0d2bd5fa-15cc-4b27-b94e-83614f9e5b38.31ab2b89-4cad-4ccb-bc6c-988623ea5223@emailsignatures365.codetwo.com>
@ 2021-03-22 16:08 ` Mike Looijmans
2021-03-22 16:54 ` Anton Kachalov
0 siblings, 1 reply; 6+ messages in thread
From: Mike Looijmans @ 2021-03-22 16:08 UTC (permalink / raw)
To: gmouse, openembedded-core
Two remarks below...
Met vriendelijke groet / kind regards,
Mike Looijmans
System Expert
TOPIC Embedded Products B.V.
Materiaalweg 4, 5681 RJ Best
The Netherlands
T: +31 (0) 499 33 69 69
E: mike.looijmans@topicproducts.com
W: www.topic.nl
Please consider the environment before printing this e-mail
On 22-03-2021 16:45, Anton Kachalov via lists.openembedded.org wrote:
> From: "Anton D. Kachalov" <gmouse@google.com>
>
> When running on read-only rootfs backed with overlayfs, the processed scripts
> are being marked as deleted on upperdir of overlayfs. When it comes to
> upgrade the main read-only image, it might contain new postinst scripts that
> are different from the already processed. Introduced suffix (sha256 of
> the content) allows to distinct updated scripts of the same package.
So your usecase is that you have a read-only rootfs with an overlay on
top, and then just replace the rootfs. And then expect things to still
work?
That doesn't sound valid to me - normally I'd wipe the overlay clean
when updating the underlying read-only part. I'd be very interested in
what you're actually trying to accomplish here?
And further, I don't think appending 64 characters to filenames is going
to make anyone happy.
In your case, I'd opt for a script that just resets the postinst scripts
(and maybe more?) in the overlay. Or have 3 overlays. Whenever the
package manager runs, activate the second layer. The "user" stuff in the
third layer is normally active. Then when you upgrade the system, wipe
the second layer, to get rid of obsolete packages. Or something like
that. Again, tell us what you're really building...
>
> Signed-off-by: Anton D. Kachalov <gmouse@google.com>
> ---
> meta/lib/oe/package_manager/deb/rootfs.py | 11 ++++++++++-
> meta/lib/oe/package_manager/ipk/rootfs.py | 11 ++++++++++-
> meta/lib/oe/package_manager/rpm/__init__.py | 4 +++-
> 3 files changed, 23 insertions(+), 3 deletions(-)
>
> diff --git a/meta/lib/oe/package_manager/deb/rootfs.py b/meta/lib/oe/package_manager/deb/rootfs.py
> index 8fbaca11d6..704d213626 100644
> --- a/meta/lib/oe/package_manager/deb/rootfs.py
> +++ b/meta/lib/oe/package_manager/deb/rootfs.py
> @@ -4,6 +4,7 @@
>
> import re
> import shutil
> +from hashlib import sha256
> from oe.rootfs import Rootfs
> from oe.manifest import Manifest
> from oe.utils import execute_pre_post_process
> @@ -115,8 +116,16 @@ class DpkgOpkgRootfs(Rootfs):
> bb.utils.mkdirhier(dst_postinst_dir)
>
> if os.path.exists(os.path.join(src_postinst_dir, p + ".postinst")):
> + csum = sha256()
> + with open(src_postinst_dir, p + ".postinst", "rb") as f:
> + while True:
> + data = f.read(65536)
> + if not data:
> + break
> + csum.update(data)
> +
> shutil.copy(os.path.join(src_postinst_dir, p + ".postinst"),
> - os.path.join(dst_postinst_dir, "%03d-%s" % (num, p)))
> + os.path.join(dst_postinst_dir, "%03d-%s.%s" % (num, p, csum.hexdigest())))
>
> num += 1
>
> diff --git a/meta/lib/oe/package_manager/ipk/rootfs.py b/meta/lib/oe/package_manager/ipk/rootfs.py
> index 26dbee6f6a..6ead6ddef3 100644
> --- a/meta/lib/oe/package_manager/ipk/rootfs.py
> +++ b/meta/lib/oe/package_manager/ipk/rootfs.py
> @@ -5,6 +5,7 @@
> import re
> import filecmp
> import shutil
> +from hashlib import sha256
> from oe.rootfs import Rootfs
> from oe.manifest import Manifest
> from oe.utils import execute_pre_post_process
> @@ -116,8 +117,16 @@ class DpkgOpkgRootfs(Rootfs):
> bb.utils.mkdirhier(dst_postinst_dir)
>
> if os.path.exists(os.path.join(src_postinst_dir, p + ".postinst")):
> + csum = sha256()
> + with open(src_postinst_dir, p + ".postinst", "rb") as f:
> + while True:
> + data = f.read(65536)
> + if not data:
> + break
> + csum.update(data)
> +
> shutil.copy(os.path.join(src_postinst_dir, p + ".postinst"),
> - os.path.join(dst_postinst_dir, "%03d-%s" % (num, p)))
> + os.path.join(dst_postinst_dir, "%03d-%s.%s" % (num, p, csum.hexdigest())))
>
> num += 1
>
> diff --git a/meta/lib/oe/package_manager/rpm/__init__.py b/meta/lib/oe/package_manager/rpm/__init__.py
> index 6df0092281..4746e96ce8 100644
> --- a/meta/lib/oe/package_manager/rpm/__init__.py
> +++ b/meta/lib/oe/package_manager/rpm/__init__.py
> @@ -4,6 +4,7 @@
>
> import shutil
> import subprocess
> +from hashlib import sha256
> from oe.package_manager import *
>
> class RpmIndexer(Indexer):
> @@ -359,7 +360,8 @@ class RpmPM(PackageManager):
> target_path = oe.path.join(self.target_rootfs, self.d.expand('${sysconfdir}/rpm-postinsts/'))
> bb.utils.mkdirhier(target_path)
> num = self._script_num_prefix(target_path)
> - saved_script_name = oe.path.join(target_path, "%d-%s" % (num, pkg))
> + csum = sha256(output.encode("utf-8")).hexdigest()
> + saved_script_name = oe.path.join(target_path, "%d-%s.%s" % (num, pkg, csum))
> open(saved_script_name, 'w').write(output)
> os.chmod(saved_script_name, 0o755)
>
>
>
>
--
Mike Looijmans
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [OE-core] [poky][PATCH] package_manager: install versioned postinst scripts
2021-03-22 16:08 ` Mike Looijmans
@ 2021-03-22 16:54 ` Anton Kachalov
2021-03-23 6:56 ` Mike Looijmans
0 siblings, 1 reply; 6+ messages in thread
From: Anton Kachalov @ 2021-03-22 16:54 UTC (permalink / raw)
To: Mike Looijmans; +Cc: openembedded-core
[-- Attachment #1: Type: text/plain, Size: 6709 bytes --]
Hi, Mike.
That is how OpenBMC is designed. Most of the targets are read-only rootfs
(squashfs) and overlayfs for RW.
It is not expected to overwrite binaries through overlayfs and there are
plans to scope RW areas (e.g. /etc).
Those files with 64-chars suffix are mostly hidden from regular users: the
files are removed during the first startup. No one will notice them :)
Anyway, alternatively within the openbmc update script it's possible to
force clean the deleted files on overlayfs upperdir. However, it will mean
that all the scripts regardless if they are updated or not, will be
re-executed after rootfs update.
On Mon, 22 Mar 2021 at 17:08, Mike Looijmans <mike.looijmans@topic.nl>
wrote:
> Two remarks below...
>
>
> Met vriendelijke groet / kind regards,
>
> Mike Looijmans
> System Expert
>
>
> TOPIC Embedded Products B.V.
> Materiaalweg 4, 5681 RJ Best
> The Netherlands
>
> T: +31 (0) 499 33 69 69 <+31%20499%20336%20969>
> E: mike.looijmans@topicproducts.com
> W: www.topic.nl
>
> Please consider the environment before printing this e-mail
> On 22-03-2021 16:45, Anton Kachalov via lists.openembedded.org wrote:
> > From: "Anton D. Kachalov" <gmouse@google.com>
> >
> > When running on read-only rootfs backed with overlayfs, the processed
> scripts
> > are being marked as deleted on upperdir of overlayfs. When it comes to
> > upgrade the main read-only image, it might contain new postinst scripts
> that
> > are different from the already processed. Introduced suffix (sha256 of
> > the content) allows to distinct updated scripts of the same package.
>
> So your usecase is that you have a read-only rootfs with an overlay on
> top, and then just replace the rootfs. And then expect things to still
> work?
>
> That doesn't sound valid to me - normally I'd wipe the overlay clean
> when updating the underlying read-only part. I'd be very interested in
> what you're actually trying to accomplish here?
>
> And further, I don't think appending 64 characters to filenames is going
> to make anyone happy.
>
> In your case, I'd opt for a script that just resets the postinst scripts
> (and maybe more?) in the overlay. Or have 3 overlays. Whenever the
> package manager runs, activate the second layer. The "user" stuff in the
> third layer is normally active. Then when you upgrade the system, wipe
> the second layer, to get rid of obsolete packages. Or something like
> that. Again, tell us what you're really building...
>
>
> >
> > Signed-off-by: Anton D. Kachalov <gmouse@google.com>
> > ---
> > meta/lib/oe/package_manager/deb/rootfs.py | 11 ++++++++++-
> > meta/lib/oe/package_manager/ipk/rootfs.py | 11 ++++++++++-
> > meta/lib/oe/package_manager/rpm/__init__.py | 4 +++-
> > 3 files changed, 23 insertions(+), 3 deletions(-)
> >
> > diff --git a/meta/lib/oe/package_manager/deb/rootfs.py
> b/meta/lib/oe/package_manager/deb/rootfs.py
> > index 8fbaca11d6..704d213626 100644
> > --- a/meta/lib/oe/package_manager/deb/rootfs.py
> > +++ b/meta/lib/oe/package_manager/deb/rootfs.py
> > @@ -4,6 +4,7 @@
> >
> > import re
> > import shutil
> > +from hashlib import sha256
> > from oe.rootfs import Rootfs
> > from oe.manifest import Manifest
> > from oe.utils import execute_pre_post_process
> > @@ -115,8 +116,16 @@ class DpkgOpkgRootfs(Rootfs):
> > bb.utils.mkdirhier(dst_postinst_dir)
> >
> > if os.path.exists(os.path.join(src_postinst_dir, p +
> ".postinst")):
> > + csum = sha256()
> > + with open(src_postinst_dir, p + ".postinst", "rb") as f:
> > + while True:
> > + data = f.read(65536)
> > + if not data:
> > + break
> > + csum.update(data)
> > +
> > shutil.copy(os.path.join(src_postinst_dir, p +
> ".postinst"),
> > - os.path.join(dst_postinst_dir, "%03d-%s" %
> (num, p)))
> > + os.path.join(dst_postinst_dir, "%03d-%s.%s"
> % (num, p, csum.hexdigest())))
> >
> > num += 1
> >
> > diff --git a/meta/lib/oe/package_manager/ipk/rootfs.py
> b/meta/lib/oe/package_manager/ipk/rootfs.py
> > index 26dbee6f6a..6ead6ddef3 100644
> > --- a/meta/lib/oe/package_manager/ipk/rootfs.py
> > +++ b/meta/lib/oe/package_manager/ipk/rootfs.py
> > @@ -5,6 +5,7 @@
> > import re
> > import filecmp
> > import shutil
> > +from hashlib import sha256
> > from oe.rootfs import Rootfs
> > from oe.manifest import Manifest
> > from oe.utils import execute_pre_post_process
> > @@ -116,8 +117,16 @@ class DpkgOpkgRootfs(Rootfs):
> > bb.utils.mkdirhier(dst_postinst_dir)
> >
> > if os.path.exists(os.path.join(src_postinst_dir, p +
> ".postinst")):
> > + csum = sha256()
> > + with open(src_postinst_dir, p + ".postinst", "rb") as f:
> > + while True:
> > + data = f.read(65536)
> > + if not data:
> > + break
> > + csum.update(data)
> > +
> > shutil.copy(os.path.join(src_postinst_dir, p +
> ".postinst"),
> > - os.path.join(dst_postinst_dir, "%03d-%s" %
> (num, p)))
> > + os.path.join(dst_postinst_dir, "%03d-%s.%s"
> % (num, p, csum.hexdigest())))
> >
> > num += 1
> >
> > diff --git a/meta/lib/oe/package_manager/rpm/__init__.py
> b/meta/lib/oe/package_manager/rpm/__init__.py
> > index 6df0092281..4746e96ce8 100644
> > --- a/meta/lib/oe/package_manager/rpm/__init__.py
> > +++ b/meta/lib/oe/package_manager/rpm/__init__.py
> > @@ -4,6 +4,7 @@
> >
> > import shutil
> > import subprocess
> > +from hashlib import sha256
> > from oe.package_manager import *
> >
> > class RpmIndexer(Indexer):
> > @@ -359,7 +360,8 @@ class RpmPM(PackageManager):
> > target_path = oe.path.join(self.target_rootfs,
> self.d.expand('${sysconfdir}/rpm-postinsts/'))
> > bb.utils.mkdirhier(target_path)
> > num = self._script_num_prefix(target_path)
> > - saved_script_name = oe.path.join(target_path, "%d-%s" % (num,
> pkg))
> > + csum = sha256(output.encode("utf-8")).hexdigest()
> > + saved_script_name = oe.path.join(target_path, "%d-%s.%s" %
> (num, pkg, csum))
> > open(saved_script_name, 'w').write(output)
> > os.chmod(saved_script_name, 0o755)
> >
> >
> >
> >
>
> --
> Mike Looijmans
>
>
[-- Attachment #2: Type: text/html, Size: 8635 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [OE-core] [poky][PATCH] package_manager: install versioned postinst scripts
2021-03-22 16:54 ` Anton Kachalov
@ 2021-03-23 6:56 ` Mike Looijmans
2021-03-23 10:48 ` Anton Kachalov
0 siblings, 1 reply; 6+ messages in thread
From: Mike Looijmans @ 2021-03-23 6:56 UTC (permalink / raw)
To: Anton Kachalov; +Cc: openembedded-core
So it's something exclusively for OpenBMC.
As for running the scripts, you cannot simply conclude that a script
doesn't need to run if its contents did not change. It may very well
call the executable from its package to do the actual work and that may
have changed as well.
Met vriendelijke groet / kind regards,
Mike Looijmans
System Expert
TOPIC Embedded Products B.V.
Materiaalweg 4, 5681 RJ Best
The Netherlands
T: +31 (0) 499 33 69 69
E: mike.looijmans@topicproducts.com
W: www.topic.nl
Please consider the environment before printing this e-mail
On 22-03-2021 17:54, Anton Kachalov wrote:
> Hi, Mike.
>
> That is how OpenBMC is designed. Most of the targets are read-only
> rootfs (squashfs) and overlayfs for RW.
> It is not expected to overwrite binaries through overlayfs and there
> are plans to scope RW areas (e.g. /etc).
>
> Those files with 64-chars suffix are mostly hidden from regular users:
> the files are removed during the first startup. No one will notice them :)
>
> Anyway, alternatively within the openbmc update script it's possible
> to force clean the deleted files on overlayfs upperdir. However, it
> will mean that all the scripts regardless if they are updated or not,
> will be re-executed after rootfs update.
>
> On Mon, 22 Mar 2021 at 17:08, Mike Looijmans <mike.looijmans@topic.nl
> <mailto:mike.looijmans@topic.nl>> wrote:
>
> Two remarks below...
> On 22-03-2021 16:45, Anton Kachalov via lists.openembedded.org
> <http://lists.openembedded.org> wrote:
> > From: "Anton D. Kachalov" <gmouse@google.com
> <mailto:gmouse@google.com>>
> >
> > When running on read-only rootfs backed with overlayfs, the
> processed scripts
> > are being marked as deleted on upperdir of overlayfs. When it
> comes to
> > upgrade the main read-only image, it might contain new postinst
> scripts that
> > are different from the already processed. Introduced suffix
> (sha256 of
> > the content) allows to distinct updated scripts of the same package.
>
> So your usecase is that you have a read-only rootfs with an
> overlay on
> top, and then just replace the rootfs. And then expect things to
> still
> work?
>
> That doesn't sound valid to me - normally I'd wipe the overlay clean
> when updating the underlying read-only part. I'd be very
> interested in
> what you're actually trying to accomplish here?
>
> And further, I don't think appending 64 characters to filenames is
> going
> to make anyone happy.
>
> In your case, I'd opt for a script that just resets the postinst
> scripts
> (and maybe more?) in the overlay. Or have 3 overlays. Whenever the
> package manager runs, activate the second layer. The "user" stuff
> in the
> third layer is normally active. Then when you upgrade the system,
> wipe
> the second layer, to get rid of obsolete packages. Or something like
> that. Again, tell us what you're really building...
>
>
> >
> > Signed-off-by: Anton D. Kachalov <gmouse@google.com
> <mailto:gmouse@google.com>>
> > ---
> > meta/lib/oe/package_manager/deb/rootfs.py | 11 ++++++++++-
> > meta/lib/oe/package_manager/ipk/rootfs.py | 11 ++++++++++-
> > meta/lib/oe/package_manager/rpm/__init__.py | 4 +++-
> > 3 files changed, 23 insertions(+), 3 deletions(-)
> >
> > diff --git a/meta/lib/oe/package_manager/deb/rootfs.py
> b/meta/lib/oe/package_manager/deb/rootfs.py
> > index 8fbaca11d6..704d213626 100644
> > --- a/meta/lib/oe/package_manager/deb/rootfs.py
> > +++ b/meta/lib/oe/package_manager/deb/rootfs.py
> > @@ -4,6 +4,7 @@
> >
> > import re
> > import shutil
> > +from hashlib import sha256
> > from oe.rootfs import Rootfs
> > from oe.manifest import Manifest
> > from oe.utils import execute_pre_post_process
> > @@ -115,8 +116,16 @@ class DpkgOpkgRootfs(Rootfs):
> > bb.utils.mkdirhier(dst_postinst_dir)
> >
> > if os.path.exists(os.path.join(src_postinst_dir, p
> + ".postinst")):
> > + csum = sha256()
> > + with open(src_postinst_dir, p + ".postinst",
> "rb") as f:
> > + while True:
> > + data = f.read(65536)
> > + if not data:
> > + break
> > + csum.update(data)
> > +
> > shutil.copy(os.path.join(src_postinst_dir, p + ".postinst"),
> > - os.path.join(dst_postinst_dir, "%03d-%s" % (num, p)))
> > + os.path.join(dst_postinst_dir, "%03d-%s.%s" % (num, p,
> csum.hexdigest())))
> >
> > num += 1
> >
> > diff --git a/meta/lib/oe/package_manager/ipk/rootfs.py
> b/meta/lib/oe/package_manager/ipk/rootfs.py
> > index 26dbee6f6a..6ead6ddef3 100644
> > --- a/meta/lib/oe/package_manager/ipk/rootfs.py
> > +++ b/meta/lib/oe/package_manager/ipk/rootfs.py
> > @@ -5,6 +5,7 @@
> > import re
> > import filecmp
> > import shutil
> > +from hashlib import sha256
> > from oe.rootfs import Rootfs
> > from oe.manifest import Manifest
> > from oe.utils import execute_pre_post_process
> > @@ -116,8 +117,16 @@ class DpkgOpkgRootfs(Rootfs):
> > bb.utils.mkdirhier(dst_postinst_dir)
> >
> > if os.path.exists(os.path.join(src_postinst_dir, p
> + ".postinst")):
> > + csum = sha256()
> > + with open(src_postinst_dir, p + ".postinst",
> "rb") as f:
> > + while True:
> > + data = f.read(65536)
> > + if not data:
> > + break
> > + csum.update(data)
> > +
> > shutil.copy(os.path.join(src_postinst_dir, p + ".postinst"),
> > - os.path.join(dst_postinst_dir, "%03d-%s" % (num, p)))
> > + os.path.join(dst_postinst_dir, "%03d-%s.%s" % (num, p,
> csum.hexdigest())))
> >
> > num += 1
> >
> > diff --git a/meta/lib/oe/package_manager/rpm/__init__.py
> b/meta/lib/oe/package_manager/rpm/__init__.py
> > index 6df0092281..4746e96ce8 100644
> > --- a/meta/lib/oe/package_manager/rpm/__init__.py
> > +++ b/meta/lib/oe/package_manager/rpm/__init__.py
> > @@ -4,6 +4,7 @@
> >
> > import shutil
> > import subprocess
> > +from hashlib import sha256
> > from oe.package_manager import *
> >
> > class RpmIndexer(Indexer):
> > @@ -359,7 +360,8 @@ class RpmPM(PackageManager):
> > target_path = oe.path.join(self.target_rootfs,
> self.d.expand('${sysconfdir}/rpm-postinsts/'))
> > bb.utils.mkdirhier(target_path)
> > num = self._script_num_prefix(target_path)
> > - saved_script_name = oe.path.join(target_path, "%d-%s" %
> (num, pkg))
> > + csum = sha256(output.encode("utf-8")).hexdigest()
> > + saved_script_name = oe.path.join(target_path,
> "%d-%s.%s" % (num, pkg, csum))
> > open(saved_script_name, 'w').write(output)
> > os.chmod(saved_script_name, 0o755)
> >
> >
> >
> >
>
> --
> Mike Looijmans
>
--
Mike Looijmans
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [OE-core] [poky][PATCH] package_manager: install versioned postinst scripts
2021-03-23 6:56 ` Mike Looijmans
@ 2021-03-23 10:48 ` Anton Kachalov
0 siblings, 0 replies; 6+ messages in thread
From: Anton Kachalov @ 2021-03-23 10:48 UTC (permalink / raw)
To: Mike Looijmans; +Cc: openembedded-core
[-- Attachment #1: Type: text/plain, Size: 8248 bytes --]
Yes, this use case makes sense. Then the reliable way is to cleanup
upperdir of overlayfs during image upgrade and make postinsts scripts
re-entrant.
On Tue, Mar 23, 2021, 07:56 Mike Looijmans <mike.looijmans@topic.nl> wrote:
> So it's something exclusively for OpenBMC.
>
> As for running the scripts, you cannot simply conclude that a script
> doesn't need to run if its contents did not change. It may very well
> call the executable from its package to do the actual work and that may
> have changed as well.
>
>
> Met vriendelijke groet / kind regards,
>
> Mike Looijmans
> System Expert
>
>
> TOPIC Embedded Products B.V.
> Materiaalweg 4, 5681 RJ Best
> The Netherlands
>
> T: +31 (0) 499 33 69 69
> E: mike.looijmans@topicproducts.com
> W: www.topic.nl
>
> Please consider the environment before printing this e-mail
> On 22-03-2021 17:54, Anton Kachalov wrote:
> > Hi, Mike.
> >
> > That is how OpenBMC is designed. Most of the targets are read-only
> > rootfs (squashfs) and overlayfs for RW.
> > It is not expected to overwrite binaries through overlayfs and there
> > are plans to scope RW areas (e.g. /etc).
> >
> > Those files with 64-chars suffix are mostly hidden from regular users:
> > the files are removed during the first startup. No one will notice them
> :)
> >
> > Anyway, alternatively within the openbmc update script it's possible
> > to force clean the deleted files on overlayfs upperdir. However, it
> > will mean that all the scripts regardless if they are updated or not,
> > will be re-executed after rootfs update.
> >
> > On Mon, 22 Mar 2021 at 17:08, Mike Looijmans <mike.looijmans@topic.nl
> > <mailto:mike.looijmans@topic.nl>> wrote:
> >
> > Two remarks below...
> > On 22-03-2021 16:45, Anton Kachalov via lists.openembedded.org
> > <http://lists.openembedded.org> wrote:
> > > From: "Anton D. Kachalov" <gmouse@google.com
> > <mailto:gmouse@google.com>>
> > >
> > > When running on read-only rootfs backed with overlayfs, the
> > processed scripts
> > > are being marked as deleted on upperdir of overlayfs. When it
> > comes to
> > > upgrade the main read-only image, it might contain new postinst
> > scripts that
> > > are different from the already processed. Introduced suffix
> > (sha256 of
> > > the content) allows to distinct updated scripts of the same
> package.
> >
> > So your usecase is that you have a read-only rootfs with an
> > overlay on
> > top, and then just replace the rootfs. And then expect things to
> > still
> > work?
> >
> > That doesn't sound valid to me - normally I'd wipe the overlay clean
> > when updating the underlying read-only part. I'd be very
> > interested in
> > what you're actually trying to accomplish here?
> >
> > And further, I don't think appending 64 characters to filenames is
> > going
> > to make anyone happy.
> >
> > In your case, I'd opt for a script that just resets the postinst
> > scripts
> > (and maybe more?) in the overlay. Or have 3 overlays. Whenever the
> > package manager runs, activate the second layer. The "user" stuff
> > in the
> > third layer is normally active. Then when you upgrade the system,
> > wipe
> > the second layer, to get rid of obsolete packages. Or something like
> > that. Again, tell us what you're really building...
> >
> >
> > >
> > > Signed-off-by: Anton D. Kachalov <gmouse@google.com
> > <mailto:gmouse@google.com>>
> > > ---
> > > meta/lib/oe/package_manager/deb/rootfs.py | 11 ++++++++++-
> > > meta/lib/oe/package_manager/ipk/rootfs.py | 11 ++++++++++-
> > > meta/lib/oe/package_manager/rpm/__init__.py | 4 +++-
> > > 3 files changed, 23 insertions(+), 3 deletions(-)
> > >
> > > diff --git a/meta/lib/oe/package_manager/deb/rootfs.py
> > b/meta/lib/oe/package_manager/deb/rootfs.py
> > > index 8fbaca11d6..704d213626 100644
> > > --- a/meta/lib/oe/package_manager/deb/rootfs.py
> > > +++ b/meta/lib/oe/package_manager/deb/rootfs.py
> > > @@ -4,6 +4,7 @@
> > >
> > > import re
> > > import shutil
> > > +from hashlib import sha256
> > > from oe.rootfs import Rootfs
> > > from oe.manifest import Manifest
> > > from oe.utils import execute_pre_post_process
> > > @@ -115,8 +116,16 @@ class DpkgOpkgRootfs(Rootfs):
> > > bb.utils.mkdirhier(dst_postinst_dir)
> > >
> > > if os.path.exists(os.path.join(src_postinst_dir, p
> > + ".postinst")):
> > > + csum = sha256()
> > > + with open(src_postinst_dir, p + ".postinst",
> > "rb") as f:
> > > + while True:
> > > + data = f.read(65536)
> > > + if not data:
> > > + break
> > > + csum.update(data)
> > > +
> > > shutil.copy(os.path.join(src_postinst_dir, p + ".postinst"),
> > > - os.path.join(dst_postinst_dir, "%03d-%s" % (num, p)))
> > > + os.path.join(dst_postinst_dir, "%03d-%s.%s" % (num, p,
> > csum.hexdigest())))
> > >
> > > num += 1
> > >
> > > diff --git a/meta/lib/oe/package_manager/ipk/rootfs.py
> > b/meta/lib/oe/package_manager/ipk/rootfs.py
> > > index 26dbee6f6a..6ead6ddef3 100644
> > > --- a/meta/lib/oe/package_manager/ipk/rootfs.py
> > > +++ b/meta/lib/oe/package_manager/ipk/rootfs.py
> > > @@ -5,6 +5,7 @@
> > > import re
> > > import filecmp
> > > import shutil
> > > +from hashlib import sha256
> > > from oe.rootfs import Rootfs
> > > from oe.manifest import Manifest
> > > from oe.utils import execute_pre_post_process
> > > @@ -116,8 +117,16 @@ class DpkgOpkgRootfs(Rootfs):
> > > bb.utils.mkdirhier(dst_postinst_dir)
> > >
> > > if os.path.exists(os.path.join(src_postinst_dir, p
> > + ".postinst")):
> > > + csum = sha256()
> > > + with open(src_postinst_dir, p + ".postinst",
> > "rb") as f:
> > > + while True:
> > > + data = f.read(65536)
> > > + if not data:
> > > + break
> > > + csum.update(data)
> > > +
> > > shutil.copy(os.path.join(src_postinst_dir, p + ".postinst"),
> > > - os.path.join(dst_postinst_dir, "%03d-%s" % (num, p)))
> > > + os.path.join(dst_postinst_dir, "%03d-%s.%s" % (num, p,
> > csum.hexdigest())))
> > >
> > > num += 1
> > >
> > > diff --git a/meta/lib/oe/package_manager/rpm/__init__.py
> > b/meta/lib/oe/package_manager/rpm/__init__.py
> > > index 6df0092281..4746e96ce8 100644
> > > --- a/meta/lib/oe/package_manager/rpm/__init__.py
> > > +++ b/meta/lib/oe/package_manager/rpm/__init__.py
> > > @@ -4,6 +4,7 @@
> > >
> > > import shutil
> > > import subprocess
> > > +from hashlib import sha256
> > > from oe.package_manager import *
> > >
> > > class RpmIndexer(Indexer):
> > > @@ -359,7 +360,8 @@ class RpmPM(PackageManager):
> > > target_path = oe.path.join(self.target_rootfs,
> > self.d.expand('${sysconfdir}/rpm-postinsts/'))
> > > bb.utils.mkdirhier(target_path)
> > > num = self._script_num_prefix(target_path)
> > > - saved_script_name = oe.path.join(target_path, "%d-%s" %
> > (num, pkg))
> > > + csum = sha256(output.encode("utf-8")).hexdigest()
> > > + saved_script_name = oe.path.join(target_path,
> > "%d-%s.%s" % (num, pkg, csum))
> > > open(saved_script_name, 'w').write(output)
> > > os.chmod(saved_script_name, 0o755)
> > >
> > >
> > >
> > >
> >
> > --
> > Mike Looijmans
> >
>
> --
> Mike Looijmans
>
>
[-- Attachment #2: Type: text/html, Size: 11511 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-03-23 10:49 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-22 15:45 [poky][PATCH] package_manager: install versioned postinst scripts Anton Kachalov
2021-03-22 15:54 ` [OE-core] " Richard Purdie
[not found] ` <1b153bce-a66a-45ee-a5c6-963ea6fb1c82.949ef384-8293-46b8-903f-40a477c056ae.f4dbd25e-5dbe-43cd-9759-142455aef0eb@emailsignatures365.codetwo.com>
[not found] ` <1b153bce-a66a-45ee-a5c6-963ea6fb1c82.0d2bd5fa-15cc-4b27-b94e-83614f9e5b38.31ab2b89-4cad-4ccb-bc6c-988623ea5223@emailsignatures365.codetwo.com>
2021-03-22 16:08 ` Mike Looijmans
2021-03-22 16:54 ` Anton Kachalov
2021-03-23 6:56 ` Mike Looijmans
2021-03-23 10:48 ` Anton Kachalov
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.