All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH 1/2] staging: Add extra hash handling code
@ 2022-02-03 11:25 Richard Purdie
  2022-02-03 11:25 ` [PATCH 2/2] abi_version/sstate: Bump for hash equivalence fix Richard Purdie
  2022-02-03 11:31 ` [OE-core] [PATCH 1/2] staging: Add extra hash handling code Matt Madison
  0 siblings, 2 replies; 4+ messages in thread
From: Richard Purdie @ 2022-02-03 11:25 UTC (permalink / raw)
  To: openembedded-core

Target build output, stored in do_populate_sysroot or do_package can depend
not only upon direct dependencies but also indirect ones. A good example is
linux-libc-headers. The toolchain depends on this but most target recipes do
not. There are some headers which are not used by the toolchain build and do
not change the toolchain task output, hence the task hashes can change without
changing the sysroot output of that recipe yet they can influence others.

A specific example is rtc.h which can change rtcwake.c in util-linux but is not
used in the glibc or gcc build. To account for this, we need to account for the
populate_sysroot hashes in the task output hashes.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/sstate.bbclass  |  2 +-
 meta/classes/staging.bbclass | 35 ++++++++++++++++++++++++++++++++++-
 2 files changed, 35 insertions(+), 2 deletions(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 17dcf4cc175..fd9acbab9b4 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -94,7 +94,7 @@ SSTATE_ARCHS[vardepsexclude] = "ORIGNATIVELSBSTRING"
 
 SSTATE_MANMACH ?= "${SSTATE_PKGARCH}"
 
-SSTATECREATEFUNCS = "sstate_hardcode_path"
+SSTATECREATEFUNCS += "sstate_hardcode_path"
 SSTATECREATEFUNCS[vardeps] = "SSTATE_SCAN_FILES"
 SSTATEPOSTCREATEFUNCS = ""
 SSTATEPREINSTFUNCS = ""
diff --git a/meta/classes/staging.bbclass b/meta/classes/staging.bbclass
index 25f77c77350..3ea9c974eb5 100644
--- a/meta/classes/staging.bbclass
+++ b/meta/classes/staging.bbclass
@@ -21,7 +21,7 @@ SYSROOT_DIRS_NATIVE = " \
 "
 SYSROOT_DIRS:append:class-native = " ${SYSROOT_DIRS_NATIVE}"
 SYSROOT_DIRS:append:class-cross = " ${SYSROOT_DIRS_NATIVE}"
-SYSROOT_DIRS:append:class-crosssdk = " ${SYSROOT_DIRS_NATIVE}"
+SYSROOT_DIRS:app:class-crosssdk = " ${SYSROOT_DIRS_NATIVE}"
 
 # These directories will not be staged in the sysroot
 SYSROOT_DIRS_BLACKLIST = " \
@@ -624,3 +624,36 @@ python staging_taskhandler() {
 }
 staging_taskhandler[eventmask] = "bb.event.RecipeTaskPreProcess"
 addhandler staging_taskhandler
+
+
+#
+# Target build output, stored in do_populate_sysroot or do_package can depend
+# not only upon direct dependencies but also indirect ones. A good example is
+# linux-libc-headers. The toolchain depends on this but most target recipes do
+# not. There are some headers which are not used by the toolchain build and do
+# not change the toolchain task output, hence the task hashes can change without
+# changing the sysroot output of that recipe yet they can influence others.
+#
+# A specific example is rtc.h which can change rtcwake.c in util-linux but is not
+# used in the glibc or gcc build. To account for this, we need to account for the
+# populate_sysroot hashes in the task output hashes.
+#
+python target_add_sysroot_deps () {
+    current_task = "do_" + d.getVar("BB_CURRENTTASK")
+    if current_task not in ["do_populate_sysroot", "do_package"]:
+        return
+
+    pn = d.getVar("PN")
+    if pn.endswith("-native"):
+        return
+
+    taskdepdata = d.getVar("BB_TASKDEPDATA", False)
+    deps = {}
+    for dep in taskdepdata.values():
+        if dep[1] == "do_populate_sysroot" and not dep[0].endswith(("-native", "-initial")) and "-cross-" not in dep[0]:
+            deps[dep[0]] = dep[6]
+
+    d.setVar("HASHEQUIV_EXTRA_SIGDATA", "\n".join("%s: %s" % (k, deps[k]) for k in sorted(deps.keys())))
+}
+SSTATECREATEFUNCS += "target_add_sysroot_deps"
+
-- 
2.32.0



^ permalink raw reply related	[flat|nested] 4+ messages in thread

* [PATCH 2/2] abi_version/sstate: Bump for hash equivalence fix
  2022-02-03 11:25 [PATCH 1/2] staging: Add extra hash handling code Richard Purdie
@ 2022-02-03 11:25 ` Richard Purdie
  2022-02-03 11:31 ` [OE-core] [PATCH 1/2] staging: Add extra hash handling code Matt Madison
  1 sibling, 0 replies; 4+ messages in thread
From: Richard Purdie @ 2022-02-03 11:25 UTC (permalink / raw)
  To: openembedded-core

With the hash equivalence fix, we need to bump the sstate and hash equivalence
version numbers to ensure older task hashes aren't matched into the new namespace.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/sstate.bbclass | 2 +-
 meta/conf/abi_version.conf  | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index fd9acbab9b4..85d0ee2e6a5 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -1,4 +1,4 @@
-SSTATE_VERSION = "7"
+SSTATE_VERSION = "8"
 
 SSTATE_ZSTD_CLEVEL ??= "8"
 
diff --git a/meta/conf/abi_version.conf b/meta/conf/abi_version.conf
index 20ca1098ecb..66df69bb8d5 100644
--- a/meta/conf/abi_version.conf
+++ b/meta/conf/abi_version.conf
@@ -12,4 +12,4 @@ OELAYOUT_ABI = "14"
 # a reset of the equivalence, for example when reproducibility issues break the
 # existing match data. Distros can also append to this value for the same effect.
 #
-HASHEQUIV_HASH_VERSION  = "11"
+HASHEQUIV_HASH_VERSION  = "12"
-- 
2.32.0



^ permalink raw reply related	[flat|nested] 4+ messages in thread

* Re: [OE-core] [PATCH 1/2] staging: Add extra hash handling code
  2022-02-03 11:25 [PATCH 1/2] staging: Add extra hash handling code Richard Purdie
  2022-02-03 11:25 ` [PATCH 2/2] abi_version/sstate: Bump for hash equivalence fix Richard Purdie
@ 2022-02-03 11:31 ` Matt Madison
  2022-02-03 11:33   ` Richard Purdie
  1 sibling, 1 reply; 4+ messages in thread
From: Matt Madison @ 2022-02-03 11:31 UTC (permalink / raw)
  To: Richard Purdie; +Cc: Patches and discussions about the oe-core layer

On Thu, Feb 3, 2022 at 3:25 AM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> Target build output, stored in do_populate_sysroot or do_package can depend
> not only upon direct dependencies but also indirect ones. A good example is
> linux-libc-headers. The toolchain depends on this but most target recipes do
> not. There are some headers which are not used by the toolchain build and do
> not change the toolchain task output, hence the task hashes can change without
> changing the sysroot output of that recipe yet they can influence others.
>
> A specific example is rtc.h which can change rtcwake.c in util-linux but is not
> used in the glibc or gcc build. To account for this, we need to account for the
> populate_sysroot hashes in the task output hashes.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>  meta/classes/sstate.bbclass  |  2 +-
>  meta/classes/staging.bbclass | 35 ++++++++++++++++++++++++++++++++++-
>  2 files changed, 35 insertions(+), 2 deletions(-)
>
> diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
> index 17dcf4cc175..fd9acbab9b4 100644
> --- a/meta/classes/sstate.bbclass
> +++ b/meta/classes/sstate.bbclass
> @@ -94,7 +94,7 @@ SSTATE_ARCHS[vardepsexclude] = "ORIGNATIVELSBSTRING"
>
>  SSTATE_MANMACH ?= "${SSTATE_PKGARCH}"
>
> -SSTATECREATEFUNCS = "sstate_hardcode_path"
> +SSTATECREATEFUNCS += "sstate_hardcode_path"
>  SSTATECREATEFUNCS[vardeps] = "SSTATE_SCAN_FILES"
>  SSTATEPOSTCREATEFUNCS = ""
>  SSTATEPREINSTFUNCS = ""
> diff --git a/meta/classes/staging.bbclass b/meta/classes/staging.bbclass
> index 25f77c77350..3ea9c974eb5 100644
> --- a/meta/classes/staging.bbclass
> +++ b/meta/classes/staging.bbclass
> @@ -21,7 +21,7 @@ SYSROOT_DIRS_NATIVE = " \
>  "
>  SYSROOT_DIRS:append:class-native = " ${SYSROOT_DIRS_NATIVE}"
>  SYSROOT_DIRS:append:class-cross = " ${SYSROOT_DIRS_NATIVE}"
> -SYSROOT_DIRS:append:class-crosssdk = " ${SYSROOT_DIRS_NATIVE}"
> +SYSROOT_DIRS:app:class-crosssdk = " ${SYSROOT_DIRS_NATIVE}"

Did you intend to drop the "end" from "append" here?

-M

>
>  # These directories will not be staged in the sysroot
>  SYSROOT_DIRS_BLACKLIST = " \
> @@ -624,3 +624,36 @@ python staging_taskhandler() {
>  }
>  staging_taskhandler[eventmask] = "bb.event.RecipeTaskPreProcess"
>  addhandler staging_taskhandler
> +
> +
> +#
> +# Target build output, stored in do_populate_sysroot or do_package can depend
> +# not only upon direct dependencies but also indirect ones. A good example is
> +# linux-libc-headers. The toolchain depends on this but most target recipes do
> +# not. There are some headers which are not used by the toolchain build and do
> +# not change the toolchain task output, hence the task hashes can change without
> +# changing the sysroot output of that recipe yet they can influence others.
> +#
> +# A specific example is rtc.h which can change rtcwake.c in util-linux but is not
> +# used in the glibc or gcc build. To account for this, we need to account for the
> +# populate_sysroot hashes in the task output hashes.
> +#
> +python target_add_sysroot_deps () {
> +    current_task = "do_" + d.getVar("BB_CURRENTTASK")
> +    if current_task not in ["do_populate_sysroot", "do_package"]:
> +        return
> +
> +    pn = d.getVar("PN")
> +    if pn.endswith("-native"):
> +        return
> +
> +    taskdepdata = d.getVar("BB_TASKDEPDATA", False)
> +    deps = {}
> +    for dep in taskdepdata.values():
> +        if dep[1] == "do_populate_sysroot" and not dep[0].endswith(("-native", "-initial")) and "-cross-" not in dep[0]:
> +            deps[dep[0]] = dep[6]
> +
> +    d.setVar("HASHEQUIV_EXTRA_SIGDATA", "\n".join("%s: %s" % (k, deps[k]) for k in sorted(deps.keys())))
> +}
> +SSTATECREATEFUNCS += "target_add_sysroot_deps"
> +
> --
> 2.32.0
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#161273): https://lists.openembedded.org/g/openembedded-core/message/161273
> Mute This Topic: https://lists.openembedded.org/mt/88880595/3618418
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [matt@madison.systems]
> -=-=-=-=-=-=-=-=-=-=-=-
>


^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [OE-core] [PATCH 1/2] staging: Add extra hash handling code
  2022-02-03 11:31 ` [OE-core] [PATCH 1/2] staging: Add extra hash handling code Matt Madison
@ 2022-02-03 11:33   ` Richard Purdie
  0 siblings, 0 replies; 4+ messages in thread
From: Richard Purdie @ 2022-02-03 11:33 UTC (permalink / raw)
  To: Matt Madison; +Cc: Patches and discussions about the oe-core layer

On Thu, 2022-02-03 at 03:31 -0800, Matt Madison wrote:
> On Thu, Feb 3, 2022 at 3:25 AM Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > 
> > Target build output, stored in do_populate_sysroot or do_package can depend
> > not only upon direct dependencies but also indirect ones. A good example is
> > linux-libc-headers. The toolchain depends on this but most target recipes do
> > not. There are some headers which are not used by the toolchain build and do
> > not change the toolchain task output, hence the task hashes can change without
> > changing the sysroot output of that recipe yet they can influence others.
> > 
> > A specific example is rtc.h which can change rtcwake.c in util-linux but is not
> > used in the glibc or gcc build. To account for this, we need to account for the
> > populate_sysroot hashes in the task output hashes.
> > 
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > ---
> >  meta/classes/sstate.bbclass  |  2 +-
> >  meta/classes/staging.bbclass | 35 ++++++++++++++++++++++++++++++++++-
> >  2 files changed, 35 insertions(+), 2 deletions(-)
> > 
> > diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
> > index 17dcf4cc175..fd9acbab9b4 100644
> > --- a/meta/classes/sstate.bbclass
> > +++ b/meta/classes/sstate.bbclass
> > @@ -94,7 +94,7 @@ SSTATE_ARCHS[vardepsexclude] = "ORIGNATIVELSBSTRING"
> > 
> >  SSTATE_MANMACH ?= "${SSTATE_PKGARCH}"
> > 
> > -SSTATECREATEFUNCS = "sstate_hardcode_path"
> > +SSTATECREATEFUNCS += "sstate_hardcode_path"
> >  SSTATECREATEFUNCS[vardeps] = "SSTATE_SCAN_FILES"
> >  SSTATEPOSTCREATEFUNCS = ""
> >  SSTATEPREINSTFUNCS = ""
> > diff --git a/meta/classes/staging.bbclass b/meta/classes/staging.bbclass
> > index 25f77c77350..3ea9c974eb5 100644
> > --- a/meta/classes/staging.bbclass
> > +++ b/meta/classes/staging.bbclass
> > @@ -21,7 +21,7 @@ SYSROOT_DIRS_NATIVE = " \
> >  "
> >  SYSROOT_DIRS:append:class-native = " ${SYSROOT_DIRS_NATIVE}"
> >  SYSROOT_DIRS:append:class-cross = " ${SYSROOT_DIRS_NATIVE}"
> > -SYSROOT_DIRS:append:class-crosssdk = " ${SYSROOT_DIRS_NATIVE}"
> > +SYSROOT_DIRS:app:class-crosssdk = " ${SYSROOT_DIRS_NATIVE}"
> 
> Did you intend to drop the "end" from "append" here?

Definitely not, well spotted. I think that happened as I was improving comments
before sending to the list and this is why we have review. I've fixed in master-
next, thanks!

Cheers,

Richard



^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2022-02-03 11:33 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-02-03 11:25 [PATCH 1/2] staging: Add extra hash handling code Richard Purdie
2022-02-03 11:25 ` [PATCH 2/2] abi_version/sstate: Bump for hash equivalence fix Richard Purdie
2022-02-03 11:31 ` [OE-core] [PATCH 1/2] staging: Add extra hash handling code Matt Madison
2022-02-03 11:33   ` Richard Purdie

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.