[OE-core][PATCH v2] package: moving field data process before variable process in process_pkgconfig

[OE-core][PATCH v2] package: moving field data process before variable process in process_pkgconfig

[Xiangyu Chen]

From: Xiangyu Chen <xiangyu.chen@windriver.com>

Currently, the latest version abseil-cpp contains a new library named "absl_log_internal_format", it's
basic package config(.pc file) as below:

prefix=/usr
exec_prefix=${prefix}

......

Requires: absl_config = 20230125, absl_core_headers = 20230125, absl_log_internal_append_truncated = 20230125,
absl_log_internal_config = 20230125, absl_log_internal_globals = 20230125, absl_log_severity = 20230125,
absl_strings = 20230125, absl_str_format = 20230125, absl_time = 20230125, absl_span = 20230125
......

Normally, the process_pkgconfig() would process variable data before field data in a .pc file, but in the
absl_log_internal_format, the field data in "Requires" section contains "xxxx = xxxx" format, the
process_pkgconfig() treats them as normal variable and using the setVar() in bitbake's data_smart.py
try to process. The absl_log_internal_format field data contains "_append_", this hit the setVar() checking
and finally bitbake stop building and reporting an error as below:

"Variable xxx contains an operation using the old override syntax. Please convert this layer/metadata before attempting to use with a newer bitbake."

This patch move the field data process before variable process to avoid the process_pkgconfig() treat the field
data as variable.

Signed-off-by: Xiangyu Chen <xiangyu.chen@windriver.com>
---

Changes:
V1 -> V2: changing the package.py instead of bitbake's data_smart.py
v1 thread please refer:https://lists.openembedded.org/g/bitbake-devel/message/14525

---
 meta/lib/oe/package.py | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index c9eb75d852..7a6b31957a 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -1823,18 +1823,18 @@ def process_pkgconfig(pkgfiles, d):
                     with open(file, 'r') as f:
                         lines = f.readlines()
                     for l in lines:
-                        m = var_re.match(l)
-                        if m:
-                            name = m.group(1)
-                            val = m.group(2)
-                            pd.setVar(name, pd.expand(val))
-                            continue
                         m = field_re.match(l)
                         if m:
                             hdr = m.group(1)
                             exp = pd.expand(m.group(2))
                             if hdr == 'Requires':
                                 pkgconfig_needed[pkg] += exp.replace(',', ' ').split()
+                                continue
+                        m = var_re.match(l)
+                        if m:
+                            name = m.group(1)
+                            val = m.group(2)
+                            pd.setVar(name, pd.expand(val))
 
     for pkg in packages.split():
         pkgs_file = os.path.join(shlibswork_dir, pkg + ".pclist")
-- 
2.34.1

Re: [OE-core][PATCH v2] package: moving field data process before variable process in process_pkgconfig

[Richard Purdie]

On Fri, 2023-03-24 at 16:36 +0800, Xiangyu Chen wrote:
> From: Xiangyu Chen <xiangyu.chen@windriver.com>
> 
> Currently, the latest version abseil-cpp contains a new library named "absl_log_internal_format", it's
> basic package config(.pc file) as below:
> 
> prefix=/usr
> exec_prefix=${prefix}
> 
> ......
> 
> Requires: absl_config = 20230125, absl_core_headers = 20230125, absl_log_internal_append_truncated = 20230125,
> absl_log_internal_config = 20230125, absl_log_internal_globals = 20230125, absl_log_severity = 20230125,
> absl_strings = 20230125, absl_str_format = 20230125, absl_time = 20230125, absl_span = 20230125
> ......
> 
> Normally, the process_pkgconfig() would process variable data before field data in a .pc file, but in the
> absl_log_internal_format, the field data in "Requires" section contains "xxxx = xxxx" format, the
> process_pkgconfig() treats them as normal variable and using the setVar() in bitbake's data_smart.py
> try to process. The absl_log_internal_format field data contains "_append_", this hit the setVar() checking
> and finally bitbake stop building and reporting an error as below:
> 
> "Variable xxx contains an operation using the old override syntax. Please convert this layer/metadata before attempting to use with a newer bitbake."
> 
> This patch move the field data process before variable process to avoid the process_pkgconfig() treat the field
> data as variable.
> 
> Signed-off-by: Xiangyu Chen <xiangyu.chen@windriver.com>
> ---
> 
> Changes:
> V1 -> V2: changing the package.py instead of bitbake's data_smart.py
> v1 thread please refer:https://lists.openembedded.org/g/bitbake-devel/message/14525

This looks like a much better fix than the workaround in v1, thanks!

Cheers,

Richard

Re: [OE-core][PATCH v2] package: moving field data process before variable process in process_pkgconfig

[Richard Purdie]

On Fri, 2023-03-24 at 16:36 +0800, Xiangyu Chen wrote:
> From: Xiangyu Chen <xiangyu.chen@windriver.com>
> 
> Currently, the latest version abseil-cpp contains a new library named "absl_log_internal_format", it's
> basic package config(.pc file) as below:
> 
> prefix=/usr
> exec_prefix=${prefix}
> 
> ......
> 
> Requires: absl_config = 20230125, absl_core_headers = 20230125, absl_log_internal_append_truncated = 20230125,
> absl_log_internal_config = 20230125, absl_log_internal_globals = 20230125, absl_log_severity = 20230125,
> absl_strings = 20230125, absl_str_format = 20230125, absl_time = 20230125, absl_span = 20230125
> ......
> 
> Normally, the process_pkgconfig() would process variable data before field data in a .pc file, but in the
> absl_log_internal_format, the field data in "Requires" section contains "xxxx = xxxx" format, the
> process_pkgconfig() treats them as normal variable and using the setVar() in bitbake's data_smart.py
> try to process. The absl_log_internal_format field data contains "_append_", this hit the setVar() checking
> and finally bitbake stop building and reporting an error as below:
> 
> "Variable xxx contains an operation using the old override syntax. Please convert this layer/metadata before attempting to use with a newer bitbake."
> 
> This patch move the field data process before variable process to avoid the process_pkgconfig() treat the field
> data as variable.
> 
> Signed-off-by: Xiangyu Chen <xiangyu.chen@windriver.com>
> ---
> 
> Changes:
> V1 -> V2: changing the package.py instead of bitbake's data_smart.py
> v1 thread please refer:https://lists.openembedded.org/g/bitbake-devel/message/14525
> 
> ---
>  meta/lib/oe/package.py | 12 ++++++------
>  1 file changed, 6 insertions(+), 6 deletions(-)
> 
> diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
> index c9eb75d852..7a6b31957a 100644
> --- a/meta/lib/oe/package.py
> +++ b/meta/lib/oe/package.py
> @@ -1823,18 +1823,18 @@ def process_pkgconfig(pkgfiles, d):
>                      with open(file, 'r') as f:
>                          lines = f.readlines()
>                      for l in lines:
> -                        m = var_re.match(l)
> -                        if m:
> -                            name = m.group(1)
> -                            val = m.group(2)
> -                            pd.setVar(name, pd.expand(val))
> -                            continue
>                          m = field_re.match(l)
>                          if m:
>                              hdr = m.group(1)
>                              exp = pd.expand(m.group(2))
>                              if hdr == 'Requires':
>                                  pkgconfig_needed[pkg] += exp.replace(',', ' ').split()
> +                                continue
> +                        m = var_re.match(l)
> +                        if m:
> +                            name = m.group(1)
> +                            val = m.group(2)
> +                            pd.setVar(name, pd.expand(val))
>  
>      for pkg in packages.split():
>          pkgs_file = os.path.join(shlibswork_dir, pkg + ".pclist")

It is worth noting this fix actually resolved a number of latent bugs
too. This breakage in reproducibility was as a result of this change:

http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20230325-0k1j3_4w/packages/diff-html/

which was because the output changed but the sstate hashes didn't. That
is because package.py code isn't part of the sstate hashes (currently).

This isn't a fault of this patch. I've a patch in progress to bump
various versions to make sure this doesn't break anything further and
it does raise a question on whether we should checksum the python
functions and track that due to the adverse impact an issue like this
can have.

The issue here was missing dependencies, which were missing from -dev
packages when there was a version constraint present. I wanted to
highlight why fixing things at the root cause is good! If we'd hacked
this in bitbake, the other issues would have remained unseen.

Cheers,

Richard