[PATCH 0/2] data_smart: simple matching of curly brackets inside Python expressions

[PATCH 0/2] data_smart: simple matching of curly brackets inside Python expressions

[Markus Lehtonen]

This patchset aims at making the parsing of Python expressions in variable
definitions a bit more reliable by adding simple matching of curly brackets.

[YOCTO #8849]

Markus Lehtonen (2):
  data_smart: simple bracket matching inside python expressions
  data_smart: handle '\x7d' in python parsing

 lib/bb/data_smart.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
2.1.4

[PATCH 1/2] data_smart: simple bracket matching inside python expressions

[Markus Lehtonen]

This expands the python expansion regex by matching curly brackets
inside python expressions. In its simplicity it is limited: e.g. it does
not correctly handle nested curly brackets or open curly brackets (i.e.
'{' without a matching '}') inside python expressions. Making python
parsing inside data_smart fully error-free would require significantly
more work, basically using/implementing python parser to correctly
handle python strings etc.

[YOCTO #8849]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 lib/bb/data_smart.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py
index ca5774b..6069499 100644
--- a/lib/bb/data_smart.py
+++ b/lib/bb/data_smart.py
@@ -41,7 +41,7 @@ logger = logging.getLogger("BitBake.Data")
 __setvar_keyword__ = ["_append", "_prepend", "_remove"]
 __setvar_regexp__ = re.compile('(?P<base>.*?)(?P<keyword>_append|_prepend|_remove)(_(?P<add>.*))?$')
 __expand_var_regexp__ = re.compile(r"\${[^{}@\n\t ]+}")
-__expand_python_regexp__ = re.compile(r"\${@.+?}")
+__expand_python_regexp__ = re.compile(r"\${@(({[^{]*?})|.)+?}")
 
 def infer_caller_details(loginfo, parent = False, varval = True):
     """Save the caller the trouble of specifying everything."""
-- 
2.1.4

[PATCH 2/2] data_smart: handle '\x7d' in python parsing

[Markus Lehtonen]

At least ncurses recipe uses a workaround (for former bitbake python
parser) where closing curly bracket inside a python string was replaced
by '\x7d' ascii code. The previous "curly bracket matching" patch broke
parsing of constructs like this as bitbake wouldn't correctly find the
closing bracket. This patch fixes this by accepting '\x7d' as a closing
bracket.

Hopefully, this patch could be reverted (and the python regexp thus
simplified) in the future after making sure that no recipes use the
'\x7d' workaround, anymore.

[YOCTO #8849]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 lib/bb/data_smart.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py
index 6069499..5f48fe3 100644
--- a/lib/bb/data_smart.py
+++ b/lib/bb/data_smart.py
@@ -41,7 +41,7 @@ logger = logging.getLogger("BitBake.Data")
 __setvar_keyword__ = ["_append", "_prepend", "_remove"]
 __setvar_regexp__ = re.compile('(?P<base>.*?)(?P<keyword>_append|_prepend|_remove)(_(?P<add>.*))?$')
 __expand_var_regexp__ = re.compile(r"\${[^{}@\n\t ]+}")
-__expand_python_regexp__ = re.compile(r"\${@(({[^{]*?})|.)+?}")
+__expand_python_regexp__ = re.compile(r"\${@(({[^{]*?(}|\\x7d))|.)+?}")
 
 def infer_caller_details(loginfo, parent = False, varval = True):
     """Save the caller the trouble of specifying everything."""
-- 
2.1.4

Re: [PATCH 1/2] data_smart: simple bracket matching inside python expressions

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 950 bytes --]

On Wed, Feb 3, 2016 at 3:33 AM, Markus Lehtonen <
markus.lehtonen@linux.intel.com> wrote:

> This expands the python expansion regex by matching curly brackets
> inside python expressions. In its simplicity it is limited: e.g. it does
> not correctly handle nested curly brackets or open curly brackets (i.e.
> '{' without a matching '}') inside python expressions. Making python
> parsing inside data_smart fully error-free would require significantly
> more work, basically using/implementing python parser to correctly
> handle python strings etc.
>
> [YOCTO #8849]
>
> Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
>

Will this cause problems with regular variable expansion nested inside of
inline python, which is allowed? I.e. ${@ ... '${FOO}'}
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1484 bytes --]

Re: [PATCH 2/2] data_smart: handle '\x7d' in python parsing

[Richard Purdie]

On Wed, 2016-02-03 at 12:33 +0200, Markus Lehtonen wrote:
> At least ncurses recipe uses a workaround (for former bitbake python
> parser) where closing curly bracket inside a python string was
> replaced
> by '\x7d' ascii code. The previous "curly bracket matching" patch
> broke
> parsing of constructs like this as bitbake wouldn't correctly find
> the
> closing bracket. This patch fixes this by accepting '\x7d' as a
> closing
> bracket.
> 
> Hopefully, this patch could be reverted (and the python regexp thus
> simplified) in the future after making sure that no recipes use the
> '\x7d' workaround, anymore.
> 
> [YOCTO #8849]
> 
> Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
> ---
>  lib/bb/data_smart.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

To be honest, the number of recipes which do this is hopefully very
very small and we should just go and fix them now?

I'm not sure we want to go ahead and support this.

Cheers,

Richard

Re: [PATCH 1/2] data_smart: simple bracket matching inside python expressions

[Markus Lehtonen]

[-- Attachment #1: Type: text/plain, Size: 1224 bytes --]

Hi Christopher,

On 03/02/16 17:56, "Christopher Larson" <kergoth@gmail.com on behalf of clarson@kergoth.com> wrote:


On Wed, Feb 3, 2016 at 3:33 AM, Markus Lehtonen <markus.lehtonen@linux.intel.com> wrote:
This expands the python expansion regex by matching curly brackets
inside python expressions. In its simplicity it is limited: e.g. it does
not correctly handle nested curly brackets or open curly brackets (i.e.
'{' without a matching '}') inside python expressions. Making python
parsing inside data_smart fully error-free would require significantly
more work, basically using/implementing python parser to correctly
handle python strings etc.

[YOCTO #8849]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

Will this cause problems with regular variable expansion nested inside of inline python, which is allowed? I.e. ${@ ... '${FOO}'}

No, it wont affect that. First of all because the variable expansion is done before python expansion – ${FOO} would be expanded before the Python regexp is evaluated. Second, the limitation about nested brackets is only about nested brackets inside the python expression (i.e. something like '${FOO${BAR}}').

Thanks,
  Markus



[-- Attachment #2: Type: text/html, Size: 2307 bytes --]

Re: [PATCH 2/2] data_smart: handle '\x7d' in python parsing

[Markus Lehtonen]

Hi Richard,



On 03/02/16 18:52, "Richard Purdie" <richard.purdie@linuxfoundation.org> wrote:

>On Wed, 2016-02-03 at 12:33 +0200, Markus Lehtonen wrote:
>> At least ncurses recipe uses a workaround (for former bitbake python
>> parser) where closing curly bracket inside a python string was
>> replaced
>> by '\x7d' ascii code. The previous "curly bracket matching" patch
>> broke
>> parsing of constructs like this as bitbake wouldn't correctly find
>> the
>> closing bracket. This patch fixes this by accepting '\x7d' as a
>> closing
>> bracket.
>> 
>> Hopefully, this patch could be reverted (and the python regexp thus
>> simplified) in the future after making sure that no recipes use the
>> '\x7d' workaround, anymore.
>> 
>> [YOCTO #8849]
>> 
>> Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
>> ---
>>  lib/bb/data_smart.py | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>
>To be honest, the number of recipes which do this is hopefully very
>very small and we should just go and fix them now?

I was pondering this, too, but tried to make sure not to break anything. I can e.g. grep through the layers in git.openembedded.org and git.yoctoproject.org and see if there are any other recipes like this and submit patches to change those.


>I'm not sure we want to go ahead and support this.

Me neither ;)

Thanks,
  Markus

Re: [PATCH 2/2] data_smart: handle '\x7d' in python parsing

[Richard Purdie]

On Thu, 2016-02-04 at 08:39 +0200, Markus Lehtonen wrote:
> On 03/02/16 18:52, "Richard Purdie" <
> richard.purdie@linuxfoundation.org> wrote:
> 
> > On Wed, 2016-02-03 at 12:33 +0200, Markus Lehtonen wrote:
> > > At least ncurses recipe uses a workaround (for former bitbake
> > > python
> > > parser) where closing curly bracket inside a python string was
> > > replaced
> > > by '\x7d' ascii code. The previous "curly bracket matching" patch
> > > broke
> > > parsing of constructs like this as bitbake wouldn't correctly
> > > find
> > > the
> > > closing bracket. This patch fixes this by accepting '\x7d' as a
> > > closing
> > > bracket.
> > > 
> > > Hopefully, this patch could be reverted (and the python regexp
> > > thus
> > > simplified) in the future after making sure that no recipes use
> > > the
> > > '\x7d' workaround, anymore.
> > > 
> > > [YOCTO #8849]
> > > 
> > > Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
> > > ---
> > >  lib/bb/data_smart.py | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > To be honest, the number of recipes which do this is hopefully very
> > very small and we should just go and fix them now?
> 
> I was pondering this, too, but tried to make sure not to break
> anything. I can e.g. grep through the layers in git.openembedded.org
> and git.yoctoproject.org and see if there are any other recipes like
> this and submit patches to change those.

I think that might be the better option, thanks!

Richard

Re: [PATCH 2/2] data_smart: handle '\x7d' in python parsing

[Markus Lehtonen]

On 04/02/16 10:21, "Richard Purdie" <richard.purdie@linuxfoundation.org> wrote:

>On Thu, 2016-02-04 at 08:39 +0200, Markus Lehtonen wrote:
>> On 03/02/16 18:52, "Richard Purdie" <
>> richard.purdie@linuxfoundation.org> wrote:
>> 
>> > On Wed, 2016-02-03 at 12:33 +0200, Markus Lehtonen wrote:
>> > > At least ncurses recipe uses a workaround (for former bitbake
>> > > python
>> > > parser) where closing curly bracket inside a python string was
>> > > replaced
>> > > by '\x7d' ascii code. The previous "curly bracket matching" patch
>> > > broke
>> > > parsing of constructs like this as bitbake wouldn't correctly
>> > > find
>> > > the
>> > > closing bracket. This patch fixes this by accepting '\x7d' as a
>> > > closing
>> > > bracket.
>> > > 
>> > > Hopefully, this patch could be reverted (and the python regexp
>> > > thus
>> > > simplified) in the future after making sure that no recipes use
>> > > the
>> > > '\x7d' workaround, anymore.
>> > > 
>> > > [YOCTO #8849]
>> > > 
>> > > Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
>> > > ---
>> > >  lib/bb/data_smart.py | 2 +-
>> > >  1 file changed, 1 insertion(+), 1 deletion(-)
>> > 
>> > To be honest, the number of recipes which do this is hopefully very
>> > very small and we should just go and fix them now?
>> 
>> I was pondering this, too, but tried to make sure not to break
>> anything. I can e.g. grep through the layers in git.openembedded.org
>> and git.yoctoproject.org and see if there are any other recipes like
>> this and submit patches to change those.
>
>I think that might be the better option, thanks!

You were right. At least I didn’t find any other problematic recipes than ncurses. I sent patch for review:
http://lists.openembedded.org/pipermail/openembedded-core/2016-February/116904.html

That should be merged at the same time as the [1/2] bitbake patch in order to avoid build failure.

Cheers,
   Markus