* [PATCH 0/2] data_smart: simple matching of curly brackets inside Python expressions @ 2016-02-03 10:33 Markus Lehtonen 2016-02-03 10:33 ` [PATCH 1/2] data_smart: simple bracket matching inside python expressions Markus Lehtonen 2016-02-03 10:33 ` [PATCH 2/2] data_smart: handle '\x7d' in python parsing Markus Lehtonen 0 siblings, 2 replies; 9+ messages in thread From: Markus Lehtonen @ 2016-02-03 10:33 UTC (permalink / raw) To: bitbake-devel This patchset aims at making the parsing of Python expressions in variable definitions a bit more reliable by adding simple matching of curly brackets. [YOCTO #8849] Markus Lehtonen (2): data_smart: simple bracket matching inside python expressions data_smart: handle '\x7d' in python parsing lib/bb/data_smart.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- 2.1.4 ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 1/2] data_smart: simple bracket matching inside python expressions 2016-02-03 10:33 [PATCH 0/2] data_smart: simple matching of curly brackets inside Python expressions Markus Lehtonen @ 2016-02-03 10:33 ` Markus Lehtonen 2016-02-03 15:56 ` Christopher Larson 2016-02-03 10:33 ` [PATCH 2/2] data_smart: handle '\x7d' in python parsing Markus Lehtonen 1 sibling, 1 reply; 9+ messages in thread From: Markus Lehtonen @ 2016-02-03 10:33 UTC (permalink / raw) To: bitbake-devel This expands the python expansion regex by matching curly brackets inside python expressions. In its simplicity it is limited: e.g. it does not correctly handle nested curly brackets or open curly brackets (i.e. '{' without a matching '}') inside python expressions. Making python parsing inside data_smart fully error-free would require significantly more work, basically using/implementing python parser to correctly handle python strings etc. [YOCTO #8849] Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> --- lib/bb/data_smart.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py index ca5774b..6069499 100644 --- a/lib/bb/data_smart.py +++ b/lib/bb/data_smart.py @@ -41,7 +41,7 @@ logger = logging.getLogger("BitBake.Data") __setvar_keyword__ = ["_append", "_prepend", "_remove"] __setvar_regexp__ = re.compile('(?P<base>.*?)(?P<keyword>_append|_prepend|_remove)(_(?P<add>.*))?$') __expand_var_regexp__ = re.compile(r"\${[^{}@\n\t ]+}") -__expand_python_regexp__ = re.compile(r"\${@.+?}") +__expand_python_regexp__ = re.compile(r"\${@(({[^{]*?})|.)+?}") def infer_caller_details(loginfo, parent = False, varval = True): """Save the caller the trouble of specifying everything.""" -- 2.1.4 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH 1/2] data_smart: simple bracket matching inside python expressions 2016-02-03 10:33 ` [PATCH 1/2] data_smart: simple bracket matching inside python expressions Markus Lehtonen @ 2016-02-03 15:56 ` Christopher Larson 2016-02-04 6:33 ` Markus Lehtonen 0 siblings, 1 reply; 9+ messages in thread From: Christopher Larson @ 2016-02-03 15:56 UTC (permalink / raw) To: Markus Lehtonen; +Cc: bitbake-devel [-- Attachment #1: Type: text/plain, Size: 950 bytes --] On Wed, Feb 3, 2016 at 3:33 AM, Markus Lehtonen < markus.lehtonen@linux.intel.com> wrote: > This expands the python expansion regex by matching curly brackets > inside python expressions. In its simplicity it is limited: e.g. it does > not correctly handle nested curly brackets or open curly brackets (i.e. > '{' without a matching '}') inside python expressions. Making python > parsing inside data_smart fully error-free would require significantly > more work, basically using/implementing python parser to correctly > handle python strings etc. > > [YOCTO #8849] > > Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> > Will this cause problems with regular variable expansion nested inside of inline python, which is allowed? I.e. ${@ ... '${FOO}'} -- Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics [-- Attachment #2: Type: text/html, Size: 1484 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 1/2] data_smart: simple bracket matching inside python expressions 2016-02-03 15:56 ` Christopher Larson @ 2016-02-04 6:33 ` Markus Lehtonen 0 siblings, 0 replies; 9+ messages in thread From: Markus Lehtonen @ 2016-02-04 6:33 UTC (permalink / raw) To: Christopher Larson; +Cc: bitbake-devel [-- Attachment #1: Type: text/plain, Size: 1224 bytes --] Hi Christopher, On 03/02/16 17:56, "Christopher Larson" <kergoth@gmail.com on behalf of clarson@kergoth.com> wrote: On Wed, Feb 3, 2016 at 3:33 AM, Markus Lehtonen <markus.lehtonen@linux.intel.com> wrote: This expands the python expansion regex by matching curly brackets inside python expressions. In its simplicity it is limited: e.g. it does not correctly handle nested curly brackets or open curly brackets (i.e. '{' without a matching '}') inside python expressions. Making python parsing inside data_smart fully error-free would require significantly more work, basically using/implementing python parser to correctly handle python strings etc. [YOCTO #8849] Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Will this cause problems with regular variable expansion nested inside of inline python, which is allowed? I.e. ${@ ... '${FOO}'} No, it wont affect that. First of all because the variable expansion is done before python expansion – ${FOO} would be expanded before the Python regexp is evaluated. Second, the limitation about nested brackets is only about nested brackets inside the python expression (i.e. something like '${FOO${BAR}}'). Thanks, Markus [-- Attachment #2: Type: text/html, Size: 2307 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 2/2] data_smart: handle '\x7d' in python parsing 2016-02-03 10:33 [PATCH 0/2] data_smart: simple matching of curly brackets inside Python expressions Markus Lehtonen 2016-02-03 10:33 ` [PATCH 1/2] data_smart: simple bracket matching inside python expressions Markus Lehtonen @ 2016-02-03 10:33 ` Markus Lehtonen 2016-02-03 16:52 ` Richard Purdie 1 sibling, 1 reply; 9+ messages in thread From: Markus Lehtonen @ 2016-02-03 10:33 UTC (permalink / raw) To: bitbake-devel At least ncurses recipe uses a workaround (for former bitbake python parser) where closing curly bracket inside a python string was replaced by '\x7d' ascii code. The previous "curly bracket matching" patch broke parsing of constructs like this as bitbake wouldn't correctly find the closing bracket. This patch fixes this by accepting '\x7d' as a closing bracket. Hopefully, this patch could be reverted (and the python regexp thus simplified) in the future after making sure that no recipes use the '\x7d' workaround, anymore. [YOCTO #8849] Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> --- lib/bb/data_smart.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py index 6069499..5f48fe3 100644 --- a/lib/bb/data_smart.py +++ b/lib/bb/data_smart.py @@ -41,7 +41,7 @@ logger = logging.getLogger("BitBake.Data") __setvar_keyword__ = ["_append", "_prepend", "_remove"] __setvar_regexp__ = re.compile('(?P<base>.*?)(?P<keyword>_append|_prepend|_remove)(_(?P<add>.*))?$') __expand_var_regexp__ = re.compile(r"\${[^{}@\n\t ]+}") -__expand_python_regexp__ = re.compile(r"\${@(({[^{]*?})|.)+?}") +__expand_python_regexp__ = re.compile(r"\${@(({[^{]*?(}|\\x7d))|.)+?}") def infer_caller_details(loginfo, parent = False, varval = True): """Save the caller the trouble of specifying everything.""" -- 2.1.4 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH 2/2] data_smart: handle '\x7d' in python parsing 2016-02-03 10:33 ` [PATCH 2/2] data_smart: handle '\x7d' in python parsing Markus Lehtonen @ 2016-02-03 16:52 ` Richard Purdie 2016-02-04 6:39 ` Markus Lehtonen 0 siblings, 1 reply; 9+ messages in thread From: Richard Purdie @ 2016-02-03 16:52 UTC (permalink / raw) To: Markus Lehtonen, bitbake-devel On Wed, 2016-02-03 at 12:33 +0200, Markus Lehtonen wrote: > At least ncurses recipe uses a workaround (for former bitbake python > parser) where closing curly bracket inside a python string was > replaced > by '\x7d' ascii code. The previous "curly bracket matching" patch > broke > parsing of constructs like this as bitbake wouldn't correctly find > the > closing bracket. This patch fixes this by accepting '\x7d' as a > closing > bracket. > > Hopefully, this patch could be reverted (and the python regexp thus > simplified) in the future after making sure that no recipes use the > '\x7d' workaround, anymore. > > [YOCTO #8849] > > Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> > --- > lib/bb/data_smart.py | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) To be honest, the number of recipes which do this is hopefully very very small and we should just go and fix them now? I'm not sure we want to go ahead and support this. Cheers, Richard ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 2/2] data_smart: handle '\x7d' in python parsing 2016-02-03 16:52 ` Richard Purdie @ 2016-02-04 6:39 ` Markus Lehtonen 2016-02-04 8:21 ` Richard Purdie 0 siblings, 1 reply; 9+ messages in thread From: Markus Lehtonen @ 2016-02-04 6:39 UTC (permalink / raw) To: Richard Purdie, bitbake-devel Hi Richard, On 03/02/16 18:52, "Richard Purdie" <richard.purdie@linuxfoundation.org> wrote: >On Wed, 2016-02-03 at 12:33 +0200, Markus Lehtonen wrote: >> At least ncurses recipe uses a workaround (for former bitbake python >> parser) where closing curly bracket inside a python string was >> replaced >> by '\x7d' ascii code. The previous "curly bracket matching" patch >> broke >> parsing of constructs like this as bitbake wouldn't correctly find >> the >> closing bracket. This patch fixes this by accepting '\x7d' as a >> closing >> bracket. >> >> Hopefully, this patch could be reverted (and the python regexp thus >> simplified) in the future after making sure that no recipes use the >> '\x7d' workaround, anymore. >> >> [YOCTO #8849] >> >> Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> >> --- >> lib/bb/data_smart.py | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-) > >To be honest, the number of recipes which do this is hopefully very >very small and we should just go and fix them now? I was pondering this, too, but tried to make sure not to break anything. I can e.g. grep through the layers in git.openembedded.org and git.yoctoproject.org and see if there are any other recipes like this and submit patches to change those. >I'm not sure we want to go ahead and support this. Me neither ;) Thanks, Markus ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 2/2] data_smart: handle '\x7d' in python parsing 2016-02-04 6:39 ` Markus Lehtonen @ 2016-02-04 8:21 ` Richard Purdie 2016-02-04 17:56 ` Markus Lehtonen 0 siblings, 1 reply; 9+ messages in thread From: Richard Purdie @ 2016-02-04 8:21 UTC (permalink / raw) To: Markus Lehtonen, bitbake-devel On Thu, 2016-02-04 at 08:39 +0200, Markus Lehtonen wrote: > On 03/02/16 18:52, "Richard Purdie" < > richard.purdie@linuxfoundation.org> wrote: > > > On Wed, 2016-02-03 at 12:33 +0200, Markus Lehtonen wrote: > > > At least ncurses recipe uses a workaround (for former bitbake > > > python > > > parser) where closing curly bracket inside a python string was > > > replaced > > > by '\x7d' ascii code. The previous "curly bracket matching" patch > > > broke > > > parsing of constructs like this as bitbake wouldn't correctly > > > find > > > the > > > closing bracket. This patch fixes this by accepting '\x7d' as a > > > closing > > > bracket. > > > > > > Hopefully, this patch could be reverted (and the python regexp > > > thus > > > simplified) in the future after making sure that no recipes use > > > the > > > '\x7d' workaround, anymore. > > > > > > [YOCTO #8849] > > > > > > Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> > > > --- > > > lib/bb/data_smart.py | 2 +- > > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > To be honest, the number of recipes which do this is hopefully very > > very small and we should just go and fix them now? > > I was pondering this, too, but tried to make sure not to break > anything. I can e.g. grep through the layers in git.openembedded.org > and git.yoctoproject.org and see if there are any other recipes like > this and submit patches to change those. I think that might be the better option, thanks! Richard ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 2/2] data_smart: handle '\x7d' in python parsing 2016-02-04 8:21 ` Richard Purdie @ 2016-02-04 17:56 ` Markus Lehtonen 0 siblings, 0 replies; 9+ messages in thread From: Markus Lehtonen @ 2016-02-04 17:56 UTC (permalink / raw) To: Richard Purdie, bitbake-devel On 04/02/16 10:21, "Richard Purdie" <richard.purdie@linuxfoundation.org> wrote: >On Thu, 2016-02-04 at 08:39 +0200, Markus Lehtonen wrote: >> On 03/02/16 18:52, "Richard Purdie" < >> richard.purdie@linuxfoundation.org> wrote: >> >> > On Wed, 2016-02-03 at 12:33 +0200, Markus Lehtonen wrote: >> > > At least ncurses recipe uses a workaround (for former bitbake >> > > python >> > > parser) where closing curly bracket inside a python string was >> > > replaced >> > > by '\x7d' ascii code. The previous "curly bracket matching" patch >> > > broke >> > > parsing of constructs like this as bitbake wouldn't correctly >> > > find >> > > the >> > > closing bracket. This patch fixes this by accepting '\x7d' as a >> > > closing >> > > bracket. >> > > >> > > Hopefully, this patch could be reverted (and the python regexp >> > > thus >> > > simplified) in the future after making sure that no recipes use >> > > the >> > > '\x7d' workaround, anymore. >> > > >> > > [YOCTO #8849] >> > > >> > > Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> >> > > --- >> > > lib/bb/data_smart.py | 2 +- >> > > 1 file changed, 1 insertion(+), 1 deletion(-) >> > >> > To be honest, the number of recipes which do this is hopefully very >> > very small and we should just go and fix them now? >> >> I was pondering this, too, but tried to make sure not to break >> anything. I can e.g. grep through the layers in git.openembedded.org >> and git.yoctoproject.org and see if there are any other recipes like >> this and submit patches to change those. > >I think that might be the better option, thanks! You were right. At least I didn’t find any other problematic recipes than ncurses. I sent patch for review: http://lists.openembedded.org/pipermail/openembedded-core/2016-February/116904.html That should be merged at the same time as the [1/2] bitbake patch in order to avoid build failure. Cheers, Markus ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2016-02-04 17:56 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-02-03 10:33 [PATCH 0/2] data_smart: simple matching of curly brackets inside Python expressions Markus Lehtonen 2016-02-03 10:33 ` [PATCH 1/2] data_smart: simple bracket matching inside python expressions Markus Lehtonen 2016-02-03 15:56 ` Christopher Larson 2016-02-04 6:33 ` Markus Lehtonen 2016-02-03 10:33 ` [PATCH 2/2] data_smart: handle '\x7d' in python parsing Markus Lehtonen 2016-02-03 16:52 ` Richard Purdie 2016-02-04 6:39 ` Markus Lehtonen 2016-02-04 8:21 ` Richard Purdie 2016-02-04 17:56 ` Markus Lehtonen
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.