[PATCH] build.py: support custom task [progress] handlers

[PATCH] build.py: support custom task [progress] handlers

[Chris Laplante]

To use this mechanism, you need to inject your progress handler (i.e.
something derived from bb.progress.ProgressHandler) into __builtins__.
Here's one way to do it (from recipe-space):

    def install_my_progress_handler():
        from bb.progress import ProgressHandler

        class MyProgressHandler(ProgressHandler):
            pass

        if "MyProgressHandler" not in __builtins__:
            __builtins__["MyProgressHandler"] = MyProgressHandler

        return "OK"

    _INSTALL_MY_PROGRESS_HANDLERS := "${@install_my_progress_handler()}"

To install on a task:
    do_task[progress] = "custom:MyProgressHandler"

To install on a task and pass extra arguments:
    do_task[progress] = "custom:MyProgressHandler:my-arg"

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
---
 lib/bb/build.py | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/lib/bb/build.py b/lib/bb/build.py
index 3e2a94e..2e14572 100644
--- a/lib/bb/build.py
+++ b/lib/bb/build.py
@@ -387,6 +387,14 @@ exit $ret
         elif progress.startswith('outof:'):
             # Use specified regex
             logfile = bb.progress.OutOfProgressHandler(d, regex=progress.split(':', 1)[1], outfile=logfile)
+        elif progress.startswith("custom:"):
+            # Use a custom progress handler
+            parts = progress.split(":", 2)
+            _, cls, otherargs = parts[0], parts[1], (parts[2] or None) if parts[2:] else None
+            if cls and cls in __builtins__:
+                logfile = __builtins__[cls](d, outfile=logfile, otherargs=otherargs)
+            else:
+                bb.warn('%s: unknown custom progress handler in task progress varflag value "%s", ignoring' % (func, cls))
         else:
             bb.warn('%s: invalid task progress varflag value "%s", ignoring' % (func, progress))
 
-- 
2.7.4

Re: [PATCH] build.py: support custom task [progress] handlers

[Richard Purdie]

On Wed, 2018-10-03 at 22:48 -0400, Chris Laplante via bitbake-devel
wrote:
> To use this mechanism, you need to inject your progress handler (i.e.
> something derived from bb.progress.ProgressHandler) into
> __builtins__.
> Here's one way to do it (from recipe-space):
> 
>     def install_my_progress_handler():
>         from bb.progress import ProgressHandler
> 
>         class MyProgressHandler(ProgressHandler):
>             pass
> 
>         if "MyProgressHandler" not in __builtins__:
>             __builtins__["MyProgressHandler"] = MyProgressHandler
> 
>         return "OK"
> 
>     _INSTALL_MY_PROGRESS_HANDLERS := "${@install_my_progress_handler(
> )}"

Sorry about the lack of review.

My concern on this is the fact we have to poke around __builtins__. I
understand why, I just can't help wonder if there is a better way
somehow.

I'm not sure I have a better one, not sure if Chris or others might?

Cheers,

Richard


> To install on a task:
>     do_task[progress] = "custom:MyProgressHandler"
> 
> To install on a task and pass extra arguments:
>     do_task[progress] = "custom:MyProgressHandler:my-arg"
> 
> Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
> ---
>  lib/bb/build.py | 8 ++++++++
>  1 file changed, 8 insertions(+)
> 
> diff --git a/lib/bb/build.py b/lib/bb/build.py
> index 3e2a94e..2e14572 100644
> --- a/lib/bb/build.py
> +++ b/lib/bb/build.py
> @@ -387,6 +387,14 @@ exit $ret
>          elif progress.startswith('outof:'):
>              # Use specified regex
>              logfile = bb.progress.OutOfProgressHandler(d,
> regex=progress.split(':', 1)[1], outfile=logfile)
> +        elif progress.startswith("custom:"):
> +            # Use a custom progress handler
> +            parts = progress.split(":", 2)
> +            _, cls, otherargs = parts[0], parts[1], (parts[2] or
> None) if parts[2:] else None
> +            if cls and cls in __builtins__:
> +                logfile = __builtins__[cls](d, outfile=logfile,
> otherargs=otherargs)
> +            else:
> +                bb.warn('%s: unknown custom progress handler in task
> progress varflag value "%s", ignoring' % (func, cls))
>          else:
>              bb.warn('%s: invalid task progress varflag value "%s",
> ignoring' % (func, progress))
>  
> -- 
> 2.7.4
> 

Re: [PATCH] build.py: support custom task [progress] handlers

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 3737 bytes --]

This is a great idea, thanks a lot for contribution. As Richard says, I
think using and hardcoding __builtins__ as the injection point isn’t ideal.
Ideally I think we'd be able to provide a python module in a layer with the
progress handler, let OE_IMPORTS import it, and then let bitbake use it as
is.

I think we just need to tweak it to accept the full name with imports, i.e.
custom:oe.myprogressmodule.MyProgressHandler. Then if you want to inject it
you can (though I think bb.context is better than __builtins__), or use an
imported module, whichever is preferred, rather than hardcoding the context.

Something like `functools.reduce(lambda x, y: x[y], cls.split("."),
globals())` might do for resolving the dot-separated name to the class
object, assuming it’s already been imported.

On Thu, Feb 21, 2019 at 2:13 PM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Wed, 2018-10-03 at 22:48 -0400, Chris Laplante via bitbake-devel
> wrote:
> > To use this mechanism, you need to inject your progress handler (i.e.
> > something derived from bb.progress.ProgressHandler) into
> > __builtins__.
> > Here's one way to do it (from recipe-space):
> >
> >     def install_my_progress_handler():
> >         from bb.progress import ProgressHandler
> >
> >         class MyProgressHandler(ProgressHandler):
> >             pass
> >
> >         if "MyProgressHandler" not in __builtins__:
> >             __builtins__["MyProgressHandler"] = MyProgressHandler
> >
> >         return "OK"
> >
> >     _INSTALL_MY_PROGRESS_HANDLERS := "${@install_my_progress_handler(
> > )}"
>
> Sorry about the lack of review.
>
> My concern on this is the fact we have to poke around __builtins__. I
> understand why, I just can't help wonder if there is a better way
> somehow.
>
> I'm not sure I have a better one, not sure if Chris or others might?
>
> Cheers,
>
> Richard
>
>
> > To install on a task:
> >     do_task[progress] = "custom:MyProgressHandler"
> >
> > To install on a task and pass extra arguments:
> >     do_task[progress] = "custom:MyProgressHandler:my-arg"
> >
> > Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
> > ---
> >  lib/bb/build.py | 8 ++++++++
> >  1 file changed, 8 insertions(+)
> >
> > diff --git a/lib/bb/build.py b/lib/bb/build.py
> > index 3e2a94e..2e14572 100644
> > --- a/lib/bb/build.py
> > +++ b/lib/bb/build.py
> > @@ -387,6 +387,14 @@ exit $ret
> >          elif progress.startswith('outof:'):
> >              # Use specified regex
> >              logfile = bb.progress.OutOfProgressHandler(d,
> > regex=progress.split(':', 1)[1], outfile=logfile)
> > +        elif progress.startswith("custom:"):
> > +            # Use a custom progress handler
> > +            parts = progress.split(":", 2)
> > +            _, cls, otherargs = parts[0], parts[1], (parts[2] or
> > None) if parts[2:] else None
> > +            if cls and cls in __builtins__:
> > +                logfile = __builtins__[cls](d, outfile=logfile,
> > otherargs=otherargs)
> > +            else:
> > +                bb.warn('%s: unknown custom progress handler in task
> > progress varflag value "%s", ignoring' % (func, cls))
> >          else:
> >              bb.warn('%s: invalid task progress varflag value "%s",
> > ignoring' % (func, progress))
> >
> > --
> > 2.7.4
> >
>
> --
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
>


-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 5223 bytes --]

Re: [PATCH] build.py: support custom task [progress] handlers

[chris.laplante]

[-- Attachment #1: Type: text/plain, Size: 4729 bytes --]

Thanks Chris. I will work on prototyping this. Since the documentation currently describes OE_IMPORTS as being “internal only”, I will modify it in base.bbclass from this:

OE_IMPORTS += "os sys time oe.path oe.utils oe.types oe.package oe.packagegroup oe.sstatesig oe.lsb oe.cachedpath oe.license"

To something like this:

OE_IMPORTS += "os sys time oe.path oe.utils oe.types oe.package oe.packagegroup oe.sstatesig oe.lsb oe.cachedpath oe.license ${BB_PROGRESS_IMPORTS}"

Is BB_PROGRESS_IMPORTS a suitable variable name?

P.S. Sorry for terrible email formatting. I’m stuck with Outlook + Exchange server at work. Will try to figure out something better.

Chris

From: Christopher Larson <kergoth@gmail.com>
Sent: Thursday, February 21, 2019 4:46 PM
To: Richard Purdie <richard.purdie@linuxfoundation.org>
Cc: LAPLANTE,CHRIS (A-Little Falls,ex1) <chris.laplante@agilent.com>; bitbake-devel@lists.openembedded.org
Subject: Re: [bitbake-devel] [PATCH] build.py: support custom task [progress] handlers

This is a great idea, thanks a lot for contribution. As Richard says, I think using and hardcoding __builtins__ as the injection point isn’t ideal. Ideally I think we'd be able to provide a python module in a layer with the progress handler, let OE_IMPORTS import it, and then let bitbake use it as is.

I think we just need to tweak it to accept the full name with imports, i.e. custom:oe.myprogressmodule.MyProgressHandler. Then if you want to inject it you can (though I think bb.context is better than __builtins__), or use an imported module, whichever is preferred, rather than hardcoding the context.

Something like `functools.reduce(lambda x, y: x[y], cls.split("."), globals())` might do for resolving the dot-separated name to the class object, assuming it’s already been imported.

On Thu, Feb 21, 2019 at 2:13 PM Richard Purdie <richard.purdie@linuxfoundation.org<mailto:richard.purdie@linuxfoundation.org>> wrote:
On Wed, 2018-10-03 at 22:48 -0400, Chris Laplante via bitbake-devel
wrote:
> To use this mechanism, you need to inject your progress handler (i.e.
> something derived from bb.progress.ProgressHandler) into
> __builtins__.
> Here's one way to do it (from recipe-space):
>
>     def install_my_progress_handler():
>         from bb.progress import ProgressHandler
>
>         class MyProgressHandler(ProgressHandler):
>             pass
>
>         if "MyProgressHandler" not in __builtins__:
>             __builtins__["MyProgressHandler"] = MyProgressHandler
>
>         return "OK"
>
>     _INSTALL_MY_PROGRESS_HANDLERS := "${@install_my_progress_handler(
> )}"

Sorry about the lack of review.

My concern on this is the fact we have to poke around __builtins__. I
understand why, I just can't help wonder if there is a better way
somehow.

I'm not sure I have a better one, not sure if Chris or others might?

Cheers,

Richard


> To install on a task:
>     do_task[progress] = "custom:MyProgressHandler"
>
> To install on a task and pass extra arguments:
>     do_task[progress] = "custom:MyProgressHandler:my-arg"
>
> Signed-off-by: Chris Laplante <chris.laplante@agilent.com<mailto:chris.laplante@agilent.com>>
> ---
>  lib/bb/build.py | 8 ++++++++
>  1 file changed, 8 insertions(+)
>
> diff --git a/lib/bb/build.py b/lib/bb/build.py
> index 3e2a94e..2e14572 100644
> --- a/lib/bb/build.py
> +++ b/lib/bb/build.py
> @@ -387,6 +387,14 @@ exit $ret
>          elif progress.startswith('outof:'):
>              # Use specified regex
>              logfile = bb.progress.OutOfProgressHandler(d,
> regex=progress.split(':', 1)[1], outfile=logfile)
> +        elif progress.startswith("custom:"):
> +            # Use a custom progress handler
> +            parts = progress.split(":", 2)
> +            _, cls, otherargs = parts[0], parts[1], (parts[2] or
> None) if parts[2:] else None
> +            if cls and cls in __builtins__:
> +                logfile = __builtins__[cls](d, outfile=logfile,
> otherargs=otherargs)
> +            else:
> +                bb.warn('%s: unknown custom progress handler in task
> progress varflag value "%s", ignoring' % (func, cls))
>          else:
>              bb.warn('%s: invalid task progress varflag value "%s",
> ignoring' % (func, progress))
>
> --
> 2.7.4
>

--
_______________________________________________
bitbake-devel mailing list
bitbake-devel@lists.openembedded.org<mailto:bitbake-devel@lists.openembedded.org>
http://lists.openembedded.org/mailman/listinfo/bitbake-devel



--
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Senior Software Engineer, Mentor Graphics


[-- Attachment #2: Type: text/html, Size: 8177 bytes --]