All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH userspace] sepolicy: generate man pages in parallel
@ 2019-10-14  8:06 Ondrej Mosnacek
  2019-10-17 17:14 ` Stephen Smalley
  0 siblings, 1 reply; 6+ messages in thread
From: Ondrej Mosnacek @ 2019-10-14  8:06 UTC (permalink / raw)
  To: selinux

Generating man pages takes a lot of time. Do it in parallel to speed up
the process.

Signed-off-by: Ondrej Mosnacek <omosnace@redhat.com>
---
 python/sepolicy/sepolicy.py | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/python/sepolicy/sepolicy.py b/python/sepolicy/sepolicy.py
index 1934cd86..02094013 100755
--- a/python/sepolicy/sepolicy.py
+++ b/python/sepolicy/sepolicy.py
@@ -25,6 +25,7 @@ import os
 import sys
 import selinux
 import sepolicy
+from concurrent.futures import ProcessPoolExecutor
 from sepolicy import get_os_version, get_conditionals, get_conditionals_format_text
 import argparse
 PROGNAME = "policycoreutils"
@@ -326,8 +327,13 @@ def gen_gui_args(parser):
     gui.set_defaults(func=gui_run)
 
 
+def manpage_work(domain, path, root, source_files, web):
+    from sepolicy.manpage import ManPage
+    m = ManPage(domain, path, root, source_files, web)
+    print(m.get_man_page_path())
+
 def manpage(args):
-    from sepolicy.manpage import ManPage, HTMLManPages, manpage_domains, manpage_roles, gen_domains
+    from sepolicy.manpage import HTMLManPages, manpage_domains, manpage_roles, gen_domains
 
     path = args.path
     if not args.policy and args.root != "/":
@@ -340,9 +346,9 @@ def manpage(args):
     else:
         test_domains = args.domain
 
-    for domain in test_domains:
-        m = ManPage(domain, path, args.root, args.source_files, args.web)
-        print(m.get_man_page_path())
+    with ProcessPoolExecutor() as e:
+        for domain in test_domains:
+            e.submit(manpage_work, domain, path, args.root, args.source_files, args.web)
 
     if args.web:
         HTMLManPages(manpage_roles, manpage_domains, path, args.os)
-- 
2.21.0


^ permalink raw reply related	[flat|nested] 6+ messages in thread

* Re: [PATCH userspace] sepolicy: generate man pages in parallel
  2019-10-14  8:06 [PATCH userspace] sepolicy: generate man pages in parallel Ondrej Mosnacek
@ 2019-10-17 17:14 ` Stephen Smalley
  2019-10-18  7:44   ` Ondrej Mosnacek
  0 siblings, 1 reply; 6+ messages in thread
From: Stephen Smalley @ 2019-10-17 17:14 UTC (permalink / raw)
  To: Ondrej Mosnacek, selinux

On 10/14/19 4:06 AM, Ondrej Mosnacek wrote:
> Generating man pages takes a lot of time. Do it in parallel to speed up
> the process.
> 
> Signed-off-by: Ondrej Mosnacek <omosnace@redhat.com>

Acked-by: Stephen Smalley <sds@tycho.nsa.gov>

> ---
>   python/sepolicy/sepolicy.py | 14 ++++++++++----
>   1 file changed, 10 insertions(+), 4 deletions(-)
> 
> diff --git a/python/sepolicy/sepolicy.py b/python/sepolicy/sepolicy.py
> index 1934cd86..02094013 100755
> --- a/python/sepolicy/sepolicy.py
> +++ b/python/sepolicy/sepolicy.py
> @@ -25,6 +25,7 @@ import os
>   import sys
>   import selinux
>   import sepolicy
> +from concurrent.futures import ProcessPoolExecutor
>   from sepolicy import get_os_version, get_conditionals, get_conditionals_format_text
>   import argparse
>   PROGNAME = "policycoreutils"
> @@ -326,8 +327,13 @@ def gen_gui_args(parser):
>       gui.set_defaults(func=gui_run)
>   
>   
> +def manpage_work(domain, path, root, source_files, web):
> +    from sepolicy.manpage import ManPage
> +    m = ManPage(domain, path, root, source_files, web)
> +    print(m.get_man_page_path())
> +
>   def manpage(args):
> -    from sepolicy.manpage import ManPage, HTMLManPages, manpage_domains, manpage_roles, gen_domains
> +    from sepolicy.manpage import HTMLManPages, manpage_domains, manpage_roles, gen_domains
>   
>       path = args.path
>       if not args.policy and args.root != "/":
> @@ -340,9 +346,9 @@ def manpage(args):
>       else:
>           test_domains = args.domain
>   
> -    for domain in test_domains:
> -        m = ManPage(domain, path, args.root, args.source_files, args.web)
> -        print(m.get_man_page_path())
> +    with ProcessPoolExecutor() as e:
> +        for domain in test_domains:
> +            e.submit(manpage_work, domain, path, args.root, args.source_files, args.web)
>   
>       if args.web:
>           HTMLManPages(manpage_roles, manpage_domains, path, args.os)
> 


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH userspace] sepolicy: generate man pages in parallel
  2019-10-17 17:14 ` Stephen Smalley
@ 2019-10-18  7:44   ` Ondrej Mosnacek
  2019-10-18  9:00     ` Chris PeBenito
  0 siblings, 1 reply; 6+ messages in thread
From: Ondrej Mosnacek @ 2019-10-18  7:44 UTC (permalink / raw)
  To: Stephen Smalley; +Cc: SElinux list

On Thu, Oct 17, 2019 at 7:15 PM Stephen Smalley <sds@tycho.nsa.gov> wrote:
> On 10/14/19 4:06 AM, Ondrej Mosnacek wrote:
> > Generating man pages takes a lot of time. Do it in parallel to speed up
> > the process.
> >
> > Signed-off-by: Ondrej Mosnacek <omosnace@redhat.com>
>
> Acked-by: Stephen Smalley <sds@tycho.nsa.gov>

Thank you for the ack, however I discovered that after this change it
becomes more difficult to end the program via KeyboardInterrupt
(SIGINT). The first interrupt only stops the main process and you need
to send several more to take down the background processes as well...

I found a different way (multiprocessing.Pool) to do the same, which
ends the processing gracefully on interrupt, but that one behaves even
worse under Python 2 (each interrupt only cancels one work item and
the processing happily continues...). Since there are plans to support
only Python 3 in 3.0+ this may not be an issue, but I could also add a
few lines to fallback to sequential execution under Python 2 for the
sake of compatibility. Would that be OK or should I not bother?

Either way I'd like to send a v2 that uses multiprocessing instead of
concurrent.futures, so please don't merge this yet :)

FYI, here is a preliminary diff for a switch to multiprocessing.Pool:
https://github.com/WOnder93/selinux/commit/a33acec8c298c112f5412b8b61b5b09058a267ee

...and here is what the Python 2 fallback would look like:
https://github.com/WOnder93/selinux/commit/b39a12120656b50eb0a1ee01227646ba3cd63f15

>
> > ---
> >   python/sepolicy/sepolicy.py | 14 ++++++++++----
> >   1 file changed, 10 insertions(+), 4 deletions(-)
> >
> > diff --git a/python/sepolicy/sepolicy.py b/python/sepolicy/sepolicy.py
> > index 1934cd86..02094013 100755
> > --- a/python/sepolicy/sepolicy.py
> > +++ b/python/sepolicy/sepolicy.py
> > @@ -25,6 +25,7 @@ import os
> >   import sys
> >   import selinux
> >   import sepolicy
> > +from concurrent.futures import ProcessPoolExecutor
> >   from sepolicy import get_os_version, get_conditionals, get_conditionals_format_text
> >   import argparse
> >   PROGNAME = "policycoreutils"
> > @@ -326,8 +327,13 @@ def gen_gui_args(parser):
> >       gui.set_defaults(func=gui_run)
> >
> >
> > +def manpage_work(domain, path, root, source_files, web):
> > +    from sepolicy.manpage import ManPage
> > +    m = ManPage(domain, path, root, source_files, web)
> > +    print(m.get_man_page_path())
> > +
> >   def manpage(args):
> > -    from sepolicy.manpage import ManPage, HTMLManPages, manpage_domains, manpage_roles, gen_domains
> > +    from sepolicy.manpage import HTMLManPages, manpage_domains, manpage_roles, gen_domains
> >
> >       path = args.path
> >       if not args.policy and args.root != "/":
> > @@ -340,9 +346,9 @@ def manpage(args):
> >       else:
> >           test_domains = args.domain
> >
> > -    for domain in test_domains:
> > -        m = ManPage(domain, path, args.root, args.source_files, args.web)
> > -        print(m.get_man_page_path())
> > +    with ProcessPoolExecutor() as e:
> > +        for domain in test_domains:
> > +            e.submit(manpage_work, domain, path, args.root, args.source_files, args.web)
> >
> >       if args.web:
> >           HTMLManPages(manpage_roles, manpage_domains, path, args.os)
> >
>


-- 
Ondrej Mosnacek <omosnace at redhat dot com>
Software Engineer, Security Technologies
Red Hat, Inc.


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH userspace] sepolicy: generate man pages in parallel
  2019-10-18  7:44   ` Ondrej Mosnacek
@ 2019-10-18  9:00     ` Chris PeBenito
  2019-10-18  9:01       ` Chris PeBenito
  0 siblings, 1 reply; 6+ messages in thread
From: Chris PeBenito @ 2019-10-18  9:00 UTC (permalink / raw)
  To: Ondrej Mosnacek, Stephen Smalley; +Cc: SElinux list

On 10/18/19 3:44 AM, Ondrej Mosnacek wrote:
> Since there are plans to support
> only Python 3 in 3.0+ this may not be an issue, but I could also add a
> few lines to fallback to sequential execution under Python 2 for the
> sake of compatibility. Would that be OK or should I not bother?

Python 2 end of life is in less than 2 months.  Please don't add new 
code only for Python 2 compatibility.


-- 
Chris PeBenito

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH userspace] sepolicy: generate man pages in parallel
  2019-10-18  9:00     ` Chris PeBenito
@ 2019-10-18  9:01       ` Chris PeBenito
  2019-10-18  9:22         ` Ondrej Mosnacek
  0 siblings, 1 reply; 6+ messages in thread
From: Chris PeBenito @ 2019-10-18  9:01 UTC (permalink / raw)
  To: Ondrej Mosnacek, Stephen Smalley; +Cc: SElinux list

On 10/18/19 5:00 AM, Chris PeBenito wrote:
> On 10/18/19 3:44 AM, Ondrej Mosnacek wrote:
>> Since there are plans to support
>> only Python 3 in 3.0+ this may not be an issue, but I could also add a
>> few lines to fallback to sequential execution under Python 2 for the
>> sake of compatibility. Would that be OK or should I not bother?
> 
> Python 2 end of life is in less than 2 months.  Please don't add new 
> code only for Python 2 compatibility.

I can't count.  It's a little over 2 months.  The point still stands :)

-- 
Chris PeBenito

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH userspace] sepolicy: generate man pages in parallel
  2019-10-18  9:01       ` Chris PeBenito
@ 2019-10-18  9:22         ` Ondrej Mosnacek
  0 siblings, 0 replies; 6+ messages in thread
From: Ondrej Mosnacek @ 2019-10-18  9:22 UTC (permalink / raw)
  To: Chris PeBenito; +Cc: Stephen Smalley, SElinux list

On Fri, Oct 18, 2019 at 11:01 AM Chris PeBenito <pebenito@ieee.org> wrote:
> On 10/18/19 5:00 AM, Chris PeBenito wrote:
> > On 10/18/19 3:44 AM, Ondrej Mosnacek wrote:
> >> Since there are plans to support
> >> only Python 3 in 3.0+ this may not be an issue, but I could also add a
> >> few lines to fallback to sequential execution under Python 2 for the
> >> sake of compatibility. Would that be OK or should I not bother?
> >
> > Python 2 end of life is in less than 2 months.  Please don't add new
> > code only for Python 2 compatibility.
>
> I can't count.  It's a little over 2 months.  The point still stands :)

OK, I posted a v2 without the fallback.

-- 
Ondrej Mosnacek <omosnace at redhat dot com>
Software Engineer, Security Technologies
Red Hat, Inc.


^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2019-10-18  9:22 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-10-14  8:06 [PATCH userspace] sepolicy: generate man pages in parallel Ondrej Mosnacek
2019-10-17 17:14 ` Stephen Smalley
2019-10-18  7:44   ` Ondrej Mosnacek
2019-10-18  9:00     ` Chris PeBenito
2019-10-18  9:01       ` Chris PeBenito
2019-10-18  9:22         ` Ondrej Mosnacek

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.