[PATCH 1/4] libselinux: compile Python bytecode when installing Python files

[PATCH 1/4] libselinux: compile Python bytecode when installing Python files

[Nicolas Iooss]

When selinux module is imported from a Python script, the content of
__init__.py is compiled into bytecode and the result is saved into a
file if it is allowed. For example, when root runs with Python 3.7 a
script that uses "import selinux" on a system where SELinux is in
permissive mode, this file may be created:

    /usr/lib/python3.7/site-packages/selinux/__pycache__/__init__.cpython-37.pyc

Prevent this file from being dynamically created by creating it when
libselinux is installed, using "python -m compileall".

Signed-off-by: Nicolas Iooss <nicolas.iooss@m4x.org>
---
 libselinux/src/Makefile | 1 +
 1 file changed, 1 insertion(+)

diff --git a/libselinux/src/Makefile b/libselinux/src/Makefile
index 3b8bad810de0..349f957355c1 100644
--- a/libselinux/src/Makefile
+++ b/libselinux/src/Makefile
@@ -176,6 +176,7 @@ install-pywrap: pywrap
 	$(PYTHON) setup.py install --prefix=$(PREFIX) `test -n "$(DESTDIR)" && echo --root $(DESTDIR)`
 	install -m 644 $(SWIGPYOUT) $(DESTDIR)$(PYTHONLIBDIR)/selinux/__init__.py
 	ln -sf --relative $(DESTDIR)$(PYTHONLIBDIR)/selinux/_selinux$(PYCEXT) $(DESTDIR)$(PYTHONLIBDIR)/_selinux$(PYCEXT)
+	$(PYTHON) -m compileall $(DESTDIR)$(PYTHONLIBDIR)/selinux
 
 install-rubywrap: rubywrap
 	test -d $(DESTDIR)$(RUBYINSTALL) || install -m 755 -d $(DESTDIR)$(RUBYINSTALL) 
-- 
2.23.0

[PATCH 2/4] libsemanage: compile Python bytecode when installing Python files

[Nicolas Iooss]

Create a cache file such as
/usr/lib/python3.7/site-packages/__pycache__/semanage.cpython-37.pyc
in order to prevent it from being dynamically created when semanage
module is imported.

Signed-off-by: Nicolas Iooss <nicolas.iooss@m4x.org>
---
 libsemanage/src/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libsemanage/src/Makefile b/libsemanage/src/Makefile
index e029f0988dd8..a9adfa47bdc8 100644
--- a/libsemanage/src/Makefile
+++ b/libsemanage/src/Makefile
@@ -145,7 +145,7 @@ install-pywrap: pywrap
 	test -d $(DESTDIR)$(PYTHONLIBDIR) || install -m 755 -d $(DESTDIR)$(PYTHONLIBDIR)
 	install -m 755 $(SWIGSO) $(DESTDIR)$(PYTHONLIBDIR)/_semanage$(PYCEXT)
 	install -m 644 semanage.py $(DESTDIR)$(PYTHONLIBDIR)
-
+	$(PYTHON) -m compileall $(DESTDIR)$(PYTHONLIBDIR)/semanage.py
 
 install-rubywrap: rubywrap
 	test -d $(DESTDIR)$(RUBYINSTALL) || install -m 755 -d $(DESTDIR)$(RUBYINSTALL) 
-- 
2.23.0

[PATCH 3/4] python: compile Python bytecode when installing Python files

[Nicolas Iooss]

Create a cache directory such as
/usr/lib/python3.7/site-packages/sepolgen/__pycache__/ in order to
prevent it from being dynamically created when sepolgen module is
imported.

Signed-off-by: Nicolas Iooss <nicolas.iooss@m4x.org>
---
 python/semanage/Makefile              | 1 +
 python/sepolgen/src/sepolgen/Makefile | 1 +
 2 files changed, 2 insertions(+)

diff --git a/python/semanage/Makefile b/python/semanage/Makefile
index 024e9640fed0..c6a16918360d 100644
--- a/python/semanage/Makefile
+++ b/python/semanage/Makefile
@@ -28,6 +28,7 @@ install: all
 	done
 	test -d $(DESTDIR)/$(PACKAGEDIR) || install -m 755 -d $(DESTDIR)/$(PACKAGEDIR)
 	install -m 644 seobject.py $(DESTDIR)/$(PACKAGEDIR)
+	$(PYTHON) -m compileall $(DESTDIR)/$(PACKAGEDIR)/seobject.py
 	-mkdir -p $(DESTDIR)$(BASHCOMPLETIONDIR)
 	install -m 644 $(BASHCOMPLETIONS) $(DESTDIR)$(BASHCOMPLETIONDIR)/semanage
 
diff --git a/python/sepolgen/src/sepolgen/Makefile b/python/sepolgen/src/sepolgen/Makefile
index cac8def7204e..95a9d41571e7 100644
--- a/python/sepolgen/src/sepolgen/Makefile
+++ b/python/sepolgen/src/sepolgen/Makefile
@@ -8,6 +8,7 @@ all:
 install: all
 	-mkdir -p $(DESTDIR)$(PACKAGEDIR)
 	install -m 644 *.py $(DESTDIR)$(PACKAGEDIR)
+	$(PYTHON) -m compileall $(DESTDIR)$(PACKAGEDIR)
 
 clean:
 	rm -f parser.out parsetab.py
-- 
2.23.0

[PATCH 4/4] gui: compile Python bytecode when installing Python files

[Nicolas Iooss]

Create a cache directory such as
/usr/share/system-config-selinux/__pycache__/ in order to prevent it
from being dynamically created when the gui is used.

Signed-off-by: Nicolas Iooss <nicolas.iooss@m4x.org>
---
 gui/Makefile | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/gui/Makefile b/gui/Makefile
index ca965c942912..c3fd196fcd2c 100644
--- a/gui/Makefile
+++ b/gui/Makefile
@@ -1,3 +1,5 @@
+PYTHON ?= python3
+
 # Installation directories.
 LINGUAS ?= ru
 PREFIX ?= /usr
@@ -54,6 +56,8 @@ install: all
 		install -m 644 sepolicy_$${i}.png $(DESTDIR)$(DATADIR)/icons/hicolor/$${i}x$${i}/apps/sepolicy.png; \
 	done
 	install -m 644 org.selinux.config.policy $(DESTDIR)$(DATADIR)/polkit-1/actions/
+	$(PYTHON) -m compileall $(DESTDIR)$(SHAREDIR)
+
 clean:
 
 indent:
-- 
2.23.0

Re: [PATCH 1/4] libselinux: compile Python bytecode when installing Python files

[Thomas Petazzoni]

Hello Nicolas,

On Sat,  2 Nov 2019 18:28:09 +0100
Nicolas Iooss <nicolas.iooss@m4x.org> wrote:

> When selinux module is imported from a Python script, the content of
> __init__.py is compiled into bytecode and the result is saved into a
> file if it is allowed. For example, when root runs with Python 3.7 a
> script that uses "import selinux" on a system where SELinux is in
> permissive mode, this file may be created:
> 
>     /usr/lib/python3.7/site-packages/selinux/__pycache__/__init__.cpython-37.pyc
> 
> Prevent this file from being dynamically created by creating it when
> libselinux is installed, using "python -m compileall".
> 
> Signed-off-by: Nicolas Iooss <nicolas.iooss@m4x.org>

As far as I know, this not typically done by "setup.py install", and
this is generally left to distributions.

In the context of Buildroot [1], we do the byte-compilation all at once
at the very end of the build of all packages. Having individual
packages do their own byte-compilation would be annoying.

If you would like to have this byte-compilation done by the SELinux
Makefile, could you make it optional (even if you decide to enable it
by default) ?

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [PATCH 1/4] libselinux: compile Python bytecode when installing Python files

[Nicolas Iooss]

On Sat, Nov 2, 2019 at 9:18 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> Hello Nicolas,
>
> On Sat,  2 Nov 2019 18:28:09 +0100
> Nicolas Iooss <nicolas.iooss@m4x.org> wrote:
>
> > When selinux module is imported from a Python script, the content of
> > __init__.py is compiled into bytecode and the result is saved into a
> > file if it is allowed. For example, when root runs with Python 3.7 a
> > script that uses "import selinux" on a system where SELinux is in
> > permissive mode, this file may be created:
> >
> >     /usr/lib/python3.7/site-packages/selinux/__pycache__/__init__.cpython-37.pyc
> >
> > Prevent this file from being dynamically created by creating it when
> > libselinux is installed, using "python -m compileall".
> >
> > Signed-off-by: Nicolas Iooss <nicolas.iooss@m4x.org>
>
> As far as I know, this not typically done by "setup.py install", and
> this is generally left to distributions.

Hello, this is done when a parameter such as "--optimize=1" is given
to "python setup.py install". Such a parameter is described in Arch
Linux packaging guidelines
(https://wiki.archlinux.org/index.php/Python_package_guidelines#distutils).
Nevertheless I agree it seems to be left to distributions whether to
bytecode-compile Python modules.

> In the context of Buildroot [1], we do the byte-compilation all at once
> at the very end of the build of all packages. Having individual
> packages do their own byte-compilation would be annoying.

What is the reference you are using for [1]? Why would this be
annoying? For example, do you use a non-standard way of generating the
bytecode which is not compatible with the one that would be generated
by the package itself?

> If you would like to have this byte-compilation done by the SELinux
> Makefile, could you make it optional (even if you decide to enable it
> by default) ?

My main motivation behind this patch is to remove lines such as this
one from Arch Linux's PKGBUILD:

/usr/bin/python3 -m compileall "${pkgdir}/$(/usr/bin/python3 -c
'import site; print(site.getsitepackages()[0])')"

(Example from https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=selinux-python&id=dd87f7ae1d636a02bf404ed527a27022ee5d2ab3)
As every Makefile installing Python scripts has everything that is
needed to build this command line, it is much simpler for "make
install" to perform the bytecode-compilation directly instead.
Nevertheless, as there is an interest to keep bytecode-compilation out
of the "make && make install" process, I will rework my patches to
make it optional (so that Arch Linux and maybe other distributions can
invoke it, but not Buildroot).

Thanks,
Nicolas

Re: [PATCH 1/4] libselinux: compile Python bytecode when installing Python files

[Nicolas Iooss]

On Sun, Nov 3, 2019 at 9:57 PM Nicolas Iooss <nicolas.iooss@m4x.org> wrote:
>
> On Sat, Nov 2, 2019 at 9:18 PM Thomas Petazzoni
> <thomas.petazzoni@bootlin.com> wrote:
> >
> > Hello Nicolas,
> >
> > On Sat,  2 Nov 2019 18:28:09 +0100
> > Nicolas Iooss <nicolas.iooss@m4x.org> wrote:
> >
> > > When selinux module is imported from a Python script, the content of
> > > __init__.py is compiled into bytecode and the result is saved into a
> > > file if it is allowed. For example, when root runs with Python 3.7 a
> > > script that uses "import selinux" on a system where SELinux is in
> > > permissive mode, this file may be created:
> > >
> > >     /usr/lib/python3.7/site-packages/selinux/__pycache__/__init__.cpython-37.pyc
> > >
> > > Prevent this file from being dynamically created by creating it when
> > > libselinux is installed, using "python -m compileall".
> > >
> > > Signed-off-by: Nicolas Iooss <nicolas.iooss@m4x.org>
> >
> > As far as I know, this not typically done by "setup.py install", and
> > this is generally left to distributions.
>
> Hello, this is done when a parameter such as "--optimize=1" is given
> to "python setup.py install". Such a parameter is described in Arch
> Linux packaging guidelines
> (https://wiki.archlinux.org/index.php/Python_package_guidelines#distutils).
> Nevertheless I agree it seems to be left to distributions whether to
> bytecode-compile Python modules.

I have spent some time working on these patches, and it appears that
the base assumption of "this not typically done by setup.py install"
is not true on my system (Arch Linux with Python 3.7): setup.py *DOES*
compile bytecode into .pyc files. What the distribution I'm using does
when packaging is giving a parameter such as --optimize=1 in order to
create .opt-1.pyc files too. Currently, python/sepolicy/Makefile
already uses setup.py (cf.
https://github.com/SELinuxProject/selinux/blob/selinux-python-3.0-rc1/python/sepolicy/Makefile#L30),
so .pyc files are already generated in some Makefiles of the project.

Could you please confirm that "setup.py install" does not compile
Python scripts into bytecode on your system? This test can be achieved
for example by cloning https://github.com/SELinuxProject/selinux,
running "make -C python/sepolicy DESTDIR=/tmp/selinux install" and
searching for .pyc files in /tmp/selinux.

If "setup.py install" produces .pyc files on every system, from my
point of view there are several alternatives that can be considered:

1. Keep the current behavior by default and introduce a COMPILE_PY
boolean variable in Makefiles that produces both .pyc and .opt-1.pyc
files in Makefiles that do not use setup.py.
2. Generate .pyc files everywhere (what my patches do), and introduce
a OPTIMIZE_PY boolean variable in Makefiles that is used to trigger
the compilation to .opt-1.pyc files.
3. Mix 2 and 3 by introducing both COMPILE_PY and OPTIMIZE_PY.
4. Find a way for "setup.py install" not to produce such files in
python/sepolicy/Makefile and do as 1 or 3.

(A 5th option would be to compile both .pyc and .opt-1.pyc files
without introducing new variables, but this would go against what
appears to be a legitimate user request)

My personal preference here would be either 1. or 2. Would one of
these options suits Buildroot maintainers better?

Thanks,
Nicolas