[Buildroot] Python pyc only and problem with scripts

[Buildroot] Python pyc only and problem with scripts

[Federico Pellegrin]

[-- Attachment #1.1: Type: text/plain, Size: 1621 bytes --]

Hello all,
I've found a minor problem related to the rt-tests package, but that maybe
is slightly more generic and wanted to report and in case understand which
approach would be best to fix it (and contribute a patch then if possible).

The specific problem: rt-tests, besides a few compiled executables,
installs also two tools which are written in Python, namely hwlatdetect and
get_cyclictest_snapshot. These are Python scripts which get installed in
the PYLIB and then a symbolic link is done from /usr/bin to them, so ie we
have: /usr/bin/hwlatdetect ->
/usr/lib/python3.10/site-packages//hwlatdetect.py

Needless to say that once the pycompile.py script is run and original .py
files are removed, of course in the "PYC only" setup (which is the
default), then the link is broken and it will not work.
Additionally that pyc files will not work for scripts like that, so just
somehow fixing the link mentioned above would anyhow not work, as you need
to execute them eventually by explicitly naming the interpreter.

I could imagine that we could have some similar other cases when we are
installing Python scripts directly executed in PYLIB and not modules. But
I'm not sure that installing the scripts there is a good idea as well,
comparing with other distros that doesn't seem to be the case.
So maybe the problem is really in rt-tests after all and the location those
scripts get installed and I may just fix it that way (and contact upstream)?
Otherwise should it be more generic maybe could make sense to have a
facility to list some files that should not be converted to PYC?

Thanks for any input,
Federico

[-- Attachment #1.2: Type: text/html, Size: 1918 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Python pyc only and problem with scripts

[Marcus Hoffmann]

Hi Federico,

On 08.08.22 09:40, Federico Pellegrin wrote:
> 
> 
> Hello all,
> I've found a minor problem related to the rt-tests package, but that 
> maybe is slightly more generic and wanted to report and in case 
> understand which approach would be best to fix it (and contribute a 
> patch then if possible).

I've noticed a similar problem for pip's PEP517 backend as well, which 
normally calls some internal helper script via python subprocess (and 
uses the .py name for that). This is fixable by using the .pyc suffix 
instead in that case though.

> 
> The specific problem: rt-tests, besides a few compiled executables, 
> installs also two tools which are written in Python, namely hwlatdetect 
> and get_cyclictest_snapshot. These are Python scripts which get 
> installed in the PYLIB and then a symbolic link is done from /usr/bin to 
> them, so ie we have: /usr/bin/hwlatdetect -> 
> /usr/lib/python3.10/site-packages//hwlatdetect.py
> 
> Needless to say that once the pycompile.py script is run and original 
> .py files are removed, of course in the "PYC only" setup (which is the 
> default), then the link is broken and it will not work.
> Additionally that pyc files will not work for scripts like that, so just 
> somehow fixing the link mentioned above would anyhow not work, as you 
> need to execute them eventually by explicitly naming the interpreter.
> 
> I could imagine that we could have some similar other cases when we are 
> installing Python scripts directly executed in PYLIB and not modules. 
> But I'm not sure that installing the scripts there is a good idea as 
> well, comparing with other distros that doesn't seem to be the case.
> So maybe the problem is really in rt-tests after all and the location 
> those scripts get installed and I may just fix it that way (and contact 
> upstream)?
> Otherwise should it be more generic maybe could make sense to have a 
> facility to list some files that should not be converted to PYC?

I honestly don't know what the best generic fix for this is, or if it 
even makes sense to try doing that.

> 
> Thanks for any input,
> Federico
> 

Best,
Marcus
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Python pyc only and problem with scripts

[Thomas Petazzoni via buildroot]

Hello Marcus,

On Mon, 8 Aug 2022 12:05:54 +0200
Marcus Hoffmann <marcus.hoffmann@othermo.de> wrote:

> I've noticed a similar problem for pip's PEP517 backend as well, which 
> normally calls some internal helper script via python subprocess (and 
> uses the .py name for that). This is fixable by using the .pyc suffix 
> instead in that case though.

Do you have some details on this?

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Python pyc only and problem with scripts

[Thomas Petazzoni via buildroot]

Hello Federico,

On Mon, 8 Aug 2022 09:40:02 +0200
Federico Pellegrin <fede@evolware.org> wrote:

> I've found a minor problem related to the rt-tests package, but that maybe
> is slightly more generic and wanted to report and in case understand which
> approach would be best to fix it (and contribute a patch then if possible).
> 
> The specific problem: rt-tests, besides a few compiled executables,
> installs also two tools which are written in Python, namely hwlatdetect and
> get_cyclictest_snapshot. These are Python scripts which get installed in
> the PYLIB and then a symbolic link is done from /usr/bin to them, so ie we
> have: /usr/bin/hwlatdetect ->
> /usr/lib/python3.10/site-packages//hwlatdetect.py
> 
> Needless to say that once the pycompile.py script is run and original .py
> files are removed, of course in the "PYC only" setup (which is the
> default), then the link is broken and it will not work.
> Additionally that pyc files will not work for scripts like that, so just
> somehow fixing the link mentioned above would anyhow not work, as you need
> to execute them eventually by explicitly naming the interpreter.
> 
> I could imagine that we could have some similar other cases when we are
> installing Python scripts directly executed in PYLIB and not modules. But
> I'm not sure that installing the scripts there is a good idea as well,
> comparing with other distros that doesn't seem to be the case.
> So maybe the problem is really in rt-tests after all and the location those
> scripts get installed and I may just fix it that way (and contact upstream)?
> Otherwise should it be more generic maybe could make sense to have a
> facility to list some files that should not be converted to PYC?

Thanks for your bug report.

My initial gut feeling would indeed be that it is strange to install
scripts in /usr/lib/python, and they should be installed in /usr/bin
instead, so that /usr/lib/python is only for Python modules that can be
byte-compiled. However, I am not an expert at all in terms of Python
best practices and latest standards.

But from a Buildroot perspective, it really makes a lot of sense to be
able to do the byte-compilation at build-time (to save on startup time)
and drop the .py files (to save on filesystem footprint).

For the rt-tests case, I would therefore suggest to move those scripts
to /usr/bin.

Of course, if that problem shows up in more packages, then yes we could
think of a mechanism for packages to declare a number of py files that
should not be byte-compiled, and that should be preserved on the target.

Best regards,

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Python pyc only and problem with scripts

[Marcus Hoffmann]

Hi Thomas,

On 08.08.22 12:17, Thomas Petazzoni via buildroot wrote:
> Hello Marcus,
> 
> On Mon, 8 Aug 2022 12:05:54 +0200
> Marcus Hoffmann <marcus.hoffmann@othermo.de> wrote:
> 
>> I've noticed a similar problem for pip's PEP517 backend as well, which
>> normally calls some internal helper script via python subprocess (and
>> uses the .py name for that). This is fixable by using the .pyc suffix
>> instead in that case though.
> 
> Do you have some details on this?

Sorry, I hadn't really gotten around to properly reporting but here's 
some details:

https://github.com/pypa/pip/blob/main/src/pip/_vendor/pep517/in_process/__init__.py#L13 
returns the path to the .py file in question and is called by

https://github.com/pypa/pip/blob/1880f4a58b4e4ed577a6337d666767784f6831d5/src/pip/_vendor/pep517/wrappers.py#L320-L326 
via the subprocess call.

Why they are doing that, I couldn't figure out in the time I spent 
looking at this.

This is happening for us when we `pip install` an internal package that 
uses flit[1] as it's pep517 build backend.


[1]: https://github.com/pypa/flit

> 
> Thomas

Marcus
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Python pyc only and problem with scripts

[Thomas Petazzoni via buildroot]

Hello Marcus,

On Mon, 8 Aug 2022 12:30:10 +0200
Marcus Hoffmann <marcus.hoffmann@othermo.de> wrote:

> Sorry, I hadn't really gotten around to properly reporting but here's 
> some details:
> 
> https://github.com/pypa/pip/blob/main/src/pip/_vendor/pep517/in_process/__init__.py#L13 
> returns the path to the .py file in question and is called by
> 
> https://github.com/pypa/pip/blob/1880f4a58b4e4ed577a6337d666767784f6831d5/src/pip/_vendor/pep517/wrappers.py#L320-L326 
> via the subprocess call.
> 
> Why they are doing that, I couldn't figure out in the time I spent 
> looking at this.
> 
> This is happening for us when we `pip install` an internal package that 
> uses flit[1] as it's pep517 build backend.
> 
> [1]: https://github.com/pypa/flit

Ah, but you're doing "pip install" on the target?

Without saying it's not supported in Buildroot, it's not really the
recommended way of doing things: we encourage to create proper
Buildroot packages for Python modules.

Of course, since we support installing pip on the target, ideally this
should also work, but I'm not too surprised it can be broken for some
packages as I'm not sure many people use "pip install" on the target in
a Buildroot context.

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Python pyc only and problem with scripts

[Federico Pellegrin]

[-- Attachment #1.1: Type: text/plain, Size: 3884 bytes --]

Hi Thomas, Marcus,
Many thanks for your feedback and ideas!

I've looked a bit more now after your feedback online and indeed found that
likely it's mostly rt-tests (or other packages with a similar behaviour, as
mentioned by Marcus) that may be wrong-doing.

Indeed according to:
https://docs.python.org/3/install/#alternate-installation-unix-the-prefix-scheme
my understanding would be that they should indeed go to  *prefix*/bin and
not with the modules.
This link
https://python-packaging.readthedocs.io/en/latest/command-line-scripts.html#the-scripts-keyword-argument
also somehow confirms it I believe as also setuptools assume the
possibility to deliver them.
So after all if rt-tests (or similar) would follow that, Buildroot py->pyc
should be safe, assuming that those script will not be converted.

I'll try to contact the rt-tests team and see what they think of this and
come back with a patch for Buildroot!

Cheers,
Federico


Il giorno lun 8 ago 2022 alle ore 12:20 Thomas Petazzoni <
thomas.petazzoni@bootlin.com> ha scritto:

> Hello Federico,
>
> On Mon, 8 Aug 2022 09:40:02 +0200
> Federico Pellegrin <fede@evolware.org> wrote:
>
> > I've found a minor problem related to the rt-tests package, but that
> maybe
> > is slightly more generic and wanted to report and in case understand
> which
> > approach would be best to fix it (and contribute a patch then if
> possible).
> >
> > The specific problem: rt-tests, besides a few compiled executables,
> > installs also two tools which are written in Python, namely hwlatdetect
> and
> > get_cyclictest_snapshot. These are Python scripts which get installed in
> > the PYLIB and then a symbolic link is done from /usr/bin to them, so ie
> we
> > have: /usr/bin/hwlatdetect ->
> > /usr/lib/python3.10/site-packages//hwlatdetect.py
> >
> > Needless to say that once the pycompile.py script is run and original .py
> > files are removed, of course in the "PYC only" setup (which is the
> > default), then the link is broken and it will not work.
> > Additionally that pyc files will not work for scripts like that, so just
> > somehow fixing the link mentioned above would anyhow not work, as you
> need
> > to execute them eventually by explicitly naming the interpreter.
> >
> > I could imagine that we could have some similar other cases when we are
> > installing Python scripts directly executed in PYLIB and not modules. But
> > I'm not sure that installing the scripts there is a good idea as well,
> > comparing with other distros that doesn't seem to be the case.
> > So maybe the problem is really in rt-tests after all and the location
> those
> > scripts get installed and I may just fix it that way (and contact
> upstream)?
> > Otherwise should it be more generic maybe could make sense to have a
> > facility to list some files that should not be converted to PYC?
>
> Thanks for your bug report.
>
> My initial gut feeling would indeed be that it is strange to install
> scripts in /usr/lib/python, and they should be installed in /usr/bin
> instead, so that /usr/lib/python is only for Python modules that can be
> byte-compiled. However, I am not an expert at all in terms of Python
> best practices and latest standards.
>
> But from a Buildroot perspective, it really makes a lot of sense to be
> able to do the byte-compilation at build-time (to save on startup time)
> and drop the .py files (to save on filesystem footprint).
>
> For the rt-tests case, I would therefore suggest to move those scripts
> to /usr/bin.
>
> Of course, if that problem shows up in more packages, then yes we could
> think of a mechanism for packages to declare a number of py files that
> should not be byte-compiled, and that should be preserved on the target.
>
> Best regards,
>
> Thomas
> --
> Thomas Petazzoni, co-owner and CEO, Bootlin
> Embedded Linux and Kernel engineering and training
> https://bootlin.com
>

[-- Attachment #1.2: Type: text/html, Size: 6352 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Python pyc only and problem with scripts

[Marcus Hoffmann]

Hi Thomas,

On 08.08.22 12:47, Thomas Petazzoni wrote:
> Hello Marcus,
> 
> On Mon, 8 Aug 2022 12:30:10 +0200
> Marcus Hoffmann <marcus.hoffmann@othermo.de> wrote:
> 
>> Sorry, I hadn't really gotten around to properly reporting but here's
>> some details:
>>
>> https://github.com/pypa/pip/blob/main/src/pip/_vendor/pep517/in_process/__init__.py#L13
>> returns the path to the .py file in question and is called by
>>
>> https://github.com/pypa/pip/blob/1880f4a58b4e4ed577a6337d666767784f6831d5/src/pip/_vendor/pep517/wrappers.py#L320-L326
>> via the subprocess call.
>>
>> Why they are doing that, I couldn't figure out in the time I spent
>> looking at this.
>>
>> This is happening for us when we `pip install` an internal package that
>> uses flit[1] as it's pep517 build backend.
>>
>> [1]: https://github.com/pypa/flit
> 
> Ah, but you're doing "pip install" on the target?

Yes, indeed.

> 
> Without saying it's not supported in Buildroot, it's not really the
> recommended way of doing things: we encourage to create proper
> Buildroot packages for Python modules.

I'm aware of that :-). This is used for development purposes, which 
sometimes needs access to real hardware as well as being able to easily 
install new python dependencies.

> 
> Of course, since we support installing pip on the target, ideally this
> should also work, but I'm not too surprised it can be broken for some
> packages as I'm not sure many people use "pip install" on the target in
> a Buildroot context.

(I ran into this at one more point: everything that wants to compile 
native modules as part of the package installation doesn't work for 
obvious reasons. Every pure-python package should ideally "just work" 
but the pep517 world is still somewhat new and somewhat broken :D)

But now I wonder where the right place to fix this particular bug is. 
Locally I've just patched the .py -> .pyc but for upstream buildroot 
we'd need a conditional solution depending on the setting fopr what of 
py/pyc gets installed.

Or do you think this is something that should be reported/solved in 
upstream pip?

> 
> Thomas
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Python pyc only and problem with scripts

[Arnout Vandecappelle]

  Hi Federico, all,

  [Please don't top-post, but reply inline like I do below. Note that this 
requires configuring your mailer to quote replies properly.]


On 08/08/2022 15:19, Federico Pellegrin wrote:
> 
> Hi Thomas, Marcus,
> Many thanks for your feedback and ideas!
> 
> I've looked a bit more now after your feedback online and indeed found that 
> likely it's mostly rt-tests (or other packages with a similar behaviour, as 
> mentioned by Marcus) that may be wrong-doing.

  Well, rt-tests is certainly not the only package that does that - I've used 
other (proprietary) software that does similar things.


> Indeed according to: 
> https://docs.python.org/3/install/#alternate-installation-unix-the-prefix-scheme 
> <https://docs.python.org/3/install/#alternate-installation-unix-the-prefix-scheme> 
> my understanding would be that they should indeed go to |/prefix//bin and not 
> with the modules.|
> |This link 
> https://python-packaging.readthedocs.io/en/latest/command-line-scripts.html#the-scripts-keyword-argument 
> <https://python-packaging.readthedocs.io/en/latest/command-line-scripts.html#the-scripts-keyword-argument>  
> also somehow confirms it I believe as also setuptools assume the possibility to 
> deliver them.

  I believe the proper way to install is to have a small script in /usr/bin/ 
that simply imports the full script (relying on PYTHONPATH) and calls its main 
function.

> So after all if rt-tests (or similar) would follow that, Buildroot py->pyc 
> should be safe, assuming that those script will not be converted.

  Given that rt-tests is not alone, I think it would be good if we could fix the 
issue on Buildroot side.

  I think the problem can be diagnosed when we encounter a .py file that is 
executable - normally they shouldn't be. We could simply leave those alone 
instead of removing them. Alternatively, perhaps it's possible to replace them 
with a symlink to the pyc file - although that may break things in horrible ways 
as well.

  In either case, it's nice we have this rt-tests failure, because it means we 
can have a runtime test for it!

  Regards,
  Arnout


> |
> |
> |
> |I'll try to contact the rt-tests team and see what they think of this and come 
> back with a patch for Buildroot!
> |
> |
> |
> |Cheers,|
> |Federico
> |
> |
> |
> 
> Il giorno lun 8 ago 2022 alle ore 12:20 Thomas Petazzoni 
> <thomas.petazzoni@bootlin.com <mailto:thomas.petazzoni@bootlin.com>> ha scritto:
> 
>     Hello Federico,
> 
>     On Mon, 8 Aug 2022 09:40:02 +0200
>     Federico Pellegrin <fede@evolware.org <mailto:fede@evolware.org>> wrote:
> 
>      > I've found a minor problem related to the rt-tests package, but that maybe
>      > is slightly more generic and wanted to report and in case understand which
>      > approach would be best to fix it (and contribute a patch then if possible).
>      >
>      > The specific problem: rt-tests, besides a few compiled executables,
>      > installs also two tools which are written in Python, namely hwlatdetect and
>      > get_cyclictest_snapshot. These are Python scripts which get installed in
>      > the PYLIB and then a symbolic link is done from /usr/bin to them, so ie we
>      > have: /usr/bin/hwlatdetect ->
>      > /usr/lib/python3.10/site-packages//hwlatdetect.py
>      >
>      > Needless to say that once the pycompile.py script is run and original .py
>      > files are removed, of course in the "PYC only" setup (which is the
>      > default), then the link is broken and it will not work.
>      > Additionally that pyc files will not work for scripts like that, so just
>      > somehow fixing the link mentioned above would anyhow not work, as you need
>      > to execute them eventually by explicitly naming the interpreter.
>      >
>      > I could imagine that we could have some similar other cases when we are
>      > installing Python scripts directly executed in PYLIB and not modules. But
>      > I'm not sure that installing the scripts there is a good idea as well,
>      > comparing with other distros that doesn't seem to be the case.
>      > So maybe the problem is really in rt-tests after all and the location those
>      > scripts get installed and I may just fix it that way (and contact upstream)?
>      > Otherwise should it be more generic maybe could make sense to have a
>      > facility to list some files that should not be converted to PYC?
> 
>     Thanks for your bug report.
> 
>     My initial gut feeling would indeed be that it is strange to install
>     scripts in /usr/lib/python, and they should be installed in /usr/bin
>     instead, so that /usr/lib/python is only for Python modules that can be
>     byte-compiled. However, I am not an expert at all in terms of Python
>     best practices and latest standards.
> 
>     But from a Buildroot perspective, it really makes a lot of sense to be
>     able to do the byte-compilation at build-time (to save on startup time)
>     and drop the .py files (to save on filesystem footprint).
> 
>     For the rt-tests case, I would therefore suggest to move those scripts
>     to /usr/bin.
> 
>     Of course, if that problem shows up in more packages, then yes we could
>     think of a mechanism for packages to declare a number of py files that
>     should not be byte-compiled, and that should be preserved on the target.
> 
>     Best regards,
> 
>     Thomas
>     -- 
>     Thomas Petazzoni, co-owner and CEO, Bootlin
>     Embedded Linux and Kernel engineering and training
>     https://bootlin.com <https://bootlin.com>
> 
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Python pyc only and problem with scripts

[Federico Pellegrin]

[-- Attachment #1.1: Type: text/plain, Size: 9633 bytes --]

Hi Arnout, all!

Il giorno lun 8 ago 2022 alle ore 22:48 Arnout Vandecappelle <arnout@mind.be>
ha scritto:

>
> > Indeed according to:
> >
> https://docs.python.org/3/install/#alternate-installation-unix-the-prefix-scheme
> > <
> https://docs.python.org/3/install/#alternate-installation-unix-the-prefix-scheme>
>
> > my understanding would be that they should indeed go to |/prefix//bin
> and not
> > with the modules.|
> > |This link
> >
> https://python-packaging.readthedocs.io/en/latest/command-line-scripts.html#the-scripts-keyword-argument
> > <
> https://python-packaging.readthedocs.io/en/latest/command-line-scripts.html#the-scripts-keyword-argument>
>
> > also somehow confirms it I believe as also setuptools assume the
> possibility to
> > deliver them.
>
>   I believe the proper way to install is to have a small script in
> /usr/bin/
> that simply imports the full script (relying on PYTHONPATH) and calls its
> main
> function.
>

Yes in principle that is something that you often see around, although I
guess is mostly pointed at uses when you would like to both reuse some code
as module and script. For Python scripts that don't have any potentially
shareable code it may be an overhead and indeed you can find many around
(will later try to catch some other in Buildroot besides rt-tests so we can
have some more use-cases which maybe show us some other details).


>
> > So after all if rt-tests (or similar) would follow that, Buildroot
> py->pyc
> > should be safe, assuming that those script will not be converted.
>
>   Given that rt-tests is not alone, I think it would be good if we could
> fix the
> issue on Buildroot side.
>
>   I think the problem can be diagnosed when we encounter a .py file that
> is
> executable - normally they shouldn't be. We could simply leave those alone
> instead of removing them. Alternatively, perhaps it's possible to replace
> them
> with a symlink to the pyc file - although that may break things in
> horrible ways
> as well.
>

The first solution to not delete those py files (or not even generate pyc)
sounds good to me in principle!
The second one to put a symlink to the pyc file will not work, as in that
case you just get an error from the shell since the interpreter is not
known if you try to execute it directly.


As for implementing the first solution I gave a fast look:

-) Not to remove the executable .py should be very easy, we should just add
a check to the find that removes them in PYTHON3_REMOVE_PY_FILES, something
along the lines of:

diff -ruN a/python3.mk b/python3.mk
--- a/python3.mk        2022-08-02 03:59:45.897891945 +0200
+++ b/python3.mk        2022-08-09 04:36:40.798305233 +0200
@@ -264,7 +264,7 @@

 ifeq ($(BR2_PACKAGE_PYTHON3_PYC_ONLY),y)
 define PYTHON3_REMOVE_PY_FILES
-       find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name
'*.py' \
+       find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name
'*.py' \! -executable \
                $(if $(strip $(KEEP_PYTHON_PY_FILES)),-not \( $(call
finddirclauses,$(TARGET_DIR),$(KEEP_PYTHON_PY_FILES)) \) ) \
                -print0 | \
                xargs -0 --no-run-if-empty rm -f

I did a small test with the above change and that would solve rt-tests
case. On the other side I see that from a basic Python config I did this
means also including/keeping a bunch of other files, which I'm not sure if
they are executable by purpose or by chance/negligence, for example:

./config-3.10-x86_64-linux-gnu/python-config.py
./profile.py
./tarfile.py
./quopri.py
./cgi.py
./socket.py
./uu.py
./timeit.py
./tabnanny.py
./turtledemo/paint.py
./turtledemo/penrose.py
./turtledemo/minimal_hanoi.py
./turtledemo/forest.py
./turtledemo/lindenmayer.py
./turtledemo/fractalcurves.py
./turtledemo/yinyang.py
./turtledemo/peace.py
./turtledemo/planet_and_moon.py
./turtledemo/tree.py
./turtledemo/__main__.py
./turtledemo/sorting_animate.py
./turtledemo/clock.py
./turtledemo/bytedesign.py
./webbrowser.py
./smtplib.py
./encodings/rot_13.py
./cProfile.py
./smtpd.py
./trace.py
./pydoc.py
./base64.py
./site-packages/hwlatdetect.py
./site-packages/get_cyclictest_snapshot.py
./platform.py
./pdb.py

Just a fast look and some (ie. pdb.py) could be interesting to have as .py
indeed, as this way you can easily invoke the Python debugger (otherwise
with just the pyc you need to manually import and call main as the pyc will
not execute directly). Same ie. for smtpd or base64 or webbrowser as you
can then use them immediately (and with an additional command line parsing
part). So after all maybe we were losing with the deletion of the py files
also some other possibly useful delivered features.

-) If we would like not to create the pyc at all as well it may be a little
more work in recent versions. Indeed recently (end of July, so in master)
Buildroot went from using the custom internal script
"support/scripts/pycompile.py" to the Python shipped "compileall.py".
For the old internal pycompile.py it would be very easy to just add a check
on the executable-ness and skip the path (as we have a bunch of other
checks, ie. is a link). But for the compileall.py as far as I can see there
is no option to skip them and therefore patching this may be a bit more
complex.


But maybe for the scripts, which I guess should be a relatively small
number of cases (in the above example out of 550 pyc files 35 would have
the py left in place due to execution bit), just not removing the .py so
they can still work correctly could be good enough after all ? (so, for
executable Python files, one ends up with both .py and .pyc)


If you think the first point proposed makes sense, I can send an official
patch.

Cheers,
Federico




>   In either case, it's nice we have this rt-tests failure, because it
> means we
> can have a runtime test for it!
>
>   Regards,
>   Arnout
>
>
> > |
> > |
> > |
> > |I'll try to contact the rt-tests team and see what they think of this
> and come
> > back with a patch for Buildroot!
> > |
> > |
> > |
> > |Cheers,|
> > |Federico
> > |
> > |
> > |
> >
> > Il giorno lun 8 ago 2022 alle ore 12:20 Thomas Petazzoni
> > <thomas.petazzoni@bootlin.com <mailto:thomas.petazzoni@bootlin.com>> ha
> scritto:
> >
> >     Hello Federico,
> >
> >     On Mon, 8 Aug 2022 09:40:02 +0200
> >     Federico Pellegrin <fede@evolware.org <mailto:fede@evolware.org>>
> wrote:
> >
> >      > I've found a minor problem related to the rt-tests package, but
> that maybe
> >      > is slightly more generic and wanted to report and in case
> understand which
> >      > approach would be best to fix it (and contribute a patch then if
> possible).
> >      >
> >      > The specific problem: rt-tests, besides a few compiled
> executables,
> >      > installs also two tools which are written in Python, namely
> hwlatdetect and
> >      > get_cyclictest_snapshot. These are Python scripts which get
> installed in
> >      > the PYLIB and then a symbolic link is done from /usr/bin to them,
> so ie we
> >      > have: /usr/bin/hwlatdetect ->
> >      > /usr/lib/python3.10/site-packages//hwlatdetect.py
> >      >
> >      > Needless to say that once the pycompile.py script is run and
> original .py
> >      > files are removed, of course in the "PYC only" setup (which is the
> >      > default), then the link is broken and it will not work.
> >      > Additionally that pyc files will not work for scripts like that,
> so just
> >      > somehow fixing the link mentioned above would anyhow not work, as
> you need
> >      > to execute them eventually by explicitly naming the interpreter.
> >      >
> >      > I could imagine that we could have some similar other cases when
> we are
> >      > installing Python scripts directly executed in PYLIB and not
> modules. But
> >      > I'm not sure that installing the scripts there is a good idea as
> well,
> >      > comparing with other distros that doesn't seem to be the case.
> >      > So maybe the problem is really in rt-tests after all and the
> location those
> >      > scripts get installed and I may just fix it that way (and contact
> upstream)?
> >      > Otherwise should it be more generic maybe could make sense to
> have a
> >      > facility to list some files that should not be converted to PYC?
> >
> >     Thanks for your bug report.
> >
> >     My initial gut feeling would indeed be that it is strange to install
> >     scripts in /usr/lib/python, and they should be installed in /usr/bin
> >     instead, so that /usr/lib/python is only for Python modules that can
> be
> >     byte-compiled. However, I am not an expert at all in terms of Python
> >     best practices and latest standards.
> >
> >     But from a Buildroot perspective, it really makes a lot of sense to
> be
> >     able to do the byte-compilation at build-time (to save on startup
> time)
> >     and drop the .py files (to save on filesystem footprint).
> >
> >     For the rt-tests case, I would therefore suggest to move those
> scripts
> >     to /usr/bin.
> >
> >     Of course, if that problem shows up in more packages, then yes we
> could
> >     think of a mechanism for packages to declare a number of py files
> that
> >     should not be byte-compiled, and that should be preserved on the
> target.
> >
> >     Best regards,
> >
> >     Thomas
> >     --
> >     Thomas Petazzoni, co-owner and CEO, Bootlin
> >     Embedded Linux and Kernel engineering and training
> >     https://bootlin.com <https://bootlin.com>
> >
> >
> > _______________________________________________
> > buildroot mailing list
> > buildroot@buildroot.org
> > https://lists.buildroot.org/mailman/listinfo/buildroot
>

[-- Attachment #1.2: Type: text/html, Size: 13067 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [External] - Re: Python pyc only and problem with scripts

[Vincent Fazio]

All

> -----Original Message-----
> From: buildroot <buildroot-bounces@buildroot.org> On Behalf Of Federico
> Pellegrin
> Sent: Monday, August 8, 2022 11:50 PM
> To: Arnout Vandecappelle <arnout@mind.be>
> Cc: Marcus Hoffmann <marcus.hoffmann@othermo.de>; Thomas Petazzoni
> <thomas.petazzoni@bootlin.com>; buildroot <buildroot@buildroot.org>
> Subject: [External] - Re: [Buildroot] Python pyc only and problem with scripts
> 
> 
> Hi Arnout, all!
> 
> 
> Il giorno lun 8 ago 2022 alle ore 22:48 Arnout Vandecappelle
> <arnout@mind.be <mailto:arnout@mind.be> > ha scritto:
> 
> 
> 
> 	> Indeed according to:
> 	>
> https://docs.python.org/3/install/#alternate-installation-unix-the-prefi
> x-scheme
> <https://docs.python.org/3/install/#alternate-installation-unix-the-pref
> ix-scheme>
> 	>
> <https://docs.python.org/3/install/#alternate-installation-unix-the-pref
> ix-scheme>
> 	> my understanding would be that they should indeed go to
> |/prefix//bin and not
> 	> with the modules.|
> 	> |This link
> 	>
> https://python-packaging.readthedocs.io/en/latest/command-line-scripts.h
> tml#the-scripts-keyword-argument
> <https://python-packaging.readthedocs.io/en/latest/command-line-scripts.
> html#the-scripts-keyword-argument>
> 	>
> <https://python-packaging.readthedocs.io/en/latest/command-line-scripts.
> html#the-scripts-keyword-argument>
> 	> also somehow confirms it I believe as also setuptools assume
> the possibility to
> 	> deliver them.
> 
> 	  I believe the proper way to install is to have a small script
> in /usr/bin/
> 	that simply imports the full script (relying on PYTHONPATH) and
> calls its main
> 	function.
> 
> 
> 
> Yes in principle that is something that you often see around, although
> I guess is mostly pointed at uses when you would like to both reuse some
> code as module and script. For Python scripts that don't have any
> potentially shareable code it may be an overhead and indeed you can find
> many around (will later try to catch some other in Buildroot besides
> rt-tests so we can have some more use-cases which maybe show us some
> other details).
> 
> 
> 
> 
> 	> So after all if rt-tests (or similar) would follow that,
> Buildroot py->pyc
> 	> should be safe, assuming that those script will not be
> converted.
> 
> 	  Given that rt-tests is not alone, I think it would be good if
> we could fix the
> 	issue on Buildroot side.
> 
> 	  I think the problem can be diagnosed when we encounter a .py
> file that is
> 	executable - normally they shouldn't be. We could simply leave
> those alone
> 	instead of removing them. Alternatively, perhaps it's possible
> to replace them
> 	with a symlink to the pyc file - although that may break things
> in horrible ways
> 	as well.
> 
> 
> 
> The first solution to not delete those py files (or not even generate
> pyc) sounds good to me in principle!

I'd argue that these modules should have a wrapper generated that imports the module and
executes it (as is standard) or that it should be invoked via `python3 -m <module> <args>`
instead of relying on a symlink. The pyc file will be imported as appropriate in these cases.

> 
> The second one to put a symlink to the pyc file will not work, as in
> that case you just get an error from the shell since the interpreter is
> not known if you try to execute it directly.
> 
> 
> 
> As for implementing the first solution I gave a fast look:
> 
> -) Not to remove the executable .py should be very easy, we should just
> add a check to the find that removes them in PYTHON3_REMOVE_PY_FILES,
> something along the lines of:
> 
> diff -ruN a/python3.mk <http://python3.mk>  b/python3.mk
> <http://python3.mk>
> --- a/python3.mk <http://python3.mk>         2022-08-02
> 03:59:45.897891945 +0200
> +++ b/python3.mk <http://python3.mk>         2022-08-09
> 04:36:40.798305233 +0200
> @@ -264,7 +264,7 @@
> 
>  ifeq ($(BR2_PACKAGE_PYTHON3_PYC_ONLY),y)
>  define PYTHON3_REMOVE_PY_FILES
> -       find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -
> name
> '*.py' \
> +       find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -
> name
> '*.py' \! -executable \
>                 $(if $(strip $(KEEP_PYTHON_PY_FILES)),-not \( $(call
> finddirclauses,$(TARGET_DIR),$(KEEP_PYTHON_PY_FILES)) \) ) \
>                 -print0 | \
>                 xargs -0 --no-run-if-empty rm -f
> 
> I did a small test with the above change and that would solve rt-tests
> case. On the other side I see that from a basic Python config I did this
> means also including/keeping a bunch of other files, which I'm not sure
> if they are executable by purpose or by chance/negligence, for example:
> 
> ./config-3.10-x86_64-linux-gnu/python-config.py
> ./profile.py
> ./tarfile.py
> ./quopri.py
> ./cgi.py
> ./socket.py
> ./uu.py
> ./timeit.py
> ./tabnanny.py
> ./turtledemo/paint.py
> ./turtledemo/penrose.py
> ./turtledemo/minimal_hanoi.py
> ./turtledemo/forest.py
> ./turtledemo/lindenmayer.py
> ./turtledemo/fractalcurves.py
> ./turtledemo/yinyang.py
> ./turtledemo/peace.py
> ./turtledemo/planet_and_moon.py
> ./turtledemo/tree.py
> ./turtledemo/__main__.py
> ./turtledemo/sorting_animate.py
> ./turtledemo/clock.py
> ./turtledemo/bytedesign.py
> ./webbrowser.py
> ./smtplib.py
> ./encodings/rot_13.py
> ./cProfile.py
> ./smtpd.py
> ./trace.py
> ./pydoc.py
> ./base64.py
> ./site-packages/hwlatdetect.py
> ./site-packages/get_cyclictest_snapshot.py
> ./platform.py
> ./pdb.py

Most, if not all, of these scripts support the `python -m <module> <args>` syntax which is important
since it then becomes unnecessary to know which explicit file provides the module you're trying to
execute. That said,  if you read PEP 3147, it's pretty clear that sourceless distributions are "legacy",
so anything we do is likely outside of the scope of what upstream intends to support. It may still be
worth having a discussion about long term support options

> 
> Just a fast look and some (ie. pdb.py) could be interesting to have as
> .py indeed, as this way you can easily invoke the Python debugger
> (otherwise with just the pyc you need to manually import and call main
> as the pyc will not execute directly). Same ie. for smtpd or base64 or
> webbrowser as you can then use them immediately (and with an additional
> command line parsing part). So after all maybe we were losing with the
> deletion of the py files also some other possibly useful delivered
> features.

See above.

It's also important to note that excluding executable files will not resolve the issue with PEP 517

https://github.com/pypa/pip/blob/main/src/pip/_vendor/pep517/in_process/__init__.py
```
return resources.path(__package__, '_in_process.py')
```

Where the path to a .py file is returned. This is probably fixable by stripping the extension and passing `-m` to the call in `wrappers.py`

```
            with _in_proc_script_path() as script:
                python = self.python_executable
                self._subprocess_runner(
                    [python, abspath(str(script)), hook_name, td],
                    cwd=self.source_dir,
                    extra_environ=extra_environ
                )
```

The other option is to explicitly exclude this special file from the compileall call via the `-x` argument.

> 
> 
> -) If we would like not to create the pyc at all as well it may be a
> little more work in recent versions. Indeed recently (end of July, so in
> master) Buildroot went from using the custom internal script
> "support/scripts/pycompile.py" to the Python shipped "compileall.py".
> For the old internal pycompile.py it would be very easy to just add a
> check on the executable-ness and skip the path (as we have a bunch of
> other checks, ie. is a link). But for the compileall.py as far as I can
> see there is no option to skip them and therefore patching this may be a
> bit more complex.
> 
> 
> But maybe for the scripts, which I guess should be a relatively small
> number of cases (in the above example out of 550 pyc files 35 would have
> the py left in place due to execution bit), just not removing the .py so
> they can still work correctly could be good enough after all ? (so, for
> executable Python files, one ends up with both .py and .pyc)
> 

Shouldn't known broken cases either be patched to work or prevent sourceless distribution?

> 
> 
> If you think the first point proposed makes sense, I can send an
> official patch.
> 
> 
> Cheers,
> Federico
> 
> 
> 
> 
> 
> 	  In either case, it's nice we have this rt-tests failure,
> because it means we
> 	can have a runtime test for it!
> 
> 	  Regards,
> 	  Arnout
> 
> 
> 	> |
> 	> |
> 	> |
> 	> |I'll try to contact the rt-tests team and see what they think
> of this and come
> 	> back with a patch for Buildroot!
> 	> |
> 	> |
> 	> |
> 	> |Cheers,|
> 	> |Federico
> 	> |
> 	> |
> 	> |
> 	>
> 	> Il giorno lun 8 ago 2022 alle ore 12:20 Thomas Petazzoni
> 	> <thomas.petazzoni@bootlin.com
> <mailto:thomas.petazzoni@bootlin.com>
> <mailto:thomas.petazzoni@bootlin.com
> <mailto:thomas.petazzoni@bootlin.com> >> ha scritto:
> 	>
> 	>     Hello Federico,
> 	>
> 	>     On Mon, 8 Aug 2022 09:40:02 +0200
> 	>     Federico Pellegrin <fede@evolware.org
> <mailto:fede@evolware.org>  <mailto:fede@evolware.org
> <mailto:fede@evolware.org> >> wrote:
> 	>
> 	>      > I've found a minor problem related to the rt-tests
> package, but that maybe
> 	>      > is slightly more generic and wanted to report and in
> case understand which
> 	>      > approach would be best to fix it (and contribute a
> patch then if possible).
> 	>      >
> 	>      > The specific problem: rt-tests, besides a few compiled
> executables,
> 	>      > installs also two tools which are written in Python,
> namely hwlatdetect and
> 	>      > get_cyclictest_snapshot. These are Python scripts which
> get installed in
> 	>      > the PYLIB and then a symbolic link is done from
> /usr/bin to them, so ie we
> 	>      > have: /usr/bin/hwlatdetect ->
> 	>      > /usr/lib/python3.10/site-packages//hwlatdetect.py
> 	>      >
> 	>      > Needless to say that once the pycompile.py script is
> run and original .py
> 	>      > files are removed, of course in the "PYC only" setup
> (which is the
> 	>      > default), then the link is broken and it will not work.
> 	>      > Additionally that pyc files will not work for scripts
> like that, so just
> 	>      > somehow fixing the link mentioned above would anyhow
> not work, as you need
> 	>      > to execute them eventually by explicitly naming the
> interpreter.
> 	>      >
> 	>      > I could imagine that we could have some similar other
> cases when we are
> 	>      > installing Python scripts directly executed in PYLIB
> and not modules. But
> 	>      > I'm not sure that installing the scripts there is a
> good idea as well,
> 	>      > comparing with other distros that doesn't seem to be
> the case.
> 	>      > So maybe the problem is really in rt-tests after all
> and the location those
> 	>      > scripts get installed and I may just fix it that way
> (and contact upstream)?
> 	>      > Otherwise should it be more generic maybe could make
> sense to have a
> 	>      > facility to list some files that should not be
> converted to PYC?
> 	>
> 	>     Thanks for your bug report.
> 	>
> 	>     My initial gut feeling would indeed be that it is strange
> to install
> 	>     scripts in /usr/lib/python, and they should be installed
> in /usr/bin
> 	>     instead, so that /usr/lib/python is only for Python
> modules that can be
> 	>     byte-compiled. However, I am not an expert at all in terms
> of Python
> 	>     best practices and latest standards.
> 	>
> 	>     But from a Buildroot perspective, it really makes a lot of
> sense to be
> 	>     able to do the byte-compilation at build-time (to save on
> startup time)
> 	>     and drop the .py files (to save on filesystem footprint).
> 	>
> 	>     For the rt-tests case, I would therefore suggest to move
> those scripts
> 	>     to /usr/bin.
> 	>
> 	>     Of course, if that problem shows up in more packages, then
> yes we could
> 	>     think of a mechanism for packages to declare a number of
> py files that
> 	>     should not be byte-compiled, and that should be preserved
> on the target.
> 	>
> 	>     Best regards,
> 	>
> 	>     Thomas
> 	>     --
> 	>     Thomas Petazzoni, co-owner and CEO, Bootlin
> 	>     Embedded Linux and Kernel engineering and training
> 	>     https://bootlin.com <https://bootlin.com>
> 	>
> 	>
> 	> _______________________________________________
> 	> buildroot mailing list
> 	> buildroot@buildroot.org <mailto:buildroot@buildroot.org>
> 	> https://lists.buildroot.org/mailman/listinfo/buildroot
> <https://lists.buildroot.org/mailman/listinfo/buildroot>
> 
> 
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [External] - Re: Python pyc only and problem with scripts

[Yann E. MORIN]

Vincent, All,

On 2022-08-09 12:54 +0000, Vincent Fazio spake thusly:
[--SNIP--]
> That said,  if you read PEP 3147, it's pretty clear that sourceless distributions are "legacy",
> so anything we do is likely outside of the scope of what upstream intends to support. It may still be
> worth having a discussion about long term support options

Ref;
    https://peps.pythondiscord.com/pep-3147/#case-4-legacy-pyc-files-and-source-less-imports

    ---8<---
    Case 4: legacy pyc files and source-less imports

    Python will ignore all legacy pyc files when a source file exists
    next to it. In other words, if a foo.pyc file exists next to the
    foo.py file, the pyc file will be ignored in all cases

    In order to continue to support source-less distributions though,
    if the source file is missing, Python will import a lone pyc file
    if it lives where the source file would have been.
    ---8<---

Vincent, Thomas, and I have had a quick discussion on IRC, which boils
down to reading the PEP3147 in two (or three) ways:

  - Vincent: legacy (pyc files and source-less imports)
    => both pyc files on their own *and* source-less distributions
       are considered legacy

  - Yann: (legacy pyc files) and (source-less imports)
    => only pyc files on thier own are legacy; source-less distribution
       is still supported

  - Thomas: in-between
    => "Because we can't break the world overnight, we continue to
       support this, but it sucks"

So, Vincent has opened an issue to ask for clarification:
    https://github.com/python/cpython/issues/95827

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Python pyc only and problem with scripts

[Marcus Hoffmann]

On 08.08.22 12:30, Marcus Hoffmann wrote:
> Hi Thomas,
> 
> On 08.08.22 12:17, Thomas Petazzoni via buildroot wrote:
>> Hello Marcus,
>>
>> On Mon, 8 Aug 2022 12:05:54 +0200
>> Marcus Hoffmann <marcus.hoffmann@othermo.de> wrote:
>>
>>> I've noticed a similar problem for pip's PEP517 backend as well, which
>>> normally calls some internal helper script via python subprocess (and
>>> uses the .py name for that). This is fixable by using the .pyc suffix
>>> instead in that case though.
>>
>> Do you have some details on this?
> 

I filed https://github.com/pypa/pep517/issues/148 now. It seems that 
openwrt has run into this problem as well but the patch the submitted 
didn't get any feedback for quite a while now.


> 
>>
>> Thomas
> 
> Marcus
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Python pyc only and problem with scripts

[James Hilliard]

On Mon, Aug 8, 2022 at 1:40 AM Federico Pellegrin <fede@evolware.org> wrote:
>
>
>
> Hello all,
> I've found a minor problem related to the rt-tests package, but that maybe is slightly more generic and wanted to report and in case understand which approach would be best to fix it (and contribute a patch then if possible).
>
> The specific problem: rt-tests, besides a few compiled executables, installs also two tools which are written in Python, namely hwlatdetect and get_cyclictest_snapshot. These are Python scripts which get installed in the PYLIB and then a symbolic link is done from /usr/bin to them, so ie we have: /usr/bin/hwlatdetect -> /usr/lib/python3.10/site-packages//hwlatdetect.py

This looks wrong, hwlatdetect is just a script, it's not a python
package so it shouldn't even go in site-packages at all, just directly
install it to /usr/bin/hwlatdetect.

>
> Needless to say that once the pycompile.py script is run and original .py files are removed, of course in the "PYC only" setup (which is the default), then the link is broken and it will not work.
> Additionally that pyc files will not work for scripts like that, so just somehow fixing the link mentioned above would anyhow not work, as you need to execute them eventually by explicitly naming the interpreter.
>
> I could imagine that we could have some similar other cases when we are installing Python scripts directly executed in PYLIB and not modules. But I'm not sure that installing the scripts there is a good idea as well, comparing with other distros that doesn't seem to be the case.

Our infrastructure should handle entry points already which generates
wrappers that shouldn't care about the extension of the module in
site-packages:
https://packaging.python.org/en/latest/specifications/entry-points/

> So maybe the problem is really in rt-tests after all and the location those scripts get installed and I may just fix it that way (and contact upstream)?

Yeah, not sure why rt-tests would be touching PYLIB in the first place
if it isn't installing a python package.

> Otherwise should it be more generic maybe could make sense to have a facility to list some files that should not be converted to PYC?
>
> Thanks for any input,
> Federico
>
>
>
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Python pyc only and problem with scripts

[James Hilliard]

On Mon, Aug 8, 2022 at 2:48 PM Arnout Vandecappelle <arnout@mind.be> wrote:
>
>   Hi Federico, all,
>
>   [Please don't top-post, but reply inline like I do below. Note that this
> requires configuring your mailer to quote replies properly.]
>
>
> On 08/08/2022 15:19, Federico Pellegrin wrote:
> >
> > Hi Thomas, Marcus,
> > Many thanks for your feedback and ideas!
> >
> > I've looked a bit more now after your feedback online and indeed found that
> > likely it's mostly rt-tests (or other packages with a similar behaviour, as
> > mentioned by Marcus) that may be wrong-doing.
>
>   Well, rt-tests is certainly not the only package that does that - I've used
> other (proprietary) software that does similar things.
>
>
> > Indeed according to:
> > https://docs.python.org/3/install/#alternate-installation-unix-the-prefix-scheme
> > <https://docs.python.org/3/install/#alternate-installation-unix-the-prefix-scheme>
> > my understanding would be that they should indeed go to |/prefix//bin and not
> > with the modules.|
> > |This link
> > https://python-packaging.readthedocs.io/en/latest/command-line-scripts.html#the-scripts-keyword-argument
> > <https://python-packaging.readthedocs.io/en/latest/command-line-scripts.html#the-scripts-keyword-argument>
> > also somehow confirms it I believe as also setuptools assume the possibility to
> > deliver them.
>
>   I believe the proper way to install is to have a small script in /usr/bin/
> that simply imports the full script (relying on PYTHONPATH) and calls its main
> function.

That's basically how python package entry points work:
https://packaging.python.org/en/latest/specifications/entry-points/

>
> > So after all if rt-tests (or similar) would follow that, Buildroot py->pyc
> > should be safe, assuming that those script will not be converted.
>
>   Given that rt-tests is not alone, I think it would be good if we could fix the
> issue on Buildroot side.

The bug is in rt-tests, it's installing a script to site-packages when it should
just be installing it like we would for a shell script(as the script
isn't a python
package but rather just a standalone python script).

>
>   I think the problem can be diagnosed when we encounter a .py file that is
> executable - normally they shouldn't be. We could simply leave those alone
> instead of removing them. Alternatively, perhaps it's possible to replace them
> with a symlink to the pyc file - although that may break things in horrible ways
> as well.

I don't think it's normal to execute .py files directly from
site-packages, packages
can provide entry points if needed which will handle both .py and .pyc
files without
issues.

>
>   In either case, it's nice we have this rt-tests failure, because it means we
> can have a runtime test for it!
>
>   Regards,
>   Arnout
>
>
> > |
> > |
> > |
> > |I'll try to contact the rt-tests team and see what they think of this and come
> > back with a patch for Buildroot!
> > |
> > |
> > |
> > |Cheers,|
> > |Federico
> > |
> > |
> > |
> >
> > Il giorno lun 8 ago 2022 alle ore 12:20 Thomas Petazzoni
> > <thomas.petazzoni@bootlin.com <mailto:thomas.petazzoni@bootlin.com>> ha scritto:
> >
> >     Hello Federico,
> >
> >     On Mon, 8 Aug 2022 09:40:02 +0200
> >     Federico Pellegrin <fede@evolware.org <mailto:fede@evolware.org>> wrote:
> >
> >      > I've found a minor problem related to the rt-tests package, but that maybe
> >      > is slightly more generic and wanted to report and in case understand which
> >      > approach would be best to fix it (and contribute a patch then if possible).
> >      >
> >      > The specific problem: rt-tests, besides a few compiled executables,
> >      > installs also two tools which are written in Python, namely hwlatdetect and
> >      > get_cyclictest_snapshot. These are Python scripts which get installed in
> >      > the PYLIB and then a symbolic link is done from /usr/bin to them, so ie we
> >      > have: /usr/bin/hwlatdetect ->
> >      > /usr/lib/python3.10/site-packages//hwlatdetect.py
> >      >
> >      > Needless to say that once the pycompile.py script is run and original .py
> >      > files are removed, of course in the "PYC only" setup (which is the
> >      > default), then the link is broken and it will not work.
> >      > Additionally that pyc files will not work for scripts like that, so just
> >      > somehow fixing the link mentioned above would anyhow not work, as you need
> >      > to execute them eventually by explicitly naming the interpreter.
> >      >
> >      > I could imagine that we could have some similar other cases when we are
> >      > installing Python scripts directly executed in PYLIB and not modules. But
> >      > I'm not sure that installing the scripts there is a good idea as well,
> >      > comparing with other distros that doesn't seem to be the case.
> >      > So maybe the problem is really in rt-tests after all and the location those
> >      > scripts get installed and I may just fix it that way (and contact upstream)?
> >      > Otherwise should it be more generic maybe could make sense to have a
> >      > facility to list some files that should not be converted to PYC?
> >
> >     Thanks for your bug report.
> >
> >     My initial gut feeling would indeed be that it is strange to install
> >     scripts in /usr/lib/python, and they should be installed in /usr/bin
> >     instead, so that /usr/lib/python is only for Python modules that can be
> >     byte-compiled. However, I am not an expert at all in terms of Python
> >     best practices and latest standards.
> >
> >     But from a Buildroot perspective, it really makes a lot of sense to be
> >     able to do the byte-compilation at build-time (to save on startup time)
> >     and drop the .py files (to save on filesystem footprint).
> >
> >     For the rt-tests case, I would therefore suggest to move those scripts
> >     to /usr/bin.
> >
> >     Of course, if that problem shows up in more packages, then yes we could
> >     think of a mechanism for packages to declare a number of py files that
> >     should not be byte-compiled, and that should be preserved on the target.
> >
> >     Best regards,
> >
> >     Thomas
> >     --
> >     Thomas Petazzoni, co-owner and CEO, Bootlin
> >     Embedded Linux and Kernel engineering and training
> >     https://bootlin.com <https://bootlin.com>
> >
> >
> > _______________________________________________
> > buildroot mailing list
> > buildroot@buildroot.org
> > https://lists.buildroot.org/mailman/listinfo/buildroot
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot