Missing venv module in Python 3

Missing venv module in Python 3

[Jonathan Haigh]

[-- Attachment #1: Type: text/plain, Size: 677 bytes --]

Hello,

I have noticed that the venv module is not available with Python 3 in Yocto.
This is possibly because there is no entry for it in openembedded-core/meta/recipes-devtools/python/python3/python3-manifest.json

Could  you please provide guidance or a fix to make the module available?

Thank you

Jonathan Haigh on behalf of Hugues Kamba
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

[-- Attachment #2: Outlook-o5x4232g.png --]
[-- Type: image/png, Size: 2551 bytes --]

[-- Attachment #3: Outlook-3dhoboya.png --]
[-- Type: image/png, Size: 2551 bytes --]

Re: Missing venv module in Python 3

[Burton, Ross]

venv is packaged in python3-pyvenv.  Patches welcome to get that
installed out of the box.

Ross
On Thu, 18 Oct 2018 at 14:35, Jonathan Haigh <Jonathan.Haigh@arm.com> wrote:
>
> Hello,
>
> I have noticed that the venv module is not available with Python 3 in Yocto.
> This is possibly because there is no entry for it in openembedded-core/meta/recipes-devtools/python/python3/python3-manifest.json
>
> Could  you please provide guidance or a fix to make the module available?
>
> Thank you
>
> Jonathan Haigh on behalf of Hugues Kamba
> IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: Missing venv module in Python 3

[Jonathan Haigh]

[-- Attachment #1: Type: text/plain, Size: 3411 bytes --]

Hi Ross,
Thank you for you prompt reply.

Apologies if I was not clear enough in my previous message.

python3-pyvenv is indeed a package that gets built with python3. However, pyvenv is just a short script that imports the venv module. Including the python3-pyvenv package does not solve the issue as the creation of a virtual environment using "python3 -m venv my_env" or "pyvenv my_env" results in a error about the missing venv module.

See the entire content of pyvenv:
---------------------------------------------------
#! /usr/bin/python3.5
if __name__ == '__main__':
    import sys
    rc = 1
    try:
        import venv
        venv.main()
        rc = 0
    except Exception as e:
        print('Error: %s' % e, file=sys.stderr)
    sys.exit(rc)
----------------------------------------------------

The venv package (https://docs.python.org/3/library/venv.html) itself is missing although its source is available within Python 3.5 (https://www.python.org/ftp/python/3.5.6/Python-3.5.6.tgz). As mentioned earlier, it seems the issue is because there is no entry in the manifest from which the various Python 3 modules are created (https://github.com/openembedded/openembedded-core/blob/master/meta/recipes-devtools/python/python3/python3-manifest.json).

The manifest is generated from https://github.com/openembedded/openembedded-core/blob/master/meta/recipes-devtools/python/python3/create_manifest3.py, and it does not seem obvious to me how to go about correctly adding a new module to the manifest, using create_manifest3.py or otherwise.

Can you, or others on the mailing list, provide some guidance on how to go about adding the venv module to the manifest?

Regards,

Hugues/Jonathan


________________________________
From: Burton, Ross <ross.burton@intel.com>
Sent: 18 October 2018 14:56
To: Jonathan Haigh
Cc: OE-core; Jeremy Johnson; Alejandro Enedino Hernandez Samaniego; Hugues Kamba
Subject: Re: [OE-core] Missing venv module in Python 3

venv is packaged in python3-pyvenv.  Patches welcome to get that
installed out of the box.

Ross
On Thu, 18 Oct 2018 at 14:35, Jonathan Haigh <Jonathan.Haigh@arm.com> wrote:
>
> Hello,
>
> I have noticed that the venv module is not available with Python 3 in Yocto.
> This is possibly because there is no entry for it in openembedded-core/meta/recipes-devtools/python/python3/python3-manifest.json
>
> Could  you please provide guidance or a fix to make the module available?
>
> Thank you
>
> Jonathan Haigh on behalf of Hugues Kamba
> IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

[-- Attachment #2: Type: text/html, Size: 13113 bytes --]

Re: Missing venv module in Python 3

[Burton, Ross]

On Thu, 18 Oct 2018 at 16:19, Jonathan Haigh <Jonathan.Haigh@arm.com> wrote:
> The venv package (https://docs.python.org/3/library/venv.html) itself is missing although its source is available within Python 3.5 (https://www.python.org/ftp/python/3.5.6/Python-3.5.6.tgz). As mentioned earlier, it seems the issue is because there is no entry in the manifest from which the various Python 3 modules are created (https://github.com/openembedded/openembedded-core/blob/master/meta/recipes-devtools/python/python3/python3-manifest.json).
>
> The manifest is generated from https://github.com/openembedded/openembedded-core/blob/master/meta/recipes-devtools/python/python3/create_manifest3.py, and it does not seem obvious to me how to go about correctly adding a new module to the manifest, using create_manifest3.py or otherwise.

Ah, right.

In that case:

$ oe-pkgdata-util find-path */*venv*py
python3-tests: /usr/lib/python3.5/test/test_venv.py
python3-misc: /usr/lib/python3.5/venv/__main__.py
python3-misc: /usr/lib/python3.5/venv/__init__.py

The files are in the python3-misc package, which is the fallback for
everything which isn't explicitly put into another package.  I'd say
the best fix for this would be to edit the manifest so the venv module
goes into the python3-pyvenv package too.

Ross

Re: Missing venv module in Python 3

[Hugues Kamba]

[-- Attachment #1.1: Type: text/plain, Size: 3914 bytes --]

Hi Alejandro,


Thank you for your suggestion.


The following has been added to a python3_%.bbapend:

# Add missing venv module dependency.
FILES_${PN}-pyvenv += "${libdir}/${PYTHON_MAJMIN}/venv"


However, it is not quite obvious how to successfully run the create_manifest3.py script.
Attempts were made to run it with "python3 create_manifest3.py python3.5" but it complains about an undefined variable (nativelibfolder).
I also tried to run it using "bitbake pyhton -c create_manifest3.py" as specified in the docstring but execution fails because of missing tasks (no do_..).


Could you or others on the mailing list, provide some guidance on how to go about running (arguments, etc..) the create_manifest3.py script.


[cid:bbaed11e-d9b2-456f-bd15-200669ad237e]

Hugues Kamba | Senior Software Engineer
hugues.kamba@arm.com<mailto:sandi.shannon@arm.com>
CPC1, Capital Park, Cambridge Road, Fulbourn, CB21 5XE, UK
Tel: (+44)1223 400400
Visit the new ARM Connected Community<https://urldefense.proofpoint.com/v2/url?u=http-3A__community.arm.com_welcome&d=DwMFAg&c=bKNseOHsszsRCtLFfd6XtA&r=h1JGsEgvfYBPUFak8b6Fs-oV6irVsuuFc4RBESlsTFE&m=N8x4mnijl-wwLmtz3tbzVbe9R-QRW9bVZHfjOv3YY0o&s=FTtAextvaqUli74ujYjbDxp7jK9aaA6G7NOsVfryuyw&e=>



________________________________
From: Alejandro Enedino Hernandez Samaniego <aehs29@gmail.com>
Sent: 18 October 2018 19:18
To: Burton, Ross
Cc: Jonathan Haigh; OE-core; Jeremy Johnson; Hugues Kamba
Subject: Re: [OE-core] Missing venv module in Python 3

Hello folks,

On Thu, Oct 18, 2018, 3:34 PM Burton, Ross <ross.burton@intel.com<mailto:ross.burton@intel.com>> wrote:
On Thu, 18 Oct 2018 at 16:19, Jonathan Haigh <Jonathan.Haigh@arm.com<mailto:Jonathan.Haigh@arm.com>> wrote:
> The venv package (https://docs.python.org/3/library/venv.html) itself is missing although its source is available within Python 3.5 (https://www.python.org/ftp/python/3.5.6/Python-3.5.6.tgz). As mentioned earlier, it seems the issue is because there is no entry in the manifest from which the various Python 3 modules are created (https://github.com/openembedded/openembedded-core/blob/master/meta/recipes-devtools/python/python3/python3-manifest.json).
>
> The manifest is generated from https://github.com/openembedded/openembedded-core/blob/master/meta/recipes-devtools/python/python3/create_manifest3.py, and it does not seem obvious to me how to go about correctly adding a new module to the manifest, using create_manifest3.py or otherwise.

The script itself should get the venv dependency automatically (my guess is that this isn't getting detected by the script because the module itself its coming from a directory) but I'd have to debug what is exactly happening, although I don't have access to my computer at the moment.

An easy fix (for now) is to add the

/usr/lib/python3.5/venv/

Path to the manifest on the FILES for the pyenv module.

After that running the create_manifest task should get its dependencies correctly and update the manifest so it can be pushed to the repo.

Cheers,

Alejandro



Ah, right.

In that case:

$ oe-pkgdata-util find-path */*venv*py
python3-tests: /usr/lib/python3.5/test/test_venv.py
python3-misc: /usr/lib/python3.5/venv/__main__.py
python3-misc: /usr/lib/python3.5/venv/__init__.py

The files are in the python3-misc package, which is the fallback for
everything which isn't explicitly put into another package.  I'd say
the best fix for this would be to edit the manifest so the venv module
goes into the python3-pyvenv package too.

Ross
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

[-- Attachment #1.2: Type: text/html, Size: 10057 bytes --]

[-- Attachment #2: Outlook-3v34lh5k.png --]
[-- Type: image/png, Size: 2551 bytes --]

Re: Missing venv module in Python 3

[Hugues Kamba]

[-- Attachment #1: Type: text/plain, Size: 6274 bytes --]

Hello Alejandro,

Thank you for your help.

Unfortunately, the pyvenv package is not in the manifest, I cannot therefore add the path where you are suggesting.

It is not immediately clear to me where the package is created, however, files are added to the pyvenv package (using FILES_${PN}-pyvenv += ) in the python3_3.5.6.bb recipe. This is why I added the ${libdir}/${PYTHON_MAJMIN}/venv file in a bbapend to the python3_3.5.6.bb recipe.

I am able to successfully run the create_manifest3.py script but the generated manifest content has not changed. I did not expect it to change because pyvenv was not there in the first place.

On a different note; I managed to include the venv module by including the python3-misc package to the build. I can create a virtual environment using python -m venv. The issue with that is that python3-misc includes other modules I do not need and thus needlessly increases size. Another issue is that I have to include python3-runpy (to use the -m option when creating the virtual environment).

I also realised that python3-pip is a depencency to venv.

In conclusion, can you think of how dependencies (venv and pip) can be added to the python3-pyvenv package? This will allow reduction of the image size by removing python3-misc and python3-runpy.


Hugues
________________________________
From: Alejandro Enedino Hernandez Samaniego <aehs29@gmail.com>
Sent: 19 October 2018 13:37:14
To: Hugues Kamba
Cc: Burton, Ross; Jonathan Haigh; OE-core; Jeremy Johnson
Subject: Re: [OE-core] Missing venv module in Python 3

Hello Huges,

The path needs to be added to the manifest itself (the json file) under the files section for the pyvenv package not in a bbappend.

And to run the create manifest task you would do:

$ bitbake python3 -c create_manifest

Once I get to my computer I will be able to provide more help.

Alejandro


On Fri, Oct 19, 2018, 10:08 AM Hugues Kamba <Hugues.Kamba@arm.com<mailto:Hugues.Kamba@arm.com>> wrote:

Hi Alejandro,


Thank you for your suggestion.


The following has been added to a python3_%.bbapend:

# Add missing venv module dependency.
FILES_${PN}-pyvenv += "${libdir}/${PYTHON_MAJMIN}/venv"


However, it is not quite obvious how to successfully run the create_manifest3.py script.
Attempts were made to run it with "python3 create_manifest3.py python3.5" but it complains about an undefined variable (nativelibfolder).
I also tried to run it using "bitbake pyhton -c create_manifest3.py" as specified in the docstring but execution fails because of missing tasks (no do_..).


Could you or others on the mailing list, provide some guidance on how to go about running (arguments, etc..) the create_manifest3.py script.


[cid:bbaed11e-d9b2-456f-bd15-200669ad237e]

Hugues Kamba | Senior Software Engineer
hugues.kamba@arm.com<mailto:sandi.shannon@arm.com>
CPC1, Capital Park, Cambridge Road, Fulbourn, CB21 5XE, UK
Tel: (+44)1223 400400
Visit the new ARM Connected Community<https://urldefense.proofpoint.com/v2/url?u=http-3A__community.arm.com_welcome&d=DwMFAg&c=bKNseOHsszsRCtLFfd6XtA&r=h1JGsEgvfYBPUFak8b6Fs-oV6irVsuuFc4RBESlsTFE&m=N8x4mnijl-wwLmtz3tbzVbe9R-QRW9bVZHfjOv3YY0o&s=FTtAextvaqUli74ujYjbDxp7jK9aaA6G7NOsVfryuyw&e=>



________________________________
From: Alejandro Enedino Hernandez Samaniego <aehs29@gmail.com<mailto:aehs29@gmail.com>>
Sent: 18 October 2018 19:18
To: Burton, Ross
Cc: Jonathan Haigh; OE-core; Jeremy Johnson; Hugues Kamba
Subject: Re: [OE-core] Missing venv module in Python 3

Hello folks,

On Thu, Oct 18, 2018, 3:34 PM Burton, Ross <ross.burton@intel.com<mailto:ross.burton@intel.com>> wrote:
On Thu, 18 Oct 2018 at 16:19, Jonathan Haigh <Jonathan.Haigh@arm.com<mailto:Jonathan.Haigh@arm.com>> wrote:
> The venv package (https://docs.python.org/3/library/venv.html) itself is missing although its source is available within Python 3.5 (https://www.python.org/ftp/python/3.5.6/Python-3.5.6.tgz). As mentioned earlier, it seems the issue is because there is no entry in the manifest from which the various Python 3 modules are created (https://github.com/openembedded/openembedded-core/blob/master/meta/recipes-devtools/python/python3/python3-manifest.json).
>
> The manifest is generated from https://github.com/openembedded/openembedded-core/blob/master/meta/recipes-devtools/python/python3/create_manifest3.py, and it does not seem obvious to me how to go about correctly adding a new module to the manifest, using create_manifest3.py or otherwise.

The script itself should get the venv dependency automatically (my guess is that this isn't getting detected by the script because the module itself its coming from a directory) but I'd have to debug what is exactly happening, although I don't have access to my computer at the moment.

An easy fix (for now) is to add the

/usr/lib/python3.5/venv/

Path to the manifest on the FILES for the pyenv module.

After that running the create_manifest task should get its dependencies correctly and update the manifest so it can be pushed to the repo.

Cheers,

Alejandro



Ah, right.

In that case:

$ oe-pkgdata-util find-path */*venv*py
python3-tests: /usr/lib/python3.5/test/test_venv.py
python3-misc: /usr/lib/python3.5/venv/__main__.py
python3-misc: /usr/lib/python3.5/venv/__init__.py

The files are in the python3-misc package, which is the fallback for
everything which isn't explicitly put into another package.  I'd say
the best fix for this would be to edit the manifest so the venv module
goes into the python3-pyvenv package too.

Ross
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

[-- Attachment #2: Type: text/html, Size: 15605 bytes --]