Re: [PATCH v2] python3{,-native}: update to 3.7.0

[Alejandro Enedino Hernandez Samaniego <alejandro.enedino.hernandez-samaniego@xilinx.com>]

[-- Attachment #1: Type: text/plain, Size: 8089 bytes --]

Hey Jens,


On 09/10/2018 11:58 PM, Jens Rehsack wrote:
>
>
>> Am 10.09.2018 um 23:33 schrieb Alejandro Enedino Hernandez Samaniego 
>> <alejandro.enedino.hernandez-samaniego@xilinx.com 
>> <mailto:alejandro.enedino.hernandez-samaniego@xilinx.com>>:
>>
>> Hey Jens,
>>
>>
>> As I explained before, when you create a manifest for python 
>> (target), it uses the native build as base (it literally runs the 
>> native python that was just built), it is assumed its the same 
>> version as target and contains all the modules provided by upstream, 
>> otherwise the missing modules cannot be checked for dependencies, and 
>> the manifest becomes incoherent, so its not an option to have an 
>> incomplete python native build.
>>
> In that case, uuid for target never gets deployed, but it is. And I 
> didn't see any packaging issues for `python3` nor for `nativesdk-python3`


I don't see what that has to do with anything, fixing the native build 
should not affect what gets deployed on target, thats exactly why we 
have a manifest, so they user can decide what to install and what not to 
install.
I know you didn't see any packaging issues, but that doesn't mean they 
don't exist, just from the log I showed you, I can tell you that the 
python3-crypt package is not created correctly, for example, if you do:

IMAGE_INSTALL_append = " python3-crypt"

Boot the image, run python3 and you try to import sha3, it will fail, 
because the sha3*.so library won't be on the filesystem.


And thats because the sha3.*.so library was just introduced in this 
upgrade, and our manifest isn't aware it exists, hence it'll end up on 
python3-misc and you have just created an unnecessary dependency from 
python3-crypt to python3-misc (and worse, a dependency which were not 
even aware of, until we test it manually), which beats the whole purpose 
of the manifest.

The do_package function is not gonna fail just because, so you won't see 
errors, but the files will be packaged incorrectly, causing runtime 
errors as a consequence, the create_manifest task tries to solve these 
runtime issues before they happen.

Cheers,

Alejandro


>> Yes you probably need a patch to look at the correct directories for 
>> the h files, as well as a dependency to make the h files available on 
>> recipe-sysroot-native.
>>
> Please check the submission.

I did, its not checking inside recipe-sysroot-native.


>
> Jens
>>
>> Alejandro
>>
>>
>> On 09/10/2018 02:05 PM, Jens Rehsack wrote:
>>> Hey Alejandro,
>>>
>>> I fixed that for cross-compile only, since I would need add a patch 
>>> and a dependency python3-native for one thing: calculate uuids.
>>> When you can explain to me why the python-native needs that, I'll 
>>> change that from -target & nativesdk to all.
>>>
>>> From my point of view it's not a question of having every (unneeded) 
>>> python module being built for the native python, which is used for 
>>> cross-compiling python and some modules only.
>>>
>>> Cheers,
>>> Jens
>>>
>>>> Am 10.09.2018 um 22:17 schrieb Alejandro Enedino Hernandez 
>>>> Samaniego <alejandro.enedino.hernandez-samaniego@xilinx.com 
>>>> <mailto:alejandro.enedino.hernandez-samaniego@xilinx.com>>:
>>>>
>>>> Hey Jens,
>>>>
>>>> The compilation log for python3-native still shows that it didn't 
>>>> build the uuid module
>>>>
>>>>
>>>> Python build finished successfully!
>>>> The necessary bits to build these optional modules were not found:
>>>> _uuid
>>>>
>>>> Please look at my previous reply to find how this can be solved 
>>>> (its likely a missing DEPENDS).
>>>>
>>>>
>>>> Also, this patch is missing the new python3 manifest for this 
>>>> release, there appears to be a few new modules that we need to 
>>>> decide which package they belong to, this is the output of bitbake 
>>>> python3 -c create_manifest:
>>>>
>>>>
>>>> | ERROR:
>>>> | The following files are repeated (contained in more than one 
>>>> package),
>>>> | this is likely to happen when new files are introduced after an 
>>>> upgrade,
>>>> | please check which package should get it,
>>>> |  modify the manifest accordingly and re-run the create_manifest task:
>>>> | ${libdir}/python${PYTHON_MAJMIN}/lib-dynload/_blake2.*.so
>>>> | ${libdir}/python${PYTHON_MAJMIN}/lib-dynload/_sha3.*.so
>>>> | ${libdir}/python${PYTHON_MAJMIN}/lib-dynload/_contextvars.*.so
>>>> | ${libdir}/python${PYTHON_MAJMIN}/contextvars.py
>>>> | ${libdir}/python${PYTHON_MAJMIN}/__pycache__/contextvars.*.pyc
>>>> | ${libdir}/python${PYTHON_MAJMIN}/lib-dynload/_queue.*.so
>>>>
>>>>
>>>> Cheers,
>>>>
>>>>
>>>> Alejandro
>>>>
>>>>
>>>> On 09/10/2018 09:38 AM, Burton, Ross wrote:
>>>>> One thing to be aware of is that I've been fixing up Python's PGO
>>>>> support and there's a slew of patches in master-next and more just
>>>>> posted that this needs to be rebased on top of.  Good news is that my
>>>>> patches remove two of the patches we've been carrying!
>>>>>
>>>>> Ross
>>>>>
>>>>> On 10 September 2018 at 17:36, Jens Rehsack <sno@netbsd.org 
>>>>> <mailto:sno@netbsd.org>> wrote:
>>>>>>
>>>>>> Am 10.09.2018 um 11:35 schrieb Alexander Kanavin 
>>>>>> <alex.kanavin@gmail.com <mailto:alex.kanavin@gmail.com>>:
>>>>>>
>>>>>> Large parts of dnf and friends have been rewritten in c++. I have not
>>>>>> yet updated and reviewed that, that will happen in the next cycle.
>>>>>>
>>>>>>
>>>>>> If I can prepare something for you - drop me a note.
>>>>>> Otherwise - the perl-5.28 update ("." in @INC, regex buffer 
>>>>>> overflow, ...)
>>>>>> is also
>>>>>> awaiting some progress (I can keep "myself" busy).
>>>>>>
>>>>>> There's already enough disruption to deal with (postinsts errors,
>>>>>> openssl 1.1, both caused by me :)
>>>>>>
>>>>>>
>>>>>> You know, corner, ash, ... things happen. But there is progress! 
>>>>>> Great!
>>>>>>
>>>>>> Good that we got all the way to do_rootfs though with 3.7.
>>>>>>
>>>>>>
>>>>>> Yeah, but than came postinst (coreutils :P) :D
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> Alex
>>>>>>
>>>>>> 2018-09-10 0:38 GMT+02:00 Tim Orling <ticotimo@gmail.com 
>>>>>> <mailto:ticotimo@gmail.com>>:
>>>>>>
>>>>>> I did not review the patches closely, but I did try to build
>>>>>> core-image-full-cmdline with the tip of poky and these patches 
>>>>>> applied.
>>>>>> Everything was fine until do_rootfs... I've attached the log.
>>>>>>
>>>>>> Essentially, there are some bits of dnf and so on which are not 
>>>>>> ready for
>>>>>> Python 3.7. We have dnf version 2.7.5, but the latest upstream 
>>>>>> release is
>>>>>> 3.4.0 (with a 3.5.0 just 3 days ago). Not sure yet if that would have
>>>>>> helped.
>>>>>>
>>>>>> --
>>>>>> _______________________________________________
>>>>>> Openembedded-core mailing list
>>>>>> Openembedded-core@lists.openembedded.org 
>>>>>> <mailto:Openembedded-core@lists.openembedded.org>
>>>>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Jens Rehsack - rehsack@gmail.com
>>>>>>
>>>>>>
>>>>>> --
>>>>>> _______________________________________________
>>>>>> Openembedded-core mailing list
>>>>>> Openembedded-core@lists.openembedded.org
>>>>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>>>>>
>>>>
>>>> -- 
>>>> _______________________________________________
>>>> Openembedded-core mailing list
>>>> Openembedded-core@lists.openembedded.org 
>>>> <mailto:Openembedded-core@lists.openembedded.org>
>>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>>
>>> -- 
>>> Jens Rehsack - rehsack@gmail.com <mailto:rehsack@gmail.com>
>>>
>>
>> -- 
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org 
>> <mailto:Openembedded-core@lists.openembedded.org>
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
> -- 
> Jens Rehsack - rehsack@gmail.com <mailto:rehsack@gmail.com>
>


[-- Attachment #2: Type: text/html, Size: 21294 bytes --]