On Tue, 14 Feb 2023 at 21:56, Arnout Vandecappelle <
arnout@mind.be> wrote:
On 14/02/2023 08:27, Sourabh Hegde wrote:
> Hello James,
>
> Thanks for the update.
>
> awscrt also has precompiled wheels <https://pypi.org/project/awscrt/#files> such
> as awscrt-0.16.10-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
>
> Can Buildroot use them instead of cross compiling?
If cross-compiling is difficult, it's fairly likely that there are many cases
where the binary wheel won't work either. Therefore, no, we (probably) can't use
them.
> If not, is it possible
> to extract it and install manually?
You can do that in a post-build script. It's going to be something that is
tested in your specific use case only, so not something that we would support
upstream.
A wheel file is actually a zip file, so you just need to unzip it in the
site_packages directory. And test if it works, of course.
To understand it correctly, should I unzip the wheel file into "/target/lib/python3.10/site-packages/" after the make?
And now how can I select this dependency during build?
Regards,
Arnout
>
> Thanks in advance
>
> On Thu, 9 Feb 2023 at 21:12, James Hilliard <james.hilliard1@gmail.com
> <mailto:james.hilliard1@gmail.com>> wrote:
>
> On Thu, Feb 9, 2023 at 5:10 AM Sourabh Hegde <hrsourabh011@gmail.com
> <mailto:hrsourabh011@gmail.com>> wrote:
> >
> > Hello Maxim,
> >
> > While doing make with new Buildroot-packages I getting:
> >
> > [ 51%] Building ASM object
> aws-lc/crypto/CMakeFiles/crypto_objects.dir/hrss/asm/poly_rq_mul.S.o
> >
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S: Assembler messages:
> >
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:304: Error: unknown pseudo-op: `.att_syntax'
> >
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:307: Error: unknown mnemonic `push' -- `push %rbp'
> >
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:309: Error: bad register expression
> >
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:310: Error: unknown mnemonic `movq' -- `movq %rsp,%rbp'
> >
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:311: Error: bad register expression
> >
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:312: Error: unknown mnemonic `push' -- `push %r12'
> >
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:313: Error: bad register expression
> >
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:321: Error: operand 1 must be an integer register -- `mov %rcx,%r8'
> >
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:322: Error: unknown mnemonic `addq' -- `addq $6144+12288+512+9408+32,%r8'
> >
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:323: Error: operand 1 must be an integer register -- `mov %r8,%rax'
> >
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:324: Error: unknown mnemonic `subq' -- `subq $6144,%r8'
> >
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:325: Error: operand 1 must be an integer register -- `mov %r8,%r11'
> >
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:326: Error: unknown mnemonic `subq' -- `subq $12288,%r8'
> >
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:327: Error: operand 1 must be an integer register -- `mov %r8,%r12'
> >
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:328: Error: unknown mnemonic `subq' -- `subq $512,%r8'
> >
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:329: Error: unknown mnemonic `vmovdqa' -- `vmovdqa const3(%rip),%ymm3'
> >
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:330: Error: unknown mnemonic `vmovdqu' -- `vmovdqu 0(%rsi),%ymm0'
> >
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:331: Error: unknown mnemonic `vmovdqu' -- `vmovdqu 88(%rsi),%ymm1'
> > .
> > .
> > .
> > Any idea what it is complaining about?
>
> The problem here is that while the python awscrt package uses setuptools
> it's not a normal python setuptools package, the setup.py installs both a normal
> setuptools package and a vendored cmake package(meta package really).
>
> See cmake build wrapper functionality here in the setuptools setup.py file:
> https://github.com/awslabs/aws-crt-python/blob/v0.16.10/setup.py#L156-L272
> <https://github.com/awslabs/aws-crt-python/blob/v0.16.10/setup.py#L156-L272>
>
> This is the cmake meta project root which is vendored inside of awscrt:
> https://github.com/awslabs/aws-crt-python/tree/v0.16.10/crt
> <https://github.com/awslabs/aws-crt-python/tree/v0.16.10/crt>
>
> See meta cmakelists build:
> https://github.com/awslabs/aws-crt-python/blob/v0.16.10/crt/CMakeLists.txt
> <https://github.com/awslabs/aws-crt-python/blob/v0.16.10/crt/CMakeLists.txt>
>
> This meta project is made up of multiple subprojects:
> aws-c-auth
> aws-c-cal
> aws-c-common
> aws-c-compression
> aws-c-event-stream
> aws-checksums
> aws-c-http
> aws-c-io
> aws-c-mqtt
> aws-c-s3
> aws-c-sdkutils
> aws-lc
> s2n
>
> In buildroot we expect cmake packages to use our cmake infrastructure
> as it will set up the cross compilation environment correctly while the
> awscrt setuptools cmake wrapper does not.
>
> I confirmed that the cmake packages build correctly when using our
> cmake infrastructure:
> ################################################################################
> #
> # python-awscrt
> #
> ################################################################################
>
> PYTHON_AWSCRT_VERSION = 0.16.10
> PYTHON_AWSCRT_SOURCE = awscrt-$(PYTHON_AWSCRT_VERSION).tar.gz
> PYTHON_AWSCRT_SITE =
> https://files.pythonhosted.org/packages/ae/98/dda092078f2d47bad08e184ab9d364b8f89166497062bc1cf811c0bbbde2 <https://files.pythonhosted.org/packages/ae/98/dda092078f2d47bad08e184ab9d364b8f89166497062bc1cf811c0bbbde2>
> PYTHON_AWSCRT_SUBDIR = crt
>
> $(eval $(cmake-package))
>
> See:
> https://buildroot.org/downloads/manual/manual.html#_infrastructure_for_cmake_based_packages <https://buildroot.org/downloads/manual/manual.html#_infrastructure_for_cmake_based_packages>
>
> So what I think you need to do is create buildroot cmake packages for all
> these subprojects using buildroot's cmake infrastructure:
> https://github.com/awslabs/aws-c-auth <https://github.com/awslabs/aws-c-auth>
> https://github.com/awslabs/aws-c-cal <https://github.com/awslabs/aws-c-cal>
> https://github.com/awslabs/aws-c-common
> <https://github.com/awslabs/aws-c-common>
> https://github.com/awslabs/aws-c-compression
> <https://github.com/awslabs/aws-c-compression>
> https://github.com/awslabs/aws-c-event-stream
> <https://github.com/awslabs/aws-c-event-stream>
> https://github.com/awslabs/aws-checksums
> <https://github.com/awslabs/aws-checksums>
> https://github.com/awslabs/aws-c-http <https://github.com/awslabs/aws-c-http>
> https://github.com/awslabs/aws-c-io <https://github.com/awslabs/aws-c-io>
> https://github.com/awslabs/aws-c-mqtt <https://github.com/awslabs/aws-c-mqtt>
> https://github.com/awslabs/aws-c-s3 <https://github.com/awslabs/aws-c-s3>
> https://github.com/awslabs/aws-c-sdkutils
> <https://github.com/awslabs/aws-c-sdkutils>
> https://github.com/awslabs/aws-lc <https://github.com/awslabs/aws-lc>
> https://github.com/aws/s2n-tls <https://github.com/aws/s2n-tls>
>
> You will then probably need to modify the setup.py file
> (https://github.com/awslabs/aws-crt-python/blob/v0.16.10/setup.py
> <https://github.com/awslabs/aws-crt-python/blob/v0.16.10/setup.py>) so that
> instead of using the cmake build functions have it use the system versions
> of these libraries pre-installed via the buildroot cmake infrastructure.
>
> You can probably use and check for an env variable such as this in the setup.py:
> PYTHON_AWSCRT_USE_SYSTEM_CRT=1
>
> When present have it disable building the cmake libraries itself and search
> for the installed system crt libraries instead of using the vendored
> cmake build.
>
> You should upstream this setup.py functionality to the aws-crt-project as well
> so that we don't need to carry a patch in buildroot for python-awscrt to use
> the system crt libraries.
>
> Then from the python-awscrt buildroot package you would set that env
> variable and add all the cmake subprojects as buildroot package dependencies.
>
> First thing you should start with is adding the cmake buildroot packages for
> the crt subprojects to buildroot then work on modifying aws-crt-python to
> work correctly with them instead of trying to use its broken cmake setuptools
> wrapper.
>
> >
> > Thanks in advance
> >
> > On Thu, 9 Feb 2023 at 10:28, Sourabh Hegde <hrsourabh011@gmail.com
> <mailto:hrsourabh011@gmail.com>> wrote:
> >>
> >> Hi Maxim,
> >>
> >> I was missing python setuptools on my host. After installing it, I can
> add the package. Also, the added package is available in menuconfig
> >>
> >> Thank you for the support!
> >>
> >> On Thu, 9 Feb 2023 at 10:10, Maxim Kochetkov <fido_max@inbox.ru
> <mailto:fido_max@inbox.ru>> wrote:
> >>>
> >>>
> >>>
> >>> On 09.02.2023 12:03, Sourabh Hegde wrote:
> >>> > Hello Maxim,
> >>> >
> >>> > Update:
> >>> >
> >>> > The above issue resolved by itself!
> >>> >
> >>> > Now when I do: utils/scanpypi awscrt -o package , it says
> >>> >
> >>> > spdx_lookup module is not installed. This can lead to an inaccurate
> >>> > licence detection. Please install it via
> >>> > pip install spdx_lookup
> >>> > buildroot package name for awscrt: python-awscrt
> >>> > Package: python-awscrt
> >>> > Fetching package awscrt
> >>> > ERROR: Could not find package awscrt.
> >>> > Check syntax inside the python package index:
> >>> > https://pypi.python.org/pypi/ <https://pypi.python.org/pypi/>
> <https://pypi.python.org/pypi/ <https://pypi.python.org/pypi/>>
> >>> >
> >>> > But the package is available at https://pypi.org/project/awscrt/
> <https://pypi.org/project/awscrt/>
> >>> > <https://pypi.org/project/awscrt/ <https://pypi.org/project/awscrt/>>
> >>>
> >>> It works fine for me:
> >>>
> >>> buildroot package name for awscrt: python-awscrt
> >>> Package: python-awscrt
> >>> Fetching package awscrt
> >>> Downloading package awscrt from
> >>>
> https://files.pythonhosted.org/packages/ae/98/dda092078f2d47bad08e184ab9d364b8f89166497062bc1cf811c0bbbde2/awscrt-0.16.10.tar.gz. <https://files.pythonhosted.org/packages/ae/98/dda092078f2d47bad08e184ab9d364b8f89166497062bc1cf811c0bbbde2/awscrt-0.16.10.tar.gz.>..
> >>> Checking if package package/python-awscrt already exists...
> >>> Creating package/python-awscrt/python-awscrt.mk...
> >>> More than one file found for license: LICENSE, crt/aws-c-common/LICENSE,
> >>> crt/aws-c-io/LICENSE, crt/aws-lc/LICENSE,
> >>> crt/aws-lc/third_party/fiat/LICENSE, crt/aws-c-mqtt/LICENSE,
> >>> crt/aws-checksums/LICENSE, crt/aws-c-s3/LICENSE,
> >>> crt/aws-c-compression/LICENSE, crt/aws-c-http/LICENSE,
> >>> crt/aws-c-sdkutils/LICENSE, crt/s2n/LICENSE, crt/aws-c-cal/LICENSE,
> >>> crt/aws-c-auth/LICENSE, crt/aws-c-event-stream/LICENSE
> >>> Creating package/python-awscrt/python-awscrt.hash...
> >>> Creating package/python-awscrt/Config.in...
> >>> NOTE: Remember to also make an update to the DEVELOPERS file
> >>> and include an entry for the pkg in packages/Config.in
> >>>
> > _______________________________________________
> > buildroot mailing list
> > buildroot@buildroot.org <mailto:buildroot@buildroot.org>
> > https://lists.buildroot.org/mailman/listinfo/buildroot
> <https://lists.buildroot.org/mailman/listinfo/buildroot>
>
>
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot