Re: [Buildroot] How to install custom sdk in buildroot?

[Sourabh Hegde <hrsourabh011@gmail.com>]

[-- Attachment #1.1: Type: text/plain, Size: 13961 bytes --]

Hello Arnout,

Thanks for the update.

On Tue, 14 Feb 2023 at 21:56, Arnout Vandecappelle <arnout@mind.be> wrote:

>
>
> On 14/02/2023 08:27, Sourabh Hegde wrote:
> > Hello James,
> >
> > Thanks for the update.
> >
> > awscrt also has precompiled wheels <
> https://pypi.org/project/awscrt/#files> such
> > as
> awscrt-0.16.10-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
> >
> > Can Buildroot use them instead of cross compiling?
>
>   If cross-compiling is difficult, it's fairly likely that there are many
> cases
> where the binary wheel won't work either. Therefore, no, we (probably)
> can't use
> them.
>
> > If not, is it possible
> > to extract it and install manually?
>
>   You can do that in a post-build script. It's going to be something that
> is
> tested in your specific use case only, so not something that we would
> support
> upstream.
>
>   A wheel file is actually a zip file, so you just need to unzip it in the
> site_packages directory. And test if it works, of course.
>

To understand it correctly, should I unzip the wheel file into
"/target/lib/python3.10/site-packages/" after the make?

But, this "awscrt" is the dependency for https://pypi.org/project/awsiotsdk/
 https://github.com/aws/aws-iot-device-sdk-python-v2#installation-issues
And now how can I select this dependency during build?


>
>   Regards,
>   Arnout
>
> >
> > Thanks in advance
> >
> > On Thu, 9 Feb 2023 at 21:12, James Hilliard <james.hilliard1@gmail.com
> > <mailto:james.hilliard1@gmail.com>> wrote:
> >
> >     On Thu, Feb 9, 2023 at 5:10 AM Sourabh Hegde <hrsourabh011@gmail.com
> >     <mailto:hrsourabh011@gmail.com>> wrote:
> >      >
> >      > Hello Maxim,
> >      >
> >      > While doing make with new Buildroot-packages I getting:
> >      >
> >      > [ 51%] Building ASM object
> >     aws-lc/crypto/CMakeFiles/crypto_objects.dir/hrss/asm/poly_rq_mul.S.o
> >      >
> >
>  /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:
> Assembler messages:
> >      >
> >
>  /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:304:
> Error: unknown pseudo-op: `.att_syntax'
> >      >
> >
>  /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:307:
> Error: unknown mnemonic `push' -- `push %rbp'
> >      >
> >
>  /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:309:
> Error: bad register expression
> >      >
> >
>  /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:310:
> Error: unknown mnemonic `movq' -- `movq %rsp,%rbp'
> >      >
> >
>  /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:311:
> Error: bad register expression
> >      >
> >
>  /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:312:
> Error: unknown mnemonic `push' -- `push %r12'
> >      >
> >
>  /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:313:
> Error: bad register expression
> >      >
> >
>  /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:321:
> Error: operand 1 must be an integer register -- `mov %rcx,%r8'
> >      >
> >
>  /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:322:
> Error: unknown mnemonic `addq' -- `addq $6144+12288+512+9408+32,%r8'
> >      >
> >
>  /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:323:
> Error: operand 1 must be an integer register -- `mov %r8,%rax'
> >      >
> >
>  /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:324:
> Error: unknown mnemonic `subq' -- `subq $6144,%r8'
> >      >
> >
>  /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:325:
> Error: operand 1 must be an integer register -- `mov %r8,%r11'
> >      >
> >
>  /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:326:
> Error: unknown mnemonic `subq' -- `subq $12288,%r8'
> >      >
> >
>  /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:327:
> Error: operand 1 must be an integer register -- `mov %r8,%r12'
> >      >
> >
>  /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:328:
> Error: unknown mnemonic `subq' -- `subq $512,%r8'
> >      >
> >
>  /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:329:
> Error: unknown mnemonic `vmovdqa' -- `vmovdqa const3(%rip),%ymm3'
> >      >
> >
>  /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:330:
> Error: unknown mnemonic `vmovdqu' -- `vmovdqu 0(%rsi),%ymm0'
> >      >
> >
>  /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:331:
> Error: unknown mnemonic `vmovdqu' -- `vmovdqu 88(%rsi),%ymm1'
> >      > .
> >      > .
> >      > .
> >      > Any idea what it is complaining about?
> >
> >     The problem here is that while the python awscrt package uses
> setuptools
> >     it's not a normal python setuptools package, the setup.py installs
> both a normal
> >     setuptools package and a vendored cmake package(meta package really).
> >
> >     See cmake build wrapper functionality here in the setuptools
> setup.py file:
> >
> https://github.com/awslabs/aws-crt-python/blob/v0.16.10/setup.py#L156-L272
> >     <
> https://github.com/awslabs/aws-crt-python/blob/v0.16.10/setup.py#L156-L272
> >
> >
> >     This is the cmake meta project root which is vendored inside of
> awscrt:
> >     https://github.com/awslabs/aws-crt-python/tree/v0.16.10/crt
> >     <https://github.com/awslabs/aws-crt-python/tree/v0.16.10/crt>
> >
> >     See meta cmakelists build:
> >
> https://github.com/awslabs/aws-crt-python/blob/v0.16.10/crt/CMakeLists.txt
> >     <
> https://github.com/awslabs/aws-crt-python/blob/v0.16.10/crt/CMakeLists.txt
> >
> >
> >     This meta project is made up of multiple subprojects:
> >     aws-c-auth
> >     aws-c-cal
> >     aws-c-common
> >     aws-c-compression
> >     aws-c-event-stream
> >     aws-checksums
> >     aws-c-http
> >     aws-c-io
> >     aws-c-mqtt
> >     aws-c-s3
> >     aws-c-sdkutils
> >     aws-lc
> >     s2n
> >
> >     In buildroot we expect cmake packages to use our cmake infrastructure
> >     as it will set up the cross compilation environment correctly while
> the
> >     awscrt setuptools cmake wrapper does not.
> >
> >     I confirmed that the cmake packages build correctly when using our
> >     cmake infrastructure:
> >
>  ################################################################################
> >     #
> >     # python-awscrt
> >     #
> >
>  ################################################################################
> >
> >     PYTHON_AWSCRT_VERSION = 0.16.10
> >     PYTHON_AWSCRT_SOURCE = awscrt-$(PYTHON_AWSCRT_VERSION).tar.gz
> >     PYTHON_AWSCRT_SITE =
> >
> https://files.pythonhosted.org/packages/ae/98/dda092078f2d47bad08e184ab9d364b8f89166497062bc1cf811c0bbbde2
> <
> https://files.pythonhosted.org/packages/ae/98/dda092078f2d47bad08e184ab9d364b8f89166497062bc1cf811c0bbbde2
> >
> >     PYTHON_AWSCRT_SUBDIR = crt
> >
> >     $(eval $(cmake-package))
> >
> >     See:
> >
> https://buildroot.org/downloads/manual/manual.html#_infrastructure_for_cmake_based_packages
> <
> https://buildroot.org/downloads/manual/manual.html#_infrastructure_for_cmake_based_packages
> >
> >
> >     So what I think you need to do is create buildroot cmake packages
> for all
> >     these subprojects using buildroot's cmake infrastructure:
> >     https://github.com/awslabs/aws-c-auth <
> https://github.com/awslabs/aws-c-auth>
> >     https://github.com/awslabs/aws-c-cal <
> https://github.com/awslabs/aws-c-cal>
> >     https://github.com/awslabs/aws-c-common
> >     <https://github.com/awslabs/aws-c-common>
> >     https://github.com/awslabs/aws-c-compression
> >     <https://github.com/awslabs/aws-c-compression>
> >     https://github.com/awslabs/aws-c-event-stream
> >     <https://github.com/awslabs/aws-c-event-stream>
> >     https://github.com/awslabs/aws-checksums
> >     <https://github.com/awslabs/aws-checksums>
> >     https://github.com/awslabs/aws-c-http <
> https://github.com/awslabs/aws-c-http>
> >     https://github.com/awslabs/aws-c-io <
> https://github.com/awslabs/aws-c-io>
> >     https://github.com/awslabs/aws-c-mqtt <
> https://github.com/awslabs/aws-c-mqtt>
> >     https://github.com/awslabs/aws-c-s3 <
> https://github.com/awslabs/aws-c-s3>
> >     https://github.com/awslabs/aws-c-sdkutils
> >     <https://github.com/awslabs/aws-c-sdkutils>
> >     https://github.com/awslabs/aws-lc <https://github.com/awslabs/aws-lc
> >
> >     https://github.com/aws/s2n-tls <https://github.com/aws/s2n-tls>
> >
> >     You will then probably need to modify the setup.py file
> >     (https://github.com/awslabs/aws-crt-python/blob/v0.16.10/setup.py
> >     <https://github.com/awslabs/aws-crt-python/blob/v0.16.10/setup.py>)
> so that
> >     instead of using the cmake build functions have it use the system
> versions
> >     of these libraries pre-installed via the buildroot cmake
> infrastructure.
> >
> >     You can probably use and check for an env variable such as this in
> the setup.py:
> >     PYTHON_AWSCRT_USE_SYSTEM_CRT=1
> >
> >     When present have it disable building the cmake libraries itself and
> search
> >     for the installed system crt libraries instead of using the vendored
> >     cmake build.
> >
> >     You should upstream this setup.py functionality to the
> aws-crt-project as well
> >     so that we don't need to carry a patch in buildroot for
> python-awscrt to use
> >     the system crt libraries.
> >
> >     Then from the python-awscrt buildroot package you would set that env
> >     variable and add all the cmake subprojects as buildroot package
> dependencies.
> >
> >     First thing you should start with is adding the cmake buildroot
> packages for
> >     the crt subprojects to buildroot then work on modifying
> aws-crt-python to
> >     work correctly with them instead of trying to use its broken cmake
> setuptools
> >     wrapper.
> >
> >      >
> >      > Thanks in advance
> >      >
> >      > On Thu, 9 Feb 2023 at 10:28, Sourabh Hegde <
> hrsourabh011@gmail.com
> >     <mailto:hrsourabh011@gmail.com>> wrote:
> >      >>
> >      >> Hi Maxim,
> >      >>
> >      >> I was missing python setuptools on my host. After installing it,
> I can
> >     add the package. Also, the added package is available in menuconfig
> >      >>
> >      >> Thank you for the support!
> >      >>
> >      >> On Thu, 9 Feb 2023 at 10:10, Maxim Kochetkov <fido_max@inbox.ru
> >     <mailto:fido_max@inbox.ru>> wrote:
> >      >>>
> >      >>>
> >      >>>
> >      >>> On 09.02.2023 12:03, Sourabh Hegde wrote:
> >      >>> > Hello Maxim,
> >      >>> >
> >      >>> > Update:
> >      >>> >
> >      >>> > The above issue resolved by itself!
> >      >>> >
> >      >>> > Now when I do: utils/scanpypi awscrt -o package , it says
> >      >>> >
> >      >>> > spdx_lookup module is not installed. This can lead to an
> inaccurate
> >      >>> > licence detection. Please install it via
> >      >>> > pip install spdx_lookup
> >      >>> > buildroot package name for awscrt: python-awscrt
> >      >>> > Package: python-awscrt
> >      >>> > Fetching package awscrt
> >      >>> > ERROR: Could not find package awscrt.
> >      >>> > Check syntax inside the python package index:
> >      >>> > https://pypi.python.org/pypi/ <https://pypi.python.org/pypi/>
> >     <https://pypi.python.org/pypi/ <https://pypi.python.org/pypi/>>
> >      >>> >
> >      >>> > But the package is available at
> https://pypi.org/project/awscrt/
> >     <https://pypi.org/project/awscrt/>
> >      >>> > <https://pypi.org/project/awscrt/ <
> https://pypi.org/project/awscrt/>>
> >      >>>
> >      >>> It works fine for me:
> >      >>>
> >      >>> buildroot package name for awscrt: python-awscrt
> >      >>> Package: python-awscrt
> >      >>> Fetching package awscrt
> >      >>> Downloading package awscrt from
> >      >>>
> >
> https://files.pythonhosted.org/packages/ae/98/dda092078f2d47bad08e184ab9d364b8f89166497062bc1cf811c0bbbde2/awscrt-0.16.10.tar.gz.
> <
> https://files.pythonhosted.org/packages/ae/98/dda092078f2d47bad08e184ab9d364b8f89166497062bc1cf811c0bbbde2/awscrt-0.16.10.tar.gz
> .>..
> >      >>> Checking if package package/python-awscrt already exists...
> >      >>> Creating package/python-awscrt/python-awscrt.mk...
> >      >>> More than one file found for license: LICENSE,
> crt/aws-c-common/LICENSE,
> >      >>> crt/aws-c-io/LICENSE, crt/aws-lc/LICENSE,
> >      >>> crt/aws-lc/third_party/fiat/LICENSE, crt/aws-c-mqtt/LICENSE,
> >      >>> crt/aws-checksums/LICENSE, crt/aws-c-s3/LICENSE,
> >      >>> crt/aws-c-compression/LICENSE, crt/aws-c-http/LICENSE,
> >      >>> crt/aws-c-sdkutils/LICENSE, crt/s2n/LICENSE,
> crt/aws-c-cal/LICENSE,
> >      >>> crt/aws-c-auth/LICENSE, crt/aws-c-event-stream/LICENSE
> >      >>> Creating package/python-awscrt/python-awscrt.hash...
> >      >>> Creating package/python-awscrt/Config.in...
> >      >>> NOTE: Remember to also make an update to the DEVELOPERS file
> >      >>>        and include an entry for the pkg in packages/Config.in
> >      >>>
> >      > _______________________________________________
> >      > buildroot mailing list
> >      > buildroot@buildroot.org <mailto:buildroot@buildroot.org>
> >      > https://lists.buildroot.org/mailman/listinfo/buildroot
> >     <https://lists.buildroot.org/mailman/listinfo/buildroot>
> >
> >
> > _______________________________________________
> > buildroot mailing list
> > buildroot@buildroot.org
> > https://lists.buildroot.org/mailman/listinfo/buildroot
>

[-- Attachment #1.2: Type: text/html, Size: 23450 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot