Re: [Buildroot] How to install custom sdk in buildroot?

[James Hilliard <james.hilliard1@gmail.com>]

On Thu, Feb 9, 2023 at 5:10 AM Sourabh Hegde <hrsourabh011@gmail.com> wrote:
>
> Hello Maxim,
>
> While doing make with new Buildroot-packages I getting:
>
> [ 51%] Building ASM object aws-lc/crypto/CMakeFiles/crypto_objects.dir/hrss/asm/poly_rq_mul.S.o
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S: Assembler messages:
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:304: Error: unknown pseudo-op: `.att_syntax'
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:307: Error: unknown mnemonic `push' -- `push %rbp'
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:309: Error: bad register expression
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:310: Error: unknown mnemonic `movq' -- `movq %rsp,%rbp'
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:311: Error: bad register expression
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:312: Error: unknown mnemonic `push' -- `push %r12'
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:313: Error: bad register expression
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:321: Error: operand 1 must be an integer register -- `mov %rcx,%r8'
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:322: Error: unknown mnemonic `addq' -- `addq $6144+12288+512+9408+32,%r8'
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:323: Error: operand 1 must be an integer register -- `mov %r8,%rax'
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:324: Error: unknown mnemonic `subq' -- `subq $6144,%r8'
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:325: Error: operand 1 must be an integer register -- `mov %r8,%r11'
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:326: Error: unknown mnemonic `subq' -- `subq $12288,%r8'
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:327: Error: operand 1 must be an integer register -- `mov %r8,%r12'
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:328: Error: unknown mnemonic `subq' -- `subq $512,%r8'
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:329: Error: unknown mnemonic `vmovdqa' -- `vmovdqa const3(%rip),%ymm3'
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:330: Error: unknown mnemonic `vmovdqu' -- `vmovdqu 0(%rsi),%ymm0'
> /home/amazone/output/build/python-awscrt-0.16.10/crt/aws-lc/crypto/hrss/asm/poly_rq_mul.S:331: Error: unknown mnemonic `vmovdqu' -- `vmovdqu 88(%rsi),%ymm1'
> .
> .
> .
> Any idea what it is complaining about?

The problem here is that while the python awscrt package uses setuptools
it's not a normal python setuptools package, the setup.py installs both a normal
setuptools package and a vendored cmake package(meta package really).

See cmake build wrapper functionality here in the setuptools setup.py file:
https://github.com/awslabs/aws-crt-python/blob/v0.16.10/setup.py#L156-L272

This is the cmake meta project root which is vendored inside of awscrt:
https://github.com/awslabs/aws-crt-python/tree/v0.16.10/crt

See meta cmakelists build:
https://github.com/awslabs/aws-crt-python/blob/v0.16.10/crt/CMakeLists.txt

This meta project is made up of multiple subprojects:
aws-c-auth
aws-c-cal
aws-c-common
aws-c-compression
aws-c-event-stream
aws-checksums
aws-c-http
aws-c-io
aws-c-mqtt
aws-c-s3
aws-c-sdkutils
aws-lc
s2n

In buildroot we expect cmake packages to use our cmake infrastructure
as it will set up the cross compilation environment correctly while the
awscrt setuptools cmake wrapper does not.

I confirmed that the cmake packages build correctly when using our
cmake infrastructure:
################################################################################
#
# python-awscrt
#
################################################################################

PYTHON_AWSCRT_VERSION = 0.16.10
PYTHON_AWSCRT_SOURCE = awscrt-$(PYTHON_AWSCRT_VERSION).tar.gz
PYTHON_AWSCRT_SITE =
https://files.pythonhosted.org/packages/ae/98/dda092078f2d47bad08e184ab9d364b8f89166497062bc1cf811c0bbbde2
PYTHON_AWSCRT_SUBDIR = crt

$(eval $(cmake-package))

See:
https://buildroot.org/downloads/manual/manual.html#_infrastructure_for_cmake_based_packages

So what I think you need to do is create buildroot cmake packages for all
these subprojects using buildroot's cmake infrastructure:
https://github.com/awslabs/aws-c-auth
https://github.com/awslabs/aws-c-cal
https://github.com/awslabs/aws-c-common
https://github.com/awslabs/aws-c-compression
https://github.com/awslabs/aws-c-event-stream
https://github.com/awslabs/aws-checksums
https://github.com/awslabs/aws-c-http
https://github.com/awslabs/aws-c-io
https://github.com/awslabs/aws-c-mqtt
https://github.com/awslabs/aws-c-s3
https://github.com/awslabs/aws-c-sdkutils
https://github.com/awslabs/aws-lc
https://github.com/aws/s2n-tls

You will then probably need to modify the setup.py file
(https://github.com/awslabs/aws-crt-python/blob/v0.16.10/setup.py) so that
instead of using the cmake build functions have it use the system versions
of these libraries pre-installed via the buildroot cmake infrastructure.

You can probably use and check for an env variable such as this in the setup.py:
PYTHON_AWSCRT_USE_SYSTEM_CRT=1

When present have it disable building the cmake libraries itself and search
for the installed system crt libraries instead of using the vendored
cmake build.

You should upstream this setup.py functionality to the aws-crt-project as well
so that we don't need to carry a patch in buildroot for python-awscrt to use
the system crt libraries.

Then from the python-awscrt buildroot package you would set that env
variable and add all the cmake subprojects as buildroot package dependencies.

First thing you should start with is adding the cmake buildroot packages for
the crt subprojects to buildroot then work on modifying aws-crt-python to
work correctly with them instead of trying to use its broken cmake setuptools
wrapper.

>
> Thanks in advance
>
> On Thu, 9 Feb 2023 at 10:28, Sourabh Hegde <hrsourabh011@gmail.com> wrote:
>>
>> Hi Maxim,
>>
>> I was missing python setuptools on my host. After installing it, I can add the package. Also, the added package is available in menuconfig
>>
>> Thank you for the support!
>>
>> On Thu, 9 Feb 2023 at 10:10, Maxim Kochetkov <fido_max@inbox.ru> wrote:
>>>
>>>
>>>
>>> On 09.02.2023 12:03, Sourabh Hegde wrote:
>>> > Hello Maxim,
>>> >
>>> > Update:
>>> >
>>> > The above issue resolved by itself!
>>> >
>>> > Now when I do: utils/scanpypi awscrt -o package , it says
>>> >
>>> > spdx_lookup module is not installed. This can lead to an inaccurate
>>> > licence detection. Please install it via
>>> > pip install spdx_lookup
>>> > buildroot package name for awscrt: python-awscrt
>>> > Package: python-awscrt
>>> > Fetching package awscrt
>>> > ERROR: Could not find package awscrt.
>>> > Check syntax inside the python package index:
>>> > https://pypi.python.org/pypi/ <https://pypi.python.org/pypi/>
>>> >
>>> > But the package is available at https://pypi.org/project/awscrt/
>>> > <https://pypi.org/project/awscrt/>
>>>
>>> It works fine for me:
>>>
>>> buildroot package name for awscrt: python-awscrt
>>> Package: python-awscrt
>>> Fetching package awscrt
>>> Downloading package awscrt from
>>> https://files.pythonhosted.org/packages/ae/98/dda092078f2d47bad08e184ab9d364b8f89166497062bc1cf811c0bbbde2/awscrt-0.16.10.tar.gz...
>>> Checking if package package/python-awscrt already exists...
>>> Creating package/python-awscrt/python-awscrt.mk...
>>> More than one file found for license: LICENSE, crt/aws-c-common/LICENSE,
>>> crt/aws-c-io/LICENSE, crt/aws-lc/LICENSE,
>>> crt/aws-lc/third_party/fiat/LICENSE, crt/aws-c-mqtt/LICENSE,
>>> crt/aws-checksums/LICENSE, crt/aws-c-s3/LICENSE,
>>> crt/aws-c-compression/LICENSE, crt/aws-c-http/LICENSE,
>>> crt/aws-c-sdkutils/LICENSE, crt/s2n/LICENSE, crt/aws-c-cal/LICENSE,
>>> crt/aws-c-auth/LICENSE, crt/aws-c-event-stream/LICENSE
>>> Creating package/python-awscrt/python-awscrt.hash...
>>> Creating package/python-awscrt/Config.in...
>>> NOTE: Remember to also make an update to the DEVELOPERS file
>>>        and include an entry for the pkg in packages/Config.in
>>>
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot