[Buildroot] [Feature Request] Host cmake built with the --system-curl option

[Buildroot] [Feature Request] Host cmake built with the --system-curl option

[Emile Cormier]

[-- Attachment #1.1: Type: text/plain, Size: 901 bytes --]

When using cmake while running a post-build script, the ExternalProject_add
command fails when the URL option is used to download a source tarball via
https. This is because the host cmake is not built using the --system-curl
option (https://github.com/Kitware/CMake/blob/master/bootstrap#L660).

Without the system-curl option, the cmake build will use its own bundled
libcurl without OpenSSL support, and thus downloading tarballs via https
fails.

Adding the --system-curl option would entail that libcurl also be built as
a host library. I don't think it currently is.

I use CMake's ExternalProject_add in our project to download and build the
static library dependencies that our embedded application needs. I know
that buildroot provides various packages for cross-compiled libraries, but
I have no control over the versions (and compile-time options) of these
packages.

Cheers,
Emile Cormier

[-- Attachment #1.2: Type: text/html, Size: 1141 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [Feature Request] Host cmake built with the --system-curl option

[Arnout Vandecappelle]

On 06/04/2022 04:06, Emile Cormier wrote:
> When using cmake while running a post-build script, the ExternalProject_add 
> command fails when the URL option is used to download a source tarball via 
> https. This is because the host cmake is not built using the --system-curl 
> option (https://github.com/Kitware/CMake/blob/master/bootstrap#L660 
> <https://github.com/Kitware/CMake/blob/master/bootstrap#L660>).
> 
> Without the system-curl option, the cmake build will use its own bundled libcurl 
> without OpenSSL support, and thus downloading tarballs via https fails.
> 
> Adding the --system-curl option would entail that libcurl also be built as a 
> host library. I don't think it currently is.

  Not only that, we'd need to configure host-curl to pull in host-openssl as well.

  That is not really great. It means that cmake (which in many configurations is 
a very basic dependency) would pull in two more packages, which significantly 
increases build time.

  And the only added value is that it allows you to build something outside of 
buildroot. If you need to do that, simply install cmake on your build system and 
use that instead of the buildroot-built cmake.

  Regards,
  Arnout


> 
> I use CMake's ExternalProject_add in our project to download and build the 
> static library dependencies that our embedded application needs. I know that 
> buildroot provides various packages for cross-compiled libraries, but I have no 
> control over the versions (and compile-time options) of these packages.
> 
> Cheers,
> Emile Cormier
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [Feature Request] Host cmake built with the --system-curl option

[Emile Cormier]

[-- Attachment #1.1: Type: text/plain, Size: 2074 bytes --]

Thank you for your feedback on this request, Arnout.

Buildroot's cmake is being executed in my post-build script instead of my
system's cmake. How do I make my system's cmake take precedence?

On Wed, Apr 6, 2022 at 11:29 AM Arnout Vandecappelle <arnout@mind.be> wrote:

>
>
> On 06/04/2022 04:06, Emile Cormier wrote:
> > When using cmake while running a post-build script, the
> ExternalProject_add
> > command fails when the URL option is used to download a source tarball
> via
> > https. This is because the host cmake is not built using the
> --system-curl
> > option (https://github.com/Kitware/CMake/blob/master/bootstrap#L660
> > <https://github.com/Kitware/CMake/blob/master/bootstrap#L660>).
> >
> > Without the system-curl option, the cmake build will use its own bundled
> libcurl
> > without OpenSSL support, and thus downloading tarballs via https fails.
> >
> > Adding the --system-curl option would entail that libcurl also be built
> as a
> > host library. I don't think it currently is.
>
>   Not only that, we'd need to configure host-curl to pull in host-openssl
> as well.
>
>   That is not really great. It means that cmake (which in many
> configurations is
> a very basic dependency) would pull in two more packages, which
> significantly
> increases build time.
>
>   And the only added value is that it allows you to build something
> outside of
> buildroot. If you need to do that, simply install cmake on your build
> system and
> use that instead of the buildroot-built cmake.
>
>   Regards,
>   Arnout
>
>
> >
> > I use CMake's ExternalProject_add in our project to download and build
> the
> > static library dependencies that our embedded application needs. I know
> that
> > buildroot provides various packages for cross-compiled libraries, but I
> have no
> > control over the versions (and compile-time options) of these packages.
> >
> > Cheers,
> > Emile Cormier
> >
> > _______________________________________________
> > buildroot mailing list
> > buildroot@buildroot.org
> > https://lists.buildroot.org/mailman/listinfo/buildroot
>

[-- Attachment #1.2: Type: text/html, Size: 2995 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [Feature Request] Host cmake built with the --system-curl option

[James Hilliard]

On Wed, Apr 6, 2022 at 2:13 PM Emile Cormier <emile.cormier.jr@gmail.com> wrote:
>
> Thank you for your feedback on this request, Arnout.
>
> Buildroot's cmake is being executed in my post-build script instead of my system's cmake. How do I make my system's cmake take precedence?

I think you just need to update your system's cmake to something that meets the
required minimum version:
https://github.com/buildroot/buildroot/blob/ac91235797fe9b234fb820c0112f21c26be1349f/support/dependencies/check-host-cmake.mk#L6

>
> On Wed, Apr 6, 2022 at 11:29 AM Arnout Vandecappelle <arnout@mind.be> wrote:
>>
>>
>>
>> On 06/04/2022 04:06, Emile Cormier wrote:
>> > When using cmake while running a post-build script, the ExternalProject_add
>> > command fails when the URL option is used to download a source tarball via
>> > https. This is because the host cmake is not built using the --system-curl
>> > option (https://github.com/Kitware/CMake/blob/master/bootstrap#L660
>> > <https://github.com/Kitware/CMake/blob/master/bootstrap#L660>).
>> >
>> > Without the system-curl option, the cmake build will use its own bundled libcurl
>> > without OpenSSL support, and thus downloading tarballs via https fails.
>> >
>> > Adding the --system-curl option would entail that libcurl also be built as a
>> > host library. I don't think it currently is.
>>
>>   Not only that, we'd need to configure host-curl to pull in host-openssl as well.
>>
>>   That is not really great. It means that cmake (which in many configurations is
>> a very basic dependency) would pull in two more packages, which significantly
>> increases build time.
>>
>>   And the only added value is that it allows you to build something outside of
>> buildroot. If you need to do that, simply install cmake on your build system and
>> use that instead of the buildroot-built cmake.
>>
>>   Regards,
>>   Arnout
>>
>>
>> >
>> > I use CMake's ExternalProject_add in our project to download and build the
>> > static library dependencies that our embedded application needs. I know that
>> > buildroot provides various packages for cross-compiled libraries, but I have no
>> > control over the versions (and compile-time options) of these packages.
>> >
>> > Cheers,
>> > Emile Cormier
>> >
>> > _______________________________________________
>> > buildroot mailing list
>> > buildroot@buildroot.org
>> > https://lists.buildroot.org/mailman/listinfo/buildroot
>
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [Feature Request] Host cmake built with the --system-curl option

[Emile Cormier]

[-- Attachment #1.1: Type: text/plain, Size: 1353 bytes --]

On Wed, Apr 6, 2022 at 6:50 PM James Hilliard <james.hilliard1@gmail.com>
wrote:

> On Wed, Apr 6, 2022 at 2:13 PM Emile Cormier <emile.cormier.jr@gmail.com>
> wrote:
> > Buildroot's cmake is being executed in my post-build script instead of
> my system's cmake. How do I make my system's cmake take precedence?
>
> I think you just need to update your system's cmake to something that
> meets the
> required minimum version:
>
> https://github.com/buildroot/buildroot/blob/ac91235797fe9b234fb820c0112f21c26be1349f/support/dependencies/check-host-cmake.mk#L6
>

Ah, that explains why I started getting CMake https download failures
_after_ the buildroot cmake version was bumped. My system's cmake version
was in between buildroot's versions for cmake.

However, I think having to rely on the system's CMake version goes against
the buildroot philosophy of an image build not relying on the tools and
libraries available on the build system.

What I have done as a workaround in my post-build script is pre-download
the needed source tarballs via wget. Later, when ExternalProject_add
attempts to download the same source tarball, it sees that it's already
downloaded and matches the expected hash. It then skips the https download,
thus preventing the error. This way, whatever version of wget is installed
on the build system makes little difference.

[-- Attachment #1.2: Type: text/html, Size: 2011 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [Feature Request] Host cmake built with the --system-curl option

[James Hilliard]

On Wed, Apr 6, 2022 at 4:15 PM Emile Cormier <emile.cormier.jr@gmail.com> wrote:
>
> On Wed, Apr 6, 2022 at 6:50 PM James Hilliard <james.hilliard1@gmail.com> wrote:
>>
>> On Wed, Apr 6, 2022 at 2:13 PM Emile Cormier <emile.cormier.jr@gmail.com> wrote:
>> > Buildroot's cmake is being executed in my post-build script instead of my system's cmake. How do I make my system's cmake take precedence?

Why are you executing cmake in a post-build script?

You should probably create a proper cmake package in your BR2_EXTERNAL:
https://buildroot.org/downloads/manual/manual.html#_infrastructure_for_cmake_based_packages
https://buildroot.org/downloads/manual/manual.html#outside-br-custom

>>
>> I think you just need to update your system's cmake to something that meets the
>> required minimum version:
>> https://github.com/buildroot/buildroot/blob/ac91235797fe9b234fb820c0112f21c26be1349f/support/dependencies/check-host-cmake.mk#L6
>
>
> Ah, that explains why I started getting CMake https download failures _after_ the buildroot cmake version was bumped. My system's cmake version was in between buildroot's versions for cmake.
>
> However, I think having to rely on the system's CMake version goes against the buildroot philosophy of an image build not relying on the tools and libraries available on the build system.

Yes, however the issue here is that you are bypassing buildroot's package
download infrastructure.

>
> What I have done as a workaround in my post-build script is pre-download the needed source tarballs via wget. Later, when ExternalProject_add attempts to download the same source tarball, it sees that it's already downloaded and matches the expected hash. It then skips the https download, thus preventing the error. This way, whatever version of wget is installed on the build system makes little difference.

So create a separate buildroot package for those dependencies that your main
package selects.

Use the cmake package infrastructure instead of trying to use cmake to download
the dependency.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [Feature Request] Host cmake built with the --system-curl option

[Emile Cormier]

[-- Attachment #1.1: Type: text/plain, Size: 1449 bytes --]

On Wed, Apr 6, 2022 at 7:27 PM James Hilliard <james.hilliard1@gmail.com>
wrote:

> Why are you executing cmake in a post-build script?
>
> You should probably create a proper cmake package in your BR2_EXTERNAL:
>
> https://buildroot.org/downloads/manual/manual.html#_infrastructure_for_cmake_based_packages
> https://buildroot.org/downloads/manual/manual.html#outside-br-custom
>

I should have clarified that the same CMake-based superbuild project is
used to compile desktop versions of the same app for development purposes.
All the (permissive-licenced) dependencies are statically linked to avoid
headaches where the desktop's dynamic libraries are used instead of the
ones intended for the app. As I'm sure you know, the edit/run/debug cycle
is much faster when it doesn't have to be uploaded and remotely debugged on
an embedded device.

Nonetheless, I'll consider your advice and will check if the CMake
superbuild of our app can be hooked into a BR2_EXTERNAL cmake package
instead of a post-build script. Thank you.

So create a separate buildroot package for those dependencies that your main
> package selects.
>
> Use the cmake package infrastructure instead of trying to use cmake to
> download
> the dependency.
>

Again, the same CMake superbuild is used for desktop builds and I don't
want to duplicate all that effort into equivalent buildroot cmake packages.
Sorry for not making clearer the dual purpose of our CMake superbuild.

[-- Attachment #1.2: Type: text/html, Size: 2259 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [Feature Request] Host cmake built with the --system-curl option

[James Hilliard]

On Wed, Apr 6, 2022 at 5:02 PM Emile Cormier <emile.cormier.jr@gmail.com> wrote:
>
> On Wed, Apr 6, 2022 at 7:27 PM James Hilliard <james.hilliard1@gmail.com> wrote:
>>
>> Why are you executing cmake in a post-build script?
>>
>> You should probably create a proper cmake package in your BR2_EXTERNAL:
>> https://buildroot.org/downloads/manual/manual.html#_infrastructure_for_cmake_based_packages
>> https://buildroot.org/downloads/manual/manual.html#outside-br-custom
>
>
> I should have clarified that the same CMake-based superbuild project is used to compile desktop versions of the same app for development purposes. All the (permissive-licenced) dependencies are statically linked to avoid headaches where the desktop's dynamic libraries are used instead of the ones intended for the app. As I'm sure you know, the edit/run/debug cycle is much faster when it doesn't have to be uploaded and remotely debugged on an embedded device.
>
> Nonetheless, I'll consider your advice and will check if the CMake superbuild of our app can be hooked into a BR2_EXTERNAL cmake package instead of a post-build script. Thank you.
>
>> So create a separate buildroot package for those dependencies that your main
>> package selects.
>>
>> Use the cmake package infrastructure instead of trying to use cmake to download
>> the dependency.
>
>
> Again, the same CMake superbuild is used for desktop builds and I don't want to duplicate all that effort into equivalent buildroot cmake packages. Sorry for not making clearer the dual purpose of our CMake superbuild.

You might still be able to share it for desktop builds for development
by creating
packages and then overriding them using the OVERRIDE_SRCDIR feature:
https://buildroot.org/downloads/manual/manual.html#_using_buildroot_during_development
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [Feature Request] Host cmake built with the --system-curl option

[Emile Cormier]

[-- Attachment #1.1: Type: text/plain, Size: 1429 bytes --]

On Wed, Apr 6, 2022 at 8:11 PM James Hilliard <james.hilliard1@gmail.com>
wrote:

> You might still be able to share it for desktop builds for development
> by creating
> packages and then overriding them using the OVERRIDE_SRCDIR feature:
>
> https://buildroot.org/downloads/manual/manual.html#_using_buildroot_during_development
>

We use Qt Creator as an IDE, and it integrates very well with CMake-based
projects, especially for debugging. I'm afraid that introducing buildroot
into the mix would wreck our productivity for development.

The way I have it set up now with buildroot being exclusively used to
produce "release" images for our embedded device is working well. Before
buildroot, release images were assembled manually using a Debian ARM image
provided by the board vendor, and then tailored to suit our needs. It goes
without saying this was very tedious and error-prone. Using buildroot to
generate device images is way easier and more consistent, plus we can
tailor the operating system exactly as we need it.

It turns out that the problem with buildroot's cmake not supporting https
downloads is a minor one that I can solve it two ways:
1. Pre-download the tarballs using wget via build scripts
2. Install a more recent cmake on the build system that meets buildroot's
requirements

If this feature request is declined (as perhaps it should be), then it's no
big deal, really.

Thanks again for the advice.

[-- Attachment #1.2: Type: text/html, Size: 2026 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [Feature Request] Host cmake built with the --system-curl option

[James Hilliard]

On Wed, Apr 6, 2022 at 6:36 PM Emile Cormier <emile.cormier.jr@gmail.com> wrote:
>
> On Wed, Apr 6, 2022 at 8:11 PM James Hilliard <james.hilliard1@gmail.com> wrote:
>>
>> You might still be able to share it for desktop builds for development
>> by creating
>> packages and then overriding them using the OVERRIDE_SRCDIR feature:
>> https://buildroot.org/downloads/manual/manual.html#_using_buildroot_during_development
>
>
> We use Qt Creator as an IDE, and it integrates very well with CMake-based projects, especially for debugging. I'm afraid that introducing buildroot into the mix would wreck our productivity for development.
>
> The way I have it set up now with buildroot being exclusively used to produce "release" images for our embedded device is working well. Before buildroot, release images were assembled manually using a Debian ARM image provided by the board vendor, and then tailored to suit our needs. It goes without saying this was very tedious and error-prone. Using buildroot to generate device images is way easier and more consistent, plus we can tailor the operating system exactly as we need it.

Using OVERRIDE_SRCDIR would just allow you to easily generate release images
from your cmake projects working tree using buildroot infrastructure
which shouldn't
conflict with Qt Creator if that's only being used for developing the
cmake project.

The idea is that you use your existing debugging setup pretty much as is but
redirect the buildroot package infrastructure to source your cmake package from
your local development folder when you want to build a test release image.

This way your release process would properly use buildroot package
infrastructure
for dependency handling while still allowing debugging to be done
separately most
of the time using existing workflows.

>
> It turns out that the problem with buildroot's cmake not supporting https downloads is a minor one that I can solve it two ways:
> 1. Pre-download the tarballs using wget via build scripts
> 2. Install a more recent cmake on the build system that meets buildroot's requirements
>
> If this feature request is declined (as perhaps it should be), then it's no big deal, really.
>
> Thanks again for the advice.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot