[Buildroot] [PATCH 0/2] package/grpc: 1.31.0

[Buildroot] [PATCH 0/2] package/grpc: 1.31.0

[Michael Nosthoff]

grpc 1.31.0 introduces an new dependency on Google RE2
for Regular Expression parsing.

This series adds RE2 as a new package and bumps
grpc to 1.31.0 with a dependency on RE2

Michael Nosthoff (2):
  package/re2: new package
  package/grpc: bump version to 1.31.0

 package/Config.in                             |  1 +
 .../0001-target-build-using-host-plugin.patch |  4 +--
 ...the-availability-of-pthread_setname_.patch |  7 +++---
 package/grpc/Config.in                        |  3 ++-
 package/grpc/grpc.hash                        |  2 +-
 package/grpc/grpc.mk                          | 12 +++++----
 package/re2/Config.in                         | 12 +++++++++
 package/re2/re2.hash                          |  3 +++
 package/re2/re2.mk                            | 25 +++++++++++++++++++
 9 files changed, 56 insertions(+), 13 deletions(-)
 create mode 100644 package/re2/Config.in
 create mode 100644 package/re2/re2.hash
 create mode 100644 package/re2/re2.mk

-- 
2.25.1

[Buildroot] [PATCH 1/2] package/re2: new package

[Michael Nosthoff]

RE2 is a fast, safe, thread-friendly alternative to backtracking
regular expression engines like those used in PCRE, Perl, and
Python. It is a C++ library.

Signed-off-by: Michael Nosthoff <buildroot@heine.tech>
---
 package/Config.in     |  1 +
 package/re2/Config.in | 12 ++++++++++++
 package/re2/re2.hash  |  3 +++
 package/re2/re2.mk    | 25 +++++++++++++++++++++++++
 4 files changed, 41 insertions(+)
 create mode 100644 package/re2/Config.in
 create mode 100644 package/re2/re2.hash
 create mode 100644 package/re2/re2.mk

diff --git a/package/Config.in b/package/Config.in
index c637b5427f..f6e9524a0f 100644
--- a/package/Config.in
+++ b/package/Config.in
@@ -1942,6 +1942,7 @@ menu "Text and terminal handling"
 	source "package/pcre/Config.in"
 	source "package/pcre2/Config.in"
 	source "package/popt/Config.in"
+	source "package/re2/Config.in"
 	source "package/readline/Config.in"
 	source "package/slang/Config.in"
 	source "package/tclap/Config.in"
diff --git a/package/re2/Config.in b/package/re2/Config.in
new file mode 100644
index 0000000000..d1dcb693ea
--- /dev/null
+++ b/package/re2/Config.in
@@ -0,0 +1,12 @@
+config BR2_PACKAGE_RE2
+	bool "re2"
+	depends on BR2_INSTALL_LIBSTDCPP
+	depends on BR2_TOOLCHAIN_HAS_ATOMIC
+	depends on BR2_TOOLCHAIN_GCC_AT_LEAST_4_8 # needs C++11
+	help
+	  RE2 is a fast, safe, thread-friendly alternative
+	  to backtracking regular expression engines like
+	  those used in PCRE, Perl, and Python.
+	  It is a C++ library.
+
+	  https://github.com/google/re2
diff --git a/package/re2/re2.hash b/package/re2/re2.hash
new file mode 100644
index 0000000000..b8a80513b7
--- /dev/null
+++ b/package/re2/re2.hash
@@ -0,0 +1,3 @@
+# locally calculated
+sha256  6f4c8514249cd65b9e85d3e6f4c35595809a63ad71c5d93083e4d1dcdf9e0cd6  re2-2020-08-01.tar.gz
+sha256  6040cda75d90b1738292a631d89934c411ef7ffd543c4d6a1b7edfc8edf29449  LICENSE
diff --git a/package/re2/re2.mk b/package/re2/re2.mk
new file mode 100644
index 0000000000..d2791a460b
--- /dev/null
+++ b/package/re2/re2.mk
@@ -0,0 +1,25 @@
+################################################################################
+#
+# Google RE2
+#
+################################################################################
+
+RE2_VERSION = 2020-08-01
+RE2_SITE = $(call github,google,re2,$(RE2_VERSION))
+RE2_LICENSE = BSD
+RE2_LICENSE_FILES = LICENSE
+
+RE2_INSTALL_STAGING = YES
+
+RE2_CONF_OPTS += -DRE2_BUILD_TESTING=OFF
+HOST_RE2_CONF_OPTS += -DRE2_BUILD_TESTING=OFF
+
+ifeq ($(BR2_SHARED_LIBS)$(BR2_SHARED_STATIC_LIBS),y)
+RE2_CONF_OPTS += \
+	-DBUILD_SHARED_LIBS=ON
+HOST_RE2_CONF_OPTS += \
+	-DBUILD_SHARED_LIBS=ON
+endif
+
+$(eval $(cmake-package))
+$(eval $(host-cmake-package))
-- 
2.25.1

[Buildroot] [PATCH 2/2] package/grpc: bump version to 1.31.0

[Michael Nosthoff]

Related changes:
 - add dependency on Google RE2 package
 - update patches to new offsets

Tested on Ubuntu 20.04

Signed-off-by: Michael Nosthoff <buildroot@heine.tech>
---
 .../grpc/0001-target-build-using-host-plugin.patch   |  4 ++--
 ...detect-the-availability-of-pthread_setname_.patch |  7 +++----
 package/grpc/Config.in                               |  3 ++-
 package/grpc/grpc.hash                               |  2 +-
 package/grpc/grpc.mk                                 | 12 +++++++-----
 5 files changed, 15 insertions(+), 13 deletions(-)

diff --git a/package/grpc/0001-target-build-using-host-plugin.patch b/package/grpc/0001-target-build-using-host-plugin.patch
index ac2bf8cc38..5b3c6a6fae 100644
--- a/package/grpc/0001-target-build-using-host-plugin.patch
+++ b/package/grpc/0001-target-build-using-host-plugin.patch
@@ -20,7 +20,7 @@ diff --git a/CMakeLists.txt b/CMakeLists.txt
 index be695c2..45c2fcb 100644
 --- a/CMakeLists.txt
 +++ b/CMakeLists.txt
-@@ -10844,6 +10844,18 @@ target_link_libraries(grpc_cli
+@@ -11043,6 +11043,18 @@ target_link_libraries(grpc_cli
  endif()
  if(gRPC_BUILD_CODEGEN AND gRPC_BUILD_GRPC_CPP_PLUGIN)
  
@@ -39,7 +39,7 @@ index be695c2..45c2fcb 100644
  add_executable(grpc_cpp_plugin
    src/compiler/cpp_plugin.cc
  )
-@@ -10877,6 +10889,7 @@ if(gRPC_INSTALL)
+@@ -11077,6 +11089,7 @@ if(gRPC_INSTALL)
      ARCHIVE DESTINATION ${gRPC_INSTALL_LIBDIR}
    )
  endif()
diff --git a/package/grpc/0002-Properly-detect-the-availability-of-pthread_setname_.patch b/package/grpc/0002-Properly-detect-the-availability-of-pthread_setname_.patch
index 4b7253f7aa..c8b1dc8e73 100644
--- a/package/grpc/0002-Properly-detect-the-availability-of-pthread_setname_.patch
+++ b/package/grpc/0002-Properly-detect-the-availability-of-pthread_setname_.patch
@@ -20,7 +20,7 @@ diff --git a/CMakeLists.txt b/CMakeLists.txt
 index 45c2fcb..0b2be4f 100644
 --- a/CMakeLists.txt
 +++ b/CMakeLists.txt
-@@ -177,6 +177,12 @@ if (NOT DEFINED CMAKE_POSITION_INDEPENDENT_CODE)
+@@ -180,6 +180,12 @@ if (NOT DEFINED CMAKE_POSITION_INDEPENDENT_CODE)
  endif()
  list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake/modules")
  
@@ -37,7 +37,7 @@ diff --git a/include/grpc/impl/codegen/port_platform.h b/include/grpc/impl/codeg
 index 4f213ff..55ecd9d 100644
 --- a/include/grpc/impl/codegen/port_platform.h
 +++ b/include/grpc/impl/codegen/port_platform.h
-@@ -195,7 +195,9 @@
+@@ -163,7 +163,9 @@
  #endif /* _LP64 */
  #ifdef __GLIBC__
  #define GPR_POSIX_CRASH_HANDLER 1
@@ -48,5 +48,4 @@ index 4f213ff..55ecd9d 100644
  #else /* musl libc */
  #define GPR_MUSL_LIBC_COMPAT 1
 -- 
-2.26.2
-
+2.26.63
diff --git a/package/grpc/Config.in b/package/grpc/Config.in
index a2da3f4c53..2eb7c1de1c 100644
--- a/package/grpc/Config.in
+++ b/package/grpc/Config.in
@@ -3,7 +3,7 @@ config BR2_PACKAGE_GRPC
 	depends on BR2_INSTALL_LIBSTDCPP
 	depends on BR2_TOOLCHAIN_HAS_THREADS # protobuf
 	depends on BR2_PACKAGE_PROTOBUF_ARCH_SUPPORTS # protobuf
-	depends on BR2_TOOLCHAIN_GCC_AT_LEAST_4_8 # protobuf
+	depends on BR2_TOOLCHAIN_GCC_AT_LEAST_4_8 # protobuf, re2
 	depends on !BR2_STATIC_LIBS # protobuf, libabseil-cpp
 	depends on BR2_TOOLCHAIN_HAS_SYNC_4 || BR2_TOOLCHAIN_HAS_ATOMIC
 	depends on BR2_PACKAGE_LIBABSEIL_CPP_ARCH_SUPPORTS
@@ -11,6 +11,7 @@ config BR2_PACKAGE_GRPC
 	select BR2_PACKAGE_LIBABSEIL_CPP
 	select BR2_PACKAGE_OPENSSL
 	select BR2_PACKAGE_PROTOBUF
+	select BR2_PACKAGE_RE2
 	select BR2_PACKAGE_ZLIB
 	help
 	  A language-neutral, platform-neutral, open source, remote
diff --git a/package/grpc/grpc.hash b/package/grpc/grpc.hash
index f0586e4b18..20368068d9 100644
--- a/package/grpc/grpc.hash
+++ b/package/grpc/grpc.hash
@@ -1,3 +1,3 @@
 # Locally computed
-sha256  ba74b97a2f1b4e22ec5fb69d639d849d2069fb58ea7d6579a31f800af6fe3b6c  grpc-1.30.2.tar.gz
+sha256  1236514199d3deb111a6dd7f6092f67617cd2b147f7eda7adbafccea95de7381  grpc-1.31.0.tar.gz
 sha256  cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30  LICENSE
diff --git a/package/grpc/grpc.mk b/package/grpc/grpc.mk
index 087deac0c6..208abd90c8 100644
--- a/package/grpc/grpc.mk
+++ b/package/grpc/grpc.mk
@@ -4,7 +4,7 @@
 #
 ################################################################################
 
-GRPC_VERSION = 1.30.2
+GRPC_VERSION = 1.31.0
 GRPC_SITE = $(call github,grpc,grpc,v$(GRPC_VERSION))
 GRPC_LICENSE = Apache-2.0
 GRPC_LICENSE_FILES = LICENSE
@@ -12,10 +12,10 @@ GRPC_LICENSE_FILES = LICENSE
 GRPC_INSTALL_STAGING = YES
 
 # Need to use host grpc_cpp_plugin during cross compilation.
-GRPC_DEPENDENCIES = c-ares host-grpc openssl protobuf zlib libabseil-cpp
-HOST_GRPC_DEPENDENCIES = host-c-ares host-openssl host-protobuf host-zlib \
-	host-libabseil-cpp
-
+GRPC_DEPENDENCIES = c-ares host-grpc openssl protobuf re2 zlib libabseil-cpp
+HOST_GRPC_DEPENDENCIES = host-c-ares host-libabseil-cpp host-openssl host-protobuf \
+	host-re2 host-zlib
+	
 # gRPC_CARES_PROVIDER=package won't work because it requires c-ares to have
 # installed a cmake config file, but buildroot uses c-ares' autotools build,
 # which doesn't do this.  These CARES settings trick the gRPC cmake code into
@@ -24,6 +24,7 @@ GRPC_CONF_OPTS = \
 	-D_gRPC_CARES_LIBRARIES=cares \
 	-DgRPC_CARES_PROVIDER=none \
 	-DgRPC_PROTOBUF_PROVIDER=package \
+	-DgRPC_RE2_PROVIDER=package \
 	-DgRPC_SSL_PROVIDER=package \
 	-DgRPC_ZLIB_PROVIDER=package \
 	-DgRPC_ABSL_PROVIDER=package \
@@ -66,6 +67,7 @@ HOST_GRPC_CONF_OPTS = \
 	-D_gRPC_CARES_LIBRARIES=cares \
 	-DgRPC_CARES_PROVIDER=none \
 	-DgRPC_PROTOBUF_PROVIDER=package \
+	-DgRPC_RE2_PROVIDER=package \
 	-DgRPC_SSL_PROVIDER=package \
 	-DgRPC_ZLIB_PROVIDER=package \
 	-DgRPC_ABSL_PROVIDER=package
-- 
2.25.1

[Buildroot] [PATCH 1/2] package/re2: new package

[Thomas Petazzoni]

On Thu, 13 Aug 2020 16:45:48 +0200
Michael Nosthoff via buildroot <buildroot@busybox.net> wrote:

> RE2 is a fast, safe, thread-friendly alternative to backtracking
> regular expression engines like those used in PCRE, Perl, and
> Python. It is a C++ library.
> 
> Signed-off-by: Michael Nosthoff <buildroot@heine.tech>

Thanks, I've applied to master, with a few changes.

> ---
>  package/Config.in     |  1 +
>  package/re2/Config.in | 12 ++++++++++++
>  package/re2/re2.hash  |  3 +++
>  package/re2/re2.mk    | 25 +++++++++++++++++++++++++
>  4 files changed, 41 insertions(+)

You forgot to add an entry in the DEVELOPERS file, so I've done so.

> diff --git a/package/re2/Config.in b/package/re2/Config.in
> new file mode 100644
> index 0000000000..d1dcb693ea
> --- /dev/null
> +++ b/package/re2/Config.in
> @@ -0,0 +1,12 @@
> +config BR2_PACKAGE_RE2
> +	bool "re2"
> +	depends on BR2_INSTALL_LIBSTDCPP
> +	depends on BR2_TOOLCHAIN_HAS_ATOMIC

Even though it uses std::atomic, I'm not sure it needs the atomic
intrinsics of the compiler. Indeed, I tried building for SPARC, where
the atomic built-ins are implemented in the libatomic.so library, and
even without linking to this library, re2 builds fine.

So I've dropped this dependency. We'll see if we have any build
failure, and we can adjust.

> +	depends on BR2_TOOLCHAIN_GCC_AT_LEAST_4_8 # needs C++11
> +	help
> +	  RE2 is a fast, safe, thread-friendly alternative
> +	  to backtracking regular expression engines like
> +	  those used in PCRE, Perl, and Python.
> +	  It is a C++ library.
> +
> +	  https://github.com/google/re2

You forgot a Config.in comment about the dependency (C++, gcc >= 4.8),
so I've added that.


> +RE2_VERSION = 2020-08-01
> +RE2_SITE = $(call github,google,re2,$(RE2_VERSION))
> +RE2_LICENSE = BSD

BSD is not specific enough. In this case, it should have been BSD-3-Clause.

> +ifeq ($(BR2_SHARED_LIBS)$(BR2_SHARED_STATIC_LIBS),y)
> +RE2_CONF_OPTS += \
> +	-DBUILD_SHARED_LIBS=ON
> +HOST_RE2_CONF_OPTS += \
> +	-DBUILD_SHARED_LIBS=ON

This was not needed: BUILD_SHARED_LIBS is handled by the cmake-package
infrastructure.

Thanks, applied with those changes!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 2/2] package/grpc: bump version to 1.31.0

[Thomas Petazzoni]

On Thu, 13 Aug 2020 16:45:49 +0200
Michael Nosthoff via buildroot <buildroot@busybox.net> wrote:

> Related changes:
>  - add dependency on Google RE2 package
>  - update patches to new offsets
> 
> Tested on Ubuntu 20.04
> 
> Signed-off-by: Michael Nosthoff <buildroot@heine.tech>
> ---
>  .../grpc/0001-target-build-using-host-plugin.patch   |  4 ++--
>  ...detect-the-availability-of-pthread_setname_.patch |  7 +++----
>  package/grpc/Config.in                               |  3 ++-
>  package/grpc/grpc.hash                               |  2 +-
>  package/grpc/grpc.mk                                 | 12 +++++++-----
>  5 files changed, 15 insertions(+), 13 deletions(-)

Applied to next, thanks. Note that you had forgotten to propagate the
BR2_TOOLCHAIN_HAS_ATOMIC dependency of the re2 package, but since I
dropped that dependency for now, that's OK.

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 1/2] package/re2: new package

[Thomas Petazzoni]

On Thu, 13 Aug 2020 16:45:48 +0200
Michael Nosthoff via buildroot <buildroot@busybox.net> wrote:

> RE2 is a fast, safe, thread-friendly alternative to backtracking
> regular expression engines like those used in PCRE, Perl, and
> Python. It is a C++ library.
> 
> Signed-off-by: Michael Nosthoff <buildroot@heine.tech>
> ---
>  package/Config.in     |  1 +
>  package/re2/Config.in | 12 ++++++++++++
>  package/re2/re2.hash  |  3 +++
>  package/re2/re2.mk    | 25 +++++++++++++++++++++++++
>  4 files changed, 41 insertions(+)
>  create mode 100644 package/re2/Config.in
>  create mode 100644 package/re2/re2.hash
>  create mode 100644 package/re2/re2.mk

It seems like there is an issue with this package:

  http://autobuild.buildroot.net/results/a98/a98d3203f68f0f929c544537244e7621e80ce0a1/build-end.log

Apparently, only a static version of libre2 gets installed, which is
built without -fPIC (makes sense for a static library). But then it
gets linked into a shared library, which booms as all the code in a
shared library must be compiled with -fPIC.

Perhaps the issue is that we're not explicitly passing
-DBUILD_SHARED_LIBS=ON to host CMake packages in package/pkg-cmake.mk ?

Could you have a look into this ?

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 1/2] package/re2: new package

[Michael Nosthoff]

Hi Thomas,

On 26.08.20 23:30, Thomas Petazzoni wrote:
> On Thu, 13 Aug 2020 16:45:48 +0200
> Michael Nosthoff via buildroot <buildroot@busybox.net> wrote:
>
>> RE2 is a fast, safe, thread-friendly alternative to backtracking
>> regular expression engines like those used in PCRE, Perl, and
>> Python. It is a C++ library.
>>
>> Signed-off-by: Michael Nosthoff <buildroot@heine.tech>
>> ---
>>   package/Config.in     |  1 +
>>   package/re2/Config.in | 12 ++++++++++++
>>   package/re2/re2.hash  |  3 +++
>>   package/re2/re2.mk    | 25 +++++++++++++++++++++++++
>>   4 files changed, 41 insertions(+)
>>   create mode 100644 package/re2/Config.in
>>   create mode 100644 package/re2/re2.hash
>>   create mode 100644 package/re2/re2.mk
> It seems like there is an issue with this package:
>
>    http://autobuild.buildroot.net/results/a98/a98d3203f68f0f929c544537244e7621e80ce0a1/build-end.log
that's not good...
>
> Apparently, only a static version of libre2 gets installed, which is
> built without -fPIC (makes sense for a static library). But then it
> gets linked into a shared library, which booms as all the code in a
> shared library must be compiled with -fPIC.
>
> Perhaps the issue is that we're not explicitly passing
> -DBUILD_SHARED_LIBS=ON to host CMake packages in package/pkg-cmake.mk ?
>
> Could you have a look into this ?


What I'm wondering: Should all host packages always be built as shared libs?


 From the CMake Documentation [1] (which is a bit unclear) I assume that 
as default

CMake will build libs as static. So it might be a coincidence that all 
existing host cmake packages

built as shared as their default.

When comparing to package/pkg-autotools.mk I see that for host packages

 ??? --enable-shared --disable-static

is always set. So it would make sense to also add

 ?? -DBUILD_SHARED_LIBS=ON


to cmake host build flags.



I'm currently on vacation with no access to a proper machine for 
building. So I would

be able to take care of this further earliest next week.


Regards,

Michael



[1] https://cmake.org/cmake/help/latest/variable/BUILD_SHARED_LIBS.html

[Buildroot] [PATCH 1/2] package/re2: new package

[Romain Naour]

Hello,

Le 07/09/2020 ? 17:19, Thomas Petazzoni a ?crit?:
> Hello Michael,
> 
> Romain: there are some LLVM questions below.
> 
> On Mon, 07 Sep 2020 16:43:42 +0200
> "Michael Nosthoff" <buildroot@heine.tech> wrote:
> 
>> I did a check on what might happen when we add -DBUILD_SHARED_LIBS to package/pkg-cmake.mk for host packages.
>>
>> In Summary: a few packages change their output. Most stay stable. (see at the end for details)
> 
> First, thanks a lot for this extensive research.
> 
>> Builds that change their output from .a to .so: host-clang, host-grpc, host-flatcc, host-libabseil-cpp, host-lld, host-pugixml
>> builds that break: host-doxygen
>>
>> So for doxygen that is not good. They mix add_library calls which explicitly set STATIC and one that sets nothing. 
>> So this creates a .so but when it should be linked a .a is needed because the lib to be linked is static.
>> If we change the default doxygen would need an explicit -DBUILD_SHARED_LIBS=OFF Flag.
> 
> I think we could live with one host package explicitly disabling shared
> libraries.
> 
>> What I'm not sure about is the output of clang. Should this be built
>> as a shared lib? For llvm this is explicitly set to static in the
>> .mk. For clang this is not done for host builds. Is this maybe an
>> oversight?
> 
> On this, I don't know, but I've added Romain in Cc.

IIRC, we had some issues when BUILD_SHARED_LIBS was ON:
https://git.buildroot.net/buildroot/tree/package/llvm/llvm.mk#n70
https://git.buildroot.net/buildroot/tree/package/clang/clang.mk#n26

Indeed BUILD_SHARED_LIBS is missing for host-clang, I guess we shoud disable it.

Here is static libraries installed in host directory :
host]$ find -name "libclang*"
./lib/libclangCodeGen.a
./lib/libclangAnalysis.a
./lib/libclangStaticAnalyzerFrontend.a
./lib/libclangAST.a
./lib/libclangBasic.a
./lib/libclangRewrite.a
./lib/libclangHandleLLVM.a
./lib/libclangStaticAnalyzerCheckers.a
./lib/libclangSema.a
./lib/libclangIndex.a
./lib/libclangDriver.a
./lib/libclangFormat.a
./lib/libclangARCMigrate.a
./lib/libclangCrossTU.a
./lib/libclangASTMatchers.a
./lib/libclangFrontend.a
./lib/libclangParse.a
./lib/libclang.so
./lib/libclangHandleCXX.a
./lib/libclangStaticAnalyzerCore.a
./lib/libclangSerialization.a
./lib/libclangToolingRefactor.a
./lib/libclangTooling.a
./lib/libclangDynamicASTMatchers.a
./lib/libclangRewriteFrontend.a
./lib/libclangLex.a
./lib/libclangToolingASTDiff.a
./lib/libclangFrontendTool.a
./lib/libclangToolingCore.a
./lib/libclang.so.8
./lib/libclangEdit.a
./lib/libclangToolingInclusions.a


> 
>> So what do you think? Is this something we should go forward on or
>> should I just change the re2 package?
> 
> I think we should have the simple fix that tweaks just the re2 package,
> and then a more global fix in pkg-cmake.
> 
>> How is -DBUILD_SHARED_LIBS set and/or handled currently on the host packages?
>>
>> host-cdrkit always uses ADD_LIBRARY with STATIC flag
>> host-clang not set explicitly for host, so it might change from STATIC to SHARED here. Is this intentional or an oversight (compared to llvm)? [1]
>> host-doxygen adds most libs with STATIC but not all -> breaks on linking
>> host-fatcat Does not build libs
>> host-flatbuffers uses its own FLATBUFFERS_BUILD_SHAREDLIB, host .mk build sets it to OFF
>> host-flatcc has no preference, CMakeList.txt states explicitly that it honors BUILD_SHARED_LIBS
>> host-grpc honors the flag, .mk sets nothing
>> host-json-c Builds static and shared by default, setting flag doesn't change that
>> host-kodi-jsonschemabuilder only builds executable
>> host-kodi-texturepacker only builds executable
>> host-libabseil-cpp honors the flag, .mk sets nothing
>> host-libnetconf2 always uses ADD_LIBRARY with SHARED flag [2]
>> host-libyang Builds SHARED when ENABLE_STATIC is not set, which is the case [3]
>> host-libzip builds as shared as default
>> host-lld honors the flag, .mk sets nothing
>> host-llvm set to OFF by BR .mk file [4]
>> host-lzo .mk sets -DENABLE_SHARED=ON -DENABLE_STATIC=OFF
>> host-mariadb ignores DBUILD_SHARED_LIBS
>> host-mfgtools .mk defines its own build step (and builds shared lib)
>> host-ninja only build executable
>> host-pugixml honors -DBUILD_SHARED_LIBS
>> host-sysrepo sets shared as default [5]
>> host-thrift Sets to ON when not on windows [6]
>> host-waylandpp Defaults to OFF but doesn't build libs for host (-DBUILD_LIBRARIES=OFF) [7]
> 
> Again, thanks for this extensive research!

Ideed, Thanks!

Best regards,
Romain

> 
> Thomas
> 

[Buildroot] [PATCH 1/2] package/re2: new package

[Michael Nosthoff]

Hi,

okay, then I'll submit a patch for re2 shortly and we can put together a patch series when there is
clarity on how to handle clang.

Regards,
Michael 

On Monday, September 07, 2020 17:19 CEST, Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote: 
 
> Hello Michael,
> 
> Romain: there are some LLVM questions below.
> 
> On Mon, 07 Sep 2020 16:43:42 +0200
> "Michael Nosthoff" <buildroot@heine.tech> wrote:
> 
> > I did a check on what might happen when we add -DBUILD_SHARED_LIBS to package/pkg-cmake.mk for host packages.
> > 
> > In Summary: a few packages change their output. Most stay stable. (see at the end for details)
> 
> First, thanks a lot for this extensive research.
> 
> > Builds that change their output from .a to .so: host-clang, host-grpc, host-flatcc, host-libabseil-cpp, host-lld, host-pugixml
> > builds that break: host-doxygen
> > 
> > So for doxygen that is not good. They mix add_library calls which explicitly set STATIC and one that sets nothing. 
> > So this creates a .so but when it should be linked a .a is needed because the lib to be linked is static.
> > If we change the default doxygen would need an explicit -DBUILD_SHARED_LIBS=OFF Flag.
> 
> I think we could live with one host package explicitly disabling shared
> libraries.
> 
> > What I'm not sure about is the output of clang. Should this be built
> > as a shared lib? For llvm this is explicitly set to static in the
> > .mk. For clang this is not done for host builds. Is this maybe an
> > oversight?
> 
> On this, I don't know, but I've added Romain in Cc.
> 
> > So what do you think? Is this something we should go forward on or
> > should I just change the re2 package?
> 
> I think we should have the simple fix that tweaks just the re2 package,
> and then a more global fix in pkg-cmake.
> 
> > How is -DBUILD_SHARED_LIBS set and/or handled currently on the host packages?
> > 
> > host-cdrkit always uses ADD_LIBRARY with STATIC flag
> > host-clang not set explicitly for host, so it might change from STATIC to SHARED here. Is this intentional or an oversight (compared to llvm)? [1]
> > host-doxygen adds most libs with STATIC but not all -> breaks on linking
> > host-fatcat Does not build libs
> > host-flatbuffers uses its own FLATBUFFERS_BUILD_SHAREDLIB, host .mk build sets it to OFF
> > host-flatcc has no preference, CMakeList.txt states explicitly that it honors BUILD_SHARED_LIBS
> > host-grpc honors the flag, .mk sets nothing
> > host-json-c Builds static and shared by default, setting flag doesn't change that
> > host-kodi-jsonschemabuilder only builds executable
> > host-kodi-texturepacker only builds executable
> > host-libabseil-cpp honors the flag, .mk sets nothing
> > host-libnetconf2 always uses ADD_LIBRARY with SHARED flag [2]
> > host-libyang Builds SHARED when ENABLE_STATIC is not set, which is the case [3]
> > host-libzip builds as shared as default
> > host-lld honors the flag, .mk sets nothing
> > host-llvm set to OFF by BR .mk file [4]
> > host-lzo .mk sets -DENABLE_SHARED=ON -DENABLE_STATIC=OFF
> > host-mariadb ignores DBUILD_SHARED_LIBS
> > host-mfgtools .mk defines its own build step (and builds shared lib)
> > host-ninja only build executable
> > host-pugixml honors -DBUILD_SHARED_LIBS
> > host-sysrepo sets shared as default [5]
> > host-thrift Sets to ON when not on windows [6]
> > host-waylandpp Defaults to OFF but doesn't build libs for host (-DBUILD_LIBRARIES=OFF) [7]
> 
> Again, thanks for this extensive research!
> 
> Thomas
> -- 
> Thomas Petazzoni, CTO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com

[Buildroot] [PATCH 1/2] package/re2: new package

[Thomas Petazzoni]

Hello Michael,

Romain: there are some LLVM questions below.

On Mon, 07 Sep 2020 16:43:42 +0200
"Michael Nosthoff" <buildroot@heine.tech> wrote:

> I did a check on what might happen when we add -DBUILD_SHARED_LIBS to package/pkg-cmake.mk for host packages.
> 
> In Summary: a few packages change their output. Most stay stable. (see at the end for details)

First, thanks a lot for this extensive research.

> Builds that change their output from .a to .so: host-clang, host-grpc, host-flatcc, host-libabseil-cpp, host-lld, host-pugixml
> builds that break: host-doxygen
> 
> So for doxygen that is not good. They mix add_library calls which explicitly set STATIC and one that sets nothing. 
> So this creates a .so but when it should be linked a .a is needed because the lib to be linked is static.
> If we change the default doxygen would need an explicit -DBUILD_SHARED_LIBS=OFF Flag.

I think we could live with one host package explicitly disabling shared
libraries.

> What I'm not sure about is the output of clang. Should this be built
> as a shared lib? For llvm this is explicitly set to static in the
> .mk. For clang this is not done for host builds. Is this maybe an
> oversight?

On this, I don't know, but I've added Romain in Cc.

> So what do you think? Is this something we should go forward on or
> should I just change the re2 package?

I think we should have the simple fix that tweaks just the re2 package,
and then a more global fix in pkg-cmake.

> How is -DBUILD_SHARED_LIBS set and/or handled currently on the host packages?
> 
> host-cdrkit always uses ADD_LIBRARY with STATIC flag
> host-clang not set explicitly for host, so it might change from STATIC to SHARED here. Is this intentional or an oversight (compared to llvm)? [1]
> host-doxygen adds most libs with STATIC but not all -> breaks on linking
> host-fatcat Does not build libs
> host-flatbuffers uses its own FLATBUFFERS_BUILD_SHAREDLIB, host .mk build sets it to OFF
> host-flatcc has no preference, CMakeList.txt states explicitly that it honors BUILD_SHARED_LIBS
> host-grpc honors the flag, .mk sets nothing
> host-json-c Builds static and shared by default, setting flag doesn't change that
> host-kodi-jsonschemabuilder only builds executable
> host-kodi-texturepacker only builds executable
> host-libabseil-cpp honors the flag, .mk sets nothing
> host-libnetconf2 always uses ADD_LIBRARY with SHARED flag [2]
> host-libyang Builds SHARED when ENABLE_STATIC is not set, which is the case [3]
> host-libzip builds as shared as default
> host-lld honors the flag, .mk sets nothing
> host-llvm set to OFF by BR .mk file [4]
> host-lzo .mk sets -DENABLE_SHARED=ON -DENABLE_STATIC=OFF
> host-mariadb ignores DBUILD_SHARED_LIBS
> host-mfgtools .mk defines its own build step (and builds shared lib)
> host-ninja only build executable
> host-pugixml honors -DBUILD_SHARED_LIBS
> host-sysrepo sets shared as default [5]
> host-thrift Sets to ON when not on windows [6]
> host-waylandpp Defaults to OFF but doesn't build libs for host (-DBUILD_LIBRARIES=OFF) [7]

Again, thanks for this extensive research!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 1/2] package/re2: new package

[Michael Nosthoff]

Hi Thomas,

I'm back from vacation so I had a look at the pkg-cmake.mk.

On Sunday, August 30, 2020 16:34 CEST, Michael Nosthoff <buildroot@heine.tech> wrote: 

> >
> > Perhaps the issue is that we're not explicitly passing
> > -DBUILD_SHARED_LIBS=ON to host CMake packages in package/pkg-cmake.mk ?
> >
> > Could you have a look into this ?
> 
> 
> What I'm wondering: Should all host packages always be built as shared libs?
> 
> 
>  From the CMake Documentation [1] (which is a bit unclear) I assume that 
> as default
> CMake will build libs as static. So it might be a coincidence that all 
> existing host cmake packages
> built as shared as their default.
> 
> When comparing to package/pkg-autotools.mk I see that for host packages
> 
>  ??? --enable-shared --disable-static
> 
> is always set. So it would make sense to also add
> 
>  ?? -DBUILD_SHARED_LIBS=ON
> 
> 
> to cmake host build flags.
> 


I did a check on what might happen when we add -DBUILD_SHARED_LIBS to package/pkg-cmake.mk for host packages.

In Summary: a few packages change their output. Most stay stable. (see at the end for details)

Builds that change their output from .a to .so: host-clang, host-grpc, host-flatcc, host-libabseil-cpp, host-lld, host-pugixml
builds that break: host-doxygen

So for doxygen that is not good. They mix add_library calls which explicitly set STATIC and one that sets nothing. 
So this creates a .so but when it should be linked a .a is needed because the lib to be linked is static.
If we change the default doxygen would need an explicit -DBUILD_SHARED_LIBS=OFF Flag.

What I'm not sure about is the output of clang. Should this be built as a shared lib? For llvm this is explicitly set to static in the .mk.
For clang this is not done for host builds. Is this maybe an oversight?

So what do you think? Is this something we should go forward on or should I just change the re2 package?

Regards,
Michael


Detailed overview per package:

How is -DBUILD_SHARED_LIBS set and/or handled currently on the host packages?

host-cdrkit always uses ADD_LIBRARY with STATIC flag
host-clang not set explicitly for host, so it might change from STATIC to SHARED here. Is this intentional or an oversight (compared to llvm)? [1]
host-doxygen adds most libs with STATIC but not all -> breaks on linking
host-fatcat Does not build libs
host-flatbuffers uses its own FLATBUFFERS_BUILD_SHAREDLIB, host .mk build sets it to OFF
host-flatcc has no preference, CMakeList.txt states explicitly that it honors BUILD_SHARED_LIBS
host-grpc honors the flag, .mk sets nothing
host-json-c Builds static and shared by default, setting flag doesn't change that
host-kodi-jsonschemabuilder only builds executable
host-kodi-texturepacker only builds executable
host-libabseil-cpp honors the flag, .mk sets nothing
host-libnetconf2 always uses ADD_LIBRARY with SHARED flag [2]
host-libyang Builds SHARED when ENABLE_STATIC is not set, which is the case [3]
host-libzip builds as shared as default
host-lld honors the flag, .mk sets nothing
host-llvm set to OFF by BR .mk file [4]
host-lzo .mk sets -DENABLE_SHARED=ON -DENABLE_STATIC=OFF
host-mariadb ignores DBUILD_SHARED_LIBS
host-mfgtools .mk defines its own build step (and builds shared lib)
host-ninja only build executable
host-pugixml honors -DBUILD_SHARED_LIBS
host-sysrepo sets shared as default [5]
host-thrift Sets to ON when not on windows [6]
host-waylandpp Defaults to OFF but doesn't build libs for host (-DBUILD_LIBRARIES=OFF) [7]


[1] https://git.busybox.net/buildroot/tree/package/clang/clang.mk#n36
[2] https://github.com/CESNET/libnetconf2/blob/master/CMakeLists.txt#L117
[3] https://github.com/CESNET/libyang/blob/master/CMakeLists.txt#L204
[4] https://git.busybox.net/buildroot/tree/package/llvm/llvm.mk#n75
[5] https://github.com/sysrepo/sysrepo/blob/master/CMakeLists.txt#L187
[6] https://github.com/apache/thrift/blob/master/build/cmake/DefineOptions.cmake#L135
[7] https://github.com/NilsBrause/waylandpp/blob/master/CMakeLists.txt#L44