[Buildroot] [PATCH 1/1] Makefile: workaround make 4.3 issue for 'printvars and 'show-vars'

[Buildroot] [PATCH 1/1] Makefile: workaround make 4.3 issue for 'printvars and 'show-vars'

[Roosen Henri]

[-- Attachment #1.1: Type: text/plain, Size: 2132 bytes --]

Event though the bug with make 4.3 has been reported and fixed, there
has not been a release of make with the fix for a long time, see [1].

As the root cause seems the 'filter' command cannot handle large
chunks of data, like .VARIABLES, we can workaround the problem by
using a foreach command over .VARIABLES, then use the filter command.

It might not be logical to program it that way, but at least the
functionality is now usable.

[1] https://savannah.gnu.org/bugs/?59093#comment10

Signed-off-by: Henri Roosen <henri.roosen@ginzinger.com>
---
 Makefile | 15 ++++-----------
 1 file changed, 4 insertions(+), 11 deletions(-)

diff --git a/Makefile b/Makefile
index bf101ff770..a856f69685 100644
--- a/Makefile
+++ b/Makefile
@@ -1056,23 +1056,16 @@ ifeq ($(NEED_WRAPPER),y)
 	$(Q)$(TOPDIR)/support/scripts/mkmakefile $(TOPDIR) $(O)
 endif
 
-.PHONY: check-make-version
-check-make-version:
-ifneq ($(filter $(RUNNING_MAKE_VERSION),4.3),)
-	@echo "Make 4.3 doesn't support 'printvars' and 'show-vars' recipes"
-	@exit 1
-endif
-
 # printvars prints all the variables currently defined in our
 # Makefiles. Alternatively, if a non-empty VARS variable is passed,
 # only the variables matching the make pattern passed in VARS are
 # displayed.
 # show-vars does the same, but as a JSON dictionnary.
 .PHONY: printvars
-printvars: check-make-version
+printvars:
 	@:
 	$(foreach V, \
-		$(sort $(filter $(VARS),$(.VARIABLES))), \
+		$(sort $(foreach X, $(.VARIABLES), $(filter $(VARS),$(X)))), \
 		$(if $(filter-out environment% default automatic, \
 				$(origin $V)), \
 		$(if $(QUOTED_VARS),\
@@ -1082,11 +1075,11 @@ printvars: check-make-version
 
 .PHONY: show-vars
 show-vars: VARS?=%
-show-vars: check-make-version
+show-vars:
 	@:
 	$(info $(call clean-json, { \
 			$(foreach V, \
-				$(sort $(filter $(VARS),$(.VARIABLES))), \
+				$(sort $(foreach X, $(.VARIABLES), $(filter $(VARS),$(X)))), \
 				$(if $(filter-out environment% default automatic, $(origin $V)), \
 					"$V": { \
 						"expanded": $(call mk-json-str,$($V))$(comma) \
-- 
2.30.2



[-- Attachment #1.2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 7961 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] Makefile: workaround make 4.3 issue for 'printvars and 'show-vars'

[Yann E. MORIN]

Henri, All,

On 2022-07-29 21:49 +0000, Roosen Henri spake thusly:
> Event though the bug with make 4.3 has been reported and fixed, there
> has not been a release of make with the fix for a long time, see [1].
> 
> As the root cause seems the 'filter' command cannot handle large
> chunks of data, like .VARIABLES, we can workaround the problem by
> using a foreach command over .VARIABLES, then use the filter command.
> 
> It might not be logical to program it that way, but at least the
> functionality is now usable.

Awesome! Just awesome! 👍

I just added a small comment in the Makefile to explain that, and
reference the bug.

> [1] https://savannah.gnu.org/bugs/?59093#comment10
> 
> Signed-off-by: Henri Roosen <henri.roosen@ginzinger.com>

Applied to master, thanks.

Regards,
Yann E. MORIN.

> ---
>  Makefile | 15 ++++-----------
>  1 file changed, 4 insertions(+), 11 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index bf101ff770..a856f69685 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -1056,23 +1056,16 @@ ifeq ($(NEED_WRAPPER),y)
>  	$(Q)$(TOPDIR)/support/scripts/mkmakefile $(TOPDIR) $(O)
>  endif
>  
> -.PHONY: check-make-version
> -check-make-version:
> -ifneq ($(filter $(RUNNING_MAKE_VERSION),4.3),)
> -	@echo "Make 4.3 doesn't support 'printvars' and 'show-vars' recipes"
> -	@exit 1
> -endif
> -
>  # printvars prints all the variables currently defined in our
>  # Makefiles. Alternatively, if a non-empty VARS variable is passed,
>  # only the variables matching the make pattern passed in VARS are
>  # displayed.
>  # show-vars does the same, but as a JSON dictionnary.
>  .PHONY: printvars
> -printvars: check-make-version
> +printvars:
>  	@:
>  	$(foreach V, \
> -		$(sort $(filter $(VARS),$(.VARIABLES))), \
> +		$(sort $(foreach X, $(.VARIABLES), $(filter $(VARS),$(X)))), \
>  		$(if $(filter-out environment% default automatic, \
>  				$(origin $V)), \
>  		$(if $(QUOTED_VARS),\
> @@ -1082,11 +1075,11 @@ printvars: check-make-version
>  
>  .PHONY: show-vars
>  show-vars: VARS?=%
> -show-vars: check-make-version
> +show-vars:
>  	@:
>  	$(info $(call clean-json, { \
>  			$(foreach V, \
> -				$(sort $(filter $(VARS),$(.VARIABLES))), \
> +				$(sort $(foreach X, $(.VARIABLES), $(filter $(VARS),$(X)))), \
>  				$(if $(filter-out environment% default automatic, $(origin $V)), \
>  					"$V": { \
>  						"expanded": $(call mk-json-str,$($V))$(comma) \
> -- 
> 2.30.2
> 
> 



> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot


-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] Makefile: workaround make 4.3 issue for 'printvars and 'show-vars'

[Quentin Schulz]

Hi all,

On 7/30/22 14:19, Yann E. MORIN wrote:
> Henri, All,
> 
> On 2022-07-29 21:49 +0000, Roosen Henri spake thusly:
>> Event though the bug with make 4.3 has been reported and fixed, there
>> has not been a release of make with the fix for a long time, see [1].
>>
>> As the root cause seems the 'filter' command cannot handle large
>> chunks of data, like .VARIABLES, we can workaround the problem by
>> using a foreach command over .VARIABLES, then use the filter command.
>>
>> It might not be logical to program it that way, but at least the
>> functionality is now usable.
> 
> Awesome! Just awesome! 👍
> 
> I just added a small comment in the Makefile to explain that, and
> reference the bug.
> 

Just a heads-up that `make printvars` is broken (I've sent a patch for 
it just now) and also that I get a segfault for `make show-vars` (`make 
show-vars VARS="BUSYBOX_%" works just fine).

Cheers,
Quentin
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] Makefile: workaround make 4.3 issue for 'printvars and 'show-vars'

[Roosen Henri]

[-- Attachment #1.1: Type: text/plain, Size: 1801 bytes --]

On Mon, 2022-08-01 at 15:36 +0200, Quentin Schulz wrote:
> Hi all,
> 

Hi Quentin,

> On 7/30/22 14:19, Yann E. MORIN wrote:
> > Henri, All,
> > 
> > On 2022-07-29 21:49 +0000, Roosen Henri spake thusly:
> > > Event though the bug with make 4.3 has been reported and fixed,
> > > there
> > > has not been a release of make with the fix for a long time, see
> > > [1].
> > > 
> > > As the root cause seems the 'filter' command cannot handle large
> > > chunks of data, like .VARIABLES, we can workaround the problem by
> > > using a foreach command over .VARIABLES, then use the filter
> > > command.
> > > 
> > > It might not be logical to program it that way, but at least the
> > > functionality is now usable.
> > 
> > Awesome! Just awesome! 👍
> > 
> > I just added a small comment in the Makefile to explain that, and
> > reference the bug.
> > 
> 
> Just a heads-up that `make printvars` is broken (I've sent a patch
> for 
> it just now) and also that I get a segfault for `make show-vars`
> (`make 
> show-vars VARS="BUSYBOX_%" works just fine).

Thanks for the heads-up! I can reproduce your tests-results.

I made the workaround to get the 'make pkg-stats' functionality up and
running again on recent systems with make 4.3. So I did my testing for
'make printvars' and 'make show-vars' with the VARS variable set to:

"VARS=%_LICENSE %_LICENSE_FILES %_VERSION %_IGNORE_CVES %_CPE_ID"

Sorry I missed testing the empty VARS (=all).

We might revert the workaround for 'show-vars' and install the make 4.3
check again. For 'printvars' we might keep the workaround, as until now
no segfault could be reproduced and I think the 'pkg-stats'
functionality is a nice to have.

What do you think?

Thanks,
Henri

> 
> Cheers,
> Quentin


[-- Attachment #1.2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 7961 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] Makefile: workaround make 4.3 issue for 'printvars and 'show-vars'

[Yann E. MORIN]

Henri, Quentin, All,

On 2022-08-01 14:49 +0000, Roosen Henri spake thusly:
> On Mon, 2022-08-01 at 15:36 +0200, Quentin Schulz wrote:
> > On 7/30/22 14:19, Yann E. MORIN wrote:
> > > On 2022-07-29 21:49 +0000, Roosen Henri spake thusly:
> > > > As the root cause seems the 'filter' command cannot handle large
> > > > chunks of data, like .VARIABLES, we can workaround the problem by
> > > > using a foreach command over .VARIABLES, then use the filter
> > > > command.
> > Just a heads-up that `make printvars` is broken (I've sent a patch
> > for 
> > it just now) and also that I get a segfault for `make show-vars`
> > (`make 
> > show-vars VARS="BUSYBOX_%" works just fine).
> 
> Thanks for the heads-up! I can reproduce your tests-results.
> 
> I made the workaround to get the 'make pkg-stats' functionality up and
> running again on recent systems with make 4.3. So I did my testing for
> 'make printvars' and 'make show-vars' with the VARS variable set to:
> 
> "VARS=%_LICENSE %_LICENSE_FILES %_VERSION %_IGNORE_CVES %_CPE_ID"
> 
> Sorry I missed testing the empty VARS (=all).
> 
> We might revert the workaround for 'show-vars' and install the make 4.3
> check again. For 'printvars' we might keep the workaround, as until now
> no segfault could be reproduced and I think the 'pkg-stats'
> functionality is a nice to have.
> 
> What do you think?

Yes, please partially revert the change to protect just show-vars, so
that the autobuilders and plg-stats are still happy with printvars.

Thanks!

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] Makefile: workaround make 4.3 issue for 'printvars and 'show-vars'

[Roosen Henri]

[-- Attachment #1.1: Type: text/plain, Size: 1940 bytes --]

Hi Yann,

On Mon, 2022-08-01 at 17:30 +0200, Yann E. MORIN wrote:
> Henri, Quentin, All,
> 
> On 2022-08-01 14:49 +0000, Roosen Henri spake thusly:
> > On Mon, 2022-08-01 at 15:36 +0200, Quentin Schulz wrote:
> > > On 7/30/22 14:19, Yann E. MORIN wrote:
> > > > On 2022-07-29 21:49 +0000, Roosen Henri spake thusly:
> > > > > As the root cause seems the 'filter' command cannot handle
> > > > > large
> > > > > chunks of data, like .VARIABLES, we can workaround the
> > > > > problem by
> > > > > using a foreach command over .VARIABLES, then use the filter
> > > > > command.
> > > Just a heads-up that `make printvars` is broken (I've sent a
> > > patch
> > > for 
> > > it just now) and also that I get a segfault for `make show-vars`
> > > (`make 
> > > show-vars VARS="BUSYBOX_%" works just fine).
> > 
> > Thanks for the heads-up! I can reproduce your tests-results.
> > 
> > I made the workaround to get the 'make pkg-stats' functionality up
> > and
> > running again on recent systems with make 4.3. So I did my testing
> > for
> > 'make printvars' and 'make show-vars' with the VARS variable set
> > to:
> > 
> > "VARS=%_LICENSE %_LICENSE_FILES %_VERSION %_IGNORE_CVES %_CPE_ID"
> > 
> > Sorry I missed testing the empty VARS (=all).
> > 
> > We might revert the workaround for 'show-vars' and install the make
> > 4.3
> > check again. For 'printvars' we might keep the workaround, as until
> > now
> > no segfault could be reproduced and I think the 'pkg-stats'
> > functionality is a nice to have.
> > 
> > What do you think?
> 
> Yes, please partially revert the change to protect just show-vars, so
> that the autobuilders and plg-stats are still happy with printvars.

Or.. we might be able to fix show-vars; I guess the root-cause is the
first $(info ...).  If we would be able to bring it into the $(foreach
V ...)..

> 
> Thanks!
> 
> Regards,
> Yann E. MORIN.
> 


[-- Attachment #1.2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 7961 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] Makefile: workaround make 4.3 issue for 'printvars and 'show-vars'

[Yann E. MORIN]

Quentin, All,

On 2022-08-01 15:36 +0200, Quentin Schulz spake thusly:
> On 7/30/22 14:19, Yann E. MORIN wrote:
> >On 2022-07-29 21:49 +0000, Roosen Henri spake thusly:
> >>Event though the bug with make 4.3 has been reported and fixed, there
> >>has not been a release of make with the fix for a long time, see [1].
> >>
> >>As the root cause seems the 'filter' command cannot handle large
> >>chunks of data, like .VARIABLES, we can workaround the problem by
> >>using a foreach command over .VARIABLES, then use the filter command.
> >>
> >>It might not be logical to program it that way, but at least the
> >>functionality is now usable.
> >Awesome! Just awesome! 👍
> Just a heads-up that `make printvars` is broken (I've sent a patch for it
> just now) and also that I get a segfault for `make show-vars` (`make
> show-vars VARS="BUSYBOX_%" works just fine).

Gah, I was pretty sure it worked for me, but indeed show-vars fails,
while printvars does work...

    $ ./utils/docker-run
    docker$ make defconfig; make allyespackageconfig

    docker$ make printvars >/dev/null 2>&1; echo $?
    0
    docker$ make printvars VARS=% >/dev/null 2>&1; echo $?
    0

    docker$ make show-vars >/dev/null 2>&1; echo $?
    Segmentation fault (core dumped)
    139

Grr... :-(

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot