[PATCH v2] dt-bindings: Fix command line length limit calling dt-mk-schema

[PATCH v2] dt-bindings: Fix command line length limit calling dt-mk-schema

[Rob Herring]

As the number of schemas has increased, we're starting to hit the error
"execvp: /bin/sh: Argument list too long". This is due to passing all the
schema files on the command line to dt-mk-schema. It currently is only
with out of tree builds and is intermittent depending on the file path
lengths.

Commit 2ba06cd8565b ("kbuild: Always validate DT binding examples") made
hitting this proplem more likely since the example validation now always
gets the full list of schemas.

Fix this by passing the schema file list in a pipe and using xargs. We end
up doing the find twice, but the time is insignificant compared to the
dt-mk-schema time.

Reported-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Cc: Masahiro Yamada <masahiroy@kernel.org>
Signed-off-by: Rob Herring <robh@kernel.org>
---
 Documentation/devicetree/bindings/Makefile | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)

diff --git a/Documentation/devicetree/bindings/Makefile b/Documentation/devicetree/bindings/Makefile
index 87c76bdabfe6..7782d9985082 100644
--- a/Documentation/devicetree/bindings/Makefile
+++ b/Documentation/devicetree/bindings/Makefile
@@ -14,16 +14,18 @@ $(obj)/%.example.dts: $(src)/%.yaml FORCE
 # Use full schemas when checking %.example.dts
 DT_TMP_SCHEMA := $(obj)/processed-schema-examples.yaml
 
+find_cmd = find $(srctree)/$(src) \( -name '*.yaml' ! \
+		-name 'processed-schema*' ! \
+		-name '*.example.dt.yaml' \)
+
 quiet_cmd_mk_schema = SCHEMA  $@
-      cmd_mk_schema = $(DT_MK_SCHEMA) $(DT_MK_SCHEMA_FLAGS) -o $@ $(real-prereqs)
+      cmd_mk_schema = rm -f $@ ; \
+                      $(if $(DT_MK_SCHEMA_FLAGS), \
+                           echo $(real-prereqs), \
+                           $(find_cmd)) | \
+                      xargs $(DT_MK_SCHEMA) $(DT_MK_SCHEMA_FLAGS) >> $@
 
-DT_DOCS = $(addprefix $(src)/, \
-	$(shell \
-	cd $(srctree)/$(src) && \
-	find * \( -name '*.yaml' ! \
-		-name 'processed-schema*' ! \
-		-name '*.example.dt.yaml' \) \
-	))
+DT_DOCS = $(shell $(find_cmd) | sed -e 's|^$(srctree)/||')
 
 DT_SCHEMA_FILES ?= $(DT_DOCS)
 
-- 
2.20.1

Re: [PATCH v2] dt-bindings: Fix command line length limit calling dt-mk-schema

[Masahiro Yamada]

On Thu, Apr 23, 2020 at 3:57 AM Rob Herring <robh@kernel.org> wrote:
>
> As the number of schemas has increased, we're starting to hit the error
> "execvp: /bin/sh: Argument list too long". This is due to passing all the
> schema files on the command line to dt-mk-schema. It currently is only
> with out of tree builds and is intermittent depending on the file path
> lengths.
>
> Commit 2ba06cd8565b ("kbuild: Always validate DT binding examples") made
> hitting this proplem more likely since the example validation now always
> gets the full list of schemas.
>
> Fix this by passing the schema file list in a pipe and using xargs. We end
> up doing the find twice, but the time is insignificant compared to the
> dt-mk-schema time.
>
> Reported-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Cc: Masahiro Yamada <masahiroy@kernel.org>
> Signed-off-by: Rob Herring <robh@kernel.org>


Reviewed-by: Masahiro Yamada <masahiroy@kernel.org>


> ---
>  Documentation/devicetree/bindings/Makefile | 18 ++++++++++--------
>  1 file changed, 10 insertions(+), 8 deletions(-)
>
> diff --git a/Documentation/devicetree/bindings/Makefile b/Documentation/devicetree/bindings/Makefile
> index 87c76bdabfe6..7782d9985082 100644
> --- a/Documentation/devicetree/bindings/Makefile
> +++ b/Documentation/devicetree/bindings/Makefile
> @@ -14,16 +14,18 @@ $(obj)/%.example.dts: $(src)/%.yaml FORCE
>  # Use full schemas when checking %.example.dts
>  DT_TMP_SCHEMA := $(obj)/processed-schema-examples.yaml
>
> +find_cmd = find $(srctree)/$(src) \( -name '*.yaml' ! \
> +               -name 'processed-schema*' ! \
> +               -name '*.example.dt.yaml' \)
> +
>  quiet_cmd_mk_schema = SCHEMA  $@
> -      cmd_mk_schema = $(DT_MK_SCHEMA) $(DT_MK_SCHEMA_FLAGS) -o $@ $(real-prereqs)
> +      cmd_mk_schema = rm -f $@ ; \
> +                      $(if $(DT_MK_SCHEMA_FLAGS), \
> +                           echo $(real-prereqs), \
> +                           $(find_cmd)) | \
> +                      xargs $(DT_MK_SCHEMA) $(DT_MK_SCHEMA_FLAGS) >> $@
>
> -DT_DOCS = $(addprefix $(src)/, \
> -       $(shell \
> -       cd $(srctree)/$(src) && \
> -       find * \( -name '*.yaml' ! \
> -               -name 'processed-schema*' ! \
> -               -name '*.example.dt.yaml' \) \
> -       ))
> +DT_DOCS = $(shell $(find_cmd) | sed -e 's|^$(srctree)/||')
>
>  DT_SCHEMA_FILES ?= $(DT_DOCS)
>
> --
> 2.20.1
>


-- 
Best Regards
Masahiro Yamada

Re: [PATCH v2] dt-bindings: Fix command line length limit calling dt-mk-schema

[Laurent Pinchart]

Hi Rob,

Thank you for the patch.

On Wed, Apr 22, 2020 at 01:57:08PM -0500, Rob Herring wrote:
> As the number of schemas has increased, we're starting to hit the error
> "execvp: /bin/sh: Argument list too long". This is due to passing all the
> schema files on the command line to dt-mk-schema. It currently is only
> with out of tree builds and is intermittent depending on the file path
> lengths.
> 
> Commit 2ba06cd8565b ("kbuild: Always validate DT binding examples") made
> hitting this proplem more likely since the example validation now always
> gets the full list of schemas.
> 
> Fix this by passing the schema file list in a pipe and using xargs. We end
> up doing the find twice, but the time is insignificant compared to the
> dt-mk-schema time.
> 
> Reported-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Cc: Masahiro Yamada <masahiroy@kernel.org>
> Signed-off-by: Rob Herring <robh@kernel.org>

Tested-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> ---
>  Documentation/devicetree/bindings/Makefile | 18 ++++++++++--------
>  1 file changed, 10 insertions(+), 8 deletions(-)
> 
> diff --git a/Documentation/devicetree/bindings/Makefile b/Documentation/devicetree/bindings/Makefile
> index 87c76bdabfe6..7782d9985082 100644
> --- a/Documentation/devicetree/bindings/Makefile
> +++ b/Documentation/devicetree/bindings/Makefile
> @@ -14,16 +14,18 @@ $(obj)/%.example.dts: $(src)/%.yaml FORCE
>  # Use full schemas when checking %.example.dts
>  DT_TMP_SCHEMA := $(obj)/processed-schema-examples.yaml
>  
> +find_cmd = find $(srctree)/$(src) \( -name '*.yaml' ! \
> +		-name 'processed-schema*' ! \
> +		-name '*.example.dt.yaml' \)
> +
>  quiet_cmd_mk_schema = SCHEMA  $@
> -      cmd_mk_schema = $(DT_MK_SCHEMA) $(DT_MK_SCHEMA_FLAGS) -o $@ $(real-prereqs)
> +      cmd_mk_schema = rm -f $@ ; \
> +                      $(if $(DT_MK_SCHEMA_FLAGS), \
> +                           echo $(real-prereqs), \
> +                           $(find_cmd)) | \
> +                      xargs $(DT_MK_SCHEMA) $(DT_MK_SCHEMA_FLAGS) >> $@
>  
> -DT_DOCS = $(addprefix $(src)/, \
> -	$(shell \
> -	cd $(srctree)/$(src) && \
> -	find * \( -name '*.yaml' ! \
> -		-name 'processed-schema*' ! \
> -		-name '*.example.dt.yaml' \) \
> -	))
> +DT_DOCS = $(shell $(find_cmd) | sed -e 's|^$(srctree)/||')
>  
>  DT_SCHEMA_FILES ?= $(DT_DOCS)
>  

-- 
Regards,

Laurent Pinchart