[PATCH 0/2] dt-bindings: Improve yamllint performance

[PATCH 0/2] dt-bindings: Improve yamllint performance

[Geert Uytterhoeven]

	Hi Rob,

This series improves yamllint performance by parallelizing the
operation, and by restricting the checked files to those specified using
DT_SCHEMA_FILES.

Changes compared to v1:
  - New patch to parallelize yamllint,
  - Introduce find_all_cmd,
  - Only use the restricted set for yamllint.

I've been using this for the past 6 months, as it helps a lot when
writing or updating DT bindings.  Combined, this reduces the execution
time of

    make dt_binding_check DT_SCHEMA_FILES=Documentation/devicetree/bindings/your/binding/file.yaml

from ca. 30 to 10 s on i7-8700K.

Thanks for your comments!

Geert Uytterhoeven (2):
  dt-bindings: Parallelize yamllint
  dt-bindings: Consider DT_SCHEMA_FILES when finding all json-schema

 Documentation/devicetree/bindings/Makefile | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

-- 
2.25.1

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

[PATCH 1/2] dt-bindings: Parallelize yamllint

[Geert Uytterhoeven]

Use xargs sharding like "chk_bindings" does, to parallelize the
execution of yamllint.

This reduces the yamllint execution time from ca. 21 to 5 seconds on
i7-8700K.

Suggested-by: Rob Herring <robh+dt@kernel.org>
Signed-off-by: Geert Uytterhoeven <geert+renesas@glider.be>
---
v2:
  - New.
---
 Documentation/devicetree/bindings/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Documentation/devicetree/bindings/Makefile b/Documentation/devicetree/bindings/Makefile
index a072e95de626c83b..6305cfa9495ed676 100644
--- a/Documentation/devicetree/bindings/Makefile
+++ b/Documentation/devicetree/bindings/Makefile
@@ -28,7 +28,7 @@ find_cmd = find $(srctree)/$(src) \( -name '*.yaml' ! \
 
 quiet_cmd_yamllint = LINT    $(src)
       cmd_yamllint = ($(find_cmd) | \
-                     xargs $(DT_SCHEMA_LINT) -f parsable -c $(srctree)/$(src)/.yamllint >&2) || true
+                     xargs -n200 -P$$(nproc) $(DT_SCHEMA_LINT) -f parsable -c $(srctree)/$(src)/.yamllint >&2) || true
 
 quiet_cmd_chk_bindings = CHKDT   $@
       cmd_chk_bindings = ($(find_cmd) | \
-- 
2.25.1

[PATCH 2/2] dt-bindings: Consider DT_SCHEMA_FILES when finding all json-schema

[Geert Uytterhoeven]

Setting DT_SCHEMA_FILES allows the user to restrict the
"dt_binding_check" make target to a specified set of DT binding files.
However, yamllint is still run on all available files, which not only
takes time, but also outputs warnings for other binding files the
developer is not interested in.

Fix this by renaming "find_cmd" to "find_all_cmd", introducing a new
"find_cmd" to only return the files specified by DT_SCHEMA_FILES (if
present), and using the latter for yamllint.

Signed-off-by: Geert Uytterhoeven <geert+renesas@glider.be>
---
v2:
  - Introduce find_all_cmd,
  - Only use the restricted set for yamllint.
---
 Documentation/devicetree/bindings/Makefile | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/Documentation/devicetree/bindings/Makefile b/Documentation/devicetree/bindings/Makefile
index 6305cfa9495ed676..1a8c3ab9a538837a 100644
--- a/Documentation/devicetree/bindings/Makefile
+++ b/Documentation/devicetree/bindings/Makefile
@@ -22,23 +22,29 @@ $(obj)/%.example.dts: $(src)/%.yaml check_dtschema_version FORCE
 # Use full schemas when checking %.example.dts
 DT_TMP_SCHEMA := $(obj)/processed-schema-examples.json
 
-find_cmd = find $(srctree)/$(src) \( -name '*.yaml' ! \
+find_all_cmd = find $(srctree)/$(src) \( -name '*.yaml' ! \
 		-name 'processed-schema*' ! \
 		-name '*.example.dt.yaml' \)
 
+ifeq ($(DT_SCHEMA_FILES),)
+find_cmd = $(find_all_cmd)
+else
+find_cmd = echo $(addprefix $(srctree)/, $(DT_SCHEMA_FILES))
+endif
+
 quiet_cmd_yamllint = LINT    $(src)
       cmd_yamllint = ($(find_cmd) | \
                      xargs -n200 -P$$(nproc) $(DT_SCHEMA_LINT) -f parsable -c $(srctree)/$(src)/.yamllint >&2) || true
 
 quiet_cmd_chk_bindings = CHKDT   $@
-      cmd_chk_bindings = ($(find_cmd) | \
+      cmd_chk_bindings = ($(find_all_cmd) | \
                          xargs -n200 -P$$(nproc) $(DT_DOC_CHECKER) -u $(srctree)/$(src)) || true
 
 quiet_cmd_mk_schema = SCHEMA  $@
       cmd_mk_schema = f=$$(mktemp) ; \
                       $(if $(DT_MK_SCHEMA_FLAGS), \
                            printf '%s\n' $(real-prereqs), \
-                           $(find_cmd)) > $$f ; \
+                           $(find_all_cmd)) > $$f ; \
                       $(DT_MK_SCHEMA) -j $(DT_MK_SCHEMA_FLAGS) @$$f > $@ ; \
 		      rm -f $$f
 
@@ -48,7 +54,7 @@ define rule_chkdt
 	$(call cmd,mk_schema)
 endef
 
-DT_DOCS = $(patsubst $(srctree)/%,%,$(shell $(find_cmd)))
+DT_DOCS = $(patsubst $(srctree)/%,%,$(shell $(find_all_cmd)))
 
 override DTC_FLAGS := \
 	-Wno-avoid_unnecessary_addr_size \
-- 
2.25.1

Re: [PATCH 1/2] dt-bindings: Parallelize yamllint

[Rob Herring]

On Mon, 18 Oct 2021 12:54:47 +0200, Geert Uytterhoeven wrote:
> Use xargs sharding like "chk_bindings" does, to parallelize the
> execution of yamllint.
> 
> This reduces the yamllint execution time from ca. 21 to 5 seconds on
> i7-8700K.
> 
> Suggested-by: Rob Herring <robh+dt@kernel.org>
> Signed-off-by: Geert Uytterhoeven <geert+renesas@glider.be>
> ---
> v2:
>   - New.
> ---
>  Documentation/devicetree/bindings/Makefile | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 

Applied, thanks!

Re: [PATCH 2/2] dt-bindings: Consider DT_SCHEMA_FILES when finding all json-schema

[Rob Herring]

On Mon, 18 Oct 2021 12:54:48 +0200, Geert Uytterhoeven wrote:
> Setting DT_SCHEMA_FILES allows the user to restrict the
> "dt_binding_check" make target to a specified set of DT binding files.
> However, yamllint is still run on all available files, which not only
> takes time, but also outputs warnings for other binding files the
> developer is not interested in.
> 
> Fix this by renaming "find_cmd" to "find_all_cmd", introducing a new
> "find_cmd" to only return the files specified by DT_SCHEMA_FILES (if
> present), and using the latter for yamllint.

We can also do the same thing on dt-doc-validate since checking and 
preprocessing schemas are separate steps, so I did while applying. 

> 
> Signed-off-by: Geert Uytterhoeven <geert+renesas@glider.be>
> ---
> v2:
>   - Introduce find_all_cmd,
>   - Only use the restricted set for yamllint.
> ---
>  Documentation/devicetree/bindings/Makefile | 14 ++++++++++----
>  1 file changed, 10 insertions(+), 4 deletions(-)
>