[Buildroot] [Bug 14996] New: Too many directories in BR2_EXTERNAL causes hang during make

[Buildroot] [Bug 14996] New: Too many directories in BR2_EXTERNAL causes hang during make

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14996

            Bug ID: 14996
           Summary: Too many directories in BR2_EXTERNAL causes hang
                    during make
           Product: buildroot
           Version: unspecified
          Hardware: All
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P5
         Component: Other
          Assignee: unassigned@buildroot.uclibc.org
          Reporter: david.lawson1@tx.rr.com
                CC: buildroot@uclibc.org
  Target Milestone: ---

The top level Makefile in buildroot has a recursive rule which causes the
appearance of a hang as the number of directories in BR2_EXTERNAL increases.
When the number of directories in BR2_EXTERNAL is small, the recursion occurs,
but make detects the recursion and determines the target does not have to be
remade. This allows make to progress.

This is the failing rule:

define percent_defconfig
# Override the BR2_DEFCONFIG from COMMON_CONFIG_ENV with the new defconfig
%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(1)/configs/%_defconfig
outputmakefile
        @$$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(1)/configs/$$@ \
                $$< --defconfig=$(1)/configs/$$@ $$(CONFIG_CONFIG_IN)
endef
$(eval $(foreach d,$(call reverse,$(TOPDIR) $(BR2_EXTERNAL_DIRS)),$(call
percent_defconfig,$(d))$(sep)))

The rule for %defconfig is created for each directory in BR2_EXTERNAL. When the
rule is matched, the stem is 'defconfig_name'. The second prerequisite is
expanded to $(1)/configs/defconfig_name_defconfig. The rule, and all of the
other rules defined by this macro, are invoked again, but the stem is now
$(1)/configs/defconfig_name_defconfig. The second prerequisite is now expanded
to $(1)/configs/($1)/configs/defconfig_name_defconfig. This expansion continues
until make detects the infinite recursion.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14996] Too many directories in BR2_EXTERNAL causes hang during make

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14996

Yann E. MORIN <yann.morin.1998@free.fr> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Assignee|unassigned@buildroot.uclibc |yann.morin.1998@free.fr
                   |.org                        |
                 CC|                            |yann.morin.1998@free.fr
             Status|NEW                         |ASSIGNED

--- Comment #1 from Yann E. MORIN <yann.morin.1998@free.fr> ---
David, All,

Thanks for the report, and the detailed explanations (which I did not yet
entirely
groked, and will need a bit more time before I see the issue).

How many br2-external trees are you using? I've done a bit of testing
with 0 to 1000 br2-external trees, and here are the results:

$ time make BR2_EXTERNAL='' list-defconfigs
real    0m0.129s

$ time make BR2_EXTERNAL='(1 item)' list-defconfigs
real    0m0.138s

$ time make BR2_EXTERNAL='(10 item)' list-defconfigs
real    0m0.262s

$ time make BR2_EXTERNAL='(100 item)' list-defconfigs
real    0m1.912s

$ time make BR2_EXTERNAL='(1000 item)' list-defconfigs
real    0m19.344s

So, what we can see, is that it is roughly linear to the number of br2-external
trees. Even the 1000-tree case is far from appearing stuck (granted, it is a
bit
long, but after seeing 2s for 100 items, it did not feel excruciating to wait).

Also, I do not have any message from make 4.2.1 that reports any infinite
recursion.

Can you provide a bit more details on your configuration?

Regards,
Yann E. MORIN.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14996] Too many directories in BR2_EXTERNAL causes hang during make

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14996

--- Comment #2 from David Lawson <david.lawson1@tx.rr.com> ---
I am sorry, I am working on a github project to show the problem. Do you have a
defconfig file in any of the external projects? It should only take one. I am
concerned also about the name of the defconfig file.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14996] Too many directories in BR2_EXTERNAL causes hang during make

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14996

--- Comment #3 from Yann E. MORIN <yann.morin.1998@free.fr> ---
David, All,

Here's how I prepared the test br2-external trees:

$ for i in $(seq 1 1000); do
    mkdir -p br2-external-${i}/configs
    touch br2-external-${i}/{Config.in,external.mk}
    echo "name: BR_TEST_${i}" >br2-external-${i}/external.desc
    touch br2-external-${i}/configs/foo{,_${i}}_defconfig
done

So, I get minimal br2-external trees, each with two defconfigs:
one that is present in all br2external trees, and one that is
specific to each tree.

And here is exactly how I tested that:

$ make --version
GNU Make 4.2.1

$ for ext in 1 10 100 1000; do time make BR2_EXTERNAL="$(for i in $(seq 1
${ext}); do echo $(pwd)/br2-external-${i}; done)" list-defconfigs >/dev/null;
done 2>&1 |grep real
real    0m0.179s
real    0m0.245s
real    0m1.299s
real    0m18.277s

What version of make are you using? How many br2-external trees do you have?

However, please do note that indeed, it was never envisioned that more than a
few
br2-external trees be used simultaneously. But that it feels like it got stuck
is
still not a nice behaviour...

Regards,
Yann E. MORIN.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14996] Too many directories in BR2_EXTERNAL causes hang during make

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14996

--- Comment #4 from Yann E. MORIN <yann.morin.1998@free.fr> ---
David, All,

To be noted: on my machine, in the directory I was using, I was limited
to using just 2446 br2-external trees, otherwise, make would whine with:

- with 2447:
  make: execvp: /bin/sh: Argument list too long

- with 2248 and above:
  -bash: /usr/bin/make: Argument list too long

Yes, note how 2446 succeeds, 2447 has a special error, and 2448 and above
have another error... Meh...

But still, 2446 br2-external trees is clearly way more than was ever
expected when the feature was added... back in 2016!

Regards,
Yann E. MORIN.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14996] Too many directories in BR2_EXTERNAL causes hang during make

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14996

--- Comment #5 from David Lawson <david.lawson1@tx.rr.com> ---
Here is a simple project that shows the problem:

https://github.com/dclawso/buldroot-recursive

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14996] Too many directories in BR2_EXTERNAL causes hang during make

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14996

--- Comment #6 from David Lawson <david.lawson1@tx.rr.com> ---
This shows the recursion occurs for only one external package, but it is
largely benign as make detects the recursion and determines the target does not
have to be remade.

However, with as few as 6 external packages, the number of dependency checks
that are performed causes make to hang for several hours.

We currently are at 8 external packages and I left the compile running over the
weekend with no progress.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14996] Too many directories in BR2_EXTERNAL causes hang during make

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14996

David Lawson <david.lawson1@tx.rr.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
               Host|                            |ubuntu 22.04

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14996] Too many directories in BR2_EXTERNAL causes hang during make

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14996

--- Comment #7 from Yann E. MORIN <yann.morin.1998@free.fr> ---
David, All,

OK, I could reproduce the issue with a much less convoluted example than your
script.

The issue does only lanifests itself when actually trying to configure
Buildroot.
Listing the defconfig like I was doing was not enough to trigger the issue.

What's weird, is that using 5 external trees does not exhibit the issue; it is
very fast. But using 6 trees triggers the issue, and make is insanely long (I
did
not even wait for it to finish, in fact).

Note however that your comments are a bit confusing. You are talking about
external
packages, but that is incorrect; it is about _br2-external trees_. Also, your
script
is very confusing, because it does very convoluted things (instead of providing
a git
tree that was already fully prepared). But since we don't need it to reproduce
the
issue (and thus to test a fix), I'm not going to comment further. ;-)

I'll lok into devising a fix...

Regards,
Yann E. MORIN.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14996] Too many directories in BR2_EXTERNAL causes hang during make

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14996

--- Comment #8 from Yann E. MORIN <yann.morin.1998@free.fr> ---
David, All,

Could you please have a go with, and test that patch, please:
https://patchwork.ozlabs.org/project/buildroot/patch/20220917073445.2536513-1-yann.morin.1998@free.fr/

Regards,
Yann E. MORIN.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14996] Too many directories in BR2_EXTERNAL causes hang during make

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14996

--- Comment #9 from David Lawson <david.lawson1@tx.rr.com> ---
The comments for $1 and $2 appear to be reversed. I am still attempting to
incorporate this patch into the build tree I am using.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14996] Too many directories in BR2_EXTERNAL causes hang during make

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14996

--- Comment #10 from David Lawson <david.lawson1@tx.rr.com> ---
This change is working in my environment.

Thank you!

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14996] Too many directories in BR2_EXTERNAL causes hang during make

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14996

--- Comment #11 from Yann E. MORIN <yann.morin.1998@free.fr> ---
David, All,

(In reply to David Lawson from comment #10)
> This change is working in my environment.

Thanks for the feedback. I've just resent a v3 with trivial changes.
If you can validate it still works, could you please reply to the mail
with your review tag, ple

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14996] Too many directories in BR2_EXTERNAL causes hang during make

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14996

--- Comment #12 from Yann E. MORIN <yann.morin.1998@free.fr> ---
David, All,

(In reply to David Lawson from comment #10)
> This change is working in my environment.

Thanks for the feedback. I've just resent a v3 with trivial changes.
If you can validate it still works, could you please reply to the mail
with your tested tag, please:

    Tested-by: Your NAME <your@email>

Regards,
Yann E. MORIN.

PS. Sorry for the previous spurious message, PEBCAK.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14996] Too many directories in BR2_EXTERNAL causes hang during make

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14996

Yann E. MORIN <yann.morin.1998@free.fr> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|---                         |FIXED

--- Comment #13 from Yann E. MORIN <yann.morin.1998@free.fr> ---
David, All,

Thank you for the report.

We believe this is now fixed with commit e6195c53041f (Makefile: fix use
of many br2-external trees).

If you still encounter the issue, please re-open.

Regards,
Yann E. MORIN.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot