* [Buildroot] [PATCH] package/python3: add patch to fix highly parallel builds
@ 2021-08-02 21:10 Thomas Petazzoni
2021-08-04 20:24 ` Yann E. MORIN
2021-08-08 9:54 ` Peter Korsgaard
0 siblings, 2 replies; 3+ messages in thread
From: Thomas Petazzoni @ 2021-08-02 21:10 UTC (permalink / raw)
To: Buildroot List, James Hilliard; +Cc: Asaf Kahlon, Thomas Petazzoni
Highly parallel host-python3 builds sometimes fail with:
Exception in thread Thread-1:
Traceback (most recent call last):
File "/tmp/instance-3/output-1/host/lib/python3.9/threading.py", line 973, in _bootstrap_inner
self.run()
File "/tmp/instance-3/output-1/host/lib/python3.9/concurrent/futures/process.py", line 317, in run
result_item, is_broken, cause = self.wait_result_broken_or_wakeup()
File "/tmp/instance-3/output-1/host/lib/python3.9/concurrent/futures/process.py", line 376, in wait_result_broken_or_wakeup
worker_sentinels = [p.sentinel for p in self.processes.values()]
File "/tmp/instance-3/output-1/host/lib/python3.9/concurrent/futures/process.py", line 376, in <listcomp>
worker_sentinels = [p.sentinel for p in self.processes.values()]
RuntimeError: dictionary changed size during iteration
During the compile_all.py step of host-python3. This issue is reported
upstream at https://bugs.python.org/issue43498, and while not yet
fixed upstream, a PR was proposed with a possible fix for it. Seems
the PR seems reasonable, let's give it a chance and see if it improves
the situation.
Hopefully Fixes:
http://autobuild.buildroot.net/results/ae6c4ab292589a4e4442dfb0a1286349a9bf4d29/
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
...teration-error-in-_ExecutorManagerTh.patch | 29 +++++++++++++++++++
1 file changed, 29 insertions(+)
create mode 100644 package/python3/0036-Fix-dictionary-iteration-error-in-_ExecutorManagerTh.patch
diff --git a/package/python3/0036-Fix-dictionary-iteration-error-in-_ExecutorManagerTh.patch b/package/python3/0036-Fix-dictionary-iteration-error-in-_ExecutorManagerTh.patch
new file mode 100644
index 0000000000..fbe44c7753
--- /dev/null
+++ b/package/python3/0036-Fix-dictionary-iteration-error-in-_ExecutorManagerTh.patch
@@ -0,0 +1,29 @@
+From 331966be70c371b268a4fcce9e97280cd869f137 Mon Sep 17 00:00:00 2001
+From: Jakub Kulik <kulikjak@gmail.com>
+Date: Mon, 15 Mar 2021 08:49:28 +0100
+Subject: [PATCH] Fix dictionary iteration error in _ExecutorManagerThread
+
+[Thomas: Taken from upstream pull request
+https://github.com/python/cpython/pull/24868, which is aimed at fixing
+https://bugs.python.org/issue43498]
+Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
+---
+ Lib/concurrent/futures/process.py | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/Lib/concurrent/futures/process.py b/Lib/concurrent/futures/process.py
+index 90bc98bf2ec..e3b36dff572 100644
+--- a/Lib/concurrent/futures/process.py
++++ b/Lib/concurrent/futures/process.py
+@@ -373,7 +373,7 @@ class _ExecutorManagerThread(threading.Thread):
+ assert not self.thread_wakeup._closed
+ wakeup_reader = self.thread_wakeup._reader
+ readers = [result_reader, wakeup_reader]
+- worker_sentinels = [p.sentinel for p in self.processes.values()]
++ worker_sentinels = [p.sentinel for p in self.processes.copy().values()]
+ ready = mp.connection.wait(readers + worker_sentinels)
+
+ cause = None
+--
+2.31.1
+
--
2.31.1
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [Buildroot] [PATCH] package/python3: add patch to fix highly parallel builds
2021-08-02 21:10 [Buildroot] [PATCH] package/python3: add patch to fix highly parallel builds Thomas Petazzoni
@ 2021-08-04 20:24 ` Yann E. MORIN
2021-08-08 9:54 ` Peter Korsgaard
1 sibling, 0 replies; 3+ messages in thread
From: Yann E. MORIN @ 2021-08-04 20:24 UTC (permalink / raw)
To: Thomas Petazzoni; +Cc: James Hilliard, Asaf Kahlon, Buildroot List
Thomas, All,
On 2021-08-02 23:10 +0200, Thomas Petazzoni spake thusly:
> Highly parallel host-python3 builds sometimes fail with:
>
> Exception in thread Thread-1:
> Traceback (most recent call last):
> File "/tmp/instance-3/output-1/host/lib/python3.9/threading.py", line 973, in _bootstrap_inner
> self.run()
> File "/tmp/instance-3/output-1/host/lib/python3.9/concurrent/futures/process.py", line 317, in run
> result_item, is_broken, cause = self.wait_result_broken_or_wakeup()
> File "/tmp/instance-3/output-1/host/lib/python3.9/concurrent/futures/process.py", line 376, in wait_result_broken_or_wakeup
> worker_sentinels = [p.sentinel for p in self.processes.values()]
> File "/tmp/instance-3/output-1/host/lib/python3.9/concurrent/futures/process.py", line 376, in <listcomp>
> worker_sentinels = [p.sentinel for p in self.processes.values()]
> RuntimeError: dictionary changed size during iteration
>
> During the compile_all.py step of host-python3. This issue is reported
> upstream at https://bugs.python.org/issue43498, and while not yet
> fixed upstream, a PR was proposed with a possible fix for it. Seems
> the PR seems reasonable, let's give it a chance and see if it improves
> the situation.
>
> Hopefully Fixes:
>
> http://autobuild.buildroot.net/results/ae6c4ab292589a4e4442dfb0a1286349a9bf4d29/
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Applied to master, thanks.
It seems reasonable enough that we can have a shot at it in the
autobuilders.
Note that it has been reported that this is not necessarily 100%
bullet-proof, so we may still see some failures, but the situation
should be muh, much better now.
Thanks!
Regards,
Yann E. MORIN.
> ---
> ...teration-error-in-_ExecutorManagerTh.patch | 29 +++++++++++++++++++
> 1 file changed, 29 insertions(+)
> create mode 100644 package/python3/0036-Fix-dictionary-iteration-error-in-_ExecutorManagerTh.patch
>
> diff --git a/package/python3/0036-Fix-dictionary-iteration-error-in-_ExecutorManagerTh.patch b/package/python3/0036-Fix-dictionary-iteration-error-in-_ExecutorManagerTh.patch
> new file mode 100644
> index 0000000000..fbe44c7753
> --- /dev/null
> +++ b/package/python3/0036-Fix-dictionary-iteration-error-in-_ExecutorManagerTh.patch
> @@ -0,0 +1,29 @@
> +From 331966be70c371b268a4fcce9e97280cd869f137 Mon Sep 17 00:00:00 2001
> +From: Jakub Kulik <kulikjak@gmail.com>
> +Date: Mon, 15 Mar 2021 08:49:28 +0100
> +Subject: [PATCH] Fix dictionary iteration error in _ExecutorManagerThread
> +
> +[Thomas: Taken from upstream pull request
> +https://github.com/python/cpython/pull/24868, which is aimed at fixing
> +https://bugs.python.org/issue43498]
> +Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> +---
> + Lib/concurrent/futures/process.py | 2 +-
> + 1 file changed, 1 insertion(+), 1 deletion(-)
> +
> +diff --git a/Lib/concurrent/futures/process.py b/Lib/concurrent/futures/process.py
> +index 90bc98bf2ec..e3b36dff572 100644
> +--- a/Lib/concurrent/futures/process.py
> ++++ b/Lib/concurrent/futures/process.py
> +@@ -373,7 +373,7 @@ class _ExecutorManagerThread(threading.Thread):
> + assert not self.thread_wakeup._closed
> + wakeup_reader = self.thread_wakeup._reader
> + readers = [result_reader, wakeup_reader]
> +- worker_sentinels = [p.sentinel for p in self.processes.values()]
> ++ worker_sentinels = [p.sentinel for p in self.processes.copy().values()]
> + ready = mp.connection.wait(readers + worker_sentinels)
> +
> + cause = None
> +--
> +2.31.1
> +
> --
> 2.31.1
>
> _______________________________________________
> buildroot mailing list
> buildroot@busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [Buildroot] [PATCH] package/python3: add patch to fix highly parallel builds
2021-08-02 21:10 [Buildroot] [PATCH] package/python3: add patch to fix highly parallel builds Thomas Petazzoni
2021-08-04 20:24 ` Yann E. MORIN
@ 2021-08-08 9:54 ` Peter Korsgaard
1 sibling, 0 replies; 3+ messages in thread
From: Peter Korsgaard @ 2021-08-08 9:54 UTC (permalink / raw)
To: Thomas Petazzoni; +Cc: James Hilliard, Asaf Kahlon, Buildroot List
>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes:
> Highly parallel host-python3 builds sometimes fail with:
> Exception in thread Thread-1:
> Traceback (most recent call last):
> File "/tmp/instance-3/output-1/host/lib/python3.9/threading.py", line 973, in _bootstrap_inner
> self.run()
> File "/tmp/instance-3/output-1/host/lib/python3.9/concurrent/futures/process.py", line 317, in run
> result_item, is_broken, cause = self.wait_result_broken_or_wakeup()
> File "/tmp/instance-3/output-1/host/lib/python3.9/concurrent/futures/process.py", line 376, in wait_result_broken_or_wakeup
> worker_sentinels = [p.sentinel for p in self.processes.values()]
> File "/tmp/instance-3/output-1/host/lib/python3.9/concurrent/futures/process.py", line 376, in <listcomp>
> worker_sentinels = [p.sentinel for p in self.processes.values()]
> RuntimeError: dictionary changed size during iteration
> During the compile_all.py step of host-python3. This issue is reported
> upstream at https://bugs.python.org/issue43498, and while not yet
> fixed upstream, a PR was proposed with a possible fix for it. Seems
> the PR seems reasonable, let's give it a chance and see if it improves
> the situation.
> Hopefully Fixes:
> http://autobuild.buildroot.net/results/ae6c4ab292589a4e4442dfb0a1286349a9bf4d29/
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Committed to 2021.02.x and 2021.05.x, thanks.
--
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2021-08-08 9:54 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-08-02 21:10 [Buildroot] [PATCH] package/python3: add patch to fix highly parallel builds Thomas Petazzoni
2021-08-04 20:24 ` Yann E. MORIN
2021-08-08 9:54 ` Peter Korsgaard
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.