[PATCH] local.conf.sample.extended: Document how to limit parallel fetch tasks

[PATCH] local.conf.sample.extended: Document how to limit parallel fetch tasks

[Richard Purdie]

[YOCTO #13235]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/conf/local.conf.sample.extended | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/meta/conf/local.conf.sample.extended b/meta/conf/local.conf.sample.extended
index fcf4a9b7c1a..443b4e83f9a 100644
--- a/meta/conf/local.conf.sample.extended
+++ b/meta/conf/local.conf.sample.extended
@@ -23,6 +23,14 @@
 #
 # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would
 # be appropriate for example.
+#
+# Some users are behind firewalls or use servers where the number of parallel connections
+# is limited. In such cases you can limit the number of fetch tasks which run in parallel by
+# setting the option below, in this case limiting to a maximum of 4 fetch tasks in parallel:
+#
+#do_fetch[number_threads] = "4"
+#
+
 
 #DISTRO_FEATURES = "alsa bluetooth ext2 irda ipv4 ipv6 pcmcia usbgadget usbhost wifi nfs zeroconf pci"
 
-- 
2.25.0

Re: [PATCH] local.conf.sample.extended: Document how to limit parallel fetch tasks

[Andre McCurdy]

On Wed, Feb 19, 2020 at 1:43 PM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> [YOCTO #13235]
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>  meta/conf/local.conf.sample.extended | 8 ++++++++
>  1 file changed, 8 insertions(+)
>
> diff --git a/meta/conf/local.conf.sample.extended b/meta/conf/local.conf.sample.extended
> index fcf4a9b7c1a..443b4e83f9a 100644
> --- a/meta/conf/local.conf.sample.extended
> +++ b/meta/conf/local.conf.sample.extended
> @@ -23,6 +23,14 @@
>  #
>  # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would
>  # be appropriate for example.
> +#
> +# Some users are behind firewalls or use servers where the number of parallel connections
> +# is limited. In such cases you can limit the number of fetch tasks which run in parallel by
> +# setting the option below, in this case limiting to a maximum of 4 fetch tasks in parallel:
> +#
> +#do_fetch[number_threads] = "4"

How does this interact with BB_NUMBER_THREADS? If, for example, both
are set to 4 does it mean there could be 4 fetch threads and 4 compile
threads running in parallel?

A related issue when the number of parallel connections is limited is
parsing a large number of AUTOREV recipes. From what I remember the
workaround for that was to set BB_NUMBER_PARSE_THREADS. Maybe that
should be documented in local.conf.sample.extended too? Perhaps even
better would be to define a single new variable which limits the
number of parallel connections in both cases?

> +#
> +
>
>  #DISTRO_FEATURES = "alsa bluetooth ext2 irda ipv4 ipv6 pcmcia usbgadget usbhost wifi nfs zeroconf pci"
>
> --
> 2.25.0
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH] local.conf.sample.extended: Document how to limit parallel fetch tasks

[Richard Purdie]

On Wed, 2020-02-19 at 14:05 -0800, Andre McCurdy wrote:
> On Wed, Feb 19, 2020 at 1:43 PM Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > [YOCTO #13235]
> > 
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > ---
> >  meta/conf/local.conf.sample.extended | 8 ++++++++
> >  1 file changed, 8 insertions(+)
> > 
> > diff --git a/meta/conf/local.conf.sample.extended
> > b/meta/conf/local.conf.sample.extended
> > index fcf4a9b7c1a..443b4e83f9a 100644
> > --- a/meta/conf/local.conf.sample.extended
> > +++ b/meta/conf/local.conf.sample.extended
> > @@ -23,6 +23,14 @@
> >  #
> >  # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE
> > = "-j 4" would
> >  # be appropriate for example.
> > +#
> > +# Some users are behind firewalls or use servers where the number
> > of parallel connections
> > +# is limited. In such cases you can limit the number of fetch
> > tasks which run in parallel by
> > +# setting the option below, in this case limiting to a maximum of
> > 4 fetch tasks in parallel:
> > +#
> > +#do_fetch[number_threads] = "4"
> 
> How does this interact with BB_NUMBER_THREADS? If, for example, both
> are set to 4 does it mean there could be 4 fetch threads and 4
> compile threads running in parallel?

BB_NUMBER_THREADS is an upper limit which includes fetch tasks.

> A related issue when the number of parallel connections is limited is
> parsing a large number of AUTOREV recipes. From what I remember the
> workaround for that was to set BB_NUMBER_PARSE_THREADS. Maybe that
> should be documented in local.conf.sample.extended too? Perhaps even
> better would be to define a single new variable which limits the
> number of parallel connections in both cases?

I don't know how you'd implement a total number of connections limit
from a code perspective :/.

You'd probably need something like a make job server where you obtained
tokens before making connections and I can see this becoming "fun"
since you'd have to markup every code section where a network
connection was made.

We have a conflict between code complexity, maintainability,
performance and functionality and I'm not sure the use case here
warrants the complexity...

Cheers,

Richard

Re: [PATCH] local.conf.sample.extended: Document how to limit parallel fetch tasks

[Andre McCurdy]

On Wed, Feb 19, 2020 at 3:16 PM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Wed, 2020-02-19 at 14:05 -0800, Andre McCurdy wrote:
> > On Wed, Feb 19, 2020 at 1:43 PM Richard Purdie
> > <richard.purdie@linuxfoundation.org> wrote:
> > > [YOCTO #13235]
> > >
> > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > > ---
> > >  meta/conf/local.conf.sample.extended | 8 ++++++++
> > >  1 file changed, 8 insertions(+)
> > >
> > > diff --git a/meta/conf/local.conf.sample.extended
> > > b/meta/conf/local.conf.sample.extended
> > > index fcf4a9b7c1a..443b4e83f9a 100644
> > > --- a/meta/conf/local.conf.sample.extended
> > > +++ b/meta/conf/local.conf.sample.extended
> > > @@ -23,6 +23,14 @@
> > >  #
> > >  # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE
> > > = "-j 4" would
> > >  # be appropriate for example.
> > > +#
> > > +# Some users are behind firewalls or use servers where the number
> > > of parallel connections
> > > +# is limited. In such cases you can limit the number of fetch
> > > tasks which run in parallel by
> > > +# setting the option below, in this case limiting to a maximum of
> > > 4 fetch tasks in parallel:
> > > +#
> > > +#do_fetch[number_threads] = "4"
> >
> > How does this interact with BB_NUMBER_THREADS? If, for example, both
> > are set to 4 does it mean there could be 4 fetch threads and 4
> > compile threads running in parallel?
>
> BB_NUMBER_THREADS is an upper limit which includes fetch tasks.
>
> > A related issue when the number of parallel connections is limited is
> > parsing a large number of AUTOREV recipes. From what I remember the
> > workaround for that was to set BB_NUMBER_PARSE_THREADS. Maybe that
> > should be documented in local.conf.sample.extended too? Perhaps even
> > better would be to define a single new variable which limits the
> > number of parallel connections in both cases?
>
> I don't know how you'd implement a total number of connections limit
> from a code perspective :/.

A single variable such as BB_MAX_CONNECTION_THREADS which, if defined,
would set the limit for both do_fetch[number_threads] and
BB_NUMBER_PARSE_THREADS ?

> You'd probably need something like a make job server where you obtained
> tokens before making connections and I can see this becoming "fun"
> since you'd have to markup every code section where a network
> connection was made.
>
> We have a conflict between code complexity, maintainability,
> performance and functionality and I'm not sure the use case here
> warrants the complexity...

Of course. I just wanted to share it since it is a real world problem
(I've seen a development team stumped for a few days before finding
BB_NUMBER_PARSE_THREADS, which isn't nearly as well documented as
BB_NUMBER_THREADS and PARALLEL_MAKE). A comment for
BB_NUMBER_PARSE_THREADS in local.conf.sample.extended might have sped
up that debugging process.

[PATCH] local.conf.sample.extended: Document how to limit parallel fetch tasks

[Richard Purdie]

[YOCTO #13235]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/conf/local.conf.sample.extended | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/meta/conf/local.conf.sample.extended b/meta/conf/local.conf.sample.extended
index fcf4a9b7c1a..443b4e83f9a 100644
--- a/meta/conf/local.conf.sample.extended
+++ b/meta/conf/local.conf.sample.extended
@@ -23,6 +23,14 @@
 #
 # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would
 # be appropriate for example.
+#
+# Some users are behind firewalls or use servers where the number of parallel connections
+# is limited. In such cases you can limit the number of fetch tasks which run in parallel by
+# setting the option below, in this case limiting to a maximum of 4 fetch tasks in parallel:
+#
+#do_fetch[number_threads] = "4"
+#
+
 
 #DISTRO_FEATURES = "alsa bluetooth ext2 irda ipv4 ipv6 pcmcia usbgadget usbhost wifi nfs zeroconf pci"
 
-- 
2.25.0