Re: [PATCH] submodule: implement `module_name` as a builtin helper

[Stefan Beller <sbeller@google.com>]

On Fri, Aug 7, 2015 at 3:42 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Stefan Beller <sbeller@google.com> writes:
>
>> That's why I want to be a bit more generic and have this thread pool API
>> done in C, such that "any for loop" in git can be easily replaced by using
>> the thread pool. I think of "git fetch --all" specially.
>
> One more thing, as I didn't notice that you kept repeating "thread"
> pool API.

Yeah I intended to use both threads and processes for the heavy submodule
operations.

Each thread in the thread pool would setup the right argument lists
and environment
and then spawn a process for heavy weight operations, wait for the process,
maybe process its output.

Maybe I should omit the whole thread pool and only use select from a single
threaded main program.

>
> While I doubt that you would gain much by using threads in place of
> processes to perform parallel "submodule update", "submodule clone",
> "fetch all", etc., all of which are fairly well isolated and heavy
> weight operations themselves, and I suspect that the implementation
> simplicity of using separate processes would probably be huge plus
> compared to any possible upside you may gain from using threads, if
> you really want to go the "thread" route, the first thing to try
> would be to see if a few places we already use threads for
> parallelism (namely, "grep", "pack-objects", "preload-index" and
> "index-pack") can be factored out and model your new API around the
> commonality among them.