[patch] iowatcher: spwan NPROCESSORS_ONLN rsvg-convert-s

[patch] iowatcher: spwan NPROCESSORS_ONLN rsvg-convert-s

[Jeff Moyer]

iowatcher currently always spawns 8 rsvg-convert processes, no matter
how many CPUs a system has.  I did some limited testing of different
numbers of rsvg-convert processes.  Here are the results:

8 processes:
real	4m2.194s
user	23m36.665s
sys	0m38.523s

20 processes:
real	2m28.935s
user	24m51.817s
sys	0m49.227s

40 processes:
real	2m28.150s
user	24m56.994s
sys	0m49.621s

Note that this is the time it takes for a full run of iowatcher -- I
didn't separate out just the rsvg-convert portion.

Given the above results, it seems like a reasonable thing to spawn one
rsvg-convert process per cpu.

Signed-off-by: Jeff Moyer <jmoyer@redhat.com>

diff --git a/iowatcher/main.c b/iowatcher/main.c
index 54325fb..4b97013 100644
--- a/iowatcher/main.c
+++ b/iowatcher/main.c
@@ -1043,9 +1043,14 @@ static void system_check(const char *cmd)
 
 static void convert_movie_files(char *movie_dir)
 {
+	long nr_cpus = sysconf(_SC_NPROCESSORS_ONLN);
+
+	if (nr_cpus < 0)
+		nr_cpus = 8;
+
 	fprintf(stderr, "Converting svg files in %s\n", movie_dir);
-	snprintf(line, line_len, "find %s -name \\*.svg | xargs -I{} -n 1 -P 8 rsvg-convert -o {}.png {}",
-		 movie_dir);
+	snprintf(line, line_len, "find %s -name \\*.svg | xargs -I{} -n 1 -P %ld rsvg-convert -o {}.png {}",
+		 movie_dir, nr_cpus);
 	system_check(line);
 }
 

Re: [patch] iowatcher: spwan NPROCESSORS_ONLN rsvg-convert-s

[Jeff Moyer]

Jeff Moyer <jmoyer@redhat.com> writes:

> iowatcher currently always spawns 8 rsvg-convert processes, no matter
> how many CPUs a system has.  I did some limited testing of different
> numbers of rsvg-convert processes.  Here are the results:

I forgot to mention that my system has 20 hyperthreads.

Jens, can you also fix the spwan typo in the subject when you commit
this?

Thanks!
Jeff

> 8 processes:
> real	4m2.194s
> user	23m36.665s
> sys	0m38.523s
>
> 20 processes:
> real	2m28.935s
> user	24m51.817s
> sys	0m49.227s
>
> 40 processes:
> real	2m28.150s
> user	24m56.994s
> sys	0m49.621s
>
> Note that this is the time it takes for a full run of iowatcher -- I
> didn't separate out just the rsvg-convert portion.
>
> Given the above results, it seems like a reasonable thing to spawn one
> rsvg-convert process per cpu.
>
> Signed-off-by: Jeff Moyer <jmoyer@redhat.com>
>
> diff --git a/iowatcher/main.c b/iowatcher/main.c
> index 54325fb..4b97013 100644
> --- a/iowatcher/main.c
> +++ b/iowatcher/main.c
> @@ -1043,9 +1043,14 @@ static void system_check(const char *cmd)
>  
>  static void convert_movie_files(char *movie_dir)
>  {
> +	long nr_cpus = sysconf(_SC_NPROCESSORS_ONLN);
> +
> +	if (nr_cpus < 0)
> +		nr_cpus = 8;
> +
>  	fprintf(stderr, "Converting svg files in %s\n", movie_dir);
> -	snprintf(line, line_len, "find %s -name \\*.svg | xargs -I{} -n 1 -P 8 rsvg-convert -o {}.png {}",
> -		 movie_dir);
> +	snprintf(line, line_len, "find %s -name \\*.svg | xargs -I{} -n 1 -P %ld rsvg-convert -o {}.png {}",
> +		 movie_dir, nr_cpus);
>  	system_check(line);
>  }
>  

Re: [patch] iowatcher: spwan NPROCESSORS_ONLN rsvg-convert-s

[Jens Axboe]

On 8/31/18 2:01 PM, Jeff Moyer wrote:
> iowatcher currently always spawns 8 rsvg-convert processes, no matter
> how many CPUs a system has.  I did some limited testing of different
> numbers of rsvg-convert processes.  Here are the results:
> 
> 8 processes:
> real	4m2.194s
> user	23m36.665s
> sys	0m38.523s
> 
> 20 processes:
> real	2m28.935s
> user	24m51.817s
> sys	0m49.227s
> 
> 40 processes:
> real	2m28.150s
> user	24m56.994s
> sys	0m49.621s
> 
> Note that this is the time it takes for a full run of iowatcher -- I
> didn't separate out just the rsvg-convert portion.
> 
> Given the above results, it seems like a reasonable thing to spawn one
> rsvg-convert process per cpu.

Applied, with the typo fixed.

-- 
Jens Axboe

Re: [patch] iowatcher: spwan NPROCESSORS_ONLN rsvg-convert-s

[Chris Mason]

On 31 Aug 2018, at 16:56, Jeff Moyer wrote:

> Jeff Moyer <jmoyer@redhat.com> writes:
>
>> iowatcher currently always spawns 8 rsvg-convert processes, no matter
>> how many CPUs a system has.  I did some limited testing of different
>> numbers of rsvg-convert processes.  Here are the results:
>
> I forgot to mention that my system has 20 hyperthreads.
>
> Jens, can you also fix the spwan typo in the subject when you commit
> this?

Reviewed-by: Chris Mason <clm@fb.com> but only if the typo stays.

-chris

Re: [patch] iowatcher: spwan NPROCESSORS_ONLN rsvg-convert-s

[Jens Axboe]

On 8/31/18 2:59 PM, Chris Mason wrote:
> On 31 Aug 2018, at 16:56, Jeff Moyer wrote:
> 
>> Jeff Moyer <jmoyer@redhat.com> writes:
>>
>>> iowatcher currently always spawns 8 rsvg-convert processes, no matter
>>> how many CPUs a system has.  I did some limited testing of different
>>> numbers of rsvg-convert processes.  Here are the results:
>>
>> I forgot to mention that my system has 20 hyperthreads.
>>
>> Jens, can you also fix the spwan typo in the subject when you commit
>> this?
> 
> Reviewed-by: Chris Mason <clm@fb.com> but only if the typo stays.

Haha damn, well I guess that worked out, since I committed it 5 min
ago, without the reviewed-by, but with typo fixed :-)

-- 
Jens Axboe

Re: [patch] iowatcher: spwan NPROCESSORS_ONLN rsvg-convert-s

[Jeff Moyer]

Jens Axboe <axboe@kernel.dk> writes:

> On 8/31/18 2:59 PM, Chris Mason wrote:
>> On 31 Aug 2018, at 16:56, Jeff Moyer wrote:
>> 
>>> Jeff Moyer <jmoyer@redhat.com> writes:
>>>
>>>> iowatcher currently always spawns 8 rsvg-convert processes, no matter
>>>> how many CPUs a system has.  I did some limited testing of different
>>>> numbers of rsvg-convert processes.  Here are the results:
>>>
>>> I forgot to mention that my system has 20 hyperthreads.
>>>
>>> Jens, can you also fix the spwan typo in the subject when you commit
>>> this?
>> 
>> Reviewed-by: Chris Mason <clm@fb.com> but only if the typo stays.
>
> Haha damn, well I guess that worked out, since I committed it 5 min
> ago, without the reviewed-by, but with typo fixed :-)

You guys crack me up.

Jeff