[4.9-rc1] Build-time 2x slower

[4.9-rc1] Build-time 2x slower

[Sedat Dilek]

[-- Attachment #1: Type: text/plain, Size: 925 bytes --]

Hi Linus,

not sure whom to address on this issue.

I have built Linux v4.9-rc1, v4.8.2 and v4.4.25 kernels (in this
order) this morning.

Building a Linux v4.8.2 under Linux v4.9-rc1 took two times longer.

As usually I build with 2 parallel-make-jobs.
This takes approx. 30mins.
Under Linux v4.9-rc1 it took approx. an hour.

My system is a Ubuntu/precise AMD64 (WUBI installation).
I use my normal build-environment.

If you need further informations, please let me know.

Regards,
- Sedat -

P.S.: Listing content of attached tarball file.

$ LC_ALL=C ls -R
.:
build-time  configs  scripts  toolchain-amd64

./build-time:
ls-alt_building-4-4-25-under-4-8-2.txt
ls-alt_building-4-8-2-under-4-9-0-rc1.txt

./configs:
WHATS-IN-DILEKS-LINUX-KERNEL.txt  config-4.4.25-1-iniza-small
config-4.8.2-1-iniza-small  config-4.9.0-rc1-1-iniza-small

./scripts:
build_linux-upstream.sh

./toolchain-amd64:
HOST-AND-BUILD-TOOLCHAIN.txt

[-- Attachment #2: for-linus.tar.gz --]
[-- Type: application/x-gzip, Size: 99514 bytes --]

[-- Attachment #3: for-linus.tar.gz.md5sum --]
[-- Type: application/octet-stream, Size: 51 bytes --]

4202e585e9c692c406f7bc0ad85733b3  for-linus.tar.gz

Re: [4.9-rc1] Build-time 2x slower

[Linus Torvalds]

On Mon, Oct 17, 2016 at 9:04 AM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
>
> not sure whom to address on this issue.
>
> I have built Linux v4.9-rc1, v4.8.2 and v4.4.25 kernels (in this
> order) this morning.
>
> Building a Linux v4.8.2 under Linux v4.9-rc1 took two times longer.
>
> As usually I build with 2 parallel-make-jobs.
> This takes approx. 30mins.
> Under Linux v4.9-rc1 it took approx. an hour.

Hmm. Would you mind just bisecting it? I've not noticed the same thing
on my setup, but to be honest I generally don't time things very
closely because the variation for me tends to be much more along the
lines of "damn, that pull request touched <linux./fs.h> now it's
rebuilding everything" vs "40 seconds to build just the driver that
changed". Most of my builds are the "allmodconfig" builds I do in
between pulls..

You can even automate it, since it's going to take some time, using
"git bisect run" and writing a small script that looks at how long it
takes to build the kernel from scratch each time. I'd suggest trying
to write the script using a smaller repository (maybe git itself). The
script would just needs to do a clean rebuild, time it, and have
return success or error based on how long it took.

The script *might* look something like

  #!/bin/sh
  git clean -dqfx
  make oldconfig
  time -o time-file -f '%e' sh -c "make -j8 > ../output"
  # remove fractional seconds
  time=$(cat time-file | sed 's/\..*//')
  # less than 35 minutes is good?
  [ $time -lt $((35*60)) ]

but I didn't really test that very much. Anyway, you *should* be able
to do something like

  git bisect good v4.8
  git bisect bad v4.9-rc1
  git bisect run ../timing-script

(put the "timing-script" somewhere _else_ than the kernel sources so
that the "git clean" doesn't remove it - that's what my first silly
test did ;)

You may have to tweak that script a bit, but I think you get the idea..

               Linus

Re: [4.9-rc1] Build-time 2x slower

[Ming Lei]

On Tue, Oct 18, 2016 at 12:04 AM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
> Hi Linus,
>
> not sure whom to address on this issue.
>
> I have built Linux v4.9-rc1, v4.8.2 and v4.4.25 kernels (in this
> order) this morning.
>
> Building a Linux v4.8.2 under Linux v4.9-rc1 took two times longer.
>
> As usually I build with 2 parallel-make-jobs.
> This takes approx. 30mins.
> Under Linux v4.9-rc1 it took approx. an hour.
>
> My system is a Ubuntu/precise AMD64 (WUBI installation).
> I use my normal build-environment.
>
> If you need further informations, please let me know.

Kernel building is more like a CPU-bound workload, so maybe
some clues can be got by comparing results of 'perf top/record',
which should be very easy to collect.

>
> Regards,
> - Sedat -
>
> P.S.: Listing content of attached tarball file.
>
> $ LC_ALL=C ls -R
> .:
> build-time  configs  scripts  toolchain-amd64
>
> ./build-time:
> ls-alt_building-4-4-25-under-4-8-2.txt
> ls-alt_building-4-8-2-under-4-9-0-rc1.txt
>
> ./configs:
> WHATS-IN-DILEKS-LINUX-KERNEL.txt  config-4.4.25-1-iniza-small
> config-4.8.2-1-iniza-small  config-4.9.0-rc1-1-iniza-small
>
> ./scripts:
> build_linux-upstream.sh
>
> ./toolchain-amd64:
> HOST-AND-BUILD-TOOLCHAIN.txt



Thanks,
Ming Lei

Re: [4.9-rc1] Build-time 2x slower

[Sedat Dilek]

On Mon, Oct 17, 2016 at 6:46 PM, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
> On Mon, Oct 17, 2016 at 9:04 AM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>
>> not sure whom to address on this issue.
>>
>> I have built Linux v4.9-rc1, v4.8.2 and v4.4.25 kernels (in this
>> order) this morning.
>>
>> Building a Linux v4.8.2 under Linux v4.9-rc1 took two times longer.
>>
>> As usually I build with 2 parallel-make-jobs.
>> This takes approx. 30mins.
>> Under Linux v4.9-rc1 it took approx. an hour.
>
> Hmm. Would you mind just bisecting it? I've not noticed the same thing
> on my setup, but to be honest I generally don't time things very
> closely because the variation for me tends to be much more along the
> lines of "damn, that pull request touched <linux./fs.h> now it's
> rebuilding everything" vs "40 seconds to build just the driver that
> changed". Most of my builds are the "allmodconfig" builds I do in
> between pulls..
>
> You can even automate it, since it's going to take some time, using
> "git bisect run" and writing a small script that looks at how long it
> takes to build the kernel from scratch each time. I'd suggest trying
> to write the script using a smaller repository (maybe git itself). The
> script would just needs to do a clean rebuild, time it, and have
> return success or error based on how long it took.
>
> The script *might* look something like
>
>   #!/bin/sh
>   git clean -dqfx
>   make oldconfig
>   time -o time-file -f '%e' sh -c "make -j8 > ../output"
>   # remove fractional seconds
>   time=$(cat time-file | sed 's/\..*//')
>   # less than 35 minutes is good?
>   [ $time -lt $((35*60)) ]
>
> but I didn't really test that very much. Anyway, you *should* be able
> to do something like
>
>   git bisect good v4.8
>   git bisect bad v4.9-rc1
>   git bisect run ../timing-script
>
> (put the "timing-script" somewhere _else_ than the kernel sources so
> that the "git clean" doesn't remove it - that's what my first silly
> test did ;)
>
> You may have to tweak that script a bit, but I think you get the idea..
>

OK, thanks for confirming the problem.
I was not sure if this is a "local" problem.

Your timing-script looks interesting and test... will report.

- Sedat -

Re: [4.9-rc1] Build-time 2x slower

[Sedat Dilek]

On Tue, Oct 18, 2016 at 3:38 AM, Ming Lei <tom.leiming@gmail.com> wrote:
> On Tue, Oct 18, 2016 at 12:04 AM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
>> Hi Linus,
>>
>> not sure whom to address on this issue.
>>
>> I have built Linux v4.9-rc1, v4.8.2 and v4.4.25 kernels (in this
>> order) this morning.
>>
>> Building a Linux v4.8.2 under Linux v4.9-rc1 took two times longer.
>>
>> As usually I build with 2 parallel-make-jobs.
>> This takes approx. 30mins.
>> Under Linux v4.9-rc1 it took approx. an hour.
>>
>> My system is a Ubuntu/precise AMD64 (WUBI installation).
>> I use my normal build-environment.
>>
>> If you need further informations, please let me know.
>
> Kernel building is more like a CPU-bound workload, so maybe
> some clues can be got by comparing results of 'perf top/record',
> which should be very easy to collect.
>

I have no big experiences with perf.
Last I played was testing early days of "lockdep" feature.
Can you give some clear instructions on how to use perf top/record in
this scenario?

- Sedat -

>>
>> Regards,
>> - Sedat -
>>
>> P.S.: Listing content of attached tarball file.
>>
>> $ LC_ALL=C ls -R
>> .:
>> build-time  configs  scripts  toolchain-amd64
>>
>> ./build-time:
>> ls-alt_building-4-4-25-under-4-8-2.txt
>> ls-alt_building-4-8-2-under-4-9-0-rc1.txt
>>
>> ./configs:
>> WHATS-IN-DILEKS-LINUX-KERNEL.txt  config-4.4.25-1-iniza-small
>> config-4.8.2-1-iniza-small  config-4.9.0-rc1-1-iniza-small
>>
>> ./scripts:
>> build_linux-upstream.sh
>>
>> ./toolchain-amd64:
>> HOST-AND-BUILD-TOOLCHAIN.txt
>
>
>
> Thanks,
> Ming Lei

Re: [4.9-rc1] Build-time 2x slower

[Jörg Otte]

2016-10-18 11:30 GMT+02:00 Sedat Dilek <sedat.dilek@gmail.com>:
> On Tue, Oct 18, 2016 at 3:38 AM, Ming Lei <tom.leiming@gmail.com> wrote:
>> On Tue, Oct 18, 2016 at 12:04 AM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>> Hi Linus,
>>>
>>> not sure whom to address on this issue.
>>>
>>> I have built Linux v4.9-rc1, v4.8.2 and v4.4.25 kernels (in this
>>> order) this morning.
>>>
>>> Building a Linux v4.8.2 under Linux v4.9-rc1 took two times longer.
>>>
>>> As usually I build with 2 parallel-make-jobs.
>>> This takes approx. 30mins.
>>> Under Linux v4.9-rc1 it took approx. an hour.
>>>
>>> My system is a Ubuntu/precise AMD64 (WUBI installation).
>>> I use my normal build-environment.
>>>
>>> If you need further informations, please let me know.
>>
>> Kernel building is more like a CPU-bound workload, so maybe
>> some clues can be got by comparing results of 'perf top/record',
>> which should be very easy to collect.
>>
>
> I have no big experiences with perf.
> Last I played was testing early days of "lockdep" feature.
> Can you give some clear instructions on how to use perf top/record in
> this scenario?
>
> - Sedat -
>
>>>
>>> Regards,
>>> - Sedat -
>>>
>>> P.S.: Listing content of attached tarball file.
>>>
>>> $ LC_ALL=C ls -R
>>> .:
>>> build-time  configs  scripts  toolchain-amd64
>>>
>>> ./build-time:
>>> ls-alt_building-4-4-25-under-4-8-2.txt
>>> ls-alt_building-4-8-2-under-4-9-0-rc1.txt
>>>
>>> ./configs:
>>> WHATS-IN-DILEKS-LINUX-KERNEL.txt  config-4.4.25-1-iniza-small
>>> config-4.8.2-1-iniza-small  config-4.9.0-rc1-1-iniza-small
>>>
>>> ./scripts:
>>> build_linux-upstream.sh
>>>
>>> ./toolchain-amd64:
>>> HOST-AND-BUILD-TOOLCHAIN.txt
>>
>>
>>
>> Thanks,
>> Ming Lei

I can confirm the problem. I use 3 build jobs in parallel
and the kernel build takes 2,5 times longer.

I'm only seeing 1 (of 4) cores are running with max frequency.
The other are running in minimum frequency. And this seems not
to be limited to build jobs however.

The last known good kernel for me is  ..-4.8.0-14604-g29fbff8

Jörg

Re: [4.9-rc1] Build-time 2x slower

[Linus Torvalds]

On Tue, Oct 18, 2016 at 9:49 AM, Jörg Otte <jrg.otte@gmail.com> wrote:
> On Tue, Oct 18, 2016 at 12:04 AM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>
>> not sure whom to address on this issue.
>>
>> I have built Linux v4.9-rc1, v4.8.2 and v4.4.25 kernels (in this
>> order) this morning.
>>
>> Building a Linux v4.8.2 under Linux v4.9-rc1 took two times longer.
>>
>> As usually I build with 2 parallel-make-jobs.
>> This takes approx. 30mins.
>> Under Linux v4.9-rc1 it took approx. an hour.
>>
>> My system is a Ubuntu/precise AMD64 (WUBI installation).
>> I use my normal build-environment.
>
> I can confirm the problem. I use 3 build jobs in parallel
> and the kernel build takes 2,5 times longer.
>
> I'm only seeing 1 (of 4) cores are running with max frequency.
> The other are running in minimum frequency. And this seems not
> to be limited to build jobs however.
>
> The last known good kernel for me is  ..-4.8.0-14604-g29fbff8

Well, there are a few merges in 4.9-rc1 since that
4.8.0-14604-g29fbff8 version, but the obvious ones are my pulls from:

  Michal Marek (2):
     kbuild updates
     misc kbuild changes

(My merge commit ID's are 50cff89837a4 and 84d69848c97f) with
everything else looking like "normal code updates".

Michal: a 2.5x slowdown of the kernel build was presumably *not* intentional.

I'm not seeing anything obvious, but if it's spending a lot more time
in fixdep, then it's that "strstr()" change. That commit seems to
assume that strstr() is fast, which is a debatable assumption and
might be wrong in some environments.

But even with a "strstr()" written by a sloth that was dropped on its
head a few too many times when young, I can't see it being *that* much
slower.

Can you do just a silly

   perf record make -j8

of the bad build, and see if something stands out when you do "perf report"?

But maybe Michal has some ideas.

              Linus

Re: [4.9-rc1] Build-time 2x slower

[Jörg Otte]

2016-10-18 19:19 GMT+02:00 Linus Torvalds <torvalds@linux-foundation.org>:
> On Tue, Oct 18, 2016 at 9:49 AM, Jörg Otte <jrg.otte@gmail.com> wrote:
>> On Tue, Oct 18, 2016 at 12:04 AM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>>
>>> not sure whom to address on this issue.
>>>
>>> I have built Linux v4.9-rc1, v4.8.2 and v4.4.25 kernels (in this
>>> order) this morning.
>>>
>>> Building a Linux v4.8.2 under Linux v4.9-rc1 took two times longer.
>>>
>>> As usually I build with 2 parallel-make-jobs.
>>> This takes approx. 30mins.
>>> Under Linux v4.9-rc1 it took approx. an hour.
>>>
>>> My system is a Ubuntu/precise AMD64 (WUBI installation).
>>> I use my normal build-environment.
>>
>> I can confirm the problem. I use 3 build jobs in parallel
>> and the kernel build takes 2,5 times longer.
>>
>> I'm only seeing 1 (of 4) cores are running with max frequency.
>> The other are running in minimum frequency. And this seems not
>> to be limited to build jobs however.
>>
>> The last known good kernel for me is  ..-4.8.0-14604-g29fbff8
>
> Well, there are a few merges in 4.9-rc1 since that
> 4.8.0-14604-g29fbff8 version, but the obvious ones are my pulls from:
>
>   Michal Marek (2):
>      kbuild updates
>      misc kbuild changes
>
> (My merge commit ID's are 50cff89837a4 and 84d69848c97f) with
> everything else looking like "normal code updates".
>
> Michal: a 2.5x slowdown of the kernel build was presumably *not* intentional.
>
> I'm not seeing anything obvious, but if it's spending a lot more time
> in fixdep, then it's that "strstr()" change. That commit seems to
> assume that strstr() is fast, which is a debatable assumption and
> might be wrong in some environments.
>
> But even with a "strstr()" written by a sloth that was dropped on its
> head a few too many times when young, I can't see it being *that* much
> slower.
>
> Can you do just a silly
>
>    perf record make -j8
>
> of the bad build, and see if something stands out when you do "perf report"?
>
> But maybe Michal has some ideas.
>
>               Linus

Additional info: I usally use schedutil governor.
If I switch to performance governor problems go away.
Maybe a cpufreq problem?

Jörg

Re: [4.9-rc1] Build-time 2x slower

[Linus Torvalds]

On Wed, Oct 19, 2016 at 4:07 AM, Jörg Otte <jrg.otte@gmail.com> wrote:
>
> Additional info: I usally use schedutil governor.
> If I switch to performance governor problems go away.
> Maybe a cpufreq problem?

Oh, I completely misread the original bug report, and then didn't read
your confirmation email right.

I thought you had a slower build of the different kernels (when
building on the same kernel), and that the _build_ itself had slowed
down for some reason. But you're actually saying that doing the _same_
build actually takes longer when running on 4.9-rc1.

My bad.

There are a few small cpufreq changes there in between commit
29fbff8698fc (that you reported was fine - please tell me I got _that_
right, at least?) and 4.9-rc1. Adding Rafael to the cc.

That said, none of them look all that likely to me. It *would* be good
if you could bisect it a bit (perhaps not fully, but a couple of
bisection steps to narrow down what area it is).

                 Linus

Re: [4.9-rc1] Build-time 2x slower

[Jörg Otte]

2016-10-19 17:29 GMT+02:00 Linus Torvalds <torvalds@linux-foundation.org>:
> On Wed, Oct 19, 2016 at 4:07 AM, Jörg Otte <jrg.otte@gmail.com> wrote:
>>
>> Additional info: I usally use schedutil governor.
>> If I switch to performance governor problems go away.
>> Maybe a cpufreq problem?
>
> Oh, I completely misread the original bug report, and then didn't read
> your confirmation email right.
>
> I thought you had a slower build of the different kernels (when
> building on the same kernel), and that the _build_ itself had slowed
> down for some reason. But you're actually saying that doing the _same_
> build actually takes longer when running on 4.9-rc1.

Exactly!

Btw: ondemand governor is also good.

> There are a few small cpufreq changes there in between commit
> 29fbff8698fc (that you reported was fine - please tell me I got _that_
> right, at least?) and 4.9-rc1.

Perfect! That's what I mean.

> Adding Rafael to the cc.
>
> That said, none of them look all that likely to me. It *would* be good
> if you could bisect it a bit (perhaps not fully, but a couple of
> bisection steps to narrow down what area it is).

I try that tomorrow.

Jörg

Re: [4.9-rc1] Build-time 2x slower

[Rafael J. Wysocki]

On Wednesday, October 19, 2016 06:59:35 PM Jörg Otte wrote:
> 2016-10-19 17:29 GMT+02:00 Linus Torvalds <torvalds@linux-foundation.org>:
> > On Wed, Oct 19, 2016 at 4:07 AM, Jörg Otte <jrg.otte@gmail.com> wrote:
> >>
> >> Additional info: I usally use schedutil governor.
> >> If I switch to performance governor problems go away.
> >> Maybe a cpufreq problem?
> >
> > Oh, I completely misread the original bug report, and then didn't read
> > your confirmation email right.
> >
> > I thought you had a slower build of the different kernels (when
> > building on the same kernel), and that the _build_ itself had slowed
> > down for some reason. But you're actually saying that doing the _same_
> > build actually takes longer when running on 4.9-rc1.
> 
> Exactly!
> 
> Btw: ondemand governor is also good.
> 
> > There are a few small cpufreq changes there in between commit
> > 29fbff8698fc (that you reported was fine - please tell me I got _that_
> > right, at least?) and 4.9-rc1.
> 
> Perfect! That's what I mean.
> 
> > Adding Rafael to the cc.
> >
> > That said, none of them look all that likely to me. It *would* be good
> > if you could bisect it a bit (perhaps not fully, but a couple of
> > bisection steps to narrow down what area it is).
> 
> I try that tomorrow.

Well, please try commit ef98988ba369 (Merge tag 'pm-extra-4.9-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm) which is the
merge introducing the late cpufreq changes.  If the issue is there, please
try to revert commit 899bb6642f2a (cpufreq: skip invalid entries when searching
the frequency) which is the only cpufreq one that may matter for the schedutil
governor (and I have one fix for that commit queued up already).

Thanks,
Rafael

Re: [4.9-rc1] Build-time 2x slower

[Sedat Dilek]

On Wed, Oct 19, 2016 at 6:59 PM, Jörg Otte <jrg.otte@gmail.com> wrote:
> 2016-10-19 17:29 GMT+02:00 Linus Torvalds <torvalds@linux-foundation.org>:
>> On Wed, Oct 19, 2016 at 4:07 AM, Jörg Otte <jrg.otte@gmail.com> wrote:
>>>
>>> Additional info: I usally use schedutil governor.
>>> If I switch to performance governor problems go away.
>>> Maybe a cpufreq problem?
>>
>> Oh, I completely misread the original bug report, and then didn't read
>> your confirmation email right.
>>
>> I thought you had a slower build of the different kernels (when
>> building on the same kernel), and that the _build_ itself had slowed
>> down for some reason. But you're actually saying that doing the _same_
>> build actually takes longer when running on 4.9-rc1.
>
> Exactly!
>
> Btw: ondemand governor is also good.
>
>> There are a few small cpufreq changes there in between commit
>> 29fbff8698fc (that you reported was fine - please tell me I got _that_
>> right, at least?) and 4.9-rc1.
>
> Perfect! That's what I mean.
>
>> Adding Rafael to the cc.
>>
>> That said, none of them look all that likely to me. It *would* be good
>> if you could bisect it a bit (perhaps not fully, but a couple of
>> bisection steps to narrow down what area it is).
>
> I try that tomorrow.
>

I switched over to ***schedutil*** as default-cpufreq-governor with
v4.8 in my init-scripts.
Didn't try any other cpufreq-governor and had no time to play more
with this issue.

- Sedat -

Re: [4.9-rc1] Build-time 2x slower

[Sedat Dilek]

On Wed, Oct 19, 2016 at 10:55 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> On Wednesday, October 19, 2016 06:59:35 PM Jörg Otte wrote:
>> 2016-10-19 17:29 GMT+02:00 Linus Torvalds <torvalds@linux-foundation.org>:
>> > On Wed, Oct 19, 2016 at 4:07 AM, Jörg Otte <jrg.otte@gmail.com> wrote:
>> >>
>> >> Additional info: I usally use schedutil governor.
>> >> If I switch to performance governor problems go away.
>> >> Maybe a cpufreq problem?
>> >
>> > Oh, I completely misread the original bug report, and then didn't read
>> > your confirmation email right.
>> >
>> > I thought you had a slower build of the different kernels (when
>> > building on the same kernel), and that the _build_ itself had slowed
>> > down for some reason. But you're actually saying that doing the _same_
>> > build actually takes longer when running on 4.9-rc1.
>>
>> Exactly!
>>
>> Btw: ondemand governor is also good.
>>
>> > There are a few small cpufreq changes there in between commit
>> > 29fbff8698fc (that you reported was fine - please tell me I got _that_
>> > right, at least?) and 4.9-rc1.
>>
>> Perfect! That's what I mean.
>>
>> > Adding Rafael to the cc.
>> >
>> > That said, none of them look all that likely to me. It *would* be good
>> > if you could bisect it a bit (perhaps not fully, but a couple of
>> > bisection steps to narrow down what area it is).
>>
>> I try that tomorrow.
>
> Well, please try commit ef98988ba369 (Merge tag 'pm-extra-4.9-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm) which is the
> merge introducing the late cpufreq changes.  If the issue is there, please
> try to revert commit 899bb6642f2a (cpufreq: skip invalid entries when searching
> the frequency) which is the only cpufreq one that may matter for the schedutil
> governor (and I have one fix for that commit queued up already).
>

Is "cpufreq: fix overflow in cpufreq_table_find_index_dl()" the fix
you are speaking of?

Fixes: 899bb6642f2a (cpufreq: skip invalid entries when searching the frequency)

If yes, can you add a hint in the commit message describing the impact
like here a slow-down of building a linux-kernel.
With a reference to this ML-thread?

I try to test this but don't be disappointed if you get no answer.

- Sedat -

[1] https://git.kernel.org/cgit/linux/kernel/git/rafael/linux-pm.git/commit/?h=linux-next&id=f7a7a80ae30521b65a6dfc98df45d3ec9e238d73
[2] https://git.kernel.org/cgit/linux/kernel/git/rafael/linux-pm.git/patch/?id=f7a7a80ae30521b65a6dfc98df45d3ec9e238d73

Re: [4.9-rc1] Build-time 2x slower

[Sedat Dilek]

On Wed, Oct 19, 2016 at 5:29 PM, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
> On Wed, Oct 19, 2016 at 4:07 AM, Jörg Otte <jrg.otte@gmail.com> wrote:
>>
>> Additional info: I usally use schedutil governor.
>> If I switch to performance governor problems go away.
>> Maybe a cpufreq problem?
>
> Oh, I completely misread the original bug report, and then didn't read
> your confirmation email right.
>
> I thought you had a slower build of the different kernels (when
> building on the same kernel), and that the _build_ itself had slowed
> down for some reason. But you're actually saying that doing the _same_
> build actually takes longer when running on 4.9-rc1.
>
> My bad.
>

English is not my native language.
To clarify building whatever Linux-version "under Linux v4.9-rc1" means...
Running Linux v4.9-rc1...
( ...with schedutil as cpufreq-governor... )
and building software in this environment.

- Sedat -

> There are a few small cpufreq changes there in between commit
> 29fbff8698fc (that you reported was fine - please tell me I got _that_
> right, at least?) and 4.9-rc1. Adding Rafael to the cc.
>
> That said, none of them look all that likely to me. It *would* be good
> if you could bisect it a bit (perhaps not fully, but a couple of
> bisection steps to narrow down what area it is).
>
>                  Linus

Re: [4.9-rc1] Build-time 2x slower

[Jörg Otte]

2016-10-19 22:55 GMT+02:00 Rafael J. Wysocki <rjw@rjwysocki.net>:
> On Wednesday, October 19, 2016 06:59:35 PM Jörg Otte wrote:
>> 2016-10-19 17:29 GMT+02:00 Linus Torvalds <torvalds@linux-foundation.org>:
>> > On Wed, Oct 19, 2016 at 4:07 AM, Jörg Otte <jrg.otte@gmail.com> wrote:
>> >>
>> >> Additional info: I usally use schedutil governor.
>> >> If I switch to performance governor problems go away.
>> >> Maybe a cpufreq problem?
>> >
>> > Oh, I completely misread the original bug report, and then didn't read
>> > your confirmation email right.
>> >
>> > I thought you had a slower build of the different kernels (when
>> > building on the same kernel), and that the _build_ itself had slowed
>> > down for some reason. But you're actually saying that doing the _same_
>> > build actually takes longer when running on 4.9-rc1.
>>
>> Exactly!
>>
>> Btw: ondemand governor is also good.
>>
>> > There are a few small cpufreq changes there in between commit
>> > 29fbff8698fc (that you reported was fine - please tell me I got _that_
>> > right, at least?) and 4.9-rc1.
>>
>> Perfect! That's what I mean.
>>
>> > Adding Rafael to the cc.
>> >
>> > That said, none of them look all that likely to me. It *would* be good
>> > if you could bisect it a bit (perhaps not fully, but a couple of
>> > bisection steps to narrow down what area it is).
>>
>> I try that tomorrow.
>
> Well, please try commit ef98988ba369 (Merge tag 'pm-extra-4.9-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm) which is the
> merge introducing the late cpufreq changes.  If the issue is there, please
> try to revert commit 899bb6642f2a (cpufreq: skip invalid entries when searching
> the frequency) which is the only cpufreq one that may matter for the schedutil
> governor (and I have one fix for that commit queued up already).
>

I first tried the merge but git said I'm already uptodate (my tree
is at 1a1891d Merge tag 'for-f2fs-4.9-rc2' of
git://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs already)

Then I did the revert of 899bb6642f2a and
that worked fine for me.

Thanks, Jörg

Re: [4.9-rc1] Build-time 2x slower

[Sedat Dilek]

On Thu, Oct 20, 2016 at 9:57 AM, Jörg Otte <jrg.otte@gmail.com> wrote:
> 2016-10-19 22:55 GMT+02:00 Rafael J. Wysocki <rjw@rjwysocki.net>:
>> On Wednesday, October 19, 2016 06:59:35 PM Jörg Otte wrote:
>>> 2016-10-19 17:29 GMT+02:00 Linus Torvalds <torvalds@linux-foundation.org>:
>>> > On Wed, Oct 19, 2016 at 4:07 AM, Jörg Otte <jrg.otte@gmail.com> wrote:
>>> >>
>>> >> Additional info: I usally use schedutil governor.
>>> >> If I switch to performance governor problems go away.
>>> >> Maybe a cpufreq problem?
>>> >
>>> > Oh, I completely misread the original bug report, and then didn't read
>>> > your confirmation email right.
>>> >
>>> > I thought you had a slower build of the different kernels (when
>>> > building on the same kernel), and that the _build_ itself had slowed
>>> > down for some reason. But you're actually saying that doing the _same_
>>> > build actually takes longer when running on 4.9-rc1.
>>>
>>> Exactly!
>>>
>>> Btw: ondemand governor is also good.
>>>
>>> > There are a few small cpufreq changes there in between commit
>>> > 29fbff8698fc (that you reported was fine - please tell me I got _that_
>>> > right, at least?) and 4.9-rc1.
>>>
>>> Perfect! That's what I mean.
>>>
>>> > Adding Rafael to the cc.
>>> >
>>> > That said, none of them look all that likely to me. It *would* be good
>>> > if you could bisect it a bit (perhaps not fully, but a couple of
>>> > bisection steps to narrow down what area it is).
>>>
>>> I try that tomorrow.
>>
>> Well, please try commit ef98988ba369 (Merge tag 'pm-extra-4.9-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm) which is the
>> merge introducing the late cpufreq changes.  If the issue is there, please
>> try to revert commit 899bb6642f2a (cpufreq: skip invalid entries when searching
>> the frequency) which is the only cpufreq one that may matter for the schedutil
>> governor (and I have one fix for that commit queued up already).
>>
>
> I first tried the merge but git said I'm already uptodate (my tree
> is at 1a1891d Merge tag 'for-f2fs-4.9-rc2' of
> git://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs already)
>

I guess Rafael meant try his latest linux-pm Git tree with branch and
look if the problem still exists.

> Then I did the revert of 899bb6642f2a and
> that worked fine for me.
>

Thanks for the feedback!
Hope I can try the revert and in a 2nd run the possible fix pending in
Rafael's linux-pm.git#linux-next.

- Sedat -

[1] https://git.kernel.org/cgit/linux/kernel/git/rafael/linux-pm.git/patch/?id=f7a7a80ae30521b65a6dfc98df45d3ec9e238d73

Re: [4.9-rc1] Build-time 2x slower

[Rafael J. Wysocki]

On Thursday, October 20, 2016 09:57:45 AM Jörg Otte wrote:
> 2016-10-19 22:55 GMT+02:00 Rafael J. Wysocki <rjw@rjwysocki.net>:
> > On Wednesday, October 19, 2016 06:59:35 PM Jörg Otte wrote:
> >> 2016-10-19 17:29 GMT+02:00 Linus Torvalds <torvalds@linux-foundation.org>:
> >> > On Wed, Oct 19, 2016 at 4:07 AM, Jörg Otte <jrg.otte@gmail.com> wrote:
> >> >>
> >> >> Additional info: I usally use schedutil governor.
> >> >> If I switch to performance governor problems go away.
> >> >> Maybe a cpufreq problem?
> >> >
> >> > Oh, I completely misread the original bug report, and then didn't read
> >> > your confirmation email right.
> >> >
> >> > I thought you had a slower build of the different kernels (when
> >> > building on the same kernel), and that the _build_ itself had slowed
> >> > down for some reason. But you're actually saying that doing the _same_
> >> > build actually takes longer when running on 4.9-rc1.
> >>
> >> Exactly!
> >>
> >> Btw: ondemand governor is also good.
> >>
> >> > There are a few small cpufreq changes there in between commit
> >> > 29fbff8698fc (that you reported was fine - please tell me I got _that_
> >> > right, at least?) and 4.9-rc1.
> >>
> >> Perfect! That's what I mean.
> >>
> >> > Adding Rafael to the cc.
> >> >
> >> > That said, none of them look all that likely to me. It *would* be good
> >> > if you could bisect it a bit (perhaps not fully, but a couple of
> >> > bisection steps to narrow down what area it is).
> >>
> >> I try that tomorrow.
> >
> > Well, please try commit ef98988ba369 (Merge tag 'pm-extra-4.9-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm) which is the
> > merge introducing the late cpufreq changes.  If the issue is there, please
> > try to revert commit 899bb6642f2a (cpufreq: skip invalid entries when searching
> > the frequency) which is the only cpufreq one that may matter for the schedutil
> > governor (and I have one fix for that commit queued up already).
> >
> 
> I first tried the merge but git said I'm already uptodate (my tree
> is at 1a1891d Merge tag 'for-f2fs-4.9-rc2' of
> git://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs already)
> 
> Then I did the revert of 899bb6642f2a and
> that worked fine for me.

OK, thanks!

Pointer arithmetics is messed up in that commit somewhat which may be the
reason for what you see.

Please check if this patch helps (instead of the revert):

https://patchwork.kernel.org/patch/9379639/

Thanks,
Rafael

Re: [4.9-rc1] Build-time 2x slower

[Rafael J. Wysocki]

On Thursday, October 20, 2016 09:41:34 AM Sedat Dilek wrote:
> On Wed, Oct 19, 2016 at 10:55 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> > On Wednesday, October 19, 2016 06:59:35 PM Jörg Otte wrote:
> >> 2016-10-19 17:29 GMT+02:00 Linus Torvalds <torvalds@linux-foundation.org>:
> >> > On Wed, Oct 19, 2016 at 4:07 AM, Jörg Otte <jrg.otte@gmail.com> wrote:
> >> >>
> >> >> Additional info: I usally use schedutil governor.
> >> >> If I switch to performance governor problems go away.
> >> >> Maybe a cpufreq problem?
> >> >
> >> > Oh, I completely misread the original bug report, and then didn't read
> >> > your confirmation email right.
> >> >
> >> > I thought you had a slower build of the different kernels (when
> >> > building on the same kernel), and that the _build_ itself had slowed
> >> > down for some reason. But you're actually saying that doing the _same_
> >> > build actually takes longer when running on 4.9-rc1.
> >>
> >> Exactly!
> >>
> >> Btw: ondemand governor is also good.
> >>
> >> > There are a few small cpufreq changes there in between commit
> >> > 29fbff8698fc (that you reported was fine - please tell me I got _that_
> >> > right, at least?) and 4.9-rc1.
> >>
> >> Perfect! That's what I mean.
> >>
> >> > Adding Rafael to the cc.
> >> >
> >> > That said, none of them look all that likely to me. It *would* be good
> >> > if you could bisect it a bit (perhaps not fully, but a couple of
> >> > bisection steps to narrow down what area it is).
> >>
> >> I try that tomorrow.
> >
> > Well, please try commit ef98988ba369 (Merge tag 'pm-extra-4.9-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm) which is the
> > merge introducing the late cpufreq changes.  If the issue is there, please
> > try to revert commit 899bb6642f2a (cpufreq: skip invalid entries when searching
> > the frequency) which is the only cpufreq one that may matter for the schedutil
> > governor (and I have one fix for that commit queued up already).
> >
> 
> Is "cpufreq: fix overflow in cpufreq_table_find_index_dl()" the fix
> you are speaking of?
> 
> Fixes: 899bb6642f2a (cpufreq: skip invalid entries when searching the frequency)

Yes.

> If yes, can you add a hint in the commit message describing the impact
> like here a slow-down of building a linux-kernel.
> With a reference to this ML-thread?

I will if that turns out to be the case.

Thanks,
Rafael

Re: [4.9-rc1] Build-time 2x slower

[Sedat Dilek]

On Thu, Oct 20, 2016 at 1:15 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> On Thursday, October 20, 2016 09:41:34 AM Sedat Dilek wrote:
>> On Wed, Oct 19, 2016 at 10:55 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
>> > On Wednesday, October 19, 2016 06:59:35 PM Jörg Otte wrote:
>> >> 2016-10-19 17:29 GMT+02:00 Linus Torvalds <torvalds@linux-foundation.org>:
>> >> > On Wed, Oct 19, 2016 at 4:07 AM, Jörg Otte <jrg.otte@gmail.com> wrote:
>> >> >>
>> >> >> Additional info: I usally use schedutil governor.
>> >> >> If I switch to performance governor problems go away.
>> >> >> Maybe a cpufreq problem?
>> >> >
>> >> > Oh, I completely misread the original bug report, and then didn't read
>> >> > your confirmation email right.
>> >> >
>> >> > I thought you had a slower build of the different kernels (when
>> >> > building on the same kernel), and that the _build_ itself had slowed
>> >> > down for some reason. But you're actually saying that doing the _same_
>> >> > build actually takes longer when running on 4.9-rc1.
>> >>
>> >> Exactly!
>> >>
>> >> Btw: ondemand governor is also good.
>> >>
>> >> > There are a few small cpufreq changes there in between commit
>> >> > 29fbff8698fc (that you reported was fine - please tell me I got _that_
>> >> > right, at least?) and 4.9-rc1.
>> >>
>> >> Perfect! That's what I mean.
>> >>
>> >> > Adding Rafael to the cc.
>> >> >
>> >> > That said, none of them look all that likely to me. It *would* be good
>> >> > if you could bisect it a bit (perhaps not fully, but a couple of
>> >> > bisection steps to narrow down what area it is).
>> >>
>> >> I try that tomorrow.
>> >
>> > Well, please try commit ef98988ba369 (Merge tag 'pm-extra-4.9-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm) which is the
>> > merge introducing the late cpufreq changes.  If the issue is there, please
>> > try to revert commit 899bb6642f2a (cpufreq: skip invalid entries when searching
>> > the frequency) which is the only cpufreq one that may matter for the schedutil
>> > governor (and I have one fix for that commit queued up already).
>> >
>>
>> Is "cpufreq: fix overflow in cpufreq_table_find_index_dl()" the fix
>> you are speaking of?
>>
>> Fixes: 899bb6642f2a (cpufreq: skip invalid entries when searching the frequency)
>
> Yes.
>
>> If yes, can you add a hint in the commit message describing the impact
>> like here a slow-down of building a linux-kernel.
>> With a reference to this ML-thread?
>
> I will if that turns out to be the case.
>

I have tried the revert and the patch from Sergey Senozhatsk pending
in linux-pm.git#linux-next.
Both fixes the issue for me.

Feel free to give appropriate credits and many thanks to Jörg.

I tried 'make -j3' in my last build and it was approx. 5mins faster in
my customized setup.
Will switch back to 2 parallel-make-jobs - it's safer for me.

Can you explain why this issue was not seen when building under Linux v4.8.x?
[1] says...
Cc: 4.8+ <stable@vger.kernel.org> # 4.8+

Thanks.

- sed@ -

[1] http://git.kernel.org/cgit/linux/kernel/git/rafael/linux-pm.git/commit/?h=linux-next&id=f7a7a80ae30521b65a6dfc98df45d3ec9e238d73

Re: [4.9-rc1] Build-time 2x slower

[Jörg Otte]

2016-10-20 13:14 GMT+02:00 Rafael J. Wysocki <rjw@rjwysocki.net>:
> On Thursday, October 20, 2016 09:57:45 AM Jörg Otte wrote:
>> 2016-10-19 22:55 GMT+02:00 Rafael J. Wysocki <rjw@rjwysocki.net>:
>> > On Wednesday, October 19, 2016 06:59:35 PM Jörg Otte wrote:
>> >> 2016-10-19 17:29 GMT+02:00 Linus Torvalds <torvalds@linux-foundation.org>:
>> >> > On Wed, Oct 19, 2016 at 4:07 AM, Jörg Otte <jrg.otte@gmail.com> wrote:
>> >> >>
>> >> >> Additional info: I usally use schedutil governor.
>> >> >> If I switch to performance governor problems go away.
>> >> >> Maybe a cpufreq problem?
>> >> >
>> >> > Oh, I completely misread the original bug report, and then didn't read
>> >> > your confirmation email right.
>> >> >
>> >> > I thought you had a slower build of the different kernels (when
>> >> > building on the same kernel), and that the _build_ itself had slowed
>> >> > down for some reason. But you're actually saying that doing the _same_
>> >> > build actually takes longer when running on 4.9-rc1.
>> >>
>> >> Exactly!
>> >>
>> >> Btw: ondemand governor is also good.
>> >>
>> >> > There are a few small cpufreq changes there in between commit
>> >> > 29fbff8698fc (that you reported was fine - please tell me I got _that_
>> >> > right, at least?) and 4.9-rc1.
>> >>
>> >> Perfect! That's what I mean.
>> >>
>> >> > Adding Rafael to the cc.
>> >> >
>> >> > That said, none of them look all that likely to me. It *would* be good
>> >> > if you could bisect it a bit (perhaps not fully, but a couple of
>> >> > bisection steps to narrow down what area it is).
>> >>
>> >> I try that tomorrow.
>> >
>> > Well, please try commit ef98988ba369 (Merge tag 'pm-extra-4.9-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm) which is the
>> > merge introducing the late cpufreq changes.  If the issue is there, please
>> > try to revert commit 899bb6642f2a (cpufreq: skip invalid entries when searching
>> > the frequency) which is the only cpufreq one that may matter for the schedutil
>> > governor (and I have one fix for that commit queued up already).
>> >
>>
>> I first tried the merge but git said I'm already uptodate (my tree
>> is at 1a1891d Merge tag 'for-f2fs-4.9-rc2' of
>> git://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs already)
>>
>> Then I did the revert of 899bb6642f2a and
>> that worked fine for me.
>
> OK, thanks!
>
> Pointer arithmetics is messed up in that commit somewhat which may be the
> reason for what you see.
>
> Please check if this patch helps (instead of the revert):
>
> https://patchwork.kernel.org/patch/9379639/
>
Yes, works well for me.

Thanks, Jörg

Re: [4.9-rc1] Build-time 2x slower

[Rafael J. Wysocki]

On Thursday, October 20, 2016 04:20:44 PM Sedat Dilek wrote:
> On Thu, Oct 20, 2016 at 1:15 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> > On Thursday, October 20, 2016 09:41:34 AM Sedat Dilek wrote:
> >> On Wed, Oct 19, 2016 at 10:55 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> >> > On Wednesday, October 19, 2016 06:59:35 PM Jörg Otte wrote:
> >> >> 2016-10-19 17:29 GMT+02:00 Linus Torvalds <torvalds@linux-foundation.org>:
> >> >> > On Wed, Oct 19, 2016 at 4:07 AM, Jörg Otte <jrg.otte@gmail.com> wrote:
> >> >> >>
> >> >> >> Additional info: I usally use schedutil governor.
> >> >> >> If I switch to performance governor problems go away.
> >> >> >> Maybe a cpufreq problem?
> >> >> >
> >> >> > Oh, I completely misread the original bug report, and then didn't read
> >> >> > your confirmation email right.
> >> >> >
> >> >> > I thought you had a slower build of the different kernels (when
> >> >> > building on the same kernel), and that the _build_ itself had slowed
> >> >> > down for some reason. But you're actually saying that doing the _same_
> >> >> > build actually takes longer when running on 4.9-rc1.
> >> >>
> >> >> Exactly!
> >> >>
> >> >> Btw: ondemand governor is also good.
> >> >>
> >> >> > There are a few small cpufreq changes there in between commit
> >> >> > 29fbff8698fc (that you reported was fine - please tell me I got _that_
> >> >> > right, at least?) and 4.9-rc1.
> >> >>
> >> >> Perfect! That's what I mean.
> >> >>
> >> >> > Adding Rafael to the cc.
> >> >> >
> >> >> > That said, none of them look all that likely to me. It *would* be good
> >> >> > if you could bisect it a bit (perhaps not fully, but a couple of
> >> >> > bisection steps to narrow down what area it is).
> >> >>
> >> >> I try that tomorrow.
> >> >
> >> > Well, please try commit ef98988ba369 (Merge tag 'pm-extra-4.9-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm) which is the
> >> > merge introducing the late cpufreq changes.  If the issue is there, please
> >> > try to revert commit 899bb6642f2a (cpufreq: skip invalid entries when searching
> >> > the frequency) which is the only cpufreq one that may matter for the schedutil
> >> > governor (and I have one fix for that commit queued up already).
> >> >
> >>
> >> Is "cpufreq: fix overflow in cpufreq_table_find_index_dl()" the fix
> >> you are speaking of?
> >>
> >> Fixes: 899bb6642f2a (cpufreq: skip invalid entries when searching the frequency)
> >
> > Yes.
> >
> >> If yes, can you add a hint in the commit message describing the impact
> >> like here a slow-down of building a linux-kernel.
> >> With a reference to this ML-thread?
> >
> > I will if that turns out to be the case.
> >
> 
> I have tried the revert and the patch from Sergey Senozhatsk pending
> in linux-pm.git#linux-next.
> Both fixes the issue for me.

OK, thanks for the confirmation!

> Feel free to give appropriate credits and many thanks to Jörg.
> 
> I tried 'make -j3' in my last build and it was approx. 5mins faster in
> my customized setup.
> Will switch back to 2 parallel-make-jobs - it's safer for me.
> 
> Can you explain why this issue was not seen when building under Linux v4.8.x?
> [1] says...
> Cc: 4.8+ <stable@vger.kernel.org> # 4.8+

The commit in question might not make it into 4.8.y yet.

Thanks,
Rafael

Re: [4.9-rc1] Build-time 2x slower

[Sedat Dilek]

On Thu, Oct 20, 2016 at 4:32 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> On Thursday, October 20, 2016 04:20:44 PM Sedat Dilek wrote:
>> On Thu, Oct 20, 2016 at 1:15 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
>> > On Thursday, October 20, 2016 09:41:34 AM Sedat Dilek wrote:
>> >> On Wed, Oct 19, 2016 at 10:55 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
>> >> > On Wednesday, October 19, 2016 06:59:35 PM Jörg Otte wrote:
>> >> >> 2016-10-19 17:29 GMT+02:00 Linus Torvalds <torvalds@linux-foundation.org>:
>> >> >> > On Wed, Oct 19, 2016 at 4:07 AM, Jörg Otte <jrg.otte@gmail.com> wrote:
>> >> >> >>
>> >> >> >> Additional info: I usally use schedutil governor.
>> >> >> >> If I switch to performance governor problems go away.
>> >> >> >> Maybe a cpufreq problem?
>> >> >> >
>> >> >> > Oh, I completely misread the original bug report, and then didn't read
>> >> >> > your confirmation email right.
>> >> >> >
>> >> >> > I thought you had a slower build of the different kernels (when
>> >> >> > building on the same kernel), and that the _build_ itself had slowed
>> >> >> > down for some reason. But you're actually saying that doing the _same_
>> >> >> > build actually takes longer when running on 4.9-rc1.
>> >> >>
>> >> >> Exactly!
>> >> >>
>> >> >> Btw: ondemand governor is also good.
>> >> >>
>> >> >> > There are a few small cpufreq changes there in between commit
>> >> >> > 29fbff8698fc (that you reported was fine - please tell me I got _that_
>> >> >> > right, at least?) and 4.9-rc1.
>> >> >>
>> >> >> Perfect! That's what I mean.
>> >> >>
>> >> >> > Adding Rafael to the cc.
>> >> >> >
>> >> >> > That said, none of them look all that likely to me. It *would* be good
>> >> >> > if you could bisect it a bit (perhaps not fully, but a couple of
>> >> >> > bisection steps to narrow down what area it is).
>> >> >>
>> >> >> I try that tomorrow.
>> >> >
>> >> > Well, please try commit ef98988ba369 (Merge tag 'pm-extra-4.9-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm) which is the
>> >> > merge introducing the late cpufreq changes.  If the issue is there, please
>> >> > try to revert commit 899bb6642f2a (cpufreq: skip invalid entries when searching
>> >> > the frequency) which is the only cpufreq one that may matter for the schedutil
>> >> > governor (and I have one fix for that commit queued up already).
>> >> >
>> >>
>> >> Is "cpufreq: fix overflow in cpufreq_table_find_index_dl()" the fix
>> >> you are speaking of?
>> >>
>> >> Fixes: 899bb6642f2a (cpufreq: skip invalid entries when searching the frequency)
>> >
>> > Yes.
>> >
>> >> If yes, can you add a hint in the commit message describing the impact
>> >> like here a slow-down of building a linux-kernel.
>> >> With a reference to this ML-thread?
>> >
>> > I will if that turns out to be the case.
>> >
>>
>> I have tried the revert and the patch from Sergey Senozhatsk pending
>> in linux-pm.git#linux-next.
>> Both fixes the issue for me.
>
> OK, thanks for the confirmation!
>
>> Feel free to give appropriate credits and many thanks to Jörg.
>>
>> I tried 'make -j3' in my last build and it was approx. 5mins faster in
>> my customized setup.
>> Will switch back to 2 parallel-make-jobs - it's safer for me.
>>
>> Can you explain why this issue was not seen when building under Linux v4.8.x?
>> [1] says...
>> Cc: 4.8+ <stable@vger.kernel.org> # 4.8+
>
> The commit in question might not make it into 4.8.y yet.
>

It's a bit terrifying to see these impacts of schedutil cpufreq-governor.

Do you have a test-case or how do you test with / for schedutil?

- sed@ -

Re: [4.9-rc1] Build-time 2x slower

[Rafael J. Wysocki]

On Thursday, October 20, 2016 04:38:39 PM Sedat Dilek wrote:
> On Thu, Oct 20, 2016 at 4:32 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> > On Thursday, October 20, 2016 04:20:44 PM Sedat Dilek wrote:
> >> On Thu, Oct 20, 2016 at 1:15 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> >> > On Thursday, October 20, 2016 09:41:34 AM Sedat Dilek wrote:
> >> >> On Wed, Oct 19, 2016 at 10:55 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> >> >> > On Wednesday, October 19, 2016 06:59:35 PM Jörg Otte wrote:
> >> >> >> 2016-10-19 17:29 GMT+02:00 Linus Torvalds <torvalds@linux-foundation.org>:
> >> >> >> > On Wed, Oct 19, 2016 at 4:07 AM, Jörg Otte <jrg.otte@gmail.com> wrote:
> >> >> >> >>
> >> >> >> >> Additional info: I usally use schedutil governor.
> >> >> >> >> If I switch to performance governor problems go away.
> >> >> >> >> Maybe a cpufreq problem?
> >> >> >> >
> >> >> >> > Oh, I completely misread the original bug report, and then didn't read
> >> >> >> > your confirmation email right.
> >> >> >> >
> >> >> >> > I thought you had a slower build of the different kernels (when
> >> >> >> > building on the same kernel), and that the _build_ itself had slowed
> >> >> >> > down for some reason. But you're actually saying that doing the _same_
> >> >> >> > build actually takes longer when running on 4.9-rc1.
> >> >> >>
> >> >> >> Exactly!
> >> >> >>
> >> >> >> Btw: ondemand governor is also good.
> >> >> >>
> >> >> >> > There are a few small cpufreq changes there in between commit
> >> >> >> > 29fbff8698fc (that you reported was fine - please tell me I got _that_
> >> >> >> > right, at least?) and 4.9-rc1.
> >> >> >>
> >> >> >> Perfect! That's what I mean.
> >> >> >>
> >> >> >> > Adding Rafael to the cc.
> >> >> >> >
> >> >> >> > That said, none of them look all that likely to me. It *would* be good
> >> >> >> > if you could bisect it a bit (perhaps not fully, but a couple of
> >> >> >> > bisection steps to narrow down what area it is).
> >> >> >>
> >> >> >> I try that tomorrow.
> >> >> >
> >> >> > Well, please try commit ef98988ba369 (Merge tag 'pm-extra-4.9-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm) which is the
> >> >> > merge introducing the late cpufreq changes.  If the issue is there, please
> >> >> > try to revert commit 899bb6642f2a (cpufreq: skip invalid entries when searching
> >> >> > the frequency) which is the only cpufreq one that may matter for the schedutil
> >> >> > governor (and I have one fix for that commit queued up already).
> >> >> >
> >> >>
> >> >> Is "cpufreq: fix overflow in cpufreq_table_find_index_dl()" the fix
> >> >> you are speaking of?
> >> >>
> >> >> Fixes: 899bb6642f2a (cpufreq: skip invalid entries when searching the frequency)
> >> >
> >> > Yes.
> >> >
> >> >> If yes, can you add a hint in the commit message describing the impact
> >> >> like here a slow-down of building a linux-kernel.
> >> >> With a reference to this ML-thread?
> >> >
> >> > I will if that turns out to be the case.
> >> >
> >>
> >> I have tried the revert and the patch from Sergey Senozhatsk pending
> >> in linux-pm.git#linux-next.
> >> Both fixes the issue for me.
> >
> > OK, thanks for the confirmation!
> >
> >> Feel free to give appropriate credits and many thanks to Jörg.
> >>
> >> I tried 'make -j3' in my last build and it was approx. 5mins faster in
> >> my customized setup.
> >> Will switch back to 2 parallel-make-jobs - it's safer for me.
> >>
> >> Can you explain why this issue was not seen when building under Linux v4.8.x?
> >> [1] says...
> >> Cc: 4.8+ <stable@vger.kernel.org> # 4.8+
> >
> > The commit in question might not make it into 4.8.y yet.
> >
> 
> It's a bit terrifying to see these impacts of schedutil cpufreq-governor.
> 
> Do you have a test-case or how do you test with / for schedutil?

In the same way as for the other governors: run a bunch of workloads and
see if there's any impact on the execution times etc.

And of course schedutil was not the only thing affected in this case
quite obviously which you can see from the fix changelog.

Thanks,
Rafael

Re: [4.9-rc1] Build-time 2x slower

[Rafael J. Wysocki]

On Thursday, October 20, 2016 04:32:10 PM Jörg Otte wrote:
> 2016-10-20 13:14 GMT+02:00 Rafael J. Wysocki <rjw@rjwysocki.net>:
> > On Thursday, October 20, 2016 09:57:45 AM Jörg Otte wrote:
> >> 2016-10-19 22:55 GMT+02:00 Rafael J. Wysocki <rjw@rjwysocki.net>:
> >> > On Wednesday, October 19, 2016 06:59:35 PM Jörg Otte wrote:
> >> >> 2016-10-19 17:29 GMT+02:00 Linus Torvalds <torvalds@linux-foundation.org>:
> >> >> > On Wed, Oct 19, 2016 at 4:07 AM, Jörg Otte <jrg.otte@gmail.com> wrote:
> >> >> >>
> >> >> >> Additional info: I usally use schedutil governor.
> >> >> >> If I switch to performance governor problems go away.
> >> >> >> Maybe a cpufreq problem?
> >> >> >
> >> >> > Oh, I completely misread the original bug report, and then didn't read
> >> >> > your confirmation email right.
> >> >> >
> >> >> > I thought you had a slower build of the different kernels (when
> >> >> > building on the same kernel), and that the _build_ itself had slowed
> >> >> > down for some reason. But you're actually saying that doing the _same_
> >> >> > build actually takes longer when running on 4.9-rc1.
> >> >>
> >> >> Exactly!
> >> >>
> >> >> Btw: ondemand governor is also good.
> >> >>
> >> >> > There are a few small cpufreq changes there in between commit
> >> >> > 29fbff8698fc (that you reported was fine - please tell me I got _that_
> >> >> > right, at least?) and 4.9-rc1.
> >> >>
> >> >> Perfect! That's what I mean.
> >> >>
> >> >> > Adding Rafael to the cc.
> >> >> >
> >> >> > That said, none of them look all that likely to me. It *would* be good
> >> >> > if you could bisect it a bit (perhaps not fully, but a couple of
> >> >> > bisection steps to narrow down what area it is).
> >> >>
> >> >> I try that tomorrow.
> >> >
> >> > Well, please try commit ef98988ba369 (Merge tag 'pm-extra-4.9-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm) which is the
> >> > merge introducing the late cpufreq changes.  If the issue is there, please
> >> > try to revert commit 899bb6642f2a (cpufreq: skip invalid entries when searching
> >> > the frequency) which is the only cpufreq one that may matter for the schedutil
> >> > governor (and I have one fix for that commit queued up already).
> >> >
> >>
> >> I first tried the merge but git said I'm already uptodate (my tree
> >> is at 1a1891d Merge tag 'for-f2fs-4.9-rc2' of
> >> git://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs already)
> >>
> >> Then I did the revert of 899bb6642f2a and
> >> that worked fine for me.
> >
> > OK, thanks!
> >
> > Pointer arithmetics is messed up in that commit somewhat which may be the
> > reason for what you see.
> >
> > Please check if this patch helps (instead of the revert):
> >
> > https://patchwork.kernel.org/patch/9379639/
> >
> Yes, works well for me.

OK, thanks!

Rafael

Re: [4.9-rc1] Build-time 2x slower

[Sedat Dilek]

On Thu, Oct 20, 2016 at 4:32 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> On Thursday, October 20, 2016 04:20:44 PM Sedat Dilek wrote:
>> On Thu, Oct 20, 2016 at 1:15 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
>> > On Thursday, October 20, 2016 09:41:34 AM Sedat Dilek wrote:
>> >> On Wed, Oct 19, 2016 at 10:55 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
>> >> > On Wednesday, October 19, 2016 06:59:35 PM Jörg Otte wrote:
>> >> >> 2016-10-19 17:29 GMT+02:00 Linus Torvalds <torvalds@linux-foundation.org>:
>> >> >> > On Wed, Oct 19, 2016 at 4:07 AM, Jörg Otte <jrg.otte@gmail.com> wrote:
>> >> >> >>
>> >> >> >> Additional info: I usally use schedutil governor.
>> >> >> >> If I switch to performance governor problems go away.
>> >> >> >> Maybe a cpufreq problem?
>> >> >> >
>> >> >> > Oh, I completely misread the original bug report, and then didn't read
>> >> >> > your confirmation email right.
>> >> >> >
>> >> >> > I thought you had a slower build of the different kernels (when
>> >> >> > building on the same kernel), and that the _build_ itself had slowed
>> >> >> > down for some reason. But you're actually saying that doing the _same_
>> >> >> > build actually takes longer when running on 4.9-rc1.
>> >> >>
>> >> >> Exactly!
>> >> >>
>> >> >> Btw: ondemand governor is also good.
>> >> >>
>> >> >> > There are a few small cpufreq changes there in between commit
>> >> >> > 29fbff8698fc (that you reported was fine - please tell me I got _that_
>> >> >> > right, at least?) and 4.9-rc1.
>> >> >>
>> >> >> Perfect! That's what I mean.
>> >> >>
>> >> >> > Adding Rafael to the cc.
>> >> >> >
>> >> >> > That said, none of them look all that likely to me. It *would* be good
>> >> >> > if you could bisect it a bit (perhaps not fully, but a couple of
>> >> >> > bisection steps to narrow down what area it is).
>> >> >>
>> >> >> I try that tomorrow.
>> >> >
>> >> > Well, please try commit ef98988ba369 (Merge tag 'pm-extra-4.9-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm) which is the
>> >> > merge introducing the late cpufreq changes.  If the issue is there, please
>> >> > try to revert commit 899bb6642f2a (cpufreq: skip invalid entries when searching
>> >> > the frequency) which is the only cpufreq one that may matter for the schedutil
>> >> > governor (and I have one fix for that commit queued up already).
>> >> >
>> >>
>> >> Is "cpufreq: fix overflow in cpufreq_table_find_index_dl()" the fix
>> >> you are speaking of?
>> >>
>> >> Fixes: 899bb6642f2a (cpufreq: skip invalid entries when searching the frequency)
>> >
>> > Yes.
>> >
>> >> If yes, can you add a hint in the commit message describing the impact
>> >> like here a slow-down of building a linux-kernel.
>> >> With a reference to this ML-thread?
>> >
>> > I will if that turns out to be the case.
>> >
>>
>> I have tried the revert and the patch from Sergey Senozhatsk pending
>> in linux-pm.git#linux-next.
>> Both fixes the issue for me.
>
> OK, thanks for the confirmation!
>
>> Feel free to give appropriate credits and many thanks to Jörg.
>>
>> I tried 'make -j3' in my last build and it was approx. 5mins faster in
>> my customized setup.
>> Will switch back to 2 parallel-make-jobs - it's safer for me.
>>
>> Can you explain why this issue was not seen when building under Linux v4.8.x?
>> [1] says...
>> Cc: 4.8+ <stable@vger.kernel.org> # 4.8+
>
> The commit in question might not make it into 4.8.y yet.
>

For all followers...

The fix is now in Linus tree (see [1]).

- Sedat -

[1] http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=c6fe46a79ecd79606bb96fada4515f6b23f87b62