makedepend in Mesa

makedepend in Mesa

[Xavier Chantry]

While keeping up-to-date the nouveau mesa driver (either classic or
gallium), or doing regression testing, the big majority of my rebuilds
resulted in segfaults.
I am not talking about autogen or configure detection. I believe this
also works automatically in other projects and doesn't with mesa, but
forgetting to do this usually causes a build failure. Then
autogen/configure can be run and make can resume the build.
What is more problematic is when an apparently successful build does
not work. The reply I used to get is just to make clean or
distclean/realclean. clean is usually works, but rebuilding everything
takes time. And regression testing needs a lot of rebuilding :)
Anyway I found this IRC discussion yesterday quite interesting :

20:30  * jbarnes curses the mesa build system
20:31 < jbarnes> change intel_fbo.h, nothing rebuilds
20:32 < Dr_Jakob> makedepend installed?
20:33 < jbarnes> yeah but it looks like we don't symlink intel_fbo.h
into i915 and include it in the makefile
20:33 < jbarnes> so I guess the deps don't get picked up
20:34 < Dr_Jakob> Hmm
20:34 < jbarnes> oh no I guess I don't have makedepend
20:34 -!- jsgf [~jeremy-ZcXrCSuhvlktFdR4KeJ2yPcEJKo2Qz8o1NOjf9nqTsILN5Mb03AwvQ@public.gmane.org] has
joined #dri-devel
20:34  * jbarnes tries again with that installed
20:35 < jbarnes> don't most projects just use gcc's dep tracking?
20:35 < Dr_Jakob> yes
20:36 < Dr_Jakob> I tried to add that but it mostly turned into fail.
20:37 < jbarnes> Dr_Jakob: you guys mostly build with scons these days right?
20:37 < jbarnes> and that has a separate set of files for tracking sources?
20:37 < Dr_Jakob> For windows yes, but I use make for linux.
20:37 < Dr_Jakob> Then again I'm pretty good at installing makedepend ;-)
20:37 < jbarnes> heh
20:39 < Dr_Jakob> I don't often have to do a make clean.
20:40 < suokko> I guess there is something broken in radeon makefiles
because make clean is required quite often
20:40 < suokko> Luckily ccache is very fast
20:40 < MostAwesomeDude> Yeah, ccache is a much better friend than makedepend.


21:48 < shining> still not very clear to me, should everyone have
makedepend installed for building mesa ?
21:49 < Dr_Jakob> it is higly recommended yes.
21:51 < shining> does it help with mesa build failures ? I have seen
many many times in #nouveau weird mesa behavior
                 because of build failure, and a make
distclean/realclean fixed it
21:51 < shining> sorry its not build failure
21:51 < Dr_Jakob> yes
21:51 < shining> it *seems* to build fun but it doesnt work correctly,
most of the time it simply segfaults
21:52 < shining> huh
21:52 < shining> s/fun/fine
21:54 < suokko> shining: Problem without makedepend is that in rebuild
make doesn't build all files that should be
                rebuild because it doesn't know about included files.
So if you link old and new object files and
                some memory layout changed you will get segfaults

22:49 < shining> then I would workaround it by requiring makedepend
22:51 < zackr> we don't have a configure step so you'll need to most
likely rewrite the build system to do that


I probably should look into ccache (and I probably will), but if
makedepend improves the rebuild situation, I believe it should be
better advertised.
And unless I am mistaken, mesa does have a configure step, I even
believed most people use that.
So my simple suggestion would be to simply print a warning if
makedepend is not detected.

I saw a report saying make clean was still needed with makedepend
installed, but maybe not every parts of mesa uses makedepend
correctly, like the nouveau driver ?

diff --git a/configure.ac b/configure.ac
index 485836a..ba5f267 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1441,6 +1441,11 @@ echo "        CFLAGS:          $cflags"
 echo "        CXXFLAGS:        $cxxflags"
 echo "        Macros:          $defines"

+if test "x$MKDEP" = x; then
+       echo ""
+       echo "        warning: makedepend is not installed, so it is
recommended to make distclean after any code change."
+fi
+
 echo ""
 echo "        Run '${MAKE-make}' to build Mesa"
 echo ""

Re: makedepend in Mesa

[Dan Nicholson]

On Tue, Feb 23, 2010 at 1:29 PM, Xavier Chantry
<chantry.xavier@gmail.com> wrote:
> While keeping up-to-date the nouveau mesa driver (either classic or
> gallium), or doing regression testing, the big majority of my rebuilds
> resulted in segfaults.
> I am not talking about autogen or configure detection. I believe this
> also works automatically in other projects and doesn't with mesa, but
> forgetting to do this usually causes a build failure. Then
> autogen/configure can be run and make can resume the build.
> What is more problematic is when an apparently successful build does
> not work. The reply I used to get is just to make clean or
> distclean/realclean. clean is usually works, but rebuilding everything
> takes time. And regression testing needs a lot of rebuilding :)
> Anyway I found this IRC discussion yesterday quite interesting :
>
> 20:30  * jbarnes curses the mesa build system
> 20:31 < jbarnes> change intel_fbo.h, nothing rebuilds
> 20:32 < Dr_Jakob> makedepend installed?
> 20:33 < jbarnes> yeah but it looks like we don't symlink intel_fbo.h
> into i915 and include it in the makefile
> 20:33 < jbarnes> so I guess the deps don't get picked up
> 20:34 < Dr_Jakob> Hmm
> 20:34 < jbarnes> oh no I guess I don't have makedepend
> 20:34 -!- jsgf [~jeremy@adsl-69-107-81-54.dsl.pltn13.pacbell.net] has
> joined #dri-devel
> 20:34  * jbarnes tries again with that installed
> 20:35 < jbarnes> don't most projects just use gcc's dep tracking?
> 20:35 < Dr_Jakob> yes
> 20:36 < Dr_Jakob> I tried to add that but it mostly turned into fail.

We could use gcc directly for depends (I have a patch to do it), but:

1. I don't think it would actually help much in terms of rebuilds
since makedepend seems to do a perfectly adequate job of finding the
needed headers.

2. gcc expects to output a single make target for a single source
file. mesa just tosses all the source files at makedepend which
generates a depend file with a bunch of targets. gcc is phenomenally
slow doing the same. Changing the build to include one dep file per
source file would be a ton of churn.

3. The fast way automake does gcc dep tracking is to do it as a step
during the compile. Since gcc fully preprocesses the source file
before generating the make target, you can get it for free during the
compile. Doing as a separate step like mesa means you're throwing away
the preprocessing and then doing it again immediately afterward. The
time adds up.

I guess what I'm saying is that I wouldn't bother with gcc dep
tracking unless it's coming as part of automake.

> 20:37 < jbarnes> Dr_Jakob: you guys mostly build with scons these days right?
> 20:37 < jbarnes> and that has a separate set of files for tracking sources?
> 20:37 < Dr_Jakob> For windows yes, but I use make for linux.
> 20:37 < Dr_Jakob> Then again I'm pretty good at installing makedepend ;-)
> 20:37 < jbarnes> heh
> 20:39 < Dr_Jakob> I don't often have to do a make clean.
> 20:40 < suokko> I guess there is something broken in radeon makefiles
> because make clean is required quite often
> 20:40 < suokko> Luckily ccache is very fast
> 20:40 < MostAwesomeDude> Yeah, ccache is a much better friend than makedepend.
>
>
> 21:48 < shining> still not very clear to me, should everyone have
> makedepend installed for building mesa ?
> 21:49 < Dr_Jakob> it is higly recommended yes.
> 21:51 < shining> does it help with mesa build failures ? I have seen
> many many times in #nouveau weird mesa behavior
>                 because of build failure, and a make
> distclean/realclean fixed it
> 21:51 < shining> sorry its not build failure
> 21:51 < Dr_Jakob> yes
> 21:51 < shining> it *seems* to build fun but it doesnt work correctly,
> most of the time it simply segfaults
> 21:52 < shining> huh
> 21:52 < shining> s/fun/fine
> 21:54 < suokko> shining: Problem without makedepend is that in rebuild
> make doesn't build all files that should be
>                rebuild because it doesn't know about included files.
> So if you link old and new object files and
>                some memory layout changed you will get segfaults
>
> 22:49 < shining> then I would workaround it by requiring makedepend
> 22:51 < zackr> we don't have a configure step so you'll need to most
> likely rewrite the build system to do that
>
>
> I probably should look into ccache (and I probably will), but if
> makedepend improves the rebuild situation, I believe it should be
> better advertised.
> And unless I am mistaken, mesa does have a configure step, I even
> believed most people use that.
> So my simple suggestion would be to simply print a warning if
> makedepend is not detected.

configure checks for makedepend, but doesn't error if it's not found.
It probably should. Likewise, the commands running makedepend should
probably not be silenced with stderr redirected to /dev/null. That
would break builds for people using the static configs without
makedepend.

> I saw a report saying make clean was still needed with makedepend
> installed, but maybe not every parts of mesa uses makedepend
> correctly, like the nouveau driver ?

I think the problem is more that it's not used consistently since it
has to be setup manually in the Makefile vs. automatically like
automake.

> diff --git a/configure.ac b/configure.ac
> index 485836a..ba5f267 100644
> --- a/configure.ac
> +++ b/configure.ac
> @@ -1441,6 +1441,11 @@ echo "        CFLAGS:          $cflags"
>  echo "        CXXFLAGS:        $cxxflags"
>  echo "        Macros:          $defines"
>
> +if test "x$MKDEP" = x; then
> +       echo ""
> +       echo "        warning: makedepend is not installed, so it is
> recommended to make distclean after any code change."
> +fi
> +

I'd be comfortable having makedepend be a hard requirement in
configure. Otherwise, I'd rather have this be AC_MSG_WARN([your
message]) right after the AC_PATH_PROG for makedepend.

--
Dan

------------------------------------------------------------------------------
Download Intel&#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Mesa3d-dev mailing list
Mesa3d-dev@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mesa3d-dev

Re: [Mesa3d-dev] makedepend in Mesa

[Xavier Chantry]

On Tue, Feb 23, 2010 at 11:14 PM, Dan Nicholson <dbn.lists-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> We could use gcc directly for depends (I have a patch to do it), but:
>
> 1. I don't think it would actually help much in terms of rebuilds
> since makedepend seems to do a perfectly adequate job of finding the
> needed headers.
>
> 2. gcc expects to output a single make target for a single source
> file. mesa just tosses all the source files at makedepend which
> generates a depend file with a bunch of targets. gcc is phenomenally
> slow doing the same. Changing the build to include one dep file per
> source file would be a ton of churn.
>
> 3. The fast way automake does gcc dep tracking is to do it as a step
> during the compile. Since gcc fully preprocesses the source file
> before generating the make target, you can get it for free during the
> compile. Doing as a separate step like mesa means you're throwing away
> the preprocessing and then doing it again immediately afterward. The
> time adds up.
>
> I guess what I'm saying is that I wouldn't bother with gcc dep
> tracking unless it's coming as part of automake.
>

Thanks for all these explanations !

>
> configure checks for makedepend, but doesn't error if it's not found.
> It probably should. Likewise, the commands running makedepend should
> probably not be silenced with stderr redirected to /dev/null. That
> would break builds for people using the static configs without
> makedepend.
>

Why do we need static configs ?
If I understand correctly, we have the choice between 1 autoconf file,
and more than 100 static config files ?
And autoconf can potentially capture all these static configurations and more ?

>
> I think the problem is more that it's not used consistently since it
> has to be setup manually in the Makefile vs. automatically like
> automake.
>

My understanding of autotools is still very limited. mesa currently
doesn't use automake, does it ?
A git grep automake lets me believe that it was dropped in 5.1

So currently makedepend needs to be setup manually in all the Makefiles ?

And using automake would give a big advantage on the two sides :
1) automatic makedepend setup
2) efficient use of gcc dep tracking
?

>
> I'd be comfortable having makedepend be a hard requirement in
> configure. Otherwise, I'd rather have this be AC_MSG_WARN([your
> message]) right after the AC_PATH_PROG for makedepend.
>

Actually I wanted to propose a hard requirement but then thought some
people might complain.
If I just put a AC_MSG_WARN right after AC_PATH_PROG, I am afraid it
won't make a big difference as it won't be very visible.
To make a hard requirement, I could just use AC_MSG_ERROR instead ?

[I wonder if it wouldn't be easier to make scons work like I want to
avoid all these problems.
I believe I would just need to add nouveau support, and probably look
into this :
# TODO: Build several variants at the same time?
# http://www.scons.org/wiki/SimultaneousVariantBuilds]

diff --git a/configure.ac b/configure.ac
index 485836a..a582337 100644
--- a/configure.ac
+++ b/configure.ac
@@ -28,8 +28,11 @@ AC_PROG_CPP
 AC_PROG_CC
 AC_PROG_CXX
 AC_CHECK_PROGS([MAKE], [gmake make])
-AC_PATH_PROG([MKDEP], [makedepend])
 AC_PATH_PROG([SED], [sed])
+AC_PATH_PROG([MKDEP], [makedepend], no)
+if test "x$MKDEP" = xno; then
+    AC_MSG_ERROR([makedepend not found])
+fi

 dnl Our fallback install-sh is a symlink to minstall. Use the existing
 dnl configuration in that case.

Re: [Mesa3d-dev] makedepend in Mesa

[Dan Nicholson]

On Thu, Feb 25, 2010 at 11:18 AM, Xavier Chantry
<chantry.xavier@gmail.com> wrote:
> On Tue, Feb 23, 2010 at 11:14 PM, Dan Nicholson <dbn.lists@gmail.com> wrote:
>>
>> We could use gcc directly for depends (I have a patch to do it), but:
>>
>> 1. I don't think it would actually help much in terms of rebuilds
>> since makedepend seems to do a perfectly adequate job of finding the
>> needed headers.
>>
>> 2. gcc expects to output a single make target for a single source
>> file. mesa just tosses all the source files at makedepend which
>> generates a depend file with a bunch of targets. gcc is phenomenally
>> slow doing the same. Changing the build to include one dep file per
>> source file would be a ton of churn.
>>
>> 3. The fast way automake does gcc dep tracking is to do it as a step
>> during the compile. Since gcc fully preprocesses the source file
>> before generating the make target, you can get it for free during the
>> compile. Doing as a separate step like mesa means you're throwing away
>> the preprocessing and then doing it again immediately afterward. The
>> time adds up.
>>
>> I guess what I'm saying is that I wouldn't bother with gcc dep
>> tracking unless it's coming as part of automake.
>>
>
> Thanks for all these explanations !
>
>>
>> configure checks for makedepend, but doesn't error if it's not found.
>> It probably should. Likewise, the commands running makedepend should
>> probably not be silenced with stderr redirected to /dev/null. That
>> would break builds for people using the static configs without
>> makedepend.
>>
>
> Why do we need static configs ?
> If I understand correctly, we have the choice between 1 autoconf file,
> and more than 100 static config files ?
> And autoconf can potentially capture all these static configurations and more ?

Because a lot of the vmware guys use the static configs. They like
being able to just do "make linux".

>> I think the problem is more that it's not used consistently since it
>> has to be setup manually in the Makefile vs. automatically like
>> automake.
>>
>
> My understanding of autotools is still very limited. mesa currently
> doesn't use automake, does it ?
> A git grep automake lets me believe that it was dropped in 5.1

Yeah, we just use autoconf to generate configs/autoconf. After that
it's treated just like one of the static configs. All the Makefiles
are manual.

> So currently makedepend needs to be setup manually in all the Makefiles ?
>
> And using automake would give a big advantage on the two sides :
> 1) automatic makedepend setup
> 2) efficient use of gcc dep tracking
> ?

Right, except automake doesn't use makedepend. It does the equivalent
with gcc's generation of make dependency rules. Look for the -M option
in gcc(1). Using automake and libtool would have a lot of advantages
over mesa's homegrown system. The big drawback is the added complexity
and the fact that Brian has historically wanted nothing to do with it.

>> I'd be comfortable having makedepend be a hard requirement in
>> configure. Otherwise, I'd rather have this be AC_MSG_WARN([your
>> message]) right after the AC_PATH_PROG for makedepend.
>>
>
> Actually I wanted to propose a hard requirement but then thought some
> people might complain.
> If I just put a AC_MSG_WARN right after AC_PATH_PROG, I am afraid it
> won't make a big difference as it won't be very visible.
> To make a hard requirement, I could just use AC_MSG_ERROR instead ?
>
> [I wonder if it wouldn't be easier to make scons work like I want to
> avoid all these problems.
> I believe I would just need to add nouveau support, and probably look
> into this :
> # TODO: Build several variants at the same time?
> # http://www.scons.org/wiki/SimultaneousVariantBuilds]

Well, that's for scons, which is completely independent of
autoconf/configs/Makefile. I believe the vmware guys use scons
primarily for building on windows, which the configs system can't
handle.

> diff --git a/configure.ac b/configure.ac
> index 485836a..a582337 100644
> --- a/configure.ac
> +++ b/configure.ac
> @@ -28,8 +28,11 @@ AC_PROG_CPP
>  AC_PROG_CC
>  AC_PROG_CXX
>  AC_CHECK_PROGS([MAKE], [gmake make])
> -AC_PATH_PROG([MKDEP], [makedepend])
>  AC_PATH_PROG([SED], [sed])
> +AC_PATH_PROG([MKDEP], [makedepend], no)
> +if test "x$MKDEP" = xno; then
> +    AC_MSG_ERROR([makedepend not found])
> +fi

Yep, that would do it. In fact, when I wrote configure.ac originally,
I wanted it to be a hard requirement and assumed that AC_PATH_PROG
would error by default if it didn't find the program.

--
Dan
_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/nouveau

Re: [Mesa3d-dev] makedepend in Mesa

[Xavier Chantry]

On Thu, Feb 25, 2010 at 9:41 PM, Dan Nicholson <dbn.lists-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
>> diff --git a/configure.ac b/configure.ac
>> index 485836a..a582337 100644
>> --- a/configure.ac
>> +++ b/configure.ac
>> @@ -28,8 +28,11 @@ AC_PROG_CPP
>>  AC_PROG_CC
>>  AC_PROG_CXX
>>  AC_CHECK_PROGS([MAKE], [gmake make])
>> -AC_PATH_PROG([MKDEP], [makedepend])
>>  AC_PATH_PROG([SED], [sed])
>> +AC_PATH_PROG([MKDEP], [makedepend], no)
>> +if test "x$MKDEP" = xno; then
>> +    AC_MSG_ERROR([makedepend not found])
>> +fi
>
> Yep, that would do it. In fact, when I wrote configure.ac originally,
> I wanted it to be a hard requirement and assumed that AC_PATH_PROG
> would error by default if it didn't find the program.
>

Should I resubmit a proper git patch ?

I have been running with makedepend installed on my 3 systems for 1
month, it has worked very well. I don't think I got ONE weird failure
in the past month, while I used to get more than once per week.
The only thing I need is to sometimes run autogen.sh and/or configure,
when there are some structure changes (e.g. nv30-nv40 merge to nvfx).
AFAIK this is also handled automatically in other projects. But this
is much less annoying, because it fails at build time, instead of
segfaulting at runtime.

Everyone complains about mesa crappy build system so I thought all the
failures I got were expected. Now I see they weren't, just that
makedepend is badly required :)

I am just worried some people might run into the same troubles as me.
But I wonder if there are any since only Dan answered in that thread.

Re: [Mesa3d-dev] makedepend in Mesa

[Dan Nicholson]

On Fri, Mar 19, 2010 at 12:23 PM, Xavier Chantry
<chantry.xavier@gmail.com> wrote:
> On Thu, Feb 25, 2010 at 9:41 PM, Dan Nicholson <dbn.lists@gmail.com> wrote:
>>
>>> diff --git a/configure.ac b/configure.ac
>>> index 485836a..a582337 100644
>>> --- a/configure.ac
>>> +++ b/configure.ac
>>> @@ -28,8 +28,11 @@ AC_PROG_CPP
>>>  AC_PROG_CC
>>>  AC_PROG_CXX
>>>  AC_CHECK_PROGS([MAKE], [gmake make])
>>> -AC_PATH_PROG([MKDEP], [makedepend])
>>>  AC_PATH_PROG([SED], [sed])
>>> +AC_PATH_PROG([MKDEP], [makedepend], no)
>>> +if test "x$MKDEP" = xno; then
>>> +    AC_MSG_ERROR([makedepend not found])
>>> +fi
>>
>> Yep, that would do it. In fact, when I wrote configure.ac originally,
>> I wanted it to be a hard requirement and assumed that AC_PATH_PROG
>> would error by default if it didn't find the program.
>>
>
> Should I resubmit a proper git patch ?

Sure.

> I have been running with makedepend installed on my 3 systems for 1
> month, it has worked very well. I don't think I got ONE weird failure
> in the past month, while I used to get more than once per week.
> The only thing I need is to sometimes run autogen.sh and/or configure,
> when there are some structure changes (e.g. nv30-nv40 merge to nvfx).
> AFAIK this is also handled automatically in other projects. But this
> is much less annoying, because it fails at build time, instead of
> segfaulting at runtime.

Yeah, automatic autotools updates could maybe be handled in the
top-level Makefile, but really that's more of a job that automake
takes care of.

> Everyone complains about mesa crappy build system so I thought all the
> failures I got were expected. Now I see they weren't, just that
> makedepend is badly required :)
>
> I am just worried some people might run into the same troubles as me.
> But I wonder if there are any since only Dan answered in that thread.

The only drawback I know of is adding a hard requirement on makedepend
where there wasn't one before, but distros usually have it.

--
Dan
_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/nouveau