[PATCH 0/4] buildsys: Build quicker (mostly tools and linux-user)

[PATCH 0/4] buildsys: Build quicker (mostly tools and linux-user)

[Philippe Mathieu-Daudé]

In some configuration (linux-user, tools) we can ignore building
various objects (and the libfdt).

Philippe Mathieu-Daudé (4):
  configure: Do not build libfdt is not required
  Makefile: Clarify all the codebase requires qom/ objects
  Makefile: Restrict system emulation and tools objects
  Makefile: Remove unhelpful comment

 configure     |  2 ++
 Makefile.objs | 31 ++++++++++---------------------
 2 files changed, 12 insertions(+), 21 deletions(-)

-- 
2.21.1

[PATCH 1/4] configure: Do not build libfdt is not required

[Philippe Mathieu-Daudé]

We only require libfdt for system emulation, in a small set
of architecture:

4077  # fdt support is mandatory for at least some target architectures,
4078  # so insist on it if we're building those system emulators.
4079  fdt_required=no
4080  for target in $target_list; do
4081    case $target in
4082      aarch64*-softmmu|arm*-softmmu|ppc*-softmmu|microblaze*-softmmu|mips64el-softmmu|riscv*-softmmu)
4083        fdt_required=yes

Do not build libfdt if we did not manually specified --enable-fdt.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 configure | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/configure b/configure
index 0ce2c0354a..266a8386d1 100755
--- a/configure
+++ b/configure
@@ -4092,6 +4092,8 @@ if test "$fdt_required" = "yes"; then
       "targets which need it (by specifying a cut down --target-list)."
   fi
   fdt=yes
+elif test "$fdt" != "yes" ; then
+  fdt=no
 fi
 
 if test "$fdt" != "no" ; then
-- 
2.21.1

[PATCH 2/4] Makefile: Clarify all the codebase requires qom/ objects

[Philippe Mathieu-Daudé]

QEMU user-mode also requires the qom/ objects, it is not only
used by "system emulation and qemu-img". As we will use a big
if() block, move it upper in the "Common libraries for tools
and emulators" section.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 Makefile.objs | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/Makefile.objs b/Makefile.objs
index 7c1e50f9d6..5aae561984 100644
--- a/Makefile.objs
+++ b/Makefile.objs
@@ -2,6 +2,7 @@
 # Common libraries for tools and emulators
 stub-obj-y = stubs/
 util-obj-y = crypto/ util/ qobject/ qapi/
+qom-obj-y = qom/
 
 chardev-obj-y = chardev/
 
@@ -26,11 +27,6 @@ block-obj-m = block/
 
 crypto-obj-y = crypto/
 
-#######################################################################
-# qom-obj-y is code used by both qemu system emulation and qemu-img
-
-qom-obj-y = qom/
-
 #######################################################################
 # io-obj-y is code used by both qemu system emulation and qemu-img
 
-- 
2.21.1

[PATCH 3/4] Makefile: Restrict system emulation and tools objects

[Philippe Mathieu-Daudé]

Restrict all the system emulation and tools objects with a
Makefile IF (CONFIG_SOFTMMU OR CONFIG_TOOLS) check.

Using the same description over and over is not very helpful.
Use it once, just before the if() block.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 Makefile.objs | 19 +++++++------------
 1 file changed, 7 insertions(+), 12 deletions(-)

diff --git a/Makefile.objs b/Makefile.objs
index 5aae561984..395dd1e670 100644
--- a/Makefile.objs
+++ b/Makefile.objs
@@ -4,16 +4,15 @@ stub-obj-y = stubs/
 util-obj-y = crypto/ util/ qobject/ qapi/
 qom-obj-y = qom/
 
+#######################################################################
+# code used by both qemu system emulation and qemu-img
+
+ifeq ($(call lor,$(CONFIG_SOFTMMU),$(CONFIG_TOOLS)),y)
+
 chardev-obj-y = chardev/
 
-#######################################################################
-# authz-obj-y is code used by both qemu system emulation and qemu-img
-
 authz-obj-y = authz/
 
-#######################################################################
-# block-obj-y is code used by both qemu system emulation and qemu-img
-
 block-obj-y = nbd/
 block-obj-y += block.o blockjob.o job.o
 block-obj-y += block/ scsi/
@@ -22,16 +21,12 @@ block-obj-$(CONFIG_REPLICATION) += replication.o
 
 block-obj-m = block/
 
-#######################################################################
-# crypto-obj-y is code used by both qemu system emulation and qemu-img
-
 crypto-obj-y = crypto/
 
-#######################################################################
-# io-obj-y is code used by both qemu system emulation and qemu-img
-
 io-obj-y = io/
 
+endif # CONFIG_SOFTMMU or CONFIG_TOOLS
+
 ######################################################################
 # Target independent part of system emulation. The long term path is to
 # suppress *all* target specific code in case of system emulation, i.e. a
-- 
2.21.1

[PATCH 4/4] Makefile: Remove unhelpful comment

[Philippe Mathieu-Daudé]

It is pointless to keep qapi/ object separate from the other
common-objects. Drop the comment.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 Makefile.objs | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/Makefile.objs b/Makefile.objs
index 395dd1e670..c6321d0465 100644
--- a/Makefile.objs
+++ b/Makefile.objs
@@ -71,11 +71,9 @@ qemu-seccomp.o-libs := $(SECCOMP_LIBS)
 
 common-obj-$(CONFIG_FDT) += device_tree.o
 
-######################################################################
-# qapi
-
 common-obj-y += qapi/
-endif
+
+endif # CONFIG_SOFTMMU
 
 #######################################################################
 # Target-independent parts used in system and user emulation
-- 
2.21.1

Re: [PATCH 1/4] configure: Do not build libfdt is not required

[Thomas Huth]

On 09/01/2020 16.39, Philippe Mathieu-Daudé wrote:
> We only require libfdt for system emulation, in a small set
> of architecture:
> 
> 4077  # fdt support is mandatory for at least some target architectures,
> 4078  # so insist on it if we're building those system emulators.
> 4079  fdt_required=no
> 4080  for target in $target_list; do
> 4081    case $target in
> 4082      aarch64*-softmmu|arm*-softmmu|ppc*-softmmu|microblaze*-softmmu|mips64el-softmmu|riscv*-softmmu)
> 4083        fdt_required=yes
> 
> Do not build libfdt if we did not manually specified --enable-fdt.

I suggest to add:

"... or have one of the platforms that require it in our target list."

> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  configure | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/configure b/configure
> index 0ce2c0354a..266a8386d1 100755
> --- a/configure
> +++ b/configure
> @@ -4092,6 +4092,8 @@ if test "$fdt_required" = "yes"; then
>        "targets which need it (by specifying a cut down --target-list)."
>    fi
>    fdt=yes
> +elif test "$fdt" != "yes" ; then
> +  fdt=no
>  fi
>  
>  if test "$fdt" != "no" ; then
> 

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH 2/4] Makefile: Clarify all the codebase requires qom/ objects

[Thomas Huth]

On 09/01/2020 16.39, Philippe Mathieu-Daudé wrote:
> QEMU user-mode also requires the qom/ objects, it is not only
> used by "system emulation and qemu-img". As we will use a big
> if() block, move it upper in the "Common libraries for tools
> and emulators" section.
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  Makefile.objs | 6 +-----
>  1 file changed, 1 insertion(+), 5 deletions(-)
> 
> diff --git a/Makefile.objs b/Makefile.objs
> index 7c1e50f9d6..5aae561984 100644
> --- a/Makefile.objs
> +++ b/Makefile.objs
> @@ -2,6 +2,7 @@
>  # Common libraries for tools and emulators
>  stub-obj-y = stubs/
>  util-obj-y = crypto/ util/ qobject/ qapi/
> +qom-obj-y = qom/
>  
>  chardev-obj-y = chardev/
>  
> @@ -26,11 +27,6 @@ block-obj-m = block/
>  
>  crypto-obj-y = crypto/
>  
> -#######################################################################
> -# qom-obj-y is code used by both qemu system emulation and qemu-img
> -
> -qom-obj-y = qom/
> -
>  #######################################################################
>  # io-obj-y is code used by both qemu system emulation and qemu-img
>  
> 

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH 3/4] Makefile: Restrict system emulation and tools objects

[Thomas Huth]

On 09/01/2020 16.39, Philippe Mathieu-Daudé wrote:
> Restrict all the system emulation and tools objects with a
> Makefile IF (CONFIG_SOFTMMU OR CONFIG_TOOLS) check.
> 
> Using the same description over and over is not very helpful.
> Use it once, just before the if() block.
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  Makefile.objs | 19 +++++++------------
>  1 file changed, 7 insertions(+), 12 deletions(-)
> 
> diff --git a/Makefile.objs b/Makefile.objs
> index 5aae561984..395dd1e670 100644
> --- a/Makefile.objs
> +++ b/Makefile.objs
> @@ -4,16 +4,15 @@ stub-obj-y = stubs/
>  util-obj-y = crypto/ util/ qobject/ qapi/
>  qom-obj-y = qom/
>  
> +#######################################################################
> +# code used by both qemu system emulation and qemu-img
> +
> +ifeq ($(call lor,$(CONFIG_SOFTMMU),$(CONFIG_TOOLS)),y)

That ",y" at the end looks wrong?

 Thomas

Re: [PATCH 4/4] Makefile: Remove unhelpful comment

[Thomas Huth]

On 09/01/2020 16.39, Philippe Mathieu-Daudé wrote:
> It is pointless to keep qapi/ object separate from the other
> common-objects. Drop the comment.
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  Makefile.objs | 6 ++----
>  1 file changed, 2 insertions(+), 4 deletions(-)
> 
> diff --git a/Makefile.objs b/Makefile.objs
> index 395dd1e670..c6321d0465 100644
> --- a/Makefile.objs
> +++ b/Makefile.objs
> @@ -71,11 +71,9 @@ qemu-seccomp.o-libs := $(SECCOMP_LIBS)
>  
>  common-obj-$(CONFIG_FDT) += device_tree.o
>  
> -######################################################################
> -# qapi
> -
>  common-obj-y += qapi/
> -endif
> +
> +endif # CONFIG_SOFTMMU
>  
>  #######################################################################
>  # Target-independent parts used in system and user emulation
> 

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH 3/4] Makefile: Restrict system emulation and tools objects

[Philippe Mathieu-Daudé]

[-- Attachment #1: Type: text/plain, Size: 1097 bytes --]

Le ven. 10 janv. 2020 09:18, Thomas Huth <thuth@redhat.com> a écrit :

> On 09/01/2020 16.39, Philippe Mathieu-Daudé wrote:
> > Restrict all the system emulation and tools objects with a
> > Makefile IF (CONFIG_SOFTMMU OR CONFIG_TOOLS) check.
> >
> > Using the same description over and over is not very helpful.
> > Use it once, just before the if() block.
> >
> > Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> > ---
> >  Makefile.objs | 19 +++++++------------
> >  1 file changed, 7 insertions(+), 12 deletions(-)
> >
> > diff --git a/Makefile.objs b/Makefile.objs
> > index 5aae561984..395dd1e670 100644
> > --- a/Makefile.objs
> > +++ b/Makefile.objs
> > @@ -4,16 +4,15 @@ stub-obj-y = stubs/
> >  util-obj-y = crypto/ util/ qobject/ qapi/
> >  qom-obj-y = qom/
> >
> > +#######################################################################
> > +# code used by both qemu system emulation and qemu-img
> > +
> > +ifeq ($(call lor,$(CONFIG_SOFTMMU),$(CONFIG_TOOLS)),y)
>
> That ",y" at the end looks wrong?
>

It's the result of the logical OR.

[-- Attachment #2: Type: text/html, Size: 1626 bytes --]

Re: [PATCH 3/4] Makefile: Restrict system emulation and tools objects

[Thomas Huth]

On 10/01/2020 09.25, Philippe Mathieu-Daudé wrote:
> Le ven. 10 janv. 2020 09:18, Thomas Huth <thuth@redhat.com
> <mailto:thuth@redhat.com>> a écrit :
> 
>     On 09/01/2020 16.39, Philippe Mathieu-Daudé wrote:
>     > Restrict all the system emulation and tools objects with a
>     > Makefile IF (CONFIG_SOFTMMU OR CONFIG_TOOLS) check.
>     >
>     > Using the same description over and over is not very helpful.
>     > Use it once, just before the if() block.
>     >
>     > Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com
>     <mailto:philmd@redhat.com>>
>     > ---
>     >  Makefile.objs | 19 +++++++------------
>     >  1 file changed, 7 insertions(+), 12 deletions(-)
>     >
>     > diff --git a/Makefile.objs b/Makefile.objs
>     > index 5aae561984..395dd1e670 100644
>     > --- a/Makefile.objs
>     > +++ b/Makefile.objs
>     > @@ -4,16 +4,15 @@ stub-obj-y = stubs/
>     >  util-obj-y = crypto/ util/ qobject/ qapi/
>     >  qom-obj-y = qom/
>     > 
>     >
>     +#######################################################################
>     > +# code used by both qemu system emulation and qemu-img
>     > +
>     > +ifeq ($(call lor,$(CONFIG_SOFTMMU),$(CONFIG_TOOLS)),y)
> 
>     That ",y" at the end looks wrong?
> 
> 
> It's the result of the logical OR.

-ENOTENOUGHCOFFEEYET

Your code is fine, of course.

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH 0/4] buildsys: Build quicker (mostly tools and linux-user)

[Laurent Vivier]

Le 09/01/2020 à 16:39, Philippe Mathieu-Daudé a écrit :
> In some configuration (linux-user, tools) we can ignore building
> various objects (and the libfdt).
> 
> Philippe Mathieu-Daudé (4):
>   configure: Do not build libfdt is not required
>   Makefile: Clarify all the codebase requires qom/ objects
>   Makefile: Restrict system emulation and tools objects
>   Makefile: Remove unhelpful comment
> 
>  configure     |  2 ++
>  Makefile.objs | 31 ++++++++++---------------------
>  2 files changed, 12 insertions(+), 21 deletions(-)
> 

Did you test this with all the combinations of --[enable|disable]-tools,
--[enable|disable]-user and --[enable|disable]-system?

Thanks,
Laurent

Re: [PATCH 0/4] buildsys: Build quicker (mostly tools and linux-user)

[Philippe Mathieu-Daudé]

[-- Attachment #1: Type: text/plain, Size: 944 bytes --]

Le ven. 10 janv. 2020 09:36, Laurent Vivier <laurent@vivier.eu> a écrit :

> Le 09/01/2020 à 16:39, Philippe Mathieu-Daudé a écrit :
> > In some configuration (linux-user, tools) we can ignore building
> > various objects (and the libfdt).
> >
> > Philippe Mathieu-Daudé (4):
> >   configure: Do not build libfdt is not required
> >   Makefile: Clarify all the codebase requires qom/ objects
> >   Makefile: Restrict system emulation and tools objects
> >   Makefile: Remove unhelpful comment
> >
> >  configure     |  2 ++
> >  Makefile.objs | 31 ++++++++++---------------------
> >  2 files changed, 12 insertions(+), 21 deletions(-)
> >
>
> Did you test this with all the combinations of --[enable|disable]-tools,
> --[enable|disable]-user and --[enable|disable]-system
>

I tested 12 of 27 because I thought some sets might overlap but I might
have missed something, what combination is giving you problem?

>

[-- Attachment #2: Type: text/html, Size: 1509 bytes --]

Re: [PATCH 1/4] configure: Do not build libfdt is not required

[Alistair Francis]

On Thu, Jan 9, 2020 at 11:40 PM Philippe Mathieu-Daudé
<philmd@redhat.com> wrote:
>
> We only require libfdt for system emulation, in a small set
> of architecture:
>
> 4077  # fdt support is mandatory for at least some target architectures,
> 4078  # so insist on it if we're building those system emulators.
> 4079  fdt_required=no
> 4080  for target in $target_list; do
> 4081    case $target in
> 4082      aarch64*-softmmu|arm*-softmmu|ppc*-softmmu|microblaze*-softmmu|mips64el-softmmu|riscv*-softmmu)
> 4083        fdt_required=yes
>
> Do not build libfdt if we did not manually specified --enable-fdt.
>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>

Reviewed-by: Alistair Francis <alistair.francis@wdc.com>

Alistair

> ---
>  configure | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/configure b/configure
> index 0ce2c0354a..266a8386d1 100755
> --- a/configure
> +++ b/configure
> @@ -4092,6 +4092,8 @@ if test "$fdt_required" = "yes"; then
>        "targets which need it (by specifying a cut down --target-list)."
>    fi
>    fdt=yes
> +elif test "$fdt" != "yes" ; then
> +  fdt=no
>  fi
>
>  if test "$fdt" != "no" ; then
> --
> 2.21.1
>
>

Re: [PATCH 0/4] buildsys: Build quicker (mostly tools and linux-user)

[Laurent Vivier]

Le 10/01/2020 à 10:17, Philippe Mathieu-Daudé a écrit :
> Le ven. 10 janv. 2020 09:36, Laurent Vivier <laurent@vivier.eu
> <mailto:laurent@vivier.eu>> a écrit :
> 
>     Le 09/01/2020 à 16:39, Philippe Mathieu-Daudé a écrit :
>     > In some configuration (linux-user, tools) we can ignore building
>     > various objects (and the libfdt).
>     >
>     > Philippe Mathieu-Daudé (4):
>     >   configure: Do not build libfdt is not required
>     >   Makefile: Clarify all the codebase requires qom/ objects
>     >   Makefile: Restrict system emulation and tools objects
>     >   Makefile: Remove unhelpful comment
>     >
>     >  configure     |  2 ++
>     >  Makefile.objs | 31 ++++++++++---------------------
>     >  2 files changed, 12 insertions(+), 21 deletions(-)
>     >
> 
>     Did you test this with all the combinations of --[enable|disable]-tools,
>     --[enable|disable]-user and --[enable|disable]-system
> 
> 
> I tested 12 of 27 because I thought some sets might overlap but I might
> have missed something, what combination is giving you problem? 
> 

I didn't test your series, but I did this kind of change in the past and
sometime enabling tools without enabling softmmu can show missing
objects at build time, or you can also see if tools are built with
softmmu while tools are disabled.

I used to test with something like

for user in enable disable; do
    for tools in enable disable; do
        for system in enable disable; do
            rm -fr build
            mkdir build && \
            (cd build && \
             ../configure --$user-user \
                          --$system-system \
                          --$tools-tools && \
             make || exit)
        done
    done
done

Thanks,
Laurent

Re: [PATCH 0/4] buildsys: Build quicker (mostly tools and linux-user)

[Philippe Mathieu-Daudé]

On 1/10/20 3:03 PM, Laurent Vivier wrote:
> Le 10/01/2020 à 10:17, Philippe Mathieu-Daudé a écrit :
>> Le ven. 10 janv. 2020 09:36, Laurent Vivier <laurent@vivier.eu
>> <mailto:laurent@vivier.eu>> a écrit :
>>
>>      Le 09/01/2020 à 16:39, Philippe Mathieu-Daudé a écrit :
>>      > In some configuration (linux-user, tools) we can ignore building
>>      > various objects (and the libfdt).
>>      >
>>      > Philippe Mathieu-Daudé (4):
>>      >   configure: Do not build libfdt is not required
>>      >   Makefile: Clarify all the codebase requires qom/ objects
>>      >   Makefile: Restrict system emulation and tools objects
>>      >   Makefile: Remove unhelpful comment
>>      >
>>      >  configure     |  2 ++
>>      >  Makefile.objs | 31 ++++++++++---------------------
>>      >  2 files changed, 12 insertions(+), 21 deletions(-)
>>      >
>>
>>      Did you test this with all the combinations of --[enable|disable]-tools,
>>      --[enable|disable]-user and --[enable|disable]-system
>>
>>
>> I tested 12 of 27 because I thought some sets might overlap but I might
>> have missed something, what combination is giving you problem?
>>
> 
> I didn't test your series, but I did this kind of change in the past and
> sometime enabling tools without enabling softmmu can show missing
> objects at build time, or you can also see if tools are built with
> softmmu while tools are disabled.
> 
> I used to test with something like
> 
> for user in enable disable; do
>      for tools in enable disable; do
>          for system in enable disable; do
>              rm -fr build
>              mkdir build && \
>              (cd build && \
>               ../configure --$user-user \
>                            --$system-system \
>                            --$tools-tools && \
>               make || exit)
>          done
>      done
> done

I tested all these 8 combinations, no problem.