All of lore.kernel.org
 help / color / mirror / Atom feed
* [Qemu-devel] [PATCH RFC 00/21] Modularize generated QAPI code
@ 2018-02-02 13:03 Markus Armbruster
  2018-02-02 13:03 ` [Qemu-devel] [PATCH RFC 01/21] qapi: Streamline boilerplate comment generation Markus Armbruster
                   ` (28 more replies)
  0 siblings, 29 replies; 87+ messages in thread
From: Markus Armbruster @ 2018-02-02 13:03 UTC (permalink / raw)
  To: qemu-devel; +Cc: marcandre.lureau, eblake, mdroth

Our qapi-schema.json is composed of modules connected by include
directives, but the generated code is monolithic all the same: one
qapi-types.h with all the types, one qapi-visit.h with all the
visitors, and so forth.  These monolithic headers get included all
over the place.  In my "build everyhing" tree, adding a QAPI type
recompiles about 4500 out of 4800 objects.

Nobody would write such monolithic headers by hand.  It stands to
reason that one shouldn't generate them, either.

This series' basic idea is to split up generated headers to mirror the
schema's modular structure: one header per module.  That way, you can
include just what you need.

The series is RFC for a number of reasons:

* The split is implemented only for qapi-types.h.  That one should
  provide the biggest benefits, though.

* There's a bit of code duplication.

* I haven't re-read my patches, yet.

Even in this incomplete state, the compile-time improvements can be
massive.  Before this series, any QAPI schema change recompiles some
4500 out of 4800 objects in my "build everything" tree.  Afterwards,
adding a type to qapi/migration.json recompiles less than 400, adding
a QMP event recompiles less than 200, and a documentation change no
longer recompiles anything.

Related: Marc-André's 'unit' pragma proposal.  That's a different way
to split off parts of the generated code, motivated by the desire to
use poisoned identifiers such as TARGET_I386.  I noted in my review of
v3 that I "can either accept it, or come up with a better solution."
This is my attempt at a better solution.  It's a bit more ambitious,
and thus more useful (I hope).  The pragma has one theoretical
advantage, though: you can modularize the generated output in
different ways than the input.  The patches using don't do that,
however.

Based-on: <20180201111846.21846-1-armbru@redhat.com>
[PATCH v3 00/19] Clean up includes to reduce compile time

Markus Armbruster (21):
  qapi: Streamline boilerplate comment generation
  qapi: Generate up-to-date copyright notice
  qapi: New classes QAPIGenC, QAPIGenH, QAPIGenDoc
  qapi: Reduce use of global variables in generators some
  qapi: Turn generators into modules
  qapi-gen: New common driver for code and doc generators
  qapi: Move parse_command_line() next to its only use
  qapi: Touch generated files only when they change
  qapi: Don't absolutize include file name in error messages
  qapi/common: Eliminate QAPISchema.exprs
  qapi: Lift error reporting from QAPISchema.__init__() to callers
  qapi: Concentrate QAPISchemaParser.exprs updates in .__init__()
  qapi: Record 'include' directives in parse tree
  qapi: Generate in source order
  qapi: Record 'include' directives in intermediate representation
  qapi/types qapi/visit: Make visitors use QAPIGen more
  qapi/types qapi/visit: Generate built-in stuff into separate files
  qapi/common: Fix guardname() for funny filenames
  qapi/types: Generate separate .h, .c for each module
  Include less of qapi-types.h
  qapi: Empty out qapi-schema.json

 Makefile                                           |  131 +-
 Makefile.objs                                      |   20 +-
 crypto/cipherpriv.h                                |    2 +-
 hw/block/block.c                                   |    1 +
 hw/block/hd-geometry.c                             |    1 +
 hw/net/rocker/rocker_fp.c                          |    2 +-
 include/block/block.h                              |    2 +-
 include/block/dirty-bitmap.h                       |    2 +-
 include/chardev/char.h                             |    1 +
 include/crypto/cipher.h                            |    2 +-
 include/crypto/hash.h                              |    2 +-
 include/crypto/hmac.h                              |    2 +-
 include/crypto/secret.h                            |    1 +
 include/crypto/tlscreds.h                          |    1 +
 include/hw/block/block.h                           |    2 +-
 include/hw/block/fdc.h                             |    2 +-
 include/hw/ppc/spapr_drc.h                         |    1 +
 include/hw/qdev-properties.h                       |    2 +
 include/io/dns-resolver.h                          |    1 +
 include/migration/colo.h                           |    2 +-
 include/migration/failover.h                       |    2 +-
 include/migration/global_state.h                   |    1 +
 include/monitor/monitor.h                          |    1 +
 include/net/filter.h                               |    1 +
 include/net/net.h                                  |    2 +-
 include/qapi/error.h                               |    2 +-
 include/qapi/qmp/qobject.h                         |    2 +-
 include/qapi/visitor.h                             |    2 +-
 include/qemu/sockets.h                             |    2 +-
 include/qemu/throttle.h                            |    2 +-
 include/qom/cpu.h                                  |    1 +
 include/qom/object.h                               |    2 +-
 include/sysemu/arch_init.h                         |    2 +-
 include/sysemu/balloon.h                           |    2 +-
 include/sysemu/dump.h                              |    2 +
 include/sysemu/hostmem.h                           |    1 +
 include/sysemu/replay.h                            |    2 +
 include/sysemu/sysemu.h                            |    1 +
 include/sysemu/tpm.h                               |    1 +
 include/sysemu/watchdog.h                          |    2 +-
 include/ui/input.h                                 |    2 +-
 migration/migration.h                              |    1 +
 migration/ram.h                                    |    2 +-
 net/tap_int.h                                      |    2 +-
 qapi-schema.json                                   | 3098 +-------------------
 qapi/misc.json                                     | 3090 +++++++++++++++++++
 qapi/run-state.json                                |   10 +
 replication.h                                      |    1 +
 scripts/qapi-gen.py                                |   95 +
 scripts/qapi/__init__.py                           |    0
 scripts/{qapi-commands.py => qapi/commands.py}     |   98 +-
 scripts/{qapi.py => qapi/common.py}                |  290 +-
 scripts/{qapi2texi.py => qapi/doc.py}              |   28 +-
 scripts/{qapi-event.py => qapi/events.py}          |  101 +-
 scripts/{qapi-introspect.py => qapi/introspect.py} |   87 +-
 scripts/{qapi-types.py => qapi/types.py}           |  218 +-
 scripts/{qapi-visit.py => qapi/visit.py}           |  208 +-
 tests/Makefile.include                             |   56 +-
 tests/qapi-schema/comments.out                     |    3 +-
 tests/qapi-schema/doc-bad-section.out              |    5 +-
 tests/qapi-schema/doc-good.out                     |   33 +-
 tests/qapi-schema/doc-good.texi                    |    3 +-
 tests/qapi-schema/empty.out                        |    2 +-
 tests/qapi-schema/event-case.out                   |    3 +-
 tests/qapi-schema/ident-with-escape.out            |    7 +-
 tests/qapi-schema/include-no-file.err              |    2 +-
 tests/qapi-schema/include-relpath.out              |    7 +-
 tests/qapi-schema/include-repetition.out           |   12 +-
 tests/qapi-schema/include-simple.out               |    5 +-
 tests/qapi-schema/indented-expr.out                |    3 +-
 tests/qapi-schema/qapi-schema-test.out             |  321 +-
 tests/qapi-schema/test-qapi.py                     |   17 +-
 ui/vnc.h                                           |    1 +
 73 files changed, 4040 insertions(+), 3984 deletions(-)
 create mode 100644 qapi/misc.json
 create mode 100755 scripts/qapi-gen.py
 create mode 100644 scripts/qapi/__init__.py
 rename scripts/{qapi-commands.py => qapi/commands.py} (78%)
 rename scripts/{qapi.py => qapi/common.py} (93%)
 rename scripts/{qapi2texi.py => qapi/doc.py} (93%)
 mode change 100755 => 100644
 rename scripts/{qapi-event.py => qapi/events.py} (69%)
 rename scripts/{qapi-introspect.py => qapi/introspect.py} (78%)
 rename scripts/{qapi-types.py => qapi/types.py} (54%)
 rename scripts/{qapi-visit.py => qapi/visit.py} (66%)

-- 
2.13.6

^ permalink raw reply	[flat|nested] 87+ messages in thread

end of thread, other threads:[~2018-02-08  9:55 UTC | newest]

Thread overview: 87+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-02-02 13:03 [Qemu-devel] [PATCH RFC 00/21] Modularize generated QAPI code Markus Armbruster
2018-02-02 13:03 ` [Qemu-devel] [PATCH RFC 01/21] qapi: Streamline boilerplate comment generation Markus Armbruster
2018-02-02 15:08   ` Eric Blake
2018-02-03  8:45     ` Markus Armbruster
2018-02-05 13:44   ` Marc-Andre Lureau
2018-02-02 13:03 ` [Qemu-devel] [PATCH RFC 02/21] qapi: Generate up-to-date copyright notice Markus Armbruster
2018-02-02 15:47   ` Eric Blake
2018-02-03  8:48     ` Markus Armbruster
2018-02-05 13:44   ` Marc-Andre Lureau
2018-02-05 15:28     ` Markus Armbruster
2018-02-05 15:45     ` Eric Blake
2018-02-02 13:03 ` [Qemu-devel] [PATCH RFC 03/21] qapi: New classes QAPIGenC, QAPIGenH, QAPIGenDoc Markus Armbruster
2018-02-02 15:59   ` Eric Blake
2018-02-03  8:49     ` Markus Armbruster
2018-02-05 15:46       ` Eric Blake
2018-02-06  7:28         ` Markus Armbruster
2018-02-05 13:44   ` Marc-Andre Lureau
2018-02-05 15:34     ` Markus Armbruster
2018-02-02 13:03 ` [Qemu-devel] [PATCH RFC 04/21] qapi: Reduce use of global variables in generators some Markus Armbruster
2018-02-02 16:03   ` Eric Blake
2018-02-05 13:44   ` Marc-Andre Lureau
2018-02-02 13:03 ` [Qemu-devel] [PATCH RFC 05/21] qapi: Turn generators into modules Markus Armbruster
2018-02-02 16:47   ` Eric Blake
2018-02-05 13:44   ` Marc-Andre Lureau
2018-02-02 13:03 ` [Qemu-devel] [PATCH RFC 06/21] qapi-gen: New common driver for code and doc generators Markus Armbruster
2018-02-02 19:27   ` Eric Blake
2018-02-03  9:03     ` Markus Armbruster
2018-02-05 15:52       ` Eric Blake
2018-02-06  7:45         ` Markus Armbruster
2018-02-06 14:56           ` Eric Blake
2018-02-05 13:44   ` Marc-Andre Lureau
2018-02-05 15:36     ` Markus Armbruster
2018-02-08  9:55       ` Markus Armbruster
2018-02-02 13:03 ` [Qemu-devel] [PATCH RFC 07/21] qapi: Move parse_command_line() next to its only use Markus Armbruster
2018-02-02 19:29   ` Eric Blake
2018-02-05 13:45   ` Marc-Andre Lureau
2018-02-02 13:03 ` [Qemu-devel] [PATCH RFC 08/21] qapi: Touch generated files only when they change Markus Armbruster
2018-02-02 19:42   ` Eric Blake
2018-02-03  9:05     ` Markus Armbruster
2018-02-02 13:03 ` [Qemu-devel] [PATCH RFC 09/21] qapi: Don't absolutize include file name in error messages Markus Armbruster
2018-02-02 20:22   ` Eric Blake
2018-02-03  9:08     ` Markus Armbruster
2018-02-05 15:55       ` Eric Blake
2018-02-06  7:49         ` Markus Armbruster
2018-02-06 15:06           ` Eric Blake
2018-02-05 13:46   ` Marc-Andre Lureau
2018-02-02 13:03 ` [Qemu-devel] [PATCH RFC 10/21] qapi/common: Eliminate QAPISchema.exprs Markus Armbruster
2018-02-02 22:02   ` Eric Blake
2018-02-05 13:45   ` Marc-Andre Lureau
2018-02-02 13:03 ` [Qemu-devel] [PATCH RFC 11/21] qapi: Lift error reporting from QAPISchema.__init__() to callers Markus Armbruster
2018-02-05 13:45   ` Marc-Andre Lureau
2018-02-02 13:03 ` [Qemu-devel] [PATCH RFC 12/21] qapi: Concentrate QAPISchemaParser.exprs updates in .__init__() Markus Armbruster
2018-02-05 13:45   ` Marc-Andre Lureau
2018-02-05 14:26     ` Markus Armbruster
2018-02-02 13:03 ` [Qemu-devel] [PATCH RFC 13/21] qapi: Record 'include' directives in parse tree Markus Armbruster
2018-02-05 13:45   ` Marc-Andre Lureau
2018-02-02 13:03 ` [Qemu-devel] [PATCH RFC 14/21] qapi: Generate in source order Markus Armbruster
2018-02-05 13:45   ` Marc-Andre Lureau
2018-02-05 14:33     ` Markus Armbruster
2018-02-05 14:40       ` Marc-Andre Lureau
2018-02-06 10:33       ` Markus Armbruster
2018-02-02 13:03 ` [Qemu-devel] [PATCH RFC 15/21] qapi: Record 'include' directives in intermediate representation Markus Armbruster
2018-02-05 13:45   ` Marc-Andre Lureau
2018-02-02 13:03 ` [Qemu-devel] [PATCH RFC 16/21] qapi/types qapi/visit: Make visitors use QAPIGen more Markus Armbruster
2018-02-05 13:46   ` Marc-Andre Lureau
2018-02-02 13:03 ` [Qemu-devel] [PATCH RFC 17/21] qapi/types qapi/visit: Generate built-in stuff into separate files Markus Armbruster
2018-02-05 13:46   ` Marc-Andre Lureau
2018-02-06 20:54   ` Eric Blake
2018-02-02 13:03 ` [Qemu-devel] [PATCH RFC 18/21] qapi/common: Fix guardname() for funny filenames Markus Armbruster
2018-02-05 13:47   ` Marc-Andre Lureau
2018-02-06 21:00   ` Eric Blake
2018-02-02 13:03 ` [Qemu-devel] [PATCH RFC 19/21] qapi/types: Generate separate .h, .c for each module Markus Armbruster
2018-02-05 13:58   ` Marc-Andre Lureau
2018-02-02 13:03 ` [Qemu-devel] [PATCH RFC 20/21] Include less of qapi-types.h Markus Armbruster
2018-02-05 13:46   ` Marc-Andre Lureau
2018-02-05 14:53     ` Markus Armbruster
2018-02-02 13:03 ` [Qemu-devel] [PATCH RFC 21/21] qapi: Empty out qapi-schema.json Markus Armbruster
2018-02-05 13:45   ` Marc-Andre Lureau
2018-02-02 14:52 ` [Qemu-devel] [PATCH RFC 00/21] Modularize generated QAPI code Markus Armbruster
2018-02-02 16:44 ` Daniel P. Berrangé
2018-02-03  9:18   ` Markus Armbruster
2018-02-02 18:36 ` no-reply
2018-02-02 19:33 ` no-reply
2018-02-02 20:11 ` no-reply
2018-02-02 21:13 ` no-reply
2018-02-02 22:14 ` no-reply
2018-02-03 11:30 ` Markus Armbruster

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.