Re: [Qemu-devel] [PATCH v11 21/28] qapi: Convert qtype_code into qapi enum type

[Markus Armbruster <armbru@redhat.com>]

Eric Blake <eblake@redhat.com> writes:

> On 11/11/2015 09:42 AM, Markus Armbruster wrote:
>> Eric Blake <eblake@redhat.com> writes:
>> 
>>> What's more meta than using qapi to define qapi? :)
>>>
>>> Convert qtype_code into a full-fledged[*] builtin qapi enum type,
>>> so that a subsequent patch can then use it as the discriminator
>>> type of qapi alternate types.  Doing so is easiest when renaming
>>> it to qapi conventions, as QTypeCode.
>> 
>> Out of curiosity: why does the rename make the conversion easier?
>
> It guarantees I found all affected instances.  (Although I guess the
> rename could be split to a separate patch from making it builtin).

Well, you have to find them only because you rename, don't you?

> It makes sure that if we later tighten rules about naming, we won't have
> to whitelist 'qtype_code' as an anomaly to our conventions.

Good point.

>> If we rename anyway, what about renaming to QType?  Hmm, we burned that
>> on a struct we use only internally in qobject/.  Oh well.
>
> Internal structs are often easy to rename.  So if we want to avoid the
> need for 'prefix', I could certainly try to achieve that (move internal
> QType out of the way, then rename qtype_code to QType, then make QType
> the builtin).  Looks like this one patch just became three :)

Not sure it's worth the bother; the patch is okay as it is.

QType is overkill.  Instead of

    typedef struct QType {
        qtype_code code;
        void (*destroy)(struct QObject *);
    } QType;

    typedef struct QObject {
        const QType *type;
        size_t refcnt;
    } QObject;

we could simply have

    typedef struct QObject {
        QTypeCode type;
        size_t refcnt;
    } QObject;

with an array mapping QTypeCode to destroy methods.  We're not going to
define additional types at run time.

Perhaps such a change would be actually worth the bother.

>>>                                        Fortunately, there are not
>>> many places in the tree that were actually spelling the type name
>>> out, and the judicious use of 'prefix' in the qapi defintion
>> 
>> definition
>
> I've got to quit coding late at night - my rate of typos increases :)
>
>>> +++ b/docs/qapi-code-gen.txt
>>> @@ -163,6 +163,7 @@ The following types are predefined, and map to C as follows:
>>>                         accepts size suffixes
>>>    bool      bool       JSON true or false
>>>    any       QObject *  any JSON value
>>> +  QTypeCode QTypeCode  JSON string of enum QTypeCode values
>> 
>> QTypeCode is currently used only internally, so the JSON values don't
>> matter.  I don't expect that to change.  However, we either enforce
>> internal use somehow, or document the JSON values.  Documenting them is
>> easier.
>> 
>> In short, your patch is fine.
>> 
>
>>> -
>>> -struct QObject;
>>> +#include "qapi-types.h"
>>>
>>>  typedef struct QType {
>>> -    qtype_code code;
>>> +    QTypeCode code;
>>>      void (*destroy)(struct QObject *);
>>>  } QType;
>>>
>>    typedef struct QObject {
>>        const QType *type;
>>        size_t refcnt;
>>    } QObject;
>> 
>> Note: typedef name QObject still defined here.
>
> Oh, I see what you're saying. Since qapi-types.h now has a forward
> declaration of the QObject typedef, this could be changed to just
>
> struct QObject {
> ...
> };
>
>>> +++ b/scripts/qapi-types.py
>>> @@ -233,8 +233,14 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
>>>          self.defn += gen_type_cleanup(name)
>>>
>>>      def visit_enum_type(self, name, info, values, prefix):
>>> -        self._fwdecl += gen_enum(name, values, prefix)
>>> -        self._fwdefn += gen_enum_lookup(name, values, prefix)
>>> +        # Special case for our lone builtin enum type
>>> +        if name == 'QTypeCode':
>> 
>> Would "if not info" work?  Same in qapi-visit.py below.
>
> Feels a bit hacky, since we just recently added is_implicit() to hide
> (and then change) the 'if not info' check on objects.  Maybe an accessor
> is_builtin() makes more sense?  But yes, same approach to both client files.

QAPISchemaEntity methods like is_implicit() or a new is_builtin() can't
work here, because we lack the entity.

We have one in visit_needed(), and we use its is_implicit() to skip
implicit object types.  We could use entity.is_builtin() to skip (some)
builtins, and handle them elsewhere, but that doesn't feel like an
improvement over your code.

Let's take a step back and reconsider how we do builtins.

>> +            self._btin += gen_enum(name, values, prefix)
>> +            if do_builtins:
>> +                self.defn += gen_enum_lookup(name, values, prefix)
>> +        else:
>> +            self._fwdecl += gen_enum(name, values, prefix)
>> +            self._fwdefn += gen_enum_lookup(name, values, prefix)
>>
>>      def visit_array_type(self, name, info, element_type):
>>          if isinstance(element_type, QAPISchemaBuiltinType):

Linking generated code from multiple schemata that share names may fail,
because multiple definitions of the same external symbol exist.

Example: two schemata both define enum BadIdea.  Both generate const
char *BadIdea_lookup[] = { ... }, and we end up with two global symbols
BadIdea_lookup.

Solution: don't do that then.  Easy enough, except *all* schemata share
the builtin symbols!  Solution:

1. For declarations, use ifdeffery to make the compiler ignore all but
   the first copy it encounters,

2. For definitions, make the programmer pick one schema to generate the
   definitions, and run qapi-types.py and qapi-visit.py with -b.

In generator code, this looks like

    self._btin += ... declarations ...
    if do_builtins:
        self.defn += ... definitions ...

instead of the normal

    self.decl += ... declarations ...
    self.defn += ... declarations ...

(or the same with ._fwdecl, ._fwdefn, doesn't matter).

This is why you need to know whether the enum is builtin in
.visit_enum_type() above.

The builtin definitions are emitted into a suitable #ifdef block by
bracketing this code with an initial

    self._btin = guardstart('QAPI_TYPES_BUILTIN')

and a final

    self._btin += guardend('QAPI_TYPES_BUILTIN')
    self.decl = self._btin + self.decl
    self._btin = None

Here's an alternative solution that permits slightly code simpler
generator code, and thus avoids the need to know:

* Generate code for builtins exactly the same as for any other entities,
  i.e. get rid of self._btin and the ifdeffery.

* If the program links just one generated schema, this just works.

* If the program links multiple generated schemata, the programmer has
  to ensure their definitions get generated just once, and their
  declarations are available everywhere anyway.  Straightforward method:

  - The programmer suppresses builtins *completely* for *all* schemata.
    The obvious way to suppress them is to filter them out in
    visit_needed().

  - Instead, he generates them once for the *empty* schema, with a
    well-known --prefix.

  - Suppressing builtins generates a suitable #include for the
    well-known .h with the builtin declarations.

  - Additionally link the .c containing the builtin definitions.

Alternatively, trade some ease-of-use for the single schema case for
ease-of-use for the multiple schemata case and fewer cases:

* The generators either generate for a schema, or they generate builtins.

* When they generate builtins, they always use well-known file names.

* When they generate for a schema, they always generate the #include for
  the well-known builtin .h.  They never generate builtins.

>>> -#include "qapi/qmp/qobject.h"
>>> +
>>> +typedef struct QObject QObject;
>> 
>> Typedef name QObject now also defined here.  GCC accepts this silently
>> without -Wpedantic, but other compilers might not.  Whether we care for
>> such compilers or not, defining things in exactly one place is neater.
>> 
>> Possible fixes:
>> 
>> * Drop the typedef from qobject.h
>> 
>> * Don't add it to qapi-types.h, and use struct QObject there
>> 
>
> I favor dropping the second typedef.

Your choice.

>>> +++ b/scripts/qapi.py
>>> @@ -33,7 +33,7 @@ builtin_types = {
>>>      'uint32':   'QTYPE_QINT',
>>>      'uint64':   'QTYPE_QINT',
>>>      'size':     'QTYPE_QINT',
>>> -    'any':      None,           # any qtype_code possible, actually
>>> +    'any':      None,           # any QTypeCode possible, actually
>>>  }
>>>
>> 
>> Should we list QTypeCode here?
>
> Yeah, probably.  This array is only used by the ad hoc parser, and may
> disappear later as we move more into check(), but we should be
> consistent in the meantime.
>
>> 
>>>  # Whitelist of commands allowed to return a non-dictionary
>>> @@ -1243,6 +1243,11 @@ class QAPISchema(object):
>>>          self.the_empty_object_type = QAPISchemaObjectType(':empty', None, None,
>>>                                                            [], None)
>>>          self._def_entity(self.the_empty_object_type)
>>> +        self._def_entity(QAPISchemaEnumType('QTypeCode', None,
>>> +                                            ['none', 'qnull', 'qint',
>>> +                                             'qstring', 'qdict', 'qlist',
>>> +                                             'qfloat', 'qbool'],
>>> +                                            'QTYPE'))
>>>
>>>      def _make_implicit_enum_type(self, name, info, values):
>>>          name = name + 'Kind'   # Use namespace reserved by add_name()
>> [Trivial changes to expected test output snipped]
>
> I debated about hacking tests/qapi-schema/test-qapi.py to omit QTypeCode
> (the way we already omit builtin types and things like 'intList'), for
> less churn in the .out files.  I can go either way, if you have a
> preference.

Omit them only if it's trivial.

I guess it would be trivial if we adopted the alternative way to do
builtins I sketched above.