[Qemu-devel] [PATCH 0/5] decodetree enhancements

[Qemu-devel] [PATCH 0/5] decodetree enhancements

[Richard Henderson]

Two of these have been sitting around on a branch for too long.
Then, split out the documentation to a new file.

Finally, support overlapping patterns.  To demonstrate, I convert two
decodetree users: hppa and riscv, rfc patches appended to the series.


r~


Philippe Mathieu-Daudé (2):
  MAINTAINERS: Add scripts/decodetree.py to the TCG section
  decodetree: Ensure build_tree does not include values outside insnmask

Richard Henderson (3):
  decodetree: Move documentation to docs/decodetree.rst
  decodetree: Do not unconditionaly return from Pattern.output_code
  decodetree: Allow grouping of overlapping patterns

 MAINTAINERS           |   2 +
 docs/decodetree.rst   | 214 +++++++++++++++++++++++++++++++
 scripts/decodetree.py | 287 ++++++++++++++++++++----------------------
 3 files changed, 355 insertions(+), 148 deletions(-)
 create mode 100644 docs/decodetree.rst

-- 
2.17.2

[Qemu-devel] [PATCH 1/5] MAINTAINERS: Add scripts/decodetree.py to the TCG section

[Richard Henderson]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20181110211313.6922-2-f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 MAINTAINERS | 1 +
 1 file changed, 1 insertion(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index b1d786cfd8..ad007748b9 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -117,6 +117,7 @@ F: cpus.c
 F: exec.c
 F: accel/tcg/
 F: accel/stubs/tcg-stub.c
+F: scripts/decodetree.py
 F: include/exec/cpu*.h
 F: include/exec/exec-all.h
 F: include/exec/helper*.h
-- 
2.17.2

[Qemu-devel] [PATCH 2/5] decodetree: Move documentation to docs/decodetree.rst

[Richard Henderson]

One great big block comment isn't the best way to document
the syntax of a language.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 MAINTAINERS           |   1 +
 docs/decodetree.rst   | 156 ++++++++++++++++++++++++++++++++++++++++++
 scripts/decodetree.py | 134 +-----------------------------------
 3 files changed, 158 insertions(+), 133 deletions(-)
 create mode 100644 docs/decodetree.rst

diff --git a/MAINTAINERS b/MAINTAINERS
index ad007748b9..fc7cddb873 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -118,6 +118,7 @@ F: exec.c
 F: accel/tcg/
 F: accel/stubs/tcg-stub.c
 F: scripts/decodetree.py
+F: docs/decodetree.rst
 F: include/exec/cpu*.h
 F: include/exec/exec-all.h
 F: include/exec/helper*.h
diff --git a/docs/decodetree.rst b/docs/decodetree.rst
new file mode 100644
index 0000000000..d9be30b2db
--- /dev/null
+++ b/docs/decodetree.rst
@@ -0,0 +1,156 @@
+========================
+Decodetree Specification
+========================
+
+A *decodetree* is built from instruction *patterns*.  A pattern may
+represent a single architectural instruction or a group of same, depending
+on what is convenient for further processing.
+
+Each pattern has both *fixedbits* and *fixedmask*, the combination of which
+describes the condition under which the pattern is matched::
+
+  (insn & fixedmask) == fixedbits
+
+Each pattern may have *fields*, which are extracted from the insn and
+passed along to the translator.  Examples of such are registers,
+immediates, and sub-opcodes.
+
+In support of patterns, one may declare *fields*, *argument sets*, and
+*formats*, each of which may be re-used to simplify further definitions.
+
+Fields
+======
+
+Syntax::
+
+  field_def     := '%' identifier ( unnamed_field )+ ( !function=identifier )?
+  unnamed_field := number ':' ( 's' ) number
+
+For *unnamed_field*, the first number is the least-significant bit position
+of the field and the second number is the length of the field.  If the 's' is
+present, the field is considered signed.  If multiple ``unnamed_fields`` are
+present, they are concatenated.  In this way one can define disjoint fields.
+
+If ``!function`` is specified, the concatenated result is passed through the
+named function, taking and returning an integral value.
+
+FIXME: the fields of the structure into which this result will be stored
+is restricted to ``int``.  Which means that we cannot expand 64-bit items.
+
+Field examples:
+
++---------------------------+---------------------------------------------+
+| Input                     | Generated code                              |
++===========================+=============================================+
+| %disp   0:s16             | sextract(i, 0, 16)                          |
++---------------------------+---------------------------------------------+
+| %imm9   16:6 10:3         | extract(i, 16, 6) << 3 | extract(i, 10, 3)  |
++---------------------------+---------------------------------------------+
+| %disp12 0:s1 1:1 2:10     | sextract(i, 0, 1) << 11 |                   |
+|                           |    extract(i, 1, 1) << 10 |                 |
+|                           |    extract(i, 2, 10)                        |
++---------------------------+---------------------------------------------+
+| %shimm8 5:s8 13:1         | expand_shimm8(sextract(i, 5, 8) << 1 |      |
+|   !function=expand_shimm8 |               extract(i, 13, 1))            |
++---------------------------+---------------------------------------------+
+
+Argument Sets
+=============
+
+Syntax::
+
+  args_def    := '&' identifier ( args_elt )+ ( !extern )?
+  args_elt    := identifier
+
+Each *args_elt* defines an argument within the argument set.
+Each argument set will be rendered as a C structure "arg_$name"
+with each of the fields being one of the member arguments.
+
+If ``!extern`` is specified, the backing structure is assumed
+to have been already declared, typically via a second decoder.
+
+Argument set examples::
+
+  &reg3       ra rb rc
+  &loadstore  reg base offset
+
+
+Formats
+=======
+
+Syntax::
+
+  fmt_def      := '@' identifier ( fmt_elt )+
+  fmt_elt      := fixedbit_elt | field_elt | field_ref | args_ref
+  fixedbit_elt := [01.-]+
+  field_elt    := identifier ':' 's'? number
+  field_ref    := '%' identifier | identifier '=' '%' identifier
+  args_ref     := '&' identifier
+
+Defining a format is a handy way to avoid replicating groups of fields
+across many instruction patterns.
+
+A *fixedbit_elt* describes a contiguous sequence of bits that must
+be 1, 0, or don't care.  The difference between '.' and '-'
+is that '.' means that the bit will be covered with a field or a
+final 0 or 1 from the pattern, and '-' means that the bit is really
+ignored by the cpu and will not be specified.
+
+A *field_elt* describes a simple field only given a width; the position of
+the field is implied by its position with respect to other *fixedbit_elt*
+and *field_elt*.
+
+If any *fixedbit_elt* or *field_elt* appear, then all bits must be defined.
+Padding with a *fixedbit_elt* of all '.' is an easy way to accomplish that.
+
+A *field_ref* incorporates a field by reference.  This is the only way to
+add a complex field to a format.  A field may be renamed in the process
+via assignment to another identifier.  This is intended to allow the
+same argument set be used with disjoint named fields.
+
+A single *args_ref* may specify an argument set to use for the format.
+The set of fields in the format must be a subset of the arguments in
+the argument set.  If an argument set is not specified, one will be
+inferred from the set of fields.
+
+It is recommended, but not required, that all *field_ref* and *args_ref*
+appear at the end of the line, not interleaving with *fixedbit_elf* or
+*field_elt*.
+
+Format examples::
+
+  @opr    ...... ra:5 rb:5 ... 0 ....... rc:5
+  @opi    ...... ra:5 lit:8    1 ....... rc:5
+
+Patterns
+========
+
+Syntax::
+
+  pat_def      := identifier ( pat_elt )+
+  pat_elt      := fixedbit_elt | field_elt | field_ref | args_ref | fmt_ref | const_elt
+  fmt_ref      := '@' identifier
+  const_elt    := identifier '=' number
+
+The *fixedbit_elt* and *field_elt* specifiers are unchanged from formats.
+A pattern that does not specify a named format will have one inferred
+from a referenced argument set (if present) and the set of fields.
+
+A *const_elt* allows a argument to be set to a constant value.  This may
+come in handy when fields overlap between patterns and one has to
+include the values in the *fixedbit_elt* instead.
+
+The decoder will call a translator function for each pattern matched.
+
+Pattern examples::
+
+  addl_r   010000 ..... ..... .... 0000000 ..... @opr
+  addl_i   010000 ..... ..... .... 0000000 ..... @opi
+
+which will, in part, invoke::
+
+  trans_addl_r(ctx, &arg_opr, insn)
+
+and::
+
+  trans_addl_i(ctx, &arg_opi, insn)
diff --git a/scripts/decodetree.py b/scripts/decodetree.py
index e342d278b8..ba203aeccd 100755
--- a/scripts/decodetree.py
+++ b/scripts/decodetree.py
@@ -17,139 +17,7 @@
 
 #
 # Generate a decoding tree from a specification file.
-#
-# The tree is built from instruction "patterns".  A pattern may represent
-# a single architectural instruction or a group of same, depending on what
-# is convenient for further processing.
-#
-# Each pattern has "fixedbits" & "fixedmask", the combination of which
-# describes the condition under which the pattern is matched:
-#
-#   (insn & fixedmask) == fixedbits
-#
-# Each pattern may have "fields", which are extracted from the insn and
-# passed along to the translator.  Examples of such are registers,
-# immediates, and sub-opcodes.
-#
-# In support of patterns, one may declare fields, argument sets, and
-# formats, each of which may be re-used to simplify further definitions.
-#
-# *** Field syntax:
-#
-# field_def     := '%' identifier ( unnamed_field )+ ( !function=identifier )?
-# unnamed_field := number ':' ( 's' ) number
-#
-# For unnamed_field, the first number is the least-significant bit position of
-# the field and the second number is the length of the field.  If the 's' is
-# present, the field is considered signed.  If multiple unnamed_fields are
-# present, they are concatenated.  In this way one can define disjoint fields.
-#
-# If !function is specified, the concatenated result is passed through the
-# named function, taking and returning an integral value.
-#
-# FIXME: the fields of the structure into which this result will be stored
-# is restricted to "int".  Which means that we cannot expand 64-bit items.
-#
-# Field examples:
-#
-#   %disp   0:s16          -- sextract(i, 0, 16)
-#   %imm9   16:6 10:3      -- extract(i, 16, 6) << 3 | extract(i, 10, 3)
-#   %disp12 0:s1 1:1 2:10  -- sextract(i, 0, 1) << 11
-#                             | extract(i, 1, 1) << 10
-#                             | extract(i, 2, 10)
-#   %shimm8 5:s8 13:1 !function=expand_shimm8
-#                          -- expand_shimm8(sextract(i, 5, 8) << 1
-#                                           | extract(i, 13, 1))
-#
-# *** Argument set syntax:
-#
-# args_def    := '&' identifier ( args_elt )+ ( !extern )?
-# args_elt    := identifier
-#
-# Each args_elt defines an argument within the argument set.
-# Each argument set will be rendered as a C structure "arg_$name"
-# with each of the fields being one of the member arguments.
-#
-# If !extern is specified, the backing structure is assumed to
-# have been already declared, typically via a second decoder.
-#
-# Argument set examples:
-#
-#   &reg3       ra rb rc
-#   &loadstore  reg base offset
-#
-# *** Format syntax:
-#
-# fmt_def      := '@' identifier ( fmt_elt )+
-# fmt_elt      := fixedbit_elt | field_elt | field_ref | args_ref
-# fixedbit_elt := [01.-]+
-# field_elt    := identifier ':' 's'? number
-# field_ref    := '%' identifier | identifier '=' '%' identifier
-# args_ref     := '&' identifier
-#
-# Defining a format is a handy way to avoid replicating groups of fields
-# across many instruction patterns.
-#
-# A fixedbit_elt describes a contiguous sequence of bits that must
-# be 1, 0, [.-] for don't care.  The difference between '.' and '-'
-# is that '.' means that the bit will be covered with a field or a
-# final [01] from the pattern, and '-' means that the bit is really
-# ignored by the cpu and will not be specified.
-#
-# A field_elt describes a simple field only given a width; the position of
-# the field is implied by its position with respect to other fixedbit_elt
-# and field_elt.
-#
-# If any fixedbit_elt or field_elt appear then all bits must be defined.
-# Padding with a fixedbit_elt of all '.' is an easy way to accomplish that.
-#
-# A field_ref incorporates a field by reference.  This is the only way to
-# add a complex field to a format.  A field may be renamed in the process
-# via assignment to another identifier.  This is intended to allow the
-# same argument set be used with disjoint named fields.
-#
-# A single args_ref may specify an argument set to use for the format.
-# The set of fields in the format must be a subset of the arguments in
-# the argument set.  If an argument set is not specified, one will be
-# inferred from the set of fields.
-#
-# It is recommended, but not required, that all field_ref and args_ref
-# appear at the end of the line, not interleaving with fixedbit_elf or
-# field_elt.
-#
-# Format examples:
-#
-#   @opr    ...... ra:5 rb:5 ... 0 ....... rc:5
-#   @opi    ...... ra:5 lit:8    1 ....... rc:5
-#
-# *** Pattern syntax:
-#
-# pat_def      := identifier ( pat_elt )+
-# pat_elt      := fixedbit_elt | field_elt | field_ref
-#               | args_ref | fmt_ref | const_elt
-# fmt_ref      := '@' identifier
-# const_elt    := identifier '=' number
-#
-# The fixedbit_elt and field_elt specifiers are unchanged from formats.
-# A pattern that does not specify a named format will have one inferred
-# from a referenced argument set (if present) and the set of fields.
-#
-# A const_elt allows a argument to be set to a constant value.  This may
-# come in handy when fields overlap between patterns and one has to
-# include the values in the fixedbit_elt instead.
-#
-# The decoder will call a translator function for each pattern matched.
-#
-# Pattern examples:
-#
-#   addl_r   010000 ..... ..... .... 0000000 ..... @opr
-#   addl_i   010000 ..... ..... .... 0000000 ..... @opi
-#
-# which will, in part, invoke
-#
-#   trans_addl_r(ctx, &arg_opr, insn)
-# and
-#   trans_addl_i(ctx, &arg_opi, insn)
+# See the syntax and semantics in docs/decodetree.rst.
 #
 
 import os
-- 
2.17.2

[Qemu-devel] [PATCH 3/5] decodetree: Ensure build_tree does not include values outside insnmask

[Richard Henderson]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Reproduced with "scripts/decodetree.py /dev/null".

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 scripts/decodetree.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/decodetree.py b/scripts/decodetree.py
index ba203aeccd..fb9a0ab3ad 100755
--- a/scripts/decodetree.py
+++ b/scripts/decodetree.py
@@ -784,7 +784,7 @@ class Tree:
 
 def build_tree(pats, outerbits, outermask):
     # Find the intersection of all remaining fixedmask.
-    innermask = ~outermask
+    innermask = ~outermask & insnmask
     for i in pats:
         innermask &= i.fixedmask
 
-- 
2.17.2

[Qemu-devel] [PATCH 4/5] decodetree: Do not unconditionaly return from Pattern.output_code

[Richard Henderson]

As a consequence, the 'return false' gets pushed up one level.

This will allow us to perform some other action when the
translator returns failure.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 scripts/decodetree.py | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/scripts/decodetree.py b/scripts/decodetree.py
index fb9a0ab3ad..dd495096fc 100755
--- a/scripts/decodetree.py
+++ b/scripts/decodetree.py
@@ -348,8 +348,8 @@ class Pattern(General):
             output(ind, self.base.extract_name(), '(&u.f_', arg, ', insn);\n')
         for n, f in self.fields.items():
             output(ind, 'u.f_', arg, '.', n, ' = ', f.str_extract(), ';\n')
-        output(ind, 'return ', translate_prefix, '_', self.name,
-               '(ctx, &u.f_', arg, ');\n')
+        output(ind, 'if (', translate_prefix, '_', self.name,
+               '(ctx, &u.f_', arg, ')) return true;\n')
 # end Pattern
 
 
@@ -777,8 +777,8 @@ class Tree:
             output(ind, '    /* ',
                    str_match_bits(innerbits, innermask), ' */\n')
             s.output_code(i + 4, extracted, innerbits, innermask)
+            output(ind, '    return false;\n')
         output(ind, '}\n')
-        output(ind, 'return false;\n')
 # end Tree
 
 
@@ -932,6 +932,7 @@ def main():
     output(i4, '} u;\n\n')
 
     t.output_code(4, False, 0, 0)
+    output(i4, 'return false;\n')
 
     output('}\n')
 
-- 
2.17.2

[Qemu-devel] [PATCH 5/5] decodetree: Allow grouping of overlapping patterns

[Richard Henderson]

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 docs/decodetree.rst   |  58 +++++++++++++++++
 scripts/decodetree.py | 144 ++++++++++++++++++++++++++++++++++++++----
 2 files changed, 191 insertions(+), 11 deletions(-)

diff --git a/docs/decodetree.rst b/docs/decodetree.rst
index d9be30b2db..391069c105 100644
--- a/docs/decodetree.rst
+++ b/docs/decodetree.rst
@@ -154,3 +154,61 @@ which will, in part, invoke::
 and::
 
   trans_addl_i(ctx, &arg_opi, insn)
+
+Pattern Groups
+==============
+
+Syntax::
+
+  group    := '{' ( pat_def | group )+ '}'
+
+A *group* begins with a lone open-brace, with all subsequent lines
+indented two spaces, and ending with a lone close-brace.  Groups
+may be nested, increasing the required indentation of the lines
+within the nested group to two spaces per nesting level.
+
+Unlike ungrouped patterns, grouped patterns are allowed to overlap.
+Conflicts are resolved by selecting the patterns in order.  If all
+of the fixedbits for a pattern match, its translate function will
+be called.  If the translate function returns false, then subsequent
+patterns within the group will be matched.
+
+The following example from PA-RISC shows specialization of the *or*
+instruction::
+
+  {
+    {
+      nop   000010 ----- ----- 0000 001001 0 00000
+      copy  000010 00000 r1:5  0000 001001 0 rt:5
+    }
+    or      000010 r2:5  r1:5  cf:4 001001 0 rt:5
+  }
+
+When the *cf* field is zero, the instruction has no side effects,
+and may be specialized.  When the *rt* field is zero, the output
+is discarded and so the instruction has no effect.  When the *rt2*
+field is zero, the operation is ``reg[rt] | 0`` and so encodes
+the canonical register copy operation.
+
+The output from the generator might look like::
+
+  switch (insn & 0xfc000fe0) {
+  case 0x08000240:
+    /* 000010.. ........ ....0010 010..... */
+    if ((insn & 0x0000f000) == 0x00000000) {
+        /* 000010.. ........ 00000010 010..... */
+        if ((insn & 0x0000001f) == 0x00000000) {
+            /* 000010.. ........ 00000010 01000000 */
+            extract_decode_Fmt_0(&u.f_decode0, insn);
+            if (trans_nop(ctx, &u.f_decode0)) return true;
+        }
+        if ((insn & 0x03e00000) == 0x00000000) {
+            /* 00001000 000..... 00000010 010..... */
+            extract_decode_Fmt_1(&u.f_decode1, insn);
+            if (trans_copy(ctx, &u.f_decode1)) return true;
+        }
+    }
+    extract_decode_Fmt_2(&u.f_decode2, insn);
+    if (trans_or(ctx, &u.f_decode2)) return true;
+    return false;
+  }
diff --git a/scripts/decodetree.py b/scripts/decodetree.py
index dd495096fc..abce58ed8f 100755
--- a/scripts/decodetree.py
+++ b/scripts/decodetree.py
@@ -31,6 +31,7 @@ fields = {}
 arguments = {}
 formats = {}
 patterns = []
+allpatterns = []
 
 translate_prefix = 'trans'
 translate_scope = 'static '
@@ -353,6 +354,46 @@ class Pattern(General):
 # end Pattern
 
 
+class MultiPattern(General):
+    """Class representing an overlapping set of instruction patterns"""
+
+    def __init__(self, lineno, pats, fixb, fixm, udfm):
+        self.lineno = lineno
+        self.pats = pats
+        self.base = None
+        self.fixedbits = fixb
+        self.fixedmask = fixm
+        self.undefmask = udfm
+
+    def __str__(self):
+        r = "{"
+        for p in self.pats:
+           r = r + ' ' + str(p)
+        return r + "}"
+
+    def output_decl(self):
+        for p in self.pats:
+            p.output_decl()
+
+    def output_code(self, i, extracted, outerbits, outermask):
+        global translate_prefix
+        ind = str_indent(i)
+        for p in self.pats:
+            if outermask != p.fixedmask:
+                innermask = p.fixedmask & ~outermask
+                innerbits = p.fixedbits & ~outermask
+                output(ind, 'if ((insn & ',
+                       '0x{0:08x}) == 0x{1:08x}'.format(innermask, innerbits),
+                       ') {\n')
+                output(ind, '    /* ',
+                       str_match_bits(p.fixedbits, p.fixedmask), ' */\n')
+                p.output_code(i + 4, extracted, p.fixedbits, p.fixedmask)
+                output(ind, '}\n')
+            else:
+                p.output_code(i, extracted, p.fixedbits, p.fixedmask)
+#end MultiPattern
+
+
 def parse_field(lineno, name, toks):
     """Parse one instruction field from TOKS at LINENO"""
     global fields
@@ -505,6 +546,7 @@ def parse_generic(lineno, is_format, name, toks):
     global arguments
     global formats
     global patterns
+    global allpatterns
     global re_ident
     global insnwidth
     global insnmask
@@ -649,6 +691,7 @@ def parse_generic(lineno, is_format, name, toks):
         pat = Pattern(name, lineno, fmt, fixedbits, fixedmask,
                       undefmask, fieldmask, flds)
         patterns.append(pat)
+        allpatterns.append(pat)
 
     # Validate the masks that we have assembled.
     if fieldmask & fixedmask:
@@ -667,17 +710,61 @@ def parse_generic(lineno, is_format, name, toks):
                           .format(allbits ^ insnmask))
 # end parse_general
 
+def build_multi_pattern(lineno, pats):
+    """Validate the Patterns going into a MultiPattern."""
+    global patterns
+    global insnmask
+
+    if len(pats) < 2:
+        error(lineno, 'less than two patterns within braces')
+
+    fixedmask = insnmask
+    undefmask = insnmask
+
+    for p in pats:
+        fixedmask &= p.fixedmask
+        undefmask &= p.undefmask
+        if p.lineno < lineno:
+            lineno = p.lineno
+
+    if fixedmask == 0:
+        error(lineno, 'no overlap in patterns within braces')
+
+    fixedbits = None
+    for p in pats:
+        thisbits = p.fixedbits & fixedmask
+        if fixedbits is None:
+            fixedbits = thisbits
+        elif fixedbits != thisbits:
+            error(p.lineno, 'fixedbits mismatch within braces',
+                  '(0x{0:08x} != 0x{1:08x})'.format(thisbits, fixedbits))
+
+    mp = MultiPattern(lineno, pats, fixedbits, fixedmask, undefmask)
+    patterns.append(mp)
+# end build_multi_pattern
 
 def parse_file(f):
     """Parse all of the patterns within a file"""
 
+    global patterns
+
     # Read all of the lines of the file.  Concatenate lines
     # ending in backslash; discard empty lines and comments.
     toks = []
     lineno = 0
+    nesting = 0
+    saved_pats = []
+
     for line in f:
         lineno += 1
 
+        # Expand and strip spaces, to find indent.
+        line = line.rstrip()
+        line = line.expandtabs()
+        len1 = len(line)
+        line = line.lstrip()
+        len2 = len(line)
+
         # Discard comments
         end = line.find('#')
         if end >= 0:
@@ -687,10 +774,18 @@ def parse_file(f):
         if len(toks) != 0:
             # Next line after continuation
             toks.extend(t)
-        elif len(t) == 0:
-            # Empty line
-            continue
         else:
+            # Allow completely blank lines.
+            if len1 == 0:
+                continue
+            indent = len1 - len2
+            # Empty line due to comment.
+            if len(t) == 0:
+                # Indentation must be correct, even for comment lines.
+                if indent != nesting:
+                    error(lineno, 'indentation ', indent, ' != ', nesting)
+                continue
+            start_lineno = lineno
             toks = t
 
         # Continuation?
@@ -698,21 +793,47 @@ def parse_file(f):
             toks.pop()
             continue
 
-        if len(toks) < 2:
-            error(lineno, 'short line')
-
         name = toks[0]
         del toks[0]
 
+        # End nesting?
+        if name == '}':
+            if nesting == 0:
+                error(start_lineno, 'mismatched close brace')
+            if len(toks) != 0:
+                error(start_lineno, 'extra tokens after close brace')
+            nesting -= 2
+            if indent != nesting:
+                error(start_lineno, 'indentation ', indent, ' != ', nesting)
+            pats = patterns
+            patterns = saved_pats.pop()
+            build_multi_pattern(lineno, pats)
+            toks = []
+            continue
+
+        # Everything else should have current indentation.
+        if indent != nesting:
+            error(start_lineno, 'indentation ', indent, ' != ', nesting)
+
+        # Start nesting?
+        if name == '{':
+            if len(toks) != 0:
+                error(start_lineno, 'extra tokens after open brace')
+            saved_pats.append(patterns)
+            patterns = []
+            nesting += 2
+            toks = []
+            continue
+
         # Determine the type of object needing to be parsed.
         if name[0] == '%':
-            parse_field(lineno, name[1:], toks)
+            parse_field(start_lineno, name[1:], toks)
         elif name[0] == '&':
-            parse_arguments(lineno, name[1:], toks)
+            parse_arguments(start_lineno, name[1:], toks)
         elif name[0] == '@':
-            parse_generic(lineno, True, name[1:], toks)
+            parse_generic(start_lineno, True, name[1:], toks)
         else:
-            parse_generic(lineno, False, name, toks)
+            parse_generic(start_lineno, False, name, toks)
         toks = []
 # end parse_file
 
@@ -846,6 +967,7 @@ def main():
     global arguments
     global formats
     global patterns
+    global allpatterns
     global translate_scope
     global translate_prefix
     global output_fd
@@ -907,7 +1029,7 @@ def main():
     # Make sure that the argument sets are the same, and declare the
     # function only once.
     out_pats = {}
-    for i in patterns:
+    for i in allpatterns:
         if i.name in out_pats:
             p = out_pats[i.name]
             if i.base.base != p.base.base:
-- 
2.17.2

[Qemu-devel] [RFC 6/7] target/hppa: Use pattern groups to decode OR

[Richard Henderson]

It seems clearer to decode specializations of OR within decodetree
instead of by hand within the translation function.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 target/hppa/translate.c  | 106 ++++++++++++++++++++-------------------
 target/hppa/insns.decode |  10 +++-
 2 files changed, 64 insertions(+), 52 deletions(-)

diff --git a/target/hppa/translate.c b/target/hppa/translate.c
index b4fd307b77..8f5010b633 100644
--- a/target/hppa/translate.c
+++ b/target/hppa/translate.c
@@ -2619,59 +2619,63 @@ static bool trans_and(DisasContext *ctx, arg_rrr_cf *a)
     return do_log_reg(ctx, a, tcg_gen_and_reg);
 }
 
+static bool trans_copy(DisasContext *ctx, arg_copy *a)
+{
+    unsigned r = a->r;
+    unsigned t = a->t;
+
+    if (r == 0) {
+        TCGv_reg dest = dest_gpr(ctx, t);
+        tcg_gen_movi_reg(dest, 0);
+        save_gpr(ctx, t, dest);
+    } else {
+        save_gpr(ctx, t, cpu_gr[r]);
+    }
+    cond_free(&ctx->null_cond);
+    return true;
+}
+
+static bool trans_pause(DisasContext *ctx, arg_pause *a)
+{
+#ifndef CONFIG_USER_ONLY
+    /*
+     * These are QEMU extensions and are nops in the real architecture:
+     *
+     * or %r10,%r10,%r10 -- idle loop; wait for interrupt
+     * or %r31,%r31,%r31 -- death loop; offline cpu
+     *                      currently implemented as idle.
+     */
+    TCGv_i32 tmp;
+
+    /*
+     * No need to check for supervisor, as userland can only pause
+     * until the next timer interrupt.
+     */
+    nullify_over(ctx);
+
+    /* Advance the instruction queue.  */
+    copy_iaoq_entry(cpu_iaoq_f, ctx->iaoq_b, cpu_iaoq_b);
+    copy_iaoq_entry(cpu_iaoq_b, ctx->iaoq_n, ctx->iaoq_n_var);
+    nullify_set(ctx, 0);
+
+    /* Tell the qemu main loop to halt until this cpu has work.  */
+    tmp = tcg_const_i32(1);
+    tcg_gen_st_i32(tmp, cpu_env, -offsetof(HPPACPU, env) +
+                   offsetof(CPUState, halted));
+    tcg_temp_free_i32(tmp);
+    gen_excp_1(EXCP_HALTED);
+    ctx->base.is_jmp = DISAS_NORETURN;
+
+    return nullify_end(ctx);
+#else
+    /* For user-only, don't pause but treat as nop.  */
+    cond_free(&ctx->null_cond);
+    return true;
+#endif
+}
+
 static bool trans_or(DisasContext *ctx, arg_rrr_cf *a)
 {
-    if (a->cf == 0) {
-        unsigned r2 = a->r2;
-        unsigned r1 = a->r1;
-        unsigned rt = a->t;
-
-        if (rt == 0) { /* NOP */
-            cond_free(&ctx->null_cond);
-            return true;
-        }
-        if (r2 == 0) { /* COPY */
-            if (r1 == 0) {
-                TCGv_reg dest = dest_gpr(ctx, rt);
-                tcg_gen_movi_reg(dest, 0);
-                save_gpr(ctx, rt, dest);
-            } else {
-                save_gpr(ctx, rt, cpu_gr[r1]);
-            }
-            cond_free(&ctx->null_cond);
-            return true;
-        }
-#ifndef CONFIG_USER_ONLY
-        /* These are QEMU extensions and are nops in the real architecture:
-         *
-         * or %r10,%r10,%r10 -- idle loop; wait for interrupt
-         * or %r31,%r31,%r31 -- death loop; offline cpu
-         *                      currently implemented as idle.
-         */
-        if ((rt == 10 || rt == 31) && r1 == rt && r2 == rt) { /* PAUSE */
-            TCGv_i32 tmp;
-
-            /* No need to check for supervisor, as userland can only pause
-               until the next timer interrupt.  */
-            nullify_over(ctx);
-
-            /* Advance the instruction queue.  */
-            copy_iaoq_entry(cpu_iaoq_f, ctx->iaoq_b, cpu_iaoq_b);
-            copy_iaoq_entry(cpu_iaoq_b, ctx->iaoq_n, ctx->iaoq_n_var);
-            nullify_set(ctx, 0);
-
-            /* Tell the qemu main loop to halt until this cpu has work.  */
-            tmp = tcg_const_i32(1);
-            tcg_gen_st_i32(tmp, cpu_env, -offsetof(HPPACPU, env) +
-                                         offsetof(CPUState, halted));
-            tcg_temp_free_i32(tmp);
-            gen_excp_1(EXCP_HALTED);
-            ctx->base.is_jmp = DISAS_NORETURN;
-
-            return nullify_end(ctx);
-        }
-#endif
-    }
     return do_log_reg(ctx, a, tcg_gen_or_reg);
 }
 
diff --git a/target/hppa/insns.decode b/target/hppa/insns.decode
index 55ff39dd05..46c64334d1 100644
--- a/target/hppa/insns.decode
+++ b/target/hppa/insns.decode
@@ -148,7 +148,15 @@ lci             000001 ----- ----- -- 01001100 0 t:5
 
 andcm           000010 ..... ..... .... 000000 0 .....  @rrr_cf
 and             000010 ..... ..... .... 001000 0 .....  @rrr_cf
-or              000010 ..... ..... .... 001001 0 .....  @rrr_cf
+{
+  {
+    nop         000010 ----- ----- 0000 001001 0 00000
+    copy        000010 00000 r:5   0000 001001 0 t:5
+    pause       000010 01010 01010 0000 001001 0 01010
+    pause       000010 11111 11111 0000 001001 0 11111
+  }
+  or            000010 ..... ..... .... 001001 0 .....  @rrr_cf
+}
 xor             000010 ..... ..... .... 001010 0 .....  @rrr_cf
 uxor            000010 ..... ..... .... 001110 0 .....  @rrr_cf
 ds              000010 ..... ..... .... 010001 0 .....  @rrr_cf
-- 
2.17.2

[Qemu-devel] [RFC 7/7] target/riscv: Use pattern groups for RVC

[Richard Henderson]

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 target/riscv/insn_trans/trans_rvc.inc.c | 60 +++++++++----------------
 target/riscv/insn16.decode              | 23 +++++++---
 target/riscv/insn32.decode              | 18 ++++----
 3 files changed, 48 insertions(+), 53 deletions(-)

diff --git a/target/riscv/insn_trans/trans_rvc.inc.c b/target/riscv/insn_trans/trans_rvc.inc.c
index 631e72c8b5..a81da2f107 100644
--- a/target/riscv/insn_trans/trans_rvc.inc.c
+++ b/target/riscv/insn_trans/trans_rvc.inc.c
@@ -48,24 +48,6 @@ static bool trans_c_li(DisasContext *ctx, arg_c_li *a)
     return trans_addi(ctx, &arg);
 }
 
-static bool trans_c_addi16sp_lui(DisasContext *ctx, arg_c_addi16sp_lui *a)
-{
-    if (a->rd == 2) {
-        /* C.ADDI16SP */
-        arg_addi arg = { .rd = 2, .rs1 = 2, .imm = a->imm_addi16sp };
-        return trans_addi(ctx, &arg);
-    } else if (a->imm_lui != 0) {
-        /* C.LUI */
-        if (a->rd == 0) {
-            /* Hint: insn is valid but does not affect state */
-            return true;
-        }
-        arg_lui arg = { .rd = a->rd, .imm = a->imm_lui };
-        return trans_lui(ctx, &arg);
-    }
-    return false;
-}
-
 static bool trans_c_srli(DisasContext *ctx, arg_c_srli *a)
 {
     int shamt = a->shamt;
@@ -114,36 +96,38 @@ static bool trans_c_slli(DisasContext *ctx, arg_c_slli *a)
     return trans_slli(ctx, &arg);
 }
 
-static bool trans_c_jr_mv(DisasContext *ctx, arg_c_jr_mv *a)
+static bool trans_c_jr(DisasContext *ctx, arg_c_jr *a)
 {
-    if (a->rd != 0 && a->rs2 == 0) {
-        /* C.JR */
+    if (a->rd != 0) {
         arg_jalr arg = { .rd = 0, .rs1 = a->rd, .imm = 0 };
         return trans_jalr(ctx, &arg);
-    } else if (a->rd != 0 && a->rs2 != 0) {
-        /* C.MV */
+    }
+    return false;
+}
+
+static bool trans_c_mv(DisasContext *ctx, arg_c_mv *a)
+{
+    if (a->rd != 0 && a->rs2 != 0) {
         arg_add arg = { .rd = a->rd, .rs1 = 0, .rs2 = a->rs2 };
         return trans_add(ctx, &arg);
     }
     return false;
 }
 
-static bool trans_c_ebreak_jalr_add(DisasContext *ctx, arg_c_ebreak_jalr_add *a)
+static bool trans_c_jalr(DisasContext *ctx, arg_c_jalr *a)
 {
-    if (a->rd == 0 && a->rs2 == 0) {
-        /* C.EBREAK */
-        arg_ebreak arg = { };
-        return trans_ebreak(ctx, &arg);
-    } else if (a->rd != 0) {
-        if (a->rs2 == 0) {
-            /* C.JALR */
-            arg_jalr arg = { .rd = 1, .rs1 = a->rd, .imm = 0 };
-            return trans_jalr(ctx, &arg);
-        } else {
-            /* C.ADD */
-            arg_add arg = { .rd = a->rd, .rs1 = a->rd, .rs2 = a->rs2 };
-            return trans_add(ctx, &arg);
-        }
+    if (a->rd != 0) {
+        arg_jalr arg = { .rd = 1, .rs1 = a->rd, .imm = 0 };
+        return trans_jalr(ctx, &arg);
+    }
+    return false;
+}
+
+static bool trans_c_add(DisasContext *ctx, arg_c_add *a)
+{
+    if (a->rd != 0 && a->rs2 != 0) {
+        arg_add arg = { .rd = a->rd, .rs1 = a->rd, .rs2 = a->rs2 };
+        return trans_add(ctx, &arg);
     }
     return false;
 }
diff --git a/target/riscv/insn16.decode b/target/riscv/insn16.decode
index d88a0c78ab..5b93051a19 100644
--- a/target/riscv/insn16.decode
+++ b/target/riscv/insn16.decode
@@ -42,11 +42,13 @@
 
 
 # Argument sets imported from insn32.decode:
+&empty                  !extern
 &r         rd rs1 rs2   !extern
 &i         imm rs1 rd   !extern
 &s         imm rs1 rs2  !extern
 &j         imm rd       !extern
 &b         imm rs2 rs1  !extern
+&u         imm rd       !extern
 
 # Argument sets:
 &ci        imm        rd
@@ -55,8 +57,6 @@
 &cr               rd  rs2
 &c_shift   shamt      rd
 
-&c_addi16sp_lui  imm_lui imm_addi16sp rd
-
 # Formats 16:
 @cr        ....  ..... .....  .. &cr                      rs2=%rs2_5  %rd
 @ci        ... . ..... .....  .. &i  imm=%imm_ci %rd rs1=%rd
@@ -74,8 +74,6 @@
 @c_sd      ... . .....  ..... .. &s  imm=%uimm_6bit_sd  rs1=2 rs2=%rs2_5
 @c_sw      ... . .....  ..... .. &s  imm=%uimm_6bit_sw  rs1=2 rs2=%rs2_5
 
-@c_addi16sp_lui ... .  ..... ..... .. &c_addi16sp_lui %imm_lui %imm_addi16sp %rd
-
 @c_shift        ... . .. ... ..... .. &c_shift rd=%rs1_3 shamt=%nzuimm_6bit
 @c_shift2       ... . .. ... ..... .. &c_shift rd=%rd    shamt=%nzuimm_6bit
 
@@ -92,7 +90,11 @@ sw                110  ... ... .. ... 00 @cs_w
 # *** RV64C Standard Extension (Quadrant 1) ***
 c_addi            000 .  .....  ..... 01 @ci
 c_li              010 .  .....  ..... 01 @ci
-c_addi16sp_lui    011 .  .....  ..... 01 @c_addi16sp_lui # shares opc with C.LUI
+{
+  # addi16sp
+  addi            011 .  00010  ..... 01 &i rd=2 rs1=2 imm=%imm_addi16sp
+  lui             011 .  .....  ..... 01 &u %rd imm=%imm_lui
+}
 c_srli            100 . 00 ...  ..... 01 @c_shift
 c_srai            100 . 01 ...  ..... 01 @c_shift
 andi              100 . 10 ...  ..... 01 @c_andi
@@ -108,7 +110,14 @@ bne               111  ... ...  ..... 01 @cb # c_bnez
 c_slli            000 .  .....  ..... 10 @c_shift2
 fld               001 .  .....  ..... 10 @c_ld # fldsp
 lw                010 .  .....  ..... 10 @c_lw # lwsp
-c_jr_mv           100 0  .....  ..... 10 @cr
-c_ebreak_jalr_add 100 1  .....  ..... 10 @cr
+{
+  c_jr            100 0  .....  00000 10 %rd
+  c_mv            100 0  .....  ..... 10 @cr
+}
+{
+  ebreak          100 1  00000  00000 10
+  c_jalr          100 1  .....  00000 10 %rd
+  c_add           100 1  .....  ..... 10 @cr
+}
 fsd               101   ......  ..... 10 @c_sd # fsdsp
 sw                110 .  .....  ..... 10 @c_sw # swsp
diff --git a/target/riscv/insn32.decode b/target/riscv/insn32.decode
index 0e098e05fe..81bcb5dbb4 100644
--- a/target/riscv/insn32.decode
+++ b/target/riscv/insn32.decode
@@ -34,20 +34,22 @@
 %imm_u    12:s20                 !function=ex_shift_12
 
 # Argument sets:
-&b    imm rs2 rs1
-&i    imm rs1 rd
-&r    rd rs1 rs2
-&s    imm rs2 rs1
-&j    imm rd
-&shift     shamt rs1 rd
-&atomic    aq rl rs2 rs1 rd
+&empty
+&b        imm rs2 rs1
+&i        imm rs1 rd
+&r        rd rs1 rs2
+&s        imm rs2 rs1
+&j        imm rd
+&u        imm rd
+&shift    shamt rs1 rd
+&atomic   aq rl rs2 rs1 rd
 
 # Formats 32:
 @r       .......   ..... ..... ... ..... ....... &r                %rs2 %rs1 %rd
 @i       ............    ..... ... ..... ....... &i      imm=%imm_i     %rs1 %rd
 @b       .......   ..... ..... ... ..... ....... &b      imm=%imm_b %rs2 %rs1
 @s       .......   ..... ..... ... ..... ....... &s      imm=%imm_s %rs2 %rs1
-@u       ....................      ..... .......         imm=%imm_u          %rd
+@u       ....................      ..... ....... &u      imm=%imm_u          %rd
 @j       ....................      ..... ....... &j      imm=%imm_j          %rd
 
 @sh      ......  ...... .....  ... ..... ....... &shift  shamt=%sh10      %rs1 %rd
-- 
2.17.2

Re: [Qemu-devel] [PATCH 4/5] decodetree: Do not unconditionaly return from Pattern.output_code

[Philippe Mathieu-Daudé]

On 2/24/19 12:29 AM, Richard Henderson wrote:
> As a consequence, the 'return false' gets pushed up one level.
> 
> This will allow us to perform some other action when the
> translator returns failure.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

> ---
>  scripts/decodetree.py | 7 ++++---
>  1 file changed, 4 insertions(+), 3 deletions(-)
> 
> diff --git a/scripts/decodetree.py b/scripts/decodetree.py
> index fb9a0ab3ad..dd495096fc 100755
> --- a/scripts/decodetree.py
> +++ b/scripts/decodetree.py
> @@ -348,8 +348,8 @@ class Pattern(General):
>              output(ind, self.base.extract_name(), '(&u.f_', arg, ', insn);\n')
>          for n, f in self.fields.items():
>              output(ind, 'u.f_', arg, '.', n, ' = ', f.str_extract(), ';\n')
> -        output(ind, 'return ', translate_prefix, '_', self.name,
> -               '(ctx, &u.f_', arg, ');\n')
> +        output(ind, 'if (', translate_prefix, '_', self.name,
> +               '(ctx, &u.f_', arg, ')) return true;\n')
>  # end Pattern
>  
>  
> @@ -777,8 +777,8 @@ class Tree:
>              output(ind, '    /* ',
>                     str_match_bits(innerbits, innermask), ' */\n')
>              s.output_code(i + 4, extracted, innerbits, innermask)
> +            output(ind, '    return false;\n')
>          output(ind, '}\n')
> -        output(ind, 'return false;\n')
>  # end Tree
>  
>  
> @@ -932,6 +932,7 @@ def main():
>      output(i4, '} u;\n\n')
>  
>      t.output_code(4, False, 0, 0)
> +    output(i4, 'return false;\n')
>  
>      output('}\n')
>  
> 

Re: [Qemu-devel] [PATCH 5/5] decodetree: Allow grouping of overlapping patterns

[Philippe Mathieu-Daudé]

On 2/24/19 12:29 AM, Richard Henderson wrote:
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>  docs/decodetree.rst   |  58 +++++++++++++++++
>  scripts/decodetree.py | 144 ++++++++++++++++++++++++++++++++++++++----
>  2 files changed, 191 insertions(+), 11 deletions(-)
> 
> diff --git a/docs/decodetree.rst b/docs/decodetree.rst
> index d9be30b2db..391069c105 100644
> --- a/docs/decodetree.rst
> +++ b/docs/decodetree.rst
> @@ -154,3 +154,61 @@ which will, in part, invoke::
>  and::
>  
>    trans_addl_i(ctx, &arg_opi, insn)
> +
> +Pattern Groups
> +==============
> +
> +Syntax::
> +
> +  group    := '{' ( pat_def | group )+ '}'
> +
> +A *group* begins with a lone open-brace, with all subsequent lines
> +indented two spaces, and ending with a lone close-brace.  Groups
> +may be nested, increasing the required indentation of the lines
> +within the nested group to two spaces per nesting level.
> +
> +Unlike ungrouped patterns, grouped patterns are allowed to overlap.
> +Conflicts are resolved by selecting the patterns in order.  If all
> +of the fixedbits for a pattern match, its translate function will
> +be called.  If the translate function returns false, then subsequent
> +patterns within the group will be matched.

Excellent! The tibetan cuisine inspired you :P

> +
> +The following example from PA-RISC shows specialization of the *or*
> +instruction::
> +
> +  {
> +    {
> +      nop   000010 ----- ----- 0000 001001 0 00000
> +      copy  000010 00000 r1:5  0000 001001 0 rt:5
> +    }
> +    or      000010 r2:5  r1:5  cf:4 001001 0 rt:5
> +  }
> +
> +When the *cf* field is zero, the instruction has no side effects,
> +and may be specialized.  When the *rt* field is zero, the output
> +is discarded and so the instruction has no effect.  When the *rt2*

*r2*?

> +field is zero, the operation is ``reg[rt] | 0`` and so encodes
> +the canonical register copy operation.
> +
> +The output from the generator might look like::
> +
> +  switch (insn & 0xfc000fe0) {
> +  case 0x08000240:
> +    /* 000010.. ........ ....0010 010..... */
> +    if ((insn & 0x0000f000) == 0x00000000) {
> +        /* 000010.. ........ 00000010 010..... */
> +        if ((insn & 0x0000001f) == 0x00000000) {
> +            /* 000010.. ........ 00000010 01000000 */
> +            extract_decode_Fmt_0(&u.f_decode0, insn);
> +            if (trans_nop(ctx, &u.f_decode0)) return true;
> +        }
> +        if ((insn & 0x03e00000) == 0x00000000) {
> +            /* 00001000 000..... 00000010 010..... */
> +            extract_decode_Fmt_1(&u.f_decode1, insn);
> +            if (trans_copy(ctx, &u.f_decode1)) return true;
> +        }
> +    }
> +    extract_decode_Fmt_2(&u.f_decode2, insn);
> +    if (trans_or(ctx, &u.f_decode2)) return true;
> +    return false;
> +  }
> diff --git a/scripts/decodetree.py b/scripts/decodetree.py
> index dd495096fc..abce58ed8f 100755
> --- a/scripts/decodetree.py
> +++ b/scripts/decodetree.py
> @@ -31,6 +31,7 @@ fields = {}
>  arguments = {}
>  formats = {}
>  patterns = []
> +allpatterns = []
>  
>  translate_prefix = 'trans'
>  translate_scope = 'static '
> @@ -353,6 +354,46 @@ class Pattern(General):
>  # end Pattern
>  
>  
> +class MultiPattern(General):
> +    """Class representing an overlapping set of instruction patterns"""
> +
> +    def __init__(self, lineno, pats, fixb, fixm, udfm):
> +        self.lineno = lineno
> +        self.pats = pats
> +        self.base = None
> +        self.fixedbits = fixb
> +        self.fixedmask = fixm
> +        self.undefmask = udfm
> +
> +    def __str__(self):
> +        r = "{"
> +        for p in self.pats:
> +           r = r + ' ' + str(p)
> +        return r + "}"
> +
> +    def output_decl(self):
> +        for p in self.pats:
> +            p.output_decl()
> +
> +    def output_code(self, i, extracted, outerbits, outermask):
> +        global translate_prefix
> +        ind = str_indent(i)
> +        for p in self.pats:
> +            if outermask != p.fixedmask:
> +                innermask = p.fixedmask & ~outermask
> +                innerbits = p.fixedbits & ~outermask
> +                output(ind, 'if ((insn & ',
> +                       '0x{0:08x}) == 0x{1:08x}'.format(innermask, innerbits),
> +                       ') {\n')
> +                output(ind, '    /* ',
> +                       str_match_bits(p.fixedbits, p.fixedmask), ' */\n')
> +                p.output_code(i + 4, extracted, p.fixedbits, p.fixedmask)
> +                output(ind, '}\n')
> +            else:
> +                p.output_code(i, extracted, p.fixedbits, p.fixedmask)
> +#end MultiPattern
> +
> +
>  def parse_field(lineno, name, toks):
>      """Parse one instruction field from TOKS at LINENO"""
>      global fields
> @@ -505,6 +546,7 @@ def parse_generic(lineno, is_format, name, toks):
>      global arguments
>      global formats
>      global patterns
> +    global allpatterns
>      global re_ident
>      global insnwidth
>      global insnmask
> @@ -649,6 +691,7 @@ def parse_generic(lineno, is_format, name, toks):
>          pat = Pattern(name, lineno, fmt, fixedbits, fixedmask,
>                        undefmask, fieldmask, flds)
>          patterns.append(pat)
> +        allpatterns.append(pat)
>  
>      # Validate the masks that we have assembled.
>      if fieldmask & fixedmask:
> @@ -667,17 +710,61 @@ def parse_generic(lineno, is_format, name, toks):
>                            .format(allbits ^ insnmask))
>  # end parse_general
>  
> +def build_multi_pattern(lineno, pats):
> +    """Validate the Patterns going into a MultiPattern."""
> +    global patterns
> +    global insnmask
> +
> +    if len(pats) < 2:
> +        error(lineno, 'less than two patterns within braces')
> +
> +    fixedmask = insnmask
> +    undefmask = insnmask
> +
> +    for p in pats:
> +        fixedmask &= p.fixedmask
> +        undefmask &= p.undefmask
> +        if p.lineno < lineno:
> +            lineno = p.lineno
> +
> +    if fixedmask == 0:
> +        error(lineno, 'no overlap in patterns within braces')
> +
> +    fixedbits = None
> +    for p in pats:
> +        thisbits = p.fixedbits & fixedmask
> +        if fixedbits is None:
> +            fixedbits = thisbits
> +        elif fixedbits != thisbits:
> +            error(p.lineno, 'fixedbits mismatch within braces',
> +                  '(0x{0:08x} != 0x{1:08x})'.format(thisbits, fixedbits))
> +
> +    mp = MultiPattern(lineno, pats, fixedbits, fixedmask, undefmask)
> +    patterns.append(mp)
> +# end build_multi_pattern
>  
>  def parse_file(f):
>      """Parse all of the patterns within a file"""
>  
> +    global patterns
> +
>      # Read all of the lines of the file.  Concatenate lines
>      # ending in backslash; discard empty lines and comments.
>      toks = []
>      lineno = 0
> +    nesting = 0
> +    saved_pats = []
> +
>      for line in f:
>          lineno += 1
>  
> +        # Expand and strip spaces, to find indent.
> +        line = line.rstrip()
> +        line = line.expandtabs()
> +        len1 = len(line)
> +        line = line.lstrip()
> +        len2 = len(line)
> +
>          # Discard comments
>          end = line.find('#')
>          if end >= 0:
> @@ -687,10 +774,18 @@ def parse_file(f):
>          if len(toks) != 0:
>              # Next line after continuation
>              toks.extend(t)
> -        elif len(t) == 0:
> -            # Empty line
> -            continue
>          else:
> +            # Allow completely blank lines.
> +            if len1 == 0:
> +                continue
> +            indent = len1 - len2
> +            # Empty line due to comment.
> +            if len(t) == 0:
> +                # Indentation must be correct, even for comment lines.
> +                if indent != nesting:
> +                    error(lineno, 'indentation ', indent, ' != ', nesting)
> +                continue
> +            start_lineno = lineno
>              toks = t
>  
>          # Continuation?
> @@ -698,21 +793,47 @@ def parse_file(f):
>              toks.pop()
>              continue
>  
> -        if len(toks) < 2:
> -            error(lineno, 'short line')
> -
>          name = toks[0]
>          del toks[0]
>  
> +        # End nesting?
> +        if name == '}':
> +            if nesting == 0:
> +                error(start_lineno, 'mismatched close brace')
> +            if len(toks) != 0:
> +                error(start_lineno, 'extra tokens after close brace')
> +            nesting -= 2
> +            if indent != nesting:
> +                error(start_lineno, 'indentation ', indent, ' != ', nesting)
> +            pats = patterns
> +            patterns = saved_pats.pop()
> +            build_multi_pattern(lineno, pats)
> +            toks = []
> +            continue
> +
> +        # Everything else should have current indentation.
> +        if indent != nesting:
> +            error(start_lineno, 'indentation ', indent, ' != ', nesting)
> +
> +        # Start nesting?
> +        if name == '{':
> +            if len(toks) != 0:
> +                error(start_lineno, 'extra tokens after open brace')
> +            saved_pats.append(patterns)
> +            patterns = []
> +            nesting += 2
> +            toks = []
> +            continue
> +
>          # Determine the type of object needing to be parsed.
>          if name[0] == '%':
> -            parse_field(lineno, name[1:], toks)
> +            parse_field(start_lineno, name[1:], toks)
>          elif name[0] == '&':
> -            parse_arguments(lineno, name[1:], toks)
> +            parse_arguments(start_lineno, name[1:], toks)
>          elif name[0] == '@':
> -            parse_generic(lineno, True, name[1:], toks)
> +            parse_generic(start_lineno, True, name[1:], toks)
>          else:
> -            parse_generic(lineno, False, name, toks)
> +            parse_generic(start_lineno, False, name, toks)
>          toks = []
>  # end parse_file
>  
> @@ -846,6 +967,7 @@ def main():
>      global arguments
>      global formats
>      global patterns
> +    global allpatterns
>      global translate_scope
>      global translate_prefix
>      global output_fd
> @@ -907,7 +1029,7 @@ def main():
>      # Make sure that the argument sets are the same, and declare the
>      # function only once.
>      out_pats = {}
> -    for i in patterns:
> +    for i in allpatterns:
>          if i.name in out_pats:
>              p = out_pats[i.name]
>              if i.base.base != p.base.base:
> 

I'm anxious to test this :)

Re: [Qemu-devel] [PATCH 2/5] decodetree: Move documentation to docs/decodetree.rst

[Peter Maydell]

On Sat, 23 Feb 2019 at 23:35, Richard Henderson
<richard.henderson@linaro.org> wrote:
>
> One great big block comment isn't the best way to document
> the syntax of a language.
>
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>  MAINTAINERS           |   1 +
>  docs/decodetree.rst   | 156 ++++++++++++++++++++++++++++++++++++++++++
>  scripts/decodetree.py | 134 +-----------------------------------
>  3 files changed, 158 insertions(+), 133 deletions(-)
>  create mode 100644 docs/decodetree.rst

Could you put this in docs/devel/, please ?
Our plan for documentation is to try to split it into
multiple manuals, each of which gets a subdirectory of
docs/, and eventually have nothing still in the docs/
directory itself. This doc should definitely be in
devel/ as it's relevant only to QEMU developers.

thanks
-- PMM

Re: [Qemu-devel] [PATCH 2/5] decodetree: Move documentation to docs/decodetree.rst

[Bastian Koppelmann]

On 2/24/19 12:29 AM, Richard Henderson wrote:
> +
> +Argument Sets
> +=============
> +
> +Syntax::
> +
> +  args_def    := '&' identifier ( args_elt )+ ( !extern )?
> +  args_elt    := identifier
> +
> +Each *args_elt* defines an argument within the argument set.
> +Each argument set will be rendered as a C structure "arg_$name"
> +with each of the fields being one of the member arguments.
> +
> +If ``!extern`` is specified, the backing structure is assumed
> +to have been already declared, typically via a second decoder.
> +
> +Argument set examples::
> +
> +  &reg3       ra rb rc
> +  &loadstore  reg base offset
> +


Can we explain why argument sets are useful? This was puzzling for me at 
first. Something like:

This is used to have shared translate functions for instructions with 
differently named arguments. For instance if two variants of the same 
instructions have different sizes in some immediate, we can use a 
argument set to group these immediates together and use the same 
translate functions for both.


Cheers,

Bastian

Re: [Qemu-devel] [PATCH 3/5] decodetree: Ensure build_tree does not include values outside insnmask

[Bastian Koppelmann]

On 2/24/19 12:29 AM, Richard Henderson wrote:
> From: Philippe Mathieu-Daudé <f4bug@amsat.org>
>
> Reproduced with "scripts/decodetree.py /dev/null".
>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   scripts/decodetree.py | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)

Reviewed-by: Bastian Koppelmann <kbastian@mail.uni-paderborn.de>

Cheers,

Bastian

Re: [Qemu-devel] [PATCH 4/5] decodetree: Do not unconditionaly return from Pattern.output_code

[Bastian Koppelmann]

On 2/24/19 12:29 AM, Richard Henderson wrote:
> As a consequence, the 'return false' gets pushed up one level.
>
> This will allow us to perform some other action when the
> translator returns failure.
>
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   scripts/decodetree.py | 7 ++++---
>   1 file changed, 4 insertions(+), 3 deletions(-)


Reviewed-by: Bastian Koppelmann <kbastian@mail.uni-paderborn.de>

Cheers,

Bastian

Re: [Qemu-devel] [PATCH 5/5] decodetree: Allow grouping of overlapping patterns

[Bastian Koppelmann]

On 2/24/19 12:29 AM, Richard Henderson wrote:
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   docs/decodetree.rst   |  58 +++++++++++++++++
>   scripts/decodetree.py | 144 ++++++++++++++++++++++++++++++++++++++----
>   2 files changed, 191 insertions(+), 11 deletions(-)
>
> diff --git a/docs/decodetree.rst b/docs/decodetree.rst


It would be great to have tests in tests/decode/ that exercise 
PatternGroups.

Cheers,

Bastian

[Qemu-devel] [PATCH] test/decode: Add tests for PatternGroups

[Bastian Koppelmann]

this adds one test that supposed to succeed to test deep nesting
of pattern groups which is rarely exercised by targets using decode
tree. The remaining tests exercise various fail conditions.

Signed-off-by: Bastian Koppelmann <kbastian@mail.uni-paderborn.de>
---
These are the tests that came to my mind. It's not complete,but a
start.

 tests/decode/check.sh                        |  6 ++++++
 tests/decode/err_pattern_group_empty.decode  |  6 ++++++
 tests/decode/err_pattern_group_ident1.decode | 10 +++++++++
 tests/decode/err_pattern_group_ident2.decode | 11 ++++++++++
 tests/decode/err_pattern_group_nest1.decode  | 13 ++++++++++++
 tests/decode/succ_pattern_group_nest1.decode | 22 ++++++++++++++++++++
 6 files changed, 68 insertions(+)
 create mode 100644 tests/decode/err_pattern_group_empty.decode
 create mode 100644 tests/decode/err_pattern_group_ident1.decode
 create mode 100644 tests/decode/err_pattern_group_ident2.decode
 create mode 100644 tests/decode/err_pattern_group_nest1.decode
 create mode 100644 tests/decode/succ_pattern_group_nest1.decode

diff --git a/tests/decode/check.sh b/tests/decode/check.sh
index 79a06c37cd..95445a0115 100755
--- a/tests/decode/check.sh
+++ b/tests/decode/check.sh
@@ -15,4 +15,10 @@ for i in err_*.decode; do
     fi
 done
 
+for i in succ_*.decode; do
+    if ! $PYTHON $DECODETREE $i > /dev/null 2> /dev/null; then
+        echo FAIL:$i 1>&2
+    fi
+done
+
 exit $E
diff --git a/tests/decode/err_pattern_group_empty.decode b/tests/decode/err_pattern_group_empty.decode
new file mode 100644
index 0000000000..abbff6b528
--- /dev/null
+++ b/tests/decode/err_pattern_group_empty.decode
@@ -0,0 +1,6 @@
+# This work is licensed under the terms of the GNU LGPL, version 2 or later.
+# See the COPYING.LIB file in the top-level directory.
+
+# empty groups are not allowed
+{
+}
diff --git a/tests/decode/err_pattern_group_ident1.decode b/tests/decode/err_pattern_group_ident1.decode
new file mode 100644
index 0000000000..3e65fab2f9
--- /dev/null
+++ b/tests/decode/err_pattern_group_ident1.decode
@@ -0,0 +1,10 @@
+# This work is licensed under the terms of the GNU LGPL, version 2 or later.
+# See the COPYING.LIB file in the top-level directory.
+
+%sub1 0:8
+
+# Make sure that indentation is enforced
+{
+    top      00000000 00000000 00000000 00000000
+    sub1     00000000 00000000 00000000 ........ %sub1
+}
diff --git a/tests/decode/err_pattern_group_ident2.decode b/tests/decode/err_pattern_group_ident2.decode
new file mode 100644
index 0000000000..bc859233b1
--- /dev/null
+++ b/tests/decode/err_pattern_group_ident2.decode
@@ -0,0 +1,11 @@
+# This work is licensed under the terms of the GNU LGPL, version 2 or later.
+# See the COPYING.LIB file in the top-level directory.
+
+%sub1 0:8
+
+# Make sure that indentation is enforced
+{
+  top      00000000 00000000 00000000 00000000
+  sub1     00000000 00000000 00000000 ........ %sub1
+# comments are suposed to be indented
+}
diff --git a/tests/decode/err_pattern_group_nest1.decode b/tests/decode/err_pattern_group_nest1.decode
new file mode 100644
index 0000000000..92e971c3c5
--- /dev/null
+++ b/tests/decode/err_pattern_group_nest1.decode
@@ -0,0 +1,13 @@
+# This work is licensed under the terms of the GNU LGPL, version 2 or later.
+# See the COPYING.LIB file in the top-level directory.
+
+%sub1 0:8
+%sub2 8:8
+%sub3 16:8
+%sub4 24:8
+
+# Groups with no overlap are supposed to fail
+{
+  top  00000000 00000000 00000000 00000000
+  sub4 ........ ........ ........ ........ %sub1 %sub2 %sub3 %sub4
+}
diff --git a/tests/decode/succ_pattern_group_nest1.decode b/tests/decode/succ_pattern_group_nest1.decode
new file mode 100644
index 0000000000..77b0f48b49
--- /dev/null
+++ b/tests/decode/succ_pattern_group_nest1.decode
@@ -0,0 +1,22 @@
+# This work is licensed under the terms of the GNU LGPL, version 2 or later.
+# See the COPYING.LIB file in the top-level directory.
+
+%sub1 0:8
+%sub2 8:8
+%sub3 16:8
+%sub4 24:7
+
+# Make sure deep netsting works, as few targets will actually exercise it
+{
+  top        00000000 00000000 00000000 00000000
+  {
+    sub1     00000000 00000000 00000000 ........ %sub1
+    {
+      sub2   00000000 00000000 ........ ........ %sub1 %sub2
+      {
+        sub3 00000000 ........ ........ ........ %sub1 %sub2 %sub3
+        sub4 0....... ........ ........ ........ %sub1 %sub2 %sub3 %sub4
+      }
+    }
+  }
+}
-- 
2.21.0

Re: [Qemu-devel] [PATCH] test/decode: Add tests for PatternGroups

[Richard Henderson]

On 2/27/19 4:02 AM, Bastian Koppelmann wrote:
> this adds one test that supposed to succeed to test deep nesting
> of pattern groups which is rarely exercised by targets using decode
> tree. The remaining tests exercise various fail conditions.
> 
> Signed-off-by: Bastian Koppelmann <kbastian@mail.uni-paderborn.de>
> ---
> These are the tests that came to my mind. It's not complete,but a
> start.

Thanks for these.  Will queue for v2.


r~