All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH 0/8] oeqa.utils.metadata: update xml schema
@ 2016-12-28 13:02 Markus Lehtonen
  2016-12-28 13:02 ` [PATCH 1/8] oeqa.utils.metadata: re-organise host distro information Markus Lehtonen
                   ` (7 more replies)
  0 siblings, 8 replies; 15+ messages in thread
From: Markus Lehtonen @ 2016-12-28 13:02 UTC (permalink / raw)
  To: openembedded-core

This patchset extends and modifies the xml format of the test metadata report.
The goal is to make the data more comprehensive in slightly more consistent
format.

The following changes since commit 425afe2484707640ac71194885fdb263e95e9950:

  lib/oe/utils: Drop python2 compatibility code (2016-12-22 08:50:21 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib marquiz/oeqa-metaxml
  http://git.openembedded.org/openembedded-core-contrib/log/?h=marquiz/oeqa-metaxml


Markus Lehtonen (8):
  oeqa.utils.metadata: re-organise host distro information
  oeqa.utils.metadata: re-organise distro information
  oeqa.utils.metadata: drop 'unknown' git data elements
  oeqa.utils.metadata: fix retrieval of git branch and revision
  oeqa.utils.metadata: rename 'revision' to 'commit'
  oeqa.utils.metadata: add commit count information
  oeqa.utils.metadata: have layer name as an attribute in xml
  oeqa.utils.metadata: add bitbake revision information

 meta/lib/oeqa/utils/metadata.py | 81 ++++++++++++++++++++++++++++-------------
 scripts/oe-selftest             |  8 ++--
 2 files changed, 59 insertions(+), 30 deletions(-)

-- 
2.6.6



^ permalink raw reply	[flat|nested] 15+ messages in thread

* [PATCH 1/8] oeqa.utils.metadata: re-organise host distro information
  2016-12-28 13:02 [PATCH 0/8] oeqa.utils.metadata: update xml schema Markus Lehtonen
@ 2016-12-28 13:02 ` Markus Lehtonen
  2016-12-28 13:02 ` [PATCH 2/8] oeqa.utils.metadata: re-organise " Markus Lehtonen
                   ` (6 subsequent siblings)
  7 siblings, 0 replies; 15+ messages in thread
From: Markus Lehtonen @ 2016-12-28 13:02 UTC (permalink / raw)
  To: openembedded-core

Put all host distro data under one <host_distro> element. In addition
take the data directly from /etc/os-release instead of the "lsb API".
The /etc/os-release file is virtually ubiquitous, now, and using its
field names and values provides a more standardized and extensible
format.

[YOCTO #10590]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/lib/oeqa/utils/metadata.py | 26 +++++++++++++++++++++-----
 1 file changed, 21 insertions(+), 5 deletions(-)

diff --git a/meta/lib/oeqa/utils/metadata.py b/meta/lib/oeqa/utils/metadata.py
index 5d8bf84..2316841 100644
--- a/meta/lib/oeqa/utils/metadata.py
+++ b/meta/lib/oeqa/utils/metadata.py
@@ -10,11 +10,22 @@ from collections.abc import MutableMapping
 from xml.dom.minidom import parseString
 from xml.etree.ElementTree import Element, tostring
 
-from oe.lsb import distro_identifier
 from oeqa.utils.commands import runCmd, get_bb_var, get_bb_vars
 
 metadata_vars = ['MACHINE', 'DISTRO', 'DISTRO_VERSION']
 
+def get_os_release():
+    """Get info from /etc/os-release as a dict"""
+    data = OrderedDict()
+    os_release_file = '/etc/os-release'
+    if not os.path.exists(os_release_file):
+        return None
+    with open(os_release_file) as fobj:
+        for line in fobj:
+            key, value = line.split('=', 1)
+            data[key.strip().lower()] = value.strip().strip('"')
+    return data
+
 def metadata_from_bb():
     """ Returns test's metadata as OrderedDict.
 
@@ -27,10 +38,15 @@ def metadata_from_bb():
     data_dict = get_bb_vars(metadata_vars)
     for var in metadata_vars:
         info_dict[var.lower()] = data_dict[var]
-    host_distro= distro_identifier()
-    host_distro, _, host_distro_release = host_distro.partition('-')
-    info_dict['host_distro'] = host_distro
-    info_dict['host_distro_release'] = host_distro_release
+
+    # Host distro information
+    os_release = get_os_release()
+    if os_release:
+        info_dict['host_distro'] = OrderedDict()
+        for key in ('id', 'version_id', 'pretty_name'):
+            if key in os_release:
+                info_dict['host_distro'][key] = os_release[key]
+
     info_dict['layers'] = get_layers(get_bb_var('BBLAYERS'))
     return info_dict
 
-- 
2.6.6



^ permalink raw reply related	[flat|nested] 15+ messages in thread

* [PATCH 2/8] oeqa.utils.metadata: re-organise distro information
  2016-12-28 13:02 [PATCH 0/8] oeqa.utils.metadata: update xml schema Markus Lehtonen
  2016-12-28 13:02 ` [PATCH 1/8] oeqa.utils.metadata: re-organise host distro information Markus Lehtonen
@ 2016-12-28 13:02 ` Markus Lehtonen
  2016-12-28 13:02 ` [PATCH 3/8] oeqa.utils.metadata: drop 'unknown' git data elements Markus Lehtonen
                   ` (5 subsequent siblings)
  7 siblings, 0 replies; 15+ messages in thread
From: Markus Lehtonen @ 2016-12-28 13:02 UTC (permalink / raw)
  To: openembedded-core

Use the same format, based on /etc/os-release, as for host distro
information.

[YOCTO #10590]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/lib/oeqa/utils/metadata.py | 17 ++++++++++-------
 scripts/oe-selftest             |  4 ++--
 2 files changed, 12 insertions(+), 9 deletions(-)

diff --git a/meta/lib/oeqa/utils/metadata.py b/meta/lib/oeqa/utils/metadata.py
index 2316841..df6ed91 100644
--- a/meta/lib/oeqa/utils/metadata.py
+++ b/meta/lib/oeqa/utils/metadata.py
@@ -10,9 +10,7 @@ from collections.abc import MutableMapping
 from xml.dom.minidom import parseString
 from xml.etree.ElementTree import Element, tostring
 
-from oeqa.utils.commands import runCmd, get_bb_var, get_bb_vars
-
-metadata_vars = ['MACHINE', 'DISTRO', 'DISTRO_VERSION']
+from oeqa.utils.commands import runCmd, get_bb_vars
 
 def get_os_release():
     """Get info from /etc/os-release as a dict"""
@@ -35,9 +33,14 @@ def metadata_from_bb():
     info_dict = OrderedDict()
     hostname = runCmd('hostname')
     info_dict['hostname'] = hostname.output
-    data_dict = get_bb_vars(metadata_vars)
-    for var in metadata_vars:
-        info_dict[var.lower()] = data_dict[var]
+    data_dict = get_bb_vars()
+
+    info_dict['machine'] = data_dict['MACHINE']
+
+    # Distro information
+    info_dict['distro'] = {'id': data_dict['DISTRO'],
+                           'version_id': data_dict['DISTRO_VERSION'],
+                           'pretty_name': '%s %s' % (data_dict['DISTRO'], data_dict['DISTRO_VERSION'])}
 
     # Host distro information
     os_release = get_os_release()
@@ -47,7 +50,7 @@ def metadata_from_bb():
             if key in os_release:
                 info_dict['host_distro'][key] = os_release[key]
 
-    info_dict['layers'] = get_layers(get_bb_var('BBLAYERS'))
+    info_dict['layers'] = get_layers(data_dict['BBLAYERS'])
     return info_dict
 
 def metadata_from_data_store(d):
diff --git a/scripts/oe-selftest b/scripts/oe-selftest
index bfcea66..b4d911e 100755
--- a/scripts/oe-selftest
+++ b/scripts/oe-selftest
@@ -624,8 +624,8 @@ def main():
             for layer, values in metadata['layers'].items():
                 layer_info = '%s%-17s = %s:%s\n' % (layer_info, layer,
                               values['branch'], values['revision'])
-            msg = 'Selftest for build %s of %s %s for machine %s on %s\n\n%s' % (
-                   log_prefix[12:], metadata['distro'], metadata['distro_version'],
+            msg = 'Selftest for build %s of %s for machine %s on %s\n\n%s' % (
+                   log_prefix[12:], metadata['distro']['pretty_name'],
                    metadata['machine'], metadata['hostname'], layer_info)
 
             log.debug('Commiting results to local repository')
-- 
2.6.6



^ permalink raw reply related	[flat|nested] 15+ messages in thread

* [PATCH 3/8] oeqa.utils.metadata: drop 'unknown' git data elements
  2016-12-28 13:02 [PATCH 0/8] oeqa.utils.metadata: update xml schema Markus Lehtonen
  2016-12-28 13:02 ` [PATCH 1/8] oeqa.utils.metadata: re-organise host distro information Markus Lehtonen
  2016-12-28 13:02 ` [PATCH 2/8] oeqa.utils.metadata: re-organise " Markus Lehtonen
@ 2016-12-28 13:02 ` Markus Lehtonen
  2017-01-04 15:30   ` Mariano Lopez
  2016-12-28 13:02 ` [PATCH 4/8] oeqa.utils.metadata: fix retrieval of git branch and revision Markus Lehtonen
                   ` (4 subsequent siblings)
  7 siblings, 1 reply; 15+ messages in thread
From: Markus Lehtonen @ 2016-12-28 13:02 UTC (permalink / raw)
  To: openembedded-core

It's better just to not have the xml elements than to have elements with
faux data. One could have git branch named 'unknown', for example.

[YOCTO #10590]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/lib/oeqa/utils/metadata.py | 7 +++----
 scripts/oe-selftest             | 4 ++--
 2 files changed, 5 insertions(+), 6 deletions(-)

diff --git a/meta/lib/oeqa/utils/metadata.py b/meta/lib/oeqa/utils/metadata.py
index df6ed91..a389c6a 100644
--- a/meta/lib/oeqa/utils/metadata.py
+++ b/meta/lib/oeqa/utils/metadata.py
@@ -73,11 +73,10 @@ def get_layers(layers):
         try:
             repo = Repo(layer, search_parent_directories=True)
             revision, branch = repo.head.object.name_rev.split()
-            layer_dict[layer_name]['branch'] = branch
-            layer_dict[layer_name]['revision'] = revision
         except (InvalidGitRepositoryError, NoSuchPathError):
-            layer_dict[layer_name]['branch'] = 'unknown'
-            layer_dict[layer_name]['revision'] = 'unknown'
+            continue
+        layer_dict[layer_name]['branch'] = branch
+        layer_dict[layer_name]['revision'] = revision
     return layer_dict
 
 def write_metadata_file(file_path, metadata):
diff --git a/scripts/oe-selftest b/scripts/oe-selftest
index b4d911e..51c52f2 100755
--- a/scripts/oe-selftest
+++ b/scripts/oe-selftest
@@ -595,7 +595,7 @@ def main():
             r_branches = set(r_branches.replace('origin/', '').split())
             l_branches = {str(branch) for branch in repo.branches}
             branch = '%s/%s/%s' % (metadata['hostname'],
-                                   metadata['layers']['meta']['branch'],
+                                   metadata['layers']['meta'].get('branch', '(nogit)'),
                                    metadata['machine'])
 
             if branch in l_branches:
@@ -623,7 +623,7 @@ def main():
             layer_info = ''
             for layer, values in metadata['layers'].items():
                 layer_info = '%s%-17s = %s:%s\n' % (layer_info, layer,
-                              values['branch'], values['revision'])
+                              values.get('branch', '(nogit)'), values.get('revision', '0'*40))
             msg = 'Selftest for build %s of %s for machine %s on %s\n\n%s' % (
                    log_prefix[12:], metadata['distro']['pretty_name'],
                    metadata['machine'], metadata['hostname'], layer_info)
-- 
2.6.6



^ permalink raw reply related	[flat|nested] 15+ messages in thread

* [PATCH 4/8] oeqa.utils.metadata: fix retrieval of git branch and revision
  2016-12-28 13:02 [PATCH 0/8] oeqa.utils.metadata: update xml schema Markus Lehtonen
                   ` (2 preceding siblings ...)
  2016-12-28 13:02 ` [PATCH 3/8] oeqa.utils.metadata: drop 'unknown' git data elements Markus Lehtonen
@ 2016-12-28 13:02 ` Markus Lehtonen
  2016-12-28 13:02 ` [PATCH 5/8] oeqa.utils.metadata: rename 'revision' to 'commit' Markus Lehtonen
                   ` (3 subsequent siblings)
  7 siblings, 0 replies; 15+ messages in thread
From: Markus Lehtonen @ 2016-12-28 13:02 UTC (permalink / raw)
  To: openembedded-core

Always return a valid branch name, or, '(nobranch)' if the current HEAD
is detached. Also, always return the hash of the commit object that HEAD
is pointing to. Previous code returned an incorrect branch name (or
crashed) e.g. in the case of detached HEAD.

[YOCTO #10590]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/lib/oeqa/utils/metadata.py | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oeqa/utils/metadata.py b/meta/lib/oeqa/utils/metadata.py
index a389c6a..b732d37 100644
--- a/meta/lib/oeqa/utils/metadata.py
+++ b/meta/lib/oeqa/utils/metadata.py
@@ -72,11 +72,13 @@ def get_layers(layers):
         layer_dict[layer_name] = OrderedDict()
         try:
             repo = Repo(layer, search_parent_directories=True)
-            revision, branch = repo.head.object.name_rev.split()
         except (InvalidGitRepositoryError, NoSuchPathError):
             continue
-        layer_dict[layer_name]['branch'] = branch
-        layer_dict[layer_name]['revision'] = revision
+        layer_dict[layer_name]['revision'] = repo.head.commit.hexsha
+        try:
+            layer_dict[layer_name]['branch'] = repo.active_branch.name
+        except TypeError:
+            layer_dict[layer_name]['branch'] = '(nobranch)'
     return layer_dict
 
 def write_metadata_file(file_path, metadata):
-- 
2.6.6



^ permalink raw reply related	[flat|nested] 15+ messages in thread

* [PATCH 5/8] oeqa.utils.metadata: rename 'revision' to 'commit'
  2016-12-28 13:02 [PATCH 0/8] oeqa.utils.metadata: update xml schema Markus Lehtonen
                   ` (3 preceding siblings ...)
  2016-12-28 13:02 ` [PATCH 4/8] oeqa.utils.metadata: fix retrieval of git branch and revision Markus Lehtonen
@ 2016-12-28 13:02 ` Markus Lehtonen
  2016-12-28 13:02 ` [PATCH 6/8] oeqa.utils.metadata: add commit count information Markus Lehtonen
                   ` (2 subsequent siblings)
  7 siblings, 0 replies; 15+ messages in thread
From: Markus Lehtonen @ 2016-12-28 13:02 UTC (permalink / raw)
  To: openembedded-core

Revision is a bit vague and could point to a tag, for example. Git
commit objects are unambiguous and persistent so be explicit that the
element should contain git commit hash.

[YOCTO #10590]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/lib/oeqa/utils/metadata.py | 2 +-
 scripts/oe-selftest             | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oeqa/utils/metadata.py b/meta/lib/oeqa/utils/metadata.py
index b732d37..2f7e8f2 100644
--- a/meta/lib/oeqa/utils/metadata.py
+++ b/meta/lib/oeqa/utils/metadata.py
@@ -74,7 +74,7 @@ def get_layers(layers):
             repo = Repo(layer, search_parent_directories=True)
         except (InvalidGitRepositoryError, NoSuchPathError):
             continue
-        layer_dict[layer_name]['revision'] = repo.head.commit.hexsha
+        layer_dict[layer_name]['commit'] = repo.head.commit.hexsha
         try:
             layer_dict[layer_name]['branch'] = repo.active_branch.name
         except TypeError:
diff --git a/scripts/oe-selftest b/scripts/oe-selftest
index 51c52f2..2092b09 100755
--- a/scripts/oe-selftest
+++ b/scripts/oe-selftest
@@ -623,7 +623,7 @@ def main():
             layer_info = ''
             for layer, values in metadata['layers'].items():
                 layer_info = '%s%-17s = %s:%s\n' % (layer_info, layer,
-                              values.get('branch', '(nogit)'), values.get('revision', '0'*40))
+                              values.get('branch', '(nogit)'), values.get('commit', '0'*40))
             msg = 'Selftest for build %s of %s for machine %s on %s\n\n%s' % (
                    log_prefix[12:], metadata['distro']['pretty_name'],
                    metadata['machine'], metadata['hostname'], layer_info)
-- 
2.6.6



^ permalink raw reply related	[flat|nested] 15+ messages in thread

* [PATCH 6/8] oeqa.utils.metadata: add commit count information
  2016-12-28 13:02 [PATCH 0/8] oeqa.utils.metadata: update xml schema Markus Lehtonen
                   ` (4 preceding siblings ...)
  2016-12-28 13:02 ` [PATCH 5/8] oeqa.utils.metadata: rename 'revision' to 'commit' Markus Lehtonen
@ 2016-12-28 13:02 ` Markus Lehtonen
  2016-12-28 13:02 ` [PATCH 7/8] oeqa.utils.metadata: have layer name as an attribute in xml Markus Lehtonen
  2016-12-28 13:02 ` [PATCH 8/8] oeqa.utils.metadata: add bitbake revision information Markus Lehtonen
  7 siblings, 0 replies; 15+ messages in thread
From: Markus Lehtonen @ 2016-12-28 13:02 UTC (permalink / raw)
  To: openembedded-core

Makes it easier to put the commits into a timeline.

[YOCTO #10590]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/lib/oeqa/utils/metadata.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/utils/metadata.py b/meta/lib/oeqa/utils/metadata.py
index 2f7e8f2..d5cc290 100644
--- a/meta/lib/oeqa/utils/metadata.py
+++ b/meta/lib/oeqa/utils/metadata.py
@@ -63,7 +63,7 @@ def metadata_from_data_store(d):
     pass
 
 def get_layers(layers):
-    """ Returns layer name, branch, and revision as OrderedDict. """
+    """Returns layer information in dict format"""
     from git import Repo, InvalidGitRepositoryError, NoSuchPathError
 
     layer_dict = OrderedDict()
@@ -75,6 +75,7 @@ def get_layers(layers):
         except (InvalidGitRepositoryError, NoSuchPathError):
             continue
         layer_dict[layer_name]['commit'] = repo.head.commit.hexsha
+        layer_dict[layer_name]['commit_count'] = repo.head.commit.count()
         try:
             layer_dict[layer_name]['branch'] = repo.active_branch.name
         except TypeError:
-- 
2.6.6



^ permalink raw reply related	[flat|nested] 15+ messages in thread

* [PATCH 7/8] oeqa.utils.metadata: have layer name as an attribute in xml
  2016-12-28 13:02 [PATCH 0/8] oeqa.utils.metadata: update xml schema Markus Lehtonen
                   ` (5 preceding siblings ...)
  2016-12-28 13:02 ` [PATCH 6/8] oeqa.utils.metadata: add commit count information Markus Lehtonen
@ 2016-12-28 13:02 ` Markus Lehtonen
  2017-01-04 15:43   ` Mariano Lopez
  2016-12-28 13:02 ` [PATCH 8/8] oeqa.utils.metadata: add bitbake revision information Markus Lehtonen
  7 siblings, 1 reply; 15+ messages in thread
From: Markus Lehtonen @ 2016-12-28 13:02 UTC (permalink / raw)
  To: openembedded-core

Have the layer name as an attribute instead of of the name of the
element itself. That is, have <layer name="layer_name"/> instead of
<layer_name/>. A bit better XML design.

[YOCTO #10590]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/lib/oeqa/utils/metadata.py | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oeqa/utils/metadata.py b/meta/lib/oeqa/utils/metadata.py
index d5cc290..6331c21 100644
--- a/meta/lib/oeqa/utils/metadata.py
+++ b/meta/lib/oeqa/utils/metadata.py
@@ -90,12 +90,14 @@ def write_metadata_file(file_path, metadata):
     with open(file_path, 'w') as f:
         f.write(xml_doc.toprettyxml())
 
-def dict_to_XML(tag, dictionary):
+def dict_to_XML(tag, dictionary, **kwargs):
     """ Return XML element converting dicts recursively. """
 
-    elem = Element(tag)
+    elem = Element(tag, **kwargs)
     for key, val in dictionary.items():
-        if isinstance(val, MutableMapping):
+        if tag == 'layers':
+            child = (dict_to_XML('layer', val, name=key))
+        elif isinstance(val, MutableMapping):
             child = (dict_to_XML(key, val))
         else:
             child = Element(key)
-- 
2.6.6



^ permalink raw reply related	[flat|nested] 15+ messages in thread

* [PATCH 8/8] oeqa.utils.metadata: add bitbake revision information
  2016-12-28 13:02 [PATCH 0/8] oeqa.utils.metadata: update xml schema Markus Lehtonen
                   ` (6 preceding siblings ...)
  2016-12-28 13:02 ` [PATCH 7/8] oeqa.utils.metadata: have layer name as an attribute in xml Markus Lehtonen
@ 2016-12-28 13:02 ` Markus Lehtonen
  2017-01-04 15:45   ` Mariano Lopez
  7 siblings, 1 reply; 15+ messages in thread
From: Markus Lehtonen @ 2016-12-28 13:02 UTC (permalink / raw)
  To: openembedded-core

[YOCTO #10590]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/lib/oeqa/utils/metadata.py | 32 +++++++++++++++++++-------------
 1 file changed, 19 insertions(+), 13 deletions(-)

diff --git a/meta/lib/oeqa/utils/metadata.py b/meta/lib/oeqa/utils/metadata.py
index 6331c21..23449fc 100644
--- a/meta/lib/oeqa/utils/metadata.py
+++ b/meta/lib/oeqa/utils/metadata.py
@@ -10,6 +10,8 @@ from collections.abc import MutableMapping
 from xml.dom.minidom import parseString
 from xml.etree.ElementTree import Element, tostring
 
+from git import Repo, InvalidGitRepositoryError, NoSuchPathError
+
 from oeqa.utils.commands import runCmd, get_bb_vars
 
 def get_os_release():
@@ -51,6 +53,7 @@ def metadata_from_bb():
                 info_dict['host_distro'][key] = os_release[key]
 
     info_dict['layers'] = get_layers(data_dict['BBLAYERS'])
+    info_dict['bitbake'] = git_rev_info(os.path.dirname(bb.__file__))
     return info_dict
 
 def metadata_from_data_store(d):
@@ -62,24 +65,27 @@ def metadata_from_data_store(d):
     # be useful when running within bitbake.
     pass
 
+def git_rev_info(path):
+    """Get git revision information as a dict"""
+    info = OrderedDict()
+    try:
+        repo = Repo(path, search_parent_directories=True)
+    except (InvalidGitRepositoryError, NoSuchPathError):
+        return info
+    info['commit'] = repo.head.commit.hexsha
+    info['commit_count'] = repo.head.commit.count()
+    try:
+        info['branch'] = repo.active_branch.name
+    except TypeError:
+        info['branch'] = '(nobranch)'
+    return info
+
 def get_layers(layers):
     """Returns layer information in dict format"""
-    from git import Repo, InvalidGitRepositoryError, NoSuchPathError
-
     layer_dict = OrderedDict()
     for layer in layers.split():
         layer_name = os.path.basename(layer)
-        layer_dict[layer_name] = OrderedDict()
-        try:
-            repo = Repo(layer, search_parent_directories=True)
-        except (InvalidGitRepositoryError, NoSuchPathError):
-            continue
-        layer_dict[layer_name]['commit'] = repo.head.commit.hexsha
-        layer_dict[layer_name]['commit_count'] = repo.head.commit.count()
-        try:
-            layer_dict[layer_name]['branch'] = repo.active_branch.name
-        except TypeError:
-            layer_dict[layer_name]['branch'] = '(nobranch)'
+        layer_dict[layer_name] = git_rev_info(layer)
     return layer_dict
 
 def write_metadata_file(file_path, metadata):
-- 
2.6.6



^ permalink raw reply related	[flat|nested] 15+ messages in thread

* Re: [PATCH 3/8] oeqa.utils.metadata: drop 'unknown' git data elements
  2016-12-28 13:02 ` [PATCH 3/8] oeqa.utils.metadata: drop 'unknown' git data elements Markus Lehtonen
@ 2017-01-04 15:30   ` Mariano Lopez
  2017-01-05  9:19     ` Markus Lehtonen
  0 siblings, 1 reply; 15+ messages in thread
From: Mariano Lopez @ 2017-01-04 15:30 UTC (permalink / raw)
  To: Markus Lehtonen, openembedded-core



On 28/12/16 07:02, Markus Lehtonen wrote:
> It's better just to not have the xml elements than to have elements with
> faux data. One could have git branch named 'unknown', for example.
>
>

I don't think is a good idea to completely remove the layer from the
metadata when such layer is not a repository, it would remove valuable
information for later analysis. And if you ask would do such a thing, I
would, I have my own debug layers that can mess with testing.


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH 7/8] oeqa.utils.metadata: have layer name as an attribute in xml
  2016-12-28 13:02 ` [PATCH 7/8] oeqa.utils.metadata: have layer name as an attribute in xml Markus Lehtonen
@ 2017-01-04 15:43   ` Mariano Lopez
  2017-01-05  9:35     ` Markus Lehtonen
  0 siblings, 1 reply; 15+ messages in thread
From: Mariano Lopez @ 2017-01-04 15:43 UTC (permalink / raw)
  To: Markus Lehtonen, openembedded-core



On 28/12/16 07:02, Markus Lehtonen wrote:
> -def dict_to_XML(tag, dictionary):
> +def dict_to_XML(tag, dictionary, **kwargs):
>      """ Return XML element converting dicts recursively. """
>  
> -    elem = Element(tag)
> +    elem = Element(tag, **kwargs)
>      for key, val in dictionary.items():
> -        if isinstance(val, MutableMapping):
> +        if tag == 'layers':
> +            child = (dict_to_XML('layer', val, name=key))
> +        elif isinstance(val, MutableMapping):
>              child = (dict_to_XML(key, val))
>          else:
>              child = Element(key)

I was thinking that this function would be more generic, adding a
comparison to a tag name surely won't keep the function generic, is
there another way to implement this keeping this function generic enough?


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH 8/8] oeqa.utils.metadata: add bitbake revision information
  2016-12-28 13:02 ` [PATCH 8/8] oeqa.utils.metadata: add bitbake revision information Markus Lehtonen
@ 2017-01-04 15:45   ` Mariano Lopez
  2017-01-05  9:40     ` Markus Lehtonen
  0 siblings, 1 reply; 15+ messages in thread
From: Mariano Lopez @ 2017-01-04 15:45 UTC (permalink / raw)
  To: Markus Lehtonen, openembedded-core



On 28/12/16 07:02, Markus Lehtonen wrote:
> [YOCTO #10590]
>
> Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
> ---
>  meta/lib/oeqa/utils/metadata.py | 32 +++++++++++++++++++-------------
>  1 file changed, 19 insertions(+), 13 deletions(-)
>
> diff --git a/meta/lib/oeqa/utils/metadata.py b/meta/lib/oeqa/utils/metadata.py
> index 6331c21..23449fc 100644
> --- a/meta/lib/oeqa/utils/metadata.py
> +++ b/meta/lib/oeqa/utils/metadata.py
> @@ -10,6 +10,8 @@ from collections.abc import MutableMapping
>  from xml.dom.minidom import parseString
>  from xml.etree.ElementTree import Element, tostring
>  
> +from git import Repo, InvalidGitRepositoryError, NoSuchPathError
> +

It seems not every user running selftest appreciated the requirement of
gitpython, so there was a patch to keep this dependency out. This will
introduce the need of gitpython again.


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH 3/8] oeqa.utils.metadata: drop 'unknown' git data elements
  2017-01-04 15:30   ` Mariano Lopez
@ 2017-01-05  9:19     ` Markus Lehtonen
  0 siblings, 0 replies; 15+ messages in thread
From: Markus Lehtonen @ 2017-01-05  9:19 UTC (permalink / raw)
  To: Mariano Lopez, openembedded-core

On 04/01/2017, 17.30, "Mariano Lopez" <mariano.lopez@linux.intel.com>
wrote:
>On 28/12/16 07:02, Markus Lehtonen wrote:
>> It's better just to not have the xml elements than to have elements with
>> faux data. One could have git branch named 'unknown', for example.
>>
>>
>
>I don't think is a good idea to completely remove the layer from the
>metadata when such layer is not a repository, it would remove valuable
>information for later analysis. And if you ask would do such a thing, I
>would, I have my own debug layers that can mess with testing.

I'm not removing the entire layer. I'm just removing the
<branch>unknown</branch> and <revisio>unknown</revision> elements. So the
layer elements are still there.

Thanks,
   Markus




^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH 7/8] oeqa.utils.metadata: have layer name as an attribute in xml
  2017-01-04 15:43   ` Mariano Lopez
@ 2017-01-05  9:35     ` Markus Lehtonen
  0 siblings, 0 replies; 15+ messages in thread
From: Markus Lehtonen @ 2017-01-05  9:35 UTC (permalink / raw)
  To: Mariano Lopez, openembedded-core

On 04/01/2017, 17.43, "Mariano Lopez" <mariano.lopez@linux.intel.com>
wrote:
>
>On 28/12/16 07:02, Markus Lehtonen wrote:
>> -def dict_to_XML(tag, dictionary):
>> +def dict_to_XML(tag, dictionary, **kwargs):
>>      """ Return XML element converting dicts recursively. """
>>  
>> -    elem = Element(tag)
>> +    elem = Element(tag, **kwargs)
>>      for key, val in dictionary.items():
>> -        if isinstance(val, MutableMapping):
>> +        if tag == 'layers':
>> +            child = (dict_to_XML('layer', val, name=key))
>> +        elif isinstance(val, MutableMapping):
>>              child = (dict_to_XML(key, val))
>>          else:
>>              child = Element(key)
>
>I was thinking that this function would be more generic, adding a
>comparison to a tag name surely won't keep the function generic, is
>there another way to implement this keeping this function generic enough?

I don't think it's possible to make a nice generic json to xml converter.
The formats are just so different. I think the current method just
generates bad xml. In the future, it would probably be better to just get
rid of the json-to-xml conversion completely and make metadata_from_bb()
to directly return an ElementTree.

Thanks, 
  Markus




^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH 8/8] oeqa.utils.metadata: add bitbake revision information
  2017-01-04 15:45   ` Mariano Lopez
@ 2017-01-05  9:40     ` Markus Lehtonen
  0 siblings, 0 replies; 15+ messages in thread
From: Markus Lehtonen @ 2017-01-05  9:40 UTC (permalink / raw)
  To: Mariano Lopez, openembedded-core

On 04/01/2017, 17.45, "Mariano Lopez" <mariano.lopez@linux.intel.com>
wrote:
>On 28/12/16 07:02, Markus Lehtonen wrote:
>> [YOCTO #10590]
>>
>> Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
>> ---
>>  meta/lib/oeqa/utils/metadata.py | 32 +++++++++++++++++++-------------
>>  1 file changed, 19 insertions(+), 13 deletions(-)
>>
>> diff --git a/meta/lib/oeqa/utils/metadata.py
>>b/meta/lib/oeqa/utils/metadata.py
>> index 6331c21..23449fc 100644
>> --- a/meta/lib/oeqa/utils/metadata.py
>> +++ b/meta/lib/oeqa/utils/metadata.py
>> @@ -10,6 +10,8 @@ from collections.abc import MutableMapping
>>  from xml.dom.minidom import parseString
>>  from xml.etree.ElementTree import Element, tostring
>>  
>> +from git import Repo, InvalidGitRepositoryError, NoSuchPathError
>> +
>
>It seems not every user running selftest appreciated the requirement of
>gitpython, so there was a patch to keep this dependency out. This will
>introduce the need of gitpython again.

Yes. I don't know why I moved this import to module level in the first
place. A corrected version of the patch in now found  in my
oe-core-contrib branch
(git://git.openembedded.org/openembedded-core-contrib marquiz/oeqa-metaxml)


Thanks,
  Markus




^ permalink raw reply	[flat|nested] 15+ messages in thread

end of thread, other threads:[~2017-01-05  9:40 UTC | newest]

Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-12-28 13:02 [PATCH 0/8] oeqa.utils.metadata: update xml schema Markus Lehtonen
2016-12-28 13:02 ` [PATCH 1/8] oeqa.utils.metadata: re-organise host distro information Markus Lehtonen
2016-12-28 13:02 ` [PATCH 2/8] oeqa.utils.metadata: re-organise " Markus Lehtonen
2016-12-28 13:02 ` [PATCH 3/8] oeqa.utils.metadata: drop 'unknown' git data elements Markus Lehtonen
2017-01-04 15:30   ` Mariano Lopez
2017-01-05  9:19     ` Markus Lehtonen
2016-12-28 13:02 ` [PATCH 4/8] oeqa.utils.metadata: fix retrieval of git branch and revision Markus Lehtonen
2016-12-28 13:02 ` [PATCH 5/8] oeqa.utils.metadata: rename 'revision' to 'commit' Markus Lehtonen
2016-12-28 13:02 ` [PATCH 6/8] oeqa.utils.metadata: add commit count information Markus Lehtonen
2016-12-28 13:02 ` [PATCH 7/8] oeqa.utils.metadata: have layer name as an attribute in xml Markus Lehtonen
2017-01-04 15:43   ` Mariano Lopez
2017-01-05  9:35     ` Markus Lehtonen
2016-12-28 13:02 ` [PATCH 8/8] oeqa.utils.metadata: add bitbake revision information Markus Lehtonen
2017-01-04 15:45   ` Mariano Lopez
2017-01-05  9:40     ` Markus Lehtonen

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.