[PATCH 00/11] combo-layer enhancements

[PATCH 00/11] combo-layer enhancements

[Patrick Ohly]

This replaces the four individual patches sent recently. It addresses
the comments that Paul had, fixes one issue in file_exclude filtering
of patches when combined with dest_dir, and last but not least,
introduces full history importing during "init".

For the separate fixes see https://github.com/pohly/openembedded-core/commits/pending

The following changes since commit fb29441216435b9bae47ca9cd42db5a6b1fe77d8:

  oeqa/parselogs: Skip hda opcode errors (2015-03-12 12:49:22 +0000)

are available in the git repository at:

  git://github.com/pohly/openembedded-core combo-layer
  https://github.com/pohly/openembedded-core/tree/combo-layer

Patrick Ohly (11):
  combo-layer: let user choose where properties get updated
  combo-layer: make Signed-off-by optional
  combo-layer: runcmd() with separate output
  combo-layer: exclude files
  combo-layer: update() also instance property
  combo-layer: init with full history
  combo-layer: combine trees via replacement objects
  combo-layer: partial import for '--history init'
  combo-layer-hook-default.sh: avoid duplicating prefix
  combo-layer-hook-default.sh: beware of embedded patches
  combo-layer-hook-default.sh: handle patches without Signed-off-by

 scripts/combo-layer                 | 277 ++++++++++++++++++++++++++++++++++--
 scripts/combo-layer-hook-default.sh |  11 +-
 scripts/combo-layer.conf.example    |  37 +++++
 3 files changed, 311 insertions(+), 14 deletions(-)

-- 
2.1.4

[PATCH 01/11] combo-layer: let user choose where properties get updated

[Patrick Ohly]

When updating a property (at the moment, only 'last_revision'),
first check whether the user already added it to the main config.
If so, update there even if there is a local config.

This way, 'last_revision' can be shared between developers
as part of the repository while still configuring per-developer
repo paths outside of the git repository in a local config.

An example of a repository which is set up like that is tizen-distro,
with instructions for such a collaborative maintenance found here:
https://review.tizen.org/git/?p=scm/bb/tizen-distro.git;a=blob;f=README;hb=refs/heads/tizen-ivi

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 scripts/combo-layer | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/combo-layer b/scripts/combo-layer
index 851003d..71fa7b127 100755
--- a/scripts/combo-layer
+++ b/scripts/combo-layer
@@ -109,7 +109,9 @@ class Configuration(object):
                         readsection(self.localparser, section, repo)
 
     def update(self, repo, option, value, initmode=False):
-        if self.localparser:
+        # If the main config has the option already, that is what we
+        # are expected to modify.
+        if self.localparser and not self.parser.has_option(repo, option):
             parser = self.localparser
             section = "%s|%s" % (repo, self.combobranch)
             conffile = self.localconffile
-- 
2.1.4

[PATCH 02/11] combo-layer: make Signed-off-by optional

[Patrick Ohly]

It depends on the diligence of the person running the combo-layer tool
whether the Signed-off-by line added to each commit actually indicates
that the person was involved in validating the change.

When the import is purely automatic, it is better to not add the line,
because the history is more useful without it (searching for the person
really only lists changes he or she was involved with) and it would
be a false statement.

The 'signoff' property can be set per repository, like every
other property. But setting it in the special [DEFAULT] section
is more useful, so that is what the example shows.
---
 scripts/combo-layer              |  7 ++++++-
 scripts/combo-layer.conf.example | 10 ++++++++++
 2 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/scripts/combo-layer b/scripts/combo-layer
index 71fa7b127..62f2cf8 100755
--- a/scripts/combo-layer
+++ b/scripts/combo-layer
@@ -68,6 +68,11 @@ class Configuration(object):
                 if value.startswith("@"):
                     self.repos[repo][name] = eval(value.strip("@"))
                 else:
+                    # Apply special type transformations for some properties.
+                    # Type matches the RawConfigParser.get*() methods.
+                    types = {'signoff': 'boolean'}
+                    if name in types:
+                        value = getattr(parser, 'get' + types[name])(section, name)
                     self.repos[repo][name] = value
 
         logger.debug("Loading config file %s" % self.conffile)
@@ -482,7 +487,7 @@ def apply_patchlist(conf, repos):
                 if os.path.getsize(patchfile) == 0:
                     logger.info("(skipping %d/%d %s - no changes)" % (i, linecount, patchdisp))
                 else:
-                    cmd = "git am --keep-cr -s -p1 %s" % patchfile
+                    cmd = "git am --keep-cr %s-p1 %s" % ('-s ' if repo.get('signoff', True) else '', patchfile)
                     logger.info("Applying %d/%d: %s" % (i, linecount, patchdisp))
                     try:
                         runcmd(cmd)
diff --git a/scripts/combo-layer.conf.example b/scripts/combo-layer.conf.example
index 010a692..427c1b3 100644
--- a/scripts/combo-layer.conf.example
+++ b/scripts/combo-layer.conf.example
@@ -1,7 +1,17 @@
 # combo-layer example configuration file
 
+# Default values for all sections.
+[DEFAULT]
+
+# Add 'Signed-off-by' to all commits that get imported automatically.
+signoff = True
+
 # component name
 [bitbake]
+
+# Override signedoff default above (not very useful, but possible).
+signoff = False
+
 # mandatory options
 # git upstream uri
 src_uri = git://git.openembedded.org/bitbake
-- 
2.1.4

[PATCH 03/11] combo-layer: runcmd() with separate output

[Patrick Ohly]

Allow the caller to specify a separate output stream. stderr is always
a temporary file opened by runcmd(), so read from that to capture
output for error reporting *and* the return value.

The reasoning for the latter is a) that this preserves the traditional
behavior when out=None and b) if the caller wants the content of
stdout, it can read from the stream itself, which is not possible for
the temporary stderr.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 scripts/combo-layer | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)

diff --git a/scripts/combo-layer b/scripts/combo-layer
index 62f2cf8..3ee9eb2 100755
--- a/scripts/combo-layer
+++ b/scripts/combo-layer
@@ -152,23 +152,27 @@ class Configuration(object):
             logger.error("ERROR: patchutils package is missing, please install it (e.g. # apt-get install patchutils)")
             sys.exit(1)
 
-def runcmd(cmd,destdir=None,printerr=True):
+def runcmd(cmd,destdir=None,printerr=True,out=None):
     """
         execute command, raise CalledProcessError if fail
         return output if succeed
     """
     logger.debug("run cmd '%s' in %s" % (cmd, os.getcwd() if destdir is None else destdir))
-    out = os.tmpfile()
+    if not out:
+        out = os.tmpfile()
+        err = out
+    else:
+        err = os.tmpfile()
     try:
-        subprocess.check_call(cmd, stdout=out, stderr=out, cwd=destdir, shell=True)
+        subprocess.check_call(cmd, stdout=out, stderr=err, cwd=destdir, shell=isinstance(cmd, str))
     except subprocess.CalledProcessError,e:
-        out.seek(0)
+        err.seek(0)
         if printerr:
-            logger.error("%s" % out.read())
+            logger.error("%s" % err.read())
         raise e
 
-    out.seek(0)
-    output = out.read()
+    err.seek(0)
+    output = err.read()
     logger.debug("output: %s" % output )
     return output
 
-- 
2.1.4

[PATCH 04/11] combo-layer: exclude files

[Patrick Ohly]

Some combined repos intentionally do not include certain files.
For example, Poky does not include bitbake's setup files and
OE-core's sample files under meta/conf.

When these files get modified in the upstream repository, applying the
patches fails and requires manual intervention. That is merely a
nuisance for someone familiar with the problem, but a real show
stopper when having the import run automatically or by someone less
experienced.

Therefore this change introduces "file_exclude", a new per-repo list
of file patterns which removes all matching files when initializing or
updating a combined repository. Because fnmatch is used under the hood
to match full path strings, removing entire directories must be done
with a pattern ending in a '/*' (in contrast to file_filter).

For Poky, the additional configuration looks like this:

[bitbake]
...
file_exclude = classes/base.bbclass
	conf/bitbake.conf
	.gitignore
	MANIFEST.in
	setup.py
	TODO

[openembedded-core]
...
file_exclude = meta/conf/bblayers.conf.sample
	meta/conf/local.conf.sample
	meta/conf/local.conf.sample.extended
	meta/conf/site.conf.sample

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 scripts/combo-layer              | 42 +++++++++++++++++++++++++++++++++++++++-
 scripts/combo-layer.conf.example | 14 ++++++++++++++
 2 files changed, 55 insertions(+), 1 deletion(-)

diff --git a/scripts/combo-layer b/scripts/combo-layer
index 3ee9eb2..8db5d7e 100755
--- a/scripts/combo-layer
+++ b/scripts/combo-layer
@@ -20,6 +20,7 @@
 # with this program; if not, write to the Free Software Foundation, Inc.,
 # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
+import fnmatch
 import os, sys
 import optparse
 import logging
@@ -211,7 +212,18 @@ def action_init(conf, args):
             else:
                 extract_dir = os.getcwd()
             file_filter = repo.get('file_filter', "")
-            runcmd("git archive %s | tar -x -C %s %s" % (initialrev, extract_dir, file_filter), ldir)
+            files = runcmd("git archive %s | tar -x -v -C %s %s" % (initialrev, extract_dir, file_filter), ldir)
+            exclude_patterns = repo.get('file_exclude', '').split()
+            if exclude_patterns:
+                # Implement file removal by letting tar create the
+                # file and then deleting it in the file system
+                # again. Uses the list of files created by tar (easier
+                # than walking the tree).
+                for file in files.split('\n'):
+                    for pattern in exclude_patterns:
+                        if fnmatch.fnmatch(file, pattern):
+                            os.unlink(os.path.join(extract_dir, file))
+                            break
             if not lastrev:
                 lastrev = runcmd("git rev-parse %s" % initialrev, ldir).strip()
                 conf.update(name, "last_revision", lastrev, initmode=True)
@@ -426,6 +438,34 @@ def action_update(conf, args):
                 runcmd("%s %s %s %s" % (repo['hook'], patch, revlist[count], name))
                 count=count-1
 
+        # Step 3a: Filter out unwanted files and patches.
+        exclude = repo.get('file_exclude', '')
+        if exclude:
+            filter = ['filterdiff', '-p1']
+            for path in exclude.split():
+                filter.append('-x')
+                filter.append('%s/%s' % (dest_dir, path) if dest_dir else path)
+            for patch in patchlist[:]:
+                filtered = patch + '.tmp'
+                with open(filtered, 'w') as f:
+                    runcmd(filter + [patch], out=f)
+                # Now check for empty patches.
+                if runcmd(['filterdiff', '--list', filtered]):
+                    # Possibly modified.
+                    os.unlink(patch)
+                    os.rename(filtered, patch)
+                else:
+                    # Empty, ignore it. Must also remove from revlist.
+                    with open(patch, 'r') as f:
+                        fromline = f.readline()
+                    m = re.match(r'''^From ([0-9a-fA-F]+) .*\n''', fromline)
+                    rev = m.group(1)
+                    logger.debug('skipping empty patch %s = %s' % (patch, rev))
+                    os.unlink(patch)
+                    os.unlink(filtered)
+                    patchlist.remove(patch)
+                    revlist.remove(rev)
+
         # Step 4: write patch list and revision list to file, for user to edit later
         patchlist_file = os.path.join(os.getcwd(), patch_dir, "patchlist-%s" % name)
         repo['patchlist'] = patchlist_file
diff --git a/scripts/combo-layer.conf.example b/scripts/combo-layer.conf.example
index 427c1b3..38bc53c 100644
--- a/scripts/combo-layer.conf.example
+++ b/scripts/combo-layer.conf.example
@@ -42,6 +42,20 @@ last_revision =
 #   file_filter = src/*.c : only include the src *.c file
 #   file_filter = src/main.c src/Makefile.am : only include these two files
 
+# file_exclude: filter out these file(s)
+# file_exclude = [path] [path] ...
+#
+# Each entry must match a file name. In contrast do file_filter, matching
+# a directory has no effect. To achieve that, use append a * wildcard
+# at the end.
+#
+# Wildcards are applied to the complete path and also match slashes.
+#
+# example:
+#   file_exclude = src/foobar/*  : exclude everything under src/foobar
+#   file_exclude = src/main.c : filter out main.c after including it with file_filter = src/*.c
+#   file_exclude = *~ : exclude backup files
+
 # hook: if provided, the tool will call the hook to process the generated
 #     patch from upstream, and then apply the modified patch to the combo
 #     repo.
-- 
2.1.4

[PATCH 05/11] combo-layer: update() also instance property

[Patrick Ohly]

The Configuration class mirrors all properties in local hashes.
When updating the configuration, also update these properties
to remain consistent.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 scripts/combo-layer | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/combo-layer b/scripts/combo-layer
index 8db5d7e..8ed9be8 100755
--- a/scripts/combo-layer
+++ b/scripts/combo-layer
@@ -130,6 +130,7 @@ class Configuration(object):
         parser.set(section, option, value)
         with open(conffile, "w") as f:
             parser.write(f)
+        self.repos[repo][option] = value
 
     def sanity_check(self, initmode=False):
         required_options=["src_uri", "local_repo_dir", "dest_dir", "last_revision"]
-- 
2.1.4

[PATCH 06/11] combo-layer: init with full history

[Patrick Ohly]

The new --history parameter enables a new mode in "combo-layer init"
where it copies the entire history of the components into the new
combined repository. This also imports merge commits.

Moving into a destination directory and applying commit hooks
is done via "git filter-branch" of the upstream branch. File
filtering uses the same code as before and just applies it
to that filtered branch to create the final commit which
then gets merged into the master branch of the new repository.

When multiple components are involved, they all get merged
into a single commit with an octopus merge. This depends
on a common ancestor, which is grafted onto the filtered
branches via .git/info/grafts.

These grafts are currently left in place. However, they do not get
pushed, so the local view on the entire history (all branches rooted
in the initial, empty commit, temporarily diverging and then
converging) is not the same as what others will see (branches starting
independently and converging). Perhaps "git replace" should be used
instead.

The final commit needs to be done manually, as before. A commit
message with some tracking information is ready for use as-is. This
information should be sufficient to implement also "combo-layer
update" using this approach, if desired. The advantage would be that
merge commits with conflict resolution would not longer break
the update.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 scripts/combo-layer | 187 ++++++++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 174 insertions(+), 13 deletions(-)

diff --git a/scripts/combo-layer b/scripts/combo-layer
index 8ed9be8..d11274e 100755
--- a/scripts/combo-layer
+++ b/scripts/combo-layer
@@ -25,6 +25,7 @@ import os, sys
 import optparse
 import logging
 import subprocess
+import tempfile
 import ConfigParser
 import re
 from collections import OrderedDict
@@ -190,6 +191,11 @@ def action_init(conf, args):
             subprocess.check_call("git clone %s %s" % (conf.repos[name]['src_uri'], ldir), shell=True)
     if not os.path.exists(".git"):
         runcmd("git init")
+        if conf.history:
+            # Need a common ref for all trees.
+            runcmd('git commit -m "initial empty commit" --allow-empty')
+            startrev = runcmd('git rev-parse master').strip()
+
         for name in conf.repos:
             repo = conf.repos[name]
             ldir = repo['local_repo_dir']
@@ -205,6 +211,25 @@ def action_init(conf, args):
                 lastrev = None
                 initialrev = branch
                 logger.info("Copying data from %s..." % name)
+            # Sanity check initialrev and turn it into hash (required for copying history,
+            # because resolving a name ref only works in the component repo).
+            rev = runcmd('git rev-parse %s' % initialrev, ldir).strip()
+            if rev != initialrev:
+                try:
+                    refs = runcmd('git show-ref -s %s' % initialrev, ldir).split('\n')
+                    if len(set(refs)) > 1:
+                        # Happens for example when configured to track
+                        # "master" and there is a refs/heads/master. The
+                        # traditional behavior from "git archive" (preserved
+                        # here) it to choose the first one. This might not be
+                        # intended, so at least warn about it.
+                        logger.warn("%s: initial revision '%s' not unique, picking result of rev-parse = %s" %
+                                    (name, initialrev, refs[0]))
+                        initialrev = rev
+                except:
+                    # show-ref fails for hashes. Skip the sanity warning in that case.
+                    pass
+                initialrev = rev
             dest_dir = repo['dest_dir']
             if dest_dir and dest_dir != ".":
                 extract_dir = os.path.join(os.getcwd(), dest_dir)
@@ -213,22 +238,155 @@ def action_init(conf, args):
             else:
                 extract_dir = os.getcwd()
             file_filter = repo.get('file_filter', "")
-            files = runcmd("git archive %s | tar -x -v -C %s %s" % (initialrev, extract_dir, file_filter), ldir)
             exclude_patterns = repo.get('file_exclude', '').split()
-            if exclude_patterns:
-                # Implement file removal by letting tar create the
-                # file and then deleting it in the file system
-                # again. Uses the list of files created by tar (easier
-                # than walking the tree).
-                for file in files.split('\n'):
-                    for pattern in exclude_patterns:
-                        if fnmatch.fnmatch(file, pattern):
-                            os.unlink(os.path.join(extract_dir, file))
-                            break
+            def copy_selected_files(initialrev, extract_dir, file_filter, exclude_patterns, ldir,
+                                    subdir=""):
+                # When working inside a filtered branch which had the
+                # files already moved, we need to prepend the
+                # subdirectory to all filters, otherwise they would
+                # not match.
+                if subdir:
+                    file_filter = ' '.join([subdir + '/' + x for x in file_filter.split()])
+                    exclude_patterns = [subdir + '/' + x for x in exclude_patterns]
+                # To handle both cases, we cd into the target
+                # directory and optionally tell tar to strip the path
+                # prefix when the files were already moved.
+                subdir_components = len(os.path.normpath(subdir).split(os.path.sep)) if subdir else 0
+                strip=('--strip-components=%d' % subdir_components) if subdir else ''
+                # TODO: file_filter wild cards do not work (and haven't worked before either), because
+                # a) GNU tar requires a --wildcards parameter before turning on wild card matching.
+                # b) The semantic is not as intendend (src/*.c also matches src/foo/bar.c,
+                #    in contrast to the other use of file_filter as parameter of "git archive"
+                #    where it only matches .c files directly in src).
+                files = runcmd("git archive %s %s | tar -x -v %s -C %s %s" %
+                               (initialrev, subdir,
+                                strip, extract_dir, file_filter),
+                               ldir)
+                if exclude_patterns:
+                    # Implement file removal by letting tar create the
+                    # file and then deleting it in the file system
+                    # again. Uses the list of files created by tar (easier
+                    # than walking the tree).
+                    for file in files.split('\n'):
+                        for pattern in exclude_patterns:
+                            if fnmatch.fnmatch(file, pattern):
+                                os.unlink(os.path.join(*([extract_dir] + ['..'] * subdir_components + [file])))
+                                break
+
+            if not conf.history:
+                copy_selected_files(initialrev, extract_dir, file_filter, exclude_patterns, ldir)
+            else:
+                # First fetch remote history into local repository.
+                # We need a ref for that, so ensure that there is one.
+                refname = "combo-layer-init-%s" % name
+                runcmd("git branch -f %s %s" % (refname, initialrev), ldir)
+                runcmd("git fetch %s %s" % (ldir, refname))
+                runcmd("git branch -D %s" % refname, ldir)
+                # Make that the head revision.
+                runcmd("git checkout -b %s %s" % (name, initialrev))
+                # Optional: rewrite history to change commit messages or to move files.
+                if 'hook' in repo or dest_dir and dest_dir != ".":
+                    filter_branch = ['git', 'filter-branch', '--force']
+                    with tempfile.NamedTemporaryFile() as hookwrapper:
+                        if 'hook' in repo:
+                            # Create a shell script wrapper around the original hook that
+                            # can be used by git filter-branch. Hook may or may not have
+                            # an absolute path.
+                            hook = repo['hook']
+                            hook = os.path.join(os.path.dirname(conf.conffile), '..', hook)
+                            # The wrappers turns the commit message
+                            # from stdin into a fake patch header.
+                            # This is good enough for changing Subject
+                            # and commit msg body with normal
+                            # combo-layer hooks.
+                            hookwrapper.write('''set -e
+tmpname=$(mktemp)
+trap "rm $tmpname" EXIT
+echo -n 'Subject: [PATCH] ' >>$tmpname
+cat >>$tmpname
+if ! [ $(tail -c 1 $tmpname | od -A n -t x1) == '0a' ]; then
+    echo >>$tmpname
+fi
+echo '---' >>$tmpname
+%s $tmpname $GIT_COMMIT %s
+tail -c +18 $tmpname | head -c -4
+''' % (hook, name))
+                            hookwrapper.flush()
+                            filter_branch.extend(['--msg-filter', 'bash %s' % hookwrapper.name])
+                        if dest_dir and dest_dir != ".":
+                            parent = os.path.dirname(dest_dir)
+                            if not parent:
+                                parent = '.'
+                            # May run outside of the current directory, so do not assume that .git exists.
+                            filter_branch.extend(['--tree-filter', 'mkdir -p .git/tmptree && mv $(ls -1 -a | grep -v -e ^.git$ -e ^.$ -e ^..$) .git/tmptree && mkdir -p %s && mv .git/tmptree %s' % (parent, dest_dir)])
+                        filter_branch.append('HEAD')
+                        runcmd(filter_branch)
+                        runcmd('git update-ref -d refs/original/refs/heads/%s' % name)
+                repo['rewritten_revision'] = runcmd('git rev-parse HEAD').strip()
+                repo['stripped_revision'] = repo['rewritten_revision']
+                # Optional filter files: remove everything and re-populate using the normal filtering code.
+                # Override any potential .gitignore.
+                if file_filter or exclude_patterns:
+                    runcmd('git rm -rf .')
+                    if not os.path.exists(extract_dir):
+                        os.makedirs(extract_dir)
+                    copy_selected_files('HEAD', extract_dir, file_filter, exclude_patterns, '.',
+                                        subdir=dest_dir if dest_dir and dest_dir != '.' else '')
+                    runcmd('git add --all --force .')
+                    if runcmd('git status --porcelain'):
+                        # Something to commit.
+                        runcmd(['git', 'commit', '-m',
+                                '''%s: select file subset
+
+Files from the component repository were chosen based on
+the following filters:
+file_filter = %s
+file_exclude = %s''' % (name, file_filter or '<empty>', repo.get('file_exclude', '<empty>'))])
+                        repo['stripped_revision'] = runcmd('git rev-parse HEAD').strip()
+
             if not lastrev:
-                lastrev = runcmd("git rev-parse %s" % initialrev, ldir).strip()
+                lastrev = runcmd('git rev-parse %s' % initialrev, ldir).strip()
                 conf.update(name, "last_revision", lastrev, initmode=True)
-        runcmd("git add .")
+
+        if not conf.history:
+            runcmd("git add .")
+        else:
+            # Create Octopus merge commit according to http://stackoverflow.com/questions/10874149/git-octopus-merge-with-unrelated-repositoies
+            runcmd('git checkout master')
+            merge = ['git', 'merge', '--no-commit']
+            with open('.git/info/grafts', 'w') as grafts:
+                grafts.write('%s\n' % startrev)
+                for name in conf.repos:
+                    repo = conf.repos[name]
+                    # Use branch created earlier.
+                    merge.append(name)
+                    for start in runcmd('git log --pretty=format:%%H --max-parents=0 %s' % name).split('\n'):
+                        grafts.write('%s %s\n' % (start, startrev))
+            try:
+                runcmd(merge)
+            except Exception, error:
+                logger.info('''Merging component repository history failed, perhaps because of merge conflicts.
+It may be possible to commit anyway after resolving these conflicts.
+
+%s''' % error)
+            # Create MERGE_HEAD and MERGE_MSG. "git merge" itself
+            # does not create MERGE_HEAD in case of a (harmless) failure,
+            # and we want certain auto-generated information in the
+            # commit message for future reference and/or automation.
+            with open('.git/MERGE_HEAD', 'w') as head:
+                with open('.git/MERGE_MSG', 'w') as msg:
+                    msg.write('repo: initial import of components\n\n')
+                    # head.write('%s\n' % startrev)
+                    for name in conf.repos:
+                        repo = conf.repos[name]
+                        # <upstream ref> <rewritten ref> <rewritten + files removed>
+                        msg.write('combo-layer-%s: %s %s %s\n' % (name,
+                                                                  repo['last_revision'],
+                                                                  repo['rewritten_revision'],
+                                                                  repo['stripped_revision']))
+                        rev = runcmd('git rev-parse %s' % name).strip()
+                        head.write('%s\n' % rev)
+
         if conf.localconffile:
             localadded = True
             try:
@@ -631,6 +789,9 @@ Action:
     parser.add_option("-n", "--no-pull", help = "skip pulling component repos during update",
                action = "store_true", dest = "nopull", default = False)
 
+    parser.add_option("-H", "--history", help = "import full history of components during init",
+                      action = "store_true", default = False)
+
     options, args = parser.parse_args(sys.argv)
 
     # Dispatch to action handler
-- 
2.1.4

[PATCH 07/11] combo-layer: combine trees via replacement objects

[Patrick Ohly]

Instead of local graft entries rooting the imported branches in the
shared root commit, use replacement objects.

The advantage is that they get moved around by "git push" and "git
fetch", so everyone has the same, nicer view with everything starting
at the beginning of the combined repository.

If undesired, these objects can be removed with "git replace".

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 scripts/combo-layer | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/scripts/combo-layer b/scripts/combo-layer
index d11274e..6d24ce3 100755
--- a/scripts/combo-layer
+++ b/scripts/combo-layer
@@ -354,14 +354,14 @@ file_exclude = %s''' % (name, file_filter or '<empty>', repo.get('file_exclude',
             # Create Octopus merge commit according to http://stackoverflow.com/questions/10874149/git-octopus-merge-with-unrelated-repositoies
             runcmd('git checkout master')
             merge = ['git', 'merge', '--no-commit']
-            with open('.git/info/grafts', 'w') as grafts:
-                grafts.write('%s\n' % startrev)
-                for name in conf.repos:
-                    repo = conf.repos[name]
-                    # Use branch created earlier.
-                    merge.append(name)
-                    for start in runcmd('git log --pretty=format:%%H --max-parents=0 %s' % name).split('\n'):
-                        grafts.write('%s %s\n' % (start, startrev))
+            for name in conf.repos:
+                repo = conf.repos[name]
+                # Use branch created earlier.
+                merge.append(name)
+                # Root all commits which have no parent in the common
+                # ancestor in the new repository.
+                for start in runcmd('git log --pretty=format:%%H --max-parents=0 %s' % name).split('\n'):
+                    runcmd('git replace --graft %s %s' % (start, startrev))
             try:
                 runcmd(merge)
             except Exception, error:
-- 
2.1.4

[PATCH 08/11] combo-layer: partial import for '--history init'

[Patrick Ohly]

The new "since_revision" property can be used to cut off the imported
history at some point. This is useful to keep the resulting repository
smaller while still preserving enough history that "git annotate"
reports the right author and commit for most lines.

The initial, squashed import commit shows up with "unknown" as author
in the "git annotate" output. It has the repository name as prefix
in the subject line; importing that commit works best with a
layer hook which does not add the repository name again when
it is already present. Adding it here is useful for hooks
which do not extend the subject line.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 scripts/combo-layer              | 40 ++++++++++++++++++++++++++++++++++++++++
 scripts/combo-layer.conf.example | 13 +++++++++++++
 2 files changed, 53 insertions(+)

diff --git a/scripts/combo-layer b/scripts/combo-layer
index 6d24ce3..cbff618 100755
--- a/scripts/combo-layer
+++ b/scripts/combo-layer
@@ -284,6 +284,46 @@ def action_init(conf, args):
                 runcmd("git branch -D %s" % refname, ldir)
                 # Make that the head revision.
                 runcmd("git checkout -b %s %s" % (name, initialrev))
+                # Optional: cut the history by replacing the given
+                # start point(s) with commits providing the same
+                # content (aka tree), but with commit information that
+                # makes it clear that this is an artifically created
+                # commit and nothing the original authors had anything
+                # to do with.
+                since_rev = repo.get('since_revision', '')
+                if since_rev:
+                    committer = runcmd('git var GIT_AUTHOR_IDENT').strip()
+                    # Same time stamp, no name.
+                    author = re.sub('.* (\d+ [+-]\d+)', r'unknown <unknown> \1', committer)
+                    logger.info('author %s' % author)
+                    for rev in since_rev.split():
+                        # Resolve in component repo...
+                        rev = runcmd('git log --oneline --no-abbrev-commit -n1 %s' % rev, ldir).split()[0]
+                        # ... and then get the tree in current
+                        # one. The commit should be in both repos with
+                        # the same tree, but better check here.
+                        tree = runcmd('git show -s --pretty=format:%%T %s' % rev).strip()
+                        with tempfile.NamedTemporaryFile() as editor:
+                            editor.write('''cat >$1 <<EOF
+tree %s
+author %s
+committer %s
+
+%s: squashed import of component
+
+This commit copies the entire set of files as found in
+%s %s
+
+For more information about previous commits, see the
+upstream repository.
+
+Commit created by combo-layer.
+EOF
+''' % (tree, author, committer, name, name, since_rev))
+                            editor.flush()
+                            os.environ['GIT_EDITOR'] = 'sh %s' % editor.name
+                            runcmd('git replace --edit %s' % rev)
+
                 # Optional: rewrite history to change commit messages or to move files.
                 if 'hook' in repo or dest_dir and dest_dir != ".":
                     filter_branch = ['git', 'filter-branch', '--force']
diff --git a/scripts/combo-layer.conf.example b/scripts/combo-layer.conf.example
index 38bc53c..90e2b58 100644
--- a/scripts/combo-layer.conf.example
+++ b/scripts/combo-layer.conf.example
@@ -63,11 +63,24 @@ last_revision =
 # example:
 #     hook = combo-layer-hook-default.sh
 
+# since_revision:
+#   since_revision = release-1-2
+#   since_revision = 12345 abcdf
+#
+# If provided, truncate imported history during "combo-layer --history
+# init" at the specified revision(s).  More than one can be specified
+# to cut off multiple component branches.
+#
+# The specified commits themselves do not get imported. Instead, an
+# artificial commit with "unknown" author is created with a content
+# that matches the original commit.
+
 [oe-core]
 src_uri = git://git.openembedded.org/openembedded-core
 local_repo_dir = /home/kyu3/src/test/oecore
 dest_dir = .
 last_revision =
+since_revision = some-tag-or-commit-on-master-branch
 
 # It is also possible to embed python code in the config values. Similar
 # to bitbake it considers every value starting with @ to be a python
-- 
2.1.4

[PATCH 09/11] combo-layer-hook-default.sh: avoid duplicating prefix

[Patrick Ohly]

The existing patch might already have the desired prefix, perhaps
even multiple times (due to some previous import error). Ensure
that after the replace, the prefix is present exactly once.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 scripts/combo-layer-hook-default.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/combo-layer-hook-default.sh b/scripts/combo-layer-hook-default.sh
index 8b148ac..9dd5aed 100755
--- a/scripts/combo-layer-hook-default.sh
+++ b/scripts/combo-layer-hook-default.sh
@@ -9,5 +9,5 @@ patchfile=$1
 rev=$2
 reponame=$3
 
-sed -i -e "s#^Subject: \[PATCH\] \(.*\)#Subject: \[PATCH\] $reponame: \1#" $patchfile
+sed -i -e "s#^Subject: \[PATCH\] \($reponame: \)*\(.*\)#Subject: \[PATCH\] $reponame: \2#" $patchfile
 sed -i -e "0,/^Signed-off-by:/s#\(^Signed-off-by:.*\)#\($reponame rev: $rev\)\n\n\1#" $patchfile
-- 
2.1.4

[PATCH 10/11] combo-layer-hook-default.sh: beware of embedded patches

[Patrick Ohly]

Patching the subject line must be limited to the subject of
the main patch itself. In particular, git formatted patches embedded in the patch must not be changed.

Achieved by limiting the replacement to the lines until the first
subject in the patch, just as it is done for modifying the first
Signed-off-by.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 scripts/combo-layer-hook-default.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/combo-layer-hook-default.sh b/scripts/combo-layer-hook-default.sh
index 9dd5aed..46f2b70 100755
--- a/scripts/combo-layer-hook-default.sh
+++ b/scripts/combo-layer-hook-default.sh
@@ -9,5 +9,5 @@ patchfile=$1
 rev=$2
 reponame=$3
 
-sed -i -e "s#^Subject: \[PATCH\] \($reponame: \)*\(.*\)#Subject: \[PATCH\] $reponame: \2#" $patchfile
+sed -i -e "0,/^Subject:/s#^Subject: \[PATCH\] \($reponame: \)*\(.*\)#Subject: \[PATCH\] $reponame: \2#" $patchfile
 sed -i -e "0,/^Signed-off-by:/s#\(^Signed-off-by:.*\)#\($reponame rev: $rev\)\n\n\1#" $patchfile
-- 
2.1.4

[PATCH 11/11] combo-layer-hook-default.sh: handle patches without Signed-off-by

[Patrick Ohly]

Inserting the "From rev" comment depended on having at least one
Signed-off-by line in the patch header. Some old repository commits in
openembedded-core and bitbake do not have those.

When inporting those, just insert at the end of the patch
header. While doing so, ensure that there's exactly one blank line
since the last non-blank line.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 scripts/combo-layer-hook-default.sh | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/scripts/combo-layer-hook-default.sh b/scripts/combo-layer-hook-default.sh
index 46f2b70..1e3a3b9 100755
--- a/scripts/combo-layer-hook-default.sh
+++ b/scripts/combo-layer-hook-default.sh
@@ -10,4 +10,11 @@ rev=$2
 reponame=$3
 
 sed -i -e "0,/^Subject:/s#^Subject: \[PATCH\] \($reponame: \)*\(.*\)#Subject: \[PATCH\] $reponame: \2#" $patchfile
-sed -i -e "0,/^Signed-off-by:/s#\(^Signed-off-by:.*\)#\($reponame rev: $rev\)\n\n\1#" $patchfile
+if grep -q '^Signed-off-by:' $patchfile; then
+    # Insert before Signed-off-by.
+    sed -i -e "0,/^Signed-off-by:/s#\(^Signed-off-by:.*\)#\(From $reponame rev: $rev\)\n\n\1#" $patchfile
+else
+    # Insert before final --- separator, with extra blank lines removed.
+    perl -e "\$_ = join('', <>); s/^(.*\S[ \t]*)(\n|\n\s*\n)---\n/\$1\n\nFrom $reponame rev: $rev\n---\n/s; print;" $patchfile >$patchfile.tmp
+    mv $patchfile.tmp $patchfile
+fi
-- 
2.1.4

Re: [PATCH 02/11] combo-layer: make Signed-off-by optional

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 352 bytes --]

On 13 March 2015 at 15:29, Patrick Ohly <patrick.ohly@intel.com> wrote:

> It depends on the diligence of the person running the combo-layer tool
> whether the Signed-off-by line added to each commit actually indicates
> that the person was involved in validating the change.
>

Ironically, this commit is missing the signed-off-by. :)

Ross

[-- Attachment #2: Type: text/html, Size: 785 bytes --]