[yocto-kernel-tools][PATCH 0/1] Rework merge of feature branches

[yocto-kernel-tools][PATCH 0/1] Rework merge of feature branches

[David Vincent]

I've recently noticed that, on Morty, feature branches of a Yocto kernel are not
merged anymore. This is due to the fact that the meta-series file does not
process the git merge commands anymore.

This commit creates a new merge command to be executed when processing the
kernel metadata. To solve the issue, there is also a patch to send to oe-core
for kernel-yocto.bbclass, but I wait for this patch to be included since there
is no point without it.

David Vincent (1):
  scc: Move merge command

 tools/kgit-meta          | 14 ++++++++++++++
 tools/scc                | 13 -------------
 tools/scc-cmds/merge.cmd | 10 ++++++++++
 3 files changed, 24 insertions(+), 13 deletions(-)
 create mode 100644 tools/scc-cmds/merge.cmd

-- 
2.11.0

[yocto-kernel-tools][PATCH 1/1] scc: Move merge command

[David Vincent]

Create a new command to merge a feature branch inside the current branch
when processing meta-series.

Signed-off-by: David Vincent <freesilicon@gmail.com>
---
 tools/kgit-meta          | 14 ++++++++++++++
 tools/scc                | 13 -------------
 tools/scc-cmds/merge.cmd | 10 ++++++++++
 3 files changed, 24 insertions(+), 13 deletions(-)
 create mode 100644 tools/scc-cmds/merge.cmd

diff --git a/tools/kgit-meta b/tools/kgit-meta
index 1c19ef5..aa5e210 100755
--- a/tools/kgit-meta
+++ b/tools/kgit-meta
@@ -254,6 +254,20 @@ for fline in `cat ${meta_series}`; do
                 fi
             fi
             ;;
+        merge:*)
+            b=$(echo ${fline} | cut -d: -f2 | sed 's%^\ %%')
+            check_branch ${b}
+            if [ $? -ne 0 ]; then
+                echo "ERROR. branch ${b} does not exist, can't merge"
+                exit 1
+            else
+                echo "[INFO]: branch merge: ${b}"
+                eval git merge -q --no-ff -m \"Merge branch ${b}\" ${b}
+                if [ $? -ne 0 ]; then
+                exit 1
+                fi
+            fi
+            ;;
         *)
             ;;
     esac
diff --git a/tools/scc b/tools/scc
index 3d17496..cee9e0d 100755
--- a/tools/scc
+++ b/tools/scc
@@ -286,19 +286,6 @@ set_kernel_version()
     KERNEL_VERSION=$1
 }
 
-# Used in feature scripts
-#  - convenience wrapper for "git merge" 
-#  - arguments are git merge <args> and are passed directly to git
-merge()
-{
-    args="$@"
-
-    # call the git wrapper function. Currently with no modification or
-    # checking on the args, but the option of checking in the future is
-    # trivial once users have converted.
-    git merge $args
-}
-
 # used in feature scripts
 #  - encodes a generic git command
 git()
diff --git a/tools/scc-cmds/merge.cmd b/tools/scc-cmds/merge.cmd
new file mode 100644
index 0000000..02be6ca
--- /dev/null
+++ b/tools/scc-cmds/merge.cmd
@@ -0,0 +1,10 @@
+# used in feature scripts
+#    arg1: the branch name
+merge()
+{
+    local text
+    local mbranch_name=$1
+
+    text="merge: ${mbranch_name}"
+    eval echo "\$text" $outfile_append
+}
-- 
2.11.0

Re: [yocto-kernel-tools][PATCH 0/1] Rework merge of feature branches

[Bruce Ashfield]

[-- Attachment #1: Type: text/plain, Size: 1743 bytes --]

On Wed, Jan 25, 2017 at 5:37 AM, David Vincent <freesilicon@gmail.com>
wrote:

> I've recently noticed that, on Morty, feature branches of a Yocto kernel
> are not
> merged anymore. This is due to the fact that the meta-series file does not
> process the git merge commands anymore.
>

Indeed. I dropped that functionality on purpose when I did the
simplification
and streamlining of the processing .. since I wanted to see if anyone was
using it.

I now have my answer :D

I need to have a closer look at the patch and run it through some tests
here, but  on my glance, you've done it correctly.


> This commit creates a new merge command to be executed when processing the
> kernel metadata. To solve the issue, there is also a patch to send to
> oe-core
> for kernel-yocto.bbclass, but I wait for this patch to be included since
> there
> is no point without it.
>

Send that patch to me, or I'll just do the patch myself .. since yes, we do
need
to add merge to the list of valid comments when processing the meta data.

I'll submit all the changes in my next consolidated pull request for the
kernels + tools.

Bruce


>
> David Vincent (1):
>   scc: Move merge command
>
>  tools/kgit-meta          | 14 ++++++++++++++
>  tools/scc                | 13 -------------
>  tools/scc-cmds/merge.cmd | 10 ++++++++++
>  3 files changed, 24 insertions(+), 13 deletions(-)
>  create mode 100644 tools/scc-cmds/merge.cmd
>
> --
> 2.11.0
>
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end"

[-- Attachment #2: Type: text/html, Size: 2855 bytes --]

[PATCH] kernel-yocto: Update source tree with kgit-meta

[David Vincent]

kgit-meta processes a file called meta-series to update the source tree
(create, merge branches, ...). This fixes the merging of feature
branches using the new merge command of yocto-kernel-tools.

Signed-off-by: David Vincent <freesilicon@gmail.com>
---
 meta/classes/kernel-yocto.bbclass | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/meta/classes/kernel-yocto.bbclass b/meta/classes/kernel-yocto.bbclass
index 5cfd8aff50..b37ac01a84 100644
--- a/meta/classes/kernel-yocto.bbclass
+++ b/meta/classes/kernel-yocto.bbclass
@@ -162,6 +162,19 @@ do_kernel_metadata() {
 			bbfatal_log "Could not generate configuration queue for ${KMACHINE}."
 		fi
 	fi
+
+	# Update git repository by running kgit-meta
+	cd ${S}
+
+	check_git_config
+	meta_dir=$(kgit --meta)
+	if [ -f "${meta_dir}/meta-series" ]; then
+		kgit-meta ${meta_dir}/meta-series
+		if [ $? -ne 0 ]; then
+			bberror "Could not apply metadata for ${KMACHINE}."
+			bbfatal_log "Failures can be resolved in the linux source directory ${S})"
+		fi
+	fi
 }
 
 do_patch() {
-- 
2.11.0

Re: [PATCH] kernel-yocto: Update source tree with kgit-meta

[Bruce Ashfield]

On 2017-01-26 03:38 AM, David Vincent wrote:
> kgit-meta processes a file called meta-series to update the source tree
> (create, merge branches, ...). This fixes the merging of feature
> branches using the new merge command of yocto-kernel-tools.

I have this queued, with some modifications. I as traveling this week,
so it'll be early next week before this goes out for merge.

cheers,

Bruce

>
> Signed-off-by: David Vincent <freesilicon@gmail.com>
> ---
>  meta/classes/kernel-yocto.bbclass | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
>
> diff --git a/meta/classes/kernel-yocto.bbclass b/meta/classes/kernel-yocto.bbclass
> index 5cfd8aff50..b37ac01a84 100644
> --- a/meta/classes/kernel-yocto.bbclass
> +++ b/meta/classes/kernel-yocto.bbclass
> @@ -162,6 +162,19 @@ do_kernel_metadata() {
>  			bbfatal_log "Could not generate configuration queue for ${KMACHINE}."
>  		fi
>  	fi
> +
> +	# Update git repository by running kgit-meta
> +	cd ${S}
> +
> +	check_git_config
> +	meta_dir=$(kgit --meta)
> +	if [ -f "${meta_dir}/meta-series" ]; then
> +		kgit-meta ${meta_dir}/meta-series
> +		if [ $? -ne 0 ]; then
> +			bberror "Could not apply metadata for ${KMACHINE}."
> +			bbfatal_log "Failures can be resolved in the linux source directory ${S})"
> +		fi
> +	fi
>  }
>
>  do_patch() {
>

Re: [yocto-kernel-tools][PATCH 0/1] Rework merge of feature branches

[Bruce Ashfield]

[-- Attachment #1: Type: text/plain, Size: 3051 bytes --]

On Wed, Jan 25, 2017 at 1:05 PM, Bruce Ashfield <bruce.ashfield@gmail.com>
wrote:

>
>
> On Wed, Jan 25, 2017 at 5:37 AM, David Vincent <freesilicon@gmail.com>
> wrote:
>
>> I've recently noticed that, on Morty, feature branches of a Yocto kernel
>> are not
>> merged anymore. This is due to the fact that the meta-series file does not
>> process the git merge commands anymore.
>>
>
> Indeed. I dropped that functionality on purpose when I did the
> simplification
> and streamlining of the processing .. since I wanted to see if anyone was
> using it.
>
> I now have my answer :D
>
> I need to have a closer look at the patch and run it through some tests
> here, but  on my glance, you've done it correctly.
>
>
>> This commit creates a new merge command to be executed when processing the
>> kernel metadata. To solve the issue, there is also a patch to send to
>> oe-core
>> for kernel-yocto.bbclass, but I wait for this patch to be included since
>> there
>> is no point without it.
>>
>
> Send that patch to me, or I'll just do the patch myself .. since yes, we
> do need
> to add merge to the list of valid comments when processing the meta data.
>
> I'll submit all the changes in my next consolidated pull request for the
> kernels + tools.
>

I spent Friday looking into this in more detail, and to keep things
consistent
with how the path and config queues are handled, with all the tree
manipulations
pull out of the various scripts and exposed in the bbclass itself, I ended
up with
a different implementation of the support.

The kern-tools change is the same as you did (and is your change), the
difference
is in the bbclass processing, where do_patch() modifies the tree, and it
consumes
the output of the meta-data routine .. versus the kgit-meta script itself.

The work can be seen in my poky-contrib tree:

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=zedd/kernel,
in the
commit "kern-tools: re-enable scc merge command".

These changes are on master, and passed my testing to merge feature branches
into the branch being built.

If you are able to try it out (on master), that would be great. I'm going
to let it
soak for a few days, and once it merges to master, it can be ported to
morty.

Cheers,

Bruce



>
> Bruce
>
>
>>
>> David Vincent (1):
>>   scc: Move merge command
>>
>>  tools/kgit-meta          | 14 ++++++++++++++
>>  tools/scc                | 13 -------------
>>  tools/scc-cmds/merge.cmd | 10 ++++++++++
>>  3 files changed, 24 insertions(+), 13 deletions(-)
>>  create mode 100644 tools/scc-cmds/merge.cmd
>>
>> --
>> 2.11.0
>>
>> --
>> _______________________________________________
>> yocto mailing list
>> yocto@yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/yocto
>>
>
>
>
> --
> "Thou shalt not follow the NULL pointer, for chaos and madness await thee
> at its end"
>



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end"

[-- Attachment #2: Type: text/html, Size: 5300 bytes --]

Re: [yocto-kernel-tools][PATCH 0/1] Rework merge of feature branches

[David Vincent]

On dimanche 29 janvier 2017 23:31:49 CET Bruce Ashfield wrote:
> On Wed, Jan 25, 2017 at 1:05 PM, Bruce Ashfield <bruce.ashfield@gmail.com>
> wrote:
> 
> I spent Friday looking into this in more detail, and to keep things
> consistent
> with how the path and config queues are handled, with all the tree
> manipulations
> pull out of the various scripts and exposed in the bbclass itself, I ended
> up with
> a different implementation of the support.
> 
> The kern-tools change is the same as you did (and is your change), the
> difference
> is in the bbclass processing, where do_patch() modifies the tree, and it
> consumes
> the output of the meta-data routine .. versus the kgit-meta script itself.
Thanks for looking into this and come up with a better implementation. I agree 
that mine was a little harsh.
> The work can be seen in my poky-contrib tree:
> 
> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=zedd/kernel,
> in the
> commit "kern-tools: re-enable scc merge command".
> 
> These changes are on master, and passed my testing to merge feature branches
> into the branch being built.
Saw it, read it and saw that it was now in master-next, ready to be pulled.
> 
> If you are able to try it out (on master), that would be great. I'm going
> to let it
> soak for a few days, and once it merges to master, it can be ported to
> morty.
Sorry, I don't have a master build ready but as soon as I can test it whether 
on morty or master, I will report back.

David

Re: [yocto-kernel-tools][PATCH 0/1] Rework merge of feature branches

[Bruce Ashfield]

On 2017-02-03 03:42 AM, David Vincent wrote:
> On dimanche 29 janvier 2017 23:31:49 CET Bruce Ashfield wrote:
>> On Wed, Jan 25, 2017 at 1:05 PM, Bruce Ashfield <bruce.ashfield@gmail.com>
>> wrote:
>>
>> I spent Friday looking into this in more detail, and to keep things
>> consistent
>> with how the path and config queues are handled, with all the tree
>> manipulations
>> pull out of the various scripts and exposed in the bbclass itself, I ended
>> up with
>> a different implementation of the support.
>>
>> The kern-tools change is the same as you did (and is your change), the
>> difference
>> is in the bbclass processing, where do_patch() modifies the tree, and it
>> consumes
>> the output of the meta-data routine .. versus the kgit-meta script itself.
> Thanks for looking into this and come up with a better implementation. I agree
> that mine was a little harsh.
>> The work can be seen in my poky-contrib tree:
>>
>> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=zedd/kernel,
>> in the
>> commit "kern-tools: re-enable scc merge command".
>>
>> These changes are on master, and passed my testing to merge feature branches
>> into the branch being built.
> Saw it, read it and saw that it was now in master-next, ready to be pulled.
>>
>> If you are able to try it out (on master), that would be great. I'm going
>> to let it
>> soak for a few days, and once it merges to master, it can be ported to
>> morty.
> Sorry, I don't have a master build ready but as soon as I can test it whether
> on morty or master, I will report back.

Perfect. Since I just sent my kernel series to master, I can flip my
builder to morty and have a look as well. I'd still like to confirm that
it does work in your scenario, but I can do the lifting to get it ready
for morty with a run of my own.

Cheers,

Bruce

>
> David
>

Re: [yocto-kernel-tools][PATCH 0/1] Rework merge of feature branches

[Bruce Ashfield]

[-- Attachment #1: Type: text/plain, Size: 1704 bytes --]

On 2017-02-03 03:42 AM, David Vincent wrote:
> On dimanche 29 janvier 2017 23:31:49 CET Bruce Ashfield wrote:
>> On Wed, Jan 25, 2017 at 1:05 PM, Bruce Ashfield <bruce.ashfield@gmail.com>
>> wrote:
>>
>> I spent Friday looking into this in more detail, and to keep things
>> consistent
>> with how the path and config queues are handled, with all the tree
>> manipulations
>> pull out of the various scripts and exposed in the bbclass itself, I ended
>> up with
>> a different implementation of the support.
>>
>> The kern-tools change is the same as you did (and is your change), the
>> difference
>> is in the bbclass processing, where do_patch() modifies the tree, and it
>> consumes
>> the output of the meta-data routine .. versus the kgit-meta script itself.
> Thanks for looking into this and come up with a better implementation. I agree
> that mine was a little harsh.
>> The work can be seen in my poky-contrib tree:
>>
>> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=zedd/kernel,
>> in the
>> commit "kern-tools: re-enable scc merge command".
>>
>> These changes are on master, and passed my testing to merge feature branches
>> into the branch being built.
> Saw it, read it and saw that it was now in master-next, ready to be pulled.
>>
>> If you are able to try it out (on master), that would be great. I'm going
>> to let it
>> soak for a few days, and once it merges to master, it can be ported to
>> morty.
> Sorry, I don't have a master build ready but as soon as I can test it whether
> on morty or master, I will report back.

I was able to merge branches with this patch applied to Morty.

Cheers,

Bruce

>
> David
>


[-- Attachment #2: 0001-kern-tools-re-enable-scc-merge-command.patch --]
[-- Type: text/x-patch, Size: 2789 bytes --]

From 8e08efa4b567e58386247575ae28835841f484c0 Mon Sep 17 00:00:00 2001
From: Bruce Ashfield <bruce.ashfield@windriver.com>
Date: Wed, 25 Jan 2017 13:29:34 -0500
Subject: [PATCH] kern-tools: re-enable scc merge command

The ability to merge two branches directly from a .scc file was
dropped during the streamlining of the tools.

As was pointed out by David Vincent <freesilicon@gmail.com>, there is
once again a valid use case for this functionality, so we restore the
capability.

Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
---
 meta/classes/kernel-yocto.bbclass                       | 16 +++++++++++++++-
 meta/recipes-kernel/kern-tools/kern-tools-native_git.bb |  3 ++-
 2 files changed, 17 insertions(+), 2 deletions(-)

diff --git a/meta/classes/kernel-yocto.bbclass b/meta/classes/kernel-yocto.bbclass
index a60327a..eb35dce 100644
--- a/meta/classes/kernel-yocto.bbclass
+++ b/meta/classes/kernel-yocto.bbclass
@@ -148,7 +148,7 @@ do_kernel_metadata() {
 	# run1: pull all the configuration fragments, no matter where they come from
 	elements="`echo -n ${bsp_definition} ${sccs} ${patches} ${KERNEL_FEATURES}`"
 	if [ -n "${elements}" ]; then
-		scc --force -o ${S}/${meta_dir}:cfg,meta ${includes} ${bsp_definition} ${sccs} ${patches} ${KERNEL_FEATURES}
+		scc --force -o ${S}/${meta_dir}:cfg,merge,meta ${includes} ${bsp_definition} ${sccs} ${patches} ${KERNEL_FEATURES}
 		if [ $? -ne 0 ]; then
 			bbfatal_log "Could not generate configuration queue for ${KMACHINE}."
 		fi
@@ -165,6 +165,7 @@ do_kernel_metadata() {
 }
 
 do_patch() {
+	set +e
 	cd ${S}
 
 	check_git_config
@@ -177,6 +178,19 @@ do_patch() {
 			bbfatal_log "Patch failures can be resolved in the linux source directory ${S})"
 		fi
 	fi
+
+	if [ -f "${meta_dir}/merge.queue" ]; then
+		# we need to merge all these branches
+		for b in $(cat ${meta_dir}/merge.queue); do
+			git show-ref --verify --quiet refs/heads/${b}
+			if [ $? -eq 0 ]; then
+				bbnote "Merging branch ${b}"
+				git merge -q --no-ff -m "Merge branch ${b}" ${b}
+			else
+				bbfatal "branch ${b} does not exist, cannot merge"
+			fi
+		done
+	fi
 }
 
 do_kernel_checkout() {
diff --git a/meta/recipes-kernel/kern-tools/kern-tools-native_git.bb b/meta/recipes-kernel/kern-tools/kern-tools-native_git.bb
index 8af0155..aa27ec8 100644
--- a/meta/recipes-kernel/kern-tools/kern-tools-native_git.bb
+++ b/meta/recipes-kernel/kern-tools/kern-tools-native_git.bb
@@ -4,7 +4,8 @@ LIC_FILES_CHKSUM = "file://git/tools/kgit;beginline=5;endline=9;md5=a6c2fa8aef1b
 
 DEPENDS = "git-native"
 
-SRCREV = "85564e69555b713c2759d58ec5ade54424d051d8"
+SRCREV = "c14440d4e7ae0160c260ed65c3e123be5dc97ae8"
+
 PR = "r12"
 PV = "0.2+git${SRCPV}"
 
-- 
2.7.4

Re: [yocto-kernel-tools][PATCH 0/1] Rework merge of feature branches

[David Vincent]

On vendredi 3 février 2017 13:51:34 CET Bruce Ashfield wrote:
> 
> I was able to merge branches with this patch applied to Morty.
> 
> Cheers,
> 
> Bruce
> 
I've just successfully tested it on Morty myself. Works for me too, branches 
are merged just like they were before !

Thank you
David

Re: [yocto-kernel-tools][PATCH 0/1] Rework merge of feature branches

[Bruce Ashfield]

On 2017-02-07 06:21 AM, David Vincent wrote:
> On vendredi 3 février 2017 13:51:34 CET Bruce Ashfield wrote:
>>
>> I was able to merge branches with this patch applied to Morty.
>>
>> Cheers,
>>
>> Bruce
>>
> I've just successfully tested it on Morty myself. Works for me too, branches
> are merged just like they were before !

Good news!

I'll send the patch to the morty maintainer so it can be merged.

Cheers,

Bruce

>
> Thank you
> David
>
>