[PATCH OSSTEST+XEN 0/1+1] Switching to one qemu tree per qemu version (trad vs upstream)

[PATCH OSSTEST+XEN 0/1+1] Switching to one qemu tree per qemu version (trad vs upstream)

[Ian Campbell]

As discussed[0] I think we should move away from having a pair of qemu
repositories for each Xen branch and instead have a single pair (one for
qemu-xen and one for qemu-xen-traditional).

I've prepared a pair of repositories:

    http://xenbits.xen.org/gitweb/?p=people/ianc/single-qemu-repo/qemu-xen-traditional.git;a=summary
http://xenbits.xen.org/gitweb/?p=people/ianc/single-qemu-repo/qemu-xen.git;a=summary

These will eventually become, respectively:
    git://xenbits.xen.org/qemu-xen-traditional.git
    git    ://xenbits.xen.org/qemu-xen.git

Note that qemu-xen-traditional does not have an osstest gate, it is gated
by changes to Config.mk in xen.git pulling in the specific revision.

In addition to the scheme described in [0] the qemu-mainline test output
gate changes from:
    git    ://xenbits.xen.org/osstest/qemu.git#mainline/xen-tested-master
to
    git    ://xenbits.xen.org/qemu-xen.git#upstream    -tested

Thereby sharing a tree with our upstream branches, which I think makes
sense. The input remains git://git.qemu.org/qemu.git#master, of course.

I will post two patches in reply to this, the first is for xen.git to make
the obvious change to Config.mk.

The second is for osstest to change it to fetch from these new trees and
push to them, and in addition push to the old trees for existing stable
branches only (including 4.6).

I believe the plan for deployment should be:

 * Today we can already just remove the old staging/qemu-xen-* trees, they
   are unused (apart from being manually pushed to along with the staging
   trees, I think).
 * Move the two new trees out of people/ianc into the correct places.
   Ensure git-http etc all works and that Stefano + IanJ have appropriate
   permissions.
 * Push the osstest patch (possibly consider a force push? An osstest
   flight doesn't actually exercise this and it would make coordination
   with the next step easier...)
 * Once that change is in osstest.git#production Stefano and Ian would
   switch to pushing to the appropriate staging* branches new trees'.
   osstest will ignore the old staging trees and osstest will update both
   the new master/stable branches and the old stable trees (but not the old
   qemu-upstream-unstable#master branch).
 * ASAP after the osstest patch reaches production push the patch to
   xen.git#staging. This will cause the xen-unstable builds to use the new
   output gate. Until this is done unstable builds will continue using the
   old master push gate, which is not updated after the osstest update
   (only stable branches are), this isn't a big deal but obviously a
   smaller window would be better.
 * At this point we could remove the old staging/qemu-upstream-* trees,
   they are not referenced by anything.
 * At our leisure backport the xen.git change to stable branches, probably
   back as far as 4.2 (when qemu-xen was introduced).
 * Do stable releases of each of the above.
 * Drop (one by one or all at once) the push to the legacy stable branches
   from osstest as stable releases are made referencing the new trees.
 * Consider hiding (or removing) the old output trees from xenbits as well.

I don't think this has any impact on the 4.6 release, apart from the point
where Stefano+Ian start pushing to the new trees, so we could consider
starting whenever we like.

I don't propose that we try and do the backport of the Config.mk change to
4.6 for 4.6.0, it can wait for 4.6.1.

Ian.

[0] http://mid.gmane.org/<1438266156.11600.347.camel@citrix.com>

[PATCH OSSTEST] Switch to merged qemu-xen{, -traditional}.git trees

[Ian Campbell]

The qemu-mainline flights now push to the upstream-tested branch of
qemu-xen.git (while still pulling from upstream).

The qemu-upstream-unstable flights pull from staging and push to
master in qemu-xen.git

All of the qemu-upstream-X.Y-testing pull from staging-X.Y and push to
stable-X.Y branch in qemu-xen.git.

For now we also continue pushing to the old trees for qemu-upstream
4.2 through to 4.6-testing. Once those branches have updated their
Config.mk and done a point release we can consider removing these.

Abolish $LOCALREV_QEMU_MAINLINE in favour of $LOCALREV_QEMU_UPSTREAM,
it was used inconsistently.

While changing things ensure all pushes are done to refs/heads/$thing
to avoid issues when output branches to not exist.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 ap-common            | 10 +++-------
 ap-fetch-version     | 10 ++++++++--
 ap-fetch-version-old | 14 +++++++++-----
 ap-push              | 25 ++++++++++++++++++++-----
 4 files changed, 40 insertions(+), 19 deletions(-)

diff --git a/ap-common b/ap-common
index dfeab9e..2cba7f8 100644
--- a/ap-common
+++ b/ap-common
@@ -26,9 +26,7 @@
 : ${TREE_XEN:=git://xenbits.xen.org/xen.git}
 : ${PUSH_TREE_XEN:=$XENBITS:/home/xen/git/xen.git}
 
-#: ${TREE_QEMU:=git://mariner.uk.xensource.com/qemu-$xenbranch.git}
-: ${TREE_QEMU:=git://xenbits.xen.org/staging/qemu-$xenbranch.git}
-
+: ${TREE_QEMU:=git://xenbits.xen.org/qemu-xen-traditional.git}
 
 : ${GIT_KERNEL_ORG:=git://git.kernel.org}
 : ${KERNEL_SCM:=${GIT_KERNEL_ORG}/pub/scm/linux/kernel/git}
@@ -87,13 +85,11 @@ fi
 
 : ${TREEBASE_LINUX_XCP:=http://hg.uk.xensource.com/carbon/trunk/linux-2.6.27}
 
-: ${TREE_QEMU_UPSTREAM:=git://xenbits.xen.org/staging/qemu-upstream-${xenbranch#xen-}.git}
+: ${TREE_QEMU_UPSTREAM:=git://xenbits.xen.org/qemu-xen.git}
+: ${PUSH_TREE_QEMU_UPSTREAM=$XENBITS:/home/xen/git/qemu-xen.git
 : ${LOCALREV_QEMU_UPSTREAM:=daily-cron.$branch}
 
 : ${TREE_QEMU_MAINLINE:=git://git.qemu.org/qemu.git}
-: ${BASE_TREE_QEMU_MAINLINE:=git://xenbits.xen.org/osstest/qemu.git}
-: ${PUSH_TREE_QEMU_MAINLINE:=$XENBITS:/home/xen/git/osstest/qemu.git}
-: ${LOCALREV_QEMU_MAINLINE:=daily-cron.$branch}
 
 info_linux_tree () {
 	case $1 in
diff --git a/ap-fetch-version b/ap-fetch-version
index 8b41acf..754a398 100755
--- a/ap-fetch-version
+++ b/ap-fetch-version
@@ -53,9 +53,15 @@ qemu-mainline)
 	repo_tree_rev_fetch_git $branch \
 		$TREE_QEMU_MAINLINE master $LOCALREV_QEMU_UPSTREAM
 	;;
-qemu-upstream-*)
+qemu-upstream-unstable)
         repo_tree_rev_fetch_git $branch \
-		$TREE_QEMU_UPSTREAM master $LOCALREV_QEMU_UPSTREAM
+		$TREE_QEMU_UPSTREAM staging $LOCALREV_QEMU_UPSTREAM
+        ;;
+qemu-upstream-*-testing)
+	branchcore=${branch#qemu-upstream-}
+	branchcore=${branchcore%-testing}
+        repo_tree_rev_fetch_git $branch \
+		$TREE_QEMU_UPSTREAM staging-$branchcore $LOCALREV_QEMU_UPSTREAM
         ;;
 linux)
 	repo_tree_rev_fetch_git linux \
diff --git a/ap-fetch-version-old b/ap-fetch-version-old
index 0b4739b..464d1ab 100755
--- a/ap-fetch-version-old
+++ b/ap-fetch-version-old
@@ -32,8 +32,6 @@ select_xenbranch
 : ${BASE_LOCALREV_SEABIOS:=daily-cron.$branch.old}
 : ${BASE_LOCALREV_OVMF:=daily-cron.$branch.old}
 
-: ${BASE_TREE_QEMU_UPSTREAM:=${TREE_QEMU_UPSTREAM/\/staging\//\/}}
-
 if info_linux_tree "$branch"; then
 	repo_tree_rev_fetch_git linux \
 		$BASE_TREE_LINUX_THIS $BASE_TAG_LINUX_THIS $BASE_LOCALREV_LINUX
@@ -59,11 +57,17 @@ xen-4.*-testing)
 	;;
 qemu-mainline)
         repo_tree_rev_fetch_git $branch \
-		$BASE_TREE_QEMU_MAINLINE mainline/xen-tested-master $LOCALREV_QEMU_MAINLINE
+		$TREE_QEMU_UPSTREAM upstream-tested $LOCALREV_QEMU_UPSTREAM
+        ;;
+qemu-upstream-unstable)
+        repo_tree_rev_fetch_git $branch \
+		$TREE_QEMU_UPSTREAM master $LOCALREV_QEMU_UPSTREAM
         ;;
-qemu-upstream-*)
+qemu-upstream-*-testing)
+	branchcore=${branch#qemu-upstream-}
+	branchcore=${branchcore%-testing}
         repo_tree_rev_fetch_git $branch \
-		$BASE_TREE_QEMU_UPSTREAM master $LOCALREV_QEMU_UPSTREAM
+		$TREE_QEMU_UPSTREAM stable-$branchcore $LOCALREV_QEMU_UPSTREAM
         ;;
 linux)
 	repo_tree_rev_fetch_git linux \
diff --git a/ap-push b/ap-push
index 5967b42..c3bc814 100755
--- a/ap-push
+++ b/ap-push
@@ -30,8 +30,7 @@ select_xenbranch
 . ap-common
 
 TREE_LINUX=$PUSH_TREE_LINUX
-TREE_QEMU_MAINLINE=$PUSH_TREE_QEMU_MAINLINE
-TREE_QEMU_UPSTREAM=$XENBITS:/home/xen/git/qemu-upstream-${xenbranch#xen-}.git
+TREE_QEMU_UPSTREAM=$PUSH_TREE_QEMU_UPSTREAM
 TREE_XEN=$PUSH_TREE_XEN
 TREE_LIBVIRT=$PUSH_TREE_LIBVIRT
 TREE_RUMPUSERXEN=$PUSH_TREE_RUMPUSERXEN
@@ -63,11 +62,27 @@ xen-*-testing)
 	;;
 qemu-mainline)
 	cd $repos/qemu-mainline
-	git push $TREE_QEMU_MAINLINE $revision:refs/heads/mainline/xen-tested-master
+	git push $TREE_QEMU_UPSTREAM $revision:refs/heads/upstream-tested
 	;;
-qemu-upstream-*)
+qemu-upstream-unstable)
 	cd $repos/$branch
-        git push $TREE_QEMU_UPSTREAM $revision:master
+        git push $TREE_QEMU_UPSTREAM $revision:refs/heads/master
+        ;;
+qemu-upstream-*-testing)
+	branchcore=${branch#qemu-upstream-}
+	branchcore=${branchcore%-testing}
+	cd $repos/$branch
+        git push $TREE_QEMU_UPSTREAM $revision:refs/heads/stable-$branchcore
+
+	# For now also push to the old split trees for historical
+	# branches only (qemu-upstream started with xen-4.2-testing
+	# and the split trees end at xen-4.6-testing)
+	case "$xenbranch" in
+	    xen-4.[23456]-testing)
+		tree=$XENBITS:/home/xen/git/qemu-upstream-${xenbranch#xen-}.git
+		git push $tree $revision:refs/heads/master
+		;;
+	esac
         ;;
 linux-3.*)
 	cd $repos/linux
-- 
2.5.1

[PATCH XEN] Config: Switch to unified qemu trees.

[Ian Campbell]

Upstream qemu is not in qemu-xen.git and the trad fork is in
qemu-xen-traditional.h

QEMU_UPSTREAM_REVISION is currently a tag and
QEMU_TRADITIONAL_REVISION is a specific revision, so no changes are
required to those.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
(cherry picked from commit dff5c395c1d23c21238ce17ddcd6f7abe2efd08d)

Conflicts:
	Config.mk
---
 Config.mk | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/Config.mk b/Config.mk
index 9936090..1b559b0 100644
--- a/Config.mk
+++ b/Config.mk
@@ -242,13 +242,13 @@ endif
 
 ifeq ($(GIT_HTTP),y)
 OVMF_UPSTREAM_URL ?= http://xenbits.xen.org/git-http/ovmf.git
-QEMU_UPSTREAM_URL ?= http://xenbits.xen.org/git-http/qemu-upstream-4.5-testing.git
-QEMU_TRADITIONAL_URL ?= http://xenbits.xen.org/git-http/qemu-xen-4.5-testing.git
+QEMU_UPSTREAM_URL ?= http://xenbits.xen.org/git-http/qemu-xen.git
+QEMU_TRADITIONAL_URL ?= http://xenbits.xen.org/git-http/qemu-xen-traditional.git
 SEABIOS_UPSTREAM_URL ?= http://xenbits.xen.org/git-http/seabios.git
 else
 OVMF_UPSTREAM_URL ?= git://xenbits.xen.org/ovmf.git
-QEMU_UPSTREAM_URL ?= git://xenbits.xen.org/qemu-upstream-4.5-testing.git
-QEMU_TRADITIONAL_URL ?= git://xenbits.xen.org/qemu-xen-4.5-testing.git
+QEMU_UPSTREAM_URL ?= git://xenbits.xen.org/qemu-xen.git
+QEMU_TRADITIONAL_URL ?= git://xenbits.xen.org/qemu-xen-traditional.git
 SEABIOS_UPSTREAM_URL ?= git://xenbits.xen.org/seabios.git
 endif
 OVMF_UPSTREAM_REVISION ?= cb9a7ebabcd6b8a49dc0854b2f9592d732b5afbd
-- 
2.1.4

Re: [PATCH OSSTEST+XEN 0/1+1] Switching to one qemu tree per qemu version (trad vs upstream)

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST+XEN 0/1+1] Switching to one qemu tree per qemu version (trad vs upstream)"):
> As discussed[0] I think we should move away from having a pair of qemu
> repositories for each Xen branch and instead have a single pair (one for
> qemu-xen and one for qemu-xen-traditional).
> 
> I've prepared a pair of repositories:
> 
>     http://xenbits.xen.org/gitweb/?p=people/ianc/single-qemu-repo/qemu-xen-traditional.git;a=summary
> http://xenbits.xen.org/gitweb/?p=people/ianc/single-qemu-repo/qemu-xen.git;a=summary
> 
> These will eventually become, respectively:
>     git://xenbits.xen.org/qemu-xen-traditional.git
>     git    ://xenbits.xen.org/qemu-xen.git
> 
> Note that qemu-xen-traditional does not have an osstest gate, it is gated
> by changes to Config.mk in xen.git pulling in the specific revision.
> 
> In addition to the scheme described in [0] the qemu-mainline test output
> gate changes from:
>     git    ://xenbits.xen.org/osstest/qemu.git#mainline/xen-tested-master
> to
>     git    ://xenbits.xen.org/qemu-xen.git#upstream    -tested
> 
> Thereby sharing a tree with our upstream branches, which I think makes
> sense. The input remains git://git.qemu.org/qemu.git#master, of course.
> 
> I will post two patches in reply to this, the first is for xen.git to make
> the obvious change to Config.mk.
> 
> The second is for osstest to change it to fetch from these new trees and
> push to them, and in addition push to the old trees for existing stable
> branches only (including 4.6).
> 
> I believe the plan for deployment should be:
> 
>  * Today we can already just remove the old staging/qemu-xen-* trees, they
>    are unused (apart from being manually pushed to along with the staging
>    trees, I think).

Yes.  (it needs some consequential changes to my ad-hoc scripts but
that's fine.)

>  * Push the osstest patch (possibly consider a force push? An osstest
>    flight doesn't actually exercise this and it would make coordination
>    with the next step easier...)

A force push would seem fine to me.

>  * ASAP after the osstest patch reaches production push the patch to
>    xen.git#staging. This will cause the xen-unstable builds to use the new
>    output gate. Until this is done unstable builds will continue using the
>    old master push gate, which is not updated after the osstest update
>    (only stable branches are), this isn't a big deal but obviously a
>    smaller window would be better.

Right.

>  * At this point we could remove the old staging/qemu-upstream-* trees,
>    they are not referenced by anything.

Promptly removing things that are unreferenced and not updated would
be a good idea :-).

>  * At our leisure backport the xen.git change to stable branches, probably
>    back as far as 4.2 (when qemu-xen was introduced).
>  * Do stable releases of each of the above.

This will take quite some time.

>  * Drop (one by one or all at once) the push to the legacy stable branches
>    from osstest as stable releases are made referencing the new trees.

I think it would be sensible to do this all at once and to expect to
do it perhaps 12 months from when we start.

>  * Consider hiding (or removing) the old output trees from xenbits as well.

Hiding them would be a good idea.  I wonder if that's possible.

Ian.

Re: [PATCH OSSTEST] Switch to merged qemu-xen{, -traditional}.git trees

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST] Switch to merged qemu-xen{,-traditional}.git trees"):
> The qemu-mainline flights now push to the upstream-tested branch of
> qemu-xen.git (while still pulling from upstream).

This mostly looks fine.

How are we going to test this ?  I think the most practical approach
is to add a step to your deployment plan (after the new trees are in
place, but before the osstest force push) where you explicitly ap-push
and ap-fetch the relevant trees.  You could do the ap-push as a user
without the appropriate permissions to get a dry run.

Then if this patch needs updating it can be fixed up quite easily.

> --- a/ap-push
> +++ b/ap-push
...
> +qemu-upstream-*-testing)
...
> +	# For now also push to the old split trees for historical
> +	# branches only (qemu-upstream started with xen-4.2-testing
> +	# and the split trees end at xen-4.6-testing)
> +	case "$xenbranch" in
> +	    xen-4.[23456]-testing)
> +		tree=$XENBITS:/home/xen/git/qemu-upstream-${xenbranch#xen-}.git
> +		git push $tree $revision:refs/heads/master

I think this would be clearer if not done in terms of xenbranch.  I
would use case $branch and then $branchcore.  What do you think ?

Ian.

Re: [PATCH XEN] Config: Switch to unified qemu trees.

[Ian Jackson]

Ian Campbell writes ("[PATCH XEN] Config: Switch to unified qemu trees."):
> Upstream qemu is not in qemu-xen.git and the trad fork is in
> qemu-xen-traditional.h
> 
> QEMU_UPSTREAM_REVISION is currently a tag and
> QEMU_TRADITIONAL_REVISION is a specific revision, so no changes are
> required to those.

When this is backported, something will have to be done about that, of
course.  Otherwise stable branches might pick up qemu-*#master

> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> (cherry picked from commit dff5c395c1d23c21238ce17ddcd6f7abe2efd08d)
> 
> Conflicts:
> 	Config.mk

Did you intend that to appear here ?

Ian.

Re: [PATCH OSSTEST+XEN 0/1+1] Switching to one qemu tree per qemu version (trad vs upstream)

[Ian Campbell]

On Thu, 2015-09-17 at 11:23 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST+XEN 0/1+1] Switching to one qemu tree per qemu version (trad vs upstream)"):
> > I believe the plan for deployment should be:
> > 
> >  * Today we can already just remove the old staging/qemu-xen-* trees, they
> >    are unused (apart from being manually pushed to along with the staging
> >    trees, I think).
> 
> Yes.  (it needs some consequential changes to my ad-hoc scripts but
> that's fine.)

Which reminds me: there will no doubt be some knock on effects on the
release check list both for 4.7 and for existing stable branches which
pickup the Config.mk change.

> 
> >  * Push the osstest patch (possibly consider a force push? An osstest
> >    flight doesn't actually exercise this and it would make coordination
> >    with the next step easier...)
> 
> A force push would seem fine to me.
> 
> >  * ASAP after the osstest patch reaches production push the patch to
> >    xen.git#staging. This will cause the xen-unstable builds to use the
> > new
> >    output gate. Until this is done unstable builds will continue using
> > the
> >    old master push gate, which is not updated after the osstest update
> >    (only stable branches are), this isn't a big deal but obviously a
> >    smaller window would be better.
> 
> Right.
> 
> >  * At this point we could remove the old staging/qemu-upstream-* trees,
> >    they are not referenced by anything.
> 
> Promptly removing things that are unreferenced and not updated would
> be a good idea :-).

Right. My concern would be people with trees cloned from it, I suppose they
will get the git equivalent of a 404 and can update.

> 
> >  * At our leisure backport the xen.git change to stable branches,
> > probably
> >    back as far as 4.2 (when qemu-xen was introduced).
> >  * Do stable releases of each of the above.
> 
> This will take quite some time.

Yes, I don't envisage this stage happening quickly. More "in the natural
course of things".

> 
> >  * Drop (one by one or all at once) the push to the legacy stable
> > branches
> >    from osstest as stable releases are made referencing the new trees.
> 
> I think it would be sensible to do this all at once and to expect to
> do it perhaps 12 months from when we start.

Ack.

> >  * Consider hiding (or removing) the old output trees from xenbits as
> > well.
> 
> Hiding them would be a good idea.  I wonder if that's possible.

Sadly not in the mode which we use gitweb in which is to effectively do
"find $root" and export everything, with no blacklist support.

It is possible to switch to a mode which is default deny with an explicit
whitelist, which would involve some faff for each tree we create. I'm not
sure if it is worth it.

NB in that mode it should be possible to have repos which are present (i.e.
can be cloned) but not listed on the gitweb index page, and which still
have their own subpage if you know where to look.

I've deployed gitolite on my own VPS not so long ago, perhaps we should
look at this, as a medium term goal, it would also make it easier to give
people trees without ssh accounts and to do access control on various
shared trees etc.

Ian.

Re: [PATCH XEN] Config: Switch to unified qemu trees.

[Ian Campbell]

On Thu, 2015-09-17 at 11:32 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH XEN] Config: Switch to unified qemu
> trees."):
> > Upstream qemu is not in qemu-xen.git and the trad fork is in
> > qemu-xen-traditional.h
> > 
> > QEMU_UPSTREAM_REVISION is currently a tag and
> > QEMU_TRADITIONAL_REVISION is a specific revision, so no changes are
> > required to those.
> 
> When this is backported, something will have to be done about that, of
> course.  Otherwise stable branches might pick up qemu-*#master

Stable branches already have this switched out for a specific versioned tag
as part of the branching process, I believe.

> > Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> > (cherry picked from commit dff5c395c1d23c21238ce17ddcd6f7abe2efd08d)
> > 
> > Conflicts:
> > 	Config.mk
> 
> Did you intend that to appear here ?

No. In fact I ran git send-email with my staging-4.5 branch checked out
where I had been checking how noisey the cherry pick was etc. Sorry.

The proper commit for staging is below.

FWIW the cherry-pick conflicts because the branch has the different split
tree, just accepting the version from the cherry-picked patch is correct.

Ian.

>From dff5c395c1d23c21238ce17ddcd6f7abe2efd08d Mon Sep 17 00:00:00 2001
From: Ian Campbell <ian.campbell@citrix.com>
Date: Thu, 10 Sep 2015 14:31:34 +0100
Subject: [PATCH] Config: Switch to unified qemu trees.

Upstream qemu is now in qemu-xen.git and the trad fork is in
qemu-xen-traditional.h

QEMU_UPSTREAM_REVISION is currently a tag and
QEMU_TRADITIONAL_REVISION is a specific revision, so no changes are
required to those.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 Config.mk | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/Config.mk b/Config.mk
index 59607b4..a371cd9 100644
--- a/Config.mk
+++ b/Config.mk
@@ -242,14 +242,14 @@ endif
 
 ifeq ($(GIT_HTTP),y)
 OVMF_UPSTREAM_URL ?= http://xenbits.xen.org/git-http/ovmf.git
-QEMU_UPSTREAM_URL ?= http://xenbits.xen.org/git-http/qemu-upstream-unstable.git
-QEMU_TRADITIONAL_URL ?= http://xenbits.xen.org/git-http/qemu-xen-unstable.git
+QEMU_UPSTREAM_URL ?= http://xenbits.xen.org/git-http/qemu-xen.git
+QEMU_TRADITIONAL_URL ?= http://xenbits.xen.org/git-http/qemu-xen-traditional.git
 SEABIOS_UPSTREAM_URL ?= http://xenbits.xen.org/git-http/seabios.git
 MINIOS_UPSTREAM_URL ?= http://xenbits.xen.org/git-http/mini-os.git
 else
 OVMF_UPSTREAM_URL ?= git://xenbits.xen.org/ovmf.git
-QEMU_UPSTREAM_URL ?= git://xenbits.xen.org/qemu-upstream-unstable.git
-QEMU_TRADITIONAL_URL ?= git://xenbits.xen.org/qemu-xen-unstable.git
+QEMU_UPSTREAM_URL ?= git://xenbits.xen.org/qemu-xen.git
+QEMU_TRADITIONAL_URL ?= git://xenbits.xen.org/qemu-xen-traditional.git
 SEABIOS_UPSTREAM_URL ?= git://xenbits.xen.org/seabios.git
 MINIOS_UPSTREAM_URL ?= git://xenbits.xen.org/mini-os.git
 endif
-- 
2.1.4

Re: [PATCH OSSTEST] Switch to merged qemu-xen{, -traditional}.git trees

[Ian Campbell]

On Thu, 2015-09-17 at 11:31 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST] Switch to merged qemu-xen{,
> -traditional}.git trees"):
> > The qemu-mainline flights now push to the upstream-tested branch of
> > qemu-xen.git (while still pulling from upstream).
> 
> This mostly looks fine.
> 
> How are we going to test this ?  I think the most practical approach
> is to add a step to your deployment plan (after the new trees are in
> place, but before the osstest force push) where you explicitly ap-push
> and ap-fetch the relevant trees.

This sounds workable.

>   You could do the ap-push as a user
> without the appropriate permissions to get a dry run.

Since this would be pre-deployment it wouldn't be too hard to just let it
do it and then force a return to whatever baseline manually.
> 
> Then if this patch needs updating it can be fixed up quite easily.

> > --- a/ap-push
> > +++ b/ap-push
> ...
> > +qemu-upstream-*-testing)
> ...
> > +	# For now also push to the old split trees for historical
> > +	# branches only (qemu-upstream started with xen-4.2-testing
> > +	# and the split trees end at xen-4.6-testing)
> > +	case "$xenbranch" in
> > +	    xen-4.[23456]-testing)
> > +		tree=$XENBITS:/home/xen/git/qemu-upstream
> > -${xenbranch#xen-}.git
> > +		git push $tree $revision:refs/heads/master
> 
> I think this would be clearer if not done in terms of xenbranch.  I
> would use case $branch and then $branchcore.  What do you think ?

Good idea.

Ian.

Re: [PATCH OSSTEST] Switch to merged qemu-xen{, -traditional}.git trees

[Ian Campbell]

On Thu, 2015-09-17 at 11:43 +0100, Ian Campbell wrote:
> > > +qemu-upstream-*-testing)
> > ...
> > > +	# For now also push to the old split trees for historical
> > > +	# branches only (qemu-upstream started with xen-4.2-testing
> > > +	# and the split trees end at xen-4.6-testing)
> > > +	case "$xenbranch" in
> > > +	    xen-4.[23456]-testing)
> > > +		tree=$XENBITS:/home/xen/git/qemu-upstream
> > > -${xenbranch#xen-}.git
> > > +		git push $tree $revision:refs/heads/master
> > 
> > I think this would be clearer if not done in terms of xenbranch.  I
> > would use case $branch and then $branchcore.  What do you think ?
> 
> Good idea.

$branch is a bit wordy at this point, but $branchcore is usable for both
cases incrementally:

diff --git a/ap-push b/ap-push
index c3bc814..41acb39 100755
--- a/ap-push
+++ b/ap-push
@@ -77,9 +77,9 @@ qemu-upstream-*-testing)
        # For now also push to the old split trees for historical
        # branches only (qemu-upstream started with xen-4.2-testing
        # and the split trees end at xen-4.6-testing)
-       case "$xenbranch" in
-           xen-4.[23456]-testing)
-               tree=$XENBITS:/home/xen/git/qemu-upstream-${xenbranch#xen-}.git
+       case "$branchcore" in
+           4.[23456])
+               tree=$XENBITS:/home/xen/git/qemu-upstream-${branchcore}-testing.git
                git push $tree $revision:refs/heads/master
                ;;
        esac

Look ok?

Re: [PATCH OSSTEST] Switch to merged qemu-xen{, -traditional}.git trees

[Ian Jackson]

Ian Campbell writes ("Re: [PATCH OSSTEST] Switch to merged qemu-xen{,-traditional}.git trees"):
> $branch is a bit wordy at this point, but $branchcore is usable for both
> cases incrementally:

LGTM.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

(I assume you'll fold this in.)

Ian.

Re: [PATCH OSSTEST] Switch to merged qemu-xen{, -traditional}.git trees

[Ian Campbell]

On Fri, 2015-10-02 at 14:43 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [PATCH OSSTEST] Switch to merged qemu-xen{,
> -traditional}.git trees"):
> > $branch is a bit wordy at this point, but $branchcore is usable for
> > both
> > cases incrementally:
> 
> LGTM.
> 
> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
> 
> (I assume you'll fold this in.)

Yes, v2 is below for completeness. I rebased this over the smoketest stuff
which added $xenbranch_forqemu which of course is now no longer needed so
the rebase just involved taking my version of each conflict and removing
the initialisation of xenbranch_forqemu. Accordingly I've retained your
Ack, I hope that is ok.

We discussed on IRC with you and Stefano and are going to aim to push this
in the w/c 19 October.

Cheers,
Ian.

-----------8<-------

>From faa826a104a1e6c7e357870a843777887b55a3ef Mon Sep 17 00:00:00 2001
From: Ian Campbell <ian.campbell@citrix.com>
Date: Thu, 30 Jul 2015 13:42:03 +0100
Subject: [PATCH v2] Switch to merged qemu-xen{,-traditional}.git trees

The qemu-mainline flights now push to the upstream-tested branch of
qemu-xen.git (while still pulling from upstream).

The qemu-upstream-unstable flights pull from staging and push to
master in qemu-xen.git

All of the qemu-upstream-X.Y-testing pull from staging-X.Y and push to
stable-X.Y branch in qemu-xen.git.

For now we also continue pushing to the old trees for qemu-upstream
4.2 through to 4.6-testing. Once those branches have updated their
Config.mk and done a point release we can consider removing these.

Abolish $LOCALREV_QEMU_MAINLINE in favour of $LOCALREV_QEMU_UPSTREAM,
it was used inconsistently.

While changing things ensure all pushes are done to refs/heads/$thing
to avoid issues when output branches to not exist.

Note that xenbranch_forqemu is no longer needed.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v2: Use branchcore for old tree pushes.
---
 ap-common            | 12 +++---------
 ap-fetch-version     | 10 ++++++++--
 ap-fetch-version-old | 14 +++++++++-----
 ap-push              | 23 +++++++++++++++++++----
 4 files changed, 39 insertions(+), 20 deletions(-)

diff --git a/ap-common b/ap-common
index 91425a9..2cba7f8 100644
--- a/ap-common
+++ b/ap-common
@@ -19,8 +19,6 @@
 
 # $xenbranch must already be set
 
-xenbranch_forqemu=${xenbranch/xen-unstable-smoke/xen-unstable}
-
 : ${XENBITS:=osstest@xenbits.xen.org}
 
 : ${TREEBRANCH_OSSTEST_UPSTREAM=`getconfig OsstestUpstream`}
@@ -28,9 +26,7 @@ xenbranch_forqemu=${xenbranch/xen-unstable-smoke/xen-unstable}
 : ${TREE_XEN:=git://xenbits.xen.org/xen.git}
 : ${PUSH_TREE_XEN:=$XENBITS:/home/xen/git/xen.git}
 
-#: ${TREE_QEMU:=git://mariner.uk.xensource.com/qemu-$xenbranch_forqemu.git}
-: ${TREE_QEMU:=git://xenbits.xen.org/staging/qemu-$xenbranch_forqemu.git}
-
+: ${TREE_QEMU:=git://xenbits.xen.org/qemu-xen-traditional.git}
 
 : ${GIT_KERNEL_ORG:=git://git.kernel.org}
 : ${KERNEL_SCM:=${GIT_KERNEL_ORG}/pub/scm/linux/kernel/git}
@@ -89,13 +85,11 @@ fi
 
 : ${TREEBASE_LINUX_XCP:=http://hg.uk.xensource.com/carbon/trunk/linux-2.6.27}
 
-: ${TREE_QEMU_UPSTREAM:=git://xenbits.xen.org/staging/qemu-upstream-${xenbranch_forqemu#xen-}.git}
+: ${TREE_QEMU_UPSTREAM:=git://xenbits.xen.org/qemu-xen.git}
+: ${PUSH_TREE_QEMU_UPSTREAM=$XENBITS:/home/xen/git/qemu-xen.git
 : ${LOCALREV_QEMU_UPSTREAM:=daily-cron.$branch}
 
 : ${TREE_QEMU_MAINLINE:=git://git.qemu.org/qemu.git}
-: ${BASE_TREE_QEMU_MAINLINE:=git://xenbits.xen.org/osstest/qemu.git}
-: ${PUSH_TREE_QEMU_MAINLINE:=$XENBITS:/home/xen/git/osstest/qemu.git}
-: ${LOCALREV_QEMU_MAINLINE:=daily-cron.$branch}
 
 info_linux_tree () {
 	case $1 in
diff --git a/ap-fetch-version b/ap-fetch-version
index 6fa7588..973c30e 100755
--- a/ap-fetch-version
+++ b/ap-fetch-version
@@ -55,9 +55,15 @@ qemu-mainline)
 	repo_tree_rev_fetch_git $branch \
 		$TREE_QEMU_MAINLINE master $LOCALREV_QEMU_UPSTREAM
 	;;
-qemu-upstream-*)
+qemu-upstream-unstable)
         repo_tree_rev_fetch_git $branch \
-		$TREE_QEMU_UPSTREAM master $LOCALREV_QEMU_UPSTREAM
+		$TREE_QEMU_UPSTREAM staging $LOCALREV_QEMU_UPSTREAM
+        ;;
+qemu-upstream-*-testing)
+	branchcore=${branch#qemu-upstream-}
+	branchcore=${branchcore%-testing}
+        repo_tree_rev_fetch_git $branch \
+		$TREE_QEMU_UPSTREAM staging-$branchcore $LOCALREV_QEMU_UPSTREAM
         ;;
 linux)
 	repo_tree_rev_fetch_git linux \
diff --git a/ap-fetch-version-old b/ap-fetch-version-old
index 66d51f8..511c1fc 100755
--- a/ap-fetch-version-old
+++ b/ap-fetch-version-old
@@ -32,8 +32,6 @@ select_xenbranch
 : ${BASE_LOCALREV_SEABIOS:=daily-cron.$branch.old}
 : ${BASE_LOCALREV_OVMF:=daily-cron.$branch.old}
 
-: ${BASE_TREE_QEMU_UPSTREAM:=${TREE_QEMU_UPSTREAM/\/staging\//\/}}
-
 if info_linux_tree "$branch"; then
 	repo_tree_rev_fetch_git linux \
 		$BASE_TREE_LINUX_THIS $BASE_TAG_LINUX_THIS $BASE_LOCALREV_LINUX
@@ -61,11 +59,17 @@ xen-4.*-testing)
 	;;
 qemu-mainline)
         repo_tree_rev_fetch_git $branch \
-		$BASE_TREE_QEMU_MAINLINE mainline/xen-tested-master $LOCALREV_QEMU_MAINLINE
+		$TREE_QEMU_UPSTREAM upstream-tested $LOCALREV_QEMU_UPSTREAM
+        ;;
+qemu-upstream-unstable)
+        repo_tree_rev_fetch_git $branch \
+		$TREE_QEMU_UPSTREAM master $LOCALREV_QEMU_UPSTREAM
         ;;
-qemu-upstream-*)
+qemu-upstream-*-testing)
+	branchcore=${branch#qemu-upstream-}
+	branchcore=${branchcore%-testing}
         repo_tree_rev_fetch_git $branch \
-		$BASE_TREE_QEMU_UPSTREAM master $LOCALREV_QEMU_UPSTREAM
+		$TREE_QEMU_UPSTREAM stable-$branchcore $LOCALREV_QEMU_UPSTREAM
         ;;
 linux)
 	repo_tree_rev_fetch_git linux \
diff --git a/ap-push b/ap-push
index d2195f2..eacf96b 100755
--- a/ap-push
+++ b/ap-push
@@ -30,8 +30,7 @@ select_xenbranch
 . ./ap-common
 
 TREE_LINUX=$PUSH_TREE_LINUX
-TREE_QEMU_MAINLINE=$PUSH_TREE_QEMU_MAINLINE
-TREE_QEMU_UPSTREAM=$XENBITS:/home/xen/git/qemu-upstream-${xenbranch#xen-}.git
+TREE_QEMU_UPSTREAM=$PUSH_TREE_QEMU_UPSTREAM
 TREE_XEN=$PUSH_TREE_XEN
 TREE_LIBVIRT=$PUSH_TREE_LIBVIRT
 TREE_RUMPUSERXEN=$PUSH_TREE_RUMPUSERXEN
@@ -67,12 +66,28 @@ xen-*-testing)
 	;;
 qemu-mainline)
 	cd $repos/qemu-mainline
-	git push $TREE_QEMU_MAINLINE $revision:refs/heads/mainline/xen-tested-master
+	git push $TREE_QEMU_UPSTREAM $revision:refs/heads/upstream-tested
 	;;
-qemu-upstream-*)
+qemu-upstream-unstable)
 	cd $repos/$branch
         git push $TREE_QEMU_UPSTREAM $revision:refs/heads/master
         ;;
+qemu-upstream-*-testing)
+	branchcore=${branch#qemu-upstream-}
+	branchcore=${branchcore%-testing}
+	cd $repos/$branch
+	git push $TREE_QEMU_UPSTREAM $revision:refs/heads/stable-$branchcore
+
+	# For now also push to the old split trees for historical
+	# branches only (qemu-upstream started with xen-4.2-testing
+	# and the split trees end at xen-4.6-testing)
+	case "$branchcore" in
+	    4.[23456])
+		tree=$XENBITS:/home/xen/git/qemu-upstream-${branchcore}-testing.git
+		git push $tree $revision:refs/heads/master
+		;;
+	esac
+        ;;
 linux-3.*)
 	cd $repos/linux
 	git push $TREE_LINUX $revision:refs/heads/tested/$branch
-- 
2.5.3

Re: [PATCH OSSTEST] Switch to merged qemu-xen{, -traditional}.git trees

[Ian Jackson]

Ian Campbell writes ("Re: [PATCH OSSTEST] Switch to merged qemu-xen{,-traditional}.git trees"):
> Yes, v2 is below for completeness. I rebased this over the smoketest stuff
> which added $xenbranch_forqemu which of course is now no longer needed so
> the rebase just involved taking my version of each conflict and removing
> the initialisation of xenbranch_forqemu. Accordingly I've retained your
> Ack, I hope that is ok.

Fair enough.

> We discussed on IRC with you and Stefano and are going to aim to push this
> in the w/c 19 October.

Right.

Ian.

Re: [PATCH XEN] Config: Switch to unified qemu trees.

[Ian Jackson]

Ian Campbell writes ("Re: [PATCH XEN] Config: Switch to unified qemu trees."):
> >From dff5c395c1d23c21238ce17ddcd6f7abe2efd08d Mon Sep 17 00:00:00 2001
> From: Ian Campbell <ian.campbell@citrix.com>
> Date: Thu, 10 Sep 2015 14:31:34 +0100
> Subject: [PATCH] Config: Switch to unified qemu trees.
> 
> Upstream qemu is now in qemu-xen.git and the trad fork is in
> qemu-xen-traditional.h

As you say on irc, you mean .git.

> QEMU_UPSTREAM_REVISION is currently a tag and
> QEMU_TRADITIONAL_REVISION is a specific revision, so no changes are
> required to those.

Apart from that,

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Re: [PATCH OSSTEST] Switch to merged qemu-xen{, -traditional}.git trees

[Ian Jackson]

Ian Campbell writes ("Re: [PATCH OSSTEST] Switch to merged qemu-xen{,-traditional}.git trees"):
> We discussed on IRC with you and Stefano and are going to aim to push this
> in the w/c 19 October.

We have decided under the circumstances to postpone this to next week.

It would probably have been possible for me to pick things up from
the deployment plan in your mail
   [PATCH OSSTEST+XEN 0/1+1] Switching to one qemu tree per qemu version (trad vs upstream)
But it seems better to wait for you to be back.

Ian.

Re: [PATCH OSSTEST] Switch to merged qemu-xen{, -traditional}.git trees

[Ian Campbell]

On Mon, 2015-10-19 at 12:44 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [PATCH OSSTEST] Switch to merged qemu-xen{,
> -traditional}.git trees"):
> > We discussed on IRC with you and Stefano and are going to aim to push
> > this
> > in the w/c 19 October.
> 
> We have decided under the circumstances to postpone this to next week.
> 
> It would probably have been possible for me to pick things up from
> the deployment plan in your mail
>    [PATCH OSSTEST+XEN 0/1+1] Switching to one qemu tree per qemu version
> (trad vs upstream)
> But it seems better to wait for you to be back.

That was probably wise, although I'm on vacation next week.

I thought maybe I'd be able to manage it today given the existence of the
checklist, but I think actually I'm too fuzzy headed even for that.

I made some updates to the plan last week, see below.

Ian.


 * Today we can already just remove the old staging/qemu-xen-* trees, they
   are unused (apart from being manually pushed to along with the staging
   trees, I think).
   
 * Move the two new trees out of people/ianc into the correct places.
   Ensure git-http etc all works and that Stefano + IanJ have appropriate
   permissions.

 * See if there is a way to prevent pushes to the old trees (e.g. a
   setting in their .git/config file).

 * Test osstest patch <1443793989.11707.121.camel@citrix.com>. Quoth iwj:

   * explicitly ap-push and ap-fetch the relevant trees. Perhaps do
     the ap-push as a user without the appropriate permissions to get
     a dry run.

   * Update patch as necessary.
   
 * Push resulting tested osstest patch. Probably force push
 
 * Once that change is in osstest.git#production Stefano and Ian would
   switch to pushing to the appropriate staging* branches new trees.

   osstest will ignore the old staging trees and osstest will update both
   the new master/stable branches and the old stable trees (but not the old
   qemu-upstream-unstable#master branch).
   
 * ASAP after the osstest patch reaches production push the patch
   <1442486509.18856.166.camel@citrix.com> to xen.git#staging.

   NOTE: s/qemu-xen-traditional\.h/qemu-xen-traditional.git./ on the commit
   message and add Ian J's ack.
   
   This will cause the xen-unstable builds to use the new output gate.

   Until this is done unstable builds will continue using the old
   master push gate, which is not updated after the osstest update
   (only stable branches are).
   
 * Remove the old staging/qemu-upstream-* trees, they are not
   referenced by anything.
   
 * At our leisure backport the xen.git change to stable branches, probably
   back as far as 4.2 (when qemu-xen was introduced).
   
 * Do stable releases of each of the above.
 
 * Drop (one by one or all at once) the push to the legacy stable branches
   from osstest as stable releases are made referencing the new trees.
   
 * Consider hiding (or removing) the old output trees from xenbits as well.
   => Not possible with current gitweb setup.

Re: [PATCH OSSTEST] Switch to merged qemu-xen{, -traditional}.git trees

[Ian Campbell]

On Tue, 2015-10-20 at 11:34 +0100, Ian Campbell wrote:

>  * See if there is a way to prevent pushes to the old trees (e.g. a
>    setting in their .git/config file).

It appears that it is trivial to block force pushes or pushes to the
current branch (which I think might be meaningless for a bare repo) but
that blocking all pushes involves using .git/hooks. I think this is
probably too much faff, or at least not worth putting on the critical path.

Ian.

Re: [PATCH OSSTEST] Switch to merged qemu-xen{, -traditional}.git trees

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [PATCH OSSTEST] Switch to merged qemu-xen{, -traditional}.git trees"):
> On Tue, 2015-10-20 at 11:34 +0100, Ian Campbell wrote:
> >  * See if there is a way to prevent pushes to the old trees (e.g. a
> >    setting in their .git/config file).
> 
> It appears that it is trivial to block force pushes or pushes to the
> current branch (which I think might be meaningless for a bare repo) but
> that blocking all pushes involves using .git/hooks. I think this is
> probably too much faff, or at least not worth putting on the critical path.

These hooks aren't too hard to write.  See
  xenbits:~xen/git/xen.git/hooks/pre-receive
for an example.

Ian.

Re: [PATCH OSSTEST] Switch to merged qemu-xen{, -traditional}.git trees

[Ian Campbell]

On Wed, 2015-10-21 at 11:35 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [Xen-devel] [PATCH OSSTEST] Switch to merged
> qemu-xen{, -traditional}.git trees"):
> > On Tue, 2015-10-20 at 11:34 +0100, Ian Campbell wrote:
> > >  * See if there is a way to prevent pushes to the old trees (e.g. a
> > >    setting in their .git/config file).
> > 
> > It appears that it is trivial to block force pushes or pushes to the
> > current branch (which I think might be meaningless for a bare repo) but
> > that blocking all pushes involves using .git/hooks. I think this is
> > probably too much faff, or at least not worth putting on the critical
> > path.
> 
> These hooks aren't too hard to write.  See
>   xenbits:~xen/git/xen.git/hooks/pre-receive
> for an example.

In this case it looks then like:
    #!/bin/sh
    set -e
    echo >&2 "Pushes are not allowed to this tree"
    exit 1

is all that is needed.

Ian.

Re: [PATCH OSSTEST] Switch to merged qemu-xen{, -traditional}.git trees

[Ian Campbell]

On Tue, 2015-10-20 at 11:34 +0100, Ian Campbell wrote:
> On Mon, 2015-10-19 at 12:44 +0100, Ian Jackson wrote:
> > Ian Campbell writes ("Re: [PATCH OSSTEST] Switch to merged qemu-xen{,
> > -traditional}.git trees"):
> > > We discussed on IRC with you and Stefano and are going to aim to push
> > > this
> > > in the w/c 19 October.
> > 
> > We have decided under the circumstances to postpone this to next week.
> > 
> > It would probably have been possible for me to pick things up from
> > the deployment plan in your mail
> >    [PATCH OSSTEST+XEN 0/1+1] Switching to one qemu tree per qemu
> > version
> > (trad vs upstream)
> > But it seems better to wait for you to be back.
> 
> That was probably wise, although I'm on vacation next week.
> 
> I thought maybe I'd be able to manage it today given the existence of the
> checklist, but I think actually I'm too fuzzy headed even for that.

I am back and well enough to begin, so I am doing this now.

Ian.

Re: [PATCH OSSTEST] Switch to merged qemu-xen{, -traditional}.git trees

[Ian Campbell]

On Tue, 2015-10-20 at 11:34 +0100, Ian Campbell wrote:
> On Mon, 2015-10-19 at 12:44 +0100, Ian Jackson wrote:
> > Ian Campbell writes ("Re: [PATCH OSSTEST] Switch to merged qemu-xen{,
> > -traditional}.git trees"):
> > > We discussed on IRC with you and Stefano and are going to aim to push
> > > this
> > > in the w/c 19 October.
> > 
> > We have decided under the circumstances to postpone this to next week.
> > 
> > It would probably have been possible for me to pick things up from
> > the deployment plan in your mail
> >    [PATCH OSSTEST+XEN 0/1+1] Switching to one qemu tree per qemu
> > version
> > (trad vs upstream)
> > But it seems better to wait for you to be back.
> 
> That was probably wise, although I'm on vacation next week.
> 
> I thought maybe I'd be able to manage it today given the existence of the
> checklist, but I think actually I'm too fuzzy headed even for that.
> 
> I made some updates to the plan last week, see below.
> 
> Ian.
> 
> 
>  * Today we can already just remove the old staging/qemu-xen-* trees, they
>    are unused (apart from being manually pushed to along with the staging
>    trees, I think).

I have interpreted remove here (and everywhere else) as "move to
~xen/git.single-qemu.deleteme".

I did however delete the corresponding symlinks from /var/xenbits
-www/html/git-http/

>  * Move the two new trees out of people/ianc into the correct places.
>    Ensure git-http etc all works and that Stefano + IanJ have appropriate
>    permissions.

Appropriate permissions == owned by xen:xenmaint and g+s set for the
directories.

Also, in *.git/config:
    [receive]
            denyNonFastForwards = true
            unpackLimit = 10000
    [gc]
            autopacklimit = 25

Copied from xen.git/config. I also copied xen.git/hooks/pre-receive which
prevents pushes to non-staging branches (except by osstest) and prevents
pushes of unannottated tags.

I also in both cases moved hooks/post-update.sample to hooks/post-update,
which runs update-server-info. I then ran this by hand in both repos.

I also updated config/description of new and old trees.

I also edited ~xen/HG/patchbot/versions to add the appropriate lines for
all the new branches. That is I removed them and added:

/home/xen/git qemu-xen.git#master               xen-changelog@lists.xensource.com       xen-devel@lists.xensource.com
/home/xen/git qemu-xen.git#stable-4.6           xen-changelog@lists.xensource.com       xen-devel@lists.xensource.com
[...]
/home/xen/git qemu-xen.git#stable-4.2           xen-changelog@lists.xensource.com       xen-devel@lists.xensource.com
/home/xen/git qemu-xen-traditional.git#master           xen-changelog@lists.xensource.com       xen-devel@lists.xensource.com
/home/xen/git qemu-xen-traditional.git#stable-4.6               xen-changelog@lists.xensource.com       xen-devel@lists.xensource.com
[...]
/home/xen/git qemu-xen-traditional.git#stable-3.3               xen-changelog@lists.xensource.com       xen-devel@lists.xensource.com

and

/home/xen/git qemu-xen.git#staging              xen-staging@lists.xensource.com         xen-devel@lists.xensource.com
/home/xen/git qemu-xen.git#staging-4.6          xen-staging@lists.xensource.com         xen-devel@lists.xensource.com
[...]
/home/xen/git qemu-xen.git#staging-4.2          xen-staging@lists.xensource.com         xen-devel@lists.xensource.com
/home/xen/git qemu-xen-traditional.git#staging          xen-staging@lists.xensource.com         xen-devel@lists.xensource.com
/home/xen/git qemu-xen-traditional.git#staging-4.6              xen-staging@lists.xensource.com         xen-devel@lists.xensource.com
[...]
/home/xen/git qemu-xen-traditional.git#staging-3.3              xen-staging@lists.xensource.com         xen-devel@lists.xensource.com

This follows the pattern for the existing xen.git and old qemu ones, which
I've removed.

I also did moved ~xen/HG/patchbot/qemu-* to ~xen/git.single-qemu.deleteme/
_after_ having made a botched attempt to rename them appropriately for the
new scheme.

To unbotch it I regenerated from scratch for qemu-xen-traditional:

$ for i in master stable-3.{3,4} stable-4.{0,1,2,3,4,5,6} ; do r=$(git rev-parse $i); echo "echo $r > qemu-xen-traditional--$i.patchbot-reported-heads" ; done
echo bc00cad75d8bcc3ba696992bec219c21db8406aa > qemu-xen-traditional--master.patchbot-reported-heads
echo f3115dc6719e4d21c426b9395b2b30e7062b97cf > qemu-xen-traditional--stable-3.3.patchbot-reported-heads
echo 62539a7d5974cdc0a448d469cda458761940cd33 > qemu-xen-traditional--stable-3.4.patchbot-reported-heads
echo eaa1bd612f50d2f253738ed19e14981e4ede98a5 > qemu-xen-traditional--stable-4.0.patchbot-reported-heads
echo 77d9bdb27c4237a007ba93a6f159791eed317abc > qemu-xen-traditional--stable-4.1.patchbot-reported-heads
echo 112599882987da1afbbe4c16f6b049065a64f065 > qemu-xen-traditional--stable-4.2.patchbot-reported-heads
echo 1e5099d596b6f7a977d4bc040a54edc2a6a3c6a4 > qemu-xen-traditional--stable-4.3.patchbot-reported-heads
echo 5ae0569d964ad1a6d8dc781e5566d39210a5d063 > qemu-xen-traditional--stable-4.4.patchbot-reported-heads
echo dfe880e8d5fdc863ce6bbcdcaebaf918f8689cc0 > qemu-xen-traditional--stable-4.5.patchbot-reported-heads
echo 1c8d43cbdf0fc01a8f05acfbf55b805a83da34bb > qemu-xen-traditional--stable-4.6.patchbot-reported-heads

And for qemu-xen:

$ for i in master staging {staging,stable}-4.{2,3,4,5,6} ; do r=$(git rev-parse $i); echo "echo $r > qemu-xen--$i.patchbot-reported-heads" ; done
echo 816609b2841297925a223ec377c336360e044ee5 > qemu-xen--master.patchbot-reported-heads
echo 816609b2841297925a223ec377c336360e044ee5 > qemu-xen--staging.patchbot-reported-heads
echo c17e602ae64fb24405ebd256679ba9a6f5819086 > qemu-xen--staging-4.2.patchbot-reported-heads
echo b188780861662e8cf1847ec562799b32bb44f05e > qemu-xen--staging-4.3.patchbot-reported-heads
echo 5fe74249f5ab528fe84a26fa60438a6de4c787f0 > qemu-xen--staging-4.4.patchbot-reported-heads
echo e5a1bb22cfb307db909dbd3404c48e5bbeb9e66d > qemu-xen--staging-4.5.patchbot-reported-heads
echo 980dfcf5b8d8161871f81d1987b2f8ea5d7d4db9 > qemu-xen--staging-4.6.patchbot-reported-heads
echo c17e602ae64fb24405ebd256679ba9a6f5819086 > qemu-xen--stable-4.2.patchbot-reported-heads
echo b188780861662e8cf1847ec562799b32bb44f05e > qemu-xen--stable-4.3.patchbot-reported-heads
echo 5fe74249f5ab528fe84a26fa60438a6de4c787f0 > qemu-xen--stable-4.4.patchbot-reported-heads
echo e5a1bb22cfb307db909dbd3404c48e5bbeb9e66d > qemu-xen--stable-4.5.patchbot-reported-heads
echo 980dfcf5b8d8161871f81d1987b2f8ea5d7d4db9 > qemu-xen--stable-4.6.patchbot-reported-heads

The above was run in the relevant git repo and then the resulting commands
were then run in ~xen/HG/patchbot/.

Previously for qemu-xen it looks like only qemu-upstream-unstable.git
(staging and master) generated these mails, I have deliberately changed
that here so all branches should have mails.

http://lists.xen.org/archives/html/xen-devel/2015-02/msg03547.html and the
various followups were quite useful here.

I manually ran:
git clone git://xenbits.xen.org/qemu-xen.git  
git clone git://xenbits.xen.org/qemu-xen-traditional.git  
git clone http://xenbits.xen.org/git-http/qemu-xen.git qemu-xen.http
git clone http://xenbits.xen.org/git-http/qemu-xen-traditional.git qemu-xen-traditional.http

and observed that they seemed to be fetching stuff.

>  * See if there is a way to prevent pushes to the old trees (e.g. a
>    setting in their .git/config file).

I deployed the snippet in <1445424646.9563.86.camel@citrix.com> into all
the old trees. I didn't actually try anything until later during ap-push
testing, when I realised that these would stop osstest's ap-push too.
Furthermore the old qemu-xen-trad trees would not be updated to do anything
at all. After discussion with IanJ I remove the hook from the old trad
trees (qemu-xen-unstable.git and qemu-xen-X.Y-testing.git) and replaced the
one in qemu-upstream-* with:
    #!/bin/sh
    set -e
    whoami=`whoami`
    reject () {
            echo >&2 "
    *** push rejected: $1 ***
    "
            exit 1
    }
    if [ "x$whoami" != xosstest ]; then
    	    reject 'only osstest may push here'
    fi


staging/qemu-upstream-*.git retain the original hook which rejects
everything.


>  * Test osstest patch <1443793989.11707.121.camel@citrix.com>. Quoth iwj:
> 
>    * explicitly ap-push and ap-fetch the relevant trees. Perhaps do
>      the ap-push as a user without the appropriate permissions to get
>      a dry run.
> 
>    * Update patch as necessary.

This highlighted a missing } on:

+: ${PUSH_TREE_QEMU_UPSTREAM=$XENBITS:/home/xen/git/qemu-xen.git

With that fixed I ran ap-fetch-version, ap-fetch-version-baseline and ap
-fetch-version-old on a representative set of branches
    qemu-upstream-unstable
    qemu-upstream-4.6-testing
    qemu-upstream-4.2-testing

I skipped ap-fetch-version-baseline-late since that only does anything for
linux-next. There is nothing to test for qemu-xen-traditional since there
is no gate.

Since everything is quiescent at the moment all of the above got the same
answer for a given branch.

I repeated with qemu-mainline and as expected ap-fetch-version returned
something different to the other two (which were the same as each other).

I then for each branch above list above I ran, as osstest on the production
vm:
    ./ap-fetch-version-baseline $branch
    ./ap-push $branch <result of fetch>

For qemu-upstream-unstable this did:
+ git push osstest@xenbits.xen.org:/home/xen/git/qemu-xen.git 816609b2841297925a223ec377c336360e044ee5:refs/heads/master

For qemu-upstream-4.6-testing it did:
+ git push osstest@xenbits.xen.org:/home/xen/git/qemu-xen.git 980dfcf5b8d8161871f81d1987b2f8ea5d7d4db9:refs/heads/stable-4.6
+ git push osstest@xenbits.xen.org:/home/xen/git/qemu-upstream-4.6-testing.git 980dfcf5b8d8161871f81d1987b2f8ea5d7d4db9:refs/heads/master

qemu-upstream-4.2-testing similarly.

For qemu-mainline this did:
+ git push osstest@xenbits.xen.org:/home/xen/git/qemu-xen.git 26c7be842637ee65a79cd77f96a99c23ddcd90ad:refs/heads/upstream-tested

In all cases the push was "Everything up-to-date" and the paths and
branches were as expected.
   
>  * Push resulting tested osstest patch. Probably force push

Done, by force push.

>  * Once that change is in osstest.git#production Stefano and Ian would
>    switch to pushing to the appropriate staging* branches new trees.
> 
>    osstest will ignore the old staging trees and osstest will update both
>    the new master/stable branches and the old stable trees (but not the
> old
>    qemu-upstream-unstable#master branch).
>    
>  * ASAP after the osstest patch reaches production push the patch
>    <1442486509.18856.166.camel@citrix.com> to xen.git#staging.
> 
>    NOTE: s/qemu-xen-traditional\.h/qemu-xen-traditional.git./ on the
> commit
>    message and add Ian J's ack.
>    
>    This will cause the xen-unstable builds to use the new output gate.
> 
>    Until this is done unstable builds will continue using the old
>    master push gate, which is not updated after the osstest update
>    (only stable branches are).

Patch pushed to staging.

>  * Remove the old staging/qemu-upstream-* trees, they are not
>    referenced by anything.

Done (by move aside a discussed above).

The remainder are things which should not happen immediately.

Including a new step:

 * Remove non-staging qemu-upstream-unstable.git and qemu-xen-unstable.git
   which can occur once any flights which might use them have been and gone
   (including possible bisect attempts)

>    
>  * At our leisure backport the xen.git change to stable branches,
> probably
>    back as far as 4.2 (when qemu-xen was introduced).
>    
>  * Do stable releases of each of the above.
>  
>  * Drop (one by one or all at once) the push to the legacy stable
> branches
>    from osstest as stable releases are made referencing the new trees.
>    
>  * Consider hiding (or removing) the old output trees from xenbits as
> well.
>    => Not possible with current gitweb setup.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH XEN] Config: Switch to unified qemu trees.

[Ian Campbell]

On Fri, 2015-10-16 at 12:34 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [PATCH XEN] Config: Switch to unified qemu
> trees."):
> > > From dff5c395c1d23c21238ce17ddcd6f7abe2efd08d Mon Sep 17 00:00:00
> > > 2001
> > From: Ian Campbell <ian.campbell@citrix.com>
> > Date: Thu, 10 Sep 2015 14:31:34 +0100
> > Subject: [PATCH] Config: Switch to unified qemu trees.
> > 
> > Upstream qemu is now in qemu-xen.git and the trad fork is in
> > qemu-xen-traditional.h
> 
> As you say on irc, you mean .git.
> 
> > QEMU_UPSTREAM_REVISION is currently a tag and
> > QEMU_TRADITIONAL_REVISION is a specific revision, so no changes are
> > required to those.
> 
> Apart from that,
> 
> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

As I've just announce separately this is now done and this patch is in the
tree.

Please can you queue this up for backport back to 4.2.

It will conflict at every step, due to the version number in the old URLs,
but you can trivially take the new version in every case.

It would be worth double checking that the REVISION variables point to a
sha1 or to a tag and not to a branch name. I believe this is the case in
all stable branches.

Ian.

Re: [PATCH OSSTEST] Switch to merged qemu-xen{, -traditional}.git trees

[Ian Campbell]

On Wed, 2015-10-21 at 14:15 +0100, Ian Campbell wrote:
> /home/xen/git qemu-xen.git#staging              xen-staging@lists.xensource.com         xen-devel@lists.xensource.com
> /home/xen/git qemu-xen.git#staging-4.6          xen-staging@lists.xensource.com         xen-devel@lists.xensource.com
> [...]
> /home/xen/git qemu-xen.git#staging-4.2          xen-staging@lists.xensource.com         xen-devel@lists.xensource.com
> /home/xen/git qemu-xen-traditional.git#staging          xen-staging@lists.xensource.com         xen-devel@lists.xensource.com
> /home/xen/git qemu-xen-traditional.git#staging-4.6              xen-staging@lists.xensource.com         xen-devel@lists.xensource.com
> [...]
> /home/xen/git qemu-xen-traditional.git#staging-3.3              xen-staging@lists.xensource.com         xen-devel@lists.xensource.com


There are no staging trees for qemu-xen-traditional. I have removed these
lines.

Re: [PATCH OSSTEST] Switch to merged qemu-xen{, -traditional}.git trees

[Ian Campbell]

On Wed, 2015-10-21 at 14:15 +0100, Ian Campbell wrote:
> >  * Remove the old staging/qemu-upstream-* trees, they are not
> >    referenced by anything.
> 
> Done (by move aside a discussed above).

This was premature and I have put them back.

The reason is that all osstest flights prior to the updated patch were
cloning from the staging URL.

This is because ap-print-url doesn't distinguish e.g. flights on qemu
-upstream-4-2-testing from flights on xen-4.2-testing and used staging/
since that would always have all the commits in it.

This ends up in tree_qemuu in the 4.2-testing flight and then tries to be
used for bisections and breaks.

These trees can be removed as part of:

> Including a new step:
> 
>  * Remove non-staging qemu-upstream-unstable.git and qemu-xen-unstable.git
>    which can occur once any flights which might use them have been and gone
>    (including possible bisect attempts)

I am also going to go an unbless some bisection flights which have failed
due to this.

Ian.

Re: [PATCH OSSTEST] Switch to merged qemu-xen{, -traditional}.git trees

[Ian Campbell]

On Wed, 2015-10-21 at 15:12 +0100, Ian Campbell wrote:
> On Wed, 2015-10-21 at 14:15 +0100, Ian Campbell wrote:
> > >  * Remove the old staging/qemu-upstream-* trees, they are not
> > >    referenced by anything.
> > 
> > Done (by move aside a discussed above).
> 
> This was premature and I have put them back.
> 
> The reason is that all osstest flights prior to the updated patch were
> cloning from the staging URL.
> 
> This is because ap-print-url doesn't distinguish e.g. flights on qemu
> -upstream-4-2-testing from flights on xen-4.2-testing and used staging/
> since that would always have all the commits in it.
> 
> This ends up in tree_qemuu in the 4.2-testing flight and then tries to be
> used for bisections and breaks.

This reasoning also applies to the qemu-xen-trad trees, which end up in
tree_qemu.

Hence I've put those ones back too.

Ian.

Re: [PATCH OSSTEST] Switch to merged qemu-xen{, -traditional}.git trees

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [PATCH OSSTEST] Switch to merged qemu-xen{, -traditional}.git trees"):
> Copied from xen.git/config. I also copied xen.git/hooks/pre-receive which
> prevents pushes to non-staging branches (except by osstest) and prevents
> pushes of unannottated tags.

This prevented my pushes to qemu-xen-traditional.git, since it doesn't
have staging branches.

I edited the hooks/pre-receive and the relevant bit now looks like
this:

10      while read old new refname rest; do
11              case $refname in
12              refs/heads/master)      ;;
13              refs/heads/stable-*)    ;;
14      #       refs/heads/*staging*)   ;;
15      #       refs/heads/*)
16      #               if [ "x$whoami" != xosstest ]; then
17      #                       reject 'push only to staging branches'
18      #               fi
19      #                ;;
20              refs/tags/*)

Ian.