From: "Roger Pau Monné" <roger.pau@citrix.com>
To: Ian Jackson <ian.jackson@citrix.com>
Cc: xen-devel@lists.xenproject.org
Subject: Re: [Xen-devel] [PATCH 5/6] osstest: introduce a script to build a FreeBSD package repository
Date: Fri, 24 May 2019 11:13:43 +0100 [thread overview]
Message-ID: <20190524101343.2rerk7u5jf5shtsi@Air-de-Roger> (raw)
Message-ID: <20190524101343.9dcOp1hIcSvYbaiLDfT3zekK1qKKFOQJ5MchpVWZuO0@z> (raw)
In-Reply-To: <23782.30785.859446.290967@mariner.uk.xensource.com>
On Thu, May 23, 2019 at 11:38:57AM +0100, Ian Jackson wrote:
> Roger Pau Monne writes ("[Xen-devel] [PATCH 5/6] osstest: introduce a script to build a FreeBSD package repository"):
> > diff --git a/make-freebsd-flight b/make-freebsd-flight
> > index d3c413b5..fc3d2d83 100755
> > --- a/make-freebsd-flight
> > +++ b/make-freebsd-flight
> > @@ -38,13 +38,15 @@ job_create_build_filter_callback () {
> >
> > for arch in "$arches"; do
> > set_freebsd_runvars
> > -
> > create_freebsd_build_job build-$arch-freebsd
> >
> > - # Create an identical job that's going to use the build output from
> > - # the previous one.
> > + # Create a job to build the packages against the new world.
> > freebsd_runvars="$freebsd_runvars freebsdbuildjob=build-$arch-freebsd \
> > recipe_testinstall=true"
> > + create_freebsd_pkg_build_job build-$arch-freebsd-packages
> > +
> > + # Create an identical job that's going to use the build output from
> > + # the previous one.
> > create_freebsd_build_job build-$arch-freebsd-again
> >
> > # Create a Xen build job that's going to use the output from the first
>
> This looks OK.
>
> > @@ -768,7 +773,9 @@ proc prepare-build-host-freebsd {} {
> > global jobinfo
> > if {[recipe-flag testinstall]} { set broken fail } { set broken broken }
> > run-ts $broken host-install(*) ts-freebsd-host-install
> > - run-ts . host-build-prep ts-build-prep-freebsd
> > + if {![recipe-flag skipbuildprep]} {
> > + run-ts . host-build-prep ts-build-prep-freebsd
>
> What's this for ? Oh, I see.
The job that creates the package repository cannot use build-prep
because the packages are not yet built.
> I notice that none of your freebsd build jobs pass any share- hostflag
> so they always use a fresh installation. Is that necessary ?
Hm, I don't think so. build-amd64-xen-freebsd and
build-amd64-freebsd-again could share a host. I need to take a look at
how to do this, I could send this as a separate fix for the existing
jobs.
> > proc need-hosts/coverity {} { return BUILD_LINUX }
> > diff --git a/ts-freebsd-build-packages b/ts-freebsd-build-packages
> > new file mode 100755
> > index 00000000..9202dd9f
> > --- /dev/null
> > +++ b/ts-freebsd-build-packages
> > @@ -0,0 +1,145 @@
> > +#!/usr/bin/perl -w
> > +# This is part of "osstest", an automated testing framework for Xen.
> > +# Copyright (C) 2019 Citrix Inc.
> ...
> > +# Consumes the following input runvars:
> > +# svnrevision_freebsdports: ports svn revision id to use.
> > +# svntree_freebsdports ports svn tree to fetch the source code from.
>
> More regular in osstest terms would be
> tree_freebsdports
> revision_freebsdports
> treevcs_freebsdports=svn
> But I guess svn is sufficiently unlike what osstest expects out of a
> vcs that this is not feasible, and it is better to do it this way.
I don't really have an opinion, I somehow assumed that using the same
format might interfere with things like bisection, so I've decided to
pass the git revision using tree_freebsdports &c and the svn revision
using the newly introduced flags.
> > +sub checkout () {
> > + my $u = URI->new($c{HttpProxy});
> > + my $host = $u->host;
> > + my $port = $u->port;
> > + prepbuilddirs();
> > +
> > + logm("Checkout ports tree from svn");
> > + target_cmd_build($ho, 4000, $builddir, <<END);
> > +cd $builddir
> > +rm -rf ports
> > +# svn ignores HTTP_PROXY envvar
> > +svnlite checkout --config-option servers:global:http-proxy-host=$host \\
> > + --config-option servers:global:http-proxy-port=$port \\
> > + --trust-server-cert \\
> > + $r{"svntree_freebsdports"} \\
> > + -r $r{"svnrevision_freebsdports"} ports
>
> Will this work to cache the checkout ?
I think so? Would https somehow prevent the caching?
> All of this says http but I
> assume it's really https ?
AFAIK svn uses the http-proxy options for both http and https.
> Typically, https clients expect to do the
> TLS themselves but I think you're using our squid mitm and that's what
> "--trust-server-cert" is doing ?
I can't really remember why I've added this option, but I'm quite
sure it was failing without it. As you say the proxy is acting as a
mitm, so that's likely why trust-server-cert is required.
> Rather than "--trust-server-cert" which disables TLS's own mitm
> protection it would be rather better to inject the osstest mitm squid
> cert into the testbed, but that may be difficult, and the risk is only
> from internal things between the build (test) box and the proxy.
I can look into this, but at the end of day this is all internal, so
I'm not sure there's a lot of risk here.
> > +sub create_jail() {
> > + my $src_prefix = $r{"freebsd_distpath"} ||
> > + get_stashed("path_freebsddist", $r{"freebsdbuildjob"});
> > + my $dst_prefix = "/root/sets";
>
> Do we need a jail for this ? We have a whole baremetal OS install
> whose entire purpose is to do this build ...
Yes, that's how the repository package builder (poudriere) works, it
requires a jail to do the package building. In our case it's not so
important, but I assume this is mostly done to always use a clean
install, so that currently installed packages on the system don't
interfere with package building.
> > +logm("FreeBSD packages built successful");
> ^ly :-)
Thanks.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
next prev parent reply other threads:[~2019-05-24 10:14 UTC|newest]
Thread overview: 35+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-02-20 16:59 [PATCH 0/6] osstest: create a local binary FreeBSD package repository Roger Pau Monne
2019-02-20 16:59 ` [PATCH 1/6] osstest: introduce a helper to stash a whole directory Roger Pau Monne
2019-05-23 9:48 ` Ian Jackson
2019-05-23 9:48 ` [Xen-devel] " Ian Jackson
2019-05-24 9:42 ` Roger Pau Monné
2019-05-24 9:42 ` [Xen-devel] " Roger Pau Monné
2019-02-20 16:59 ` [PATCH 2/6] osstest: introduce a helper to create a weblink to a directory Roger Pau Monne
2019-05-23 9:57 ` Ian Jackson
2019-05-23 9:57 ` [Xen-devel] " Ian Jackson
2019-02-20 16:59 ` [PATCH 3/6] osstest: allow to perform multiple anoints in the same transaction Roger Pau Monne
2019-05-23 10:00 ` Ian Jackson
2019-05-23 10:00 ` [Xen-devel] " Ian Jackson
2019-02-20 16:59 ` [PATCH 4/6] osstest: introduce a helper to get the svn revision of a git commit Roger Pau Monne
2019-05-23 10:03 ` Ian Jackson
2019-05-23 10:03 ` [Xen-devel] " Ian Jackson
2019-05-24 9:57 ` Roger Pau Monné
2019-05-24 9:57 ` [Xen-devel] " Roger Pau Monné
2019-05-24 10:35 ` Ian Jackson
2019-05-24 10:35 ` [Xen-devel] " Ian Jackson
2019-02-20 17:00 ` [PATCH 5/6] osstest: introduce a script to build a FreeBSD package repository Roger Pau Monne
2019-05-23 10:19 ` Ian Jackson
2019-05-23 10:19 ` [Xen-devel] " Ian Jackson
2019-05-23 10:38 ` Ian Jackson
2019-05-23 10:38 ` [Xen-devel] " Ian Jackson
2019-05-24 10:13 ` Roger Pau Monné [this message]
2019-05-24 10:13 ` Roger Pau Monné
2019-05-24 11:21 ` Ian Jackson
2019-05-24 11:21 ` [Xen-devel] " Ian Jackson
2019-02-20 17:00 ` [PATCH 6/6] osstest: use a locally built pkg repository for FreeBSD Roger Pau Monne
2019-05-23 11:06 ` Ian Jackson
2019-05-23 11:06 ` [Xen-devel] " Ian Jackson
2019-05-24 10:45 ` Roger Pau Monné
2019-05-24 10:45 ` [Xen-devel] " Roger Pau Monné
2019-05-24 17:26 ` Ian Jackson
2019-05-24 17:26 ` [Xen-devel] " Ian Jackson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190524101343.2rerk7u5jf5shtsi@Air-de-Roger \
--to=roger.pau@citrix.com \
--cc=ian.jackson@citrix.com \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).