[HACKATHON] Toolstack session

* [HACKATHON] Toolstack session
@ 2016-04-20 18:20 Roger Pau Monné
  2016-04-26 13:44 ` xen.git build system (Re: [HACKATHON] Toolstack session) Wei Liu
  0 siblings, 1 reply; 6+ messages in thread
From: Roger Pau Monné @ 2016-04-20 18:20 UTC (permalink / raw)
  To: xen-devel

Toolstack session
=================

Topics:

 - Build system
 - Stubdom
 - Testing
 - {o/c}xenstored

Wei: what downstream consumers expect from the build system. Xen has a top 
level makefile that builds everything, by pulling other projects source 
code. Trying to make it cleaner.

George: someone recommended to pull grub2 into the Xen build system, but it 
was seen as too much. Try to remove other pieces that Xen build system pull 
in in order to perform a build. Package Raisin in a way that includes all 
the needed dependencies (source).

Doug: Gentoo/Yocto build system based on the one from Debian. It's not good 
to pull things from the internet when performing a build. Yocto build system 
disables network access when performing a build, custom patches are needed 
in order to fix that. XenServer has to do the same.

George: In general people packaging Xen needs to work around the fact that 
Xen pulls things from the internet while building

Doug: this doesn't work for stubdom

Olaf: it's possible to do the same for stubdom, Suse does it.

Doug: stubdom build always call the git binary, even if the code it's 
already there.

Ian: Debian doesn't build stubdom.

Wei: it's a PITA that the Xen build system downloads stuff while building. 
Need to solve this.

Ian: two different things in the build system:

 - Our build process downloads stuff while building. This annoys packagers, 
but it's probably important for users that build Xen from source. We cannot 
force them to download stuff manually.
 - Release tarballs _still_ need to download more stuff while building 
(firmware?).

How to fix:
 - Everything controlled by the Xen build system, make it clear what will be 
downloaded, have a target to download the required sources.
 - Use Raisin or something similar, that pulls in everything.

George: difficulties building things from outside, like blktap. Requires 
libxc, which is inside the Xen build system.

Juergen: add the download section to configure.

Ian: seems worse than adding it to the makefile.

Olaf: configure to select what to download/install.

Ian: add download options to configure, add a download target to the build 
makefile. Default to stay the same as it is now.

Konrad: configure to create a list of what Xen build wants to download.

Ian: hard to make it declarative.

****: the build process is taking very long. Sometimes it fails, shouldn't 
re-download things again.

Ian: clean only cleans objects, distclean cleans everything downloaded. 
Maybe add a configure or make target to pull and build pvgrub2?

George: some distros might only want to compile core, and use Qemu/pvgrub2.. 
from other distro packages.

Doug: important to keep in sync with upstream, having custom (cherry-picked) 
patches on top of upstream repos makes it harder for distro to use the 
already present-packages.

George: try to clean things up, clear separation between download and build 
phases.

Andrew: it would be interesting to be able to pass libraries or object files 
to the Xen build system, instead of only pointing it to source 
trees/packages.

George: Raisin would help with that.

Ian: hard to modify Raisin to do this, will take a long time to get there. 
Maybe will take care of those changes, will try to make things not harder 
for people that wants to build things independently.

Stubdom
=======

Wei: stubdom in tree, it's not getting much attention.

Ian: not to be confused with rump-kernels. Should Wei continue working on 
splitting up stubdom? Should it be split from the Xen source tree?

Doug: different release cadence.

Ian: would make it easier for distros to consume maybe.

Wei: I will post patches to split stubdom from the repo.

****: What about using Linux in stubdoms?

Roger: Linux kernel can only be built on Linux.

George: have a script that you can call to build it?

David: use things already present in order to build Linux distros, like 
Yocto or even the Debian packages.

Ian: in any case, it should be an opt-in disabled by default, since it's not 
possible to build Linux on != Linux. We should also have a clear protocol 
about how a stubdom interacts with the toolstack. Toolstack needs to tell 
stubdom which devices will be available to the guest.

Wei: I've done some fixes to it, seems better ATM.

Doug: get the protocol into a standard, in a way that stubdoms would be 
interchangeable (Linux/MiniOS/rump).

Wei: Linux stubdom can become a separate project, but there needs to be a 
clear and standard protocol.

Testing
=======

Wei: Upstream relies on OSSTest, tracks both Xen and other projects (Linux, 
QEMU, SeaBIOS...). Tight on resources, more machines would help testing 
things faster. 

Doug: other projects do more tests before anything even reaching staging.

Luis: 0-day testing has been offered to Xen, need to speak about which Xen 
or Linux versions should be tested.

David: that runs on top of KVM, so it would only allow us to the test PV 
Dom0.

Andrew: that's what needs more testing.

David: reduce some load, reduce the number of Linux branches that we test.

Ian: remove maybe every Linux branch except for Linus master and the stable 
branches. Improve performance by reducing the number of re-installs of 
certain jobs, we would have a performance boost of 20% maybe.

Ian: a dedicated build box doesn't seem to have a lot of difference. The 
real delay is availability of test boxes.

Andrew: smoke test only runs every 4h, and takes 3h to finish. XenServer 
test system is much faster.

Ian: OSSTest lacks machines in order to test more often/faster. Maybe split 
to a model where there are several input branches that get merged into the 
main repository, this would force committers to keep their branch in good 
state.

****: How to split the threes?

****: by committers, or by code areas (x86, tools...).

{o/c}xenstore
=============

Luis: oxenstore is smaller, more performant, is it the default?

Andrew: if ocaml is found, it is compiled by default.

George: also start the ocaml one by default.

Luis: can we get rid of the C one?

Andrew: no, it's used by the xenstore stubdom, would need someone to port 
the xenstore to MirageOS.

****: almost every distro/OS has a ocaml compiler, so it *could* be made the 
default.

Andrew: XenServer has been using it for ~9 years.

Dario: add a test for cxenstored-stubdom, so that we can make the ocaml one 
the default, and still get test coverage on the C one.

Ross: it should be quite easy to get the MirageOS oxenstored stubdom.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 6+ messages in thread