[OSSTEST PATCH] README.hardware-acquisition

[OSSTEST PATCH] README.hardware-acquisition

[Ian Jackson]

New document-cum-checklist, for helping with hardware procurement.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
CC: infra@xenproject.org
---
 README.hardware-acquisition | 310 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 310 insertions(+)
 create mode 100644 README.hardware-acquisition

diff --git a/README.hardware-acquisition b/README.hardware-acquisition
new file mode 100644
index 00000000..8f196109
--- /dev/null
+++ b/README.hardware-acquisition
@@ -0,0 +1,310 @@
+====================================
+# HARDWARE ACQUISITION FOR OSSTEST #
+====================================
+
+This document can be used as a checklist when procuring hardware for
+an osstest instance.  A few of the points have details specific to the
+Xen Project test lab in Massachusetts, but most of it will be relevant
+to all osstest installations.
+
+
+POWER
+=====
+
+osstest needs to turn each host on and off under program control.
+
+When a host is power cycled, all state in it must be reset.  This
+includes onboard control and management software (eg IPMI), since such
+systems can be buggy and bugs in them can be provoked by bugs in
+system software (ie, buggy versions of Xen can break the LOM, even if
+the LOM, unusually, is not simply flaky).
+
+However, it is often necessary to use the LOM (Lights Out Management)
+as part of the poweron/poweroff sequence as otherwise some machines
+draw enough current to wear out our mains PDU contacts too quickly.
+
+(I use the English word `mains' for the single phase 110V/220V-240V AC
+electrical power supply prevalent in datacentres.)
+
+Requirements for typical server hardware
+----------------------------------------
+
+ * If the system has a LOM it should be driveable with Free Software,
+   eg via the IPMI protocol.
+
+ * Redundant PSUs are not required.
+
+ * Provisioning: One PDU port is required per host.
+
+Requirements for embedded or devboard hardware
+----------------------------------------------
+
+ * There must be arrangements to control the actual power supply
+   to each board (node).  Options include:
+
+     (i) Each node has a separate mains power supply, each of which
+         we will plug into a PDU port.
+
+     (ii) A separate management or PDU board or backplane, which
+         has one single mains power input and which has relays
+         or similar to control power to individual nodes.
+         The management system must have its own separate network
+         connection and not be at risk of corruption from
+         bad software on nodes.
+
+ * Provisioning:
+    + Number of PDU ports required depends on the approach taken.
+    + With a separate PDU controller, a switch port is required.
+
+
+SERIAL
+======
+
+We always use hardware serial for console output.  This is essential
+to capture kernel and hypervisor crash messages, including from early
+boot; as well as bootloader output, and so on.  We use our own serial
+concentrator hardware, separate from the systems under test.  Built-in
+console-over-LAN systems (eg IPMI serial over LAN) are not reliable
+enough for our purposes.
+
+Requirements for typical server hardware
+----------------------------------------
+
+ * At least one conventional RS232 UART, accessible to system
+   software in the conventional way.
+
+ * For ARM, supported as console by both Xen[1] and Linux[2].
+
+ * Presented on a standard 9-pin D connector.  (RJ45 is acceptable
+   if we know the pinout.)
+
+ * Provisioning: one serial concentrator port required per host.
+
+Requirements for a embedded or devboard hardware
+------------------------------------------------
+
+ * At least one suitable UART
+
+ * Supported in software by both Xen[1] and Linux[2]
+
+ * With suitable physical presentation:
+    (i)
+       + Proper RS232 (full voltage, not TTL or 3.3V)
+       + presented on a 9-pin D or RJ45 connector
+       + with known pinout;
+   or
+    (ii)
+       + Connected somehow to a USB-to-serial adapter
+       + Adapter supported by Linux[2]
+       + Multiple adapters, giving one physical USB port
+         for all nodes (ie built-in hub) preferred
+   or
+    (iii) Some other suitable arrangement to be discussed.
+
+ * Provisioning: Requires serial concentrator port(s) and/or spare USB
+   port(s) on appropriate infrastructure host(s).
+
+
+PHYSICAL PRESENTATION
+=====================
+
+ * All equipment should be mounted inside one or more 19" rack
+   mount cases.
+
+ * In as few U as possible: usually 1U (or, exceptionally, maybe 2U)
+   for a single server-type host.  
+
+ * Forbidden: External power adapters (laptop-style mains power supply
+   bricks); external USB hubs; any equipment not physically
+   restrained.  There is no shelf in the rack.
+
+ * Pair principle: Every host or node must be part of a set of several
+   identical hosts.  This allows us to distinguish hardware faults
+   from software bugs.  (In the cases of chassis with backplane, one
+   backplane is OK.)  Conversely, we want diversity to find the most
+   host-specific bugs, so usually around two of each type is best.
+
+ * Provisioning: Enough rack space must be available.
+
+
+MASS STORAGE
+============
+
+Each host needs some locally attached mass storage of its own.
+
+Requirements for typical server hardware
+----------------------------------------
+
+ * SATA controller supported by Linux[2]
+
+ * If SATA controller has multiple modes (eg, AHCI vs RAID)
+   it is sufficient for it to be supported in one mode.
+
+ * Storage redundancy is not required: one disk will do.
+
+ * SSD is not required: rotating rust is cheaper and will do.
+
+Requirements for embedded or devboard hardware
+----------------------------------------------
+
+ * Some mass storage supported by Linux[2].  Best is an onboard SATA
+   controller, connected to a SATA HDD in the same enclosure.
+   High-endurance flash drives are another possibility.
+
+ * If the hardware always starts by boot from a mass storage device,
+   that boot device must be physically read-only and separate from the
+   primary mass storage.  See BOOT ARRANGEMENTS.
+
+
+REMOTE FIRMWARE ACCESS VIA SERIAL
+=================================
+
+Configuration of the primary system firmware must be possible remotely
+using only the power and serial accesses just described.
+Specifically, interaction with the firmware via the serial port.
+
+Requirements for typical server hardware with UEFI or BIOS
+----------------------------------------------------------
+
+ * `BIOS' configuration (including the UEFI equivalent) accessible and
+   useable via BIOS `serial console redirection'.
+
+ * UEFI shell (if provided) also available via serial.
+
+ * Specifically, boot order configuration available via serial.
+
+
+Requirements for embedded or devboard hardware
+----------------------------------------------
+
+ * See BOOT ARRANGEMENTS.
+
+
+BOOT ARRANGEMENTS, NETBOOT
+==========================
+
+Every host must netboot as its first boot source.  The netboot
+configuration must be able to `chain' to the local writeable mass
+storage.  This ensures that a host can be completely wiped, even if
+bad software has corrupted the mass storage.
+
+Requirements for typical server hardware with UEFI or BIOS
+----------------------------------------------------------
+
+ * PXE and/or UEFI netboot.
+
+Requirements for embedded or devboard hardware
+----------------------------------------------
+
+ * Some firmware must be available and provided which is capable of
+   netbooting Xen[1] and Linux[2], under control from the netboot
+   server.  A suitable version of u-boot can meet this need.
+
+ * The firmware which performs the netbooting must be on a read-only
+   storage device (flagged as such in hardware, not software) so that
+   it cannot be corrupted by system software.  So it must be on a
+   separate physical storage device to the primary mass storage (see
+   MASS STORAGE, above).
+
+ * This firmware will not usually be updated.
+
+
+NETWORKING
+==========
+
+Requirements
+------------
+
+ * Each host must have at least one RJ45 ethernet port compatible
+   with ordinary 100Mbit ethernet.   xxx
+
+ * The primary ethernet port must be compatible with Linux[2].
+
+ * In the case of a chassis with backplane, it is acceptable if the
+   chassis contains an ethernet switch, provided that it is a normal
+   and reliable ethernet switch (not a proprietary interconnect).
+
+ * In the case of a system with IPMI or similar LOM, it is best if the
+   LOM has its own physical ethernet port.
+
+
+CPU, CHIPSET, MOTHERBOARD, ETC.
+===============================
+
+General advice and preferences
+------------------------------
+
+ * We prefer multicore, multisocket and NUMA systems because they
+   expose a greater variety of exciting bugs.  But we don't care much
+   about performance and we want a wide variety of different hosts.
+   We want a mixture of systems with different CPU variants and
+   feature support.
+
+ * Memory requirements are modest.  8G or 16G per host is fine. xxx
+
+Compatibility with Xen and Linux - requirements
+-----------------------------------------------
+
+(Normally these issues are not a problem for x86, except perhaps for
+the network and storage controllers - see MASS STORAGE and NETWORKING,
+above.)
+
+ * [1] Xen: The CPU and other hardware must be supported by current
+   versions of xen-unstable, at the very least.
+
+ * [2] Linux: The CPU and other hardware must be supported by existing
+   widely available versions of Linux.  There are two principal
+   requirements:
+
+   + Baremetal boot from Debian stable or stable-backports:
+
+     A suitable Linux kernel binary which can boot baremetal on the
+     proposed hardware must be available from Debian (at least
+     `stable', or, if that is not possible `stable-backports').  It is
+     not OK to require a patched version of Linux, or a version of
+     Linux built from a particular git branch, or some such.  If the
+     required kernel is not available in Debian, the vendor should
+     first work with the Debian project to ensure and validate that
+     the Debian stable-backports kernel binaries boot on the proposed
+     hardware.
+
+   + Boot under Xen with Linux kernel built from source code.
+
+     For x86, recent Linux LTS or mainline kernel source code must be
+     able to boot under Xen, on the proposed hardware.
+
+     For ARM, there is a special Xen ARM kernel branch.  It must be
+     able to boot under Xen, on the proposed hardware.
+
+ * Board-specific Linux and Xen versions are not acceptable.
+
+ * Hardware vendor offering a "board support package" is a red flag.
+   We will not be using a "board support package".  If we are offered
+   one we will need explicit confirmation, and perhaps verification,
+   of the points above.
+
+ * For ARM systems using Device Tree: xxx what to write here ?
+
+
+RELIABILITY
+===========
+
+ * osstest stresses systems in unusual ways.  The need to completely
+   wipe the machine for each test means test hosts are power cycled
+   more often than usual.
+
+ * Random failures due to unreliable hardware are not tolerable.  Some
+   hosts do not boot reliably.  Even a very small probability of a
+   random boot failure, per boot, is intolerable in this CI
+   environment: hosts are rebooted many times a day, and a random boot
+   failure looks just like a `hypervisor could not boot' bug.  (The
+   same bug would not be noticeable in a server farm where hosts are
+   nearly never rebooted.)
+
+
+NON-REQUIREMENTS
+================
+
+ * No VGA console needed.
+ * Redundant PSUs are not needed (see POWER, above).
+ * RAID is not needed (or wanted) (see MASS STORAGE, above).
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition

[Jan Beulich]

>>> On 30.10.18 at 17:13, <ian.jackson@eu.citrix.com> wrote:
> New document-cum-checklist, for helping with hardware procurement.
> 
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>

I don't think any acks should be needed here, but if so:
Acked-by: Jan Beulich <jbeulich@suse.com>

Jan



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition

[Julien Grall]

Hi Ian,

On 10/30/18 4:13 PM, Ian Jackson wrote:
> +Compatibility with Xen and Linux - requirements
> +-----------------------------------------------
> +
> +(Normally these issues are not a problem for x86, except perhaps for
> +the network and storage controllers - see MASS STORAGE and NETWORKING,
> +above.)
> +
> + * [1] Xen: The CPU and other hardware must be supported by current
> +   versions of xen-unstable, at the very least.
> +
> + * [2] Linux: The CPU and other hardware must be supported by existing
> +   widely available versions of Linux.  There are two principal
> +   requirements:
> +
> +   + Baremetal boot from Debian stable or stable-backports:
> +
> +     A suitable Linux kernel binary which can boot baremetal on the
> +     proposed hardware must be available from Debian (at least
> +     `stable', or, if that is not possible `stable-backports').  It is
> +     not OK to require a patched version of Linux, or a version of
> +     Linux built from a particular git branch, or some such.  If the
> +     required kernel is not available in Debian, the vendor should
> +     first work with the Debian project to ensure and validate that
> +     the Debian stable-backports kernel binaries boot on the proposed
> +     hardware.

While I understand this point, I think this is going to limit us a lot 
on the choice of hardware for Arm. While Debian is quite famous on the 
Server world, this is less the case on the automotive/embedded world 
where they tend to have there custom distribution.

Most of the time, the issue to boot Debian is related to the kernel. 
Yet, the price might be too high for some of the vendors to donate the 
hardware. Furthermore, at the moment Osstest is stuck on Jessie. This 
version does not support Thunder-X (stealing power and heating data for 
the past year and half...) and hardwares that will be donated in the 
next few months (i.e Renesas R-Car, and Xilinx MPSoc).

I still want to encourage vendor to work upstream but I would relax the 
requirements here to "Xen Arm kernel branch".

> +
> +   + Boot under Xen with Linux kernel built from source code.
> +
> +     For x86, recent Linux LTS or mainline kernel source code must be
> +     able to boot under Xen, on the proposed hardware.
> +
> +     For ARM, there is a special Xen ARM kernel branch.  It must be
> +     able to boot under Xen, on the proposed hardware.

I still want to keep that branch very close to Linux upstream. So it 
might be worth to say we have the liberty to not accept there patch...

> +
> + * Board-specific Linux and Xen versions are not acceptable.

... That will avoid a vendor to try to workaround this by asking to 
merge hundreds of patch in "Xen Arm kernel" branch.

> +
> + * Hardware vendor offering a "board support package" is a red flag.
> +   We will not be using a "board support package".  If we are offered
> +   one we will need explicit confirmation, and perhaps verification,
> +   of the points above.
> +
> + * For ARM systems using Device Tree: xxx what to write here ?
IIRC we are using the DT from the Linux tree. This was to workaround 
broken backward compatibility on the cubieboard. Am I correct?

I would write: "The firmware should provide a Device-Tree working for 
all version of Linux or provide a mechanisms to load a different 
Device-Tree at boot."

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition

[Stefano Stabellini]

On Tue, 30 Oct 2018, Julien Grall wrote:
> Hi Ian,
> 
> On 10/30/18 4:13 PM, Ian Jackson wrote:
> > +Compatibility with Xen and Linux - requirements
> > +-----------------------------------------------
> > +
> > +(Normally these issues are not a problem for x86, except perhaps for
> > +the network and storage controllers - see MASS STORAGE and NETWORKING,
> > +above.)
> > +
> > + * [1] Xen: The CPU and other hardware must be supported by current
> > +   versions of xen-unstable, at the very least.
> > +
> > + * [2] Linux: The CPU and other hardware must be supported by existing
> > +   widely available versions of Linux.  There are two principal
> > +   requirements:
> > +
> > +   + Baremetal boot from Debian stable or stable-backports:
> > +
> > +     A suitable Linux kernel binary which can boot baremetal on the
> > +     proposed hardware must be available from Debian (at least
> > +     `stable', or, if that is not possible `stable-backports').  It is
> > +     not OK to require a patched version of Linux, or a version of
> > +     Linux built from a particular git branch, or some such.  If the
> > +     required kernel is not available in Debian, the vendor should
> > +     first work with the Debian project to ensure and validate that
> > +     the Debian stable-backports kernel binaries boot on the proposed
> > +     hardware.
> 
> While I understand this point, I think this is going to limit us a lot on the
> choice of hardware for Arm. While Debian is quite famous on the Server world,
> this is less the case on the automotive/embedded world where they tend to have
> there custom distribution.

That's right, it's a Yocto world around here. That's also what EPAM
uses.


> Most of the time, the issue to boot Debian is related to the kernel. Yet, the
> price might be too high for some of the vendors to donate the hardware.
> Furthermore, at the moment Osstest is stuck on Jessie. This version does not
> support Thunder-X (stealing power and heating data for the past year and
> half...) and hardwares that will be donated in the next few months (i.e
> Renesas R-Car, and Xilinx MPSoc).
>
> I still want to encourage vendor to work upstream but I would relax the
> requirements here to "Xen Arm kernel branch".

+1

Xilinx MPSoC support is upstream in vanilla Linux (defconfig), and the
MPSoC is fully supported in Yocto. We did issue a ticket in the Debian
system to add support for the Xilinx MPSoC in their kernel but is hasn't
happened yet. (The fact that Debian support hasn't come up as an issue
up until now tells us that embedded folks tend to use other distros.)

In general, I think vanilla Linux kernel support would be a rather
strict requirement for many embedded boards (especially the sub 100$
ones), but it would still be better than asking for Debian kernel
support.

Overall, Julien's suggestion to require "Xen Arm kernel branch" is best.



> > +
> > +   + Boot under Xen with Linux kernel built from source code.
> > +
> > +     For x86, recent Linux LTS or mainline kernel source code must be
> > +     able to boot under Xen, on the proposed hardware.
> > +
> > +     For ARM, there is a special Xen ARM kernel branch.  It must be
> > +     able to boot under Xen, on the proposed hardware.
> 
> I still want to keep that branch very close to Linux upstream. So it might be
> worth to say we have the liberty to not accept there patch...
> 
> > +
> > + * Board-specific Linux and Xen versions are not acceptable.
> 
> ... That will avoid a vendor to try to workaround this by asking to merge
> hundreds of patch in "Xen Arm kernel" branch.

Yes, good idea. We don't want to deviate the "Xen Arm kernel" branch
significantly.


> > +
> > + * Hardware vendor offering a "board support package" is a red flag.
> > +   We will not be using a "board support package".  If we are offered
> > +   one we will need explicit confirmation, and perhaps verification,
> > +   of the points above.
> > +
> > + * For ARM systems using Device Tree: xxx what to write here ?
> IIRC we are using the DT from the Linux tree. This was to workaround broken
> backward compatibility on the cubieboard. Am I correct?
> 
> I would write: "The firmware should provide a Device-Tree working for all
> version of Linux or provide a mechanisms to load a different Device-Tree at
> boot."

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition

[Ian Jackson]

I'm going to reply to this email twice.  This email is about what seem
to be the noncontentious issues: everything *except* the requirement
in my document that the board is supported by Debian -backports
kernels, at least.


Julien Grall writes ("Re: [OSSTEST PATCH] README.hardware-acquisition"):
>   Furthermore, at the moment Osstest is stuck on Jessie. This 
> version does not support Thunder-X (stealing power and heating data for 
> the past year and half...) and hardwares that will be donated in the 
> next few months (i.e Renesas R-Car, and Xilinx MPSoc).

This is indeed a serious nuisance and is getting quite high on my
priority list.


> > +   + Boot under Xen with Linux kernel built from source code.
> > +
> > +     For x86, recent Linux LTS or mainline kernel source code must be
> > +     able to boot under Xen, on the proposed hardware.
> > +
> > +     For ARM, there is a special Xen ARM kernel branch.  It must be
> > +     able to boot under Xen, on the proposed hardware.
> 
> I still want to keep that branch very close to Linux upstream. So it 
> might be worth to say we have the liberty to not accept there patch...

My wording was unclear.  I should have written something like this:

       For ARM, there is a special Xen ARM kernel branch. The proposed
       hardware must be able to boot that version of Linux under Xen.

       If the Xen ARM Linux branch does not support the proposed
       hardware yet, the hardware should not be accepted until that is
       remedied.  Where this involves adding kernel patches to that
       branch this is subject to the approval of its maintainers,
       considering the need to keep it very close to upstream.

What do you think ?


> > + * Hardware vendor offering a "board support package" is a red flag.
> > +   We will not be using a "board support package".  If we are offered
> > +   one we will need explicit confirmation, and perhaps verification,
> > +   of the points above.
> > +
> > + * For ARM systems using Device Tree: xxx what to write here ?
> 
> IIRC we are using the DT from the Linux tree. This was to workaround 
> broken backward compatibility on the cubieboard. Am I correct?

I looked at mg-debian-installer-update.  It seems that where we use a
Debian -backports kernel we are using the dt from the backports kernel
package.

> I would write: "The firmware should provide a Device-Tree working for 
> all version of Linux or provide a mechanisms to load a different 
> Device-Tree at boot."

I take it that if a vendor comes up with a novel "mechanisms to load a
different Device-Tree at boot" the work to teach osstest about it will
be simple ?

But, I was more concerned about the possible need for a board-specific
device tree file.  Is that likely ?  There doesn't seem to be any code
in osstest right now to handle such a thing.  (I say `doesn't seem'
because it wasn't me that did the osstest ARM support.)


Thanks,
Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages]

[Ian Jackson]

I'm going to reply to this email twice.  This email is about the
requirement in my document that the board is supported by Debian
-backports kernels, at least.

I'm replying to two mail@: first Julien's; then Stefano's.


Julien Grall writes ("Re: [OSSTEST PATCH] README.hardware-acquisition"):
> Most of the time, the issue to boot Debian is related to the kernel. 
> Yet, the price might be too high for some of the vendors to donate the 
> hardware. [...]
> 
> I still want to encourage vendor to work upstream but I would relax the 
> requirements here to "Xen Arm kernel branch".

When you say `I would relax the requirements' you seem to be under the
impression that this requirement is here for some kind of
sociopolitical reason.  It is not, or at least, not only.

It is an actual requirement stemming from the way that osstest does
its host installs (for both builds and tests).

To build a kernel from source code involves an OS install that can be
used for the kernel build, since otherwise there is nowhere to do the
build.

Clearly we do not want to manually create master build environment
images, or have some kind of manually-updated build host.  I certainly
don't have time to maintain such a thing.  Everything must be
automated.

Nor should the approach to setting up a build environment involve
uncontrolled downloading of a pile of varying binaries from various
bits of the internet (eg, a dockerfile).  We need a single reliable
and reputable place where we get the binaries for the base image, so
that we have some idea what is going on and to eliminate or at least
minimise uncontrolled and untracked changes to the test environment.

Currently that one place we get those binaries is a Debian release
(possibly a -backports kernel) and the files are the Debian installer,
Debian kernel, and the Debian binary packages from the Debian archive.
So the builds are done on a Debian host.

This doesn't, in principle, have to be Debian.

Take FreeBSD.  It can only be built on FreeBSD and for complicated
reasons a suitable precooked binary installer is not available.  So
for FreeBSD we make our own image using the previous FreeBSD image, in
a kind of chicken-and-egg setup.  This is quite complicated, and still
does not work entirely right because all the inter-repo dependencies
are apparently not captured properly yet in osstest's automation.

I would be open to doing something similar for ARM but someone would
have to write the code.


> While I understand this point, I think this is going to limit us a lot 
> on the choice of hardware for Arm. While Debian is quite famous on the 
> Server world, this is less the case on the automotive/embedded world 
> where they tend to have there custom distribution.

Obviously I'll take your word for that, although Debian (and its
ecosystem) is also widely used in embedded contexts.

In any case using the hardware vendor's "custom distribution" is
unmanageable for a CI system, even if we were willing to go that far
in the direction of testing using un-upstreamed (and perhaps
un-upstreamable) code.  We can't cope with one OS per kind of board.

I am open to supporting other operating systems as the baseline for
builds and tests.  FreeBSD contributors are already working on support
for that in osstest, on x86 at least.

(I didn't just use Debian because I'm familiar with it.  Debian is
simply much more mature and tractable than most of the alternatives.
It has excellent autoinstallation support, a very high level of
organisation including corresponding source code availability and
excellent release management.  It has wide architecture support and
generally wide support for a large variety of use cases.  From a
community point of view it is welcoming and helpful.  And over the
years it has been a good partner for the Xen Project.  For all I know
many of these things are true of Yocto.)


Stefano Stabellini writes ("Re: [OSSTEST PATCH] README.hardware-acquisition"):
> That's right, it's a Yocto world around here. That's also what EPAM
> uses.

I am not against using Yocto but I know nothing about it.

Is Yocto a binary distribution, or is it source code ?  Does it do
releases ?

Is the plan then to write code to make osstest install its build hosts
using Yocto ?  Or is the proposal just to use a kernel from Yocto ?
Would we want to do that for all ARM hosts ?

Who will write this code and who will fix it when it goes wrong ?


> > I still want to encourage vendor to work upstream but I would relax the
> > requirements here to "Xen Arm kernel branch".
> 
> +1

I am very open to this.

It does need a solution to the chicken-and-egg problem.  As I say, the
required underlying concepts (for using an existing properly anointed
egg to raise a new chicken so as to generate the next egg) are in
osstest already.

Ie perhaps we could install our build hosts with a Linux image we
prepared earlier.  We would pick up our own-built Linux images, and
when they pass the tests, anoint them for use in future installs.

ISTM that this would be a fairly easy way to improve this area.  If
someone wants to write the actual code to do this I would be very
happy to help with design, advice, review, etc., as I did for the
FreeBSD support (which introduced the anointed-previous-build
concept).

If you are interested then please do say and we can have a
conversation about technical details.


> Xilinx MPSoC support is upstream in vanilla Linux (defconfig), and the
> MPSoC is fully supported in Yocto. We did issue a ticket in the Debian
> system to add support for the Xilinx MPSoC in their kernel but is hasn't
> happened yet. (The fact that Debian support hasn't come up as an issue
> up until now tells us that embedded folks tend to use other distros.)

Can you please point me to the corresponding Debian bug ?

> In general, I think vanilla Linux kernel support would be a rather
> strict requirement for many embedded boards (especially the sub 100$
> ones), but it would still be better than asking for Debian kernel
> support.

Debian is generally willing to backport hardware-support patches from
(say) linux master to its released and -backports kernels.

> Overall, Julien's suggestion to require "Xen Arm kernel branch" is best.

See my discussion above.  This requirement is not just policy; without
it, osstest cannot function because it can't build anything: so it
couldn't build the Xen ARM kernel branch because it would have no host
to build it on until it had built it.


So overall, for the reasons I explain, I'm going to commit this
document (subject to the other comments etc.) *with* the requirement
that hardware must be supported by Debian (at least, in -backports).

If someone wants to do the work to relax that requirement, and to be
around to fix it when things go wrong, then I would be very happy to
help and support that and then this requirements document can be
amended to reflect whatever the new reality is.

I have tried to be clear about this need throughout our past
discussions, ever since we have been trying to acquire more ARM
hardware for the test lab.  I have also tried to be clear in my offers
to accept contributions to change the way things work.

I guess at least stating it again so explicitly and unambiguously in a
high-profile document may produce a different outcome this time.


Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition

[George Dunlap]

On Tue, Oct 30, 2018 at 4:14 PM Ian Jackson <ian.jackson@eu.citrix.com> wrote:
> +   + Baremetal boot from Debian stable or stable-backports:
> +
> +     A suitable Linux kernel binary which can boot baremetal on the
> +     proposed hardware must be available from Debian (at least
> +     `stable', or, if that is not possible `stable-backports').  It is
> +     not OK to require a patched version of Linux, or a version of
> +     Linux built from a particular git branch, or some such.  If the
> +     required kernel is not available in Debian, the vendor should
> +     first work with the Debian project to ensure and validate that
> +     the Debian stable-backports kernel binaries boot on the proposed
> +     hardware.

So it sounds like (from the rest of this discussion) the real
requirement is "osstest must be able to install a system such that it
can build the target versions of Linux and Xen".  At the moment, that
means that the proposed hardware must be supported by debian +
stable-backports.  Osstest does not have functionality to build custom
versions of Linux for build boxes, nor does it have support for Yocto.

Would it make sense to reword it this way:

---
+ Baremetal boot from Debian stable or stable-backports:

In order to avoid cross-compilation, Osstest must be able to install a
bare-metal system on the host itself in order to build Linux and Xen
test binaries for that host. At the moment osstest uses Debian for
this, and there is no facility in osstest for building custom kernels
for this purpose.  As such, a suitable Linux kernel binary which can
boot baremetal on the proposed hardware must be available from Debian
(at least `stable', or, if that is not possible, `stable-backports').
Osstest cannot install using a patched version of Linux, or one built
from a particular git branch, or some such.  If the required kernel is
not available in Debian, the vendor should ideally work with the
Debian project to ensure and validate that Debian stable-backports
kernel binaries boot on the proposed hardware.  Alternately, the
vendor can work with the community to implement the necessary
functionality within osstest to enable it to build custom kernels for
build installs, or use alternate distributions which have better
baremetal support for the hardware.
---

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition

[George Dunlap]

[CC'ing Stefano and Julien]
On Wed, Oct 31, 2018 at 5:49 PM George Dunlap <dunlapg@umich.edu> wrote:
>
> On Tue, Oct 30, 2018 at 4:14 PM Ian Jackson <ian.jackson@eu.citrix.com> wrote:
> > +   + Baremetal boot from Debian stable or stable-backports:
> > +
> > +     A suitable Linux kernel binary which can boot baremetal on the
> > +     proposed hardware must be available from Debian (at least
> > +     `stable', or, if that is not possible `stable-backports').  It is
> > +     not OK to require a patched version of Linux, or a version of
> > +     Linux built from a particular git branch, or some such.  If the
> > +     required kernel is not available in Debian, the vendor should
> > +     first work with the Debian project to ensure and validate that
> > +     the Debian stable-backports kernel binaries boot on the proposed
> > +     hardware.
>
> So it sounds like (from the rest of this discussion) the real
> requirement is "osstest must be able to install a system such that it
> can build the target versions of Linux and Xen".  At the moment, that
> means that the proposed hardware must be supported by debian +
> stable-backports.  Osstest does not have functionality to build custom
> versions of Linux for build boxes, nor does it have support for Yocto.
>
> Would it make sense to reword it this way:
>
> ---
> + Baremetal boot from Debian stable or stable-backports:
>
> In order to avoid cross-compilation, Osstest must be able to install a
> bare-metal system on the host itself in order to build Linux and Xen
> test binaries for that host. At the moment osstest uses Debian for
> this, and there is no facility in osstest for building custom kernels
> for this purpose.  As such, a suitable Linux kernel binary which can
> boot baremetal on the proposed hardware must be available from Debian
> (at least `stable', or, if that is not possible, `stable-backports').
> Osstest cannot install using a patched version of Linux, or one built
> from a particular git branch, or some such.  If the required kernel is
> not available in Debian, the vendor should ideally work with the
> Debian project to ensure and validate that Debian stable-backports
> kernel binaries boot on the proposed hardware.  Alternately, the
> vendor can work with the community to implement the necessary
> functionality within osstest to enable it to build custom kernels for
> build installs, or use alternate distributions which have better
> baremetal support for the hardware.
> ---
>
>  -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition

[Julien Grall]

Hi Ian,

On 10/31/18 2:44 PM, Ian Jackson wrote:
> I'm going to reply to this email twice.  This email is about what seem
> to be the noncontentious issues: everything *except* the requirement
> in my document that the board is supported by Debian -backports
> kernels, at least.
> 
> 
> Julien Grall writes ("Re: [OSSTEST PATCH] README.hardware-acquisition"):
>>    Furthermore, at the moment Osstest is stuck on Jessie. This
>> version does not support Thunder-X (stealing power and heating data for
>> the past year and half...) and hardwares that will be donated in the
>> next few months (i.e Renesas R-Car, and Xilinx MPSoc).
> 
> This is indeed a serious nuisance and is getting quite high on my
> priority list.

I don't have the knowledge to help on the Debian side, but I am happy to 
help testing the series on Thunder-X.

>>> +   + Boot under Xen with Linux kernel built from source code.
>>> +
>>> +     For x86, recent Linux LTS or mainline kernel source code must be
>>> +     able to boot under Xen, on the proposed hardware.
>>> +
>>> +     For ARM, there is a special Xen ARM kernel branch.  It must be
>>> +     able to boot under Xen, on the proposed hardware.
>>
>> I still want to keep that branch very close to Linux upstream. So it
>> might be worth to say we have the liberty to not accept there patch...
> 
> My wording was unclear.  I should have written something like this:
> 
>         For ARM, there is a special Xen ARM kernel branch. The proposed
>         hardware must be able to boot that version of Linux under Xen.
> 
>         If the Xen ARM Linux branch does not support the proposed
>         hardware yet, the hardware should not be accepted until that is
>         remedied.  Where this involves adding kernel patches to that
>         branch this is subject to the approval of its maintainers,
>         considering the need to keep it very close to upstream.
> 
> What do you think ?

This sounds good to me.

> 
> 
>>> + * Hardware vendor offering a "board support package" is a red flag.
>>> +   We will not be using a "board support package".  If we are offered
>>> +   one we will need explicit confirmation, and perhaps verification,
>>> +   of the points above.
>>> +
>>> + * For ARM systems using Device Tree: xxx what to write here ?
>>
>> IIRC we are using the DT from the Linux tree. This was to workaround
>> broken backward compatibility on the cubieboard. Am I correct?
> 
> I looked at mg-debian-installer-update.  It seems that where we use a
> Debian -backports kernel we are using the dt from the backports kernel
> package.
> 
>> I would write: "The firmware should provide a Device-Tree working for
>> all version of Linux or provide a mechanisms to load a different
>> Device-Tree at boot."
> 
> I take it that if a vendor comes up with a novel "mechanisms to load a
> different Device-Tree at boot" the work to teach osstest about it will
> be simple ?

U-boot and Grub provides a  way to load a different device-tree. Looking 
at the osstest logs, we already use it on U-boot.

For Grub, we rely on the DT provided by the firmware.

What I want to make sure is the DT can be loaded and be updated (i.e 
adding more nodes) via U-boot command line. Although, IHMO, that would 
be a broken firmware...

> 
> But, I was more concerned about the possible need for a board-specific
> device tree file.  Is that likely ?  There doesn't seem to be any code
> in osstest right now to handle such a thing.  (I say `doesn't seem'
> because it wasn't me that did the osstest ARM support.)

Device Tree are already board specific. Even worst, some of them are 
even depending on the version of the kernel. We should have some runes 
today to deal with that.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages]

[Stefano Stabellini]

> Julien Grall writes ("Re: [OSSTEST PATCH] README.hardware-acquisition"):
> > Most of the time, the issue to boot Debian is related to the kernel. 
> > Yet, the price might be too high for some of the vendors to donate the 
> > hardware. [...]
> > 
> > I still want to encourage vendor to work upstream but I would relax the 
> > requirements here to "Xen Arm kernel branch".
> 
> When you say `I would relax the requirements' you seem to be under the
> impression that this requirement is here for some kind of
> sociopolitical reason.  It is not, or at least, not only.
> 
> It is an actual requirement stemming from the way that osstest does
> its host installs (for both builds and tests).
> 
> To build a kernel from source code involves an OS install that can be
> used for the kernel build, since otherwise there is nowhere to do the
> build.
> 
> Clearly we do not want to manually create master build environment
> images, or have some kind of manually-updated build host.  I certainly
> don't have time to maintain such a thing.  Everything must be
> automated.
> 
> Nor should the approach to setting up a build environment involve
> uncontrolled downloading of a pile of varying binaries from various
> bits of the internet (eg, a dockerfile).  We need a single reliable
> and reputable place where we get the binaries for the base image, so
> that we have some idea what is going on and to eliminate or at least
> minimise uncontrolled and untracked changes to the test environment.
> 
> Currently that one place we get those binaries is a Debian release
> (possibly a -backports kernel) and the files are the Debian installer,
> Debian kernel, and the Debian binary packages from the Debian archive.
> So the builds are done on a Debian host.
> 
> This doesn't, in principle, have to be Debian.
> 
> Take FreeBSD.  It can only be built on FreeBSD and for complicated
> reasons a suitable precooked binary installer is not available.  So
> for FreeBSD we make our own image using the previous FreeBSD image, in
> a kind of chicken-and-egg setup.  This is quite complicated, and still
> does not work entirely right because all the inter-repo dependencies
> are apparently not captured properly yet in osstest's automation.
> 
> I would be open to doing something similar for ARM but someone would
> have to write the code.

I am suggesting to use Debian for the installer and rootfs, but to use a
different kernel for it. The kernel could come from the same Xen ARM
kernel branch that we'll use later for dom0 (Julien's suggestion).

Building the kernel could be done in a number of ways we can discuss,
including cross-compiling it on x86 hosts. It is simple to cross-compile
kernels.

The ARM kernel branch won't change very often, we could start by
building Debian packages by hand for it and pushing them to a known
location. OSSTest could fetch the Debian installer kernel from there. I
am sure we could find other, better, ways to do this.


> > While I understand this point, I think this is going to limit us a lot 
> > on the choice of hardware for Arm. While Debian is quite famous on the 
> > Server world, this is less the case on the automotive/embedded world 
> > where they tend to have there custom distribution.
> 
> Obviously I'll take your word for that, although Debian (and its
> ecosystem) is also widely used in embedded contexts.
> 
> In any case using the hardware vendor's "custom distribution" is
> unmanageable for a CI system, even if we were willing to go that far
> in the direction of testing using un-upstreamed (and perhaps
> un-upstreamable) code.  We can't cope with one OS per kind of board.
> 
> I am open to supporting other operating systems as the baseline for
> builds and tests.  FreeBSD contributors are already working on support
> for that in osstest, on x86 at least.
> 
> (I didn't just use Debian because I'm familiar with it.  Debian is
> simply much more mature and tractable than most of the alternatives.
> It has excellent autoinstallation support, a very high level of
> organisation including corresponding source code availability and
> excellent release management.  It has wide architecture support and
> generally wide support for a large variety of use cases.  From a
> community point of view it is welcoming and helpful.  And over the
> years it has been a good partner for the Xen Project.  For all I know
> many of these things are true of Yocto.)

Neither Julien nor me are suggesting to use a Vendor's distro, not even
a Vendor's kernel. We are suggesting to use our own kernel branch
together with Debian.

 
> Stefano Stabellini writes ("Re: [OSSTEST PATCH] README.hardware-acquisition"):
> > That's right, it's a Yocto world around here. That's also what EPAM
> > uses.
> 
> I am not against using Yocto but I know nothing about it.
> 
> Is Yocto a binary distribution, or is it source code ?  Does it do
> releases ?
> 
> Is the plan then to write code to make osstest install its build hosts
> using Yocto ?  Or is the proposal just to use a kernel from Yocto ?
> Would we want to do that for all ARM hosts ?
> 
> Who will write this code and who will fix it when it goes wrong ?

Actually, I think it would be great to have Yocto support in OSSTest, it
would work well on ARM and x86 but, also because it is a source
distribution, I am not suggesting it at the moment.

It would be possible to only use the kernel from Yocto, and that would
solve our problem, but at that point it is easier to just use our own
Linux branch. It is more productive to discuss that option.


> > > I still want to encourage vendor to work upstream but I would relax the
> > > requirements here to "Xen Arm kernel branch".
> > 
> > +1
> 
> I am very open to this.
> 
> It does need a solution to the chicken-and-egg problem.  As I say, the
> required underlying concepts (for using an existing properly anointed
> egg to raise a new chicken so as to generate the next egg) are in
> osstest already.
> 
> Ie perhaps we could install our build hosts with a Linux image we
> prepared earlier.  We would pick up our own-built Linux images, and
> when they pass the tests, anoint them for use in future installs.
> 
> ISTM that this would be a fairly easy way to improve this area.  If
> someone wants to write the actual code to do this I would be very
> happy to help with design, advice, review, etc., as I did for the
> FreeBSD support (which introduced the anointed-previous-build
> concept).
> 
> If you are interested then please do say and we can have a
> conversation about technical details.

Yes, we should discuss the technical details on how to use our own
quasi-vanilla Linux branch together with the Debian installer. That's
all we need AFAICT.

Could you build and push kernel binary (no required modules) to a known
location?


> > Xilinx MPSoC support is upstream in vanilla Linux (defconfig), and the
> > MPSoC is fully supported in Yocto. We did issue a ticket in the Debian
> > system to add support for the Xilinx MPSoC in their kernel but is hasn't
> > happened yet. (The fact that Debian support hasn't come up as an issue
> > up until now tells us that embedded folks tend to use other distros.)
> 
> Can you please point me to the corresponding Debian bug ?

https://salsa.debian.org/kernel-team/linux/merge_requests/67

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition

[Stefano Stabellini]

On Wed, 31 Oct 2018, George Dunlap wrote:
> [CC'ing Stefano and Julien]
> On Wed, Oct 31, 2018 at 5:49 PM George Dunlap <dunlapg@umich.edu> wrote:
> >
> > On Tue, Oct 30, 2018 at 4:14 PM Ian Jackson <ian.jackson@eu.citrix.com> wrote:
> > > +   + Baremetal boot from Debian stable or stable-backports:
> > > +
> > > +     A suitable Linux kernel binary which can boot baremetal on the
> > > +     proposed hardware must be available from Debian (at least
> > > +     `stable', or, if that is not possible `stable-backports').  It is
> > > +     not OK to require a patched version of Linux, or a version of
> > > +     Linux built from a particular git branch, or some such.  If the
> > > +     required kernel is not available in Debian, the vendor should
> > > +     first work with the Debian project to ensure and validate that
> > > +     the Debian stable-backports kernel binaries boot on the proposed
> > > +     hardware.
> >
> > So it sounds like (from the rest of this discussion) the real
> > requirement is "osstest must be able to install a system such that it
> > can build the target versions of Linux and Xen".  At the moment, that
> > means that the proposed hardware must be supported by debian +
> > stable-backports.  Osstest does not have functionality to build custom
> > versions of Linux for build boxes, nor does it have support for Yocto.
> >
> > Would it make sense to reword it this way:
> >
> > ---
> > + Baremetal boot from Debian stable or stable-backports:
> >
> > In order to avoid cross-compilation, Osstest must be able to install a
> > bare-metal system on the host itself in order to build Linux and Xen
> > test binaries for that host. At the moment osstest uses Debian for
> > this, and there is no facility in osstest for building custom kernels
> > for this purpose.  As such, a suitable Linux kernel binary which can
> > boot baremetal on the proposed hardware must be available from Debian
> > (at least `stable', or, if that is not possible, `stable-backports').
> > Osstest cannot install using a patched version of Linux, or one built
> > from a particular git branch, or some such.  If the required kernel is
> > not available in Debian, the vendor should ideally work with the
> > Debian project to ensure and validate that Debian stable-backports
> > kernel binaries boot on the proposed hardware.  Alternately, the
> > vendor can work with the community to implement the necessary
> > functionality within osstest to enable it to build custom kernels for
> > build installs, or use alternate distributions which have better
> > baremetal support for the hardware.

If we want to grow Xen on ARM testing in OSSTest for embedded boards, I
think that requiring Debian kernel support is unrealistic, as both me
with my Xilinx hat on, and Artem with his EPAM/Renesas hat on, wrote in
the past. Xilinx and Renesas are two of the best ARM vendors, imagine
the others. The Debian kernel requirement disqualifies all boards we
care about on embedded I know about at the moment. (Unrelated: this is
why for ViryaOS we went with the Yocto kernel.)

Vendors would certainly push for the usage of their own Linux trees.

The best compromise is to use our own Xen Project Linux tree for
testing. We could build, by hand if necessary, kernel binaries out of
it, push them to a known location and have OSSTest use them.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition

[George Dunlap]

On Wed, Oct 31, 2018 at 6:46 PM Stefano Stabellini
<sstabellini@kernel.org> wrote:
> > > ---
> > > + Baremetal boot from Debian stable or stable-backports:
> > >
> > > In order to avoid cross-compilation, Osstest must be able to install a
> > > bare-metal system on the host itself in order to build Linux and Xen
> > > test binaries for that host. At the moment osstest uses Debian for
> > > this, and there is no facility in osstest for building custom kernels
> > > for this purpose.  As such, a suitable Linux kernel binary which can
> > > boot baremetal on the proposed hardware must be available from Debian
> > > (at least `stable', or, if that is not possible, `stable-backports').
> > > Osstest cannot install using a patched version of Linux, or one built
> > > from a particular git branch, or some such.  If the required kernel is
> > > not available in Debian, the vendor should ideally work with the
> > > Debian project to ensure and validate that Debian stable-backports
> > > kernel binaries boot on the proposed hardware.  Alternately, the
> > > vendor can work with the community to implement the necessary
> > > functionality within osstest to enable it to build custom kernels for
> > > build installs, or use alternate distributions which have better
> > > baremetal support for the hardware.
>
> If we want to grow Xen on ARM testing in OSSTest for embedded boards, I
> think that requiring Debian kernel support is unrealistic,

You keep using the word "requirement" as though it's an active choice
that is being made.  This is a checklist for what kind of hardware can
currently be integrated into the XenProject test lab; it is not a
policy document or a design document.  As such, it should reflect the
situation as it exists at the moment, not how we would like it to be,
or how we think it may be at some point in the future.  At the moment,
only kind of hardware which can actually be integrated is one on which
Debian will boot; so this is listed as a criterion.  There's no point
buying hardware which only boots on the XenProject Linux tree until
osstest can actually boot such hardware.

It also includes pointers for how to change the situation.  If and
when the situation changes, we can change the document.

> The best compromise is to use our own Xen Project Linux tree for
> testing. We could build, by hand if necessary, kernel binaries out of
> it, push them to a known location and have OSSTest use them.

Ian objects to having binary blobs built by hand for an automated
testing system, and I tend to agree with him.  What would be required
for using the XenProject Linux tree, then, is to have a system set up
such that osstest builds its own build kernels on the target hardware
from the XenProject Linux tree.  Of course, in order to build a buiild
kernel, it needs a build kernel; so osstest needs the following
additional functionality:
 * a way to bootstrap the first build kernel, bk_1, for a particular
hardware line
 * a system within osstest to generate bk_N+1 from bk_N
Once we have those, we can change the criteria.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition

[Lars Kurth]

On 01/11/2018, 11:30, "George Dunlap" <dunlapg@umich.edu> wrote:

    On Wed, Oct 31, 2018 at 6:46 PM Stefano Stabellini
    <sstabellini@kernel.org> wrote:
    > > > ---
    > > > + Baremetal boot from Debian stable or stable-backports:
    > > >
    > > > In order to avoid cross-compilation, Osstest must be able to install a
    > > > bare-metal system on the host itself in order to build Linux and Xen
    > > > test binaries for that host. At the moment osstest uses Debian for
    > > > this, and there is no facility in osstest for building custom kernels
    > > > for this purpose.  As such, a suitable Linux kernel binary which can
    > > > boot baremetal on the proposed hardware must be available from Debian
    > > > (at least `stable', or, if that is not possible, `stable-backports').
    > > > Osstest cannot install using a patched version of Linux, or one built
    > > > from a particular git branch, or some such.  If the required kernel is
    > > > not available in Debian, the vendor should ideally work with the
    > > > Debian project to ensure and validate that Debian stable-backports
    > > > kernel binaries boot on the proposed hardware.  Alternately, the
    > > > vendor can work with the community to implement the necessary
    > > > functionality within osstest to enable it to build custom kernels for
    > > > build installs, or use alternate distributions which have better
    > > > baremetal support for the hardware.
    >
    > If we want to grow Xen on ARM testing in OSSTest for embedded boards, I
    > think that requiring Debian kernel support is unrealistic,
    
    You keep using the word "requirement" as though it's an active choice
    that is being made.  This is a checklist for what kind of hardware can
    currently be integrated into the XenProject test lab; it is not a
    policy document or a design document.  As such, it should reflect the
    situation as it exists at the moment, not how we would like it to be,
    or how we think it may be at some point in the future.  At the moment,
    only kind of hardware which can actually be integrated is one on which
    Debian will boot; so this is listed as a criterion.  There's no point
    buying hardware which only boots on the XenProject Linux tree until
    osstest can actually boot such hardware.
    
    It also includes pointers for how to change the situation.  If and
    when the situation changes, we can change the document.
    
I agree with George and Ian. The document describes what is possible now. Part of the rationale for this document, is to enable Ian to off-load more responsibility related to adding machines to OSSTEST to Credativ (from procurement to putting machines into service). This will free up some of Ian's time to focus on development work. The project pays for the service by Credativ by the hour: to manage costs we need to ensure that we don't blow the allocated hours budget (which already happened once this year). Anything which does not work with OSSTEST out-of-the box or cannot work because there is no OSSTEST support will very quickly burn through the project's support budget. Thus, we need to define boundaries to avoid running up costs for tickets in the order of several thousands of USD.

If we want to increase Arm testing, the necessary functionality has to be implemented in OSSTEST. I don’t think you can expect Ian to implement such functionality in OSSTEST, given all the other work which is already on his plate: although I believe Ian stated several times that he would help someone else to do this.

Best Regards
Lars




   

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages]

[Ian Jackson]

Stefano Stabellini writes ("Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages]"):
> I am suggesting to use Debian for the installer and rootfs, but to use a
> different kernel for it.

osstest already knows how to do that in principle, because it knows
how to insert a Debian backports kernel into the Debian d-i image.

> Actually, I think it would be great to have Yocto support in OSSTest, it
> would work well on ARM and x86 but, also because it is a source
> distribution, I am not suggesting it at the moment.

`Source distribution' means that they don't distribute binaries, so
the thing would have to be built from source.  On an x86 host I
guess.  I see no fundamental difficulties with osstest supporting
that, but it would be some work to implement.

> It would be possible to only use the kernel from Yocto, and that would
> solve our problem, but at that point it is easier to just use our own
> Linux branch. It is more productive to discuss that option.

Right.

> Yes, we should discuss the technical details on how to use our own
> quasi-vanilla Linux branch together with the Debian installer. That's
> all we need AFAICT.

OK.  So:


I see two possible approaches:

Firstly, chicken-and-egg: Use osstest's `anointed job' mechanism to
chain one Xen ARM kernel build from the next.  (The anointed job
feature in osstest allows a certain build to be declared generally
good for use by other jobs.  The anointment typically takes place at
the end of a push gate flight, when the build job that is being
anointed has been shown to work properly.)

Secondly, cross-compilation on x86.

I think cross-compilation on x86 is probably going to be easier
because it is conceptually simpler.  It also avoids difficulties if
the anointed build should turn out to be broken on some hosts (this
ought to be detected by the push gate system, but...).  And, frankly,
our x86 hardware is a lot faster.

So, assuming the plan is to do cross-compilation on x86.

The prerequisite is obviously an appropriate cross-compiler.  Will the
Debian cross-compilers do ?  If not then maybe this is not the best
approach because otherwise it's not clear where we'll get a suitable
compiler.


If the Debian cross compilers are OK, then I think the necessary
changes to osstest are:

1. Introduce a distinction between the host (GCC terminology: build)
   and target (GCC terminology: host) architectures, in ts-xen-build.
   This includes adding a call to target_install_packages to install
   the cross compiler, and appropriately amending the configure and
   make runes.  Perhaps some of this will want to be in
   Osstest/BuildSupport.pm.  The runvars for build jobs will need to
   be reviewed to decide whether a new runvar is needed or whether
   cross-compilation can be inferred from a currently-unsupported
   combination of runvars (particularly, arch vs., hostflags).

2. Maybe change ts-kernel-build to be able to additionally produce a
   .deb, or cpio full of modules, for use by step 5.  (This should be
   optional, controlled by a runvar, since it probably doubles the
   size of the build output...)

3. Change make*flight and mfi-* to, on ARM, run the existing kernel
   build job on x86 by setting the job runvars appropriately.

4a. Teach the debian-installer driver in Debian.pm how to pick up a
   kernel image from another job.  It would look at a runvar
   dikernelbuildjob or something I guess.

4b. Teach it to pick up a kernel modules from another job and stuff
   them into its installer cpio before use.

4c. Teach it to put the kernel and modules onto the being-installed
   system.

   This would be a variant of, or amendment to, or alternative to,
   Osstest/Debian.pm:di_special_kernel or its call site.  The kernel's
   ability to handle concatenated cpio images may be useful.

   We will want to refactor into a utility library (probably a file
   of shell functions) at least some of the code in
   mg-debian-installer-update for unpicking a kernel .deb (usually
   from -backports) and fishing out the kernel image and the modules,
   and stuffing the modules into an existing installer cpio archive.

   Whatever approach is taking, the modules in the installer must be a
   subset because the whole set of modules is very large and may make
   the initramfs too big to be booted.  See the list of module paths
   in mg-debian-installer-update.

   NB overall there are four aspects to (4): (i) arranging to boot the
   right kernel; (ii) getting the modules into the installer
   environment; and getting both (iii) kernel and (iv) modules into
   the being-installed system.

5. Change make*flight and mfi-* on ARM to add the new runvar so that
   ARM flights use our own kernels rather than Debian's.

6. Review the arrangements for reuse of existing build jobs, to maybe
   reuse ARM kernel builds more often.  Search cr-daily-branch for
   mg-adjust-flight-makexrefs.  Probably, an additional call should be
   added with some appropriate conditions.



> Could you build and push kernel binary (no required modules) to a known
> location?

This is a rather vague statement but basically, yes.  The question is
(i) where would it come from, and (ii) what would the known location
be.

In my proposal above, (i) it would come from a cross compilation
kernel job in the same flight (but maybe reusing the results of an
identical job elsewhere) (ii) the known location is the osstest build
outputs stash for that build job.


> > > Xilinx MPSoC support is upstream in vanilla Linux (defconfig), and the
> > > MPSoC is fully supported in Yocto. We did issue a ticket in the Debian
> > > system to add support for the Xilinx MPSoC in their kernel but is hasn't
> > > happened yet. (The fact that Debian support hasn't come up as an issue
> > > up until now tells us that embedded folks tend to use other distros.)

Maybe they compile their own kernels (or use a Debian derivative,
which is one way of sharing the effort of doing that...)

> > Can you please point me to the corresponding Debian bug ?
> 
> https://salsa.debian.org/kernel-team/linux/merge_requests/67

Hrm, you may want to file a bug in the Debian BTS.  I'm not sure
whether the stable kernel maintainers are the same people.  OTOH it
has only been there 2 weeks and it wouldn't be released until the next
Debian stable update anyway, after which it would probably go to
backports.

Right now the situation with MRs in Salsa is not always ideal;
sometimes it doesn't email the right people.  It would be worth going
to #debian-kernel (on oftc) and checking that you have made your
request via the right channel.


Thanks,
Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition

[Stefano Stabellini]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 4359 bytes --]

On Thu, 1 Nov 2018, Lars Kurth wrote:
> On 01/11/2018, 11:30, "George Dunlap" <dunlapg@umich.edu> wrote:
> 
>     On Wed, Oct 31, 2018 at 6:46 PM Stefano Stabellini
>     <sstabellini@kernel.org> wrote:
>     > > > ---
>     > > > + Baremetal boot from Debian stable or stable-backports:
>     > > >
>     > > > In order to avoid cross-compilation, Osstest must be able to install a
>     > > > bare-metal system on the host itself in order to build Linux and Xen
>     > > > test binaries for that host. At the moment osstest uses Debian for
>     > > > this, and there is no facility in osstest for building custom kernels
>     > > > for this purpose.  As such, a suitable Linux kernel binary which can
>     > > > boot baremetal on the proposed hardware must be available from Debian
>     > > > (at least `stable', or, if that is not possible, `stable-backports').
>     > > > Osstest cannot install using a patched version of Linux, or one built
>     > > > from a particular git branch, or some such.  If the required kernel is
>     > > > not available in Debian, the vendor should ideally work with the
>     > > > Debian project to ensure and validate that Debian stable-backports
>     > > > kernel binaries boot on the proposed hardware.  Alternately, the
>     > > > vendor can work with the community to implement the necessary
>     > > > functionality within osstest to enable it to build custom kernels for
>     > > > build installs, or use alternate distributions which have better
>     > > > baremetal support for the hardware.
>     >
>     > If we want to grow Xen on ARM testing in OSSTest for embedded boards, I
>     > think that requiring Debian kernel support is unrealistic,
>     
>     You keep using the word "requirement" as though it's an active choice
>     that is being made.  This is a checklist for what kind of hardware can
>     currently be integrated into the XenProject test lab; it is not a
>     policy document or a design document.  As such, it should reflect the
>     situation as it exists at the moment, not how we would like it to be,
>     or how we think it may be at some point in the future.  At the moment,
>     only kind of hardware which can actually be integrated is one on which
>     Debian will boot; so this is listed as a criterion.  There's no point
>     buying hardware which only boots on the XenProject Linux tree until
>     osstest can actually boot such hardware.
>     
>     It also includes pointers for how to change the situation.  If and
>     when the situation changes, we can change the document.

It looks like this thread wasn't the right one to raise my concerns
about embedded testing in OSSTests. If this doc is just describing what
the situation is today, then, of course, Ian should go ahead and commit
it as is. He knows best what the situation is like.

FYI I have been talking with Ian about sending Xilinx MPSoC boards to
the COLO for OSSTests usage for a while now. Ian wrote that he would
send "a checklist for OSSTests hardware". When I saw this email I
thought that this was it, and I interpreted it as requirements.


> I agree with George and Ian. The document describes what is possible now. Part of the rationale for this document, is to enable Ian to off-load more responsibility related to adding machines to OSSTEST to Credativ (from procurement to putting machines into service). This will free up some of Ian's time to focus on development work. The project pays for the service by Credativ by the hour: to manage costs we need to ensure that we don't blow the allocated hours budget (which already happened once this year). Anything which does not work with OSSTEST out-of-the box or cannot work because there is no OSSTEST support will very quickly burn through the project's support budget. Thus, we need to define boundaries to avoid running up costs for tickets in the order of several thousands of USD.
> 
> If we want to increase Arm testing, the necessary functionality has to be implemented in OSSTEST. I don’t think you can expect Ian to implement such functionality in OSSTEST, given all the other work which is already on his plate: although I believe Ian stated several times that he would help someone else to do this.
 
Unfortunately, looking at my schedule, I am unable to provide help on
this front in the foreseeable feature :-(

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages]

[Stefano Stabellini]

Hi Ian,

Thank you for the detailed answer and the willingness to see OSSTest
changed in this respect.

Let me premise that as much as I would like this to be done, I had a
look at my schedule, and, realistically, I can only volunteer very
little time on this. In regards to the two Xilinx boards, it looks like
we'll just have to wait for Debian.

For the sake of this discussion and brainstorming solutions, I have a
couple of questions and answers on how to support different kernels with
Debian below.


On Thu, 1 Nov 2018, Ian Jackson wrote:
> > Yes, we should discuss the technical details on how to use our own
> > quasi-vanilla Linux branch together with the Debian installer. That's
> > all we need AFAICT.
> 
> OK.  So:
> 
> 
> I see two possible approaches:
> 
> Firstly, chicken-and-egg: Use osstest's `anointed job' mechanism to
> chain one Xen ARM kernel build from the next.  (The anointed job
> feature in osstest allows a certain build to be declared generally
> good for use by other jobs.  The anointment typically takes place at
> the end of a push gate flight, when the build job that is being
> anointed has been shown to work properly.)
> 
> Secondly, cross-compilation on x86.
> 
> I think cross-compilation on x86 is probably going to be easier
> because it is conceptually simpler.  It also avoids difficulties if
> the anointed build should turn out to be broken on some hosts (this
> ought to be detected by the push gate system, but...).  And, frankly,
> our x86 hardware is a lot faster.
> 
> So, assuming the plan is to do cross-compilation on x86.
> 
> The prerequisite is obviously an appropriate cross-compiler.  Will the
> Debian cross-compilers do ?

Probably it would work, but I don't know for sure. Most people use the
Linaro compiler and toolchain:

https://releases.linaro.org/components/toolchain/binaries/latest-7/aarch64-linux-gnu/
https://releases.linaro.org/components/toolchain/gcc-linaro/latest-7/

Testing the Debian cross-compiler would be very easy.


> If not then maybe this is not the best
> approach because otherwise it's not clear where we'll get a suitable
> compiler.
> 
> If the Debian cross compilers are OK, then I think the necessary
> changes to osstest are:
> 
> 1. Introduce a distinction between the host (GCC terminology: build)
>    and target (GCC terminology: host) architectures, in ts-xen-build.
>    This includes adding a call to target_install_packages to install
>    the cross compiler, and appropriately amending the configure and
>    make runes.  Perhaps some of this will want to be in
>    Osstest/BuildSupport.pm.  The runvars for build jobs will need to
>    be reviewed to decide whether a new runvar is needed or whether
>    cross-compilation can be inferred from a currently-unsupported
>    combination of runvars (particularly, arch vs., hostflags).
> 
> 2. Maybe change ts-kernel-build to be able to additionally produce a
>    .deb, or cpio full of modules, for use by step 5.  (This should be
>    optional, controlled by a runvar, since it probably doubles the
>    size of the build output...)
> 
> 3. Change make*flight and mfi-* to, on ARM, run the existing kernel
>    build job on x86 by setting the job runvars appropriately.
> 
> 4a. Teach the debian-installer driver in Debian.pm how to pick up a
>    kernel image from another job.  It would look at a runvar
>    dikernelbuildjob or something I guess.
> 
> 4b. Teach it to pick up a kernel modules from another job and stuff
>    them into its installer cpio before use.
> 
> 4c. Teach it to put the kernel and modules onto the being-installed
>    system.
> 
>    This would be a variant of, or amendment to, or alternative to,
>    Osstest/Debian.pm:di_special_kernel or its call site.  The kernel's
>    ability to handle concatenated cpio images may be useful.
> 
>    We will want to refactor into a utility library (probably a file
>    of shell functions) at least some of the code in
>    mg-debian-installer-update for unpicking a kernel .deb (usually
>    from -backports) and fishing out the kernel image and the modules,
>    and stuffing the modules into an existing installer cpio archive.
> 
>    Whatever approach is taking, the modules in the installer must be a
>    subset because the whole set of modules is very large and may make
>    the initramfs too big to be booted.  See the list of module paths
>    in mg-debian-installer-update.
> 
>    NB overall there are four aspects to (4): (i) arranging to boot the
>    right kernel; (ii) getting the modules into the installer
>    environment; and getting both (iii) kernel and (iv) modules into
>    the being-installed system.
> 
> 5. Change make*flight and mfi-* on ARM to add the new runvar so that
>    ARM flights use our own kernels rather than Debian's.
> 
> 6. Review the arrangements for reuse of existing build jobs, to maybe
>    reuse ARM kernel builds more often.  Search cr-daily-branch for
>    mg-adjust-flight-makexrefs.  Probably, an additional call should be
>    added with some appropriate conditions.

I thought that we could have provided a deb repository with alternative
kernels for OSSTests to use. We would have scripts to generate those deb
packages from the Xen ARM Linux tree in a repository on xenbits, but we
wouldn't necessarily have OSSTest run the script. Initially, we could
run the scripts by hand, then, we could run them automatically in
OSSTest or elsewhere. Is that a possibility? I already have Dockerfiles
(AKA bash scripts) to build an ARM kernel on a few distros, that's
something I could make available.

This morning Julien had one more different suggestion: building the
kernel with OSSTest on SoftIron, that we know it works, it would be a
native compilation. Then we could use the built kernel together with the
Debian installer on the other boards (Xilinx, Renesas, etc.)

Either way, the kernel to be used with the embedded boards doesn't need
to be rebuilt often, only once a month or so.

 
> > > Can you please point me to the corresponding Debian bug ?
> > 
> > https://salsa.debian.org/kernel-team/linux/merge_requests/67
> 
> Hrm, you may want to file a bug in the Debian BTS.  I'm not sure
> whether the stable kernel maintainers are the same people.  OTOH it
> has only been there 2 weeks and it wouldn't be released until the next
> Debian stable update anyway, after which it would probably go to
> backports.
> 
> Right now the situation with MRs in Salsa is not always ideal;
> sometimes it doesn't email the right people.  It would be worth going
> to #debian-kernel (on oftc) and checking that you have made your
> request via the right channel.

I'll forward your suggestion to the Xilinx person that created that
ticket.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages]

[Lars Kurth]

Hi all, 

adding Wei because of  ...

User facing part: https://gitlab.com/xen-project/xen/pipelines
Back-end: https://gitlab.com/xen-project/xen-gitlab-ci
There are also some scripts in http://xenbits.xen.org/gitweb/?p=xen.git;a=tree;f=automation;hb=HEAD related to this

On 01/11/2018, 18:12, "Stefano Stabellini" <sstabellini@kernel.org> wrote:

    Hi Ian,
    
    Thank you for the detailed answer and the willingness to see OSSTest
    changed in this respect.
    
    Let me premise that as much as I would like this to be done, I had a
    look at my schedule, and, realistically, I can only volunteer very
    little time on this. In regards to the two Xilinx boards, it looks like
    we'll just have to wait for Debian.
    
    For the sake of this discussion and brainstorming solutions, I have a
    couple of questions and answers on how to support different kernels with
    Debian below.
    
    
    On Thu, 1 Nov 2018, Ian Jackson wrote:
    > > Yes, we should discuss the technical details on how to use our own
    > > quasi-vanilla Linux branch together with the Debian installer. That's
    > > all we need AFAICT.
    > 
    > OK.  So:
    > 
    > 
    > I see two possible approaches:
    > 
    > Firstly, chicken-and-egg: Use osstest's `anointed job' mechanism to
    > chain one Xen ARM kernel build from the next.  (The anointed job
    > feature in osstest allows a certain build to be declared generally
    > good for use by other jobs.  The anointment typically takes place at
    > the end of a push gate flight, when the build job that is being
    > anointed has been shown to work properly.)
    > 
    > Secondly, cross-compilation on x86.
    > 
    > I think cross-compilation on x86 is probably going to be easier
    > because it is conceptually simpler.  It also avoids difficulties if
    > the anointed build should turn out to be broken on some hosts (this
    > ought to be detected by the push gate system, but...).  And, frankly,
    > our x86 hardware is a lot faster.
    > 
    > So, assuming the plan is to do cross-compilation on x86.
    > 
    > The prerequisite is obviously an appropriate cross-compiler.  Will the
    > Debian cross-compilers do ?
    
    Probably it would work, but I don't know for sure. Most people use the
    Linaro compiler and toolchain:
    
    https://releases.linaro.org/components/toolchain/binaries/latest-7/aarch64-linux-gnu/
    https://releases.linaro.org/components/toolchain/gcc-linaro/latest-7/
    
    Testing the Debian cross-compiler would be very easy.
    
I was wondering whether we could use images in https://gitlab.com/xen-project/xen/container_registry as baseline for OSSTESTIN in these instances
We may be close to solving the build issues (via a WorksOnArm) via the GitLab CI
And it should be possible to create some infrastructure to build some custom images and put them into https://gitlab.com/xen-project/xen/container_registry and pull them from there. 

I don’t know whether that solves the full problem and how easy it would be: e.g. would we still need the cross-compiler for Xen
But we could separate the Dom0 kernel / distro build from OSSTEST 
    
    > If not then maybe this is not the best
    > approach because otherwise it's not clear where we'll get a suitable
    > compiler.
    > 
    > If the Debian cross compilers are OK, then I think the necessary
    > changes to osstest are:
    > 
    > 1. Introduce a distinction between the host (GCC terminology: build)
    >    and target (GCC terminology: host) architectures, in ts-xen-build.
    >    This includes adding a call to target_install_packages to install
    >    the cross compiler, and appropriately amending the configure and
    >    make runes.  Perhaps some of this will want to be in
    >    Osstest/BuildSupport.pm.  The runvars for build jobs will need to
    >    be reviewed to decide whether a new runvar is needed or whether
    >    cross-compilation can be inferred from a currently-unsupported
    >    combination of runvars (particularly, arch vs., hostflags).
    > 
    > 2. Maybe change ts-kernel-build to be able to additionally produce a
    >    .deb, or cpio full of modules, for use by step 5.  (This should be
    >    optional, controlled by a runvar, since it probably doubles the
    >    size of the build output...)
    > 
    > 3. Change make*flight and mfi-* to, on ARM, run the existing kernel
    >    build job on x86 by setting the job runvars appropriately.
    > 
    > 4a. Teach the debian-installer driver in Debian.pm how to pick up a
    >    kernel image from another job.  It would look at a runvar
    >    dikernelbuildjob or something I guess.
    > 
    > 4b. Teach it to pick up a kernel modules from another job and stuff
    >    them into its installer cpio before use.
    > 
    > 4c. Teach it to put the kernel and modules onto the being-installed
    >    system.
    > 
    >    This would be a variant of, or amendment to, or alternative to,
    >    Osstest/Debian.pm:di_special_kernel or its call site.  The kernel's
    >    ability to handle concatenated cpio images may be useful.
    > 
    >    We will want to refactor into a utility library (probably a file
    >    of shell functions) at least some of the code in
    >    mg-debian-installer-update for unpicking a kernel .deb (usually
    >    from -backports) and fishing out the kernel image and the modules,
    >    and stuffing the modules into an existing installer cpio archive.
    > 
    >    Whatever approach is taking, the modules in the installer must be a
    >    subset because the whole set of modules is very large and may make
    >    the initramfs too big to be booted.  See the list of module paths
    >    in mg-debian-installer-update.
    > 
    >    NB overall there are four aspects to (4): (i) arranging to boot the
    >    right kernel; (ii) getting the modules into the installer
    >    environment; and getting both (iii) kernel and (iv) modules into
    >    the being-installed system.
    > 
    > 5. Change make*flight and mfi-* on ARM to add the new runvar so that
    >    ARM flights use our own kernels rather than Debian's.
    > 
    > 6. Review the arrangements for reuse of existing build jobs, to maybe
    >    reuse ARM kernel builds more often.  Search cr-daily-branch for
    >    mg-adjust-flight-makexrefs.  Probably, an additional call should be
    >    added with some appropriate conditions.
    
    I thought that we could have provided a deb repository with alternative
    kernels for OSSTests to use. We would have scripts to generate those deb
    packages from the Xen ARM Linux tree in a repository on xenbits, but we
    wouldn't necessarily have OSSTest run the script. Initially, we could
    run the scripts by hand, then, we could run them automatically in
    OSSTest or elsewhere. Is that a possibility? I already have Dockerfiles
    (AKA bash scripts) to build an ARM kernel on a few distros, that's
    something I could make available.
    
    This morning Julien had one more different suggestion: building the
    kernel with OSSTest on SoftIron, that we know it works, it would be a
    native compilation. Then we could use the built kernel together with the
    Debian installer on the other boards (Xilinx, Renesas, etc.)
    
    Either way, the kernel to be used with the embedded boards doesn't need
    to be rebuilt often, only once a month or so.
    
That would fit with the https://gitlab.com/xen-project/xen/container_registry model 
where we store Dom0 baselines as containers for builds via the Gitlab CI 

This may be a stupid idea, but I wanted to make sure that we consider all options

Regards
Lars

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition

[Ian Jackson]

Stefano Stabellini writes ("Re: [Xen-devel] [OSSTEST PATCH] README.hardware-acquisition"):
> FYI I have been talking with Ian about sending Xilinx MPSoC boards to
> the COLO for OSSTests usage for a while now. Ian wrote that he would
> send "a checklist for OSSTests hardware". When I saw this email I
> thought that this was it, and I interpreted it as requirements.

It is indeed this document that I was referring to.  Clearly it was
useful to write it, because it has identified this issue.

> Unfortunately, looking at my schedule, I am unable to provide help on
> this front in the foreseeable feature :-(

Of course we all have priorities.  If we had a team of build system
and CI engineers, project managers, etc., I would certainly expect
that team to take on this kind of enhancement work.  But we don't; we
have about half of me.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages]

[Wei Liu]

On Fri, Nov 02, 2018 at 10:16:48AM +0000, Lars Kurth wrote:
> Hi all, 
> 
> adding Wei because of  ...
> 
> User facing part: https://gitlab.com/xen-project/xen/pipelines
> Back-end: https://gitlab.com/xen-project/xen-gitlab-ci
> There are also some scripts in http://xenbits.xen.org/gitweb/?p=xen.git;a=tree;f=automation;hb=HEAD related to this
> 
> On 01/11/2018, 18:12, "Stefano Stabellini" <sstabellini@kernel.org> wrote:
> 
>     Hi Ian,
>     
>     Thank you for the detailed answer and the willingness to see OSSTest
>     changed in this respect.
>     
>     Let me premise that as much as I would like this to be done, I had a
>     look at my schedule, and, realistically, I can only volunteer very
>     little time on this. In regards to the two Xilinx boards, it looks like
>     we'll just have to wait for Debian.
>     
>     For the sake of this discussion and brainstorming solutions, I have a
>     couple of questions and answers on how to support different kernels with
>     Debian below.
>     
>     
>     On Thu, 1 Nov 2018, Ian Jackson wrote:
>     > > Yes, we should discuss the technical details on how to use our own
>     > > quasi-vanilla Linux branch together with the Debian installer. That's
>     > > all we need AFAICT.
>     > 
>     > OK.  So:
>     > 
>     > 
>     > I see two possible approaches:
>     > 
>     > Firstly, chicken-and-egg: Use osstest's `anointed job' mechanism to
>     > chain one Xen ARM kernel build from the next.  (The anointed job
>     > feature in osstest allows a certain build to be declared generally
>     > good for use by other jobs.  The anointment typically takes place at
>     > the end of a push gate flight, when the build job that is being
>     > anointed has been shown to work properly.)
>     > 
>     > Secondly, cross-compilation on x86.
>     > 
>     > I think cross-compilation on x86 is probably going to be easier
>     > because it is conceptually simpler.  It also avoids difficulties if
>     > the anointed build should turn out to be broken on some hosts (this
>     > ought to be detected by the push gate system, but...).  And, frankly,
>     > our x86 hardware is a lot faster.
>     > 
>     > So, assuming the plan is to do cross-compilation on x86.
>     > 
>     > The prerequisite is obviously an appropriate cross-compiler.  Will the
>     > Debian cross-compilers do ?
>     
>     Probably it would work, but I don't know for sure. Most people use the
>     Linaro compiler and toolchain:
>     
>     https://releases.linaro.org/components/toolchain/binaries/latest-7/aarch64-linux-gnu/
>     https://releases.linaro.org/components/toolchain/gcc-linaro/latest-7/
>     
>     Testing the Debian cross-compiler would be very easy.
>     
> I was wondering whether we could use images in https://gitlab.com/xen-project/xen/container_registry as baseline for OSSTESTIN in these instances
> We may be close to solving the build issues (via a WorksOnArm) via the GitLab CI

I would be wary to depend on WorksOnArm at this stage because there
isn't enough information to make an informed decision.


> And it should be possible to create some infrastructure to build some custom images and put them into https://gitlab.com/xen-project/xen/container_registry and pull them from there. 
> 
> I don’t know whether that solves the full problem and how easy it would be: e.g. would we still need the cross-compiler for Xen
> But we could separate the Dom0 kernel / distro build from OSSTEST 

Debian's cross-compiler package conflicts with its native compiler,
that's why Doug and I couldn't get Arm build in Gitlab CI.

>     
>     > If not then maybe this is not the best
>     > approach because otherwise it's not clear where we'll get a suitable
>     > compiler.
>     > 
>     > If the Debian cross compilers are OK, then I think the necessary
>     > changes to osstest are:
>     > 
>     > 1. Introduce a distinction between the host (GCC terminology: build)
>     >    and target (GCC terminology: host) architectures, in ts-xen-build.
>     >    This includes adding a call to target_install_packages to install
>     >    the cross compiler, and appropriately amending the configure and
>     >    make runes.  Perhaps some of this will want to be in
>     >    Osstest/BuildSupport.pm.  The runvars for build jobs will need to
>     >    be reviewed to decide whether a new runvar is needed or whether
>     >    cross-compilation can be inferred from a currently-unsupported
>     >    combination of runvars (particularly, arch vs., hostflags).
>     > 
>     > 2. Maybe change ts-kernel-build to be able to additionally produce a
>     >    .deb, or cpio full of modules, for use by step 5.  (This should be
>     >    optional, controlled by a runvar, since it probably doubles the
>     >    size of the build output...)
>     > 
>     > 3. Change make*flight and mfi-* to, on ARM, run the existing kernel
>     >    build job on x86 by setting the job runvars appropriately.
>     > 
>     > 4a. Teach the debian-installer driver in Debian.pm how to pick up a
>     >    kernel image from another job.  It would look at a runvar
>     >    dikernelbuildjob or something I guess.
>     > 
>     > 4b. Teach it to pick up a kernel modules from another job and stuff
>     >    them into its installer cpio before use.
>     > 
>     > 4c. Teach it to put the kernel and modules onto the being-installed
>     >    system.
>     > 
>     >    This would be a variant of, or amendment to, or alternative to,
>     >    Osstest/Debian.pm:di_special_kernel or its call site.  The kernel's
>     >    ability to handle concatenated cpio images may be useful.
>     > 
>     >    We will want to refactor into a utility library (probably a file
>     >    of shell functions) at least some of the code in
>     >    mg-debian-installer-update for unpicking a kernel .deb (usually
>     >    from -backports) and fishing out the kernel image and the modules,
>     >    and stuffing the modules into an existing installer cpio archive.
>     > 
>     >    Whatever approach is taking, the modules in the installer must be a
>     >    subset because the whole set of modules is very large and may make
>     >    the initramfs too big to be booted.  See the list of module paths
>     >    in mg-debian-installer-update.
>     > 
>     >    NB overall there are four aspects to (4): (i) arranging to boot the
>     >    right kernel; (ii) getting the modules into the installer
>     >    environment; and getting both (iii) kernel and (iv) modules into
>     >    the being-installed system.
>     > 
>     > 5. Change make*flight and mfi-* on ARM to add the new runvar so that
>     >    ARM flights use our own kernels rather than Debian's.
>     > 
>     > 6. Review the arrangements for reuse of existing build jobs, to maybe
>     >    reuse ARM kernel builds more often.  Search cr-daily-branch for
>     >    mg-adjust-flight-makexrefs.  Probably, an additional call should be
>     >    added with some appropriate conditions.
>     
>     I thought that we could have provided a deb repository with alternative
>     kernels for OSSTests to use. We would have scripts to generate those deb
>     packages from the Xen ARM Linux tree in a repository on xenbits, but we
>     wouldn't necessarily have OSSTest run the script. Initially, we could
>     run the scripts by hand, then, we could run them automatically in
>     OSSTest or elsewhere. Is that a possibility? I already have Dockerfiles
>     (AKA bash scripts) to build an ARM kernel on a few distros, that's
>     something I could make available.
>     
>     This morning Julien had one more different suggestion: building the
>     kernel with OSSTest on SoftIron, that we know it works, it would be a
>     native compilation. Then we could use the built kernel together with the
>     Debian installer on the other boards (Xilinx, Renesas, etc.)
>     
>     Either way, the kernel to be used with the embedded boards doesn't need
>     to be rebuilt often, only once a month or so.
>     
> That would fit with the https://gitlab.com/xen-project/xen/container_registry model 
> where we store Dom0 baselines as containers for builds via the Gitlab CI 
> 
> This may be a stupid idea, but I wanted to make sure that we consider all options

Gitlab CI is still "another system", albeit the maintenance may be lower
compared to other solutions. I will let Ian to decide what is the best
approach.

Wei.

> 
> Regards
> Lars
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages] [and 2 more messages]

[Ian Jackson]

Stefano Stabellini writes ("Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages]"):
> Thank you for the detailed answer and the willingness to see osstest
> changed in this respect.

Sure.  It's not fixed in stone, and a variety of approachs will fit
into it I think.

> Let me premise that as much as I would like this to be done, I had a
> look at my schedule, and, realistically, I can only volunteer very
> little time on this. In regards to the two Xilinx boards, it looks like
> we'll just have to wait for Debian.

OK.  That's perhaps less work overall at our end anyway.  Let me know
if you think any aspect of this is getting stuck somehow; I have
contacts in Debian which I could use to enquire or whatever.

> On Thu, 1 Nov 2018, Ian Jackson wrote:
> > The prerequisite is obviously an appropriate cross-compiler.  Will the
> > Debian cross-compilers do ?
> 
> Probably it would work, but I don't know for sure. Most people use the
> Linaro compiler and toolchain:
> 
> https://releases.linaro.org/components/toolchain/binaries/latest-7/aarch64-linux-gnu/
> https://releases.linaro.org/components/toolchain/gcc-linaro/latest-7/

I guess that's probably tolerable.  We'd have to think about whether
we should snapshot them, or install them directly from upstream.  The
latter is less coding effort in osstest (just add an apt source and
run the install rune I guess) but it would imply tracking these in an
uncontrolled way, so we would want to be confident that they are
maintained to a high standard, since problems with the compiler would
break everything for us.  Also I would ask: how reliable is their apt
repository hosting ?  If it goes down, likewise, everything would
break for us.

> Testing the Debian cross-compiler would be very easy.

Perhaps Julien has time to do that.

> > If the Debian cross compilers are OK, then I think the necessary
> > changes to osstest are:
...
> I thought that we could have provided a deb repository with alternative
> kernels for OSSTests to use. We would have scripts to generate those deb
> packages from the Xen ARM Linux tree in a repository on xenbits, but we
> wouldn't necessarily have OSSTest run the script. Initially, we could
> run the scripts by hand, then, we could run them automatically in
> OSSTest or elsewhere. Is that a possibility? I already have Dockerfiles
> (AKA bash scripts) to build an ARM kernel on a few distros, that's
> something I could make available.

Everything that osstest consumes should either be either:
(i) snapshotted and push gated, so that osstest always uses a version
   that it itself has previously tested
(ii) maintained to a very high standard of quality and availability.

For things for which we take approach (ii), regressions of any kind,
or unavailability of the download server, cause what is effectively a
whole-system outage for osstest: every test run ends up plagued by
whatever lossage is going on.  Implying no disrespect, but doing (ii)
for a kernel apt repository is very hard.

So I think (i) would be more appropriate.  That would mean generating,
periodically, a whole new apt repository, alongside the old one.
Presumably they would have the generation date in the filename, like
our Debian installer images do.  Updating would involve a commit to
the osstest config file, which is push-gate-controlled.

Overall I think, though, that this is probably not the best approach.

What you are saying is that you do not have the effort to automate the
building of kernel binaries, and instead you propose to do it
manually.  That seems like a false economy.

The task of automating the building of kernel binaries is points 1-3
of my plan to cross build the kernels and use the result for baremetal
builds; that's not the hardest part.

> This morning Julien had one more different suggestion: building the
> kernel with OSSTest on SoftIron, that we know it works, it would be a
> native compilation. Then we could use the built kernel together with the
> Debian installer on the other boards (Xilinx, Renesas, etc.)

Yes, that is also a possibility.  But it still involves steps 4-6 from
my plan.

> Either way, the kernel to be used with the embedded boards doesn't need
> to be rebuilt often, only once a month or so.

Of course we shouldn't waste it, but computer time is much cheaper
than human time.  osstest already has mechanisms to optimise by
reusing builds where appropriate.  The easiest thing is to tell
osstest `we want a kernel built like this' and it will DTRT.

Lars Kurth writes ("Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages]"):
>     Either way, the kernel to be used with the embedded boards doesn't need
>     to be rebuilt often, only once a month or so.
>     
> That would fit with the https://gitlab.com/xen-project/xen/container_registry model 
> where we store Dom0 baselines as containers for builds via the Gitlab CI 
> 
> This may be a stupid idea, but I wanted to make sure that we consider all options

I don't know much about this.

I have no objection to osstest consuming things that came out of the
gitlab CI.

We would have to think about whether those things would (by virtue of
their origin) be necessarily of good enough quality, so that osstest
could simply use them, or whether we would want osstest to have a QA
gate before it accepted a new build.

We would also have to think about where these images (or whatever they
are) were hosted: should that be on xenbits or in a test lab VM (for
fate-sharing reasons) ?

But overall I think getting osstest to build these binaries is not the
hardest part.  If we use osstest-built binaries then we can reuse all
the existing push-gate-linked snapshotting and hosting and so on; if
we use externally built binaries we have an integration challenge.

Ie the biggest work here of all kinds is is glue.  Adding more
entities to the problem will increase rather than reduce the amount of
glue code that needs to be written.


Wei Liu writes ("Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages]"):
> Debian's cross-compiler package conflicts with its native compiler,

Specifically it conflicts with the multilib style native compiler used
for Xen.  This is a bug in Debian but it is intractable because of the
approach of the Debiabn GCC maintainer, so we will have to live with
it.

However, this is not a blocker because osstest could use a dedicated
x86 setup for the ARM cross builds.  That's not ideal because it
shares resources less well, but it's certainly workable.

But maybe we just want to build on ThunderX.  That leaves steps 4-6 of
my plan, which almost any approach (other than fixing the kernels in
Debian) require.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages] [and 2 more messages]

[Julien Grall]

Hi,

On 02/11/2018 15:05, Ian Jackson wrote:
> Stefano Stabellini writes ("Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages]"):
>> Thank you for the detailed answer and the willingness to see osstest
>> changed in this respect.
> 
> Sure.  It's not fixed in stone, and a variety of approachs will fit
> into it I think.
> 
>> Let me premise that as much as I would like this to be done, I had a
>> look at my schedule, and, realistically, I can only volunteer very
>> little time on this. In regards to the two Xilinx boards, it looks like
>> we'll just have to wait for Debian.
> 
> OK.  That's perhaps less work overall at our end anyway.  Let me know
> if you think any aspect of this is getting stuck somehow; I have
> contacts in Debian which I could use to enquire or whatever.
> 
>> On Thu, 1 Nov 2018, Ian Jackson wrote:
>>> The prerequisite is obviously an appropriate cross-compiler.  Will the
>>> Debian cross-compilers do ?
>>
>> Probably it would work, but I don't know for sure. Most people use the
>> Linaro compiler and toolchain:
>>
>> https://releases.linaro.org/components/toolchain/binaries/latest-7/aarch64-linux-gnu/
>> https://releases.linaro.org/components/toolchain/gcc-linaro/latest-7/
> 
> I guess that's probably tolerable.  We'd have to think about whether
> we should snapshot them, or install them directly from upstream.  The
> latter is less coding effort in osstest (just add an apt source and
> run the install rune I guess) but it would imply tracking these in an
> uncontrolled way, so we would want to be confident that they are
> maintained to a high standard, since problems with the compiler would
> break everything for us.  Also I would ask: how reliable is their apt
> repository hosting ?  If it goes down, likewise, everything would
> break for us.

Linaro hosting is pretty reliable. But AFAIK, they don't have an apt repo for 
the toolchains. We would need to download a tarball and unpack it.

> 
>> Testing the Debian cross-compiler would be very easy.
> 
> Perhaps Julien has time to do that.

Arm64 cross-compiler in Debian should be suitable enough for building the 
kernel. If there are any issue with them, then a bug report should be filled.

[...]

> Wei Liu writes ("Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages]"):
>> Debian's cross-compiler package conflicts with its native compiler,
> 
> Specifically it conflicts with the multilib style native compiler used
> for Xen.  This is a bug in Debian but it is intractable because of the
> approach of the Debiabn GCC maintainer, so we will have to live with
> it.
> 
> However, this is not a blocker because osstest could use a dedicated
> x86 setup for the ARM cross builds.  That's not ideal because it
> shares resources less well, but it's certainly workable.
> 
> But maybe we just want to build on ThunderX.  That leaves steps 4-6 of
> my plan, which almost any approach (other than fixing the kernels in
> Debian) require.

My preference is to avoid cross-compiling if we can. This would avoid to use x86 
resource for building kernel that could be re-used for testing Xen itself.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages] [and 2 more messages]

[Ian Jackson]

Julien Grall writes ("Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages] [and 2 more messages]"):
> On 02/11/2018 15:05, Ian Jackson wrote:
> > I guess that's probably tolerable.  We'd have to think about whether
> > we should snapshot them, or install them directly from upstream.  The
> > latter is less coding effort in osstest (just add an apt source and
> > run the install rune I guess) but it would imply tracking these in an
> > uncontrolled way, so we would want to be confident that they are
> > maintained to a high standard, since problems with the compiler would
> > break everything for us.  Also I would ask: how reliable is their apt
> > repository hosting ?  If it goes down, likewise, everything would
> > break for us.
> 
> Linaro hosting is pretty reliable. But AFAIK, they don't have an apt repo for 
> the toolchains. We would need to download a tarball and unpack it.

That would be doable.  If the tarballs have versions we could put the
version number in the osstest config and then it would be push gated.

> > Perhaps Julien has time to do that.
> 
> Arm64 cross-compiler in Debian should be suitable enough for building the 
> kernel. If there are any issue with them, then a bug report should be filled.

OK, that's definitely going to be easier then.

> My preference is to avoid cross-compiling if we can. This would avoid to use x86 
> resource for building kernel that could be re-used for testing Xen itself.

We have more x86 capacity and are deploying some dedicated build
hosts.  But cross-compilation is more complex (particularly the
compiler conflict...)

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages] [and 2 more messages]

[Julien Grall]

Hi,

On 02/11/2018 15:44, Ian Jackson wrote:
> Julien Grall writes ("Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages] [and 2 more messages]"):
>> On 02/11/2018 15:05, Ian Jackson wrote:
>>> I guess that's probably tolerable.  We'd have to think about whether
>>> we should snapshot them, or install them directly from upstream.  The
>>> latter is less coding effort in osstest (just add an apt source and
>>> run the install rune I guess) but it would imply tracking these in an
>>> uncontrolled way, so we would want to be confident that they are
>>> maintained to a high standard, since problems with the compiler would
>>> break everything for us.  Also I would ask: how reliable is their apt
>>> repository hosting ?  If it goes down, likewise, everything would
>>> break for us.
>>
>> Linaro hosting is pretty reliable. But AFAIK, they don't have an apt repo for
>> the toolchains. We would need to download a tarball and unpack it.
> 
> That would be doable.  If the tarballs have versions we could put the
> version number in the osstest config and then it would be push gated.
> 
>>> Perhaps Julien has time to do that.
>>
>> Arm64 cross-compiler in Debian should be suitable enough for building the
>> kernel. If there are any issue with them, then a bug report should be filled.
> 
> OK, that's definitely going to be easier then.
> 
>> My preference is to avoid cross-compiling if we can. This would avoid to use x86
>> resource for building kernel that could be re-used for testing Xen itself.
> 
> We have more x86 capacity and are deploying some dedicated build
> hosts.  But cross-compilation is more complex (particularly the
> compiler conflict...)

Do you mean the multilib problem? This issue is only when cross-compiling tools 
(which I would not recommend anyway...). For things like kernel and hypervisor, 
you don't need multilib.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages]

[Stefano Stabellini]

On Fri, 2 Nov 2018, Lars Kurth wrote:
>     I thought that we could have provided a deb repository with alternative
>     kernels for OSSTests to use. We would have scripts to generate those deb
>     packages from the Xen ARM Linux tree in a repository on xenbits, but we
>     wouldn't necessarily have OSSTest run the script. Initially, we could
>     run the scripts by hand, then, we could run them automatically in
>     OSSTest or elsewhere. Is that a possibility? I already have Dockerfiles
>     (AKA bash scripts) to build an ARM kernel on a few distros, that's
>     something I could make available.
>     
>     This morning Julien had one more different suggestion: building the
>     kernel with OSSTest on SoftIron, that we know it works, it would be a
>     native compilation. Then we could use the built kernel together with the
>     Debian installer on the other boards (Xilinx, Renesas, etc.)
>     
>     Either way, the kernel to be used with the embedded boards doesn't need
>     to be rebuilt often, only once a month or so.
>     
> That would fit with the https://gitlab.com/xen-project/xen/container_registry model 
> where we store Dom0 baselines as containers for builds via the Gitlab CI 
> 
> This may be a stupid idea, but I wanted to make sure that we consider all options

This is not a stupid idea -- it is exactly the kind of thing I had in
mind too

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages] [and 2 more messages]

[Ian Jackson]

Julien Grall writes ("Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages] [and 2 more messages]"):
> On 02/11/2018 15:44, Ian Jackson wrote:
> > We have more x86 capacity and are deploying some dedicated build
> > hosts.  But cross-compilation is more complex (particularly the
> > compiler conflict...)
> 
> Do you mean the multilib problem? This issue is only when cross-compiling tools 
> (which I would not recommend anyway...). For things like kernel and hypervisor, 
> you don't need multilib.

Yes, that problem.

I think by `cross-compiling' here you mean to include running gcc -m32
on an amd64 host or vice versa, which is not quite real
cross-compiling in my view.  But it does require gcc-multilib.

But, as it turns out, you are right that this is not a problem for
osstest.  When osstest does builds for an i386 dom0, it does the
hypervisor build on an amd64 host and the tools build separately on an
i386 host.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages] [and 2 more messages]

[Stefano Stabellini]

On Fri, 2 Nov 2018, Ian Jackson wrote:
> > > If the Debian cross compilers are OK, then I think the necessary
> > > changes to osstest are:
> ...
> > I thought that we could have provided a deb repository with alternative
> > kernels for OSSTests to use. We would have scripts to generate those deb
> > packages from the Xen ARM Linux tree in a repository on xenbits, but we
> > wouldn't necessarily have OSSTest run the script. Initially, we could
> > run the scripts by hand, then, we could run them automatically in
> > OSSTest or elsewhere. Is that a possibility? I already have Dockerfiles
> > (AKA bash scripts) to build an ARM kernel on a few distros, that's
> > something I could make available.
> 
> Everything that osstest consumes should either be either:
> (i) snapshotted and push gated, so that osstest always uses a version
>    that it itself has previously tested
> (ii) maintained to a very high standard of quality and availability.
> 
> For things for which we take approach (ii), regressions of any kind,
> or unavailability of the download server, cause what is effectively a
> whole-system outage for osstest: every test run ends up plagued by
> whatever lossage is going on.  Implying no disrespect, but doing (ii)
> for a kernel apt repository is very hard.
> 
> So I think (i) would be more appropriate.  That would mean generating,
> periodically, a whole new apt repository, alongside the old one.
> Presumably they would have the generation date in the filename, like
> our Debian installer images do.  Updating would involve a commit to
> the osstest config file, which is push-gate-controlled.
> 
> Overall I think, though, that this is probably not the best approach.
> 
> What you are saying is that you do not have the effort to automate the
> building of kernel binaries, and instead you propose to do it
> manually.  That seems like a false economy.
> 
> The task of automating the building of kernel binaries is points 1-3
> of my plan to cross build the kernels and use the result for baremetal
> builds; that's not the hardest part.

[...]

> Ie the biggest work here of all kinds is is glue.  Adding more
> entities to the problem will increase rather than reduce the amount of
> glue code that needs to be written.

Basically, you are saying that even if had a well maintained deb
repository of kernel packages for OSSTest to use, doesn't matter how we
achieve this goal, it would still require a non trivial amount of work
to do the integration with OSSTest.

I would have thought that it would have made OSSTest work (almost) out
of the box. Too bad. In that case, I'll leave this thread and the
implementation choices to the people that are actually going to do work
on it.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages] [and 2 more messages]

[Julien Grall]

On 02/11/2018 17:56, Stefano Stabellini wrote:
> On Fri, 2 Nov 2018, Ian Jackson wrote:
>> Ie the biggest work here of all kinds is is glue.  Adding more
>> entities to the problem will increase rather than reduce the amount of
>> glue code that needs to be written.
> 
> Basically, you are saying that even if had a well maintained deb
> repository of kernel packages for OSSTest to use, doesn't matter how we
> achieve this goal, it would still require a non trivial amount of work
> to do the integration with OSSTest.

This is not how I understood the thread. I believe this was related to use an 
external CI loop.

In our case, once we have a deb package. Then this is not much different than 
using a backport kernel. We already have such support in Osstest, so it should 
not be too difficult to adapt it for a different repo.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages] [and 2 more messages]

[Stefano Stabellini]

On Fri, 2 Nov 2018, Julien Grall wrote:
> On 02/11/2018 17:56, Stefano Stabellini wrote:
> > On Fri, 2 Nov 2018, Ian Jackson wrote:
> > > Ie the biggest work here of all kinds is is glue.  Adding more
> > > entities to the problem will increase rather than reduce the amount of
> > > glue code that needs to be written.
> > 
> > Basically, you are saying that even if had a well maintained deb
> > repository of kernel packages for OSSTest to use, doesn't matter how we
> > achieve this goal, it would still require a non trivial amount of work
> > to do the integration with OSSTest.
> 
> This is not how I understood the thread. I believe this was related to use an
> external CI loop.
> 
> In our case, once we have a deb package. Then this is not much different than
> using a backport kernel. We already have such support in Osstest, so it should
> not be too difficult to adapt it for a different repo.

OK, I misunderstood Ian, thanks for stepping in. If that is the case, then:

1) We already have a Xen ARM Linux tree which we have to maintain for Dom0
The maintenance model and testing of this tree doesn't change regardless.

2) We already have Dockerfiles and scripts by Doug and ViryaOS to build kernels on gitlab
We could improve them to build a full deb repository.

What's stopping us from using (2) to setup up a Xen Project deb repo?

Does it really matter who executes scripts (2) and whether they are run
on your laptop or in the cloud as long as the build is fully
reproducible? We are not talking about hacking config files around and
using who knows what compiler to build something. docker build will give
you the same output no matter the host.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages] [and 2 more messages]

[Ian Jackson]

Stefano Stabellini writes ("Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages] [and 2 more messages]"):
> Basically, you are saying that even if had a well maintained deb
> repository of kernel packages for OSSTest to use, doesn't matter how we
> achieve this goal, it would still require a non trivial amount of work
> to do the integration with OSSTest.

No, sorry.  The one thing that osstest can easily consume is a
well-maintained deb repository.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages] [and 2 more messages]

[Julien Grall]

Hi Stefano,

On 02/11/2018 23:44, Stefano Stabellini wrote:
> On Fri, 2 Nov 2018, Julien Grall wrote:
>> On 02/11/2018 17:56, Stefano Stabellini wrote:
>>> On Fri, 2 Nov 2018, Ian Jackson wrote:
>>>> Ie the biggest work here of all kinds is is glue.  Adding more
>>>> entities to the problem will increase rather than reduce the amount of
>>>> glue code that needs to be written.
>>>
>>> Basically, you are saying that even if had a well maintained deb
>>> repository of kernel packages for OSSTest to use, doesn't matter how we
>>> achieve this goal, it would still require a non trivial amount of work
>>> to do the integration with OSSTest.
>>
>> This is not how I understood the thread. I believe this was related to use an
>> external CI loop.
>>
>> In our case, once we have a deb package. Then this is not much different than
>> using a backport kernel. We already have such support in Osstest, so it should
>> not be too difficult to adapt it for a different repo.
> 
> OK, I misunderstood Ian, thanks for stepping in. If that is the case, then:
> 
> 1) We already have a Xen ARM Linux tree which we have to maintain for Dom0
> The maintenance model and testing of this tree doesn't change regardless.
> 
> 2) We already have Dockerfiles and scripts by Doug and ViryaOS to build kernels on gitlab
> We could improve them to build a full deb repository.

Which does not work on Arm today... (see the previous discussions). Lars and Wei 
are working towards addressing this.

> 
> What's stopping us from using (2) to setup up a Xen Project deb repo?
> 
> Does it really matter who executes scripts (2) and whether they are run
> on your laptop or in the cloud as long as the build is fully
> reproducible? We are not talking about hacking config files around and
> using who knows what compiler to build something. docker build will give
> you the same output no matter the host.

AFAICT, Ian's main concern was adding yet another dependency on external entity.

However, OSStest already provides scripts to build kernel. So why would you want 
to use Dockerfiles/ViryaOS?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages] [and 2 more messages]

[Ian Jackson]

Julien Grall writes ("Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages] [and 2 more messages]"):
> AFAICT, Ian's main concern was adding yet another dependency on external entity.

We could put the .deb repository on xenbits.
There remains the question of who will do the manual rebuilds.

Another thing that occurred to me is that not all kernel .debs are
created equal.  Do ones that come from the kernel source tree, without
any of the Debian packaging code, interact properly with Debian's
initramfs and bootloader update machinery ?

> However, OSStest already provides scripts to build kernel. So why would you want 
> to use Dockerfiles/ViryaOS?

That seems to me to be a very salient point.  Earlier I wrote:

   Ie the biggest work here of all kinds is is glue.  Adding more
   entities to the problem will increase rather than reduce the amount of
   glue code that needs to be written.

The amount of new glue that is needed depends also on how much of the
existing systems can be reused.


I am starting to think that it may have been a mistake of me to
explain in clear and precise detail what would be needed, to have
osstest directly use its own-built kernels for baremetal install.

If you look at the length of my description it looks like a lot of
work.  But the competing proposals are being described by some
handwaving.  Obviously they look simpler then!

No-one has written a comparabily detailed plan for exactly what work
would need to be done, and what risks there are, for example in this
scheme to use docker and ViryaOS and an apt repository.  It's true
that such a plan would have fewer changes to osstest; but more of it
would be manual steps.  In the case of manual steps IMO a comparably
detailed plan would include a rough set of proposed command line
runes, etc.  I'm not even sure that this scheme has been thought
through enough for us to write such a plan with confidence that it
will work.

Almost certainly such a detailed plan, if we can write it, would be
considerably longer and more complex than the plan I posted earlier.

And of course it has a higher overall maintenance burden because
updates would be manual forever more - not only manual, but also
dependent on the dockerfile continuing to work, which means the manual
steps depend on the availability (and integrity!) of whatever external
entities the dockerfile uses.

Thanks,
Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages] [and 2 more messages]

[Stefano Stabellini]

Hi Ian,

Given that the Debian bug is filed so either way we have a path forward,
should I get started with the discussion about shipping the hardware
(get various approvals, prepare boxes for shipping, etc.)? That's going
to take a few weeks for sure. Or would you like to wait for the Debian
bug to be resolved / alternative code to be written ?

Cheers,

Stefano


On Mon, 5 Nov 2018, Ian Jackson wrote:
> Julien Grall writes ("Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages] [and 2 more messages] [and 2 more messages]"):
> > AFAICT, Ian's main concern was adding yet another dependency on external entity.
> 
> We could put the .deb repository on xenbits.
> There remains the question of who will do the manual rebuilds.
> 
> Another thing that occurred to me is that not all kernel .debs are
> created equal.  Do ones that come from the kernel source tree, without
> any of the Debian packaging code, interact properly with Debian's
> initramfs and bootloader update machinery ?
> 
> > However, OSStest already provides scripts to build kernel. So why would you want 
> > to use Dockerfiles/ViryaOS?
> 
> That seems to me to be a very salient point.  Earlier I wrote:
> 
>    Ie the biggest work here of all kinds is is glue.  Adding more
>    entities to the problem will increase rather than reduce the amount of
>    glue code that needs to be written.
> 
> The amount of new glue that is needed depends also on how much of the
> existing systems can be reused.
> 
> 
> I am starting to think that it may have been a mistake of me to
> explain in clear and precise detail what would be needed, to have
> osstest directly use its own-built kernels for baremetal install.
> 
> If you look at the length of my description it looks like a lot of
> work.  But the competing proposals are being described by some
> handwaving.  Obviously they look simpler then!
> 
> No-one has written a comparabily detailed plan for exactly what work
> would need to be done, and what risks there are, for example in this
> scheme to use docker and ViryaOS and an apt repository.  It's true
> that such a plan would have fewer changes to osstest; but more of it
> would be manual steps.  In the case of manual steps IMO a comparably
> detailed plan would include a rough set of proposed command line
> runes, etc.  I'm not even sure that this scheme has been thought
> through enough for us to write such a plan with confidence that it
> will work.
> 
> Almost certainly such a detailed plan, if we can write it, would be
> considerably longer and more complex than the plan I posted earlier.
> 
> And of course it has a higher overall maintenance burden because
> updates would be manual forever more - not only manual, but also
> dependent on the dockerfile continuing to work, which means the manual
> steps depend on the availability (and integrity!) of whatever external
> entities the dockerfile uses.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages]

[Ian Jackson]

Ian Jackson writes ("Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages]"):
> So overall, for the reasons I explain, I'm going to commit this
> document (subject to the other comments etc.) *with* the requirement
> that hardware must be supported by Debian (at least, in -backports).

This didn't happen.  THere was considerable further discussion.  The
fact that various kinds of uncertainty meant this document didn't get
committed is now blocking us giving the go-ahead for some new hardware
acquisition:

Ie, I can't answer the question "should we accept hardware XYZ"
without reference to at least an implied a checklist like this.
Having written it down I ought to use the one I've written down,
because to do otherwise is simply to pointlessly invite mistakes.  And
if I'm to use a written-down checklist it should be one which is
actually official.

Accordingly, I intend to commit this to osstest now.  Juergen, this is
just a document: can I have your release ack for it ?

I will then reply separately about the specific new hardware, using
the checklist as a guide.  Obviously a checklist is always a
guidelines document: if we find that a point is best answered a
different way than the checklist expects, or that the checklist ought
to be changed, then changes to the checklist are a reasonable part of
the outcome of such a process; that would be in the form of further
patches to this document in osstest.

Ian.

From fae48bd584a0b58934a2df97b6db1d06eacf1724 Mon Sep 17 00:00:00 2001
From: Ian Jackson <ian.jackson@eu.citrix.com>
Date: Tue, 30 Oct 2018 16:12:27 +0000
Subject: [OSSTEST PATCH] README.hardware-acquisition

New document-cum-checklist, for helping with hardware procurement.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
CC: infra@xenproject.org
CC: George Dunlap <dunlapg@umich.edu>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Julien Grall <julien.grall@arm.com
--
v2: Add caveats about the Xen ARM Linux branch
    Say something, albeit rather vague, about device trees
---
 README.hardware-acquisition | 317 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 317 insertions(+)
 create mode 100644 README.hardware-acquisition

diff --git a/README.hardware-acquisition b/README.hardware-acquisition
new file mode 100644
index 00000000..0a429db3
--- /dev/null
+++ b/README.hardware-acquisition
@@ -0,0 +1,317 @@
+====================================
+# HARDWARE ACQUISITION FOR OSSTEST #
+====================================
+
+This document can be used as a checklist when procuring hardware for
+an osstest instance.  A few of the points have details specific to the
+Xen Project test lab in Massachusetts, but most of it will be relevant
+to all osstest installations.
+
+
+POWER
+=====
+
+osstest needs to turn each host on and off under program control.
+
+When a host is power cycled, all state in it must be reset.  This
+includes onboard control and management software (eg IPMI), since such
+systems can be buggy and bugs in them can be provoked by bugs in
+system software (ie, buggy versions of Xen can break the LOM, even if
+the LOM, unusually, is not simply flaky).
+
+However, it is often necessary to use the LOM (Lights Out Management)
+as part of the poweron/poweroff sequence as otherwise some machines
+draw enough current to wear out our mains PDU contacts too quickly.
+
+(I use the English word `mains' for the single phase 110V/220V-240V AC
+electrical power supply prevalent in datacentres.)
+
+Requirements for typical server hardware
+----------------------------------------
+
+ * If the system has a LOM it should be driveable with Free Software,
+   eg via the IPMI protocol.
+
+ * Redundant PSUs are not required.
+
+ * Provisioning: One PDU port is required per host.
+
+Requirements for embedded or devboard hardware
+----------------------------------------------
+
+ * There must be arrangements to control the actual power supply
+   to each board (node).  Options include:
+
+     (i) Each node has a separate mains power supply, each of which
+         we will plug into a PDU port.
+
+     (ii) A separate management or PDU board or backplane, which
+         has one single mains power input and which has relays
+         or similar to control power to individual nodes.
+         The management system must have its own separate network
+         connection and not be at risk of corruption from
+         bad software on nodes.
+
+ * Provisioning:
+    + Number of PDU ports required depends on the approach taken.
+    + With a separate PDU controller, a switch port is required.
+
+
+SERIAL
+======
+
+We always use hardware serial for console output.  This is essential
+to capture kernel and hypervisor crash messages, including from early
+boot; as well as bootloader output, and so on.  We use our own serial
+concentrator hardware, separate from the systems under test.  Built-in
+console-over-LAN systems (eg IPMI serial over LAN) are not reliable
+enough for our purposes.
+
+Requirements for typical server hardware
+----------------------------------------
+
+ * At least one conventional RS232 UART, accessible to system
+   software in the conventional way.
+
+ * For ARM, supported as console by both Xen[1] and Linux[2].
+
+ * Presented on a standard 9-pin D connector.  (RJ45 is acceptable
+   if we know the pinout.)
+
+ * Provisioning: one serial concentrator port required per host.
+
+Requirements for a embedded or devboard hardware
+------------------------------------------------
+
+ * At least one suitable UART
+
+ * Supported in software by both Xen[1] and Linux[2]
+
+ * With suitable physical presentation:
+    (i)
+       + Proper RS232 (full voltage, not TTL or 3.3V)
+       + presented on a 9-pin D or RJ45 connector
+       + with known pinout;
+   or
+    (ii)
+       + Connected somehow to a USB-to-serial adapter
+       + Adapter supported by Linux[2]
+       + Multiple adapters, giving one physical USB port
+         for all nodes (ie built-in hub) preferred
+   or
+    (iii) Some other suitable arrangement to be discussed.
+
+ * Provisioning: Requires serial concentrator port(s) and/or spare USB
+   port(s) on appropriate infrastructure host(s).
+
+
+PHYSICAL PRESENTATION
+=====================
+
+ * All equipment should be mounted inside one or more 19" rack
+   mount cases.
+
+ * In as few U as possible: usually 1U (or, exceptionally, maybe 2U)
+   for a single server-type host.  
+
+ * Forbidden: External power adapters (laptop-style mains power supply
+   bricks); external USB hubs; any equipment not physically
+   restrained.  There is no shelf in the rack.
+
+ * Pair principle: Every host or node must be part of a set of several
+   identical hosts.  This allows us to distinguish hardware faults
+   from software bugs.  (In the cases of chassis with backplane, one
+   backplane is OK.)  Conversely, we want diversity to find the most
+   host-specific bugs, so usually around two of each type is best.
+
+ * Provisioning: Enough rack space must be available.
+
+
+MASS STORAGE
+============
+
+Each host needs some locally attached mass storage of its own.
+
+Requirements for typical server hardware
+----------------------------------------
+
+ * SATA controller supported by Linux[2]
+
+ * If SATA controller has multiple modes (eg, AHCI vs RAID)
+   it is sufficient for it to be supported in one mode.
+
+ * Storage redundancy is not required: one disk will do.
+
+ * SSD is not required: rotating rust is cheaper and will do.
+
+Requirements for embedded or devboard hardware
+----------------------------------------------
+
+ * Some mass storage supported by Linux[2].  Best is an onboard SATA
+   controller, connected to a SATA HDD in the same enclosure.
+   High-endurance flash drives are another possibility.
+
+ * If the hardware always starts by boot from a mass storage device,
+   that boot device must be physically read-only and separate from the
+   primary mass storage.  See BOOT ARRANGEMENTS.
+
+
+REMOTE FIRMWARE ACCESS VIA SERIAL
+=================================
+
+Configuration of the primary system firmware must be possible remotely
+using only the power and serial accesses just described.
+Specifically, interaction with the firmware via the serial port.
+
+Requirements for typical server hardware with UEFI or BIOS
+----------------------------------------------------------
+
+ * `BIOS' configuration (including the UEFI equivalent) accessible and
+   useable via BIOS `serial console redirection'.
+
+ * UEFI shell (if provided) also available via serial.
+
+ * Specifically, boot order configuration available via serial.
+
+
+Requirements for embedded or devboard hardware
+----------------------------------------------
+
+ * See BOOT ARRANGEMENTS.
+
+
+BOOT ARRANGEMENTS, NETBOOT
+==========================
+
+Every host must netboot as its first boot source.  The netboot
+configuration must be able to `chain' to the local writeable mass
+storage.  This ensures that a host can be completely wiped, even if
+bad software has corrupted the mass storage.
+
+Requirements for typical server hardware with UEFI or BIOS
+----------------------------------------------------------
+
+ * PXE and/or UEFI netboot.
+
+Requirements for embedded or devboard hardware
+----------------------------------------------
+
+ * Some firmware must be available and provided which is capable of
+   netbooting Xen[1] and Linux[2], under control from the netboot
+   server.  A suitable version of u-boot can meet this need.
+
+ * The firmware which performs the netbooting must be on a read-only
+   storage device (flagged as such in hardware, not software) so that
+   it cannot be corrupted by system software.  So it must be on a
+   separate physical storage device to the primary mass storage (see
+   MASS STORAGE, above).
+
+ * This firmware will not usually be updated.
+
+
+NETWORKING
+==========
+
+Requirements
+------------
+
+ * Each host must have at least one RJ45 ethernet port compatible
+   with ordinary 100Mbit ethernet.   xxx
+
+ * The primary ethernet port must be compatible with Linux[2].
+
+ * In the case of a chassis with backplane, it is acceptable if the
+   chassis contains an ethernet switch, provided that it is a normal
+   and reliable ethernet switch (not a proprietary interconnect).
+
+ * In the case of a system with IPMI or similar LOM, it is best if the
+   LOM has its own physical ethernet port.
+
+
+CPU, CHIPSET, MOTHERBOARD, ETC.
+===============================
+
+General advice and preferences
+------------------------------
+
+ * We prefer multicore, multisocket and NUMA systems because they
+   expose a greater variety of exciting bugs.  But we don't care much
+   about performance and we want a wide variety of different hosts.
+   We want a mixture of systems with different CPU variants and
+   feature support.
+
+ * Memory requirements are modest.  8G or 16G per host is fine. xxx
+
+Compatibility with Xen and Linux - requirements
+-----------------------------------------------
+
+(Normally these issues are not a problem for x86, except perhaps for
+the network and storage controllers - see MASS STORAGE and NETWORKING,
+above.)
+
+ * [1] Xen: The CPU and other hardware must be supported by current
+   versions of xen-unstable, at the very least.
+
+ * [2] Linux: The CPU and other hardware must be supported by existing
+   widely available versions of Linux.  There are two principal
+   requirements:
+
+   + Baremetal boot from Debian stable or stable-backports:
+
+     A suitable Linux kernel binary which can boot baremetal on the
+     proposed hardware must be available from Debian (at least
+     `stable', or, if that is not possible `stable-backports').  It is
+     not OK to require a patched version of Linux, or a version of
+     Linux built from a particular git branch, or some such.  If the
+     required kernel is not available in Debian, the vendor should
+     first work with the Debian project to ensure and validate that
+     the Debian stable-backports kernel binaries boot on the proposed
+     hardware.
+
+   + Boot under Xen with Linux kernel built from source code.
+
+     For x86, recent Linux LTS or mainline kernel source code must be
+     able to boot under Xen, on the proposed hardware.
+
+     For ARM, there is a special Xen ARM kernel branch. The proposed
+     hardware must be able to boot that version of Linux under Xen.
+
+     If the Xen ARM Linux branch does not support the proposed
+     hardware yet, the hardware should not be accepted until that is
+     remedied.  Where this involves adding kernel patches to that
+     branch this is subject to the approval of its maintainers,
+     considering the need to keep it very close to upstream.
+
+ * Board-specific Linux and Xen versions are not acceptable.
+
+ * Hardware vendor offering a "board support package" is a red flag.
+   We will not be using a "board support package".  If we are offered
+   one we will need explicit confirmation, and perhaps verification,
+   of the points above.
+
+ * For ARM systems using Device Tree: check what DT is expected to be
+   used, and where and how we are expecting osstest to get it from.
+
+
+RELIABILITY
+===========
+
+ * osstest stresses systems in unusual ways.  The need to completely
+   wipe the machine for each test means test hosts are power cycled
+   more often than usual.
+
+ * Random failures due to unreliable hardware are not tolerable.  Some
+   hosts do not boot reliably.  Even a very small probability of a
+   random boot failure, per boot, is intolerable in this CI
+   environment: hosts are rebooted many times a day, and a random boot
+   failure looks just like a `hypervisor could not boot' bug.  (The
+   same bug would not be noticeable in a server farm where hosts are
+   nearly never rebooted.)
+
+
+NON-REQUIREMENTS
+================
+
+ * No VGA console needed.
+ * Redundant PSUs are not needed (see POWER, above).
+ * RAID is not needed (or wanted) (see MASS STORAGE, above).
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages]

[Juergen Gross]

On 15/02/2019 12:56, Ian Jackson wrote:
> Ian Jackson writes ("Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages]"):
>> So overall, for the reasons I explain, I'm going to commit this
>> document (subject to the other comments etc.) *with* the requirement
>> that hardware must be supported by Debian (at least, in -backports).
> 
> This didn't happen.  THere was considerable further discussion.  The
> fact that various kinds of uncertainty meant this document didn't get
> committed is now blocking us giving the go-ahead for some new hardware
> acquisition:
> 
> Ie, I can't answer the question "should we accept hardware XYZ"
> without reference to at least an implied a checklist like this.
> Having written it down I ought to use the one I've written down,
> because to do otherwise is simply to pointlessly invite mistakes.  And
> if I'm to use a written-down checklist it should be one which is
> actually official.
> 
> Accordingly, I intend to commit this to osstest now.  Juergen, this is
> just a document: can I have your release ack for it ?

Yes, sure:

Release-acked-by: Juergen Gross <jgross@suse.com>


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages]

[Lars Kurth]

Ian,

as mentioned earlier, I think we need a legal/admin item for hardware and donations. See proposed addition below.

On 15/02/2019, 11:57, "Ian Jackson" <ian.jackson@citrix.com> wrote:

    Ian Jackson writes ("Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages]"):
    > So overall, for the reasons I explain, I'm going to commit this
    > document (subject to the other comments etc.) *with* the requirement
    > that hardware must be supported by Debian (at least, in -backports).
    
    This didn't happen.  THere was considerable further discussion.  The
    fact that various kinds of uncertainty meant this document didn't get
    committed is now blocking us giving the go-ahead for some new hardware
    acquisition:
    
...
    
    From fae48bd584a0b58934a2df97b6db1d06eacf1724 Mon Sep 17 00:00:00 2001
    From: Ian Jackson <ian.jackson@eu.citrix.com>
    Date: Tue, 30 Oct 2018 16:12:27 +0000
    Subject: [OSSTEST PATCH] README.hardware-acquisition
    
    New document-cum-checklist, for helping with hardware procurement.
    
    Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
    CC: infra@xenproject.org
    CC: George Dunlap <dunlapg@umich.edu>
    CC: Stefano Stabellini <sstabellini@kernel.org>
    CC: Julien Grall <julien.grall@arm.com
    --
    v2: Add caveats about the Xen ARM Linux branch
        Say something, albeit rather vague, about device trees
    ---
     README.hardware-acquisition | 317 ++++++++++++++++++++++++++++++++++++++++++++
     1 file changed, 317 insertions(+)
     create mode 100644 README.hardware-acquisition
    
    diff --git a/README.hardware-acquisition b/README.hardware-acquisition
    new file mode 100644
    index 00000000..0a429db3
    --- /dev/null
    +++ b/README.hardware-acquisition
    @@ -0,0 +1,317 @@
    +====================================
    +# HARDWARE ACQUISITION FOR OSSTEST #
    +====================================
    +
    +This document can be used as a checklist when procuring hardware for
    +an osstest instance.  A few of the points have details specific to the
    +Xen Project test lab in Massachusetts, but most of it will be relevant
    +to all osstest installations.
    +
...
    +
    +RELIABILITY
    +===========
    +
    + * osstest stresses systems in unusual ways.  The need to completely
    +   wipe the machine for each test means test hosts are power cycled
    +   more often than usual.
    +
    + * Random failures due to unreliable hardware are not tolerable.  Some
    +   hosts do not boot reliably.  Even a very small probability of a
    +   random boot failure, per boot, is intolerable in this CI
    +   environment: hosts are rebooted many times a day, and a random boot
    +   failure looks just like a `hypervisor could not boot' bug.  (The
    +   same bug would not be noticeable in a server farm where hosts are
    +   nearly never rebooted.)
    +
    +

LEGAL
=====
* There cannot be *any* restrictions, such as NDAs or other restrictions for hardware 
   that runs public tests such as osstest. The project also does not accept hardware on
   loan.

* '''Hardware Donations''': should an organisation or individual want to donate Hardware to the Xen 
   Project, it is necessary for the donor of the Hardware to provides a letter (also referred to as ''donation 
   form''), which states the following:
   + Date

   + Linux Foundation address: 
      c/o Xen Project 
      Linux Foundation
      1 Letterman Drive
      Building D, Suite D4700
      San Francisco
      CA 94129

   + Donor Information: is address details related to the donor

   + A name, title and signature of the donor, or a representative of a donor organisation who is 
      authorized to approve the donation

   + A description of the donated items, ideally with correct model and serial numbers

   Typically donating organisations have their own template for donation forms, which is why we
   have not provided our own template.

Best Regards
Lars

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages]

[Ian Jackson]

Lars Kurth writes ("Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages]"):
> as mentioned earlier, I think we need a legal/admin item for hardware and donations. See proposed addition below.
...
> LEGAL
> =====

I would prefer not to bake the Linux Foundation address into this
document, nor to have it be the primary repository for legal details,
if that's OK.

How about this:

  LEGAL - XEN PROJECT
  ===================

  The Xen Project Test Lab runs public tests, and provides access to its
  hardware to Xen community members so that they can debug the code.
  Accordingly there must not be any restrictions on publication of
  information such as test results, nor any restrictions on access to
  the hardware.  Ie, the Xen Project is not able to agree to any
  nondisclosure agreement (NDA).

  The Xen Project Test Lab does not accept hardware on loan.  We
  purchase it, or, following prior consultation and approval, can accept
  donations.  For donations, the Linux Foundation requires a letter
  confirming some legal details.  The Community Manager will provide
  a list of specific requirements for that `donation form' letter.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages]

[Lars Kurth]

On 15/02/2019, 15:40, "Ian Jackson" <ian.jackson@citrix.com> wrote:

    Lars Kurth writes ("Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages]"):
    > as mentioned earlier, I think we need a legal/admin item for hardware and donations. See proposed addition below.
    ...
    > LEGAL
    > =====
    
    I would prefer not to bake the Linux Foundation address into this
    document, nor to have it be the primary repository for legal details,
    if that's OK.
    
    How about this:
    
      LEGAL - XEN PROJECT
      ===================
    
      The Xen Project Test Lab runs public tests, and provides access to its
      hardware to Xen community members so that they can debug the code.
      Accordingly there must not be any restrictions on publication of
      information such as test results, nor any restrictions on access to
      the hardware.  Ie, the Xen Project is not able to agree to any
      nondisclosure agreement (NDA).
    
      The Xen Project Test Lab does not accept hardware on loan.  We
      purchase it, or, following prior consultation and approval, can accept
      donations.  For donations, the Linux Foundation requires a letter
      confirming some legal details.  The Community Manager will provide
      a list of specific requirements for that `donation form' letter.
    
This is fine by me
Lars    

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages]

[Ian Jackson]

Lars Kurth writes ("Re: [OSSTEST PATCH] README.hardware-acquisition [and 1 more messages]"):
> This is fine by me

Thanks, pushed now.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel