osstest going live in new test colo

osstest going live in new test colo

[Ian Jackson]

Based on our commissioning tests, the new facility now has greater
capacity and better reliability than the existing setup in the Citrix
Cambridge office.

I'm therefore going to set the new system going in production mode,
and shut down the old one.

There are a few loose ends to get the cron jobs working right which I
expect to have sorted out very soon.  In the meantime I have already
instructed the old system to drain its queue, with the expectation of
going fully live with the new system on Friday.

You will be able to tell when test reports are coming from the new
system because they will have flight numbers >=50000.


There are still some problems with the new facility.  Of the 24 x86
test hosts, only 12 are currently live.  Of the remainder:

 * 2 pairs (4 hosts, chardonnay[01] and elbling[01]) are affected by 2
   different kernel/hypervisor bugs.  I have not yet reported these
   properly, but will do so shortly.  I hope to get community help
   with resolving them.

 * 1 pair (2 hosts, oseleta[01]) are affected by what appears to be a
   BIOS bug that sometimes prevents the machine rebooting; this is
   with our supplier.

 * 1 pair (2 hosts, godello[01]) cannot boot Debian wheezy i386, nor
   boot under Xen.  I suspect buggy ACPI tables in the BIOS but have
   not yet investigated in detail.

 * For 2 pairs (4 hosts, huxelrebe[01] and rimava[01]) we have yet to
   gain useable remote access to the BIOS configuration.  This is
   being worked on by our supplier with advice from hardware vendors.

However, the existing setup has only x86 10 hosts, so the new lab is
bigger.  The new colo has better (more, and more reliable) ARM hosts
(cubietruck-* and arndale-*).  And the existing setup is affected by
various problems with the underlying infrastructure.


Thanks very much to Ian Campbell and Lars Kurth for their help, and
for the support and patience of the Xen Project Test Working Group and
the community in general.

Ian.

Re: [Wg-test-framework] osstest going live in new test colo

[Chris Shepherd]

And thanks Ian for your vision and dogged persistence on this. Great to have this up and running!


-----Original Message-----
From: wg-test-framework-bounces@lists.xenproject.org [mailto:wg-test-framework-bounces@lists.xenproject.org] On Behalf Of Ian Jackson
Sent: 26 March 2015 13:25
To: wg-test-framework@lists.xenproject.org; xen-devel@lists.xenproject.org
Cc: Ian Campbell
Subject: [Wg-test-framework] osstest going live in new test colo

Based on our commissioning tests, the new facility now has greater capacity and better reliability than the existing setup in the Citrix Cambridge office.

I'm therefore going to set the new system going in production mode, and shut down the old one.

There are a few loose ends to get the cron jobs working right which I expect to have sorted out very soon.  In the meantime I have already instructed the old system to drain its queue, with the expectation of going fully live with the new system on Friday.

You will be able to tell when test reports are coming from the new system because they will have flight numbers >=50000.


There are still some problems with the new facility.  Of the 24 x86 test hosts, only 12 are currently live.  Of the remainder:

 * 2 pairs (4 hosts, chardonnay[01] and elbling[01]) are affected by 2
   different kernel/hypervisor bugs.  I have not yet reported these
   properly, but will do so shortly.  I hope to get community help
   with resolving them.

 * 1 pair (2 hosts, oseleta[01]) are affected by what appears to be a
   BIOS bug that sometimes prevents the machine rebooting; this is
   with our supplier.

 * 1 pair (2 hosts, godello[01]) cannot boot Debian wheezy i386, nor
   boot under Xen.  I suspect buggy ACPI tables in the BIOS but have
   not yet investigated in detail.

 * For 2 pairs (4 hosts, huxelrebe[01] and rimava[01]) we have yet to
   gain useable remote access to the BIOS configuration.  This is
   being worked on by our supplier with advice from hardware vendors.

However, the existing setup has only x86 10 hosts, so the new lab is bigger.  The new colo has better (more, and more reliable) ARM hosts
(cubietruck-* and arndale-*).  And the existing setup is affected by various problems with the underlying infrastructure.


Thanks very much to Ian Campbell and Lars Kurth for their help, and for the support and patience of the Xen Project Test Working Group and the community in general.

Ian.

_______________________________________________
Wg-test-framework mailing list
Wg-test-framework@lists.xenproject.org
http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework