[Xen-devel] Stopping much Linux testing in Xen Project CI

* [Xen-devel] Stopping much Linux testing in Xen Project CI
@ 2020-03-12 16:49 Ian Jackson
  2020-03-12 16:51 ` Ian Jackson
                   ` (2 more replies)
  0 siblings, 3 replies; 6+ messages in thread
From: Ian Jackson @ 2020-03-12 16:49 UTC (permalink / raw)
  To: Konrad Rzeszutek Wilk, Stefano Stabellini,
	Oleksandr Andrushchenko, Roger Pau Monné,
	Boris Ostrovsky, Juergen Gross, Wei Liu, Paul Durrant
  Cc: xen-devel

Linux stable branches, and Linux upstream tip, are badly broken and
have been for months.  Apparently no-one is able to (or has time to)
to investigate and fix.

  linux-4.4          218 days         to be suspended
  linux-4.9          134 days         to be suspended
  linux-4.14         134 days         to be suspended
  linux-4.19         134 days         to be suspended
  linux-5.4           55 days
  linux-arm-xen     up to date
  linux-linus        372 days         to be suspended

These are times since the last push - ie, how long it has been broken.
Evidently no-one is paying any attention to this.[1]  I looked at the
reports myself and:

Nested HVM is broken on Intel in all of the 4.x branches.
Additionally:

Linux 4.4 has some intermittent guest start failure for 32-bit PV.

Linux 4.14 does not boot on 32-bit ARM.  There are also some 64-bit
x86 migration failures.

The most recent reports (last week or two) are afflicted by underlying
CI problems - what look like sticky PDU relays, or what may be
problems in the Debian mirror network (I have definitely seen problems
there), so the reports are rather noisy.  Sorry about that.  I am
trying to improve this situation but it is quite difficult [2].  But
overall it is clear that the underlying code is broken.

The repeated almost-certainly-doomed retestes are using too much of
osstest's capacity.  I am going to stop testing all of these 4.x
branches, and of linux-linus, until someone tells me they think the
fix(es) are in the relevant branch(es).

This means that we will *no longer have any visibility of breakage in
much of upstream Linux*.  I think this is "fine" because right now
no-one appears to be doing anything with the information.

I didn't look at master or at 5.4.

Thanks for your attention.

Ian.

[1] osstest does not have a triage team monitoring the reports.  There
is just about half of me to run the whole system.  We are relying on
maintainers of Xen-related code noticing when things break and
proactively trying to investigate and fix them.

[2] Amongst other things, someone has managed to patent, in the USA,
the obvious[3] solution to PDU relays sticking.  PDUs which do zero
crossing switching are available in Europe but are hens' teeth in the
US.  The patentholder doesn't actually make and sell the things of
course so we can't even solve the problem by paying them a bad-law
tax.

[3] Obvious anyone with a reasaonble electrical engineering background
like, for example, me.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

^ permalink raw reply	[flat|nested] 6+ messages in thread