[OSSTEST PATCH 0/5] Support for PDU *and* IPMI

[OSSTEST PATCH 0/5] Support for PDU *and* IPMI

[Ian Jackson]

From: Ian Jackson <Ian.Jackson@eu.citrix.com>

We have been having trouble with our PDU ports: relays are getting
stuck.  This is affecting a distressing variety of machines and both
our PDUs.

We have a theory the relays will work a lot better if we do not open
and close them under the load of the main system.

Deployment of this change for any one test box is as follows:
 * plug the test box's management port into the test network
 * give it an ip address and check that ipmi seems to work
 * set the test box not to come on after AC power loss
 * configure the ansible playbook to specify the appropriate
   config rune (see the final patch).

Currently, I have done this for two of our boxes (rimava[01]) which
are out of service because of relay trouble.  I am about to run
a full commissioning test with these changes.

Ian Jackson (5):
  PDU::ipmi: Do not return until the power state has changed
  PDU::ipmi: Better username/password configuration
  PowerMethod: New "nest" psuedo-method
  PowerMethod: New pause functionality
  PDU::pause: Better pause time configuration

 Osstest/PDU/ipmi.pm    | 28 ++++++++++++++++++--------
 Osstest/PDU/pause.pm   | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++
 Osstest/TestSupport.pm | 16 +++++++++++++--
 3 files changed, 88 insertions(+), 10 deletions(-)
 create mode 100644 Osstest/PDU/pause.pm

-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[OSSTEST PATCH 1/5] PDU::ipmi: Do not return until the power state has changed

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/PDU/ipmi.pm | 26 ++++++++++++++++++--------
 1 file changed, 18 insertions(+), 8 deletions(-)

diff --git a/Osstest/PDU/ipmi.pm b/Osstest/PDU/ipmi.pm
index bff1143..0cbc5b9 100644
--- a/Osstest/PDU/ipmi.pm
+++ b/Osstest/PDU/ipmi.pm
@@ -49,19 +49,29 @@ sub pdu_power_state {
 
     my $cmd = "ipmitool -H $mo->{Mgmt} -U $mo->{User} -P $mo->{Pass}";
 
-    my $status = `$cmd power status`
-	or die "Cannot retrieve current power status";
-    chomp($status);
-    logm("$status (want $onoff)");
-    $status =~ s/^Chassis Power is (on|off)$/$1/
-	or die "Cannot parse current power status: $status";
+    my $getstatus = sub {
+        my $status = `$cmd power status`
+            or die "Cannot retrieve current power status";
+        chomp($status);
+        logm("$status (want $onoff)");
+        $status =~ s/^Chassis Power is (on|off)$/$1/
+            or die "Cannot parse current power status: $status";
+        return $status;
+    };
 
-    if ( $status eq $onoff ) {
+    if ( $getstatus->() eq $onoff ) {
 	logm("Current power status is correct");
 	return;
     }
 
-    system_checked("$cmd power $onoff")
+    system_checked("$cmd power $onoff");
+
+    my $count = 60;
+    for (;;) {
+        last if $getstatus->() eq $onoff;
+        die "did not power $onoff" unless --$count > 0;
+        sleep(1);
+    }
 }
 
 1;
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[OSSTEST PATCH 2/5] PDU::ipmi: Better username/password configuration

[Ian Jackson]

This can now come from host properties, or failing that we use
ADMIN/ADMIN which seems common.

No functional change with working existing configs.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 Osstest/PDU/ipmi.pm | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/Osstest/PDU/ipmi.pm b/Osstest/PDU/ipmi.pm
index 0cbc5b9..dbf211f 100644
--- a/Osstest/PDU/ipmi.pm
+++ b/Osstest/PDU/ipmi.pm
@@ -36,6 +36,8 @@ BEGIN {
 
 sub new {
     my ($class, $ho, $methname, $mgmt, $user, $pass, @opts) = @_;
+    $user ||= get_host_property($ho, 'IpmiUser') || 'ADMIN';
+    $pass ||= get_host_property($ho, 'IpmiPassword') || 'ADMIN';
     return bless { Host => $ho,
 		   Mgmt => $mgmt,
 		   User => $user,
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[OSSTEST PATCH 3/5] PowerMethod: New "nest" psuedo-method

[Ian Jackson]

If specified as the first method, it means to reverse the order when
powering off.

There don't seem to be any docs for PowerMethod, so I have not
documented this right now.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/TestSupport.pm | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 79737ea..ad877e2 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -920,7 +920,11 @@ sub power_cycle_host_setup ($) {
     my ($ho) = @_;
     my $methobjs = [ ];
     foreach my $meth (split /\;\s*/, ($ho->{Power} // 'unsupported')) {
-	push @$methobjs, get_host_method_object($ho,'PDU',$meth);
+        if ($meth eq 'nest') {
+            push @$methobjs, $meth;
+        } else {
+            push @$methobjs, get_host_method_object($ho,'PDU',$meth);
+        }
     }
     $ho->{PowerMethobjs} = $methobjs;
 }
@@ -946,7 +950,12 @@ sub power_cycle ($) {
 sub power_state ($$) {
     my ($ho, $on) = @_;
     logm("power: setting $on for $ho->{Name}");
-    foreach my $mo (@{ $ho->{PowerMethobjs} }) {
+    my @methobjs = @{ $ho->{PowerMethobjs} };
+    if ($methobjs[0] eq 'nest') {
+        shift @methobjs;
+        @methobjs = reverse @methobjs if !$on;
+    }
+    foreach my $mo (@methobjs) {
 	$mo->pdu_power_state($on);
     }
 }
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[OSSTEST PATCH 4/5] PowerMethod: New pause functionality

[Ian Jackson]

And, an abbreviation for invoking it: just writing numbers (or
<on>/<off> pair of numbers).

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/PDU/pause.pm   | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
 Osstest/TestSupport.pm |  3 +++
 2 files changed, 52 insertions(+)
 create mode 100644 Osstest/PDU/pause.pm

diff --git a/Osstest/PDU/pause.pm b/Osstest/PDU/pause.pm
new file mode 100644
index 0000000..b1160c0
--- /dev/null
+++ b/Osstest/PDU/pause.pm
@@ -0,0 +1,49 @@
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2014 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package Osstest::PDU::pause;
+
+use strict;
+use warnings;
+
+use Osstest;
+use Osstest::TestSupport;
+use IO::File;
+
+BEGIN {
+    use Exporter ();
+    our ($VERSION, @ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
+    $VERSION     = 1.00;
+    @ISA         = qw(Exporter);
+    @EXPORT      = qw();
+    %EXPORT_TAGS = ( );
+
+    @EXPORT_OK   = qw();
+}
+
+sub new {
+    my ($class, $ho, $methname, $on, $off) = @_;
+    return bless [ $off, $on ], $class;
+}
+
+sub pdu_power_state {
+    my ($mo, $on) = @_;
+    my $delay = $mo->[!!$on];
+    logm("power: pdu operation pausing for ${delay}s");
+    sleep $delay;
+}
+
+1;
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index ad877e2..004e644 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -922,6 +922,9 @@ sub power_cycle_host_setup ($) {
     foreach my $meth (split /\;\s*/, ($ho->{Power} // 'unsupported')) {
         if ($meth eq 'nest') {
             push @$methobjs, $meth;
+        } elsif ($meth =~ m{^(\d+)(?:/(\d+))$}) {
+            require Osstest::PDU::pause;
+            push @$methobjs, new Osstest::PDU::pause $ho, 'pause', $1, $2//$1;
         } else {
             push @$methobjs, get_host_method_object($ho,'PDU',$meth);
         }
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[OSSTEST PATCH 5/5] PDU::pause: Better pause time configuration

[Ian Jackson]

If no time is specified, use 120 (or the PowerOnTime host property
value) when powering on, or 10 when powering off.

This is all intended to be used like this:
  HostProp_rimava0_PowerMethod nest; msw --apc6 pdu2 19; pause; ipmi rimava0m

This means:
  * For power on, first turn on the APC PDU pdu2 port 19
  * Wait 120s (or HostProp_rimava0_PowerOnTime)
  * Then ipmi to the host name rimava0m with ADMIN/ADMIN
    (or HostProp_rimava0_IpmiUser/HostProp_rimava0_IpmiPassword)
  * For power off, do that in reverse, only with a 10s pause.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 Osstest/PDU/pause.pm   | 5 +++++
 Osstest/TestSupport.pm | 2 +-
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/Osstest/PDU/pause.pm b/Osstest/PDU/pause.pm
index b1160c0..9e839c6 100644
--- a/Osstest/PDU/pause.pm
+++ b/Osstest/PDU/pause.pm
@@ -36,6 +36,11 @@ BEGIN {
 
 sub new {
     my ($class, $ho, $methname, $on, $off) = @_;
+    if (!defined $on) {
+        $on = get_host_property($ho, 'PowerOnTime', 120);
+        $off = 10;
+    }
+    $off //= $on;
     return bless [ $off, $on ], $class;
 }
 
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 004e644..ea54601 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -924,7 +924,7 @@ sub power_cycle_host_setup ($) {
             push @$methobjs, $meth;
         } elsif ($meth =~ m{^(\d+)(?:/(\d+))$}) {
             require Osstest::PDU::pause;
-            push @$methobjs, new Osstest::PDU::pause $ho, 'pause', $1, $2//$1;
+            push @$methobjs, new Osstest::PDU::pause $ho, 'pause', $1, $2;
         } else {
             push @$methobjs, get_host_method_object($ho,'PDU',$meth);
         }
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[OSSTEST PATCH 6/5] PDU::ipmi_try: New method

[Ian Jackson]

For power on, this is just like ipmi.

For power off, it doesn't mind errors (although it has three
attempts).  This is useful when combined with pdu power: if the
chassis power is already off, ipmi will fail.

If the ipmi operation does fail, then the chassing pdu power operation
will do the job.  So this will also fix things if the test case
crashed the bmc or something.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/PDU/ipmi_try.pm | 55 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 55 insertions(+)
 create mode 100644 Osstest/PDU/ipmi_try.pm

diff --git a/Osstest/PDU/ipmi_try.pm b/Osstest/PDU/ipmi_try.pm
new file mode 100644
index 0000000..172f4ec
--- /dev/null
+++ b/Osstest/PDU/ipmi_try.pm
@@ -0,0 +1,55 @@
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2014 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package Osstest::PDU::ipmi_try;
+
+use strict;
+use warnings;
+
+use Osstest;
+use Osstest::TestSupport;
+use IO::File;
+use Osstest::PDU::ipmi;
+
+BEGIN {
+    use Exporter ();
+    our ($VERSION, @ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
+    $VERSION     = 1.00;
+    @ISA         = qw(Exporter Osstest::PDU::ipmi);
+    @EXPORT      = qw();
+    %EXPORT_TAGS = ( );
+
+    @EXPORT_OK   = qw();
+}
+
+sub pdu_power_state {
+    my ($mo, $on) = @_;
+
+    if ($on) {
+	$mo->Osstest::PDU::ipmi::pdu_power_state($on);
+    } else {
+        my $attempts = 3;
+	foreach my $attempt (1..$attempts) {
+	    eval {
+		$mo->Osstest::PDU::ipmi::pdu_power_state($on);
+	    };
+	    last unless $@;
+	    warn "(attempt $attempt/$attempts) $@";
+	}
+    }
+}
+
+1;
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel