All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH 01/16] hwsim: return radio ID on create
@ 2020-08-27 17:32 James Prestwood
  2020-08-27 17:32 ` [PATCH 02/16] auto-t: prepare autotests for test-runner re-write James Prestwood
                   ` (15 more replies)
  0 siblings, 16 replies; 18+ messages in thread
From: James Prestwood @ 2020-08-27 17:32 UTC (permalink / raw)
  To: iwd

[-- Attachment #1: Type: text/plain, Size: 1921 bytes --]

Hwsim was relying on some internal behavior of mac80211_hwsim where new
radios are created with ID's starting at zero and incremented. While
this is a reasonable assumption its better to obtain the actual radio
ID since mac80211_hwsim returns it in the radio creation callback.

Now hwsim will return the radio ID when creating radios. Negative returns
are still errors, but any value >= 0 indicates the radio ID of the newly
created radio.
---
 tools/hwsim.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/tools/hwsim.c b/tools/hwsim.c
index 02053fa1..467ccd2d 100644
--- a/tools/hwsim.c
+++ b/tools/hwsim.c
@@ -216,6 +216,7 @@ static void create_callback(struct l_genl_msg *msg, void *user_data)
 		radio_id = err;
 
 		l_info("Created new radio with id %u", radio_id);
+		exit_status = radio_id;
 	} else {
 		l_warn("Failed to get create return value");
 		exit_status = EXIT_FAILURE;
@@ -2605,7 +2606,8 @@ static void usage(void)
 	printf("\thwsim [options]\n");
 	printf("Options:\n"
 		"\t-L, --list [id]        List simulated radios\n"
-		"\t-C, --create           Create new simulated radio\n"
+		"\t-C, --create           Create new simulated radio. The "
+						"return value is the new radio ID\n"
 		"\t-D, --destroy <id>     Destroy existing radio\n"
 		"\t-n, --name <name>      Name of a radio to be created\n"
 		"\t-i, --nointerface      Do not create VIF\n"
@@ -2632,6 +2634,7 @@ static const struct option main_options[] = {
 int main(int argc, char *argv[])
 {
 	int actions = 0;
+	int ret;
 
 	for (;;) {
 		int opt;
@@ -2729,7 +2732,9 @@ int main(int argc, char *argv[])
 		goto done;
 	}
 
-	exit_status = l_main_run_with_signal(signal_handler, NULL);
+	ret = l_main_run_with_signal(signal_handler, NULL);
+	if (ret)
+		exit_status = ret;
 
 	l_genl_family_free(hwsim);
 	l_genl_family_free(nl80211);
-- 
2.21.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [PATCH 02/16] auto-t: prepare autotests for test-runner re-write
  2020-08-27 17:32 [PATCH 01/16] hwsim: return radio ID on create James Prestwood
@ 2020-08-27 17:32 ` James Prestwood
  2020-08-27 17:32 ` [PATCH 03/16] auto-t: introduce pure python " James Prestwood
                   ` (14 subsequent siblings)
  15 siblings, 0 replies; 18+ messages in thread
From: James Prestwood @ 2020-08-27 17:32 UTC (permalink / raw)
  To: iwd

[-- Attachment #1: Type: text/plain, Size: 34507 bytes --]

The tests basically remained the same with a few minor changes.
The wiphy_map and in turn hostapd_map are no longer used. This
was already partially converted a long time ago when the 'config'
parameter was added to HostapdCLI. This patch fully converts all
autotests to use 'config' rather than looking up by interface.

Some test scripts were named 'test.py' which was fine before but
the new rewrite actually loads each python test as a module. The
name 'test' is too ambiguous and causes issues due to a native
python module with the same name. All of these files were
renamed to 'connection_test.py'.
---
 .../{test.py => connection_test.py}           |  0
 .../testAP/{test.py => connection_test.py}    |  9 ++--
 autotests/testAP/failure_test.py              |  6 +--
 autotests/testAPRoam/connection_test.py       |  2 -
 .../testBSSBlacklist/all_blacklisted_test.py  |  4 --
 autotests/testBSSBlacklist/bad_pass_test.py   |  4 --
 autotests/testBSSBlacklist/connection_test.py |  1 -
 .../testBSSBlacklist/temp_blacklist_test.py   |  4 --
 .../connect_command_test.py                   | 42 +++++++------------
 .../disconnect_by_ap_test.py                  |  4 +-
 autotests/testEAP-AKA/connection_test.py      |  1 -
 .../testEAP-PEAPv0-CryptoBinding/ISK_test.py  |  1 -
 .../NoISK_test.py                             |  1 -
 .../testEAP-PEAPv0-CryptoBinding/main.conf    |  2 -
 autotests/testEAP-PWD/connection_test.py      |  1 -
 autotests/testEAP-SIM/connection_test.py      |  1 -
 .../connection_test.py                        | 11 ++---
 .../testEAP-TTLS-CHAP/connection_test.py      |  1 -
 .../testEAP-TTLS-MSCHAP/connection_test.py    |  1 -
 autotests/testEAP-TTLS-PAP/connection_test.py |  1 -
 autotests/testEAP-WPS-Frag/wps_frag_test.py   |  4 +-
 autotests/testEAP-WPS/four_digit_pin_test.py  |  4 +-
 autotests/testEAP-WPS/pin_test.py             |  4 +-
 autotests/testEAP-WPS/push_button_test.py     |  5 +--
 autotests/testFILS/fils_256_test.py           |  1 -
 autotests/testFILS/fils_384_test.py           |  1 -
 .../{test.py => connection_test.py}           |  2 +-
 .../{test.py => connection_test.py}           |  2 +-
 .../{test.py => connection_test.py}           |  2 +-
 .../{test.py => connection_test.py}           |  2 +-
 .../{test.py => connection_test.py}           | 18 +++++---
 .../{test.py => connection_test.py}           |  2 +-
 .../{test.py => connection_test.py}           |  2 +-
 autotests/testOWE/connection_test.py          |  1 -
 autotests/testOWE/renegotiate_test.py         |  1 -
 .../{test.py => connection_test.py}           |  2 +-
 .../{test.py => connection_test.py}           |  0
 autotests/testSAE/connection_test.py          | 10 ++---
 autotests/testSAE/group_20_connection_test.py | 10 ++---
 .../testSAQuery-spoofing/connection_test.py   |  2 +-
 autotests/testSAQuery/connection_test.py      |  2 +-
 41 files changed, 57 insertions(+), 117 deletions(-)
 rename autotests/testAP-no-support/{test.py => connection_test.py} (100%)
 rename autotests/testAP/{test.py => connection_test.py} (94%)
 delete mode 100644 autotests/testEAP-PEAPv0-CryptoBinding/main.conf
 rename autotests/testFT-8021x-roam/{test.py => connection_test.py} (99%)
 rename autotests/testFT-FILS-SHA256/{test.py => connection_test.py} (99%)
 rename autotests/testFT-FILS-SHA384/{test.py => connection_test.py} (99%)
 rename autotests/testFT-PSK-over-DS/{test.py => connection_test.py} (99%)
 rename autotests/testFT-PSK-roam/{test.py => connection_test.py} (95%)
 rename autotests/testFT-SAE-roam/{test.py => connection_test.py} (99%)
 rename autotests/testHT-VHT/{test.py => connection_test.py} (98%)
 rename autotests/testPreauth-roam/{test.py => connection_test.py} (98%)
 rename autotests/testRSSIAgent/{test.py => connection_test.py} (100%)

diff --git a/autotests/testAP-no-support/test.py b/autotests/testAP-no-support/connection_test.py
similarity index 100%
rename from autotests/testAP-no-support/test.py
rename to autotests/testAP-no-support/connection_test.py
diff --git a/autotests/testAP/test.py b/autotests/testAP/connection_test.py
similarity index 94%
rename from autotests/testAP/test.py
rename to autotests/testAP/connection_test.py
index 4cacf9ad..36302da1 100644
--- a/autotests/testAP/test.py
+++ b/autotests/testAP/connection_test.py
@@ -3,17 +3,17 @@
 import unittest
 import sys, os
 
-sys.path.append('../util')
 import iwd
 from iwd import IWD
 from iwd import PSKAgent
 from iwd import NetworkType
-import hostapd
+from hostapd import HostapdCLI
 import testutil
 
 class Test(unittest.TestCase):
 
     def client_connect(self, wd, dev):
+        hostapd = HostapdCLI(config='psk-ccmp.conf')
 
         ordered_network = dev.get_ordered_network('TestAP1', True)
 
@@ -30,8 +30,7 @@ class Test(unittest.TestCase):
         wd.unregister_psk_agent(psk_agent)
 
         testutil.test_iface_operstate(dev.name)
-        testutil.test_ifaces_connected(list(hostapd.hostapd_map.keys())[0],
-                                       dev.name)
+        testutil.test_ifaces_connected(hostapd.ifname, dev.name)
 
         dev.disconnect()
 
@@ -57,6 +56,7 @@ class Test(unittest.TestCase):
             wd.wait_for_object_condition(dev2, condition)
 
             ordered_networks = dev2.get_ordered_networks()
+
             networks = { n.name: n for n in ordered_networks }
             self.assertEqual(networks['TestAP1'].type, NetworkType.psk)
             self.assertEqual(networks['TestAP2'].type, NetworkType.psk)
@@ -66,6 +66,7 @@ class Test(unittest.TestCase):
 
             try:
                 dev2.disconnect()
+
                 condition = 'not obj.connected'
                 wd.wait_for_object_condition(dev2, condition)
             except:
diff --git a/autotests/testAP/failure_test.py b/autotests/testAP/failure_test.py
index d30bccb4..a2622f45 100644
--- a/autotests/testAP/failure_test.py
+++ b/autotests/testAP/failure_test.py
@@ -8,12 +8,13 @@ import iwd
 from iwd import IWD
 from iwd import PSKAgent
 from iwd import NetworkType
-import hostapd
+from hostapd import HostapdCLI
 import testutil
 
 class Test(unittest.TestCase):
 
     def client_connect(self, wd, dev):
+        hostapd = HostapdCLI(config='psk-ccmp.conf')
 
         ordered_network = dev.get_ordered_network('TestAP1', True)
 
@@ -30,8 +31,7 @@ class Test(unittest.TestCase):
         wd.unregister_psk_agent(psk_agent)
 
         testutil.test_iface_operstate(dev.name)
-        testutil.test_ifaces_connected(list(hostapd.hostapd_map.keys())[0],
-                                       dev.name)
+        testutil.test_ifaces_connected(hostapd.ifname, dev.name)
 
         dev.disconnect()
 
diff --git a/autotests/testAPRoam/connection_test.py b/autotests/testAPRoam/connection_test.py
index 292754c9..c8c2ca9f 100644
--- a/autotests/testAPRoam/connection_test.py
+++ b/autotests/testAPRoam/connection_test.py
@@ -10,8 +10,6 @@ from iwd import PSKAgent
 from iwd import NetworkType
 
 from hostapd import HostapdCLI
-from hostapd import hostapd_map
-
 from hwsim import Hwsim
 
 class Test(unittest.TestCase):
diff --git a/autotests/testBSSBlacklist/all_blacklisted_test.py b/autotests/testBSSBlacklist/all_blacklisted_test.py
index 78239309..099804a8 100644
--- a/autotests/testBSSBlacklist/all_blacklisted_test.py
+++ b/autotests/testBSSBlacklist/all_blacklisted_test.py
@@ -10,12 +10,8 @@ from iwd import PSKAgent
 from iwd import NetworkType
 
 from hostapd import HostapdCLI
-from hostapd import hostapd_map
-
 from hwsim import Hwsim
 
-import time
-
 class Test(unittest.TestCase):
 
     def test_connection_success(self):
diff --git a/autotests/testBSSBlacklist/bad_pass_test.py b/autotests/testBSSBlacklist/bad_pass_test.py
index 3516f3be..220ffcc8 100644
--- a/autotests/testBSSBlacklist/bad_pass_test.py
+++ b/autotests/testBSSBlacklist/bad_pass_test.py
@@ -10,12 +10,8 @@ from iwd import PSKAgent
 from iwd import NetworkType
 
 from hostapd import HostapdCLI
-from hostapd import hostapd_map
-
 from hwsim import Hwsim
 
-import time
-
 class Test(unittest.TestCase):
 
     def test_connection_success(self):
diff --git a/autotests/testBSSBlacklist/connection_test.py b/autotests/testBSSBlacklist/connection_test.py
index 7433d5f3..825439d4 100644
--- a/autotests/testBSSBlacklist/connection_test.py
+++ b/autotests/testBSSBlacklist/connection_test.py
@@ -10,7 +10,6 @@ from iwd import PSKAgent
 from iwd import NetworkType
 
 from hostapd import HostapdCLI
-from hostapd import hostapd_map
 
 from hwsim import Hwsim
 
diff --git a/autotests/testBSSBlacklist/temp_blacklist_test.py b/autotests/testBSSBlacklist/temp_blacklist_test.py
index f43f807c..b2b57035 100644
--- a/autotests/testBSSBlacklist/temp_blacklist_test.py
+++ b/autotests/testBSSBlacklist/temp_blacklist_test.py
@@ -10,12 +10,8 @@ from iwd import PSKAgent
 from iwd import NetworkType
 
 from hostapd import HostapdCLI
-from hostapd import hostapd_map
-
 from hwsim import Hwsim
 
-import time
-
 class Test(unittest.TestCase):
 
     def test_connection_success(self):
diff --git a/autotests/testClientNonInteractive/connect_command_test.py b/autotests/testClientNonInteractive/connect_command_test.py
index 494f05d2..886d7a76 100644
--- a/autotests/testClientNonInteractive/connect_command_test.py
+++ b/autotests/testClientNonInteractive/connect_command_test.py
@@ -12,29 +12,25 @@ import subprocess
 class Test(unittest.TestCase):
 
     def check_connection_success(self, ssid):
-        wd = IWD()
-
-        device = wd.list_devices(1)[0]
+        device = self.wd.list_devices(1)[0]
 
         condition = 'not obj.scanning'
-        wd.wait_for_object_condition(device, condition)
+        self.wd.wait_for_object_condition(device, condition)
 
         ordered_network = device.get_ordered_network(ssid)
 
         condition = 'obj.connected'
-        wd.wait_for_object_condition(ordered_network.network_object, condition)
+        self.wd.wait_for_object_condition(ordered_network.network_object, condition)
 
         device.disconnect()
 
         condition = 'not obj.connected'
-        wd.wait_for_object_condition(ordered_network.network_object, condition)
+        self.wd.wait_for_object_condition(ordered_network.network_object, condition)
 
     def test_connection_with_passphrase(self):
         ssid = 'ssidPassphrase'
 
-        wd = IWD()
-
-        device = wd.list_devices(1)[0]
+        device = self.wd.list_devices(1)[0]
 
         # Use --dontaks cmd-line option
         with self.assertRaises(subprocess.CalledProcessError):
@@ -49,9 +45,7 @@ class Test(unittest.TestCase):
     def test_connection_with_username_and_password(self):
         ssid = 'ssidUNameAndPWord'
 
-        wd = IWD()
-
-        device = wd.list_devices(1)[0]
+        device = self.wd.list_devices(1)[0]
 
         subprocess.check_call(['iwctl', '-u', 'user', '-p', 'password',
                                 'station', device.name, 'connect', ssid])
@@ -61,9 +55,7 @@ class Test(unittest.TestCase):
     def test_connection_with_password(self):
         ssid = 'ssidPWord'
 
-        wd = IWD()
-
-        device = wd.list_devices(1)[0]
+        device = self.wd.list_devices(1)[0]
 
         subprocess.check_call(['iwctl', '-p', 'password',
                                 'station', device.name, 'connect', ssid])
@@ -73,9 +65,7 @@ class Test(unittest.TestCase):
     def test_connection_failure(self):
         ssid = 'ssidPassphrase'
 
-        wd = IWD()
-
-        device = wd.list_devices(1)[0]
+        device = self.wd.list_devices(1)[0]
 
         with self.assertRaises(subprocess.CalledProcessError):
                 subprocess.check_call(['iwctl', '-P', 'incorrect_passphrase',
@@ -84,18 +74,14 @@ class Test(unittest.TestCase):
     def test_invalid_command_line_option(self):
         ssid = 'ssidPassphrase'
 
-        wd = IWD()
-
-        device = wd.list_devices(1)[0]
+        device = self.wd.list_devices(1)[0]
 
         with self.assertRaises(subprocess.CalledProcessError):
                 subprocess.check_call(['iwctl', '-z',
                                 'station', device.name, 'connect', ssid])
 
     def test_invalid_command(self):
-        wd = IWD()
-
-        device = wd.list_devices(1)[0]
+        device = self.wd.list_devices(1)[0]
 
         with self.assertRaises(subprocess.CalledProcessError):
                 subprocess.check_call(['iwctl', 'inexistent', 'command'])
@@ -105,17 +91,17 @@ class Test(unittest.TestCase):
         IWD.copy_to_storage('ssidUNameAndPWord.8021x')
         IWD.copy_to_storage('ssidPWord.8021x')
 
-        wd = IWD()
+        cls.wd = IWD()
 
-        device = wd.list_devices(1)[0]
+        device = cls.wd.list_devices(1)[0]
 
         condition = 'not obj.scanning'
-        wd.wait_for_object_condition(device, condition)
+        cls.wd.wait_for_object_condition(device, condition)
 
         device.scan()
 
         condition = 'not obj.scanning'
-        wd.wait_for_object_condition(device, condition)
+        cls.wd.wait_for_object_condition(device, condition)
 
     @classmethod
     def tearDownClass(cls):
diff --git a/autotests/testDisconnectByAP/disconnect_by_ap_test.py b/autotests/testDisconnectByAP/disconnect_by_ap_test.py
index 298d75c4..f003ec0c 100644
--- a/autotests/testDisconnectByAP/disconnect_by_ap_test.py
+++ b/autotests/testDisconnectByAP/disconnect_by_ap_test.py
@@ -10,7 +10,6 @@ from iwd import DeviceState
 from iwd import NetworkType
 
 from hostapd import HostapdCLI
-from hostapd import hostapd_map
 
 class Test(unittest.TestCase):
 
@@ -20,8 +19,7 @@ class Test(unittest.TestCase):
         devices = wd.list_devices(1)
         device = devices[0]
 
-        hostapd_if = list(hostapd_map.values())[0]
-        hostapd = HostapdCLI(hostapd_if)
+        hostapd = HostapdCLI(config='ssidOpen.conf')
 
         device.scan()
 
diff --git a/autotests/testEAP-AKA/connection_test.py b/autotests/testEAP-AKA/connection_test.py
index bcdb1d92..c4b7c2f9 100644
--- a/autotests/testEAP-AKA/connection_test.py
+++ b/autotests/testEAP-AKA/connection_test.py
@@ -10,7 +10,6 @@ from iwd import NetworkType
 from hlrauc import AuthCenter
 
 from hostapd import HostapdCLI
-from hostapd import hostapd_map
 
 class Test(unittest.TestCase):
 
diff --git a/autotests/testEAP-PEAPv0-CryptoBinding/ISK_test.py b/autotests/testEAP-PEAPv0-CryptoBinding/ISK_test.py
index 7ba49384..a25f99ef 100644
--- a/autotests/testEAP-PEAPv0-CryptoBinding/ISK_test.py
+++ b/autotests/testEAP-PEAPv0-CryptoBinding/ISK_test.py
@@ -11,7 +11,6 @@ from iwd import NetworkType
 import testutil
 
 from hostapd import HostapdCLI
-from hostapd import hostapd_map
 
 class Test(unittest.TestCase):
 
diff --git a/autotests/testEAP-PEAPv0-CryptoBinding/NoISK_test.py b/autotests/testEAP-PEAPv0-CryptoBinding/NoISK_test.py
index 0f7d432c..33ff0285 100644
--- a/autotests/testEAP-PEAPv0-CryptoBinding/NoISK_test.py
+++ b/autotests/testEAP-PEAPv0-CryptoBinding/NoISK_test.py
@@ -11,7 +11,6 @@ from iwd import NetworkType
 import testutil
 
 from hostapd import HostapdCLI
-from hostapd import hostapd_map
 
 class Test(unittest.TestCase):
 
diff --git a/autotests/testEAP-PEAPv0-CryptoBinding/main.conf b/autotests/testEAP-PEAPv0-CryptoBinding/main.conf
deleted file mode 100644
index 55a5543e..00000000
--- a/autotests/testEAP-PEAPv0-CryptoBinding/main.conf
+++ /dev/null
@@ -1,2 +0,0 @@
-[General]
-UseDefaultInterface=true
diff --git a/autotests/testEAP-PWD/connection_test.py b/autotests/testEAP-PWD/connection_test.py
index 0c722c44..2edc45b8 100644
--- a/autotests/testEAP-PWD/connection_test.py
+++ b/autotests/testEAP-PWD/connection_test.py
@@ -10,7 +10,6 @@ from iwd import PSKAgent
 from iwd import NetworkType
 
 from hostapd import HostapdCLI
-from hostapd import hostapd_map
 
 class Test(unittest.TestCase):
 
diff --git a/autotests/testEAP-SIM/connection_test.py b/autotests/testEAP-SIM/connection_test.py
index 97956d0c..617d833b 100644
--- a/autotests/testEAP-SIM/connection_test.py
+++ b/autotests/testEAP-SIM/connection_test.py
@@ -10,7 +10,6 @@ from iwd import NetworkType
 from hlrauc import AuthCenter
 
 from hostapd import HostapdCLI
-from hostapd import hostapd_map
 
 class Test(unittest.TestCase):
 
diff --git a/autotests/testEAP-TLS-embedded-pems/connection_test.py b/autotests/testEAP-TLS-embedded-pems/connection_test.py
index 1102b810..5d46868d 100644
--- a/autotests/testEAP-TLS-embedded-pems/connection_test.py
+++ b/autotests/testEAP-TLS-embedded-pems/connection_test.py
@@ -9,23 +9,18 @@ from iwd import IWD
 from iwd import PSKAgent
 from iwd import NetworkType
 import testutil
-import hostapd
+from hostapd import HostapdCLI
 
 class Test(unittest.TestCase):
 
     def do_test_connection_success(self, ssid, passphrase=None):
+        hostapd = HostapdCLI(config=ssid + '.conf')
         wd = IWD()
 
         if passphrase:
             psk_agent = PSKAgent(passphrase)
             wd.register_psk_agent(psk_agent)
 
-        hostapd_ifname = None
-        for ifname in hostapd.hostapd_map:
-            if ssid + '.conf' in hostapd.hostapd_map[ifname].config:
-                hostapd_ifname = ifname
-                break
-
         devices = wd.list_devices(1)
         device = devices[0]
 
@@ -52,7 +47,7 @@ class Test(unittest.TestCase):
         wd.wait_for_object_condition(ordered_network.network_object, condition)
 
         testutil.test_iface_operstate()
-        testutil.test_ifaces_connected(hostapd_ifname, device.name)
+        testutil.test_ifaces_connected(hostapd.ifname, device.name)
 
         device.disconnect()
 
diff --git a/autotests/testEAP-TTLS-CHAP/connection_test.py b/autotests/testEAP-TTLS-CHAP/connection_test.py
index e3719669..331e799f 100644
--- a/autotests/testEAP-TTLS-CHAP/connection_test.py
+++ b/autotests/testEAP-TTLS-CHAP/connection_test.py
@@ -11,7 +11,6 @@ from iwd import PSKAgent
 from iwd import NetworkType
 
 from hostapd import HostapdCLI
-from hostapd import hostapd_map
 
 class Test(unittest.TestCase):
 
diff --git a/autotests/testEAP-TTLS-MSCHAP/connection_test.py b/autotests/testEAP-TTLS-MSCHAP/connection_test.py
index ea245a38..973370a8 100644
--- a/autotests/testEAP-TTLS-MSCHAP/connection_test.py
+++ b/autotests/testEAP-TTLS-MSCHAP/connection_test.py
@@ -11,7 +11,6 @@ from iwd import PSKAgent
 from iwd import NetworkType
 
 from hostapd import HostapdCLI
-from hostapd import hostapd_map
 
 class Test(unittest.TestCase):
 
diff --git a/autotests/testEAP-TTLS-PAP/connection_test.py b/autotests/testEAP-TTLS-PAP/connection_test.py
index ce787b01..a8f6fde7 100644
--- a/autotests/testEAP-TTLS-PAP/connection_test.py
+++ b/autotests/testEAP-TTLS-PAP/connection_test.py
@@ -11,7 +11,6 @@ from iwd import PSKAgent
 from iwd import NetworkType
 
 from hostapd import HostapdCLI
-from hostapd import hostapd_map
 
 class Test(unittest.TestCase):
 
diff --git a/autotests/testEAP-WPS-Frag/wps_frag_test.py b/autotests/testEAP-WPS-Frag/wps_frag_test.py
index 9d106a75..b9be0159 100644
--- a/autotests/testEAP-WPS-Frag/wps_frag_test.py
+++ b/autotests/testEAP-WPS-Frag/wps_frag_test.py
@@ -9,7 +9,6 @@ from iwd import IWD
 from iwd import DeviceState
 
 from hostapd import HostapdCLI
-from hostapd import hostapd_map
 
 class Test(unittest.TestCase):
 
@@ -32,8 +31,7 @@ class Test(unittest.TestCase):
 
     @classmethod
     def setUpClass(cls):
-        cls.hostapd_if = list(hostapd_map.values())[0]
-        cls.hostapd = HostapdCLI(cls.hostapd_if)
+        cls.hostapd = HostapdCLI(config='ssid-wps-small-mtu.conf')
 
         cls.hostapd.wps_push_button()
 
diff --git a/autotests/testEAP-WPS/four_digit_pin_test.py b/autotests/testEAP-WPS/four_digit_pin_test.py
index feb4222d..10afaa5d 100644
--- a/autotests/testEAP-WPS/four_digit_pin_test.py
+++ b/autotests/testEAP-WPS/four_digit_pin_test.py
@@ -9,7 +9,6 @@ from iwd import IWD
 from iwd import DeviceState
 
 from hostapd import HostapdCLI
-from hostapd import hostapd_map
 
 class Test(unittest.TestCase):
 
@@ -39,8 +38,7 @@ class Test(unittest.TestCase):
 
     @classmethod
     def setUpClass(cls):
-        cls.hostapd_if = list(hostapd_map.values())[0]
-        cls.hostapd = HostapdCLI(cls.hostapd_if)
+        cls.hostapd = HostapdCLI(config='ssidWPS.conf')
 
     @classmethod
     def tearDownClass(cls):
diff --git a/autotests/testEAP-WPS/pin_test.py b/autotests/testEAP-WPS/pin_test.py
index 080a91e1..706dc20b 100644
--- a/autotests/testEAP-WPS/pin_test.py
+++ b/autotests/testEAP-WPS/pin_test.py
@@ -9,7 +9,6 @@ from iwd import IWD
 from iwd import DeviceState
 
 from hostapd import HostapdCLI
-from hostapd import hostapd_map
 
 class Test(unittest.TestCase):
 
@@ -42,8 +41,7 @@ class Test(unittest.TestCase):
 
     @classmethod
     def setUpClass(cls):
-        cls.hostapd_if = list(hostapd_map.values())[0]
-        cls.hostapd = HostapdCLI(cls.hostapd_if)
+        cls.hostapd = HostapdCLI(config='ssidWPS.conf')
 
     @classmethod
     def tearDownClass(cls):
diff --git a/autotests/testEAP-WPS/push_button_test.py b/autotests/testEAP-WPS/push_button_test.py
index bbf09dcb..f82a56b5 100644
--- a/autotests/testEAP-WPS/push_button_test.py
+++ b/autotests/testEAP-WPS/push_button_test.py
@@ -9,8 +9,6 @@ from iwd import IWD
 from iwd import DeviceState
 
 from hostapd import HostapdCLI
-from hostapd import hostapd_map
-
 class Test(unittest.TestCase):
 
     def push_button_success(self, wd):
@@ -38,8 +36,7 @@ class Test(unittest.TestCase):
 
     @classmethod
     def setUpClass(cls):
-        cls.hostapd_if = list(hostapd_map.values())[0]
-        cls.hostapd = HostapdCLI(cls.hostapd_if)
+        cls.hostapd = HostapdCLI(config='ssidWPS.conf')
 
     @classmethod
     def tearDownClass(cls):
diff --git a/autotests/testFILS/fils_256_test.py b/autotests/testFILS/fils_256_test.py
index aa9e441d..af033d4a 100644
--- a/autotests/testFILS/fils_256_test.py
+++ b/autotests/testFILS/fils_256_test.py
@@ -9,7 +9,6 @@ import iwd
 from iwd import IWD
 from iwd import PSKAgent
 from iwd import NetworkType
-from hostapd import hostapd_map
 from hostapd import HostapdCLI
 import testutil
 
diff --git a/autotests/testFILS/fils_384_test.py b/autotests/testFILS/fils_384_test.py
index ed5cc3b0..70e3bc87 100644
--- a/autotests/testFILS/fils_384_test.py
+++ b/autotests/testFILS/fils_384_test.py
@@ -9,7 +9,6 @@ import iwd
 from iwd import IWD
 from iwd import PSKAgent
 from iwd import NetworkType
-from hostapd import hostapd_map
 from hostapd import HostapdCLI
 import testutil
 
diff --git a/autotests/testFT-8021x-roam/test.py b/autotests/testFT-8021x-roam/connection_test.py
similarity index 99%
rename from autotests/testFT-8021x-roam/test.py
rename to autotests/testFT-8021x-roam/connection_test.py
index 0e6f733e..65c73492 100644
--- a/autotests/testFT-8021x-roam/test.py
+++ b/autotests/testFT-8021x-roam/connection_test.py
@@ -8,7 +8,7 @@ import iwd
 from iwd import IWD
 from iwd import NetworkType
 from hwsim import Hwsim
-from hostapd import HostapdCLI, hostapd_map
+from hostapd import HostapdCLI
 import testutil
 
 class Test(unittest.TestCase):
diff --git a/autotests/testFT-FILS-SHA256/test.py b/autotests/testFT-FILS-SHA256/connection_test.py
similarity index 99%
rename from autotests/testFT-FILS-SHA256/test.py
rename to autotests/testFT-FILS-SHA256/connection_test.py
index 7b800f1c..16f46210 100644
--- a/autotests/testFT-FILS-SHA256/test.py
+++ b/autotests/testFT-FILS-SHA256/connection_test.py
@@ -9,7 +9,7 @@ from iwd import IWD
 from iwd import PSKAgent
 from iwd import NetworkType
 from hwsim import Hwsim
-from hostapd import HostapdCLI, hostapd_map
+from hostapd import HostapdCLI
 import testutil
 
 class Test(unittest.TestCase):
diff --git a/autotests/testFT-FILS-SHA384/test.py b/autotests/testFT-FILS-SHA384/connection_test.py
similarity index 99%
rename from autotests/testFT-FILS-SHA384/test.py
rename to autotests/testFT-FILS-SHA384/connection_test.py
index 7b800f1c..16f46210 100644
--- a/autotests/testFT-FILS-SHA384/test.py
+++ b/autotests/testFT-FILS-SHA384/connection_test.py
@@ -9,7 +9,7 @@ from iwd import IWD
 from iwd import PSKAgent
 from iwd import NetworkType
 from hwsim import Hwsim
-from hostapd import HostapdCLI, hostapd_map
+from hostapd import HostapdCLI
 import testutil
 
 class Test(unittest.TestCase):
diff --git a/autotests/testFT-PSK-over-DS/test.py b/autotests/testFT-PSK-over-DS/connection_test.py
similarity index 99%
rename from autotests/testFT-PSK-over-DS/test.py
rename to autotests/testFT-PSK-over-DS/connection_test.py
index f3dd088c..2cfdf245 100644
--- a/autotests/testFT-PSK-over-DS/test.py
+++ b/autotests/testFT-PSK-over-DS/connection_test.py
@@ -9,7 +9,7 @@ from iwd import IWD
 from iwd import PSKAgent
 from iwd import NetworkType
 from hwsim import Hwsim
-from hostapd import HostapdCLI, hostapd_map
+from hostapd import HostapdCLI
 import testutil
 
 class Test(unittest.TestCase):
diff --git a/autotests/testFT-PSK-roam/test.py b/autotests/testFT-PSK-roam/connection_test.py
similarity index 95%
rename from autotests/testFT-PSK-roam/test.py
rename to autotests/testFT-PSK-roam/connection_test.py
index 2443593d..1b32cc0b 100644
--- a/autotests/testFT-PSK-roam/test.py
+++ b/autotests/testFT-PSK-roam/connection_test.py
@@ -9,7 +9,7 @@ from iwd import IWD
 from iwd import PSKAgent
 from iwd import NetworkType
 from hwsim import Hwsim
-from hostapd import HostapdCLI, hostapd_map
+from hostapd import HostapdCLI
 import testutil
 
 class Test(unittest.TestCase):
@@ -24,12 +24,14 @@ class Test(unittest.TestCase):
         rule1.source = self.bss_radio[1].addresses[0]
         rule1.bidirectional = True
 
-        wd = IWD()
+        wd = self.wd
 
         psk_agent = PSKAgent("EasilyGuessedPassword")
         wd.register_psk_agent(psk_agent)
 
         device = wd.list_devices(1)[0]
+        # prevent autoconnect
+        device.disconnect()
 
         # Check that iwd selects BSS 0 first
         rule0.signal = -2000
@@ -97,6 +99,7 @@ class Test(unittest.TestCase):
                           (self.bss_hostapd[0].ifname, device.name))
 
     def test_roam_on_beacon_loss(self):
+        print("Starting test_roam_on_beacon_loss")
         hwsim = Hwsim()
 
         rule0 = hwsim.rules.create()
@@ -107,12 +110,14 @@ class Test(unittest.TestCase):
         rule1.source = self.bss_radio[1].addresses[0]
         rule1.bidirectional = True
 
-        wd = IWD()
+        wd = self.wd
 
         psk_agent = PSKAgent("EasilyGuessedPassword")
         wd.register_psk_agent(psk_agent)
 
         device = wd.list_devices(1)[0]
+        # prevent autoconnect
+        device.disconnect()
 
         # Check that iwd selects BSS 0 first
         rule0.signal = -2000
@@ -176,6 +181,8 @@ class Test(unittest.TestCase):
         testutil.test_ifaces_connected(self.bss_hostapd[1].ifname, device.name)
         self.assertRaises(Exception, testutil.test_ifaces_connected,
                           (self.bss_hostapd[0].ifname, device.name))
+    def setUp(self):
+        self.wd = IWD()
 
     def tearDown(self):
         os.system('ifconfig "' + self.bss_hostapd[0].ifname + '" down')
@@ -184,15 +191,14 @@ class Test(unittest.TestCase):
         os.system('ifconfig "' + self.bss_hostapd[1].ifname + '" up')
 
         hwsim = Hwsim()
-        wd = IWD()
-        device = wd.list_devices(1)[0]
+        device = self.wd.list_devices(1)[0]
         try:
             device.disconnect()
         except:
             pass
 
         condition = 'obj.state == DeviceState.disconnected'
-        wd.wait_for_object_condition(device, condition)
+        self.wd.wait_for_object_condition(device, condition)
 
         for rule in list(hwsim.rules.keys()):
             del hwsim.rules[rule]
diff --git a/autotests/testFT-SAE-roam/test.py b/autotests/testFT-SAE-roam/connection_test.py
similarity index 99%
rename from autotests/testFT-SAE-roam/test.py
rename to autotests/testFT-SAE-roam/connection_test.py
index 9c2b3ee1..3e209560 100644
--- a/autotests/testFT-SAE-roam/test.py
+++ b/autotests/testFT-SAE-roam/connection_test.py
@@ -9,7 +9,7 @@ from iwd import IWD
 from iwd import PSKAgent
 from iwd import NetworkType
 from hwsim import Hwsim
-from hostapd import HostapdCLI, hostapd_map
+from hostapd import HostapdCLI
 import testutil
 
 class Test(unittest.TestCase):
diff --git a/autotests/testHT-VHT/test.py b/autotests/testHT-VHT/connection_test.py
similarity index 98%
rename from autotests/testHT-VHT/test.py
rename to autotests/testHT-VHT/connection_test.py
index 0129d8a0..e4781ea0 100644
--- a/autotests/testHT-VHT/test.py
+++ b/autotests/testHT-VHT/connection_test.py
@@ -9,9 +9,9 @@ from iwd import IWD
 from iwd import PSKAgent
 from iwd import NetworkType
 from hwsim import Hwsim
-from hostapd import HostapdCLI, hostapd_map
 import testutil
 from time import sleep
+from hostapd import HostapdCLI
 
 class Test(unittest.TestCase):
     def do_connect(self, wd, device, hostapd):
diff --git a/autotests/testOWE/connection_test.py b/autotests/testOWE/connection_test.py
index e4482bc0..539b06d0 100644
--- a/autotests/testOWE/connection_test.py
+++ b/autotests/testOWE/connection_test.py
@@ -7,7 +7,6 @@ sys.path.append('../util')
 import iwd
 from iwd import IWD
 from iwd import NetworkType
-from hostapd import hostapd_map
 from hostapd import HostapdCLI
 import testutil
 
diff --git a/autotests/testOWE/renegotiate_test.py b/autotests/testOWE/renegotiate_test.py
index 57705a59..7c736acd 100644
--- a/autotests/testOWE/renegotiate_test.py
+++ b/autotests/testOWE/renegotiate_test.py
@@ -7,7 +7,6 @@ sys.path.append('../util')
 import iwd
 from iwd import IWD
 from iwd import NetworkType
-from hostapd import hostapd_map
 from hostapd import HostapdCLI
 import testutil
 
diff --git a/autotests/testPreauth-roam/test.py b/autotests/testPreauth-roam/connection_test.py
similarity index 98%
rename from autotests/testPreauth-roam/test.py
rename to autotests/testPreauth-roam/connection_test.py
index 06babcea..ed043c3f 100644
--- a/autotests/testPreauth-roam/test.py
+++ b/autotests/testPreauth-roam/connection_test.py
@@ -9,7 +9,7 @@ from iwd import IWD
 from iwd import PSKAgent
 from iwd import NetworkType
 from hwsim import Hwsim
-from hostapd import HostapdCLI, hostapd_map
+from hostapd import HostapdCLI
 import testutil
 
 class Test(unittest.TestCase):
diff --git a/autotests/testRSSIAgent/test.py b/autotests/testRSSIAgent/connection_test.py
similarity index 100%
rename from autotests/testRSSIAgent/test.py
rename to autotests/testRSSIAgent/connection_test.py
diff --git a/autotests/testSAE/connection_test.py b/autotests/testSAE/connection_test.py
index 874eb833..56bb1f6d 100644
--- a/autotests/testSAE/connection_test.py
+++ b/autotests/testSAE/connection_test.py
@@ -8,17 +8,13 @@ import iwd
 from iwd import IWD
 from iwd import PSKAgent
 from iwd import NetworkType
-from hostapd import hostapd_map
+from hostapd import HostapdCLI
 import testutil
 
 class Test(unittest.TestCase):
 
     def validate_connection(self, wd):
-        hostapd_if = None
-
-        for hostapd in hostapd_map.values():
-            if hostapd.config == 'ssidSAE.conf':
-                hostapd_if = hostapd.name
+        hostapd = HostapdCLI(config='ssidSAE.conf')
 
         psk_agent = PSKAgent("secret123")
         wd.register_psk_agent(psk_agent)
@@ -56,7 +52,7 @@ class Test(unittest.TestCase):
         wd.wait(2)
 
         testutil.test_iface_operstate(intf=device.name)
-        testutil.test_ifaces_connected(if0=device.name, if1=hostapd_if)
+        testutil.test_ifaces_connected(if0=device.name, if1=hostapd.ifname)
 
         device.disconnect()
 
diff --git a/autotests/testSAE/group_20_connection_test.py b/autotests/testSAE/group_20_connection_test.py
index 033c8850..5631878a 100644
--- a/autotests/testSAE/group_20_connection_test.py
+++ b/autotests/testSAE/group_20_connection_test.py
@@ -8,17 +8,13 @@ import iwd
 from iwd import IWD
 from iwd import PSKAgent
 from iwd import NetworkType
-from hostapd import hostapd_map
+from hostapd import HostapdCLI
 import testutil
 
 class Test(unittest.TestCase):
 
     def validate_connection(self, wd):
-        hostapd_if = None
-
-        for hostapd in hostapd_map.values():
-            if hostapd.config == 'ssidSAE-20.conf':
-                hostapd_if = hostapd.name
+        hostapd = HostapdCLI(config='ssidSAE-20.conf')
 
         psk_agent = PSKAgent("secret123")
         wd.register_psk_agent(psk_agent)
@@ -56,7 +52,7 @@ class Test(unittest.TestCase):
         wd.wait(2)
 
         testutil.test_iface_operstate(intf=device.name)
-        testutil.test_ifaces_connected(if0=device.name, if1=hostapd_if)
+        testutil.test_ifaces_connected(if0=device.name, if1=hostapd.ifname)
 
         device.disconnect()
 
diff --git a/autotests/testSAQuery-spoofing/connection_test.py b/autotests/testSAQuery-spoofing/connection_test.py
index e9d31c15..4887b172 100644
--- a/autotests/testSAQuery-spoofing/connection_test.py
+++ b/autotests/testSAQuery-spoofing/connection_test.py
@@ -9,7 +9,7 @@ from iwd import IWD
 from iwd import PSKAgent
 from iwd import NetworkType
 from hwsim import Hwsim
-from hostapd import HostapdCLI, hostapd_map
+from hostapd import HostapdCLI
 
 from time import sleep
 
diff --git a/autotests/testSAQuery/connection_test.py b/autotests/testSAQuery/connection_test.py
index de0fe2cd..9ea95e2b 100644
--- a/autotests/testSAQuery/connection_test.py
+++ b/autotests/testSAQuery/connection_test.py
@@ -8,7 +8,7 @@ import iwd
 from iwd import IWD
 from iwd import PSKAgent
 from iwd import NetworkType
-from hostapd import HostapdCLI, hostapd_map
+from hostapd import HostapdCLI
 
 class Test(unittest.TestCase):
 
-- 
2.21.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [PATCH 03/16] auto-t: introduce pure python test-runner re-write
  2020-08-27 17:32 [PATCH 01/16] hwsim: return radio ID on create James Prestwood
  2020-08-27 17:32 ` [PATCH 02/16] auto-t: prepare autotests for test-runner re-write James Prestwood
@ 2020-08-27 17:32 ` James Prestwood
  2020-08-27 17:32 ` [PATCH 04/16] auto-t: hostapd.py: update to work with test-runner rewrite James Prestwood
                   ` (13 subsequent siblings)
  15 siblings, 0 replies; 18+ messages in thread
From: James Prestwood @ 2020-08-27 17:32 UTC (permalink / raw)
  To: iwd

[-- Attachment #1: Type: text/plain, Size: 39563 bytes --]

This patch completely re-writes test-runner in Python. This was done
because the existing C test-runner had some clunky work arounds and
maintaining or adding new features was starting to become a huge pain.

There were a few aspects of test-runner which continually had to
be dealt with when adding any new functionality:

 * Argument parsing: Adding new arguments to test-runner wasn't so
   bad, but if you wanted those arguments passed into the VM it
   became a huge pain. Arguments needed to be parsed, then re-formatted
   into the qemu command line, then re-parsed in a special order
   (backwards) once in the VM. The burden for adding new arguments was
   quite high so it was avoided (at least by me) at all costs.
 * The separation between C and Python: The tests are all written in
   python, but the executables, radios, and interfaces were all created
   from C. The way we solved this was by encoding the require info as
   environment variables, then parsing those from Python. It worked,
   but it was, again, a huge pain.
 * Process management: It started with all processes being launched
   from C, but eventually tests required the ability to start IWD, or
   kill hostapd ungracefully in order to test certain functionality.
   Since the processes were tracked in C, Python had no way of
   signalling that it killed a process and when it started one C had
   no idea. This was mitigated (basically by killall), but it was
   no where close to an elegant solution.

Re-writing test-runner in python solves all these problems and will
be much easier to maintain.

 * Argument parsing: Now all arguments are forwarded automatically
   to the VM. The ArgParse library takes care of parsing and each
   argument is stored in a dictionary.
 * Separation between C and Python: No more C, so no more separation.
 * Process management: Python will now manage all processes. This
   allows a test to kill, restart, or start a new process and not
   have to remember the PID or to kill it after the test.

There are a few more important aspects of the python implementation
that should now be considered when writing new tests:

 * The IWD constructor now has different default arugments. IWD
   will always be started unless specified and the configuration
   directory will always be /tmp
 * Any non *.py file in the test directory will be copied to /tmp.
   This avoids the need for 'tmpfs_extra_stuff' completely.
 * ctrl_interface will automatically be appended to every hostapd
   config. There is no need to include this in a config file from
   now on.
 * Test cleanup is extremely important. All tests get run in the
   same interpreter now and the tests themselves are actually loaded
   as python modules. This means e.g. if you somehow kept a reference
   to IWD() any subsequent tests would not start since IWD is still
   running.
 * For debugging, the test context can be printed which shows running
   processes, radios, and interfaces.

Three non-native python modules were used: PrettyTable, colored, and
pyroute2

$ pip3 install prettytable
$ pip3 install termcolor
$ pip3 install pyroute2
---
 autotests/util/config.py |    6 +
 tools/py_runner.py       | 1245 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 1251 insertions(+)
 create mode 100644 autotests/util/config.py
 create mode 100755 tools/py_runner.py

diff --git a/autotests/util/config.py b/autotests/util/config.py
new file mode 100644
index 00000000..be8edbfc
--- /dev/null
+++ b/autotests/util/config.py
@@ -0,0 +1,6 @@
+#
+# Acts as a global store for test information. 'ctx' is set by the creator of
+# of the TestContext (test-runner).
+#
+
+ctx = {}
diff --git a/tools/py_runner.py b/tools/py_runner.py
new file mode 100755
index 00000000..67863c2d
--- /dev/null
+++ b/tools/py_runner.py
@@ -0,0 +1,1245 @@
+#!/bin/python3
+
+import argparse
+import os
+import shutil
+import ctypes
+import fcntl
+import shlex
+import sys
+import subprocess
+import atexit
+import time
+import unittest
+import importlib
+import signal
+import pyroute2
+import multiprocessing
+import re
+
+from configparser import ConfigParser
+from prettytable import PrettyTable
+from termcolor import colored
+from glob import glob
+from collections import namedtuple
+from time import sleep
+
+libc = ctypes.cdll['libc.so.6']
+libc.mount.argtypes = (ctypes.c_char_p, ctypes.c_char_p, ctypes.c_char_p, \
+			ctypes.c_ulong, ctypes.c_char_p)
+
+# Using ctypes to load the libc library is somewhat low level. Because of this
+# we need to define our own flags/options for use with mounting.
+MS_NOSUID = 2
+MS_NODEV = 4
+MS_NOEXEC = 8
+MS_STRICTATIME = 1 << 24
+STDIN_FILENO = 0
+TIOCSTTY = 0x540E
+
+config = None
+intf_id = 0
+
+TEST_MAX_TIMEOUT = 45
+
+def dbg(*s):
+	'''
+		Allows prints if stdout has been re-directed
+	'''
+	print(*s, file=sys.__stdout__)
+
+def exit_vm():
+	if config:
+		for p in config.ctx.processes:
+			print("Process %s still running!" % p.name)
+
+	os.sync()
+
+	RB_AUTOBOOT = 0x01234567
+	#
+	# Calling 'reboot' or 'shutdown' from a shell (e.g. os.system('reboot'))
+	# is not the same the POSIX reboot() and will cause a kernel panic since
+	# we are the init process. The libc.reboot() allows the VM to exit
+	# gracefully.
+	#
+	libc.reboot(RB_AUTOBOOT)
+
+def path_exists(path):
+	'''
+		Searches PATH as well as absolute paths.
+	'''
+	if shutil.which(path):
+		return True
+	try:
+		os.stat(path)
+	except:
+		return False
+	return True
+
+def find_binary(list):
+	'''
+		Returns a binary from 'list' if its found in PATH or on a
+		valid absolute path.
+	'''
+	for path in list:
+		if path_exists(path):
+			return path
+	return None
+
+def mount(source, target, fs, flags, options=''):
+	'''
+		Python wrapper for libc mount()
+	'''
+	ret = libc.mount(source.encode(), target.encode(), fs.encode(), flags,
+				options.encode())
+	if ret < 0:
+		errno = ctypes.get_errno()
+		raise Exception("Could not mount %s (%d)" % (target, errno))
+
+MountInfo = namedtuple('MountInfo', 'fstype target options flags')
+
+mount_table = [
+	MountInfo('sysfs', '/sys', '', MS_NOSUID|MS_NOEXEC|MS_NODEV),
+	MountInfo('proc', '/proc', '', MS_NOSUID|MS_NOEXEC|MS_NODEV),
+	MountInfo('devpts', '/dev/pts', 'mode=0620', MS_NOSUID|MS_NOEXEC),
+	MountInfo('tmpfs', '/dev/shm', 'mode=1777', MS_NOSUID|MS_NODEV|MS_STRICTATIME),
+	MountInfo('tmpfs', '/run', 'mode=0755', MS_NOSUID|MS_NODEV|MS_STRICTATIME),
+	MountInfo('tmpfs', '/var/lib/iwd', 'mode=0755', 0),
+	MountInfo('tmpfs', '/tmp', '', 0),
+	MountInfo('tmpfs', '/usr/share/dbus-1', 'mode=0755', MS_NOSUID|MS_NOEXEC|MS_NODEV|MS_STRICTATIME),
+	MountInfo('debugfs', '/sys/kernel/debug', '', 0)
+]
+
+DevInfo = namedtuple('DevInfo', 'target linkpath')
+
+dev_table = [
+	DevInfo('/proc/self/fd', '/dev/fd'),
+	DevInfo('/proc/self/fd/0', '/dev/stdin'),
+	DevInfo('/proc/self/fd/1', '/dev/stdout'),
+	DevInfo('/proc/self/fd/2', '/dev/stderr')
+]
+
+dbus_config = '''
+<!DOCTYPE busconfig PUBLIC \
+"-//freedesktop//DTD D-Bus Bus Configuration 1.0//EN" \
+"http://www.freedesktop.org/standards/dbus/1.0/\
+busconfig.dtd\">
+<busconfig>
+<type>system</type>
+<listen>unix:path=/run/dbus/system_bus_socket</listen>
+<limit name=\"reply_timeout\">2147483647</limit>
+<auth>ANONYMOUS</auth>
+<allow_anonymous/>
+<policy context=\"default\">
+<allow user=\"*\"/>
+<allow own=\"*\"/>
+<allow send_type=\"method_call\"/>
+<allow send_type=\"signal\"/>
+<allow send_type=\"method_return\"/>
+<allow send_type=\"error\"/>
+<allow receive_type=\"method_call\"/>
+<allow receive_type=\"signal\"/>
+<allow receive_type=\"method_return\"/>
+<allow receive_type=\"error\"/>
+</policy>
+</busconfig>
+'''
+class Process:
+	'''
+		Start a process. If 'wait' is True the constructor will start
+		the process and wait for it to exit. No PID is tracked in this
+		case. If 'multi_test' is True this indicates the process is
+		run over the entire test run and will not be killed after each
+		test exits.
+	'''
+	def __init__(self, args, wait=False, multi_test=False, env=None, ctx=None):
+		self.args = args
+		self.wait = wait
+		self.name = args[0]
+		self.multi_test = multi_test
+		self.stdout = subprocess.PIPE
+		self.stderr = subprocess.PIPE
+		self.ret = None
+		self.ctx = ctx
+
+		if ctx:
+			set_stdout = False
+
+			if ctx.is_verbose(args[0]):
+				dbg("Verbose on for %s" % args[0])
+				set_stdout = True
+
+			if os.path.basename(args[0]) == ctx.args.gdb:
+				self.args = ['gdb', '--args']
+				self.args.extend(args)
+				set_stdout = True
+
+			# Anything labeled as multi_test isn't important to
+			# log. These are processes such as dbus-daemon and
+			# haveged.
+			if set_stdout:
+				if ctx.args.log:
+					test = os.path.basename(os.getcwd())
+					test_dir = '%s/%s' % (ctx.args.log, test)
+
+					if not path_exists(test_dir):
+						os.mkdir(test_dir)
+						os.chown(test_dir, int(ctx.args.log_uid), \
+								int(ctx.args.log_gid))
+
+					self.stdout = open('%s/%s' % (test_dir, args[0]), 'w')
+					self.stderr = open('%s/%s' % (test_dir, args[0]), 'w')
+				else:
+					self.stdout = sys.__stdout__
+					self.stderr = sys.__stderr__
+
+		if not wait:
+			self.pid = subprocess.Popen(self.args, stdout=self.stdout, \
+							stderr=self.stderr, env=env, \
+							cwd=os.getcwd())
+			print("Starting process {}".format(self.pid.args))
+		else:
+			self.ret = subprocess.call(self.args, stdout=self.stdout, \
+							stderr=self.stderr)
+			print("%s returned %d" % (args[0], self.ret))
+
+	def __del__(self):
+		print("Del process %s" % self.args)
+		if self.ctx and self.ctx.args.log:
+			self.stdout.close()
+			self.stderr.close()
+
+	def kill(self, force=False):
+		print("Killing process %s" % self.args)
+
+		if force:
+			os.kill(self.pid.pid, signal.SIGKILL)
+		else:
+			self.pid.kill()
+
+		self.pid.wait(timeout=15)
+
+	def wait_for_socket(self, socket, wait):
+		waited = 0
+		while not os.path.exists(socket):
+			sleep(0.5)
+			waited += 0.5
+			if waited > wait:
+				raise Exception("Timed out waiting for socket")
+
+class Interface:
+	def __init__(self, name, config):
+		self.name = name
+		self.ctrl_interface = '/var/run/hostapd/' + name
+		self.config = config
+
+	def __del__(self):
+		Process(['iw', 'dev', self.name, 'del'], True)
+
+	def set_interface_state(self, state):
+		Process(['ifconfig', self.name, state], True)
+
+class Radio:
+	def __init__(self, name):
+		self.name = name
+		# hostapd will reset this if this radio is used by it
+		self.use = 'iwd'
+		self.interface = None
+
+	def __del__(self):
+		print("Removing radio %s" % self.name)
+		self.interface = None
+
+	def create_interface(self, hapd):
+		global intf_id
+
+		ifname = 'wln%s' % intf_id
+
+		intf_id += 1
+
+		self.interface = Interface(ifname, hapd.config)
+		# IWD does not use interfaces in test-runner so any created
+		# interface is assumed to be used by hostapd.
+		self.use = 'hostapd'
+
+		Process(['iw', 'phy', self.name, 'interface', 'add', ifname,
+				'type', 'managed'], True)
+
+		return self.interface
+
+	def __str__(self):
+		ret = self.name + ':\n'
+		ret += '\tUsed By: %s ' % self.use
+		if self.interface:
+			ret += '(%s)' % self.interface.name
+
+		ret += '\n'
+
+		return ret
+
+class VirtualRadio(Radio):
+	'''
+		A subclass of 'Radio' specific to mac80211_hwsim radios.
+
+		TODO: Using D-Bus to create and destroy radios is more desireable
+		than the command line.
+	'''
+	def __init__(self, name, config=None):
+		super().__init__(name)
+
+		self.disable_cipher = None
+		self.disable_iftype = None
+
+		args = ['hwsim', '--create', '--name', self.name, '--nointerface']
+
+		if config:
+			self.disable_iftype = config.get('iftype_disable', False)
+			if self.disable_iftype:
+				args.append('--iftype-disable')
+				args.append(self.disable_iftype)
+
+			self.disable_cipher = config.get('cipher_disable', False)
+			if self.disable_cipher:
+				args.append('--cipher-disable')
+				args.append(self.disable_cipher)
+
+		p = Process(args, wait=True)
+
+		if p.ret < 0:
+			raise Exception('Failed to create radio (%d)' % p.ret)
+
+		self.id = p.ret
+
+	def __del__(self):
+		super().__del__()
+
+		Process(['hwsim', '--destroy=%s' % self.id])
+
+	def __str__(self):
+		ret = super().__str__()
+
+		if self.disable_iftype:
+			ret += '\tDisabled interface types: %s\n' % self.disable_iftype
+
+		if self.disable_cipher:
+			ret += '\tDisabled ciphers: %s\n' % self.disable_cipher
+
+		ret += '\n'
+
+		return ret
+
+class HostapdInstance:
+	'''
+		A single instance of hostapd. In reality all hostapd instances
+		are started as a single process. This class just makes things
+		convenient for communicating with one of the hostapd APs.
+	'''
+	def __init__(self, config, radio):
+		self.radio = radio
+		self.config = config
+
+		self.intf = radio.create_interface(self)
+		self.intf.set_interface_state('up')
+
+	def __del__(self):
+		print("Removing HostapdInstance %s" % self.config)
+		self.intf.set_interface_state('down')
+		self.radio = None
+		self.intf = None
+
+	def __str__(self):
+		ret = 'Hostapd (%s)\n' % self.intf.name
+		ret += '\tConfig: %s\n' % self.config
+
+		return ret
+
+class Hostapd:
+	'''
+		A set of running hostapd instances. This is really just a single
+		process since hostapd can be started with multiple config files.
+	'''
+	def __init__(self, ctx, radios, configs, radius):
+		if len(configs) != len(radios):
+			raise Exception("Config (%d) and radio (%d) list length not equal" % \
+						(len(configs), len(radios)))
+
+		print("Initializing hostapd instances")
+
+		self.global_ctrl_iface = '/var/run/hostapd/ctrl'
+
+		self.instances = [HostapdInstance(c, r) for c, r in zip(configs, radios)]
+
+		ifaces = [rad.interface.name for rad in radios]
+		ifaces = ','.join(ifaces)
+
+		args = ['hostapd', '-i', ifaces, '-g', self.global_ctrl_iface]
+
+		#
+		# Config files should already be present in /tmp. This appends
+		# ctrl_interface and does any variable replacement. Currently
+		# this is just any $ifaceN occurrences.
+		#
+		for c in configs:
+			full_path = '/tmp/%s' % c
+			args.append(full_path)
+
+			self._rewrite_config(full_path)
+
+		if radius:
+			args.append(radius)
+
+		if ctx.is_verbose('hostapd'):
+			args.append('-d')
+
+		self.process = ctx.start_process(args)
+
+		self.process.wait_for_socket(self.global_ctrl_iface, 10)
+
+	def _rewrite_config(self, config):
+		'''
+			Replaces any $ifaceN values with the correct interface
+			names as well as appends the ctrl_interface path to
+			the config file.
+		'''
+		with open(config, 'r+') as f:
+			data = f.read()
+			to_replace = []
+			for match in re.finditer(r'\$iface[0-9]+', data):
+				tag = data[match.start():match.end()]
+				idx = tag.split('iface')[1]
+
+				to_replace.append((tag, self.instances[int(idx)].intf.name))
+
+			for r in to_replace:
+				data = data.replace(r[0], r[1], 1)
+
+			data += '\nctrl_interface=/var/run/hostapd\n'
+
+			f.write(data)
+
+	def __getitem__(self, config):
+		if not config:
+			return self.instances[0]
+
+		for hapd in self.instances:
+			if hapd.config == config:
+				return hapd
+
+		return None
+
+	def __del__(self):
+		print("Removing Hostapd")
+		try:
+			os.remove(self.global_ctrl_iface)
+		except:
+			dbg("Failed to remove %s" % self.global_ctrl_iface)
+
+		self.instances = None
+		self.process.kill()
+
+class TestContext:
+	'''
+		Contains all information for a given set of tests being run
+		such as processes, radios, interfaces and test results.
+	'''
+	def __init__(self, args):
+		self.processes = []
+		self.args = args
+		self.hw_config = None
+		self.hostapd = None
+		self.cur_radio_id = 0
+		self.cur_iface_id = 0
+		self.radios = []
+		self.loopback_started = False
+		self.iwd_extra_options = None
+		self.results = {}
+
+	def start_process(self, args, wait=False, multi_test=False, env=None):
+		p = Process(args, wait, multi_test, env, ctx=self)
+
+		if not wait:
+			self.processes.append(p)
+
+		return p
+
+	def start_dbus(self):
+		with open('/usr/share/dbus-1/system.conf', 'w+') as f:
+			f.write(dbus_config)
+
+		os.mkdir('/run/dbus', 755)
+
+		self.start_process(['dbus-daemon', '--system', '--nosyslog'], multi_test=True)
+
+	def start_dbus_monitor(self):
+		if not self.is_verbose('dbus-monitor'):
+			return
+
+		self.start_process(['dbus-monitor', '--system'])
+
+	def start_haveged(self):
+		self.start_process(['haveged'], multi_test=True)
+
+	def create_radios(self):
+		setup = self.hw_config['SETUP']
+		nradios = int(setup['num_radios'])
+
+		for i in range(nradios):
+			name = 'rad%u' % i
+
+			# Get any [radX] sections. These are for configuring
+			# any special radios. This no longer requires a
+			# radio_conf list, we just assume radios start rad0
+			# and increment.
+			rad_config = None
+			if self.hw_config.has_section(name):
+				rad_config = self.hw_config[name]
+
+			self.radios.append(VirtualRadio(name, rad_config))
+			self.cur_radio_id += 1
+
+		# register hwsim as medium
+		self.start_process(['hwsim'])
+
+	def discover_radios(self):
+		phys = []
+		iw = pyroute2.iwutil.IW()
+
+		attrs = [phy['attrs'] for phy in iw.list_wiphy()]
+
+		for attr in attrs:
+			for key, value in attr:
+				if key == 'NL80211_ATTR_WIPHY_NAME':
+					if value not in phys:
+						phys.append(value)
+					break
+
+		print('Discovered radios: %s' % str(phys))
+		self.radios = [Radio(name) for name in phys]
+
+	def start_radios(self):
+		reg_domain = self.hw_config['SETUP'].get('reg_domain', None)
+		if reg_domain:
+			Process(['iw', 'reg', 'set', reg_domain], True)
+
+		if self.args.hw:
+			self.discover_radios()
+		else:
+			self.create_radios()
+
+	def start_iwd(self, config_dir = '/tmp'):
+		args = []
+		iwd_radios = ','.join([r.name for r in self.radios if r.use == 'iwd'])
+
+		if self.args.valgrind:
+			args.extend(['valgrind', '--leak-check=full', '--log-file=%s' % \
+					'/tmp/valgrind.log'])
+
+		args.extend(['iwd', '-p', iwd_radios])
+
+		if self.is_verbose(args[0]):
+			args.append('-d')
+
+		if self.iwd_extra_options:
+			args.append(self.iwd_extra_options)
+
+		env = os.environ.copy()
+		env['CONFIGURATION_DIRECTORY'] = config_dir
+		env['STATE_DIRECTORY'] = '/var/lib/iwd'
+
+		pid = self.start_process(args, env=env)
+		return pid
+
+	def start_hostapd(self):
+		if not 'HOSTAPD' in self.hw_config:
+			return
+
+		settings = self.hw_config['HOSTAPD']
+
+		if self.args.hw:
+			# Just grab the first N radios. It gets rather
+			# complicated trying to map radX radios specified in
+			# hw.conf so any passed through physical adapters are
+			# just given to hostapd/IWD as they appear during
+			# discovery.
+			#
+			# TODO: It may be desireable to map PCI/USB adapters to
+			#       specific radX radios specified in the config but
+			#       there are really 2 separate use cases here.
+			#       1. You want to test a *specific* radio with IWD
+			#          or hostapd. For this you would want radX
+			#          to map to a specific radio
+			#       2. You have many adapters in use to run multiple
+			#          tests. In this case you would not care what
+			#          was using each radio, just that there was
+			#          enough to run all tests.
+			nradios = 0
+			for k, _ in settings.items():
+				if k == 'radius_server':
+					continue
+				nradios += 1
+
+			hapd_radios = self.radios[:nradios]
+
+		else:
+			hapd_radios = [rad for rad in self.radios if rad.name in settings]
+
+		hapd_configs = [conf for rad, conf in settings.items() if rad != 'radius_server']
+
+		radius_config = settings.get('radius_server', None)
+
+		self.hostapd = Hostapd(self, hapd_radios, hapd_configs, radius_config)
+
+	def start_ofono(self):
+		sim_keys = self.hw_config['SETUP'].get('sim_keys', None)
+		if not sim_keys:
+			print("Ofono not requred")
+			return
+		elif sim_keys != 'ofono':
+			os.environ['IWD_SIM_KEYS'] = sim_keys
+			self.iwd_extra_options = '--plugin=sim_hardcoded'
+			return
+
+		if not find_binary(['ofonod']) or not find_binary(['phonesim']):
+			print("Ofono or Phonesim not found, skipping test")
+			return
+
+		Process(['ifconfig', 'lo', 'up'], wait=True)
+
+		self.iwd_extra_options = '--plugin=ofono'
+
+		os.environ['OFONO_PHONESIM_CONFIG'] = '/tmp/phonesim.conf'
+
+		phonesim_args = ['phonesim', '-p', '12345', '/usr/share/phonesim/default.xml']
+
+		self.start_process(phonesim_args)
+
+		#
+		# TODO:
+		# Is there something to wait for? Without this phonesim rejects
+		# connections on all but the fist test.
+		#
+		time.sleep(3)
+
+		ofono_args = ['ofonod', '-n', '--plugin=atmodem,phonesim']
+		if self.is_verbose('ofonod'):
+			ofono_args.append('-d')
+
+		self.start_process(ofono_args)
+
+		print("Ofono started")
+
+	def is_verbose(self, process):
+		process = os.path.basename(process)
+
+		if self.args is None:
+			return False
+
+		# every process is verbose when logging is enabled
+		if self.args.log:
+			return True
+
+		if process in self.args.verbose:
+			return True
+
+		# Special case here to enable verbose output with valgrind running
+		if process == 'valgrind' and 'iwd' in self.args.verbose:
+			return True
+
+		# Handle any glob matches
+		for item in self.args.verbose:
+			if process in glob(item):
+				return True
+
+		return False
+
+	def stop_process(self, p, force=False):
+		p.kill(force)
+		self.processes.remove(p)
+
+	def stop_test_processes(self):
+		self.radios = []
+		self.hostapd = None
+		self.iwd_extra_options = None
+
+		for p in [p for p in self.processes if p.multi_test is False]:
+			print("Killing process %s" % p.name)
+			self.stop_process(p)
+
+	def is_process_running(self, process):
+		for p in self.processes:
+			if p.name == process:
+				return True
+		return False
+
+	def __str__(self):
+		ret = 'Arguments:\n'
+		for arg in vars(self.args):
+			ret += '\t --%s %s\n' % (arg, str(getattr(self.args, arg)))
+
+		ret += 'Processes:\n'
+		for p in self.processes:
+			ret += '\t%s\n' % str(p.args)
+
+		ret += 'Radios:\n'
+		if len(self.radios) > 0:
+			for r in self.radios:
+				ret += '\t%s\n' % str(r)
+		else:
+			ret += '\tNo Radios\n'
+
+		ret += 'Hostapd:\n'
+		if self.hostapd:
+			for h in self.hostapd.instances:
+				ret += '\t%s\n' % str(h)
+		else:
+			ret += '\tNo Hostapd instances\n'
+
+		return ret
+
+def prepare_sandbox():
+	print('Preparing sandbox')
+
+	for entry in mount_table:
+		try:
+			os.lstat(entry.target)
+		except:
+			os.mkdir(entry.target, 755)
+
+		mount(entry.fstype, entry.target, entry.fstype, entry.flags,
+			entry.options)
+
+	for entry in dev_table:
+		os.symlink(entry.target, entry.linkpath)
+
+	os.setsid()
+
+	fcntl.ioctl(STDIN_FILENO, TIOCSTTY, 1)
+
+def build_unit_list(args):
+	'''
+		Build list of unit tests based on passed arguments. This first
+		checks for literal names provided in the arguments, then if
+		no matches were found, checks for a glob match.
+	'''
+	tests = []
+	test_root = args.testhome + '/unit'
+
+	for unit in args.unit_tests.split(','):
+		path = '%s/%s' % (test_root, unit)
+		if os.access(unit, os.X_OK):
+			tests.append(unit)
+		elif os.access(path, os.X_OK):
+			tests.append(path)
+		else:
+			# Full list or glob, first build up valid list of tests
+			matches = glob(path)
+			if matches == []:
+				raise Exception("Could not find test %s" % unit)
+
+			matches = [exe for exe in matches if os.access(exe, os.X_OK)]
+
+			tests.extend(matches)
+
+	return sorted(tests)
+
+def build_test_list(args):
+	'''
+		Build list of auto test directories based on passed arguments.
+		First check for absolute paths, then look in <iwd>/autotests,
+		then glob match.
+	'''
+	tests = []
+	test_root = args.testhome + '/autotests'
+
+	# Run all tests
+	if not args.auto_tests:
+		# --shell with no tests implies 'shell' test
+		if args.shell:
+			return [test_root + '/shell']
+
+		tests = os.listdir(test_root)
+		# Pair down any non-tests and append full path
+		tests = [test_root + '/' + t for t in tests if t.startswith('test')]
+	else:
+		print("Generating partial test list")
+		for t in args.auto_tests.split(','):
+			path = '%s/%s' % (test_root, t)
+			# Full test path specified
+			if os.path.exists(t):
+				tests.append(t)
+			elif os.path.exists(path):
+				tests.append(path)
+			else:
+				matches = glob(path)
+				if matches == []:
+					raise Exception("Could not find test %s" % t)
+
+				tests.extend(matches)
+
+	return sorted(tests)
+
+SimpleResult = namedtuple('SimpleResult', 'run failures errors skipped time')
+
+def start_test(ctx, subtests, rqueue):
+	'''
+		Run an individual test. 'subtests' are parsed prior to calling
+		but these effectively make up a single test. 'rqueue' is the
+		results queue which is required since this is using
+		multiprocessing.
+	'''
+	suite = unittest.TestSuite()
+
+	#
+	# Iterate through each individual python test.
+	#
+	for s in subtests:
+		loader = unittest.TestLoader()
+		subtest = importlib.import_module(os.path.splitext(s)[0])
+		suite.addTests(loader.loadTestsFromModule(subtest))
+
+		# Prevents future test modules with the same name (e.g.
+		# connection_test.py) from being loaded from the cache
+		sys.modules.pop(subtest.__name__)
+
+	start = time.time()
+	runner = unittest.TextTestRunner()
+	result = runner.run(suite)
+	#
+	# The multiprocessing queue is picky with what objects it will serialize
+	# and send between processes. Because of this we put the important bits
+	# of the result into our own 'SimpleResult' tuple.
+	#
+	sresult = SimpleResult(run=result.testsRun, failures=len(result.failures),
+				errors=len(result.errors), skipped=len(result.skipped),
+				time=time.time() - start)
+	rqueue.put(sresult)
+
+	# This may not be required since we are manually popping sys.modules
+	importlib.invalidate_caches()
+
+def pre_test(ctx, test):
+	'''
+		Copy test files, start processes, and any other pre test work.
+	'''
+	os.chdir(test)
+
+	dbg("Starting test %s" % test)
+	if not os.path.exists(test + '/hw.conf'):
+		print("No hw.conf found for %s" % test)
+		exit()
+
+	ctx.hw_config = ConfigParser()
+	ctx.hw_config.read(test + '/hw.conf')
+	#
+	# We have two types of test files: tests and everything else. Rather
+	# than require each test to specify the files needing to be copied to
+	# /tmp (previously 'tmpfs_extra_stuff'), we just copy everything which
+	# isn't a test. There is really no reason not to do this as any file
+	# present in a test directory should be needed by the test.
+	#
+	# All files
+	files = os.listdir(test)
+	# Tests (starts or ends with 'test')
+	subtests = [f for f in files if f.startswith('test') or \
+			os.path.splitext(f)[0].endswith('test')]
+	# Everything else (except .py files)
+	to_copy = [f for f in list(set(files) - set(subtests)) if not f.endswith('.py')]
+	for f in to_copy:
+		if os.path.isdir(f):
+			shutil.copytree(f, '/tmp/' + f)
+		else:
+			shutil.copy(f, '/tmp')
+
+	ctx.start_dbus_monitor()
+	ctx.start_radios()
+	ctx.start_hostapd()
+	ctx.start_ofono()
+
+	print(ctx)
+
+	sys.path.insert(1, test)
+
+	return (to_copy, subtests)
+
+def post_test(ctx, to_copy):
+	'''
+		Remove copied files, and stop test processes.
+	'''
+	for f in to_copy:
+		if os.path.isdir('/tmp/' + f):
+			shutil.rmtree('/tmp/' + f)
+		else:
+			os.remove('/tmp/' + f)
+
+	Process(['ifconfig', 'lo', 'down'], wait=True)
+
+	ctx.stop_test_processes()
+	if ctx.args.valgrind:
+		with open('/tmp/valgrind.log', 'r') as f:
+				dbg(f.read())
+		dbg("\n")
+
+def print_results(results):
+	table = PrettyTable(['Test', colored('Passed', 'green'), colored('Failed', 'red'), \
+				colored('Skipped', 'cyan'), colored('Time', 'yellow')])
+
+	total_pass = 0
+	total_fail = 0
+	total_skip = 0
+	total_time = 0
+
+	for test, result in results.items():
+		if result.time != TEST_MAX_TIMEOUT:
+			failed = result.failures + result.errors
+			passed = result.run - failed
+
+			total_pass += passed
+			total_fail += failed
+			total_skip += result.skipped
+		else:
+			failed = "Timed out"
+			passed = "Timed out"
+
+		total_time += result.time
+
+		time = '%.2f' % result.time
+
+		table.add_row([test, colored(passed, 'green'), colored(failed, 'red'), \
+				colored(result.skipped, 'cyan'), colored(time, 'yellow')])
+
+	total_time = '%.2f' % total_time
+
+	table.add_row(['Total', colored(total_pass, 'green'), colored(total_fail, 'red'), \
+			colored(total_skip, 'cyan'), colored(total_time, 'yellow')])
+
+	dbg(table)
+
+def run_auto_tests(ctx, args):
+	tests = build_test_list(args)
+
+	ctx.start_dbus()
+	ctx.start_haveged()
+
+	# Copy autotests/misc/{certs,secrets,phonesim} so any test can refer to them
+	shutil.copytree(args.testhome + '/autotests/misc/certs', '/tmp/certs')
+	shutil.copytree(args.testhome + '/autotests/misc/secrets', '/tmp/secrets')
+	shutil.copy(args.testhome + '/autotests/misc/phonesim/phonesim.conf', '/tmp')
+
+	if args.shell:
+		#
+		# Shell really isn't meant to be used with multiple tests. If
+		# a set of tests was passed in just start out in the first.
+		#
+		os.chdir(tests[0])
+		os.system('/bin/bash')
+		exit()
+
+	for test in tests:
+		copied, subtests = pre_test(ctx, test)
+
+		rqueue = multiprocessing.Queue()
+		p = multiprocessing.Process(target=start_test, args=(ctx, subtests, rqueue))
+		p.start()
+		# Rather than time each subtest we just time the total but
+		# mutiply the default time by the number of tests being run.
+		p.join(TEST_MAX_TIMEOUT * len(subtests))
+
+		if p.is_alive():
+			# Timeout
+			p.terminate()
+
+			ctx.results[os.path.basename(test)] = SimpleResult(run=0,
+								failures=0, errors=0,
+								skipped=0, time=TEST_MAX_TIMEOUT)
+		else:
+			ctx.results[os.path.basename(test)] = rqueue.get()
+
+		post_test(ctx, copied)
+
+	# Write out kernel log
+	if ctx.args.log:
+		Process(["dmesg"], ctx=ctx, wait=True)
+
+	print_results(ctx.results)
+
+def run_unit_tests(ctx, args):
+	os.chdir(args.testhome + '/unit')
+	units = build_unit_list(args)
+
+	for u in units:
+		if ctx.start_process([u], wait=True).ret != 0:
+			dbg("Unit test %s failed" % os.path.basename(u))
+		else:
+			dbg("Unit test %s passed" % os.path.basename(u))
+
+def run_tests():
+	global config
+
+	with open('/proc/cmdline', 'r') as f:
+		cmdline = f.read()
+
+	start = cmdline.find('--testhome')
+
+	options = shlex.split(cmdline[start:])
+
+	parser = argparse.ArgumentParser()
+	parser.add_argument('--testhome')
+	parser.add_argument('--auto_tests')
+	parser.add_argument('--unit_tests')
+	parser.add_argument('--verbose', default=[])
+	parser.add_argument('--debug')
+	parser.add_argument('--path')
+	parser.add_argument('--valgrind')
+	parser.add_argument('--gdb')
+	parser.add_argument('--shell')
+	parser.add_argument('--log')
+	parser.add_argument('--log-gid')
+	parser.add_argument('--log-uid')
+	parser.add_argument('--hw')
+
+	args = parser.parse_args(options)
+
+	#
+	# This prevents any print() calls in this script from printing unless
+	# --debug is passed. For an 'always print' option use dbg()
+	#
+	if not args.debug:
+		sys.stdout = open(os.devnull, 'w')
+
+	os.environ['PATH'] = args.path
+	os.environ['PATH'] += ':' + args.testhome + '/src'
+
+	sys.path.append(args.testhome + '/autotests/util')
+
+	#
+	# This allows all autotest utils (iwd/hostapd/etc) to access the
+	# TestContext. Any other module or script (in the same interpreter) can
+	# simply import config.ctx and access all live test information,
+	# start/stop processes, see active radios etc.
+	#
+	config = importlib.import_module('config')
+	config.ctx = TestContext(args)
+
+	if args.log:
+		mount('logdir', args.log, '9p', 0, 'trans=virtio,version=9p2000.L')
+
+	if config.ctx.args.unit_tests is None:
+		run_auto_tests(config.ctx, args)
+	else:
+		run_unit_tests(config.ctx, args)
+
+class Main:
+	def __init__(self):
+		self.parser = argparse.ArgumentParser(
+				description='IWD Test Runner')
+
+		self.parser.add_argument('--qemu', '-q',
+				metavar='<QEMU binary>', type=str,
+				help='QEMU binary to use',
+				dest='qemu')
+		self.parser.add_argument('--kernel', '-k', metavar='<kernel>',
+				type=str,
+				help='Path to kernel image',
+				dest='kernel')
+		self.parser.add_argument('--verbose', '-v', metavar='<list>',
+				type=str,
+				help='Comma separated list of applications',
+				dest='verbose',
+				default=[])
+		self.parser.add_argument('--debug', '-d',
+				action='store_true',
+				help='Enable test-runner debugging',
+				dest='debug')
+		self.parser.add_argument('--shell', '-s', action='store_true',
+				help='Boot into shell', dest='shell')
+		self.parser.add_argument('--log', '-l', type=str,
+				help='Directory for log files')
+		self.parser.add_argument('--hw', '-w', type=str, nargs=1,
+				help='Use physical adapters for tests (passthrough)')
+
+		# Prevent --autotest/--unittest from being used together
+		auto_unit_group = self.parser.add_mutually_exclusive_group()
+		auto_unit_group.add_argument('--auto-tests', '-A',
+				metavar='<tests>', type=str, nargs=1,
+				help='List of tests to run',
+				default=None,
+				dest='auto_tests')
+		auto_unit_group.add_argument('--unit-tests', '-U',
+				metavar='<tests>', type=str, nargs='?',
+				const='*',
+				help='List of unit tests to run',
+				dest='unit_tests')
+
+		# Prevent --valgrind/--gdb from being used together
+		valgrind_gdb_group = self.parser.add_mutually_exclusive_group()
+		valgrind_gdb_group.add_argument('--gdb', '-g', metavar='<exec>',
+				type=str, nargs=1,
+				help='Run gdb on specified executable',
+				dest='gdb')
+		valgrind_gdb_group.add_argument('--valgrind', '-V', action='store_true',
+				help='Run valgrind on IWD', dest='valgrind')
+
+		self.args = self.parser.parse_args()
+
+		print(self.args)
+
+		if self.args.log and self.args.unit_tests:
+			dbg("Cannot use --log with --unit-tests")
+			quit()
+
+	def start(self):
+		usb_adapters = None
+
+		qemu_table = [
+			'qemu-system-x86_64',
+			'/usr/bin/qemu-system-x86_64'
+		]
+
+		kernel_table = [
+			'bzImage',
+			'arch/x86/boot/bzImage',
+			'vmlinux',
+			'arch/x86/boot/vmlinux'
+		]
+
+		if self.args.qemu is None:
+			qemu_binary = find_binary(qemu_table)
+		else:
+			if path_exists(self.args.qemu):
+				qemu_binary = self.args.qemu
+			else:
+				print("QEMU binary %s does not exist" % \
+						self.args.qemu)
+				quit()
+
+		if self.args.kernel is None:
+			kernel_binary = find_binary(kernel_table)
+		else:
+			if path_exists(self.args.kernel):
+				kernel_binary = self.args.kernel
+			else:
+				print("Kernel image %s does not exist" % \
+						self.args.kernel)
+				quit()
+
+		if self.args.hw:
+			hw_conf = ConfigParser()
+			hw_conf.read(self.args.hw)
+			# TODO: Parse PCI adapters
+			if hw_conf.has_section('USBAdapters'):
+				# The actual key name of the adapter
+				# doesn't matter since all we need is the
+				# bus/address. This gets named by the kernel
+				# anyways once in the VM.
+				usb_adapters = [v for v in hw_conf['USBAdapters'].values()]
+
+		#
+		# Additional arguments not provided to test-runner which are
+		# needed once booted into the kernel.
+		#
+		options = 'init=%s' % os.path.realpath(sys.argv[0])
+
+		# Support running from top level as well as tools
+		if os.getcwd().endswith('tools'):
+			options += ' --testhome %s/../' % os.getcwd()
+		else:
+			options += ' --testhome %s' % os.getcwd()
+
+		options += ' --path "%s"' % os.environ['PATH']
+
+		if self.args.auto_tests:
+			options += ' --auto_tests %s' % ','.join(self.args.auto_tests)
+
+		if self.args.log:
+			if os.environ.get('SUDO_GID', None) is None:
+				print("--log can only be used as root user")
+				quit()
+
+			self.args.log = os.path.abspath(self.args.log)
+			uid = int(os.environ['SUDO_UID'])
+			gid = int(os.environ['SUDO_GID'])
+
+			if not path_exists(self.args.log):
+				os.mkdir(self.args.log)
+				os.chown(self.args.log, uid, gid)
+
+			options += ' --log-gid %u' % gid
+			options += ' --log-uid %u' % uid
+
+		denylist = [
+			'auto_tests',
+			'qemu',
+			'kernel'
+		]
+
+		#
+		# This passes through most of the command line options to
+		# the kernel command line. Some are not relevant (e.g. qemu)
+		# so similar options are added in the denylist above. This excludes
+		# any unset options which are assumed to be None or False. This
+		# is done so default arguments can be filled once in the VM. If
+		# we pass and basic types (None, False etc.) they are turned into
+		# a string representation ('None', 'False', etc.) which is not
+		# desirable.
+		#
+		for arg in vars(self.args):
+			if arg in denylist or getattr(self.args, arg) in [None, False, []]:
+				continue
+			options += ' --%s %s' % (arg, str(getattr(self.args, arg)))
+
+		kern_log = "ignore_loglevel" if "kernel" in self.args.verbose else "quiet"
+
+		qemu_cmdline = [
+			qemu_binary,
+			'-machine', 'type=q35,accel=kvm:tcg',
+			'-nodefaults', '-no-user-config', '-monitor', 'none',
+			'-display', 'none', '-m', '192M', '-nographic', '-vga',
+			'none', '-net', 'none', '-no-acpi', '-no-hpet',
+			'-no-reboot', '-fsdev',
+			'local,id=fsdev-root,path=/,readonly,security_model=none',
+			'-device',
+			'virtio-9p-pci,fsdev=fsdev-root,mount_tag=/dev/root',
+			'-chardev', 'stdio,id=chardev-serial0,signal=off',
+			'-device', 'pci-serial,chardev=chardev-serial0',
+			'-device', 'virtio-rng-pci',
+			'-kernel',
+			kernel_binary,
+			'-append',
+			'console=ttyS0,115200n8 earlyprintk=serial \
+				rootfstype=9p root=/dev/root \
+				rootflags=trans=virtio,version=9p2000.u \
+				acpi=off pci=noacpi %s ro \
+				mac80211_hwsim.radios=0 %s' % (kern_log, options),
+			'-cpu', 'host'
+		]
+
+		if usb_adapters:
+			for bus, addr in [s.split(',') for s in usb_adapters]:
+				qemu_cmdline.extend(['-usb',
+							'-device',
+							'usb-host,hostbus=%s,hostaddr=%s' % \
+							(bus, addr)])
+		if self.args.log:
+			#
+			# Creates a virtfs device that can be mounted. This mount
+			# will point back to the provided log directory and is
+			# writable unlike the rest of the mounted file system.
+			#
+			qemu_cmdline.extend([
+				'-virtfs',
+				'local,path=%s,mount_tag=logdir,security_model=passthrough,id=logdir' \
+						% self.args.log
+			])
+
+		os.execlp(qemu_cmdline[0], *qemu_cmdline)
+
+if __name__ == '__main__':
+	if os.getpid() == 1 and os.getppid() == 0:
+		atexit.register(exit_vm)
+		prepare_sandbox()
+		run_tests()
+
+		exit()
+
+	main = Main()
+	main.start()
-- 
2.21.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [PATCH 04/16] auto-t: hostapd.py: update to work with test-runner rewrite
  2020-08-27 17:32 [PATCH 01/16] hwsim: return radio ID on create James Prestwood
  2020-08-27 17:32 ` [PATCH 02/16] auto-t: prepare autotests for test-runner re-write James Prestwood
  2020-08-27 17:32 ` [PATCH 03/16] auto-t: introduce pure python " James Prestwood
@ 2020-08-27 17:32 ` James Prestwood
  2020-08-27 17:32 ` [PATCH 05/16] auto-t: testutil.py: " James Prestwood
                   ` (12 subsequent siblings)
  15 siblings, 0 replies; 18+ messages in thread
From: James Prestwood @ 2020-08-27 17:32 UTC (permalink / raw)
  To: iwd

[-- Attachment #1: Type: text/plain, Size: 5683 bytes --]

Before hostapd was initialized using the wiphy_map which has now
gone away. Instead we have a global config module which contains
a single 'ctx'. This is the centeral store for all test information.

This patch converts hostapd.py to lookup instances by already
initialized Hostapd object. The interface parameter was removed
since all tests have been converted to use config= instead.

In addition HostapdCLI was changed to allow no parameters if there
is only a single hostapd instance.
---
 autotests/util/hostapd.py | 57 +++++++++++++++++----------------------
 1 file changed, 24 insertions(+), 33 deletions(-)

diff --git a/autotests/util/hostapd.py b/autotests/util/hostapd.py
index 0ed4e004..b8cfe231 100644
--- a/autotests/util/hostapd.py
+++ b/autotests/util/hostapd.py
@@ -1,11 +1,11 @@
 #!/usr/bin/python3
 import os, os.path
-from wiphy import wiphy_map
 import re
 import socket
 import select
 import time
 from gi.repository import GLib
+from config import ctx
 
 chan_freq_map = [
     None,
@@ -28,28 +28,24 @@ chan_freq_map = [
 ctrl_count = 0
 mainloop = GLib.MainLoop()
 
-hostapd_map = {ifname: intf for wname, wiphy in wiphy_map.items()
-        for ifname, intf in wiphy.interface_map.items()
-        if wiphy.use == 'hostapd'}
-
 class HostapdCLI:
-    def _init_hostapd(self, interface=None, config=None):
+    def _init_hostapd(self, config=None):
         global ctrl_count
+        interface = None
+
+        if not config and len(ctx.hostapd.instances) > 1:
+            raise Exception('config must be provided if more than one hostapd instance exists')
 
-        if not interface and not config:
-            raise Exception('interface or config must be provided')
+        hapd = ctx.hostapd[config]
 
-        if not interface:
-            for intf in hostapd_map.values():
-                if intf.config == config:
-                    interface = intf
-                    break
+        self.interface = hapd.intf
+        self.config = hapd.config
 
-        if not interface:
+        if not self.interface:
             raise Exception('config %s not found' % config)
 
-        self.ifname = interface.name
-        self.socket_path = os.path.dirname(interface.ctrl_interface)
+        self.ifname = self.interface.name
+        self.socket_path = os.path.dirname(self.interface.ctrl_interface)
 
         self.cmdline = 'hostapd_cli -p"' + self.socket_path + '" -i"' + \
                 self.ifname + '"'
@@ -61,6 +57,7 @@ class HostapdCLI:
                             str(ctrl_count)
         self.ctrl_sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
         self.ctrl_sock.bind(self.local_ctrl)
+
         self.ctrl_sock.connect(self.socket_path + '/' + self.ifname)
 
         if 'OK' not in self._ctrl_request('ATTACH'):
@@ -68,8 +65,8 @@ class HostapdCLI:
 
         ctrl_count = ctrl_count + 1
 
-    def __init__(self, interface=None, config=None):
-        self._init_hostapd(interface, config)
+    def __init__(self, config=None):
+        self._init_hostapd(config)
 
     def wait_for_event(self, event, timeout=10):
         global mainloop
@@ -115,15 +112,13 @@ class HostapdCLI:
 
     def _del_hostapd(self, force=False):
         self.ctrl_sock.close()
+        os.remove(self.local_ctrl)
 
         if self._hostapd_restarted:
-            if force:
-                os.system('killall -9 hostapd')
-            else:
-                os.system('killall hostapd')
+            ctx.stop_process(ctx.hostapd.process, force)
 
-            os.system('ifconfig %s down' % self.ifname)
-            os.system('ifconfig %s up' % self.ifname)
+            self.interface.set_interface_state('down')
+            self.interface.set_interface_state('up')
 
     def __del__(self):
         self._del_hostapd()
@@ -139,10 +134,7 @@ class HostapdCLI:
 
     def eapol_reauth(self, client_address):
         cmd = 'IFNAME=' + self.ifname + ' EAPOL_REAUTH ' + client_address
-        s = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
-        s.connect(self.socket_path + '/' + self.ifname)
-        s.sendall(cmd.encode('utf-8'))
-        s.close()
+        self.ctrl_sock.sendall(cmd.encode('utf-8'))
 
     def reload(self):
         # Seemingly all three commands needed for the instance to notice
@@ -192,7 +184,7 @@ class HostapdCLI:
 
     def get_config_value(self, key):
         # first find the right config file
-        with open(hostapd_map[self.ifname].config, 'r') as f:
+        with open(self.config, 'r') as f:
             # read in config file and search for key
             cfg = f.read()
             match = re.search(r'%s=.*' % key, cfg)
@@ -200,6 +192,7 @@ class HostapdCLI:
                 return match.group(0).split('=')[1]
         return None
 
+
     def get_freq(self):
         return chan_freq_map[int(self.get_config_value('channel'))]
 
@@ -209,18 +202,16 @@ class HostapdCLI:
         '''
         # set flag so hostapd can be killed after the test
         self._hostapd_restarted = True
-        intf = hostapd_map[self.ifname]
 
         self._del_hostapd(force=True)
 
-        os.system('hostapd -g %s -i %s %s &' %
-                  (intf.ctrl_interface, intf.name, intf.config))
+        ctx.start_hostapd()
 
         # Give hostapd a second to start and initialize the control interface
         time.sleep(1)
 
         # New hostapd process, so re-init
-        self._init_hostapd(intf)
+        self._init_hostapd(config=self.config)
 
     def req_beacon(self, addr, request):
         '''
-- 
2.21.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [PATCH 05/16] auto-t: testutil.py: update to work with test-runner rewrite
  2020-08-27 17:32 [PATCH 01/16] hwsim: return radio ID on create James Prestwood
                   ` (2 preceding siblings ...)
  2020-08-27 17:32 ` [PATCH 04/16] auto-t: hostapd.py: update to work with test-runner rewrite James Prestwood
@ 2020-08-27 17:32 ` James Prestwood
  2020-08-27 17:32 ` [PATCH 06/16] auto-t: ofono.py: fix timeout cleanup and wait for service James Prestwood
                   ` (11 subsequent siblings)
  15 siblings, 0 replies; 18+ messages in thread
From: James Prestwood @ 2020-08-27 17:32 UTC (permalink / raw)
  To: iwd

[-- Attachment #1: Type: text/plain, Size: 1142 bytes --]

Similar to the others, testutil.py was converted to use the global
test context to lookup radios/interfaces rather than the wiphy_map
---
 autotests/util/testutil.py | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/autotests/util/testutil.py b/autotests/util/testutil.py
index aaa4e7e1..0a94e23a 100644
--- a/autotests/util/testutil.py
+++ b/autotests/util/testutil.py
@@ -5,8 +5,8 @@ import fcntl
 import struct
 import select
 
-import wiphy
 import iwd
+from config import ctx
 
 HWSIM_ETHERTYPE = 0x0800
 HWSIM_PACKETLEN = 250
@@ -54,8 +54,9 @@ def tx(fromsock, tosock, src, dst):
 def test_connected(if0=None, if1=None, group=True):
     if if0 is None or if1 is None:
         iwd_list = [dev.name for dev in iwd.IWD.get_instance().list_devices()]
-        non_iwd_list = [ifname for w in wiphy.wiphy_map.values()
-                for ifname in w.interface_map]
+
+        non_iwd_list = [rad.interface.name for rad in ctx.radios if rad.interface is not None]
+
         for intf in iwd_list + non_iwd_list:
             if if0 is None:
                 if0 = intf
-- 
2.21.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [PATCH 06/16] auto-t: ofono.py: fix timeout cleanup and wait for service
  2020-08-27 17:32 [PATCH 01/16] hwsim: return radio ID on create James Prestwood
                   ` (3 preceding siblings ...)
  2020-08-27 17:32 ` [PATCH 05/16] auto-t: testutil.py: " James Prestwood
@ 2020-08-27 17:32 ` James Prestwood
  2020-08-27 17:32 ` [PATCH 07/16] auto-t: iwd.py: update to work with test-runner rewrite James Prestwood
                   ` (10 subsequent siblings)
  15 siblings, 0 replies; 18+ messages in thread
From: James Prestwood @ 2020-08-27 17:32 UTC (permalink / raw)
  To: iwd

[-- Attachment #1: Type: text/plain, Size: 1929 bytes --]

Ofono.py was not cleaning up the timeout, nor waiting for the ofono.org
service to come up before continuing.
---
 autotests/util/ofono.py | 25 ++++++++++++++++++-------
 1 file changed, 18 insertions(+), 7 deletions(-)

diff --git a/autotests/util/ofono.py b/autotests/util/ofono.py
index 2bc6338b..f1ed14ee 100644
--- a/autotests/util/ofono.py
+++ b/autotests/util/ofono.py
@@ -1,4 +1,5 @@
 import dbus
+import time
 from gi.repository import GLib
 
 SIM_AUTH_IFACE = 'org.ofono.SimAuthentication'
@@ -7,6 +8,14 @@ class Ofono(dbus.service.Object):
     def __init__(self):
         self._bus = dbus.SystemBus()
 
+        tries = 0
+
+        while not self._bus.name_has_owner('org.ofono'):
+            if tries > 100:
+                raise TimeoutError('Waiting for org.ofono service timed out')
+            tries += 1
+            time.sleep(0.1)
+
     def enable_modem(self, path):
         self._modem_path = path
         self._modem_iface = dbus.Interface(
@@ -37,11 +46,13 @@ class Ofono(dbus.service.Object):
             self._wait_timed_out = True
             return False
 
-        timeout = GLib.timeout_add_seconds(max_wait, wait_timeout_cb)
-        context = mainloop.get_context()
-        while (not self._sim_auth_up):
-            context.iteration(may_block=True)
-            if self._wait_timed_out:
-                raise TimeoutError('waiting for SimAuthentication timed out')
+        try:
+            timeout = GLib.timeout_add_seconds(max_wait, wait_timeout_cb)
+            context = mainloop.get_context()
+            while (not self._sim_auth_up):
+                context.iteration(may_block=True)
+                if self._wait_timed_out:
+                    raise TimeoutError('waiting for SimAuthentication timed out')
 
-        GLib.source_remove(timeout)
+        finally:
+            GLib.source_remove(timeout)
-- 
2.21.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [PATCH 07/16] auto-t: iwd.py: update to work with test-runner rewrite
  2020-08-27 17:32 [PATCH 01/16] hwsim: return radio ID on create James Prestwood
                   ` (4 preceding siblings ...)
  2020-08-27 17:32 ` [PATCH 06/16] auto-t: ofono.py: fix timeout cleanup and wait for service James Prestwood
@ 2020-08-27 17:32 ` James Prestwood
  2020-08-27 17:32 ` [PATCH 08/16] auto-t: iwd.py: fix multiple timeout cleanup issues James Prestwood
                   ` (9 subsequent siblings)
  15 siblings, 0 replies; 18+ messages in thread
From: James Prestwood @ 2020-08-27 17:32 UTC (permalink / raw)
  To: iwd

[-- Attachment #1: Type: text/plain, Size: 3510 bytes --]

The class initializer was changed to make starting IWD on by default.
This allows us to not have multiple places where IWD is started. You
can still pass False which will initialize the class but not start
IWD itself. The configuration director was also chaged to /tmp by
default. This was done by all tests requiring main.conf anyways, so
we might as well make it the default.
---
 autotests/util/iwd.py | 48 ++++++++++++++-----------------------------
 1 file changed, 15 insertions(+), 33 deletions(-)

diff --git a/autotests/util/iwd.py b/autotests/util/iwd.py
index ef1ea80e..37daaeef 100755
--- a/autotests/util/iwd.py
+++ b/autotests/util/iwd.py
@@ -15,7 +15,7 @@ import weakref
 from abc import ABCMeta, abstractmethod
 from enum import Enum
 
-import wiphy
+from config import ctx
 
 IWD_STORAGE_DIR =               '/var/lib/iwd'
 IWD_CONFIG_DIR =                '/etc/iwd'
@@ -873,7 +873,10 @@ class DeviceList(collections.Mapping):
 
 
 class IWD(AsyncOpAbstract):
-    ''''''
+    '''
+        Start an IWD instance. By default IWD will be started and with the
+        config directory set to /tmp.
+    '''
     _bus = dbus.SystemBus()
 
     _object_manager_if = None
@@ -882,40 +885,22 @@ class IWD(AsyncOpAbstract):
     _devices = None
     _instance = None
 
-    def __init__(self, start_iwd_daemon = False,
-                                               iwd_config_dir = IWD_CONFIG_DIR):
+    def __init__(self, start_iwd_daemon = True, iwd_config_dir = '/tmp'):
         global mainloop
         mainloop = GLib.MainLoop()
 
-        if start_iwd_daemon:
-            args = []
-            iwd_wiphys = [wname for wname, wiphy in wiphy.wiphy_map.items()
-                          if wiphy.use == 'iwd']
-            whitelist = ','.join(iwd_wiphys)
-
-            if os.environ.get('IWD_TEST_VALGRIND', None) == 'on':
-                    args.append('valgrind')
-                    args.append('--leak-check=full')
-                    args.append('--log-file=/tmp/valgrind.log')
-
-            os.environ["CONFIGURATION_DIRECTORY"] = iwd_config_dir;
-
-            args.append('iwd')
-            args.append('-p')
-            args.append(whitelist)
-            args.append('-d')
-
-            import subprocess
-            iwd_proc = subprocess.Popen(args)
+        self._iwd_proc = None
 
-            self._iwd_proc = iwd_proc
+        if start_iwd_daemon:
+            self._iwd_proc = ctx.start_iwd(iwd_config_dir)
 
         tries = 0
         while not self._bus.name_has_owner(IWD_SERVICE):
-            if os.environ['IWD_TEST_TIMEOUTS'] == 'on':
+            if ctx.args.gdb == 'None':
                 if tries > 100:
                     if start_iwd_daemon:
-                        iwd_proc.terminate()
+                        self._iwd_proc.kill()
+                        self._iwd_proc = None
                     raise TimeoutError('IWD has failed to start')
                 tries += 1
             time.sleep(0.1)
@@ -935,12 +920,9 @@ class IWD(AsyncOpAbstract):
         self._known_networks = None
         self._devices = None
 
-        self._iwd_proc.terminate()
-        self._iwd_proc.wait()
-
-        self._iwd_proc = None
-
-        del os.environ["CONFIGURATION_DIRECTORY"]
+        if self._iwd_proc is not None:
+            self._iwd_proc.kill()
+            self._iwd_proc = None
 
     @property
     def _object_manager(self):
-- 
2.21.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [PATCH 08/16] auto-t: iwd.py: fix multiple timeout cleanup issues
  2020-08-27 17:32 [PATCH 01/16] hwsim: return radio ID on create James Prestwood
                   ` (5 preceding siblings ...)
  2020-08-27 17:32 ` [PATCH 07/16] auto-t: iwd.py: update to work with test-runner rewrite James Prestwood
@ 2020-08-27 17:32 ` James Prestwood
  2020-08-27 17:32 ` [PATCH 09/16] auto-t: remove wiphy.py James Prestwood
                   ` (8 subsequent siblings)
  15 siblings, 0 replies; 18+ messages in thread
From: James Prestwood @ 2020-08-27 17:32 UTC (permalink / raw)
  To: iwd

[-- Attachment #1: Type: text/plain, Size: 3539 bytes --]

There were a few more places where the GLib timeout was not being
removed which could cause the IWD object to hang around longer than
expected causing problems for subsequent tests.
---
 autotests/util/iwd.py | 51 +++++++++++++++++++++++++------------------
 1 file changed, 30 insertions(+), 21 deletions(-)

diff --git a/autotests/util/iwd.py b/autotests/util/iwd.py
index 37daaeef..113039b6 100755
--- a/autotests/util/iwd.py
+++ b/autotests/util/iwd.py
@@ -948,15 +948,18 @@ class IWD(AsyncOpAbstract):
             self._wait_timed_out = True
             return False
 
-        timeout = GLib.timeout_add_seconds(max_wait, wait_timeout_cb)
-        context = mainloop.get_context()
-        while not eval(condition_str):
-            context.iteration(may_block=True)
-            if self._wait_timed_out and os.environ['IWD_TEST_TIMEOUTS'] == 'on':
-                raise TimeoutError('[' + condition_str + ']'\
-                                   ' condition was not met in '\
-                                   + str(max_wait) + ' sec')
-        GLib.source_remove(timeout)
+        try:
+            timeout = GLib.timeout_add_seconds(max_wait, wait_timeout_cb)
+            context = mainloop.get_context()
+            while not eval(condition_str):
+                context.iteration(may_block=True)
+                if self._wait_timed_out and ctx.args.gdb == 'None':
+                    raise TimeoutError('[' + condition_str + ']'\
+                                       ' condition was not met in '\
+                                       + str(max_wait) + ' sec')
+        finally:
+            if not self._wait_timed_out:
+                GLib.source_remove(timeout)
 
     def wait(self, time):
         self._wait_timed_out = False
@@ -964,10 +967,14 @@ class IWD(AsyncOpAbstract):
             self._wait_timed_out = True
             return False
 
-        GLib.timeout_add(int(time * 1000), wait_timeout_cb)
-        context = mainloop.get_context()
-        while not self._wait_timed_out:
-            context.iteration(may_block=True)
+        try:
+            timeout = GLib.timeout_add(int(time * 1000), wait_timeout_cb)
+            context = mainloop.get_context()
+            while not self._wait_timed_out:
+                context.iteration(may_block=True)
+        finally:
+            if not self._wait_timed_out:
+                GLib.source_remove(timeout)
 
     @staticmethod
     def clear_storage():
@@ -1013,14 +1020,16 @@ class IWD(AsyncOpAbstract):
             self._wait_timed_out = True
             return False
 
-        timeout = GLib.timeout_add_seconds(max_wait, wait_timeout_cb)
-        context = mainloop.get_context()
-        while len(self._devices) < wait_to_appear:
-            context.iteration(may_block=True)
-            if self._wait_timed_out:
-                raise TimeoutError('IWD has no associated devices')
-
-        GLib.source_remove(timeout)
+        try:
+            timeout = GLib.timeout_add_seconds(max_wait, wait_timeout_cb)
+            context = mainloop.get_context()
+            while len(self._devices) < wait_to_appear:
+                context.iteration(may_block=True)
+                if self._wait_timed_out:
+                    raise TimeoutError('IWD has no associated devices')
+        finally:
+            if not self._wait_timed_out:
+                GLib.source_remove(timeout)
 
         return list(self._devices.values())
 
-- 
2.21.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [PATCH 09/16] auto-t: remove wiphy.py
  2020-08-27 17:32 [PATCH 01/16] hwsim: return radio ID on create James Prestwood
                   ` (6 preceding siblings ...)
  2020-08-27 17:32 ` [PATCH 08/16] auto-t: iwd.py: fix multiple timeout cleanup issues James Prestwood
@ 2020-08-27 17:32 ` James Prestwood
  2020-08-27 17:32 ` [PATCH 10/16] auto-t: fix hidden network test James Prestwood
                   ` (7 subsequent siblings)
  15 siblings, 0 replies; 18+ messages in thread
From: James Prestwood @ 2020-08-27 17:32 UTC (permalink / raw)
  To: iwd

[-- Attachment #1: Type: text/plain, Size: 1296 bytes --]

This is no longer needed
---
 autotests/util/wiphy.py | 32 --------------------------------
 1 file changed, 32 deletions(-)
 delete mode 100644 autotests/util/wiphy.py

diff --git a/autotests/util/wiphy.py b/autotests/util/wiphy.py
deleted file mode 100644
index 3f94d87b..00000000
--- a/autotests/util/wiphy.py
+++ /dev/null
@@ -1,32 +0,0 @@
-#! /usr/bin/python3
-import os
-import collections
-
-wiphy_map = {}
-
-Wiphy = collections.namedtuple('Wiphy', ['name', 'use', 'interface_map'])
-
-Intf = collections.namedtuple('Intf',
-        ['name', 'wiphy', 'ctrl_interface', 'config'])
-
-def parse_list():
-    for entry in os.environ['TEST_WIPHY_LIST'].split('\n'):
-        wname, use_str = entry.split('=', 1)
-        use = use_str.split(',')
-
-        if wname not in wiphy_map:
-            wiphy_map[wname] = Wiphy(use=use[0], name=wname, interface_map={})
-
-        if len(use) <= 1:
-            continue
-
-        intf = {}
-        intf['name'] = None
-        intf['wiphy'] = wiphy_map[wname]
-        intf['ctrl_interface'] = None
-        intf['config'] = None
-        intf.update(dict([param.split('=', 1) for param in use[1:]]))
-
-        wiphy_map[wname].interface_map[intf['name']] = Intf(**intf)
-
-parse_list()
-- 
2.21.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [PATCH 10/16] auto-t: fix hidden network test
  2020-08-27 17:32 [PATCH 01/16] hwsim: return radio ID on create James Prestwood
                   ` (7 preceding siblings ...)
  2020-08-27 17:32 ` [PATCH 09/16] auto-t: remove wiphy.py James Prestwood
@ 2020-08-27 17:32 ` James Prestwood
  2020-08-27 17:32 ` [PATCH 11/16] auto-t: fix testSAE autoconnect_test.py James Prestwood
                   ` (6 subsequent siblings)
  15 siblings, 0 replies; 18+ messages in thread
From: James Prestwood @ 2020-08-27 17:32 UTC (permalink / raw)
  To: iwd

[-- Attachment #1: Type: text/plain, Size: 2281 bytes --]

This test was never 100% reliable, and after the test-runner re-write
it became extremely unreliable. The issue came down to the very common
block of code thats present in many tests where we wait for obj.scanning
then not obj.scanning. This is fine when a dbus scan() is explicitly
done before, otherwise it could lead to problems. Without a dbus scan
explicitly called we are assuming a periodic scan will happen. If it
already happen the initial wait for obj.scanning will never return and
time out.

This probably needs to be changed in several tests, but for this specific
case we can remove the waits completely. Since
check_autoconnect_hidden_network has a 30 second wait on
DeviceState.connected this will ultimately time out if anything goes
wrong. There isn't any great reason to wait for scanning (for this test
specifically).

A minor style change was also made when initializing IWD. The values
passed in this test are now the default, so no arguments need to be
passed.
---
 autotests/testHiddenNetworks/validation.py | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)

diff --git a/autotests/testHiddenNetworks/validation.py b/autotests/testHiddenNetworks/validation.py
index 9c433623..991a5b32 100644
--- a/autotests/testHiddenNetworks/validation.py
+++ b/autotests/testHiddenNetworks/validation.py
@@ -50,11 +50,6 @@ class TestConnectAutoConnect(unittest.TestCase):
         device = devices[0]
 
         if autoconnect:
-            condition = 'obj.scanning'
-            wd.wait_for_object_condition(device, condition)
-            condition = 'not obj.scanning'
-            wd.wait_for_object_condition(device, condition)
-
             self.check_autoconnect_hidden_network(wd, device, ssid, throws)
         else:
             if wait_periodic_scan:
@@ -71,7 +66,6 @@ class TestConnectAutoConnect(unittest.TestCase):
 
     def validate(self, ssid, autoconnect, throws = None, use_agent = False,
                                                     wait_periodic_scan = False):
-        wd = IWD(True, '/tmp')
-
+        wd = IWD()
         self.validate_connection(wd, ssid, autoconnect, throws, use_agent,
                                                         wait_periodic_scan)
-- 
2.21.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [PATCH 11/16] auto-t: fix testSAE autoconnect_test.py
  2020-08-27 17:32 [PATCH 01/16] hwsim: return radio ID on create James Prestwood
                   ` (8 preceding siblings ...)
  2020-08-27 17:32 ` [PATCH 10/16] auto-t: fix hidden network test James Prestwood
@ 2020-08-27 17:32 ` James Prestwood
  2020-08-27 17:32 ` [PATCH 12/16] auto-t: skip ofono tests if ofonod isn't running James Prestwood
                   ` (5 subsequent siblings)
  15 siblings, 0 replies; 18+ messages in thread
From: James Prestwood @ 2020-08-27 17:32 UTC (permalink / raw)
  To: iwd

[-- Attachment #1: Type: text/plain, Size: 1769 bytes --]

This test was unreliable since it was assuming a periodic scan would
happen at just the right time. Instead since we are expecting autoconnect
we can just wait for DeviceState.connected then after we are connected
verify the network was correct.
---
 autotests/testSAE/autoconnect_test.py | 15 +++++----------
 1 file changed, 5 insertions(+), 10 deletions(-)

diff --git a/autotests/testSAE/autoconnect_test.py b/autotests/testSAE/autoconnect_test.py
index a45fd09d..2eb73b7f 100644
--- a/autotests/testSAE/autoconnect_test.py
+++ b/autotests/testSAE/autoconnect_test.py
@@ -23,20 +23,15 @@ class Test(unittest.TestCase):
         devices[2].disconnect()
         devices[3].disconnect()
 
-        condition = 'obj.scanning'
-        wd.wait_for_object_condition(device, condition)
+        condition = 'obj.state == DeviceState.connected'
+        wd.wait_for_object_condition(device, condition, 30)
 
-        condition = 'not obj.scanning'
+        condition = 'obj.connected_network is not None'
         wd.wait_for_object_condition(device, condition)
 
         ordered_network = device.get_ordered_network('ssidSAE')
 
-        self.assertEqual(ordered_network.type, NetworkType.psk)
-
-        condition = 'obj.connected'
-        wd.wait_for_object_condition(ordered_network.network_object, condition)
-
-        device.wait_for_connected()
+        self.assertTrue(ordered_network.network_object.connected)
 
         device.disconnect()
 
@@ -44,7 +39,7 @@ class Test(unittest.TestCase):
         wd.wait_for_object_condition(ordered_network.network_object, condition)
 
     def test_connection_success(self):
-        wd = IWD(True)
+        wd = IWD()
 
         self.validate_connection(wd)
 
-- 
2.21.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [PATCH 12/16] auto-t: skip ofono tests if ofonod isn't running
  2020-08-27 17:32 [PATCH 01/16] hwsim: return radio ID on create James Prestwood
                   ` (9 preceding siblings ...)
  2020-08-27 17:32 ` [PATCH 11/16] auto-t: fix testSAE autoconnect_test.py James Prestwood
@ 2020-08-27 17:32 ` James Prestwood
  2020-08-27 17:32 ` [PATCH 13/16] auto-t: replace hard-coded interfaces James Prestwood
                   ` (4 subsequent siblings)
  15 siblings, 0 replies; 18+ messages in thread
From: James Prestwood @ 2020-08-27 17:32 UTC (permalink / raw)
  To: iwd

[-- Attachment #1: Type: text/plain, Size: 2448 bytes --]

---
 autotests/testEAP-AKA-ofono/connection_test.py       | 4 ++++
 autotests/testEAP-AKA-prime-ofono/connection_test.py | 4 ++++
 autotests/testEAP-SIM-ofono/connection_test.py       | 4 ++++
 3 files changed, 12 insertions(+)

diff --git a/autotests/testEAP-AKA-ofono/connection_test.py b/autotests/testEAP-AKA-ofono/connection_test.py
index 94066f76..27f736f6 100644
--- a/autotests/testEAP-AKA-ofono/connection_test.py
+++ b/autotests/testEAP-AKA-ofono/connection_test.py
@@ -9,6 +9,7 @@ from iwd import IWD
 from iwd import NetworkType
 from hlrauc import AuthCenter
 from ofono import Ofono
+from config import ctx
 
 class Test(unittest.TestCase):
 
@@ -56,6 +57,9 @@ class Test(unittest.TestCase):
         auth.stop()
     @classmethod
     def setUpClass(cls):
+        if not ctx.is_process_running('ofonod'):
+            cls.skipTest(cls, "ofono not running")
+
         IWD.copy_to_storage('ssidEAP-AKA.8021x')
 
     @classmethod
diff --git a/autotests/testEAP-AKA-prime-ofono/connection_test.py b/autotests/testEAP-AKA-prime-ofono/connection_test.py
index 2ff99894..967ba352 100644
--- a/autotests/testEAP-AKA-prime-ofono/connection_test.py
+++ b/autotests/testEAP-AKA-prime-ofono/connection_test.py
@@ -9,6 +9,7 @@ from iwd import IWD
 from iwd import NetworkType
 from hlrauc import AuthCenter
 from ofono import Ofono
+from config import ctx
 
 class Test(unittest.TestCase):
 
@@ -57,6 +58,9 @@ class Test(unittest.TestCase):
 
     @classmethod
     def setUpClass(cls):
+        if not ctx.is_process_running('ofonod'):
+            cls.skipTest(cls, "ofono not running")
+
         IWD.copy_to_storage('ssidEAP-AKA.8021x')
 
     @classmethod
diff --git a/autotests/testEAP-SIM-ofono/connection_test.py b/autotests/testEAP-SIM-ofono/connection_test.py
index 95242943..de7c4b48 100644
--- a/autotests/testEAP-SIM-ofono/connection_test.py
+++ b/autotests/testEAP-SIM-ofono/connection_test.py
@@ -9,6 +9,7 @@ from iwd import IWD
 from iwd import NetworkType
 from hlrauc import AuthCenter
 from ofono import Ofono
+from config import ctx
 
 class Test(unittest.TestCase):
 
@@ -57,6 +58,9 @@ class Test(unittest.TestCase):
 
     @classmethod
     def setUpClass(cls):
+        if not ctx.is_process_running('ofonod'):
+            cls.skipTest(cls, "ofono not running")
+
         IWD.copy_to_storage('ssidEAP-SIM.8021x')
 
     @classmethod
-- 
2.21.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [PATCH 13/16] auto-t: replace hard-coded interfaces
  2020-08-27 17:32 [PATCH 01/16] hwsim: return radio ID on create James Prestwood
                   ` (10 preceding siblings ...)
  2020-08-27 17:32 ` [PATCH 12/16] auto-t: skip ofono tests if ofonod isn't running James Prestwood
@ 2020-08-27 17:32 ` James Prestwood
  2020-08-27 17:32 ` [PATCH 14/16] auto-t: remove device.wait_for_connected James Prestwood
                   ` (3 subsequent siblings)
  15 siblings, 0 replies; 18+ messages in thread
From: James Prestwood @ 2020-08-27 17:32 UTC (permalink / raw)
  To: iwd

[-- Attachment #1: Type: text/plain, Size: 1987 bytes --]

test-runner now supports interface name replacement inside hostapd
config files. Since a given test configuration doesn't know what
interface names there will be $ifaceN can be specified instead e.g.

rsn_preauth_interfaces=$iface0 $iface1

The $ifaceN values will be replace with actual interface names when
the test is started.

This patch also removes ctrl_interface inside the hostapd config
files as this is no longer required.
---
 autotests/testPreauth-roam/eaptls-preauth-1.conf | 3 +--
 autotests/testPreauth-roam/eaptls-preauth-2.conf | 3 +--
 2 files changed, 2 insertions(+), 4 deletions(-)

diff --git a/autotests/testPreauth-roam/eaptls-preauth-1.conf b/autotests/testPreauth-roam/eaptls-preauth-1.conf
index 0c7f96a7..bc808ed7 100644
--- a/autotests/testPreauth-roam/eaptls-preauth-1.conf
+++ b/autotests/testPreauth-roam/eaptls-preauth-1.conf
@@ -2,7 +2,6 @@ hw_mode=g
 channel=1
 ssid=TestPreauth
 utf8_ssid=1
-ctrl_interface=/var/run/hostapd
 
 wpa=2
 wpa_key_mgmt=WPA-EAP
@@ -24,7 +23,7 @@ radius_server_auth_port=1812
 nas_identifier=testeap1
 
 rsn_preauth=1
-rsn_preauth_interfaces=wln0 wln1
+rsn_preauth_interfaces=$iface0 $iface1
 disable_pmksa_caching=0
 
 # Allow PMK cache to be shared opportunistically among configured interfaces
diff --git a/autotests/testPreauth-roam/eaptls-preauth-2.conf b/autotests/testPreauth-roam/eaptls-preauth-2.conf
index bddcca69..6d3a18e7 100644
--- a/autotests/testPreauth-roam/eaptls-preauth-2.conf
+++ b/autotests/testPreauth-roam/eaptls-preauth-2.conf
@@ -2,7 +2,6 @@ hw_mode=g
 channel=2
 ssid=TestPreauth
 utf8_ssid=1
-ctrl_interface=/var/run/hostapd
 
 wpa=2
 wpa_key_mgmt=WPA-EAP
@@ -20,7 +19,7 @@ auth_server_port=1812
 auth_server_shared_secret=secret
 
 rsn_preauth=1
-rsn_preauth_interfaces=wln0 wln1
+rsn_preauth_interfaces=$iface0 $iface1
 disable_pmksa_caching=0
 
 # Allow PMK cache to be shared opportunistically among configured interfaces
-- 
2.21.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [PATCH 14/16] auto-t: remove device.wait_for_connected
  2020-08-27 17:32 [PATCH 01/16] hwsim: return radio ID on create James Prestwood
                   ` (11 preceding siblings ...)
  2020-08-27 17:32 ` [PATCH 13/16] auto-t: replace hard-coded interfaces James Prestwood
@ 2020-08-27 17:32 ` James Prestwood
  2020-08-27 17:32 ` [PATCH 15/16] tools: post test-runner rewrite cleanup James Prestwood
                   ` (2 subsequent siblings)
  15 siblings, 0 replies; 18+ messages in thread
From: James Prestwood @ 2020-08-27 17:32 UTC (permalink / raw)
  To: iwd

[-- Attachment #1: Type: text/plain, Size: 2328 bytes --]

I had added this without realizing wait_for_object_condition
can do this already using the device object.
---
 autotests/testSAE/clogging_test.py |  3 ++-
 autotests/util/iwd.py              | 25 -------------------------
 2 files changed, 2 insertions(+), 26 deletions(-)

diff --git a/autotests/testSAE/clogging_test.py b/autotests/testSAE/clogging_test.py
index e94e2ad6..c46c5821 100644
--- a/autotests/testSAE/clogging_test.py
+++ b/autotests/testSAE/clogging_test.py
@@ -48,7 +48,8 @@ class Test(unittest.TestCase):
             wd.wait_for_object_condition(n.network_object, condition)
 
         for d in devices:
-            d.wait_for_connected()
+            condition = 'obj.state == DeviceState.connected'
+            wd.wait_for_object_condition(d, condition)
 
         for d in devices:
             d.disconnect()
diff --git a/autotests/util/iwd.py b/autotests/util/iwd.py
index 113039b6..9c106b89 100755
--- a/autotests/util/iwd.py
+++ b/autotests/util/iwd.py
@@ -568,31 +568,6 @@ class Device(IWDDBusAbstract):
             if self._adhoc_timed_out:
                 raise TimeoutError("Timed out waiting for peer %s" % addr)
 
-    def wait_for_connected(self):
-        if str(self.state) == "connected":
-            return
-
-        self._connected_success = False
-        self._connected_timed_out = False
-
-        def wait_timeout_cb():
-            self._connected_timed_out = True
-            return False
-
-        def connected_prop_changed(iface, changed, invalid):
-            if changed.get('State', None):
-                if changed['State'] == 'connected':
-                    self._connected_success = True
-
-        self._prop_proxy.connect_to_signal('PropertiesChanged',
-                                            connected_prop_changed)
-        GLib.timeout_add(int(15 * 1000), wait_timeout_cb)
-        context = mainloop.get_context()
-        while not self._connected_success:
-            context.iteration(may_block=True)
-            if self._connected_timed_out:
-                raise TimeoutError("Timed out waiting for connected")
-
     def __str__(self, prefix = ''):
         return prefix + 'Device: ' + self.device_path + '\n'\
                + prefix + '\tName:\t\t' + self.name + '\n'\
-- 
2.21.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [PATCH 15/16] tools: post test-runner rewrite cleanup
  2020-08-27 17:32 [PATCH 01/16] hwsim: return radio ID on create James Prestwood
                   ` (12 preceding siblings ...)
  2020-08-27 17:32 ` [PATCH 14/16] auto-t: remove device.wait_for_connected James Prestwood
@ 2020-08-27 17:32 ` James Prestwood
  2020-08-27 17:32 ` [PATCH 16/16] doc: update test runner docs James Prestwood
  2020-08-27 20:08 ` [PATCH 01/16] hwsim: return radio ID on create Denis Kenzior
  15 siblings, 0 replies; 18+ messages in thread
From: James Prestwood @ 2020-08-27 17:32 UTC (permalink / raw)
  To: iwd

[-- Attachment #1: Type: text/plain, Size: 81167 bytes --]

Removed test-runner.c, and renamed py_runner to test-runner. Removed
tools/test-runner from .gitignore.

This was done as a separate commit to avoid a nasty diff between the
existing test runner, and the new python version
---
 .gitignore                          |    1 -
 Makefile.am                         |    7 -
 tools/{py_runner.py => test-runner} |    0
 tools/test-runner.c                 | 3310 ---------------------------
 4 files changed, 3318 deletions(-)
 rename tools/{py_runner.py => test-runner} (100%)
 delete mode 100644 tools/test-runner.c

diff --git a/.gitignore b/.gitignore
index 93fec4f7..33405af8 100644
--- a/.gitignore
+++ b/.gitignore
@@ -33,7 +33,6 @@ wired/ead.8
 wired/ead.service
 tools/hwsim
 tools/hwsim.1
-tools/test-runner
 tools/probe-req
 unit/test-cmac-aes
 unit/test-arc4
diff --git a/Makefile.am b/Makefile.am
index e83dbeee..5f16920f 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -366,13 +366,6 @@ man_MANS += tools/hwsim.1
 endif
 endif
 
-if TOOLS
-noinst_PROGRAMS += tools/test-runner
-
-tools_test_runner_SOURCES = tools/test-runner.c
-tools_test_runner_LDADD = $(ell_ldadd)
-endif
-
 unit_tests = unit/test-cmac-aes \
 		unit/test-hmac-md5 unit/test-hmac-sha1 unit/test-hmac-sha256 \
 		unit/test-prf-sha1 unit/test-kdf-sha256 \
diff --git a/tools/py_runner.py b/tools/test-runner
similarity index 100%
rename from tools/py_runner.py
rename to tools/test-runner
diff --git a/tools/test-runner.c b/tools/test-runner.c
deleted file mode 100644
index 40148241..00000000
--- a/tools/test-runner.c
+++ /dev/null
@@ -1,3310 +0,0 @@
-/*
- *
- *  BlueZ - Bluetooth protocol stack for Linux
- *
- *  Copyright (C) 2012-2019  Intel Corporation. All rights reserved.
- *
- *
- *  This library is free software; you can redistribute it and/or
- *  modify it under the terms of the GNU Lesser General Public
- *  License as published by the Free Software Foundation; either
- *  version 2.1 of the License, or (at your option) any later version.
- *
- *  This library is distributed in the hope that it will be useful,
- *  but WITHOUT ANY WARRANTY; without even the implied warranty of
- *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
- *  Lesser General Public License for more details.
- *
- *  You should have received a copy of the GNU Lesser General Public
- *  License along with this library; if not, write to the Free Software
- *  Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
- *
- */
-
-#ifdef HAVE_CONFIG_H
-#include <config.h>
-#endif
-
-#define _GNU_SOURCE
-#include <stdio.h>
-#include <errno.h>
-#include <fcntl.h>
-#include <unistd.h>
-#include <stdlib.h>
-#include <stdbool.h>
-#include <stdint.h>
-#include <string.h>
-#include <getopt.h>
-#include <poll.h>
-#include <dirent.h>
-#include <sys/wait.h>
-#include <sys/stat.h>
-#include <sys/types.h>
-#include <sys/mount.h>
-#include <sys/param.h>
-#include <sys/reboot.h>
-#include <sys/time.h>
-#include <glob.h>
-#include <time.h>
-#include <ell/ell.h>
-
-#include "linux/nl80211.h"
-
-#ifndef WAIT_ANY
-#define WAIT_ANY (-1)
-#endif
-
-#define CMDLINE_MAX			2048
-
-#define BIN_IW				"iw"
-#define BIN_HWSIM			"hwsim"
-#define BIN_OFONO			"ofonod"
-#define BIN_PHONESIM			"phonesim"
-#define BIN_HOSTAPD			"hostapd"
-#define BIN_IWD				"iwd"
-
-#define HWSIM_RADIOS_MAX		100
-#define TEST_MAX_EXEC_TIME_SEC		20
-
-static enum action {
-	ACTION_AUTO_TEST,
-	ACTION_UNIT_TEST,
-} test_action;
-
-static const char *own_binary;
-static char **test_argv;
-static int test_argc;
-static char **verbose_apps;
-static char *verbose_opt;
-static bool valgrind;
-static char *gdb_opt;
-static bool enable_debug;
-const char *debug_filter;
-static struct l_settings *hw_config;
-static bool native_hw;
-static bool shell;
-static bool log;
-static char log_dir[PATH_MAX];
-static uid_t log_uid;
-static gid_t log_gid;
-static const char *qemu_binary;
-static const char *kernel_image;
-static const char *exec_home;
-static const char *test_action_params;
-static char top_level_path[PATH_MAX];
-static struct l_queue *wiphy_list;
-
-#if defined(__i386__)
-/*
- * If iwd is being compiled for i386, prefer the i386 qemu but try the
- * X86-64 version as a fallback.
- */
-static const char * const qemu_table[] = {
-	"qemu-system-i386",
-	"/usr/bin/qemu-system-i386",
-	"qemu-system-x86_64",
-	"/usr/bin/qemu-system-x86_64",
-	NULL
-};
-#elif defined(__x86_64__)
-/*
- * If iwd is being built for X86-64 bits there's no point booting a 32-bit
- * only system.
- */
-static const char * const qemu_table[] = {
-	"qemu-system-x86_64",
-	"/usr/bin/qemu-system-x86_64",
-	NULL
-};
-#elif defined(__arm__)
-/*
- * If iwd is being built for ARM look for 32-bit version.
- */
-static const char * const qemu_table[] = {
-	"qemu-system-arm",
-	"/usr/bin/qemu-system-arm",
-	NULL
-};
-#elif defined(__aarch64__)
-/*
- * If iwd is being built for AARCH64 look for 64-bit version.
- */
-static const char * const qemu_table[] = {
-	"qemu-system-aarch64",
-	"/usr/bin/qemu-system-aarch64",
-	NULL
-};
-#elif defined(__powerpc__)
-/*
- * If iwd is being built for PowerPC look for 32-bit version.
- */
-static const char * const qemu_table[] = {
-	"qemu-system-ppc",
-	"/usr/bin/qemu-system-ppc",
-	NULL
-};
-#elif defined(__powerpc64__)
-/*
- * If iwd is being built for PowerPC-64 look for 64-bit version.
- */
-static const char * const qemu_table[] = {
-	"qemu-system-ppc64",
-	"/usr/bin/qemu-system-ppc64",
-	NULL
-};
-#else
-#warning Qemu binary name not defined for this architecture yet
-static const char * const qemu_table[] = { NULL };
-#endif
-
-struct wiphy {
-	char name[20];
-	int id;
-	unsigned int interface_index;
-	bool interface_created : 1;
-	bool used_by_hostapd : 1;
-	char *interface_name;
-	char *hostapd_ctrl_interface;
-	char *hostapd_config;
-	bool can_ap;
-};
-
-static bool check_verbosity(const char *app)
-{
-	char **apps = verbose_apps;
-
-	/*
-	 * All processes are verbose if logging is enabled. Kernel is a bit
-	 * different as we just pipe dmesg into a log file at the end of
-	 * execution.
-	 */
-	if (log && strcmp(app, "kernel") != 0)
-		return true;
-
-	/*
-	 * Turn on output if this is a unit test run. Nothing should output
-	 * anything except the tests themselves and the kernel.
-	 */
-	if (test_action == ACTION_UNIT_TEST && strcmp(app, "kernel") != 0)
-		return true;
-
-	if (!apps)
-		return false;
-
-	while (*apps) {
-		if (!strcmp(app, *apps))
-			return true;
-
-		apps++;
-	}
-
-	return false;
-}
-
-static bool path_exist(const char *path_name)
-{
-	struct stat st;
-
-	if (!stat(path_name, &st))
-		return true;
-
-	return false;
-}
-
-static const char *find_qemu(void)
-{
-	int i;
-
-	for (i = 0; qemu_table[i]; i++)
-		if (path_exist(qemu_table[i]))
-			return qemu_table[i];
-
-	return NULL;
-}
-
-static const char * const kernel_table[] = {
-	"bzImage",
-	"arch/x86/boot/bzImage",
-	"vmlinux",
-	"arch/x86/boot/vmlinux",
-	NULL
-};
-
-static const char *find_kernel(void)
-{
-	int i;
-
-	for (i = 0; kernel_table[i]; i++)
-		if (path_exist(kernel_table[i]))
-			return kernel_table[i];
-
-	return NULL;
-}
-
-static const struct {
-	const char *target;
-	const char *linkpath;
-} dev_table[] = {
-	{ "/proc/self/fd",	"/dev/fd"	},
-	{ "/proc/self/fd/0",	"/dev/stdin"	},
-	{ "/proc/self/fd/1",	"/dev/stdout"	},
-	{ "/proc/self/fd/2",	"/dev/stderr"	},
-	{ }
-};
-
-static const struct {
-	const char *fstype;
-	const char *target;
-	const char *options;
-	unsigned long flags;
-} mount_table[] = {
-	{ "sysfs",    "/sys",     NULL,        MS_NOSUID|MS_NOEXEC|MS_NODEV },
-	{ "proc",     "/proc",    NULL,        MS_NOSUID|MS_NOEXEC|MS_NODEV },
-	{ "devpts",   "/dev/pts", "mode=0620", MS_NOSUID|MS_NOEXEC },
-	{ "tmpfs",    "/dev/shm", "mode=1777",
-					MS_NOSUID|MS_NODEV|MS_STRICTATIME },
-	{ "tmpfs",    "/run",     "mode=0755",
-					MS_NOSUID|MS_NODEV|MS_STRICTATIME },
-	{ "tmpfs",    "/var/lib/iwd", "mode=0755", 0 },
-	{ "tmpfs",    "/tmp",              NULL, 0 },
-	{ "debugfs",  "/sys/kernel/debug", NULL, 0 },
-	{ }
-};
-
-static const char * const config_table[] = {
-	"/usr/share/dbus-1",
-	NULL
-};
-
-static void prepare_sandbox(void)
-{
-	int i;
-
-	for (i = 0; mount_table[i].fstype; i++) {
-		struct stat st;
-
-		if (lstat(mount_table[i].target, &st) < 0) {
-			mkdir(mount_table[i].target, 0755);
-		}
-
-		if (mount(mount_table[i].fstype,
-				mount_table[i].target,
-				mount_table[i].fstype,
-				mount_table[i].flags,
-				mount_table[i].options) < 0) {
-			l_error("Error: Failed to mount filesystem %s",
-							mount_table[i].target);
-		}
-	}
-
-	for (i = 0; dev_table[i].target; i++) {
-
-		if (symlink(dev_table[i].target, dev_table[i].linkpath) < 0)
-			l_error("Failed to create device symlink: %s",
-							strerror(errno));
-	}
-
-	setsid();
-
-	ioctl(STDIN_FILENO, TIOCSCTTY, 1);
-
-	for (i = 0; config_table[i]; i++) {
-		if (mount("tmpfs", config_table[i], "tmpfs",
-				MS_NOSUID|MS_NOEXEC|MS_NODEV|MS_STRICTATIME,
-				"mode=0755") < 0)
-			l_error("Failed to create filesystem: %s",
-							strerror(errno));
-	}
-}
-
-static char *const qemu_argv[] = {
-	"",
-	"-machine", "type=q35,accel=kvm:tcg",
-	"-nodefaults",
-	"-no-user-config",
-	"-monitor", "none",
-	"-display", "none",
-	"-m", "192M",
-	"-nographic",
-	"-vga", "none",
-	"-net", "none",
-	"-no-acpi",
-	"-no-hpet",
-	"-no-reboot",
-	"-fsdev", "local,id=fsdev-root,path=/,readonly,security_model=none",
-	"-device", "virtio-9p-pci,fsdev=fsdev-root,mount_tag=/dev/root",
-	"-chardev", "stdio,id=chardev-serial0,signal=off",
-	"-device", "pci-serial,chardev=chardev-serial0",
-	"-device", "virtio-rng-pci",
-	NULL
-};
-
-static char *const qemu_envp[] = {
-	"HOME=/",
-	NULL
-};
-
-static bool check_virtualization(void)
-{
-#if defined(__GNUC__) && (defined(__i386__) || defined(__amd64__))
-	uint32_t ecx;
-
-	__asm__ __volatile__("cpuid" : "=c" (ecx) :
-				"a" (1) : "%ebx", "%edx");
-
-	if (!!(ecx & (1 << 5))) {
-		l_info("Found support for Virtual Machine eXtensions");
-		return true;
-	}
-
-	__asm__ __volatile__("cpuid" : "=c" (ecx) :
-				"a" (0x80000001) : "%ebx", "%edx");
-
-	if (ecx & (1 << 2)) {
-		l_info("Found support for Secure Virtual Machine extension");
-		return true;
-	}
-#endif
-	return false;
-}
-
-static bool start_qemu(void)
-{
-	char cwd[PATH_MAX], testargs[PATH_MAX];
-	char *initcmd, *cmdline;
-	char **argv;
-	int i, pos;
-	bool has_virt;
-	int num_pci = 0, num_usb = 0;
-	char **pci_keys = NULL;
-	char **usb_keys = NULL;
-	L_AUTO_FREE_VAR(char *, log_option) = NULL;
-
-	has_virt = check_virtualization();
-
-	if (!getcwd(cwd, sizeof(cwd)))
-		strcat(cwd, "/");
-
-	if (own_binary[0] == '/')
-		initcmd = l_strdup_printf("%s", own_binary);
-	else
-		initcmd = l_strdup_printf("%s/%s", cwd, own_binary);
-
-	pos = snprintf(testargs, sizeof(testargs), "%s", test_argv[0]);
-
-	for (i = 1; i < test_argc; i++) {
-		int len;
-
-		len = sizeof(testargs) - pos;
-		pos += snprintf(testargs + pos, len, " %s", test_argv[i]);
-	}
-
-	cmdline = l_strdup_printf(
-			"console=ttyS0,115200n8 earlyprintk=serial "
-			"rootfstype=9p "
-			"root=/dev/root "
-			"rootflags=trans=virtio,version=9p2000.u "
-			"acpi=off pci=noacpi noapic %s ro "
-			"mac80211_hwsim.radios=0 init=%s TESTHOME=%s "
-			"TESTVERBOUT=\'%s\' DEBUG_FILTER=\'%s\'"
-			"TEST_ACTION=%u TEST_ACTION_PARAMS=\'%s\' "
-			"TESTARGS=\'%s\' PATH=\'%s\' VALGRIND=%u "
-			"GDB=\'%s\' HW=\'%s\' SHELL=%u "
-			"LOG_PATH=\'%s\' LOG_UID=\'%d\' LOG_GID=\'%d\'",
-			check_verbosity("kernel") ? "ignore_loglevel" : "quiet",
-			initcmd, cwd, verbose_opt ? verbose_opt : "none",
-			enable_debug ? debug_filter : "",
-			test_action,
-			test_action_params ? test_action_params : "",
-			testargs,
-			getenv("PATH"),
-			valgrind,
-			gdb_opt ? gdb_opt : "none",
-			hw_config ? "real" : "virtual",
-			shell,
-			log ? log_dir : "none",
-			log_uid, log_gid);
-
-	if (hw_config) {
-		if (l_settings_has_group(hw_config, "PCIAdapters")) {
-			pci_keys = l_settings_get_keys(hw_config, "PCIAdapters");
-
-			for (num_pci = 0; pci_keys[num_pci]; num_pci++);
-		}
-
-		if (l_settings_has_group(hw_config, "USBAdapters")) {
-			usb_keys = l_settings_get_keys(hw_config, "USBAdapters");
-
-			for (num_usb = 0; usb_keys[num_usb]; num_usb++);
-		}
-
-		if (!pci_keys && !usb_keys) {
-			l_error("hs config had no PCIAdapters or USBAdapters");
-			l_free(initcmd);
-			l_free(cmdline);
-			return false;
-		}
-	}
-
-	/*
-	 * This got quite confusing. We need enough room for:
-	 *
-	 * qemu_argv (static list above with default parameters)
-	 * -kernel,-append,-cpu,-host parameters (7)
-	 * -enable-kvm and/or -usb (2)
-	 * PCI and/or USB parameters (num_pci * 2) (num_usb * 2)
-	 * Logging directory/device (2)
-	 */
-	argv = alloca(sizeof(qemu_argv) + sizeof(char *) *
-				(7 + (2 + (num_pci * 2) + (num_usb * 2) + 2)));
-	memcpy(argv, qemu_argv, sizeof(qemu_argv));
-
-	pos = (sizeof(qemu_argv) / sizeof(char *)) - 1;
-
-	argv[0] = (char *) qemu_binary;
-
-	argv[pos++] = "-kernel";
-	argv[pos++] = (char *) kernel_image;
-	argv[pos++] = "-append";
-	argv[pos++] = (char *) cmdline;
-	argv[pos++] = "-cpu";
-	argv[pos++] = has_virt ? "host" : "max";
-
-	if (pci_keys) {
-		argv[pos++] = "-enable-kvm";
-		for (i = 0; pci_keys[i]; i++) {
-			argv[pos++] = "-device";
-			argv[pos] = alloca(22);
-			sprintf(argv[pos], "vfio-pci,host=%s",
-					l_settings_get_value(hw_config,
-					"PCIAdapters", pci_keys[i]));
-			pos++;
-		}
-	}
-
-	if (usb_keys) {
-		argv[pos++] = "-usb";
-		for (i = 0; usb_keys[i]; i++) {
-			const char *value = l_settings_get_value(hw_config,
-						"USBAdapters", usb_keys[i]);
-			char **info = l_strsplit(value, ',');
-
-			if (l_strv_length(info) != 2) {
-				l_error("hw config formatting error");
-				l_strv_free(info);
-				return false;
-			}
-
-			argv[pos++] = "-device";
-			argv[pos] = alloca(32);
-			sprintf(argv[pos], "usb-host,hostbus=%s,hostaddr=%s",
-						info[0], info[1]);
-			pos++;
-
-			l_strv_free(info);
-		}
-	}
-
-	if (log) {
-		/*
-		 * Create a virtfs device and tag it. This allows the guest to
-		 * mount 'logdir' in the path specified with --log.
-		 */
-		log_option = l_strdup_printf("local,path=%s,mount_tag=logdir,"
-					"security_model=passthrough,id=logdir",
-					log_dir);
-		argv[pos++] = "-virtfs";
-		argv[pos++] = log_option;
-	}
-
-	argv[pos] = NULL;
-
-	execve(argv[0], argv, qemu_envp);
-
-	/* Don't expect to reach here */
-	free(initcmd);
-	free(cmdline);
-
-	return true;
-}
-
-static pid_t execute_program(char *argv[], char *envp[], bool wait,
-				const char *test_name)
-{
-	int status;
-	pid_t pid, child_pid;
-	char *str;
-	bool verbose;
-	char *log_name = argv[0];
-
-	if (!argv[0])
-		return -1;
-
-	/*
-	 * We have a few special cases here:
-	 *
-	 * Since execute_program automatically logs to <process>.log this would
-	 * put all iwd output into valgrind.log rather than iwd.log. Since we
-	 * are explicitly having valgrind output to a log file we can assume any
-	 * output from this is only IWD, and not valgrind.
-	 *
-	 * python3 is special cased so that tests which start IWD manually can
-	 * still show IWD output when using -v iwd.
-	 */
-	if (!strcmp(log_name, "valgrind") || !strcmp(log_name, "python3"))
-		log_name = "iwd";
-
-	str = l_strjoinv(argv, ' ');
-	l_debug("Executing: %s", str);
-	l_free(str);
-
-	child_pid = fork();
-	if (child_pid < 0) {
-		l_error("Failed to fork new process");
-		return -1;
-	}
-
-	if (child_pid == 0) {
-		int fd = -1;
-		L_AUTO_FREE_VAR(char *, log_file) = NULL;
-
-		verbose = check_verbosity(log_name);
-
-		/* No stdout and no logging */
-		if (!verbose && !log)
-			fd = open("/dev/null", O_WRONLY);
-		else if (log && verbose) {
-			/*
-			 * Create the log file for this process. If no test name
-			 * was specified this is a 'global' process (only run
-			 * once, not per-test).
-			 */
-			if (test_name) {
-				log_file = l_strdup_printf("%s/%s/%s.log",
-						log_dir, test_name, log_name);
-			} else
-				log_file = l_strdup_printf("%s/%s.log",
-						log_dir, log_name);
-
-			fd = open(log_file, O_WRONLY | O_CREAT | O_APPEND,
-					S_IRUSR | S_IWUSR);
-			if (fchown(fd, log_uid, log_gid) < 0)
-				l_error("failed to fchown %s", log_file);
-		}
-
-		if (fd > -1) {
-			dup2(fd, 1);
-			dup2(fd, 2);
-
-			close(fd);
-		}
-
-		execvpe(argv[0], argv, envp);
-
-		l_error("Failed to call execvpe for: %s. Error: %s", argv[0],
-							strerror(errno));
-
-		exit(EXIT_FAILURE);
-	}
-
-	if (!wait)
-		goto exit;
-
-	do {
-		pid = waitpid(child_pid, &status, 0);
-	} while (!WIFEXITED(status) && pid == child_pid);
-
-	if (WEXITSTATUS(status) != EXIT_SUCCESS)
-		return -1;
-
-exit:
-	return child_pid;
-}
-
-static void kill_process(pid_t pid)
-{
-	int status;
-	int i = 0;
-
-	l_debug("Terminate pid: %d", pid);
-
-	kill(pid, SIGTERM);
-
-	do {
-		if (waitpid(pid, &status, WNOHANG) == pid)
-			return;
-
-		usleep(100000);
-	} while (!WIFEXITED(status) && !WIFSIGNALED(status) && i++ < 20);
-
-	l_error("Failed to kill process %d gracefully", pid);
-	kill(pid, SIGKILL);
-
-	/* SIGKILL shouldn't need WNOHANG */
-	waitpid(pid, &status, 0);
-}
-
-static bool wait_for_socket(const char *socket, useconds_t wait_time)
-{
-	int i = 0;
-
-	do {
-		if (path_exist(socket))
-			return true;
-
-		usleep(wait_time);
-	} while ((wait_time > 0) ? i++ < 20 : true);
-
-	l_error("Error: cannot find socket: %s", socket);
-	return false;
-}
-
-static void create_dbus_system_conf(void)
-{
-	FILE *fp;
-
-	fp = fopen("/usr/share/dbus-1/system.conf", "we");
-	if (!fp)
-		return;
-
-	fputs("<!DOCTYPE busconfig PUBLIC ", fp);
-	fputs("\"-//freedesktop//DTD D-Bus Bus Configuration 1.0//EN\" ", fp);
-	fputs("\"http://www.freedesktop.org/standards/dbus/1.0/", fp);
-	fputs("busconfig.dtd\">\n", fp);
-	fputs("<busconfig>\n", fp);
-	fputs("<type>system</type>\n", fp);
-	fputs("<listen>unix:path=/run/dbus/system_bus_socket</listen>\n", fp);
-	fputs("<limit name=\"reply_timeout\">2147483647</limit>", fp);
-	fputs("<auth>ANONYMOUS</auth>\n", fp);
-	fputs("<allow_anonymous/>\n", fp);
-	fputs("<policy context=\"default\">\n", fp);
-	fputs("<allow user=\"*\"/>\n", fp);
-	fputs("<allow own=\"*\"/>\n", fp);
-	fputs("<allow send_type=\"method_call\"/>\n", fp);
-	fputs("<allow send_type=\"signal\"/>\n", fp);
-	fputs("<allow send_type=\"method_return\"/>\n", fp);
-	fputs("<allow send_type=\"error\"/>\n", fp);
-	fputs("<allow receive_type=\"method_call\"/>\n", fp);
-	fputs("<allow receive_type=\"signal\"/>\n", fp);
-	fputs("<allow receive_type=\"method_return\"/>\n", fp);
-	fputs("<allow receive_type=\"error\"/>\n", fp);
-	fputs("</policy>\n", fp);
-	fputs("</busconfig>\n", fp);
-
-	fclose(fp);
-
-	mkdir("/run/dbus", 0755);
-}
-
-static bool start_dbus_daemon(void)
-{
-	char *argv[4];
-	pid_t pid;
-
-	argv[0] = "dbus-daemon";
-	argv[1] = "--system";
-	argv[2] = "--nosyslog";
-	argv[3] = NULL;
-
-	if (check_verbosity("dbus"))
-		setenv("DBUS_VERBOSE", "1", true);
-
-	pid = execute_program(argv, environ, false, NULL);
-	if (pid < 0)
-		return false;
-
-	if (!wait_for_socket("/run/dbus/system_bus_socket", 25 * 10000))
-		return false;
-
-	if (check_verbosity("dbus-monitor")) {
-		argv[0] = "dbus-monitor";
-		argv[1] = "--system";
-		argv[2] = NULL;
-		execute_program(argv, environ, false, NULL);
-	}
-
-	l_debug("D-Bus is running");
-
-	return true;
-}
-
-static bool start_haveged(void)
-{
-	char *argv[2];
-	pid_t pid;
-
-	argv[0] = "haveged";
-	argv[1] = NULL;
-
-	pid = execute_program(argv, environ, true, NULL);
-	if (pid < 0)
-		return false;
-
-	return true;
-}
-
-static bool set_interface_state(const char *if_name, bool isUp)
-{
-	char *state, *argv[4];
-	pid_t pid;
-
-	if (isUp)
-		state = "up";
-	else
-		state = "down";
-
-	argv[0] = "ifconfig";
-	argv[1] = (char *) if_name;
-	argv[2] = state;
-	argv[3] = NULL;
-
-	pid = execute_program(argv, environ, true, NULL);
-	if (pid < 0)
-		return false;
-
-	return true;
-}
-
-static bool create_interface(const char *if_name, const char *phy_name)
-{
-	char *argv[9];
-	pid_t pid;
-
-	argv[0] = BIN_IW;
-	argv[1] = "phy";
-	argv[2] = (char *) phy_name;
-	argv[3] = "interface";
-	argv[4] = "add";
-	argv[5] = (char *) if_name;
-	argv[6] = "type";
-	argv[7] = "managed";
-	argv[8] = NULL;
-
-	pid = execute_program(argv, environ, true, NULL);
-	if (pid < 0)
-		return false;
-
-	return true;
-}
-
-static bool delete_interface(const char *if_name)
-{
-	char *argv[5];
-	pid_t pid;
-
-	argv[0] = BIN_IW;
-	argv[1] = "dev";
-	argv[2] = (char *) if_name;
-	argv[3] = "del";
-	argv[4] = NULL;
-
-	pid = execute_program(argv, environ, true, NULL);
-	if (pid < 0)
-		return false;
-
-	return true;
-}
-
-static bool list_interfaces(void)
-{
-	char *argv[3];
-	pid_t pid;
-
-	argv[0] = "ifconfig";
-	argv[1] = "-a";
-	argv[2] = NULL;
-
-	pid = execute_program(argv, environ, true, NULL);
-	if (pid < 0)
-		return false;
-
-	return true;
-}
-
-static bool list_hwsim_radios(void)
-{
-	char *argv[3];
-	pid_t pid;
-
-	argv[0] = BIN_HWSIM;
-	argv[1] = "--list";
-	argv[2] = NULL;
-
-	pid = execute_program(argv, environ, true, NULL);
-	if (pid < 0)
-		return false;
-
-	return true;
-}
-
-static int read_radio_id(void)
-{
-	static int current_radio_id;
-
-	return current_radio_id++;
-}
-
-struct hwsim_radio_params {
-	unsigned int channels;
-	bool p2p_device;
-	bool use_chanctx;
-	char *iftype_disable;
-	char *cipher_disable;
-};
-
-static int create_hwsim_radio(const char *radio_name,
-				struct hwsim_radio_params *params)
-{
-	char *argv[10];
-	pid_t pid;
-	int idx = 0;
-
-	/*TODO add the rest of params*/
-	argv[idx++] = BIN_HWSIM;
-	argv[idx++] = "--create";
-	argv[idx++] = "--name";
-	argv[idx++] = (char *) radio_name;
-	argv[idx++] = "--nointerface";
-
-	if (params->iftype_disable) {
-		argv[idx++] = "--iftype-disable";
-		argv[idx++] = params->iftype_disable;
-	}
-
-	if (params->cipher_disable) {
-		argv[idx++] = "--cipher-disable";
-		argv[idx++] = params->cipher_disable;
-	}
-
-	argv[idx] = NULL;
-
-	pid = execute_program(argv, environ, true, NULL);
-	if (pid < 0)
-		return -1;
-
-	return read_radio_id();
-}
-
-static bool destroy_hwsim_radio(int radio_id)
-{
-	char *argv[4];
-	char destroy_param[20];
-	pid_t pid;
-
-	sprintf(destroy_param, "--destroy=%d", radio_id);
-
-	argv[0] = BIN_HWSIM;
-	argv[1] = destroy_param;
-	argv[2] = NULL;
-
-	pid = execute_program(argv, environ, true, NULL);
-	if (pid < 0)
-		return false;
-
-	return true;
-}
-
-static pid_t register_hwsim_as_trans_medium(void)
-{
-	char *argv[16];
-	unsigned int idx = 0;
-
-	if (strcmp(gdb_opt, "hwsim") == 0) {
-		argv[idx++] = "gdb";
-		argv[idx++] = "--args";
-	}
-
-	argv[idx++] = BIN_HWSIM;
-	argv[idx++] = NULL;
-
-	return execute_program(argv, environ, false, NULL);
-}
-
-static void terminate_medium(pid_t medium_pid)
-{
-	kill_process(medium_pid);
-}
-
-#define HOSTAPD_CTRL_INTERFACE_PREFIX "/var/run/hostapd"
-
-static bool loopback_started;
-
-static void start_loopback(void)
-{
-	char *argv[7];
-
-	if (loopback_started)
-		return;
-
-	argv[0] = "ifconfig";
-	argv[1] = "lo";
-	argv[2] = "127.0.0.1";
-	argv[3] = "up";
-	argv[4] = NULL;
-	execute_program(argv, environ, false, NULL);
-
-	argv[0] = "route";
-	argv[1] = "add";
-	argv[2] = "127.0.0.1";
-	argv[3] = NULL;
-	execute_program(argv, environ, false, NULL);
-
-	loopback_started = true;
-}
-
-static pid_t start_phonesim(const char *test_name)
-{
-	char *argv[5];
-
-	argv[0] = BIN_PHONESIM;
-	argv[1] = "-p";
-	argv[2] = "12345";
-	argv[3] = "/usr/share/phonesim/default.xml";
-	argv[4] = NULL;
-
-	start_loopback();
-
-	setenv("OFONO_PHONESIM_CONFIG", "/tmp/phonesim.conf", true);
-
-	return execute_program(argv, environ, false, test_name);
-}
-
-static void stop_phonesim(pid_t pid)
-{
-	kill_process(pid);
-}
-
-static pid_t start_ofono(const char *test_name)
-{
-	char *argv[5];
-	bool verbose = check_verbosity(BIN_OFONO);
-
-	argv[0] = BIN_OFONO;
-	argv[1] = "-n";
-	argv[2] = "--plugin=atmodem,phonesim";
-
-	if (verbose)
-		argv[3] = "-d";
-	else
-		argv[3] = NULL;
-
-	argv[4] = NULL;
-
-	start_loopback();
-
-	return execute_program(argv, environ, false, test_name);
-}
-
-static void stop_ofono(pid_t pid)
-{
-	kill_process(pid);
-}
-
-static pid_t start_hostapd(char **config_files, struct wiphy **wiphys,
-				const char *test_name, const char *radius_conf)
-{
-	char **argv;
-	pid_t pid;
-	int idx = 0;
-	uint32_t wait = 25 * 10000;
-	bool verbose = check_verbosity(BIN_HOSTAPD);
-	size_t ifnames_size;
-	char *ifnames;
-	int i;
-
-	for (i = 0, ifnames_size = 0; wiphys[i]; i++)
-		ifnames_size += 1 + strlen(wiphys[i]->interface_name);
-
-	argv = alloca(sizeof(char *) * (9 + i));
-
-	if (strcmp(gdb_opt, "hostapd") == 0) {
-		argv[idx++] = "gdb";
-		argv[idx++] = "--args";
-		wait = 0;
-	}
-
-	argv[idx++] = BIN_HOSTAPD;
-
-	ifnames = alloca(ifnames_size);
-	argv[idx++] = "-i";
-	argv[idx++] = ifnames;
-
-	argv[idx++] = "-g";
-	argv[idx++] = wiphys[0]->hostapd_ctrl_interface;
-
-	for (i = 0, ifnames_size = 0; wiphys[i]; i++) {
-		if (ifnames_size)
-			ifnames[ifnames_size++] = ',';
-		strcpy(ifnames + ifnames_size, wiphys[i]->interface_name);
-		ifnames_size += strlen(wiphys[i]->interface_name);
-
-		argv[idx++] = config_files[i];
-	}
-
-	if (radius_conf)
-		argv[idx++] = (void *)radius_conf;
-
-	if (verbose) {
-		argv[idx++] = "-d";
-		argv[idx++] = NULL;
-	} else {
-		argv[idx++] = NULL;
-	}
-
-	pid = execute_program(argv, environ, false, log ? test_name : NULL);
-	if (pid < 0) {
-		goto exit;
-	}
-
-	if (!wait_for_socket(wiphys[0]->hostapd_ctrl_interface, wait))
-		pid = -1;
-exit:
-	return pid;
-}
-
-static void destroy_hostapd_instances(pid_t hostapd_pids[])
-{
-	int i = 0;
-
-	while (hostapd_pids[i] != -1) {
-		kill_process(hostapd_pids[i]);
-
-		l_debug("hostapd instance with pid=%d is destroyed",
-			hostapd_pids[i]);
-
-		hostapd_pids[i] = -1;
-
-		i++;
-	}
-}
-
-#define TEST_TOP_DIR_DEFAULT_NAME	"autotests"
-#define TEST_DIR_PREFIX			"test"
-
-static bool is_test_file(const char *file)
-{
-	size_t i;
-	static const char * const test_file_extension_table[] = {
-		"test",
-		"test.py",
-		"Test",
-		"Test.py",
-		NULL
-	};
-
-	for (i = 0; test_file_extension_table[i]; i++) {
-		if (l_str_has_suffix(file, test_file_extension_table[i]))
-			return true;
-	}
-
-	return false;
-}
-
-static int is_test_dir(const char *dir)
-{
-	return strncmp(dir, TEST_DIR_PREFIX, strlen(TEST_DIR_PREFIX)) == 0;
-}
-
-static bool find_test_configuration(const char *path, int level,
-						struct l_queue *config_queue);
-
-struct test_entry {
-	struct l_queue *test_queue;
-	char *path;
-};
-
-static int insert_py_test(const void *a, const void *b, void *user_data)
-{
-	return strcmp((const char *)a, (const char *)b);
-}
-
-static int insert_test_entry(const void *a, const void *b, void *user_data)
-{
-	const struct test_entry *entry_a = a;
-	const struct test_entry *entry_b = b;
-
-	return strcmp(entry_a->path, entry_b->path);
-}
-
-static bool add_path(const char *path, int level, struct l_queue *config_queue)
-{
-	DIR *dir = NULL;
-	struct l_queue *py_test_queue = NULL;
-	struct dirent *entry;
-	char *npath;
-
-	dir = opendir(path);
-	if (!dir) {
-		l_error("Test directory does not exist: %s", path);
-		return false;
-	}
-
-	while ((entry = readdir(dir))) {
-		if (entry->d_type == DT_DIR) {
-			if (!strcmp(entry->d_name, ".") ||
-					!strcmp(entry->d_name, ".."))
-				continue;
-
-			if (level == 0 && is_test_dir(entry->d_name)) {
-				npath = l_strdup_printf("%s/%s", path,
-								entry->d_name);
-
-				find_test_configuration(npath, 1, config_queue);
-
-				l_free(npath);
-			}
-		} else if (level == 1 && is_test_file(entry->d_name)) {
-			if (!py_test_queue)
-				py_test_queue = l_queue_new();
-
-			l_queue_insert(py_test_queue, l_strdup(entry->d_name),
-						insert_py_test, NULL);
-		}
-	}
-
-	if (py_test_queue && !l_queue_isempty(py_test_queue)) {
-		struct test_entry *entry = l_new(struct test_entry, 1);
-
-		entry->test_queue = py_test_queue;
-		entry->path = l_strdup(path);
-
-		l_queue_insert(config_queue, entry, insert_test_entry, NULL);
-	}
-
-	closedir(dir);
-	return true;
-}
-
-static bool find_test_configuration(const char *path, int level,
-						struct l_queue *config_queue)
-{
-	glob_t glist;
-	int i = 0;
-	int ret;
-
-	if (!config_queue)
-		return false;
-
-	ret = glob(path, 0, NULL, &glist);
-	if (ret != 0) {
-		l_error("Could not match glob %s", path);
-		return false;
-	}
-
-	while (glist.gl_pathv[i]) {
-		if (!add_path(glist.gl_pathv[i], level, config_queue))
-			return false;
-
-		i++;
-	}
-
-	return true;
-}
-
-#define HW_CONFIG_FILE_NAME		"hw.conf"
-#define HW_CONFIG_GROUP_HOSTAPD		"HOSTAPD"
-#define HW_CONFIG_GROUP_SETUP		"SETUP"
-
-#define HW_CONFIG_SETUP_NUM_RADIOS	"num_radios"
-#define HW_CONFIG_SETUP_RADIO_CONFS	"radio_confs"
-#define HW_CONFIG_SETUP_MAX_EXEC_SEC	"max_test_exec_interval_sec"
-#define HW_CONFIG_SETUP_TMPFS_EXTRAS	"tmpfs_extra_stuff"
-#define HW_CONFIG_SETUP_START_IWD	"start_iwd"
-#define HW_CONFIG_SETUP_IWD_CONF_DIR	"iwd_config_dir"
-#define HW_CONFIG_SETUP_REG_DOMAIN	"reg_domain"
-#define HW_CONFIG_SETUP_NEEDS_HWSIM	"needs_hwsim"
-
-static struct l_settings *read_hw_config(const char *test_dir_path)
-{
-	struct l_settings *hw_settings;
-	char *hw_file;
-
-	hw_file = l_strdup_printf("%s/%s", test_dir_path, HW_CONFIG_FILE_NAME);
-
-	hw_settings = l_settings_new();
-
-	if (!l_settings_load_from_file(hw_settings, hw_file)) {
-		l_error("No %s file found", HW_CONFIG_FILE_NAME);
-		goto error_exit;
-	}
-
-	if (!l_settings_has_group(hw_settings, HW_CONFIG_GROUP_SETUP)) {
-		l_error("No %s setting group found in %s",
-						HW_CONFIG_GROUP_SETUP, hw_file);
-		goto error_exit;
-	}
-
-	l_free(hw_file);
-	return hw_settings;
-
-error_exit:
-	l_free(hw_file);
-	l_settings_free(hw_settings);
-	return NULL;
-}
-
-#define HW_CONFIG_PHY_CHANNELS	"channels"
-#define HW_CONFIG_PHY_CHANCTX	"use_chanctx"
-#define HW_CONFIG_PHY_P2P	"p2p_device"
-#define HW_CONFIG_PHY_IFTYPE_DISABLE "iftype_disable"
-#define HW_CONFIG_PHY_CIPHER_DISABLE "cipher_disable"
-
-#define HW_MIN_NUM_RADIOS	1
-
-#define HW_INTERFACE_PREFIX	"wln"
-#define HW_INTERFACE_STATE_UP   true
-#define HW_INTERFACE_STATE_DOWN false
-
-static bool configure_hw_radios(struct l_settings *hw_settings,
-						struct l_queue *wiphy_list)
-{
-	char **radio_conf_list;
-	int i, num_radios_requested;
-	bool status = false;
-
-	l_settings_get_int(hw_settings, HW_CONFIG_GROUP_SETUP,
-						HW_CONFIG_SETUP_NUM_RADIOS,
-							&num_radios_requested);
-
-	if (num_radios_requested < HW_MIN_NUM_RADIOS) {
-		l_error("%s must be greater or equal to %d",
-			HW_CONFIG_SETUP_NUM_RADIOS, HW_MIN_NUM_RADIOS);
-		return false;
-	}
-
-	radio_conf_list =
-		l_settings_get_string_list(hw_settings, HW_CONFIG_GROUP_SETUP,
-						HW_CONFIG_SETUP_RADIO_CONFS,
-									':');
-	for (i = 0; i < num_radios_requested; i++) {
-		struct wiphy *wiphy;
-		struct hwsim_radio_params params = { 0 };
-
-		wiphy = l_new(struct wiphy, 1);
-
-		sprintf(wiphy->name, "rad%d", i);
-
-		/* radio not in radio_confs, use default parameters */
-		if (!l_strv_contains(radio_conf_list, wiphy->name)) {
-			params.channels = 1;
-			params.p2p_device = true;
-			params.use_chanctx = true;
-			goto create;
-		}
-
-		if (!l_settings_get_uint(hw_settings, wiphy->name,
-					HW_CONFIG_PHY_CHANNELS,
-					&params.channels))
-			params.channels = 1;
-
-		if (!l_settings_get_bool(hw_settings, wiphy->name,
-					HW_CONFIG_PHY_P2P, &params.p2p_device))
-			params.p2p_device = true;
-
-		if (!l_settings_get_bool(hw_settings, wiphy->name,
-					HW_CONFIG_PHY_CHANCTX,
-					&params.use_chanctx))
-			params.use_chanctx = true;
-
-		params.iftype_disable = l_settings_get_string(hw_settings,
-					wiphy->name,
-					HW_CONFIG_PHY_IFTYPE_DISABLE);
-		params.cipher_disable = l_settings_get_string(hw_settings,
-					wiphy->name,
-					HW_CONFIG_PHY_CIPHER_DISABLE);
-
-create:
-		wiphy->id = create_hwsim_radio(wiphy->name, &params);
-		wiphy->can_ap = true;
-
-		if (wiphy->id < 0) {
-			l_free(wiphy);
-			goto exit;
-		}
-
-		l_queue_push_tail(wiphy_list, wiphy);
-	}
-
-	status = true;
-
-exit:
-	l_strfreev(radio_conf_list);
-	return status;
-}
-
-static void wiphy_free(void *data)
-{
-	struct wiphy *wiphy = data;
-
-	if (wiphy->interface_created) {
-		set_interface_state(wiphy->interface_name,
-					HW_INTERFACE_STATE_DOWN);
-
-		if (delete_interface(wiphy->interface_name))
-			l_debug("Removed interface %s", wiphy->interface_name);
-		else
-			l_error("Failed to remove interface %s",
-				wiphy->interface_name);
-	}
-
-	/* Native interfaces cannot be destroyed */
-	if (native_hw) {
-		set_interface_state(wiphy->interface_name,
-					HW_INTERFACE_STATE_DOWN);
-	} else {
-		destroy_hwsim_radio(wiphy->id);
-		l_debug("Removed radio id %d", wiphy->id);
-	}
-
-	l_free(wiphy->hostapd_config);
-	l_free(wiphy->hostapd_ctrl_interface);
-	l_free(wiphy->interface_name);
-
-	l_free(wiphy);
-}
-
-static bool configure_hostapd_instances(struct l_settings *hw_settings,
-						char *config_dir_path,
-						struct l_queue *wiphy_list,
-						pid_t hostapd_pids_out[],
-						int *phys_used)
-{
-	char **hostap_keys;
-	int i;
-	char **hostapd_config_file_paths;
-	struct wiphy **wiphys;
-	const char *radius_config = NULL;
-
-	*phys_used = 0;
-
-	if (!l_settings_has_group(hw_settings, HW_CONFIG_GROUP_HOSTAPD)) {
-		l_info("No hostapd instances to create");
-		return true;
-	}
-
-	hostap_keys =
-		l_settings_get_keys(hw_settings, HW_CONFIG_GROUP_HOSTAPD);
-
-	for (i = 0; hostap_keys[i]; i++);
-
-	hostapd_config_file_paths = l_new(char *, i + 1);
-	wiphys = alloca(sizeof(struct wiphy *) * (i + 1));
-	memset(wiphys, 0, sizeof(struct wiphy *) * (i + 1));
-
-	hostapd_pids_out[0] = -1;
-
-	for (i = 0; hostap_keys[i]; i++) {
-		const struct l_queue_entry *wiphy_entry;
-		const char *hostapd_config_file;
-		unsigned wiphy_idx = 0;
-
-		hostapd_config_file =
-			l_settings_get_value(hw_settings,
-						HW_CONFIG_GROUP_HOSTAPD,
-						hostap_keys[i]);
-
-		hostapd_config_file_paths[i] =
-			l_strdup_printf("%s/%s", config_dir_path,
-					hostapd_config_file);
-
-		if (!path_exist(hostapd_config_file_paths[i])) {
-			l_error("%s : hostapd configuration file [%s] "
-				"does not exist.", HW_CONFIG_FILE_NAME,
-						hostapd_config_file_paths[i]);
-			goto done;
-		}
-
-		if (!strcmp(hostap_keys[i], "radius_server")) {
-			radius_config = l_settings_get_value(hw_settings,
-						HW_CONFIG_GROUP_HOSTAPD,
-						"radius_server");
-			continue;
-		}
-
-		for (wiphy_entry = l_queue_get_entries(wiphy_list);
-					wiphy_entry;
-					wiphy_entry = wiphy_entry->next,
-					wiphy_idx++) {
-			struct wiphy *wiphy = wiphy_entry->data;
-
-			/*
-			 * We can skip this check in native mode since we have
-			 * no control over the phy name. Any test requiring a
-			 * "special" radio should not be ran in native mode.
-			 */
-			if (!native_hw && strcmp(wiphy->name, hostap_keys[i]))
-				continue;
-
-			if (wiphy->used_by_hostapd) {
-				/*
-				 * Since we bypass the above check in native
-				 * mode we could still get here. We can just
-				 * continue searching for more adapters if this
-				 * one is already in use.
-				 */
-				if (native_hw)
-					continue;
-
-				l_error("Wiphy %s already used by hostapd",
-					wiphy->name);
-				goto done;
-			}
-
-			if (!wiphy->can_ap)
-				continue;
-
-			wiphys[i] = wiphy;
-			break;
-		}
-
-		if (!wiphy_entry) {
-			l_error("Failed to find available wiphy.");
-			goto done;
-		}
-
-		if (native_hw)
-			goto hostapd_done;
-
-		wiphys[i]->interface_name = l_strdup_printf("%s%d",
-							HW_INTERFACE_PREFIX,
-							wiphy_idx);
-		if (!create_interface(wiphys[i]->interface_name,
-					wiphys[i]->name)) {
-			l_error("Failed to create hostapd interface %s on "
-				"radio %s",
-				wiphys[i]->interface_name, wiphys[i]->name);
-			goto done;
-		}
-
-		wiphys[i]->interface_created = true;
-		l_info("Created hostapd interface %s on %s radio",
-			wiphys[i]->interface_name, wiphys[i]->name);
-
-		if (!native_hw && !set_interface_state(wiphys[i]->interface_name,
-						HW_INTERFACE_STATE_UP)) {
-			l_error("Failed to set %s state UP",
-				wiphys[i]->interface_name);
-			goto done;
-		}
-
-hostapd_done:
-		wiphys[i]->used_by_hostapd = true;
-		wiphys[i]->hostapd_ctrl_interface =
-			l_strdup_printf("%s/%s", HOSTAPD_CTRL_INTERFACE_PREFIX,
-					wiphys[0]->interface_name);
-		wiphys[i]->hostapd_config = l_strdup(hostapd_config_file);
-
-		(*phys_used)++;
-	}
-
-	hostapd_pids_out[0] = start_hostapd(hostapd_config_file_paths, wiphys,
-						basename(config_dir_path),
-						radius_config);
-	hostapd_pids_out[1] = -1;
-
-done:
-	l_strfreev(hostapd_config_file_paths);
-
-	if (hostapd_pids_out[0] < 1)
-		return false;
-
-	return true;
-}
-
-static pid_t start_iwd(const char *config_dir, struct l_queue *wiphy_list,
-		const char *ext_options, int num_phys, const char *test_name)
-{
-	char *argv[13], **envp;
-	char *iwd_phys = NULL;
-	pid_t ret;
-	int idx = 0;
-	L_AUTO_FREE_VAR(char *, fd_option) = NULL;
-
-	if (valgrind) {
-		L_AUTO_FREE_VAR(char *, valgrind_log);
-		int fd;
-
-		argv[idx++] = "valgrind";
-		argv[idx++] = "--leak-check=full";
-
-		/*
-		 * Valgrind needs --log-fd if we want both stderr and stdout
-		 */
-		if (log)
-			valgrind_log = l_strdup_printf("%s/%s/valgrind.log",
-							log_dir, test_name);
-		else
-			valgrind_log = l_strdup("/tmp/valgrind.log");
-
-		fd = open(valgrind_log, O_WRONLY | O_CREAT | O_APPEND,
-					S_IRUSR | S_IWUSR);
-
-		if (log) {
-			if (fchown(fd, log_uid, log_gid) < 0)
-				l_error("chown failed");
-		}
-
-		fd_option = l_strdup_printf("--log-fd=%d", fd);
-		argv[idx++] = fd_option;
-	}
-
-	if (strcmp(gdb_opt, "iwd") == 0) {
-		argv[idx++] = "gdb";
-		argv[idx++] = "--args";
-	}
-
-	argv[idx++] = BIN_IWD;
-
-	if (check_verbosity(BIN_IWD) || shell)
-		argv[idx++] = "-d";
-
-	argv[idx] = NULL;
-
-	if (wiphy_list) {
-		const struct l_queue_entry *wiphy_entry;
-		struct l_string *list = l_string_new(64);
-
-		for (wiphy_entry = l_queue_get_entries(wiphy_list);
-					wiphy_entry;
-					wiphy_entry = wiphy_entry->next) {
-			struct wiphy *wiphy = wiphy_entry->data;
-
-			if (wiphy->used_by_hostapd)
-				continue;
-
-			/*
-			 * Break out, only adding the required number of phys
-			 * for this test.
-			 */
-			if (num_phys == 0)
-				break;
-
-			l_string_append_printf(list, "%s,", wiphy->name);
-
-			num_phys--;
-		}
-
-		iwd_phys = l_string_unwrap(list);
-		/* Take care of last comma */
-		iwd_phys[strlen(iwd_phys) - 1] = '\0';
-
-		argv[idx++] = "-p";
-		argv[idx++] = iwd_phys;
-		argv[idx] = NULL;
-	}
-
-	argv[idx++] = (char *)ext_options;
-	argv[idx] = NULL;
-
-	envp = l_strv_copy(environ);
-	envp = l_strv_append_printf(envp, "CONFIGURATION_DIRECTORY=%s",
-							config_dir);
-	envp = l_strv_append_printf(envp, "STATE_DIRECTORY=%s",
-							DAEMON_STORAGEDIR);
-
-	ret = execute_program(argv, envp, false, test_name);
-
-	l_strv_free(envp);
-
-	l_free(iwd_phys);
-
-	return ret;
-}
-
-static void terminate_iwd(pid_t iwd_pid)
-{
-	kill_process(iwd_pid);
-}
-
-static pid_t start_monitor(const char *test_name)
-{
-	char *argv[6];
-	char *write_arg;
-	pid_t pid;
-
-	write_arg = l_strdup_printf("%s/%s/monitor.pcap", log_dir, test_name);
-
-	argv[0] = "iwmon";
-	argv[1] = "--nortnl";
-	argv[2] = "--nowiphy";
-	argv[3] = "--write";
-	argv[4] = write_arg;
-	argv[5] = NULL;
-
-	pid = execute_program(argv, environ, false, test_name);
-
-	l_free(write_arg);
-
-	return pid;
-}
-
-static bool create_tmpfs_extra_stuff(char **tmpfs_extra_stuff)
-{
-	size_t i = 0;
-
-	if (!tmpfs_extra_stuff)
-		return true;
-
-	while (tmpfs_extra_stuff[i]) {
-		char *link_dir;
-		char *target_dir;
-
-		target_dir = realpath(tmpfs_extra_stuff[i], NULL);
-
-		if (!path_exist(target_dir)) {
-			l_error("No such directory: %s", target_dir);
-			l_free(target_dir);
-			return false;
-		}
-
-		link_dir = l_strdup_printf("%s%s", "/tmp",
-						rindex(target_dir, '/'));
-
-		if (symlink(target_dir, link_dir) < 0) {
-			l_error("Failed to create symlink %s for %s: %s",
-					link_dir, target_dir, strerror(errno));
-
-			l_free(target_dir);
-			l_free(link_dir);
-			return false;
-		}
-
-		l_free(tmpfs_extra_stuff[i]);
-		l_free(target_dir);
-
-		tmpfs_extra_stuff[i] = link_dir;
-		i++;
-	}
-
-	return true;
-}
-
-static bool remove_absolute_path_dirs(char **tmpfs_extra_stuff)
-{
-	size_t i = 0;
-
-	if (!tmpfs_extra_stuff)
-		return true;
-
-	while (tmpfs_extra_stuff[i]) {
-		if (unlink(tmpfs_extra_stuff[i]) < 0) {
-			l_error("Failed to remove symlink for %s: %s",
-					tmpfs_extra_stuff[i], strerror(errno));
-
-			return false;
-		}
-
-		i++;
-	}
-
-	return true;
-}
-
-#define CONSOLE_LN_DEFAULT	"\x1B[0m"
-#define CONSOLE_LN_RED		"\x1B[31m"
-#define CONSOLE_LN_GREEN	"\x1B[32m"
-#define CONSOLE_LN_BLACK	"\x1B[30m"
-#define CONSOLE_LN_YELLOW	"\x1B[33m"
-#define CONSOLE_LN_RESET	"\033[0m"
-
-#define CONSOLE_LN_BOLD		"\x1b[1m"
-
-#define CONSOLE_BG_WHITE	"\e[47m"
-#define CONSOLE_BG_DEFAULT	"\e[0m"
-
-enum test_status {
-	TEST_STATUS_STARTED,
-	TEST_STATUS_PASSED,
-	TEST_STATUS_FAILED,
-	TEST_STATUS_TIMEDOUT,
-};
-
-static void print_test_status(char *test_name, enum test_status ts,
-								double interval)
-{
-	const char *clear_line = "\r";
-	int int_len;
-	char *color_str;
-	char *status_str;
-	char *interval_str;
-	char *line_end = "";
-
-	switch (ts) {
-	case TEST_STATUS_STARTED:
-		color_str = CONSOLE_LN_RESET;
-		status_str = "STARTED   ";
-
-		if (strcmp(verbose_opt, "none"))
-			line_end = "\n";
-
-		break;
-	case TEST_STATUS_PASSED:
-		printf("%s", clear_line);
-		color_str = CONSOLE_LN_GREEN;
-		status_str = "PASSED    ";
-		line_end = "\n";
-
-		break;
-	case TEST_STATUS_FAILED:
-		printf("%s", clear_line);
-		color_str = CONSOLE_LN_RED;
-		status_str = "FAILED    ";
-		line_end = "\n";
-
-		break;
-	case TEST_STATUS_TIMEDOUT:
-		printf("%s", clear_line);
-		color_str = CONSOLE_LN_YELLOW;
-		status_str = "TIMED OUT ";
-		line_end = "\n";
-
-		break;
-	}
-
-	if (interval > 0)
-		int_len = snprintf(NULL, 0, "%.3f", interval);
-	else
-		int_len = 3;
-
-	int_len++;
-
-	interval_str = l_malloc(int_len);
-	memset(interval_str, ' ', int_len);
-	interval_str[int_len - 1] = '\0';
-
-	if (interval > 0)
-		sprintf(interval_str, "%.3f sec", interval);
-	else
-		sprintf(interval_str, "%s", "...");
-
-	printf("%s%s%s%-60s%7s%s", color_str, status_str, CONSOLE_LN_RESET,
-		test_name, interval_str, line_end);
-
-	fflush(stdout);
-
-	l_free(interval_str);
-}
-
-static void test_timeout_timer_tick(struct l_timeout *timeout, void *user_data)
-{
-	pid_t *test_exec_pid = (pid_t *) user_data;
-
-	kill_process(*test_exec_pid);
-
-	l_main_quit();
-}
-
-static void test_timeout_signal_handler(uint32_t signo, void *user_data)
-{
-	switch (signo) {
-	case SIGINT:
-	case SIGTERM:
-		l_main_quit();
-		break;
-	}
-}
-
-static pid_t start_execution_timeout_timer(unsigned int max_exec_interval_sec,
-							pid_t *test_exec_pid)
-{
-	struct l_timeout *test_exec_timeout;
-	pid_t test_timer_pid;
-
-	test_timer_pid = fork();
-	if (test_timer_pid < 0) {
-		l_error("Failed to fork new process");
-		return -1;
-	}
-
-	if (test_timer_pid == 0) {
-		if (!l_main_init())
-			exit(EXIT_FAILURE);
-
-		test_exec_timeout =
-			l_timeout_create(max_exec_interval_sec,
-						test_timeout_timer_tick,
-						test_exec_pid,
-						NULL);
-
-		l_main_run_with_signal(test_timeout_signal_handler, NULL);
-
-		l_timeout_remove(test_exec_timeout);
-
-		l_main_exit();
-
-		exit(EXIT_SUCCESS);
-	}
-
-	return test_timer_pid;
-}
-
-struct test_stats {
-	char *config_cycle_name;
-	unsigned int num_passed;
-	unsigned int num_failed;
-	unsigned int num_timedout;
-	double py_run_time;
-};
-
-static void run_py_tests(struct l_settings *hw_settings,
-					struct l_queue *test_queue,
-					struct l_queue *test_stats_queue,
-					const char *test_name)
-{
-	char *argv[3];
-	pid_t test_exec_pid, test_timer_pid = -1;
-	struct timeval time_before, time_after, time_elapsed;
-	unsigned int max_exec_interval;
-	char *py_test = NULL;
-	struct test_stats *test_stats;
-	pid_t monitor_pid = -1;
-
-	if (!l_settings_get_uint(hw_settings, HW_CONFIG_GROUP_SETUP,
-						HW_CONFIG_SETUP_MAX_EXEC_SEC,
-							&max_exec_interval))
-		max_exec_interval = TEST_MAX_EXEC_TIME_SEC;
-
-	l_info(CONSOLE_LN_BOLD "%-10s%-60s%s" CONSOLE_LN_RESET, "Status",
-							"Test", "Duration");
-
-start_next_test:
-
-	if (l_queue_isempty(test_queue))
-		return;
-
-	py_test = (char *) l_queue_pop_head(test_queue);
-	if (!py_test)
-		return;
-
-	if (log) {
-		char *test_path;
-		char *ext;
-		char *full_path;
-
-		test_path = l_strdup_printf("%s/%s", test_name, py_test);
-		ext = strchr(test_path, '.');
-		ext[0] = '\0';
-
-		full_path = l_strdup_printf("%s/%s", log_dir, test_path);
-
-		mkdir(full_path, 0755);
-		if (chown(full_path, log_uid, log_gid) < 0)
-			l_error("chown failed %s", full_path);
-
-		l_free(full_path);
-
-		monitor_pid = start_monitor(test_path);
-
-		l_free(test_path);
-	}
-
-	argv[0] = "python3";
-	argv[1] = py_test;
-	argv[2] = NULL;
-
-	print_test_status(py_test, TEST_STATUS_STARTED, 0);
-	test_exec_pid = execute_program(argv, environ, false, test_name);
-
-	gettimeofday(&time_before, NULL);
-
-	if (!strcmp(gdb_opt, "none"))
-		test_timer_pid = start_execution_timeout_timer(
-				max_exec_interval, &test_exec_pid);
-
-	test_stats = (struct test_stats *) l_queue_peek_tail(test_stats_queue);
-
-	while (true) {
-		pid_t corpse;
-		int status;
-		double interval;
-
-		corpse = waitpid(WAIT_ANY, &status, 0);
-
-		if (corpse < 0 || corpse == 0)
-			continue;
-
-		if (test_exec_pid == corpse) {
-			gettimeofday(&time_after, NULL);
-
-			if (test_timer_pid != -1)
-				kill_process(test_timer_pid);
-
-			timersub(&time_after, &time_before, &time_elapsed);
-			interval = time_elapsed.tv_sec +
-					1e-6 * time_elapsed.tv_usec;
-
-			if (WIFEXITED(status) &&
-					WEXITSTATUS(status) == EXIT_SUCCESS) {
-				print_test_status(py_test, TEST_STATUS_PASSED,
-							interval);
-				test_stats->num_passed++;
-			} else if (WIFSIGNALED(status)) {
-				print_test_status(py_test, TEST_STATUS_TIMEDOUT,
-								interval);
-				test_stats->num_timedout++;
-			} else {
-				print_test_status(py_test, TEST_STATUS_FAILED,
-								interval);
-				test_stats->num_failed++;
-			}
-
-			test_stats->py_run_time += interval;
-
-			break;
-		} else if (WIFSTOPPED(status))
-			l_info("Process %d stopped with signal %d", corpse,
-			       WSTOPSIG(status));
-		else if (WIFCONTINUED(status))
-			l_info("Process %d continued", corpse);
-	}
-
-	l_free(py_test);
-	py_test = NULL;
-
-	if (monitor_pid != -1) {
-		kill_process(monitor_pid);
-		monitor_pid = -1;
-	}
-
-	goto start_next_test;
-}
-
-static void set_config_cycle_info(const char *config_dir_path,
-					struct l_queue *test_stats_queue)
-{
-	char sep_line[80];
-	char *config_name_ptr;
-	struct test_stats *test_stats;
-
-	memset(sep_line, '_', sizeof(sep_line) - 1);
-	sep_line[sizeof(sep_line) - 1] = '\0';
-
-	config_name_ptr = strrchr(config_dir_path, '/');
-	config_name_ptr++;
-
-	l_info("%s", sep_line);
-	l_info(CONSOLE_LN_BOLD "Starting configuration cycle No: %d [%s]"
-		CONSOLE_LN_RESET, l_queue_length(test_stats_queue) + 1,
-							config_name_ptr);
-
-	test_stats = l_new(struct test_stats, 1);
-	test_stats->config_cycle_name = strdup(config_name_ptr);
-
-	l_queue_push_tail(test_stats_queue, test_stats);
-}
-
-static void set_wiphy_list(struct l_queue *wiphy_list)
-{
-	const struct l_queue_entry *wiphy_entry;
-	int size = 32;
-	char *var;
-
-	for (wiphy_entry = l_queue_get_entries(wiphy_list);
-				wiphy_entry; wiphy_entry = wiphy_entry->next) {
-		struct wiphy *wiphy = wiphy_entry->data;
-
-		size += 32 + strlen(wiphy->name);
-		if (wiphy->used_by_hostapd) {
-			size += 32 + strlen(wiphy->interface_name) +
-				strlen(wiphy->hostapd_ctrl_interface) +
-				strlen(wiphy->hostapd_config);
-		}
-	}
-
-	var = alloca(size);
-	size = 0;
-
-	for (wiphy_entry = l_queue_get_entries(wiphy_list);
-				wiphy_entry; wiphy_entry = wiphy_entry->next) {
-		struct wiphy *wiphy = wiphy_entry->data;
-
-		if (size)
-			var[size++] = '\n';
-
-		size += sprintf(var + size, "%s=", wiphy->name);
-
-		if (wiphy->used_by_hostapd)
-			size += sprintf(var + size,
-					"hostapd,name=%s,ctrl_interface=%s,"
-					"config=%s",
-					wiphy->interface_name,
-					wiphy->hostapd_ctrl_interface,
-					wiphy->hostapd_config);
-		else
-			size += sprintf(var + size, "iwd");
-	}
-
-	var[size++] = '\0';
-
-	setenv("TEST_WIPHY_LIST", var, true);
-}
-
-static void set_reg_domain(const char *domain)
-{
-	char *argv[5];
-
-	argv[0] = "iw";
-	argv[1] = "reg";
-	argv[2] = "set";
-	argv[3] = (char *) domain;
-	argv[4] = NULL;
-
-	execute_program(argv, environ, false, NULL);
-}
-
-static void wiphy_up(void *data, void *user_data)
-{
-	struct wiphy *wiphy = data;
-
-	set_interface_state(wiphy->interface_name, true);
-}
-
-static void wiphy_reset(void *data, void *user_data)
-{
-	struct wiphy *wiphy = data;
-
-	wiphy->used_by_hostapd = false;
-
-	l_free(wiphy->hostapd_config);
-	wiphy->hostapd_config = NULL;
-	l_free(wiphy->hostapd_ctrl_interface);
-	wiphy->hostapd_ctrl_interface = NULL;
-}
-
-static void create_network_and_run_tests(void *data, void *user_data)
-{
-	pid_t hostapd_pids[HWSIM_RADIOS_MAX];
-	pid_t iwd_pid = -1;
-	pid_t medium_pid = -1;
-	pid_t ofono_pid = -1;
-	pid_t phonesim_pid = -1;
-	char *config_dir_path;
-	char *iwd_config_dir;
-	char **tmpfs_extra_stuff = NULL;
-	struct l_settings *hw_settings;
-	struct l_queue *test_queue;
-	struct l_queue *test_stats_queue;
-	bool start_iwd_daemon = true;
-	bool needs_hwsim = false;
-	bool ofono_req = false;
-	const char *sim_keys;
-	const char *iwd_ext_options = NULL;
-	const char *reg_domain;
-	int phys_used;
-	int num_radios;
-	struct test_entry *entry = data;
-	char *test_name = NULL;
-
-	memset(hostapd_pids, -1, sizeof(hostapd_pids));
-
-	config_dir_path = (char *) entry->path;
-	test_queue = (struct l_queue *) entry->test_queue;
-	test_stats_queue = (struct l_queue *) user_data;
-
-	if (l_queue_isempty(test_queue)) {
-		l_error("No Python IWD tests have been found in %s",
-							config_dir_path);
-		return;
-	}
-
-	set_config_cycle_info(config_dir_path, test_stats_queue);
-
-	hw_settings = read_hw_config(config_dir_path);
-	if (!hw_settings)
-		return;
-
-	l_info("Configuring network...");
-
-	if (log) {
-		char *log_path;
-
-		test_name = basename(config_dir_path);
-		log_path = l_strdup_printf("%s/%s", log_dir, test_name);
-
-		mkdir(log_path, 0755);
-		if (chown(log_path, log_uid, log_gid) < 0)
-			l_error("chown failed");
-
-		l_free(log_path);
-	}
-
-	if (chdir(config_dir_path) < 0) {
-		l_error("Failed to change to test directory: %s",
-							strerror(errno));
-		goto free_hw_settings;
-	}
-
-	tmpfs_extra_stuff =
-		l_settings_get_string_list(hw_settings, HW_CONFIG_GROUP_SETUP,
-						HW_CONFIG_SETUP_TMPFS_EXTRAS,
-									':');
-
-	sim_keys = l_settings_get_value(hw_settings, HW_CONFIG_GROUP_SETUP,
-								"sim_keys");
-
-	if (sim_keys) {
-		if (!strcmp(sim_keys, "ofono")) {
-			bool ofono_found = false;
-			bool phonesim_found = false;
-
-			if (!system("which ofonod > /dev/null 2>&1"))
-				ofono_found = true;
-
-			if (!system("which phonesim > /dev/null 2>&1"))
-				phonesim_found = true;
-
-			if (!ofono_found || !phonesim_found) {
-				l_info("ofono or phonesim not found, skipping");
-				goto free_tmpfs_extra;
-			}
-
-			ofono_req = true;
-			iwd_ext_options = "--plugin=ofono";
-		} else {
-			setenv("IWD_SIM_KEYS", sim_keys, true);
-			iwd_ext_options = "--plugin=sim_hardcoded";
-		}
-	}
-
-	/* turn on/off timeouts if GDB is being used */
-	if (!strcmp(gdb_opt, "none"))
-		setenv("IWD_TEST_TIMEOUTS", "on", true);
-	else
-		setenv("IWD_TEST_TIMEOUTS", "off", true);
-
-	if (!create_tmpfs_extra_stuff(tmpfs_extra_stuff))
-		goto remove_abs_paths;
-
-	l_settings_get_int(hw_settings, HW_CONFIG_GROUP_SETUP,
-					HW_CONFIG_SETUP_NUM_RADIOS,
-					&num_radios);
-
-	if (!native_hw) {
-		reg_domain = l_settings_get_value(hw_settings,
-						HW_CONFIG_GROUP_SETUP,
-						HW_CONFIG_SETUP_REG_DOMAIN);
-		if (reg_domain)
-			set_reg_domain(reg_domain);
-
-		wiphy_list = l_queue_new();
-
-		if (!configure_hw_radios(hw_settings, wiphy_list))
-			goto remove_abs_paths;
-
-		medium_pid = register_hwsim_as_trans_medium();
-		if (medium_pid < 0)
-			goto remove_abs_paths;
-
-		if (check_verbosity("hwsim")) {
-			list_hwsim_radios();
-			list_interfaces();
-		}
-	} else {
-		int len;
-
-		l_settings_get_bool(hw_settings, HW_CONFIG_GROUP_SETUP,
-				HW_CONFIG_SETUP_NEEDS_HWSIM, &needs_hwsim);
-
-		/* Skip test that require hwsim dbus APIs (hwsim not running) */
-		if (needs_hwsim) {
-			l_error("test requires hwsim, skipping");
-			goto remove_abs_paths;
-		}
-
-		len = l_queue_length(wiphy_list);
-
-		/* Skip tests that need more radios than we have */
-		if (num_radios > len) {
-			l_error("test requires %d radios, only %d found",
-					num_radios, len);
-			goto remove_abs_paths;
-		}
-
-		l_queue_foreach(wiphy_list, wiphy_up, NULL);
-	}
-
-	if (check_verbosity("tls"))
-		setenv("IWD_TLS_DEBUG", "on", true);
-
-	if (!configure_hostapd_instances(hw_settings, config_dir_path,
-						wiphy_list, hostapd_pids,
-						&phys_used))
-		goto exit_hostapd;
-
-	l_settings_get_bool(hw_settings, HW_CONFIG_GROUP_SETUP,
-				HW_CONFIG_SETUP_START_IWD, &start_iwd_daemon);
-
-	if (start_iwd_daemon) {
-		/*
-		 * In native mode we may have more radios than a test actually
-		 * needs. This would result in IWD managing all phys that
-		 * hostapd wasn't using, which could throw off test results.
-		 * By passing the number of phys the test expects IWD to have
-		 * we can leave the remaining (unneeded) phys unmanaged.
-		 */
-		int iwd_phys = num_radios - phys_used;
-
-		iwd_config_dir =
-			l_settings_get_string(hw_settings,
-						HW_CONFIG_GROUP_SETUP,
-						HW_CONFIG_SETUP_IWD_CONF_DIR);
-		if (!iwd_config_dir)
-			iwd_config_dir = "/tmp";
-
-		iwd_pid = start_iwd(iwd_config_dir, wiphy_list,
-				iwd_ext_options, iwd_phys, test_name);
-
-		if (iwd_pid == -1)
-			goto exit_hostapd;
-	} else {
-		/* tells pytest to start iwd with valgrind */
-		if (valgrind)
-			setenv("IWD_TEST_VALGRIND", "on", true);
-	}
-
-	if (ofono_req) {
-		phonesim_pid = start_phonesim(test_name);
-		ofono_pid = start_ofono(test_name);
-	}
-
-	set_wiphy_list(wiphy_list);
-
-	if (!shell)
-		run_py_tests(hw_settings, test_queue, test_stats_queue,
-				test_name);
-	else {
-		if (system("/bin/sh"))
-			l_info("executing /bin/sh failed");
-	}
-
-	l_info("Destructing network...");
-
-	/* Script has responsibility to cleanup any iwd instances it started */
-	if (iwd_pid > 0)
-		terminate_iwd(iwd_pid);
-
-	/* /tmp/valgrind.log will only exist without logging turned on */
-	if (valgrind && !log) {
-		if (system("cat /tmp/valgrind.log"))
-			l_info("cat /tmp/valgrind.log failed");
-
-		if (system("echo \"\" > /tmp/valgrind.log"))
-			l_info("Failed to reset /tmp/valgrind.log");
-	}
-
-	if (log) {
-		L_AUTO_FREE_VAR(char *, dmesg);
-		L_AUTO_FREE_VAR(char *, kernel_log);
-
-		kernel_log = l_strdup_printf("%s/kernel.log", log_dir);
-		dmesg = l_strdup_printf("dmesg > %s", kernel_log);
-
-		if (system(dmesg))
-			l_error("dmesg failed");
-		if (chown(kernel_log, log_uid, log_gid))
-			l_error("chown failed");
-	}
-
-	if (ofono_req) {
-		loopback_started = false;
-		stop_ofono(ofono_pid);
-		stop_phonesim(phonesim_pid);
-	}
-
-exit_hostapd:
-	destroy_hostapd_instances(hostapd_pids);
-
-	if (!native_hw)
-		terminate_medium(medium_pid);
-
-remove_abs_paths:
-	remove_absolute_path_dirs(tmpfs_extra_stuff);
-
-	/*
-	 * If running in hwsim mode, we want to completely free/destroy the
-	 * wiphy list since it will be re-populated on the next test. For the
-	 * native case we want to reset the list as if it was freshly
-	 * discovered. This ensures that all the hostapd flags get reset.
-	 */
-	if (!native_hw)
-		l_queue_destroy(wiphy_list, wiphy_free);
-	else
-		l_queue_foreach(wiphy_list, wiphy_reset, NULL);
-
-free_tmpfs_extra:
-	l_strfreev(tmpfs_extra_stuff);
-free_hw_settings:
-	l_settings_free(hw_settings);
-}
-
-struct stat_totals {
-	unsigned int total_passed;
-	unsigned int total_failed;
-	unsigned int total_timedout;
-	double total_duration;
-};
-
-static void print_test_stat(void *data, void *user_data)
-{
-	struct test_stats *test_stats;
-	struct stat_totals *stat_totals;
-	char *str_runtime, *str_passed, *str_failed, *str_timedout;
-
-	test_stats = (struct test_stats *) data;
-	stat_totals = (struct stat_totals *) user_data;
-
-	stat_totals->total_duration	+= test_stats->py_run_time;
-	stat_totals->total_passed	+= test_stats->num_passed;
-	stat_totals->total_failed	+= test_stats->num_failed;
-	stat_totals->total_timedout	+= test_stats->num_timedout;
-
-	if (test_stats->py_run_time)
-		str_runtime = l_strdup_printf("| %9.3f sec",
-						test_stats->py_run_time);
-	else
-		str_runtime = l_strdup_printf("| %9s", "Skipped");
-
-	if (test_stats->num_passed)
-		str_passed = l_strdup_printf(" %6d ", test_stats->num_passed);
-	else
-		str_passed = l_strdup_printf(" %6c ", '-');
-
-	if (test_stats->num_failed)
-		str_failed = l_strdup_printf(" %6d ", test_stats->num_failed);
-	else
-		str_failed = l_strdup_printf(" %6c ", '-');
-
-	if (test_stats->num_timedout)
-		str_timedout = l_strdup_printf(" %9d ",
-						test_stats->num_timedout);
-	else
-		str_timedout = l_strdup_printf(" %9c ", '-');
-
-	l_info(CONSOLE_LN_BOLD "%27s "
-			CONSOLE_LN_DEFAULT "|" CONSOLE_LN_GREEN "%s"
-			CONSOLE_LN_DEFAULT "|" CONSOLE_LN_RED "%s"
-			CONSOLE_LN_DEFAULT "|" CONSOLE_LN_YELLOW "%s"
-			CONSOLE_LN_RESET "%s", test_stats->config_cycle_name,
-			str_passed, str_failed, str_timedout, str_runtime);
-
-	l_free(str_passed);
-	l_free(str_failed);
-	l_free(str_timedout);
-	l_free(str_runtime);
-}
-
-static void print_results(struct l_queue *test_stat_queue)
-{
-	struct stat_totals stat_totals = { 0, 0, 0, 0 };
-	char sep_line[80];
-
-	memset(sep_line, '_', sizeof(sep_line) - 1);
-	sep_line[sizeof(sep_line) - 1] = '\0';
-
-	l_info("%s\n" CONSOLE_LN_RESET, sep_line);
-	l_info("%27s " CONSOLE_LN_DEFAULT "|" CONSOLE_LN_GREEN " %s "
-		CONSOLE_LN_DEFAULT "|" CONSOLE_LN_RED " %5s "
-		CONSOLE_LN_DEFAULT "|" CONSOLE_LN_YELLOW " %9s "
-		CONSOLE_LN_RESET "| Duration",
-		"Configuration cycle", "PASSED", "FAILED", "TIMED OUT");
-
-	memset(sep_line, '-', sizeof(sep_line) - 1);
-	sep_line[sizeof(sep_line) - 1] = '\0';
-	l_info("%s" CONSOLE_LN_RESET, sep_line);
-
-	l_queue_foreach(test_stat_queue, print_test_stat, &stat_totals);
-
-	l_info("%s" CONSOLE_LN_RESET, sep_line);
-	l_info("%27s "
-		CONSOLE_LN_DEFAULT "|" CONSOLE_LN_GREEN " %6d "
-		CONSOLE_LN_DEFAULT "|" CONSOLE_LN_RED " %6d "
-		CONSOLE_LN_DEFAULT "|" CONSOLE_LN_YELLOW " %9d "
-		CONSOLE_LN_RESET "| %9.3f sec",
-		"Total", stat_totals.total_passed, stat_totals.total_failed,
-			stat_totals.total_timedout, stat_totals.total_duration);
-
-	memset(sep_line, '_', sizeof(sep_line) - 1);
-	sep_line[sizeof(sep_line) - 1] = '\0';
-	l_info("%s" CONSOLE_LN_RESET, sep_line);
-}
-
-static void test_stat_queue_entry_destroy(void *data)
-{
-	struct test_stats *ts;
-
-	ts = (struct test_stats *) data;
-
-	l_free(ts->config_cycle_name);
-	l_free(ts);
-}
-
-static void free_test_entry(void *data)
-{
-	struct test_entry *entry = data;
-
-	l_free(entry->path);
-	l_free(entry);
-}
-
-static void run_auto_tests(void)
-{
-	L_AUTO_FREE_VAR(char*, test_home_path) = NULL;
-	L_AUTO_FREE_VAR(char*, env_path) = NULL;
-	int i;
-	struct l_queue *test_config_queue;
-	struct l_queue *test_stat_queue;
-	char **test_config_dirs;
-
-	if (log) {
-		if (mount("logdir", log_dir, "9p", 0,
-					"trans=virtio,version=9p2000.L") < 0) {
-			l_error("Mounting %s failed", log_dir);
-			return;
-		}
-	}
-
-	env_path = l_strdup_printf("%s/src:%s/tools:%s", top_level_path,
-					top_level_path, getenv("PATH"));
-
-	setenv("PATH", env_path, true);
-
-	test_home_path = l_strdup_printf("%s/%s", top_level_path,
-						TEST_TOP_DIR_DEFAULT_NAME);
-
-	if (!path_exist(test_home_path)) {
-		l_error("Test directory %s does not exist", test_home_path);
-		return;
-	}
-
-	test_config_queue = l_queue_new();
-	if (!test_config_queue)
-		return;
-
-	test_config_dirs = l_strsplit(test_action_params, ',');
-
-	if (test_config_dirs[0]) {
-		i = 0;
-
-		while (test_config_dirs[i]) {
-			if (strchr(test_config_dirs[i], '/')) {
-				if (!find_test_configuration(
-							test_config_dirs[i], 1,
-							test_config_queue))
-					goto exit;
-			} else {
-				char *config_dir_path;
-
-				config_dir_path =
-					l_strdup_printf("%s/%s", test_home_path,
-							test_config_dirs[i]);
-
-				if (!find_test_configuration(config_dir_path, 1,
-							test_config_queue)) {
-					l_free(config_dir_path);
-
-					goto exit;
-				}
-
-				l_free(config_dir_path);
-			}
-
-			i++;
-		}
-	} else {
-		/*
-		 * --shell without any specific tests implies 'shell' test
-		 */
-		if (shell) {
-			char *config_dir_path;
-			config_dir_path = l_strdup_printf("%s/shell",
-								test_home_path);
-
-			if (!find_test_configuration(config_dir_path, 1,
-							test_config_queue)) {
-				l_free(config_dir_path);
-				goto exit;
-			}
-
-			l_free(config_dir_path);
-		} else {
-			l_info("Automatic test execution requested");
-			l_info("Searching for the test configurations...");
-
-			if (!find_test_configuration(test_home_path, 0,
-							test_config_queue))
-				goto exit;
-		}
-	}
-
-	if (l_queue_isempty(test_config_queue)) {
-		l_error("No test configuration discovered");
-		goto exit;
-	}
-
-	create_dbus_system_conf();
-
-	if (!start_dbus_daemon())
-		goto exit;
-
-	if (!start_haveged()) {
-		l_error("Failed to start haveged");
-		goto exit;
-	}
-
-	test_stat_queue = l_queue_new();
-
-	l_queue_foreach(test_config_queue, create_network_and_run_tests,
-							test_stat_queue);
-
-	print_results(test_stat_queue);
-
-	l_queue_destroy(test_stat_queue, test_stat_queue_entry_destroy);
-
-exit:
-	l_strfreev(verbose_apps);
-	l_strfreev(test_config_dirs);
-	l_queue_destroy(test_config_queue, free_test_entry);
-}
-
-static void run_unit_tests(void)
-{
-	DIR *d;
-	struct dirent *dirent;
-	char *argv[2];
-	char *unit_test_abs_path = NULL;
-	char **unit_tests = NULL;
-
-	if (strcmp(test_action_params, "")) {
-		unit_tests = l_strsplit(test_action_params, ',');
-
-		if (!unit_tests || !unit_tests[0])
-			goto exit;
-	}
-
-	if (chdir(top_level_path) < 0)
-		goto exit;
-
-	d = opendir("unit/");
-	if (!d)
-		goto exit;
-
-	while ((dirent = readdir(d)) != NULL) {
-		struct stat st;
-
-		if (dirent->d_type != DT_REG)
-			continue;
-
-		unit_test_abs_path = l_strdup_printf("%s%s%s", top_level_path,
-						"/unit/", dirent->d_name);
-
-		if (stat(unit_test_abs_path, &st) < 0)
-			goto next;
-
-		if (!(st.st_mode & S_IEXEC))
-			goto next;
-
-		if (unit_tests) {
-			if (!l_strv_contains(unit_tests, dirent->d_name))
-				goto next;
-		}
-
-		argv[0] = unit_test_abs_path;
-		argv[1] = NULL;
-
-		l_info("\n---------- Unit %s ----------", dirent->d_name);
-		execute_program(argv, environ, true, NULL);
-
-next:
-		l_free(unit_test_abs_path);
-	}
-
-	closedir(d);
-
-exit:
-	l_strfreev(unit_tests);
-}
-
-static bool wiphy_match(const void *a, const void *b)
-{
-	const struct wiphy *wiphy = a;
-	int id = L_PTR_TO_INT(b);
-
-	return (wiphy->id == id);
-}
-
-static struct wiphy *wiphy_find(int wiphy_id)
-{
-	return l_queue_find(wiphy_list, wiphy_match, L_INT_TO_PTR(wiphy_id));
-}
-
-static void parse_supported_iftypes(uint16_t *iftypes,
-						struct l_genl_attr *attr)
-{
-	uint16_t type, len;
-	const void *data;
-
-	while (l_genl_attr_next(attr, &type, &len, &data)) {
-		/*
-		 * NL80211_IFTYPE_UNSPECIFIED can be ignored, so we start
-		 * at the first bit
-		 */
-		if (type > sizeof(uint16_t) * 8) {
-			l_warn("unsupported iftype: %u", type);
-			continue;
-		}
-
-		*iftypes |= 1 << (type - 1);
-	}
-}
-
-static void wiphy_dump_callback(struct l_genl_msg *msg, void *user_data)
-{
-	struct wiphy *wiphy;
-	struct l_genl_attr attr;
-	struct l_genl_attr nested;
-	uint32_t id = UINT32_MAX;
-	uint16_t type, len;
-	const void *data;
-	const char *name = NULL;
-	uint32_t name_len = 0;
-	uint16_t iftypes = 0;
-
-	if (!l_genl_attr_init(&attr, msg))
-		return;
-
-	while (l_genl_attr_next(&attr, &type, &len, &data)) {
-		switch (type) {
-		case NL80211_ATTR_WIPHY:
-			if (len != sizeof(uint32_t))
-				return;
-
-			id = *((uint32_t *) data);
-
-			if (wiphy_find(id))
-				return;
-
-			break;
-		case NL80211_ATTR_WIPHY_NAME:
-			if (len > sizeof(((struct wiphy *) 0)->name))
-				return;
-
-			name = data;
-			name_len = len;
-
-			break;
-		case NL80211_ATTR_SUPPORTED_IFTYPES:
-			if (l_genl_attr_recurse(&attr, &nested))
-				parse_supported_iftypes(&iftypes, &nested);
-
-			break;
-		}
-	}
-
-	if (id == UINT32_MAX || !name)
-		return;
-
-	wiphy = l_new(struct wiphy, 1);
-	strncpy(wiphy->name, name, name_len);
-	wiphy->id = id;
-	wiphy->can_ap = iftypes & (1 << NL80211_IFTYPE_AP);
-
-	l_queue_push_tail(wiphy_list, wiphy);
-}
-
-static void iface_dump_callback(struct l_genl_msg *msg, void *user_data)
-{
-	struct l_genl_attr attr;
-	uint16_t type, len;
-	const void *data;
-	const char *ifname = NULL;
-	struct wiphy *wiphy = NULL;
-
-	if (!l_genl_attr_init(&attr, msg))
-		return;
-
-	while (l_genl_attr_next(&attr, &type, &len, &data)) {
-		switch (type) {
-
-		case NL80211_ATTR_IFNAME:
-			if (len > 16) {
-				l_warn("Invalid interface name attribute");
-				return;
-			}
-
-			ifname = data;
-			break;
-
-		case NL80211_ATTR_WIPHY:
-			if (len != sizeof(uint32_t)) {
-				l_warn("Invalid wiphy attribute");
-				return;
-			}
-
-			wiphy = wiphy_find(*((uint32_t *) data));
-			break;
-		}
-	}
-
-	if (!ifname || !wiphy)
-		return;
-
-	wiphy->interface_name = l_strdup(ifname);
-	wiphy->interface_created = false;
-
-	l_info("Discovered interface %s", wiphy->interface_name);
-}
-
-struct nl_data {
-	struct l_genl *genl;
-	struct l_genl_family *nl80211;
-};
-
-static void iface_dump_done(void *user_data)
-{
-	struct nl_data *data = user_data;
-
-	l_debug("Interface discovery complete, running tests");
-
-	list_interfaces();
-
-	run_auto_tests();
-
-	l_queue_destroy(wiphy_list, wiphy_free);
-
-	l_genl_family_free(data->nl80211);
-	l_genl_unref(data->genl);
-	l_free(data);
-
-	l_main_quit();
-}
-
-static void wiphy_dump_done(void *user_data)
-{
-	struct nl_data *data = user_data;
-	struct l_genl_msg *msg;
-
-	l_debug("Wiphy discovery complete, discovering interfaces");
-
-	msg = l_genl_msg_new(NL80211_CMD_GET_INTERFACE);
-	if (!l_genl_family_dump(data->nl80211, msg, iface_dump_callback,
-						data, iface_dump_done))
-		l_error("Getting all interface information failed");
-}
-
-static void nl80211_requested(const struct l_genl_family_info *info,
-							void *user_data)
-{
-	struct nl_data *data = user_data;
-	struct l_genl_msg *msg;
-
-	if (info == NULL) {
-		l_info("No nl80211 family found");
-		goto done;
-	}
-
-	l_debug("Found nl80211 interface");
-
-	data->nl80211 = l_genl_family_new(data->genl, NL80211_GENL_NAME);
-	wiphy_list = l_queue_new();
-
-	msg = l_genl_msg_new(NL80211_CMD_GET_WIPHY);
-	if (!l_genl_family_dump(data->nl80211, msg, wiphy_dump_callback,
-						data, wiphy_dump_done))
-		l_error("Getting all wiphy devices failed");
-
-	return;
-done:
-	l_main_quit();
-}
-
-static void start_hw_discovery(void)
-{
-	struct nl_data *data = l_new(struct nl_data, 1);
-
-	data->genl = l_genl_new();
-	l_genl_request_family(data->genl, NL80211_GENL_NAME,
-				nl80211_requested, data, NULL);
-	/*
-	 * This is somewhat of a mystery, but it appears that
-	 * calling lshw causes the OS to re-enumerate the USB
-	 * bus. Without this no USB adapters are found when
-	 * doing the wiphy/iface dump from nl80211.
-	 *
-	 * This also conveniently prints all the network
-	 * adapters and their iface name, so its much easier
-	 * to know which adapter are being used by iwd/hostapd
-	 * after the test.
-	 */
-	if (system("lshw -C network"))
-		l_info("lshw failed");
-
-	l_main_run();
-}
-
-static void run_tests(void)
-{
-	char cmdline[CMDLINE_MAX], *ptr, *cmds;
-	char *test_action_str;
-	FILE *fp;
-	int i;
-
-	fp = fopen("/proc/cmdline", "re");
-
-	if (!fp) {
-		l_error("Failed to open kernel command line");
-		return;
-	}
-
-	ptr = fgets(cmdline, sizeof(cmdline), fp);
-	fclose(fp);
-
-	if (!ptr) {
-		l_error("Failed to read kernel command line");
-		return;
-	}
-
-	ptr = strstr(cmdline, "LOG_GID=");
-	if (ptr) {
-		*ptr = '\0';
-		test_action_str = ptr + 9;
-		ptr = strchr(test_action_str, '\'');
-		*ptr = '\0';
-		log_gid = atoi(test_action_str);
-	}
-
-	ptr = strstr(cmdline, "LOG_UID=");
-	if (ptr) {
-		*ptr = '\0';
-		test_action_str = ptr + 9;
-		ptr = strchr(test_action_str, '\'');
-		*ptr = '\0';
-		log_uid = atoi(test_action_str);
-	}
-
-	ptr = strstr(cmdline, "LOG_PATH=");
-	if (ptr) {
-		*ptr = '\0';
-		test_action_str = ptr + 10;
-
-		ptr = strchr(test_action_str, '\'');
-		*ptr = '\0';
-
-		if (strcmp(test_action_str, "none")) {
-			log = true;
-			strcpy(log_dir, test_action_str);
-		}
-	}
-
-	ptr = strstr(cmdline, "SHELL=");
-	if (ptr) {
-		*ptr = '\0';
-		test_action_str = ptr + 6;
-
-		shell = atoi(test_action_str);
-	}
-
-	ptr = strstr(cmdline, "HW=");
-	if (ptr) {
-		*ptr = '\0';
-		test_action_str = ptr + 4;
-
-		ptr = strchr(test_action_str, '\'');
-		if (ptr)
-			*ptr = '\0';
-
-		if (!strcmp(test_action_str, "virtual"))
-			native_hw = false;
-		else
-			native_hw = true;
-	}
-
-	ptr = strstr(cmdline, "GDB=");
-	if (ptr) {
-		*ptr = '\0';
-		test_action_str = ptr + 5;
-
-		ptr = strchr(test_action_str, '\'');
-		*ptr = '\0';
-		gdb_opt = l_strdup(test_action_str);
-	}
-
-	ptr = strstr(cmdline, "VALGRIND=");
-	if (ptr) {
-		char *end;
-		unsigned long v;
-
-		*ptr = '\0';
-		test_action_str = ptr + 9;
-		v = strtoul(test_action_str, &end, 10);
-		if ((v != 0 && v != 1) || end != test_action_str + 1) {
-			l_error("malformed valgrind option");
-			return;
-		}
-
-		valgrind = (bool) v;
-	}
-
-	ptr = strstr(cmdline, "PATH=");
-	if (!ptr) {
-		l_error("No $PATH section found");
-		return;
-	}
-
-	*ptr = '\0';
-	test_action_str = ptr + 6;
-	ptr = strchr(test_action_str, '\'');
-	*ptr = '\0';
-	l_info("%s", test_action_str);
-	setenv("PATH", test_action_str, true);
-
-	ptr = strstr(cmdline, "TESTARGS=");
-
-	if (!ptr) {
-		l_error("No test command section found");
-		return;
-	}
-
-	cmds = ptr + 10;
-	ptr = strchr(cmds, '\'');
-
-	if (!ptr) {
-		l_error("Malformed test command section");
-		return;
-	}
-
-	*ptr = '\0';
-
-	ptr = strstr(cmdline, "TEST_ACTION_PARAMS=");
-
-	if (ptr) {
-		test_action_params = ptr + 20;
-		ptr = strchr(test_action_params, '\'');
-
-		if (!ptr) {
-			l_error("Malformed test action parameters section");
-			return;
-		}
-
-		*ptr = '\0';
-	}
-
-	ptr = strstr(cmdline, "TEST_ACTION=");
-
-	if (ptr) {
-		test_action_str = ptr + 12;
-		ptr = strchr(test_action_str, ' ');
-
-		if (!ptr) {
-			l_error("Malformed test action parameters section");
-			return;
-		}
-
-		*ptr = '\0';
-
-		test_action = (enum action) atoi(test_action_str);
-	}
-
-	ptr = strstr(cmdline, "DEBUG_FILTER=");
-
-	if (ptr) {
-		debug_filter = ptr + 14;
-
-		ptr = strchr(debug_filter, '\'');
-
-		if (!ptr) {
-			l_error("Malformed debug filter section");
-			return;
-		}
-
-		*ptr = '\0';
-
-		if (debug_filter[0] != '\0') {
-			enable_debug = true;
-			l_debug_enable(debug_filter);
-			setenv("HWSIM_DEBUG", "", true);
-		}
-	}
-
-	ptr = strstr(cmdline, "TESTVERBOUT=");
-
-	if (ptr) {
-		verbose_opt = ptr + 13;
-
-		ptr = strchr(verbose_opt, '\'');
-		if (!ptr) {
-			l_error("Malformed verbose parameter");
-			return;
-		}
-
-		*ptr = '\0';
-
-		l_info("Enable verbose output for %s", verbose_opt);
-
-		verbose_apps = l_strsplit(verbose_opt, ',');
-	}
-
-	ptr = strstr(cmdline, "TESTHOME=");
-
-	if (ptr) {
-		exec_home = ptr + 4;
-		ptr = strpbrk(exec_home + 9, " \r\n");
-
-		if (ptr)
-			*ptr = '\0';
-	}
-
-	ptr = strrchr(exec_home, '/');
-
-	if (!ptr)
-		exit(EXIT_FAILURE);
-
-	i = ptr - exec_home;
-
-	strncpy(top_level_path, exec_home + 5, i - 5);
-	top_level_path[i - 5] = '\0';
-
-	switch (test_action) {
-	case ACTION_AUTO_TEST:
-		if (native_hw)
-			start_hw_discovery();
-		else
-			run_auto_tests();
-		break;
-	case ACTION_UNIT_TEST:
-		run_unit_tests();
-		break;
-	}
-}
-
-static void usage(void)
-{
-	l_info("test-runner - Automated test execution utility\n"
-		"Usage:\n");
-	l_info("\ttest-runner [options] [--] <command> [args]\n");
-	l_info("Options:\n"
-		"\t-q, --qemu <path>	QEMU binary\n"
-		"\t-k, --kernel <image>	Kernel image (bzImage)\n"
-		"\t-v, --verbose <apps>	Comma separated list of "
-						"applications to enable\n"
-						"\t\t\t\tverbose output\n"
-		"\t-h, --help		Show help options\n"
-		"\t-V, --valgrind		Run valgrind on iwd. Note: \"-v"
-						" iwd\" is required\n"
-						"\t\t\t\tto see valgrind"
-						" output\n"
-		"\t-g, --gdb <iwd|hostapd>	Run gdb on the specified"
-						" executable\n"
-		"\t-w, --hw <config>	Run using a physical hardware "
-					"configuration\n"
-		"\t-s, --shell		Boot into shell. If -A is used the"
-					" environment\n"
-					"\t\t\t\twill be setup exactly as it is"
-					" in the test,\n"
-					"\t\t\t\tbut no test will be run. If no"
-					" test is specified\n"
-					"\t\t\t\tthe 'shell' test"
-					" will be used\n"
-		"\t-l, --log <dir>		Directory used for log output. "
-					"This option sets \n"
-					"\t\t\t\t--verbose on all apps");
-	l_info("Commands:\n"
-		"\t-A, --auto-tests <dirs>	Comma separated list of the "
-						"test configuration\n\t\t\t\t"
-						"directories to run\n"
-		"\t-U, --unit-tests <tests>	Comma separated list of the "
-						"unit tests to run\n");
-}
-
-static const struct option main_options[] = {
-	{ "auto-tests",	required_argument, NULL, 'A' },
-	{ "unit-tests",	optional_argument, NULL, 'U' },
-	{ "qemu",	required_argument, NULL, 'q' },
-	{ "kernel",	required_argument, NULL, 'k' },
-	{ "verbose",	required_argument, NULL, 'v' },
-	{ "debug",	optional_argument, NULL, 'd' },
-	{ "gdb",	required_argument, NULL, 'g' },
-	{ "valgrind",	no_argument,       NULL, 'V' },
-	{ "hw",		required_argument, NULL, 'w' },
-	{ "shell",	optional_argument, NULL, 's' },
-	{ "log",	required_argument, NULL, 'l' },
-	{ "help",	no_argument,       NULL, 'h' },
-	{ }
-};
-
-int main(int argc, char *argv[])
-{
-	uint8_t actions = 0;
-	struct tm *timeinfo;
-	time_t t;
-	char *gid;
-	char *uid;
-
-	l_log_set_stderr();
-
-	if (getpid() == 1 && getppid() == 0) {
-		if (!l_main_init())
-			return EXIT_FAILURE;
-
-		prepare_sandbox();
-
-		run_tests();
-
-		sync();
-		l_info("Done running tests. Rebooting...");
-
-		reboot(RB_AUTOBOOT);
-		return EXIT_SUCCESS;
-	}
-
-	for (;;) {
-		int opt;
-
-		opt = getopt_long(argc, argv, "A:q:k:v:g:sl:UVdh", main_options,
-									NULL);
-		if (opt < 0)
-			break;
-
-		switch (opt) {
-		case 'A':
-			test_action = ACTION_AUTO_TEST;
-			test_action_params = optarg;
-			actions++;
-			break;
-		case 'U':
-			test_action = ACTION_UNIT_TEST;
-			test_action_params = optarg;
-			actions++;
-			break;
-		case 'q':
-			qemu_binary = optarg;
-			break;
-		case 'k':
-			kernel_image = optarg;
-			break;
-		case 'd':
-			enable_debug = true;
-
-			if (optarg)
-				debug_filter = optarg;
-			else
-				debug_filter = "*";
-
-			l_debug_enable(debug_filter);
-			break;
-		case 'v':
-			verbose_opt = optarg;
-			verbose_apps = l_strsplit(optarg, ',');
-			break;
-		case 'V':
-			valgrind = true;
-			break;
-		case 'g':
-			gdb_opt = optarg;
-			if (!gdb_opt || (strcmp(gdb_opt, "iwd") &&
-					strcmp(gdb_opt, "hostapd") &&
-					strcmp(gdb_opt, "hwsim"))) {
-				l_error("--gdb can only be used with iwd"
-					", hwsim or hostapd");
-				return EXIT_FAILURE;
-			}
-			break;
-		case 'w':
-			hw_config = l_settings_new();
-			if (!l_settings_load_from_file(hw_config, optarg)) {
-				l_error("could not read hw config from %s",
-						optarg);
-				l_settings_free(hw_config);
-				return EXIT_FAILURE;
-			}
-			break;
-		case 's':
-			shell = true;
-			break;
-		case 'l':
-			/*
-			 * Setup the log directory. This is created under the
-			 * passed in log dir (--log) in the format:
-			 * <logdir>/run-<year>-<month>-<day>-<PID>
-			 *
-			 * The created log dir is then chown'ed to the user
-			 * who started test-runner, as are all files created
-			 * under this directory.
-			 */
-			log = true;
-
-			if (!optarg)
-				optarg = ".";
-
-			time(&t);
-			timeinfo = localtime(&t);
-
-			gid = getenv("SUDO_GID");
-			uid = getenv("SUDO_UID");
-
-			if (!gid || !uid) {
-				log_gid = getgid();
-				log_uid = getuid();
-			} else {
-				log_gid = strtol(gid, NULL, 10);
-				log_uid = strtol(uid, NULL, 10);
-			}
-
-			snprintf(log_dir, sizeof(log_dir), "%s/run-%d-%d-%d-%d",
-					optarg, timeinfo->tm_year + 1900,
-					timeinfo->tm_mon + 1, timeinfo->tm_mday,
-					getpid());
-			mkdir(log_dir, 0755);
-
-			if (chown(log_dir, log_uid, log_gid) < 0)
-				l_error("failed to fchown %s", log_dir);
-
-			break;
-		case 'h':
-			usage();
-			return EXIT_SUCCESS;
-		default:
-			return EXIT_FAILURE;
-		}
-	}
-
-	if (argc - optind > 0) {
-		l_error("Invalid command line parameters");
-		return EXIT_FAILURE;
-	}
-
-	if (actions > 1) {
-		l_error("Only one action can be specified");
-		return EXIT_FAILURE;
-	}
-
-	if (!actions)
-		test_action = ACTION_AUTO_TEST;
-
-	own_binary = argv[0];
-	test_argv = argv + optind;
-	test_argc = argc - optind;
-
-	if (!qemu_binary) {
-		qemu_binary = find_qemu();
-		if (!qemu_binary) {
-			l_error("No default QEMU binary found");
-			return EXIT_FAILURE;
-		}
-	}
-
-	if (!kernel_image) {
-		kernel_image = find_kernel();
-		if (!kernel_image) {
-			l_error("No default kernel image found");
-			return EXIT_FAILURE;
-		}
-	}
-
-	l_info("Using QEMU binary %s", qemu_binary);
-	l_info("Using kernel image %s", kernel_image);
-
-	if (!start_qemu())
-		return EXIT_FAILURE;
-
-	return EXIT_SUCCESS;
-}
-- 
2.21.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [PATCH 16/16] doc: update test runner docs
  2020-08-27 17:32 [PATCH 01/16] hwsim: return radio ID on create James Prestwood
                   ` (13 preceding siblings ...)
  2020-08-27 17:32 ` [PATCH 15/16] tools: post test-runner rewrite cleanup James Prestwood
@ 2020-08-27 17:32 ` James Prestwood
  2020-08-27 20:08 ` [PATCH 01/16] hwsim: return radio ID on create Denis Kenzior
  15 siblings, 0 replies; 18+ messages in thread
From: James Prestwood @ 2020-08-27 17:32 UTC (permalink / raw)
  To: iwd

[-- Attachment #1: Type: text/plain, Size: 7245 bytes --]

---
 doc/test-runner.txt | 117 +++++++++++++-------------------------------
 1 file changed, 34 insertions(+), 83 deletions(-)

diff --git a/doc/test-runner.txt b/doc/test-runner.txt
index 2d45bedc..17d7bbb2 100644
--- a/doc/test-runner.txt
+++ b/doc/test-runner.txt
@@ -17,18 +17,21 @@ OS:
 
 	Name:			Tested ver.:
 	 1. qemu		2.4.1
-	 2. Linux kernel	4.20
+	 2. Linux kernel	4.20+
 	 3. dbus-daemon		1.11.18
 	 4. ifconfig		2.10-alpha
 	 5. iw			3.17
 	 6. python		2.7
 	 7. haveged		no ver. avail.
-	 8. hostapd		commit id: 31d3692
+	 8. hostapd		recent
 	 9. <iwd>/tools/hwsim	0.0
 	10. <iwd>/src/iwd	0.0
+	11. ofono (optional)
+	12. phonesim (optional)
 
-Note: You will need ell-key-crypto branch, not the master branch from the tree
-in step 2 above.
+Note: The version for hostapd is listed as 'recent'. Older hostapd versions
+will work but we are continually adopting new features from hostapd and using
+those to test. Its recommended to use a recent release of hostapd.
 
 Note: The test-runner mounts host's file system in readonly mode and executes
 the above binaries inside of an emulated environment directly from it.
@@ -100,7 +103,6 @@ By default the test-runner will search for the kernel image in these locations:
 An arbitrary kernel image location can be specified by using '--kernel <path>'
 parameter into test-runner.
 
-
 Running Automated Tests
 =======================
 Before running any tests, its expected that the folder /var/lib/iwd exists on
@@ -113,8 +115,8 @@ By default, the automated test configuration directories reside in
 '<iwd>/autotests' and have a mandatory prefix of 'test'.
 
 	<iwd>/autotests/test1
-                       /test2
-                         ...
+			/test2
+			...
 
 The test configurations along with test cases in <iwd>/autotests/test*
 directories will be discovered and executed by test-runner in sequential
@@ -133,6 +135,10 @@ The command line may look as follows:
 
 	$ sudo ./test-runner -t test1,test3,/home/test4
 
+Glob matching is also supported when specifying a test list:
+
+	$ sudo ./test-runner -t testWPA*
+
 
 Creating Test Configurations
 ============================
@@ -145,7 +151,11 @@ A typical test configuration directory may consist of these types of files:
 	*Test or *Test.py	The set of test cases for IWD functionality
 				implemented using Python scripting language.
 				These files must have one of the two predefined
-				suffixes: 'Test' or 'Test.py'
+				suffixes: 'test' or 'test.py'. The file name
+				must also be more descriptive than simply 'test'.
+				e.g. 'connection_test' or 'failure_test'. Files
+				named 'test' will not be dynamically loaded due
+				to python module naming overlap.
 
 	*.conf			A configuration file for an instance of hostapd
 	(Defined in hw.conf)	service.
@@ -164,6 +174,9 @@ A typical contents of a test configuration directory may look as follows:
 		networkScanTest
 		networkConnectTest.py
 
+Note: Any additional files in your test directory will be copied to /tmp inside
+the VM. These additional file should be referenced using /tmp/<file> inside
+any configuration file for hostapd.
 
 Defining Network
 ----------------
@@ -187,61 +200,26 @@ supplicant running IWD:
 # has a range of [1, 100].
 num_radios=3
 
-# Definition of the radio configuration identifiers. These identifiers are used
-# to map the APs and IWD to the radios with the particular hardware properties.
-# This field is optional. If identifier is omitted from this list, then the
-# default radio properties will be used as they are defined in mac80211_hwsim
-# driver to satisfy a total number of radios requested in 'num_radios' field.
-radio_confs=rad0:rad1
-
 # Maximum execution interval per Python script file in seconds. This field is
 # optional.
 # Default: 20 seconds.
 #max_test_exec_interval_sec=5
 
-# List of paths inside of a test configuration directory for which
-# the symlinks will be created inside of /tmp. Such paths can be used
-# to specify an absolute path to the needed files inside of IWD and Hostapd
-# configuration files.
-# Example:
-#	 <some path>/test1/certs
-#		           misc
-#
-# certs and misc directories will be respectively mapped to:
-#
-#	 /tmp/certs
-#	      misc
-#
-# This field is optional.
-#tmpfs_extra_stuff=certs:misc
-
-# Flag to prevent test-runner from starting IWD. Therefore, it may later be
-# started from the python test cases.
-# This field is optional. Default: 1 (true)
-#start_iwd=0
-
-# Configuration directory to use for IWD daemon. IWD expects 'main.conf' to be
-# inside of the specified directory.
-# This field is optional. Default: /etc/iwd
-#iwd_config_dir=/etc/iwd
-
 #
 # The following two configuration groups are examples of the radio
 # configurations.
 #
 # This group of settings allows to specify a set of properties for a radio. The
-# name of the group represents a radio identifier and must be predefined in
-# 'radio_confs' field inside of 'SETUP' group. This configuration group is
-# optional.
-# TODO: explain each one of the properties.
+# name of the group represents a radio identifier. These configuration groups
+# are optional.
 [rad0]
-channels=2
-p2p_device=1
-use_chanctx=1
 
-# Properties of the second radio. This configuration group is optional.
-[rad1]
-p2p=0
+# For each radX group you can specify radio properties:
+# Disables the provided interface type(s):
+disable_iftype=ap,p2p
+
+# Disables the provided cipher types:
+disable_cipher=ccmp,bip
 
 # 'HOSTAPD' configuration group identifies a set of access points (AP) for the
 # current network topology. Each key/value pair represents a single AP that is
@@ -253,8 +231,12 @@ p2p=0
 [HOSTAPD]
 rad0=ap1.conf
 rad1=ap2.conf
-#~~~~~~~~~~~~~~~~~~ end of hw.conf ~~~~~~~~~~~~~~~~~~~~~~~~~
 
+# 'radius_server' is a special option used to specify a hostapd instance which
+# acts as a radius server only (not an access point). This option does not
+# require an additional 'num_radios' to be used.
+radius_server=radius.conf
+#~~~~~~~~~~~~~~~~~~ end of hw.conf ~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Configuring Access Points
 -------------------------
@@ -358,37 +340,6 @@ For more information on Python unit test framework refer to the following page:
 
 	http://pyunit.sourceforge.net/pyunit.html
 
-
-Examples of the framework usage:
-
-#~~~~~~~~~~~~~~~~~~~~~~~~~ alwaysPassingTest.py ~~~~~~~~~~~~~~~~~~~~~~~~~
-#!/usr/bin/python
-
-import unittest
-
-class TestPassingCase(unittest.TestCase):
-
-	def test_pass(self):
-		self.assertTrue(True)
-
-if __name__ == '__main__':
-	unittest.main()
-#~~~~~~~~~~~~~~~~~~ end of alwaysPassingTest.py ~~~~~~~~~~~~~~~~~~~~~~~~~
-
-#~~~~~~~~~~~~~~~~~~~~~~~~~ alwaysFailingTest.py ~~~~~~~~~~~~~~~~~~~~~~~~~
-#!/usr/bin/python
-
-import unittest
-
-class TestFailingCase(unittest.TestCase):
-
-	def test_fail(self):
-		self.assertTrue(False)
-
-if __name__ == '__main__':
-	unittest.main()
-#~~~~~~~~~~~~~~~~~~ end of alwaysFailingTest.py ~~~~~~~~~~~~~~~~~~~~~~~~~
-
 Using hardware passthrough
 ---------------------------
 
-- 
2.21.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* Re: [PATCH 01/16] hwsim: return radio ID on create
  2020-08-27 17:32 [PATCH 01/16] hwsim: return radio ID on create James Prestwood
                   ` (14 preceding siblings ...)
  2020-08-27 17:32 ` [PATCH 16/16] doc: update test runner docs James Prestwood
@ 2020-08-27 20:08 ` Denis Kenzior
  2020-08-27 20:21   ` James Prestwood
  15 siblings, 1 reply; 18+ messages in thread
From: Denis Kenzior @ 2020-08-27 20:08 UTC (permalink / raw)
  To: iwd

[-- Attachment #1: Type: text/plain, Size: 964 bytes --]

Hi James,

On 8/27/20 12:32 PM, James Prestwood wrote:
> Hwsim was relying on some internal behavior of mac80211_hwsim where new
> radios are created with ID's starting at zero and incremented. While
> this is a reasonable assumption its better to obtain the actual radio
> ID since mac80211_hwsim returns it in the radio creation callback.
> 
> Now hwsim will return the radio ID when creating radios. Negative returns
> are still errors, but any value >= 0 indicates the radio ID of the newly
> created radio.

Ah, now I see what you meant by exit code.  Sorry I should have caught this earlier.

So the problem is that UNIX doesn't really understand negative exit codes.  exit 
code of 0 is a success and anything else is a failure.

I think you will get people screaming obscenities if they try to use hwsim in a 
script...

> ---
>   tools/hwsim.c | 9 +++++++--
>   1 file changed, 7 insertions(+), 2 deletions(-)
> 

Regards,
-Denis

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [PATCH 01/16] hwsim: return radio ID on create
  2020-08-27 20:08 ` [PATCH 01/16] hwsim: return radio ID on create Denis Kenzior
@ 2020-08-27 20:21   ` James Prestwood
  0 siblings, 0 replies; 18+ messages in thread
From: James Prestwood @ 2020-08-27 20:21 UTC (permalink / raw)
  To: iwd

[-- Attachment #1: Type: text/plain, Size: 1208 bytes --]

On Thu, 2020-08-27 at 15:08 -0500, Denis Kenzior wrote:
> Hi James,
> 
> On 8/27/20 12:32 PM, James Prestwood wrote:
> > Hwsim was relying on some internal behavior of mac80211_hwsim where
> > new
> > radios are created with ID's starting at zero and incremented.
> > While
> > this is a reasonable assumption its better to obtain the actual
> > radio
> > ID since mac80211_hwsim returns it in the radio creation callback.
> > 
> > Now hwsim will return the radio ID when creating radios. Negative
> > returns
> > are still errors, but any value >= 0 indicates the radio ID of the
> > newly
> > created radio.
> 
> Ah, now I see what you meant by exit code.  Sorry I should have
> caught this earlier.
> 
> So the problem is that UNIX doesn't really understand negative exit
> codes.  exit 
> code of 0 is a success and anything else is a failure.
> 
> I think you will get people screaming obscenities if they try to use
> hwsim in a 
> script...

Ok, makes sense. I can just track the ID internally until I get the
dbus stuff sorted out.

> 
> > ---
> >   tools/hwsim.c | 9 +++++++--
> >   1 file changed, 7 insertions(+), 2 deletions(-)
> > 
> 
> Regards,
> -Denis

^ permalink raw reply	[flat|nested] 18+ messages in thread

end of thread, other threads:[~2020-08-27 20:21 UTC | newest]

Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-08-27 17:32 [PATCH 01/16] hwsim: return radio ID on create James Prestwood
2020-08-27 17:32 ` [PATCH 02/16] auto-t: prepare autotests for test-runner re-write James Prestwood
2020-08-27 17:32 ` [PATCH 03/16] auto-t: introduce pure python " James Prestwood
2020-08-27 17:32 ` [PATCH 04/16] auto-t: hostapd.py: update to work with test-runner rewrite James Prestwood
2020-08-27 17:32 ` [PATCH 05/16] auto-t: testutil.py: " James Prestwood
2020-08-27 17:32 ` [PATCH 06/16] auto-t: ofono.py: fix timeout cleanup and wait for service James Prestwood
2020-08-27 17:32 ` [PATCH 07/16] auto-t: iwd.py: update to work with test-runner rewrite James Prestwood
2020-08-27 17:32 ` [PATCH 08/16] auto-t: iwd.py: fix multiple timeout cleanup issues James Prestwood
2020-08-27 17:32 ` [PATCH 09/16] auto-t: remove wiphy.py James Prestwood
2020-08-27 17:32 ` [PATCH 10/16] auto-t: fix hidden network test James Prestwood
2020-08-27 17:32 ` [PATCH 11/16] auto-t: fix testSAE autoconnect_test.py James Prestwood
2020-08-27 17:32 ` [PATCH 12/16] auto-t: skip ofono tests if ofonod isn't running James Prestwood
2020-08-27 17:32 ` [PATCH 13/16] auto-t: replace hard-coded interfaces James Prestwood
2020-08-27 17:32 ` [PATCH 14/16] auto-t: remove device.wait_for_connected James Prestwood
2020-08-27 17:32 ` [PATCH 15/16] tools: post test-runner rewrite cleanup James Prestwood
2020-08-27 17:32 ` [PATCH 16/16] doc: update test runner docs James Prestwood
2020-08-27 20:08 ` [PATCH 01/16] hwsim: return radio ID on create Denis Kenzior
2020-08-27 20:21   ` James Prestwood

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.